2024-03-24T10:18:30.124 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T10:18:30.124 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} --verbose -- /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048/orig.config.yaml 2024-03-24T10:18:30.182 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T10:18:30.278 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048 branch: wip-batrick-testing-20240323.003144-squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} email: pdonnell@redhat.com first_in_suite: false job_id: '7619048' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240323.003144-squid ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 21b3bd672163092b4e542f772797fcba02d3d4de subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 21b3bd672163092b4e542f772797fcba02d3d4de kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240323.003144-squid sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4284 sha1: 21b3bd672163092b4e542f772797fcba02d3d4de sleep_before_teardown: 0 subset: 405/512 suite: fs suite_branch: wip-batrick-testing-20240323.003144-squid suite_path: /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi023.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFz//K9/uxvwtd/dn2kIuJgguQItPWZvjS/bTfHZLeUy8O0lrTfHSbPY96xhGanB6CHhZvJhDHfwl2LzBT84ZwM= smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMVMb3emcxPjzlzZK0ROwX1kMluUy7TScFma5MqLnZq4clQHyRD9Bs1LHx7vkYOSezWsaadnMz5l3egWLF/kA8w= smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN9q4Z7n2upgxsNmEi6Qmkm6Ii9ePAVs2Lqax+P+SdVfRXfKwUHf4Zj99lMAhPCbVZPu5aiDeWLM7VkW4PiOIco= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-24_04:56:01 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T10:18:30.278 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa; will attempt to use it 2024-03-24T10:18:30.279 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks 2024-03-24T10:18:30.279 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T10:18:30.280 INFO:teuthology.task.internal:Checking packages... 2024-03-24T10:18:30.305 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '21b3bd672163092b4e542f772797fcba02d3d4de' 2024-03-24T10:18:30.305 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T10:18:30.305 INFO:teuthology.packaging:ref: None 2024-03-24T10:18:30.305 INFO:teuthology.packaging:tag: None 2024-03-24T10:18:30.305 INFO:teuthology.packaging:branch: wip-batrick-testing-20240323.003144-squid 2024-03-24T10:18:30.305 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:18:30.305 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240323.003144-squid 2024-03-24T10:18:30.523 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1855.g21b3bd67 2024-03-24T10:18:30.524 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T10:18:30.531 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T10:18:30.531 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T10:18:30.536 INFO:teuthology.task.internal:Saving configuration 2024-03-24T10:18:30.558 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T10:18:30.564 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T10:18:30.589 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:12:26.468685', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFz//K9/uxvwtd/dn2kIuJgguQItPWZvjS/bTfHZLeUy8O0lrTfHSbPY96xhGanB6CHhZvJhDHfwl2LzBT84ZwM='} 2024-03-24T10:18:30.609 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:12:26.471945', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMVMb3emcxPjzlzZK0ROwX1kMluUy7TScFma5MqLnZq4clQHyRD9Bs1LHx7vkYOSezWsaadnMz5l3egWLF/kA8w='} 2024-03-24T10:18:30.628 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:12:26.470526', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN9q4Z7n2upgxsNmEi6Qmkm6Ii9ePAVs2Lqax+P+SdVfRXfKwUHf4Zj99lMAhPCbVZPu5aiDeWLM7VkW4PiOIco='} 2024-03-24T10:18:30.628 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T10:18:30.634 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-24T10:18:30.634 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-24T10:18:30.634 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-24T10:18:30.634 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T10:18:30.729 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8a15ef6160>, signals=[15]) 2024-03-24T10:18:30.729 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T10:18:30.736 INFO:teuthology.task.internal:Opening connections... 2024-03-24T10:18:30.736 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T10:18:30.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:18:30.846 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T10:18:30.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:18:30.934 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-03-24T10:18:30.934 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:18:31.023 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T10:18:31.031 DEBUG:teuthology.orchestra.run.smithi023:> uname -m 2024-03-24T10:18:31.049 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2024-03-24T10:18:31.049 DEBUG:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2024-03-24T10:18:31.107 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="CentOS Stream" 2024-03-24T10:18:31.107 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="9" 2024-03-24T10:18:31.107 INFO:teuthology.orchestra.run.smithi023.stdout:ID="centos" 2024-03-24T10:18:31.107 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE="rhel fedora" 2024-03-24T10:18:31.107 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="9" 2024-03-24T10:18:31.107 INFO:teuthology.orchestra.run.smithi023.stdout:PLATFORM_ID="platform:el9" 2024-03-24T10:18:31.107 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T10:18:31.107 INFO:teuthology.orchestra.run.smithi023.stdout:ANSI_COLOR="0;31" 2024-03-24T10:18:31.108 INFO:teuthology.orchestra.run.smithi023.stdout:LOGO="fedora-logo-icon" 2024-03-24T10:18:31.108 INFO:teuthology.orchestra.run.smithi023.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T10:18:31.108 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://centos.org/" 2024-03-24T10:18:31.108 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T10:18:31.108 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T10:18:31.108 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T10:18:31.109 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2024-03-24T10:18:31.131 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-03-24T10:18:31.149 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-03-24T10:18:31.149 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-03-24T10:18:31.206 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-03-24T10:18:31.206 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-03-24T10:18:31.206 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-03-24T10:18:31.206 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-03-24T10:18:31.206 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-03-24T10:18:31.206 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-03-24T10:18:31.206 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T10:18:31.206 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-03-24T10:18:31.207 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-03-24T10:18:31.207 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T10:18:31.207 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-03-24T10:18:31.207 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T10:18:31.207 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T10:18:31.207 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T10:18:31.209 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-03-24T10:18:31.231 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-03-24T10:18:31.248 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-03-24T10:18:31.248 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-03-24T10:18:31.305 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream" 2024-03-24T10:18:31.306 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="9" 2024-03-24T10:18:31.306 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos" 2024-03-24T10:18:31.306 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora" 2024-03-24T10:18:31.306 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="9" 2024-03-24T10:18:31.306 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el9" 2024-03-24T10:18:31.306 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T10:18:31.307 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31" 2024-03-24T10:18:31.307 INFO:teuthology.orchestra.run.smithi117.stdout:LOGO="fedora-logo-icon" 2024-03-24T10:18:31.307 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T10:18:31.307 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/" 2024-03-24T10:18:31.307 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T10:18:31.308 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T10:18:31.308 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T10:18:31.308 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-03-24T10:18:31.339 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T10:18:31.349 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T10:18:31.354 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T10:18:31.354 DEBUG:teuthology.orchestra.run.smithi023:> test '!' -e /home/ubuntu/cephtest 2024-03-24T10:18:31.356 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-03-24T10:18:31.359 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-03-24T10:18:31.375 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T10:18:31.380 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T10:18:31.380 DEBUG:teuthology.orchestra.run.smithi023:> test -z $(ls -A /var/lib/ceph) 2024-03-24T10:18:31.415 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-03-24T10:18:31.417 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-03-24T10:18:31.509 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T10:18:31.583 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T10:18:31.596 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-24T10:18:31.596 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-24T10:18:31.596 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-24T10:18:31.597 DEBUG:teuthology.orchestra.run.smithi023:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T10:18:31.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:18:31.614 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2024-03-24T10:18:31.672 INFO:teuthology.orchestra.run.smithi023.stdout:5.14.0-361.el9.x86_64 2024-03-24T10:18:31.672 INFO:teuthology.task.kernel:Running kernel on smithi023: 5.14.0-361.el9.x86_64 2024-03-24T10:18:31.673 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum install -y kernel 2024-03-24T10:18:33.723 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-24T10:18:35.123 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.0 MB 00:00 2024-03-24T10:18:44.221 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - BaseOS 940 kB/s | 8.0 MB 00:08 2024-03-24T10:18:46.133 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-24T10:18:50.367 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-24T10:18:51.907 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-03-24T10:18:52.933 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-24T10:18:58.042 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-24T10:19:00.199 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T10:19:00.199 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T10:19:00.284 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-24T10:19:00.291 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-24T10:19:00.291 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2024-03-24T10:19:00.291 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-24T10:19:00.291 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2024-03-24T10:19:00.291 INFO:teuthology.orchestra.run.smithi023.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T10:19:00.291 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2024-03-24T10:19:00.292 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T10:19:00.292 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T10:19:00.292 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T10:19:00.292 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:19:00.292 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2024-03-24T10:19:00.292 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-24T10:19:00.292 INFO:teuthology.orchestra.run.smithi023.stdout:Install 4 Packages 2024-03-24T10:19:00.292 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:19:00.293 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 76 M 2024-03-24T10:19:00.293 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 123 M 2024-03-24T10:19:00.293 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2024-03-24T10:19:00.618 INFO:teuthology.orchestra.run.smithi023.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 212 kB/s | 44 kB 00:00 2024-03-24T10:19:01.177 INFO:teuthology.orchestra.run.smithi023.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-03-24T10:19:01.710 INFO:teuthology.orchestra.run.smithi023.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 26 MB/s | 33 MB 00:01 2024-03-24T10:19:02.035 INFO:teuthology.orchestra.run.smithi023.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 19 MB/s | 27 MB 00:01 2024-03-24T10:19:02.038 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:19:02.039 INFO:teuthology.orchestra.run.smithi023.stdout:Total 43 MB/s | 76 MB 00:01 2024-03-24T10:19:02.730 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2024-03-24T10:19:03.179 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2024-03-24T10:19:03.179 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2024-03-24T10:19:04.217 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2024-03-24T10:19:04.217 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2024-03-24T10:19:06.830 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2024-03-24T10:19:07.465 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:19:07.587 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:19:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:19:10.660 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:19:15.740 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:19:15.781 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:19:20.870 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:19:51.621 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:19:51.621 INFO:teuthology.orchestra.run.smithi023.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T10:19:51.621 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:19:51.680 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:19:52.024 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:19:52.024 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:19:52.024 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:19:52.024 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:19:52.468 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:19:52.468 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:19:52.468 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2024-03-24T10:19:52.468 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T10:19:52.468 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T10:19:52.468 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T10:19:52.468 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T10:19:52.468 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:19:52.468 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-24T10:19:52.782 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T10:19:52.906 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T10:19:52.906 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T10:19:52.906 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T10:19:52.906 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T10:19:52.906 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T10:19:52.907 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-24T10:19:52.907 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi023.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T10:19:52.907 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T10:19:52.988 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T10:19:52.988 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T10:19:52.988 DEBUG:teuthology.orchestra.run.smithi023:> sudo rpm -qi grub2-tools 2024-03-24T10:19:53.025 INFO:teuthology.orchestra.run.smithi023.stdout:Name : grub2-tools 2024-03-24T10:19:53.025 INFO:teuthology.orchestra.run.smithi023.stdout:Epoch : 1 2024-03-24T10:19:53.025 INFO:teuthology.orchestra.run.smithi023.stdout:Version : 2.06 2024-03-24T10:19:53.025 INFO:teuthology.orchestra.run.smithi023.stdout:Release : 68.el9 2024-03-24T10:19:53.025 INFO:teuthology.orchestra.run.smithi023.stdout:Architecture: x86_64 2024-03-24T10:19:53.025 INFO:teuthology.orchestra.run.smithi023.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T10:19:53.026 INFO:teuthology.orchestra.run.smithi023.stdout:Group : Unspecified 2024-03-24T10:19:53.026 INFO:teuthology.orchestra.run.smithi023.stdout:Size : 8264254 2024-03-24T10:19:53.026 INFO:teuthology.orchestra.run.smithi023.stdout:License : GPLv3+ 2024-03-24T10:19:53.026 INFO:teuthology.orchestra.run.smithi023.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T10:19:53.026 INFO:teuthology.orchestra.run.smithi023.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T10:19:53.026 INFO:teuthology.orchestra.run.smithi023.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T10:19:53.026 INFO:teuthology.orchestra.run.smithi023.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T10:19:53.026 INFO:teuthology.orchestra.run.smithi023.stdout:Packager : builder@centos.org 2024-03-24T10:19:53.026 INFO:teuthology.orchestra.run.smithi023.stdout:Vendor : CentOS 2024-03-24T10:19:53.026 INFO:teuthology.orchestra.run.smithi023.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T10:19:53.026 INFO:teuthology.orchestra.run.smithi023.stdout:Summary : Support tools for GRUB. 2024-03-24T10:19:53.026 INFO:teuthology.orchestra.run.smithi023.stdout:Description : 2024-03-24T10:19:53.026 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:19:53.026 INFO:teuthology.orchestra.run.smithi023.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T10:19:53.027 INFO:teuthology.orchestra.run.smithi023.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T10:19:53.027 INFO:teuthology.orchestra.run.smithi023.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T10:19:53.027 INFO:teuthology.orchestra.run.smithi023.stdout:hardware devices. 2024-03-24T10:19:53.027 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:19:53.027 INFO:teuthology.orchestra.run.smithi023.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T10:19:53.028 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T10:19:53.028 INFO:teuthology.task.kernel:Updating grub on smithi023 to boot 5.14.0-430.el9.x86_64 2024-03-24T10:19:53.028 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T10:19:53.637 INFO:teuthology.orchestra.run.smithi023.stderr:Generating grub configuration file ... 2024-03-24T10:19:55.249 INFO:teuthology.orchestra.run.smithi023.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T10:19:55.278 INFO:teuthology.orchestra.run.smithi023.stderr:done 2024-03-24T10:19:55.280 DEBUG:teuthology.orchestra.run.smithi023:> mktemp 2024-03-24T10:19:55.297 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.FzMpYCtawZ 2024-03-24T10:19:55.297 DEBUG:teuthology.orchestra.run.smithi023:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FzMpYCtawZ 2024-03-24T10:19:55.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.FzMpYCtawZ 2024-03-24T10:19:55.516 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.FzMpYCtawZ is 6KB 2024-03-24T10:19:55.566 DEBUG:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.FzMpYCtawZ 2024-03-24T10:19:55.580 DEBUG:teuthology.orchestra.run.smithi023:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T10:19:55.648 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T10:19:55.648 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T10:19:55.648 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T10:19:55.648 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T10:19:55.650 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T10:19:55.803 DEBUG:teuthology.orchestra.run.smithi023:> sudo shutdown -r now 2024-03-24T10:19:55.888 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-03-24T10:20:25.805 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-24T10:20:25.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T10:20:25.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:21:00.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-24T10:21:09.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T10:21:09.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:21:12.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-24T10:21:24.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T10:21:24.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:21:27.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-24T10:21:42.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T10:21:42.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:22:00.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-24T10:22:18.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T10:22:18.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:22:19.754 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-24T10:22:20.283 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T10:22:20.283 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-430.el9.x86_64"... 2024-03-24T10:22:20.283 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2024-03-24T10:22:20.308 INFO:teuthology.orchestra.run.smithi023.stdout:5.14.0-430.el9.x86_64 2024-03-24T10:22:20.308 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T10:22:20.308 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T10:22:20.308 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T10:22:21.309 DEBUG:teuthology.parallel:result is None 2024-03-24T10:22:21.310 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T10:22:21.317 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T10:22:21.317 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T10:22:21.320 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T10:22:21.322 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T10:22:21.340 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T10:22:21.346 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T10:22:21.355 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T10:22:21.355 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T10:22:21.375 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T10:22:21.380 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T10:22:21.497 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T10:22:21.503 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T10:22:21.504 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T10:22:21.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:22:21.520 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T10:22:21.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:22:21.536 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T10:22:21.539 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T10:22:21.563 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T10:22:21.601 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:22:21.608 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:22:21.621 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:22:21.626 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:22:21.650 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:22:21.657 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:22:21.659 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T10:22:21.666 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T10:22:21.666 DEBUG:teuthology.orchestra.run.smithi023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T10:22:21.668 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T10:22:21.670 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T10:22:21.700 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T10:22:21.708 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T10:22:21.709 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T10:22:21.735 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T10:22:21.741 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T10:22:21.757 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:22:21.831 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:22:21.890 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:22:21.931 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:22:21.972 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:22:21.972 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T10:22:22.038 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:22:22.078 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:22:22.132 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:22:22.208 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:22:22.260 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T10:22:22.260 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T10:22:22.330 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:22:22.375 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:22:22.436 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:22:22.479 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:22:22.523 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T10:22:22.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T10:22:22.592 DEBUG:teuthology.orchestra.run.smithi023:> sudo service rsyslog restart 2024-03-24T10:22:22.594 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-03-24T10:22:22.597 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-03-24T10:22:22.637 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:22:22.640 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:22:22.668 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:22:23.160 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T10:22:23.168 INFO:teuthology.task.internal:Starting timer... 2024-03-24T10:22:23.168 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T10:22:23.179 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T10:22:23.188 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-24T10:22:23.188 DEBUG:teuthology.orchestra.run.smithi023:> sudo service auditd rotate 2024-03-24T10:22:23.240 INFO:teuthology.orchestra.run.smithi023.stdout:Rotating logs: 2024-03-24T10:22:23.243 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-03-24T10:22:23.290 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-03-24T10:22:23.292 DEBUG:teuthology.orchestra.run.smithi117:> sudo service auditd rotate 2024-03-24T10:22:23.355 INFO:teuthology.orchestra.run.smithi117.stdout:Rotating logs: 2024-03-24T10:22:23.357 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T10:22:23.357 DEBUG:teuthology.orchestra.run.smithi023:> /usr/sbin/getenforce 2024-03-24T10:22:23.377 INFO:teuthology.orchestra.run.smithi023.stdout:Permissive 2024-03-24T10:22:23.377 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-03-24T10:22:23.400 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-03-24T10:22:23.400 DEBUG:teuthology.orchestra.run.smithi117:> /usr/sbin/getenforce 2024-03-24T10:22:23.432 INFO:teuthology.orchestra.run.smithi117.stdout:Permissive 2024-03-24T10:22:23.433 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi023.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi117.front.sepia.ceph.com': 'permissive'} 2024-03-24T10:22:23.433 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T10:22:23.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:22:23.462 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T10:22:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:22:23.490 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T10:22:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:22:23.518 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T10:22:23.518 DEBUG:teuthology.orchestra.run.smithi023:> sudo /usr/sbin/setenforce permissive 2024-03-24T10:22:23.553 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-03-24T10:22:23.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo /usr/sbin/setenforce permissive 2024-03-24T10:22:23.626 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T10:22:23.638 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-24T10:22:23.643 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-24T10:22:23.754 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T10:22:23.768 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T10:22:23.769 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi023.front.sepia.ceph.com,smithi116.front.sepia.ceph.com,smithi117.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T10:26:58.896 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi023.front.sepia.ceph.com'), Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi117.front.sepia.ceph.com')] 2024-03-24T10:26:58.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T10:26:58.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:26:58.994 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-24T10:26:59.068 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-24T10:26:59.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T10:26:59.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:26:59.162 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-24T10:26:59.233 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T10:26:59.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-24T10:26:59.234 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:26:59.321 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-03-24T10:26:59.391 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-24T10:26:59.391 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T10:26:59.404 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T10:26:59.404 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T10:26:59.404 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:26:59.407 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T10:26:59.407 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:26:59.409 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T10:26:59.409 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:26:59.435 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T10:26:59.438 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T10:26:59.452 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T10:26:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T10:26:59.465 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T10:26:59.481 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: ntpd: command not found 2024-03-24T10:26:59.483 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T10:26:59.483 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-03-24T10:26:59.494 INFO:teuthology.orchestra.run.smithi023.stdout:506 Cannot talk to daemon 2024-03-24T10:26:59.497 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-03-24T10:26:59.510 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T10:26:59.514 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T10:26:59.514 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: ntpd: command not found 2024-03-24T10:26:59.526 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T10:26:59.527 INFO:teuthology.orchestra.run.smithi117.stdout:506 Cannot talk to daemon 2024-03-24T10:26:59.530 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T10:26:59.545 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T10:26:59.561 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T10:26:59.573 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:26:59.577 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:26:59.577 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-24T10:26:59.577 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:26:59.577 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:26:59.577 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:26:59.577 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:26:59.585 INFO:teuthology.orchestra.run.smithi023.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:26:59.588 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:26:59.588 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2024-03-24T10:26:59.589 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:26:59.589 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:26:59.589 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:26:59.589 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:26:59.605 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:26:59.608 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:26:59.608 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-03-24T10:26:59.608 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:26:59.608 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:26:59.608 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:26:59.608 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:26:59.609 INFO:teuthology.run_tasks:Running task install... 2024-03-24T10:26:59.617 DEBUG:teuthology.task.install:project ceph 2024-03-24T10:26:59.618 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}} 2024-03-24T10:26:59.618 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'} 2024-03-24T10:26:59.618 INFO:teuthology.task.install:Using flavor: default 2024-03-24T10:26:59.624 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T10:26:59.624 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-24T10:26:59.624 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T10:26:59.625 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:26:59.626 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T10:26:59.627 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:26:59.628 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T10:26:59.628 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:26:59.771 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-24T10:26:59.772 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-24T10:26:59.776 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-24T10:26:59.776 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-24T10:26:59.839 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-24T10:26:59.839 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-24T10:26:59.883 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T10:26:59.883 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T10:26:59.883 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T10:26:59.910 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T10:26:59.911 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T10:26:59.945 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T10:26:59.945 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:26:59.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T10:26:59.968 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T10:26:59.968 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T10:26:59.968 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T10:26:59.972 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T10:26:59.972 DEBUG:teuthology.orchestra.run.smithi023:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T10:26:59.978 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T10:26:59.995 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T10:26:59.995 DEBUG:teuthology.orchestra.run.smithi117:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T10:27:00.040 DEBUG:teuthology.orchestra.run.smithi023:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T10:27:00.057 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T10:27:00.064 DEBUG:teuthology.orchestra.run.smithi117:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T10:27:00.120 DEBUG:teuthology.orchestra.run.smithi023:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T10:27:00.124 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-03-24T10:27:00.125 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-03-24T10:27:00.143 DEBUG:teuthology.orchestra.run.smithi117:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T10:27:00.186 INFO:teuthology.orchestra.run.smithi023.stdout:check_obsoletes = 1 2024-03-24T10:27:00.187 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum clean all 2024-03-24T10:27:00.210 INFO:teuthology.orchestra.run.smithi117.stdout:check_obsoletes = 1 2024-03-24T10:27:00.211 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum clean all 2024-03-24T10:27:00.357 INFO:teuthology.orchestra.run.smithi116.stdout:74 files removed 2024-03-24T10:27:00.383 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T10:27:00.437 INFO:teuthology.orchestra.run.smithi023.stdout:74 files removed 2024-03-24T10:27:00.447 INFO:teuthology.orchestra.run.smithi117.stdout:74 files removed 2024-03-24T10:27:00.464 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T10:27:00.474 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T10:27:01.102 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 193 kB/s | 83 kB 00:00 2024-03-24T10:27:01.151 INFO:teuthology.orchestra.run.smithi023.stdout:ceph packages for x86_64 217 kB/s | 83 kB 00:00 2024-03-24T10:27:01.183 INFO:teuthology.orchestra.run.smithi117.stdout:ceph packages for x86_64 198 kB/s | 83 kB 00:00 2024-03-24T10:27:01.537 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-24T10:27:01.558 INFO:teuthology.orchestra.run.smithi117.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-24T10:27:01.643 INFO:teuthology.orchestra.run.smithi023.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-24T10:27:01.862 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 6.4 kB/s | 1.9 kB 00:00 2024-03-24T10:27:01.942 INFO:teuthology.orchestra.run.smithi117.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-03-24T10:27:02.069 INFO:teuthology.orchestra.run.smithi023.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-24T10:27:02.456 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-24T10:27:02.745 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - AppStream 8.6 MB/s | 6.7 MB 00:00 2024-03-24T10:27:02.804 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - AppStream 9.6 MB/s | 6.7 MB 00:00 2024-03-24T10:27:03.940 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-24T10:27:04.178 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-24T10:27:04.229 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-24T10:27:04.806 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-24T10:27:05.185 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-24T10:27:07.137 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-24T10:27:07.190 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-24T10:27:08.928 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-03-24T10:27:12.421 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - AppStream 7.9 MB/s | 19 MB 00:02 2024-03-24T10:27:17.738 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.1 MB 00:01 2024-03-24T10:27:18.432 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 809 kB/s | 6.1 MB 00:07 2024-03-24T10:27:18.611 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 779 kB/s | 6.1 MB 00:08 2024-03-24T10:27:19.749 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 16 kB 00:00 2024-03-24T10:27:19.989 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-03-24T10:27:20.070 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-24T10:27:20.736 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-24T10:27:21.046 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-24T10:27:22.057 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-24T10:27:25.568 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-03-24T10:27:25.879 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-24T10:27:27.123 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-24T10:27:27.889 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T10:27:27.890 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T10:27:27.904 INFO:teuthology.orchestra.run.smithi116.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T10:27:27.905 INFO:teuthology.orchestra.run.smithi116.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T10:27:27.906 INFO:teuthology.orchestra.run.smithi116.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T10:27:27.907 INFO:teuthology.orchestra.run.smithi116.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T10:27:27.909 INFO:teuthology.orchestra.run.smithi116.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T10:27:27.912 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T10:27:27.913 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T10:27:27.991 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-24T10:27:28.013 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-24T10:27:28.013 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-24T10:27:28.013 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-24T10:27:28.013 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-24T10:27:28.014 INFO:teuthology.orchestra.run.smithi116.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T10:27:28.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-24T10:27:28.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-24T10:27:28.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-24T10:27:28.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-24T10:27:28.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T10:27:28.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-24T10:27:28.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-24T10:27:28.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-24T10:27:28.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-24T10:27:28.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-24T10:27:28.014 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-24T10:27:28.015 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-24T10:27:28.015 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-24T10:27:28.015 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-24T10:27:28.015 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-24T10:27:28.015 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-24T10:27:28.015 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T10:27:28.015 INFO:teuthology.orchestra.run.smithi116.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T10:27:28.015 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T10:27:28.015 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-24T10:27:28.015 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-24T10:27:28.015 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-24T10:27:28.015 INFO:teuthology.orchestra.run.smithi116.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T10:27:28.015 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T10:27:28.015 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T10:27:28.016 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T10:27:28.016 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T10:27:28.016 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T10:27:28.016 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-24T10:27:28.016 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-24T10:27:28.016 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-24T10:27:28.016 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-24T10:27:28.016 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-24T10:27:28.016 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T10:27:28.016 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-24T10:27:28.016 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T10:27:28.016 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-03-24T10:27:28.017 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T10:27:28.017 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T10:27:28.017 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T10:27:28.017 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T10:27:28.017 INFO:teuthology.orchestra.run.smithi116.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T10:27:28.017 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T10:27:28.017 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T10:27:28.017 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T10:27:28.017 INFO:teuthology.orchestra.run.smithi116.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T10:27:28.017 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T10:27:28.017 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T10:27:28.018 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T10:27:28.018 INFO:teuthology.orchestra.run.smithi116.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T10:27:28.018 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-24T10:27:28.018 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T10:27:28.018 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T10:27:28.018 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T10:27:28.018 INFO:teuthology.orchestra.run.smithi116.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T10:27:28.018 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T10:27:28.018 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T10:27:28.018 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T10:27:28.018 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T10:27:28.018 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T10:27:28.019 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T10:27:28.019 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-24T10:27:28.019 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T10:27:28.019 INFO:teuthology.orchestra.run.smithi116.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T10:27:28.019 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T10:27:28.019 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-24T10:27:28.019 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-24T10:27:28.019 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-24T10:27:28.019 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-24T10:27:28.019 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-24T10:27:28.019 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-24T10:27:28.019 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-24T10:27:28.019 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-24T10:27:28.019 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T10:27:28.020 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T10:27:28.020 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T10:27:28.020 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T10:27:28.020 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T10:27:28.020 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T10:27:28.020 INFO:teuthology.orchestra.run.smithi116.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T10:27:28.020 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T10:27:28.020 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T10:27:28.020 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-24T10:27:28.020 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T10:27:28.020 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T10:27:28.020 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T10:27:28.021 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T10:27:28.021 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T10:27:28.021 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-24T10:27:28.021 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T10:27:28.021 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-24T10:27:28.021 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T10:27:28.021 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T10:27:28.021 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T10:27:28.021 INFO:teuthology.orchestra.run.smithi116.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T10:27:28.021 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T10:27:28.021 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T10:27:28.021 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T10:27:28.021 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T10:27:28.021 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T10:27:28.022 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T10:27:28.022 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T10:27:28.022 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T10:27:28.022 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T10:27:28.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T10:27:28.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T10:27:28.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T10:27:28.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T10:27:28.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T10:27:28.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-24T10:27:28.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-24T10:27:28.022 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T10:27:28.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T10:27:28.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T10:27:28.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T10:27:28.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T10:27:28.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T10:27:28.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T10:27:28.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T10:27:28.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T10:27:28.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T10:27:28.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T10:27:28.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T10:27:28.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T10:27:28.023 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T10:27:28.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T10:27:28.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T10:27:28.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T10:27:28.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T10:27:28.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T10:27:28.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T10:27:28.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T10:27:28.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T10:27:28.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T10:27:28.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T10:27:28.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T10:27:28.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T10:27:28.024 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T10:27:28.025 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T10:27:28.025 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T10:27:28.025 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T10:27:28.025 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T10:27:28.025 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T10:27:28.025 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T10:27:28.025 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T10:27:28.025 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T10:27:28.025 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T10:27:28.025 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T10:27:28.025 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T10:27:28.025 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T10:27:28.025 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T10:27:28.026 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T10:27:28.026 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T10:27:28.026 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T10:27:28.026 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T10:27:28.026 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T10:27:28.026 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T10:27:28.026 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T10:27:28.026 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T10:27:28.026 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T10:27:28.026 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T10:27:28.026 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T10:27:28.026 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T10:27:28.026 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T10:27:28.026 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T10:27:28.027 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T10:27:28.027 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T10:27:28.027 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T10:27:28.027 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T10:27:28.027 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T10:27:28.027 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T10:27:28.027 INFO:teuthology.orchestra.run.smithi116.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T10:27:28.027 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T10:27:28.027 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T10:27:28.027 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-03-24T10:27:28.027 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T10:27:28.027 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:27:28.028 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-24T10:27:28.028 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-24T10:27:28.028 INFO:teuthology.orchestra.run.smithi116.stdout:Install 150 Packages 2024-03-24T10:27:28.028 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 24 Packages 2024-03-24T10:27:28.028 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:27:28.028 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 224 M 2024-03-24T10:27:28.028 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-24T10:27:28.225 INFO:teuthology.orchestra.run.smithi117.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T10:27:28.226 INFO:teuthology.orchestra.run.smithi117.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T10:27:28.239 INFO:teuthology.orchestra.run.smithi117.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T10:27:28.240 INFO:teuthology.orchestra.run.smithi117.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T10:27:28.241 INFO:teuthology.orchestra.run.smithi117.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T10:27:28.242 INFO:teuthology.orchestra.run.smithi117.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T10:27:28.244 INFO:teuthology.orchestra.run.smithi117.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T10:27:28.247 INFO:teuthology.orchestra.run.smithi117.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T10:27:28.248 INFO:teuthology.orchestra.run.smithi117.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T10:27:28.327 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-24T10:27:28.339 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-24T10:27:28.339 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-24T10:27:28.339 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-24T10:27:28.339 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-24T10:27:28.339 INFO:teuthology.orchestra.run.smithi117.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T10:27:28.339 INFO:teuthology.orchestra.run.smithi117.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-24T10:27:28.340 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-24T10:27:28.340 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-24T10:27:28.340 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-24T10:27:28.340 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T10:27:28.340 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-24T10:27:28.340 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-24T10:27:28.340 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-24T10:27:28.340 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-24T10:27:28.340 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-24T10:27:28.340 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-24T10:27:28.340 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-24T10:27:28.340 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-24T10:27:28.340 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-24T10:27:28.341 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-24T10:27:28.341 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-24T10:27:28.341 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T10:27:28.341 INFO:teuthology.orchestra.run.smithi117.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T10:27:28.341 INFO:teuthology.orchestra.run.smithi117.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T10:27:28.341 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-24T10:27:28.341 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-24T10:27:28.341 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-24T10:27:28.341 INFO:teuthology.orchestra.run.smithi117.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T10:27:28.341 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T10:27:28.341 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T10:27:28.341 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T10:27:28.341 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T10:27:28.341 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T10:27:28.342 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-24T10:27:28.342 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-24T10:27:28.342 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-24T10:27:28.342 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-24T10:27:28.342 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-24T10:27:28.342 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T10:27:28.342 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-24T10:27:28.342 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T10:27:28.342 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrading: 2024-03-24T10:27:28.342 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T10:27:28.342 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T10:27:28.342 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T10:27:28.342 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T10:27:28.343 INFO:teuthology.orchestra.run.smithi117.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T10:27:28.343 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T10:27:28.343 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T10:27:28.343 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T10:27:28.343 INFO:teuthology.orchestra.run.smithi117.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T10:27:28.343 INFO:teuthology.orchestra.run.smithi117.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T10:27:28.343 INFO:teuthology.orchestra.run.smithi117.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T10:27:28.343 INFO:teuthology.orchestra.run.smithi117.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T10:27:28.343 INFO:teuthology.orchestra.run.smithi117.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T10:27:28.343 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-24T10:27:28.343 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T10:27:28.343 INFO:teuthology.orchestra.run.smithi117.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T10:27:28.343 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T10:27:28.343 INFO:teuthology.orchestra.run.smithi117.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-24T10:27:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-24T10:27:28.345 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-24T10:27:28.345 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-24T10:27:28.345 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-24T10:27:28.345 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T10:27:28.345 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T10:27:28.345 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T10:27:28.345 INFO:teuthology.orchestra.run.smithi117.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T10:27:28.345 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T10:27:28.345 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T10:27:28.345 INFO:teuthology.orchestra.run.smithi117.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T10:27:28.345 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T10:27:28.345 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T10:27:28.345 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-24T10:27:28.345 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T10:27:28.346 INFO:teuthology.orchestra.run.smithi117.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T10:27:28.346 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T10:27:28.346 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T10:27:28.346 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T10:27:28.346 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-24T10:27:28.346 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T10:27:28.346 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-24T10:27:28.346 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T10:27:28.346 INFO:teuthology.orchestra.run.smithi117.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T10:27:28.346 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T10:27:28.346 INFO:teuthology.orchestra.run.smithi117.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T10:27:28.346 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T10:27:28.346 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T10:27:28.347 INFO:teuthology.orchestra.run.smithi117.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T10:27:28.347 INFO:teuthology.orchestra.run.smithi117.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T10:27:28.347 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T10:27:28.347 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T10:27:28.347 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T10:27:28.347 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T10:27:28.347 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T10:27:28.347 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T10:27:28.347 INFO:teuthology.orchestra.run.smithi117.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T10:27:28.347 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T10:27:28.347 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T10:27:28.347 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T10:27:28.347 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-24T10:27:28.347 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-24T10:27:28.348 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T10:27:28.348 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T10:27:28.348 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T10:27:28.348 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T10:27:28.348 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T10:27:28.348 INFO:teuthology.orchestra.run.smithi117.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T10:27:28.348 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T10:27:28.348 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T10:27:28.348 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T10:27:28.348 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T10:27:28.348 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T10:27:28.348 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T10:27:28.348 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T10:27:28.348 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T10:27:28.349 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T10:27:28.349 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T10:27:28.349 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T10:27:28.349 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T10:27:28.349 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T10:27:28.349 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T10:27:28.349 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T10:27:28.349 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T10:27:28.349 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T10:27:28.349 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T10:27:28.349 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T10:27:28.349 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T10:27:28.349 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T10:27:28.349 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T10:27:28.350 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T10:27:28.351 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T10:27:28.351 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T10:27:28.351 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T10:27:28.351 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T10:27:28.351 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T10:27:28.351 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T10:27:28.351 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T10:27:28.351 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T10:27:28.351 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T10:27:28.351 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T10:27:28.351 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T10:27:28.351 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T10:27:28.351 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T10:27:28.351 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T10:27:28.352 INFO:teuthology.orchestra.run.smithi117.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T10:27:28.352 INFO:teuthology.orchestra.run.smithi117.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T10:27:28.352 INFO:teuthology.orchestra.run.smithi117.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T10:27:28.352 INFO:teuthology.orchestra.run.smithi117.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T10:27:28.352 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T10:27:28.352 INFO:teuthology.orchestra.run.smithi117.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T10:27:28.352 INFO:teuthology.orchestra.run.smithi117.stdout:Installing weak dependencies: 2024-03-24T10:27:28.352 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T10:27:28.352 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:27:28.352 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-24T10:27:28.352 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-24T10:27:28.352 INFO:teuthology.orchestra.run.smithi117.stdout:Install 150 Packages 2024-03-24T10:27:28.353 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrade 24 Packages 2024-03-24T10:27:28.353 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:27:28.353 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 224 M 2024-03-24T10:27:28.353 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-24T10:27:28.518 INFO:teuthology.orchestra.run.smithi116.stdout:(1/174): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 33 kB/s | 6.6 kB 00:00 2024-03-24T10:27:28.822 INFO:teuthology.orchestra.run.smithi117.stdout:(1/174): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 32 kB/s | 6.6 kB 00:00 2024-03-24T10:27:28.869 INFO:teuthology.orchestra.run.smithi116.stdout:(2/174): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 2.5 MB/s | 888 kB 00:00 2024-03-24T10:27:28.969 INFO:teuthology.orchestra.run.smithi116.stdout:(3/174): ceph-immutable-object-cache-19.0.0-185 1.4 MB/s | 145 kB 00:00 2024-03-24T10:27:29.169 INFO:teuthology.orchestra.run.smithi116.stdout:(4/174): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 6.3 MB/s | 5.3 MB 00:00 2024-03-24T10:27:29.180 INFO:teuthology.orchestra.run.smithi117.stdout:(2/174): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 2.4 MB/s | 888 kB 00:00 2024-03-24T10:27:29.303 INFO:teuthology.orchestra.run.smithi116.stdout:(5/174): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 7.1 MB/s | 2.4 MB 00:00 2024-03-24T10:27:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:(3/174): ceph-immutable-object-cache-19.0.0-185 1.0 MB/s | 145 kB 00:00 2024-03-24T10:27:29.378 INFO:teuthology.orchestra.run.smithi116.stdout:(6/174): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x86 7.2 MB/s | 1.5 MB 00:00 2024-03-24T10:27:29.594 INFO:teuthology.orchestra.run.smithi023.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T10:27:29.594 INFO:teuthology.orchestra.run.smithi023.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T10:27:29.607 INFO:teuthology.orchestra.run.smithi023.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T10:27:29.608 INFO:teuthology.orchestra.run.smithi023.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T10:27:29.608 INFO:teuthology.orchestra.run.smithi023.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T10:27:29.609 INFO:teuthology.orchestra.run.smithi023.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T10:27:29.611 INFO:teuthology.orchestra.run.smithi023.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T10:27:29.614 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T10:27:29.615 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T10:27:29.700 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-24T10:27:29.721 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-24T10:27:29.721 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2024-03-24T10:27:29.721 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-24T10:27:29.721 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2024-03-24T10:27:29.721 INFO:teuthology.orchestra.run.smithi023.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T10:27:29.722 INFO:teuthology.orchestra.run.smithi023.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-24T10:27:29.722 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-24T10:27:29.722 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-24T10:27:29.722 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-24T10:27:29.722 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T10:27:29.722 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-24T10:27:29.722 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-24T10:27:29.722 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-24T10:27:29.722 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-24T10:27:29.722 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-24T10:27:29.722 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-24T10:27:29.723 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-24T10:27:29.723 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-24T10:27:29.723 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-24T10:27:29.723 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-24T10:27:29.723 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-24T10:27:29.723 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T10:27:29.723 INFO:teuthology.orchestra.run.smithi023.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T10:27:29.723 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T10:27:29.723 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-24T10:27:29.723 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-24T10:27:29.723 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-24T10:27:29.723 INFO:teuthology.orchestra.run.smithi023.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T10:27:29.723 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T10:27:29.723 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T10:27:29.724 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T10:27:29.724 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T10:27:29.724 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T10:27:29.724 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-24T10:27:29.724 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-24T10:27:29.724 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-24T10:27:29.724 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-24T10:27:29.724 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-24T10:27:29.724 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T10:27:29.724 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-24T10:27:29.724 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T10:27:29.724 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrading: 2024-03-24T10:27:29.724 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T10:27:29.725 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T10:27:29.725 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T10:27:29.725 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T10:27:29.725 INFO:teuthology.orchestra.run.smithi023.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T10:27:29.725 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T10:27:29.725 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T10:27:29.725 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T10:27:29.725 INFO:teuthology.orchestra.run.smithi023.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T10:27:29.725 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T10:27:29.725 INFO:teuthology.orchestra.run.smithi023.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T10:27:29.725 INFO:teuthology.orchestra.run.smithi023.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T10:27:29.725 INFO:teuthology.orchestra.run.smithi023.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T10:27:29.725 INFO:teuthology.orchestra.run.smithi023.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-24T10:27:29.726 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T10:27:29.726 INFO:teuthology.orchestra.run.smithi023.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T10:27:29.726 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T10:27:29.726 INFO:teuthology.orchestra.run.smithi023.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T10:27:29.726 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T10:27:29.726 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T10:27:29.726 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T10:27:29.726 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T10:27:29.726 INFO:teuthology.orchestra.run.smithi023.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T10:27:29.726 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T10:27:29.726 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2024-03-24T10:27:29.726 INFO:teuthology.orchestra.run.smithi023.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T10:27:29.726 INFO:teuthology.orchestra.run.smithi023.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T10:27:29.727 INFO:teuthology.orchestra.run.smithi023.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T10:27:29.727 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-24T10:27:29.727 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-24T10:27:29.727 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-24T10:27:29.727 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-24T10:27:29.727 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-24T10:27:29.727 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-24T10:27:29.727 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-24T10:27:29.727 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-24T10:27:29.727 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T10:27:29.727 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T10:27:29.727 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T10:27:29.727 INFO:teuthology.orchestra.run.smithi023.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T10:27:29.727 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T10:27:29.728 INFO:teuthology.orchestra.run.smithi023.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T10:27:29.728 INFO:teuthology.orchestra.run.smithi023.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T10:27:29.728 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T10:27:29.728 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T10:27:29.728 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-24T10:27:29.728 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T10:27:29.728 INFO:teuthology.orchestra.run.smithi023.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T10:27:29.728 INFO:teuthology.orchestra.run.smithi023.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T10:27:29.728 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T10:27:29.728 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T10:27:29.728 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-24T10:27:29.728 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T10:27:29.728 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-24T10:27:29.729 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T10:27:29.729 INFO:teuthology.orchestra.run.smithi023.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T10:27:29.729 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T10:27:29.729 INFO:teuthology.orchestra.run.smithi023.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T10:27:29.729 INFO:teuthology.orchestra.run.smithi023.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T10:27:29.729 INFO:teuthology.orchestra.run.smithi023.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T10:27:29.729 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T10:27:29.729 INFO:teuthology.orchestra.run.smithi023.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T10:27:29.729 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T10:27:29.729 INFO:teuthology.orchestra.run.smithi023.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T10:27:29.729 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T10:27:29.729 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T10:27:29.729 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T10:27:29.729 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T10:27:29.730 INFO:teuthology.orchestra.run.smithi023.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T10:27:29.730 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T10:27:29.730 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T10:27:29.730 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T10:27:29.730 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-24T10:27:29.730 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-24T10:27:29.730 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T10:27:29.730 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T10:27:29.730 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T10:27:29.730 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T10:27:29.730 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T10:27:29.730 INFO:teuthology.orchestra.run.smithi023.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T10:27:29.730 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T10:27:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T10:27:29.732 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T10:27:29.732 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T10:27:29.732 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T10:27:29.732 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T10:27:29.732 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T10:27:29.732 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T10:27:29.732 INFO:teuthology.orchestra.run.smithi023.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T10:27:29.732 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T10:27:29.732 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T10:27:29.732 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T10:27:29.732 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T10:27:29.732 INFO:teuthology.orchestra.run.smithi023.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T10:27:29.732 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T10:27:29.733 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T10:27:29.733 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T10:27:29.733 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T10:27:29.733 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T10:27:29.733 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T10:27:29.733 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T10:27:29.733 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T10:27:29.733 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T10:27:29.733 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T10:27:29.733 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T10:27:29.733 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T10:27:29.733 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T10:27:29.733 INFO:teuthology.orchestra.run.smithi023.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T10:27:29.734 INFO:teuthology.orchestra.run.smithi023.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T10:27:29.734 INFO:teuthology.orchestra.run.smithi023.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T10:27:29.734 INFO:teuthology.orchestra.run.smithi023.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T10:27:29.734 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T10:27:29.734 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T10:27:29.734 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T10:27:29.734 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T10:27:29.734 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T10:27:29.735 INFO:teuthology.orchestra.run.smithi023.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T10:27:29.735 INFO:teuthology.orchestra.run.smithi023.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T10:27:29.735 INFO:teuthology.orchestra.run.smithi023.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T10:27:29.735 INFO:teuthology.orchestra.run.smithi023.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T10:27:29.735 INFO:teuthology.orchestra.run.smithi023.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T10:27:29.735 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T10:27:29.735 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T10:27:29.735 INFO:teuthology.orchestra.run.smithi023.stdout:Installing weak dependencies: 2024-03-24T10:27:29.735 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T10:27:29.735 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:27:29.735 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2024-03-24T10:27:29.735 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-24T10:27:29.735 INFO:teuthology.orchestra.run.smithi023.stdout:Install 150 Packages 2024-03-24T10:27:29.735 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrade 24 Packages 2024-03-24T10:27:29.736 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:27:29.736 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 224 M 2024-03-24T10:27:29.736 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2024-03-24T10:27:30.087 INFO:teuthology.orchestra.run.smithi116.stdout:(7/174): ceph-mon-19.0.0-1855.g21b3bd67.el9.x86 6.0 MB/s | 4.7 MB 00:00 2024-03-24T10:27:30.656 INFO:teuthology.orchestra.run.smithi117.stdout:(4/174): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 1.8 MB/s | 2.4 MB 00:01 2024-03-24T10:27:30.933 INFO:teuthology.orchestra.run.smithi023.stdout:(1/174): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-03-24T10:27:30.954 INFO:teuthology.orchestra.run.smithi116.stdout:(8/174): ceph-osd-19.0.0-1855.g21b3bd67.el9.x86 11 MB/s | 17 MB 00:01 2024-03-24T10:27:31.020 INFO:teuthology.orchestra.run.smithi116.stdout:(9/174): ceph-selinux-19.0.0-1855.g21b3bd67.el9 379 kB/s | 25 kB 00:00 2024-03-24T10:27:31.372 INFO:teuthology.orchestra.run.smithi117.stdout:(5/174): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x86 2.1 MB/s | 1.5 MB 00:00 2024-03-24T10:27:31.464 INFO:teuthology.orchestra.run.smithi117.stdout:(6/174): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 1.9 MB/s | 5.3 MB 00:02 2024-03-24T10:27:31.866 INFO:teuthology.orchestra.run.smithi023.stdout:(2/174): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 952 kB/s | 888 kB 00:00 2024-03-24T10:27:31.962 INFO:teuthology.orchestra.run.smithi116.stdout:(10/174): ceph-common-19.0.0-1855.g21b3bd67.el9 5.6 MB/s | 21 MB 00:03 2024-03-24T10:27:32.017 INFO:teuthology.orchestra.run.smithi023.stdout:(3/174): ceph-immutable-object-cache-19.0.0-185 966 kB/s | 145 kB 00:00 2024-03-24T10:27:32.163 INFO:teuthology.orchestra.run.smithi116.stdout:(11/174): cephfs-mirror-19.0.0-1855.g21b3bd67.e 1.1 MB/s | 219 kB 00:00 2024-03-24T10:27:32.263 INFO:teuthology.orchestra.run.smithi116.stdout:(12/174): ceph-radosgw-19.0.0-1855.g21b3bd67.el 4.5 MB/s | 9.8 MB 00:02 2024-03-24T10:27:32.280 INFO:teuthology.orchestra.run.smithi116.stdout:(13/174): cephfs-shell-19.0.0-1855.g21b3bd67.el 232 kB/s | 27 kB 00:00 2024-03-24T10:27:32.355 INFO:teuthology.orchestra.run.smithi116.stdout:(14/174): libcephfs-devel-19.0.0-1855.g21b3bd67 346 kB/s | 32 kB 00:00 2024-03-24T10:27:32.513 INFO:teuthology.orchestra.run.smithi116.stdout:(15/174): libcephsqlite-19.0.0-1855.g21b3bd67.e 1.0 MB/s | 163 kB 00:00 2024-03-24T10:27:32.564 INFO:teuthology.orchestra.run.smithi116.stdout:(16/174): libcephfs2-19.0.0-1855.g21b3bd67.el9. 2.6 MB/s | 744 kB 00:00 2024-03-24T10:27:32.647 INFO:teuthology.orchestra.run.smithi116.stdout:(17/174): librados-devel-19.0.0-1855.g21b3bd67. 948 kB/s | 127 kB 00:00 2024-03-24T10:27:32.881 INFO:teuthology.orchestra.run.smithi116.stdout:(18/174): libradosstriper1-19.0.0-1855.g21b3bd6 1.5 MB/s | 499 kB 00:00 2024-03-24T10:27:32.989 INFO:teuthology.orchestra.run.smithi116.stdout:(19/174): python3-ceph-argparse-19.0.0-1855.g21 417 kB/s | 45 kB 00:00 2024-03-24T10:27:33.115 INFO:teuthology.orchestra.run.smithi116.stdout:(20/174): python3-ceph-common-19.0.0-1855.g21b3 1.0 MB/s | 131 kB 00:00 2024-03-24T10:27:33.240 INFO:teuthology.orchestra.run.smithi116.stdout:(21/174): python3-cephfs-19.0.0-1855.g21b3bd67. 1.3 MB/s | 161 kB 00:00 2024-03-24T10:27:33.292 INFO:teuthology.orchestra.run.smithi023.stdout:(4/174): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 1.8 MB/s | 2.4 MB 00:01 2024-03-24T10:27:33.490 INFO:teuthology.orchestra.run.smithi116.stdout:(22/174): python3-rados-19.0.0-1855.g21b3bd67.e 1.3 MB/s | 320 kB 00:00 2024-03-24T10:27:33.615 INFO:teuthology.orchestra.run.smithi116.stdout:(23/174): python3-rbd-19.0.0-1855.g21b3bd67.el9 2.3 MB/s | 298 kB 00:00 2024-03-24T10:27:33.675 INFO:teuthology.orchestra.run.smithi023.stdout:(5/174): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 1.8 MB/s | 5.3 MB 00:03 2024-03-24T10:27:33.706 INFO:teuthology.orchestra.run.smithi117.stdout:(7/174): ceph-common-19.0.0-1855.g21b3bd67.el9. 4.0 MB/s | 21 MB 00:05 2024-03-24T10:27:33.749 INFO:teuthology.orchestra.run.smithi116.stdout:(24/174): python3-rgw-19.0.0-1855.g21b3bd67.el9 743 kB/s | 99 kB 00:00 2024-03-24T10:27:33.841 INFO:teuthology.orchestra.run.smithi116.stdout:(25/174): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 933 kB/s | 85 kB 00:00 2024-03-24T10:27:33.984 INFO:teuthology.orchestra.run.smithi023.stdout:(6/174): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x86 2.2 MB/s | 1.5 MB 00:00 2024-03-24T10:27:34.864 INFO:teuthology.orchestra.run.smithi117.stdout:(8/174): ceph-radosgw-19.0.0-1855.g21b3bd67.el9 8.4 MB/s | 9.8 MB 00:01 2024-03-24T10:27:34.917 INFO:teuthology.orchestra.run.smithi116.stdout:(26/174): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 2.9 MB/s | 3.1 MB 00:01 2024-03-24T10:27:34.939 INFO:teuthology.orchestra.run.smithi117.stdout:(9/174): ceph-selinux-19.0.0-1855.g21b3bd67.el9 337 kB/s | 25 kB 00:00 2024-03-24T10:27:34.992 INFO:teuthology.orchestra.run.smithi116.stdout:(27/174): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 2.2 MB/s | 170 kB 00:00 2024-03-24T10:27:35.092 INFO:teuthology.orchestra.run.smithi116.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1855.g 315 kB/s | 31 kB 00:00 2024-03-24T10:27:35.115 INFO:teuthology.orchestra.run.smithi117.stdout:(10/174): ceph-mon-19.0.0-1855.g21b3bd67.el9.x8 1.3 MB/s | 4.7 MB 00:03 2024-03-24T10:27:35.184 INFO:teuthology.orchestra.run.smithi116.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 1.5 MB/s | 138 kB 00:00 2024-03-24T10:27:35.210 INFO:teuthology.orchestra.run.smithi023.stdout:(7/174): ceph-mon-19.0.0-1855.g21b3bd67.el9.x86 3.1 MB/s | 4.7 MB 00:01 2024-03-24T10:27:35.340 INFO:teuthology.orchestra.run.smithi117.stdout:(11/174): cephfs-mirror-19.0.0-1855.g21b3bd67.e 975 kB/s | 219 kB 00:00 2024-03-24T10:27:35.440 INFO:teuthology.orchestra.run.smithi117.stdout:(12/174): cephfs-shell-19.0.0-1855.g21b3bd67.el 270 kB/s | 27 kB 00:00 2024-03-24T10:27:35.540 INFO:teuthology.orchestra.run.smithi117.stdout:(13/174): libcephfs-devel-19.0.0-1855.g21b3bd67 317 kB/s | 32 kB 00:00 2024-03-24T10:27:35.826 INFO:teuthology.orchestra.run.smithi116.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1855.g21b3b 2.8 MB/s | 1.8 MB 00:00 2024-03-24T10:27:36.241 INFO:teuthology.orchestra.run.smithi117.stdout:(14/174): libcephfs2-19.0.0-1855.g21b3bd67.el9. 1.0 MB/s | 744 kB 00:00 2024-03-24T10:27:36.351 INFO:teuthology.orchestra.run.smithi116.stdout:(31/174): librgw2-19.0.0-1855.g21b3bd67.el9.x86 1.4 MB/s | 5.1 MB 00:03 2024-03-24T10:27:36.458 INFO:teuthology.orchestra.run.smithi117.stdout:(15/174): libcephsqlite-19.0.0-1855.g21b3bd67.e 754 kB/s | 163 kB 00:00 2024-03-24T10:27:36.625 INFO:teuthology.orchestra.run.smithi117.stdout:(16/174): librados-devel-19.0.0-1855.g21b3bd67. 761 kB/s | 127 kB 00:00 2024-03-24T10:27:36.626 INFO:teuthology.orchestra.run.smithi116.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1855.g21 904 kB/s | 248 kB 00:00 2024-03-24T10:27:36.735 INFO:teuthology.orchestra.run.smithi116.stdout:(33/174): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 455 kB/s | 49 kB 00:00 2024-03-24T10:27:36.835 INFO:teuthology.orchestra.run.smithi116.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1855.g2 153 kB/s | 15 kB 00:00 2024-03-24T10:27:37.169 INFO:teuthology.orchestra.run.smithi116.stdout:(35/174): ceph-volume-19.0.0-1855.g21b3bd67.el9 791 kB/s | 264 kB 00:00 2024-03-24T10:27:37.191 INFO:teuthology.orchestra.run.smithi117.stdout:(17/174): libradosstriper1-19.0.0-1855.g21b3bd6 882 kB/s | 499 kB 00:00 2024-03-24T10:27:37.736 INFO:teuthology.orchestra.run.smithi116.stdout:(36/174): cephadm-19.0.0-1855.g21b3bd67.el9.noa 1.3 MB/s | 748 kB 00:00 2024-03-24T10:27:37.836 INFO:teuthology.orchestra.run.smithi116.stdout:(37/174): cephfs-top-19.0.0-1855.g21b3bd67.el9. 182 kB/s | 18 kB 00:00 2024-03-24T10:27:37.995 INFO:teuthology.orchestra.run.smithi116.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.1 MB/s | 662 kB 00:00 2024-03-24T10:27:38.045 INFO:teuthology.orchestra.run.smithi116.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-24T10:27:38.070 INFO:teuthology.orchestra.run.smithi116.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 4.1 MB/s | 106 kB 00:00 2024-03-24T10:27:38.120 INFO:teuthology.orchestra.run.smithi116.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-24T10:27:38.171 INFO:teuthology.orchestra.run.smithi116.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 633 kB/s | 32 kB 00:00 2024-03-24T10:27:38.234 INFO:teuthology.orchestra.run.smithi023.stdout:(8/174): ceph-radosgw-19.0.0-1855.g21b3bd67.el9 3.2 MB/s | 9.8 MB 00:03 2024-03-24T10:27:38.254 INFO:teuthology.orchestra.run.smithi116.stdout:(43/174): ceph-mgr-diskprediction-local-19.0.0- 3.0 MB/s | 7.4 MB 00:02 2024-03-24T10:27:38.305 INFO:teuthology.orchestra.run.smithi116.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 22 MB/s | 3.0 MB 00:00 2024-03-24T10:27:38.309 INFO:teuthology.orchestra.run.smithi023.stdout:(9/174): ceph-selinux-19.0.0-1855.g21b3bd67.el9 336 kB/s | 25 kB 00:00 2024-03-24T10:27:38.321 INFO:teuthology.orchestra.run.smithi116.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 266 kB/s | 17 kB 00:00 2024-03-24T10:27:38.338 INFO:teuthology.orchestra.run.smithi116.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 300 kB/s | 10 kB 00:00 2024-03-24T10:27:38.364 INFO:teuthology.orchestra.run.smithi116.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-24T10:27:38.389 INFO:teuthology.orchestra.run.smithi116.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-24T10:27:38.414 INFO:teuthology.orchestra.run.smithi116.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-24T10:27:38.448 INFO:teuthology.orchestra.run.smithi116.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-24T10:27:38.473 INFO:teuthology.orchestra.run.smithi116.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 806 kB/s | 46 kB 00:00 2024-03-24T10:27:38.503 INFO:teuthology.orchestra.run.smithi116.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.0 MB/s | 282 kB 00:00 2024-03-24T10:27:38.529 INFO:teuthology.orchestra.run.smithi116.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.3 MB/s | 187 kB 00:00 2024-03-24T10:27:38.543 INFO:teuthology.orchestra.run.smithi023.stdout:(10/174): ceph-common-19.0.0-1855.g21b3bd67.el9 2.6 MB/s | 21 MB 00:07 2024-03-24T10:27:38.554 INFO:teuthology.orchestra.run.smithi116.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-24T10:27:38.571 INFO:teuthology.orchestra.run.smithi116.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 823 kB/s | 34 kB 00:00 2024-03-24T10:27:38.660 INFO:teuthology.orchestra.run.smithi023.stdout:(11/174): cephfs-mirror-19.0.0-1855.g21b3bd67.e 1.8 MB/s | 219 kB 00:00 2024-03-24T10:27:38.727 INFO:teuthology.orchestra.run.smithi023.stdout:(12/174): cephfs-shell-19.0.0-1855.g21b3bd67.el 406 kB/s | 27 kB 00:00 2024-03-24T10:27:38.802 INFO:teuthology.orchestra.run.smithi023.stdout:(13/174): libcephfs-devel-19.0.0-1855.g21b3bd67 422 kB/s | 32 kB 00:00 2024-03-24T10:27:39.038 INFO:teuthology.orchestra.run.smithi116.stdout:(56/174): ceph-test-19.0.0-1855.g21b3bd67.el9.x 5.9 MB/s | 47 MB 00:08 2024-03-24T10:27:39.077 INFO:teuthology.orchestra.run.smithi023.stdout:(14/174): libcephfs2-19.0.0-1855.g21b3bd67.el9. 2.6 MB/s | 744 kB 00:00 2024-03-24T10:27:39.114 INFO:teuthology.orchestra.run.smithi116.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 8.5 MB/s | 4.7 MB 00:00 2024-03-24T10:27:39.147 INFO:teuthology.orchestra.run.smithi116.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 2.6 MB/s | 1.5 MB 00:00 2024-03-24T10:27:39.170 INFO:teuthology.orchestra.run.smithi023.stdout:(15/174): libcephsqlite-19.0.0-1855.g21b3bd67.e 1.7 MB/s | 163 kB 00:00 2024-03-24T10:27:39.172 INFO:teuthology.orchestra.run.smithi116.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 6.1 MB/s | 811 kB 00:00 2024-03-24T10:27:39.206 INFO:teuthology.orchestra.run.smithi116.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.4 MB/s | 134 kB 00:00 2024-03-24T10:27:39.248 INFO:teuthology.orchestra.run.smithi116.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 14 MB/s | 1.0 MB 00:00 2024-03-24T10:27:39.261 INFO:teuthology.orchestra.run.smithi023.stdout:(16/174): librados-devel-19.0.0-1855.g21b3bd67. 1.4 MB/s | 127 kB 00:00 2024-03-24T10:27:39.340 INFO:teuthology.orchestra.run.smithi116.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-24T10:27:39.357 INFO:teuthology.orchestra.run.smithi116.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 522 kB/s | 78 kB 00:00 2024-03-24T10:27:39.449 INFO:teuthology.orchestra.run.smithi116.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-24T10:27:39.470 INFO:teuthology.orchestra.run.smithi023.stdout:(17/174): libradosstriper1-19.0.0-1855.g21b3bd6 2.3 MB/s | 499 kB 00:00 2024-03-24T10:27:39.474 INFO:teuthology.orchestra.run.smithi116.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 186 kB/s | 24 kB 00:00 2024-03-24T10:27:39.499 INFO:teuthology.orchestra.run.smithi116.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 129 kB/s | 18 kB 00:00 2024-03-24T10:27:39.524 INFO:teuthology.orchestra.run.smithi116.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-24T10:27:39.549 INFO:teuthology.orchestra.run.smithi116.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 635 kB/s | 32 kB 00:00 2024-03-24T10:27:39.642 INFO:teuthology.orchestra.run.smithi116.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-24T10:27:39.666 INFO:teuthology.orchestra.run.smithi117.stdout:(18/174): ceph-osd-19.0.0-1855.g21b3bd67.el9.x8 2.0 MB/s | 17 MB 00:08 2024-03-24T10:27:39.675 INFO:teuthology.orchestra.run.smithi116.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-24T10:27:39.700 INFO:teuthology.orchestra.run.smithi116.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-24T10:27:39.725 INFO:teuthology.orchestra.run.smithi116.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 829 kB/s | 69 kB 00:00 2024-03-24T10:27:39.750 INFO:teuthology.orchestra.run.smithi117.stdout:(19/174): python3-ceph-argparse-19.0.0-1855.g21 543 kB/s | 45 kB 00:00 2024-03-24T10:27:39.750 INFO:teuthology.orchestra.run.smithi116.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-03-24T10:27:39.762 INFO:teuthology.orchestra.run.smithi023.stdout:(18/174): ceph-osd-19.0.0-1855.g21b3bd67.el9.x8 2.9 MB/s | 17 MB 00:05 2024-03-24T10:27:39.776 INFO:teuthology.orchestra.run.smithi116.stdout:(74/174): python3-prettytable-0.7.2-27.el9.noar 548 kB/s | 41 kB 00:00 2024-03-24T10:27:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-24T10:27:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-24T10:27:39.829 INFO:teuthology.orchestra.run.smithi023.stdout:(19/174): python3-ceph-argparse-19.0.0-1855.g21 680 kB/s | 45 kB 00:00 2024-03-24T10:27:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-24T10:27:39.860 INFO:teuthology.orchestra.run.smithi116.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 734 kB/s | 43 kB 00:00 2024-03-24T10:27:39.887 INFO:teuthology.orchestra.run.smithi023.stdout:(20/174): python3-ceph-common-19.0.0-1855.g21b3 2.2 MB/s | 131 kB 00:00 2024-03-24T10:27:39.971 INFO:teuthology.orchestra.run.smithi023.stdout:(21/174): python3-cephfs-19.0.0-1855.g21b3bd67. 1.9 MB/s | 161 kB 00:00 2024-03-24T10:27:40.019 INFO:teuthology.orchestra.run.smithi116.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 248 kB/s | 39 kB 00:00 2024-03-24T10:27:40.036 INFO:teuthology.orchestra.run.smithi116.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 237 kB/s | 41 kB 00:00 2024-03-24T10:27:40.088 INFO:teuthology.orchestra.run.smithi023.stdout:(22/174): python3-rados-19.0.0-1855.g21b3bd67.e 2.7 MB/s | 320 kB 00:00 2024-03-24T10:27:40.158 INFO:teuthology.orchestra.run.smithi117.stdout:(20/174): ceph-test-19.0.0-1855.g21b3bd67.el9.x 9.1 MB/s | 47 MB 00:05 2024-03-24T10:27:40.183 INFO:teuthology.orchestra.run.smithi117.stdout:(21/174): python3-ceph-common-19.0.0-1855.g21b3 303 kB/s | 131 kB 00:00 2024-03-24T10:27:40.195 INFO:teuthology.orchestra.run.smithi116.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-24T10:27:40.196 INFO:teuthology.orchestra.run.smithi023.stdout:(23/174): python3-rbd-19.0.0-1855.g21b3bd67.el9 2.7 MB/s | 298 kB 00:00 2024-03-24T10:27:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-03-24T10:27:40.237 INFO:teuthology.orchestra.run.smithi116.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 285 kB/s | 56 kB 00:00 2024-03-24T10:27:40.242 INFO:teuthology.orchestra.run.smithi117.stdout:(22/174): librgw2-19.0.0-1855.g21b3bd67.el9.x86 1.7 MB/s | 5.1 MB 00:03 2024-03-24T10:27:40.253 INFO:teuthology.orchestra.run.smithi116.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 63 kB 00:00 2024-03-24T10:27:40.255 INFO:teuthology.orchestra.run.smithi023.stdout:(24/174): python3-rgw-19.0.0-1855.g21b3bd67.el9 1.7 MB/s | 99 kB 00:00 2024-03-24T10:27:40.267 INFO:teuthology.orchestra.run.smithi117.stdout:(23/174): python3-cephfs-19.0.0-1855.g21b3bd67. 1.5 MB/s | 161 kB 00:00 2024-03-24T10:27:40.270 INFO:teuthology.orchestra.run.smithi116.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-24T10:27:40.296 INFO:teuthology.orchestra.run.smithi116.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-24T10:27:40.321 INFO:teuthology.orchestra.run.smithi116.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-24T10:27:40.322 INFO:teuthology.orchestra.run.smithi023.stdout:(25/174): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-24T10:27:40.326 INFO:teuthology.orchestra.run.smithi117.stdout:(24/174): python3-rados-19.0.0-1855.g21b3bd67.e 2.2 MB/s | 320 kB 00:00 2024-03-24T10:27:40.346 INFO:teuthology.orchestra.run.smithi116.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-24T10:27:40.351 INFO:teuthology.orchestra.run.smithi117.stdout:(25/174): python3-rgw-19.0.0-1855.g21b3bd67.el9 1.2 MB/s | 99 kB 00:00 2024-03-24T10:27:40.376 INFO:teuthology.orchestra.run.smithi117.stdout:(26/174): python3-rbd-19.0.0-1855.g21b3bd67.el9 2.2 MB/s | 298 kB 00:00 2024-03-24T10:27:40.380 INFO:teuthology.orchestra.run.smithi116.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-24T10:27:40.402 INFO:teuthology.orchestra.run.smithi117.stdout:(27/174): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-24T10:27:40.405 INFO:teuthology.orchestra.run.smithi116.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2024-03-24T10:27:40.422 INFO:teuthology.orchestra.run.smithi116.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 187 kB 00:00 2024-03-24T10:27:40.469 INFO:teuthology.orchestra.run.smithi117.stdout:(28/174): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-24T10:27:40.494 INFO:teuthology.orchestra.run.smithi117.stdout:(29/174): ceph-grafana-dashboards-19.0.0-1855.g 343 kB/s | 31 kB 00:00 2024-03-24T10:27:40.547 INFO:teuthology.orchestra.run.smithi023.stdout:(26/174): librgw2-19.0.0-1855.g21b3bd67.el9.x86 4.8 MB/s | 5.1 MB 00:01 2024-03-24T10:27:40.553 INFO:teuthology.orchestra.run.smithi117.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 1.6 MB/s | 138 kB 00:00 2024-03-24T10:27:40.555 INFO:teuthology.orchestra.run.smithi116.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-24T10:27:40.581 INFO:teuthology.orchestra.run.smithi116.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-03-24T10:27:40.606 INFO:teuthology.orchestra.run.smithi116.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-24T10:27:40.614 INFO:teuthology.orchestra.run.smithi023.stdout:(27/174): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 2.5 MB/s | 170 kB 00:00 2024-03-24T10:27:40.665 INFO:teuthology.orchestra.run.smithi116.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-03-24T10:27:40.669 INFO:teuthology.orchestra.run.smithi117.stdout:(31/174): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 9.7 MB/s | 3.1 MB 00:00 2024-03-24T10:27:40.673 INFO:teuthology.orchestra.run.smithi023.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1855.g 538 kB/s | 31 kB 00:00 2024-03-24T10:27:40.745 INFO:teuthology.orchestra.run.smithi117.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1855.g21 3.3 MB/s | 248 kB 00:00 2024-03-24T10:27:40.756 INFO:teuthology.orchestra.run.smithi116.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 122 kB/s | 21 kB 00:00 2024-03-24T10:27:40.765 INFO:teuthology.orchestra.run.smithi023.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 1.5 MB/s | 138 kB 00:00 2024-03-24T10:27:40.803 INFO:teuthology.orchestra.run.smithi117.stdout:(33/174): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 847 kB/s | 49 kB 00:00 2024-03-24T10:27:40.815 INFO:teuthology.orchestra.run.smithi116.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 139 kB/s | 25 kB 00:00 2024-03-24T10:27:40.862 INFO:teuthology.orchestra.run.smithi117.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1855.g2 262 kB/s | 15 kB 00:00 2024-03-24T10:27:40.932 INFO:teuthology.orchestra.run.smithi116.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-24T10:27:40.937 INFO:teuthology.orchestra.run.smithi117.stdout:(35/174): ceph-volume-19.0.0-1855.g21b3bd67.el9 3.4 MB/s | 264 kB 00:00 2024-03-24T10:27:40.987 INFO:teuthology.orchestra.run.smithi117.stdout:(36/174): ceph-mgr-dashboard-19.0.0-1855.g21b3b 3.6 MB/s | 1.8 MB 00:00 2024-03-24T10:27:41.006 INFO:teuthology.orchestra.run.smithi023.stdout:(30/174): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 4.5 MB/s | 3.1 MB 00:00 2024-03-24T10:27:41.007 INFO:teuthology.orchestra.run.smithi116.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.3 MB/s | 248 kB 00:00 2024-03-24T10:27:41.029 INFO:teuthology.orchestra.run.smithi117.stdout:(37/174): cephadm-19.0.0-1855.g21b3bd67.el9.noa 8.0 MB/s | 748 kB 00:00 2024-03-24T10:27:41.046 INFO:teuthology.orchestra.run.smithi117.stdout:(38/174): cephfs-top-19.0.0-1855.g21b3bd67.el9. 313 kB/s | 18 kB 00:00 2024-03-24T10:27:41.108 INFO:teuthology.orchestra.run.smithi116.stdout:(100/174): openssl-devel-3.0.7-27.el9.x86_64.rp 9.9 MB/s | 4.1 MB 00:00 2024-03-24T10:27:41.223 INFO:teuthology.orchestra.run.smithi023.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1855.g21b3b 3.9 MB/s | 1.8 MB 00:00 2024-03-24T10:27:41.241 INFO:teuthology.orchestra.run.smithi116.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 702 kB/s | 216 kB 00:00 2024-03-24T10:27:41.263 INFO:teuthology.orchestra.run.smithi117.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.8 MB/s | 662 kB 00:00 2024-03-24T10:27:41.305 INFO:teuthology.orchestra.run.smithi117.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 3.5 MB/s | 920 kB 00:00 2024-03-24T10:27:41.307 INFO:teuthology.orchestra.run.smithi023.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1855.g21 2.9 MB/s | 248 kB 00:00 2024-03-24T10:27:41.325 INFO:teuthology.orchestra.run.smithi116.stdout:(102/174): qatlib-service-23.11.0-1.el9.x86_64. 110 kB/s | 35 kB 00:00 2024-03-24T10:27:41.330 INFO:teuthology.orchestra.run.smithi117.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-24T10:27:41.356 INFO:teuthology.orchestra.run.smithi117.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-24T10:27:41.375 INFO:teuthology.orchestra.run.smithi116.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 167 kB/s | 44 kB 00:00 2024-03-24T10:27:41.381 INFO:teuthology.orchestra.run.smithi117.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 637 kB/s | 32 kB 00:00 2024-03-24T10:27:41.382 INFO:teuthology.orchestra.run.smithi023.stdout:(33/174): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 657 kB/s | 49 kB 00:00 2024-03-24T10:27:41.431 INFO:teuthology.orchestra.run.smithi117.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 353 kB/s | 17 kB 00:00 2024-03-24T10:27:41.449 INFO:teuthology.orchestra.run.smithi023.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1855.g2 230 kB/s | 15 kB 00:00 2024-03-24T10:27:41.482 INFO:teuthology.orchestra.run.smithi117.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-03-24T10:27:41.507 INFO:teuthology.orchestra.run.smithi117.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 135 kB/s | 10 kB 00:00 2024-03-24T10:27:41.509 INFO:teuthology.orchestra.run.smithi116.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 352 kB/s | 94 kB 00:00 2024-03-24T10:27:41.532 INFO:teuthology.orchestra.run.smithi117.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-24T10:27:41.549 INFO:teuthology.orchestra.run.smithi023.stdout:(35/174): ceph-volume-19.0.0-1855.g21b3bd67.el9 2.6 MB/s | 264 kB 00:00 2024-03-24T10:27:41.550 INFO:teuthology.orchestra.run.smithi116.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 111 kB/s | 22 kB 00:00 2024-03-24T10:27:41.557 INFO:teuthology.orchestra.run.smithi117.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-24T10:27:41.583 INFO:teuthology.orchestra.run.smithi117.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-24T10:27:41.617 INFO:teuthology.orchestra.run.smithi117.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-24T10:27:41.642 INFO:teuthology.orchestra.run.smithi117.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 808 kB/s | 46 kB 00:00 2024-03-24T10:27:41.667 INFO:teuthology.orchestra.run.smithi117.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-24T10:27:41.692 INFO:teuthology.orchestra.run.smithi117.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-24T10:27:41.692 INFO:teuthology.orchestra.run.smithi116.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-03-24T10:27:41.700 INFO:teuthology.orchestra.run.smithi023.stdout:(36/174): cephadm-19.0.0-1855.g21b3bd67.el9.noa 4.9 MB/s | 748 kB 00:00 2024-03-24T10:27:41.718 INFO:teuthology.orchestra.run.smithi117.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-24T10:27:41.743 INFO:teuthology.orchestra.run.smithi117.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 687 kB/s | 34 kB 00:00 2024-03-24T10:27:41.784 INFO:teuthology.orchestra.run.smithi023.stdout:(37/174): cephfs-top-19.0.0-1855.g21b3bd67.el9. 219 kB/s | 18 kB 00:00 2024-03-24T10:27:41.843 INFO:teuthology.orchestra.run.smithi117.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-24T10:27:41.880 INFO:teuthology.orchestra.run.smithi117.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-03-24T10:27:41.913 INFO:teuthology.orchestra.run.smithi117.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-24T10:27:41.934 INFO:teuthology.orchestra.run.smithi023.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 662 kB 00:00 2024-03-24T10:27:41.938 INFO:teuthology.orchestra.run.smithi117.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-24T10:27:41.968 INFO:teuthology.orchestra.run.smithi116.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-03-24T10:27:42.034 INFO:teuthology.orchestra.run.smithi023.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 9.0 MB/s | 920 kB 00:00 2024-03-24T10:27:42.064 INFO:teuthology.orchestra.run.smithi117.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-03-24T10:27:42.068 INFO:teuthology.orchestra.run.smithi116.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 983 kB/s | 548 kB 00:00 2024-03-24T10:27:42.068 INFO:teuthology.orchestra.run.smithi023.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-24T10:27:42.098 INFO:teuthology.orchestra.run.smithi117.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.5 MB/s | 1.0 MB 00:00 2024-03-24T10:27:42.110 INFO:teuthology.orchestra.run.smithi023.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-24T10:27:42.123 INFO:teuthology.orchestra.run.smithi117.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-24T10:27:42.144 INFO:teuthology.orchestra.run.smithi023.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 944 kB/s | 32 kB 00:00 2024-03-24T10:27:42.169 INFO:teuthology.orchestra.run.smithi116.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 256 kB/s | 26 kB 00:00 2024-03-24T10:27:42.245 INFO:teuthology.orchestra.run.smithi116.stdout:(110/174): gperftools-libs-2.9.1-2.el9.x86_64.r 445 kB/s | 309 kB 00:00 2024-03-24T10:27:42.256 INFO:teuthology.orchestra.run.smithi117.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 37 MB/s | 5.8 MB 00:00 2024-03-24T10:27:42.270 INFO:teuthology.orchestra.run.smithi116.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-03-24T10:27:42.277 INFO:teuthology.orchestra.run.smithi023.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 22 MB/s | 3.0 MB 00:00 2024-03-24T10:27:42.281 INFO:teuthology.orchestra.run.smithi117.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 156 kB/s | 24 kB 00:00 2024-03-24T10:27:42.302 INFO:teuthology.orchestra.run.smithi023.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 691 kB/s | 17 kB 00:00 2024-03-24T10:27:42.315 INFO:teuthology.orchestra.run.smithi117.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 547 kB/s | 18 kB 00:00 2024-03-24T10:27:42.328 INFO:teuthology.orchestra.run.smithi023.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 402 kB/s | 10 kB 00:00 2024-03-24T10:27:42.340 INFO:teuthology.orchestra.run.smithi117.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-24T10:27:42.361 INFO:teuthology.orchestra.run.smithi023.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-24T10:27:42.366 INFO:teuthology.orchestra.run.smithi117.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 630 kB/s | 32 kB 00:00 2024-03-24T10:27:42.370 INFO:teuthology.orchestra.run.smithi116.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-24T10:27:42.395 INFO:teuthology.orchestra.run.smithi023.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-24T10:27:42.404 INFO:teuthology.orchestra.run.smithi116.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 424 kB/s | 67 kB 00:00 2024-03-24T10:27:42.466 INFO:teuthology.orchestra.run.smithi117.stdout:(68/174): ceph-mgr-diskprediction-local-19.0.0- 3.9 MB/s | 7.4 MB 00:01 2024-03-24T10:27:42.495 INFO:teuthology.orchestra.run.smithi023.stdout:(48/174): ceph-mgr-diskprediction-local-19.0.0- 5.0 MB/s | 7.4 MB 00:01 2024-03-24T10:27:42.529 INFO:teuthology.orchestra.run.smithi023.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 661 kB 00:00 2024-03-24T10:27:42.533 INFO:teuthology.orchestra.run.smithi117.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-24T10:27:42.554 INFO:teuthology.orchestra.run.smithi023.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-24T10:27:42.554 INFO:teuthology.orchestra.run.smithi116.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-03-24T10:27:42.566 INFO:teuthology.orchestra.run.smithi117.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-03-24T10:27:42.581 INFO:teuthology.orchestra.run.smithi023.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 940 kB/s | 46 kB 00:00 2024-03-24T10:27:42.591 INFO:teuthology.orchestra.run.smithi117.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2024-03-24T10:27:42.605 INFO:teuthology.orchestra.run.smithi023.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-24T10:27:42.616 INFO:teuthology.orchestra.run.smithi117.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 831 kB/s | 41 kB 00:00 2024-03-24T10:27:42.630 INFO:teuthology.orchestra.run.smithi023.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-24T10:27:42.633 INFO:teuthology.orchestra.run.smithi117.stdout:(73/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-24T10:27:42.655 INFO:teuthology.orchestra.run.smithi023.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-24T10:27:42.659 INFO:teuthology.orchestra.run.smithi117.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-24T10:27:42.663 INFO:teuthology.orchestra.run.smithi116.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-03-24T10:27:42.684 INFO:teuthology.orchestra.run.smithi117.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-24T10:27:42.689 INFO:teuthology.orchestra.run.smithi023.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 586 kB/s | 34 kB 00:00 2024-03-24T10:27:42.743 INFO:teuthology.orchestra.run.smithi117.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 569 kB/s | 47 kB 00:00 2024-03-24T10:27:42.788 INFO:teuthology.orchestra.run.smithi116.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 258 kB/s | 32 kB 00:00 2024-03-24T10:27:42.810 INFO:teuthology.orchestra.run.smithi117.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 1.4 MB/s | 211 kB 00:00 2024-03-24T10:27:42.835 INFO:teuthology.orchestra.run.smithi117.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 284 kB/s | 43 kB 00:00 2024-03-24T10:27:42.881 INFO:teuthology.orchestra.run.smithi023.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2024-03-24T10:27:42.896 INFO:teuthology.orchestra.run.smithi116.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-24T10:27:42.923 INFO:teuthology.orchestra.run.smithi023.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 6.5 MB/s | 1.5 MB 00:00 2024-03-24T10:27:42.957 INFO:teuthology.orchestra.run.smithi023.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-24T10:27:42.982 INFO:teuthology.orchestra.run.smithi023.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-24T10:27:43.005 INFO:teuthology.orchestra.run.smithi116.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-24T10:27:43.106 INFO:teuthology.orchestra.run.smithi116.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-03-24T10:27:43.131 INFO:teuthology.orchestra.run.smithi116.stdout:(120/174): python3-asyncssh-2.13.2-5.el9.noarch 754 kB/s | 548 kB 00:00 2024-03-24T10:27:43.148 INFO:teuthology.orchestra.run.smithi023.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-24T10:27:43.160 INFO:teuthology.orchestra.run.smithi117.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-24T10:27:43.177 INFO:teuthology.orchestra.run.smithi117.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 121 kB/s | 41 kB 00:00 2024-03-24T10:27:43.190 INFO:teuthology.orchestra.run.smithi023.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-24T10:27:43.194 INFO:teuthology.orchestra.run.smithi117.stdout:(81/174): python3-toml-0.10.2-6.el9.noarch.rpm 109 kB/s | 39 kB 00:00 2024-03-24T10:27:43.206 INFO:teuthology.orchestra.run.smithi116.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-03-24T10:27:43.216 INFO:teuthology.orchestra.run.smithi023.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 78 kB 00:00 2024-03-24T10:27:43.219 INFO:teuthology.orchestra.run.smithi117.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-24T10:27:43.231 INFO:teuthology.orchestra.run.smithi116.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-03-24T10:27:43.236 INFO:teuthology.orchestra.run.smithi117.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 971 kB/s | 56 kB 00:00 2024-03-24T10:27:43.253 INFO:teuthology.orchestra.run.smithi117.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-24T10:27:43.270 INFO:teuthology.orchestra.run.smithi117.stdout:(85/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-24T10:27:43.295 INFO:teuthology.orchestra.run.smithi117.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-24T10:27:43.307 INFO:teuthology.orchestra.run.smithi116.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-24T10:27:43.320 INFO:teuthology.orchestra.run.smithi117.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-24T10:27:43.346 INFO:teuthology.orchestra.run.smithi117.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-24T10:27:43.358 INFO:teuthology.orchestra.run.smithi023.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-24T10:27:43.379 INFO:teuthology.orchestra.run.smithi117.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-24T10:27:43.383 INFO:teuthology.orchestra.run.smithi023.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 148 kB/s | 24 kB 00:00 2024-03-24T10:27:43.404 INFO:teuthology.orchestra.run.smithi117.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-24T10:27:43.407 INFO:teuthology.orchestra.run.smithi116.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-24T10:27:43.408 INFO:teuthology.orchestra.run.smithi023.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 359 kB/s | 18 kB 00:00 2024-03-24T10:27:43.430 INFO:teuthology.orchestra.run.smithi117.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-24T10:27:43.434 INFO:teuthology.orchestra.run.smithi023.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-24T10:27:43.459 INFO:teuthology.orchestra.run.smithi023.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 633 kB/s | 32 kB 00:00 2024-03-24T10:27:43.516 INFO:teuthology.orchestra.run.smithi116.stdout:(125/174): libarrow-9.0.0-10.el9.x86_64.rpm 2.9 MB/s | 4.4 MB 00:01 2024-03-24T10:27:43.541 INFO:teuthology.orchestra.run.smithi116.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2024-03-24T10:27:43.568 INFO:teuthology.orchestra.run.smithi023.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-24T10:27:43.593 INFO:teuthology.orchestra.run.smithi023.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2024-03-24T10:27:43.608 INFO:teuthology.orchestra.run.smithi116.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-03-24T10:27:43.618 INFO:teuthology.orchestra.run.smithi023.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-24T10:27:43.633 INFO:teuthology.orchestra.run.smithi116.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-03-24T10:27:43.644 INFO:teuthology.orchestra.run.smithi023.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-24T10:27:43.655 INFO:teuthology.orchestra.run.smithi117.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 750 kB/s | 187 kB 00:00 2024-03-24T10:27:43.669 INFO:teuthology.orchestra.run.smithi023.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 822 kB/s | 41 kB 00:00 2024-03-24T10:27:43.694 INFO:teuthology.orchestra.run.smithi023.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-03-24T10:27:43.701 INFO:teuthology.orchestra.run.smithi116.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-03-24T10:27:43.713 INFO:teuthology.orchestra.run.smithi117.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 690 kB/s | 195 kB 00:00 2024-03-24T10:27:43.719 INFO:teuthology.orchestra.run.smithi023.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-24T10:27:43.725 INFO:teuthology.orchestra.run.smithi116.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-03-24T10:27:43.738 INFO:teuthology.orchestra.run.smithi117.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-24T10:27:43.744 INFO:teuthology.orchestra.run.smithi023.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-24T10:27:43.770 INFO:teuthology.orchestra.run.smithi023.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 951 kB/s | 47 kB 00:00 2024-03-24T10:27:43.772 INFO:teuthology.orchestra.run.smithi117.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-24T10:27:43.795 INFO:teuthology.orchestra.run.smithi023.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 852 kB/s | 43 kB 00:00 2024-03-24T10:27:43.809 INFO:teuthology.orchestra.run.smithi116.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 9.4 MB/s | 1.0 MB 00:00 2024-03-24T10:27:43.834 INFO:teuthology.orchestra.run.smithi116.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2024-03-24T10:27:43.901 INFO:teuthology.orchestra.run.smithi116.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 859 kB/s | 79 kB 00:00 2024-03-24T10:27:43.943 INFO:teuthology.orchestra.run.smithi116.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 531 kB/s | 58 kB 00:00 2024-03-24T10:27:43.962 INFO:teuthology.orchestra.run.smithi023.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 234 kB/s | 39 kB 00:00 2024-03-24T10:27:43.997 INFO:teuthology.orchestra.run.smithi117.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 76 kB/s | 21 kB 00:00 2024-03-24T10:27:44.001 INFO:teuthology.orchestra.run.smithi116.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-24T10:27:44.022 INFO:teuthology.orchestra.run.smithi117.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 90 kB/s | 25 kB 00:00 2024-03-24T10:27:44.035 INFO:teuthology.orchestra.run.smithi116.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2024-03-24T10:27:44.094 INFO:teuthology.orchestra.run.smithi116.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 978 kB/s | 90 kB 00:00 2024-03-24T10:27:44.121 INFO:teuthology.orchestra.run.smithi023.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-24T10:27:44.127 INFO:teuthology.orchestra.run.smithi116.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 306 kB/s | 28 kB 00:00 2024-03-24T10:27:44.146 INFO:teuthology.orchestra.run.smithi023.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 226 kB/s | 41 kB 00:00 2024-03-24T10:27:44.171 INFO:teuthology.orchestra.run.smithi023.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-24T10:27:44.186 INFO:teuthology.orchestra.run.smithi116.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 335 kB/s | 31 kB 00:00 2024-03-24T10:27:44.188 INFO:teuthology.orchestra.run.smithi023.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-24T10:27:44.205 INFO:teuthology.orchestra.run.smithi023.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-24T10:27:44.222 INFO:teuthology.orchestra.run.smithi023.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-24T10:27:44.228 INFO:teuthology.orchestra.run.smithi116.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-24T10:27:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-24T10:27:44.273 INFO:teuthology.orchestra.run.smithi023.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-24T10:27:44.278 INFO:teuthology.orchestra.run.smithi116.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 641 kB/s | 59 kB 00:00 2024-03-24T10:27:44.306 INFO:teuthology.orchestra.run.smithi023.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-24T10:27:44.320 INFO:teuthology.orchestra.run.smithi116.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-03-24T10:27:44.331 INFO:teuthology.orchestra.run.smithi023.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-24T10:27:44.357 INFO:teuthology.orchestra.run.smithi023.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-24T10:27:44.373 INFO:teuthology.orchestra.run.smithi117.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 708 kB/s | 248 kB 00:00 2024-03-24T10:27:44.379 INFO:teuthology.orchestra.run.smithi116.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 509 kB/s | 51 kB 00:00 2024-03-24T10:27:44.382 INFO:teuthology.orchestra.run.smithi023.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-24T10:27:44.421 INFO:teuthology.orchestra.run.smithi116.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-24T10:27:44.423 INFO:teuthology.orchestra.run.smithi117.stdout:(99/174): python3-devel-3.9.17-2.el9.x86_64.rpm 577 kB/s | 245 kB 00:00 2024-03-24T10:27:44.471 INFO:teuthology.orchestra.run.smithi116.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 976 kB/s | 90 kB 00:00 2024-03-24T10:27:44.482 INFO:teuthology.orchestra.run.smithi117.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.9 MB/s | 216 kB 00:00 2024-03-24T10:27:44.513 INFO:teuthology.orchestra.run.smithi116.stdout:(146/174): python3-grpcio-1.46.7-10.el9.x86_64. 1.6 MB/s | 2.0 MB 00:01 2024-03-24T10:27:44.524 INFO:teuthology.orchestra.run.smithi117.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 347 kB/s | 35 kB 00:00 2024-03-24T10:27:44.524 INFO:teuthology.orchestra.run.smithi023.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-24T10:27:44.555 INFO:teuthology.orchestra.run.smithi116.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 239 kB/s | 20 kB 00:00 2024-03-24T10:27:44.591 INFO:teuthology.orchestra.run.smithi117.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 410 kB/s | 44 kB 00:00 2024-03-24T10:27:44.622 INFO:teuthology.orchestra.run.smithi116.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.1 MB/s | 427 kB 00:00 2024-03-24T10:27:44.641 INFO:teuthology.orchestra.run.smithi117.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 803 kB/s | 94 kB 00:00 2024-03-24T10:27:44.647 INFO:teuthology.orchestra.run.smithi116.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-24T10:27:44.694 INFO:teuthology.orchestra.run.smithi023.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-24T10:27:44.724 INFO:teuthology.orchestra.run.smithi023.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-03-24T10:27:44.756 INFO:teuthology.orchestra.run.smithi116.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-03-24T10:27:44.758 INFO:teuthology.orchestra.run.smithi023.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.9 MB/s | 125 kB 00:00 2024-03-24T10:27:44.766 INFO:teuthology.orchestra.run.smithi117.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-03-24T10:27:44.817 INFO:teuthology.orchestra.run.smithi023.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 233 kB/s | 21 kB 00:00 2024-03-24T10:27:44.842 INFO:teuthology.orchestra.run.smithi023.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 305 kB/s | 25 kB 00:00 2024-03-24T10:27:44.850 INFO:teuthology.orchestra.run.smithi117.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.8 MB/s | 4.1 MB 00:01 2024-03-24T10:27:44.873 INFO:teuthology.orchestra.run.smithi116.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.7 MB/s | 321 kB 00:00 2024-03-24T10:27:44.951 INFO:teuthology.orchestra.run.smithi023.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-03-24T10:27:44.966 INFO:teuthology.orchestra.run.smithi117.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.6 MB/s | 863 kB 00:00 2024-03-24T10:27:45.001 INFO:teuthology.orchestra.run.smithi023.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-03-24T10:27:45.076 INFO:teuthology.orchestra.run.smithi023.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 16 MB/s | 4.1 MB 00:00 2024-03-24T10:27:45.098 INFO:teuthology.orchestra.run.smithi116.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 10 MB/s | 2.3 MB 00:00 2024-03-24T10:27:45.102 INFO:teuthology.orchestra.run.smithi023.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.1 MB/s | 216 kB 00:00 2024-03-24T10:27:45.127 INFO:teuthology.orchestra.run.smithi023.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 695 kB/s | 35 kB 00:00 2024-03-24T10:27:45.152 INFO:teuthology.orchestra.run.smithi023.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 888 kB/s | 44 kB 00:00 2024-03-24T10:27:45.173 INFO:teuthology.orchestra.run.smithi116.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 25 MB/s | 1.8 MB 00:00 2024-03-24T10:27:45.178 INFO:teuthology.orchestra.run.smithi023.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-24T10:27:45.217 INFO:teuthology.orchestra.run.smithi117.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-03-24T10:27:45.223 INFO:teuthology.orchestra.run.smithi116.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 844 kB/s | 42 kB 00:00 2024-03-24T10:27:45.290 INFO:teuthology.orchestra.run.smithi116.stdout:(155/174): librados2-19.0.0-1855.g21b3bd67.el9. 5.1 MB/s | 3.4 MB 00:00 2024-03-24T10:27:45.341 INFO:teuthology.orchestra.run.smithi116.stdout:(156/174): librbd1-19.0.0-1855.g21b3bd67.el9.x8 4.5 MB/s | 3.1 MB 00:00 2024-03-24T10:27:45.353 INFO:teuthology.orchestra.run.smithi023.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 111 kB/s | 22 kB 00:00 2024-03-24T10:27:45.366 INFO:teuthology.orchestra.run.smithi116.stdout:(157/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.7 MB/s | 248 kB 00:00 2024-03-24T10:27:45.375 INFO:teuthology.orchestra.run.smithi117.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 900 kB/s | 548 kB 00:00 2024-03-24T10:27:45.391 INFO:teuthology.orchestra.run.smithi116.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 710 kB/s | 71 kB 00:00 2024-03-24T10:27:45.417 INFO:teuthology.orchestra.run.smithi117.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 545 kB/s | 309 kB 00:00 2024-03-24T10:27:45.425 INFO:teuthology.orchestra.run.smithi116.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.2 MB/s | 192 kB 00:00 2024-03-24T10:27:45.450 INFO:teuthology.orchestra.run.smithi116.stdout:(160/174): elfutils-debuginfod-client-0.190-2.e 334 kB/s | 36 kB 00:00 2024-03-24T10:27:45.476 INFO:teuthology.orchestra.run.smithi117.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-03-24T10:27:45.492 INFO:teuthology.orchestra.run.smithi116.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 345 kB/s | 23 kB 00:00 2024-03-24T10:27:45.518 INFO:teuthology.orchestra.run.smithi117.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 483 kB/s | 49 kB 00:00 2024-03-24T10:27:45.542 INFO:teuthology.orchestra.run.smithi116.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.7 MB/s | 256 kB 00:00 2024-03-24T10:27:45.568 INFO:teuthology.orchestra.run.smithi116.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 926 kB/s | 108 kB 00:00 2024-03-24T10:27:45.576 INFO:teuthology.orchestra.run.smithi117.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-03-24T10:27:45.593 INFO:teuthology.orchestra.run.smithi116.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-24T10:27:45.643 INFO:teuthology.orchestra.run.smithi116.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 849 kB/s | 64 kB 00:00 2024-03-24T10:27:45.644 INFO:teuthology.orchestra.run.smithi117.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-03-24T10:27:45.668 INFO:teuthology.orchestra.run.smithi116.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 382 kB/s | 29 kB 00:00 2024-03-24T10:27:45.694 INFO:teuthology.orchestra.run.smithi116.stdout:(167/174): libmount-2.37.4-18.el9.x86_64.rpm 899 kB/s | 136 kB 00:00 2024-03-24T10:27:45.769 INFO:teuthology.orchestra.run.smithi116.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 22 MB/s | 2.2 MB 00:00 2024-03-24T10:27:45.771 INFO:teuthology.orchestra.run.smithi117.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.2 MB/s | 838 kB 00:00 2024-03-24T10:27:45.830 INFO:teuthology.orchestra.run.smithi117.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 2.9 MB/s | 548 kB 00:00 2024-03-24T10:27:45.871 INFO:teuthology.orchestra.run.smithi117.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 434 kB/s | 43 kB 00:00 2024-03-24T10:27:45.930 INFO:teuthology.orchestra.run.smithi117.stdout:(117/174): libarrow-9.0.0-10.el9.x86_64.rpm 6.2 MB/s | 4.4 MB 00:00 2024-03-24T10:27:45.955 INFO:teuthology.orchestra.run.smithi117.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 257 kB/s | 32 kB 00:00 2024-03-24T10:27:45.981 INFO:teuthology.orchestra.run.smithi117.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-24T10:27:46.031 INFO:teuthology.orchestra.run.smithi117.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-03-24T10:27:46.073 INFO:teuthology.orchestra.run.smithi117.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-03-24T10:27:46.098 INFO:teuthology.orchestra.run.smithi117.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-03-24T10:27:46.132 INFO:teuthology.orchestra.run.smithi117.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 351 kB/s | 35 kB 00:00 2024-03-24T10:27:46.136 INFO:teuthology.orchestra.run.smithi023.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 700 kB/s | 548 kB 00:00 2024-03-24T10:27:46.170 INFO:teuthology.orchestra.run.smithi023.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 870 kB/s | 863 kB 00:00 2024-03-24T10:27:46.174 INFO:teuthology.orchestra.run.smithi117.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-24T10:27:46.232 INFO:teuthology.orchestra.run.smithi117.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-24T10:27:46.274 INFO:teuthology.orchestra.run.smithi117.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-03-24T10:27:46.333 INFO:teuthology.orchestra.run.smithi117.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-03-24T10:27:46.358 INFO:teuthology.orchestra.run.smithi117.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 141 kB/s | 18 kB 00:00 2024-03-24T10:27:46.383 INFO:teuthology.orchestra.run.smithi117.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-03-24T10:27:46.394 INFO:teuthology.orchestra.run.smithi116.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 744 kB/s | 465 kB 00:00 2024-03-24T10:27:46.434 INFO:teuthology.orchestra.run.smithi117.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-03-24T10:27:46.436 INFO:teuthology.orchestra.run.smithi116.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 3.1 MB/s | 2.3 MB 00:00 2024-03-24T10:27:46.459 INFO:teuthology.orchestra.run.smithi117.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-03-24T10:27:46.461 INFO:teuthology.orchestra.run.smithi116.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 465 kB/s | 31 kB 00:00 2024-03-24T10:27:46.526 INFO:teuthology.orchestra.run.smithi117.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-03-24T10:27:46.551 INFO:teuthology.orchestra.run.smithi117.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 395 kB/s | 46 kB 00:00 2024-03-24T10:27:46.576 INFO:teuthology.orchestra.run.smithi117.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 672 kB/s | 79 kB 00:00 2024-03-24T10:27:46.589 INFO:teuthology.orchestra.run.smithi116.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 585 kB/s | 74 kB 00:00 2024-03-24T10:27:46.627 INFO:teuthology.orchestra.run.smithi117.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-03-24T10:27:46.652 INFO:teuthology.orchestra.run.smithi117.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-24T10:27:46.654 INFO:teuthology.orchestra.run.smithi023.stdout:(107/174): ceph-test-19.0.0-1855.g21b3bd67.el9. 5.7 MB/s | 47 MB 00:08 2024-03-24T10:27:46.677 INFO:teuthology.orchestra.run.smithi117.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-03-24T10:27:46.679 INFO:teuthology.orchestra.run.smithi023.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 570 kB/s | 309 kB 00:00 2024-03-24T10:27:46.727 INFO:teuthology.orchestra.run.smithi117.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 897 kB/s | 90 kB 00:00 2024-03-24T10:27:46.753 INFO:teuthology.orchestra.run.smithi117.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-03-24T10:27:46.764 INFO:teuthology.orchestra.run.smithi116.stdout:(173/174): openssl-3.0.7-27.el9.x86_64.rpm 1.1 MB/s | 1.2 MB 00:01 2024-03-24T10:27:46.778 INFO:teuthology.orchestra.run.smithi117.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-03-24T10:27:46.779 INFO:teuthology.orchestra.run.smithi023.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 261 kB/s | 26 kB 00:00 2024-03-24T10:27:46.813 INFO:teuthology.orchestra.run.smithi023.stdout:(110/174): grpc-data-1.46.7-10.el9.noarch.rpm 30 kB/s | 19 kB 00:00 2024-03-24T10:27:46.828 INFO:teuthology.orchestra.run.smithi117.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-24T10:27:46.853 INFO:teuthology.orchestra.run.smithi117.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-03-24T10:27:46.855 INFO:teuthology.orchestra.run.smithi023.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 645 kB/s | 49 kB 00:00 2024-03-24T10:27:46.878 INFO:teuthology.orchestra.run.smithi117.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-03-24T10:27:46.897 INFO:teuthology.orchestra.run.smithi116.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 2.3 MB/s | 1.1 MB 00:00 2024-03-24T10:27:46.902 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:27:46.903 INFO:teuthology.orchestra.run.smithi116.stdout:Total 12 MB/s | 224 MB 00:18 2024-03-24T10:27:46.929 INFO:teuthology.orchestra.run.smithi117.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-03-24T10:27:46.955 INFO:teuthology.orchestra.run.smithi023.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-24T10:27:46.963 INFO:teuthology.orchestra.run.smithi117.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-24T10:27:46.988 INFO:teuthology.orchestra.run.smithi117.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 823 kB/s | 90 kB 00:00 2024-03-24T10:27:47.013 INFO:teuthology.orchestra.run.smithi023.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 336 kB/s | 67 kB 00:00 2024-03-24T10:27:47.047 INFO:teuthology.orchestra.run.smithi117.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-24T10:27:47.072 INFO:teuthology.orchestra.run.smithi117.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-03-24T10:27:47.097 INFO:teuthology.orchestra.run.smithi117.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-24T10:27:47.155 INFO:teuthology.orchestra.run.smithi023.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-03-24T10:27:47.214 INFO:teuthology.orchestra.run.smithi117.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-03-24T10:27:47.247 INFO:teuthology.orchestra.run.smithi023.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 474 kB/s | 43 kB 00:00 2024-03-24T10:27:47.339 INFO:teuthology.orchestra.run.smithi023.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 351 kB/s | 32 kB 00:00 2024-03-24T10:27:47.361 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-24T10:27:47.440 INFO:teuthology.orchestra.run.smithi023.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-03-24T10:27:47.489 INFO:teuthology.orchestra.run.smithi117.stdout:(151/174): librbd1-19.0.0-1855.g21b3bd67.el9.x8 8.0 MB/s | 3.1 MB 00:00 2024-03-24T10:27:47.532 INFO:teuthology.orchestra.run.smithi023.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-03-24T10:27:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:(119/174): python3-asyncssh-2.13.2-5.el9.noarch 1.0 MB/s | 548 kB 00:00 2024-03-24T10:27:47.598 INFO:teuthology.orchestra.run.smithi117.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 839 kB/s | 321 kB 00:00 2024-03-24T10:27:47.624 INFO:teuthology.orchestra.run.smithi023.stdout:(120/174): libarrow-9.0.0-10.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-03-24T10:27:47.633 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-24T10:27:47.633 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-24T10:27:47.649 INFO:teuthology.orchestra.run.smithi023.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-03-24T10:27:47.691 INFO:teuthology.orchestra.run.smithi023.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 529 kB/s | 35 kB 00:00 2024-03-24T10:27:47.716 INFO:teuthology.orchestra.run.smithi023.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.9 MB/s | 303 kB 00:00 2024-03-24T10:27:47.741 INFO:teuthology.orchestra.run.smithi023.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.1 MB/s | 203 kB 00:00 2024-03-24T10:27:47.900 INFO:teuthology.orchestra.run.smithi023.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.7 MB/s | 2.0 MB 00:00 2024-03-24T10:27:47.917 INFO:teuthology.orchestra.run.smithi023.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 61 kB/s | 11 kB 00:00 2024-03-24T10:27:47.942 INFO:teuthology.orchestra.run.smithi023.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 639 kB/s | 144 kB 00:00 2024-03-24T10:27:47.969 INFO:teuthology.orchestra.run.smithi023.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 264 kB/s | 18 kB 00:00 2024-03-24T10:27:47.985 INFO:teuthology.orchestra.run.smithi023.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 345 kB/s | 23 kB 00:00 2024-03-24T10:27:48.001 INFO:teuthology.orchestra.run.smithi023.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 330 kB/s | 19 kB 00:00 2024-03-24T10:27:48.027 INFO:teuthology.orchestra.run.smithi023.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 334 kB/s | 20 kB 00:00 2024-03-24T10:27:48.056 INFO:teuthology.orchestra.run.smithi117.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 4.2 MB/s | 1.8 MB 00:00 2024-03-24T10:27:48.085 INFO:teuthology.orchestra.run.smithi023.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-03-24T10:27:48.111 INFO:teuthology.orchestra.run.smithi023.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2024-03-24T10:27:48.131 INFO:teuthology.orchestra.run.smithi117.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 563 kB/s | 42 kB 00:00 2024-03-24T10:27:48.136 INFO:teuthology.orchestra.run.smithi023.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 725 kB/s | 79 kB 00:00 2024-03-24T10:27:48.153 INFO:teuthology.orchestra.run.smithi023.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 863 kB/s | 58 kB 00:00 2024-03-24T10:27:48.186 INFO:teuthology.orchestra.run.smithi023.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-03-24T10:27:48.199 INFO:teuthology.orchestra.run.smithi117.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 3.6 MB/s | 248 kB 00:00 2024-03-24T10:27:48.212 INFO:teuthology.orchestra.run.smithi023.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-03-24T10:27:48.237 INFO:teuthology.orchestra.run.smithi023.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-03-24T10:27:48.262 INFO:teuthology.orchestra.run.smithi023.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-03-24T10:27:48.265 INFO:teuthology.orchestra.run.smithi117.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-03-24T10:27:48.305 INFO:teuthology.orchestra.run.smithi023.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 332 kB/s | 31 kB 00:00 2024-03-24T10:27:48.329 INFO:teuthology.orchestra.run.smithi023.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 882 kB/s | 59 kB 00:00 2024-03-24T10:27:48.355 INFO:teuthology.orchestra.run.smithi023.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-24T10:27:48.357 INFO:teuthology.orchestra.run.smithi117.stdout:(157/174): librados2-19.0.0-1855.g21b3bd67.el9. 2.6 MB/s | 3.4 MB 00:01 2024-03-24T10:27:48.380 INFO:teuthology.orchestra.run.smithi023.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 477 kB/s | 36 kB 00:00 2024-03-24T10:27:48.382 INFO:teuthology.orchestra.run.smithi117.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 312 kB/s | 36 kB 00:00 2024-03-24T10:27:48.406 INFO:teuthology.orchestra.run.smithi023.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-03-24T10:27:48.424 INFO:teuthology.orchestra.run.smithi117.stdout:(159/174): NetworkManager-1.47.2-1.el9.x86_64.r 2.5 MB/s | 2.3 MB 00:00 2024-03-24T10:27:48.430 INFO:teuthology.orchestra.run.smithi023.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-24T10:27:48.450 INFO:teuthology.orchestra.run.smithi117.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-03-24T10:27:48.456 INFO:teuthology.orchestra.run.smithi023.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-24T10:27:48.481 INFO:teuthology.orchestra.run.smithi023.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-03-24T10:27:48.500 INFO:teuthology.orchestra.run.smithi117.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.1 MB/s | 256 kB 00:00 2024-03-24T10:27:48.506 INFO:teuthology.orchestra.run.smithi023.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-24T10:27:48.531 INFO:teuthology.orchestra.run.smithi023.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-24T10:27:48.550 INFO:teuthology.orchestra.run.smithi117.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 184 kB/s | 23 kB 00:00 2024-03-24T10:27:48.617 INFO:teuthology.orchestra.run.smithi117.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.7 MB/s | 155 kB 00:00 2024-03-24T10:27:48.648 INFO:teuthology.orchestra.run.smithi023.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-03-24T10:27:48.701 INFO:teuthology.orchestra.run.smithi117.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 903 kB/s | 136 kB 00:00 2024-03-24T10:27:48.723 INFO:teuthology.orchestra.run.smithi023.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 321 kB 00:00 2024-03-24T10:27:48.734 INFO:teuthology.orchestra.run.smithi117.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 547 kB/s | 64 kB 00:00 2024-03-24T10:27:48.760 INFO:teuthology.orchestra.run.smithi117.stdout:(166/174): libblkid-2.37.4-18.el9.x86_64.rpm 350 kB/s | 108 kB 00:00 2024-03-24T10:27:48.785 INFO:teuthology.orchestra.run.smithi117.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 342 kB/s | 29 kB 00:00 2024-03-24T10:27:48.844 INFO:teuthology.orchestra.run.smithi117.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 11 MB/s | 1.2 MB 00:00 2024-03-24T10:27:48.927 INFO:teuthology.orchestra.run.smithi117.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 5.5 MB/s | 465 kB 00:00 2024-03-24T10:27:49.002 INFO:teuthology.orchestra.run.smithi117.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 415 kB/s | 31 kB 00:00 2024-03-24T10:27:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:(171/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.6 MB/s | 2.2 MB 00:00 2024-03-24T10:27:49.128 INFO:teuthology.orchestra.run.smithi117.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.6 MB/s | 1.1 MB 00:00 2024-03-24T10:27:49.224 INFO:teuthology.orchestra.run.smithi023.stdout:(152/174): librbd1-19.0.0-1855.g21b3bd67.el9.x8 4.5 MB/s | 3.1 MB 00:00 2024-03-24T10:27:49.274 INFO:teuthology.orchestra.run.smithi023.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 4.2 MB/s | 2.3 MB 00:00 2024-03-24T10:27:49.328 INFO:teuthology.orchestra.run.smithi117.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.2 MB/s | 2.3 MB 00:00 2024-03-24T10:27:49.333 INFO:teuthology.orchestra.run.smithi023.stdout:(154/174): librados2-19.0.0-1855.g21b3bd67.el9. 4.1 MB/s | 3.4 MB 00:00 2024-03-24T10:27:49.342 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-24T10:27:49.343 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-24T10:27:49.366 INFO:teuthology.orchestra.run.smithi023.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 459 kB/s | 42 kB 00:00 2024-03-24T10:27:49.437 INFO:teuthology.orchestra.run.smithi117.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 212 kB/s | 74 kB 00:00 2024-03-24T10:27:49.444 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:27:49.445 INFO:teuthology.orchestra.run.smithi117.stdout:Total 11 MB/s | 224 MB 00:21 2024-03-24T10:27:49.450 INFO:teuthology.orchestra.run.smithi023.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 854 kB/s | 71 kB 00:00 2024-03-24T10:27:49.492 INFO:teuthology.orchestra.run.smithi023.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 879 kB/s | 36 kB 00:00 2024-03-24T10:27:49.534 INFO:teuthology.orchestra.run.smithi023.stdout:(158/174): NetworkManager-libnm-1.47.2-1.el9.x8 5.9 MB/s | 1.8 MB 00:00 2024-03-24T10:27:49.559 INFO:teuthology.orchestra.run.smithi023.stdout:(159/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.1 MB/s | 248 kB 00:00 2024-03-24T10:27:49.585 INFO:teuthology.orchestra.run.smithi023.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-03-24T10:27:49.601 INFO:teuthology.orchestra.run.smithi023.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 551 kB/s | 23 kB 00:00 2024-03-24T10:27:49.627 INFO:teuthology.orchestra.run.smithi023.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-03-24T10:27:49.652 INFO:teuthology.orchestra.run.smithi023.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-24T10:27:49.677 INFO:teuthology.orchestra.run.smithi023.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-03-24T10:27:49.703 INFO:teuthology.orchestra.run.smithi023.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-24T10:27:49.728 INFO:teuthology.orchestra.run.smithi023.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 136 kB 00:00 2024-03-24T10:27:49.745 INFO:teuthology.orchestra.run.smithi023.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 427 kB/s | 29 kB 00:00 2024-03-24T10:27:49.828 INFO:teuthology.orchestra.run.smithi023.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 1.2 MB 00:00 2024-03-24T10:27:49.870 INFO:teuthology.orchestra.run.smithi023.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 15 MB/s | 2.2 MB 00:00 2024-03-24T10:27:49.912 INFO:teuthology.orchestra.run.smithi023.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 14 MB/s | 2.3 MB 00:00 2024-03-24T10:27:49.925 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-24T10:27:49.929 INFO:teuthology.orchestra.run.smithi023.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 533 kB/s | 31 kB 00:00 2024-03-24T10:27:49.954 INFO:teuthology.orchestra.run.smithi023.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 3.6 MB/s | 465 kB 00:00 2024-03-24T10:27:49.988 INFO:teuthology.orchestra.run.smithi023.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 14 MB/s | 1.1 MB 00:00 2024-03-24T10:27:50.080 INFO:teuthology.orchestra.run.smithi023.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 494 kB/s | 74 kB 00:00 2024-03-24T10:27:50.083 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:27:50.083 INFO:teuthology.orchestra.run.smithi023.stdout:Total 11 MB/s | 224 MB 00:20 2024-03-24T10:27:50.197 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-24T10:27:50.197 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-24T10:27:50.551 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2024-03-24T10:27:50.845 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2024-03-24T10:27:50.846 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2024-03-24T10:27:51.906 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-24T10:27:51.906 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-24T10:27:52.928 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2024-03-24T10:27:52.928 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2024-03-24T10:27:56.218 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-24T10:27:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T10:27:56.352 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T10:27:56.383 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T10:27:56.391 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T10:27:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T10:27:56.485 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T10:27:56.519 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T10:27:56.554 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T10:27:56.595 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T10:27:56.913 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T10:27:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:27:57.067 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:27:57.097 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:27:57.188 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:27:57.218 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T10:27:57.269 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T10:27:57.440 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T10:27:57.490 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T10:27:57.519 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:27:57.621 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:27:57.645 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T10:27:57.692 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T10:27:57.740 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T10:27:57.763 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T10:27:57.778 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T10:27:57.962 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T10:27:57.994 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T10:27:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T10:27:58.123 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T10:27:58.219 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T10:27:58.288 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T10:27:58.354 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:27:58.431 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:27:58.646 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:27:58.764 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T10:27:58.855 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T10:27:58.886 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T10:27:58.895 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-24T10:27:58.912 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T10:27:58.939 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T10:27:58.998 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T10:27:59.012 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T10:27:59.050 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T10:27:59.067 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T10:27:59.106 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T10:27:59.160 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T10:27:59.170 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T10:27:59.170 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T10:27:59.205 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T10:27:59.249 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T10:27:59.281 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T10:27:59.309 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T10:27:59.379 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T10:27:59.398 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T10:27:59.414 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 39/198 2024-03-24T10:27:59.453 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T10:27:59.457 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 40/198 2024-03-24T10:27:59.486 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 41/198 2024-03-24T10:27:59.493 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T10:27:59.542 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T10:27:59.554 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T10:27:59.595 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T10:27:59.600 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T10:27:59.673 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T10:27:59.806 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T10:27:59.860 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T10:27:59.921 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T10:27:59.977 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:28:00.054 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2024-03-24T10:28:00.098 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:28:00.136 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:28:00.151 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T10:28:00.186 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T10:28:00.228 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:28:00.257 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T10:28:00.265 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T10:28:00.295 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T10:28:00.296 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T10:28:00.378 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T10:28:00.456 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T10:28:00.459 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T10:28:00.477 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T10:28:00.521 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T10:28:00.530 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T10:28:00.537 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T10:28:00.559 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:28:00.564 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T10:28:00.585 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T10:28:00.589 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T10:28:00.593 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T10:28:00.625 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T10:28:00.634 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T10:28:00.645 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:28:00.666 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T10:28:00.666 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T10:28:00.703 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 53/198 2024-03-24T10:28:00.731 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T10:28:00.733 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T10:28:00.770 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T10:28:00.793 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T10:28:00.809 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T10:28:00.965 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T10:28:00.980 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T10:28:00.993 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T10:28:01.024 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T10:28:01.031 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:28:01.056 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T10:28:01.149 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T10:28:01.155 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:28:01.164 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T10:28:01.191 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:28:01.202 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T10:28:01.246 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T10:28:01.267 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T10:28:01.283 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:28:01.319 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T10:28:01.326 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T10:28:01.365 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T10:28:01.381 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T10:28:01.395 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:28:01.424 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T10:28:01.453 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:28:01.504 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T10:28:01.543 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T10:28:01.576 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T10:28:01.602 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T10:28:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T10:28:01.629 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:28:01.668 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:28:01.719 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T10:28:01.730 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:28:01.755 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T10:28:01.774 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T10:28:01.803 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T10:28:01.843 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T10:28:01.861 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T10:28:01.873 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T10:28:01.888 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T10:28:01.891 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T10:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T10:28:01.918 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T10:28:01.954 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T10:28:01.954 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T10:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T10:28:02.027 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T10:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T10:28:02.048 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T10:28:02.082 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T10:28:02.082 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T10:28:02.120 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T10:28:02.128 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T10:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T10:28:02.192 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T10:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T10:28:02.307 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T10:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T10:28:02.308 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T10:28:02.308 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T10:28:02.308 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T10:28:02.308 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T10:28:02.331 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T10:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T10:28:02.401 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T10:28:02.405 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T10:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T10:28:02.436 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 39/198 2024-03-24T10:28:02.449 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T10:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T10:28:02.479 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 40/198 2024-03-24T10:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T10:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T10:28:02.508 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 41/198 2024-03-24T10:28:02.529 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:28:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T10:28:02.577 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T10:28:02.631 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T10:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T10:28:02.705 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T10:28:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T10:28:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T10:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T10:28:02.839 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T10:28:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T10:28:02.892 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T10:28:02.925 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T10:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T10:28:03.028 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T10:28:03.063 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T10:28:03.090 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T10:28:03.124 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T10:28:03.134 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T10:28:03.167 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T10:28:03.210 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T10:28:03.263 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T10:28:03.301 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T10:28:03.304 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T10:28:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T10:28:03.367 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T10:28:03.380 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T10:28:03.408 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T10:28:03.416 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T10:28:03.419 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T10:28:03.448 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T10:28:03.451 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T10:28:03.482 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T10:28:03.484 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T10:28:03.496 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T10:28:03.511 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T10:28:03.521 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T10:28:03.552 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T10:28:03.582 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T10:28:03.583 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T10:28:03.584 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T10:28:03.617 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 39/198 2024-03-24T10:28:03.624 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T10:28:03.632 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T10:28:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T10:28:03.660 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 40/198 2024-03-24T10:28:03.664 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T10:28:03.685 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T10:28:03.688 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 41/198 2024-03-24T10:28:03.713 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T10:28:03.726 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T10:28:03.751 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 53/198 2024-03-24T10:28:03.759 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T10:28:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T10:28:03.780 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T10:28:03.793 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T10:28:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T10:28:03.879 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T10:28:03.882 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T10:28:03.963 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T10:28:04.008 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T10:28:04.011 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T10:28:04.017 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T10:28:04.048 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T10:28:04.074 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T10:28:04.095 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T10:28:04.191 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T10:28:04.213 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T10:28:04.249 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T10:28:04.283 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T10:28:04.327 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T10:28:04.477 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T10:28:04.530 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T10:28:04.585 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T10:28:04.623 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T10:28:04.658 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T10:28:04.689 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T10:28:04.703 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T10:28:04.726 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T10:28:04.773 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T10:28:04.777 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T10:28:04.808 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T10:28:04.823 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T10:28:04.831 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T10:28:04.860 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T10:28:04.869 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T10:28:04.903 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T10:28:04.913 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T10:28:04.950 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 53/198 2024-03-24T10:28:04.966 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T10:28:04.980 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T10:28:05.012 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T10:28:05.038 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T10:28:05.046 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T10:28:05.092 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T10:28:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T10:28:05.122 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T10:28:05.164 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T10:28:05.174 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T10:28:05.206 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T10:28:05.237 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T10:28:05.259 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T10:28:05.305 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T10:28:05.346 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T10:28:05.371 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T10:28:05.382 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T10:28:05.396 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T10:28:05.417 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T10:28:05.420 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T10:28:05.461 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T10:28:05.497 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T10:28:05.507 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T10:28:05.560 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T10:28:05.590 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T10:28:05.697 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T10:28:05.702 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T10:28:05.737 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T10:28:05.746 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T10:28:05.786 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T10:28:05.801 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T10:28:05.841 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T10:28:05.874 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T10:28:05.904 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T10:28:05.950 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T10:28:05.997 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T10:28:06.033 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T10:28:06.040 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T10:28:06.075 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T10:28:06.084 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T10:28:06.126 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T10:28:06.130 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T10:28:06.204 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T10:28:06.240 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T10:28:06.242 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T10:28:06.265 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T10:28:06.272 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T10:28:06.303 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T10:28:06.306 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T10:28:06.331 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T10:28:06.336 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T10:28:06.365 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T10:28:06.378 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T10:28:06.404 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T10:28:06.436 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T10:28:06.466 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T10:28:06.475 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T10:28:06.519 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T10:28:06.542 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T10:28:06.561 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T10:28:06.576 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T10:28:06.585 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T10:28:06.609 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T10:28:06.610 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T10:28:06.651 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T10:28:06.686 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T10:28:06.697 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T10:28:06.774 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T10:28:06.823 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T10:28:06.865 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T10:28:06.908 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T10:28:06.929 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T10:28:06.940 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T10:28:06.968 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T10:28:06.987 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T10:28:07.018 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T10:28:07.073 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T10:28:07.087 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T10:28:07.133 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T10:28:07.264 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T10:28:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T10:28:07.453 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T10:28:07.456 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T10:28:07.498 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T10:28:07.527 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T10:28:07.537 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T10:28:07.558 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T10:28:07.568 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T10:28:07.586 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T10:28:07.599 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T10:28:07.620 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T10:28:07.626 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T10:28:07.660 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T10:28:07.692 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T10:28:07.722 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T10:28:07.801 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T10:28:07.840 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T10:28:07.850 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T10:28:07.873 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T10:28:07.897 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T10:28:07.915 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T10:28:07.930 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T10:28:07.936 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T10:28:07.949 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T10:28:07.965 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T10:28:07.978 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T10:28:08.049 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T10:28:08.055 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T10:28:08.066 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T10:28:08.086 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/198 2024-03-24T10:28:08.098 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T10:28:08.140 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T10:28:08.152 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T10:28:08.180 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T10:28:08.212 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T10:28:08.260 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T10:28:08.266 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T10:28:08.304 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T10:28:08.368 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T10:28:08.382 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T10:28:08.423 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T10:28:08.600 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T10:28:08.626 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T10:28:08.697 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T10:28:08.719 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/198 2024-03-24T10:28:09.017 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T10:28:09.127 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T10:28:09.181 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T10:28:09.227 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T10:28:09.269 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T10:28:09.319 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T10:28:09.407 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T10:28:09.445 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T10:28:09.628 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T10:28:09.659 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T10:28:09.739 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T10:28:10.342 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T10:28:10.410 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T10:28:10.429 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T10:28:10.461 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T10:28:10.492 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T10:28:10.519 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T10:28:10.535 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T10:28:10.844 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T10:28:10.874 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T10:28:10.965 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T10:28:10.996 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/198 2024-03-24T10:28:11.032 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T10:28:11.337 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:28:11.374 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:28:11.492 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T10:28:11.519 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T10:28:11.593 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T10:28:11.612 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/198 2024-03-24T10:28:11.821 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T10:28:11.909 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:28:11.909 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T10:28:11.941 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T10:28:11.944 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T10:28:11.972 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T10:28:11.999 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T10:28:12.016 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T10:28:12.016 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T10:28:12.016 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T10:28:12.017 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:28:12.045 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T10:28:12.327 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T10:28:12.354 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T10:28:12.445 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T10:28:12.475 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/198 2024-03-24T10:28:12.512 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T10:28:12.991 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T10:28:13.015 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T10:28:13.078 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T10:28:13.100 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/198 2024-03-24T10:28:13.840 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:28:13.875 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:28:14.413 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:28:14.462 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T10:28:14.527 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T10:28:14.527 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T10:28:14.527 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T10:28:14.527 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:28:14.555 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T10:28:15.449 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:28:15.480 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:28:16.046 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T10:28:16.091 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T10:28:16.194 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T10:28:16.194 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T10:28:16.195 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T10:28:16.195 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:28:16.234 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T10:28:23.610 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T10:28:23.610 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-03-24T10:28:23.610 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-03-24T10:28:23.611 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-03-24T10:28:23.611 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-03-24T10:28:23.611 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-03-24T10:28:23.611 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2024-03-24T10:28:23.611 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-03-24T10:28:23.611 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:28:23.774 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T10:28:23.789 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T10:28:23.789 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:23.790 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T10:28:23.790 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:28:25.277 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T10:28:25.303 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T10:28:25.303 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:25.303 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T10:28:25.303 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:28:25.303 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:28:25.303 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:28:25.528 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T10:28:25.549 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T10:28:25.549 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:25.550 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T10:28:25.550 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:28:25.550 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:28:25.550 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:28:25.987 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T10:28:26.009 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T10:28:26.009 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:26.009 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T10:28:26.009 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:28:26.009 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:28:26.009 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:28:26.105 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T10:28:26.388 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T10:28:26.389 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /sys 2024-03-24T10:28:26.389 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /proc 2024-03-24T10:28:26.389 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /mnt 2024-03-24T10:28:26.389 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /var/tmp 2024-03-24T10:28:26.389 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /home 2024-03-24T10:28:26.389 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /root 2024-03-24T10:28:26.389 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /tmp 2024-03-24T10:28:26.389 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:28:26.526 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T10:28:26.556 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T10:28:26.567 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T10:28:26.578 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 139/198 2024-03-24T10:28:26.583 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T10:28:26.583 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:26.583 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T10:28:26.583 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:28:26.639 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 140/198 2024-03-24T10:28:26.670 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T10:28:26.903 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T10:28:26.929 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T10:28:27.890 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T10:28:27.921 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T10:28:28.020 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T10:28:28.062 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T10:28:28.087 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T10:28:28.087 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:28.087 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T10:28:28.087 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:28:28.087 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:28:28.088 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:28:28.184 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 144/198 2024-03-24T10:28:28.206 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T10:28:28.229 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T10:28:28.229 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:28.229 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T10:28:28.230 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:28:28.230 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:28:28.230 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:28:28.307 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T10:28:28.313 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T10:28:28.318 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T10:28:28.335 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T10:28:28.335 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:28.335 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T10:28:28.335 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:28:28.335 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:28:28.335 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:28:28.352 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T10:28:28.352 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /sys 2024-03-24T10:28:28.352 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /proc 2024-03-24T10:28:28.352 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /mnt 2024-03-24T10:28:28.352 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /var/tmp 2024-03-24T10:28:28.352 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /home 2024-03-24T10:28:28.353 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /root 2024-03-24T10:28:28.353 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /tmp 2024-03-24T10:28:28.353 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:28:28.372 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 147/198 2024-03-24T10:28:28.521 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T10:28:28.535 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T10:28:28.536 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:28.536 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T10:28:28.536 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:28:28.780 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T10:28:28.802 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T10:28:28.802 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:28.802 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T10:28:28.802 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:28:28.802 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:28:28.802 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:28:28.890 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T10:28:29.223 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T10:28:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T10:28:29.282 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T10:28:29.282 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:29.282 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T10:28:29.282 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:28:29.282 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:28:29.282 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:28:29.307 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T10:28:29.327 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T10:28:29.338 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T10:28:29.348 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T10:28:29.349 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:29.349 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T10:28:29.349 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:28:29.370 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 139/198 2024-03-24T10:28:29.394 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:28:29.416 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:28:29.416 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:29.417 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T10:28:29.417 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T10:28:29.417 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T10:28:29.417 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:28:29.430 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 140/198 2024-03-24T10:28:29.463 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T10:28:29.693 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T10:28:29.722 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T10:28:29.741 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T10:28:29.763 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T10:28:29.763 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:29.763 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T10:28:29.763 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:28:29.763 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:28:29.764 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:28:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T10:28:30.114 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T10:28:30.115 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:30.115 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T10:28:30.115 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:28:30.115 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:28:30.115 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:28:30.357 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T10:28:30.379 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T10:28:30.379 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:30.379 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T10:28:30.380 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:28:30.380 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:28:30.380 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:28:30.681 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T10:28:30.706 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T10:28:30.806 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T10:28:30.843 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T10:28:30.865 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T10:28:30.865 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:30.865 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T10:28:30.865 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:28:30.865 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:28:30.865 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:28:30.961 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T10:28:30.969 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 144/198 2024-03-24T10:28:31.008 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T10:28:31.031 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T10:28:31.031 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:31.031 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T10:28:31.031 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:28:31.031 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:28:31.031 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:28:31.075 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T10:28:31.085 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T10:28:31.131 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 147/198 2024-03-24T10:28:31.346 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T10:28:31.392 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T10:28:31.433 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 139/198 2024-03-24T10:28:31.495 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 140/198 2024-03-24T10:28:31.524 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T10:28:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T10:28:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T10:28:31.986 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T10:28:32.019 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T10:28:32.041 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T10:28:32.042 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:32.042 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T10:28:32.042 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:28:32.042 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:28:32.042 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:28:32.086 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T10:28:32.108 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T10:28:32.108 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:32.109 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T10:28:32.109 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:28:32.153 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:28:32.176 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:28:32.176 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:32.176 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T10:28:32.176 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T10:28:32.176 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T10:28:32.176 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:28:32.498 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T10:28:32.520 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T10:28:32.520 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:32.520 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T10:28:32.520 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:28:32.521 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:28:32.521 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:28:32.817 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T10:28:32.842 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T10:28:32.945 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T10:28:33.112 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 144/198 2024-03-24T10:28:33.136 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T10:28:33.158 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T10:28:33.158 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:33.158 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T10:28:33.158 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:28:33.158 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:28:33.159 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:28:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T10:28:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T10:28:33.264 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 147/198 2024-03-24T10:28:34.157 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T10:28:34.175 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T10:28:34.199 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/198 2024-03-24T10:28:34.205 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T10:28:34.227 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T10:28:34.227 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:34.227 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T10:28:34.227 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:28:34.228 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:28:34.228 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:28:34.229 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/198 2024-03-24T10:28:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T10:28:34.272 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T10:28:34.294 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T10:28:34.294 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:34.294 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T10:28:34.294 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:28:34.340 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:28:34.361 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:28:34.361 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:34.361 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T10:28:34.362 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T10:28:34.362 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T10:28:34.362 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:28:34.439 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 157/198 2024-03-24T10:28:34.547 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T10:28:34.587 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 159/198 2024-03-24T10:28:34.626 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T10:28:34.679 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T10:28:34.693 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T10:28:34.714 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T10:28:34.714 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:28:34.714 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T10:28:34.715 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:28:34.715 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:28:34.715 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:28:34.722 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T10:28:34.755 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 163/198 2024-03-24T10:28:34.787 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T10:28:34.817 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T10:28:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T10:28:34.918 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T10:28:34.982 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T10:28:35.595 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:28:35.630 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:28:35.649 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:28:36.461 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 170/198 2024-03-24T10:28:36.533 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T10:28:36.573 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T10:28:36.654 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T10:28:36.695 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T10:28:36.720 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T10:28:36.745 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T10:28:36.772 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T10:28:36.772 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T10:28:36.851 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T10:28:36.881 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T10:28:36.881 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T10:28:36.925 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T10:28:36.965 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T10:28:36.966 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/198 2024-03-24T10:28:36.991 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T10:28:36.998 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/198 2024-03-24T10:28:37.023 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T10:28:37.025 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T10:28:37.050 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T10:28:37.075 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T10:28:37.108 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T10:28:37.147 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T10:28:37.180 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T10:28:37.202 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 157/198 2024-03-24T10:28:37.205 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T10:28:37.230 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T10:28:37.256 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T10:28:37.281 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T10:28:37.287 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:28:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:28:37.332 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T10:28:37.371 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 159/198 2024-03-24T10:28:37.411 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T10:28:37.455 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T10:28:37.467 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:28:37.467 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T10:28:37.498 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T10:28:37.531 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 163/198 2024-03-24T10:28:37.550 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T10:28:37.571 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T10:28:37.575 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T10:28:37.601 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T10:28:37.607 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T10:28:37.634 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T10:28:37.659 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T10:28:37.663 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T10:28:37.723 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T10:28:37.733 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T10:28:37.766 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T10:28:38.293 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:28:38.331 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:28:38.350 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:28:39.170 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 170/198 2024-03-24T10:28:39.253 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T10:28:39.291 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T10:28:39.371 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T10:28:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T10:28:39.421 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T10:28:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/198 2024-03-24T10:28:39.446 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T10:28:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/198 2024-03-24T10:28:39.480 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T10:28:39.506 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T10:28:39.506 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T10:28:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T10:28:39.594 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T10:28:39.624 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T10:28:39.624 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T10:28:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 157/198 2024-03-24T10:28:39.708 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T10:28:39.735 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T10:28:39.776 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T10:28:39.801 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T10:28:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T10:28:39.826 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T10:28:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 159/198 2024-03-24T10:28:39.860 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T10:28:39.907 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T10:28:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T10:28:39.948 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T10:28:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T10:28:39.973 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T10:28:39.998 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T10:28:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T10:28:40.024 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T10:28:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 163/198 2024-03-24T10:28:40.049 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T10:28:40.056 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:28:40.056 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:28:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T10:28:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T10:28:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T10:28:40.231 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T10:28:40.235 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:28:40.235 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T10:28:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T10:28:40.318 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T10:28:40.343 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T10:28:40.366 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T10:28:40.394 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T10:28:40.419 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T10:28:40.482 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T10:28:40.829 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:28:40.867 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:28:40.886 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T10:28:41.751 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 170/198 2024-03-24T10:28:41.838 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T10:28:41.879 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T10:28:41.972 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T10:28:41.999 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T10:28:42.024 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T10:28:42.050 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T10:28:42.076 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T10:28:42.076 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T10:28:42.164 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T10:28:42.210 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T10:28:42.210 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T10:28:42.295 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T10:28:42.331 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T10:28:42.363 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T10:28:42.388 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T10:28:42.413 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T10:28:42.446 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T10:28:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T10:28:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/198 2024-03-24T10:28:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/198 2024-03-24T10:28:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 3/198 2024-03-24T10:28:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/198 2024-03-24T10:28:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 5/198 2024-03-24T10:28:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 6/198 2024-03-24T10:28:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/198 2024-03-24T10:28:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 8/198 2024-03-24T10:28:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 9/198 2024-03-24T10:28:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:28:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:28:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T10:28:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/198 2024-03-24T10:28:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 14/198 2024-03-24T10:28:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 15/198 2024-03-24T10:28:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:28:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/198 2024-03-24T10:28:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 18/198 2024-03-24T10:28:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 19/198 2024-03-24T10:28:42.495 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 20/198 2024-03-24T10:28:42.495 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 21/198 2024-03-24T10:28:42.495 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 22/198 2024-03-24T10:28:42.495 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 23/198 2024-03-24T10:28:42.495 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/198 2024-03-24T10:28:42.495 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/198 2024-03-24T10:28:42.495 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 26/198 2024-03-24T10:28:42.495 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/198 2024-03-24T10:28:42.495 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 28/198 2024-03-24T10:28:42.495 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/198 2024-03-24T10:28:42.496 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 30/198 2024-03-24T10:28:42.496 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 31/198 2024-03-24T10:28:42.496 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 32/198 2024-03-24T10:28:42.496 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 33/198 2024-03-24T10:28:42.496 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 34/198 2024-03-24T10:28:42.496 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 35/198 2024-03-24T10:28:42.496 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 36/198 2024-03-24T10:28:42.496 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 37/198 2024-03-24T10:28:42.496 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 38/198 2024-03-24T10:28:42.496 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 39/198 2024-03-24T10:28:42.496 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T10:28:42.496 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T10:28:42.497 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T10:28:42.497 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T10:28:42.497 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T10:28:42.497 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T10:28:42.497 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T10:28:42.497 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T10:28:42.497 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T10:28:42.497 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T10:28:42.497 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T10:28:42.497 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T10:28:42.497 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T10:28:42.498 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T10:28:42.498 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T10:28:42.498 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T10:28:42.498 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T10:28:42.498 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T10:28:42.498 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T10:28:42.498 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T10:28:42.498 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T10:28:42.498 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T10:28:42.498 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T10:28:42.498 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T10:28:42.498 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T10:28:42.498 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T10:28:42.498 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T10:28:42.499 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T10:28:42.499 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T10:28:42.499 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T10:28:42.499 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T10:28:42.499 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T10:28:42.499 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T10:28:42.499 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T10:28:42.499 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T10:28:42.499 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T10:28:42.499 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T10:28:42.499 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T10:28:42.499 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T10:28:42.499 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T10:28:42.500 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T10:28:42.500 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T10:28:42.500 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T10:28:42.500 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T10:28:42.500 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T10:28:42.500 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T10:28:42.500 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T10:28:42.500 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T10:28:42.500 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T10:28:42.500 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T10:28:42.500 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T10:28:42.500 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T10:28:42.500 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T10:28:42.500 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T10:28:42.501 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T10:28:42.501 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T10:28:42.501 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T10:28:42.501 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T10:28:42.501 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T10:28:42.501 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T10:28:42.501 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T10:28:42.501 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T10:28:42.501 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T10:28:42.501 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T10:28:42.501 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T10:28:42.501 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T10:28:42.502 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T10:28:42.502 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T10:28:42.502 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T10:28:42.502 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T10:28:42.502 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T10:28:42.502 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T10:28:42.502 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T10:28:42.502 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T10:28:42.502 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T10:28:42.502 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T10:28:42.502 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T10:28:42.502 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T10:28:42.502 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T10:28:42.503 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T10:28:42.503 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T10:28:42.503 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T10:28:42.503 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T10:28:42.503 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T10:28:42.503 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T10:28:42.503 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T10:28:42.503 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T10:28:42.503 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T10:28:42.503 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T10:28:42.504 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T10:28:42.504 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T10:28:42.504 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T10:28:42.504 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T10:28:42.504 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T10:28:42.504 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T10:28:42.504 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T10:28:42.504 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T10:28:42.504 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T10:28:42.504 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T10:28:42.504 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T10:28:42.504 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T10:28:42.504 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T10:28:42.504 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T10:28:42.505 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T10:28:42.505 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T10:28:42.505 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T10:28:42.505 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T10:28:42.505 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T10:28:42.505 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T10:28:42.505 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T10:28:42.505 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T10:28:42.505 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:28:42.505 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T10:28:42.505 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T10:28:42.505 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T10:28:42.505 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T10:28:42.506 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T10:28:42.506 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T10:28:42.506 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T10:28:42.506 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T10:28:42.506 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T10:28:42.506 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T10:28:42.506 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T10:28:42.506 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T10:28:42.506 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T10:28:42.506 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T10:28:42.506 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T10:28:42.506 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T10:28:42.506 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T10:28:42.506 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T10:28:42.507 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T10:28:42.507 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T10:28:42.507 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T10:28:42.507 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T10:28:42.507 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T10:28:42.507 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T10:28:42.507 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T10:28:42.507 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T10:28:42.507 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T10:28:42.507 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T10:28:42.507 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T10:28:42.507 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T10:28:42.507 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T10:28:42.507 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T10:28:42.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T10:28:42.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T10:28:42.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T10:28:42.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T10:28:42.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T10:28:42.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T10:28:42.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T10:28:42.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T10:28:42.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T10:28:42.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T10:28:42.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T10:28:42.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T10:28:42.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T10:28:42.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T10:28:42.509 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T10:28:42.534 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T10:28:42.561 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T10:28:42.587 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T10:28:42.612 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T10:28:42.639 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T10:28:42.643 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:28:42.643 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:28:42.823 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T10:28:42.823 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T10:28:42.889 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T10:28:42.914 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T10:28:42.947 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T10:28:42.974 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T10:28:42.999 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T10:28:43.069 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T10:28:45.204 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T10:28:45.204 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/198 2024-03-24T10:28:45.204 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/198 2024-03-24T10:28:45.204 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 3/198 2024-03-24T10:28:45.204 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/198 2024-03-24T10:28:45.204 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 5/198 2024-03-24T10:28:45.204 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 6/198 2024-03-24T10:28:45.204 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/198 2024-03-24T10:28:45.204 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 8/198 2024-03-24T10:28:45.204 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 9/198 2024-03-24T10:28:45.204 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:28:45.204 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:28:45.205 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T10:28:45.205 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/198 2024-03-24T10:28:45.205 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 14/198 2024-03-24T10:28:45.205 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 15/198 2024-03-24T10:28:45.205 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:28:45.205 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/198 2024-03-24T10:28:45.205 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 18/198 2024-03-24T10:28:45.205 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 19/198 2024-03-24T10:28:45.205 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 20/198 2024-03-24T10:28:45.205 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 21/198 2024-03-24T10:28:45.205 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 22/198 2024-03-24T10:28:45.205 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 23/198 2024-03-24T10:28:45.205 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/198 2024-03-24T10:28:45.205 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/198 2024-03-24T10:28:45.206 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 26/198 2024-03-24T10:28:45.206 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/198 2024-03-24T10:28:45.206 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 28/198 2024-03-24T10:28:45.206 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/198 2024-03-24T10:28:45.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 30/198 2024-03-24T10:28:45.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 31/198 2024-03-24T10:28:45.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 32/198 2024-03-24T10:28:45.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 33/198 2024-03-24T10:28:45.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 34/198 2024-03-24T10:28:45.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 35/198 2024-03-24T10:28:45.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 36/198 2024-03-24T10:28:45.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 37/198 2024-03-24T10:28:45.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 38/198 2024-03-24T10:28:45.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 39/198 2024-03-24T10:28:45.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T10:28:45.223 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T10:28:45.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T10:28:45.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T10:28:45.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T10:28:45.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T10:28:45.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T10:28:45.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T10:28:45.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T10:28:45.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T10:28:45.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T10:28:45.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T10:28:45.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T10:28:45.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T10:28:45.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T10:28:45.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T10:28:45.225 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T10:28:45.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T10:28:45.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T10:28:45.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T10:28:45.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T10:28:45.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T10:28:45.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T10:28:45.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T10:28:45.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T10:28:45.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T10:28:45.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T10:28:45.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T10:28:45.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T10:28:45.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T10:28:45.226 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T10:28:45.227 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T10:28:45.227 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T10:28:45.227 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T10:28:45.227 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T10:28:45.227 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T10:28:45.227 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T10:28:45.227 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T10:28:45.227 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T10:28:45.227 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T10:28:45.227 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T10:28:45.227 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T10:28:45.227 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T10:28:45.227 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T10:28:45.227 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T10:28:45.228 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T10:28:45.228 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T10:28:45.228 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T10:28:45.228 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T10:28:45.228 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T10:28:45.228 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T10:28:45.228 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T10:28:45.228 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T10:28:45.228 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T10:28:45.228 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T10:28:45.228 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T10:28:45.228 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T10:28:45.228 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T10:28:45.228 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T10:28:45.229 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T10:28:45.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T10:28:45.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T10:28:45.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T10:28:45.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T10:28:45.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T10:28:45.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T10:28:45.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T10:28:45.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T10:28:45.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T10:28:45.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T10:28:45.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T10:28:45.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T10:28:45.230 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T10:28:45.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T10:28:45.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T10:28:45.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T10:28:45.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T10:28:45.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T10:28:45.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T10:28:45.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T10:28:45.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T10:28:45.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T10:28:45.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T10:28:45.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T10:28:45.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T10:28:45.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T10:28:45.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T10:28:45.232 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T10:28:45.233 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T10:28:45.234 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T10:28:45.234 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T10:28:45.234 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T10:28:45.234 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T10:28:45.234 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T10:28:45.234 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T10:28:45.234 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T10:28:45.234 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T10:28:45.234 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T10:28:45.234 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T10:28:45.234 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T10:28:45.234 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T10:28:45.234 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T10:28:45.669 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T10:28:45.669 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:28:45.669 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-03-24T10:28:45.669 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T10:28:45.669 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T10:28:45.669 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T10:28:45.669 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T10:28:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T10:28:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T10:28:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T10:28:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T10:28:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T10:28:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T10:28:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T10:28:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T10:28:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T10:28:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T10:28:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T10:28:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T10:28:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T10:28:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T10:28:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T10:28:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T10:28:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T10:28:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T10:28:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: jq-1.6-16.el9.x86_64 2024-03-24T10:28:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T10:28:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T10:28:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T10:28:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T10:28:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T10:28:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T10:28:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T10:28:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T10:28:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.675 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.675 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T10:28:45.675 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.675 INFO:teuthology.orchestra.run.smithi116.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T10:28:45.675 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T10:28:45.675 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T10:28:45.675 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T10:28:45.675 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T10:28:45.675 INFO:teuthology.orchestra.run.smithi116.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T10:28:45.675 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T10:28:45.675 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T10:28:45.675 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T10:28:45.675 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T10:28:45.675 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T10:28:45.676 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T10:28:45.676 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T10:28:45.676 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T10:28:45.676 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T10:28:45.676 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T10:28:45.676 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T10:28:45.676 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T10:28:45.676 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T10:28:45.676 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T10:28:45.676 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T10:28:45.676 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T10:28:45.676 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T10:28:45.676 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T10:28:45.676 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T10:28:45.677 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T10:28:45.678 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T10:28:45.678 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T10:28:45.678 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T10:28:45.678 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T10:28:45.678 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T10:28:45.678 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T10:28:45.678 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T10:28:45.678 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T10:28:45.678 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T10:28:45.678 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T10:28:45.678 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T10:28:45.678 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T10:28:45.678 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T10:28:45.678 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T10:28:45.679 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T10:28:45.679 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T10:28:45.679 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T10:28:45.679 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T10:28:45.679 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T10:28:45.679 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T10:28:45.679 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T10:28:45.679 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T10:28:45.679 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T10:28:45.679 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T10:28:45.679 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T10:28:45.679 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T10:28:45.679 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T10:28:45.679 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.680 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.680 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T10:28:45.680 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T10:28:45.680 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T10:28:45.680 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.680 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T10:28:45.680 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T10:28:45.680 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T10:28:45.680 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T10:28:45.680 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T10:28:45.680 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T10:28:45.680 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T10:28:45.680 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T10:28:45.680 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T10:28:45.681 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T10:28:45.682 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T10:28:45.682 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T10:28:45.682 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-24T10:28:46.179 DEBUG:teuthology.parallel:result is None 2024-03-24T10:28:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T10:28:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/198 2024-03-24T10:28:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/198 2024-03-24T10:28:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 3/198 2024-03-24T10:28:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/198 2024-03-24T10:28:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 5/198 2024-03-24T10:28:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 6/198 2024-03-24T10:28:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/198 2024-03-24T10:28:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 8/198 2024-03-24T10:28:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 9/198 2024-03-24T10:28:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T10:28:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T10:28:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 14/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 15/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 18/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 19/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 20/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 21/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 22/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 23/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 26/198 2024-03-24T10:28:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/198 2024-03-24T10:28:47.726 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 28/198 2024-03-24T10:28:47.726 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/198 2024-03-24T10:28:47.726 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 30/198 2024-03-24T10:28:47.740 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 31/198 2024-03-24T10:28:47.741 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 32/198 2024-03-24T10:28:47.741 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 33/198 2024-03-24T10:28:47.741 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 34/198 2024-03-24T10:28:47.741 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 35/198 2024-03-24T10:28:47.741 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 36/198 2024-03-24T10:28:47.741 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 37/198 2024-03-24T10:28:47.741 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 38/198 2024-03-24T10:28:47.741 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 39/198 2024-03-24T10:28:47.741 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T10:28:47.741 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T10:28:47.741 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T10:28:47.741 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T10:28:47.741 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T10:28:47.742 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T10:28:47.743 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T10:28:47.743 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T10:28:47.743 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T10:28:47.743 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T10:28:47.743 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T10:28:47.743 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T10:28:47.743 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T10:28:47.743 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T10:28:47.743 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T10:28:47.743 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T10:28:47.743 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T10:28:47.743 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T10:28:47.743 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T10:28:47.743 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T10:28:47.744 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T10:28:47.745 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T10:28:47.746 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T10:28:47.746 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T10:28:47.746 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T10:28:47.746 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T10:28:47.746 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T10:28:47.746 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T10:28:47.746 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T10:28:47.746 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T10:28:47.746 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T10:28:47.746 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T10:28:47.746 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T10:28:47.746 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T10:28:47.746 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T10:28:47.746 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T10:28:47.747 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T10:28:47.748 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T10:28:47.749 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T10:28:47.750 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T10:28:47.750 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T10:28:47.750 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T10:28:47.750 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T10:28:47.750 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T10:28:47.750 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T10:28:47.750 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T10:28:47.750 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T10:28:47.750 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T10:28:47.750 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T10:28:47.750 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T10:28:47.750 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T10:28:47.750 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T10:28:47.750 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T10:28:47.751 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T10:28:47.752 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T10:28:47.752 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T10:28:47.752 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T10:28:47.752 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T10:28:47.752 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T10:28:48.017 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T10:28:48.018 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:28:48.018 INFO:teuthology.orchestra.run.smithi117.stdout:Upgraded: 2024-03-24T10:28:48.018 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T10:28:48.018 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T10:28:48.018 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T10:28:48.018 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T10:28:48.018 INFO:teuthology.orchestra.run.smithi117.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T10:28:48.018 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T10:28:48.018 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T10:28:48.019 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T10:28:48.019 INFO:teuthology.orchestra.run.smithi117.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T10:28:48.019 INFO:teuthology.orchestra.run.smithi117.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T10:28:48.019 INFO:teuthology.orchestra.run.smithi117.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T10:28:48.019 INFO:teuthology.orchestra.run.smithi117.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T10:28:48.019 INFO:teuthology.orchestra.run.smithi117.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T10:28:48.019 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.019 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.019 INFO:teuthology.orchestra.run.smithi117.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T10:28:48.019 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T10:28:48.019 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T10:28:48.019 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T10:28:48.019 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T10:28:48.019 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T10:28:48.020 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T10:28:48.020 INFO:teuthology.orchestra.run.smithi117.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T10:28:48.020 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T10:28:48.020 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-24T10:28:48.020 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T10:28:48.020 INFO:teuthology.orchestra.run.smithi117.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T10:28:48.020 INFO:teuthology.orchestra.run.smithi117.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T10:28:48.020 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T10:28:48.020 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.020 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.020 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.020 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.020 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:48.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:48.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:48.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:48.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:48.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:48.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:48.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:48.022 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:48.022 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.022 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.022 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:48.022 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T10:28:48.022 INFO:teuthology.orchestra.run.smithi117.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T10:28:48.022 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T10:28:48.022 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T10:28:48.022 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T10:28:48.022 INFO:teuthology.orchestra.run.smithi117.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T10:28:48.022 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T10:28:48.022 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T10:28:48.022 INFO:teuthology.orchestra.run.smithi117.stdout: jq-1.6-16.el9.x86_64 2024-03-24T10:28:48.023 INFO:teuthology.orchestra.run.smithi117.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T10:28:48.023 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T10:28:48.023 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T10:28:48.023 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.023 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.023 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.023 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T10:28:48.023 INFO:teuthology.orchestra.run.smithi117.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T10:28:48.023 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T10:28:48.023 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T10:28:48.023 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T10:28:48.023 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.023 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T10:28:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T10:28:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T10:28:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T10:28:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T10:28:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T10:28:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T10:28:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T10:28:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T10:28:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T10:28:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T10:28:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T10:28:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.025 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T10:28:48.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T10:28:48.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T10:28:48.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T10:28:48.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T10:28:48.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T10:28:48.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T10:28:48.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T10:28:48.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T10:28:48.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T10:28:48.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T10:28:48.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T10:28:48.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T10:28:48.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T10:28:48.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T10:28:48.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T10:28:48.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T10:28:48.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T10:28:48.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T10:28:48.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T10:28:48.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T10:28:48.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T10:28:48.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T10:28:48.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T10:28:48.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T10:28:48.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T10:28:48.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T10:28:48.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T10:28:48.030 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T10:28:48.030 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T10:28:48.030 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T10:28:48.030 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T10:28:48.030 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.030 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.030 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:48.030 INFO:teuthology.orchestra.run.smithi117.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T10:28:48.030 INFO:teuthology.orchestra.run.smithi117.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T10:28:48.030 INFO:teuthology.orchestra.run.smithi117.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T10:28:48.030 INFO:teuthology.orchestra.run.smithi117.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T10:28:48.030 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T10:28:48.030 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T10:28:48.031 INFO:teuthology.orchestra.run.smithi117.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T10:28:48.031 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:28:48.031 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-24T10:28:48.472 DEBUG:teuthology.parallel:result is None 2024-03-24T10:28:50.536 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T10:28:50.536 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:28:50.536 INFO:teuthology.orchestra.run.smithi023.stdout:Upgraded: 2024-03-24T10:28:50.536 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T10:28:50.536 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T10:28:50.537 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T10:28:50.537 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T10:28:50.537 INFO:teuthology.orchestra.run.smithi023.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T10:28:50.537 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T10:28:50.537 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T10:28:50.537 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T10:28:50.537 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T10:28:50.537 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T10:28:50.538 INFO:teuthology.orchestra.run.smithi023.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T10:28:50.538 INFO:teuthology.orchestra.run.smithi023.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T10:28:50.538 INFO:teuthology.orchestra.run.smithi023.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T10:28:50.538 INFO:teuthology.orchestra.run.smithi023.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.538 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.538 INFO:teuthology.orchestra.run.smithi023.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T10:28:50.538 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T10:28:50.538 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T10:28:50.539 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T10:28:50.539 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T10:28:50.539 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T10:28:50.539 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T10:28:50.539 INFO:teuthology.orchestra.run.smithi023.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T10:28:50.539 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T10:28:50.539 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2024-03-24T10:28:50.539 INFO:teuthology.orchestra.run.smithi023.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T10:28:50.540 INFO:teuthology.orchestra.run.smithi023.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T10:28:50.540 INFO:teuthology.orchestra.run.smithi023.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T10:28:50.540 INFO:teuthology.orchestra.run.smithi023.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T10:28:50.540 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.540 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.540 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.540 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.540 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:50.540 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.540 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.540 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.540 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:50.540 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:50.540 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:50.541 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:50.541 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:50.541 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.541 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.541 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:50.541 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.541 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.541 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.541 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:50.541 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:50.541 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.541 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.541 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:28:50.541 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T10:28:50.542 INFO:teuthology.orchestra.run.smithi023.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T10:28:50.542 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T10:28:50.542 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T10:28:50.542 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T10:28:50.542 INFO:teuthology.orchestra.run.smithi023.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T10:28:50.542 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T10:28:50.542 INFO:teuthology.orchestra.run.smithi023.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T10:28:50.542 INFO:teuthology.orchestra.run.smithi023.stdout: jq-1.6-16.el9.x86_64 2024-03-24T10:28:50.542 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T10:28:50.542 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T10:28:50.542 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T10:28:50.542 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.542 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.543 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.543 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T10:28:50.543 INFO:teuthology.orchestra.run.smithi023.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T10:28:50.543 INFO:teuthology.orchestra.run.smithi023.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T10:28:50.543 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T10:28:50.543 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T10:28:50.543 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.543 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.543 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T10:28:50.544 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.544 INFO:teuthology.orchestra.run.smithi023.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T10:28:50.544 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T10:28:50.544 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T10:28:50.544 INFO:teuthology.orchestra.run.smithi023.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T10:28:50.544 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T10:28:50.544 INFO:teuthology.orchestra.run.smithi023.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T10:28:50.544 INFO:teuthology.orchestra.run.smithi023.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T10:28:50.544 INFO:teuthology.orchestra.run.smithi023.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T10:28:50.544 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T10:28:50.544 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T10:28:50.544 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T10:28:50.544 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T10:28:50.544 INFO:teuthology.orchestra.run.smithi023.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T10:28:50.545 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T10:28:50.545 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T10:28:50.545 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T10:28:50.545 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T10:28:50.545 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T10:28:50.545 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T10:28:50.545 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T10:28:50.545 INFO:teuthology.orchestra.run.smithi023.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T10:28:50.546 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T10:28:50.546 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T10:28:50.547 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T10:28:50.547 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.547 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.547 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.547 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T10:28:50.547 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T10:28:50.547 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T10:28:50.547 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T10:28:50.547 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T10:28:50.547 INFO:teuthology.orchestra.run.smithi023.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T10:28:50.547 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T10:28:50.547 INFO:teuthology.orchestra.run.smithi023.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T10:28:50.547 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T10:28:50.548 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T10:28:50.548 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T10:28:50.548 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T10:28:50.548 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T10:28:50.548 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T10:28:50.548 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T10:28:50.548 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T10:28:50.548 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T10:28:50.548 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T10:28:50.548 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T10:28:50.548 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T10:28:50.548 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T10:28:50.548 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T10:28:50.548 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T10:28:50.549 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T10:28:50.549 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T10:28:50.549 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T10:28:50.549 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T10:28:50.549 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T10:28:50.549 INFO:teuthology.orchestra.run.smithi023.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T10:28:50.549 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T10:28:50.549 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T10:28:50.549 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T10:28:50.549 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T10:28:50.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T10:28:50.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T10:28:50.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T10:28:50.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T10:28:50.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T10:28:50.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T10:28:50.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T10:28:50.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T10:28:50.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T10:28:50.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T10:28:50.550 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T10:28:50.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T10:28:50.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T10:28:50.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T10:28:50.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T10:28:50.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T10:28:50.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T10:28:50.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T10:28:50.551 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T10:28:50.552 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T10:28:50.553 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T10:28:50.553 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:28:50.553 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-24T10:28:51.014 DEBUG:teuthology.parallel:result is None 2024-03-24T10:28:51.015 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:28:51.156 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T10:28:51.177 INFO:teuthology.orchestra.run.smithi023.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-24T10:28:51.178 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-24T10:28:51.178 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-24T10:28:51.180 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:28:51.394 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T10:28:51.419 INFO:teuthology.orchestra.run.smithi116.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-24T10:28:51.419 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-24T10:28:51.419 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-24T10:28:51.421 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:28:51.638 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T10:28:51.664 INFO:teuthology.orchestra.run.smithi117.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-24T10:28:51.664 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-24T10:28:51.664 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-24T10:28:51.666 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T10:28:51.666 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:28:51.666 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T10:28:51.696 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T10:28:51.697 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T10:28:51.731 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T10:28:51.731 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T10:28:51.765 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T10:28:51.766 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:28:51.766 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T10:28:51.793 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T10:28:51.860 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T10:28:51.860 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T10:28:51.891 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T10:28:51.962 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T10:28:51.962 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T10:28:51.994 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T10:28:52.063 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T10:28:52.063 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:28:52.064 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T10:28:52.090 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T10:28:52.158 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T10:28:52.158 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T10:28:52.190 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T10:28:52.259 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T10:28:52.259 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T10:28:52.290 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T10:28:52.361 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T10:28:52.361 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:28:52.361 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T10:28:52.388 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T10:28:52.454 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T10:28:52.454 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T10:28:52.486 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T10:28:52.556 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T10:28:52.556 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T10:28:52.586 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T10:28:52.657 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T10:28:52.757 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'subvols': {'create': 2}} 2024-03-24T10:28:52.757 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:28:52.758 INFO:tasks.cephadm:Cluster fsid is 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:28:52.758 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T10:28:52.758 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.23', 'mon.b': '172.21.15.116', 'mon.c': '172.21.15.117'} 2024-03-24T10:28:52.758 INFO:tasks.cephadm:First mon is mon.a on smithi023 2024-03-24T10:28:52.758 INFO:tasks.cephadm:First mgr is x 2024-03-24T10:28:52.758 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T10:28:52.759 DEBUG:teuthology.orchestra.run.smithi023:> sudo hostname $(hostname -s) 2024-03-24T10:28:52.786 DEBUG:teuthology.orchestra.run.smithi116:> sudo hostname $(hostname -s) 2024-03-24T10:28:52.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo hostname $(hostname -s) 2024-03-24T10:28:52.849 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-24T10:28:52.850 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:28:53.068 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77900/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1855-g21b3bd67', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1855.g21b3bd67'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-23 04:48:23.862772', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240323.003144-squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-24T10:28:53.283 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-batrick-testing-20240323.003144-squid, sha1 21b3bd672163092b4e542f772797fcba02d3d4de from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:28:53.285 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/x86_64/flavors/default/cephadm 2024-03-24T10:28:53.285 DEBUG:teuthology.orchestra.run.smithi023:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T10:28:54.026 INFO:teuthology.orchestra.run.smithi023.stdout:-rw-r--r--. 1 ubuntu ubuntu 766828 Mar 24 10:28 /home/ubuntu/cephtest/cephadm 2024-03-24T10:28:54.031 DEBUG:teuthology.orchestra.run.smithi116:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T10:28:54.531 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r--. 1 ubuntu ubuntu 766828 Mar 24 10:28 /home/ubuntu/cephtest/cephadm 2024-03-24T10:28:54.531 DEBUG:teuthology.orchestra.run.smithi117:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T10:28:55.040 INFO:teuthology.orchestra.run.smithi117.stdout:-rw-r--r--. 1 ubuntu ubuntu 766828 Mar 24 10:28 /home/ubuntu/cephtest/cephadm 2024-03-24T10:28:55.041 DEBUG:teuthology.orchestra.run.smithi023:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T10:28:55.059 DEBUG:teuthology.orchestra.run.smithi116:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T10:28:55.080 DEBUG:teuthology.orchestra.run.smithi117:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T10:28:55.117 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de on all hosts... 2024-03-24T10:28:55.117 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de pull 2024-03-24T10:28:55.120 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de pull 2024-03-24T10:28:55.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de pull 2024-03-24T10:28:55.984 INFO:teuthology.orchestra.run.smithi117.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T10:28:56.037 INFO:teuthology.orchestra.run.smithi023.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T10:28:56.055 INFO:teuthology.orchestra.run.smithi116.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T10:30:01.233 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-24T10:30:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T10:30:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: "image_id": "553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12", 2024-03-24T10:30:01.233 INFO:teuthology.orchestra.run.smithi117.stdout: "repo_digests": [ 2024-03-24T10:30:01.234 INFO:teuthology.orchestra.run.smithi117.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919" 2024-03-24T10:30:01.234 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-03-24T10:30:01.234 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-24T10:30:01.645 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:30:01.645 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T10:30:01.645 INFO:teuthology.orchestra.run.smithi023.stdout: "image_id": "553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12", 2024-03-24T10:30:01.646 INFO:teuthology.orchestra.run.smithi023.stdout: "repo_digests": [ 2024-03-24T10:30:01.646 INFO:teuthology.orchestra.run.smithi023.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919" 2024-03-24T10:30:01.646 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-24T10:30:01.646 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:30:02.871 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T10:30:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T10:30:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "image_id": "553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12", 2024-03-24T10:30:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "repo_digests": [ 2024-03-24T10:30:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919" 2024-03-24T10:30:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-24T10:30:02.872 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T10:30:02.896 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph 2024-03-24T10:30:02.931 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph 2024-03-24T10:30:02.962 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph 2024-03-24T10:30:02.997 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 777 /etc/ceph 2024-03-24T10:30:03.027 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /etc/ceph 2024-03-24T10:30:03.053 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /etc/ceph 2024-03-24T10:30:03.084 INFO:tasks.cephadm:Writing seed config... 2024-03-24T10:30:03.085 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-24T10:30:03.085 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-24T10:30:03.085 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-24T10:30:03.085 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-24T10:30:03.085 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-24T10:30:03.085 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-24T10:30:03.085 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-24T10:30:03.086 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-24T10:30:03.087 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-24T10:30:03.087 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-03-24T10:30:03.087 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T10:30:03.087 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T10:30:03.087 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T10:30:03.087 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T10:30:03.087 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T10:30:03.087 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-03-24T10:30:03.087 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-24T10:30:03.087 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-24T10:30:03.087 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-24T10:30:03.087 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-24T10:30:03.087 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-24T10:30:03.088 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-24T10:30:03.088 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-24T10:30:03.088 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-24T10:30:03.088 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T10:30:03.088 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T10:30:03.088 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-24T10:30:03.088 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-24T10:30:03.088 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-24T10:30:03.088 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-24T10:30:03.088 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-24T10:30:03.088 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-24T10:30:03.088 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-24T10:30:03.088 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T10:30:03.088 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-24T10:30:03.089 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:30:03.089 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T10:30:03.108 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-24T10:30:03.109 DEBUG:teuthology.orchestra.run.smithi023:mon.a> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mon.a.service 2024-03-24T10:30:03.152 DEBUG:teuthology.orchestra.run.smithi023:mgr.x> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mgr.x.service 2024-03-24T10:30:03.195 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T10:30:03.195 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de -v bootstrap --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.23 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T10:30:03.393 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:30:03.393 INFO:teuthology.orchestra.run.smithi023.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de', '-v', 'bootstrap', '--fsid', '4f8fb8f6-e9c9-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.23', '--skip-admin-label'] 2024-03-24T10:30:03.416 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T10:30:03.416 INFO:teuthology.orchestra.run.smithi023.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T10:30:03.417 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying podman|docker is present... 2024-03-24T10:30:03.440 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T10:30:03.440 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying lvm2 is present... 2024-03-24T10:30:03.440 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying time synchronization is in place... 2024-03-24T10:30:03.448 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T10:30:03.448 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T10:30:03.453 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T10:30:03.454 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-24T10:30:03.460 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2024-03-24T10:30:03.466 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2024-03-24T10:30:03.466 INFO:teuthology.orchestra.run.smithi023.stdout:Unit chronyd.service is enabled and running 2024-03-24T10:30:03.466 INFO:teuthology.orchestra.run.smithi023.stdout:Repeating the final host check... 2024-03-24T10:30:03.489 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T10:30:03.490 INFO:teuthology.orchestra.run.smithi023.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T10:30:03.490 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl is present 2024-03-24T10:30:03.490 INFO:teuthology.orchestra.run.smithi023.stdout:lvcreate is present 2024-03-24T10:30:03.497 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T10:30:03.497 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T10:30:03.501 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T10:30:03.501 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-24T10:30:03.507 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2024-03-24T10:30:03.514 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2024-03-24T10:30:03.514 INFO:teuthology.orchestra.run.smithi023.stdout:Unit chronyd.service is enabled and running 2024-03-24T10:30:03.514 INFO:teuthology.orchestra.run.smithi023.stdout:Host looks OK 2024-03-24T10:30:03.514 INFO:teuthology.orchestra.run.smithi023.stdout:Cluster fsid: 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:30:03.514 INFO:teuthology.orchestra.run.smithi023.stdout:Acquiring lock 140220776865744 on /run/cephadm/4f8fb8f6-e9c9-11ee-95cd-87774f69a715.lock 2024-03-24T10:30:03.514 INFO:teuthology.orchestra.run.smithi023.stdout:Lock 140220776865744 acquired on /run/cephadm/4f8fb8f6-e9c9-11ee-95cd-87774f69a715.lock 2024-03-24T10:30:03.514 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 3300 ... 2024-03-24T10:30:03.515 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 6789 ... 2024-03-24T10:30:03.515 INFO:teuthology.orchestra.run.smithi023.stdout:Base mon IP(s) is [172.21.15.23:3300, 172.21.15.23:6789], mon addrv is [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2024-03-24T10:30:03.518 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.23 metric 100 2024-03-24T10:30:03.518 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.23 metric 100 2024-03-24T10:30:03.520 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T10:30:03.520 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-24T10:30:03.521 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-24T10:30:03.523 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T10:30:03.523 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T10:30:03.523 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T10:30:03.523 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-24T10:30:03.523 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe59:ce42/64 scope link noprefixroute 2024-03-24T10:30:03.523 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T10:30:03.524 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2024-03-24T10:30:03.524 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2024-03-24T10:30:03.524 INFO:teuthology.orchestra.run.smithi023.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-24T10:30:03.524 INFO:teuthology.orchestra.run.smithi023.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T10:30:03.525 INFO:teuthology.orchestra.run.smithi023.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T10:30:03.956 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 2024-03-24T10:30:03.956 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T10:30:03.956 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T10:30:03.956 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying blob sha256:30dc77a9f705c02d26e1a1f9e2bfdf9d08cd4086b08b221e8a27d0dd2909924d 2024-03-24T10:30:03.956 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-24T10:30:03.956 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying config sha256:553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 2024-03-24T10:30:03.956 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T10:30:04.903 INFO:teuthology.orchestra.run.smithi023.stdout:ceph: stdout ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev) 2024-03-24T10:30:04.903 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph version: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev) 2024-03-24T10:30:04.903 INFO:teuthology.orchestra.run.smithi023.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T10:30:05.725 INFO:teuthology.orchestra.run.smithi023.stdout:stat: stdout 167 167 2024-03-24T10:30:05.725 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial keys... 2024-03-24T10:30:08.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQCuAABmCACXARAAKn35B2hk1+d4Mm0ViKHRUQ== 2024-03-24T10:30:16.373 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQCzAABm67jkDhAALGAQRR6rZfT96gZbP92Xzw== 2024-03-24T10:30:17.197 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQC4AABmk4fSKBAASeLwp/dpAM23dVLTVAmVeg== 2024-03-24T10:30:17.198 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial monmap... 2024-03-24T10:30:18.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T10:30:18.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-24T10:30:18.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:30:18.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T10:30:18.095 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool for a [v2:172.21.15.23:3300,v1:172.21.15.23:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T10:30:18.096 INFO:teuthology.orchestra.run.smithi023.stdout:setting min_mon_release = quincy 2024-03-24T10:30:18.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: set fsid to 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:30:18.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T10:30:18.096 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:30:18.096 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mon... 2024-03-24T10:30:19.000 INFO:teuthology.orchestra.run.smithi023.stdout:create mon.a on 2024-03-24T10:30:19.208 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-24T10:30:19.352 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T10:30:19.533 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715.target. 2024-03-24T10:30:19.533 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715.target. 2024-03-24T10:30:19.769 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mon.a 2024-03-24T10:30:19.769 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-24T10:30:19.933 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715.target.wants/ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@.service. 2024-03-24T10:30:20.449 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T10:30:20.449 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-24T10:30:20.456 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T10:30:20.456 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-24T10:30:20.457 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-24T10:30:20.457 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T10:30:20.457 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon to start... 2024-03-24T10:30:20.457 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon... 2024-03-24T10:30:20.569 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:20 smithi023 ceph-mon[32090]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T10:30:21.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:21 smithi023 ceph-mon[32090]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T10:30:21.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:21 smithi023 ceph-mon[32090]: monmap epoch 1 2024-03-24T10:30:21.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:21 smithi023 ceph-mon[32090]: fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:30:21.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:21 smithi023 ceph-mon[32090]: last_changed 2024-03-24T10:30:17.512891+0000 2024-03-24T10:30:21.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:21 smithi023 ceph-mon[32090]: created 2024-03-24T10:30:17.512891+0000 2024-03-24T10:30:21.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:21 smithi023 ceph-mon[32090]: min_mon_release 19 (squid) 2024-03-24T10:30:21.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:21 smithi023 ceph-mon[32090]: election_strategy: 1 2024-03-24T10:30:21.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:21 smithi023 ceph-mon[32090]: 0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2024-03-24T10:30:21.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:21 smithi023 ceph-mon[32090]: fsmap 2024-03-24T10:30:21.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:21 smithi023 ceph-mon[32090]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T10:30:21.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:21 smithi023 ceph-mon[32090]: mgrmap e1: no daemons active 2024-03-24T10:30:21.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:21 smithi023 ceph-mon[32090]: from='client.? 172.21.15.23:0/2105374004' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T10:30:22.343 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T10:30:22.343 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout id: 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:30:22.343 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T10:30:22.343 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T10:30:22.344 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout services: 2024-03-24T10:30:22.344 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.543615s) 2024-03-24T10:30:22.344 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T10:30:22.344 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T10:30:22.344 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T10:30:22.344 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout data: 2024-03-24T10:30:22.344 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T10:30:22.344 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T10:30:22.344 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T10:30:22.344 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T10:30:22.345 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T10:30:22.345 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.905+0000 7fd86c3ae700 1 Processor -- start 2024-03-24T10:30:22.345 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.905+0000 7fd86c3ae700 1 -- start start 2024-03-24T10:30:22.345 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.905+0000 7fd86c3ae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd864107d80 0x7fd8641081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:22.345 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.905+0000 7fd86c3ae700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd864108770 con 0x7fd864107d80 2024-03-24T10:30:22.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.905+0000 7fd86a14a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd864107d80 0x7fd8641081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:22.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.905+0000 7fd86a14a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd864107d80 0x7fd8641081a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47548/0 (socket says 172.21.15.23:47548) 2024-03-24T10:30:22.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.905+0000 7fd86a14a700 1 -- 172.21.15.23:0/2858530791 learned_addr learned my addr 172.21.15.23:0/2858530791 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:22.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.906+0000 7fd86a14a700 1 -- 172.21.15.23:0/2858530791 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd864108f80 con 0x7fd864107d80 2024-03-24T10:30:22.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.906+0000 7fd86a14a700 1 --2- 172.21.15.23:0/2858530791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd864107d80 0x7fd8641081a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd8600098a0 tx=0x7fd860009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797912da91a70781 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:22.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.906+0000 7fd869148700 1 -- 172.21.15.23:0/2858530791 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd860004160 con 0x7fd864107d80 2024-03-24T10:30:22.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.906+0000 7fd869148700 1 -- 172.21.15.23:0/2858530791 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8600042c0 con 0x7fd864107d80 2024-03-24T10:30:22.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.906+0000 7fd869148700 1 -- 172.21.15.23:0/2858530791 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8600045a0 con 0x7fd864107d80 2024-03-24T10:30:22.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.906+0000 7fd86c3ae700 1 -- 172.21.15.23:0/2858530791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd864107d80 msgr2=0x7fd8641081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:22.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.906+0000 7fd86c3ae700 1 --2- 172.21.15.23:0/2858530791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd864107d80 0x7fd8641081a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd8600098a0 tx=0x7fd860009bb0 comp rx=0 tx=0).stop 2024-03-24T10:30:22.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.907+0000 7fd86c3ae700 1 -- 172.21.15.23:0/2858530791 shutdown_connections 2024-03-24T10:30:22.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.907+0000 7fd86c3ae700 1 --2- 172.21.15.23:0/2858530791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd864107d80 0x7fd8641081a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:22.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.907+0000 7fd86c3ae700 1 -- 172.21.15.23:0/2858530791 >> 172.21.15.23:0/2858530791 conn(0x7fd864103360 msgr2=0x7fd864105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:22.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.907+0000 7fd86c3ae700 1 -- 172.21.15.23:0/2858530791 shutdown_connections 2024-03-24T10:30:22.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.907+0000 7fd86c3ae700 1 -- 172.21.15.23:0/2858530791 wait complete. 2024-03-24T10:30:22.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.907+0000 7fd86c3ae700 1 Processor -- start 2024-03-24T10:30:22.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.907+0000 7fd86c3ae700 1 -- start start 2024-03-24T10:30:22.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.907+0000 7fd86c3ae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd864107d80 0x7fd86419b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:22.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.907+0000 7fd86c3ae700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd86419b8d0 con 0x7fd864107d80 2024-03-24T10:30:22.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.907+0000 7fd86a14a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd864107d80 0x7fd86419b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:22.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.907+0000 7fd86a14a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd864107d80 0x7fd86419b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47554/0 (socket says 172.21.15.23:47554) 2024-03-24T10:30:22.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.907+0000 7fd86a14a700 1 -- 172.21.15.23:0/2105374004 learned_addr learned my addr 172.21.15.23:0/2105374004 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:22.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.908+0000 7fd86a14a700 1 -- 172.21.15.23:0/2105374004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd860009550 con 0x7fd864107d80 2024-03-24T10:30:22.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.908+0000 7fd86a14a700 1 --2- 172.21.15.23:0/2105374004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd864107d80 0x7fd86419b390 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd860004000 tx=0x7fd860004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:22.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.908+0000 7fd85b7fe700 1 -- 172.21.15.23:0/2105374004 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd860011630 con 0x7fd864107d80 2024-03-24T10:30:22.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.908+0000 7fd85b7fe700 1 -- 172.21.15.23:0/2105374004 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd860011790 con 0x7fd864107d80 2024-03-24T10:30:22.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.908+0000 7fd86c3ae700 1 -- 172.21.15.23:0/2105374004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd86419bad0 con 0x7fd864107d80 2024-03-24T10:30:22.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.908+0000 7fd85b7fe700 1 -- 172.21.15.23:0/2105374004 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd860011a30 con 0x7fd864107d80 2024-03-24T10:30:22.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.908+0000 7fd86c3ae700 1 -- 172.21.15.23:0/2105374004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd86419c680 con 0x7fd864107d80 2024-03-24T10:30:22.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.908+0000 7fd85b7fe700 1 -- 172.21.15.23:0/2105374004 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd860022070 con 0x7fd864107d80 2024-03-24T10:30:22.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.908+0000 7fd85b7fe700 1 -- 172.21.15.23:0/2105374004 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd86001c450 con 0x7fd864107d80 2024-03-24T10:30:22.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.909+0000 7fd86c3ae700 1 -- 172.21.15.23:0/2105374004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd86419bf70 con 0x7fd864107d80 2024-03-24T10:30:22.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.911+0000 7fd85b7fe700 1 -- 172.21.15.23:0/2105374004 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd86419bf70 con 0x7fd864107d80 2024-03-24T10:30:22.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.985+0000 7fd86c3ae700 1 -- 172.21.15.23:0/2105374004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fd86406a850 con 0x7fd864107d80 2024-03-24T10:30:22.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.986+0000 7fd85b7fe700 1 -- 172.21.15.23:0/2105374004 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fd86406a850 con 0x7fd864107d80 2024-03-24T10:30:22.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.987+0000 7fd86c3ae700 1 -- 172.21.15.23:0/2105374004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd864107d80 msgr2=0x7fd86419b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:22.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.987+0000 7fd86c3ae700 1 --2- 172.21.15.23:0/2105374004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd864107d80 0x7fd86419b390 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd860004000 tx=0x7fd860004750 comp rx=0 tx=0).stop 2024-03-24T10:30:22.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.987+0000 7fd86c3ae700 1 -- 172.21.15.23:0/2105374004 shutdown_connections 2024-03-24T10:30:22.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.987+0000 7fd86c3ae700 1 --2- 172.21.15.23:0/2105374004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd864107d80 0x7fd86419b390 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:22.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.987+0000 7fd86c3ae700 1 -- 172.21.15.23:0/2105374004 >> 172.21.15.23:0/2105374004 conn(0x7fd864103360 msgr2=0x7fd8641920e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:22.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.987+0000 7fd86c3ae700 1 -- 172.21.15.23:0/2105374004 shutdown_connections 2024-03-24T10:30:22.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:20.987+0000 7fd86c3ae700 1 -- 172.21.15.23:0/2105374004 wait complete. 2024-03-24T10:30:22.349 INFO:teuthology.orchestra.run.smithi023.stdout:mon is available 2024-03-24T10:30:22.349 INFO:teuthology.orchestra.run.smithi023.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T10:30:23.130 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:22 smithi023 ceph-mon[32090]: from='client.? 172.21.15.23:0/4108136946' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T10:30:23.130 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:22 smithi023 ceph-mon[32090]: from='client.? 172.21.15.23:0/4108136946' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T10:30:23.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T10:30:23.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T10:30:23.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:30:23.465 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2024-03-24T10:30:23.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T10:30:23.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T10:30:23.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T10:30:23.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T10:30:23.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T10:30:23.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T10:30:23.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T10:30:23.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T10:30:23.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T10:30:23.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T10:30:23.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T10:30:23.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T10:30:23.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T10:30:23.466 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.751+0000 7fd6bd130700 1 Processor -- start 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.751+0000 7fd6bd130700 1 -- start start 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.751+0000 7fd6bd130700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6b8107d80 0x7fd6b81081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.751+0000 7fd6bd130700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b8108770 con 0x7fd6b8107d80 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.751+0000 7fd6b6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6b8107d80 0x7fd6b81081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.751+0000 7fd6b6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6b8107d80 0x7fd6b81081a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47560/0 (socket says 172.21.15.23:47560) 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.751+0000 7fd6b6d9d700 1 -- 172.21.15.23:0/769297943 learned_addr learned my addr 172.21.15.23:0/769297943 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.752+0000 7fd6b6d9d700 1 -- 172.21.15.23:0/769297943 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6b8108f80 con 0x7fd6b8107d80 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.752+0000 7fd6b6d9d700 1 --2- 172.21.15.23:0/769297943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6b8107d80 0x7fd6b81081a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80098a0 tx=0x7fd6a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5252990e1070482 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.752+0000 7fd6b5d9b700 1 -- 172.21.15.23:0/769297943 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6a8004160 con 0x7fd6b8107d80 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.752+0000 7fd6b5d9b700 1 -- 172.21.15.23:0/769297943 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a80042c0 con 0x7fd6b8107d80 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.752+0000 7fd6b5d9b700 1 -- 172.21.15.23:0/769297943 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6a80045a0 con 0x7fd6b8107d80 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.752+0000 7fd6bd130700 1 -- 172.21.15.23:0/769297943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6b8107d80 msgr2=0x7fd6b81081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.752+0000 7fd6bd130700 1 --2- 172.21.15.23:0/769297943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6b8107d80 0x7fd6b81081a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80098a0 tx=0x7fd6a8009bb0 comp rx=0 tx=0).stop 2024-03-24T10:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.753+0000 7fd6bd130700 1 -- 172.21.15.23:0/769297943 shutdown_connections 2024-03-24T10:30:23.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.753+0000 7fd6bd130700 1 --2- 172.21.15.23:0/769297943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6b8107d80 0x7fd6b81081a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:23.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.753+0000 7fd6bd130700 1 -- 172.21.15.23:0/769297943 >> 172.21.15.23:0/769297943 conn(0x7fd6b8103360 msgr2=0x7fd6b8105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:23.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.753+0000 7fd6bd130700 1 -- 172.21.15.23:0/769297943 shutdown_connections 2024-03-24T10:30:23.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.753+0000 7fd6bd130700 1 -- 172.21.15.23:0/769297943 wait complete. 2024-03-24T10:30:23.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.753+0000 7fd6bd130700 1 Processor -- start 2024-03-24T10:30:23.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.754+0000 7fd6bd130700 1 -- start start 2024-03-24T10:30:23.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.754+0000 7fd6bd130700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6b8107d80 0x7fd6b819b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:23.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.754+0000 7fd6bd130700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b819b8d0 con 0x7fd6b8107d80 2024-03-24T10:30:23.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.754+0000 7fd6b6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6b8107d80 0x7fd6b819b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:23.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.754+0000 7fd6b6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6b8107d80 0x7fd6b819b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47570/0 (socket says 172.21.15.23:47570) 2024-03-24T10:30:23.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.754+0000 7fd6b6d9d700 1 -- 172.21.15.23:0/4108136946 learned_addr learned my addr 172.21.15.23:0/4108136946 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:23.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.754+0000 7fd6b6d9d700 1 -- 172.21.15.23:0/4108136946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a8009550 con 0x7fd6b8107d80 2024-03-24T10:30:23.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.754+0000 7fd6b6d9d700 1 --2- 172.21.15.23:0/4108136946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6b8107d80 0x7fd6b819b390 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8004000 tx=0x7fd6a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:23.468 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.754+0000 7fd69ffff700 1 -- 172.21.15.23:0/4108136946 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6a8011630 con 0x7fd6b8107d80 2024-03-24T10:30:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.754+0000 7fd69ffff700 1 -- 172.21.15.23:0/4108136946 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a8011790 con 0x7fd6b8107d80 2024-03-24T10:30:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.754+0000 7fd6bd130700 1 -- 172.21.15.23:0/4108136946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6b819bad0 con 0x7fd6b8107d80 2024-03-24T10:30:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.755+0000 7fd69ffff700 1 -- 172.21.15.23:0/4108136946 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6a8011a30 con 0x7fd6b8107d80 2024-03-24T10:30:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.755+0000 7fd6bd130700 1 -- 172.21.15.23:0/4108136946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6b819c680 con 0x7fd6b8107d80 2024-03-24T10:30:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.755+0000 7fd69ffff700 1 -- 172.21.15.23:0/4108136946 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd6a8022070 con 0x7fd6b8107d80 2024-03-24T10:30:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.755+0000 7fd69ffff700 1 -- 172.21.15.23:0/4108136946 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd6a801c450 con 0x7fd6b8107d80 2024-03-24T10:30:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.756+0000 7fd6bd130700 1 -- 172.21.15.23:0/4108136946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6b819bf70 con 0x7fd6b8107d80 2024-03-24T10:30:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.758+0000 7fd69ffff700 1 -- 172.21.15.23:0/4108136946 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd6b819bf70 con 0x7fd6b8107d80 2024-03-24T10:30:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.829+0000 7fd6bd130700 1 -- 172.21.15.23:0/4108136946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fd6b806a850 con 0x7fd6b8107d80 2024-03-24T10:30:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.832+0000 7fd69ffff700 1 -- 172.21.15.23:0/4108136946 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7fd6b806a850 con 0x7fd6b8107d80 2024-03-24T10:30:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.833+0000 7fd69ffff700 1 -- 172.21.15.23:0/4108136946 <== mon.0 v2:172.21.15.23:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd6a8011c40 con 0x7fd6b8107d80 2024-03-24T10:30:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.834+0000 7fd6bd130700 1 -- 172.21.15.23:0/4108136946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6b8107d80 msgr2=0x7fd6b819b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.834+0000 7fd6bd130700 1 --2- 172.21.15.23:0/4108136946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6b8107d80 0x7fd6b819b390 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8004000 tx=0x7fd6a8004750 comp rx=0 tx=0).stop 2024-03-24T10:30:23.470 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.834+0000 7fd6bd130700 1 -- 172.21.15.23:0/4108136946 shutdown_connections 2024-03-24T10:30:23.470 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.834+0000 7fd6bd130700 1 --2- 172.21.15.23:0/4108136946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6b8107d80 0x7fd6b819b390 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:23.470 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.834+0000 7fd6bd130700 1 -- 172.21.15.23:0/4108136946 >> 172.21.15.23:0/4108136946 conn(0x7fd6b8103360 msgr2=0x7fd6b81920e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:23.470 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.834+0000 7fd6bd130700 1 -- 172.21.15.23:0/4108136946 shutdown_connections 2024-03-24T10:30:23.470 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:22.834+0000 7fd6bd130700 1 -- 172.21.15.23:0/4108136946 wait complete. 2024-03-24T10:30:23.470 INFO:teuthology.orchestra.run.smithi023.stdout:Generating new minimal ceph.conf... 2024-03-24T10:30:24.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:24 smithi023 ceph-mon[32090]: from='client.? 172.21.15.23:0/2715312438' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:30:24.529 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.874+0000 7f6b4e1a6700 1 Processor -- start 2024-03-24T10:30:24.529 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.874+0000 7f6b4e1a6700 1 -- start start 2024-03-24T10:30:24.529 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.874+0000 7f6b4e1a6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b48106c80 0x7f6b481070a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:24.529 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.874+0000 7f6b4e1a6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f6b48107670 con 0x7f6b48106c80 2024-03-24T10:30:24.529 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.874+0000 7f6b477fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b48106c80 0x7f6b481070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:24.529 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.875+0000 7f6b477fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b48106c80 0x7f6b481070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47580/0 (socket says 172.21.15.23:47580) 2024-03-24T10:30:24.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.875+0000 7f6b477fe700 1 -- 172.21.15.23:0/3459412889 learned_addr learned my addr 172.21.15.23:0/3459412889 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:24.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.875+0000 7f6b477fe700 1 -- 172.21.15.23:0/3459412889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b48107e80 con 0x7f6b48106c80 2024-03-24T10:30:24.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.875+0000 7f6b477fe700 1 --2- 172.21.15.23:0/3459412889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b48106c80 0x7f6b481070a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009e80 tx=0x7f6b3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b92f8e3ede1517 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:24.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.875+0000 7f6b467fc700 1 -- 172.21.15.23:0/3459412889 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b3800e070 con 0x7f6b48106c80 2024-03-24T10:30:24.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.875+0000 7f6b467fc700 1 -- 172.21.15.23:0/3459412889 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6b3800b8c0 con 0x7f6b48106c80 2024-03-24T10:30:24.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.875+0000 7f6b467fc700 1 -- 172.21.15.23:0/3459412889 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b38004550 con 0x7f6b48106c80 2024-03-24T10:30:24.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.876+0000 7f6b4e1a6700 1 -- 172.21.15.23:0/3459412889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b48106c80 msgr2=0x7f6b481070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:24.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.876+0000 7f6b4e1a6700 1 --2- 172.21.15.23:0/3459412889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b48106c80 0x7f6b481070a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009e80 tx=0x7f6b3800b330 comp rx=0 tx=0).stop 2024-03-24T10:30:24.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.876+0000 7f6b4e1a6700 1 -- 172.21.15.23:0/3459412889 shutdown_connections 2024-03-24T10:30:24.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.876+0000 7f6b4e1a6700 1 --2- 172.21.15.23:0/3459412889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b48106c80 0x7f6b481070a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:24.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.876+0000 7f6b4e1a6700 1 -- 172.21.15.23:0/3459412889 >> 172.21.15.23:0/3459412889 conn(0x7f6b48102200 msgr2=0x7f6b48104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:24.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.876+0000 7f6b4e1a6700 1 -- 172.21.15.23:0/3459412889 shutdown_connections 2024-03-24T10:30:24.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.876+0000 7f6b4e1a6700 1 -- 172.21.15.23:0/3459412889 wait complete. 2024-03-24T10:30:24.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.876+0000 7f6b4e1a6700 1 Processor -- start 2024-03-24T10:30:24.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.876+0000 7f6b4e1a6700 1 -- start start 2024-03-24T10:30:24.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.877+0000 7f6b4e1a6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b48106c80 0x7f6b4819bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:24.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.877+0000 7f6b4e1a6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4819c0f0 con 0x7f6b48106c80 2024-03-24T10:30:24.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.877+0000 7f6b477fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b48106c80 0x7f6b4819bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:24.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.877+0000 7f6b477fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b48106c80 0x7f6b4819bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47586/0 (socket says 172.21.15.23:47586) 2024-03-24T10:30:24.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.877+0000 7f6b477fe700 1 -- 172.21.15.23:0/2715312438 learned_addr learned my addr 172.21.15.23:0/2715312438 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:24.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.877+0000 7f6b477fe700 1 -- 172.21.15.23:0/2715312438 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b38009b30 con 0x7f6b48106c80 2024-03-24T10:30:24.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.877+0000 7f6b477fe700 1 --2- 172.21.15.23:0/2715312438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b48106c80 0x7f6b4819bbb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009fd0 tx=0x7f6b38003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:24.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.877+0000 7f6b44ff9700 1 -- 172.21.15.23:0/2715312438 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b3800e040 con 0x7f6b48106c80 2024-03-24T10:30:24.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.877+0000 7f6b44ff9700 1 -- 172.21.15.23:0/2715312438 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6b380128f0 con 0x7f6b48106c80 2024-03-24T10:30:24.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.877+0000 7f6b44ff9700 1 -- 172.21.15.23:0/2715312438 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b3801c890 con 0x7f6b48106c80 2024-03-24T10:30:24.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.877+0000 7f6b4e1a6700 1 -- 172.21.15.23:0/2715312438 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b4819c2f0 con 0x7f6b48106c80 2024-03-24T10:30:24.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.877+0000 7f6b4e1a6700 1 -- 172.21.15.23:0/2715312438 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b4819cea0 con 0x7f6b48106c80 2024-03-24T10:30:24.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.877+0000 7f6b44ff9700 1 -- 172.21.15.23:0/2715312438 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6b3801cb00 con 0x7f6b48106c80 2024-03-24T10:30:24.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.878+0000 7f6b44ff9700 1 -- 172.21.15.23:0/2715312438 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6b3801b850 con 0x7f6b48106c80 2024-03-24T10:30:24.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.879+0000 7f6b4e1a6700 1 -- 172.21.15.23:0/2715312438 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b4819c770 con 0x7f6b48106c80 2024-03-24T10:30:24.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.881+0000 7f6b44ff9700 1 -- 172.21.15.23:0/2715312438 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6b3802a060 con 0x7f6b48106c80 2024-03-24T10:30:24.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.950+0000 7f6b4e1a6700 1 -- 172.21.15.23:0/2715312438 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f6b480638b0 con 0x7f6b48106c80 2024-03-24T10:30:24.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.951+0000 7f6b44ff9700 1 -- 172.21.15.23:0/2715312438 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f6b38020070 con 0x7f6b48106c80 2024-03-24T10:30:24.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.952+0000 7f6b4e1a6700 1 -- 172.21.15.23:0/2715312438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b48106c80 msgr2=0x7f6b4819bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:24.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.952+0000 7f6b4e1a6700 1 --2- 172.21.15.23:0/2715312438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b48106c80 0x7f6b4819bbb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009fd0 tx=0x7f6b38003d70 comp rx=0 tx=0).stop 2024-03-24T10:30:24.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.952+0000 7f6b4e1a6700 1 -- 172.21.15.23:0/2715312438 shutdown_connections 2024-03-24T10:30:24.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.952+0000 7f6b4e1a6700 1 --2- 172.21.15.23:0/2715312438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b48106c80 0x7f6b4819bbb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:24.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.952+0000 7f6b4e1a6700 1 -- 172.21.15.23:0/2715312438 >> 172.21.15.23:0/2715312438 conn(0x7f6b48102200 msgr2=0x7f6b481927f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:24.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.952+0000 7f6b4e1a6700 1 -- 172.21.15.23:0/2715312438 shutdown_connections 2024-03-24T10:30:24.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:23.952+0000 7f6b4e1a6700 1 -- 172.21.15.23:0/2715312438 wait complete. 2024-03-24T10:30:24.536 INFO:teuthology.orchestra.run.smithi023.stdout:Restarting the monitor... 2024-03-24T10:30:24.791 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:24 smithi023 systemd[1]: Stopping Ceph mon.a for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:30:25.152 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:24 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a[32086]: 2024-03-24T10:30:24.790+0000 7fe8dc40f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:30:25.152 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:24 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a[32086]: 2024-03-24T10:30:24.790+0000 7fe8dc40f700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T10:30:25.152 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:24 smithi023 podman[32336]: 2024-03-24 10:30:24.873036 +0000 UTC m=+0.186887570 container died 976a386f48d8a710db6c4151d46dacbd28f0a4eafe3001235c05eba26398414b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T10:30:25.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:25 smithi023 podman[32336]: 2024-03-24 10:30:25.232650625 +0000 UTC m=+0.546502238 container cleanup 976a386f48d8a710db6c4151d46dacbd28f0a4eafe3001235c05eba26398414b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T10:30:25.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:25 smithi023 bash[32336]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a 2024-03-24T10:30:25.800 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:25 smithi023 podman[32350]: 2024-03-24 10:30:25.467107543 +0000 UTC m=+0.595701774 container remove 976a386f48d8a710db6c4151d46dacbd28f0a4eafe3001235c05eba26398414b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-24T10:30:25.801 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:25 smithi023 systemd[1]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T10:30:25.801 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:25 smithi023 systemd[1]: Stopped Ceph mon.a for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:30:25.801 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:25 smithi023 systemd[1]: Starting Ceph mon.a for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:30:26.048 INFO:teuthology.orchestra.run.smithi023.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T10:30:26.073 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:25 smithi023 podman[32447]: 2024-03-24 10:30:25.801354447 +0000 UTC m=+0.093915500 container create fc7c0651aa5d9f239e23074707cb733c8310e78c6cea809c0babdd5f5d7ac629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T10:30:26.073 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:25 smithi023 podman[32447]: 2024-03-24 10:30:25.724922334 +0000 UTC m=+0.017483395 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:30:26.073 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:25 smithi023 podman[32447]: 2024-03-24 10:30:25.970405201 +0000 UTC m=+0.262966255 container init fc7c0651aa5d9f239e23074707cb733c8310e78c6cea809c0babdd5f5d7ac629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T10:30:26.073 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:25 smithi023 podman[32447]: 2024-03-24 10:30:25.973182789 +0000 UTC m=+0.265743845 container start fc7c0651aa5d9f239e23074707cb733c8310e78c6cea809c0babdd5f5d7ac629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:30:26.073 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:25 smithi023 ceph-mon[32466]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T10:30:26.073 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:25 smithi023 ceph-mon[32466]: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev), process ceph-mon, pid 2 2024-03-24T10:30:26.073 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:25 smithi023 ceph-mon[32466]: pidfile_write: ignore empty --pid-file 2024-03-24T10:30:26.073 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:25 smithi023 ceph-mon[32466]: load: jerasure load: lrc 2024-03-24T10:30:26.074 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: RocksDB version: 7.9.2 2024-03-24T10:30:26.074 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Git sha 0 2024-03-24T10:30:26.074 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Compile date 2024-03-23 03:28:38 2024-03-24T10:30:26.074 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: DB SUMMARY 2024-03-24T10:30:26.074 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: DB Session ID: MLE4E55QSI5WXSN2OPVR 2024-03-24T10:30:26.074 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: CURRENT file: CURRENT 2024-03-24T10:30:26.074 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T10:30:26.074 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T10:30:26.074 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T10:30:26.074 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 106122 ; 2024-03-24T10:30:26.075 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.error_if_exists: 0 2024-03-24T10:30:26.075 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.create_if_missing: 0 2024-03-24T10:30:26.075 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.paranoid_checks: 1 2024-03-24T10:30:26.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T10:30:26.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T10:30:26.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T10:30:26.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.env: 0x55e7664aec20 2024-03-24T10:30:26.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T10:30:26.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.info_log: 0x55e7689cbac0 2024-03-24T10:30:26.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T10:30:26.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.statistics: (nil) 2024-03-24T10:30:26.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.use_fsync: 0 2024-03-24T10:30:26.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_log_file_size: 0 2024-03-24T10:30:26.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T10:30:26.076 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T10:30:26.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T10:30:26.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T10:30:26.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.allow_fallocate: 1 2024-03-24T10:30:26.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T10:30:26.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T10:30:26.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.use_direct_reads: 0 2024-03-24T10:30:26.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T10:30:26.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T10:30:26.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.db_log_dir: 2024-03-24T10:30:26.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.wal_dir: 2024-03-24T10:30:26.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T10:30:26.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T10:30:26.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T10:30:26.077 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T10:30:26.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T10:30:26.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T10:30:26.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T10:30:26.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T10:30:26.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.write_buffer_manager: 0x55e76899e1e0 2024-03-24T10:30:26.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T10:30:26.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T10:30:26.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T10:30:26.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T10:30:26.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T10:30:26.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T10:30:26.078 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T10:30:26.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T10:30:26.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.unordered_write: 0 2024-03-24T10:30:26.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T10:30:26.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T10:30:26.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T10:30:26.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T10:30:26.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.row_cache: None 2024-03-24T10:30:26.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.wal_filter: None 2024-03-24T10:30:26.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T10:30:26.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T10:30:26.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.two_write_queues: 0 2024-03-24T10:30:26.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T10:30:26.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.wal_compression: 0 2024-03-24T10:30:26.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.atomic_flush: 0 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.log_readahead_size: 0 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_background_jobs: 2 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_background_compactions: -1 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_subcompactions: 1 2024-03-24T10:30:26.080 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T10:30:26.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T10:30:26.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T10:30:26.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T10:30:26.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T10:30:26.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T10:30:26.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T10:30:26.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T10:30:26.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_open_files: -1 2024-03-24T10:30:26.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T10:30:26.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T10:30:26.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T10:30:26.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T10:30:26.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_background_flushes: -1 2024-03-24T10:30:26.081 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Compression algorithms supported: 2024-03-24T10:30:26.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T10:30:26.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: kZSTD supported: 0 2024-03-24T10:30:26.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: kXpressCompression supported: 0 2024-03-24T10:30:26.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T10:30:26.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: kZlibCompression supported: 1 2024-03-24T10:30:26.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: kSnappyCompression supported: 1 2024-03-24T10:30:26.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: kLZ4Compression supported: 1 2024-03-24T10:30:26.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: kBZip2Compression supported: 0 2024-03-24T10:30:26.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T10:30:26.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T10:30:26.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-24T10:30:26.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T10:30:26.082 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T10:30:26.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.merge_operator: 2024-03-24T10:30:26.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compaction_filter: None 2024-03-24T10:30:26.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T10:30:26.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T10:30:26.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T10:30:26.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T10:30:26.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e7689cbbc0) 2024-03-24T10:30:26.083 INFO:journalctl@ceph.mon.a.smithi023.stdout: cache_index_and_filter_blocks: 1 2024-03-24T10:30:26.083 INFO:journalctl@ceph.mon.a.smithi023.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T10:30:26.083 INFO:journalctl@ceph.mon.a.smithi023.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T10:30:26.083 INFO:journalctl@ceph.mon.a.smithi023.stdout: pin_top_level_index_and_filter: 1 2024-03-24T10:30:26.083 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_type: 0 2024-03-24T10:30:26.083 INFO:journalctl@ceph.mon.a.smithi023.stdout: data_block_index_type: 0 2024-03-24T10:30:26.083 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_shortening: 1 2024-03-24T10:30:26.084 INFO:journalctl@ceph.mon.a.smithi023.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T10:30:26.084 INFO:journalctl@ceph.mon.a.smithi023.stdout: checksum: 4 2024-03-24T10:30:26.084 INFO:journalctl@ceph.mon.a.smithi023.stdout: no_block_cache: 0 2024-03-24T10:30:26.084 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache: 0x55e7689cf090 2024-03-24T10:30:26.084 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_name: BinnedLRUCache 2024-03-24T10:30:26.084 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_options: 2024-03-24T10:30:26.084 INFO:journalctl@ceph.mon.a.smithi023.stdout: capacity : 536870912 2024-03-24T10:30:26.084 INFO:journalctl@ceph.mon.a.smithi023.stdout: num_shard_bits : 4 2024-03-24T10:30:26.084 INFO:journalctl@ceph.mon.a.smithi023.stdout: strict_capacity_limit : 0 2024-03-24T10:30:26.084 INFO:journalctl@ceph.mon.a.smithi023.stdout: high_pri_pool_ratio: 0.000 2024-03-24T10:30:26.084 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_compressed: (nil) 2024-03-24T10:30:26.084 INFO:journalctl@ceph.mon.a.smithi023.stdout: persistent_cache: (nil) 2024-03-24T10:30:26.084 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_size: 4096 2024-03-24T10:30:26.084 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_size_deviation: 10 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_restart_interval: 16 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_block_restart_interval: 1 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: metadata_block_size: 4096 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: partition_filters: 0 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: use_delta_encoding: 1 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: filter_policy: bloomfilter 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: whole_key_filtering: 1 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: verify_compression: 0 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: read_amp_bytes_per_bit: 0 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: format_version: 5 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: enable_index_compression: 1 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_align: 0 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: max_auto_readahead_size: 262144 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: prepopulate_block_cache: 0 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: initial_auto_readahead_size: 8192 2024-03-24T10:30:26.085 INFO:journalctl@ceph.mon.a.smithi023.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T10:30:26.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T10:30:26.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T10:30:26.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compression: NoCompression 2024-03-24T10:30:26.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T10:30:26.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T10:30:26.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T10:30:26.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.num_levels: 7 2024-03-24T10:30:26.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T10:30:26.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T10:30:26.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T10:30:26.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T10:30:26.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T10:30:26.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T10:30:26.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T10:30:26.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T10:30:26.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T10:30:26.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T10:30:26.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T10:30:26.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T10:30:26.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T10:30:26.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T10:30:26.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T10:30:26.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T10:30:26.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T10:30:26.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T10:30:26.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T10:30:26.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T10:30:26.089 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T10:30:26.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T10:30:26.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T10:30:26.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T10:30:26.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.table_properties_collectors: 2024-03-24T10:30:26.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.inplace_update_support: 0 2024-03-24T10:30:26.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T10:30:26.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T10:30:26.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T10:30:26.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T10:30:26.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.bloom_locality: 0 2024-03-24T10:30:26.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.max_successive_merges: 0 2024-03-24T10:30:26.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T10:30:26.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T10:30:26.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T10:30:26.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T10:30:26.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.ttl: 2592000 2024-03-24T10:30:26.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T10:30:26.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T10:30:26.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T10:30:26.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.enable_blob_files: false 2024-03-24T10:30:26.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.min_blob_size: 0 2024-03-24T10:30:26.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T10:30:26.092 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T10:30:26.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T10:30:26.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T10:30:26.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T10:30:26.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T10:30:26.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T10:30:26.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T10:30:26.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T10:30:26.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T10:30:26.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1438232c-e633-46d4-b4f1-df03d32edfa8 2024-03-24T10:30:26.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711276226001442, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T10:30:26.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T10:30:26.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711276226002486, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 101060, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 332, "table_properties": {"data_size": 98971, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14524, "raw_average_key_size": 49, "raw_value_size": 91133, "raw_average_value_size": 313, "num_data_blocks": 12, "num_entries": 291, "num_filter_entries": 291, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711276226, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1438232c-e633-46d4-b4f1-df03d32edfa8", "db_session_id": "MLE4E55QSI5WXSN2OPVR", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T10:30:26.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711276226002579, "job": 1, "event": "recovery_finished"} 2024-03-24T10:30:26.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T10:30:26.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T10:30:26.094 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e768ac0000 2024-03-24T10:30:26.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: DB pointer 0x55e768aaa000 2024-03-24T10:30:26.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T10:30:26.095 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T10:30:26.095 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** DB Stats ** 2024-03-24T10:30:26.095 INFO:journalctl@ceph.mon.a.smithi023.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T10:30:26.095 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T10:30:26.095 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T10:30:26.095 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T10:30:26.095 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T10:30:26.095 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T10:30:26.095 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T10:30:26.095 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-24T10:30:26.095 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** Compaction Stats [default] ** 2024-03-24T10:30:26.096 INFO:journalctl@ceph.mon.a.smithi023.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T10:30:26.096 INFO:journalctl@ceph.mon.a.smithi023.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T10:30:26.096 INFO:journalctl@ceph.mon.a.smithi023.stdout: L0 2/0 100.53 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T10:30:26.096 INFO:journalctl@ceph.mon.a.smithi023.stdout: Sum 2/0 100.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T10:30:26.096 INFO:journalctl@ceph.mon.a.smithi023.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T10:30:26.096 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-24T10:30:26.096 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** Compaction Stats [default] ** 2024-03-24T10:30:26.096 INFO:journalctl@ceph.mon.a.smithi023.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T10:30:26.096 INFO:journalctl@ceph.mon.a.smithi023.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T10:30:26.096 INFO:journalctl@ceph.mon.a.smithi023.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T10:30:26.096 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-24T10:30:26.096 INFO:journalctl@ceph.mon.a.smithi023.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T10:30:26.096 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-24T10:30:26.096 INFO:journalctl@ceph.mon.a.smithi023.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T10:30:26.097 INFO:journalctl@ceph.mon.a.smithi023.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T10:30:26.097 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T10:30:26.097 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T10:30:26.097 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T10:30:26.097 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T10:30:26.097 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative compaction: 0.00 GB write, 27.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T10:30:26.097 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval compaction: 0.00 GB write, 27.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T10:30:26.097 INFO:journalctl@ceph.mon.a.smithi023.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T10:30:26.097 INFO:journalctl@ceph.mon.a.smithi023.stdout: Block cache BinnedLRUCache@0x55e7689cf090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-24T10:30:26.097 INFO:journalctl@ceph.mon.a.smithi023.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-24T10:30:26.097 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-24T10:30:26.097 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T10:30:26.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: starting mon.a rank 0 at public addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] at bind addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:30:26.097 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: mon.a@-1(???) e1 preinit fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:30:26.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: mon.a@-1(???).mds e1 new map 2024-03-24T10:30:26.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: mon.a@-1(???).mds e1 print_map 2024-03-24T10:30:26.098 INFO:journalctl@ceph.mon.a.smithi023.stdout: e1 2024-03-24T10:30:26.098 INFO:journalctl@ceph.mon.a.smithi023.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T10:30:26.098 INFO:journalctl@ceph.mon.a.smithi023.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-24T10:30:26.098 INFO:journalctl@ceph.mon.a.smithi023.stdout: legacy client fscid: -1 2024-03-24T10:30:26.098 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-24T10:30:26.098 INFO:journalctl@ceph.mon.a.smithi023.stdout: No filesystems configured 2024-03-24T10:30:26.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T10:30:26.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:30:26.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:30:26.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:30:26.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T10:30:26.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 bash[32447]: fc7c0651aa5d9f239e23074707cb733c8310e78c6cea809c0babdd5f5d7ac629 2024-03-24T10:30:26.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 systemd[1]: Started Ceph mon.a for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:30:26.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T10:30:26.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: monmap epoch 1 2024-03-24T10:30:26.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:30:26.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: last_changed 2024-03-24T10:30:17.512891+0000 2024-03-24T10:30:26.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: created 2024-03-24T10:30:17.512891+0000 2024-03-24T10:30:26.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: min_mon_release 19 (squid) 2024-03-24T10:30:26.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: election_strategy: 1 2024-03-24T10:30:26.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: 0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2024-03-24T10:30:26.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: fsmap 2024-03-24T10:30:26.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T10:30:26.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:26 smithi023 ceph-mon[32466]: mgrmap e1: no daemons active 2024-03-24T10:30:27.092 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.501+0000 7f8396b9c700 1 Processor -- start 2024-03-24T10:30:27.092 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.501+0000 7f8396b9c700 1 -- start start 2024-03-24T10:30:27.092 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.501+0000 7f8396b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8390108830 0x7f839010ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:27.092 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.501+0000 7f8396b9c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f839007bdb0 con 0x7f8390108830 2024-03-24T10:30:27.092 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.502+0000 7f8394938700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8390108830 0x7f839010ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:27.093 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.502+0000 7f8394938700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8390108830 0x7f839010ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47596/0 (socket says 172.21.15.23:47596) 2024-03-24T10:30:27.093 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.502+0000 7f8394938700 1 -- 172.21.15.23:0/1052938123 learned_addr learned my addr 172.21.15.23:0/1052938123 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:27.093 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.502+0000 7f8394938700 1 -- 172.21.15.23:0/1052938123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839010b160 con 0x7f8390108830 2024-03-24T10:30:27.093 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.502+0000 7f8394938700 1 --2- 172.21.15.23:0/1052938123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8390108830 0x7f839010ac20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8388009e80 tx=0x7f838800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec1e72dfd5cc1de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:27.093 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.502+0000 7f838f7fe700 1 -- 172.21.15.23:0/1052938123 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f838800e070 con 0x7f8390108830 2024-03-24T10:30:27.093 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.503+0000 7f838f7fe700 1 -- 172.21.15.23:0/1052938123 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f838800b8c0 con 0x7f8390108830 2024-03-24T10:30:27.093 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.503+0000 7f838f7fe700 1 -- 172.21.15.23:0/1052938123 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8388004550 con 0x7f8390108830 2024-03-24T10:30:27.093 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.503+0000 7f8396b9c700 1 -- 172.21.15.23:0/1052938123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8390108830 msgr2=0x7f839010ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:27.093 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.503+0000 7f8396b9c700 1 --2- 172.21.15.23:0/1052938123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8390108830 0x7f839010ac20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8388009e80 tx=0x7f838800b330 comp rx=0 tx=0).stop 2024-03-24T10:30:27.093 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.503+0000 7f8396b9c700 1 -- 172.21.15.23:0/1052938123 shutdown_connections 2024-03-24T10:30:27.093 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.503+0000 7f8396b9c700 1 --2- 172.21.15.23:0/1052938123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8390108830 0x7f839010ac20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:27.094 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.503+0000 7f8396b9c700 1 -- 172.21.15.23:0/1052938123 >> 172.21.15.23:0/1052938123 conn(0x7f8390102200 msgr2=0x7f8390104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:27.094 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.503+0000 7f8396b9c700 1 -- 172.21.15.23:0/1052938123 shutdown_connections 2024-03-24T10:30:27.094 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.503+0000 7f8396b9c700 1 -- 172.21.15.23:0/1052938123 wait complete. 2024-03-24T10:30:27.094 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.504+0000 7f8396b9c700 1 Processor -- start 2024-03-24T10:30:27.094 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.504+0000 7f8396b9c700 1 -- start start 2024-03-24T10:30:27.094 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.504+0000 7f8396b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8390108830 0x7f8390108290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:27.094 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.504+0000 7f8396b9c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f83901067e0 con 0x7f8390108830 2024-03-24T10:30:27.094 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.504+0000 7f8394938700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8390108830 0x7f8390108290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:27.094 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.504+0000 7f8394938700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8390108830 0x7f8390108290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47612/0 (socket says 172.21.15.23:47612) 2024-03-24T10:30:27.094 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.504+0000 7f8394938700 1 -- 172.21.15.23:0/206642106 learned_addr learned my addr 172.21.15.23:0/206642106 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:27.094 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.504+0000 7f8394938700 1 -- 172.21.15.23:0/206642106 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8388009b30 con 0x7f8390108830 2024-03-24T10:30:27.094 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.504+0000 7f8394938700 1 --2- 172.21.15.23:0/206642106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8390108830 0x7f8390108290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f83880042e0 tx=0x7f8388003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:27.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.505+0000 7f838dffb700 1 -- 172.21.15.23:0/206642106 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f838800e040 con 0x7f8390108830 2024-03-24T10:30:27.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.505+0000 7f838dffb700 1 -- 172.21.15.23:0/206642106 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f838800bd70 con 0x7f8390108830 2024-03-24T10:30:27.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.505+0000 7f838dffb700 1 -- 172.21.15.23:0/206642106 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f838801c7f0 con 0x7f8390108830 2024-03-24T10:30:27.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.505+0000 7f8396b9c700 1 -- 172.21.15.23:0/206642106 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83901069e0 con 0x7f8390108830 2024-03-24T10:30:27.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.505+0000 7f8396b9c700 1 -- 172.21.15.23:0/206642106 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8390106e60 con 0x7f8390108830 2024-03-24T10:30:27.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.505+0000 7f838dffb700 1 -- 172.21.15.23:0/206642106 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f838801cab0 con 0x7f8390108830 2024-03-24T10:30:27.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.505+0000 7f838dffb700 1 -- 172.21.15.23:0/206642106 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f838801b890 con 0x7f8390108830 2024-03-24T10:30:27.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.506+0000 7f8396b9c700 1 -- 172.21.15.23:0/206642106 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8390063830 con 0x7f8390108830 2024-03-24T10:30:27.095 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.508+0000 7f838dffb700 1 -- 172.21.15.23:0/206642106 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f838802a060 con 0x7f8390108830 2024-03-24T10:30:27.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.577+0000 7f8396b9c700 1 -- 172.21.15.23:0/206642106 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f83901071d0 con 0x7f8390108830 2024-03-24T10:30:27.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.579+0000 7f838dffb700 1 -- 172.21.15.23:0/206642106 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f8388020070 con 0x7f8390108830 2024-03-24T10:30:27.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.580+0000 7f8396b9c700 1 -- 172.21.15.23:0/206642106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8390108830 msgr2=0x7f8390108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:27.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.580+0000 7f8396b9c700 1 --2- 172.21.15.23:0/206642106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8390108830 0x7f8390108290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f83880042e0 tx=0x7f8388003d00 comp rx=0 tx=0).stop 2024-03-24T10:30:27.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.580+0000 7f8396b9c700 1 -- 172.21.15.23:0/206642106 shutdown_connections 2024-03-24T10:30:27.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.580+0000 7f8396b9c700 1 --2- 172.21.15.23:0/206642106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8390108830 0x7f8390108290 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:27.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.580+0000 7f8396b9c700 1 -- 172.21.15.23:0/206642106 >> 172.21.15.23:0/206642106 conn(0x7f8390102200 msgr2=0x7f8390102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:27.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.580+0000 7f8396b9c700 1 -- 172.21.15.23:0/206642106 shutdown_connections 2024-03-24T10:30:27.096 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:26.580+0000 7f8396b9c700 1 -- 172.21.15.23:0/206642106 wait complete. 2024-03-24T10:30:27.096 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T10:30:27.109 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T10:30:27.109 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mgr... 2024-03-24T10:30:27.109 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T10:30:27.110 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T10:30:27.357 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mgr.x 2024-03-24T10:30:27.357 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mgr.x.service: Unit ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mgr.x.service not loaded. 2024-03-24T10:30:27.503 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715.target.wants/ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@.service. 2024-03-24T10:30:27.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:27 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/206642106' entity='client.admin' 2024-03-24T10:30:28.020 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T10:30:28.020 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-24T10:30:28.028 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T10:30:28.028 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-24T10:30:28.028 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-24T10:30:28.028 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T10:30:28.036 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T10:30:28.036 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-24T10:30:28.043 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T10:30:28.044 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-24T10:30:28.044 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-24T10:30:28.044 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-24T10:30:28.044 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr to start... 2024-03-24T10:30:28.044 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr... 2024-03-24T10:30:28.168 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:28 smithi023 systemd[1]: Started Ceph mgr.x for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:30:28.836 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:28 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2801722365' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T10:30:28.837 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:28 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:28.492+0000 7f4aff348200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T10:30:28.837 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:28 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:28.591+0000 7f4aff348200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T10:30:29.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T10:30:29.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:30:29.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "4f8fb8f6-e9c9-11ee-95cd-87774f69a715", 2024-03-24T10:30:29.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T10:30:29.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T10:30:29.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T10:30:29.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T10:30:29.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:29.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T10:30:29.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T10:30:29.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-24T10:30:29.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:30:29.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T10:30:29.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T10:30:29.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:30:29.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-24T10:30:29.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T10:30:29.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:29.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T10:30:29.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T10:30:29.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:29.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T10:30:29.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:29.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T10:30:29.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T10:30:29.177 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T10:30:29.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T10:30:29.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T10:30:29.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T10:30:29.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:29.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T10:30:29.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T10:30:29.178 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T10:30:29.179 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T10:30:29.179 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T10:30:29.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T10:30:29.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T10:30:29.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T10:30:29.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T10:30:29.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:29.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T10:30:29.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:29.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T10:30:29.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T10:30:29.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:29.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T10:30:29.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T10:30:29.180 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T10:30:29.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T10:30:29.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T10:30:29.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T10:30:29.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T10:30:29.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:30:29.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:30:29.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:29.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T10:30:29.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:29.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T10:30:20.457402+0000", 2024-03-24T10:30:29.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:30:29.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:29.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T10:30:29.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:30:29.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.468+0000 7fc48e73b700 1 Processor -- start 2024-03-24T10:30:29.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.468+0000 7fc48e73b700 1 -- start start 2024-03-24T10:30:29.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.469+0000 7fc48e73b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4881060c0 0x7fc4881064e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:29.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.469+0000 7fc48e73b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc488106ab0 con 0x7fc4881060c0 2024-03-24T10:30:29.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.469+0000 7fc487fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4881060c0 0x7fc4881064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:29.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.469+0000 7fc487fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4881060c0 0x7fc4881064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36614/0 (socket says 172.21.15.23:36614) 2024-03-24T10:30:29.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.469+0000 7fc487fff700 1 -- 172.21.15.23:0/1025212315 learned_addr learned my addr 172.21.15.23:0/1025212315 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:29.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.469+0000 7fc487fff700 1 -- 172.21.15.23:0/1025212315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc488109060 con 0x7fc4881060c0 2024-03-24T10:30:29.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.469+0000 7fc487fff700 1 --2- 172.21.15.23:0/1025212315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4881060c0 0x7fc4881064e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc4780098a0 tx=0x7fc478009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1f86110cb7b29c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:29.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.470+0000 7fc486ffd700 1 -- 172.21.15.23:0/1025212315 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc478004160 con 0x7fc4881060c0 2024-03-24T10:30:29.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.470+0000 7fc486ffd700 1 -- 172.21.15.23:0/1025212315 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc4780042c0 con 0x7fc4881060c0 2024-03-24T10:30:29.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.470+0000 7fc486ffd700 1 -- 172.21.15.23:0/1025212315 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc478011ac0 con 0x7fc4881060c0 2024-03-24T10:30:29.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.470+0000 7fc48e73b700 1 -- 172.21.15.23:0/1025212315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4881060c0 msgr2=0x7fc4881064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:29.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.470+0000 7fc48e73b700 1 --2- 172.21.15.23:0/1025212315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4881060c0 0x7fc4881064e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc4780098a0 tx=0x7fc478009bb0 comp rx=0 tx=0).stop 2024-03-24T10:30:29.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.470+0000 7fc48e73b700 1 -- 172.21.15.23:0/1025212315 shutdown_connections 2024-03-24T10:30:29.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.470+0000 7fc48e73b700 1 --2- 172.21.15.23:0/1025212315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4881060c0 0x7fc4881064e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:29.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.470+0000 7fc48e73b700 1 -- 172.21.15.23:0/1025212315 >> 172.21.15.23:0/1025212315 conn(0x7fc4881019e0 msgr2=0x7fc488103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:29.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.471+0000 7fc48e73b700 1 -- 172.21.15.23:0/1025212315 shutdown_connections 2024-03-24T10:30:29.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.471+0000 7fc48e73b700 1 -- 172.21.15.23:0/1025212315 wait complete. 2024-03-24T10:30:29.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.471+0000 7fc48e73b700 1 Processor -- start 2024-03-24T10:30:29.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.471+0000 7fc48e73b700 1 -- start start 2024-03-24T10:30:29.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.471+0000 7fc48e73b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4881060c0 0x7fc48819bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:29.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.471+0000 7fc48e73b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc48819c150 con 0x7fc4881060c0 2024-03-24T10:30:29.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.472+0000 7fc487fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4881060c0 0x7fc48819bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:29.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.472+0000 7fc487fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4881060c0 0x7fc48819bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36618/0 (socket says 172.21.15.23:36618) 2024-03-24T10:30:29.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.472+0000 7fc487fff700 1 -- 172.21.15.23:0/2801722365 learned_addr learned my addr 172.21.15.23:0/2801722365 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:29.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.472+0000 7fc487fff700 1 -- 172.21.15.23:0/2801722365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc478009550 con 0x7fc4881060c0 2024-03-24T10:30:29.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.472+0000 7fc487fff700 1 --2- 172.21.15.23:0/2801722365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4881060c0 0x7fc48819bc10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc478009130 tx=0x7fc478011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:29.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.472+0000 7fc4857fa700 1 -- 172.21.15.23:0/2801722365 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc478003aa0 con 0x7fc4881060c0 2024-03-24T10:30:29.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.472+0000 7fc4857fa700 1 -- 172.21.15.23:0/2801722365 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc478003c00 con 0x7fc4881060c0 2024-03-24T10:30:29.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.472+0000 7fc48e73b700 1 -- 172.21.15.23:0/2801722365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc48819c350 con 0x7fc4881060c0 2024-03-24T10:30:29.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.472+0000 7fc4857fa700 1 -- 172.21.15.23:0/2801722365 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc478024800 con 0x7fc4881060c0 2024-03-24T10:30:29.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.472+0000 7fc48e73b700 1 -- 172.21.15.23:0/2801722365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc48819cf00 con 0x7fc4881060c0 2024-03-24T10:30:29.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.473+0000 7fc4857fa700 1 -- 172.21.15.23:0/2801722365 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc47802d3f0 con 0x7fc4881060c0 2024-03-24T10:30:29.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.473+0000 7fc4857fa700 1 -- 172.21.15.23:0/2801722365 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc4780197b0 con 0x7fc4881060c0 2024-03-24T10:30:29.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.473+0000 7fc48e73b700 1 -- 172.21.15.23:0/2801722365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc488063830 con 0x7fc4881060c0 2024-03-24T10:30:29.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.476+0000 7fc4857fa700 1 -- 172.21.15.23:0/2801722365 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc47801abd0 con 0x7fc4881060c0 2024-03-24T10:30:29.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.550+0000 7fc48e73b700 1 -- 172.21.15.23:0/2801722365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc48806a7d0 con 0x7fc4881060c0 2024-03-24T10:30:29.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.550+0000 7fc4857fa700 1 -- 172.21.15.23:0/2801722365 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fc47801ad70 con 0x7fc4881060c0 2024-03-24T10:30:29.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.551+0000 7fc48e73b700 1 -- 172.21.15.23:0/2801722365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4881060c0 msgr2=0x7fc48819bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:29.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.551+0000 7fc48e73b700 1 --2- 172.21.15.23:0/2801722365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4881060c0 0x7fc48819bc10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc478009130 tx=0x7fc478011b40 comp rx=0 tx=0).stop 2024-03-24T10:30:29.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.551+0000 7fc48e73b700 1 -- 172.21.15.23:0/2801722365 shutdown_connections 2024-03-24T10:30:29.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.551+0000 7fc48e73b700 1 --2- 172.21.15.23:0/2801722365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4881060c0 0x7fc48819bc10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:29.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.551+0000 7fc48e73b700 1 -- 172.21.15.23:0/2801722365 >> 172.21.15.23:0/2801722365 conn(0x7fc4881019e0 msgr2=0x7fc4881024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:29.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.551+0000 7fc48e73b700 1 -- 172.21.15.23:0/2801722365 shutdown_connections 2024-03-24T10:30:29.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:28.551+0000 7fc48e73b700 1 -- 172.21.15.23:0/2801722365 wait complete. 2024-03-24T10:30:29.186 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (1/15)... 2024-03-24T10:30:29.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:29 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:29.257+0000 7f4aff348200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T10:30:29.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:29 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:29.352+0000 7f4aff348200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T10:30:29.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:29 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:29.494+0000 7f4aff348200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T10:30:29.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:29 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:29.591+0000 7f4aff348200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T10:30:29.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:29 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:29.676+0000 7f4aff348200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T10:30:30.240 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:29 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:29.942+0000 7f4aff348200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T10:30:30.241 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:30.056+0000 7f4aff348200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T10:30:30.241 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:30.144+0000 7f4aff348200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T10:30:30.241 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:30.240+0000 7f4aff348200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T10:30:30.631 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:30.416+0000 7f4aff348200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T10:30:31.585 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:31 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:31.448+0000 7f4aff348200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T10:30:31.585 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:31 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:31.584+0000 7f4aff348200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T10:30:32.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:31 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/119857575' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T10:30:32.168 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:31 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:31.938+0000 7f4aff348200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T10:30:32.168 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:32 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:32.031+0000 7f4aff348200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T10:30:32.536 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:32 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:32.233+0000 7f4aff348200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:30:32.536 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:32 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:32.423+0000 7f4aff348200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T10:30:32.538 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T10:30:32.540 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:30:32.540 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "4f8fb8f6-e9c9-11ee-95cd-87774f69a715", 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T10:30:32.541 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:32.542 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T10:30:32.542 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T10:30:32.545 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:32.545 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T10:30:32.545 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:32.545 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T10:30:32.545 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T10:30:32.581 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T10:30:32.581 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T10:30:32.581 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T10:30:32.581 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T10:30:32.581 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:32.581 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T10:30:32.581 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T10:30:32.581 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T10:30:32.581 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T10:30:32.581 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T10:30:32.581 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T10:30:32.581 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T10:30:32.581 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T10:30:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T10:30:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T10:30:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T10:30:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T10:30:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T10:30:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T10:30:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T10:30:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T10:30:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T10:30:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T10:30:32.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T10:30:20.457402+0000", 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.715+0000 7f6f7cebd700 1 Processor -- start 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.715+0000 7f6f7cebd700 1 -- start start 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.715+0000 7f6f7cebd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f78108820 0x7f6f7810ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.715+0000 7f6f7cebd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7807bdb0 con 0x7f6f78108820 2024-03-24T10:30:32.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.716+0000 7f6f7659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f78108820 0x7f6f7810ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:32.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.716+0000 7f6f7659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f78108820 0x7f6f7810ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36634/0 (socket says 172.21.15.23:36634) 2024-03-24T10:30:32.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.716+0000 7f6f7659c700 1 -- 172.21.15.23:0/71408172 learned_addr learned my addr 172.21.15.23:0/71408172 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:32.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.716+0000 7f6f7659c700 1 -- 172.21.15.23:0/71408172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7810b150 con 0x7f6f78108820 2024-03-24T10:30:32.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.716+0000 7f6f7659c700 1 --2- 172.21.15.23:0/71408172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f78108820 0x7f6f7810ac10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f680098a0 tx=0x7f6f68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17daa2b64676c660 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:32.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.717+0000 7f6f7559a700 1 -- 172.21.15.23:0/71408172 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f68004160 con 0x7f6f78108820 2024-03-24T10:30:32.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.717+0000 7f6f7559a700 1 -- 172.21.15.23:0/71408172 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6f680042c0 con 0x7f6f78108820 2024-03-24T10:30:32.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.717+0000 7f6f7559a700 1 -- 172.21.15.23:0/71408172 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f68011ac0 con 0x7f6f78108820 2024-03-24T10:30:32.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.717+0000 7f6f7cebd700 1 -- 172.21.15.23:0/71408172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f78108820 msgr2=0x7f6f7810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:32.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.717+0000 7f6f7cebd700 1 --2- 172.21.15.23:0/71408172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f78108820 0x7f6f7810ac10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f680098a0 tx=0x7f6f68009bb0 comp rx=0 tx=0).stop 2024-03-24T10:30:32.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.718+0000 7f6f7cebd700 1 -- 172.21.15.23:0/71408172 shutdown_connections 2024-03-24T10:30:32.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.718+0000 7f6f7cebd700 1 --2- 172.21.15.23:0/71408172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f78108820 0x7f6f7810ac10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:32.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.718+0000 7f6f7cebd700 1 -- 172.21.15.23:0/71408172 >> 172.21.15.23:0/71408172 conn(0x7f6f78102290 msgr2=0x7f6f781046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:32.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.718+0000 7f6f7cebd700 1 -- 172.21.15.23:0/71408172 shutdown_connections 2024-03-24T10:30:32.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.718+0000 7f6f7cebd700 1 -- 172.21.15.23:0/71408172 wait complete. 2024-03-24T10:30:32.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.718+0000 7f6f7cebd700 1 Processor -- start 2024-03-24T10:30:32.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.718+0000 7f6f7cebd700 1 -- start start 2024-03-24T10:30:32.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.719+0000 7f6f7cebd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f78108820 0x7f6f7819bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:32.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.719+0000 7f6f7cebd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7819c120 con 0x7f6f78108820 2024-03-24T10:30:32.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.719+0000 7f6f7659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f78108820 0x7f6f7819bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:32.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.719+0000 7f6f7659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f78108820 0x7f6f7819bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36642/0 (socket says 172.21.15.23:36642) 2024-03-24T10:30:32.592 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.719+0000 7f6f7659c700 1 -- 172.21.15.23:0/119857575 learned_addr learned my addr 172.21.15.23:0/119857575 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:32.592 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.719+0000 7f6f7659c700 1 -- 172.21.15.23:0/119857575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f68009550 con 0x7f6f78108820 2024-03-24T10:30:32.592 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.719+0000 7f6f7659c700 1 --2- 172.21.15.23:0/119857575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f78108820 0x7f6f7819bbe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f68003f70 tx=0x7f6f68011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:32.592 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.719+0000 7f6f677fe700 1 -- 172.21.15.23:0/119857575 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f680039f0 con 0x7f6f78108820 2024-03-24T10:30:32.592 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.719+0000 7f6f677fe700 1 -- 172.21.15.23:0/119857575 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6f68003b50 con 0x7f6f78108820 2024-03-24T10:30:32.592 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.719+0000 7f6f7cebd700 1 -- 172.21.15.23:0/119857575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f7819c320 con 0x7f6f78108820 2024-03-24T10:30:32.592 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.719+0000 7f6f677fe700 1 -- 172.21.15.23:0/119857575 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f6801a760 con 0x7f6f78108820 2024-03-24T10:30:32.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.720+0000 7f6f7cebd700 1 -- 172.21.15.23:0/119857575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f7819ced0 con 0x7f6f78108820 2024-03-24T10:30:32.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.720+0000 7f6f677fe700 1 -- 172.21.15.23:0/119857575 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6f6801ac50 con 0x7f6f78108820 2024-03-24T10:30:32.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.720+0000 7f6f677fe700 1 -- 172.21.15.23:0/119857575 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6f680199f0 con 0x7f6f78108820 2024-03-24T10:30:32.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.720+0000 7f6f7cebd700 1 -- 172.21.15.23:0/119857575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f7819c7c0 con 0x7f6f78108820 2024-03-24T10:30:32.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.723+0000 7f6f677fe700 1 -- 172.21.15.23:0/119857575 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6f7819c7c0 con 0x7f6f78108820 2024-03-24T10:30:32.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.797+0000 7f6f7cebd700 1 -- 172.21.15.23:0/119857575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f6f78063830 con 0x7f6f78108820 2024-03-24T10:30:32.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.797+0000 7f6f677fe700 1 -- 172.21.15.23:0/119857575 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f6f680249f0 con 0x7f6f78108820 2024-03-24T10:30:32.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.798+0000 7f6f7cebd700 1 -- 172.21.15.23:0/119857575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f78108820 msgr2=0x7f6f7819bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:32.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.798+0000 7f6f7cebd700 1 --2- 172.21.15.23:0/119857575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f78108820 0x7f6f7819bbe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f68003f70 tx=0x7f6f68011b40 comp rx=0 tx=0).stop 2024-03-24T10:30:32.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.798+0000 7f6f7cebd700 1 -- 172.21.15.23:0/119857575 shutdown_connections 2024-03-24T10:30:32.593 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.798+0000 7f6f7cebd700 1 --2- 172.21.15.23:0/119857575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f78108820 0x7f6f7819bbe0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:32.594 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.798+0000 7f6f7cebd700 1 -- 172.21.15.23:0/119857575 >> 172.21.15.23:0/119857575 conn(0x7f6f78102290 msgr2=0x7f6f78102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:32.594 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.798+0000 7f6f7cebd700 1 -- 172.21.15.23:0/119857575 shutdown_connections 2024-03-24T10:30:32.594 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:31.798+0000 7f6f7cebd700 1 -- 172.21.15.23:0/119857575 wait complete. 2024-03-24T10:30:32.594 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (2/15)... 2024-03-24T10:30:32.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:32 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:32.715+0000 7f4aff348200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:30:33.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:32 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:32.976+0000 7f4aff348200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T10:30:33.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:33 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:33.218+0000 7f4aff348200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T10:30:34.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:34 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:34.042+0000 7f4aff348200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T10:30:34.885 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:34 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:34.632+0000 7f4aff348200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T10:30:34.885 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:34 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:34.732+0000 7f4aff348200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T10:30:35.147 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:34 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:34.955+0000 7f4aff348200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T10:30:35.147 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:35 smithi023 ceph-mon[32466]: Activating manager daemon x 2024-03-24T10:30:35.147 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:35 smithi023 ceph-mon[32466]: mgrmap e2: x(active, starting, since 0.00326317s) 2024-03-24T10:30:35.147 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:35 smithi023 ceph-mon[32466]: from='mgr.14100 172.21.15.23:0/2717061222' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T10:30:35.147 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:35 smithi023 ceph-mon[32466]: from='mgr.14100 172.21.15.23:0/2717061222' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T10:30:35.147 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:35 smithi023 ceph-mon[32466]: from='mgr.14100 172.21.15.23:0/2717061222' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T10:30:35.147 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:35 smithi023 ceph-mon[32466]: from='mgr.14100 172.21.15.23:0/2717061222' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:30:35.147 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:35 smithi023 ceph-mon[32466]: from='mgr.14100 172.21.15.23:0/2717061222' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T10:30:35.147 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:35 smithi023 ceph-mon[32466]: Manager daemon x is now available 2024-03-24T10:30:35.148 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:35 smithi023 ceph-mon[32466]: from='mgr.14100 172.21.15.23:0/2717061222' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T10:30:35.148 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:35 smithi023 ceph-mon[32466]: from='mgr.14100 172.21.15.23:0/2717061222' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T10:30:35.148 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:35 smithi023 ceph-mon[32466]: from='mgr.14100 172.21.15.23:0/2717061222' entity='mgr.x' 2024-03-24T10:30:35.148 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:35 smithi023 ceph-mon[32466]: from='mgr.14100 172.21.15.23:0/2717061222' entity='mgr.x' 2024-03-24T10:30:35.148 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:35 smithi023 ceph-mon[32466]: from='mgr.14100 172.21.15.23:0/2717061222' entity='mgr.x' 2024-03-24T10:30:35.676 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T10:30:35.676 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:30:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "4f8fb8f6-e9c9-11ee-95cd-87774f69a715", 2024-03-24T10:30:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T10:30:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T10:30:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T10:30:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T10:30:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T10:30:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T10:30:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-24T10:30:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:30:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T10:30:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T10:30:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:30:35.677 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-24T10:30:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T10:30:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T10:30:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T10:30:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T10:30:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T10:30:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T10:30:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T10:30:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T10:30:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T10:30:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T10:30:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T10:30:35.679 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:35.680 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T10:30:35.680 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T10:30:35.680 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T10:30:35.680 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T10:30:35.680 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T10:30:35.680 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T10:30:35.680 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T10:30:35.680 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:30:35.680 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:30:35.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:35.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T10:30:35.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:35.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T10:30:20.457402+0000", 2024-03-24T10:30:35.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:30:35.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:35.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T10:30:35.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:30:35.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.067+0000 7f2f39a33700 1 Processor -- start 2024-03-24T10:30:35.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.067+0000 7f2f39a33700 1 -- start start 2024-03-24T10:30:35.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.067+0000 7f2f39a33700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f34108810 0x7f2f3410ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:35.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.067+0000 7f2f39a33700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3407bdf0 con 0x7f2f34108810 2024-03-24T10:30:35.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.067+0000 7f2f32ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f34108810 0x7f2f3410ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:35.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.067+0000 7f2f32ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f34108810 0x7f2f3410ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36712/0 (socket says 172.21.15.23:36712) 2024-03-24T10:30:35.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.068+0000 7f2f32ffd700 1 -- 172.21.15.23:0/361761425 learned_addr learned my addr 172.21.15.23:0/361761425 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:35.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.068+0000 7f2f32ffd700 1 -- 172.21.15.23:0/361761425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3410b140 con 0x7f2f34108810 2024-03-24T10:30:35.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.068+0000 7f2f32ffd700 1 --2- 172.21.15.23:0/361761425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f34108810 0x7f2f3410ac00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2f240098a0 tx=0x7f2f24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10cf22e15ab9cc94 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:35.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.068+0000 7f2f31ffb700 1 -- 172.21.15.23:0/361761425 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f24004160 con 0x7f2f34108810 2024-03-24T10:30:35.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.068+0000 7f2f31ffb700 1 -- 172.21.15.23:0/361761425 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2f240042c0 con 0x7f2f34108810 2024-03-24T10:30:35.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.068+0000 7f2f31ffb700 1 -- 172.21.15.23:0/361761425 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f24011ac0 con 0x7f2f34108810 2024-03-24T10:30:35.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.069+0000 7f2f39a33700 1 -- 172.21.15.23:0/361761425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f34108810 msgr2=0x7f2f3410ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:35.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.069+0000 7f2f39a33700 1 --2- 172.21.15.23:0/361761425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f34108810 0x7f2f3410ac00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2f240098a0 tx=0x7f2f24009bb0 comp rx=0 tx=0).stop 2024-03-24T10:30:35.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.069+0000 7f2f39a33700 1 -- 172.21.15.23:0/361761425 shutdown_connections 2024-03-24T10:30:35.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.069+0000 7f2f39a33700 1 --2- 172.21.15.23:0/361761425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f34108810 0x7f2f3410ac00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:35.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.069+0000 7f2f39a33700 1 -- 172.21.15.23:0/361761425 >> 172.21.15.23:0/361761425 conn(0x7f2f34102220 msgr2=0x7f2f34104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:35.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.069+0000 7f2f39a33700 1 -- 172.21.15.23:0/361761425 shutdown_connections 2024-03-24T10:30:35.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.069+0000 7f2f39a33700 1 -- 172.21.15.23:0/361761425 wait complete. 2024-03-24T10:30:35.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.069+0000 7f2f39a33700 1 Processor -- start 2024-03-24T10:30:35.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.070+0000 7f2f39a33700 1 -- start start 2024-03-24T10:30:35.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.070+0000 7f2f39a33700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f34108810 0x7f2f3419bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:35.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.070+0000 7f2f39a33700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3419c160 con 0x7f2f34108810 2024-03-24T10:30:35.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.070+0000 7f2f32ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f34108810 0x7f2f3419bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:35.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.070+0000 7f2f32ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f34108810 0x7f2f3419bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36724/0 (socket says 172.21.15.23:36724) 2024-03-24T10:30:35.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.070+0000 7f2f32ffd700 1 -- 172.21.15.23:0/2876026136 learned_addr learned my addr 172.21.15.23:0/2876026136 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:35.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.070+0000 7f2f32ffd700 1 -- 172.21.15.23:0/2876026136 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f24009550 con 0x7f2f34108810 2024-03-24T10:30:35.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.070+0000 7f2f32ffd700 1 --2- 172.21.15.23:0/2876026136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f34108810 0x7f2f3419bc20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2f24003f70 tx=0x7f2f24011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.070+0000 7f2f38a31700 1 -- 172.21.15.23:0/2876026136 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f240039f0 con 0x7f2f34108810 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.070+0000 7f2f38a31700 1 -- 172.21.15.23:0/2876026136 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2f24003b50 con 0x7f2f34108810 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.070+0000 7f2f39a33700 1 -- 172.21.15.23:0/2876026136 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f3419c360 con 0x7f2f34108810 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.070+0000 7f2f38a31700 1 -- 172.21.15.23:0/2876026136 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f2401a760 con 0x7f2f34108810 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.071+0000 7f2f39a33700 1 -- 172.21.15.23:0/2876026136 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f3419cf10 con 0x7f2f34108810 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.071+0000 7f2f38a31700 1 -- 172.21.15.23:0/2876026136 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 2) ==== 45684+0+0 (secure 0 0 0) 0x7f2f2401a900 con 0x7f2f34108810 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.072+0000 7f2f38a31700 1 -- 172.21.15.23:0/2876026136 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2f2404cea0 con 0x7f2f34108810 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.072+0000 7f2f39a33700 1 -- 172.21.15.23:0/2876026136 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f34063830 con 0x7f2f34108810 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.074+0000 7f2f38a31700 1 -- 172.21.15.23:0/2876026136 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2f2401abb0 con 0x7f2f34108810 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.146+0000 7f2f39a33700 1 -- 172.21.15.23:0/2876026136 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2f3406a7d0 con 0x7f2f34108810 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.147+0000 7f2f38a31700 1 -- 172.21.15.23:0/2876026136 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2f24018cb0 con 0x7f2f34108810 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.147+0000 7f2f39a33700 1 -- 172.21.15.23:0/2876026136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f34108810 msgr2=0x7f2f3419bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.147+0000 7f2f39a33700 1 --2- 172.21.15.23:0/2876026136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f34108810 0x7f2f3419bc20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2f24003f70 tx=0x7f2f24011b40 comp rx=0 tx=0).stop 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.147+0000 7f2f39a33700 1 -- 172.21.15.23:0/2876026136 shutdown_connections 2024-03-24T10:30:35.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.147+0000 7f2f39a33700 1 --2- 172.21.15.23:0/2876026136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f34108810 0x7f2f3419bc20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:35.685 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.147+0000 7f2f39a33700 1 -- 172.21.15.23:0/2876026136 >> 172.21.15.23:0/2876026136 conn(0x7f2f34102220 msgr2=0x7f2f34102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:35.685 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.148+0000 7f2f39a33700 1 -- 172.21.15.23:0/2876026136 shutdown_connections 2024-03-24T10:30:35.685 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:35.148+0000 7f2f39a33700 1 -- 172.21.15.23:0/2876026136 wait complete. 2024-03-24T10:30:35.685 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (3/15)... 2024-03-24T10:30:36.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:36 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2876026136' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T10:30:36.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:36 smithi023 ceph-mon[32466]: mgrmap e3: x(active, since 1.00727s) 2024-03-24T10:30:38.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:38 smithi023 ceph-mon[32466]: mgrmap e4: x(active, since 2s) 2024-03-24T10:30:38.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T10:30:38.830 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:30:38.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "4f8fb8f6-e9c9-11ee-95cd-87774f69a715", 2024-03-24T10:30:38.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T10:30:38.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T10:30:38.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T10:30:38.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T10:30:38.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:38.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T10:30:38.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T10:30:38.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-24T10:30:38.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:30:38.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T10:30:38.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T10:30:38.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:30:38.831 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:38.832 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T10:30:38.834 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T10:30:38.834 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T10:30:38.834 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T10:30:38.834 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T10:30:38.834 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T10:30:38.834 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T10:30:38.834 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-24T10:30:38.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:30:38.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:38.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T10:30:38.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T10:30:38.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T10:30:20.457402+0000", 2024-03-24T10:30:38.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T10:30:38.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-24T10:30:38.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T10:30:38.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:30:38.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.063+0000 7f2758e4e700 1 Processor -- start 2024-03-24T10:30:38.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.063+0000 7f2758e4e700 1 -- start start 2024-03-24T10:30:38.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.064+0000 7f2758e4e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2754107d80 0x7f27541081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:38.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.064+0000 7f2758e4e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2754108770 con 0x7f2754107d80 2024-03-24T10:30:38.835 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.064+0000 7f275259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2754107d80 0x7f27541081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:38.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.064+0000 7f275259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2754107d80 0x7f27541081a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34870/0 (socket says 172.21.15.23:34870) 2024-03-24T10:30:38.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.064+0000 7f275259c700 1 -- 172.21.15.23:0/518191108 learned_addr learned my addr 172.21.15.23:0/518191108 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:38.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.064+0000 7f275259c700 1 -- 172.21.15.23:0/518191108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2754108f80 con 0x7f2754107d80 2024-03-24T10:30:38.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.065+0000 7f275259c700 1 --2- 172.21.15.23:0/518191108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2754107d80 0x7f27541081a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f27440098a0 tx=0x7f2744009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712bcb0a1fcbc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:38.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.065+0000 7f275159a700 1 -- 172.21.15.23:0/518191108 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2744004160 con 0x7f2754107d80 2024-03-24T10:30:38.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.065+0000 7f275159a700 1 -- 172.21.15.23:0/518191108 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f27440042c0 con 0x7f2754107d80 2024-03-24T10:30:38.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.065+0000 7f275159a700 1 -- 172.21.15.23:0/518191108 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2744011ac0 con 0x7f2754107d80 2024-03-24T10:30:38.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.065+0000 7f2758e4e700 1 -- 172.21.15.23:0/518191108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2754107d80 msgr2=0x7f27541081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:38.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.065+0000 7f2758e4e700 1 --2- 172.21.15.23:0/518191108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2754107d80 0x7f27541081a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f27440098a0 tx=0x7f2744009bb0 comp rx=0 tx=0).stop 2024-03-24T10:30:38.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.066+0000 7f2758e4e700 1 -- 172.21.15.23:0/518191108 shutdown_connections 2024-03-24T10:30:38.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.066+0000 7f2758e4e700 1 --2- 172.21.15.23:0/518191108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2754107d80 0x7f27541081a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:38.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.066+0000 7f2758e4e700 1 -- 172.21.15.23:0/518191108 >> 172.21.15.23:0/518191108 conn(0x7f2754103360 msgr2=0x7f2754105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:38.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.066+0000 7f2758e4e700 1 -- 172.21.15.23:0/518191108 shutdown_connections 2024-03-24T10:30:38.836 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.066+0000 7f2758e4e700 1 -- 172.21.15.23:0/518191108 wait complete. 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.066+0000 7f2758e4e700 1 Processor -- start 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.066+0000 7f2758e4e700 1 -- start start 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.067+0000 7f2758e4e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2754107d80 0x7f275419bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.067+0000 7f2758e4e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f275419c1e0 con 0x7f2754107d80 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.067+0000 7f275259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2754107d80 0x7f275419bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.067+0000 7f275259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2754107d80 0x7f275419bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34884/0 (socket says 172.21.15.23:34884) 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.067+0000 7f275259c700 1 -- 172.21.15.23:0/1574188510 learned_addr learned my addr 172.21.15.23:0/1574188510 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.067+0000 7f275259c700 1 -- 172.21.15.23:0/1574188510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2744009550 con 0x7f2754107d80 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.067+0000 7f275259c700 1 --2- 172.21.15.23:0/1574188510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2754107d80 0x7f275419bca0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2744003f70 tx=0x7f2744011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.067+0000 7f274b7fe700 1 -- 172.21.15.23:0/1574188510 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27440039f0 con 0x7f2754107d80 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.067+0000 7f274b7fe700 1 -- 172.21.15.23:0/1574188510 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2744003b50 con 0x7f2754107d80 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.067+0000 7f2758e4e700 1 -- 172.21.15.23:0/1574188510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f275419c3e0 con 0x7f2754107d80 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.067+0000 7f274b7fe700 1 -- 172.21.15.23:0/1574188510 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f274401a690 con 0x7f2754107d80 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.068+0000 7f2758e4e700 1 -- 172.21.15.23:0/1574188510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f275419cf90 con 0x7f2754107d80 2024-03-24T10:30:38.837 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.068+0000 7f274b7fe700 1 -- 172.21.15.23:0/1574188510 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) ==== 45917+0+0 (secure 0 0 0) 0x7f274401a970 con 0x7f2754107d80 2024-03-24T10:30:38.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.068+0000 7f2758e4e700 1 -- 172.21.15.23:0/1574188510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2754063830 con 0x7f2754107d80 2024-03-24T10:30:38.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.069+0000 7f274b7fe700 1 --2- 172.21.15.23:0/1574188510 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f2740038e40 0x7f274003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:38.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.069+0000 7f274b7fe700 1 -- 172.21.15.23:0/1574188510 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f274404d100 con 0x7f2754107d80 2024-03-24T10:30:38.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.069+0000 7f2751d9b700 1 --2- 172.21.15.23:0/1574188510 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f2740038e40 0x7f274003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:38.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.069+0000 7f2751d9b700 1 --2- 172.21.15.23:0/1574188510 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f2740038e40 0x7f274003b300 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f273c006fd0 tx=0x7f273c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:38.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.074+0000 7f274b7fe700 1 -- 172.21.15.23:0/1574188510 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f274401ac20 con 0x7f2754107d80 2024-03-24T10:30:38.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.362+0000 7f2758e4e700 1 -- 172.21.15.23:0/1574188510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f275406a7d0 con 0x7f2754107d80 2024-03-24T10:30:38.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.362+0000 7f274b7fe700 1 -- 172.21.15.23:0/1574188510 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2744018350 con 0x7f2754107d80 2024-03-24T10:30:38.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.364+0000 7f2758e4e700 1 -- 172.21.15.23:0/1574188510 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f2740038e40 msgr2=0x7f274003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:38.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.364+0000 7f2758e4e700 1 --2- 172.21.15.23:0/1574188510 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f2740038e40 0x7f274003b300 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f273c006fd0 tx=0x7f273c006e40 comp rx=0 tx=0).stop 2024-03-24T10:30:38.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.364+0000 7f2758e4e700 1 -- 172.21.15.23:0/1574188510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2754107d80 msgr2=0x7f275419bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:38.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.364+0000 7f2758e4e700 1 --2- 172.21.15.23:0/1574188510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2754107d80 0x7f275419bca0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2744003f70 tx=0x7f2744011b40 comp rx=0 tx=0).stop 2024-03-24T10:30:38.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.365+0000 7f2758e4e700 1 -- 172.21.15.23:0/1574188510 shutdown_connections 2024-03-24T10:30:38.838 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.365+0000 7f2758e4e700 1 --2- 172.21.15.23:0/1574188510 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f2740038e40 0x7f274003b300 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:38.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.365+0000 7f2758e4e700 1 --2- 172.21.15.23:0/1574188510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2754107d80 0x7f275419bca0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:38.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.365+0000 7f2758e4e700 1 -- 172.21.15.23:0/1574188510 >> 172.21.15.23:0/1574188510 conn(0x7f2754103360 msgr2=0x7f2754103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:38.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.365+0000 7f2758e4e700 1 -- 172.21.15.23:0/1574188510 shutdown_connections 2024-03-24T10:30:38.839 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:38.365+0000 7f2758e4e700 1 -- 172.21.15.23:0/1574188510 wait complete. 2024-03-24T10:30:38.839 INFO:teuthology.orchestra.run.smithi023.stdout:mgr is available 2024-03-24T10:30:39.101 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:39 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1574188510' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T10:30:39.964 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T10:30:39.964 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T10:30:39.964 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:30:39.964 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.200+0000 7fdc33809700 1 Processor -- start 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.201+0000 7fdc33809700 1 -- start start 2024-03-24T10:30:39.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.201+0000 7fdc33809700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc2c106cc0 0x7fdc2c1070e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:39.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.201+0000 7fdc33809700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c1076b0 con 0x7fdc2c106cc0 2024-03-24T10:30:39.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.201+0000 7fdc315a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc2c106cc0 0x7fdc2c1070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:39.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.201+0000 7fdc315a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc2c106cc0 0x7fdc2c1070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34890/0 (socket says 172.21.15.23:34890) 2024-03-24T10:30:39.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.201+0000 7fdc315a5700 1 -- 172.21.15.23:0/184076644 learned_addr learned my addr 172.21.15.23:0/184076644 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:39.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.201+0000 7fdc315a5700 1 -- 172.21.15.23:0/184076644 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2c107ec0 con 0x7fdc2c106cc0 2024-03-24T10:30:39.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.201+0000 7fdc315a5700 1 --2- 172.21.15.23:0/184076644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc2c106cc0 0x7fdc2c1070e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdc280098a0 tx=0x7fdc28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4039c81e256cc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:39.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.202+0000 7fdc23fff700 1 -- 172.21.15.23:0/184076644 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc28004160 con 0x7fdc2c106cc0 2024-03-24T10:30:39.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.202+0000 7fdc23fff700 1 -- 172.21.15.23:0/184076644 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdc280042c0 con 0x7fdc2c106cc0 2024-03-24T10:30:39.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.202+0000 7fdc23fff700 1 -- 172.21.15.23:0/184076644 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc28011ac0 con 0x7fdc2c106cc0 2024-03-24T10:30:39.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.202+0000 7fdc33809700 1 -- 172.21.15.23:0/184076644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc2c106cc0 msgr2=0x7fdc2c1070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:39.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.202+0000 7fdc33809700 1 --2- 172.21.15.23:0/184076644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc2c106cc0 0x7fdc2c1070e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdc280098a0 tx=0x7fdc28009bb0 comp rx=0 tx=0).stop 2024-03-24T10:30:39.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.202+0000 7fdc33809700 1 -- 172.21.15.23:0/184076644 shutdown_connections 2024-03-24T10:30:39.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.202+0000 7fdc33809700 1 --2- 172.21.15.23:0/184076644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc2c106cc0 0x7fdc2c1070e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.202+0000 7fdc33809700 1 -- 172.21.15.23:0/184076644 >> 172.21.15.23:0/184076644 conn(0x7fdc2c1022c0 msgr2=0x7fdc2c1046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.202+0000 7fdc33809700 1 -- 172.21.15.23:0/184076644 shutdown_connections 2024-03-24T10:30:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.202+0000 7fdc33809700 1 -- 172.21.15.23:0/184076644 wait complete. 2024-03-24T10:30:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.203+0000 7fdc33809700 1 Processor -- start 2024-03-24T10:30:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.203+0000 7fdc33809700 1 -- start start 2024-03-24T10:30:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.203+0000 7fdc33809700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc2c106cc0 0x7fdc2c19bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.203+0000 7fdc33809700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c19c260 con 0x7fdc2c106cc0 2024-03-24T10:30:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.203+0000 7fdc315a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc2c106cc0 0x7fdc2c19bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.203+0000 7fdc315a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc2c106cc0 0x7fdc2c19bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34898/0 (socket says 172.21.15.23:34898) 2024-03-24T10:30:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.203+0000 7fdc315a5700 1 -- 172.21.15.23:0/734755548 learned_addr learned my addr 172.21.15.23:0/734755548 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.203+0000 7fdc315a5700 1 -- 172.21.15.23:0/734755548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc28009550 con 0x7fdc2c106cc0 2024-03-24T10:30:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.204+0000 7fdc315a5700 1 --2- 172.21.15.23:0/734755548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc2c106cc0 0x7fdc2c19bd20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009fb0 tx=0x7fdc28011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.204+0000 7fdc227fc700 1 -- 172.21.15.23:0/734755548 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc280038b0 con 0x7fdc2c106cc0 2024-03-24T10:30:39.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.204+0000 7fdc227fc700 1 -- 172.21.15.23:0/734755548 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdc28003a10 con 0x7fdc2c106cc0 2024-03-24T10:30:39.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.204+0000 7fdc33809700 1 -- 172.21.15.23:0/734755548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc2c19c460 con 0x7fdc2c106cc0 2024-03-24T10:30:39.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.204+0000 7fdc227fc700 1 -- 172.21.15.23:0/734755548 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc280195e0 con 0x7fdc2c106cc0 2024-03-24T10:30:39.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.204+0000 7fdc33809700 1 -- 172.21.15.23:0/734755548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc2c19cfc0 con 0x7fdc2c106cc0 2024-03-24T10:30:39.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.205+0000 7fdc227fc700 1 -- 172.21.15.23:0/734755548 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) ==== 45917+0+0 (secure 0 0 0) 0x7fdc28019850 con 0x7fdc2c106cc0 2024-03-24T10:30:39.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.205+0000 7fdc33809700 1 -- 172.21.15.23:0/734755548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc2c063830 con 0x7fdc2c106cc0 2024-03-24T10:30:39.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.205+0000 7fdc227fc700 1 --2- 172.21.15.23:0/734755548 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fdc18038e90 0x7fdc1803b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:39.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.205+0000 7fdc227fc700 1 -- 172.21.15.23:0/734755548 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdc2804cfa0 con 0x7fdc2c106cc0 2024-03-24T10:30:39.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.205+0000 7fdc30da4700 1 --2- 172.21.15.23:0/734755548 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fdc18038e90 0x7fdc1803b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:39.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.205+0000 7fdc30da4700 1 --2- 172.21.15.23:0/734755548 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fdc18038e90 0x7fdc1803b350 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c006fd0 tx=0x7fdc1c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:39.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.210+0000 7fdc227fc700 1 -- 172.21.15.23:0/734755548 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdc28019b00 con 0x7fdc2c106cc0 2024-03-24T10:30:39.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.413+0000 7fdc33809700 1 -- 172.21.15.23:0/734755548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fdc2c06a7d0 con 0x7fdc2c106cc0 2024-03-24T10:30:39.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.414+0000 7fdc227fc700 1 -- 172.21.15.23:0/734755548 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7fdc28018be0 con 0x7fdc2c106cc0 2024-03-24T10:30:39.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.416+0000 7fdc33809700 1 -- 172.21.15.23:0/734755548 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fdc18038e90 msgr2=0x7fdc1803b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:39.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.416+0000 7fdc33809700 1 --2- 172.21.15.23:0/734755548 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fdc18038e90 0x7fdc1803b350 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c006fd0 tx=0x7fdc1c006e40 comp rx=0 tx=0).stop 2024-03-24T10:30:39.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.416+0000 7fdc33809700 1 -- 172.21.15.23:0/734755548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc2c106cc0 msgr2=0x7fdc2c19bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:39.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.416+0000 7fdc33809700 1 --2- 172.21.15.23:0/734755548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc2c106cc0 0x7fdc2c19bd20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009fb0 tx=0x7fdc28011b20 comp rx=0 tx=0).stop 2024-03-24T10:30:39.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.416+0000 7fdc33809700 1 -- 172.21.15.23:0/734755548 shutdown_connections 2024-03-24T10:30:39.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.416+0000 7fdc33809700 1 --2- 172.21.15.23:0/734755548 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fdc18038e90 0x7fdc1803b350 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:39.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.416+0000 7fdc33809700 1 --2- 172.21.15.23:0/734755548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc2c106cc0 0x7fdc2c19bd20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:39.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.416+0000 7fdc33809700 1 -- 172.21.15.23:0/734755548 >> 172.21.15.23:0/734755548 conn(0x7fdc2c1022c0 msgr2=0x7fdc2c103180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:39.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.416+0000 7fdc33809700 1 -- 172.21.15.23:0/734755548 shutdown_connections 2024-03-24T10:30:39.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:39.416+0000 7fdc33809700 1 -- 172.21.15.23:0/734755548 wait complete. 2024-03-24T10:30:39.970 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling cephadm module... 2024-03-24T10:30:40.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:40 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/734755548' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T10:30:41.335 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:41 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2523912902' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T10:30:41.335 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:41 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: ignoring --setuser ceph since I am not root 2024-03-24T10:30:41.335 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:41 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: ignoring --setgroup ceph since I am not root 2024-03-24T10:30:41.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.352+0000 7f9652b2a700 1 Processor -- start 2024-03-24T10:30:41.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.352+0000 7f9652b2a700 1 -- start start 2024-03-24T10:30:41.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.352+0000 7f9652b2a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f964c106060 0x7f964c106480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:41.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.352+0000 7f9652b2a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f964c106a50 con 0x7f964c106060 2024-03-24T10:30:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.353+0000 7f96508c6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f964c106060 0x7f964c106480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.353+0000 7f96508c6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f964c106060 0x7f964c106480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34902/0 (socket says 172.21.15.23:34902) 2024-03-24T10:30:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.353+0000 7f96508c6700 1 -- 172.21.15.23:0/1845341153 learned_addr learned my addr 172.21.15.23:0/1845341153 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.353+0000 7f96508c6700 1 -- 172.21.15.23:0/1845341153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f964c108ff0 con 0x7f964c106060 2024-03-24T10:30:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.353+0000 7f96508c6700 1 --2- 172.21.15.23:0/1845341153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f964c106060 0x7f964c106480 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9644009e80 tx=0x7f964400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a861307767e7e2f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.353+0000 7f964b7fe700 1 -- 172.21.15.23:0/1845341153 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f964400e070 con 0x7f964c106060 2024-03-24T10:30:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.353+0000 7f964b7fe700 1 -- 172.21.15.23:0/1845341153 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f964400b8c0 con 0x7f964c106060 2024-03-24T10:30:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.353+0000 7f964b7fe700 1 -- 172.21.15.23:0/1845341153 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9644004550 con 0x7f964c106060 2024-03-24T10:30:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.354+0000 7f9652b2a700 1 -- 172.21.15.23:0/1845341153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f964c106060 msgr2=0x7f964c106480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.354+0000 7f9652b2a700 1 --2- 172.21.15.23:0/1845341153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f964c106060 0x7f964c106480 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9644009e80 tx=0x7f964400b330 comp rx=0 tx=0).stop 2024-03-24T10:30:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.354+0000 7f9652b2a700 1 -- 172.21.15.23:0/1845341153 shutdown_connections 2024-03-24T10:30:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.354+0000 7f9652b2a700 1 --2- 172.21.15.23:0/1845341153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f964c106060 0x7f964c106480 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.354+0000 7f9652b2a700 1 -- 172.21.15.23:0/1845341153 >> 172.21.15.23:0/1845341153 conn(0x7f964c1019d0 msgr2=0x7f964c103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.354+0000 7f9652b2a700 1 -- 172.21.15.23:0/1845341153 shutdown_connections 2024-03-24T10:30:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.354+0000 7f9652b2a700 1 -- 172.21.15.23:0/1845341153 wait complete. 2024-03-24T10:30:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.355+0000 7f9652b2a700 1 Processor -- start 2024-03-24T10:30:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.355+0000 7f9652b2a700 1 -- start start 2024-03-24T10:30:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.355+0000 7f9652b2a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f964c106060 0x7f964c108a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.355+0000 7f9652b2a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f964c106fa0 con 0x7f964c106060 2024-03-24T10:30:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.355+0000 7f96508c6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f964c106060 0x7f964c108a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.355+0000 7f96508c6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f964c106060 0x7f964c108a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34908/0 (socket says 172.21.15.23:34908) 2024-03-24T10:30:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.355+0000 7f96508c6700 1 -- 172.21.15.23:0/2523912902 learned_addr learned my addr 172.21.15.23:0/2523912902 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.355+0000 7f96508c6700 1 -- 172.21.15.23:0/2523912902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9644009b30 con 0x7f964c106060 2024-03-24T10:30:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.355+0000 7f96508c6700 1 --2- 172.21.15.23:0/2523912902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f964c106060 0x7f964c108a50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9644009fd0 tx=0x7f9644003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.355+0000 7f9649ffb700 1 -- 172.21.15.23:0/2523912902 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f964400e040 con 0x7f964c106060 2024-03-24T10:30:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.355+0000 7f9649ffb700 1 -- 172.21.15.23:0/2523912902 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f96440128f0 con 0x7f964c106060 2024-03-24T10:30:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.355+0000 7f9652b2a700 1 -- 172.21.15.23:0/2523912902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f964c1071a0 con 0x7f964c106060 2024-03-24T10:30:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.355+0000 7f9649ffb700 1 -- 172.21.15.23:0/2523912902 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f964401c890 con 0x7f964c106060 2024-03-24T10:30:41.591 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.356+0000 7f9652b2a700 1 -- 172.21.15.23:0/2523912902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f964c107620 con 0x7f964c106060 2024-03-24T10:30:41.591 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.356+0000 7f9649ffb700 1 -- 172.21.15.23:0/2523912902 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) ==== 45917+0+0 (secure 0 0 0) 0x7f9644012430 con 0x7f964c106060 2024-03-24T10:30:41.591 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.357+0000 7f9652b2a700 1 -- 172.21.15.23:0/2523912902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f964c06a7d0 con 0x7f964c106060 2024-03-24T10:30:41.591 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.357+0000 7f9649ffb700 1 --2- 172.21.15.23:0/2523912902 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f9638038ee0 0x7f963803b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:41.591 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.357+0000 7f9649ffb700 1 -- 172.21.15.23:0/2523912902 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f964404cf40 con 0x7f964c106060 2024-03-24T10:30:41.591 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.357+0000 7f964bfff700 1 --2- 172.21.15.23:0/2523912902 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f9638038ee0 0x7f963803b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:41.591 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.357+0000 7f964bfff700 1 --2- 172.21.15.23:0/2523912902 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f9638038ee0 0x7f963803b3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f963c006fd0 tx=0x7f963c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:41.591 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.362+0000 7f9649ffb700 1 -- 172.21.15.23:0/2523912902 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9644011390 con 0x7f964c106060 2024-03-24T10:30:41.591 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:40.626+0000 7f9652b2a700 1 -- 172.21.15.23:0/2523912902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f964c10c220 con 0x7f964c106060 2024-03-24T10:30:41.591 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:41.035+0000 7f9649ffb700 1 -- 172.21.15.23:0/2523912902 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f9644020090 con 0x7f964c106060 2024-03-24T10:30:41.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:41.037+0000 7f9649ffb700 1 -- 172.21.15.23:0/2523912902 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7f9644051070 con 0x7f964c106060 2024-03-24T10:30:41.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:41.039+0000 7f9652b2a700 1 -- 172.21.15.23:0/2523912902 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f9638038ee0 msgr2=0x7f963803b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:41.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:41.039+0000 7f9652b2a700 1 --2- 172.21.15.23:0/2523912902 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f9638038ee0 0x7f963803b3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f963c006fd0 tx=0x7f963c006e40 comp rx=0 tx=0).stop 2024-03-24T10:30:41.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:41.039+0000 7f9652b2a700 1 -- 172.21.15.23:0/2523912902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f964c106060 msgr2=0x7f964c108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:41.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:41.039+0000 7f9652b2a700 1 --2- 172.21.15.23:0/2523912902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f964c106060 0x7f964c108a50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9644009fd0 tx=0x7f9644003d70 comp rx=0 tx=0).stop 2024-03-24T10:30:41.598 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:41.039+0000 7f9652b2a700 1 -- 172.21.15.23:0/2523912902 shutdown_connections 2024-03-24T10:30:41.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:41.039+0000 7f9652b2a700 1 --2- 172.21.15.23:0/2523912902 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f9638038ee0 0x7f963803b3a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:41.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:41.039+0000 7f9652b2a700 1 --2- 172.21.15.23:0/2523912902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f964c106060 0x7f964c108a50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:41.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:41.039+0000 7f9652b2a700 1 -- 172.21.15.23:0/2523912902 >> 172.21.15.23:0/2523912902 conn(0x7f964c1019d0 msgr2=0x7f964c1023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:41.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:41.039+0000 7f9652b2a700 1 -- 172.21.15.23:0/2523912902 shutdown_connections 2024-03-24T10:30:41.599 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:41.040+0000 7f9652b2a700 1 -- 172.21.15.23:0/2523912902 wait complete. 2024-03-24T10:30:41.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:41 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:41.641+0000 7f58b1bb8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T10:30:41.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:41 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:41.737+0000 7f58b1bb8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T10:30:42.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:42 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2523912902' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T10:30:42.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:42 smithi023 ceph-mon[32466]: mgrmap e5: x(active, since 6s) 2024-03-24T10:30:42.519 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:42 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:42.423+0000 7f58b1bb8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T10:30:42.519 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:42 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:42.519+0000 7f58b1bb8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T10:30:42.852 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:42 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:42.666+0000 7f58b1bb8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T10:30:42.852 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:42 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:42.765+0000 7f58b1bb8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T10:30:42.852 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:42 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:42.852+0000 7f58b1bb8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T10:30:42.935 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:30:42.935 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T10:30:42.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T10:30:42.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-24T10:30:42.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T10:30:42.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:30:42.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.050+0000 7f1353a56700 1 Processor -- start 2024-03-24T10:30:42.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.050+0000 7f1353a56700 1 -- start start 2024-03-24T10:30:42.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.050+0000 7f1353a56700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c106c80 0x7f134c1070a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:42.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.050+0000 7f1353a56700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f134c107670 con 0x7f134c106c80 2024-03-24T10:30:42.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.050+0000 7f13517f2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c106c80 0x7f134c1070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:42.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.051+0000 7f13517f2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c106c80 0x7f134c1070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34936/0 (socket says 172.21.15.23:34936) 2024-03-24T10:30:42.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.051+0000 7f13517f2700 1 -- 172.21.15.23:0/2325945160 learned_addr learned my addr 172.21.15.23:0/2325945160 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:42.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.051+0000 7f13517f2700 1 -- 172.21.15.23:0/2325945160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134c107e80 con 0x7f134c106c80 2024-03-24T10:30:42.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.051+0000 7f13517f2700 1 --2- 172.21.15.23:0/2325945160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c106c80 0x7f134c1070a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f13400098a0 tx=0x7f1340009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9357cd01054b27f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:42.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.051+0000 7f133ffff700 1 -- 172.21.15.23:0/2325945160 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1340004160 con 0x7f134c106c80 2024-03-24T10:30:42.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.051+0000 7f133ffff700 1 -- 172.21.15.23:0/2325945160 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f13400042c0 con 0x7f134c106c80 2024-03-24T10:30:42.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.051+0000 7f133ffff700 1 -- 172.21.15.23:0/2325945160 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1340011ac0 con 0x7f134c106c80 2024-03-24T10:30:42.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.052+0000 7f1353a56700 1 -- 172.21.15.23:0/2325945160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c106c80 msgr2=0x7f134c1070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:42.949 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.052+0000 7f1353a56700 1 --2- 172.21.15.23:0/2325945160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c106c80 0x7f134c1070a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f13400098a0 tx=0x7f1340009bb0 comp rx=0 tx=0).stop 2024-03-24T10:30:42.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.052+0000 7f1353a56700 1 -- 172.21.15.23:0/2325945160 shutdown_connections 2024-03-24T10:30:42.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.052+0000 7f1353a56700 1 --2- 172.21.15.23:0/2325945160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c106c80 0x7f134c1070a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:42.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.052+0000 7f1353a56700 1 -- 172.21.15.23:0/2325945160 >> 172.21.15.23:0/2325945160 conn(0x7f134c102200 msgr2=0x7f134c104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:42.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.052+0000 7f1353a56700 1 -- 172.21.15.23:0/2325945160 shutdown_connections 2024-03-24T10:30:42.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.052+0000 7f1353a56700 1 -- 172.21.15.23:0/2325945160 wait complete. 2024-03-24T10:30:42.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.053+0000 7f1353a56700 1 Processor -- start 2024-03-24T10:30:42.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.053+0000 7f1353a56700 1 -- start start 2024-03-24T10:30:42.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.053+0000 7f1353a56700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c106c80 0x7f134c19dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:42.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.053+0000 7f1353a56700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f134c19e3e0 con 0x7f134c106c80 2024-03-24T10:30:42.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.053+0000 7f13517f2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c106c80 0x7f134c19dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:42.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.053+0000 7f13517f2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c106c80 0x7f134c19dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34946/0 (socket says 172.21.15.23:34946) 2024-03-24T10:30:42.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.053+0000 7f13517f2700 1 -- 172.21.15.23:0/3164800107 learned_addr learned my addr 172.21.15.23:0/3164800107 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:42.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.053+0000 7f13517f2700 1 -- 172.21.15.23:0/3164800107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1340009550 con 0x7f134c106c80 2024-03-24T10:30:42.950 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.053+0000 7f13517f2700 1 --2- 172.21.15.23:0/3164800107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c106c80 0x7f134c19dea0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1340003f70 tx=0x7f1340011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:42.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.054+0000 7f133e7fc700 1 -- 172.21.15.23:0/3164800107 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13400039f0 con 0x7f134c106c80 2024-03-24T10:30:42.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.054+0000 7f133e7fc700 1 -- 172.21.15.23:0/3164800107 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1340003b50 con 0x7f134c106c80 2024-03-24T10:30:42.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.054+0000 7f133e7fc700 1 -- 172.21.15.23:0/3164800107 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f134001a690 con 0x7f134c106c80 2024-03-24T10:30:42.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.054+0000 7f1353a56700 1 -- 172.21.15.23:0/3164800107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134c19e5e0 con 0x7f134c106c80 2024-03-24T10:30:42.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.054+0000 7f1353a56700 1 -- 172.21.15.23:0/3164800107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f134c19e9c0 con 0x7f134c106c80 2024-03-24T10:30:42.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.055+0000 7f133e7fc700 1 -- 172.21.15.23:0/3164800107 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7f134001a900 con 0x7f134c106c80 2024-03-24T10:30:42.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.055+0000 7f1353a56700 1 -- 172.21.15.23:0/3164800107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f134c06a7d0 con 0x7f134c106c80 2024-03-24T10:30:42.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.055+0000 7f133e7fc700 1 --2- 172.21.15.23:0/3164800107 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f1338038ee0 0x7f133803b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:42.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.055+0000 7f133e7fc700 1 -- 172.21.15.23:0/3164800107 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f134004d260 con 0x7f134c106c80 2024-03-24T10:30:42.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.055+0000 7f1350ff1700 1 -- 172.21.15.23:0/3164800107 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f1338038ee0 msgr2=0x7f133803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1774258590 2024-03-24T10:30:42.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.055+0000 7f1350ff1700 1 --2- 172.21.15.23:0/3164800107 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f1338038ee0 0x7f133803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:30:42.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.060+0000 7f133e7fc700 1 -- 172.21.15.23:0/3164800107 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f134001abb0 con 0x7f134c106c80 2024-03-24T10:30:42.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.256+0000 7f1350ff1700 1 -- 172.21.15.23:0/3164800107 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f1338038ee0 msgr2=0x7f133803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1774258590 2024-03-24T10:30:42.951 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.256+0000 7f1350ff1700 1 --2- 172.21.15.23:0/3164800107 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f1338038ee0 0x7f133803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:30:42.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.320+0000 7f1353a56700 1 -- 172.21.15.23:0/3164800107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f134c1930b0 con 0x7f134c106c80 2024-03-24T10:30:42.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.320+0000 7f133e7fc700 1 -- 172.21.15.23:0/3164800107 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f1340018be0 con 0x7f134c106c80 2024-03-24T10:30:42.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.323+0000 7f1353a56700 1 -- 172.21.15.23:0/3164800107 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f1338038ee0 msgr2=0x7f133803b3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:30:42.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.323+0000 7f1353a56700 1 --2- 172.21.15.23:0/3164800107 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f1338038ee0 0x7f133803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:42.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.323+0000 7f1353a56700 1 -- 172.21.15.23:0/3164800107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c106c80 msgr2=0x7f134c19dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:42.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.323+0000 7f1353a56700 1 --2- 172.21.15.23:0/3164800107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c106c80 0x7f134c19dea0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1340003f70 tx=0x7f1340011b40 comp rx=0 tx=0).stop 2024-03-24T10:30:42.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.323+0000 7f1353a56700 1 -- 172.21.15.23:0/3164800107 shutdown_connections 2024-03-24T10:30:42.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.323+0000 7f1353a56700 1 --2- 172.21.15.23:0/3164800107 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7f1338038ee0 0x7f133803b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:42.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.323+0000 7f1353a56700 1 --2- 172.21.15.23:0/3164800107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c106c80 0x7f134c19dea0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:42.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.323+0000 7f1353a56700 1 -- 172.21.15.23:0/3164800107 >> 172.21.15.23:0/3164800107 conn(0x7f134c102200 msgr2=0x7f134c190650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:42.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.323+0000 7f1353a56700 1 -- 172.21.15.23:0/3164800107 shutdown_connections 2024-03-24T10:30:42.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:42.324+0000 7f1353a56700 1 -- 172.21.15.23:0/3164800107 wait complete. 2024-03-24T10:30:42.953 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2024-03-24T10:30:42.953 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 5... 2024-03-24T10:30:43.130 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:43 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3164800107' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T10:30:43.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:43 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:43.125+0000 7f58b1bb8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T10:30:43.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:43 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:43.240+0000 7f58b1bb8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T10:30:43.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:43 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:43.332+0000 7f58b1bb8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T10:30:43.919 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:43 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:43.433+0000 7f58b1bb8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T10:30:43.919 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:43 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:43.612+0000 7f58b1bb8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T10:30:44.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:44 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:44.602+0000 7f58b1bb8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T10:30:44.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:44 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:44.736+0000 7f58b1bb8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T10:30:45.367 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:45 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:45.073+0000 7f58b1bb8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T10:30:45.367 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:45 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:45.165+0000 7f58b1bb8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T10:30:45.367 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:45 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:45.366+0000 7f58b1bb8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:30:45.671 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:45 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:45.556+0000 7f58b1bb8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T10:30:46.109 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:45 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:45.848+0000 7f58b1bb8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:30:46.109 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:46 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:46.108+0000 7f58b1bb8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T10:30:46.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:46 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:46.351+0000 7f58b1bb8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T10:30:47.668 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:47 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:47.181+0000 7f58b1bb8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T10:30:48.084 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:47 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:47.759+0000 7f58b1bb8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T10:30:48.084 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:47 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:47.860+0000 7f58b1bb8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T10:30:48.084 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:48 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:30:48.082+0000 7f58b1bb8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T10:30:48.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:48 smithi023 ceph-mon[32466]: Active manager daemon x restarted 2024-03-24T10:30:48.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:48 smithi023 ceph-mon[32466]: Activating manager daemon x 2024-03-24T10:30:48.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:48 smithi023 ceph-mon[32466]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T10:30:48.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:48 smithi023 ceph-mon[32466]: mgrmap e6: x(active, starting, since 0.0130661s) 2024-03-24T10:30:48.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:48 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:30:48.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:48 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T10:30:48.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:48 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T10:30:48.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:48 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T10:30:48.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:48 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T10:30:48.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:48 smithi023 ceph-mon[32466]: Manager daemon x is now available 2024-03-24T10:30:48.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:48 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' 2024-03-24T10:30:49.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:49 smithi023 ceph-mon[32466]: Found migration_current of "None". Setting to last migration. 2024-03-24T10:30:49.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:49 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' 2024-03-24T10:30:49.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:49 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:30:49.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:49 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:30:49.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:49 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:30:49.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:49 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T10:30:49.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:49 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T10:30:49.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:49 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' 2024-03-24T10:30:49.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:49 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' 2024-03-24T10:30:49.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:49 smithi023 ceph-mon[32466]: mgrmap e7: x(active, since 1.01415s) 2024-03-24T10:30:49.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:30:49.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T10:30:49.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T10:30:49.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:30:49.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.337+0000 7fa6461d5700 1 Processor -- start 2024-03-24T10:30:49.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.337+0000 7fa6461d5700 1 -- start start 2024-03-24T10:30:49.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.337+0000 7fa6461d5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa640107d80 0x7fa6401081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:49.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.337+0000 7fa6461d5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fa640108770 con 0x7fa640107d80 2024-03-24T10:30:49.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.338+0000 7fa63f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa640107d80 0x7fa6401081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:49.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.338+0000 7fa63f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa640107d80 0x7fa6401081a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34952/0 (socket says 172.21.15.23:34952) 2024-03-24T10:30:49.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.338+0000 7fa63f7fe700 1 -- 172.21.15.23:0/1195225486 learned_addr learned my addr 172.21.15.23:0/1195225486 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:49.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.338+0000 7fa63f7fe700 1 -- 172.21.15.23:0/1195225486 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa640108f80 con 0x7fa640107d80 2024-03-24T10:30:49.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.338+0000 7fa63f7fe700 1 --2- 172.21.15.23:0/1195225486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa640107d80 0x7fa6401081a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa6340098a0 tx=0x7fa634009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e896561829a906f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:49.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.338+0000 7fa63e7fc700 1 -- 172.21.15.23:0/1195225486 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa634004160 con 0x7fa640107d80 2024-03-24T10:30:49.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.338+0000 7fa63e7fc700 1 -- 172.21.15.23:0/1195225486 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa6340042c0 con 0x7fa640107d80 2024-03-24T10:30:49.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.338+0000 7fa63e7fc700 1 -- 172.21.15.23:0/1195225486 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa634011ac0 con 0x7fa640107d80 2024-03-24T10:30:49.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.339+0000 7fa6461d5700 1 -- 172.21.15.23:0/1195225486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa640107d80 msgr2=0x7fa6401081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:49.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.339+0000 7fa6461d5700 1 --2- 172.21.15.23:0/1195225486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa640107d80 0x7fa6401081a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa6340098a0 tx=0x7fa634009bb0 comp rx=0 tx=0).stop 2024-03-24T10:30:49.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.339+0000 7fa6461d5700 1 -- 172.21.15.23:0/1195225486 shutdown_connections 2024-03-24T10:30:49.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.339+0000 7fa6461d5700 1 --2- 172.21.15.23:0/1195225486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa640107d80 0x7fa6401081a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:49.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.339+0000 7fa6461d5700 1 -- 172.21.15.23:0/1195225486 >> 172.21.15.23:0/1195225486 conn(0x7fa640103360 msgr2=0x7fa640105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:49.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.339+0000 7fa6461d5700 1 -- 172.21.15.23:0/1195225486 shutdown_connections 2024-03-24T10:30:49.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.339+0000 7fa6461d5700 1 -- 172.21.15.23:0/1195225486 wait complete. 2024-03-24T10:30:49.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.340+0000 7fa6461d5700 1 Processor -- start 2024-03-24T10:30:49.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.340+0000 7fa6461d5700 1 -- start start 2024-03-24T10:30:49.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.340+0000 7fa6461d5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa640107d80 0x7fa64019bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:49.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.340+0000 7fa6461d5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fa64019c1a0 con 0x7fa640107d80 2024-03-24T10:30:49.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.340+0000 7fa63f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa640107d80 0x7fa64019bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:49.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.340+0000 7fa63f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa640107d80 0x7fa64019bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34964/0 (socket says 172.21.15.23:34964) 2024-03-24T10:30:49.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.340+0000 7fa63f7fe700 1 -- 172.21.15.23:0/1383617879 learned_addr learned my addr 172.21.15.23:0/1383617879 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:49.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.341+0000 7fa63f7fe700 1 -- 172.21.15.23:0/1383617879 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa634009550 con 0x7fa640107d80 2024-03-24T10:30:49.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.341+0000 7fa63f7fe700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa640107d80 0x7fa64019bc60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa634000c00 tx=0x7fa634011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:49.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.341+0000 7fa63cff9700 1 -- 172.21.15.23:0/1383617879 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa634003aa0 con 0x7fa640107d80 2024-03-24T10:30:49.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.341+0000 7fa63cff9700 1 -- 172.21.15.23:0/1383617879 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa634003c00 con 0x7fa640107d80 2024-03-24T10:30:49.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.341+0000 7fa63cff9700 1 -- 172.21.15.23:0/1383617879 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6340248a0 con 0x7fa640107d80 2024-03-24T10:30:49.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.341+0000 7fa6461d5700 1 -- 172.21.15.23:0/1383617879 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa64019c3a0 con 0x7fa640107d80 2024-03-24T10:30:49.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.341+0000 7fa6461d5700 1 -- 172.21.15.23:0/1383617879 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa64019cf50 con 0x7fa640107d80 2024-03-24T10:30:49.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.342+0000 7fa63cff9700 1 -- 172.21.15.23:0/1383617879 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7fa63402d3f0 con 0x7fa640107d80 2024-03-24T10:30:49.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.342+0000 7fa63cff9700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fa628038f30 0x7fa62803b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:49.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.342+0000 7fa6461d5700 1 -- 172.21.15.23:0/1383617879 --> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa640063830 con 0x7fa628038f30 2024-03-24T10:30:49.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.342+0000 7fa63cff9700 1 -- 172.21.15.23:0/1383617879 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa63404cd70 con 0x7fa640107d80 2024-03-24T10:30:49.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.342+0000 7fa63effd700 1 -- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fa628038f30 msgr2=0x7fa62803b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1774258590 2024-03-24T10:30:49.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.342+0000 7fa63effd700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fa628038f30 0x7fa62803b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:30:49.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.543+0000 7fa63effd700 1 -- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fa628038f30 msgr2=0x7fa62803b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1774258590 2024-03-24T10:30:49.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.543+0000 7fa63effd700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fa628038f30 0x7fa62803b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:30:49.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.944+0000 7fa63effd700 1 -- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fa628038f30 msgr2=0x7fa62803b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1774258590 2024-03-24T10:30:49.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:43.944+0000 7fa63effd700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fa628038f30 0x7fa62803b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:30:49.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:44.745+0000 7fa63effd700 1 -- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fa628038f30 msgr2=0x7fa62803b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1774258590 2024-03-24T10:30:49.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:44.745+0000 7fa63effd700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fa628038f30 0x7fa62803b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:30:49.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:46.346+0000 7fa63effd700 1 -- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fa628038f30 msgr2=0x7fa62803b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1774258590 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:46.346+0000 7fa63effd700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fa628038f30 0x7fa62803b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:48.107+0000 7fa63cff9700 1 -- 172.21.15.23:0/1383617879 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 6) ==== 45695+0+0 (secure 0 0 0) 0x7fa634019410 con 0x7fa640107d80 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:48.107+0000 7fa63cff9700 1 -- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fa628038f30 msgr2=0x7fa62803b3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:48.107+0000 7fa63cff9700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fa628038f30 0x7fa62803b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.108+0000 7fa63cff9700 1 -- 172.21.15.23:0/1383617879 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 7) ==== 45822+0+0 (secure 0 0 0) 0x7fa634021700 con 0x7fa640107d80 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.108+0000 7fa63cff9700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fa62803c670 0x7fa62803ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.108+0000 7fa63cff9700 1 -- 172.21.15.23:0/1383617879 --> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa640063830 con 0x7fa62803c670 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.108+0000 7fa63effd700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fa62803c670 0x7fa62803ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.109+0000 7fa63effd700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fa62803c670 0x7fa62803ea60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa630003a10 tx=0x7fa6300092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.109+0000 7fa63cff9700 1 -- 172.21.15.23:0/1383617879 <== mgr.14120 v2:172.21.15.23:6800/2159398914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fa640063830 con 0x7fa62803c670 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.116+0000 7fa6461d5700 1 -- 172.21.15.23:0/1383617879 --> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fa64019c840 con 0x7fa62803c670 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.116+0000 7fa63cff9700 1 -- 172.21.15.23:0/1383617879 <== mgr.14120 v2:172.21.15.23:6800/2159398914 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fa64019c840 con 0x7fa62803c670 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.116+0000 7fa6461d5700 1 -- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fa62803c670 msgr2=0x7fa62803ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.116+0000 7fa6461d5700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fa62803c670 0x7fa62803ea60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa630003a10 tx=0x7fa6300092b0 comp rx=0 tx=0).stop 2024-03-24T10:30:49.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.116+0000 7fa6461d5700 1 -- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa640107d80 msgr2=0x7fa64019bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:49.733 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.116+0000 7fa6461d5700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa640107d80 0x7fa64019bc60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa634000c00 tx=0x7fa634011b40 comp rx=0 tx=0).stop 2024-03-24T10:30:49.733 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.116+0000 7fa6461d5700 1 -- 172.21.15.23:0/1383617879 shutdown_connections 2024-03-24T10:30:49.733 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.116+0000 7fa6461d5700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fa62803c670 0x7fa62803ea60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:49.733 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.116+0000 7fa6461d5700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa640107d80 0x7fa64019bc60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:49.733 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.116+0000 7fa6461d5700 1 --2- 172.21.15.23:0/1383617879 >> [v2:172.21.15.23:6800/1774258590,v1:172.21.15.23:6801/1774258590] conn(0x7fa628038f30 0x7fa62803b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:49.733 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.116+0000 7fa6461d5700 1 -- 172.21.15.23:0/1383617879 >> 172.21.15.23:0/1383617879 conn(0x7fa640103360 msgr2=0x7fa640103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:49.733 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.116+0000 7fa6461d5700 1 -- 172.21.15.23:0/1383617879 shutdown_connections 2024-03-24T10:30:49.733 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:49.116+0000 7fa6461d5700 1 -- 172.21.15.23:0/1383617879 wait complete. 2024-03-24T10:30:49.733 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 5 is available 2024-03-24T10:30:49.733 INFO:teuthology.orchestra.run.smithi023.stdout:Setting orchestrator backend to cephadm... 2024-03-24T10:30:50.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:50 smithi023 ceph-mon[32466]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T10:30:50.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:50 smithi023 ceph-mon[32466]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T10:30:50.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:50 smithi023 ceph-mon[32466]: [24/Mar/2024:10:30:49] ENGINE Bus STARTING 2024-03-24T10:30:50.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:50 smithi023 ceph-mon[32466]: [24/Mar/2024:10:30:49] ENGINE Serving on http://172.21.15.23:8765 2024-03-24T10:30:50.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:50 smithi023 ceph-mon[32466]: [24/Mar/2024:10:30:49] ENGINE Serving on https://172.21.15.23:7150 2024-03-24T10:30:50.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:50 smithi023 ceph-mon[32466]: [24/Mar/2024:10:30:49] ENGINE Bus STARTED 2024-03-24T10:30:50.385 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:50 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:30:50.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.151+0000 7fb77e694700 1 Processor -- start 2024-03-24T10:30:50.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.151+0000 7fb77e694700 1 -- start start 2024-03-24T10:30:50.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.151+0000 7fb77e694700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb778106060 0x7fb778106480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:50.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.151+0000 7fb77e694700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb778106a50 con 0x7fb778106060 2024-03-24T10:30:50.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.151+0000 7fb777fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb778106060 0x7fb778106480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:50.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.151+0000 7fb777fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb778106060 0x7fb778106480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38046/0 (socket says 172.21.15.23:38046) 2024-03-24T10:30:50.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.151+0000 7fb777fff700 1 -- 172.21.15.23:0/1846786347 learned_addr learned my addr 172.21.15.23:0/1846786347 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:50.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.151+0000 7fb777fff700 1 -- 172.21.15.23:0/1846786347 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb778106b90 con 0x7fb778106060 2024-03-24T10:30:50.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.152+0000 7fb777fff700 1 --2- 172.21.15.23:0/1846786347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb778106060 0x7fb778106480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb7680098a0 tx=0x7fb768009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecad745facb44703 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:50.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.154+0000 7fb776ffd700 1 -- 172.21.15.23:0/1846786347 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb768004160 con 0x7fb778106060 2024-03-24T10:30:50.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.154+0000 7fb776ffd700 1 -- 172.21.15.23:0/1846786347 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb7680042c0 con 0x7fb778106060 2024-03-24T10:30:50.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.154+0000 7fb776ffd700 1 -- 172.21.15.23:0/1846786347 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb768011ac0 con 0x7fb778106060 2024-03-24T10:30:50.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.155+0000 7fb77e694700 1 -- 172.21.15.23:0/1846786347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb778106060 msgr2=0x7fb778106480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:50.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.155+0000 7fb77e694700 1 --2- 172.21.15.23:0/1846786347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb778106060 0x7fb778106480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb7680098a0 tx=0x7fb768009bb0 comp rx=0 tx=0).stop 2024-03-24T10:30:50.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.155+0000 7fb77e694700 1 -- 172.21.15.23:0/1846786347 shutdown_connections 2024-03-24T10:30:50.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.155+0000 7fb77e694700 1 --2- 172.21.15.23:0/1846786347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb778106060 0x7fb778106480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:50.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.155+0000 7fb77e694700 1 -- 172.21.15.23:0/1846786347 >> 172.21.15.23:0/1846786347 conn(0x7fb778101a20 msgr2=0x7fb778103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:50.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.155+0000 7fb77e694700 1 -- 172.21.15.23:0/1846786347 shutdown_connections 2024-03-24T10:30:50.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.155+0000 7fb77e694700 1 -- 172.21.15.23:0/1846786347 wait complete. 2024-03-24T10:30:50.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.156+0000 7fb77e694700 1 Processor -- start 2024-03-24T10:30:50.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.156+0000 7fb77e694700 1 -- start start 2024-03-24T10:30:50.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.156+0000 7fb77e694700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb778106060 0x7fb77819bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:50.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.156+0000 7fb77e694700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb77819c200 con 0x7fb778106060 2024-03-24T10:30:50.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.156+0000 7fb777fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb778106060 0x7fb77819bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:50.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.156+0000 7fb777fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb778106060 0x7fb77819bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38058/0 (socket says 172.21.15.23:38058) 2024-03-24T10:30:50.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.156+0000 7fb777fff700 1 -- 172.21.15.23:0/2516526413 learned_addr learned my addr 172.21.15.23:0/2516526413 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:50.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.156+0000 7fb777fff700 1 -- 172.21.15.23:0/2516526413 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb768009550 con 0x7fb778106060 2024-03-24T10:30:50.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.157+0000 7fb777fff700 1 --2- 172.21.15.23:0/2516526413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb778106060 0x7fb77819bcc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb768003f70 tx=0x7fb768011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:50.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.157+0000 7fb7757fa700 1 -- 172.21.15.23:0/2516526413 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7680039f0 con 0x7fb778106060 2024-03-24T10:30:50.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.157+0000 7fb7757fa700 1 -- 172.21.15.23:0/2516526413 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb768003b50 con 0x7fb778106060 2024-03-24T10:30:50.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.157+0000 7fb77e694700 1 -- 172.21.15.23:0/2516526413 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb77819c400 con 0x7fb778106060 2024-03-24T10:30:50.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.157+0000 7fb7757fa700 1 -- 172.21.15.23:0/2516526413 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb76801a690 con 0x7fb778106060 2024-03-24T10:30:50.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.157+0000 7fb77e694700 1 -- 172.21.15.23:0/2516526413 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb77819cfb0 con 0x7fb778106060 2024-03-24T10:30:50.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.158+0000 7fb7757fa700 1 -- 172.21.15.23:0/2516526413 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fb76801a970 con 0x7fb778106060 2024-03-24T10:30:50.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.158+0000 7fb7757fa700 1 --2- 172.21.15.23:0/2516526413 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fb760038e90 0x7fb76003b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:50.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.158+0000 7fb77e694700 1 -- 172.21.15.23:0/2516526413 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb77819c8a0 con 0x7fb778106060 2024-03-24T10:30:50.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.158+0000 7fb7757fa700 1 -- 172.21.15.23:0/2516526413 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb76804d0c0 con 0x7fb778106060 2024-03-24T10:30:50.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.158+0000 7fb7777fe700 1 --2- 172.21.15.23:0/2516526413 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fb760038e90 0x7fb76003b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:50.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.158+0000 7fb7777fe700 1 --2- 172.21.15.23:0/2516526413 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fb760038e90 0x7fb76003b350 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb76c006fd0 tx=0x7fb76c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:50.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.163+0000 7fb7757fa700 1 -- 172.21.15.23:0/2516526413 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb77819c8a0 con 0x7fb778106060 2024-03-24T10:30:50.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.383+0000 7fb77e694700 1 -- 172.21.15.23:0/2516526413 --> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fb77819c8a0 con 0x7fb760038e90 2024-03-24T10:30:50.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.394+0000 7fb7757fa700 1 -- 172.21.15.23:0/2516526413 <== mgr.14120 v2:172.21.15.23:6800/2159398914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb77819c8a0 con 0x7fb760038e90 2024-03-24T10:30:50.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.396+0000 7fb77e694700 1 -- 172.21.15.23:0/2516526413 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fb760038e90 msgr2=0x7fb76003b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:50.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.396+0000 7fb77e694700 1 --2- 172.21.15.23:0/2516526413 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fb760038e90 0x7fb76003b350 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb76c006fd0 tx=0x7fb76c006e40 comp rx=0 tx=0).stop 2024-03-24T10:30:50.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.396+0000 7fb77e694700 1 -- 172.21.15.23:0/2516526413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb778106060 msgr2=0x7fb77819bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:50.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.396+0000 7fb77e694700 1 --2- 172.21.15.23:0/2516526413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb778106060 0x7fb77819bcc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb768003f70 tx=0x7fb768011b40 comp rx=0 tx=0).stop 2024-03-24T10:30:50.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.396+0000 7fb77e694700 1 -- 172.21.15.23:0/2516526413 shutdown_connections 2024-03-24T10:30:50.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.396+0000 7fb77e694700 1 --2- 172.21.15.23:0/2516526413 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fb760038e90 0x7fb76003b350 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:50.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.396+0000 7fb77e694700 1 --2- 172.21.15.23:0/2516526413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb778106060 0x7fb77819bcc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:50.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.396+0000 7fb77e694700 1 -- 172.21.15.23:0/2516526413 >> 172.21.15.23:0/2516526413 conn(0x7fb778101a20 msgr2=0x7fb778102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:50.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.396+0000 7fb77e694700 1 -- 172.21.15.23:0/2516526413 shutdown_connections 2024-03-24T10:30:50.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:50.396+0000 7fb77e694700 1 -- 172.21.15.23:0/2516526413 wait complete. 2024-03-24T10:30:51.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:51 smithi023 ceph-mon[32466]: mgrmap e8: x(active, since 2s) 2024-03-24T10:30:51.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:51 smithi023 ceph-mon[32466]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:30:51.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:51 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' 2024-03-24T10:30:51.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:51 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:30:52.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T10:30:52.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.370+0000 7ffa0cfd9700 1 Processor -- start 2024-03-24T10:30:52.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.370+0000 7ffa0cfd9700 1 -- start start 2024-03-24T10:30:52.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.370+0000 7ffa0cfd9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa08106c80 0x7ffa081070a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:52.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.370+0000 7ffa0cfd9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ffa08107670 con 0x7ffa08106c80 2024-03-24T10:30:52.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.370+0000 7ffa0659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa08106c80 0x7ffa081070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:52.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.370+0000 7ffa0659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa08106c80 0x7ffa081070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38064/0 (socket says 172.21.15.23:38064) 2024-03-24T10:30:52.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.370+0000 7ffa0659c700 1 -- 172.21.15.23:0/2995754895 learned_addr learned my addr 172.21.15.23:0/2995754895 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:52.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.371+0000 7ffa0659c700 1 -- 172.21.15.23:0/2995754895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa08107e80 con 0x7ffa08106c80 2024-03-24T10:30:52.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.371+0000 7ffa0659c700 1 --2- 172.21.15.23:0/2995754895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa08106c80 0x7ffa081070a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff9f80098a0 tx=0x7ff9f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1490d2b98db22cde server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:52.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.371+0000 7ffa0559a700 1 -- 172.21.15.23:0/2995754895 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9f8004160 con 0x7ffa08106c80 2024-03-24T10:30:52.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.371+0000 7ffa0559a700 1 -- 172.21.15.23:0/2995754895 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff9f80042c0 con 0x7ffa08106c80 2024-03-24T10:30:52.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.371+0000 7ffa0559a700 1 -- 172.21.15.23:0/2995754895 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9f8011ac0 con 0x7ffa08106c80 2024-03-24T10:30:52.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.372+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/2995754895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa08106c80 msgr2=0x7ffa081070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:52.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.372+0000 7ffa0cfd9700 1 --2- 172.21.15.23:0/2995754895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa08106c80 0x7ffa081070a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff9f80098a0 tx=0x7ff9f8009bb0 comp rx=0 tx=0).stop 2024-03-24T10:30:52.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.372+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/2995754895 shutdown_connections 2024-03-24T10:30:52.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.372+0000 7ffa0cfd9700 1 --2- 172.21.15.23:0/2995754895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa08106c80 0x7ffa081070a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:52.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.372+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/2995754895 >> 172.21.15.23:0/2995754895 conn(0x7ffa08102200 msgr2=0x7ffa08104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:52.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.372+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/2995754895 shutdown_connections 2024-03-24T10:30:52.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.372+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/2995754895 wait complete. 2024-03-24T10:30:52.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.373+0000 7ffa0cfd9700 1 Processor -- start 2024-03-24T10:30:52.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.373+0000 7ffa0cfd9700 1 -- start start 2024-03-24T10:30:52.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.373+0000 7ffa0cfd9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa08106c80 0x7ffa0819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:52.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.373+0000 7ffa0cfd9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0819e3e0 con 0x7ffa08106c80 2024-03-24T10:30:52.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.373+0000 7ffa0659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa08106c80 0x7ffa0819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:52.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.373+0000 7ffa0659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa08106c80 0x7ffa0819dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38074/0 (socket says 172.21.15.23:38074) 2024-03-24T10:30:52.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.373+0000 7ffa0659c700 1 -- 172.21.15.23:0/3455259010 learned_addr learned my addr 172.21.15.23:0/3455259010 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:52.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.373+0000 7ffa0659c700 1 -- 172.21.15.23:0/3455259010 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f8009550 con 0x7ffa08106c80 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.373+0000 7ffa0659c700 1 --2- 172.21.15.23:0/3455259010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa08106c80 0x7ffa0819dea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8000c00 tx=0x7ff9f8011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.374+0000 7ff9f77fe700 1 -- 172.21.15.23:0/3455259010 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9f8003aa0 con 0x7ffa08106c80 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.374+0000 7ff9f77fe700 1 -- 172.21.15.23:0/3455259010 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff9f8003c00 con 0x7ffa08106c80 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.374+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/3455259010 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa0819e5e0 con 0x7ffa08106c80 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.374+0000 7ff9f77fe700 1 -- 172.21.15.23:0/3455259010 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9f80248a0 con 0x7ffa08106c80 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.374+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/3455259010 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa0819f190 con 0x7ffa08106c80 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.375+0000 7ff9f77fe700 1 -- 172.21.15.23:0/3455259010 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7ff9f801ac60 con 0x7ffa08106c80 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.375+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/3455259010 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa0819ea80 con 0x7ffa08106c80 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.375+0000 7ff9f77fe700 1 --2- 172.21.15.23:0/3455259010 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7ff9f0038f30 0x7ff9f003b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.375+0000 7ff9f77fe700 1 -- 172.21.15.23:0/3455259010 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff9f804d120 con 0x7ffa08106c80 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.375+0000 7ffa05d9b700 1 --2- 172.21.15.23:0/3455259010 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7ff9f0038f30 0x7ff9f003b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.375+0000 7ffa05d9b700 1 --2- 172.21.15.23:0/3455259010 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7ff9f0038f30 0x7ff9f003b3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc006fd0 tx=0x7ff9fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.380+0000 7ff9f77fe700 1 -- 172.21.15.23:0/3455259010 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffa0819ea80 con 0x7ffa08106c80 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.589+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/3455259010 --> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7ffa0819ea80 con 0x7ff9f0038f30 2024-03-24T10:30:52.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.589+0000 7ff9f77fe700 1 -- 172.21.15.23:0/3455259010 <== mgr.14120 v2:172.21.15.23:6800/2159398914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7ffa0819ea80 con 0x7ff9f0038f30 2024-03-24T10:30:52.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.592+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/3455259010 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7ff9f0038f30 msgr2=0x7ff9f003b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:52.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.592+0000 7ffa0cfd9700 1 --2- 172.21.15.23:0/3455259010 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7ff9f0038f30 0x7ff9f003b3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc006fd0 tx=0x7ff9fc006e40 comp rx=0 tx=0).stop 2024-03-24T10:30:52.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.592+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/3455259010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa08106c80 msgr2=0x7ffa0819dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:52.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.592+0000 7ffa0cfd9700 1 --2- 172.21.15.23:0/3455259010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa08106c80 0x7ffa0819dea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8000c00 tx=0x7ff9f8011b40 comp rx=0 tx=0).stop 2024-03-24T10:30:52.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.592+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/3455259010 shutdown_connections 2024-03-24T10:30:52.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.592+0000 7ffa0cfd9700 1 --2- 172.21.15.23:0/3455259010 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7ff9f0038f30 0x7ff9f003b3f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:52.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.592+0000 7ffa0cfd9700 1 --2- 172.21.15.23:0/3455259010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa08106c80 0x7ffa0819dea0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:52.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.592+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/3455259010 >> 172.21.15.23:0/3455259010 conn(0x7ffa08102200 msgr2=0x7ffa08190650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:52.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.592+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/3455259010 shutdown_connections 2024-03-24T10:30:52.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:51.592+0000 7ffa0cfd9700 1 -- 172.21.15.23:0/3455259010 wait complete. 2024-03-24T10:30:52.124 INFO:teuthology.orchestra.run.smithi023.stdout:Generating ssh key... 2024-03-24T10:30:52.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-mon[32466]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:30:53.168 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: Generating public/private rsa key pair. 2024-03-24T10:30:53.168 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: Your identification has been saved in /tmp/tmpywc12uoo/key. 2024-03-24T10:30:53.168 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: Your public key has been saved in /tmp/tmpywc12uoo/key.pub. 2024-03-24T10:30:53.168 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: The key fingerprint is: 2024-03-24T10:30:53.168 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: SHA256:ROrTX+CTXBUTr8KOuqjOUidXqV9yWXMlJ/aFS6ZC5Vk ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:30:53.169 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: The key's randomart image is: 2024-03-24T10:30:53.169 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: +---[RSA 3072]----+ 2024-03-24T10:30:53.169 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: | . .. E+ | 2024-03-24T10:30:53.169 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: | o .. +Bo+| 2024-03-24T10:30:53.169 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: | . .o. += Bo| 2024-03-24T10:30:53.169 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: | . ooo.=+ o..| 2024-03-24T10:30:53.169 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: | ooS =++o. | 2024-03-24T10:30:53.169 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: | o +...+= . | 2024-03-24T10:30:53.169 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: | . + . +o . | 2024-03-24T10:30:53.169 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: | .. ... | 2024-03-24T10:30:53.169 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: | o+.. o. | 2024-03-24T10:30:53.169 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:30:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: +----[SHA256]-----+ 2024-03-24T10:30:53.494 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.552+0000 7f671eb2b700 1 Processor -- start 2024-03-24T10:30:53.494 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.552+0000 7f671eb2b700 1 -- start start 2024-03-24T10:30:53.494 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.552+0000 7f671eb2b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6718108820 0x7f671810ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:53.495 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.552+0000 7f671eb2b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f671807bdb0 con 0x7f6718108820 2024-03-24T10:30:53.495 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.552+0000 7f671c8c7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6718108820 0x7f671810ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:53.495 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.553+0000 7f671c8c7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6718108820 0x7f671810ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38076/0 (socket says 172.21.15.23:38076) 2024-03-24T10:30:53.495 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.553+0000 7f671c8c7700 1 -- 172.21.15.23:0/155581147 learned_addr learned my addr 172.21.15.23:0/155581147 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:53.495 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.553+0000 7f671c8c7700 1 -- 172.21.15.23:0/155581147 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671810b150 con 0x7f6718108820 2024-03-24T10:30:53.495 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.553+0000 7f671c8c7700 1 --2- 172.21.15.23:0/155581147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6718108820 0x7f671810ac10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f670c0098a0 tx=0x7f670c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3065c91d1356ae53 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:53.495 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.553+0000 7f67177fe700 1 -- 172.21.15.23:0/155581147 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f670c004160 con 0x7f6718108820 2024-03-24T10:30:53.495 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.553+0000 7f67177fe700 1 -- 172.21.15.23:0/155581147 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f670c0042c0 con 0x7f6718108820 2024-03-24T10:30:53.495 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.553+0000 7f67177fe700 1 -- 172.21.15.23:0/155581147 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f670c011ac0 con 0x7f6718108820 2024-03-24T10:30:53.496 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.554+0000 7f671eb2b700 1 -- 172.21.15.23:0/155581147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6718108820 msgr2=0x7f671810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:53.496 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.554+0000 7f671eb2b700 1 --2- 172.21.15.23:0/155581147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6718108820 0x7f671810ac10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f670c0098a0 tx=0x7f670c009bb0 comp rx=0 tx=0).stop 2024-03-24T10:30:53.496 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.554+0000 7f671eb2b700 1 -- 172.21.15.23:0/155581147 shutdown_connections 2024-03-24T10:30:53.496 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.554+0000 7f671eb2b700 1 --2- 172.21.15.23:0/155581147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6718108820 0x7f671810ac10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:53.496 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.554+0000 7f671eb2b700 1 -- 172.21.15.23:0/155581147 >> 172.21.15.23:0/155581147 conn(0x7f6718102290 msgr2=0x7f67181046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:53.496 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.554+0000 7f671eb2b700 1 -- 172.21.15.23:0/155581147 shutdown_connections 2024-03-24T10:30:53.496 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.554+0000 7f671eb2b700 1 -- 172.21.15.23:0/155581147 wait complete. 2024-03-24T10:30:53.496 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.554+0000 7f671eb2b700 1 Processor -- start 2024-03-24T10:30:53.496 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.554+0000 7f671eb2b700 1 -- start start 2024-03-24T10:30:53.496 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.555+0000 7f671eb2b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6718108820 0x7f671819bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:53.496 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.555+0000 7f671eb2b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f671819c120 con 0x7f6718108820 2024-03-24T10:30:53.496 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.555+0000 7f671c8c7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6718108820 0x7f671819bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:53.496 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.555+0000 7f671c8c7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6718108820 0x7f671819bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38082/0 (socket says 172.21.15.23:38082) 2024-03-24T10:30:53.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.555+0000 7f671c8c7700 1 -- 172.21.15.23:0/905948790 learned_addr learned my addr 172.21.15.23:0/905948790 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:53.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.555+0000 7f671c8c7700 1 -- 172.21.15.23:0/905948790 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f670c009550 con 0x7f6718108820 2024-03-24T10:30:53.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.555+0000 7f671c8c7700 1 --2- 172.21.15.23:0/905948790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6718108820 0x7f671819bbe0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f670c009130 tx=0x7f670c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:53.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.555+0000 7f6715ffb700 1 -- 172.21.15.23:0/905948790 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f670c003aa0 con 0x7f6718108820 2024-03-24T10:30:53.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.555+0000 7f6715ffb700 1 -- 172.21.15.23:0/905948790 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f670c003c00 con 0x7f6718108820 2024-03-24T10:30:53.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.555+0000 7f6715ffb700 1 -- 172.21.15.23:0/905948790 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f670c024800 con 0x7f6718108820 2024-03-24T10:30:53.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.555+0000 7f671eb2b700 1 -- 172.21.15.23:0/905948790 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f671819c320 con 0x7f6718108820 2024-03-24T10:30:53.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.555+0000 7f671eb2b700 1 -- 172.21.15.23:0/905948790 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f671819ced0 con 0x7f6718108820 2024-03-24T10:30:53.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.556+0000 7f6715ffb700 1 -- 172.21.15.23:0/905948790 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f670c01abd0 con 0x7f6718108820 2024-03-24T10:30:53.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.556+0000 7f671eb2b700 1 -- 172.21.15.23:0/905948790 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6718063830 con 0x7f6718108820 2024-03-24T10:30:53.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.556+0000 7f6715ffb700 1 --2- 172.21.15.23:0/905948790 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f6708038f30 0x7f670803b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:53.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.556+0000 7f6715ffb700 1 -- 172.21.15.23:0/905948790 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f670c04d0a0 con 0x7f6718108820 2024-03-24T10:30:53.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.557+0000 7f6717fff700 1 --2- 172.21.15.23:0/905948790 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f6708038f30 0x7f670803b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:53.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.557+0000 7f6717fff700 1 --2- 172.21.15.23:0/905948790 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f6708038f30 0x7f670803b3f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6704006fd0 tx=0x7f6704006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:53.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.561+0000 7f6715ffb700 1 -- 172.21.15.23:0/905948790 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f670c018760 con 0x7f6718108820 2024-03-24T10:30:53.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.765+0000 7f671eb2b700 1 -- 172.21.15.23:0/905948790 --> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f6718103460 con 0x7f6708038f30 2024-03-24T10:30:53.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.958+0000 7f6715ffb700 1 -- 172.21.15.23:0/905948790 <== mgr.14120 v2:172.21.15.23:6800/2159398914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6718103460 con 0x7f6708038f30 2024-03-24T10:30:53.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.960+0000 7f671eb2b700 1 -- 172.21.15.23:0/905948790 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f6708038f30 msgr2=0x7f670803b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:53.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.960+0000 7f671eb2b700 1 --2- 172.21.15.23:0/905948790 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f6708038f30 0x7f670803b3f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6704006fd0 tx=0x7f6704006e40 comp rx=0 tx=0).stop 2024-03-24T10:30:53.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.960+0000 7f671eb2b700 1 -- 172.21.15.23:0/905948790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6718108820 msgr2=0x7f671819bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:53.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.960+0000 7f671eb2b700 1 --2- 172.21.15.23:0/905948790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6718108820 0x7f671819bbe0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f670c009130 tx=0x7f670c011b40 comp rx=0 tx=0).stop 2024-03-24T10:30:53.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.960+0000 7f671eb2b700 1 -- 172.21.15.23:0/905948790 shutdown_connections 2024-03-24T10:30:53.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.960+0000 7f671eb2b700 1 --2- 172.21.15.23:0/905948790 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f6708038f30 0x7f670803b3f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:53.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.960+0000 7f671eb2b700 1 --2- 172.21.15.23:0/905948790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6718108820 0x7f671819bbe0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:53.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.960+0000 7f671eb2b700 1 -- 172.21.15.23:0/905948790 >> 172.21.15.23:0/905948790 conn(0x7f6718102290 msgr2=0x7f6718102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:53.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.960+0000 7f671eb2b700 1 -- 172.21.15.23:0/905948790 shutdown_connections 2024-03-24T10:30:53.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:52.960+0000 7f671eb2b700 1 -- 172.21.15.23:0/905948790 wait complete. 2024-03-24T10:30:54.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:53 smithi023 ceph-mon[32466]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:30:54.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:53 smithi023 ceph-mon[32466]: Generating ssh key... 2024-03-24T10:30:54.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:53 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' 2024-03-24T10:30:54.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:53 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' 2024-03-24T10:30:54.659 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDACbbrvvqVbgW9KHxE7s+Ldi+yrxDR0DmENZu5LD716D1EtqC/w0vLqJgP8x+zGnPpcPCuXVeDT+hu7/KVTMdraTL4dr8hYYkyLFVGXZv2kwbVaumndSyaDM2IaD5SE+hNqG0/Bh1JNIxaQbcS8fIVLMUtb6oRyzbqvpK+acYr1huj2jdtGFAN7CJ+6nQxKozQMVnbC4iDpJDrFPyhV/Jv+6dl0RADnCLcjSl/rRF8ZP9crGgO1ZUoMWeKuUbwcNAuAbnJQGq29sDMLSbROD7beBPxU0lznNiYfB1Y75eMqLD5/e5OY0WHKFoezjFfHdXj+xCC+d8HA1PBuxaL75PnZHpojN0UKnPb7LDQQkwB7VsYPV6D2IDehW84RzfWWSnKugE3RFL3eC52pqrXqpVwwtEHhyspTpLSZcDz3vsfW6zh034NArP8ZApohu28xvfGSelKcaYzWIoCpShjQw3ijk/FjOQyCJdowCwGI5GBIcigCGzWP/1ZwTq7VrqprL0= ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:30:54.659 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.907+0000 7f68222f5700 1 Processor -- start 2024-03-24T10:30:54.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.907+0000 7f68222f5700 1 -- start start 2024-03-24T10:30:54.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.907+0000 7f68222f5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c07dde0 0x7f681c07c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:54.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.907+0000 7f68222f5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f681c07c810 con 0x7f681c07dde0 2024-03-24T10:30:54.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.907+0000 7f681bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c07dde0 0x7f681c07c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:54.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.907+0000 7f681bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c07dde0 0x7f681c07c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38090/0 (socket says 172.21.15.23:38090) 2024-03-24T10:30:54.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.907+0000 7f681bfff700 1 -- 172.21.15.23:0/26033047 learned_addr learned my addr 172.21.15.23:0/26033047 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:54.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.908+0000 7f681bfff700 1 -- 172.21.15.23:0/26033047 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681c07c950 con 0x7f681c07dde0 2024-03-24T10:30:54.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.908+0000 7f681bfff700 1 --2- 172.21.15.23:0/26033047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c07dde0 0x7f681c07c240 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f680c0098a0 tx=0x7f680c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8baa48bc82ab3ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:54.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.908+0000 7f681affd700 1 -- 172.21.15.23:0/26033047 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f680c004160 con 0x7f681c07dde0 2024-03-24T10:30:54.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.908+0000 7f681affd700 1 -- 172.21.15.23:0/26033047 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f680c0042c0 con 0x7f681c07dde0 2024-03-24T10:30:54.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.908+0000 7f681affd700 1 -- 172.21.15.23:0/26033047 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f680c011ac0 con 0x7f681c07dde0 2024-03-24T10:30:54.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.908+0000 7f68222f5700 1 -- 172.21.15.23:0/26033047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c07dde0 msgr2=0x7f681c07c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:54.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.908+0000 7f68222f5700 1 --2- 172.21.15.23:0/26033047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c07dde0 0x7f681c07c240 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f680c0098a0 tx=0x7f680c009bb0 comp rx=0 tx=0).stop 2024-03-24T10:30:54.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.909+0000 7f68222f5700 1 -- 172.21.15.23:0/26033047 shutdown_connections 2024-03-24T10:30:54.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.909+0000 7f68222f5700 1 --2- 172.21.15.23:0/26033047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c07dde0 0x7f681c07c240 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:54.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.909+0000 7f68222f5700 1 -- 172.21.15.23:0/26033047 >> 172.21.15.23:0/26033047 conn(0x7f681c103360 msgr2=0x7f681c105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:54.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.909+0000 7f68222f5700 1 -- 172.21.15.23:0/26033047 shutdown_connections 2024-03-24T10:30:54.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.909+0000 7f68222f5700 1 -- 172.21.15.23:0/26033047 wait complete. 2024-03-24T10:30:54.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.909+0000 7f68222f5700 1 Processor -- start 2024-03-24T10:30:54.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.909+0000 7f68222f5700 1 -- start start 2024-03-24T10:30:54.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.909+0000 7f68222f5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c07dde0 0x7f681c19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:54.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.909+0000 7f68222f5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f681c19e420 con 0x7f681c07dde0 2024-03-24T10:30:54.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.910+0000 7f681bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c07dde0 0x7f681c19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:54.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.910+0000 7f681bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c07dde0 0x7f681c19dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38098/0 (socket says 172.21.15.23:38098) 2024-03-24T10:30:54.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.910+0000 7f681bfff700 1 -- 172.21.15.23:0/2866907282 learned_addr learned my addr 172.21.15.23:0/2866907282 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:30:54.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.910+0000 7f681bfff700 1 -- 172.21.15.23:0/2866907282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f680c009550 con 0x7f681c07dde0 2024-03-24T10:30:54.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.910+0000 7f681bfff700 1 --2- 172.21.15.23:0/2866907282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c07dde0 0x7f681c19dee0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f680c003f70 tx=0x7f680c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:54.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.910+0000 7f68197fa700 1 -- 172.21.15.23:0/2866907282 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f680c0039f0 con 0x7f681c07dde0 2024-03-24T10:30:54.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.910+0000 7f68197fa700 1 -- 172.21.15.23:0/2866907282 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f680c003b50 con 0x7f681c07dde0 2024-03-24T10:30:54.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.910+0000 7f68222f5700 1 -- 172.21.15.23:0/2866907282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f681c19e620 con 0x7f681c07dde0 2024-03-24T10:30:54.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.910+0000 7f68197fa700 1 -- 172.21.15.23:0/2866907282 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f680c01a690 con 0x7f681c07dde0 2024-03-24T10:30:54.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.910+0000 7f68222f5700 1 -- 172.21.15.23:0/2866907282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f681c19f0f0 con 0x7f681c07dde0 2024-03-24T10:30:54.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.911+0000 7f68197fa700 1 -- 172.21.15.23:0/2866907282 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f680c01a900 con 0x7f681c07dde0 2024-03-24T10:30:54.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.911+0000 7f68222f5700 1 -- 172.21.15.23:0/2866907282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f681c063830 con 0x7f681c07dde0 2024-03-24T10:30:54.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.911+0000 7f68197fa700 1 --2- 172.21.15.23:0/2866907282 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f6804038ee0 0x7f680403b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:30:54.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.911+0000 7f68197fa700 1 -- 172.21.15.23:0/2866907282 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f680c04d170 con 0x7f681c07dde0 2024-03-24T10:30:54.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.911+0000 7f681b7fe700 1 --2- 172.21.15.23:0/2866907282 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f6804038ee0 0x7f680403b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:30:54.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.912+0000 7f681b7fe700 1 --2- 172.21.15.23:0/2866907282 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f6804038ee0 0x7f680403b3a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6810006fd0 tx=0x7f6810006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:30:54.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:53.916+0000 7f68197fa700 1 -- 172.21.15.23:0/2866907282 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f680c018530 con 0x7f681c07dde0 2024-03-24T10:30:54.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:54.124+0000 7f68222f5700 1 -- 172.21.15.23:0/2866907282 --> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f681c1045f0 con 0x7f6804038ee0 2024-03-24T10:30:54.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:54.124+0000 7f68197fa700 1 -- 172.21.15.23:0/2866907282 <== mgr.14120 v2:172.21.15.23:6800/2159398914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f681c1045f0 con 0x7f6804038ee0 2024-03-24T10:30:54.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:54.126+0000 7f68222f5700 1 -- 172.21.15.23:0/2866907282 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f6804038ee0 msgr2=0x7f680403b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:54.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:54.126+0000 7f68222f5700 1 --2- 172.21.15.23:0/2866907282 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f6804038ee0 0x7f680403b3a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6810006fd0 tx=0x7f6810006e40 comp rx=0 tx=0).stop 2024-03-24T10:30:54.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:54.126+0000 7f68222f5700 1 -- 172.21.15.23:0/2866907282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c07dde0 msgr2=0x7f681c19dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:30:54.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:54.126+0000 7f68222f5700 1 --2- 172.21.15.23:0/2866907282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c07dde0 0x7f681c19dee0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f680c003f70 tx=0x7f680c011b40 comp rx=0 tx=0).stop 2024-03-24T10:30:54.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:54.127+0000 7f68222f5700 1 -- 172.21.15.23:0/2866907282 shutdown_connections 2024-03-24T10:30:54.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:54.127+0000 7f68222f5700 1 --2- 172.21.15.23:0/2866907282 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f6804038ee0 0x7f680403b3a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:54.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:54.127+0000 7f68222f5700 1 --2- 172.21.15.23:0/2866907282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c07dde0 0x7f681c19dee0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:30:54.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:54.127+0000 7f68222f5700 1 -- 172.21.15.23:0/2866907282 >> 172.21.15.23:0/2866907282 conn(0x7f681c103360 msgr2=0x7f681c103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:30:54.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:54.127+0000 7f68222f5700 1 -- 172.21.15.23:0/2866907282 shutdown_connections 2024-03-24T10:30:54.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:54.127+0000 7f68222f5700 1 -- 172.21.15.23:0/2866907282 wait complete. 2024-03-24T10:30:54.664 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T10:30:54.664 INFO:teuthology.orchestra.run.smithi023.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T10:30:54.664 INFO:teuthology.orchestra.run.smithi023.stdout:Adding host smithi023... 2024-03-24T10:30:55.400 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:55 smithi023 ceph-mon[32466]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:30:56.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:56 smithi023 ceph-mon[32466]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:30:56.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:30:56 smithi023 ceph-mon[32466]: Deploying cephadm binary to smithi023 2024-03-24T10:31:00.132 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Added host 'smithi023' with addr '172.21.15.23' 2024-03-24T10:31:00.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.083+0000 7f0d12fbd700 1 Processor -- start 2024-03-24T10:31:00.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.083+0000 7f0d12fbd700 1 -- start start 2024-03-24T10:31:00.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.083+0000 7f0d12fbd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d0c108870 0x7f0d0c10ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:00.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.083+0000 7f0d12fbd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c07bdb0 con 0x7f0d0c108870 2024-03-24T10:31:00.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.083+0000 7f0d10d59700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d0c108870 0x7f0d0c10ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:00.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.083+0000 7f0d10d59700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d0c108870 0x7f0d0c10ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38102/0 (socket says 172.21.15.23:38102) 2024-03-24T10:31:00.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.083+0000 7f0d10d59700 1 -- 172.21.15.23:0/1397483035 learned_addr learned my addr 172.21.15.23:0/1397483035 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:00.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.083+0000 7f0d10d59700 1 -- 172.21.15.23:0/1397483035 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0c10b1a0 con 0x7f0d0c108870 2024-03-24T10:31:00.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.083+0000 7f0d10d59700 1 --2- 172.21.15.23:0/1397483035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d0c108870 0x7f0d0c10ac60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0d000098a0 tx=0x7f0d00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ddd962ae5fedae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:00.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.084+0000 7f0d0b7fe700 1 -- 172.21.15.23:0/1397483035 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d00004160 con 0x7f0d0c108870 2024-03-24T10:31:00.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.084+0000 7f0d0b7fe700 1 -- 172.21.15.23:0/1397483035 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0d000042c0 con 0x7f0d0c108870 2024-03-24T10:31:00.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.084+0000 7f0d0b7fe700 1 -- 172.21.15.23:0/1397483035 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d00011ac0 con 0x7f0d0c108870 2024-03-24T10:31:00.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.084+0000 7f0d12fbd700 1 -- 172.21.15.23:0/1397483035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d0c108870 msgr2=0x7f0d0c10ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:00.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.084+0000 7f0d12fbd700 1 --2- 172.21.15.23:0/1397483035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d0c108870 0x7f0d0c10ac60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0d000098a0 tx=0x7f0d00009bb0 comp rx=0 tx=0).stop 2024-03-24T10:31:00.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.084+0000 7f0d12fbd700 1 -- 172.21.15.23:0/1397483035 shutdown_connections 2024-03-24T10:31:00.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.084+0000 7f0d12fbd700 1 --2- 172.21.15.23:0/1397483035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d0c108870 0x7f0d0c10ac60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:00.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.084+0000 7f0d12fbd700 1 -- 172.21.15.23:0/1397483035 >> 172.21.15.23:0/1397483035 conn(0x7f0d0c102260 msgr2=0x7f0d0c1046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:00.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.084+0000 7f0d12fbd700 1 -- 172.21.15.23:0/1397483035 shutdown_connections 2024-03-24T10:31:00.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.084+0000 7f0d12fbd700 1 -- 172.21.15.23:0/1397483035 wait complete. 2024-03-24T10:31:00.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.085+0000 7f0d12fbd700 1 Processor -- start 2024-03-24T10:31:00.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.085+0000 7f0d12fbd700 1 -- start start 2024-03-24T10:31:00.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.085+0000 7f0d12fbd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d0c108870 0x7f0d0c108280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:00.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.085+0000 7f0d12fbd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c106820 con 0x7f0d0c108870 2024-03-24T10:31:00.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.085+0000 7f0d10d59700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d0c108870 0x7f0d0c108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:00.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.085+0000 7f0d10d59700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d0c108870 0x7f0d0c108280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38112/0 (socket says 172.21.15.23:38112) 2024-03-24T10:31:00.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.085+0000 7f0d10d59700 1 -- 172.21.15.23:0/360052650 learned_addr learned my addr 172.21.15.23:0/360052650 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.085+0000 7f0d10d59700 1 -- 172.21.15.23:0/360052650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d00009550 con 0x7f0d0c108870 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.085+0000 7f0d10d59700 1 --2- 172.21.15.23:0/360052650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d0c108870 0x7f0d0c108280 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0d00003f70 tx=0x7f0d00011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.086+0000 7f0d09ffb700 1 -- 172.21.15.23:0/360052650 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d000039f0 con 0x7f0d0c108870 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.086+0000 7f0d09ffb700 1 -- 172.21.15.23:0/360052650 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0d00003b50 con 0x7f0d0c108870 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.086+0000 7f0d09ffb700 1 -- 172.21.15.23:0/360052650 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d0001a760 con 0x7f0d0c108870 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.086+0000 7f0d12fbd700 1 -- 172.21.15.23:0/360052650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d0c106a20 con 0x7f0d0c108870 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.086+0000 7f0d12fbd700 1 -- 172.21.15.23:0/360052650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d0c106ec0 con 0x7f0d0c108870 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.087+0000 7f0d09ffb700 1 -- 172.21.15.23:0/360052650 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f0d00024430 con 0x7f0d0c108870 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.087+0000 7f0d12fbd700 1 -- 172.21.15.23:0/360052650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d0c06a7d0 con 0x7f0d0c108870 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.087+0000 7f0d09ffb700 1 --2- 172.21.15.23:0/360052650 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f0cfc038ee0 0x7f0cfc03b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.087+0000 7f0d09ffb700 1 -- 172.21.15.23:0/360052650 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0d0004d160 con 0x7f0d0c108870 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.087+0000 7f0d0bfff700 1 --2- 172.21.15.23:0/360052650 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f0cfc038ee0 0x7f0cfc03b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.087+0000 7f0d0bfff700 1 --2- 172.21.15.23:0/360052650 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f0cfc038ee0 0x7f0cfc03b3a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8006fd0 tx=0x7f0cf8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.092+0000 7f0d09ffb700 1 -- 172.21.15.23:0/360052650 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0d00018a40 con 0x7f0d0c108870 2024-03-24T10:31:00.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:55.302+0000 7f0d12fbd700 1 -- 172.21.15.23:0/360052650 --> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}) -- 0x7f0d0c1072d0 con 0x7f0cfc038ee0 2024-03-24T10:31:00.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:59.568+0000 7f0d09ffb700 1 -- 172.21.15.23:0/360052650 <== mgr.14120 v2:172.21.15.23:6800/2159398914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f0d0c1072d0 con 0x7f0cfc038ee0 2024-03-24T10:31:00.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:59.570+0000 7f0d12fbd700 1 -- 172.21.15.23:0/360052650 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f0cfc038ee0 msgr2=0x7f0cfc03b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:00.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:59.570+0000 7f0d12fbd700 1 --2- 172.21.15.23:0/360052650 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f0cfc038ee0 0x7f0cfc03b3a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8006fd0 tx=0x7f0cf8006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:00.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:59.570+0000 7f0d12fbd700 1 -- 172.21.15.23:0/360052650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d0c108870 msgr2=0x7f0d0c108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:00.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:59.570+0000 7f0d12fbd700 1 --2- 172.21.15.23:0/360052650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d0c108870 0x7f0d0c108280 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0d00003f70 tx=0x7f0d00011b40 comp rx=0 tx=0).stop 2024-03-24T10:31:00.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:59.571+0000 7f0d12fbd700 1 -- 172.21.15.23:0/360052650 shutdown_connections 2024-03-24T10:31:00.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:59.571+0000 7f0d12fbd700 1 --2- 172.21.15.23:0/360052650 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f0cfc038ee0 0x7f0cfc03b3a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:00.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:59.571+0000 7f0d12fbd700 1 --2- 172.21.15.23:0/360052650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d0c108870 0x7f0d0c108280 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:00.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:59.571+0000 7f0d12fbd700 1 -- 172.21.15.23:0/360052650 >> 172.21.15.23:0/360052650 conn(0x7f0d0c102260 msgr2=0x7f0d0c102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:00.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:59.571+0000 7f0d12fbd700 1 -- 172.21.15.23:0/360052650 shutdown_connections 2024-03-24T10:31:00.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:30:59.571+0000 7f0d12fbd700 1 -- 172.21.15.23:0/360052650 wait complete. 2024-03-24T10:31:00.136 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mon service... 2024-03-24T10:31:00.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:00 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' 2024-03-24T10:31:00.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:00 smithi023 ceph-mon[32466]: Added host smithi023 2024-03-24T10:31:00.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:00 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:31:01.550 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T10:31:01.550 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.548+0000 7fc58e4bf700 1 Processor -- start 2024-03-24T10:31:01.550 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.548+0000 7fc58e4bf700 1 -- start start 2024-03-24T10:31:01.550 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.548+0000 7fc58e4bf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc588108870 0x7fc58810ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:01.550 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.548+0000 7fc58e4bf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc58807bd70 con 0x7fc588108870 2024-03-24T10:31:01.550 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.549+0000 7fc587fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc588108870 0x7fc58810ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:01.551 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.549+0000 7fc587fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc588108870 0x7fc58810ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35340/0 (socket says 172.21.15.23:35340) 2024-03-24T10:31:01.551 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.549+0000 7fc587fff700 1 -- 172.21.15.23:0/4011396788 learned_addr learned my addr 172.21.15.23:0/4011396788 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:01.551 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.549+0000 7fc587fff700 1 -- 172.21.15.23:0/4011396788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58810b1a0 con 0x7fc588108870 2024-03-24T10:31:01.551 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.549+0000 7fc587fff700 1 --2- 172.21.15.23:0/4011396788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc588108870 0x7fc58810ac60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5780098a0 tx=0x7fc578009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8bef8fe40a151d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:01.551 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.549+0000 7fc586ffd700 1 -- 172.21.15.23:0/4011396788 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc578004160 con 0x7fc588108870 2024-03-24T10:31:01.551 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.549+0000 7fc586ffd700 1 -- 172.21.15.23:0/4011396788 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc5780042c0 con 0x7fc588108870 2024-03-24T10:31:01.551 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.549+0000 7fc586ffd700 1 -- 172.21.15.23:0/4011396788 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc578011ac0 con 0x7fc588108870 2024-03-24T10:31:01.551 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.550+0000 7fc58e4bf700 1 -- 172.21.15.23:0/4011396788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc588108870 msgr2=0x7fc58810ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:01.551 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.550+0000 7fc58e4bf700 1 --2- 172.21.15.23:0/4011396788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc588108870 0x7fc58810ac60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5780098a0 tx=0x7fc578009bb0 comp rx=0 tx=0).stop 2024-03-24T10:31:01.551 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.550+0000 7fc58e4bf700 1 -- 172.21.15.23:0/4011396788 shutdown_connections 2024-03-24T10:31:01.551 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.550+0000 7fc58e4bf700 1 --2- 172.21.15.23:0/4011396788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc588108870 0x7fc58810ac60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:01.551 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.550+0000 7fc58e4bf700 1 -- 172.21.15.23:0/4011396788 >> 172.21.15.23:0/4011396788 conn(0x7fc588102260 msgr2=0x7fc588104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:01.551 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.550+0000 7fc58e4bf700 1 -- 172.21.15.23:0/4011396788 shutdown_connections 2024-03-24T10:31:01.552 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.550+0000 7fc58e4bf700 1 -- 172.21.15.23:0/4011396788 wait complete. 2024-03-24T10:31:01.552 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.550+0000 7fc58e4bf700 1 Processor -- start 2024-03-24T10:31:01.552 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.551+0000 7fc58e4bf700 1 -- start start 2024-03-24T10:31:01.552 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.551+0000 7fc58e4bf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc588108870 0x7fc58819bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:01.552 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.551+0000 7fc58e4bf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc58819c180 con 0x7fc588108870 2024-03-24T10:31:01.552 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.551+0000 7fc587fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc588108870 0x7fc58819bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:01.552 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.551+0000 7fc587fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc588108870 0x7fc58819bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35346/0 (socket says 172.21.15.23:35346) 2024-03-24T10:31:01.552 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.551+0000 7fc587fff700 1 -- 172.21.15.23:0/390939599 learned_addr learned my addr 172.21.15.23:0/390939599 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:01.552 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.551+0000 7fc587fff700 1 -- 172.21.15.23:0/390939599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc578009550 con 0x7fc588108870 2024-03-24T10:31:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.551+0000 7fc587fff700 1 --2- 172.21.15.23:0/390939599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc588108870 0x7fc58819bc40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc578009130 tx=0x7fc578011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.551+0000 7fc5857fa700 1 -- 172.21.15.23:0/390939599 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc578003aa0 con 0x7fc588108870 2024-03-24T10:31:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.552+0000 7fc58e4bf700 1 -- 172.21.15.23:0/390939599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc58819c380 con 0x7fc588108870 2024-03-24T10:31:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.552+0000 7fc5857fa700 1 -- 172.21.15.23:0/390939599 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc578003c00 con 0x7fc588108870 2024-03-24T10:31:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.552+0000 7fc5857fa700 1 -- 172.21.15.23:0/390939599 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc578024800 con 0x7fc588108870 2024-03-24T10:31:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.552+0000 7fc58e4bf700 1 -- 172.21.15.23:0/390939599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc58819cf30 con 0x7fc588108870 2024-03-24T10:31:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.552+0000 7fc5857fa700 1 -- 172.21.15.23:0/390939599 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fc578024b00 con 0x7fc588108870 2024-03-24T10:31:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.552+0000 7fc58e4bf700 1 -- 172.21.15.23:0/390939599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc588063830 con 0x7fc588108870 2024-03-24T10:31:01.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.553+0000 7fc5857fa700 1 --2- 172.21.15.23:0/390939599 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fc570038ee0 0x7fc57003b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.553+0000 7fc5857fa700 1 -- 172.21.15.23:0/390939599 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc57804d0b0 con 0x7fc588108870 2024-03-24T10:31:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.553+0000 7fc5877fe700 1 --2- 172.21.15.23:0/390939599 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fc570038ee0 0x7fc57003b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.553+0000 7fc5877fe700 1 --2- 172.21.15.23:0/390939599 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fc570038ee0 0x7fc57003b3a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc57c006fd0 tx=0x7fc57c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:01.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.558+0000 7fc5857fa700 1 -- 172.21.15.23:0/390939599 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc578024db0 con 0x7fc588108870 2024-03-24T10:31:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.785+0000 7fc58e4bf700 1 -- 172.21.15.23:0/390939599 --> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fc588103480 con 0x7fc570038ee0 2024-03-24T10:31:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.792+0000 7fc5857fa700 1 -- 172.21.15.23:0/390939599 <== mgr.14120 v2:172.21.15.23:6800/2159398914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc588103480 con 0x7fc570038ee0 2024-03-24T10:31:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.794+0000 7fc58e4bf700 1 -- 172.21.15.23:0/390939599 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fc570038ee0 msgr2=0x7fc57003b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:01.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.794+0000 7fc58e4bf700 1 --2- 172.21.15.23:0/390939599 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fc570038ee0 0x7fc57003b3a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc57c006fd0 tx=0x7fc57c006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.794+0000 7fc58e4bf700 1 -- 172.21.15.23:0/390939599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc588108870 msgr2=0x7fc58819bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.794+0000 7fc58e4bf700 1 --2- 172.21.15.23:0/390939599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc588108870 0x7fc58819bc40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc578009130 tx=0x7fc578011b40 comp rx=0 tx=0).stop 2024-03-24T10:31:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.794+0000 7fc58e4bf700 1 -- 172.21.15.23:0/390939599 shutdown_connections 2024-03-24T10:31:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.794+0000 7fc58e4bf700 1 --2- 172.21.15.23:0/390939599 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fc570038ee0 0x7fc57003b3a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.794+0000 7fc58e4bf700 1 --2- 172.21.15.23:0/390939599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc588108870 0x7fc58819bc40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.794+0000 7fc58e4bf700 1 -- 172.21.15.23:0/390939599 >> 172.21.15.23:0/390939599 conn(0x7fc588102260 msgr2=0x7fc588102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.795+0000 7fc58e4bf700 1 -- 172.21.15.23:0/390939599 shutdown_connections 2024-03-24T10:31:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:00.795+0000 7fc58e4bf700 1 -- 172.21.15.23:0/390939599 wait complete. 2024-03-24T10:31:01.556 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mgr service... 2024-03-24T10:31:02.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:01 smithi023 ceph-mon[32466]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:31:02.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:01 smithi023 ceph-mon[32466]: Saving service mon spec with placement count:5 2024-03-24T10:31:02.093 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:01 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' 2024-03-24T10:31:03.371 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T10:31:03.371 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.932+0000 7f0b75e8c700 1 Processor -- start 2024-03-24T10:31:03.371 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.932+0000 7f0b75e8c700 1 -- start start 2024-03-24T10:31:03.371 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.933+0000 7f0b75e8c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b70106c80 0x7f0b701070a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:03.371 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.933+0000 7f0b75e8c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0b70107670 con 0x7f0b70106c80 2024-03-24T10:31:03.371 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.933+0000 7f0b6f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b70106c80 0x7f0b701070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:03.371 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.933+0000 7f0b6f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b70106c80 0x7f0b701070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35362/0 (socket says 172.21.15.23:35362) 2024-03-24T10:31:03.371 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.933+0000 7f0b6f7fe700 1 -- 172.21.15.23:0/1217329201 learned_addr learned my addr 172.21.15.23:0/1217329201 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:03.372 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.933+0000 7f0b6f7fe700 1 -- 172.21.15.23:0/1217329201 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b70107e80 con 0x7f0b70106c80 2024-03-24T10:31:03.372 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.933+0000 7f0b6f7fe700 1 --2- 172.21.15.23:0/1217329201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b70106c80 0x7f0b701070a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000d180 tx=0x7f0b6000d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cfdfc4bd986315c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:03.372 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.933+0000 7f0b6effd700 1 -- 172.21.15.23:0/1217329201 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b60010070 con 0x7f0b70106c80 2024-03-24T10:31:03.372 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.933+0000 7f0b6effd700 1 -- 172.21.15.23:0/1217329201 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0b6000de20 con 0x7f0b70106c80 2024-03-24T10:31:03.372 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.933+0000 7f0b6effd700 1 -- 172.21.15.23:0/1217329201 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b60009b90 con 0x7f0b70106c80 2024-03-24T10:31:03.372 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.934+0000 7f0b75e8c700 1 -- 172.21.15.23:0/1217329201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b70106c80 msgr2=0x7f0b701070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:03.372 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.934+0000 7f0b75e8c700 1 --2- 172.21.15.23:0/1217329201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b70106c80 0x7f0b701070a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0b6000d180 tx=0x7f0b6000d490 comp rx=0 tx=0).stop 2024-03-24T10:31:03.372 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.934+0000 7f0b75e8c700 1 -- 172.21.15.23:0/1217329201 shutdown_connections 2024-03-24T10:31:03.372 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.934+0000 7f0b75e8c700 1 --2- 172.21.15.23:0/1217329201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b70106c80 0x7f0b701070a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:03.372 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.934+0000 7f0b75e8c700 1 -- 172.21.15.23:0/1217329201 >> 172.21.15.23:0/1217329201 conn(0x7f0b70102200 msgr2=0x7f0b70104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:03.372 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.934+0000 7f0b75e8c700 1 -- 172.21.15.23:0/1217329201 shutdown_connections 2024-03-24T10:31:03.372 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.934+0000 7f0b75e8c700 1 -- 172.21.15.23:0/1217329201 wait complete. 2024-03-24T10:31:03.373 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.935+0000 7f0b75e8c700 1 Processor -- start 2024-03-24T10:31:03.373 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.935+0000 7f0b75e8c700 1 -- start start 2024-03-24T10:31:03.373 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.935+0000 7f0b75e8c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b70106c80 0x7f0b7019de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:03.373 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.935+0000 7f0b75e8c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7019e370 con 0x7f0b70106c80 2024-03-24T10:31:03.373 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.935+0000 7f0b6f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b70106c80 0x7f0b7019de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:03.373 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.935+0000 7f0b6f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b70106c80 0x7f0b7019de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35368/0 (socket says 172.21.15.23:35368) 2024-03-24T10:31:03.373 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.935+0000 7f0b6f7fe700 1 -- 172.21.15.23:0/4250282336 learned_addr learned my addr 172.21.15.23:0/4250282336 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:03.373 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.935+0000 7f0b6f7fe700 1 -- 172.21.15.23:0/4250282336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6000bd40 con 0x7f0b70106c80 2024-03-24T10:31:03.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.936+0000 7f0b6f7fe700 1 --2- 172.21.15.23:0/4250282336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b70106c80 0x7f0b7019de30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0b60009c30 tx=0x7f0b60003ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:03.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.936+0000 7f0b6d7fa700 1 -- 172.21.15.23:0/4250282336 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b60010040 con 0x7f0b70106c80 2024-03-24T10:31:03.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.936+0000 7f0b6d7fa700 1 -- 172.21.15.23:0/4250282336 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0b6000b410 con 0x7f0b70106c80 2024-03-24T10:31:03.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.936+0000 7f0b6d7fa700 1 -- 172.21.15.23:0/4250282336 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b60014870 con 0x7f0b70106c80 2024-03-24T10:31:03.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.936+0000 7f0b75e8c700 1 -- 172.21.15.23:0/4250282336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b7019e570 con 0x7f0b70106c80 2024-03-24T10:31:03.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.936+0000 7f0b75e8c700 1 -- 172.21.15.23:0/4250282336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b7019f070 con 0x7f0b70106c80 2024-03-24T10:31:03.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.937+0000 7f0b6d7fa700 1 -- 172.21.15.23:0/4250282336 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f0b6001b070 con 0x7f0b70106c80 2024-03-24T10:31:03.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.937+0000 7f0b75e8c700 1 -- 172.21.15.23:0/4250282336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b70063830 con 0x7f0b70106c80 2024-03-24T10:31:03.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.937+0000 7f0b6d7fa700 1 --2- 172.21.15.23:0/4250282336 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f0b5c038ee0 0x7f0b5c03b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:03.374 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.937+0000 7f0b6d7fa700 1 -- 172.21.15.23:0/4250282336 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0b6004d4b0 con 0x7f0b70106c80 2024-03-24T10:31:03.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.937+0000 7f0b66ffd700 1 --2- 172.21.15.23:0/4250282336 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f0b5c038ee0 0x7f0b5c03b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:03.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.938+0000 7f0b66ffd700 1 --2- 172.21.15.23:0/4250282336 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f0b5c038ee0 0x7f0b5c03b3a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0b58006fd0 tx=0x7f0b58006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:03.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:01.942+0000 7f0b6d7fa700 1 -- 172.21.15.23:0/4250282336 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0b6001b320 con 0x7f0b70106c80 2024-03-24T10:31:03.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:02.162+0000 7f0b75e8c700 1 -- 172.21.15.23:0/4250282336 --> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f0b7019f320 con 0x7f0b5c038ee0 2024-03-24T10:31:03.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:02.167+0000 7f0b6d7fa700 1 -- 172.21.15.23:0/4250282336 <== mgr.14120 v2:172.21.15.23:6800/2159398914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0b7019f320 con 0x7f0b5c038ee0 2024-03-24T10:31:03.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:02.169+0000 7f0b75e8c700 1 -- 172.21.15.23:0/4250282336 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f0b5c038ee0 msgr2=0x7f0b5c03b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:03.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:02.169+0000 7f0b75e8c700 1 --2- 172.21.15.23:0/4250282336 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f0b5c038ee0 0x7f0b5c03b3a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0b58006fd0 tx=0x7f0b58006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:03.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:02.169+0000 7f0b75e8c700 1 -- 172.21.15.23:0/4250282336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b70106c80 msgr2=0x7f0b7019de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:03.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:02.169+0000 7f0b75e8c700 1 --2- 172.21.15.23:0/4250282336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b70106c80 0x7f0b7019de30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0b60009c30 tx=0x7f0b60003ef0 comp rx=0 tx=0).stop 2024-03-24T10:31:03.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:02.169+0000 7f0b75e8c700 1 -- 172.21.15.23:0/4250282336 shutdown_connections 2024-03-24T10:31:03.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:02.169+0000 7f0b75e8c700 1 --2- 172.21.15.23:0/4250282336 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f0b5c038ee0 0x7f0b5c03b3a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:03.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:02.169+0000 7f0b75e8c700 1 --2- 172.21.15.23:0/4250282336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b70106c80 0x7f0b7019de30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:03.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:02.169+0000 7f0b75e8c700 1 -- 172.21.15.23:0/4250282336 >> 172.21.15.23:0/4250282336 conn(0x7f0b70102200 msgr2=0x7f0b701900a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:03.375 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:02.170+0000 7f0b75e8c700 1 -- 172.21.15.23:0/4250282336 shutdown_connections 2024-03-24T10:31:03.376 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:02.170+0000 7f0b75e8c700 1 -- 172.21.15.23:0/4250282336 wait complete. 2024-03-24T10:31:03.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:03 smithi023 ceph-mon[32466]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:31:03.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:03 smithi023 ceph-mon[32466]: Saving service mgr spec with placement count:2 2024-03-24T10:31:03.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:03 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' 2024-03-24T10:31:03.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:03 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' 2024-03-24T10:31:04.643 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.838+0000 7f7ce08d4700 1 Processor -- start 2024-03-24T10:31:04.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.838+0000 7f7ce08d4700 1 -- start start 2024-03-24T10:31:04.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.839+0000 7f7ce08d4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cdc106640 0x7f7cdc106a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:04.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.839+0000 7f7ce08d4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc07bdb0 con 0x7f7cdc106640 2024-03-24T10:31:04.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.839+0000 7f7cda59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cdc106640 0x7f7cdc106a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:04.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.839+0000 7f7cda59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cdc106640 0x7f7cdc106a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35382/0 (socket says 172.21.15.23:35382) 2024-03-24T10:31:04.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.839+0000 7f7cda59c700 1 -- 172.21.15.23:0/1237271557 learned_addr learned my addr 172.21.15.23:0/1237271557 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:04.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.839+0000 7f7cda59c700 1 -- 172.21.15.23:0/1237271557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cdc106fa0 con 0x7f7cdc106640 2024-03-24T10:31:04.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.840+0000 7f7cda59c700 1 --2- 172.21.15.23:0/1237271557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cdc106640 0x7f7cdc106a60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7cc40098a0 tx=0x7f7cc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5066f5e7edc4348 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:04.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.840+0000 7f7cd959a700 1 -- 172.21.15.23:0/1237271557 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cc4004160 con 0x7f7cdc106640 2024-03-24T10:31:04.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.840+0000 7f7cd959a700 1 -- 172.21.15.23:0/1237271557 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cc40042c0 con 0x7f7cdc106640 2024-03-24T10:31:04.644 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.840+0000 7f7cd959a700 1 -- 172.21.15.23:0/1237271557 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cc4011a80 con 0x7f7cdc106640 2024-03-24T10:31:04.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.840+0000 7f7ce08d4700 1 -- 172.21.15.23:0/1237271557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cdc106640 msgr2=0x7f7cdc106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:04.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.840+0000 7f7ce08d4700 1 --2- 172.21.15.23:0/1237271557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cdc106640 0x7f7cdc106a60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7cc40098a0 tx=0x7f7cc4009bb0 comp rx=0 tx=0).stop 2024-03-24T10:31:04.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.841+0000 7f7ce08d4700 1 -- 172.21.15.23:0/1237271557 shutdown_connections 2024-03-24T10:31:04.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.841+0000 7f7ce08d4700 1 --2- 172.21.15.23:0/1237271557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cdc106640 0x7f7cdc106a60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:04.645 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.841+0000 7f7ce08d4700 1 -- 172.21.15.23:0/1237271557 >> 172.21.15.23:0/1237271557 conn(0x7f7cdc102220 msgr2=0x7f7cdc104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:04.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.841+0000 7f7ce08d4700 1 -- 172.21.15.23:0/1237271557 shutdown_connections 2024-03-24T10:31:04.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.841+0000 7f7ce08d4700 1 -- 172.21.15.23:0/1237271557 wait complete. 2024-03-24T10:31:04.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.841+0000 7f7ce08d4700 1 Processor -- start 2024-03-24T10:31:04.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.841+0000 7f7ce08d4700 1 -- start start 2024-03-24T10:31:04.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.842+0000 7f7ce08d4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cdc106640 0x7f7cdc19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:04.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.842+0000 7f7ce08d4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc19e500 con 0x7f7cdc106640 2024-03-24T10:31:04.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.842+0000 7f7cda59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cdc106640 0x7f7cdc19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:04.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.842+0000 7f7cda59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cdc106640 0x7f7cdc19dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35386/0 (socket says 172.21.15.23:35386) 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.842+0000 7f7cda59c700 1 -- 172.21.15.23:0/3584767656 learned_addr learned my addr 172.21.15.23:0/3584767656 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.842+0000 7f7cda59c700 1 -- 172.21.15.23:0/3584767656 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc4009550 con 0x7f7cdc106640 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.842+0000 7f7cda59c700 1 --2- 172.21.15.23:0/3584767656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cdc106640 0x7f7cdc19dfc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4011740 tx=0x7f7cc4011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.842+0000 7f7cd37fe700 1 -- 172.21.15.23:0/3584767656 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cc4003b20 con 0x7f7cdc106640 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.842+0000 7f7ce08d4700 1 -- 172.21.15.23:0/3584767656 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cdc19e700 con 0x7f7cdc106640 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.842+0000 7f7cd37fe700 1 -- 172.21.15.23:0/3584767656 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cc401a910 con 0x7f7cdc106640 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.842+0000 7f7cd37fe700 1 -- 172.21.15.23:0/3584767656 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cc4024900 con 0x7f7cdc106640 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.843+0000 7f7ce08d4700 1 -- 172.21.15.23:0/3584767656 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cdc19f2b0 con 0x7f7cdc106640 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.843+0000 7f7cd37fe700 1 -- 172.21.15.23:0/3584767656 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f7cc4024b70 con 0x7f7cdc106640 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.843+0000 7f7cd37fe700 1 --2- 172.21.15.23:0/3584767656 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f7cc8038e90 0x7f7cc803b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.844+0000 7f7cd37fe700 1 -- 172.21.15.23:0/3584767656 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7cc404e420 con 0x7f7cdc106640 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.844+0000 7f7ce08d4700 1 -- 172.21.15.23:0/3584767656 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cdc063830 con 0x7f7cdc106640 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.844+0000 7f7cd9d9b700 1 --2- 172.21.15.23:0/3584767656 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f7cc8038e90 0x7f7cc803b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.844+0000 7f7cd9d9b700 1 --2- 172.21.15.23:0/3584767656 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f7cc8038e90 0x7f7cc803b350 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc006fd0 tx=0x7f7ccc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:04.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:03.849+0000 7f7cd37fe700 1 -- 172.21.15.23:0/3584767656 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7cc4024e20 con 0x7f7cdc106640 2024-03-24T10:31:04.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:04.053+0000 7f7ce08d4700 1 -- 172.21.15.23:0/3584767656 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f7cdc107f10 con 0x7f7cdc106640 2024-03-24T10:31:04.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:04.060+0000 7f7cd37fe700 1 -- 172.21.15.23:0/3584767656 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f7cc403ce60 con 0x7f7cdc106640 2024-03-24T10:31:04.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:04.063+0000 7f7ce08d4700 1 -- 172.21.15.23:0/3584767656 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f7cc8038e90 msgr2=0x7f7cc803b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:04.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:04.063+0000 7f7ce08d4700 1 --2- 172.21.15.23:0/3584767656 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f7cc8038e90 0x7f7cc803b350 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc006fd0 tx=0x7f7ccc006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:04.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:04.063+0000 7f7ce08d4700 1 -- 172.21.15.23:0/3584767656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cdc106640 msgr2=0x7f7cdc19dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:04.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:04.063+0000 7f7ce08d4700 1 --2- 172.21.15.23:0/3584767656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cdc106640 0x7f7cdc19dfc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4011740 tx=0x7f7cc4011a50 comp rx=0 tx=0).stop 2024-03-24T10:31:04.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:04.063+0000 7f7ce08d4700 1 -- 172.21.15.23:0/3584767656 shutdown_connections 2024-03-24T10:31:04.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:04.063+0000 7f7ce08d4700 1 --2- 172.21.15.23:0/3584767656 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f7cc8038e90 0x7f7cc803b350 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:04.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:04.063+0000 7f7ce08d4700 1 --2- 172.21.15.23:0/3584767656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cdc106640 0x7f7cdc19dfc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:04.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:04.063+0000 7f7ce08d4700 1 -- 172.21.15.23:0/3584767656 >> 172.21.15.23:0/3584767656 conn(0x7f7cdc102220 msgr2=0x7f7cdc194a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:04.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:04.063+0000 7f7ce08d4700 1 -- 172.21.15.23:0/3584767656 shutdown_connections 2024-03-24T10:31:04.655 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:04.063+0000 7f7ce08d4700 1 -- 172.21.15.23:0/3584767656 wait complete. 2024-03-24T10:31:04.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:04 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' 2024-03-24T10:31:04.989 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:04 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3584767656' entity='client.admin' 2024-03-24T10:31:06.287 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.092+0000 7f89d5faf700 1 Processor -- start 2024-03-24T10:31:06.288 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.093+0000 7f89d5faf700 1 -- start start 2024-03-24T10:31:06.288 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.093+0000 7f89d5faf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d0106070 0x7f89d0106490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:06.288 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.093+0000 7f89d5faf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f89d0106a60 con 0x7f89d0106070 2024-03-24T10:31:06.288 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.093+0000 7f89cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d0106070 0x7f89d0106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:06.288 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.093+0000 7f89cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d0106070 0x7f89d0106490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35392/0 (socket says 172.21.15.23:35392) 2024-03-24T10:31:06.288 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.093+0000 7f89cf7fe700 1 -- 172.21.15.23:0/1477829536 learned_addr learned my addr 172.21.15.23:0/1477829536 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:06.288 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.093+0000 7f89cf7fe700 1 -- 172.21.15.23:0/1477829536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d0109000 con 0x7f89d0106070 2024-03-24T10:31:06.288 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.093+0000 7f89cf7fe700 1 --2- 172.21.15.23:0/1477829536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d0106070 0x7f89d0106490 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f89c00098a0 tx=0x7f89c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7caec87f7ef38e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:06.288 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.094+0000 7f89ce7fc700 1 -- 172.21.15.23:0/1477829536 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89c0004160 con 0x7f89d0106070 2024-03-24T10:31:06.288 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.094+0000 7f89ce7fc700 1 -- 172.21.15.23:0/1477829536 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89c00042c0 con 0x7f89d0106070 2024-03-24T10:31:06.290 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.094+0000 7f89ce7fc700 1 -- 172.21.15.23:0/1477829536 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89c0011a80 con 0x7f89d0106070 2024-03-24T10:31:06.290 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.094+0000 7f89d5faf700 1 -- 172.21.15.23:0/1477829536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d0106070 msgr2=0x7f89d0106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:06.290 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.094+0000 7f89d5faf700 1 --2- 172.21.15.23:0/1477829536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d0106070 0x7f89d0106490 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f89c00098a0 tx=0x7f89c0009bb0 comp rx=0 tx=0).stop 2024-03-24T10:31:06.290 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.094+0000 7f89d5faf700 1 -- 172.21.15.23:0/1477829536 shutdown_connections 2024-03-24T10:31:06.290 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.094+0000 7f89d5faf700 1 --2- 172.21.15.23:0/1477829536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d0106070 0x7f89d0106490 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:06.290 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.094+0000 7f89d5faf700 1 -- 172.21.15.23:0/1477829536 >> 172.21.15.23:0/1477829536 conn(0x7f89d0101a20 msgr2=0x7f89d0103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:06.290 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.094+0000 7f89d5faf700 1 -- 172.21.15.23:0/1477829536 shutdown_connections 2024-03-24T10:31:06.290 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.094+0000 7f89d5faf700 1 -- 172.21.15.23:0/1477829536 wait complete. 2024-03-24T10:31:06.290 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.095+0000 7f89d5faf700 1 Processor -- start 2024-03-24T10:31:06.290 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.095+0000 7f89d5faf700 1 -- start start 2024-03-24T10:31:06.290 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.095+0000 7f89d5faf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d0106070 0x7f89d0108a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:06.290 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.095+0000 7f89d5faf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f89d0106fb0 con 0x7f89d0106070 2024-03-24T10:31:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.095+0000 7f89cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d0106070 0x7f89d0108a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.095+0000 7f89cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d0106070 0x7f89d0108a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35402/0 (socket says 172.21.15.23:35402) 2024-03-24T10:31:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.095+0000 7f89cf7fe700 1 -- 172.21.15.23:0/4076285944 learned_addr learned my addr 172.21.15.23:0/4076285944 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.096+0000 7f89cf7fe700 1 -- 172.21.15.23:0/4076285944 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c0009550 con 0x7f89d0106070 2024-03-24T10:31:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.096+0000 7f89cf7fe700 1 --2- 172.21.15.23:0/4076285944 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d0106070 0x7f89d0108a60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f89c0011740 tx=0x7f89c0011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.096+0000 7f89ccff9700 1 -- 172.21.15.23:0/4076285944 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89c0003b20 con 0x7f89d0106070 2024-03-24T10:31:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.096+0000 7f89ccff9700 1 -- 172.21.15.23:0/4076285944 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89c001a910 con 0x7f89d0106070 2024-03-24T10:31:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.096+0000 7f89d5faf700 1 -- 172.21.15.23:0/4076285944 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d01071b0 con 0x7f89d0106070 2024-03-24T10:31:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.096+0000 7f89ccff9700 1 -- 172.21.15.23:0/4076285944 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89c0024850 con 0x7f89d0106070 2024-03-24T10:31:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.096+0000 7f89d5faf700 1 -- 172.21.15.23:0/4076285944 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d0107650 con 0x7f89d0106070 2024-03-24T10:31:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.097+0000 7f89ccff9700 1 -- 172.21.15.23:0/4076285944 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f89c002d3f0 con 0x7f89d0106070 2024-03-24T10:31:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.097+0000 7f89d5faf700 1 -- 172.21.15.23:0/4076285944 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d006a7d0 con 0x7f89d0106070 2024-03-24T10:31:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.097+0000 7f89ccff9700 1 --2- 172.21.15.23:0/4076285944 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f89b8038f30 0x7f89b803b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.097+0000 7f89ccff9700 1 -- 172.21.15.23:0/4076285944 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f89c004d4e0 con 0x7f89d0106070 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.097+0000 7f89ceffd700 1 --2- 172.21.15.23:0/4076285944 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f89b8038f30 0x7f89b803b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.097+0000 7f89ceffd700 1 --2- 172.21.15.23:0/4076285944 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f89b8038f30 0x7f89b803b3f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f89c4006fd0 tx=0x7f89c4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.102+0000 7f89ccff9700 1 -- 172.21.15.23:0/4076285944 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f89c000b650 con 0x7f89d0106070 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.317+0000 7f89d5faf700 1 -- 172.21.15.23:0/4076285944 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f89d010c230 con 0x7f89d0106070 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.320+0000 7f89ccff9700 1 -- 172.21.15.23:0/4076285944 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f89c000b460 con 0x7f89d0106070 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.322+0000 7f89d5faf700 1 -- 172.21.15.23:0/4076285944 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f89b8038f30 msgr2=0x7f89b803b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.322+0000 7f89d5faf700 1 --2- 172.21.15.23:0/4076285944 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f89b8038f30 0x7f89b803b3f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f89c4006fd0 tx=0x7f89c4006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.322+0000 7f89d5faf700 1 -- 172.21.15.23:0/4076285944 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d0106070 msgr2=0x7f89d0108a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.322+0000 7f89d5faf700 1 --2- 172.21.15.23:0/4076285944 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d0106070 0x7f89d0108a60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f89c0011740 tx=0x7f89c0011b00 comp rx=0 tx=0).stop 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.322+0000 7f89d5faf700 1 -- 172.21.15.23:0/4076285944 shutdown_connections 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.322+0000 7f89d5faf700 1 --2- 172.21.15.23:0/4076285944 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7f89b8038f30 0x7f89b803b3f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.322+0000 7f89d5faf700 1 --2- 172.21.15.23:0/4076285944 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d0106070 0x7f89d0108a60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.322+0000 7f89d5faf700 1 -- 172.21.15.23:0/4076285944 >> 172.21.15.23:0/4076285944 conn(0x7f89d0101a20 msgr2=0x7f89d0102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.322+0000 7f89d5faf700 1 -- 172.21.15.23:0/4076285944 shutdown_connections 2024-03-24T10:31:06.292 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:05.322+0000 7f89d5faf700 1 -- 172.21.15.23:0/4076285944 wait complete. 2024-03-24T10:31:06.293 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling the dashboard module... 2024-03-24T10:31:06.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:06 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4076285944' entity='client.admin' 2024-03-24T10:31:07.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:07 smithi023 ceph-mon[32466]: from='mgr.14120 172.21.15.23:0/1527795833' entity='mgr.x' 2024-03-24T10:31:07.877 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:07 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4276480228' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T10:31:07.878 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:07 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: ignoring --setuser ceph since I am not root 2024-03-24T10:31:07.878 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:07 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: ignoring --setgroup ceph since I am not root 2024-03-24T10:31:08.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.745+0000 7fc0f6b06700 1 Processor -- start 2024-03-24T10:31:08.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.745+0000 7fc0f6b06700 1 -- start start 2024-03-24T10:31:08.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.745+0000 7fc0f6b06700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0f01063e0 0x7fc0f0106800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:08.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.745+0000 7fc0f6b06700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f0106dd0 con 0x7fc0f01063e0 2024-03-24T10:31:08.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.745+0000 7fc0f48a2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0f01063e0 0x7fc0f0106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:08.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.746+0000 7fc0f48a2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0f01063e0 0x7fc0f0106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35408/0 (socket says 172.21.15.23:35408) 2024-03-24T10:31:08.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.746+0000 7fc0f48a2700 1 -- 172.21.15.23:0/1088887232 learned_addr learned my addr 172.21.15.23:0/1088887232 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:08.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.746+0000 7fc0f48a2700 1 -- 172.21.15.23:0/1088887232 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f01075e0 con 0x7fc0f01063e0 2024-03-24T10:31:08.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.746+0000 7fc0f48a2700 1 --2- 172.21.15.23:0/1088887232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0f01063e0 0x7fc0f0106800 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400aa40 tx=0x7fc0e400ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49e7351cad418c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:08.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.746+0000 7fc0ef7fe700 1 -- 172.21.15.23:0/1088887232 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0e4004160 con 0x7fc0f01063e0 2024-03-24T10:31:08.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.746+0000 7fc0ef7fe700 1 -- 172.21.15.23:0/1088887232 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0e400cdd0 con 0x7fc0f01063e0 2024-03-24T10:31:08.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.746+0000 7fc0ef7fe700 1 -- 172.21.15.23:0/1088887232 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0e4003cd0 con 0x7fc0f01063e0 2024-03-24T10:31:08.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.747+0000 7fc0f6b06700 1 -- 172.21.15.23:0/1088887232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0f01063e0 msgr2=0x7fc0f0106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:08.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.747+0000 7fc0f6b06700 1 --2- 172.21.15.23:0/1088887232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0f01063e0 0x7fc0f0106800 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400aa40 tx=0x7fc0e400ad50 comp rx=0 tx=0).stop 2024-03-24T10:31:08.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.747+0000 7fc0f6b06700 1 -- 172.21.15.23:0/1088887232 shutdown_connections 2024-03-24T10:31:08.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.747+0000 7fc0f6b06700 1 --2- 172.21.15.23:0/1088887232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0f01063e0 0x7fc0f0106800 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:08.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.747+0000 7fc0f6b06700 1 -- 172.21.15.23:0/1088887232 >> 172.21.15.23:0/1088887232 conn(0x7fc0f01019e0 msgr2=0x7fc0f0103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:08.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.747+0000 7fc0f6b06700 1 -- 172.21.15.23:0/1088887232 shutdown_connections 2024-03-24T10:31:08.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.747+0000 7fc0f6b06700 1 -- 172.21.15.23:0/1088887232 wait complete. 2024-03-24T10:31:08.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.748+0000 7fc0f6b06700 1 Processor -- start 2024-03-24T10:31:08.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.748+0000 7fc0f6b06700 1 -- start start 2024-03-24T10:31:08.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.748+0000 7fc0f6b06700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0f01063e0 0x7fc0f01903f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:08.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.748+0000 7fc0f6b06700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f018e940 con 0x7fc0f01063e0 2024-03-24T10:31:08.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.748+0000 7fc0f48a2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0f01063e0 0x7fc0f01903f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:08.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.748+0000 7fc0f48a2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0f01063e0 0x7fc0f01903f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35420/0 (socket says 172.21.15.23:35420) 2024-03-24T10:31:08.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.748+0000 7fc0f48a2700 1 -- 172.21.15.23:0/4276480228 learned_addr learned my addr 172.21.15.23:0/4276480228 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:08.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.748+0000 7fc0f48a2700 1 -- 172.21.15.23:0/4276480228 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e400a6f0 con 0x7fc0f01063e0 2024-03-24T10:31:08.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.748+0000 7fc0f48a2700 1 --2- 172.21.15.23:0/4276480228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0f01063e0 0x7fc0f01903f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400a240 tx=0x7fc0e4004220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:08.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.749+0000 7fc0edffb700 1 -- 172.21.15.23:0/4276480228 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0e40103f0 con 0x7fc0f01063e0 2024-03-24T10:31:08.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.749+0000 7fc0edffb700 1 -- 172.21.15.23:0/4276480228 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0e4010a30 con 0x7fc0f01063e0 2024-03-24T10:31:08.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.749+0000 7fc0edffb700 1 -- 172.21.15.23:0/4276480228 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0e4021ad0 con 0x7fc0f01063e0 2024-03-24T10:31:08.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.749+0000 7fc0f6b06700 1 -- 172.21.15.23:0/4276480228 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f018eb40 con 0x7fc0f01063e0 2024-03-24T10:31:08.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.749+0000 7fc0f6b06700 1 -- 172.21.15.23:0/4276480228 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f018efe0 con 0x7fc0f01063e0 2024-03-24T10:31:08.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.750+0000 7fc0edffb700 1 -- 172.21.15.23:0/4276480228 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fc0e4010ba0 con 0x7fc0f01063e0 2024-03-24T10:31:08.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.750+0000 7fc0f6b06700 1 -- 172.21.15.23:0/4276480228 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0f0195860 con 0x7fc0f01063e0 2024-03-24T10:31:08.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.750+0000 7fc0edffb700 1 --2- 172.21.15.23:0/4276480228 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fc0e0038ee0 0x7fc0e003b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:08.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.750+0000 7fc0edffb700 1 -- 172.21.15.23:0/4276480228 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc0e404d3f0 con 0x7fc0f01063e0 2024-03-24T10:31:08.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.750+0000 7fc0effff700 1 --2- 172.21.15.23:0/4276480228 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fc0e0038ee0 0x7fc0e003b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:08.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.750+0000 7fc0effff700 1 --2- 172.21.15.23:0/4276480228 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fc0e0038ee0 0x7fc0e003b3a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc006fd0 tx=0x7fc0dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:08.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:06.755+0000 7fc0edffb700 1 -- 172.21.15.23:0/4276480228 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc0e400f550 con 0x7fc0f01063e0 2024-03-24T10:31:08.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:07.043+0000 7fc0f6b06700 1 -- 172.21.15.23:0/4276480228 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fc0f006a7d0 con 0x7fc0f01063e0 2024-03-24T10:31:08.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:07.613+0000 7fc0edffb700 1 -- 172.21.15.23:0/4276480228 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fc0e400f370 con 0x7fc0f01063e0 2024-03-24T10:31:08.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:07.615+0000 7fc0edffb700 1 -- 172.21.15.23:0/4276480228 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7fc0e402b440 con 0x7fc0f01063e0 2024-03-24T10:31:08.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:07.617+0000 7fc0f6b06700 1 -- 172.21.15.23:0/4276480228 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fc0e0038ee0 msgr2=0x7fc0e003b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:08.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:07.617+0000 7fc0f6b06700 1 --2- 172.21.15.23:0/4276480228 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fc0e0038ee0 0x7fc0e003b3a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc006fd0 tx=0x7fc0dc006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:08.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:07.617+0000 7fc0f6b06700 1 -- 172.21.15.23:0/4276480228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0f01063e0 msgr2=0x7fc0f01903f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:08.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:07.617+0000 7fc0f6b06700 1 --2- 172.21.15.23:0/4276480228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0f01063e0 0x7fc0f01903f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400a240 tx=0x7fc0e4004220 comp rx=0 tx=0).stop 2024-03-24T10:31:08.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:07.617+0000 7fc0f6b06700 1 -- 172.21.15.23:0/4276480228 shutdown_connections 2024-03-24T10:31:08.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:07.617+0000 7fc0f6b06700 1 --2- 172.21.15.23:0/4276480228 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fc0e0038ee0 0x7fc0e003b3a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:08.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:07.617+0000 7fc0f6b06700 1 --2- 172.21.15.23:0/4276480228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0f01063e0 0x7fc0f01903f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:08.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:07.617+0000 7fc0f6b06700 1 -- 172.21.15.23:0/4276480228 >> 172.21.15.23:0/4276480228 conn(0x7fc0f01019e0 msgr2=0x7fc0f01929a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:08.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:07.617+0000 7fc0f6b06700 1 -- 172.21.15.23:0/4276480228 shutdown_connections 2024-03-24T10:31:08.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:07.617+0000 7fc0f6b06700 1 -- 172.21.15.23:0/4276480228 wait complete. 2024-03-24T10:31:08.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:08 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:08.203+0000 7fb5a8850200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T10:31:08.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:08 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:08.300+0000 7fb5a8850200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T10:31:08.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:08 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4276480228' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T10:31:08.740 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:08 smithi023 ceph-mon[32466]: mgrmap e9: x(active, since 19s) 2024-03-24T10:31:09.069 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:08 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:08.973+0000 7fb5a8850200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T10:31:09.069 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:09 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:09.069+0000 7fb5a8850200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T10:31:09.398 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:09 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:09.212+0000 7fb5a8850200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T10:31:09.398 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:09 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:09.311+0000 7fb5a8850200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T10:31:09.398 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:09 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:09.398+0000 7fb5a8850200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T10:31:09.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:31:09.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T10:31:09.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T10:31:09.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-24T10:31:09.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T10:31:09.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:31:09.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.534+0000 7fcbe3e2f700 1 Processor -- start 2024-03-24T10:31:09.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.534+0000 7fcbe3e2f700 1 -- start start 2024-03-24T10:31:09.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.534+0000 7fcbe3e2f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbdc1088d0 0x7fcbdc10acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:09.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.534+0000 7fcbe3e2f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc07bd10 con 0x7fcbdc1088d0 2024-03-24T10:31:09.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.535+0000 7fcbe1bcb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbdc1088d0 0x7fcbdc10acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:09.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.535+0000 7fcbe1bcb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbdc1088d0 0x7fcbdc10acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53024/0 (socket says 172.21.15.23:53024) 2024-03-24T10:31:09.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.535+0000 7fcbe1bcb700 1 -- 172.21.15.23:0/2064118739 learned_addr learned my addr 172.21.15.23:0/2064118739 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:09.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.535+0000 7fcbe1bcb700 1 -- 172.21.15.23:0/2064118739 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbdc07be50 con 0x7fcbdc1088d0 2024-03-24T10:31:09.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.535+0000 7fcbe1bcb700 1 --2- 172.21.15.23:0/2064118739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbdc1088d0 0x7fcbdc10acc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80098a0 tx=0x7fcbd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5c4c1a63a25c89 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:09.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.535+0000 7fcbe0bc9700 1 -- 172.21.15.23:0/2064118739 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbd8004160 con 0x7fcbdc1088d0 2024-03-24T10:31:09.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.535+0000 7fcbe0bc9700 1 -- 172.21.15.23:0/2064118739 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbd80042c0 con 0x7fcbdc1088d0 2024-03-24T10:31:09.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.535+0000 7fcbe0bc9700 1 -- 172.21.15.23:0/2064118739 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbd8011a80 con 0x7fcbdc1088d0 2024-03-24T10:31:09.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.536+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/2064118739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbdc1088d0 msgr2=0x7fcbdc10acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:09.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.536+0000 7fcbe3e2f700 1 --2- 172.21.15.23:0/2064118739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbdc1088d0 0x7fcbdc10acc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80098a0 tx=0x7fcbd8009bb0 comp rx=0 tx=0).stop 2024-03-24T10:31:09.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.536+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/2064118739 shutdown_connections 2024-03-24T10:31:09.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.536+0000 7fcbe3e2f700 1 --2- 172.21.15.23:0/2064118739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbdc1088d0 0x7fcbdc10acc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:09.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.536+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/2064118739 >> 172.21.15.23:0/2064118739 conn(0x7fcbdc102200 msgr2=0x7fcbdc104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:09.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.536+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/2064118739 shutdown_connections 2024-03-24T10:31:09.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.536+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/2064118739 wait complete. 2024-03-24T10:31:09.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.537+0000 7fcbe3e2f700 1 Processor -- start 2024-03-24T10:31:09.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.537+0000 7fcbe3e2f700 1 -- start start 2024-03-24T10:31:09.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.537+0000 7fcbe3e2f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbdc1088d0 0x7fcbdc19bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:09.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.537+0000 7fcbe3e2f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc19c110 con 0x7fcbdc1088d0 2024-03-24T10:31:09.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.537+0000 7fcbe1bcb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbdc1088d0 0x7fcbdc19bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:09.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.537+0000 7fcbe1bcb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbdc1088d0 0x7fcbdc19bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53036/0 (socket says 172.21.15.23:53036) 2024-03-24T10:31:09.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.537+0000 7fcbe1bcb700 1 -- 172.21.15.23:0/3310697034 learned_addr learned my addr 172.21.15.23:0/3310697034 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:09.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.538+0000 7fcbe1bcb700 1 -- 172.21.15.23:0/3310697034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd8009550 con 0x7fcbdc1088d0 2024-03-24T10:31:09.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.538+0000 7fcbe1bcb700 1 --2- 172.21.15.23:0/3310697034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbdc1088d0 0x7fcbdc19bbd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8011740 tx=0x7fcbd8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:09.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.538+0000 7fcbceffd700 1 -- 172.21.15.23:0/3310697034 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbd8003b20 con 0x7fcbdc1088d0 2024-03-24T10:31:09.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.538+0000 7fcbceffd700 1 -- 172.21.15.23:0/3310697034 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbd801a910 con 0x7fcbdc1088d0 2024-03-24T10:31:09.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.538+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/3310697034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbdc19c310 con 0x7fcbdc1088d0 2024-03-24T10:31:09.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.538+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/3310697034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbdc19cec0 con 0x7fcbdc1088d0 2024-03-24T10:31:09.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.538+0000 7fcbceffd700 1 -- 172.21.15.23:0/3310697034 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbd80248c0 con 0x7fcbdc1088d0 2024-03-24T10:31:09.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.539+0000 7fcbceffd700 1 -- 172.21.15.23:0/3310697034 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7fcbd802d3f0 con 0x7fcbdc1088d0 2024-03-24T10:31:09.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.539+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/3310697034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbdc063830 con 0x7fcbdc1088d0 2024-03-24T10:31:09.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.539+0000 7fcbceffd700 1 --2- 172.21.15.23:0/3310697034 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcbc8038f90 0x7fcbc803b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:09.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.539+0000 7fcbceffd700 1 -- 172.21.15.23:0/3310697034 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcbd804d490 con 0x7fcbdc1088d0 2024-03-24T10:31:09.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.539+0000 7fcbe13ca700 1 -- 172.21.15.23:0/3310697034 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcbc8038f90 msgr2=0x7fcbc803b450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2159398914 2024-03-24T10:31:09.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.539+0000 7fcbe13ca700 1 --2- 172.21.15.23:0/3310697034 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcbc8038f90 0x7fcbc803b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:31:09.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.544+0000 7fcbceffd700 1 -- 172.21.15.23:0/3310697034 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcbd8010970 con 0x7fcbdc1088d0 2024-03-24T10:31:09.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.740+0000 7fcbe13ca700 1 -- 172.21.15.23:0/3310697034 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcbc8038f90 msgr2=0x7fcbc803b450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2159398914 2024-03-24T10:31:09.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.740+0000 7fcbe13ca700 1 --2- 172.21.15.23:0/3310697034 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcbc8038f90 0x7fcbc803b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:31:09.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.806+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/3310697034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fcbdc19d100 con 0x7fcbdc1088d0 2024-03-24T10:31:09.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.807+0000 7fcbceffd700 1 -- 172.21.15.23:0/3310697034 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fcbd8010380 con 0x7fcbdc1088d0 2024-03-24T10:31:09.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.809+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/3310697034 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcbc8038f90 msgr2=0x7fcbc803b450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:31:09.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.809+0000 7fcbe3e2f700 1 --2- 172.21.15.23:0/3310697034 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcbc8038f90 0x7fcbc803b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:09.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.809+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/3310697034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbdc1088d0 msgr2=0x7fcbdc19bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:09.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.809+0000 7fcbe3e2f700 1 --2- 172.21.15.23:0/3310697034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbdc1088d0 0x7fcbdc19bbd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8011740 tx=0x7fcbd8011b00 comp rx=0 tx=0).stop 2024-03-24T10:31:09.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.809+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/3310697034 shutdown_connections 2024-03-24T10:31:09.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.809+0000 7fcbe3e2f700 1 --2- 172.21.15.23:0/3310697034 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcbc8038f90 0x7fcbc803b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:09.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.809+0000 7fcbe3e2f700 1 --2- 172.21.15.23:0/3310697034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbdc1088d0 0x7fcbdc19bbd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:09.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.809+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/3310697034 >> 172.21.15.23:0/3310697034 conn(0x7fcbdc102200 msgr2=0x7fcbdc102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:09.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.809+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/3310697034 shutdown_connections 2024-03-24T10:31:09.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:08.809+0000 7fcbe3e2f700 1 -- 172.21.15.23:0/3310697034 wait complete. 2024-03-24T10:31:09.430 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2024-03-24T10:31:09.430 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 9... 2024-03-24T10:31:09.653 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:09 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3310697034' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T10:31:09.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:09 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:09.676+0000 7fb5a8850200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T10:31:09.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:09 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:09.790+0000 7fb5a8850200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T10:31:09.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:09 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:09.882+0000 7fb5a8850200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T10:31:10.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:09 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:09.979+0000 7fb5a8850200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T10:31:10.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:10 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:10.157+0000 7fb5a8850200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T10:31:11.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:11 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:11.144+0000 7fb5a8850200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T10:31:11.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:11 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:11.278+0000 7fb5a8850200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T10:31:11.916 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:11 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:11.620+0000 7fb5a8850200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T10:31:11.916 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:11 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:11.712+0000 7fb5a8850200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T10:31:11.916 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:11 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:11.915+0000 7fb5a8850200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:31:12.168 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:12 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:12.109+0000 7fb5a8850200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T10:31:12.668 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:12 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:12.406+0000 7fb5a8850200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:31:13.062 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:12 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:12.671+0000 7fb5a8850200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T10:31:13.062 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:12 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:12.917+0000 7fb5a8850200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T10:31:14.169 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:13 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:13.756+0000 7fb5a8850200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T10:31:14.668 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:14 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:14.368+0000 7fb5a8850200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T10:31:14.668 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:14 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:14.470+0000 7fb5a8850200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T10:31:15.168 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:31:14 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:31:14.699+0000 7fb5a8850200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T10:31:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:14 smithi023 ceph-mon[32466]: Active manager daemon x restarted 2024-03-24T10:31:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:14 smithi023 ceph-mon[32466]: Activating manager daemon x 2024-03-24T10:31:15.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:14 smithi023 ceph-mon[32466]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T10:31:15.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:14 smithi023 ceph-mon[32466]: mgrmap e10: x(active, starting, since 0.00806512s) 2024-03-24T10:31:15.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:14 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:31:15.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:14 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T10:31:15.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:14 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T10:31:15.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:14 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T10:31:15.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:14 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T10:31:15.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:14 smithi023 ceph-mon[32466]: Manager daemon x is now available 2024-03-24T10:31:15.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:14 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:31:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:31:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T10:31:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T10:31:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:15 smithi023 ceph-mon[32466]: [24/Mar/2024:10:31:14] ENGINE Bus STARTING 2024-03-24T10:31:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:15 smithi023 ceph-mon[32466]: [24/Mar/2024:10:31:15] ENGINE Serving on https://172.21.15.23:7150 2024-03-24T10:31:16.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:15 smithi023 ceph-mon[32466]: [24/Mar/2024:10:31:15] ENGINE Serving on http://172.21.15.23:8765 2024-03-24T10:31:16.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:15 smithi023 ceph-mon[32466]: [24/Mar/2024:10:31:15] ENGINE Bus STARTED 2024-03-24T10:31:16.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:15 smithi023 ceph-mon[32466]: mgrmap e11: x(active, since 1.01104s) 2024-03-24T10:31:16.495 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-24T10:31:16.495 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T10:31:16.495 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T10:31:16.495 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-24T10:31:16.496 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.871+0000 7fcc9c7e1700 1 Processor -- start 2024-03-24T10:31:16.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.871+0000 7fcc9c7e1700 1 -- start start 2024-03-24T10:31:16.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.871+0000 7fcc9c7e1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc94106660 0x7fcc94106a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:16.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.871+0000 7fcc9c7e1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9407bd70 con 0x7fcc94106660 2024-03-24T10:31:16.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.871+0000 7fcc9a57d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc94106660 0x7fcc94106a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:16.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.872+0000 7fcc9a57d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc94106660 0x7fcc94106a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53050/0 (socket says 172.21.15.23:53050) 2024-03-24T10:31:16.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.872+0000 7fcc9a57d700 1 -- 172.21.15.23:0/1776209410 learned_addr learned my addr 172.21.15.23:0/1776209410 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:16.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.872+0000 7fcc9a57d700 1 -- 172.21.15.23:0/1776209410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc94106fc0 con 0x7fcc94106660 2024-03-24T10:31:16.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.872+0000 7fcc9a57d700 1 --2- 172.21.15.23:0/1776209410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc94106660 0x7fcc94106a80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcc840098a0 tx=0x7fcc84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4482e9a04b65b51f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:16.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.872+0000 7fcc9957b700 1 -- 172.21.15.23:0/1776209410 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc84004160 con 0x7fcc94106660 2024-03-24T10:31:16.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.872+0000 7fcc9957b700 1 -- 172.21.15.23:0/1776209410 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc840042c0 con 0x7fcc94106660 2024-03-24T10:31:16.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.872+0000 7fcc9957b700 1 -- 172.21.15.23:0/1776209410 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc84011a80 con 0x7fcc94106660 2024-03-24T10:31:16.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.873+0000 7fcc9c7e1700 1 -- 172.21.15.23:0/1776209410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc94106660 msgr2=0x7fcc94106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:16.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.873+0000 7fcc9c7e1700 1 --2- 172.21.15.23:0/1776209410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc94106660 0x7fcc94106a80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcc840098a0 tx=0x7fcc84009bb0 comp rx=0 tx=0).stop 2024-03-24T10:31:16.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.873+0000 7fcc9c7e1700 1 -- 172.21.15.23:0/1776209410 shutdown_connections 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.873+0000 7fcc9c7e1700 1 --2- 172.21.15.23:0/1776209410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc94106660 0x7fcc94106a80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.873+0000 7fcc9c7e1700 1 -- 172.21.15.23:0/1776209410 >> 172.21.15.23:0/1776209410 conn(0x7fcc94102200 msgr2=0x7fcc94104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.873+0000 7fcc9c7e1700 1 -- 172.21.15.23:0/1776209410 shutdown_connections 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.873+0000 7fcc9c7e1700 1 -- 172.21.15.23:0/1776209410 wait complete. 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.873+0000 7fcc9c7e1700 1 Processor -- start 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.874+0000 7fcc9c7e1700 1 -- start start 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.874+0000 7fcc9c7e1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc94106660 0x7fcc9419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.874+0000 7fcc9c7e1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9419e520 con 0x7fcc94106660 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.874+0000 7fcc9a57d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc94106660 0x7fcc9419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.874+0000 7fcc9a57d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc94106660 0x7fcc9419dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53058/0 (socket says 172.21.15.23:53058) 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.874+0000 7fcc9a57d700 1 -- 172.21.15.23:0/2710304567 learned_addr learned my addr 172.21.15.23:0/2710304567 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.874+0000 7fcc9a57d700 1 -- 172.21.15.23:0/2710304567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc84009550 con 0x7fcc94106660 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.874+0000 7fcc9a57d700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc94106660 0x7fcc9419dfe0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcc84011740 tx=0x7fcc84011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.874+0000 7fcc8b7fe700 1 -- 172.21.15.23:0/2710304567 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc84003b20 con 0x7fcc94106660 2024-03-24T10:31:16.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.874+0000 7fcc8b7fe700 1 -- 172.21.15.23:0/2710304567 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc8401a910 con 0x7fcc94106660 2024-03-24T10:31:16.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.874+0000 7fcc9c7e1700 1 -- 172.21.15.23:0/2710304567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc9419e720 con 0x7fcc94106660 2024-03-24T10:31:16.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.874+0000 7fcc8b7fe700 1 -- 172.21.15.23:0/2710304567 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc84024900 con 0x7fcc94106660 2024-03-24T10:31:16.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.875+0000 7fcc9c7e1700 1 -- 172.21.15.23:0/2710304567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc9419f2d0 con 0x7fcc94106660 2024-03-24T10:31:16.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.875+0000 7fcc8b7fe700 1 -- 172.21.15.23:0/2710304567 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7fcc84024a60 con 0x7fcc94106660 2024-03-24T10:31:16.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.876+0000 7fcc8b7fe700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcc80038f40 0x7fcc8003b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:16.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.876+0000 7fcc8b7fe700 1 -- 172.21.15.23:0/2710304567 --> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcc8003bb10 con 0x7fcc80038f40 2024-03-24T10:31:16.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.876+0000 7fcc8b7fe700 1 -- 172.21.15.23:0/2710304567 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcc8404d3c0 con 0x7fcc94106660 2024-03-24T10:31:16.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.876+0000 7fcc99d7c700 1 -- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcc80038f40 msgr2=0x7fcc8003b400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2159398914 2024-03-24T10:31:16.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:09.876+0000 7fcc99d7c700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcc80038f40 0x7fcc8003b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:31:16.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:10.077+0000 7fcc99d7c700 1 -- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcc80038f40 msgr2=0x7fcc8003b400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2159398914 2024-03-24T10:31:16.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:10.077+0000 7fcc99d7c700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcc80038f40 0x7fcc8003b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:31:16.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:10.478+0000 7fcc99d7c700 1 -- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcc80038f40 msgr2=0x7fcc8003b400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2159398914 2024-03-24T10:31:16.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:10.478+0000 7fcc99d7c700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcc80038f40 0x7fcc8003b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:31:16.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:11.279+0000 7fcc99d7c700 1 -- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcc80038f40 msgr2=0x7fcc8003b400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2159398914 2024-03-24T10:31:16.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:11.279+0000 7fcc99d7c700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcc80038f40 0x7fcc8003b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:31:16.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:12.880+0000 7fcc99d7c700 1 -- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcc80038f40 msgr2=0x7fcc8003b400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2159398914 2024-03-24T10:31:16.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:12.880+0000 7fcc99d7c700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcc80038f40 0x7fcc8003b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T10:31:16.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:14.718+0000 7fcc8b7fe700 1 -- 172.21.15.23:0/2710304567 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 10) ==== 45708+0+0 (secure 0 0 0) 0x7fcc8401aaf0 con 0x7fcc94106660 2024-03-24T10:31:16.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:14.719+0000 7fcc8b7fe700 1 -- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcc80038f40 msgr2=0x7fcc8003b400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:31:16.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:14.719+0000 7fcc8b7fe700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcc80038f40 0x7fcc8003b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:16.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.720+0000 7fcc8b7fe700 1 -- 172.21.15.23:0/2710304567 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 11) ==== 45835+0+0 (secure 0 0 0) 0x7fcc8404efb0 con 0x7fcc94106660 2024-03-24T10:31:16.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.720+0000 7fcc8b7fe700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcc8003c9f0 0x7fcc8003ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:16.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.720+0000 7fcc8b7fe700 1 -- 172.21.15.23:0/2710304567 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcc8003bb10 con 0x7fcc8003c9f0 2024-03-24T10:31:16.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.721+0000 7fcc99d7c700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcc8003c9f0 0x7fcc8003ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:16.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.721+0000 7fcc99d7c700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcc8003c9f0 0x7fcc8003ede0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcc90003a10 tx=0x7fcc90009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:16.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.722+0000 7fcc8b7fe700 1 -- 172.21.15.23:0/2710304567 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fcc8003bb10 con 0x7fcc8003c9f0 2024-03-24T10:31:16.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.728+0000 7fcc9c7e1700 1 -- 172.21.15.23:0/2710304567 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fcc9419ebc0 con 0x7fcc8003c9f0 2024-03-24T10:31:16.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.729+0000 7fcc8b7fe700 1 -- 172.21.15.23:0/2710304567 <== mgr.14152 v2:172.21.15.23:6800/2350021632 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fcc9419ebc0 con 0x7fcc8003c9f0 2024-03-24T10:31:16.502 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.729+0000 7fcc9c7e1700 1 -- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcc8003c9f0 msgr2=0x7fcc8003ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:16.502 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.729+0000 7fcc9c7e1700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcc8003c9f0 0x7fcc8003ede0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcc90003a10 tx=0x7fcc90009270 comp rx=0 tx=0).stop 2024-03-24T10:31:16.502 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.729+0000 7fcc9c7e1700 1 -- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc94106660 msgr2=0x7fcc9419dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:16.502 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.729+0000 7fcc9c7e1700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc94106660 0x7fcc9419dfe0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcc84011740 tx=0x7fcc84011a50 comp rx=0 tx=0).stop 2024-03-24T10:31:16.502 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.729+0000 7fcc9c7e1700 1 -- 172.21.15.23:0/2710304567 shutdown_connections 2024-03-24T10:31:16.502 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.729+0000 7fcc9c7e1700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc94106660 0x7fcc9419dfe0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:16.502 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.729+0000 7fcc9c7e1700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcc8003c9f0 0x7fcc8003ede0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:16.502 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.729+0000 7fcc9c7e1700 1 --2- 172.21.15.23:0/2710304567 >> [v2:172.21.15.23:6800/2159398914,v1:172.21.15.23:6801/2159398914] conn(0x7fcc80038f40 0x7fcc8003b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:16.502 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.729+0000 7fcc9c7e1700 1 -- 172.21.15.23:0/2710304567 >> 172.21.15.23:0/2710304567 conn(0x7fcc94102200 msgr2=0x7fcc94194ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:16.502 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.729+0000 7fcc9c7e1700 1 -- 172.21.15.23:0/2710304567 shutdown_connections 2024-03-24T10:31:16.502 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:15.729+0000 7fcc9c7e1700 1 -- 172.21.15.23:0/2710304567 wait complete. 2024-03-24T10:31:16.502 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 9 is available 2024-03-24T10:31:16.503 INFO:teuthology.orchestra.run.smithi023.stdout:Generating a dashboard self-signed certificate... 2024-03-24T10:31:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:16 smithi023 ceph-mon[32466]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T10:31:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:16 smithi023 ceph-mon[32466]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T10:31:17.757 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T10:31:17.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.924+0000 7f399e8c7700 1 Processor -- start 2024-03-24T10:31:17.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.924+0000 7f399e8c7700 1 -- start start 2024-03-24T10:31:17.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.924+0000 7f399e8c7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3998108870 0x7f399810ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:17.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.924+0000 7f399e8c7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f399807bd70 con 0x7f3998108870 2024-03-24T10:31:17.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.924+0000 7f3997fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3998108870 0x7f399810ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:17.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.924+0000 7f3997fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3998108870 0x7f399810ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53122/0 (socket says 172.21.15.23:53122) 2024-03-24T10:31:17.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.924+0000 7f3997fff700 1 -- 172.21.15.23:0/195041005 learned_addr learned my addr 172.21.15.23:0/195041005 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:17.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.925+0000 7f3997fff700 1 -- 172.21.15.23:0/195041005 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399810b1a0 con 0x7f3998108870 2024-03-24T10:31:17.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.925+0000 7f3997fff700 1 --2- 172.21.15.23:0/195041005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3998108870 0x7f399810ac60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f39880098a0 tx=0x7f3988009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7284de891bb567 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:17.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.925+0000 7f3996ffd700 1 -- 172.21.15.23:0/195041005 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3988004160 con 0x7f3998108870 2024-03-24T10:31:17.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.925+0000 7f3996ffd700 1 -- 172.21.15.23:0/195041005 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39880042c0 con 0x7f3998108870 2024-03-24T10:31:17.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.925+0000 7f3996ffd700 1 -- 172.21.15.23:0/195041005 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3988011a80 con 0x7f3998108870 2024-03-24T10:31:17.758 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.925+0000 7f399e8c7700 1 -- 172.21.15.23:0/195041005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3998108870 msgr2=0x7f399810ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:17.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.925+0000 7f399e8c7700 1 --2- 172.21.15.23:0/195041005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3998108870 0x7f399810ac60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f39880098a0 tx=0x7f3988009bb0 comp rx=0 tx=0).stop 2024-03-24T10:31:17.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.926+0000 7f399e8c7700 1 -- 172.21.15.23:0/195041005 shutdown_connections 2024-03-24T10:31:17.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.926+0000 7f399e8c7700 1 --2- 172.21.15.23:0/195041005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3998108870 0x7f399810ac60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:17.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.926+0000 7f399e8c7700 1 -- 172.21.15.23:0/195041005 >> 172.21.15.23:0/195041005 conn(0x7f3998102200 msgr2=0x7f3998104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:17.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.926+0000 7f399e8c7700 1 -- 172.21.15.23:0/195041005 shutdown_connections 2024-03-24T10:31:17.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.926+0000 7f399e8c7700 1 -- 172.21.15.23:0/195041005 wait complete. 2024-03-24T10:31:17.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.926+0000 7f399e8c7700 1 Processor -- start 2024-03-24T10:31:17.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.926+0000 7f399e8c7700 1 -- start start 2024-03-24T10:31:17.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.926+0000 7f399e8c7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3998108870 0x7f399819bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:17.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.926+0000 7f399e8c7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f399819c0e0 con 0x7f3998108870 2024-03-24T10:31:17.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.927+0000 7f3997fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3998108870 0x7f399819bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:17.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.927+0000 7f3997fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3998108870 0x7f399819bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53126/0 (socket says 172.21.15.23:53126) 2024-03-24T10:31:17.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.927+0000 7f3997fff700 1 -- 172.21.15.23:0/4011960882 learned_addr learned my addr 172.21.15.23:0/4011960882 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:17.759 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.927+0000 7f3997fff700 1 -- 172.21.15.23:0/4011960882 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3988009550 con 0x7f3998108870 2024-03-24T10:31:17.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.927+0000 7f3997fff700 1 --2- 172.21.15.23:0/4011960882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3998108870 0x7f399819bba0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3988011740 tx=0x7f3988011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:17.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.927+0000 7f39957fa700 1 -- 172.21.15.23:0/4011960882 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3988003b20 con 0x7f3998108870 2024-03-24T10:31:17.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.927+0000 7f39957fa700 1 -- 172.21.15.23:0/4011960882 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f398801a910 con 0x7f3998108870 2024-03-24T10:31:17.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.927+0000 7f399e8c7700 1 -- 172.21.15.23:0/4011960882 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f399819c2e0 con 0x7f3998108870 2024-03-24T10:31:17.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.927+0000 7f39957fa700 1 -- 172.21.15.23:0/4011960882 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39880248c0 con 0x7f3998108870 2024-03-24T10:31:17.760 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.927+0000 7f399e8c7700 1 -- 172.21.15.23:0/4011960882 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f399819ce90 con 0x7f3998108870 2024-03-24T10:31:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.928+0000 7f39957fa700 1 -- 172.21.15.23:0/4011960882 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7f398802d3f0 con 0x7f3998108870 2024-03-24T10:31:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.928+0000 7f39957fa700 1 --2- 172.21.15.23:0/4011960882 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3984038f30 0x7f398403b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.928+0000 7f399e8c7700 1 -- 172.21.15.23:0/4011960882 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3998063830 con 0x7f3998108870 2024-03-24T10:31:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.928+0000 7f39957fa700 1 -- 172.21.15.23:0/4011960882 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f398804d770 con 0x7f3998108870 2024-03-24T10:31:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.928+0000 7f39977fe700 1 --2- 172.21.15.23:0/4011960882 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3984038f30 0x7f398403b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.929+0000 7f39977fe700 1 --2- 172.21.15.23:0/4011960882 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3984038f30 0x7f398403b3f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3980006fd0 tx=0x7f3980006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:16.933+0000 7f39957fa700 1 -- 172.21.15.23:0/4011960882 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3988010890 con 0x7f3998108870 2024-03-24T10:31:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:17.152+0000 7f399e8c7700 1 -- 172.21.15.23:0/4011960882 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f39981033d0 con 0x7f3984038f30 2024-03-24T10:31:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:17.251+0000 7f39957fa700 1 -- 172.21.15.23:0/4011960882 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f39981033d0 con 0x7f3984038f30 2024-03-24T10:31:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:17.253+0000 7f399e8c7700 1 -- 172.21.15.23:0/4011960882 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3984038f30 msgr2=0x7f398403b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:17.253+0000 7f399e8c7700 1 --2- 172.21.15.23:0/4011960882 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3984038f30 0x7f398403b3f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3980006fd0 tx=0x7f3980006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:17.253+0000 7f399e8c7700 1 -- 172.21.15.23:0/4011960882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3998108870 msgr2=0x7f399819bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:17.253+0000 7f399e8c7700 1 --2- 172.21.15.23:0/4011960882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3998108870 0x7f399819bba0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3988011740 tx=0x7f3988011b00 comp rx=0 tx=0).stop 2024-03-24T10:31:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:17.254+0000 7f399e8c7700 1 -- 172.21.15.23:0/4011960882 shutdown_connections 2024-03-24T10:31:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:17.254+0000 7f399e8c7700 1 --2- 172.21.15.23:0/4011960882 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3984038f30 0x7f398403b3f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:17.254+0000 7f399e8c7700 1 --2- 172.21.15.23:0/4011960882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3998108870 0x7f399819bba0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:17.254+0000 7f399e8c7700 1 -- 172.21.15.23:0/4011960882 >> 172.21.15.23:0/4011960882 conn(0x7f3998102200 msgr2=0x7f3998102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:17.254+0000 7f399e8c7700 1 -- 172.21.15.23:0/4011960882 shutdown_connections 2024-03-24T10:31:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:17.254+0000 7f399e8c7700 1 -- 172.21.15.23:0/4011960882 wait complete. 2024-03-24T10:31:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial admin user... 2024-03-24T10:31:18.052 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:17 smithi023 ceph-mon[32466]: mgrmap e12: x(active, since 2s) 2024-03-24T10:31:18.052 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:18.052 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:17 smithi023 ceph-mon[32466]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:31:18.052 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:18.052 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:19.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$wJ7nKU5s0.pI2847dgg9BupT5Ls9H1ux6mKHyDXRnmsrtfb/ikSBK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711276278, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T10:31:19.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.155+0000 7f8a9c7b7700 1 Processor -- start 2024-03-24T10:31:19.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.155+0000 7f8a9c7b7700 1 -- start start 2024-03-24T10:31:19.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.156+0000 7f8a9c7b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a94107d60 0x7f8a94108180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:19.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.156+0000 7f8a9c7b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f8a94108750 con 0x7f8a94107d60 2024-03-24T10:31:19.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.156+0000 7f8a9a553700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a94107d60 0x7f8a94108180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:19.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.156+0000 7f8a9a553700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a94107d60 0x7f8a94108180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55892/0 (socket says 172.21.15.23:55892) 2024-03-24T10:31:19.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.156+0000 7f8a9a553700 1 -- 172.21.15.23:0/1470138807 learned_addr learned my addr 172.21.15.23:0/1470138807 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:19.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.156+0000 7f8a9a553700 1 -- 172.21.15.23:0/1470138807 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a94108f60 con 0x7f8a94107d60 2024-03-24T10:31:19.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.156+0000 7f8a9a553700 1 --2- 172.21.15.23:0/1470138807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a94107d60 0x7f8a94108180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8a900098a0 tx=0x7f8a90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e03ec9c8f730f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:19.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.156+0000 7f8a99551700 1 -- 172.21.15.23:0/1470138807 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a90004160 con 0x7f8a94107d60 2024-03-24T10:31:19.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.157+0000 7f8a99551700 1 -- 172.21.15.23:0/1470138807 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a900042c0 con 0x7f8a94107d60 2024-03-24T10:31:19.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.157+0000 7f8a99551700 1 -- 172.21.15.23:0/1470138807 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a90011a80 con 0x7f8a94107d60 2024-03-24T10:31:19.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.157+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/1470138807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a94107d60 msgr2=0x7f8a94108180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:19.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.157+0000 7f8a9c7b7700 1 --2- 172.21.15.23:0/1470138807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a94107d60 0x7f8a94108180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8a900098a0 tx=0x7f8a90009bb0 comp rx=0 tx=0).stop 2024-03-24T10:31:19.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.157+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/1470138807 shutdown_connections 2024-03-24T10:31:19.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.157+0000 7f8a9c7b7700 1 --2- 172.21.15.23:0/1470138807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a94107d60 0x7f8a94108180 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:19.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.157+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/1470138807 >> 172.21.15.23:0/1470138807 conn(0x7f8a94103360 msgr2=0x7f8a94105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:19.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.157+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/1470138807 shutdown_connections 2024-03-24T10:31:19.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.157+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/1470138807 wait complete. 2024-03-24T10:31:19.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.158+0000 7f8a9c7b7700 1 Processor -- start 2024-03-24T10:31:19.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.158+0000 7f8a9c7b7700 1 -- start start 2024-03-24T10:31:19.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.158+0000 7f8a9c7b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a94107d60 0x7f8a9419bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:19.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.158+0000 7f8a9c7b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9419c160 con 0x7f8a94107d60 2024-03-24T10:31:19.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.159+0000 7f8a9a553700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a94107d60 0x7f8a9419bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:19.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.159+0000 7f8a9a553700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a94107d60 0x7f8a9419bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55904/0 (socket says 172.21.15.23:55904) 2024-03-24T10:31:19.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.159+0000 7f8a9a553700 1 -- 172.21.15.23:0/2960528589 learned_addr learned my addr 172.21.15.23:0/2960528589 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:19.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.159+0000 7f8a9a553700 1 -- 172.21.15.23:0/2960528589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a90009550 con 0x7f8a94107d60 2024-03-24T10:31:19.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.159+0000 7f8a9a553700 1 --2- 172.21.15.23:0/2960528589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a94107d60 0x7f8a9419bc20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8a90011740 tx=0x7f8a90011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:19.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.159+0000 7f8a8b7fe700 1 -- 172.21.15.23:0/2960528589 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a900039e0 con 0x7f8a94107d60 2024-03-24T10:31:19.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.159+0000 7f8a8b7fe700 1 -- 172.21.15.23:0/2960528589 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a90003b40 con 0x7f8a94107d60 2024-03-24T10:31:19.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.159+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/2960528589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a9419c360 con 0x7f8a94107d60 2024-03-24T10:31:19.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.159+0000 7f8a8b7fe700 1 -- 172.21.15.23:0/2960528589 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a90024710 con 0x7f8a94107d60 2024-03-24T10:31:19.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.159+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/2960528589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a9419cf10 con 0x7f8a94107d60 2024-03-24T10:31:19.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.160+0000 7f8a8b7fe700 1 -- 172.21.15.23:0/2960528589 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7f8a900249f0 con 0x7f8a94107d60 2024-03-24T10:31:19.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.160+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/2960528589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a94063830 con 0x7f8a94107d60 2024-03-24T10:31:19.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.160+0000 7f8a8b7fe700 1 --2- 172.21.15.23:0/2960528589 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8a80038ee0 0x7f8a8003b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:19.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.160+0000 7f8a8b7fe700 1 -- 172.21.15.23:0/2960528589 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8a9004d310 con 0x7f8a94107d60 2024-03-24T10:31:19.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.160+0000 7f8a99d52700 1 --2- 172.21.15.23:0/2960528589 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8a80038ee0 0x7f8a8003b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:19.175 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.161+0000 7f8a99d52700 1 --2- 172.21.15.23:0/2960528589 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8a80038ee0 0x7f8a8003b3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8a84006fd0 tx=0x7f8a84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:19.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.165+0000 7f8a8b7fe700 1 -- 172.21.15.23:0/2960528589 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8a90022720 con 0x7f8a94107d60 2024-03-24T10:31:19.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.404+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/2960528589 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f8a94063650 con 0x7f8a80038ee0 2024-03-24T10:31:19.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.653+0000 7f8a8b7fe700 1 -- 172.21.15.23:0/2960528589 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f8a94063650 con 0x7f8a80038ee0 2024-03-24T10:31:19.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.655+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/2960528589 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8a80038ee0 msgr2=0x7f8a8003b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:19.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.655+0000 7f8a9c7b7700 1 --2- 172.21.15.23:0/2960528589 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8a80038ee0 0x7f8a8003b3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8a84006fd0 tx=0x7f8a84006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:19.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.655+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/2960528589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a94107d60 msgr2=0x7f8a9419bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:19.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.655+0000 7f8a9c7b7700 1 --2- 172.21.15.23:0/2960528589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a94107d60 0x7f8a9419bc20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8a90011740 tx=0x7f8a90011b00 comp rx=0 tx=0).stop 2024-03-24T10:31:19.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.655+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/2960528589 shutdown_connections 2024-03-24T10:31:19.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.655+0000 7f8a9c7b7700 1 --2- 172.21.15.23:0/2960528589 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8a80038ee0 0x7f8a8003b3a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:19.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.655+0000 7f8a9c7b7700 1 --2- 172.21.15.23:0/2960528589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a94107d60 0x7f8a9419bc20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:19.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.655+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/2960528589 >> 172.21.15.23:0/2960528589 conn(0x7f8a94103360 msgr2=0x7f8a94103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:19.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.655+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/2960528589 shutdown_connections 2024-03-24T10:31:19.176 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:18.655+0000 7f8a9c7b7700 1 -- 172.21.15.23:0/2960528589 wait complete. 2024-03-24T10:31:19.176 INFO:teuthology.orchestra.run.smithi023.stdout:Fetching dashboard port number... 2024-03-24T10:31:19.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:19.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:19 smithi023 ceph-mon[32466]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:31:19.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:20.389 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T10:31:20.389 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.568+0000 7f55de7b8700 1 Processor -- start 2024-03-24T10:31:20.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.568+0000 7f55de7b8700 1 -- start start 2024-03-24T10:31:20.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.568+0000 7f55de7b8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d81060d0 0x7f55d81064f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:20.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.568+0000 7f55de7b8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f55d8106ac0 con 0x7f55d81060d0 2024-03-24T10:31:20.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.568+0000 7f55d7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d81060d0 0x7f55d81064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:20.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.568+0000 7f55d7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d81060d0 0x7f55d81064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55918/0 (socket says 172.21.15.23:55918) 2024-03-24T10:31:20.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.568+0000 7f55d7fff700 1 -- 172.21.15.23:0/2497548065 learned_addr learned my addr 172.21.15.23:0/2497548065 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:20.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.568+0000 7f55d7fff700 1 -- 172.21.15.23:0/2497548065 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d8109070 con 0x7f55d81060d0 2024-03-24T10:31:20.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.568+0000 7f55d7fff700 1 --2- 172.21.15.23:0/2497548065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d81060d0 0x7f55d81064f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f55c80098a0 tx=0x7f55c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d206d26c33b4446 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:20.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.569+0000 7f55d6ffd700 1 -- 172.21.15.23:0/2497548065 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55c8004160 con 0x7f55d81060d0 2024-03-24T10:31:20.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.569+0000 7f55d6ffd700 1 -- 172.21.15.23:0/2497548065 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55c80042c0 con 0x7f55d81060d0 2024-03-24T10:31:20.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.569+0000 7f55d6ffd700 1 -- 172.21.15.23:0/2497548065 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55c8011a80 con 0x7f55d81060d0 2024-03-24T10:31:20.390 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.569+0000 7f55de7b8700 1 -- 172.21.15.23:0/2497548065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d81060d0 msgr2=0x7f55d81064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.569+0000 7f55de7b8700 1 --2- 172.21.15.23:0/2497548065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d81060d0 0x7f55d81064f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f55c80098a0 tx=0x7f55c8009bb0 comp rx=0 tx=0).stop 2024-03-24T10:31:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.569+0000 7f55de7b8700 1 -- 172.21.15.23:0/2497548065 shutdown_connections 2024-03-24T10:31:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.569+0000 7f55de7b8700 1 --2- 172.21.15.23:0/2497548065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d81060d0 0x7f55d81064f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.569+0000 7f55de7b8700 1 -- 172.21.15.23:0/2497548065 >> 172.21.15.23:0/2497548065 conn(0x7f55d81019d0 msgr2=0x7f55d8103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.569+0000 7f55de7b8700 1 -- 172.21.15.23:0/2497548065 shutdown_connections 2024-03-24T10:31:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.569+0000 7f55de7b8700 1 -- 172.21.15.23:0/2497548065 wait complete. 2024-03-24T10:31:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.570+0000 7f55de7b8700 1 Processor -- start 2024-03-24T10:31:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.570+0000 7f55de7b8700 1 -- start start 2024-03-24T10:31:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.570+0000 7f55de7b8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d81060d0 0x7f55d819bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.570+0000 7f55de7b8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f55d819c0c0 con 0x7f55d81060d0 2024-03-24T10:31:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.570+0000 7f55d7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d81060d0 0x7f55d819bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.570+0000 7f55d7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d81060d0 0x7f55d819bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55932/0 (socket says 172.21.15.23:55932) 2024-03-24T10:31:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.570+0000 7f55d7fff700 1 -- 172.21.15.23:0/3856566183 learned_addr learned my addr 172.21.15.23:0/3856566183 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.571+0000 7f55d7fff700 1 -- 172.21.15.23:0/3856566183 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c8009550 con 0x7f55d81060d0 2024-03-24T10:31:20.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.571+0000 7f55d7fff700 1 --2- 172.21.15.23:0/3856566183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d81060d0 0x7f55d819bb80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f55c8011740 tx=0x7f55c8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:20.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.571+0000 7f55d57fa700 1 -- 172.21.15.23:0/3856566183 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55c8003b20 con 0x7f55d81060d0 2024-03-24T10:31:20.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.571+0000 7f55d57fa700 1 -- 172.21.15.23:0/3856566183 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55c801a910 con 0x7f55d81060d0 2024-03-24T10:31:20.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.571+0000 7f55de7b8700 1 -- 172.21.15.23:0/3856566183 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d819c2c0 con 0x7f55d81060d0 2024-03-24T10:31:20.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.571+0000 7f55d57fa700 1 -- 172.21.15.23:0/3856566183 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55c80248c0 con 0x7f55d81060d0 2024-03-24T10:31:20.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.571+0000 7f55de7b8700 1 -- 172.21.15.23:0/3856566183 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d819ce70 con 0x7f55d81060d0 2024-03-24T10:31:20.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.572+0000 7f55d57fa700 1 -- 172.21.15.23:0/3856566183 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7f55c802d3f0 con 0x7f55d81060d0 2024-03-24T10:31:20.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.572+0000 7f55de7b8700 1 -- 172.21.15.23:0/3856566183 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55d8063830 con 0x7f55d81060d0 2024-03-24T10:31:20.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.572+0000 7f55d57fa700 1 --2- 172.21.15.23:0/3856566183 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f55c0038f90 0x7f55c003b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:20.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.572+0000 7f55d57fa700 1 -- 172.21.15.23:0/3856566183 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f55c804d5b0 con 0x7f55d81060d0 2024-03-24T10:31:20.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.572+0000 7f55d77fe700 1 --2- 172.21.15.23:0/3856566183 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f55c0038f90 0x7f55c003b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:20.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.572+0000 7f55d77fe700 1 --2- 172.21.15.23:0/3856566183 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f55c0038f90 0x7f55c003b450 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f55cc006fd0 tx=0x7f55cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:20.392 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.577+0000 7f55d57fa700 1 -- 172.21.15.23:0/3856566183 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f55c8010a70 con 0x7f55d81060d0 2024-03-24T10:31:20.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.785+0000 7f55de7b8700 1 -- 172.21.15.23:0/3856566183 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f55d810c4e0 con 0x7f55d81060d0 2024-03-24T10:31:20.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.785+0000 7f55d57fa700 1 -- 172.21.15.23:0/3856566183 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f55c8010ed0 con 0x7f55d81060d0 2024-03-24T10:31:20.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.788+0000 7f55de7b8700 1 -- 172.21.15.23:0/3856566183 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f55c0038f90 msgr2=0x7f55c003b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:20.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.788+0000 7f55de7b8700 1 --2- 172.21.15.23:0/3856566183 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f55c0038f90 0x7f55c003b450 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f55cc006fd0 tx=0x7f55cc006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:20.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.788+0000 7f55de7b8700 1 -- 172.21.15.23:0/3856566183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d81060d0 msgr2=0x7f55d819bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:20.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.788+0000 7f55de7b8700 1 --2- 172.21.15.23:0/3856566183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d81060d0 0x7f55d819bb80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f55c8011740 tx=0x7f55c8011b00 comp rx=0 tx=0).stop 2024-03-24T10:31:20.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.788+0000 7f55de7b8700 1 -- 172.21.15.23:0/3856566183 shutdown_connections 2024-03-24T10:31:20.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.788+0000 7f55de7b8700 1 --2- 172.21.15.23:0/3856566183 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f55c0038f90 0x7f55c003b450 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:20.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.788+0000 7f55de7b8700 1 --2- 172.21.15.23:0/3856566183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55d81060d0 0x7f55d819bb80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:20.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.788+0000 7f55de7b8700 1 -- 172.21.15.23:0/3856566183 >> 172.21.15.23:0/3856566183 conn(0x7f55d81019d0 msgr2=0x7f55d8102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:20.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.788+0000 7f55de7b8700 1 -- 172.21.15.23:0/3856566183 shutdown_connections 2024-03-24T10:31:20.393 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:19.788+0000 7f55de7b8700 1 -- 172.21.15.23:0/3856566183 wait complete. 2024-03-24T10:31:20.398 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T10:31:20.398 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-24T10:31:20.401 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T10:31:20.401 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-24T10:31:20.401 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-24T10:31:20.402 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T10:31:20.402 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph Dashboard is now available at: 2024-03-24T10:31:20.403 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:31:20.403 INFO:teuthology.orchestra.run.smithi023.stdout: URL: https://smithi023.front.sepia.ceph.com:8443/ 2024-03-24T10:31:20.403 INFO:teuthology.orchestra.run.smithi023.stdout: User: admin 2024-03-24T10:31:20.403 INFO:teuthology.orchestra.run.smithi023.stdout: Password: w8msiwc1ik 2024-03-24T10:31:20.403 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:31:20.403 INFO:teuthology.orchestra.run.smithi023.stdout:Saving cluster configuration to /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config directory 2024-03-24T10:31:20.404 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling autotune for osd_memory_target 2024-03-24T10:31:20.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:20 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3856566183' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T10:31:21.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.182+0000 7fc042bc5700 1 Processor -- start 2024-03-24T10:31:21.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.182+0000 7fc042bc5700 1 -- start start 2024-03-24T10:31:21.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.182+0000 7fc042bc5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03c107d60 0x7fc03c108180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:21.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.182+0000 7fc042bc5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c108750 con 0x7fc03c107d60 2024-03-24T10:31:21.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.182+0000 7fc040961700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03c107d60 0x7fc03c108180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:21.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.182+0000 7fc040961700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03c107d60 0x7fc03c108180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55934/0 (socket says 172.21.15.23:55934) 2024-03-24T10:31:21.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.182+0000 7fc040961700 1 -- 172.21.15.23:0/1719993854 learned_addr learned my addr 172.21.15.23:0/1719993854 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:21.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.182+0000 7fc040961700 1 -- 172.21.15.23:0/1719993854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03c108f60 con 0x7fc03c107d60 2024-03-24T10:31:21.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.183+0000 7fc040961700 1 --2- 172.21.15.23:0/1719993854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03c107d60 0x7fc03c108180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc0300098a0 tx=0x7fc030009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b8d6edcd69e8764 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:21.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.183+0000 7fc03b7fe700 1 -- 172.21.15.23:0/1719993854 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc030004160 con 0x7fc03c107d60 2024-03-24T10:31:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.183+0000 7fc03b7fe700 1 -- 172.21.15.23:0/1719993854 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0300042c0 con 0x7fc03c107d60 2024-03-24T10:31:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.183+0000 7fc03b7fe700 1 -- 172.21.15.23:0/1719993854 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc030011a80 con 0x7fc03c107d60 2024-03-24T10:31:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.183+0000 7fc042bc5700 1 -- 172.21.15.23:0/1719993854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03c107d60 msgr2=0x7fc03c108180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.183+0000 7fc042bc5700 1 --2- 172.21.15.23:0/1719993854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03c107d60 0x7fc03c108180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc0300098a0 tx=0x7fc030009bb0 comp rx=0 tx=0).stop 2024-03-24T10:31:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.183+0000 7fc042bc5700 1 -- 172.21.15.23:0/1719993854 shutdown_connections 2024-03-24T10:31:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.183+0000 7fc042bc5700 1 --2- 172.21.15.23:0/1719993854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03c107d60 0x7fc03c108180 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.183+0000 7fc042bc5700 1 -- 172.21.15.23:0/1719993854 >> 172.21.15.23:0/1719993854 conn(0x7fc03c103360 msgr2=0x7fc03c105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.183+0000 7fc042bc5700 1 -- 172.21.15.23:0/1719993854 shutdown_connections 2024-03-24T10:31:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.183+0000 7fc042bc5700 1 -- 172.21.15.23:0/1719993854 wait complete. 2024-03-24T10:31:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.184+0000 7fc042bc5700 1 Processor -- start 2024-03-24T10:31:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.184+0000 7fc042bc5700 1 -- start start 2024-03-24T10:31:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.184+0000 7fc042bc5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03c107d60 0x7fc03c19bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.184+0000 7fc042bc5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c19c130 con 0x7fc03c107d60 2024-03-24T10:31:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.184+0000 7fc040961700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03c107d60 0x7fc03c19bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:21.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.184+0000 7fc040961700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03c107d60 0x7fc03c19bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55946/0 (socket says 172.21.15.23:55946) 2024-03-24T10:31:21.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.184+0000 7fc040961700 1 -- 172.21.15.23:0/1388718477 learned_addr learned my addr 172.21.15.23:0/1388718477 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:21.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.185+0000 7fc040961700 1 -- 172.21.15.23:0/1388718477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc030009550 con 0x7fc03c107d60 2024-03-24T10:31:21.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.185+0000 7fc040961700 1 --2- 172.21.15.23:0/1388718477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03c107d60 0x7fc03c19bbf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc030011740 tx=0x7fc030011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:21.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.185+0000 7fc039ffb700 1 -- 172.21.15.23:0/1388718477 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0300039e0 con 0x7fc03c107d60 2024-03-24T10:31:21.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.185+0000 7fc039ffb700 1 -- 172.21.15.23:0/1388718477 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc030003b40 con 0x7fc03c107d60 2024-03-24T10:31:21.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.185+0000 7fc042bc5700 1 -- 172.21.15.23:0/1388718477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc03c19c330 con 0x7fc03c107d60 2024-03-24T10:31:21.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.185+0000 7fc039ffb700 1 -- 172.21.15.23:0/1388718477 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc030024710 con 0x7fc03c107d60 2024-03-24T10:31:21.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.185+0000 7fc042bc5700 1 -- 172.21.15.23:0/1388718477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc03c19cee0 con 0x7fc03c107d60 2024-03-24T10:31:21.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.186+0000 7fc039ffb700 1 -- 172.21.15.23:0/1388718477 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7fc030024980 con 0x7fc03c107d60 2024-03-24T10:31:21.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.186+0000 7fc042bc5700 1 -- 172.21.15.23:0/1388718477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc03c063830 con 0x7fc03c107d60 2024-03-24T10:31:21.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.186+0000 7fc039ffb700 1 --2- 172.21.15.23:0/1388718477 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc02c038f40 0x7fc02c03b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:21.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.186+0000 7fc039ffb700 1 -- 172.21.15.23:0/1388718477 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc03004d520 con 0x7fc03c107d60 2024-03-24T10:31:21.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.186+0000 7fc03bfff700 1 --2- 172.21.15.23:0/1388718477 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc02c038f40 0x7fc02c03b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:21.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.187+0000 7fc03bfff700 1 --2- 172.21.15.23:0/1388718477 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc02c038f40 0x7fc02c03b400 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc028006fd0 tx=0x7fc028006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:21.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.191+0000 7fc039ffb700 1 -- 172.21.15.23:0/1388718477 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc030022310 con 0x7fc03c107d60 2024-03-24T10:31:21.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.231+0000 7fc039ffb700 1 -- 172.21.15.23:0/1388718477 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fc03002da00 con 0x7fc03c107d60 2024-03-24T10:31:21.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.397+0000 7fc042bc5700 1 -- 172.21.15.23:0/1388718477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fc03c06a7d0 con 0x7fc03c107d60 2024-03-24T10:31:21.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.397+0000 7fc039ffb700 1 -- 172.21.15.23:0/1388718477 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fc030022310 con 0x7fc03c107d60 2024-03-24T10:31:21.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.399+0000 7fc042bc5700 1 -- 172.21.15.23:0/1388718477 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc02c038f40 msgr2=0x7fc02c03b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:21.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.399+0000 7fc042bc5700 1 --2- 172.21.15.23:0/1388718477 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc02c038f40 0x7fc02c03b400 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc028006fd0 tx=0x7fc028006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:21.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.399+0000 7fc042bc5700 1 -- 172.21.15.23:0/1388718477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03c107d60 msgr2=0x7fc03c19bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:21.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.399+0000 7fc042bc5700 1 --2- 172.21.15.23:0/1388718477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03c107d60 0x7fc03c19bbf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc030011740 tx=0x7fc030011b00 comp rx=0 tx=0).stop 2024-03-24T10:31:21.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.399+0000 7fc042bc5700 1 -- 172.21.15.23:0/1388718477 shutdown_connections 2024-03-24T10:31:21.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.399+0000 7fc042bc5700 1 --2- 172.21.15.23:0/1388718477 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc02c038f40 0x7fc02c03b400 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:21.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.399+0000 7fc042bc5700 1 --2- 172.21.15.23:0/1388718477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03c107d60 0x7fc03c19bbf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:21.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.399+0000 7fc042bc5700 1 -- 172.21.15.23:0/1388718477 >> 172.21.15.23:0/1388718477 conn(0x7fc03c103360 msgr2=0x7fc03c103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:21.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.399+0000 7fc042bc5700 1 -- 172.21.15.23:0/1388718477 shutdown_connections 2024-03-24T10:31:21.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:21.399+0000 7fc042bc5700 1 -- 172.21.15.23:0/1388718477 wait complete. 2024-03-24T10:31:22.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:22 smithi023 ceph-mon[32466]: mgrmap e13: x(active, since 6s) 2024-03-24T10:31:23.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.363+0000 7f945286e700 1 Processor -- start 2024-03-24T10:31:23.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.363+0000 7f945286e700 1 -- start start 2024-03-24T10:31:23.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.363+0000 7f945286e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f944c106ca0 0x7f944c1070c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:23.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.363+0000 7f945286e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f944c107690 con 0x7f944c106ca0 2024-03-24T10:31:23.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.364+0000 7f944bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f944c106ca0 0x7f944c1070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:23.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.364+0000 7f944bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f944c106ca0 0x7f944c1070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55948/0 (socket says 172.21.15.23:55948) 2024-03-24T10:31:23.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.364+0000 7f944bfff700 1 -- 172.21.15.23:0/2321963300 learned_addr learned my addr 172.21.15.23:0/2321963300 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:23.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.364+0000 7f944bfff700 1 -- 172.21.15.23:0/2321963300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f944c107ea0 con 0x7f944c106ca0 2024-03-24T10:31:23.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.364+0000 7f944bfff700 1 --2- 172.21.15.23:0/2321963300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f944c106ca0 0x7f944c1070c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f943c0098a0 tx=0x7f943c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45fc0d49b5a9c221 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:23.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.364+0000 7f944affd700 1 -- 172.21.15.23:0/2321963300 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f943c004160 con 0x7f944c106ca0 2024-03-24T10:31:23.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.364+0000 7f944affd700 1 -- 172.21.15.23:0/2321963300 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f943c0042c0 con 0x7f944c106ca0 2024-03-24T10:31:23.224 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.364+0000 7f944affd700 1 -- 172.21.15.23:0/2321963300 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f943c011a80 con 0x7f944c106ca0 2024-03-24T10:31:23.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.365+0000 7f945286e700 1 -- 172.21.15.23:0/2321963300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f944c106ca0 msgr2=0x7f944c1070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:23.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.365+0000 7f945286e700 1 --2- 172.21.15.23:0/2321963300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f944c106ca0 0x7f944c1070c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f943c0098a0 tx=0x7f943c009bb0 comp rx=0 tx=0).stop 2024-03-24T10:31:23.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.365+0000 7f945286e700 1 -- 172.21.15.23:0/2321963300 shutdown_connections 2024-03-24T10:31:23.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.365+0000 7f945286e700 1 --2- 172.21.15.23:0/2321963300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f944c106ca0 0x7f944c1070c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:23.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.365+0000 7f945286e700 1 -- 172.21.15.23:0/2321963300 >> 172.21.15.23:0/2321963300 conn(0x7f944c102220 msgr2=0x7f944c104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:23.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.365+0000 7f945286e700 1 -- 172.21.15.23:0/2321963300 shutdown_connections 2024-03-24T10:31:23.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.365+0000 7f945286e700 1 -- 172.21.15.23:0/2321963300 wait complete. 2024-03-24T10:31:23.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.366+0000 7f945286e700 1 Processor -- start 2024-03-24T10:31:23.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.366+0000 7f945286e700 1 -- start start 2024-03-24T10:31:23.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.366+0000 7f945286e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f944c106ca0 0x7f944c19bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:23.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.366+0000 7f945286e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f944c19c190 con 0x7f944c106ca0 2024-03-24T10:31:23.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.366+0000 7f944bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f944c106ca0 0x7f944c19bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:23.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.366+0000 7f944bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f944c106ca0 0x7f944c19bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55954/0 (socket says 172.21.15.23:55954) 2024-03-24T10:31:23.225 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.366+0000 7f944bfff700 1 -- 172.21.15.23:0/2431607213 learned_addr learned my addr 172.21.15.23:0/2431607213 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:23.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.366+0000 7f944bfff700 1 -- 172.21.15.23:0/2431607213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943c009550 con 0x7f944c106ca0 2024-03-24T10:31:23.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.366+0000 7f944bfff700 1 --2- 172.21.15.23:0/2431607213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f944c106ca0 0x7f944c19bc50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f943c011740 tx=0x7f943c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:23.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.367+0000 7f94497fa700 1 -- 172.21.15.23:0/2431607213 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f943c0039e0 con 0x7f944c106ca0 2024-03-24T10:31:23.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.367+0000 7f94497fa700 1 -- 172.21.15.23:0/2431607213 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f943c003b40 con 0x7f944c106ca0 2024-03-24T10:31:23.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.367+0000 7f945286e700 1 -- 172.21.15.23:0/2431607213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f944c19c390 con 0x7f944c106ca0 2024-03-24T10:31:23.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.367+0000 7f945286e700 1 -- 172.21.15.23:0/2431607213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f944c19cf40 con 0x7f944c106ca0 2024-03-24T10:31:23.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.367+0000 7f94497fa700 1 -- 172.21.15.23:0/2431607213 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f943c024740 con 0x7f944c106ca0 2024-03-24T10:31:23.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.367+0000 7f94497fa700 1 -- 172.21.15.23:0/2431607213 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f943c02d3f0 con 0x7f944c106ca0 2024-03-24T10:31:23.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.368+0000 7f945286e700 1 -- 172.21.15.23:0/2431607213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9438005360 con 0x7f944c106ca0 2024-03-24T10:31:23.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.368+0000 7f94497fa700 1 --2- 172.21.15.23:0/2431607213 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9434038fd0 0x7f943403b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:23.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.368+0000 7f94497fa700 1 -- 172.21.15.23:0/2431607213 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f943c04d3f0 con 0x7f944c106ca0 2024-03-24T10:31:23.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.368+0000 7f944b7fe700 1 --2- 172.21.15.23:0/2431607213 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9434038fd0 0x7f943403b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:23.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.368+0000 7f944b7fe700 1 --2- 172.21.15.23:0/2431607213 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9434038fd0 0x7f943403b490 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9440006fd0 tx=0x7f9440006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:23.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.373+0000 7f94497fa700 1 -- 172.21.15.23:0/2431607213 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f943c010740 con 0x7f944c106ca0 2024-03-24T10:31:23.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.687+0000 7f945286e700 1 -- 172.21.15.23:0/2431607213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f9438005610 con 0x7f944c106ca0 2024-03-24T10:31:23.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.690+0000 7f94497fa700 1 -- 172.21.15.23:0/2431607213 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f943c04cc80 con 0x7f944c106ca0 2024-03-24T10:31:23.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.692+0000 7f945286e700 1 -- 172.21.15.23:0/2431607213 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9434038fd0 msgr2=0x7f943403b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:23.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.692+0000 7f945286e700 1 --2- 172.21.15.23:0/2431607213 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9434038fd0 0x7f943403b490 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9440006fd0 tx=0x7f9440006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:23.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.692+0000 7f945286e700 1 -- 172.21.15.23:0/2431607213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f944c106ca0 msgr2=0x7f944c19bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:23.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.692+0000 7f945286e700 1 --2- 172.21.15.23:0/2431607213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f944c106ca0 0x7f944c19bc50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f943c011740 tx=0x7f943c011b00 comp rx=0 tx=0).stop 2024-03-24T10:31:23.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.692+0000 7f945286e700 1 -- 172.21.15.23:0/2431607213 shutdown_connections 2024-03-24T10:31:23.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.692+0000 7f945286e700 1 --2- 172.21.15.23:0/2431607213 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9434038fd0 0x7f943403b490 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:23.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.692+0000 7f945286e700 1 --2- 172.21.15.23:0/2431607213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f944c106ca0 0x7f944c19bc50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:23.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.692+0000 7f945286e700 1 -- 172.21.15.23:0/2431607213 >> 172.21.15.23:0/2431607213 conn(0x7f944c102220 msgr2=0x7f944c102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:23.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.692+0000 7f945286e700 1 -- 172.21.15.23:0/2431607213 shutdown_connections 2024-03-24T10:31:23.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-24T10:31:22.692+0000 7f945286e700 1 -- 172.21.15.23:0/2431607213 wait complete. 2024-03-24T10:31:23.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout: ceph telemetry on 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout:For more information see: 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T10:31:23.228 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:31:23.229 INFO:teuthology.orchestra.run.smithi023.stdout:Bootstrap complete. 2024-03-24T10:31:23.262 INFO:tasks.cephadm:Fetching config... 2024-03-24T10:31:23.263 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:31:23.263 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T10:31:23.278 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T10:31:23.278 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:31:23.278 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T10:31:23.333 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T10:31:23.333 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:31:23.333 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-24T10:31:23.400 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T10:31:23.400 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:31:23.400 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T10:31:23.456 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T10:31:23.457 DEBUG:teuthology.orchestra.run.smithi023:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDACbbrvvqVbgW9KHxE7s+Ldi+yrxDR0DmENZu5LD716D1EtqC/w0vLqJgP8x+zGnPpcPCuXVeDT+hu7/KVTMdraTL4dr8hYYkyLFVGXZv2kwbVaumndSyaDM2IaD5SE+hNqG0/Bh1JNIxaQbcS8fIVLMUtb6oRyzbqvpK+acYr1huj2jdtGFAN7CJ+6nQxKozQMVnbC4iDpJDrFPyhV/Jv+6dl0RADnCLcjSl/rRF8ZP9crGgO1ZUoMWeKuUbwcNAuAbnJQGq29sDMLSbROD7beBPxU0lznNiYfB1Y75eMqLD5/e5OY0WHKFoezjFfHdXj+xCC+d8HA1PBuxaL75PnZHpojN0UKnPb7LDQQkwB7VsYPV6D2IDehW84RzfWWSnKugE3RFL3eC52pqrXqpVwwtEHhyspTpLSZcDz3vsfW6zh034NArP8ZApohu28xvfGSelKcaYzWIoCpShjQw3ijk/FjOQyCJdowCwGI5GBIcigCGzWP/1ZwTq7VrqprL0= ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T10:31:23.537 INFO:teuthology.orchestra.run.smithi023.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDACbbrvvqVbgW9KHxE7s+Ldi+yrxDR0DmENZu5LD716D1EtqC/w0vLqJgP8x+zGnPpcPCuXVeDT+hu7/KVTMdraTL4dr8hYYkyLFVGXZv2kwbVaumndSyaDM2IaD5SE+hNqG0/Bh1JNIxaQbcS8fIVLMUtb6oRyzbqvpK+acYr1huj2jdtGFAN7CJ+6nQxKozQMVnbC4iDpJDrFPyhV/Jv+6dl0RADnCLcjSl/rRF8ZP9crGgO1ZUoMWeKuUbwcNAuAbnJQGq29sDMLSbROD7beBPxU0lznNiYfB1Y75eMqLD5/e5OY0WHKFoezjFfHdXj+xCC+d8HA1PBuxaL75PnZHpojN0UKnPb7LDQQkwB7VsYPV6D2IDehW84RzfWWSnKugE3RFL3eC52pqrXqpVwwtEHhyspTpLSZcDz3vsfW6zh034NArP8ZApohu28xvfGSelKcaYzWIoCpShjQw3ijk/FjOQyCJdowCwGI5GBIcigCGzWP/1ZwTq7VrqprL0= ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:31:23.551 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDACbbrvvqVbgW9KHxE7s+Ldi+yrxDR0DmENZu5LD716D1EtqC/w0vLqJgP8x+zGnPpcPCuXVeDT+hu7/KVTMdraTL4dr8hYYkyLFVGXZv2kwbVaumndSyaDM2IaD5SE+hNqG0/Bh1JNIxaQbcS8fIVLMUtb6oRyzbqvpK+acYr1huj2jdtGFAN7CJ+6nQxKozQMVnbC4iDpJDrFPyhV/Jv+6dl0RADnCLcjSl/rRF8ZP9crGgO1ZUoMWeKuUbwcNAuAbnJQGq29sDMLSbROD7beBPxU0lznNiYfB1Y75eMqLD5/e5OY0WHKFoezjFfHdXj+xCC+d8HA1PBuxaL75PnZHpojN0UKnPb7LDQQkwB7VsYPV6D2IDehW84RzfWWSnKugE3RFL3eC52pqrXqpVwwtEHhyspTpLSZcDz3vsfW6zh034NArP8ZApohu28xvfGSelKcaYzWIoCpShjQw3ijk/FjOQyCJdowCwGI5GBIcigCGzWP/1ZwTq7VrqprL0= ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T10:31:23.597 INFO:teuthology.orchestra.run.smithi116.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDACbbrvvqVbgW9KHxE7s+Ldi+yrxDR0DmENZu5LD716D1EtqC/w0vLqJgP8x+zGnPpcPCuXVeDT+hu7/KVTMdraTL4dr8hYYkyLFVGXZv2kwbVaumndSyaDM2IaD5SE+hNqG0/Bh1JNIxaQbcS8fIVLMUtb6oRyzbqvpK+acYr1huj2jdtGFAN7CJ+6nQxKozQMVnbC4iDpJDrFPyhV/Jv+6dl0RADnCLcjSl/rRF8ZP9crGgO1ZUoMWeKuUbwcNAuAbnJQGq29sDMLSbROD7beBPxU0lznNiYfB1Y75eMqLD5/e5OY0WHKFoezjFfHdXj+xCC+d8HA1PBuxaL75PnZHpojN0UKnPb7LDQQkwB7VsYPV6D2IDehW84RzfWWSnKugE3RFL3eC52pqrXqpVwwtEHhyspTpLSZcDz3vsfW6zh034NArP8ZApohu28xvfGSelKcaYzWIoCpShjQw3ijk/FjOQyCJdowCwGI5GBIcigCGzWP/1ZwTq7VrqprL0= ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:31:23.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDACbbrvvqVbgW9KHxE7s+Ldi+yrxDR0DmENZu5LD716D1EtqC/w0vLqJgP8x+zGnPpcPCuXVeDT+hu7/KVTMdraTL4dr8hYYkyLFVGXZv2kwbVaumndSyaDM2IaD5SE+hNqG0/Bh1JNIxaQbcS8fIVLMUtb6oRyzbqvpK+acYr1huj2jdtGFAN7CJ+6nQxKozQMVnbC4iDpJDrFPyhV/Jv+6dl0RADnCLcjSl/rRF8ZP9crGgO1ZUoMWeKuUbwcNAuAbnJQGq29sDMLSbROD7beBPxU0lznNiYfB1Y75eMqLD5/e5OY0WHKFoezjFfHdXj+xCC+d8HA1PBuxaL75PnZHpojN0UKnPb7LDQQkwB7VsYPV6D2IDehW84RzfWWSnKugE3RFL3eC52pqrXqpVwwtEHhyspTpLSZcDz3vsfW6zh034NArP8ZApohu28xvfGSelKcaYzWIoCpShjQw3ijk/FjOQyCJdowCwGI5GBIcigCGzWP/1ZwTq7VrqprL0= ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T10:31:23.658 INFO:teuthology.orchestra.run.smithi117.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDACbbrvvqVbgW9KHxE7s+Ldi+yrxDR0DmENZu5LD716D1EtqC/w0vLqJgP8x+zGnPpcPCuXVeDT+hu7/KVTMdraTL4dr8hYYkyLFVGXZv2kwbVaumndSyaDM2IaD5SE+hNqG0/Bh1JNIxaQbcS8fIVLMUtb6oRyzbqvpK+acYr1huj2jdtGFAN7CJ+6nQxKozQMVnbC4iDpJDrFPyhV/Jv+6dl0RADnCLcjSl/rRF8ZP9crGgO1ZUoMWeKuUbwcNAuAbnJQGq29sDMLSbROD7beBPxU0lznNiYfB1Y75eMqLD5/e5OY0WHKFoezjFfHdXj+xCC+d8HA1PBuxaL75PnZHpojN0UKnPb7LDQQkwB7VsYPV6D2IDehW84RzfWWSnKugE3RFL3eC52pqrXqpVwwtEHhyspTpLSZcDz3vsfW6zh034NArP8ZApohu28xvfGSelKcaYzWIoCpShjQw3ijk/FjOQyCJdowCwGI5GBIcigCGzWP/1ZwTq7VrqprL0= ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:31:23.673 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T10:31:23.912 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:31:23.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:23 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2431607213' entity='client.admin' 2024-03-24T10:31:25.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.988+0000 7f9dfdda1700 1 -- 172.21.15.23:0/4057670101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df8102520 msgr2=0x7f9df8104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.988+0000 7f9dfdda1700 1 --2- 172.21.15.23:0/4057670101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df8102520 0x7f9df8104910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9dec009910 tx=0x7f9dec009c20 comp rx=0 tx=0).stop 2024-03-24T10:31:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.988+0000 7f9dfdda1700 1 -- 172.21.15.23:0/4057670101 shutdown_connections 2024-03-24T10:31:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.988+0000 7f9dfdda1700 1 --2- 172.21.15.23:0/4057670101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df8102520 0x7f9df8104910 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.988+0000 7f9dfdda1700 1 -- 172.21.15.23:0/4057670101 >> 172.21.15.23:0/4057670101 conn(0x7f9df80fbf10 msgr2=0x7f9df80fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.988+0000 7f9dfdda1700 1 -- 172.21.15.23:0/4057670101 shutdown_connections 2024-03-24T10:31:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.988+0000 7f9dfdda1700 1 -- 172.21.15.23:0/4057670101 wait complete. 2024-03-24T10:31:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.988+0000 7f9dfdda1700 1 Processor -- start 2024-03-24T10:31:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.988+0000 7f9dfdda1700 1 -- start start 2024-03-24T10:31:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.988+0000 7f9dfdda1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df8102520 0x7f9df8101da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.988+0000 7f9dfdda1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f9df81022e0 con 0x7f9df8102520 2024-03-24T10:31:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.989+0000 7f9dfcd9f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df8102520 0x7f9df8101da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.989+0000 7f9dfcd9f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df8102520 0x7f9df8101da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55968/0 (socket says 172.21.15.23:55968) 2024-03-24T10:31:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.989+0000 7f9dfcd9f700 1 -- 172.21.15.23:0/1488588829 learned_addr learned my addr 172.21.15.23:0/1488588829 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.989+0000 7f9dfcd9f700 1 -- 172.21.15.23:0/1488588829 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dec0095f0 con 0x7f9df8102520 2024-03-24T10:31:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.989+0000 7f9dfcd9f700 1 --2- 172.21.15.23:0/1488588829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df8102520 0x7f9df8101da0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9dec005b40 tx=0x7f9dec004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.989+0000 7f9df5ffb700 1 -- 172.21.15.23:0/1488588829 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dec01d070 con 0x7f9df8102520 2024-03-24T10:31:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.989+0000 7f9df5ffb700 1 -- 172.21.15.23:0/1488588829 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dec00be40 con 0x7f9df8102520 2024-03-24T10:31:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.989+0000 7f9dfdda1700 1 -- 172.21.15.23:0/1488588829 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df81004d0 con 0x7f9df8102520 2024-03-24T10:31:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.989+0000 7f9df5ffb700 1 -- 172.21.15.23:0/1488588829 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dec022dc0 con 0x7f9df8102520 2024-03-24T10:31:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.989+0000 7f9dfdda1700 1 -- 172.21.15.23:0/1488588829 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df8100970 con 0x7f9df8102520 2024-03-24T10:31:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.990+0000 7f9df5ffb700 1 -- 172.21.15.23:0/1488588829 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f9dec00f460 con 0x7f9df8102520 2024-03-24T10:31:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.990+0000 7f9df5ffb700 1 --2- 172.21.15.23:0/1488588829 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9de8038fe0 0x7f9de803b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.990+0000 7f9df5ffb700 1 -- 172.21.15.23:0/1488588829 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9dec04d230 con 0x7f9df8102520 2024-03-24T10:31:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.990+0000 7f9dfdda1700 1 -- 172.21.15.23:0/1488588829 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df8108220 con 0x7f9df8102520 2024-03-24T10:31:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.990+0000 7f9df7fff700 1 --2- 172.21.15.23:0/1488588829 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9de8038fe0 0x7f9de803b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:25.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.991+0000 7f9df7fff700 1 --2- 172.21.15.23:0/1488588829 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9de8038fe0 0x7f9de803b4a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9de4006fd0 tx=0x7f9de4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:25.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:25.995+0000 7f9df5ffb700 1 -- 172.21.15.23:0/1488588829 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9dec027020 con 0x7f9df8102520 2024-03-24T10:31:26.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:26.196+0000 7f9dfdda1700 1 -- 172.21.15.23:0/1488588829 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f9df8107c00 con 0x7f9df8102520 2024-03-24T10:31:26.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:26.199+0000 7f9df5ffb700 1 -- 172.21.15.23:0/1488588829 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f9df8107c00 con 0x7f9df8102520 2024-03-24T10:31:26.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:26.201+0000 7f9dfdda1700 1 -- 172.21.15.23:0/1488588829 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9de8038fe0 msgr2=0x7f9de803b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:26.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:26.201+0000 7f9dfdda1700 1 --2- 172.21.15.23:0/1488588829 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9de8038fe0 0x7f9de803b4a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9de4006fd0 tx=0x7f9de4006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:26.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:26.201+0000 7f9dfdda1700 1 -- 172.21.15.23:0/1488588829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df8102520 msgr2=0x7f9df8101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:26.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:26.201+0000 7f9dfdda1700 1 --2- 172.21.15.23:0/1488588829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df8102520 0x7f9df8101da0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9dec005b40 tx=0x7f9dec004ff0 comp rx=0 tx=0).stop 2024-03-24T10:31:26.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:26.201+0000 7f9dfdda1700 1 -- 172.21.15.23:0/1488588829 shutdown_connections 2024-03-24T10:31:26.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:26.201+0000 7f9dfdda1700 1 --2- 172.21.15.23:0/1488588829 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9de8038fe0 0x7f9de803b4a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:26.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:26.201+0000 7f9dfdda1700 1 --2- 172.21.15.23:0/1488588829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df8102520 0x7f9df8101da0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:26.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:26.201+0000 7f9dfdda1700 1 -- 172.21.15.23:0/1488588829 >> 172.21.15.23:0/1488588829 conn(0x7f9df80fbf10 msgr2=0x7f9df80fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:26.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:26.201+0000 7f9dfdda1700 1 -- 172.21.15.23:0/1488588829 shutdown_connections 2024-03-24T10:31:26.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:26.201+0000 7f9dfdda1700 1 -- 172.21.15.23:0/1488588829 wait complete. 2024-03-24T10:31:26.494 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:26 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:26.494 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:26 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:26.494 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:26 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi023", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:31:26.494 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:26 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:26.494 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:26 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:31:26.494 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:26 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:26.790 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T10:31:26.790 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T10:31:27.029 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:31:27.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:27 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1488588829' entity='client.admin' 2024-03-24T10:31:27.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:27 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:31:27.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:27 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:28.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.418+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/41431112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cb81002d0 msgr2=0x7f5cb81006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:28.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.418+0000 7f5cbd5aa700 1 --2- 172.21.15.23:0/41431112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cb81002d0 0x7f5cb81006f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5ca800aa20 tx=0x7f5ca800ad30 comp rx=0 tx=0).stop 2024-03-24T10:31:28.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.418+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/41431112 shutdown_connections 2024-03-24T10:31:28.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.418+0000 7f5cbd5aa700 1 --2- 172.21.15.23:0/41431112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cb81002d0 0x7f5cb81006f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:28.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.418+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/41431112 >> 172.21.15.23:0/41431112 conn(0x7f5cb80fb890 msgr2=0x7f5cb80fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:28.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.418+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/41431112 shutdown_connections 2024-03-24T10:31:28.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.418+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/41431112 wait complete. 2024-03-24T10:31:28.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.419+0000 7f5cbd5aa700 1 Processor -- start 2024-03-24T10:31:28.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.419+0000 7f5cbd5aa700 1 -- start start 2024-03-24T10:31:28.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.419+0000 7f5cbd5aa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cb81002d0 0x7f5cb81971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:28.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.419+0000 7f5cbd5aa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb8197720 con 0x7f5cb81002d0 2024-03-24T10:31:28.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.419+0000 7f5cb6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cb81002d0 0x7f5cb81971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:28.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.419+0000 7f5cb6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cb81002d0 0x7f5cb81971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43294/0 (socket says 172.21.15.23:43294) 2024-03-24T10:31:28.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.419+0000 7f5cb6ffd700 1 -- 172.21.15.23:0/4147243203 learned_addr learned my addr 172.21.15.23:0/4147243203 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:28.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.420+0000 7f5cb6ffd700 1 -- 172.21.15.23:0/4147243203 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ca800a700 con 0x7f5cb81002d0 2024-03-24T10:31:28.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.420+0000 7f5cb6ffd700 1 --2- 172.21.15.23:0/4147243203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cb81002d0 0x7f5cb81971e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8005e20 tx=0x7f5ca8004f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:28.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.420+0000 7f5caffff700 1 -- 172.21.15.23:0/4147243203 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ca801d070 con 0x7f5cb81002d0 2024-03-24T10:31:28.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.420+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/4147243203 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cb8197920 con 0x7f5cb81002d0 2024-03-24T10:31:28.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.420+0000 7f5caffff700 1 -- 172.21.15.23:0/4147243203 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ca8005430 con 0x7f5cb81002d0 2024-03-24T10:31:28.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.420+0000 7f5caffff700 1 -- 172.21.15.23:0/4147243203 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ca800e630 con 0x7f5cb81002d0 2024-03-24T10:31:28.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.420+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/4147243203 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cb8197dc0 con 0x7f5cb81002d0 2024-03-24T10:31:28.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.421+0000 7f5caffff700 1 -- 172.21.15.23:0/4147243203 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f5ca800e870 con 0x7f5cb81002d0 2024-03-24T10:31:28.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.421+0000 7f5caffff700 1 --2- 172.21.15.23:0/4147243203 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f5ca4038f80 0x7f5ca403b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:28.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.421+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/4147243203 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cb8190c80 con 0x7f5cb81002d0 2024-03-24T10:31:28.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.421+0000 7f5caffff700 1 -- 172.21.15.23:0/4147243203 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5ca8012070 con 0x7f5cb81002d0 2024-03-24T10:31:28.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.421+0000 7f5cb67fc700 1 --2- 172.21.15.23:0/4147243203 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f5ca4038f80 0x7f5ca403b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:28.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.422+0000 7f5cb67fc700 1 --2- 172.21.15.23:0/4147243203 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f5ca4038f80 0x7f5ca403b440 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0006fd0 tx=0x7f5ca0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:28.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.426+0000 7f5caffff700 1 -- 172.21.15.23:0/4147243203 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5ca80153d0 con 0x7f5cb81002d0 2024-03-24T10:31:28.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.641+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/4147243203 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f5cb8062640 con 0x7f5ca4038f80 2024-03-24T10:31:28.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.645+0000 7f5caffff700 1 -- 172.21.15.23:0/4147243203 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5cb8062640 con 0x7f5ca4038f80 2024-03-24T10:31:28.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.647+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/4147243203 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f5ca4038f80 msgr2=0x7f5ca403b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:28.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.647+0000 7f5cbd5aa700 1 --2- 172.21.15.23:0/4147243203 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f5ca4038f80 0x7f5ca403b440 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0006fd0 tx=0x7f5ca0006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:28.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.647+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/4147243203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cb81002d0 msgr2=0x7f5cb81971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:28.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.647+0000 7f5cbd5aa700 1 --2- 172.21.15.23:0/4147243203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cb81002d0 0x7f5cb81971e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8005e20 tx=0x7f5ca8004f50 comp rx=0 tx=0).stop 2024-03-24T10:31:28.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.647+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/4147243203 shutdown_connections 2024-03-24T10:31:28.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.647+0000 7f5cbd5aa700 1 --2- 172.21.15.23:0/4147243203 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f5ca4038f80 0x7f5ca403b440 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:28.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.647+0000 7f5cbd5aa700 1 --2- 172.21.15.23:0/4147243203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cb81002d0 0x7f5cb81971e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:28.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.647+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/4147243203 >> 172.21.15.23:0/4147243203 conn(0x7f5cb80fb890 msgr2=0x7f5cb818b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:28.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.647+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/4147243203 shutdown_connections 2024-03-24T10:31:28.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:28.647+0000 7f5cbd5aa700 1 -- 172.21.15.23:0/4147243203 wait complete. 2024-03-24T10:31:29.389 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi116 2024-03-24T10:31:29.389 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T10:31:29.389 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/etc/ceph/ceph.conf 2024-03-24T10:31:29.409 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T10:31:29.409 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:31:29.467 INFO:tasks.cephadm:Adding host smithi116 to orchestrator... 2024-03-24T10:31:29.467 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch host add smithi116 2024-03-24T10:31:29.707 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:31:29.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:29 smithi023 ceph-mon[32466]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:31:29.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:29 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:29.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:29 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:31:29.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:29 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:31:29.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:29 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:31:29.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:29 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:29.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:29 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:29.829 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:29 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:30.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:30 smithi023 ceph-mon[32466]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-24T10:31:30.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:30 smithi023 ceph-mon[32466]: Updating smithi023:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:31:30.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:30 smithi023 ceph-mon[32466]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:31:30.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:30 smithi023 ceph-mon[32466]: Updating smithi023:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:31:31.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.215+0000 7f81a1234700 1 -- 172.21.15.23:0/2226268089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f819c100330 msgr2=0x7f819c100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:31.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.215+0000 7f81a1234700 1 --2- 172.21.15.23:0/2226268089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f819c100330 0x7f819c100710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f818c009910 tx=0x7f818c009c20 comp rx=0 tx=0).stop 2024-03-24T10:31:31.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.215+0000 7f81a1234700 1 -- 172.21.15.23:0/2226268089 shutdown_connections 2024-03-24T10:31:31.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.215+0000 7f81a1234700 1 --2- 172.21.15.23:0/2226268089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f819c100330 0x7f819c100710 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:31.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.215+0000 7f81a1234700 1 -- 172.21.15.23:0/2226268089 >> 172.21.15.23:0/2226268089 conn(0x7f819c0fbd50 msgr2=0x7f819c0fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:31.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.215+0000 7f81a1234700 1 -- 172.21.15.23:0/2226268089 shutdown_connections 2024-03-24T10:31:31.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.215+0000 7f81a1234700 1 -- 172.21.15.23:0/2226268089 wait complete. 2024-03-24T10:31:31.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.216+0000 7f81a1234700 1 Processor -- start 2024-03-24T10:31:31.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.216+0000 7f81a1234700 1 -- start start 2024-03-24T10:31:31.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.216+0000 7f81a1234700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f819c100330 0x7f819c197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:31.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.216+0000 7f81a1234700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f819c197a90 con 0x7f819c100330 2024-03-24T10:31:31.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.216+0000 7f819ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f819c100330 0x7f819c197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:31.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.216+0000 7f819ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f819c100330 0x7f819c197550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43314/0 (socket says 172.21.15.23:43314) 2024-03-24T10:31:31.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.216+0000 7f819ad9d700 1 -- 172.21.15.23:0/2464739957 learned_addr learned my addr 172.21.15.23:0/2464739957 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:31.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.217+0000 7f819ad9d700 1 -- 172.21.15.23:0/2464739957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818c0095f0 con 0x7f819c100330 2024-03-24T10:31:31.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.217+0000 7f819ad9d700 1 --2- 172.21.15.23:0/2464739957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f819c100330 0x7f819c197550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f818c004f40 tx=0x7f818c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:31.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.217+0000 7f8183fff700 1 -- 172.21.15.23:0/2464739957 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f818c01d070 con 0x7f819c100330 2024-03-24T10:31:31.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.217+0000 7f8183fff700 1 -- 172.21.15.23:0/2464739957 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f818c00bc30 con 0x7f819c100330 2024-03-24T10:31:31.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.217+0000 7f8183fff700 1 -- 172.21.15.23:0/2464739957 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f818c003970 con 0x7f819c100330 2024-03-24T10:31:31.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.217+0000 7f81a1234700 1 -- 172.21.15.23:0/2464739957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f819c197c90 con 0x7f819c100330 2024-03-24T10:31:31.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.217+0000 7f81a1234700 1 -- 172.21.15.23:0/2464739957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f819c1980b0 con 0x7f819c100330 2024-03-24T10:31:31.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.218+0000 7f8183fff700 1 -- 172.21.15.23:0/2464739957 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f818c003c30 con 0x7f819c100330 2024-03-24T10:31:31.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.218+0000 7f8183fff700 1 --2- 172.21.15.23:0/2464739957 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8184038f80 0x7f818403b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:31.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.218+0000 7f8183fff700 1 -- 172.21.15.23:0/2464739957 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f818c04d010 con 0x7f819c100330 2024-03-24T10:31:31.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.218+0000 7f81a1234700 1 -- 172.21.15.23:0/2464739957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f819c063830 con 0x7f819c100330 2024-03-24T10:31:31.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.218+0000 7f819a59c700 1 --2- 172.21.15.23:0/2464739957 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8184038f80 0x7f818403b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:31.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.219+0000 7f819a59c700 1 --2- 172.21.15.23:0/2464739957 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8184038f80 0x7f818403b440 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8190006fd0 tx=0x7f8190006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:31.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.223+0000 7f8183fff700 1 -- 172.21.15.23:0/2464739957 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f818c026080 con 0x7f819c100330 2024-03-24T10:31:31.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:31.429+0000 7f81a1234700 1 -- 172.21.15.23:0/2464739957 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi116", "target": ["mon-mgr", ""]}) -- 0x7f819c10c090 con 0x7f8184038f80 2024-03-24T10:31:32.011 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:31 smithi023 ceph-mon[32466]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi116", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:31:33.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:32 smithi023 ceph-mon[32466]: Deploying cephadm binary to smithi116 2024-03-24T10:31:35.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:35.582+0000 7f8183fff700 1 -- 172.21.15.23:0/2464739957 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f819c10c090 con 0x7f8184038f80 2024-03-24T10:31:35.582 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi116' with addr '172.21.15.116' 2024-03-24T10:31:35.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:35.584+0000 7f81a1234700 1 -- 172.21.15.23:0/2464739957 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8184038f80 msgr2=0x7f818403b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:35.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:35.584+0000 7f81a1234700 1 --2- 172.21.15.23:0/2464739957 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8184038f80 0x7f818403b440 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8190006fd0 tx=0x7f8190006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:35.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:35.584+0000 7f81a1234700 1 -- 172.21.15.23:0/2464739957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f819c100330 msgr2=0x7f819c197550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:35.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:35.584+0000 7f81a1234700 1 --2- 172.21.15.23:0/2464739957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f819c100330 0x7f819c197550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f818c004f40 tx=0x7f818c005020 comp rx=0 tx=0).stop 2024-03-24T10:31:35.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:35.584+0000 7f81a1234700 1 -- 172.21.15.23:0/2464739957 shutdown_connections 2024-03-24T10:31:35.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:35.584+0000 7f81a1234700 1 --2- 172.21.15.23:0/2464739957 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8184038f80 0x7f818403b440 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:35.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:35.584+0000 7f81a1234700 1 --2- 172.21.15.23:0/2464739957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f819c100330 0x7f819c197550 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:35.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:35.584+0000 7f81a1234700 1 -- 172.21.15.23:0/2464739957 >> 172.21.15.23:0/2464739957 conn(0x7f819c0fbd50 msgr2=0x7f819c109760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:35.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:35.584+0000 7f81a1234700 1 -- 172.21.15.23:0/2464739957 shutdown_connections 2024-03-24T10:31:35.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:35.584+0000 7f81a1234700 1 -- 172.21.15.23:0/2464739957 wait complete. 2024-03-24T10:31:36.128 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T10:31:36.367 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:31:36.793 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:36 smithi023 ceph-mon[32466]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:36.793 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:36.793 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:36 smithi023 ceph-mon[32466]: Added host smithi116 2024-03-24T10:31:36.793 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:31:36.793 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:37.790 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:37 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.794+0000 7faf46935700 1 -- 172.21.15.23:0/2136799563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf40106b60 msgr2=0x7faf40106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.794+0000 7faf46935700 1 --2- 172.21.15.23:0/2136799563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf40106b60 0x7faf40106f40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faf30009910 tx=0x7faf30009c20 comp rx=0 tx=0).stop 2024-03-24T10:31:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.794+0000 7faf46935700 1 -- 172.21.15.23:0/2136799563 shutdown_connections 2024-03-24T10:31:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.794+0000 7faf46935700 1 --2- 172.21.15.23:0/2136799563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf40106b60 0x7faf40106f40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.794+0000 7faf46935700 1 -- 172.21.15.23:0/2136799563 >> 172.21.15.23:0/2136799563 conn(0x7faf40077740 msgr2=0x7faf400fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.794+0000 7faf46935700 1 -- 172.21.15.23:0/2136799563 shutdown_connections 2024-03-24T10:31:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.794+0000 7faf46935700 1 -- 172.21.15.23:0/2136799563 wait complete. 2024-03-24T10:31:37.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.795+0000 7faf46935700 1 Processor -- start 2024-03-24T10:31:37.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.795+0000 7faf46935700 1 -- start start 2024-03-24T10:31:37.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.795+0000 7faf46935700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf40106b60 0x7faf401961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:37.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.795+0000 7faf46935700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7faf4006a3a0 con 0x7faf40106b60 2024-03-24T10:31:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.795+0000 7faf3ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf40106b60 0x7faf401961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.795+0000 7faf3ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf40106b60 0x7faf401961c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56068/0 (socket says 172.21.15.23:56068) 2024-03-24T10:31:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.795+0000 7faf3ffff700 1 -- 172.21.15.23:0/4246829199 learned_addr learned my addr 172.21.15.23:0/4246829199 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.796+0000 7faf3ffff700 1 -- 172.21.15.23:0/4246829199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf300095f0 con 0x7faf40106b60 2024-03-24T10:31:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.796+0000 7faf3ffff700 1 --2- 172.21.15.23:0/4246829199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf40106b60 0x7faf401961c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faf30004f40 tx=0x7faf30005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.796+0000 7faf3d7fa700 1 -- 172.21.15.23:0/4246829199 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf3001d070 con 0x7faf40106b60 2024-03-24T10:31:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.796+0000 7faf46935700 1 -- 172.21.15.23:0/4246829199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf40196700 con 0x7faf40106b60 2024-03-24T10:31:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.796+0000 7faf3d7fa700 1 -- 172.21.15.23:0/4246829199 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf3000bbd0 con 0x7faf40106b60 2024-03-24T10:31:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.796+0000 7faf3d7fa700 1 -- 172.21.15.23:0/4246829199 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf30003900 con 0x7faf40106b60 2024-03-24T10:31:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.796+0000 7faf46935700 1 -- 172.21.15.23:0/4246829199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf40196b80 con 0x7faf40106b60 2024-03-24T10:31:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.797+0000 7faf3d7fa700 1 -- 172.21.15.23:0/4246829199 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7faf30003bc0 con 0x7faf40106b60 2024-03-24T10:31:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.797+0000 7faf46935700 1 -- 172.21.15.23:0/4246829199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf401021d0 con 0x7faf40106b60 2024-03-24T10:31:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.797+0000 7faf3d7fa700 1 --2- 172.21.15.23:0/4246829199 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7faf28038f80 0x7faf2803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.797+0000 7faf3d7fa700 1 -- 172.21.15.23:0/4246829199 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7faf3004cea0 con 0x7faf40106b60 2024-03-24T10:31:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.797+0000 7faf3f7fe700 1 --2- 172.21.15.23:0/4246829199 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7faf28038f80 0x7faf2803b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.798+0000 7faf3f7fe700 1 --2- 172.21.15.23:0/4246829199 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7faf28038f80 0x7faf2803b440 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faf34006fd0 tx=0x7faf34006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:37.802+0000 7faf3d7fa700 1 -- 172.21.15.23:0/4246829199 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7faf3002ae20 con 0x7faf40106b60 2024-03-24T10:31:38.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:38.010+0000 7faf46935700 1 -- 172.21.15.23:0/4246829199 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faf4006a580 con 0x7faf28038f80 2024-03-24T10:31:38.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:38.012+0000 7faf3d7fa700 1 -- 172.21.15.23:0/4246829199 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7faf34009e20 con 0x7faf28038f80 2024-03-24T10:31:38.012 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:31:38.012 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}] 2024-03-24T10:31:38.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:38.014+0000 7faf46935700 1 -- 172.21.15.23:0/4246829199 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7faf28038f80 msgr2=0x7faf2803b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:38.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:38.014+0000 7faf46935700 1 --2- 172.21.15.23:0/4246829199 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7faf28038f80 0x7faf2803b440 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faf34006fd0 tx=0x7faf34006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:38.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:38.014+0000 7faf46935700 1 -- 172.21.15.23:0/4246829199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf40106b60 msgr2=0x7faf401961c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:38.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:38.014+0000 7faf46935700 1 --2- 172.21.15.23:0/4246829199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf40106b60 0x7faf401961c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faf30004f40 tx=0x7faf30005020 comp rx=0 tx=0).stop 2024-03-24T10:31:38.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:38.014+0000 7faf46935700 1 -- 172.21.15.23:0/4246829199 shutdown_connections 2024-03-24T10:31:38.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:38.014+0000 7faf46935700 1 --2- 172.21.15.23:0/4246829199 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7faf28038f80 0x7faf2803b440 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:38.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:38.014+0000 7faf46935700 1 --2- 172.21.15.23:0/4246829199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf40106b60 0x7faf401961c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:38.014+0000 7faf46935700 1 -- 172.21.15.23:0/4246829199 >> 172.21.15.23:0/4246829199 conn(0x7faf40077740 msgr2=0x7faf40108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:38.014+0000 7faf46935700 1 -- 172.21.15.23:0/4246829199 shutdown_connections 2024-03-24T10:31:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:38.014+0000 7faf46935700 1 -- 172.21.15.23:0/4246829199 wait complete. 2024-03-24T10:31:38.663 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi117 2024-03-24T10:31:38.663 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T10:31:38.663 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/etc/ceph/ceph.conf 2024-03-24T10:31:38.683 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T10:31:38.683 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:31:38.741 INFO:tasks.cephadm:Adding host smithi117 to orchestrator... 2024-03-24T10:31:38.741 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch host add smithi117 2024-03-24T10:31:38.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:38 smithi023 ceph-mon[32466]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:38.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:38 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:38.981 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:31:39.892 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:39 smithi023 ceph-mon[32466]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:31:40.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.384+0000 7ff388a0d700 1 -- 172.21.15.23:0/3261040543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff384101e40 msgr2=0x7ff384102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.384+0000 7ff388a0d700 1 --2- 172.21.15.23:0/3261040543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff384101e40 0x7ff384102220 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009910 tx=0x7ff37c009c20 comp rx=0 tx=0).stop 2024-03-24T10:31:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.384+0000 7ff388a0d700 1 -- 172.21.15.23:0/3261040543 shutdown_connections 2024-03-24T10:31:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.384+0000 7ff388a0d700 1 --2- 172.21.15.23:0/3261040543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff384101e40 0x7ff384102220 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.384+0000 7ff388a0d700 1 -- 172.21.15.23:0/3261040543 >> 172.21.15.23:0/3261040543 conn(0x7ff3840fd710 msgr2=0x7ff3840ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.384+0000 7ff388a0d700 1 -- 172.21.15.23:0/3261040543 shutdown_connections 2024-03-24T10:31:40.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.384+0000 7ff388a0d700 1 -- 172.21.15.23:0/3261040543 wait complete. 2024-03-24T10:31:40.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.385+0000 7ff388a0d700 1 Processor -- start 2024-03-24T10:31:40.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.385+0000 7ff388a0d700 1 -- start start 2024-03-24T10:31:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.385+0000 7ff388a0d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff384101e40 0x7ff384199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.385+0000 7ff388a0d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff384199c60 con 0x7ff384101e40 2024-03-24T10:31:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.385+0000 7ff38259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff384101e40 0x7ff384199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.385+0000 7ff38259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff384101e40 0x7ff384199720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56090/0 (socket says 172.21.15.23:56090) 2024-03-24T10:31:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.385+0000 7ff38259c700 1 -- 172.21.15.23:0/1922736290 learned_addr learned my addr 172.21.15.23:0/1922736290 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.385+0000 7ff38259c700 1 -- 172.21.15.23:0/1922736290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37c0095f0 con 0x7ff384101e40 2024-03-24T10:31:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.386+0000 7ff38259c700 1 --2- 172.21.15.23:0/1922736290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff384101e40 0x7ff384199720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff37c004f40 tx=0x7ff37c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.386+0000 7ff37b7fe700 1 -- 172.21.15.23:0/1922736290 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff37c01d070 con 0x7ff384101e40 2024-03-24T10:31:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.386+0000 7ff37b7fe700 1 -- 172.21.15.23:0/1922736290 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff37c00bc30 con 0x7ff384101e40 2024-03-24T10:31:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.386+0000 7ff388a0d700 1 -- 172.21.15.23:0/1922736290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff384199e60 con 0x7ff384101e40 2024-03-24T10:31:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.386+0000 7ff37b7fe700 1 -- 172.21.15.23:0/1922736290 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff37c003970 con 0x7ff384101e40 2024-03-24T10:31:40.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.386+0000 7ff388a0d700 1 -- 172.21.15.23:0/1922736290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff38419a280 con 0x7ff384101e40 2024-03-24T10:31:40.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.387+0000 7ff37b7fe700 1 -- 172.21.15.23:0/1922736290 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7ff37c003c30 con 0x7ff384101e40 2024-03-24T10:31:40.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.387+0000 7ff388a0d700 1 -- 172.21.15.23:0/1922736290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff384063830 con 0x7ff384101e40 2024-03-24T10:31:40.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.387+0000 7ff37b7fe700 1 --2- 172.21.15.23:0/1922736290 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff370038f80 0x7ff37003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:40.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.387+0000 7ff37b7fe700 1 -- 172.21.15.23:0/1922736290 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff37c04ce90 con 0x7ff384101e40 2024-03-24T10:31:40.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.387+0000 7ff381d9b700 1 --2- 172.21.15.23:0/1922736290 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff370038f80 0x7ff37003b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:40.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.388+0000 7ff381d9b700 1 --2- 172.21.15.23:0/1922736290 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff370038f80 0x7ff37003b440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff36c006fd0 tx=0x7ff36c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:40.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.392+0000 7ff37b7fe700 1 -- 172.21.15.23:0/1922736290 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff37c029910 con 0x7ff384101e40 2024-03-24T10:31:40.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:40.601+0000 7ff388a0d700 1 -- 172.21.15.23:0/1922736290 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi117", "target": ["mon-mgr", ""]}) -- 0x7ff38410c170 con 0x7ff370038f80 2024-03-24T10:31:40.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:40 smithi023 ceph-mon[32466]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:41.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:41 smithi023 ceph-mon[32466]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi117", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:31:41.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:41.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:41.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:41.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:41.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:31:41.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:31:41.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:31:42.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:42 smithi023 ceph-mon[32466]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:42.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:42 smithi023 ceph-mon[32466]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T10:31:42.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:42 smithi023 ceph-mon[32466]: Deploying cephadm binary to smithi117 2024-03-24T10:31:42.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:42 smithi023 ceph-mon[32466]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:31:42.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:42 smithi023 ceph-mon[32466]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:31:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:44.849+0000 7ff37b7fe700 1 -- 172.21.15.23:0/1922736290 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7ff38410c170 con 0x7ff370038f80 2024-03-24T10:31:44.850 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi117' with addr '172.21.15.117' 2024-03-24T10:31:44.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:44.851+0000 7ff388a0d700 1 -- 172.21.15.23:0/1922736290 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff370038f80 msgr2=0x7ff37003b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:44.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:44.851+0000 7ff388a0d700 1 --2- 172.21.15.23:0/1922736290 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff370038f80 0x7ff37003b440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff36c006fd0 tx=0x7ff36c006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:44.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:44.851+0000 7ff388a0d700 1 -- 172.21.15.23:0/1922736290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff384101e40 msgr2=0x7ff384199720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:44.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:44.851+0000 7ff388a0d700 1 --2- 172.21.15.23:0/1922736290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff384101e40 0x7ff384199720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff37c004f40 tx=0x7ff37c005020 comp rx=0 tx=0).stop 2024-03-24T10:31:44.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:44.851+0000 7ff388a0d700 1 -- 172.21.15.23:0/1922736290 shutdown_connections 2024-03-24T10:31:44.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:44.851+0000 7ff388a0d700 1 --2- 172.21.15.23:0/1922736290 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff370038f80 0x7ff37003b440 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:44.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:44.851+0000 7ff388a0d700 1 --2- 172.21.15.23:0/1922736290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff384101e40 0x7ff384199720 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:44.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:44.851+0000 7ff388a0d700 1 -- 172.21.15.23:0/1922736290 >> 172.21.15.23:0/1922736290 conn(0x7ff3840fd710 msgr2=0x7ff384075f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:44.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:44.852+0000 7ff388a0d700 1 -- 172.21.15.23:0/1922736290 shutdown_connections 2024-03-24T10:31:44.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:44.852+0000 7ff388a0d700 1 -- 172.21.15.23:0/1922736290 wait complete. 2024-03-24T10:31:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:44 smithi023 ceph-mon[32466]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:44 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:44 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:44 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:45.423 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T10:31:45.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:45 smithi023 ceph-mon[32466]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:31:45.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:45 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:45.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:45 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:31:45.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:45 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:45.671 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:31:46.805 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:46 smithi023 ceph-mon[32466]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:46.805 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:46 smithi023 ceph-mon[32466]: Added host smithi117 2024-03-24T10:31:46.805 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:46 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:47.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.135+0000 7f219e666700 1 -- 172.21.15.23:0/2343133240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2198106ac0 msgr2=0x7f2198106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:47.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.135+0000 7f219e666700 1 --2- 172.21.15.23:0/2343133240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2198106ac0 0x7f2198106ea0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2188009910 tx=0x7f2188009c20 comp rx=0 tx=0).stop 2024-03-24T10:31:47.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.135+0000 7f219e666700 1 -- 172.21.15.23:0/2343133240 shutdown_connections 2024-03-24T10:31:47.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.135+0000 7f219e666700 1 --2- 172.21.15.23:0/2343133240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2198106ac0 0x7f2198106ea0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:47.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.135+0000 7f219e666700 1 -- 172.21.15.23:0/2343133240 >> 172.21.15.23:0/2343133240 conn(0x7f21980fc5b0 msgr2=0x7f21980fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:47.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.135+0000 7f219e666700 1 -- 172.21.15.23:0/2343133240 shutdown_connections 2024-03-24T10:31:47.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.135+0000 7f219e666700 1 -- 172.21.15.23:0/2343133240 wait complete. 2024-03-24T10:31:47.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.136+0000 7f219e666700 1 Processor -- start 2024-03-24T10:31:47.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.136+0000 7f219e666700 1 -- start start 2024-03-24T10:31:47.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.136+0000 7f219e666700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2198106ac0 0x7f21981974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:47.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.136+0000 7f219e666700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f21981979f0 con 0x7f2198106ac0 2024-03-24T10:31:47.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.136+0000 7f2197fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2198106ac0 0x7f21981974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:47.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.136+0000 7f2197fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2198106ac0 0x7f21981974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56106/0 (socket says 172.21.15.23:56106) 2024-03-24T10:31:47.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.136+0000 7f2197fff700 1 -- 172.21.15.23:0/2516672006 learned_addr learned my addr 172.21.15.23:0/2516672006 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:47.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.136+0000 7f2197fff700 1 -- 172.21.15.23:0/2516672006 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21880095f0 con 0x7f2198106ac0 2024-03-24T10:31:47.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.137+0000 7f2197fff700 1 --2- 172.21.15.23:0/2516672006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2198106ac0 0x7f21981974b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2188004f40 tx=0x7f2188005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:47.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.137+0000 7f21957fa700 1 -- 172.21.15.23:0/2516672006 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f218801d070 con 0x7f2198106ac0 2024-03-24T10:31:47.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.137+0000 7f21957fa700 1 -- 172.21.15.23:0/2516672006 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f218800bbd0 con 0x7f2198106ac0 2024-03-24T10:31:47.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.137+0000 7f219e666700 1 -- 172.21.15.23:0/2516672006 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2198197c50 con 0x7f2198106ac0 2024-03-24T10:31:47.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.137+0000 7f21957fa700 1 -- 172.21.15.23:0/2516672006 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2188003900 con 0x7f2198106ac0 2024-03-24T10:31:47.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.137+0000 7f219e666700 1 -- 172.21.15.23:0/2516672006 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21981980d0 con 0x7f2198106ac0 2024-03-24T10:31:47.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.138+0000 7f21957fa700 1 -- 172.21.15.23:0/2516672006 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f2188003bc0 con 0x7f2198106ac0 2024-03-24T10:31:47.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.138+0000 7f219e666700 1 -- 172.21.15.23:0/2516672006 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2198063830 con 0x7f2198106ac0 2024-03-24T10:31:47.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.138+0000 7f21957fa700 1 --2- 172.21.15.23:0/2516672006 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2184038f80 0x7f218403b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:47.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.138+0000 7f21957fa700 1 -- 172.21.15.23:0/2516672006 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f218804d020 con 0x7f2198106ac0 2024-03-24T10:31:47.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.138+0000 7f21977fe700 1 --2- 172.21.15.23:0/2516672006 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2184038f80 0x7f218403b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:47.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.139+0000 7f21977fe700 1 --2- 172.21.15.23:0/2516672006 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2184038f80 0x7f218403b440 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2180006fd0 tx=0x7f2180006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:47.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.143+0000 7f21957fa700 1 -- 172.21.15.23:0/2516672006 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2188029540 con 0x7f2198106ac0 2024-03-24T10:31:47.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.352+0000 7f219e666700 1 -- 172.21.15.23:0/2516672006 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f219810b0b0 con 0x7f2184038f80 2024-03-24T10:31:47.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.353+0000 7f21957fa700 1 -- 172.21.15.23:0/2516672006 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f219810b0b0 con 0x7f2184038f80 2024-03-24T10:31:47.355 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:31:47.355 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.117", "hostname": "smithi117", "labels": [], "status": ""}] 2024-03-24T10:31:47.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.355+0000 7f219e666700 1 -- 172.21.15.23:0/2516672006 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2184038f80 msgr2=0x7f218403b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:47.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.355+0000 7f219e666700 1 --2- 172.21.15.23:0/2516672006 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2184038f80 0x7f218403b440 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2180006fd0 tx=0x7f2180006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:47.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.355+0000 7f219e666700 1 -- 172.21.15.23:0/2516672006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2198106ac0 msgr2=0x7f21981974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:47.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.355+0000 7f219e666700 1 --2- 172.21.15.23:0/2516672006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2198106ac0 0x7f21981974b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2188004f40 tx=0x7f2188005020 comp rx=0 tx=0).stop 2024-03-24T10:31:47.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.356+0000 7f219e666700 1 -- 172.21.15.23:0/2516672006 shutdown_connections 2024-03-24T10:31:47.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.356+0000 7f219e666700 1 --2- 172.21.15.23:0/2516672006 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2184038f80 0x7f218403b440 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:47.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.356+0000 7f219e666700 1 --2- 172.21.15.23:0/2516672006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2198106ac0 0x7f21981974b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:47.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.356+0000 7f219e666700 1 -- 172.21.15.23:0/2516672006 >> 172.21.15.23:0/2516672006 conn(0x7f21980fc5b0 msgr2=0x7f21981045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:47.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.356+0000 7f219e666700 1 -- 172.21.15.23:0/2516672006 shutdown_connections 2024-03-24T10:31:47.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:47.356+0000 7f219e666700 1 -- 172.21.15.23:0/2516672006 wait complete. 2024-03-24T10:31:47.929 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T10:31:47.930 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T10:31:48.168 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:31:48.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:48 smithi023 ceph-mon[32466]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:48.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:48 smithi023 ceph-mon[32466]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:31:48.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:48 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:49.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.683+0000 7f2d904f7700 1 -- 172.21.15.23:0/3673306405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d88102da0 msgr2=0x7f2d88103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:49.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.683+0000 7f2d904f7700 1 --2- 172.21.15.23:0/3673306405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d88102da0 0x7f2d88103180 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009910 tx=0x7f2d84009c20 comp rx=0 tx=0).stop 2024-03-24T10:31:49.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.683+0000 7f2d904f7700 1 -- 172.21.15.23:0/3673306405 shutdown_connections 2024-03-24T10:31:49.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.683+0000 7f2d904f7700 1 --2- 172.21.15.23:0/3673306405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d88102da0 0x7f2d88103180 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:49.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.683+0000 7f2d904f7700 1 -- 172.21.15.23:0/3673306405 >> 172.21.15.23:0/3673306405 conn(0x7f2d880fc5f0 msgr2=0x7f2d880fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:49.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.683+0000 7f2d904f7700 1 -- 172.21.15.23:0/3673306405 shutdown_connections 2024-03-24T10:31:49.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.683+0000 7f2d904f7700 1 -- 172.21.15.23:0/3673306405 wait complete. 2024-03-24T10:31:49.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.684+0000 7f2d904f7700 1 Processor -- start 2024-03-24T10:31:49.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.684+0000 7f2d904f7700 1 -- start start 2024-03-24T10:31:49.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.684+0000 7f2d904f7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d88102da0 0x7f2d881975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:49.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.684+0000 7f2d904f7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2d88197ae0 con 0x7f2d88102da0 2024-03-24T10:31:49.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.684+0000 7f2d8e293700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d88102da0 0x7f2d881975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:49.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.684+0000 7f2d8e293700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d88102da0 0x7f2d881975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53886/0 (socket says 172.21.15.23:53886) 2024-03-24T10:31:49.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.684+0000 7f2d8e293700 1 -- 172.21.15.23:0/2919387357 learned_addr learned my addr 172.21.15.23:0/2919387357 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:31:49.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.684+0000 7f2d8e293700 1 -- 172.21.15.23:0/2919387357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d840095f0 con 0x7f2d88102da0 2024-03-24T10:31:49.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.684+0000 7f2d8e293700 1 --2- 172.21.15.23:0/2919387357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d88102da0 0x7f2d881975a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2d84005850 tx=0x7f2d84004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:49.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.685+0000 7f2d7b7fe700 1 -- 172.21.15.23:0/2919387357 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d8401d070 con 0x7f2d88102da0 2024-03-24T10:31:49.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.685+0000 7f2d7b7fe700 1 -- 172.21.15.23:0/2919387357 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8400bc30 con 0x7f2d88102da0 2024-03-24T10:31:49.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.685+0000 7f2d7b7fe700 1 -- 172.21.15.23:0/2919387357 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d84003a20 con 0x7f2d88102da0 2024-03-24T10:31:49.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.685+0000 7f2d904f7700 1 -- 172.21.15.23:0/2919387357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d88197ce0 con 0x7f2d88102da0 2024-03-24T10:31:49.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.685+0000 7f2d904f7700 1 -- 172.21.15.23:0/2919387357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d88198100 con 0x7f2d88102da0 2024-03-24T10:31:49.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.685+0000 7f2d7b7fe700 1 -- 172.21.15.23:0/2919387357 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f2d84003b80 con 0x7f2d88102da0 2024-03-24T10:31:49.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.686+0000 7f2d7b7fe700 1 --2- 172.21.15.23:0/2919387357 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2d74038fe0 0x7f2d7403b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:49.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.686+0000 7f2d7b7fe700 1 -- 172.21.15.23:0/2919387357 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2d8404cd40 con 0x7f2d88102da0 2024-03-24T10:31:49.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.686+0000 7f2d904f7700 1 -- 172.21.15.23:0/2919387357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d88063830 con 0x7f2d88102da0 2024-03-24T10:31:49.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.686+0000 7f2d8da92700 1 --2- 172.21.15.23:0/2919387357 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2d74038fe0 0x7f2d7403b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:49.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.686+0000 7f2d8da92700 1 --2- 172.21.15.23:0/2919387357 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2d74038fe0 0x7f2d7403b4a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c006fd0 tx=0x7f2d7c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:49.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.691+0000 7f2d7b7fe700 1 -- 172.21.15.23:0/2919387357 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2d84026080 con 0x7f2d88102da0 2024-03-24T10:31:49.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:49.898+0000 7f2d904f7700 1 -- 172.21.15.23:0/2919387357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f2d8810c080 con 0x7f2d88102da0 2024-03-24T10:31:50.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:50.409+0000 7f2d7b7fe700 1 -- 172.21.15.23:0/2919387357 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f2d8402b440 con 0x7f2d88102da0 2024-03-24T10:31:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:50.410+0000 7f2d904f7700 1 -- 172.21.15.23:0/2919387357 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2d74038fe0 msgr2=0x7f2d7403b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:50.410+0000 7f2d904f7700 1 --2- 172.21.15.23:0/2919387357 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2d74038fe0 0x7f2d7403b4a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c006fd0 tx=0x7f2d7c006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:50.410+0000 7f2d904f7700 1 -- 172.21.15.23:0/2919387357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d88102da0 msgr2=0x7f2d881975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:50.410+0000 7f2d904f7700 1 --2- 172.21.15.23:0/2919387357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d88102da0 0x7f2d881975a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2d84005850 tx=0x7f2d84004ff0 comp rx=0 tx=0).stop 2024-03-24T10:31:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:50.410+0000 7f2d904f7700 1 -- 172.21.15.23:0/2919387357 shutdown_connections 2024-03-24T10:31:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:50.410+0000 7f2d904f7700 1 --2- 172.21.15.23:0/2919387357 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2d74038fe0 0x7f2d7403b4a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:50.410+0000 7f2d904f7700 1 --2- 172.21.15.23:0/2919387357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d88102da0 0x7f2d881975a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:50.410+0000 7f2d904f7700 1 -- 172.21.15.23:0/2919387357 >> 172.21.15.23:0/2919387357 conn(0x7f2d880fc5f0 msgr2=0x7f2d88103b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:50.411+0000 7f2d904f7700 1 -- 172.21.15.23:0/2919387357 shutdown_connections 2024-03-24T10:31:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:31:50.411+0000 7f2d904f7700 1 -- 172.21.15.23:0/2919387357 wait complete. 2024-03-24T10:31:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:adjusted tunables profile to default 2024-03-24T10:31:50.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:50 smithi023 ceph-mon[32466]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:50.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:50 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2919387357' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T10:31:51.093 INFO:tasks.cephadm:Adding mon.a on smithi023 2024-03-24T10:31:51.094 INFO:tasks.cephadm:Adding mon.b on smithi116 2024-03-24T10:31:51.094 INFO:tasks.cephadm:Adding mon.c on smithi117 2024-03-24T10:31:51.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi023:172.21.15.23=a;smithi116:172.21.15.116=b;smithi117:172.21.15.117=c' 2024-03-24T10:31:51.327 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:31:51.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:51 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2919387357' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T10:31:51.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:51 smithi023 ceph-mon[32466]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T10:31:51.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:51.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:51.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:51.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:51.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi117", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:31:51.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:31:51.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:31:51.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:51 smithi023 ceph-mon[32466]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T10:31:51.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:51.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:51.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:52.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.725+0000 7fe3b882a700 1 -- 172.21.15.117:0/3552184260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b4101450 msgr2=0x7fe3b4101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:52.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.725+0000 7fe3b882a700 1 --2- 172.21.15.117:0/3552184260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b4101450 0x7fe3b4101830 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4009910 tx=0x7fe3a4009c20 comp rx=0 tx=0).stop 2024-03-24T10:31:52.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.725+0000 7fe3b882a700 1 -- 172.21.15.117:0/3552184260 shutdown_connections 2024-03-24T10:31:52.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.725+0000 7fe3b882a700 1 --2- 172.21.15.117:0/3552184260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b4101450 0x7fe3b4101830 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:52.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.725+0000 7fe3b882a700 1 -- 172.21.15.117:0/3552184260 >> 172.21.15.117:0/3552184260 conn(0x7fe3b40fbd80 msgr2=0x7fe3b40fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:52.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.725+0000 7fe3b882a700 1 -- 172.21.15.117:0/3552184260 shutdown_connections 2024-03-24T10:31:52.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.725+0000 7fe3b882a700 1 -- 172.21.15.117:0/3552184260 wait complete. 2024-03-24T10:31:52.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.726+0000 7fe3b882a700 1 Processor -- start 2024-03-24T10:31:52.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.726+0000 7fe3b882a700 1 -- start start 2024-03-24T10:31:52.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.726+0000 7fe3b882a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b4101450 0x7fe3b4102870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:52.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.726+0000 7fe3b882a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b4104760 con 0x7fe3b4101450 2024-03-24T10:31:52.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.727+0000 7fe3b259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b4101450 0x7fe3b4102870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:52.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.727+0000 7fe3b259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b4101450 0x7fe3b4102870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.117:52070/0 (socket says 172.21.15.117:52070) 2024-03-24T10:31:52.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.727+0000 7fe3b259c700 1 -- 172.21.15.117:0/2161677905 learned_addr learned my addr 172.21.15.117:0/2161677905 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T10:31:52.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.727+0000 7fe3b259c700 1 -- 172.21.15.117:0/2161677905 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a40095f0 con 0x7fe3b4101450 2024-03-24T10:31:52.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.727+0000 7fe3b259c700 1 --2- 172.21.15.117:0/2161677905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b4101450 0x7fe3b4102870 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4004f40 tx=0x7fe3a4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:52.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.727+0000 7fe3abfff700 1 -- 172.21.15.117:0/2161677905 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3a401d070 con 0x7fe3b4101450 2024-03-24T10:31:52.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.727+0000 7fe3b882a700 1 -- 172.21.15.117:0/2161677905 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3b4102db0 con 0x7fe3b4101450 2024-03-24T10:31:52.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.727+0000 7fe3abfff700 1 -- 172.21.15.117:0/2161677905 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3a400bc30 con 0x7fe3b4101450 2024-03-24T10:31:52.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.727+0000 7fe3abfff700 1 -- 172.21.15.117:0/2161677905 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3a4003970 con 0x7fe3b4101450 2024-03-24T10:31:52.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.727+0000 7fe3b882a700 1 -- 172.21.15.117:0/2161677905 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3b41031d0 con 0x7fe3b4101450 2024-03-24T10:31:52.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.728+0000 7fe3b882a700 1 -- 172.21.15.117:0/2161677905 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3b4063830 con 0x7fe3b4101450 2024-03-24T10:31:52.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.728+0000 7fe3abfff700 1 -- 172.21.15.117:0/2161677905 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fe3a4003c30 con 0x7fe3b4101450 2024-03-24T10:31:52.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.729+0000 7fe3abfff700 1 --2- 172.21.15.117:0/2161677905 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe3a0038f80 0x7fe3a003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:52.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.729+0000 7fe3abfff700 1 -- 172.21.15.117:0/2161677905 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe3a404cd80 con 0x7fe3b4101450 2024-03-24T10:31:52.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.729+0000 7fe3a9d9b700 1 --2- 172.21.15.117:0/2161677905 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe3a0038f80 0x7fe3a003b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:52.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.729+0000 7fe3a9d9b700 1 --2- 172.21.15.117:0/2161677905 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe3a0038f80 0x7fe3a003b440 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe39c006fd0 tx=0x7fe39c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:52.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.734+0000 7fe3abfff700 1 -- 172.21.15.117:0/2161677905 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe3a4005c00 con 0x7fe3b4101450 2024-03-24T10:31:52.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:52 smithi023 ceph-mon[32466]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:52.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:52 smithi023 ceph-mon[32466]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:31:52.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:52 smithi023 ceph-mon[32466]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:31:52.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:52 smithi023 ceph-mon[32466]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:31:52.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.937+0000 7fe3b882a700 1 -- 172.21.15.117:0/2161677905 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi116:172.21.15.116=b;smithi117:172.21.15.117=c", "target": ["mon-mgr", ""]}) -- 0x7fe3b410b120 con 0x7fe3a0038f80 2024-03-24T10:31:52.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.944+0000 7fe3abfff700 1 -- 172.21.15.117:0/2161677905 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe3b410b120 con 0x7fe3a0038f80 2024-03-24T10:31:52.947 INFO:teuthology.orchestra.run.smithi117.stdout:Scheduled mon update... 2024-03-24T10:31:52.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.946+0000 7fe3b882a700 1 -- 172.21.15.117:0/2161677905 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe3a0038f80 msgr2=0x7fe3a003b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:52.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.946+0000 7fe3b882a700 1 --2- 172.21.15.117:0/2161677905 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe3a0038f80 0x7fe3a003b440 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe39c006fd0 tx=0x7fe39c006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:52.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.946+0000 7fe3b882a700 1 -- 172.21.15.117:0/2161677905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b4101450 msgr2=0x7fe3b4102870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:52.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.946+0000 7fe3b882a700 1 --2- 172.21.15.117:0/2161677905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b4101450 0x7fe3b4102870 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4004f40 tx=0x7fe3a4005020 comp rx=0 tx=0).stop 2024-03-24T10:31:52.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.946+0000 7fe3b882a700 1 -- 172.21.15.117:0/2161677905 shutdown_connections 2024-03-24T10:31:52.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.946+0000 7fe3b882a700 1 --2- 172.21.15.117:0/2161677905 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe3a0038f80 0x7fe3a003b440 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:52.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.946+0000 7fe3b882a700 1 --2- 172.21.15.117:0/2161677905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3b4101450 0x7fe3b4102870 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:52.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.946+0000 7fe3b882a700 1 -- 172.21.15.117:0/2161677905 >> 172.21.15.117:0/2161677905 conn(0x7fe3b40fbd80 msgr2=0x7fe3b41000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:52.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.946+0000 7fe3b882a700 1 -- 172.21.15.117:0/2161677905 shutdown_connections 2024-03-24T10:31:52.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:52.946+0000 7fe3b882a700 1 -- 172.21.15.117:0/2161677905 wait complete. 2024-03-24T10:31:53.572 DEBUG:teuthology.orchestra.run.smithi116:mon.b> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mon.b.service 2024-03-24T10:31:53.575 DEBUG:teuthology.orchestra.run.smithi117:mon.c> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mon.c.service 2024-03-24T10:31:53.577 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T10:31:53.577 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:31:54.260 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:31:54.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:53 smithi023 ceph-mon[32466]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:54.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:53 smithi023 ceph-mon[32466]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi116:172.21.15.116=b;smithi117:172.21.15.117=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:31:54.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:53 smithi023 ceph-mon[32466]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi116:172.21.15.116=b;smithi117:172.21.15.117=c;count:3 2024-03-24T10:31:54.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:53 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:54.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:53 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:31:54.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:53 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:31:54.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:53 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:31:54.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:53 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:54.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:53 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:31:54.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:53 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:31:54.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:53 smithi023 ceph-mon[32466]: Deploying daemon mon.c on smithi117 2024-03-24T10:31:56.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.365+0000 7f4efd7b4700 1 -- 172.21.15.117:0/2748945841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ef8102e10 msgr2=0x7f4ef81031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:56.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.365+0000 7f4efd7b4700 1 --2- 172.21.15.117:0/2748945841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ef8102e10 0x7f4ef81031f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8009910 tx=0x7f4ee8009c20 comp rx=0 tx=0).stop 2024-03-24T10:31:56.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.365+0000 7f4efd7b4700 1 -- 172.21.15.117:0/2748945841 shutdown_connections 2024-03-24T10:31:56.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.365+0000 7f4efd7b4700 1 --2- 172.21.15.117:0/2748945841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ef8102e10 0x7f4ef81031f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:56.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.365+0000 7f4efd7b4700 1 -- 172.21.15.117:0/2748945841 >> 172.21.15.117:0/2748945841 conn(0x7f4ef80fc5c0 msgr2=0x7f4ef80fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:56.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.365+0000 7f4efd7b4700 1 -- 172.21.15.117:0/2748945841 shutdown_connections 2024-03-24T10:31:56.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.365+0000 7f4efd7b4700 1 -- 172.21.15.117:0/2748945841 wait complete. 2024-03-24T10:31:56.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.366+0000 7f4efd7b4700 1 Processor -- start 2024-03-24T10:31:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.366+0000 7f4efd7b4700 1 -- start start 2024-03-24T10:31:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.366+0000 7f4efd7b4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ef8102e10 0x7f4ef81975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.366+0000 7f4efd7b4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef8197b00 con 0x7f4ef8102e10 2024-03-24T10:31:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.366+0000 7f4ef6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ef8102e10 0x7f4ef81975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.366+0000 7f4ef6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ef8102e10 0x7f4ef81975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.117:52090/0 (socket says 172.21.15.117:52090) 2024-03-24T10:31:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.366+0000 7f4ef6ffd700 1 -- 172.21.15.117:0/3476100889 learned_addr learned my addr 172.21.15.117:0/3476100889 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T10:31:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.367+0000 7f4ef6ffd700 1 -- 172.21.15.117:0/3476100889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee80095f0 con 0x7f4ef8102e10 2024-03-24T10:31:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.367+0000 7f4ef6ffd700 1 --2- 172.21.15.117:0/3476100889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ef8102e10 0x7f4ef81975c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8005850 tx=0x7f4ee8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.367+0000 7f4ef4ff9700 1 -- 172.21.15.117:0/3476100889 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ee801d070 con 0x7f4ef8102e10 2024-03-24T10:31:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.367+0000 7f4ef4ff9700 1 -- 172.21.15.117:0/3476100889 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee800bc30 con 0x7f4ef8102e10 2024-03-24T10:31:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.367+0000 7f4efd7b4700 1 -- 172.21.15.117:0/3476100889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef8197d00 con 0x7f4ef8102e10 2024-03-24T10:31:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.367+0000 7f4ef4ff9700 1 -- 172.21.15.117:0/3476100889 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ee8003a20 con 0x7f4ef8102e10 2024-03-24T10:31:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.367+0000 7f4efd7b4700 1 -- 172.21.15.117:0/3476100889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef8198120 con 0x7f4ef8102e10 2024-03-24T10:31:56.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.368+0000 7f4efd7b4700 1 -- 172.21.15.117:0/3476100889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ef8063830 con 0x7f4ef8102e10 2024-03-24T10:31:56.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.368+0000 7f4ef4ff9700 1 -- 172.21.15.117:0/3476100889 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f4ee8004390 con 0x7f4ef8102e10 2024-03-24T10:31:56.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.368+0000 7f4ef4ff9700 1 --2- 172.21.15.117:0/3476100889 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4ee4038f80 0x7f4ee403b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:31:56.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.368+0000 7f4ef4ff9700 1 -- 172.21.15.117:0/3476100889 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4ee804ce40 con 0x7f4ef8102e10 2024-03-24T10:31:56.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.368+0000 7f4eee7fc700 1 --2- 172.21.15.117:0/3476100889 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4ee4038f80 0x7f4ee403b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:31:56.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.369+0000 7f4eee7fc700 1 --2- 172.21.15.117:0/3476100889 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4ee4038f80 0x7f4ee403b440 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0006fd0 tx=0x7f4ee0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:31:56.374 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.373+0000 7f4ef4ff9700 1 -- 172.21.15.117:0/3476100889 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4ee800bdf0 con 0x7f4ef8102e10 2024-03-24T10:31:56.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:55 smithi023 ceph-mon[32466]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:56.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.648+0000 7f4efd7b4700 1 -- 172.21.15.117:0/3476100889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f4ef810c0a0 con 0x7f4ef8102e10 2024-03-24T10:31:56.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.649+0000 7f4ef4ff9700 1 -- 172.21.15.117:0/3476100889 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f4ee802ac50 con 0x7f4ef8102e10 2024-03-24T10:31:56.649 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:31:56.649 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":1,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:30:17.512891Z","created":"2024-03-24T10:30:17.512891Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T10:31:56.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.651+0000 7f4efd7b4700 1 -- 172.21.15.117:0/3476100889 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4ee4038f80 msgr2=0x7f4ee403b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:56.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.651+0000 7f4efd7b4700 1 --2- 172.21.15.117:0/3476100889 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4ee4038f80 0x7f4ee403b440 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0006fd0 tx=0x7f4ee0006e40 comp rx=0 tx=0).stop 2024-03-24T10:31:56.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.651+0000 7f4efd7b4700 1 -- 172.21.15.117:0/3476100889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ef8102e10 msgr2=0x7f4ef81975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:31:56.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.651+0000 7f4efd7b4700 1 --2- 172.21.15.117:0/3476100889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ef8102e10 0x7f4ef81975c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8005850 tx=0x7f4ee8004ff0 comp rx=0 tx=0).stop 2024-03-24T10:31:56.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.651+0000 7f4efd7b4700 1 -- 172.21.15.117:0/3476100889 shutdown_connections 2024-03-24T10:31:56.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.651+0000 7f4efd7b4700 1 --2- 172.21.15.117:0/3476100889 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4ee4038f80 0x7f4ee403b440 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:56.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.651+0000 7f4efd7b4700 1 --2- 172.21.15.117:0/3476100889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ef8102e10 0x7f4ef81975c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:31:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.651+0000 7f4efd7b4700 1 -- 172.21.15.117:0/3476100889 >> 172.21.15.117:0/3476100889 conn(0x7f4ef80fc5c0 msgr2=0x7f4ef8103bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:31:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.651+0000 7f4efd7b4700 1 -- 172.21.15.117:0/3476100889 shutdown_connections 2024-03-24T10:31:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:31:56.651+0000 7f4efd7b4700 1 -- 172.21.15.117:0/3476100889 wait complete. 2024-03-24T10:31:56.652 INFO:teuthology.orchestra.run.smithi117.stderr:dumped monmap epoch 1 2024-03-24T10:31:57.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:56 smithi023 ceph-mon[32466]: from='client.? 172.21.15.117:0/3476100889' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:31:58.263 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T10:31:58.264 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T10:31:58.264 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T10:31:58.264 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T10:31:58.264 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T10:31:58.264 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T10:31:58.264 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T10:31:58.264 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:31:58.264 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:31:58.264 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:31:58.264 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:58.264 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi117", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:31:58.264 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.265 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.265 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.265 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.379 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T10:31:58.380 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:31:58.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:31:57 smithi023 ceph-mon[32466]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:58.612 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:31:58.732 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:31:58.732 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:31:58.732 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:31:58.733 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:58.733 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T10:31:58.733 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: Deploying cephadm binary to smithi117 2024-03-24T10:31:58.733 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:31:58.733 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:31:58.733 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:58.733 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.733 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.733 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.733 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:31:58.733 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.734 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:31:58.734 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.734 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:58.734 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: Added host smithi117 2024-03-24T10:31:58.734 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.734 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:58.736 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:31:58.736 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.736 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:58.737 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2919387357' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T10:31:58.737 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2919387357' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T10:31:58.737 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T10:31:58.737 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.737 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.737 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.737 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.737 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi117", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:31:58.737 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:31:58.737 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:31:58.737 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T10:31:58.737 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.737 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.737 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:58.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:31:58.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:31:58.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:31:58.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:58.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi116:172.21.15.116=b;smithi117:172.21.15.117=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:31:58.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi116:172.21.15.116=b;smithi117:172.21.15.117=c;count:3 2024-03-24T10:31:58.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:31:58.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:31:58.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:31:58.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:31:58.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:31:58.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:31:58.739 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: Deploying daemon mon.c on smithi117 2024-03-24T10:31:58.739 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:58.739 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: from='client.? 172.21.15.117:0/3476100889' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:31:58.739 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:31:58.739 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 ceph-mon[36368]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T10:31:58.739 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 bash[36349]: 08756fdf33ea003f76ab7a05a3dc11a54c7643ca45220adbb4369f6a326c9f78 2024-03-24T10:31:58.739 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:31:58 smithi117 systemd[1]: Started Ceph mon.c for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:32:03.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.323+0000 7fb4a95b7700 1 -- 172.21.15.117:0/1358311808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb48c0035c0 msgr2=0x7fb48c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:03.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.323+0000 7fb4a95b7700 1 --2- 172.21.15.117:0/1358311808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb48c0035c0 0x7fb48c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb4a406a4c0 tx=0x7fb49c009e80 comp rx=0 tx=0).stop 2024-03-24T10:32:03.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.323+0000 7fb4a95b7700 1 -- 172.21.15.117:0/1358311808 shutdown_connections 2024-03-24T10:32:03.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.323+0000 7fb4a95b7700 1 --2- 172.21.15.117:0/1358311808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb48c0035c0 0x7fb48c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:03.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.323+0000 7fb4a95b7700 1 -- 172.21.15.117:0/1358311808 >> 172.21.15.117:0/1358311808 conn(0x7fb4a40fc6a0 msgr2=0x7fb4a40feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:32:03.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.323+0000 7fb4a95b7700 1 -- 172.21.15.117:0/1358311808 shutdown_connections 2024-03-24T10:32:03.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.323+0000 7fb4a95b7700 1 -- 172.21.15.117:0/1358311808 wait complete. 2024-03-24T10:32:03.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.325+0000 7fb4a95b7700 1 Processor -- start 2024-03-24T10:32:03.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.325+0000 7fb4a95b7700 1 -- start start 2024-03-24T10:32:03.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.325+0000 7fb4a95b7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb48c0035c0 0x7fb4a4100af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:03.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.325+0000 7fb4a95b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4a4101030 0x7fb4a41014b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:03.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.325+0000 7fb4a95b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a41029e0 con 0x7fb4a4101030 2024-03-24T10:32:03.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.325+0000 7fb4a95b7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a41a37f0 con 0x7fb48c0035c0 2024-03-24T10:32:03.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb4a2ffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb48c0035c0 0x7fb4a4100af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:03.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb4a2ffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb48c0035c0 0x7fb4a4100af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:57416/0 (socket says 172.21.15.117:57416) 2024-03-24T10:32:03.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb4a27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4a4101030 0x7fb4a41014b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:03.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb4a2ffd700 1 -- 172.21.15.117:0/3362596294 learned_addr learned my addr 172.21.15.117:0/3362596294 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T10:32:03.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb4a2ffd700 1 -- 172.21.15.117:0/3362596294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb48c0035c0 msgr2=0x7fb4a4100af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T10:32:03.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb4a2ffd700 1 -- 172.21.15.117:0/3362596294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb48c0035c0 msgr2=0x7fb4a4100af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T10:32:03.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb4a2ffd700 1 --2- 172.21.15.117:0/3362596294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb48c0035c0 0x7fb4a4100af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:32:03.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb4a2ffd700 1 --2- 172.21.15.117:0/3362596294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb48c0035c0 0x7fb4a4100af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:32:03.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb4a27fc700 1 -- 172.21.15.117:0/3362596294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb48c0035c0 msgr2=0x7fb4a4100af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:32:03.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb4a27fc700 1 --2- 172.21.15.117:0/3362596294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb48c0035c0 0x7fb4a4100af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:03.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb4a27fc700 1 -- 172.21.15.117:0/3362596294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb49c0098d0 con 0x7fb4a4101030 2024-03-24T10:32:03.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb4a27fc700 1 --2- 172.21.15.117:0/3362596294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4a4101030 0x7fb4a41014b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb49800b4d0 tx=0x7fb49800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:32:03.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb48bfff700 1 -- 172.21.15.117:0/3362596294 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb498011630 con 0x7fb4a4101030 2024-03-24T10:32:03.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb48bfff700 1 -- 172.21.15.117:0/3362596294 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb498011c70 con 0x7fb4a4101030 2024-03-24T10:32:03.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.326+0000 7fb48bfff700 1 -- 172.21.15.117:0/3362596294 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb498010de0 con 0x7fb4a4101030 2024-03-24T10:32:03.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.327+0000 7fb4a95b7700 1 -- 172.21.15.117:0/3362596294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a41a3990 con 0x7fb4a4101030 2024-03-24T10:32:03.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.327+0000 7fb4a95b7700 1 -- 172.21.15.117:0/3362596294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a41a3eb0 con 0x7fb4a4101030 2024-03-24T10:32:03.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.327+0000 7fb48bfff700 1 -- 172.21.15.117:0/3362596294 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fb49801a400 con 0x7fb4a4101030 2024-03-24T10:32:03.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.328+0000 7fb4a95b7700 1 -- 172.21.15.117:0/3362596294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4a4063830 con 0x7fb4a4101030 2024-03-24T10:32:03.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.328+0000 7fb48bfff700 1 --2- 172.21.15.117:0/3362596294 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb490038d40 0x7fb49003b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:03.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.328+0000 7fb48bfff700 1 -- 172.21.15.117:0/3362596294 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb49804b730 con 0x7fb4a4101030 2024-03-24T10:32:03.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.328+0000 7fb4a2ffd700 1 --2- 172.21.15.117:0/3362596294 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb490038d40 0x7fb49003b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:03.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.329+0000 7fb4a2ffd700 1 --2- 172.21.15.117:0/3362596294 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb490038d40 0x7fb49003b200 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb49c000f80 tx=0x7fb49c007cd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:32:03.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.333+0000 7fb48bfff700 1 -- 172.21.15.117:0/3362596294 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb4980174a0 con 0x7fb4a4101030 2024-03-24T10:32:03.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.612+0000 7fb4a95b7700 1 -- 172.21.15.117:0/3362596294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fb4a4102020 con 0x7fb4a4101030 2024-03-24T10:32:03.613 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:32:03.614 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: mon.a calling monitor election 2024-03-24T10:32:03.614 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:03.614 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:03.614 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:03.614 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:03.614 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: mon.c calling monitor election 2024-03-24T10:32:03.614 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:03.614 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:03.614 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:03.614 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:03.614 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: monmap epoch 2 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: last_changed 2024-03-24T10:31:58.269625+0000 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: created 2024-03-24T10:30:17.512891+0000 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: min_mon_release 19 (squid) 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: election_strategy: 1 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: 0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: fsmap 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: mgrmap e13: x(active, since 48s) 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: overall HEALTH_OK 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:03.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:03.616 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:32:03.616 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:03.616 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.613+0000 7fb48bfff700 1 -- 172.21.15.117:0/3362596294 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fb4980172c0 con 0x7fb4a4101030 2024-03-24T10:32:03.616 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:32:03.616 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":2,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:31:58.269625Z","created":"2024-03-24T10:30:17.512891Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T10:32:03.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.615+0000 7fb4a95b7700 1 -- 172.21.15.117:0/3362596294 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb490038d40 msgr2=0x7fb49003b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:03.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.615+0000 7fb4a95b7700 1 --2- 172.21.15.117:0/3362596294 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb490038d40 0x7fb49003b200 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb49c000f80 tx=0x7fb49c007cd0 comp rx=0 tx=0).stop 2024-03-24T10:32:03.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.615+0000 7fb4a95b7700 1 -- 172.21.15.117:0/3362596294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4a4101030 msgr2=0x7fb4a41014b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:03.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.615+0000 7fb4a95b7700 1 --2- 172.21.15.117:0/3362596294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4a4101030 0x7fb4a41014b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb49800b4d0 tx=0x7fb49800b890 comp rx=0 tx=0).stop 2024-03-24T10:32:03.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.615+0000 7fb4a95b7700 1 -- 172.21.15.117:0/3362596294 shutdown_connections 2024-03-24T10:32:03.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.615+0000 7fb4a95b7700 1 --2- 172.21.15.117:0/3362596294 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb490038d40 0x7fb49003b200 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:03.617 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.615+0000 7fb4a95b7700 1 --2- 172.21.15.117:0/3362596294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb48c0035c0 0x7fb4a4100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:03.618 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.615+0000 7fb4a95b7700 1 --2- 172.21.15.117:0/3362596294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4a4101030 0x7fb4a41014b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:03.618 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.615+0000 7fb4a95b7700 1 -- 172.21.15.117:0/3362596294 >> 172.21.15.117:0/3362596294 conn(0x7fb4a40fc6a0 msgr2=0x7fb4a4103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:32:03.618 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.615+0000 7fb4a95b7700 1 -- 172.21.15.117:0/3362596294 shutdown_connections 2024-03-24T10:32:03.618 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:03.615+0000 7fb4a95b7700 1 -- 172.21.15.117:0/3362596294 wait complete. 2024-03-24T10:32:03.618 INFO:teuthology.orchestra.run.smithi117.stderr:dumped monmap epoch 2 2024-03-24T10:32:03.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:32:03.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: mon.a calling monitor election 2024-03-24T10:32:03.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:03.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: mon.c calling monitor election 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: monmap epoch 2 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: last_changed 2024-03-24T10:31:58.269625+0000 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: created 2024-03-24T10:30:17.512891+0000 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: min_mon_release 19 (squid) 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: election_strategy: 1 2024-03-24T10:32:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: 0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2024-03-24T10:32:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-03-24T10:32:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: fsmap 2024-03-24T10:32:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T10:32:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: mgrmap e13: x(active, since 48s) 2024-03-24T10:32:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: overall HEALTH_OK 2024-03-24T10:32:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:32:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:04.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:04 smithi117 ceph-mon[36368]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:04.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:04 smithi117 ceph-mon[36368]: Deploying daemon mon.b on smithi116 2024-03-24T10:32:04.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:04 smithi117 ceph-mon[36368]: from='client.? 172.21.15.117:0/3362596294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:32:04.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:04 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:04.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:04 smithi023 ceph-mon[32466]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:04.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:04 smithi023 ceph-mon[32466]: Deploying daemon mon.b on smithi116 2024-03-24T10:32:04.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:04 smithi023 ceph-mon[32466]: from='client.? 172.21.15.117:0/3362596294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:32:04.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:04 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:05.219 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T10:32:05.219 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:32:05.466 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:32:06.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:06 smithi023 ceph-mon[32466]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:06.687 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:06 smithi117 ceph-mon[36368]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:06.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.981+0000 7f9f551d3700 1 -- 172.21.15.117:0/2483318418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f50100e60 msgr2=0x7f9f50101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:06.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.981+0000 7f9f551d3700 1 --2- 172.21.15.117:0/2483318418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f50100e60 0x7f9f50101240 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9f48009910 tx=0x7f9f48009c20 comp rx=0 tx=0).stop 2024-03-24T10:32:06.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.981+0000 7f9f551d3700 1 -- 172.21.15.117:0/2483318418 shutdown_connections 2024-03-24T10:32:06.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.981+0000 7f9f551d3700 1 --2- 172.21.15.117:0/2483318418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f50100e60 0x7f9f50101240 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:06.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.981+0000 7f9f551d3700 1 -- 172.21.15.117:0/2483318418 >> 172.21.15.117:0/2483318418 conn(0x7f9f500fc710 msgr2=0x7f9f500feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:32:06.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.981+0000 7f9f551d3700 1 -- 172.21.15.117:0/2483318418 shutdown_connections 2024-03-24T10:32:06.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.981+0000 7f9f551d3700 1 -- 172.21.15.117:0/2483318418 wait complete. 2024-03-24T10:32:06.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.981+0000 7f9f551d3700 1 Processor -- start 2024-03-24T10:32:06.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f551d3700 1 -- start start 2024-03-24T10:32:06.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f551d3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f50100e60 0x7f9f5019d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:06.983 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f551d3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f5019daf0 0x7f9f50197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f551d3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5019e010 con 0x7f9f50100e60 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f551d3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9f50197bc0 con 0x7f9f5019daf0 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f4e59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f5019daf0 0x7f9f50197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f4ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f50100e60 0x7f9f5019d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f4e59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f5019daf0 0x7f9f50197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:57418/0 (socket says 172.21.15.117:57418) 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f4e59c700 1 -- 172.21.15.117:0/2172545579 learned_addr learned my addr 172.21.15.117:0/2172545579 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f4e59c700 1 -- 172.21.15.117:0/2172545579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f5019daf0 msgr2=0x7f9f50197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f4e59c700 1 -- 172.21.15.117:0/2172545579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f5019daf0 msgr2=0x7f9f50197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f4e59c700 1 --2- 172.21.15.117:0/2172545579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f5019daf0 0x7f9f50197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f4e59c700 1 --2- 172.21.15.117:0/2172545579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f5019daf0 0x7f9f50197680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f4ed9d700 1 -- 172.21.15.117:0/2172545579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f5019daf0 msgr2=0x7f9f50197680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f4ed9d700 1 --2- 172.21.15.117:0/2172545579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f5019daf0 0x7f9f50197680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.982+0000 7f9f4ed9d700 1 -- 172.21.15.117:0/2172545579 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f480095f0 con 0x7f9f50100e60 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.983+0000 7f9f4ed9d700 1 --2- 172.21.15.117:0/2172545579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f50100e60 0x7f9f5019d5b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9f48014000 tx=0x7f9f48004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:32:06.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.983+0000 7f9f47fff700 1 -- 172.21.15.117:0/2172545579 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9f4801d070 con 0x7f9f50100e60 2024-03-24T10:32:06.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.983+0000 7f9f47fff700 1 -- 172.21.15.117:0/2172545579 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f48005dc0 con 0x7f9f50100e60 2024-03-24T10:32:06.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.983+0000 7f9f551d3700 1 -- 172.21.15.117:0/2172545579 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f50197df0 con 0x7f9f50100e60 2024-03-24T10:32:06.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.983+0000 7f9f47fff700 1 -- 172.21.15.117:0/2172545579 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9f48003880 con 0x7f9f50100e60 2024-03-24T10:32:06.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.983+0000 7f9f551d3700 1 -- 172.21.15.117:0/2172545579 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f501982e0 con 0x7f9f50100e60 2024-03-24T10:32:06.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.984+0000 7f9f551d3700 1 -- 172.21.15.117:0/2172545579 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f50198550 con 0x7f9f50100e60 2024-03-24T10:32:06.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.984+0000 7f9f47fff700 1 -- 172.21.15.117:0/2172545579 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f9f48003af0 con 0x7f9f50100e60 2024-03-24T10:32:06.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.984+0000 7f9f47fff700 1 --2- 172.21.15.117:0/2172545579 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9f3c038d20 0x7f9f3c03b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:06.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.984+0000 7f9f47fff700 1 -- 172.21.15.117:0/2172545579 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9f4804cec0 con 0x7f9f50100e60 2024-03-24T10:32:06.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.984+0000 7f9f4e59c700 1 --2- 172.21.15.117:0/2172545579 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9f3c038d20 0x7f9f3c03b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:06.986 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.985+0000 7f9f4e59c700 1 --2- 172.21.15.117:0/2172545579 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9f3c038d20 0x7f9f3c03b1e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009180 tx=0x7f9f38008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:32:06.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:06.989+0000 7f9f47fff700 1 -- 172.21.15.117:0/2172545579 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9f4802ae20 con 0x7f9f50100e60 2024-03-24T10:32:07.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:07.257+0000 7f9f551d3700 1 -- 172.21.15.117:0/2172545579 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9f50198bb0 con 0x7f9f50100e60 2024-03-24T10:32:07.259 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:07.258+0000 7f9f47fff700 1 -- 172.21.15.117:0/2172545579 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f9f4802ac40 con 0x7f9f50100e60 2024-03-24T10:32:07.259 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:32:07.260 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":2,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:31:58.269625Z","created":"2024-03-24T10:30:17.512891Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T10:32:07.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:07.260+0000 7f9f551d3700 1 -- 172.21.15.117:0/2172545579 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9f3c038d20 msgr2=0x7f9f3c03b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:07.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:07.260+0000 7f9f551d3700 1 --2- 172.21.15.117:0/2172545579 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9f3c038d20 0x7f9f3c03b1e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009180 tx=0x7f9f38008040 comp rx=0 tx=0).stop 2024-03-24T10:32:07.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:07.260+0000 7f9f551d3700 1 -- 172.21.15.117:0/2172545579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f50100e60 msgr2=0x7f9f5019d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:07.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:07.260+0000 7f9f551d3700 1 --2- 172.21.15.117:0/2172545579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f50100e60 0x7f9f5019d5b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9f48014000 tx=0x7f9f48004ff0 comp rx=0 tx=0).stop 2024-03-24T10:32:07.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:07.260+0000 7f9f551d3700 1 -- 172.21.15.117:0/2172545579 shutdown_connections 2024-03-24T10:32:07.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:07.260+0000 7f9f551d3700 1 --2- 172.21.15.117:0/2172545579 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9f3c038d20 0x7f9f3c03b1e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:07.261 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:07.260+0000 7f9f551d3700 1 --2- 172.21.15.117:0/2172545579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f50100e60 0x7f9f5019d5b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:07.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:07.260+0000 7f9f551d3700 1 --2- 172.21.15.117:0/2172545579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f5019daf0 0x7f9f50197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:07.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:07.260+0000 7f9f551d3700 1 -- 172.21.15.117:0/2172545579 >> 172.21.15.117:0/2172545579 conn(0x7f9f500fc710 msgr2=0x7f9f5006a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:32:07.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:07.260+0000 7f9f551d3700 1 -- 172.21.15.117:0/2172545579 shutdown_connections 2024-03-24T10:32:07.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:07.260+0000 7f9f551d3700 1 -- 172.21.15.117:0/2172545579 wait complete. 2024-03-24T10:32:07.262 INFO:teuthology.orchestra.run.smithi117.stderr:dumped monmap epoch 2 2024-03-24T10:32:07.370 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T10:32:07.370 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T10:32:07.370 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T10:32:07.370 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T10:32:07.370 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T10:32:07.370 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T10:32:07.371 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T10:32:07.371 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:32:07.371 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:32:07.371 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T10:32:07.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:07.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi117", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:32:07.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:32:07.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:07.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:07.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:07.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T10:32:07.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: Deploying cephadm binary to smithi117 2024-03-24T10:32:07.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:32:07.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:32:07.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:07.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:32:07.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:07.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:07.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: Added host smithi117 2024-03-24T10:32:07.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:07.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:32:07.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:07.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2919387357' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T10:32:07.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2919387357' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T10:32:07.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi117", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:32:07.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi116:172.21.15.116=b;smithi117:172.21.15.117=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi116:172.21.15.116=b;smithi117:172.21.15.117=c;count:3 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: Deploying daemon mon.c on smithi117 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='client.? 172.21.15.117:0/3476100889' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:07.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:32:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mon.a calling monitor election 2024-03-24T10:32:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:07.773 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:07.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:07.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:07.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mon.c calling monitor election 2024-03-24T10:32:07.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:07.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:07.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:07.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:07.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T10:32:07.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: monmap epoch 2 2024-03-24T10:32:07.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:32:07.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: last_changed 2024-03-24T10:31:58.269625+0000 2024-03-24T10:32:07.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: created 2024-03-24T10:30:17.512891+0000 2024-03-24T10:32:07.774 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: min_mon_release 19 (squid) 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: election_strategy: 1 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: 0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: fsmap 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mgrmap e13: x(active, since 48s) 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: overall HEALTH_OK 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: Deploying daemon mon.b on smithi116 2024-03-24T10:32:07.775 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='client.? 172.21.15.117:0/3362596294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T10:32:07.776 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:07.776 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:07.776 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 ceph-mon[36248]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T10:32:07.776 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 bash[36229]: c9b7a171dc13f415698635351432a1892e06b7e154809f17150bcdcea2e56ee4 2024-03-24T10:32:07.776 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:07 smithi116 systemd[1]: Started Ceph mon.b for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:32:08.861 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T10:32:08.862 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T10:32:09.107 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:32:12.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:12.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:32:12.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: mon.c calling monitor election 2024-03-24T10:32:12.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: mon.a calling monitor election 2024-03-24T10:32:12.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:12.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:12.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:12.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:12.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:12.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:12.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:12.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:12.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:12.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T10:32:12.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: monmap epoch 3 2024-03-24T10:32:12.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:32:12.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: last_changed 2024-03-24T10:32:07.375148+0000 2024-03-24T10:32:12.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: created 2024-03-24T10:30:17.512891+0000 2024-03-24T10:32:12.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: min_mon_release 19 (squid) 2024-03-24T10:32:12.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: election_strategy: 1 2024-03-24T10:32:12.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: 0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2024-03-24T10:32:12.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-03-24T10:32:12.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-03-24T10:32:12.758 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: fsmap 2024-03-24T10:32:12.758 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T10:32:12.759 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: mgrmap e13: x(active, since 57s) 2024-03-24T10:32:12.759 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: overall HEALTH_OK 2024-03-24T10:32:12.759 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:12.759 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:12.759 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:12.759 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:12.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:12.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:32:12.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: mon.c calling monitor election 2024-03-24T10:32:12.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: mon.a calling monitor election 2024-03-24T10:32:12.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:12.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T10:32:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: monmap epoch 3 2024-03-24T10:32:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:32:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: last_changed 2024-03-24T10:32:07.375148+0000 2024-03-24T10:32:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: created 2024-03-24T10:30:17.512891+0000 2024-03-24T10:32:12.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: min_mon_release 19 (squid) 2024-03-24T10:32:12.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: election_strategy: 1 2024-03-24T10:32:12.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: 0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2024-03-24T10:32:12.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-03-24T10:32:12.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-03-24T10:32:12.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: fsmap 2024-03-24T10:32:12.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T10:32:12.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: mgrmap e13: x(active, since 57s) 2024-03-24T10:32:12.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: overall HEALTH_OK 2024-03-24T10:32:12.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:12.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:12.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:12.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:13.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.501+0000 7fe27ae55700 1 -- 172.21.15.117:0/3318366226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2640035c0 msgr2=0x7fe264005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:13.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.501+0000 7fe27ae55700 1 --2- 172.21.15.117:0/3318366226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2640035c0 0x7fe264005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe2741014a0 tx=0x7fe26800fe60 comp rx=0 tx=0).stop 2024-03-24T10:32:13.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.501+0000 7fe27ae55700 1 -- 172.21.15.117:0/3318366226 shutdown_connections 2024-03-24T10:32:13.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.501+0000 7fe27ae55700 1 --2- 172.21.15.117:0/3318366226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2640035c0 0x7fe264005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:13.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.501+0000 7fe27ae55700 1 -- 172.21.15.117:0/3318366226 >> 172.21.15.117:0/3318366226 conn(0x7fe2740771f0 msgr2=0x7fe274077600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:32:13.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.501+0000 7fe27ae55700 1 -- 172.21.15.117:0/3318366226 shutdown_connections 2024-03-24T10:32:13.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.501+0000 7fe27ae55700 1 -- 172.21.15.117:0/3318366226 wait complete. 2024-03-24T10:32:13.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.502+0000 7fe27ae55700 1 Processor -- start 2024-03-24T10:32:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.502+0000 7fe27ae55700 1 -- start start 2024-03-24T10:32:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.502+0000 7fe27ae55700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2640035c0 0x7fe2741015f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.502+0000 7fe27ae55700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe274101b30 0x7fe274101fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.502+0000 7fe27ae55700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe2741a3980 0x7fe2741a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.502+0000 7fe27ae55700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe2741a6370 con 0x7fe274101b30 2024-03-24T10:32:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.502+0000 7fe27ae55700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe2741a64e0 con 0x7fe2640035c0 2024-03-24T10:32:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.502+0000 7fe27ae55700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe2741a66a0 con 0x7fe2741a3980 2024-03-24T10:32:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.502+0000 7fe2793f2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe2741a3980 0x7fe2741a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.502+0000 7fe2793f2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe2741a3980 0x7fe2741a5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39484/0 (socket says 172.21.15.117:39484) 2024-03-24T10:32:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.502+0000 7fe278bf1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2640035c0 0x7fe2741015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.502+0000 7fe2793f2700 1 -- 172.21.15.117:0/4201245582 learned_addr learned my addr 172.21.15.117:0/4201245582 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T10:32:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.502+0000 7fe273fff700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe274101b30 0x7fe274101fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe278bf1700 1 -- 172.21.15.117:0/4201245582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2640035c0 msgr2=0x7fe2741015f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T10:32:13.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe278bf1700 1 -- 172.21.15.117:0/4201245582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2640035c0 msgr2=0x7fe2741015f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T10:32:13.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe278bf1700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2640035c0 0x7fe2741015f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:32:13.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe278bf1700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2640035c0 0x7fe2741015f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:32:13.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe2793f2700 1 -- 172.21.15.117:0/4201245582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2640035c0 msgr2=0x7fe2741015f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:32:13.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe2793f2700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2640035c0 0x7fe2741015f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:13.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe2793f2700 1 -- 172.21.15.117:0/4201245582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe274101b30 msgr2=0x7fe274101fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:13.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe2793f2700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe274101b30 0x7fe274101fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:13.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe2793f2700 1 -- 172.21.15.117:0/4201245582 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26800f8b0 con 0x7fe2741a3980 2024-03-24T10:32:13.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe273fff700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe274101b30 0x7fe274101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:32:13.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe2793f2700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe2741a3980 0x7fe2741a5d70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe26c00ba50 tx=0x7fe26c00bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:32:13.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe271ffb700 1 -- 172.21.15.117:0/4201245582 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c00c9b0 con 0x7fe2741a3980 2024-03-24T10:32:13.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe27ae55700 1 -- 172.21.15.117:0/4201245582 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2741a6980 con 0x7fe2741a3980 2024-03-24T10:32:13.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe271ffb700 1 -- 172.21.15.117:0/4201245582 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe26c014440 con 0x7fe2741a3980 2024-03-24T10:32:13.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe271ffb700 1 -- 172.21.15.117:0/4201245582 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c0075b0 con 0x7fe2741a3980 2024-03-24T10:32:13.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.503+0000 7fe27ae55700 1 -- 172.21.15.117:0/4201245582 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2741a6e80 con 0x7fe2741a3980 2024-03-24T10:32:13.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.504+0000 7fe271ffb700 1 -- 172.21.15.117:0/4201245582 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fe26c007710 con 0x7fe2741a3980 2024-03-24T10:32:13.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.504+0000 7fe271ffb700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe258039410 0x7fe25803b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:13.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.504+0000 7fe271ffb700 1 -- 172.21.15.117:0/4201245582 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe26c04c480 con 0x7fe2741a3980 2024-03-24T10:32:13.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.505+0000 7fe27ae55700 1 -- 172.21.15.117:0/4201245582 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe274063830 con 0x7fe2741a3980 2024-03-24T10:32:13.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.505+0000 7fe278bf1700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe258039410 0x7fe25803b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:13.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.505+0000 7fe278bf1700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe258039410 0x7fe25803b8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe268001eb0 tx=0x7fe268005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:32:13.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.510+0000 7fe271ffb700 1 -- 172.21.15.117:0/4201245582 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe26c01b890 con 0x7fe2741a3980 2024-03-24T10:32:13.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:13 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:13.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:13 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:13.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.785+0000 7fe27ae55700 1 -- 172.21.15.117:0/4201245582 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe274109b80 con 0x7fe2741a3980 2024-03-24T10:32:13.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.785+0000 7fe271ffb700 1 -- 172.21.15.117:0/4201245582 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7fe26c01b2c0 con 0x7fe2741a3980 2024-03-24T10:32:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T10:32:13.791 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":3,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","modified":"2024-03-24T10:32:07.375148Z","created":"2024-03-24T10:30:17.512891Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T10:32:13.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.787+0000 7fe27ae55700 1 -- 172.21.15.117:0/4201245582 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe258039410 msgr2=0x7fe25803b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:13.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.787+0000 7fe27ae55700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe258039410 0x7fe25803b8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe268001eb0 tx=0x7fe268005e30 comp rx=0 tx=0).stop 2024-03-24T10:32:13.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.787+0000 7fe27ae55700 1 -- 172.21.15.117:0/4201245582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe2741a3980 msgr2=0x7fe2741a5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:13.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.787+0000 7fe27ae55700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe2741a3980 0x7fe2741a5d70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe26c00ba50 tx=0x7fe26c00bd60 comp rx=0 tx=0).stop 2024-03-24T10:32:13.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.787+0000 7fe27ae55700 1 -- 172.21.15.117:0/4201245582 shutdown_connections 2024-03-24T10:32:13.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.787+0000 7fe27ae55700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe258039410 0x7fe25803b8d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:13.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.787+0000 7fe27ae55700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe2741a3980 0x7fe2741a5d70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:13.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.787+0000 7fe27ae55700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2640035c0 0x7fe2741015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:13.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.787+0000 7fe27ae55700 1 --2- 172.21.15.117:0/4201245582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe274101b30 0x7fe274101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:13.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.787+0000 7fe27ae55700 1 -- 172.21.15.117:0/4201245582 >> 172.21.15.117:0/4201245582 conn(0x7fe2740771f0 msgr2=0x7fe2740fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:32:13.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.788+0000 7fe27ae55700 1 -- 172.21.15.117:0/4201245582 shutdown_connections 2024-03-24T10:32:13.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:13.788+0000 7fe27ae55700 1 -- 172.21.15.117:0/4201245582 wait complete. 2024-03-24T10:32:13.793 INFO:teuthology.orchestra.run.smithi117.stderr:dumped monmap epoch 3 2024-03-24T10:32:13.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:13 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:13.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:13 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:14.453 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T10:32:14.453 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T10:32:14.706 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:32:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:32:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: mon.c calling monitor election 2024-03-24T10:32:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: mon.a calling monitor election 2024-03-24T10:32:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:32:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:14.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:14.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:14.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:14.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:14.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:14.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:14.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T10:32:14.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: monmap epoch 3 2024-03-24T10:32:14.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:32:14.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: last_changed 2024-03-24T10:32:07.375148+0000 2024-03-24T10:32:14.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: created 2024-03-24T10:30:17.512891+0000 2024-03-24T10:32:14.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: min_mon_release 19 (squid) 2024-03-24T10:32:14.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: election_strategy: 1 2024-03-24T10:32:14.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: 0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2024-03-24T10:32:14.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-03-24T10:32:14.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-03-24T10:32:14.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: fsmap 2024-03-24T10:32:14.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T10:32:14.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: mgrmap e13: x(active, since 57s) 2024-03-24T10:32:14.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: overall HEALTH_OK 2024-03-24T10:32:14.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:14.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:14.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:14.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:14.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:14.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:15.405 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: mon.b calling monitor election 2024-03-24T10:32:15.405 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-24T10:32:15.405 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T10:32:15.405 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T10:32:15.405 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: Updating smithi023:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:32:15.405 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:32:15.405 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:32:15.405 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: mon.b calling monitor election 2024-03-24T10:32:15.405 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: mon.c calling monitor election 2024-03-24T10:32:15.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: mon.a calling monitor election 2024-03-24T10:32:15.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T10:32:15.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: monmap epoch 3 2024-03-24T10:32:15.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:32:15.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: last_changed 2024-03-24T10:32:07.375148+0000 2024-03-24T10:32:15.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: created 2024-03-24T10:30:17.512891+0000 2024-03-24T10:32:15.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: min_mon_release 19 (squid) 2024-03-24T10:32:15.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: election_strategy: 1 2024-03-24T10:32:15.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: 0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2024-03-24T10:32:15.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-03-24T10:32:15.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-03-24T10:32:15.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: fsmap 2024-03-24T10:32:15.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T10:32:15.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: mgrmap e13: x(active, since 59s) 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: overall HEALTH_OK 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:15.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: Reconfiguring daemon mon.a on smithi023 2024-03-24T10:32:15.408 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: mon.b calling monitor election 2024-03-24T10:32:15.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-24T10:32:15.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T10:32:15.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T10:32:15.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: Updating smithi023:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:32:15.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:32:15.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:32:15.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: mon.b calling monitor election 2024-03-24T10:32:15.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: mon.c calling monitor election 2024-03-24T10:32:15.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: mon.a calling monitor election 2024-03-24T10:32:15.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T10:32:15.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: monmap epoch 3 2024-03-24T10:32:15.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:32:15.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: last_changed 2024-03-24T10:32:07.375148+0000 2024-03-24T10:32:15.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: created 2024-03-24T10:30:17.512891+0000 2024-03-24T10:32:15.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: min_mon_release 19 (squid) 2024-03-24T10:32:15.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: election_strategy: 1 2024-03-24T10:32:15.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: 0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: fsmap 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: mgrmap e13: x(active, since 59s) 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: overall HEALTH_OK 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T10:32:15.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:32:15.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T10:32:15.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:15.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: Reconfiguring daemon mon.a on smithi023 2024-03-24T10:32:15.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: mon.b calling monitor election 2024-03-24T10:32:15.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-24T10:32:15.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T10:32:15.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T10:32:15.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: Updating smithi023:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:32:15.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:32:15.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:32:15.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: mon.b calling monitor election 2024-03-24T10:32:15.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: mon.c calling monitor election 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: mon.a calling monitor election 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: monmap epoch 3 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: last_changed 2024-03-24T10:32:07.375148+0000 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: created 2024-03-24T10:30:17.512891+0000 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: min_mon_release 19 (squid) 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: election_strategy: 1 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: 0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: fsmap 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: mgrmap e13: x(active, since 59s) 2024-03-24T10:32:15.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: overall HEALTH_OK 2024-03-24T10:32:15.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:15.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T10:32:15.771 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:32:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T10:32:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: Reconfiguring daemon mon.a on smithi023 2024-03-24T10:32:15.772 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:16.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.259+0000 7f8478d76700 1 -- 172.21.15.23:0/1645757656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8474101e40 msgr2=0x7f8474102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:16.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.259+0000 7f8478d76700 1 --2- 172.21.15.23:0/1645757656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8474101e40 0x7f8474102220 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8464009910 tx=0x7f8464009c20 comp rx=0 tx=0).stop 2024-03-24T10:32:16.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.259+0000 7f8478d76700 1 -- 172.21.15.23:0/1645757656 shutdown_connections 2024-03-24T10:32:16.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.259+0000 7f8478d76700 1 --2- 172.21.15.23:0/1645757656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8474101e40 0x7f8474102220 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:16.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.259+0000 7f8478d76700 1 -- 172.21.15.23:0/1645757656 >> 172.21.15.23:0/1645757656 conn(0x7f84740fd710 msgr2=0x7f84740ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:32:16.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.259+0000 7f8478d76700 1 -- 172.21.15.23:0/1645757656 shutdown_connections 2024-03-24T10:32:16.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.259+0000 7f8478d76700 1 -- 172.21.15.23:0/1645757656 wait complete. 2024-03-24T10:32:16.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f8478d76700 1 Processor -- start 2024-03-24T10:32:16.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f8478d76700 1 -- start start 2024-03-24T10:32:16.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f8478d76700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8474101e40 0x7f847419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:16.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f8478d76700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f847419fbe0 0x7f8474199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:16.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f8478d76700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8474199d60 0x7f847419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:16.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f8478d76700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f84741a0250 con 0x7f8474101e40 2024-03-24T10:32:16.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f8478d76700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f84741a6500 con 0x7f8474199d60 2024-03-24T10:32:16.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f8478d76700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f84741a6670 con 0x7f847419fbe0 2024-03-24T10:32:16.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f847259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8474101e40 0x7f847419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:16.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f847259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8474101e40 0x7f847419f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55474/0 (socket says 172.21.15.23:55474) 2024-03-24T10:32:16.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f847259c700 1 -- 172.21.15.23:0/2970379474 learned_addr learned my addr 172.21.15.23:0/2970379474 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:32:16.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f8472d9d700 1 --2- 172.21.15.23:0/2970379474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8474199d60 0x7f847419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:16.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f847259c700 1 -- 172.21.15.23:0/2970379474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f847419fbe0 msgr2=0x7f8474199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:16.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f847259c700 1 --2- 172.21.15.23:0/2970379474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f847419fbe0 0x7f8474199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:16.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f847259c700 1 -- 172.21.15.23:0/2970379474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8474199d60 msgr2=0x7f847419a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:16.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f847259c700 1 --2- 172.21.15.23:0/2970379474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8474199d60 0x7f847419a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:16.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f847259c700 1 -- 172.21.15.23:0/2970379474 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84640095f0 con 0x7f8474101e40 2024-03-24T10:32:16.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.260+0000 7f8472d9d700 1 --2- 172.21.15.23:0/2970379474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8474199d60 0x7f847419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:32:16.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.261+0000 7f847259c700 1 --2- 172.21.15.23:0/2970379474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8474101e40 0x7f847419f6a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8464005b40 tx=0x7f846400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:32:16.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.261+0000 7f84637fe700 1 -- 172.21.15.23:0/2970379474 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846401d070 con 0x7f8474101e40 2024-03-24T10:32:16.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.261+0000 7f84637fe700 1 -- 172.21.15.23:0/2970379474 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f846400fbd0 con 0x7f8474101e40 2024-03-24T10:32:16.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.261+0000 7f8478d76700 1 -- 172.21.15.23:0/2970379474 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84741a68f0 con 0x7f8474101e40 2024-03-24T10:32:16.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.261+0000 7f84637fe700 1 -- 172.21.15.23:0/2970379474 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8464022d40 con 0x7f8474101e40 2024-03-24T10:32:16.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.261+0000 7f8478d76700 1 -- 172.21.15.23:0/2970379474 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84741a6d30 con 0x7f8474101e40 2024-03-24T10:32:16.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.262+0000 7f84637fe700 1 -- 172.21.15.23:0/2970379474 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f846400f6f0 con 0x7f8474101e40 2024-03-24T10:32:16.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.262+0000 7f8478d76700 1 -- 172.21.15.23:0/2970379474 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8474103320 con 0x7f8474101e40 2024-03-24T10:32:16.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.262+0000 7f84637fe700 1 --2- 172.21.15.23:0/2970379474 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f845c039410 0x7f845c03b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:16.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.262+0000 7f84637fe700 1 -- 172.21.15.23:0/2970379474 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f846404d0a0 con 0x7f8474101e40 2024-03-24T10:32:16.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.262+0000 7f8471d9b700 1 --2- 172.21.15.23:0/2970379474 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f845c039410 0x7f845c03b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:16.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.263+0000 7f8471d9b700 1 --2- 172.21.15.23:0/2970379474 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f845c039410 0x7f845c03b8d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f84680098e0 tx=0x7f8468006dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:32:16.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.267+0000 7f84637fe700 1 -- 172.21.15.23:0/2970379474 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f846402a360 con 0x7f8474101e40 2024-03-24T10:32:16.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.476+0000 7f8478d76700 1 -- 172.21.15.23:0/2970379474 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f847410c0c0 con 0x7f8474101e40 2024-03-24T10:32:16.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.476+0000 7f84637fe700 1 -- 172.21.15.23:0/2970379474 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f8464027020 con 0x7f8474101e40 2024-03-24T10:32:16.477 INFO:teuthology.orchestra.run.smithi023.stdout:# minimal ceph.conf for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:32:16.477 INFO:teuthology.orchestra.run.smithi023.stdout:[global] 2024-03-24T10:32:16.477 INFO:teuthology.orchestra.run.smithi023.stdout: fsid = 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:32:16.477 INFO:teuthology.orchestra.run.smithi023.stdout: mon_host = [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-03-24T10:32:16.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.478+0000 7f8478d76700 1 -- 172.21.15.23:0/2970379474 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f845c039410 msgr2=0x7f845c03b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:16.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.478+0000 7f8478d76700 1 --2- 172.21.15.23:0/2970379474 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f845c039410 0x7f845c03b8d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f84680098e0 tx=0x7f8468006dd0 comp rx=0 tx=0).stop 2024-03-24T10:32:16.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.478+0000 7f8478d76700 1 -- 172.21.15.23:0/2970379474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8474101e40 msgr2=0x7f847419f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:16.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.478+0000 7f8478d76700 1 --2- 172.21.15.23:0/2970379474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8474101e40 0x7f847419f6a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8464005b40 tx=0x7f846400bda0 comp rx=0 tx=0).stop 2024-03-24T10:32:16.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.478+0000 7f8478d76700 1 -- 172.21.15.23:0/2970379474 shutdown_connections 2024-03-24T10:32:16.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.478+0000 7f8478d76700 1 --2- 172.21.15.23:0/2970379474 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f845c039410 0x7f845c03b8d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:16.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.478+0000 7f8478d76700 1 --2- 172.21.15.23:0/2970379474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8474199d60 0x7f847419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:16.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.478+0000 7f8478d76700 1 --2- 172.21.15.23:0/2970379474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8474101e40 0x7f847419f6a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:16.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.478+0000 7f8478d76700 1 --2- 172.21.15.23:0/2970379474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f847419fbe0 0x7f8474199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:16.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.478+0000 7f8478d76700 1 -- 172.21.15.23:0/2970379474 >> 172.21.15.23:0/2970379474 conn(0x7f84740fd710 msgr2=0x7f8474105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:32:16.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.478+0000 7f8478d76700 1 -- 172.21.15.23:0/2970379474 shutdown_connections 2024-03-24T10:32:16.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:16.478+0000 7f8478d76700 1 -- 172.21.15.23:0/2970379474 wait complete. 2024-03-24T10:32:16.672 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:16 smithi023 ceph-mon[32466]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:16.672 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:16.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:16 smithi117 ceph-mon[36368]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:16.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:16.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:16 smithi116 ceph-mon[36248]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:16.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:32:17.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:17 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2970379474' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:17.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:17 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2970379474' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:17.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:17 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2970379474' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:17.811 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T10:32:17.812 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:32:17.812 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T10:32:17.842 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:32:17.842 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:32:17.909 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T10:32:17.910 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T10:32:17.940 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T10:32:17.941 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:32:18.008 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T10:32:18.008 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T10:32:18.042 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T10:32:18.042 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:32:18.112 INFO:tasks.cephadm:Adding mgr.x on smithi023 2024-03-24T10:32:18.112 INFO:tasks.cephadm:Adding mgr.y on smithi116 2024-03-24T10:32:18.113 INFO:tasks.cephadm:Adding mgr.z on smithi117 2024-03-24T10:32:18.113 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch apply mgr '3;smithi023=x;smithi116=y;smithi117=z' 2024-03-24T10:32:18.383 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:32:18.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:18 smithi116 ceph-mon[36248]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:18.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:18 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:18.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:18 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:18.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:18 smithi116 ceph-mon[36248]: Reconfiguring mon.b (monmap changed)... 2024-03-24T10:32:18.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:18 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:32:18.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:18 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T10:32:18.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:18 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:18.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:18 smithi116 ceph-mon[36248]: Reconfiguring daemon mon.b on smithi116 2024-03-24T10:32:18.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:18 smithi023 ceph-mon[32466]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:18.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:18 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:18.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:18 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:18.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:18 smithi023 ceph-mon[32466]: Reconfiguring mon.b (monmap changed)... 2024-03-24T10:32:18.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:18 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:32:18.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:18 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T10:32:18.702 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:18 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:18.702 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:18 smithi023 ceph-mon[32466]: Reconfiguring daemon mon.b on smithi116 2024-03-24T10:32:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:18 smithi117 ceph-mon[36368]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:18 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:18 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:18 smithi117 ceph-mon[36368]: Reconfiguring mon.b (monmap changed)... 2024-03-24T10:32:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:18 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:32:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:18 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T10:32:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:18 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:18.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:18 smithi117 ceph-mon[36368]: Reconfiguring daemon mon.b on smithi116 2024-03-24T10:32:20.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:19 smithi117 ceph-mon[36368]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:20.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:19 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:20.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:19 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:20.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:19 smithi117 ceph-mon[36368]: Reconfiguring mon.c (monmap changed)... 2024-03-24T10:32:20.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:19 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:32:20.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:19 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T10:32:20.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:19 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:20.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:19 smithi117 ceph-mon[36368]: Reconfiguring daemon mon.c on smithi117 2024-03-24T10:32:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:19 smithi023 ceph-mon[32466]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:19 smithi023 ceph-mon[32466]: Reconfiguring mon.c (monmap changed)... 2024-03-24T10:32:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:32:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T10:32:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:19 smithi023 ceph-mon[32466]: Reconfiguring daemon mon.c on smithi117 2024-03-24T10:32:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:19 smithi116 ceph-mon[36248]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:19 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:19 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:19 smithi116 ceph-mon[36248]: Reconfiguring mon.c (monmap changed)... 2024-03-24T10:32:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:19 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T10:32:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:19 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T10:32:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:19 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:19 smithi116 ceph-mon[36248]: Reconfiguring daemon mon.c on smithi117 2024-03-24T10:32:20.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.742+0000 7f2d426d8700 1 -- 172.21.15.117:0/2189469379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d3c108060 msgr2=0x7f2d3c10a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:20.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.742+0000 7f2d426d8700 1 --2- 172.21.15.117:0/2189469379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d3c108060 0x7f2d3c10a450 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d3400b0a0 tx=0x7f2d3400b3b0 comp rx=0 tx=0).stop 2024-03-24T10:32:20.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.742+0000 7f2d426d8700 1 -- 172.21.15.117:0/2189469379 shutdown_connections 2024-03-24T10:32:20.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.742+0000 7f2d426d8700 1 --2- 172.21.15.117:0/2189469379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d3c108060 0x7f2d3c10a450 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:20.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.742+0000 7f2d426d8700 1 --2- 172.21.15.117:0/2189469379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d3c102e00 0x7f2d3c1051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:20.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.742+0000 7f2d426d8700 1 --2- 172.21.15.117:0/2189469379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d3c105730 0x7f2d3c107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:20.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.742+0000 7f2d426d8700 1 -- 172.21.15.117:0/2189469379 >> 172.21.15.117:0/2189469379 conn(0x7f2d3c0fc790 msgr2=0x7f2d3c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:32:20.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.742+0000 7f2d426d8700 1 -- 172.21.15.117:0/2189469379 shutdown_connections 2024-03-24T10:32:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.742+0000 7f2d426d8700 1 -- 172.21.15.117:0/2189469379 wait complete. 2024-03-24T10:32:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.742+0000 7f2d426d8700 1 Processor -- start 2024-03-24T10:32:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.742+0000 7f2d426d8700 1 -- start start 2024-03-24T10:32:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d426d8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d3c102e00 0x7f2d3c1982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d426d8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d3c105730 0x7f2d3c198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d426d8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d3c108060 0x7f2d3c19f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d426d8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2d3c19fe60 con 0x7f2d3c105730 2024-03-24T10:32:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d426d8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2d3c19ffa0 con 0x7f2d3c108060 2024-03-24T10:32:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d426d8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2d3c1a0130 con 0x7f2d3c102e00 2024-03-24T10:32:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d3bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d3c102e00 0x7f2d3c1982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d3bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d3c102e00 0x7f2d3c1982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55720/0 (socket says 172.21.15.117:55720) 2024-03-24T10:32:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d3bfff700 1 -- 172.21.15.117:0/523547445 learned_addr learned my addr 172.21.15.117:0/523547445 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T10:32:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d3b7fe700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d3c105730 0x7f2d3c198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d40c75700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d3c108060 0x7f2d3c19f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:20.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d3bfff700 1 -- 172.21.15.117:0/523547445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d3c108060 msgr2=0x7f2d3c19f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:20.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d3bfff700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d3c108060 0x7f2d3c19f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:20.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d3bfff700 1 -- 172.21.15.117:0/523547445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d3c105730 msgr2=0x7f2d3c198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:20.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d3bfff700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d3c105730 0x7f2d3c198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:20.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d3bfff700 1 -- 172.21.15.117:0/523547445 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d34009d00 con 0x7f2d3c102e00 2024-03-24T10:32:20.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d3b7fe700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d3c105730 0x7f2d3c198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:32:20.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d3bfff700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d3c102e00 0x7f2d3c1982d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c00e9d0 tx=0x7f2d2c00ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:32:20.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d397fa700 1 -- 172.21.15.117:0/523547445 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c009790 con 0x7f2d3c102e00 2024-03-24T10:32:20.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d397fa700 1 -- 172.21.15.117:0/523547445 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d2c00ee70 con 0x7f2d3c102e00 2024-03-24T10:32:20.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d426d8700 1 -- 172.21.15.117:0/523547445 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d3c1a0410 con 0x7f2d3c102e00 2024-03-24T10:32:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.743+0000 7f2d397fa700 1 -- 172.21.15.117:0/523547445 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c018680 con 0x7f2d3c102e00 2024-03-24T10:32:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.744+0000 7f2d426d8700 1 -- 172.21.15.117:0/523547445 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d3c1a0960 con 0x7f2d3c102e00 2024-03-24T10:32:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.744+0000 7f2d397fa700 1 -- 172.21.15.117:0/523547445 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f2d2c0187e0 con 0x7f2d3c102e00 2024-03-24T10:32:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.745+0000 7f2d426d8700 1 -- 172.21.15.117:0/523547445 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d3c0fe310 con 0x7f2d3c102e00 2024-03-24T10:32:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.745+0000 7f2d397fa700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2d240391c0 0x7f2d2403b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.745+0000 7f2d397fa700 1 -- 172.21.15.117:0/523547445 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2d2c014070 con 0x7f2d3c102e00 2024-03-24T10:32:20.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.745+0000 7f2d3b7fe700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2d240391c0 0x7f2d2403b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:20.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.745+0000 7f2d3b7fe700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2d240391c0 0x7f2d2403b680 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2d30005950 tx=0x7f2d3000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:32:20.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.750+0000 7f2d397fa700 1 -- 172.21.15.117:0/523547445 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2d2c01fdb0 con 0x7f2d3c102e00 2024-03-24T10:32:20.957 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.956+0000 7f2d426d8700 1 -- 172.21.15.117:0/523547445 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi116=y;smithi117=z", "target": ["mon-mgr", ""]}) -- 0x7f2d3c02dc60 con 0x7f2d240391c0 2024-03-24T10:32:20.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.966+0000 7f2d397fa700 1 -- 172.21.15.117:0/523547445 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2d3c02dc60 con 0x7f2d240391c0 2024-03-24T10:32:20.967 INFO:teuthology.orchestra.run.smithi117.stdout:Scheduled mgr update... 2024-03-24T10:32:20.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.968+0000 7f2d426d8700 1 -- 172.21.15.117:0/523547445 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2d240391c0 msgr2=0x7f2d2403b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:20.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.968+0000 7f2d426d8700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2d240391c0 0x7f2d2403b680 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2d30005950 tx=0x7f2d3000b410 comp rx=0 tx=0).stop 2024-03-24T10:32:20.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.968+0000 7f2d426d8700 1 -- 172.21.15.117:0/523547445 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d3c102e00 msgr2=0x7f2d3c1982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:20.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.968+0000 7f2d426d8700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d3c102e00 0x7f2d3c1982d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c00e9d0 tx=0x7f2d2c00ece0 comp rx=0 tx=0).stop 2024-03-24T10:32:20.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.969+0000 7f2d426d8700 1 -- 172.21.15.117:0/523547445 shutdown_connections 2024-03-24T10:32:20.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.969+0000 7f2d426d8700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2d240391c0 0x7f2d2403b680 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:20.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.969+0000 7f2d426d8700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d3c102e00 0x7f2d3c1982d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:20.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.969+0000 7f2d426d8700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d3c108060 0x7f2d3c19f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:20.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.969+0000 7f2d426d8700 1 --2- 172.21.15.117:0/523547445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d3c105730 0x7f2d3c198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:20.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.969+0000 7f2d426d8700 1 -- 172.21.15.117:0/523547445 >> 172.21.15.117:0/523547445 conn(0x7f2d3c0fc790 msgr2=0x7f2d3c103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:32:20.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.969+0000 7f2d426d8700 1 -- 172.21.15.117:0/523547445 shutdown_connections 2024-03-24T10:32:20.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:32:20.969+0000 7f2d426d8700 1 -- 172.21.15.117:0/523547445 wait complete. 2024-03-24T10:32:21.636 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:21.637 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:21.637 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:21.637 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:21.637 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:21.637 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:21.637 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:21.637 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T10:32:21.637 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T10:32:21.637 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T10:32:21.637 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:21.638 DEBUG:teuthology.orchestra.run.smithi116:mgr.y> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mgr.y.service 2024-03-24T10:32:21.641 DEBUG:teuthology.orchestra.run.smithi117:mgr.z> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mgr.z.service 2024-03-24T10:32:21.644 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T10:32:21.644 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:32:21.644 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T10:32:21.661 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T10:32:21.662 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_1 2024-03-24T10:32:21.666 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:21.667 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:21.667 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:21.667 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:21.667 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:21.667 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:21.667 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:21.667 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T10:32:21.667 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T10:32:21.667 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T10:32:21.667 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:21.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:21.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:21.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:21.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:21.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:21.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:21.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:21.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T10:32:21.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T10:32:21.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T10:32:21.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:21.683 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T10:32:21.683 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:21.683 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T10:32:21.683 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:21.684 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:21.684 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-24 10:31:24.194139937 +0000 2024-03-24T10:32:21.684 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-24 10:26:15.135951394 +0000 2024-03-24T10:32:21.684 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-24 10:26:15.135951394 +0000 2024-03-24T10:32:21.684 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-24 10:26:15.135951394 +0000 2024-03-24T10:32:21.684 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T10:32:21.751 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-24T10:32:21.751 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-24T10:32:21.751 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000305102 s, 1.7 MB/s 2024-03-24T10:32:21.752 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T10:32:21.808 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_2 2024-03-24T10:32:21.862 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T10:32:21.863 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:21.863 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-24T10:32:21.863 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:21.863 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:21.863 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-24 10:31:24.234139662 +0000 2024-03-24T10:32:21.863 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-24 10:26:15.485949103 +0000 2024-03-24T10:32:21.863 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-24 10:26:15.485949103 +0000 2024-03-24T10:32:21.863 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-24 10:26:15.485949103 +0000 2024-03-24T10:32:21.863 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T10:32:21.929 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-24T10:32:21.929 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-24T10:32:21.929 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000283787 s, 1.8 MB/s 2024-03-24T10:32:21.930 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T10:32:21.989 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_3 2024-03-24T10:32:22.043 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T10:32:22.044 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:22.044 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-24T10:32:22.044 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:22.044 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:22.044 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-24 10:31:24.270139414 +0000 2024-03-24T10:32:22.044 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-24 10:26:15.833946825 +0000 2024-03-24T10:32:22.044 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-24 10:26:15.833946825 +0000 2024-03-24T10:32:22.044 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-24 10:26:15.833946825 +0000 2024-03-24T10:32:22.044 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T10:32:22.110 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-24T10:32:22.110 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-24T10:32:22.110 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000274545 s, 1.9 MB/s 2024-03-24T10:32:22.111 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T10:32:22.167 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_4 2024-03-24T10:32:22.222 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T10:32:22.222 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:22.222 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-24T10:32:22.223 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:22.223 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:22.223 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-24 10:31:24.304139181 +0000 2024-03-24T10:32:22.223 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-24 10:26:16.196944449 +0000 2024-03-24T10:32:22.223 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-24 10:26:16.196944449 +0000 2024-03-24T10:32:22.223 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-24 10:26:16.196944449 +0000 2024-03-24T10:32:22.223 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T10:32:22.288 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-24T10:32:22.288 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-24T10:32:22.288 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000315962 s, 1.6 MB/s 2024-03-24T10:32:22.289 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T10:32:22.345 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T10:32:22.345 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T10:32:22.364 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T10:32:22.365 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-03-24T10:32:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T10:32:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:22.423 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-24T10:32:22.423 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:22.423 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:22.423 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-24 10:32:06.816186866 +0000 2024-03-24T10:32:22.423 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-24 10:26:36.656961426 +0000 2024-03-24T10:32:22.423 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-24 10:26:36.656961426 +0000 2024-03-24T10:32:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-24 10:26:36.656961426 +0000 2024-03-24T10:32:22.424 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T10:32:22.493 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-24T10:32:22.493 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-24T10:32:22.493 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000291822 s, 1.8 MB/s 2024-03-24T10:32:22.494 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T10:32:22.553 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-03-24T10:32:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T10:32:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:22.609 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-24T10:32:22.609 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:22.609 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:22.610 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-24 10:32:06.816186866 +0000 2024-03-24T10:32:22.610 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-24 10:26:36.990955818 +0000 2024-03-24T10:32:22.610 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-24 10:26:36.990955818 +0000 2024-03-24T10:32:22.610 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-24 10:26:36.990955818 +0000 2024-03-24T10:32:22.610 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T10:32:22.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:22 smithi023 ceph-mon[32466]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:22.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:22 smithi023 ceph-mon[32466]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi116=y;smithi117=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:32:22.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:22 smithi023 ceph-mon[32466]: Saving service mgr spec with placement smithi023=x;smithi116=y;smithi117=z;count:3 2024-03-24T10:32:22.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:22 smithi023 ceph-mon[32466]: Deploying daemon mgr.z on smithi117 2024-03-24T10:32:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:22 smithi116 ceph-mon[36248]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:22 smithi116 ceph-mon[36248]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi116=y;smithi117=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:32:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:22 smithi116 ceph-mon[36248]: Saving service mgr spec with placement smithi023=x;smithi116=y;smithi117=z;count:3 2024-03-24T10:32:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:22 smithi116 ceph-mon[36248]: Deploying daemon mgr.z on smithi117 2024-03-24T10:32:22.681 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-24T10:32:22.681 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-24T10:32:22.681 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000187926 s, 2.7 MB/s 2024-03-24T10:32:22.688 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T10:32:22.734 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:22 smithi117 ceph-mon[36368]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:22.734 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:22 smithi117 ceph-mon[36368]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi116=y;smithi117=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:32:22.734 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:22 smithi117 ceph-mon[36368]: Saving service mgr spec with placement smithi023=x;smithi116=y;smithi117=z;count:3 2024-03-24T10:32:22.734 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:22 smithi117 ceph-mon[36368]: Deploying daemon mgr.z on smithi117 2024-03-24T10:32:22.746 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-03-24T10:32:22.806 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T10:32:22.806 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:22.806 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-24T10:32:22.806 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:22.806 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:22.806 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-24 10:32:06.817186850 +0000 2024-03-24T10:32:22.807 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-24 10:26:37.320950277 +0000 2024-03-24T10:32:22.807 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-24 10:26:37.320950277 +0000 2024-03-24T10:32:22.807 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-24 10:26:37.320950277 +0000 2024-03-24T10:32:22.807 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T10:32:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-24T10:32:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-24T10:32:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000190073 s, 2.7 MB/s 2024-03-24T10:32:22.876 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T10:32:22.933 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-03-24T10:32:22.989 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T10:32:22.989 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:22.989 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-24T10:32:22.989 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:22.990 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:22.990 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-24 10:32:06.817186850 +0000 2024-03-24T10:32:22.990 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-24 10:26:37.648944770 +0000 2024-03-24T10:32:22.990 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-24 10:26:37.648944770 +0000 2024-03-24T10:32:22.990 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-24 10:26:37.648944770 +0000 2024-03-24T10:32:22.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T10:32:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-24T10:32:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-24T10:32:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000297943 s, 1.7 MB/s 2024-03-24T10:32:23.058 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T10:32:23.114 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T10:32:23.115 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T10:32:23.131 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T10:32:23.132 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-03-24T10:32:23.191 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T10:32:23.191 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:23.191 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 660 Links: 1 2024-03-24T10:32:23.191 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:23.191 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:23.191 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-24 10:32:23.129258368 +0000 2024-03-24T10:32:23.191 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-24 10:26:25.446734339 +0000 2024-03-24T10:32:23.191 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-24 10:26:25.446734339 +0000 2024-03-24T10:32:23.191 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-24 10:26:25.446734339 +0000 2024-03-24T10:32:23.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T10:32:23.261 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-24T10:32:23.262 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-24T10:32:23.262 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000274667 s, 1.9 MB/s 2024-03-24T10:32:23.263 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T10:32:23.321 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-03-24T10:32:23.377 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T10:32:23.377 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:23.377 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 679 Links: 1 2024-03-24T10:32:23.377 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:23.377 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:23.377 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-24 10:32:23.130258348 +0000 2024-03-24T10:32:23.377 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-24 10:26:25.785727510 +0000 2024-03-24T10:32:23.377 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-24 10:26:25.785727510 +0000 2024-03-24T10:32:23.377 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-24 10:26:25.785727510 +0000 2024-03-24T10:32:23.378 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T10:32:23.443 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-24T10:32:23.443 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-24T10:32:23.443 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000208714 s, 2.5 MB/s 2024-03-24T10:32:23.444 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T10:32:23.500 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-03-24T10:32:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T10:32:23.557 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:23.557 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 698 Links: 1 2024-03-24T10:32:23.557 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:23.557 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:23.557 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-24 10:32:23.130258348 +0000 2024-03-24T10:32:23.557 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-24 10:26:26.109720983 +0000 2024-03-24T10:32:23.557 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-24 10:26:26.109720983 +0000 2024-03-24T10:32:23.557 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-24 10:26:26.109720983 +0000 2024-03-24T10:32:23.579 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T10:32:23.645 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-24T10:32:23.645 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-24T10:32:23.645 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000193818 s, 2.6 MB/s 2024-03-24T10:32:23.646 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T10:32:23.701 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-03-24T10:32:23.722 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:23 smithi117 ceph-mgr[38309]: -- 172.21.15.117:0/1930601112 <== mon.1 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562ce95ba1a0 con 0x562ce9591000 2024-03-24T10:32:23.736 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T10:32:23.737 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:32:23.737 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-24T10:32:23.737 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:32:23.737 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:32:23.737 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-24 10:32:23.131258329 +0000 2024-03-24T10:32:23.737 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-24 10:26:26.438714355 +0000 2024-03-24T10:32:23.737 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-24 10:26:26.438714355 +0000 2024-03-24T10:32:23.737 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-24 10:26:26.438714355 +0000 2024-03-24T10:32:23.737 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T10:32:23.805 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-24T10:32:23.805 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-24T10:32:23.805 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000172967 s, 3.0 MB/s 2024-03-24T10:32:23.806 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T10:32:23.861 INFO:tasks.cephadm:Deploying osd.0 on smithi023 with /dev/vg_nvme/lv_4... 2024-03-24T10:32:23.861 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T10:32:24.004 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:23 smithi117 bash[38184]: 574d0e6e20f704c9a979dc96f821e7c547a7603a41f8b711ff9f3ae2960b3b65 2024-03-24T10:32:24.004 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:23 smithi117 systemd[1]: Started Ceph mgr.z for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:32:24.105 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:32:24.284 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:24 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:24.191+0000 7f6746bc6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T10:32:24.285 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:24 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:24.284+0000 7f6746bc6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T10:32:24.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:24 smithi023 ceph-mon[32466]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:24.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:24.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:24.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:24.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T10:32:24.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T10:32:24.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T10:32:24.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:24.699 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:24 smithi116 ceph-mon[36248]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:24.699 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:24.699 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:24.699 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:24.699 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T10:32:24.699 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T10:32:24.699 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T10:32:24.699 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:24.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:24 smithi117 ceph-mon[36368]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:24.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:24.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:24.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:24.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T10:32:24.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T10:32:24.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T10:32:24.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:25.254 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:24 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:24.920+0000 7f6746bc6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T10:32:25.254 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:25.012+0000 7f6746bc6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T10:32:25.254 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:25.161+0000 7f6746bc6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T10:32:25.513 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:25 smithi023 ceph-mon[32466]: Deploying daemon mgr.y on smithi116 2024-03-24T10:32:25.600 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:25 smithi117 ceph-mon[36368]: Deploying daemon mgr.y on smithi116 2024-03-24T10:32:25.601 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:25.257+0000 7f6746bc6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T10:32:25.601 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:25.339+0000 7f6746bc6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T10:32:25.601 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:25.598+0000 7f6746bc6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T10:32:25.684 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:25 smithi116 ceph-mon[36248]: Deploying daemon mgr.y on smithi116 2024-03-24T10:32:25.685 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:25 smithi116 systemd[1]: Starting Ceph mgr.y for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:32:25.875 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:25.707+0000 7f6746bc6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T10:32:25.876 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:25.794+0000 7f6746bc6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T10:32:25.997 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:25 smithi116 podman[37627]: 2024-03-24 10:32:25.688403048 +0000 UTC m=+0.018730315 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:32:25.997 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:25 smithi116 podman[37627]: 2024-03-24 10:32:25.786759654 +0000 UTC m=+0.117086902 container create b69d43a495ff0e5fe9ca439b9c853074e65639d771038b300202b26358717eba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-24T10:32:25.997 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:25 smithi116 podman[37627]: 2024-03-24 10:32:25.997334928 +0000 UTC m=+0.327662184 container init b69d43a495ff0e5fe9ca439b9c853074e65639d771038b300202b26358717eba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-24T10:32:26.059 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T10:32:26.059 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T10:32:26.059 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-24T10:32:26.059 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-24T10:32:26.059 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00852989 s, 1.2 GB/s 2024-03-24T10:32:26.059 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-24T10:32:26.133 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:25.890+0000 7f6746bc6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T10:32:26.133 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:26 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:26.070+0000 7f6746bc6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T10:32:26.267 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:26 smithi116 podman[37627]: 2024-03-24 10:32:26.000066879 +0000 UTC m=+0.330394147 container start b69d43a495ff0e5fe9ca439b9c853074e65639d771038b300202b26358717eba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T10:32:26.268 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:26 smithi116 ceph-mgr[37647]: -- 172.21.15.116:0/3627668460 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5595d5c621a0 con 0x5595d5c38800 2024-03-24T10:32:26.268 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:26 smithi116 bash[37627]: b69d43a495ff0e5fe9ca439b9c853074e65639d771038b300202b26358717eba 2024-03-24T10:32:26.268 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:26 smithi116 systemd[1]: Started Ceph mgr.y for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:32:26.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:26 smithi023 ceph-mon[32466]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:26.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:26 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:26.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:26 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:26.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:26 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:26.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:26 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:26.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:26 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:26.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:26 smithi117 ceph-mon[36368]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:26.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:26 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:26.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:26 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:26.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:26 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:26.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:26 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:26.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:26 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:26.767 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:26 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:26.549+0000 7f24034da200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T10:32:26.768 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:26 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:26.643+0000 7f24034da200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T10:32:26.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:26 smithi116 ceph-mon[36248]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:26.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:26 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:26.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:26 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:26.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:26 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:26.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:26 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:26.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:26 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:26.790 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_4 2024-03-24T10:32:27.030 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:32:27.316 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:27 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:27.316+0000 7f24034da200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T10:32:27.503 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:27.105+0000 7f6746bc6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T10:32:27.503 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:27.242+0000 7f6746bc6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T10:32:27.574 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:27 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:27.409+0000 7f24034da200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T10:32:27.874 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:27.583+0000 7f6746bc6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T10:32:27.874 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:27.674+0000 7f6746bc6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T10:32:27.874 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:27.873+0000 7f6746bc6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:32:28.012 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:27 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:27.574+0000 7f24034da200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T10:32:28.012 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:27 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:27.670+0000 7f24034da200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T10:32:28.012 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:27 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:27.754+0000 7f24034da200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T10:32:28.012 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:28.011+0000 7f24034da200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T10:32:28.252 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:28.056+0000 7f6746bc6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T10:32:28.267 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:28.119+0000 7f24034da200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T10:32:28.267 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:28.203+0000 7f24034da200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T10:32:28.595 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:28.339+0000 7f6746bc6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:32:28.595 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:28.594+0000 7f6746bc6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T10:32:28.595 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-mon[36368]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:28.596 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.596 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.596 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.596 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.596 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:28.596 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:28.596 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.596 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T10:32:28.596 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T10:32:28.596 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:28.607 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:28 smithi023 ceph-mon[32466]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:28.607 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.607 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.607 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.607 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.607 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:28.607 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:28.607 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.607 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T10:32:28.607 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T10:32:28.607 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:28.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.637+0000 7f7c3148e700 1 -- 172.21.15.23:0/1862943483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c2c103df0 msgr2=0x7f7c2c10a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:28.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.637+0000 7f7c3148e700 1 --2- 172.21.15.23:0/1862943483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c2c103df0 0x7f7c2c10a6a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400b0d0 tx=0x7f7c2400b3e0 comp rx=0 tx=0).stop 2024-03-24T10:32:28.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.637+0000 7f7c3148e700 1 -- 172.21.15.23:0/1862943483 shutdown_connections 2024-03-24T10:32:28.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.637+0000 7f7c3148e700 1 --2- 172.21.15.23:0/1862943483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c2c103df0 0x7f7c2c10a6a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:28.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.637+0000 7f7c3148e700 1 --2- 172.21.15.23:0/1862943483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c2c102250 0x7f7c2c102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:28.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.637+0000 7f7c3148e700 1 --2- 172.21.15.23:0/1862943483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c2c103450 0x7f7c2c1038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:28.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.637+0000 7f7c3148e700 1 -- 172.21.15.23:0/1862943483 >> 172.21.15.23:0/1862943483 conn(0x7f7c2c0fd7b0 msgr2=0x7f7c2c0ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:32:28.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.638+0000 7f7c3148e700 1 -- 172.21.15.23:0/1862943483 shutdown_connections 2024-03-24T10:32:28.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.638+0000 7f7c3148e700 1 -- 172.21.15.23:0/1862943483 wait complete. 2024-03-24T10:32:28.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.638+0000 7f7c3148e700 1 Processor -- start 2024-03-24T10:32:28.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.638+0000 7f7c3148e700 1 -- start start 2024-03-24T10:32:28.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.638+0000 7f7c3148e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c2c102250 0x7f7c2c1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:28.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.638+0000 7f7c3148e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c2c103450 0x7f7c2c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:28.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.638+0000 7f7c3148e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c2c103df0 0x7f7c2c19f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:28.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.638+0000 7f7c3148e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c19fe40 con 0x7f7c2c102250 2024-03-24T10:32:28.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.638+0000 7f7c3148e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c19ff80 con 0x7f7c2c103df0 2024-03-24T10:32:28.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.638+0000 7f7c3148e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c1a0110 con 0x7f7c2c103450 2024-03-24T10:32:28.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c2affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c2c102250 0x7f7c2c1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:28.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c2affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c2c102250 0x7f7c2c1982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46790/0 (socket says 172.21.15.23:46790) 2024-03-24T10:32:28.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c2a7fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c2c103450 0x7f7c2c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:28.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c2b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c2c103df0 0x7f7c2c19f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:28.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c2affd700 1 -- 172.21.15.23:0/2511955596 learned_addr learned my addr 172.21.15.23:0/2511955596 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:32:28.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c2affd700 1 -- 172.21.15.23:0/2511955596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c2c103450 msgr2=0x7f7c2c1987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:28.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c2affd700 1 --2- 172.21.15.23:0/2511955596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c2c103450 0x7f7c2c1987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:28.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c2affd700 1 -- 172.21.15.23:0/2511955596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c2c103df0 msgr2=0x7f7c2c19f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:28.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c2affd700 1 --2- 172.21.15.23:0/2511955596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c2c103df0 0x7f7c2c19f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:28.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c2affd700 1 -- 172.21.15.23:0/2511955596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c24009d00 con 0x7f7c2c102250 2024-03-24T10:32:28.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c2affd700 1 --2- 172.21.15.23:0/2511955596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c2c102250 0x7f7c2c1982b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c00d9b0 tx=0x7f7c1c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:32:28.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c13fff700 1 -- 172.21.15.23:0/2511955596 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c00b950 con 0x7f7c2c102250 2024-03-24T10:32:28.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c13fff700 1 -- 172.21.15.23:0/2511955596 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c1c007620 con 0x7f7c2c102250 2024-03-24T10:32:28.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c3148e700 1 -- 172.21.15.23:0/2511955596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c2c1a03f0 con 0x7f7c2c102250 2024-03-24T10:32:28.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c13fff700 1 -- 172.21.15.23:0/2511955596 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c00f3c0 con 0x7f7c2c102250 2024-03-24T10:32:28.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.639+0000 7f7c3148e700 1 -- 172.21.15.23:0/2511955596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c2c1a08f0 con 0x7f7c2c102250 2024-03-24T10:32:28.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.640+0000 7f7c13fff700 1 -- 172.21.15.23:0/2511955596 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f7c1c00f520 con 0x7f7c2c102250 2024-03-24T10:32:28.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.640+0000 7f7c13fff700 1 --2- 172.21.15.23:0/2511955596 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7c140391c0 0x7f7c1403b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:28.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.640+0000 7f7c13fff700 1 -- 172.21.15.23:0/2511955596 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7c1c04bd80 con 0x7f7c2c102250 2024-03-24T10:32:28.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.640+0000 7f7c3148e700 1 -- 172.21.15.23:0/2511955596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c2c1095b0 con 0x7f7c2c102250 2024-03-24T10:32:28.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.641+0000 7f7c2a7fc700 1 --2- 172.21.15.23:0/2511955596 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7c140391c0 0x7f7c1403b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:28.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.641+0000 7f7c2a7fc700 1 --2- 172.21.15.23:0/2511955596 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7c140391c0 0x7f7c1403b680 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7c20007b60 tx=0x7f7c200073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:32:28.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.646+0000 7f7c13fff700 1 -- 172.21.15.23:0/2511955596 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7c1c01a970 con 0x7f7c2c102250 2024-03-24T10:32:28.767 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:28.296+0000 7f24034da200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T10:32:28.767 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:28.470+0000 7f24034da200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T10:32:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-mon[36248]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T10:32:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T10:32:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:28.873+0000 7f7c3148e700 1 -- 172.21.15.23:0/2511955596 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7c2c02db90 con 0x7f7c140391c0 2024-03-24T10:32:29.002 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:28 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:28.825+0000 7f6746bc6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T10:32:29.733 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:29 smithi023 ceph-mon[32466]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T10:32:29.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:29 smithi023 ceph-mon[32466]: Reconfiguring daemon mgr.x on smithi023 2024-03-24T10:32:29.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:29 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:32:29.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:29 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:32:29.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:29 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:29.768 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:29 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:29.418+0000 7f24034da200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T10:32:29.768 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:29 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:29.547+0000 7f24034da200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T10:32:29.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:29 smithi116 ceph-mon[36248]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T10:32:29.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:29 smithi116 ceph-mon[36248]: Reconfiguring daemon mgr.x on smithi023 2024-03-24T10:32:29.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:29 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:32:29.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:29 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:32:29.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:29 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:30.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:29 smithi117 ceph-mon[36368]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T10:32:30.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:29 smithi117 ceph-mon[36368]: Reconfiguring daemon mgr.x on smithi023 2024-03-24T10:32:30.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:29 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:32:30.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:29 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:32:30.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:29 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:30.003 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:29 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:29.615+0000 7f6746bc6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T10:32:30.158 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:29 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:29.877+0000 7f24034da200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T10:32:30.158 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:29 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:29.967+0000 7f24034da200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T10:32:30.158 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:30.157+0000 7f24034da200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:32:30.492 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:30.180+0000 7f6746bc6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T10:32:30.492 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:30.277+0000 7f6746bc6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T10:32:30.492 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:32:30.491+0000 7f6746bc6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T10:32:30.517 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:30.338+0000 7f24034da200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T10:32:30.752 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:30 smithi023 ceph-mon[32466]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:30.753 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:30 smithi023 ceph-mon[32466]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:32:30.753 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:30 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:30.753 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:30 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:30.753 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:30 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:30.753 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:30 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:30.753 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:30 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:30.753 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:30 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:30.753 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:30 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:30.753 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:30 smithi023 ceph-mon[32466]: Standby manager daemon z started 2024-03-24T10:32:30.753 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:30 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.117:0/2042666877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T10:32:30.753 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:30 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.117:0/2042666877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T10:32:30.753 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:30 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.117:0/2042666877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T10:32:30.753 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:30 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.117:0/2042666877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T10:32:30.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-mon[36368]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:30.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-mon[36368]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:32:30.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:30.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:30.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:30.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:30.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:30.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:30.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:30.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-mon[36368]: Standby manager daemon z started 2024-03-24T10:32:30.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.117:0/2042666877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T10:32:30.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.117:0/2042666877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T10:32:30.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.117:0/2042666877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T10:32:30.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:30 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.117:0/2042666877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T10:32:30.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:30.808+0000 7f7c13fff700 1 -- 172.21.15.23:0/2511955596 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 14) ==== 91342+0+0 (secure 0 0 0) 0x7f7c1c01b5e0 con 0x7f7c2c102250 2024-03-24T10:32:30.874 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:30.615+0000 7f24034da200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:32:30.874 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:30.873+0000 7f24034da200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T10:32:30.874 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-mon[36248]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:30.874 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-mon[36248]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:32:30.874 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:30.874 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:30.874 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:30.874 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:30.874 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:30.875 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:30.875 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:30.875 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-mon[36248]: Standby manager daemon z started 2024-03-24T10:32:30.875 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.117:0/2042666877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T10:32:30.875 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.117:0/2042666877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T10:32:30.875 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.117:0/2042666877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T10:32:30.875 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:30 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.117:0/2042666877' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T10:32:31.268 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:31 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:31.104+0000 7f24034da200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T10:32:32.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:31 smithi023 ceph-mon[32466]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:32.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:31 smithi023 ceph-mon[32466]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-24T10:32:32.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:31 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T10:32:32.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:31 smithi117 ceph-mon[36368]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:32.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:31 smithi117 ceph-mon[36368]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-24T10:32:32.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:31 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T10:32:32.267 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:31 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:31.890+0000 7f24034da200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T10:32:32.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:31 smithi116 ceph-mon[36248]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:32.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:31 smithi116 ceph-mon[36248]: mgrmap e14: x(active, since 76s), standbys: z 2024-03-24T10:32:32.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:31 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T10:32:32.759 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:32 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:32.451+0000 7f24034da200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T10:32:32.759 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:32 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:32.548+0000 7f24034da200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T10:32:32.759 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:32:32 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:32:32.758+0000 7f24034da200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T10:32:33.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:32 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/706840434' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6c1781a-5af4-4191-b299-02371336d02a"}]: dispatch 2024-03-24T10:32:33.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:32 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/706840434' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6c1781a-5af4-4191-b299-02371336d02a"}]': finished 2024-03-24T10:32:33.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:32 smithi116 ceph-mon[36248]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T10:32:33.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:33.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:32 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/537751747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:32:33.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:32 smithi116 ceph-mon[36248]: Standby manager daemon y started 2024-03-24T10:32:33.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:32 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.116:0/1004584575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T10:32:33.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:32 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.116:0/1004584575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T10:32:33.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:32 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.116:0/1004584575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T10:32:33.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:32 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.116:0/1004584575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T10:32:33.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:32 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/706840434' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6c1781a-5af4-4191-b299-02371336d02a"}]: dispatch 2024-03-24T10:32:33.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:32 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/706840434' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6c1781a-5af4-4191-b299-02371336d02a"}]': finished 2024-03-24T10:32:33.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:32 smithi117 ceph-mon[36368]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T10:32:33.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:33.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:32 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/537751747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:32:33.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:32 smithi117 ceph-mon[36368]: Standby manager daemon y started 2024-03-24T10:32:33.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:32 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.116:0/1004584575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T10:32:33.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:32 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.116:0/1004584575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T10:32:33.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:32 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.116:0/1004584575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T10:32:33.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:32 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.116:0/1004584575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T10:32:33.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:33.402+0000 7f7c13fff700 1 -- 172.21.15.23:0/2511955596 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7f7c1c00eca0 con 0x7f7c2c102250 2024-03-24T10:32:33.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:32 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/706840434' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6c1781a-5af4-4191-b299-02371336d02a"}]: dispatch 2024-03-24T10:32:33.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:32 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/706840434' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6c1781a-5af4-4191-b299-02371336d02a"}]': finished 2024-03-24T10:32:33.404 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:32 smithi023 ceph-mon[32466]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T10:32:33.404 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:33.404 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:32 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/537751747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:32:33.404 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:32 smithi023 ceph-mon[32466]: Standby manager daemon y started 2024-03-24T10:32:33.404 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:32 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.116:0/1004584575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T10:32:33.404 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:32 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.116:0/1004584575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T10:32:33.404 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:32 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.116:0/1004584575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T10:32:33.404 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:32 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.116:0/1004584575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T10:32:34.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:34 smithi023 ceph-mon[32466]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:34.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:34 smithi023 ceph-mon[32466]: mgrmap e15: x(active, since 78s), standbys: y, z 2024-03-24T10:32:34.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:34 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T10:32:34.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:34 smithi117 ceph-mon[36368]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:34.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:34 smithi117 ceph-mon[36368]: mgrmap e15: x(active, since 78s), standbys: y, z 2024-03-24T10:32:34.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:34 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T10:32:34.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:34 smithi116 ceph-mon[36248]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:34.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:34 smithi116 ceph-mon[36248]: mgrmap e15: x(active, since 78s), standbys: y, z 2024-03-24T10:32:34.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:34 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T10:32:36.672 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:36 smithi023 ceph-mon[32466]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:36 smithi117 ceph-mon[36368]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:36.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:36 smithi116 ceph-mon[36248]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:38.592 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:38 smithi023 ceph-mon[32466]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:38.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:38 smithi117 ceph-mon[36368]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:38.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:38 smithi116 ceph-mon[36248]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:40.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:40 smithi023 ceph-mon[32466]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:40.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T10:32:40.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:40.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:40 smithi117 ceph-mon[36368]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:40.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T10:32:40.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:40.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:40 smithi116 ceph-mon[36248]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:40.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T10:32:40.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:41.697 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:41 smithi023 ceph-mon[32466]: Deploying daemon osd.0 on smithi023 2024-03-24T10:32:41.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:41 smithi117 ceph-mon[36368]: Deploying daemon osd.0 on smithi023 2024-03-24T10:32:41.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:41 smithi116 ceph-mon[36248]: Deploying daemon osd.0 on smithi023 2024-03-24T10:32:42.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:42 smithi023 ceph-mon[32466]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:42.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:42 smithi117 ceph-mon[36368]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:42.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:42 smithi116 ceph-mon[36248]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:44.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:44 smithi117 ceph-mon[36368]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:44.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:44 smithi116 ceph-mon[36248]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:44.794 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:44 smithi023 ceph-mon[32466]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:46.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:46 smithi023 ceph-mon[32466]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:46.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:46 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:46.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:46 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:46.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:46 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:46.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:46 smithi117 ceph-mon[36368]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:46.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:46 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:46.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:46 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:46.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:46 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:46.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:46 smithi116 ceph-mon[36248]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:46.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:46 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:46.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:46 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:46.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:46 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:48.678 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 0 on host 'smithi023' 2024-03-24T10:32:48.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:48.677+0000 7f7c13fff700 1 -- 172.21.15.23:0/2511955596 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7c2c02db90 con 0x7f7c140391c0 2024-03-24T10:32:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:48.680+0000 7f7c3148e700 1 -- 172.21.15.23:0/2511955596 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7c140391c0 msgr2=0x7f7c1403b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:48.680+0000 7f7c3148e700 1 --2- 172.21.15.23:0/2511955596 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7c140391c0 0x7f7c1403b680 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7c20007b60 tx=0x7f7c200073d0 comp rx=0 tx=0).stop 2024-03-24T10:32:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:48.680+0000 7f7c3148e700 1 -- 172.21.15.23:0/2511955596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c2c102250 msgr2=0x7f7c2c1982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:48.680+0000 7f7c3148e700 1 --2- 172.21.15.23:0/2511955596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c2c102250 0x7f7c2c1982b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c00d9b0 tx=0x7f7c1c00dcc0 comp rx=0 tx=0).stop 2024-03-24T10:32:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:48.680+0000 7f7c3148e700 1 -- 172.21.15.23:0/2511955596 shutdown_connections 2024-03-24T10:32:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:48.680+0000 7f7c3148e700 1 --2- 172.21.15.23:0/2511955596 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7c140391c0 0x7f7c1403b680 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:48.680+0000 7f7c3148e700 1 --2- 172.21.15.23:0/2511955596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c2c103df0 0x7f7c2c19f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:48.680+0000 7f7c3148e700 1 --2- 172.21.15.23:0/2511955596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c2c102250 0x7f7c2c1982b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:48.680+0000 7f7c3148e700 1 --2- 172.21.15.23:0/2511955596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c2c103450 0x7f7c2c1987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:48.680+0000 7f7c3148e700 1 -- 172.21.15.23:0/2511955596 >> 172.21.15.23:0/2511955596 conn(0x7f7c2c0fd7b0 msgr2=0x7f7c2c1066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:32:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:48.680+0000 7f7c3148e700 1 -- 172.21.15.23:0/2511955596 shutdown_connections 2024-03-24T10:32:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:32:48.680+0000 7f7c3148e700 1 -- 172.21.15.23:0/2511955596 wait complete. 2024-03-24T10:32:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:48 smithi023 ceph-mon[32466]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:48 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:48 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:48 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:48 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:48 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:49.005 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:48 smithi117 ceph-mon[36368]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:49.005 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:48 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:49.005 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:48 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:49.005 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:48 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:49.005 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:48 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:49.005 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:48 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:49.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:48 smithi116 ceph-mon[36248]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:49.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:48 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:49.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:48 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:49.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:48 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:49.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:48 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:49.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:48 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:49.276 DEBUG:teuthology.orchestra.run.smithi023:osd.0> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T10:32:49.279 INFO:tasks.cephadm:Deploying osd.1 on smithi116 with /dev/vg_nvme/lv_4... 2024-03-24T10:32:49.279 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T10:32:49.522 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T10:32:49.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:49 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:49.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:49 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:49.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:49 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:49.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:49 smithi023 ceph-mon[32466]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T10:32:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:49 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:49 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:49 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:49 smithi117 ceph-mon[36368]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T10:32:50.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:49 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:50.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:49 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:50.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:49 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:32:50.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:49 smithi116 ceph-mon[36248]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T10:32:50.860 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:50 smithi023 ceph-mon[32466]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:50.860 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:50 smithi023 ceph-mon[32466]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T10:32:50.860 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:50 smithi023 ceph-mon[32466]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T10:32:50.860 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:50 smithi023 ceph-mon[32466]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T10:32:50.860 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:50 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:51.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:50 smithi117 ceph-mon[36368]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:51.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:50 smithi117 ceph-mon[36368]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T10:32:51.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:50 smithi117 ceph-mon[36368]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T10:32:51.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:50 smithi117 ceph-mon[36368]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T10:32:51.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:50 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:51.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:50 smithi116 ceph-mon[36248]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:51.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:50 smithi116 ceph-mon[36248]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T10:32:51.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:50 smithi116 ceph-mon[36248]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T10:32:51.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:50 smithi116 ceph-mon[36248]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T10:32:51.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:50 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T10:32:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T10:32:51.147 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-24T10:32:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-24T10:32:51.147 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113129 s, 927 MB/s 2024-03-24T10:32:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-24T10:32:51.700 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_4 2024-03-24T10:32:51.897 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:51 smithi023 ceph-mon[32466]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T10:32:51.897 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:51 smithi023 ceph-mon[32466]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T10:32:51.897 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:51.897 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:51.897 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:51 smithi023 ceph-mon[32466]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' 2024-03-24T10:32:51.897 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:32:51 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-0[45050]: 2024-03-24T10:32:51.582+0000 7f9645a74700 -1 osd.0 0 waiting for initial osdmap 2024-03-24T10:32:51.897 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:32:51 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-0[45050]: 2024-03-24T10:32:51.590+0000 7f963e060700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:32:51.937 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T10:32:52.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:51 smithi117 ceph-mon[36368]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T10:32:52.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:51 smithi117 ceph-mon[36368]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T10:32:52.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:51 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:52.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:51 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:52.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:51 smithi117 ceph-mon[36368]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' 2024-03-24T10:32:52.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:51 smithi116 ceph-mon[36248]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T10:32:52.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:51 smithi116 ceph-mon[36248]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T10:32:52.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:51 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:52.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:51 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:52.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:51 smithi116 ceph-mon[36248]: from='osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038]' entity='osd.0' 2024-03-24T10:32:52.985 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:52 smithi116 ceph-mon[36248]: purged_snaps scrub starts 2024-03-24T10:32:52.985 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:52 smithi116 ceph-mon[36248]: purged_snaps scrub ok 2024-03-24T10:32:52.985 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:52 smithi116 ceph-mon[36248]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:52.985 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:52.985 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:52.985 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:52.985 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:32:52.985 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:52.985 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:52.985 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:52.985 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:52.985 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:52 smithi116 ceph-mon[36248]: osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038] boot 2024-03-24T10:32:52.986 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:52 smithi116 ceph-mon[36248]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T10:32:52.986 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:52 smithi117 ceph-mon[36368]: purged_snaps scrub starts 2024-03-24T10:32:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:52 smithi117 ceph-mon[36368]: purged_snaps scrub ok 2024-03-24T10:32:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:52 smithi117 ceph-mon[36368]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:32:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:52 smithi117 ceph-mon[36368]: osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038] boot 2024-03-24T10:32:53.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:52 smithi117 ceph-mon[36368]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T10:32:53.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:52 smithi023 ceph-mon[32466]: purged_snaps scrub starts 2024-03-24T10:32:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:52 smithi023 ceph-mon[32466]: purged_snaps scrub ok 2024-03-24T10:32:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:52 smithi023 ceph-mon[32466]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T10:32:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:53.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:32:53.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:53.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:53.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:32:53.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:32:53.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:52 smithi023 ceph-mon[32466]: osd.0 [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038] boot 2024-03-24T10:32:53.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:52 smithi023 ceph-mon[32466]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T10:32:53.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:32:53.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.491+0000 7f30e0990700 1 -- 172.21.15.116:0/3509438506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30dc103df0 msgr2=0x7f30dc10a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:53.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.491+0000 7f30e0990700 1 --2- 172.21.15.116:0/3509438506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30dc103df0 0x7f30dc10a6a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f30d40099c0 tx=0x7f30d4009cd0 comp rx=0 tx=0).stop 2024-03-24T10:32:53.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.491+0000 7f30e0990700 1 -- 172.21.15.116:0/3509438506 shutdown_connections 2024-03-24T10:32:53.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.491+0000 7f30e0990700 1 --2- 172.21.15.116:0/3509438506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30dc103df0 0x7f30dc10a6a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:53.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.491+0000 7f30e0990700 1 --2- 172.21.15.116:0/3509438506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30dc102250 0x7f30dc102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:53.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.491+0000 7f30e0990700 1 --2- 172.21.15.116:0/3509438506 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30dc103450 0x7f30dc1038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:53.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.491+0000 7f30e0990700 1 -- 172.21.15.116:0/3509438506 >> 172.21.15.116:0/3509438506 conn(0x7f30dc0fd7b0 msgr2=0x7f30dc0ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:32:53.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.491+0000 7f30e0990700 1 -- 172.21.15.116:0/3509438506 shutdown_connections 2024-03-24T10:32:53.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.491+0000 7f30e0990700 1 -- 172.21.15.116:0/3509438506 wait complete. 2024-03-24T10:32:53.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30e0990700 1 Processor -- start 2024-03-24T10:32:53.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30e0990700 1 -- start start 2024-03-24T10:32:53.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30e0990700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30dc102250 0x7f30dc1981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:53.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30e0990700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30dc103450 0x7f30dc1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:53.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30e0990700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30dc103df0 0x7f30dc19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:53.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30e0990700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc19fd30 con 0x7f30dc103450 2024-03-24T10:32:53.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30e0990700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc19fe70 con 0x7f30dc102250 2024-03-24T10:32:53.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30e0990700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc1a0000 con 0x7f30dc103df0 2024-03-24T10:32:53.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30da59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30dc102250 0x7f30dc1981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:53.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30da59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30dc102250 0x7f30dc1981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41536/0 (socket says 172.21.15.116:41536) 2024-03-24T10:32:53.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30da59c700 1 -- 172.21.15.116:0/4209284621 learned_addr learned my addr 172.21.15.116:0/4209284621 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T10:32:53.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30dad9d700 1 --2- 172.21.15.116:0/4209284621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30dc103df0 0x7f30dc19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:53.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30d1d9b700 1 --2- 172.21.15.116:0/4209284621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30dc103450 0x7f30dc1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:53.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30da59c700 1 -- 172.21.15.116:0/4209284621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30dc103df0 msgr2=0x7f30dc19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:53.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30da59c700 1 --2- 172.21.15.116:0/4209284621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30dc103df0 0x7f30dc19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:53.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30da59c700 1 -- 172.21.15.116:0/4209284621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30dc103450 msgr2=0x7f30dc1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:32:53.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30da59c700 1 --2- 172.21.15.116:0/4209284621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30dc103450 0x7f30dc1986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:32:53.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.492+0000 7f30da59c700 1 -- 172.21.15.116:0/4209284621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d40095f0 con 0x7f30dc102250 2024-03-24T10:32:53.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.493+0000 7f30da59c700 1 --2- 172.21.15.116:0/4209284621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30dc102250 0x7f30dc1981a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00cfd0 tx=0x7f30cc00a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:32:53.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.493+0000 7f30d3fff700 1 -- 172.21.15.116:0/4209284621 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30cc00ada0 con 0x7f30dc102250 2024-03-24T10:32:53.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.493+0000 7f30e0990700 1 -- 172.21.15.116:0/4209284621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30dc1a02e0 con 0x7f30dc102250 2024-03-24T10:32:53.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.493+0000 7f30d3fff700 1 -- 172.21.15.116:0/4209284621 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30cc007e50 con 0x7f30dc102250 2024-03-24T10:32:53.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.493+0000 7f30d3fff700 1 -- 172.21.15.116:0/4209284621 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30cc0116e0 con 0x7f30dc102250 2024-03-24T10:32:53.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.493+0000 7f30e0990700 1 -- 172.21.15.116:0/4209284621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30dc1a07e0 con 0x7f30dc102250 2024-03-24T10:32:53.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.494+0000 7f30e0990700 1 -- 172.21.15.116:0/4209284621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30dc109540 con 0x7f30dc102250 2024-03-24T10:32:53.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.495+0000 7f30d3fff700 1 -- 172.21.15.116:0/4209284621 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7f30cc012420 con 0x7f30dc102250 2024-03-24T10:32:53.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.495+0000 7f30d3fff700 1 --2- 172.21.15.116:0/4209284621 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f30c80a20c0 0x7f30c80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:32:53.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.495+0000 7f30d3fff700 1 -- 172.21.15.116:0/4209284621 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7f30cc0cc540 con 0x7f30dc102250 2024-03-24T10:32:53.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.496+0000 7f30d1d9b700 1 --2- 172.21.15.116:0/4209284621 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f30c80a20c0 0x7f30c80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:32:53.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.496+0000 7f30d1d9b700 1 --2- 172.21.15.116:0/4209284621 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f30c80a20c0 0x7f30c80a4580 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f30c4005950 tx=0x7f30c40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:32:53.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.499+0000 7f30d3fff700 1 -- 172.21.15.116:0/4209284621 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f30cc096f80 con 0x7f30dc102250 2024-03-24T10:32:53.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:32:53.702+0000 7f30e0990700 1 -- 172.21.15.116:0/4209284621 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f30dc02dce0 con 0x7f30c80a20c0 2024-03-24T10:32:54.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:53 smithi117 ceph-mon[36368]: Detected new or changed devices on smithi023 2024-03-24T10:32:54.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:53 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-24T10:32:54.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:53 smithi116 ceph-mon[36248]: Detected new or changed devices on smithi023 2024-03-24T10:32:54.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:53 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-24T10:32:54.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:53 smithi023 ceph-mon[32466]: Detected new or changed devices on smithi023 2024-03-24T10:32:54.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:53 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-24T10:32:55.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:54 smithi117 ceph-mon[36368]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:32:55.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:54 smithi117 ceph-mon[36368]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T10:32:55.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:54 smithi117 ceph-mon[36368]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:32:55.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:54 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:32:55.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:54 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:32:55.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:54 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:55.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:54 smithi116 ceph-mon[36248]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:32:55.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:54 smithi116 ceph-mon[36248]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T10:32:55.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:54 smithi116 ceph-mon[36248]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:32:55.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:54 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:32:55.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:54 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:32:55.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:54 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:55.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:54 smithi023 ceph-mon[32466]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:32:55.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:54 smithi023 ceph-mon[32466]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T10:32:55.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:54 smithi023 ceph-mon[32466]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:32:55.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:54 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:32:55.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:54 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:32:55.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:54 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:32:56.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:55 smithi117 ceph-mon[36368]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:32:56.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:55 smithi116 ceph-mon[36248]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:32:56.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:55 smithi023 ceph-mon[32466]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:32:57.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:56 smithi117 ceph-mon[36368]: from='client.? 172.21.15.116:0/632609209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fd17c0c-e834-46bf-bb82-b247e4d5bb01"}]: dispatch 2024-03-24T10:32:57.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:56 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fd17c0c-e834-46bf-bb82-b247e4d5bb01"}]: dispatch 2024-03-24T10:32:57.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:56 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fd17c0c-e834-46bf-bb82-b247e4d5bb01"}]': finished 2024-03-24T10:32:57.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:56 smithi117 ceph-mon[36368]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T10:32:57.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:56 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:32:57.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:56 smithi116 ceph-mon[36248]: from='client.? 172.21.15.116:0/632609209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fd17c0c-e834-46bf-bb82-b247e4d5bb01"}]: dispatch 2024-03-24T10:32:57.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:56 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fd17c0c-e834-46bf-bb82-b247e4d5bb01"}]: dispatch 2024-03-24T10:32:57.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:56 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fd17c0c-e834-46bf-bb82-b247e4d5bb01"}]': finished 2024-03-24T10:32:57.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:56 smithi116 ceph-mon[36248]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T10:32:57.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:56 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:32:57.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:56 smithi023 ceph-mon[32466]: from='client.? 172.21.15.116:0/632609209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fd17c0c-e834-46bf-bb82-b247e4d5bb01"}]: dispatch 2024-03-24T10:32:57.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:56 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fd17c0c-e834-46bf-bb82-b247e4d5bb01"}]: dispatch 2024-03-24T10:32:57.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:56 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fd17c0c-e834-46bf-bb82-b247e4d5bb01"}]': finished 2024-03-24T10:32:57.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:56 smithi023 ceph-mon[32466]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T10:32:57.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:56 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:32:58.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:57 smithi117 ceph-mon[36368]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:32:58.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:57 smithi117 ceph-mon[36368]: from='client.? 172.21.15.116:0/4103039244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:32:58.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:57 smithi116 ceph-mon[36248]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:32:58.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:57 smithi116 ceph-mon[36248]: from='client.? 172.21.15.116:0/4103039244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:32:58.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:57 smithi023 ceph-mon[32466]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:32:58.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:57 smithi023 ceph-mon[32466]: from='client.? 172.21.15.116:0/4103039244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:33:00.111 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:32:59 smithi116 ceph-mon[36248]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:00.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:32:59 smithi023 ceph-mon[32466]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:00.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:32:59 smithi117 ceph-mon[36368]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:02.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:01 smithi116 ceph-mon[36248]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:02.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:01 smithi023 ceph-mon[32466]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:02.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:01 smithi117 ceph-mon[36368]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:04.061 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:03 smithi116 ceph-mon[36248]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:04.062 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:03 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T10:33:04.062 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:03 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:04.062 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:03 smithi116 ceph-mon[36248]: Deploying daemon osd.1 on smithi116 2024-03-24T10:33:04.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:03 smithi023 ceph-mon[32466]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:04.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T10:33:04.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:04.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:03 smithi023 ceph-mon[32466]: Deploying daemon osd.1 on smithi116 2024-03-24T10:33:04.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:03 smithi117 ceph-mon[36368]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:04.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T10:33:04.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:04.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:03 smithi117 ceph-mon[36368]: Deploying daemon osd.1 on smithi116 2024-03-24T10:33:06.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:05 smithi117 ceph-mon[36368]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:06.084 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:05 smithi116 ceph-mon[36248]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:06.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:05 smithi023 ceph-mon[32466]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:08.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:07 smithi023 ceph-mon[32466]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:08.235 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:07 smithi116 ceph-mon[36248]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:07 smithi117 ceph-mon[36368]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:09.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:09 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:09.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:09 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:09.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:09 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:33:09.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:09 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:09.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:09 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:09.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:09 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:33:10.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:09 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:10.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:09 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:10.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:09 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:33:10.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:10 smithi023 ceph-mon[32466]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:11.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:10 smithi117 ceph-mon[36368]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:11.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:10 smithi116 ceph-mon[36248]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:12.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:33:12.116+0000 7f30d3fff700 1 -- 172.21.15.116:0/4209284621 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f30dc02dce0 con 0x7f30c80a20c0 2024-03-24T10:33:12.117 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 1 on host 'smithi116' 2024-03-24T10:33:12.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:33:12.118+0000 7f30e0990700 1 -- 172.21.15.116:0/4209284621 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f30c80a20c0 msgr2=0x7f30c80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:12.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:33:12.118+0000 7f30e0990700 1 --2- 172.21.15.116:0/4209284621 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f30c80a20c0 0x7f30c80a4580 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f30c4005950 tx=0x7f30c40058e0 comp rx=0 tx=0).stop 2024-03-24T10:33:12.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:33:12.118+0000 7f30e0990700 1 -- 172.21.15.116:0/4209284621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30dc102250 msgr2=0x7f30dc1981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:12.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:33:12.118+0000 7f30e0990700 1 --2- 172.21.15.116:0/4209284621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30dc102250 0x7f30dc1981a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00cfd0 tx=0x7f30cc00a340 comp rx=0 tx=0).stop 2024-03-24T10:33:12.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:33:12.118+0000 7f30e0990700 1 -- 172.21.15.116:0/4209284621 shutdown_connections 2024-03-24T10:33:12.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:33:12.118+0000 7f30e0990700 1 --2- 172.21.15.116:0/4209284621 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f30c80a20c0 0x7f30c80a4580 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:12.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:33:12.118+0000 7f30e0990700 1 --2- 172.21.15.116:0/4209284621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30dc103df0 0x7f30dc19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:12.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:33:12.118+0000 7f30e0990700 1 --2- 172.21.15.116:0/4209284621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30dc102250 0x7f30dc1981a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:12.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:33:12.118+0000 7f30e0990700 1 --2- 172.21.15.116:0/4209284621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30dc103450 0x7f30dc1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:12.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:33:12.118+0000 7f30e0990700 1 -- 172.21.15.116:0/4209284621 >> 172.21.15.116:0/4209284621 conn(0x7f30dc0fd7b0 msgr2=0x7f30dc106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:12.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:33:12.119+0000 7f30e0990700 1 -- 172.21.15.116:0/4209284621 shutdown_connections 2024-03-24T10:33:12.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:33:12.119+0000 7f30e0990700 1 -- 172.21.15.116:0/4209284621 wait complete. 2024-03-24T10:33:12.416 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:12 smithi116 ceph-mon[36248]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:12.416 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:12 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.417 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:12 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.417 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:12 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:12.417 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:12 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:33:12.417 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:12 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.417 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:12 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:33:12.417 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:12 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.417 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:12 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:12 smithi117 ceph-mon[36368]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:12.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:12.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:33:12.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:33:12.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:12 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.665 DEBUG:teuthology.orchestra.run.smithi116:osd.1> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T10:33:12.668 INFO:tasks.cephadm:Deploying osd.2 on smithi117 with /dev/vg_nvme/lv_4... 2024-03-24T10:33:12.669 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T10:33:12.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:12 smithi023 ceph-mon[32466]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:12.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:12.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:33:12.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:33:12.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:12 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:12.913 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:33:13.321 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:13 smithi117 ceph-mon[36368]: from='osd.1 [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T10:33:13.321 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:13 smithi117 ceph-mon[36368]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T10:33:13.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:13 smithi116 ceph-mon[36248]: from='osd.1 [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T10:33:13.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:13 smithi116 ceph-mon[36248]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T10:33:13.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:13 smithi023 ceph-mon[32466]: from='osd.1 [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T10:33:13.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:13 smithi023 ceph-mon[32466]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T10:33:14.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:14 smithi117 ceph-mon[36368]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:14.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:14 smithi117 ceph-mon[36368]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T10:33:14.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:14 smithi117 ceph-mon[36368]: from='osd.1 [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:33:14.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:14 smithi117 ceph-mon[36368]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T10:33:14.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:14 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:33:14.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:14 smithi117 ceph-mon[36368]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:33:14.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:14 smithi023 ceph-mon[32466]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:14.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:14 smithi023 ceph-mon[32466]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T10:33:14.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:14 smithi023 ceph-mon[32466]: from='osd.1 [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:33:14.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:14 smithi023 ceph-mon[32466]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T10:33:14.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:14 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:33:14.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:14 smithi023 ceph-mon[32466]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:33:14.758 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T10:33:14.758 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T10:33:14.758 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-24T10:33:14.758 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-24T10:33:14.758 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012577 s, 834 MB/s 2024-03-24T10:33:14.758 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-24T10:33:14.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:14 smithi116 ceph-mon[36248]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:14 smithi116 ceph-mon[36248]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T10:33:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:14 smithi116 ceph-mon[36248]: from='osd.1 [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:33:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:14 smithi116 ceph-mon[36248]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T10:33:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:33:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:14 smithi116 ceph-mon[36248]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:33:15.354 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:15 smithi117 ceph-mon[36368]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T10:33:15.355 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:15 smithi117 ceph-mon[36368]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T10:33:15.355 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:33:15.355 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:15 smithi117 ceph-mon[36368]: from='osd.1 ' entity='osd.1' 2024-03-24T10:33:15.355 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:33:15.384 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_4 2024-03-24T10:33:15.393 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:15 smithi116 ceph-mon[36248]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T10:33:15.394 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:15 smithi116 ceph-mon[36248]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T10:33:15.394 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:33:15.394 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:15 smithi116 ceph-mon[36248]: from='osd.1 ' entity='osd.1' 2024-03-24T10:33:15.394 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:33:15.394 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:33:15 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-1[40858]: 2024-03-24T10:33:15.083+0000 7f8dd8dbb700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T10:33:15.394 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:33:15 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-1[40858]: 2024-03-24T10:33:15.088+0000 7f8dd13a7700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:33:15.624 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:33:15.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:15 smithi023 ceph-mon[32466]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T10:33:15.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:15 smithi023 ceph-mon[32466]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T10:33:15.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:33:15.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:15 smithi023 ceph-mon[32466]: from='osd.1 ' entity='osd.1' 2024-03-24T10:33:15.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:33:16.605 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:16 smithi117 ceph-mon[36368]: purged_snaps scrub starts 2024-03-24T10:33:16.607 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:16 smithi117 ceph-mon[36368]: purged_snaps scrub ok 2024-03-24T10:33:16.607 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:16 smithi117 ceph-mon[36368]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:16.607 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:16.607 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:16.607 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:33:16.607 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:16.607 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:16.607 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:33:16.608 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:16.608 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:16 smithi117 ceph-mon[36368]: osd.1 [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126] boot 2024-03-24T10:33:16.608 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:16 smithi117 ceph-mon[36368]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T10:33:16.608 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:33:16.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:16 smithi023 ceph-mon[32466]: purged_snaps scrub starts 2024-03-24T10:33:16.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:16 smithi023 ceph-mon[32466]: purged_snaps scrub ok 2024-03-24T10:33:16.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:16 smithi023 ceph-mon[32466]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:16.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:16.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:16.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:33:16.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:16.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:16.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:33:16.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:16.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:16 smithi023 ceph-mon[32466]: osd.1 [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126] boot 2024-03-24T10:33:16.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:16 smithi023 ceph-mon[32466]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T10:33:16.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:33:16.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:16 smithi116 ceph-mon[36248]: purged_snaps scrub starts 2024-03-24T10:33:16.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:16 smithi116 ceph-mon[36248]: purged_snaps scrub ok 2024-03-24T10:33:16.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:16 smithi116 ceph-mon[36248]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T10:33:16.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:16.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:16.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:33:16.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:16.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:16.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:33:16.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:16.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:16 smithi116 ceph-mon[36248]: osd.1 [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126] boot 2024-03-24T10:33:16.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:16 smithi116 ceph-mon[36248]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T10:33:16.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:33:17.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.183+0000 7fd57a756700 1 -- 172.21.15.117:0/1209368208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5741055f0 msgr2=0x7fd5741079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:17.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.183+0000 7fd57a756700 1 --2- 172.21.15.117:0/1209368208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5741055f0 0x7fd5741079e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd568009990 tx=0x7fd568009ca0 comp rx=0 tx=0).stop 2024-03-24T10:33:17.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.183+0000 7fd57a756700 1 -- 172.21.15.117:0/1209368208 shutdown_connections 2024-03-24T10:33:17.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.183+0000 7fd57a756700 1 --2- 172.21.15.117:0/1209368208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd574107f20 0x7fd57410a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:17.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.183+0000 7fd57a756700 1 --2- 172.21.15.117:0/1209368208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd574102cc0 0x7fd5741050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:17.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.183+0000 7fd57a756700 1 --2- 172.21.15.117:0/1209368208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5741055f0 0x7fd5741079e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:17.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.183+0000 7fd57a756700 1 -- 172.21.15.117:0/1209368208 >> 172.21.15.117:0/1209368208 conn(0x7fd5740fc6f0 msgr2=0x7fd5740feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:17.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.183+0000 7fd57a756700 1 -- 172.21.15.117:0/1209368208 shutdown_connections 2024-03-24T10:33:17.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.184+0000 7fd57a756700 1 -- 172.21.15.117:0/1209368208 wait complete. 2024-03-24T10:33:17.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.184+0000 7fd57a756700 1 Processor -- start 2024-03-24T10:33:17.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.184+0000 7fd57a756700 1 -- start start 2024-03-24T10:33:17.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd57a756700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd574102cc0 0x7fd574198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:17.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd57a756700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5741055f0 0x7fd5741987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:17.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd57a756700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd574107f20 0x7fd57419f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:17.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd57a756700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd57419fe00 con 0x7fd574102cc0 2024-03-24T10:33:17.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd57a756700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd57419ff40 con 0x7fd574107f20 2024-03-24T10:33:17.192 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd57a756700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd5741a00d0 con 0x7fd5741055f0 2024-03-24T10:33:17.192 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd5737fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5741055f0 0x7fd5741987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:17.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd5737fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5741055f0 0x7fd5741987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:34618/0 (socket says 172.21.15.117:34618) 2024-03-24T10:33:17.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd5737fe700 1 -- 172.21.15.117:0/2462329273 learned_addr learned my addr 172.21.15.117:0/2462329273 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T10:33:17.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd573fff700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd574102cc0 0x7fd574198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:17.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd578cf3700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd574107f20 0x7fd57419f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:17.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd5737fe700 1 -- 172.21.15.117:0/2462329273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd574107f20 msgr2=0x7fd57419f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:17.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd5737fe700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd574107f20 0x7fd57419f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:17.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd5737fe700 1 -- 172.21.15.117:0/2462329273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd574102cc0 msgr2=0x7fd574198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:17.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd5737fe700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd574102cc0 0x7fd574198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:17.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd5737fe700 1 -- 172.21.15.117:0/2462329273 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5680095f0 con 0x7fd5741055f0 2024-03-24T10:33:17.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd578cf3700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd574107f20 0x7fd57419f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:33:17.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd5737fe700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5741055f0 0x7fd5741987b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd568000c00 tx=0x7fd56800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:17.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd5717fa700 1 -- 172.21.15.117:0/2462329273 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd56801d070 con 0x7fd5741055f0 2024-03-24T10:33:17.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd5717fa700 1 -- 172.21.15.117:0/2462329273 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd56800fb40 con 0x7fd5741055f0 2024-03-24T10:33:17.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd57a756700 1 -- 172.21.15.117:0/2462329273 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5741a0350 con 0x7fd5741055f0 2024-03-24T10:33:17.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd5717fa700 1 -- 172.21.15.117:0/2462329273 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd568017650 con 0x7fd5741055f0 2024-03-24T10:33:17.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.185+0000 7fd57a756700 1 -- 172.21.15.117:0/2462329273 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5741a0760 con 0x7fd5741055f0 2024-03-24T10:33:17.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.186+0000 7fd57a756700 1 -- 172.21.15.117:0/2462329273 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5740fe270 con 0x7fd5741055f0 2024-03-24T10:33:17.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.187+0000 7fd5717fa700 1 -- 172.21.15.117:0/2462329273 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7fd56800fcb0 con 0x7fd5741055f0 2024-03-24T10:33:17.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.188+0000 7fd5717fa700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd55c0a20c0 0x7fd55c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:17.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.188+0000 7fd5717fa700 1 -- 172.21.15.117:0/2462329273 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7fd5680cd930 con 0x7fd5741055f0 2024-03-24T10:33:17.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.188+0000 7fd573fff700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd55c0a20c0 0x7fd55c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:17.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.189+0000 7fd573fff700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd55c0a20c0 0x7fd55c0a4580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd564009a10 tx=0x7fd564009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:17.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.192+0000 7fd5717fa700 1 -- 172.21.15.117:0/2462329273 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd5680981e0 con 0x7fd5741055f0 2024-03-24T10:33:17.395 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:17.395+0000 7fd57a756700 1 -- 172.21.15.117:0/2462329273 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fd57402db60 con 0x7fd55c0a20c0 2024-03-24T10:33:17.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:17 smithi117 ceph-mon[36368]: Detected new or changed devices on smithi116 2024-03-24T10:33:17.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:17 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi116 to 17080M 2024-03-24T10:33:17.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:17 smithi023 ceph-mon[32466]: Detected new or changed devices on smithi116 2024-03-24T10:33:17.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:17 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi116 to 17080M 2024-03-24T10:33:17.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:17 smithi116 ceph-mon[36248]: Detected new or changed devices on smithi116 2024-03-24T10:33:17.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:17 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi116 to 17080M 2024-03-24T10:33:18.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:18 smithi117 ceph-mon[36368]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:18.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:18 smithi117 ceph-mon[36368]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T10:33:18.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:18 smithi117 ceph-mon[36368]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:33:18.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:18 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:33:18.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:18 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:33:18.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:18 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:18.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:18 smithi023 ceph-mon[32466]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:18.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:18 smithi023 ceph-mon[32466]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T10:33:18.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:18 smithi023 ceph-mon[32466]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:33:18.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:18 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:33:18.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:18 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:33:18.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:18 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:18.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:18 smithi116 ceph-mon[36248]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:18.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:18 smithi116 ceph-mon[36248]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T10:33:18.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:18 smithi116 ceph-mon[36248]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:33:18.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:18 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:33:18.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:18 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:33:18.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:18 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:20.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:20 smithi023 ceph-mon[32466]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:20.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:20 smithi023 ceph-mon[32466]: from='client.? 172.21.15.117:0/621006476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ec590e7-eda9-4938-94c9-b1f61f8dfc6f"}]: dispatch 2024-03-24T10:33:20.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:20 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ec590e7-eda9-4938-94c9-b1f61f8dfc6f"}]: dispatch 2024-03-24T10:33:20.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:20 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ec590e7-eda9-4938-94c9-b1f61f8dfc6f"}]': finished 2024-03-24T10:33:20.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:20 smithi023 ceph-mon[32466]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T10:33:20.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:20 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:20.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:20 smithi117 ceph-mon[36368]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:20.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:20 smithi117 ceph-mon[36368]: from='client.? 172.21.15.117:0/621006476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ec590e7-eda9-4938-94c9-b1f61f8dfc6f"}]: dispatch 2024-03-24T10:33:20.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:20 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ec590e7-eda9-4938-94c9-b1f61f8dfc6f"}]: dispatch 2024-03-24T10:33:20.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:20 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ec590e7-eda9-4938-94c9-b1f61f8dfc6f"}]': finished 2024-03-24T10:33:20.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:20 smithi117 ceph-mon[36368]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T10:33:20.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:20 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:20.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:20 smithi116 ceph-mon[36248]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:20.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:20 smithi116 ceph-mon[36248]: from='client.? 172.21.15.117:0/621006476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ec590e7-eda9-4938-94c9-b1f61f8dfc6f"}]: dispatch 2024-03-24T10:33:20.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:20 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ec590e7-eda9-4938-94c9-b1f61f8dfc6f"}]: dispatch 2024-03-24T10:33:20.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:20 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ec590e7-eda9-4938-94c9-b1f61f8dfc6f"}]': finished 2024-03-24T10:33:20.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:20 smithi116 ceph-mon[36248]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T10:33:20.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:20 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:21.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:21 smithi023 ceph-mon[32466]: from='client.? 172.21.15.117:0/1175820134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:33:21.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:21 smithi117 ceph-mon[36368]: from='client.? 172.21.15.117:0/1175820134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:33:21.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:21 smithi116 ceph-mon[36248]: from='client.? 172.21.15.117:0/1175820134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:33:22.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:22 smithi023 ceph-mon[32466]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:22.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:22 smithi117 ceph-mon[36368]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:22.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:22 smithi116 ceph-mon[36248]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:24.428 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:24 smithi117 ceph-mon[36368]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:24.585 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:24 smithi116 ceph-mon[36248]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:24.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:24 smithi023 ceph-mon[32466]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:26.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:26 smithi117 ceph-mon[36368]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:26.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:26 smithi023 ceph-mon[32466]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:26.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:26 smithi116 ceph-mon[36248]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:27.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:27 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T10:33:27.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:27 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:27.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:27 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T10:33:27.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:27 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:27.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:27 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T10:33:27.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:27 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:28.386 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:28 smithi117 ceph-mon[36368]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:28.386 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:28 smithi117 ceph-mon[36368]: Deploying daemon osd.2 on smithi117 2024-03-24T10:33:28.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:28 smithi023 ceph-mon[32466]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:28.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:28 smithi023 ceph-mon[32466]: Deploying daemon osd.2 on smithi117 2024-03-24T10:33:28.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:28 smithi116 ceph-mon[36248]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:28 smithi116 ceph-mon[36248]: Deploying daemon osd.2 on smithi117 2024-03-24T10:33:30.656 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:30 smithi117 ceph-mon[36368]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:30.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:30 smithi023 ceph-mon[32466]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:30.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:30 smithi116 ceph-mon[36248]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:32.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:32 smithi023 ceph-mon[32466]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:32.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:32 smithi117 ceph-mon[36368]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:32.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:32 smithi116 ceph-mon[36248]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:33.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:33 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:33.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:33 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:33.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:33 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:33:33.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:33 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:33.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:33 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:33.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:33 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:33:33.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:33 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:33.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:33 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:33.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:33 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:33:34.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:34 smithi117 ceph-mon[36368]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:34.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:34 smithi023 ceph-mon[32466]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:34.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:34 smithi116 ceph-mon[36248]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:35.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:35.412+0000 7fd5717fa700 1 -- 172.21.15.117:0/2462329273 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd57402db60 con 0x7fd55c0a20c0 2024-03-24T10:33:35.413 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 2 on host 'smithi117' 2024-03-24T10:33:35.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:35.414+0000 7fd57a756700 1 -- 172.21.15.117:0/2462329273 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd55c0a20c0 msgr2=0x7fd55c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:35.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:35.414+0000 7fd57a756700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd55c0a20c0 0x7fd55c0a4580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd564009a10 tx=0x7fd564009190 comp rx=0 tx=0).stop 2024-03-24T10:33:35.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:35.414+0000 7fd57a756700 1 -- 172.21.15.117:0/2462329273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5741055f0 msgr2=0x7fd5741987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:35.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:35.414+0000 7fd57a756700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5741055f0 0x7fd5741987b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd568000c00 tx=0x7fd56800f690 comp rx=0 tx=0).stop 2024-03-24T10:33:35.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:35.414+0000 7fd57a756700 1 -- 172.21.15.117:0/2462329273 shutdown_connections 2024-03-24T10:33:35.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:35.414+0000 7fd57a756700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd55c0a20c0 0x7fd55c0a4580 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:35.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:35.414+0000 7fd57a756700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd574107f20 0x7fd57419f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:35.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:35.414+0000 7fd57a756700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd574102cc0 0x7fd574198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:35.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:35.414+0000 7fd57a756700 1 --2- 172.21.15.117:0/2462329273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5741055f0 0x7fd5741987b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:35.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:35.414+0000 7fd57a756700 1 -- 172.21.15.117:0/2462329273 >> 172.21.15.117:0/2462329273 conn(0x7fd5740fc6f0 msgr2=0x7fd5741036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:35.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:35.414+0000 7fd57a756700 1 -- 172.21.15.117:0/2462329273 shutdown_connections 2024-03-24T10:33:35.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:33:35.414+0000 7fd57a756700 1 -- 172.21.15.117:0/2462329273 wait complete. 2024-03-24T10:33:36.291 DEBUG:teuthology.orchestra.run.smithi117:osd.2> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T10:33:36.311 INFO:tasks.cephadm:Deploying osd.3 on smithi023 with /dev/vg_nvme/lv_3... 2024-03-24T10:33:36.311 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T10:33:36.573 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:33:36.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:36 smithi023 ceph-mon[32466]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:36.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:36.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:36.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:36 smithi023 ceph-mon[32466]: from='osd.2 [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T10:33:36.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:36 smithi023 ceph-mon[32466]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T10:33:36.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:36.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:36.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:36 smithi117 ceph-mon[36368]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:36 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:36 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:36 smithi117 ceph-mon[36368]: from='osd.2 [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T10:33:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:36 smithi117 ceph-mon[36368]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T10:33:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:36 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:36 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:36.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:36 smithi116 ceph-mon[36248]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:36.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:36 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:36.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:36 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:36.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:36 smithi116 ceph-mon[36248]: from='osd.2 [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T10:33:36.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:36 smithi116 ceph-mon[36248]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T10:33:36.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:36 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:36.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:36 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:37.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:37 smithi023 ceph-mon[32466]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T10:33:37.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:37 smithi023 ceph-mon[32466]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T10:33:37.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:37 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:37.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:37 smithi023 ceph-mon[32466]: from='osd.2 [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:33:37.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:37 smithi023 ceph-mon[32466]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:33:37.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:37 smithi117 ceph-mon[36368]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T10:33:37.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:37 smithi117 ceph-mon[36368]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T10:33:37.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:37 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:37.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:37 smithi117 ceph-mon[36368]: from='osd.2 [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:33:37.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:37 smithi117 ceph-mon[36368]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:33:37.753 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:33:37 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-2[41648]: 2024-03-24T10:33:37.424+0000 7fd357ba8700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T10:33:37.754 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:33:37 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-2[41648]: 2024-03-24T10:33:37.428+0000 7fd35419c700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:33:37.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:37 smithi116 ceph-mon[36248]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T10:33:37.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:37 smithi116 ceph-mon[36248]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T10:33:37.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:37 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:37.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:37 smithi116 ceph-mon[36248]: from='osd.2 [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:33:37.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:37 smithi116 ceph-mon[36248]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:33:38.496 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T10:33:38.497 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T10:33:38.497 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-24T10:33:38.497 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-24T10:33:38.497 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0165998 s, 632 MB/s 2024-03-24T10:33:38.497 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-24T10:33:38.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:38 smithi117 ceph-mon[36368]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:38.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:38 smithi117 ceph-mon[36368]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T10:33:38.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:38 smithi117 ceph-mon[36368]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T10:33:38.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:38 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:38.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:38 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:38.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:38 smithi116 ceph-mon[36248]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:38.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:38 smithi116 ceph-mon[36248]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T10:33:38.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:38 smithi116 ceph-mon[36248]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T10:33:38.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:38 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:38.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:38 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:38.833 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:38 smithi023 ceph-mon[32466]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-24T10:33:38.833 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:38 smithi023 ceph-mon[32466]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T10:33:38.833 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:38 smithi023 ceph-mon[32466]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T10:33:38.833 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:38 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:38.833 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:38 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:39.116 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_3 2024-03-24T10:33:39.354 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:33:39.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:39 smithi117 ceph-mon[36368]: purged_snaps scrub starts 2024-03-24T10:33:39.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:39 smithi117 ceph-mon[36368]: purged_snaps scrub ok 2024-03-24T10:33:39.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:39 smithi117 ceph-mon[36368]: osd.2 [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351] boot 2024-03-24T10:33:39.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:39 smithi117 ceph-mon[36368]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T10:33:39.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:39 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:39.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:39 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T10:33:39.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:39 smithi023 ceph-mon[32466]: purged_snaps scrub starts 2024-03-24T10:33:39.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:39 smithi023 ceph-mon[32466]: purged_snaps scrub ok 2024-03-24T10:33:39.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:39 smithi023 ceph-mon[32466]: osd.2 [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351] boot 2024-03-24T10:33:39.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:39 smithi023 ceph-mon[32466]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T10:33:39.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:39 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:39.690 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:39 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T10:33:39.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:39 smithi116 ceph-mon[36248]: purged_snaps scrub starts 2024-03-24T10:33:39.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:39 smithi116 ceph-mon[36248]: purged_snaps scrub ok 2024-03-24T10:33:39.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:39 smithi116 ceph-mon[36248]: osd.2 [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351] boot 2024-03-24T10:33:39.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:39 smithi116 ceph-mon[36248]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T10:33:39.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:39 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:33:39.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:39 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T10:33:40.608 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:33:40.608 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T10:33:40.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T10:33:40.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T10:33:40.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: Detected new or changed devices on smithi117 2024-03-24T10:33:40.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:33:40.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi117 to 17080M 2024-03-24T10:33:40.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:40.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:33:40.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:33:40.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:40.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:33:40.610 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:33:40.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T10:33:40.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T10:33:40.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T10:33:40.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: Detected new or changed devices on smithi117 2024-03-24T10:33:40.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:33:40.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi117 to 17080M 2024-03-24T10:33:40.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:40.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:33:40.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:33:40.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:40.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:33:40.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:33:40.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T10:33:40.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T10:33:40.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T10:33:40.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: Detected new or changed devices on smithi117 2024-03-24T10:33:40.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:33:40.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi117 to 17080M 2024-03-24T10:33:40.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:40.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:33:40.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:33:40.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:40.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:33:40.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:40.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.813+0000 7f414a787700 1 -- 172.21.15.23:0/1524259090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4144103df0 msgr2=0x7f414410a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:40.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.813+0000 7f414a787700 1 --2- 172.21.15.23:0/1524259090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4144103df0 0x7f414410a6a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f413c00b0d0 tx=0x7f413c00b3e0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.813+0000 7f414a787700 1 -- 172.21.15.23:0/1524259090 shutdown_connections 2024-03-24T10:33:40.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.813+0000 7f414a787700 1 --2- 172.21.15.23:0/1524259090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4144102250 0x7f4144102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.813+0000 7f414a787700 1 --2- 172.21.15.23:0/1524259090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4144103df0 0x7f414410a6a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.813+0000 7f414a787700 1 --2- 172.21.15.23:0/1524259090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4144103450 0x7f41441038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.813+0000 7f414a787700 1 -- 172.21.15.23:0/1524259090 >> 172.21.15.23:0/1524259090 conn(0x7f41440fd810 msgr2=0x7f41440ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:33:40.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.814+0000 7f414a787700 1 -- 172.21.15.23:0/1524259090 shutdown_connections 2024-03-24T10:33:40.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.814+0000 7f414a787700 1 -- 172.21.15.23:0/1524259090 wait complete. 2024-03-24T10:33:40.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.814+0000 7f414a787700 1 Processor -- start 2024-03-24T10:33:40.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.814+0000 7f414a787700 1 -- start start 2024-03-24T10:33:40.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f414a787700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4144102250 0x7f4144198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:40.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f414a787700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4144103450 0x7f41441987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:40.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f414a787700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4144103df0 0x7f414419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:40.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f414a787700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f414419fdf0 con 0x7f4144102250 2024-03-24T10:33:40.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f414a787700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f414419ff30 con 0x7f4144103450 2024-03-24T10:33:40.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f414a787700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f41441a00c0 con 0x7f4144103df0 2024-03-24T10:33:40.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f4143fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4144102250 0x7f4144198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:40.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f4143fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4144102250 0x7f4144198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46280/0 (socket says 172.21.15.23:46280) 2024-03-24T10:33:40.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f4143fff700 1 -- 172.21.15.23:0/3165559090 learned_addr learned my addr 172.21.15.23:0/3165559090 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:33:40.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f41437fe700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4144103450 0x7f41441987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:40.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f4148d24700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4144103df0 0x7f414419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:40.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f4143fff700 1 -- 172.21.15.23:0/3165559090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4144103df0 msgr2=0x7f414419f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:40.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f4143fff700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4144103df0 0x7f414419f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f4143fff700 1 -- 172.21.15.23:0/3165559090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4144103450 msgr2=0x7f41441987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:33:40.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f4143fff700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4144103450 0x7f41441987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:33:40.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f4143fff700 1 -- 172.21.15.23:0/3165559090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413c009d00 con 0x7f4144102250 2024-03-24T10:33:40.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f4148d24700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4144103df0 0x7f414419f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:33:40.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f41437fe700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4144103450 0x7f41441987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:33:40.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f4143fff700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4144102250 0x7f4144198260 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f413400d9b0 tx=0x7f413400dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:40.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.815+0000 7f41417fa700 1 -- 172.21.15.23:0/3165559090 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413400be80 con 0x7f4144102250 2024-03-24T10:33:40.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.816+0000 7f414a787700 1 -- 172.21.15.23:0/3165559090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41441a03a0 con 0x7f4144102250 2024-03-24T10:33:40.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.816+0000 7f41417fa700 1 -- 172.21.15.23:0/3165559090 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4134004500 con 0x7f4144102250 2024-03-24T10:33:40.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.816+0000 7f41417fa700 1 -- 172.21.15.23:0/3165559090 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134003be0 con 0x7f4144102250 2024-03-24T10:33:40.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.816+0000 7f414a787700 1 -- 172.21.15.23:0/3165559090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41441a08a0 con 0x7f4144102250 2024-03-24T10:33:40.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.817+0000 7f414a787700 1 -- 172.21.15.23:0/3165559090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41441095b0 con 0x7f4144102250 2024-03-24T10:33:40.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.818+0000 7f41417fa700 1 -- 172.21.15.23:0/3165559090 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7f4134009c00 con 0x7f4144102250 2024-03-24T10:33:40.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.818+0000 7f41417fa700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f412c0a20c0 0x7f412c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:33:40.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.818+0000 7f41437fe700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f412c0a20c0 0x7f412c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:33:40.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.818+0000 7f41417fa700 1 -- 172.21.15.23:0/3165559090 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7f41340cc830 con 0x7f4144102250 2024-03-24T10:33:40.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.819+0000 7f41437fe700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f412c0a20c0 0x7f412c0a4580 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4138005950 tx=0x7f41380058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:33:40.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:40.822+0000 7f41417fa700 1 -- 172.21.15.23:0/3165559090 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4134096cb0 con 0x7f4144102250 2024-03-24T10:33:40.918 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:33:40 smithi023 sudo[49307]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T10:33:40.918 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:33:40 smithi023 sudo[49307]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:33:40.918 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:33:40 smithi023 sudo[49307]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:33:40.918 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:33:40 smithi023 sudo[49307]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:33:41.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:41.040+0000 7f414a787700 1 -- 172.21.15.23:0/3165559090 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f414402dca0 con 0x7f412c0a20c0 2024-03-24T10:33:41.418 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:33:41 smithi023 sudo[49347]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T10:33:41.418 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:33:41 smithi023 sudo[49347]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:33:41.418 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:33:41 smithi023 sudo[49347]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:33:41.418 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:33:41 smithi023 sudo[49347]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:33:41.594 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:33:41 smithi116 sudo[44874]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T10:33:41.595 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:33:41 smithi116 sudo[44874]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:33:41.595 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:33:41 smithi116 sudo[44874]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:33:41.595 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:33:41 smithi116 sudo[44874]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:33:41.595 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T10:33:41.595 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:41 smithi116 ceph-mon[36248]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T10:33:41.595 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:33:41.595 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:33:41.595 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:41.596 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:41.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T10:33:41.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:41 smithi117 ceph-mon[36368]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T10:33:41.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:33:41.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:33:41.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:41.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:41.755 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T10:33:41.756 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:41 smithi023 ceph-mon[32466]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T10:33:41.756 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:33:41.756 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:33:41.756 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:41.756 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:42.017 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:33:41 smithi116 sudo[44877]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T10:33:42.017 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:33:41 smithi116 sudo[44877]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:33:42.018 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:33:41 smithi116 sudo[44877]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:33:42.018 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:33:41 smithi116 sudo[44877]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:33:42.120 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:33:41 smithi117 sudo[45742]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T10:33:42.120 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:33:41 smithi117 sudo[45742]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:33:42.120 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:33:41 smithi117 sudo[45742]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:33:42.120 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:33:41 smithi117 sudo[45742]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:33:42.502 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:33:42 smithi117 sudo[45745]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T10:33:42.503 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:33:42 smithi117 sudo[45745]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:33:42.503 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:33:42 smithi117 sudo[45745]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:33:42.503 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:33:42 smithi117 sudo[45745]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:33:42.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:42 smithi023 sudo[49477]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T10:33:42.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:42 smithi023 sudo[49477]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:33:42.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:42 smithi023 sudo[49477]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:33:42.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:42 smithi023 sudo[49477]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:33:42.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:33:42.686+0000 7f41417fa700 1 -- 172.21.15.23:0/3165559090 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f4134097080 con 0x7f4144102250 2024-03-24T10:33:42.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:42 smithi117 ceph-mon[36368]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:33:42.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:42 smithi117 ceph-mon[36368]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:33:42.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:42 smithi117 ceph-mon[36368]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T10:33:42.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:42 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:42.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:42 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:42.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:42 smithi117 sudo[45748]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T10:33:42.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:42 smithi117 sudo[45748]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:33:42.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:42 smithi117 sudo[45748]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:33:42.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:42 smithi117 sudo[45748]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:33:42.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:42 smithi023 sudo[49502]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T10:33:42.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:42 smithi023 sudo[49502]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:33:42.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:42 smithi023 sudo[49502]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:33:42.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:42 smithi023 sudo[49502]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:33:42.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:42 smithi023 ceph-mon[32466]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:33:42.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:42 smithi023 ceph-mon[32466]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:33:42.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:42 smithi023 ceph-mon[32466]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T10:33:42.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:42 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:42.932 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:42 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:43.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:42 smithi116 ceph-mon[36248]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:33:43.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:42 smithi116 ceph-mon[36248]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:33:43.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:42 smithi116 ceph-mon[36248]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T10:33:43.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:42 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:43.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:42 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:43.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 sudo[45751]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T10:33:43.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 sudo[45751]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:33:43.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 sudo[45751]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:33:43.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 sudo[45751]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:33:43.659 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 sudo[44880]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T10:33:43.659 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 sudo[44880]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:33:43.659 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 sudo[44880]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:33:43.659 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 sudo[44880]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:33:43.659 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 sudo[44883]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T10:33:43.659 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 sudo[44883]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T10:33:43.659 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 sudo[44883]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T10:33:43.659 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 sudo[44883]: pam_unix(sudo:session): session closed for user root 2024-03-24T10:33:43.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T10:33:43.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T10:33:43.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:33:43.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:33:43.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:33:43.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T10:33:43.993 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-24T10:33:43.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:33:43.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:33:43.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:33:43.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T10:33:43.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T10:33:43.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:33:43.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:33:43.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:33:43.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1872333109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61dcad05-5af6-4faf-860c-54f5e3566b84"}]: dispatch 2024-03-24T10:33:43.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T10:33:43.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1872333109' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61dcad05-5af6-4faf-860c-54f5e3566b84"}]': finished 2024-03-24T10:33:43.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T10:33:43.994 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:33:44.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T10:33:44.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T10:33:44.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:33:44.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:33:44.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:33:44.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T10:33:44.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-24T10:33:44.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:33:44.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:33:44.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:33:44.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T10:33:44.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T10:33:44.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:33:44.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:33:44.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:33:44.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1872333109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61dcad05-5af6-4faf-860c-54f5e3566b84"}]: dispatch 2024-03-24T10:33:44.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T10:33:44.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1872333109' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61dcad05-5af6-4faf-860c-54f5e3566b84"}]': finished 2024-03-24T10:33:44.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T10:33:44.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:33:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T10:33:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T10:33:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:33:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:33:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:33:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T10:33:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-03-24T10:33:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:33:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:33:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:33:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T10:33:44.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T10:33:44.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:33:44.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:33:44.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:33:44.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1872333109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61dcad05-5af6-4faf-860c-54f5e3566b84"}]: dispatch 2024-03-24T10:33:44.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T10:33:44.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1872333109' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61dcad05-5af6-4faf-860c-54f5e3566b84"}]': finished 2024-03-24T10:33:44.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T10:33:44.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:33:45.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:44 smithi117 ceph-mon[36368]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:33:45.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:44 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2772751940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:33:45.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:44 smithi116 ceph-mon[36248]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:33:45.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:44 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2772751940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:33:45.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:44 smithi023 ceph-mon[32466]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:33:45.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:44 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2772751940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:33:46.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:46 smithi023 ceph-mon[32466]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:33:46.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:46 smithi117 ceph-mon[36368]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:33:46.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:46 smithi116 ceph-mon[36248]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:33:48.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:48 smithi023 ceph-mon[32466]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.7 KiB/s rd, 247 KiB/s wr, 17 op/s 2024-03-24T10:33:48.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:48 smithi117 ceph-mon[36368]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.7 KiB/s rd, 247 KiB/s wr, 17 op/s 2024-03-24T10:33:48.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:48 smithi116 ceph-mon[36248]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.7 KiB/s rd, 247 KiB/s wr, 17 op/s 2024-03-24T10:33:50.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:50 smithi023 ceph-mon[32466]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.1 KiB/s rd, 179 KiB/s wr, 12 op/s 2024-03-24T10:33:50.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:50 smithi117 ceph-mon[36368]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.1 KiB/s rd, 179 KiB/s wr, 12 op/s 2024-03-24T10:33:50.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:50 smithi116 ceph-mon[36248]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.1 KiB/s rd, 179 KiB/s wr, 12 op/s 2024-03-24T10:33:52.650 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:52 smithi023 ceph-mon[32466]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 141 KiB/s wr, 9 op/s 2024-03-24T10:33:52.650 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T10:33:52.650 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:52.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:52 smithi117 ceph-mon[36368]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 141 KiB/s wr, 9 op/s 2024-03-24T10:33:52.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T10:33:52.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:52.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:52 smithi116 ceph-mon[36248]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 141 KiB/s wr, 9 op/s 2024-03-24T10:33:52.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T10:33:52.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:33:53.479 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:53 smithi023 ceph-mon[32466]: Deploying daemon osd.3 on smithi023 2024-03-24T10:33:53.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:53 smithi117 ceph-mon[36368]: Deploying daemon osd.3 on smithi023 2024-03-24T10:33:53.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:53 smithi116 ceph-mon[36248]: Deploying daemon osd.3 on smithi023 2024-03-24T10:33:54.654 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:54 smithi117 ceph-mon[36368]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T10:33:54.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:54 smithi023 ceph-mon[32466]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T10:33:54.668 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:54 smithi116 ceph-mon[36248]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-24T10:33:56.435 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:56 smithi023 ceph-mon[32466]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 118 KiB/s wr, 8 op/s 2024-03-24T10:33:56.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:56 smithi117 ceph-mon[36368]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 118 KiB/s wr, 8 op/s 2024-03-24T10:33:56.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:56 smithi116 ceph-mon[36248]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 118 KiB/s wr, 8 op/s 2024-03-24T10:33:58.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:57 smithi117 ceph-mon[36368]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-24T10:33:58.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:57 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:58.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:57 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:58.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:33:57 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:33:58.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:57 smithi116 ceph-mon[36248]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-24T10:33:58.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:57 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:58.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:57 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:58.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:33:57 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:33:58.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:57 smithi023 ceph-mon[32466]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-24T10:33:58.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:57 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:58.352 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:57 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:33:58.352 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:33:57 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:00.416 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 3 on host 'smithi023' 2024-03-24T10:34:00.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:00.415+0000 7f41417fa700 1 -- 172.21.15.23:0/3165559090 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f414402dca0 con 0x7f412c0a20c0 2024-03-24T10:34:00.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:00.417+0000 7f414a787700 1 -- 172.21.15.23:0/3165559090 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f412c0a20c0 msgr2=0x7f412c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:00.417+0000 7f414a787700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f412c0a20c0 0x7f412c0a4580 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4138005950 tx=0x7f41380058e0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:00.417+0000 7f414a787700 1 -- 172.21.15.23:0/3165559090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4144102250 msgr2=0x7f4144198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:00.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:00.417+0000 7f414a787700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4144102250 0x7f4144198260 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f413400d9b0 tx=0x7f413400dd70 comp rx=0 tx=0).stop 2024-03-24T10:34:00.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:00.417+0000 7f414a787700 1 -- 172.21.15.23:0/3165559090 shutdown_connections 2024-03-24T10:34:00.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:00.417+0000 7f414a787700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f412c0a20c0 0x7f412c0a4580 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:00.417+0000 7f414a787700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4144103df0 0x7f414419f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:00.417+0000 7f414a787700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4144102250 0x7f4144198260 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:00.417+0000 7f414a787700 1 --2- 172.21.15.23:0/3165559090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4144103450 0x7f41441987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:00.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:00.417+0000 7f414a787700 1 -- 172.21.15.23:0/3165559090 >> 172.21.15.23:0/3165559090 conn(0x7f41440fd810 msgr2=0x7f41441066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:00.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:00.417+0000 7f414a787700 1 -- 172.21.15.23:0/3165559090 shutdown_connections 2024-03-24T10:34:00.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:00.417+0000 7f414a787700 1 -- 172.21.15.23:0/3165559090 wait complete. 2024-03-24T10:34:00.883 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:00 smithi023 ceph-mon[32466]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:34:00.883 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:00.883 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:00.883 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:00.884 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:00.884 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:00.884 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:00.884 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:00.884 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:00.918 DEBUG:teuthology.orchestra.run.smithi023:osd.3> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T10:34:00.920 INFO:tasks.cephadm:Deploying osd.4 on smithi116 with /dev/vg_nvme/lv_3... 2024-03-24T10:34:00.920 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T10:34:00.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:00 smithi116 ceph-mon[36248]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:34:00.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:00.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:00.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:00.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:00.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:00.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:00.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:00.956 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:01.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:00 smithi117 ceph-mon[36368]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:34:01.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:01.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:01.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:01.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:01.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:01.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:01.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:01.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:01.165 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T10:34:01.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:01 smithi023 ceph-mon[32466]: from='osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T10:34:02.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:01 smithi117 ceph-mon[36368]: from='osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T10:34:02.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:01 smithi116 ceph-mon[36248]: from='osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T10:34:02.832 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T10:34:02.832 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T10:34:02.832 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-24T10:34:02.832 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-24T10:34:02.832 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113536 s, 924 MB/s 2024-03-24T10:34:02.832 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-24T10:34:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:02 smithi023 ceph-mon[32466]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:34:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:02 smithi023 ceph-mon[32466]: from='osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T10:34:02.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:02 smithi023 ceph-mon[32466]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T10:34:02.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:02 smithi023 ceph-mon[32466]: from='osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T10:34:02.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:02 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:02.919 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:34:02 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3[51619]: 2024-03-24T10:34:02.619+0000 7f4fda300700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T10:34:02.919 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:34:02 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3[51619]: 2024-03-24T10:34:02.624+0000 7f4fd68f4700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:34:03.008 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:02 smithi117 ceph-mon[36368]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:34:03.008 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:02 smithi117 ceph-mon[36368]: from='osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T10:34:03.008 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:02 smithi117 ceph-mon[36368]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T10:34:03.008 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:02 smithi117 ceph-mon[36368]: from='osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T10:34:03.008 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:02 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:03.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:02 smithi116 ceph-mon[36248]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:34:03.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:02 smithi116 ceph-mon[36248]: from='osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T10:34:03.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:02 smithi116 ceph-mon[36248]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T10:34:03.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:02 smithi116 ceph-mon[36248]: from='osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T10:34:03.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:02 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:03.407 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_3 2024-03-24T10:34:03.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:03 smithi023 ceph-mon[32466]: from='osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T10:34:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:03 smithi023 ceph-mon[32466]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T10:34:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:03.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:03.688 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T10:34:03.920 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:03 smithi116 ceph-mon[36248]: from='osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T10:34:03.920 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:03 smithi116 ceph-mon[36248]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T10:34:03.920 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:03 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:03.920 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:03 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:04.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:03 smithi117 ceph-mon[36368]: from='osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T10:34:04.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:03 smithi117 ceph-mon[36368]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T10:34:04.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:04.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:04.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: purged_snaps scrub starts 2024-03-24T10:34:04.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: purged_snaps scrub ok 2024-03-24T10:34:04.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:34:04.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:04.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549] boot 2024-03-24T10:34:04.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: osdmap e25: 4 total, 4 up, 4 in 2024-03-24T10:34:04.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:04.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:04.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:04.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:04.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:04.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:04.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:04.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:04.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:04 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:04.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: purged_snaps scrub starts 2024-03-24T10:34:04.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: purged_snaps scrub ok 2024-03-24T10:34:04.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:34:04.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:04.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549] boot 2024-03-24T10:34:04.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: osdmap e25: 4 total, 4 up, 4 in 2024-03-24T10:34:04.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:04.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:04.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:04.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:04.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:04.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:04.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:04.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:04.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:04 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:05.006 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: purged_snaps scrub starts 2024-03-24T10:34:05.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: purged_snaps scrub ok 2024-03-24T10:34:05.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T10:34:05.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:05.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: osd.3 [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549] boot 2024-03-24T10:34:05.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: osdmap e25: 4 total, 4 up, 4 in 2024-03-24T10:34:05.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:34:05.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:05.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:05.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:05.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:05.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:05.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:05.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:05.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:04 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:05.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.114+0000 7fbea365e700 1 -- 172.21.15.116:0/4108994264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9c074c60 msgr2=0x7fbe9c0750e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.114+0000 7fbea365e700 1 --2- 172.21.15.116:0/4108994264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9c074c60 0x7fbe9c0750e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009990 tx=0x7fbe8c009ca0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.114+0000 7fbea365e700 1 -- 172.21.15.116:0/4108994264 shutdown_connections 2024-03-24T10:34:05.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.114+0000 7fbea365e700 1 --2- 172.21.15.116:0/4108994264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe9c076230 0x7fbe9c074690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.114+0000 7fbea365e700 1 --2- 172.21.15.116:0/4108994264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c10e850 0x7fbe9c110c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.114+0000 7fbea365e700 1 --2- 172.21.15.116:0/4108994264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9c074c60 0x7fbe9c0750e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.114+0000 7fbea365e700 1 -- 172.21.15.116:0/4108994264 >> 172.21.15.116:0/4108994264 conn(0x7fbe9c0fd850 msgr2=0x7fbe9c0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:05.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.114+0000 7fbea365e700 1 -- 172.21.15.116:0/4108994264 shutdown_connections 2024-03-24T10:34:05.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.114+0000 7fbea365e700 1 -- 172.21.15.116:0/4108994264 wait complete. 2024-03-24T10:34:05.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea365e700 1 Processor -- start 2024-03-24T10:34:05.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea365e700 1 -- start start 2024-03-24T10:34:05.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea365e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe9c074c60 0x7fbe9c19e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea365e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c076230 0x7fbe9c19ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea365e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9c10e850 0x7fbe9c1a5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea365e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c1a63c0 con 0x7fbe9c076230 2024-03-24T10:34:05.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea365e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c1a6500 con 0x7fbe9c10e850 2024-03-24T10:34:05.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea365e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c1a6690 con 0x7fbe9c074c60 2024-03-24T10:34:05.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea1bfb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9c10e850 0x7fbe9c1a5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea1bfb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9c10e850 0x7fbe9c1a5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41916/0 (socket says 172.21.15.116:41916) 2024-03-24T10:34:05.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea1bfb700 1 -- 172.21.15.116:0/3131087352 learned_addr learned my addr 172.21.15.116:0/3131087352 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T10:34:05.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea0bf9700 1 --2- 172.21.15.116:0/3131087352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c076230 0x7fbe9c19ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea13fa700 1 --2- 172.21.15.116:0/3131087352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe9c074c60 0x7fbe9c19e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea1bfb700 1 -- 172.21.15.116:0/3131087352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe9c074c60 msgr2=0x7fbe9c19e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea1bfb700 1 --2- 172.21.15.116:0/3131087352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe9c074c60 0x7fbe9c19e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea1bfb700 1 -- 172.21.15.116:0/3131087352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c076230 msgr2=0x7fbe9c19ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:05.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea1bfb700 1 --2- 172.21.15.116:0/3131087352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c076230 0x7fbe9c19ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:05.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.115+0000 7fbea1bfb700 1 -- 172.21.15.116:0/3131087352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe8c0095f0 con 0x7fbe9c10e850 2024-03-24T10:34:05.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.116+0000 7fbea1bfb700 1 --2- 172.21.15.116:0/3131087352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9c10e850 0x7fbe9c1a5df0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400c970 tx=0x7fbe9400cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:05.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.116+0000 7fbe927fc700 1 -- 172.21.15.116:0/3131087352 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe94007b00 con 0x7fbe9c10e850 2024-03-24T10:34:05.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.116+0000 7fbe927fc700 1 -- 172.21.15.116:0/3131087352 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe94004d10 con 0x7fbe9c10e850 2024-03-24T10:34:05.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.116+0000 7fbea365e700 1 -- 172.21.15.116:0/3131087352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe9c1a6970 con 0x7fbe9c10e850 2024-03-24T10:34:05.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.116+0000 7fbe927fc700 1 -- 172.21.15.116:0/3131087352 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe94005580 con 0x7fbe9c10e850 2024-03-24T10:34:05.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.116+0000 7fbea365e700 1 -- 172.21.15.116:0/3131087352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe9c1a6e20 con 0x7fbe9c10e850 2024-03-24T10:34:05.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.117+0000 7fbea365e700 1 -- 172.21.15.116:0/3131087352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe9c104ab0 con 0x7fbe9c10e850 2024-03-24T10:34:05.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.118+0000 7fbe927fc700 1 -- 172.21.15.116:0/3131087352 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fbe94004750 con 0x7fbe9c10e850 2024-03-24T10:34:05.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.118+0000 7fbe927fc700 1 --2- 172.21.15.116:0/3131087352 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbe880a2150 0x7fbe880a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:05.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.119+0000 7fbe927fc700 1 -- 172.21.15.116:0/3131087352 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3698+0+0 (secure 0 0 0) 0x7fbe940ccaf0 con 0x7fbe9c10e850 2024-03-24T10:34:05.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.119+0000 7fbea13fa700 1 --2- 172.21.15.116:0/3131087352 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbe880a2150 0x7fbe880a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:05.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.119+0000 7fbea13fa700 1 --2- 172.21.15.116:0/3131087352 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbe880a2150 0x7fbe880a4610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe980060b0 tx=0x7fbe98007790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:05.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.122+0000 7fbe927fc700 1 -- 172.21.15.116:0/3131087352 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbe94096cc0 con 0x7fbe9c10e850 2024-03-24T10:34:05.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:05.325+0000 7fbea365e700 1 -- 172.21.15.116:0/3131087352 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fbe9c02dc00 con 0x7fbe880a2150 2024-03-24T10:34:05.912 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:05 smithi117 ceph-mon[36368]: Detected new or changed devices on smithi023 2024-03-24T10:34:05.912 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:05 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi023 to 8544M 2024-03-24T10:34:05.912 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:05 smithi117 ceph-mon[36368]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T10:34:05.913 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:05 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:34:05.913 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:05 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:34:05.913 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:05 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:05.913 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:05 smithi117 ceph-mon[36368]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T10:34:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:05 smithi023 ceph-mon[32466]: Detected new or changed devices on smithi023 2024-03-24T10:34:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:05 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi023 to 8544M 2024-03-24T10:34:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:05 smithi023 ceph-mon[32466]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T10:34:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:05 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:34:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:05 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:34:05.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:05 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:05.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:05 smithi023 ceph-mon[32466]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T10:34:05.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:05 smithi116 ceph-mon[36248]: Detected new or changed devices on smithi023 2024-03-24T10:34:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:05 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi023 to 8544M 2024-03-24T10:34:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:05 smithi116 ceph-mon[36248]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T10:34:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:05 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:34:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:05 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:34:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:05 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:05 smithi116 ceph-mon[36248]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T10:34:06.894 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:06 smithi116 ceph-mon[36248]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:06.894 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:06 smithi116 ceph-mon[36248]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:06.898 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:06 smithi116 ceph-mon[36248]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T10:34:06.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:06 smithi023 ceph-mon[32466]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:06.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:06 smithi023 ceph-mon[32466]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:06.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:06 smithi023 ceph-mon[32466]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T10:34:07.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:06 smithi117 ceph-mon[36368]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:07.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:06 smithi117 ceph-mon[36368]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:07.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:06 smithi117 ceph-mon[36368]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T10:34:08.396 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:08 smithi116 ceph-mon[36248]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T10:34:08.396 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:08 smithi116 ceph-mon[36248]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T10:34:08.397 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:08 smithi116 ceph-mon[36248]: from='client.? 172.21.15.116:0/75693068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d56de8f-6140-477e-b851-42928411277d"}]: dispatch 2024-03-24T10:34:08.397 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:08 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d56de8f-6140-477e-b851-42928411277d"}]: dispatch 2024-03-24T10:34:08.397 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:08 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d56de8f-6140-477e-b851-42928411277d"}]': finished 2024-03-24T10:34:08.397 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:08 smithi116 ceph-mon[36248]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T10:34:08.397 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:08 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:08.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:08 smithi023 ceph-mon[32466]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T10:34:08.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:08 smithi023 ceph-mon[32466]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T10:34:08.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:08 smithi023 ceph-mon[32466]: from='client.? 172.21.15.116:0/75693068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d56de8f-6140-477e-b851-42928411277d"}]: dispatch 2024-03-24T10:34:08.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:08 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d56de8f-6140-477e-b851-42928411277d"}]: dispatch 2024-03-24T10:34:08.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:08 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d56de8f-6140-477e-b851-42928411277d"}]': finished 2024-03-24T10:34:08.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:08 smithi023 ceph-mon[32466]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T10:34:08.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:08 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:08.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:08 smithi117 ceph-mon[36368]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T10:34:08.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:08 smithi117 ceph-mon[36368]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T10:34:08.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:08 smithi117 ceph-mon[36368]: from='client.? 172.21.15.116:0/75693068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d56de8f-6140-477e-b851-42928411277d"}]: dispatch 2024-03-24T10:34:08.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:08 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d56de8f-6140-477e-b851-42928411277d"}]: dispatch 2024-03-24T10:34:08.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:08 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d56de8f-6140-477e-b851-42928411277d"}]': finished 2024-03-24T10:34:08.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:08 smithi117 ceph-mon[36368]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T10:34:08.504 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:08 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:09.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:09 smithi023 ceph-mon[32466]: from='client.? 172.21.15.116:0/2906567465' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:34:09.504 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:09 smithi117 ceph-mon[36368]: from='client.? 172.21.15.116:0/2906567465' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:34:09.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:09 smithi116 ceph-mon[36248]: from='client.? 172.21.15.116:0/2906567465' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:34:10.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:10 smithi023 ceph-mon[32466]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T10:34:10.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:10 smithi117 ceph-mon[36368]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T10:34:10.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:10 smithi116 ceph-mon[36248]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T10:34:12.252 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:12 smithi116 ceph-mon[36248]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-24T10:34:12.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:12 smithi023 ceph-mon[32466]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-24T10:34:12.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:12 smithi117 ceph-mon[36368]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-24T10:34:14.237 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:14 smithi116 ceph-mon[36248]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T10:34:14.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:14 smithi023 ceph-mon[32466]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T10:34:14.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:14 smithi117 ceph-mon[36368]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T10:34:16.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:16 smithi116 ceph-mon[36248]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:16.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T10:34:16.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:16.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:16 smithi116 ceph-mon[36248]: Deploying daemon osd.4 on smithi116 2024-03-24T10:34:16.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:16 smithi023 ceph-mon[32466]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:16.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T10:34:16.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:16.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:16 smithi023 ceph-mon[32466]: Deploying daemon osd.4 on smithi116 2024-03-24T10:34:16.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:16 smithi117 ceph-mon[36368]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:16.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T10:34:16.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:16.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:16 smithi117 ceph-mon[36368]: Deploying daemon osd.4 on smithi116 2024-03-24T10:34:18.263 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:18 smithi116 ceph-mon[36248]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:18.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:18 smithi023 ceph-mon[32466]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:18.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:18 smithi117 ceph-mon[36368]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:20.308 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:20 smithi116 ceph-mon[36248]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:20.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:20 smithi023 ceph-mon[32466]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:20.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:20 smithi117 ceph-mon[36368]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:21.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:21.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:21.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:21.728 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:21.728 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:21.728 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:21.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:21.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:21.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:22.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:22 smithi116 ceph-mon[36248]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:22.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:22 smithi023 ceph-mon[32466]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:22.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:22 smithi117 ceph-mon[36368]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:23.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:23.988+0000 7fbe927fc700 1 -- 172.21.15.116:0/3131087352 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbe9c02dc00 con 0x7fbe880a2150 2024-03-24T10:34:23.989 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 4 on host 'smithi116' 2024-03-24T10:34:23.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:23.990+0000 7fbea365e700 1 -- 172.21.15.116:0/3131087352 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbe880a2150 msgr2=0x7fbe880a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:23.990+0000 7fbea365e700 1 --2- 172.21.15.116:0/3131087352 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbe880a2150 0x7fbe880a4610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe980060b0 tx=0x7fbe98007790 comp rx=0 tx=0).stop 2024-03-24T10:34:23.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:23.990+0000 7fbea365e700 1 -- 172.21.15.116:0/3131087352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9c10e850 msgr2=0x7fbe9c1a5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:23.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:23.990+0000 7fbea365e700 1 --2- 172.21.15.116:0/3131087352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9c10e850 0x7fbe9c1a5df0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400c970 tx=0x7fbe9400cd30 comp rx=0 tx=0).stop 2024-03-24T10:34:23.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:23.990+0000 7fbea365e700 1 -- 172.21.15.116:0/3131087352 shutdown_connections 2024-03-24T10:34:23.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:23.990+0000 7fbea365e700 1 --2- 172.21.15.116:0/3131087352 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbe880a2150 0x7fbe880a4610 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:23.990+0000 7fbea365e700 1 --2- 172.21.15.116:0/3131087352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe9c074c60 0x7fbe9c19e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:23.990+0000 7fbea365e700 1 --2- 172.21.15.116:0/3131087352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9c10e850 0x7fbe9c1a5df0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:23.990+0000 7fbea365e700 1 --2- 172.21.15.116:0/3131087352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe9c076230 0x7fbe9c19ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:23.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:23.990+0000 7fbea365e700 1 -- 172.21.15.116:0/3131087352 >> 172.21.15.116:0/3131087352 conn(0x7fbe9c0fd850 msgr2=0x7fbe9c101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:23.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:23.991+0000 7fbea365e700 1 -- 172.21.15.116:0/3131087352 shutdown_connections 2024-03-24T10:34:23.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:34:23.991+0000 7fbea365e700 1 -- 172.21.15.116:0/3131087352 wait complete. 2024-03-24T10:34:24.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:24 smithi116 ceph-mon[36248]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:24.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:24.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:24.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.243 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:24.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:24 smithi023 ceph-mon[32466]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:24.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:24.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:24.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:24.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:24 smithi117 ceph-mon[36368]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:24.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:24.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:24.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:24.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:24.573 DEBUG:teuthology.orchestra.run.smithi116:osd.4> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T10:34:24.576 INFO:tasks.cephadm:Deploying osd.5 on smithi117 with /dev/vg_nvme/lv_3... 2024-03-24T10:34:24.576 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T10:34:24.824 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:34:25.199 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:25 smithi117 ceph-mon[36368]: from='osd.4 [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T10:34:25.199 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:25 smithi117 ceph-mon[36368]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T10:34:25.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:25 smithi116 ceph-mon[36248]: from='osd.4 [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T10:34:25.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:25 smithi116 ceph-mon[36248]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T10:34:25.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:25 smithi023 ceph-mon[32466]: from='osd.4 [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T10:34:25.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:25 smithi023 ceph-mon[32466]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T10:34:26.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:26 smithi117 ceph-mon[36368]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:26.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:26 smithi117 ceph-mon[36368]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T10:34:26.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:26 smithi117 ceph-mon[36368]: from='osd.4 [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:34:26.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:26 smithi117 ceph-mon[36368]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T10:34:26.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:26 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:26.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:26 smithi117 ceph-mon[36368]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:34:26.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:26 smithi116 ceph-mon[36248]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:26.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:26 smithi116 ceph-mon[36248]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T10:34:26.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:26 smithi116 ceph-mon[36248]: from='osd.4 [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:34:26.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:26 smithi116 ceph-mon[36248]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T10:34:26.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:26 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:26.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:26 smithi116 ceph-mon[36248]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:34:26.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:26 smithi023 ceph-mon[32466]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:26.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:26 smithi023 ceph-mon[32466]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T10:34:26.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:26 smithi023 ceph-mon[32466]: from='osd.4 [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:34:26.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:26 smithi023 ceph-mon[32466]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T10:34:26.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:26 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:26.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:26 smithi023 ceph-mon[32466]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:34:26.537 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T10:34:26.537 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T10:34:26.537 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-24T10:34:26.537 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-24T10:34:26.537 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112555 s, 932 MB/s 2024-03-24T10:34:26.537 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-24T10:34:27.093 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_3 2024-03-24T10:34:27.155 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:27 smithi116 ceph-mon[36248]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T10:34:27.156 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:27 smithi116 ceph-mon[36248]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T10:34:27.156 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:27 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:27.156 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:27 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:27.156 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:27 smithi116 ceph-mon[36248]: from='osd.4 ' entity='osd.4' 2024-03-24T10:34:27.156 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:34:26 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-4[47712]: 2024-03-24T10:34:26.862+0000 7f17999be700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T10:34:27.156 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:34:26 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-4[47712]: 2024-03-24T10:34:26.871+0000 7f1791faa700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:34:27.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:27 smithi117 ceph-mon[36368]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T10:34:27.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:27 smithi117 ceph-mon[36368]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T10:34:27.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:27 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:27.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:27 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:27.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:27 smithi117 ceph-mon[36368]: from='osd.4 ' entity='osd.4' 2024-03-24T10:34:27.334 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:34:27.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:27 smithi023 ceph-mon[32466]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T10:34:27.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:27 smithi023 ceph-mon[32466]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T10:34:27.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:27 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:27.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:27 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:27.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:27 smithi023 ceph-mon[32466]: from='osd.4 ' entity='osd.4' 2024-03-24T10:34:28.314 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: purged_snaps scrub starts 2024-03-24T10:34:28.314 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: purged_snaps scrub ok 2024-03-24T10:34:28.314 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:28.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:28.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: Detected new or changed devices on smithi116 2024-03-24T10:34:28.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:28.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:28.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:28.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:28.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi116 to 8540M 2024-03-24T10:34:28.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:28.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:28.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:28.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:28.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: osd.4 [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953] boot 2024-03-24T10:34:28.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T10:34:28.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:28.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: purged_snaps scrub starts 2024-03-24T10:34:28.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: purged_snaps scrub ok 2024-03-24T10:34:28.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:28.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:28.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: Detected new or changed devices on smithi116 2024-03-24T10:34:28.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:28.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:28.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:28.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:28.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi116 to 8540M 2024-03-24T10:34:28.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:28.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:28.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:28.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:28.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: osd.4 [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953] boot 2024-03-24T10:34:28.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T10:34:28.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:28.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: purged_snaps scrub starts 2024-03-24T10:34:28.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: purged_snaps scrub ok 2024-03-24T10:34:28.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T10:34:28.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:28.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: Detected new or changed devices on smithi116 2024-03-24T10:34:28.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:28.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:28.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:28.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:28.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi116 to 8540M 2024-03-24T10:34:28.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:28.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:28.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:28.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:28.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: osd.4 [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953] boot 2024-03-24T10:34:28.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T10:34:28.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:34:28.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.852+0000 7fae0b35a700 1 -- 172.21.15.117:0/712276633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04101130 msgr2=0x7fae04101550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.852+0000 7fae0b35a700 1 --2- 172.21.15.117:0/712276633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04101130 0x7fae04101550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fadf4009990 tx=0x7fadf4009ca0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.852+0000 7fae0b35a700 1 -- 172.21.15.117:0/712276633 shutdown_connections 2024-03-24T10:34:28.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.852+0000 7fae0b35a700 1 --2- 172.21.15.117:0/712276633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04101130 0x7fae04101550 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.852+0000 7fae0b35a700 1 --2- 172.21.15.117:0/712276633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae04102cd0 0x7fae04109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.852+0000 7fae0b35a700 1 --2- 172.21.15.117:0/712276633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae04102330 0x7fae04102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.852+0000 7fae0b35a700 1 -- 172.21.15.117:0/712276633 >> 172.21.15.117:0/712276633 conn(0x7fae040fc6f0 msgr2=0x7fae040feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:28.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.852+0000 7fae0b35a700 1 -- 172.21.15.117:0/712276633 shutdown_connections 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.852+0000 7fae0b35a700 1 -- 172.21.15.117:0/712276633 wait complete. 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.853+0000 7fae0b35a700 1 Processor -- start 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.853+0000 7fae0b35a700 1 -- start start 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.853+0000 7fae0b35a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae04101130 0x7fae04198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.853+0000 7fae0b35a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04102330 0x7fae04198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.853+0000 7fae0b35a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae04102cd0 0x7fae0419f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.853+0000 7fae0b35a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fae0419fd90 con 0x7fae04101130 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.853+0000 7fae0b35a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fae0419fed0 con 0x7fae04102cd0 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.853+0000 7fae0b35a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fae041a0060 con 0x7fae04102330 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.853+0000 7fae088f5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04102330 0x7fae04198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.853+0000 7fae088f5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04102330 0x7fae04198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42232/0 (socket says 172.21.15.117:42232) 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.853+0000 7fae088f5700 1 -- 172.21.15.117:0/3670004788 learned_addr learned my addr 172.21.15.117:0/3670004788 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.853+0000 7fae098f7700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae04102cd0 0x7fae0419f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.854+0000 7fae090f6700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae04101130 0x7fae04198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.854+0000 7fae088f5700 1 -- 172.21.15.117:0/3670004788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae04102cd0 msgr2=0x7fae0419f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.854+0000 7fae088f5700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae04102cd0 0x7fae0419f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.854+0000 7fae088f5700 1 -- 172.21.15.117:0/3670004788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae04101130 msgr2=0x7fae04198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:28.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.854+0000 7fae088f5700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae04101130 0x7fae04198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:28.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.854+0000 7fae088f5700 1 -- 172.21.15.117:0/3670004788 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf40095f0 con 0x7fae04102330 2024-03-24T10:34:28.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.854+0000 7fae090f6700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae04101130 0x7fae04198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:34:28.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.854+0000 7fae088f5700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04102330 0x7fae04198740 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fae0000d980 tx=0x7fae0000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:28.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.854+0000 7fadfa7fc700 1 -- 172.21.15.117:0/3670004788 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae00009b40 con 0x7fae04102330 2024-03-24T10:34:28.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.854+0000 7fae0b35a700 1 -- 172.21.15.117:0/3670004788 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae041a0340 con 0x7fae04102330 2024-03-24T10:34:28.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.854+0000 7fadfa7fc700 1 -- 172.21.15.117:0/3670004788 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae0000b6e0 con 0x7fae04102330 2024-03-24T10:34:28.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.854+0000 7fadfa7fc700 1 -- 172.21.15.117:0/3670004788 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0000f620 con 0x7fae04102330 2024-03-24T10:34:28.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.854+0000 7fae0b35a700 1 -- 172.21.15.117:0/3670004788 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae041a0890 con 0x7fae04102330 2024-03-24T10:34:28.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.855+0000 7fae0b35a700 1 -- 172.21.15.117:0/3670004788 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae04108540 con 0x7fae04102330 2024-03-24T10:34:28.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.856+0000 7fadfa7fc700 1 -- 172.21.15.117:0/3670004788 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fae00010460 con 0x7fae04102330 2024-03-24T10:34:28.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.857+0000 7fadfa7fc700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fadf00a21d0 0x7fadf00a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:28.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.857+0000 7fadfa7fc700 1 -- 172.21.15.117:0/3670004788 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4101+0+0 (secure 0 0 0) 0x7fae000cca20 con 0x7fae04102330 2024-03-24T10:34:28.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.857+0000 7fae090f6700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fadf00a21d0 0x7fadf00a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:28.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.858+0000 7fae090f6700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fadf00a21d0 0x7fadf00a4690 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fadf4003930 tx=0x7fadf4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:28.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:28.861+0000 7fadfa7fc700 1 -- 172.21.15.117:0/3670004788 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fae00096a40 con 0x7fae04102330 2024-03-24T10:34:29.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:29.064+0000 7fae0b35a700 1 -- 172.21.15.117:0/3670004788 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fae0402dc00 con 0x7fadf00a21d0 2024-03-24T10:34:30.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:30 smithi117 ceph-mon[36368]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:30.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:30 smithi117 ceph-mon[36368]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:30.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:30 smithi117 ceph-mon[36368]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T10:34:30.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:30 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:34:30.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:30 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:34:30.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:30 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:30.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:30 smithi023 ceph-mon[32466]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:30.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:30 smithi023 ceph-mon[32466]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:30.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:30 smithi023 ceph-mon[32466]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T10:34:30.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:30 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:34:30.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:30 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:34:30.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:30 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:30.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:30 smithi116 ceph-mon[36248]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:30.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:30 smithi116 ceph-mon[36248]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:30.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:30 smithi116 ceph-mon[36248]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T10:34:30.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:30 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:34:30.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:30 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:34:30.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:30 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:32.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:32 smithi023 ceph-mon[32466]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:32.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:32 smithi023 ceph-mon[32466]: from='client.? 172.21.15.117:0/2085565231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f52e187-a40f-4360-8e5e-b665e241657e"}]: dispatch 2024-03-24T10:34:32.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:32 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f52e187-a40f-4360-8e5e-b665e241657e"}]: dispatch 2024-03-24T10:34:32.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:32 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f52e187-a40f-4360-8e5e-b665e241657e"}]': finished 2024-03-24T10:34:32.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:32 smithi023 ceph-mon[32466]: osdmap e35: 6 total, 5 up, 6 in 2024-03-24T10:34:32.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:32.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:32 smithi117 ceph-mon[36368]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:32.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:32 smithi117 ceph-mon[36368]: from='client.? 172.21.15.117:0/2085565231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f52e187-a40f-4360-8e5e-b665e241657e"}]: dispatch 2024-03-24T10:34:32.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:32 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f52e187-a40f-4360-8e5e-b665e241657e"}]: dispatch 2024-03-24T10:34:32.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:32 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f52e187-a40f-4360-8e5e-b665e241657e"}]': finished 2024-03-24T10:34:32.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:32 smithi117 ceph-mon[36368]: osdmap e35: 6 total, 5 up, 6 in 2024-03-24T10:34:32.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:32.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:32 smithi116 ceph-mon[36248]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:32.520 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:32 smithi116 ceph-mon[36248]: from='client.? 172.21.15.117:0/2085565231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f52e187-a40f-4360-8e5e-b665e241657e"}]: dispatch 2024-03-24T10:34:32.520 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:32 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f52e187-a40f-4360-8e5e-b665e241657e"}]: dispatch 2024-03-24T10:34:32.520 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:32 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f52e187-a40f-4360-8e5e-b665e241657e"}]': finished 2024-03-24T10:34:32.520 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:32 smithi116 ceph-mon[36248]: osdmap e35: 6 total, 5 up, 6 in 2024-03-24T10:34:32.520 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:33.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:33 smithi023 ceph-mon[32466]: from='client.? 172.21.15.117:0/2419051028' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:34:33.506 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:33 smithi117 ceph-mon[36368]: from='client.? 172.21.15.117:0/2419051028' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:34:33.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:33 smithi116 ceph-mon[36248]: from='client.? 172.21.15.117:0/2419051028' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:34:34.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:34 smithi023 ceph-mon[32466]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:34.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:34 smithi117 ceph-mon[36368]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:34.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:34 smithi116 ceph-mon[36248]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:36.356 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:36 smithi117 ceph-mon[36368]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:36.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:36 smithi023 ceph-mon[32466]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:36.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:36 smithi116 ceph-mon[36248]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:38.207 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:38 smithi117 ceph-mon[36368]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:38.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:38 smithi023 ceph-mon[32466]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:38.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:38 smithi116 ceph-mon[36248]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:40.138 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:40 smithi117 ceph-mon[36368]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:40.138 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T10:34:40.138 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:40.138 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:40 smithi117 ceph-mon[36368]: Deploying daemon osd.5 on smithi117 2024-03-24T10:34:40.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:40 smithi023 ceph-mon[32466]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:40.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T10:34:40.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:40.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:40 smithi023 ceph-mon[32466]: Deploying daemon osd.5 on smithi117 2024-03-24T10:34:40.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:40 smithi116 ceph-mon[36248]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:40.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T10:34:40.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:40.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:40 smithi116 ceph-mon[36248]: Deploying daemon osd.5 on smithi117 2024-03-24T10:34:42.378 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:42 smithi117 ceph-mon[36368]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:42.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:42 smithi023 ceph-mon[32466]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:42.520 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:42 smithi116 ceph-mon[36248]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:44.408 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:44 smithi117 ceph-mon[36368]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:44.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:44 smithi023 ceph-mon[32466]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:44.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:44 smithi116 ceph-mon[36248]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:45.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:45 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:45.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:45 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:45.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:45 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:46.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:45 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:46.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:45 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:46.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:45 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:46.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:45 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:46.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:45 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:46.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:45 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:46.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:46 smithi117 ceph-mon[36368]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:46.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:46 smithi023 ceph-mon[32466]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:47.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:46 smithi116 ceph-mon[36248]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:48.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:48.029+0000 7fadfa7fc700 1 -- 172.21.15.117:0/3670004788 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fae0402dc00 con 0x7fadf00a21d0 2024-03-24T10:34:48.029 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 5 on host 'smithi117' 2024-03-24T10:34:48.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:48.031+0000 7fae0b35a700 1 -- 172.21.15.117:0/3670004788 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fadf00a21d0 msgr2=0x7fadf00a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:48.031+0000 7fae0b35a700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fadf00a21d0 0x7fadf00a4690 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fadf4003930 tx=0x7fadf4005fb0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:48.031+0000 7fae0b35a700 1 -- 172.21.15.117:0/3670004788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04102330 msgr2=0x7fae04198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:48.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:48.031+0000 7fae0b35a700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04102330 0x7fae04198740 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fae0000d980 tx=0x7fae0000dc90 comp rx=0 tx=0).stop 2024-03-24T10:34:48.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:48.031+0000 7fae0b35a700 1 -- 172.21.15.117:0/3670004788 shutdown_connections 2024-03-24T10:34:48.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:48.031+0000 7fae0b35a700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fadf00a21d0 0x7fadf00a4690 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:48.031+0000 7fae0b35a700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae04102cd0 0x7fae0419f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:48.031+0000 7fae0b35a700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae04101130 0x7fae04198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:48.031+0000 7fae0b35a700 1 --2- 172.21.15.117:0/3670004788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae04102330 0x7fae04198740 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:48.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:48.031+0000 7fae0b35a700 1 -- 172.21.15.117:0/3670004788 >> 172.21.15.117:0/3670004788 conn(0x7fae040fc6f0 msgr2=0x7fae04105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:48.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:48.031+0000 7fae0b35a700 1 -- 172.21.15.117:0/3670004788 shutdown_connections 2024-03-24T10:34:48.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:34:48.031+0000 7fae0b35a700 1 -- 172.21.15.117:0/3670004788 wait complete. 2024-03-24T10:34:48.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:48 smithi117 ceph-mon[36368]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:48.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:48 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:48 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:48 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:48.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:48 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:48.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:48 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:48 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:48 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:48 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:48.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:48 smithi116 ceph-mon[36248]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:48.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:48 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:48 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:48 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:48.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:48 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:48.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:48 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:48 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:48 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:48 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:48.560 DEBUG:teuthology.orchestra.run.smithi117:osd.5> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T10:34:48.562 INFO:tasks.cephadm:Deploying osd.6 on smithi023 with /dev/vg_nvme/lv_2... 2024-03-24T10:34:48.562 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T10:34:48.592 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:48 smithi023 ceph-mon[32466]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:48.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:48 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:48 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:48 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:48.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:48 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:48.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:48 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:48 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:48 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:48.593 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:48 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:34:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:34:49.214 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:49 smithi023 ceph-mon[32466]: from='osd.5 [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T10:34:49.214 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:49 smithi023 ceph-mon[32466]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T10:34:49.441 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:49 smithi117 ceph-mon[36368]: from='osd.5 [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T10:34:49.441 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:49 smithi117 ceph-mon[36368]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T10:34:49.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:49 smithi116 ceph-mon[36248]: from='osd.5 [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T10:34:49.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:49 smithi116 ceph-mon[36248]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T10:34:50.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:50 smithi023 ceph-mon[32466]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:50.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:50 smithi023 ceph-mon[32466]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T10:34:50.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:50 smithi023 ceph-mon[32466]: from='osd.5 [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:34:50.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:50 smithi023 ceph-mon[32466]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T10:34:50.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:50 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:50.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:50 smithi023 ceph-mon[32466]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:34:50.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:50 smithi117 ceph-mon[36368]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:50.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:50 smithi117 ceph-mon[36368]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T10:34:50.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:50 smithi117 ceph-mon[36368]: from='osd.5 [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:34:50.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:50 smithi117 ceph-mon[36368]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T10:34:50.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:50 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:50.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:50 smithi117 ceph-mon[36368]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:34:50.503 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:34:50 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:34:50.188+0000 7f0502694700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T10:34:50.504 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:34:50 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:34:50.195+0000 7f04fec88700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:34:50.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:50 smithi116 ceph-mon[36248]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:50.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:50 smithi116 ceph-mon[36248]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T10:34:50.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:50 smithi116 ceph-mon[36248]: from='osd.5 [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:34:50.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:50 smithi116 ceph-mon[36248]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T10:34:50.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:50 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:50.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:50 smithi116 ceph-mon[36248]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:34:50.687 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T10:34:50.687 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T10:34:50.687 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-24T10:34:50.687 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-24T10:34:50.687 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00826299 s, 1.3 GB/s 2024-03-24T10:34:50.687 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-24T10:34:51.266 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:51 smithi023 ceph-mon[32466]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T10:34:51.266 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:51 smithi023 ceph-mon[32466]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T10:34:51.266 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:51.266 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:51.266 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:51 smithi023 ceph-mon[32466]: osd.5 [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608] boot 2024-03-24T10:34:51.266 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:51 smithi023 ceph-mon[32466]: osdmap e38: 6 total, 6 up, 6 in 2024-03-24T10:34:51.267 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:51 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:51.292 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_2 2024-03-24T10:34:51.431 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:51 smithi117 ceph-mon[36368]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T10:34:51.431 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:51 smithi117 ceph-mon[36368]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T10:34:51.431 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:51 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:51.431 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:51 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:51.431 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:51 smithi117 ceph-mon[36368]: osd.5 [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608] boot 2024-03-24T10:34:51.432 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:51 smithi117 ceph-mon[36368]: osdmap e38: 6 total, 6 up, 6 in 2024-03-24T10:34:51.432 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:51 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:51.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:51 smithi116 ceph-mon[36248]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T10:34:51.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:51 smithi116 ceph-mon[36248]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T10:34:51.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:51 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:51.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:51 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:51.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:51 smithi116 ceph-mon[36248]: osd.5 [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608] boot 2024-03-24T10:34:51.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:51 smithi116 ceph-mon[36248]: osdmap e38: 6 total, 6 up, 6 in 2024-03-24T10:34:51.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:51 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:34:51.583 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:34:52.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:52 smithi023 ceph-mon[32466]: purged_snaps scrub starts 2024-03-24T10:34:52.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:52 smithi023 ceph-mon[32466]: purged_snaps scrub ok 2024-03-24T10:34:52.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:52 smithi023 ceph-mon[32466]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:52.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:52 smithi023 ceph-mon[32466]: Detected new or changed devices on smithi117 2024-03-24T10:34:52.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:52.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:52.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:52.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:52.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:52 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi117 to 8540M 2024-03-24T10:34:52.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:52.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:52.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:52.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:52.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:52 smithi023 ceph-mon[32466]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T10:34:52.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:52 smithi117 ceph-mon[36368]: purged_snaps scrub starts 2024-03-24T10:34:52.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:52 smithi117 ceph-mon[36368]: purged_snaps scrub ok 2024-03-24T10:34:52.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:52 smithi117 ceph-mon[36368]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:52.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:52 smithi117 ceph-mon[36368]: Detected new or changed devices on smithi117 2024-03-24T10:34:52.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:52.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:52.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:52.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:52.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:52 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi117 to 8540M 2024-03-24T10:34:52.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:52.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:52.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:52.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:52.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:52 smithi117 ceph-mon[36368]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T10:34:52.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:52 smithi116 ceph-mon[36248]: purged_snaps scrub starts 2024-03-24T10:34:52.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:52 smithi116 ceph-mon[36248]: purged_snaps scrub ok 2024-03-24T10:34:52.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:52 smithi116 ceph-mon[36248]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-24T10:34:52.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:52 smithi116 ceph-mon[36248]: Detected new or changed devices on smithi117 2024-03-24T10:34:52.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:52.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:52.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:52.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:34:52.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:52 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi117 to 8540M 2024-03-24T10:34:52.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:52.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:52.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:34:52.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:34:52.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:52 smithi116 ceph-mon[36248]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T10:34:53.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.072+0000 7ff5efc9b700 1 -- 172.21.15.23:0/3928059636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5e8102d50 msgr2=0x7ff5e8105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:53.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.072+0000 7ff5efc9b700 1 --2- 172.21.15.23:0/3928059636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5e8102d50 0x7ff5e8105140 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000b0a0 tx=0x7ff5e000b3b0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.072+0000 7ff5efc9b700 1 -- 172.21.15.23:0/3928059636 shutdown_connections 2024-03-24T10:34:53.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.072+0000 7ff5efc9b700 1 --2- 172.21.15.23:0/3928059636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e8107fb0 0x7ff5e810a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.072+0000 7ff5efc9b700 1 --2- 172.21.15.23:0/3928059636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5e8102d50 0x7ff5e8105140 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.072+0000 7ff5efc9b700 1 --2- 172.21.15.23:0/3928059636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5e8105680 0x7ff5e8107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.072+0000 7ff5efc9b700 1 -- 172.21.15.23:0/3928059636 >> 172.21.15.23:0/3928059636 conn(0x7ff5e80fc6c0 msgr2=0x7ff5e80feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:34:53.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.072+0000 7ff5efc9b700 1 -- 172.21.15.23:0/3928059636 shutdown_connections 2024-03-24T10:34:53.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.072+0000 7ff5efc9b700 1 -- 172.21.15.23:0/3928059636 wait complete. 2024-03-24T10:34:53.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5efc9b700 1 Processor -- start 2024-03-24T10:34:53.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5efc9b700 1 -- start start 2024-03-24T10:34:53.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5efc9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e8102d50 0x7ff5e8198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:53.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5efc9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5e8105680 0x7ff5e8198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:53.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5efc9b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5e8107fb0 0x7ff5e819f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:53.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5efc9b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e819fea0 con 0x7ff5e8105680 2024-03-24T10:34:53.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5efc9b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e819ffe0 con 0x7ff5e8102d50 2024-03-24T10:34:53.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5efc9b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e81a0170 con 0x7ff5e8107fb0 2024-03-24T10:34:53.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5ed236700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5e8105680 0x7ff5e8198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:53.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5ed236700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5e8105680 0x7ff5e8198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56104/0 (socket says 172.21.15.23:56104) 2024-03-24T10:34:53.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5ed236700 1 -- 172.21.15.23:0/1801016478 learned_addr learned my addr 172.21.15.23:0/1801016478 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:34:53.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5ee238700 1 --2- 172.21.15.23:0/1801016478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5e8107fb0 0x7ff5e819f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:53.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5eda37700 1 --2- 172.21.15.23:0/1801016478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e8102d50 0x7ff5e8198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:53.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5ed236700 1 -- 172.21.15.23:0/1801016478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5e8107fb0 msgr2=0x7ff5e819f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:53.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5ed236700 1 --2- 172.21.15.23:0/1801016478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5e8107fb0 0x7ff5e819f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5ed236700 1 -- 172.21.15.23:0/1801016478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e8102d50 msgr2=0x7ff5e8198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:34:53.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5ed236700 1 --2- 172.21.15.23:0/1801016478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e8102d50 0x7ff5e8198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:34:53.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.073+0000 7ff5ed236700 1 -- 172.21.15.23:0/1801016478 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e0009d00 con 0x7ff5e8105680 2024-03-24T10:34:53.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.074+0000 7ff5ed236700 1 --2- 172.21.15.23:0/1801016478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5e8105680 0x7ff5e8198850 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00d980 tx=0x7ff5dc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:53.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.074+0000 7ff5e6ffd700 1 -- 172.21.15.23:0/1801016478 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc009930 con 0x7ff5e8105680 2024-03-24T10:34:53.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.074+0000 7ff5efc9b700 1 -- 172.21.15.23:0/1801016478 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e81a0450 con 0x7ff5e8105680 2024-03-24T10:34:53.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.074+0000 7ff5e6ffd700 1 -- 172.21.15.23:0/1801016478 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5dc00b6e0 con 0x7ff5e8105680 2024-03-24T10:34:53.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.074+0000 7ff5e6ffd700 1 -- 172.21.15.23:0/1801016478 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc00f3e0 con 0x7ff5e8105680 2024-03-24T10:34:53.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.074+0000 7ff5efc9b700 1 -- 172.21.15.23:0/1801016478 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e81a0950 con 0x7ff5e8105680 2024-03-24T10:34:53.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.075+0000 7ff5efc9b700 1 -- 172.21.15.23:0/1801016478 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e80fe240 con 0x7ff5e8105680 2024-03-24T10:34:53.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.076+0000 7ff5e6ffd700 1 -- 172.21.15.23:0/1801016478 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ff5dc0106f0 con 0x7ff5e8105680 2024-03-24T10:34:53.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.076+0000 7ff5e6ffd700 1 --2- 172.21.15.23:0/1801016478 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff5d00a2150 0x7ff5d00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:34:53.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.076+0000 7ff5eda37700 1 --2- 172.21.15.23:0/1801016478 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff5d00a2150 0x7ff5d00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:34:53.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.077+0000 7ff5e6ffd700 1 -- 172.21.15.23:0/1801016478 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7ff5dc0ccc20 con 0x7ff5e8105680 2024-03-24T10:34:53.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.077+0000 7ff5eda37700 1 --2- 172.21.15.23:0/1801016478 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff5d00a2150 0x7ff5d00a4610 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0007b60 tx=0x7ff5e0006000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:34:53.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.081+0000 7ff5e6ffd700 1 -- 172.21.15.23:0/1801016478 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff5dc096a90 con 0x7ff5e8105680 2024-03-24T10:34:53.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:34:53.299+0000 7ff5efc9b700 1 -- 172.21.15.23:0/1801016478 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff5e802dc00 con 0x7ff5d00a2150 2024-03-24T10:34:54.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:54 smithi023 ceph-mon[32466]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:34:54.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:54 smithi023 ceph-mon[32466]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T10:34:54.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:54 smithi023 ceph-mon[32466]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:54.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:54 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:34:54.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:54 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:34:54.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:54 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:54.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:54 smithi117 ceph-mon[36368]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:34:54.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:54 smithi117 ceph-mon[36368]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T10:34:54.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:54 smithi117 ceph-mon[36368]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:54.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:54 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:34:54.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:54 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:34:54.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:54 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:54.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:54 smithi116 ceph-mon[36248]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:34:54.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:54 smithi116 ceph-mon[36248]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T10:34:54.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:54 smithi116 ceph-mon[36248]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:34:54.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:54 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:34:54.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:54 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:34:54.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:54 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:34:55.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:55 smithi023 ceph-mon[32466]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T10:34:55.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:55 smithi117 ceph-mon[36368]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T10:34:55.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:55 smithi116 ceph-mon[36248]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T10:34:56.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:56 smithi023 ceph-mon[32466]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:34:56.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:56 smithi023 ceph-mon[32466]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T10:34:56.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:56 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2563508704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32e3081d-af77-4879-ab4c-e77a0303b741"}]: dispatch 2024-03-24T10:34:56.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:56 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2563508704' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32e3081d-af77-4879-ab4c-e77a0303b741"}]': finished 2024-03-24T10:34:56.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:56 smithi023 ceph-mon[32466]: osdmap e43: 7 total, 6 up, 7 in 2024-03-24T10:34:56.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:56 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:34:56.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:56 smithi117 ceph-mon[36368]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:34:56.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:56 smithi117 ceph-mon[36368]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T10:34:56.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:56 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2563508704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32e3081d-af77-4879-ab4c-e77a0303b741"}]: dispatch 2024-03-24T10:34:56.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:56 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2563508704' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32e3081d-af77-4879-ab4c-e77a0303b741"}]': finished 2024-03-24T10:34:56.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:56 smithi117 ceph-mon[36368]: osdmap e43: 7 total, 6 up, 7 in 2024-03-24T10:34:56.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:56 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:34:56.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:56 smithi116 ceph-mon[36248]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:34:56.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:56 smithi116 ceph-mon[36248]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T10:34:56.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:56 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2563508704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32e3081d-af77-4879-ab4c-e77a0303b741"}]: dispatch 2024-03-24T10:34:56.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:56 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2563508704' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32e3081d-af77-4879-ab4c-e77a0303b741"}]': finished 2024-03-24T10:34:56.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:56 smithi116 ceph-mon[36248]: osdmap e43: 7 total, 6 up, 7 in 2024-03-24T10:34:56.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:56 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:34:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:57 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2518438668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:34:57.770 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:57 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2518438668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:34:57.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:57 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2518438668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:34:58.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:34:58 smithi117 ceph-mon[36368]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T10:34:58.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:34:58 smithi116 ceph-mon[36248]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T10:34:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:34:58 smithi023 ceph-mon[32466]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T10:35:00.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:00 smithi117 ceph-mon[36368]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-24T10:35:00.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:00 smithi116 ceph-mon[36248]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-24T10:35:00.853 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:00 smithi023 ceph-mon[32466]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-24T10:35:02.627 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:02 smithi023 ceph-mon[32466]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T10:35:02.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:02 smithi117 ceph-mon[36368]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T10:35:02.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:02 smithi116 ceph-mon[36248]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T10:35:04.638 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:04 smithi023 ceph-mon[32466]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T10:35:04.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:04 smithi117 ceph-mon[36368]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T10:35:04.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:04 smithi116 ceph-mon[36248]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T10:35:05.566 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:05 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T10:35:05.567 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:05 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:05.567 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:05 smithi023 ceph-mon[32466]: Deploying daemon osd.6 on smithi023 2024-03-24T10:35:05.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:05 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T10:35:05.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:05 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:05.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:05 smithi117 ceph-mon[36368]: Deploying daemon osd.6 on smithi023 2024-03-24T10:35:05.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:05 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T10:35:05.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:05 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:05.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:05 smithi116 ceph-mon[36248]: Deploying daemon osd.6 on smithi023 2024-03-24T10:35:06.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:06 smithi023 ceph-mon[32466]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-24T10:35:06.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:06 smithi117 ceph-mon[36368]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-24T10:35:06.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:06 smithi116 ceph-mon[36248]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-24T10:35:08.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:08 smithi023 ceph-mon[32466]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-03-24T10:35:08.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:08 smithi117 ceph-mon[36368]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-03-24T10:35:08.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:08 smithi116 ceph-mon[36248]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-03-24T10:35:10.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:10 smithi023 ceph-mon[32466]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:10.657 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:10 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:10.657 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:10 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:10.657 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:10 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:10.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:10 smithi117 ceph-mon[36368]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:10.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:10 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:10.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:10 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:10.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:10 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:10.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:10 smithi116 ceph-mon[36248]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:10.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:10 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:10.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:10 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:10.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:10 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:12.575 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:12 smithi023 ceph-mon[32466]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:12.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:12 smithi117 ceph-mon[36368]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:12.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:12 smithi116 ceph-mon[36248]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:13.642 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 6 on host 'smithi023' 2024-03-24T10:35:13.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:35:13.642+0000 7ff5e6ffd700 1 -- 172.21.15.23:0/1801016478 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff5e802dc00 con 0x7ff5d00a2150 2024-03-24T10:35:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:35:13.644+0000 7ff5efc9b700 1 -- 172.21.15.23:0/1801016478 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff5d00a2150 msgr2=0x7ff5d00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:35:13.644+0000 7ff5efc9b700 1 --2- 172.21.15.23:0/1801016478 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff5d00a2150 0x7ff5d00a4610 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0007b60 tx=0x7ff5e0006000 comp rx=0 tx=0).stop 2024-03-24T10:35:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:35:13.644+0000 7ff5efc9b700 1 -- 172.21.15.23:0/1801016478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5e8105680 msgr2=0x7ff5e8198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:35:13.644+0000 7ff5efc9b700 1 --2- 172.21.15.23:0/1801016478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5e8105680 0x7ff5e8198850 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00d980 tx=0x7ff5dc00dc90 comp rx=0 tx=0).stop 2024-03-24T10:35:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:35:13.644+0000 7ff5efc9b700 1 -- 172.21.15.23:0/1801016478 shutdown_connections 2024-03-24T10:35:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:35:13.644+0000 7ff5efc9b700 1 --2- 172.21.15.23:0/1801016478 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff5d00a2150 0x7ff5d00a4610 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:35:13.644+0000 7ff5efc9b700 1 --2- 172.21.15.23:0/1801016478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5e8107fb0 0x7ff5e819f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:35:13.644+0000 7ff5efc9b700 1 --2- 172.21.15.23:0/1801016478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5e8102d50 0x7ff5e8198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:35:13.644+0000 7ff5efc9b700 1 --2- 172.21.15.23:0/1801016478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5e8105680 0x7ff5e8198850 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:13.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:35:13.644+0000 7ff5efc9b700 1 -- 172.21.15.23:0/1801016478 >> 172.21.15.23:0/1801016478 conn(0x7ff5e80fc6c0 msgr2=0x7ff5e81037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:13.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:35:13.644+0000 7ff5efc9b700 1 -- 172.21.15.23:0/1801016478 shutdown_connections 2024-03-24T10:35:13.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:35:13.644+0000 7ff5efc9b700 1 -- 172.21.15.23:0/1801016478 wait complete. 2024-03-24T10:35:13.911 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:13 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:13.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:13 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:13.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:13 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:13.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:13 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:35:13.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:13 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:13.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:13 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:13.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:13 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:13.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:13 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:14.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:13 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:14.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:13 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:14.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:13 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:14.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:13 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:35:14.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:13 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:14.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:13 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:14.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:13 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:14.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:13 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:14.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:13 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:14.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:13 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:14.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:13 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:14.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:13 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:35:14.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:13 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:14.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:13 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:14.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:13 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:14.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:13 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:14.284 DEBUG:teuthology.orchestra.run.smithi023:osd.6> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.6.service 2024-03-24T10:35:14.288 INFO:tasks.cephadm:Deploying osd.7 on smithi116 with /dev/vg_nvme/lv_2... 2024-03-24T10:35:14.288 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T10:35:14.533 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T10:35:14.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:14 smithi023 ceph-mon[32466]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:14.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:14 smithi023 ceph-mon[32466]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T10:35:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:14 smithi116 ceph-mon[36248]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:14.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:14 smithi116 ceph-mon[36248]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T10:35:15.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:14 smithi117 ceph-mon[36368]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:15.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:14 smithi117 ceph-mon[36368]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T10:35:15.897 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:15 smithi116 ceph-mon[36248]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T10:35:15.898 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:15 smithi116 ceph-mon[36248]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T10:35:15.898 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:15 smithi116 ceph-mon[36248]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T10:35:15.898 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:15.898 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:15 smithi116 ceph-mon[36248]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T10:35:15.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:15 smithi023 ceph-mon[32466]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T10:35:15.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:15 smithi023 ceph-mon[32466]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T10:35:15.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:15 smithi023 ceph-mon[32466]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T10:35:15.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:15.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:15 smithi023 ceph-mon[32466]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T10:35:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:15 smithi117 ceph-mon[36368]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T10:35:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:15 smithi117 ceph-mon[36368]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T10:35:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:15 smithi117 ceph-mon[36368]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T10:35:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:16.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:15 smithi117 ceph-mon[36368]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T10:35:16.425 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T10:35:16.425 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T10:35:16.425 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-24T10:35:16.426 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-24T10:35:16.426 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111908 s, 937 MB/s 2024-03-24T10:35:16.426 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-24T10:35:16.707 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:35:16 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:35:16.596+0000 7f00de70d700 -1 osd.6 0 waiting for initial osdmap 2024-03-24T10:35:16.707 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:35:16 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:35:16.606+0000 7f00d6cf9700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:35:16.821 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:16 smithi116 ceph-mon[36248]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:16.821 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:16 smithi116 ceph-mon[36248]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T10:35:16.821 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:16.821 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:16.821 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:16 smithi116 ceph-mon[36248]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' 2024-03-24T10:35:16.821 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:16 smithi117 ceph-mon[36368]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:16 smithi117 ceph-mon[36368]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T10:35:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:16 smithi117 ceph-mon[36368]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' 2024-03-24T10:35:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:17.034 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_2 2024-03-24T10:35:17.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:16 smithi023 ceph-mon[32466]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:17.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:16 smithi023 ceph-mon[32466]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T10:35:17.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:17.065 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:17.065 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:16 smithi023 ceph-mon[32466]: from='osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315]' entity='osd.6' 2024-03-24T10:35:17.065 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:17.285 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T10:35:18.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: purged_snaps scrub starts 2024-03-24T10:35:18.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: purged_snaps scrub ok 2024-03-24T10:35:18.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:18.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: Detected new or changed devices on smithi023 2024-03-24T10:35:18.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:18.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:18.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:18.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:18.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:18.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi023 to 5696M 2024-03-24T10:35:18.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:18.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:18.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:35:18.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:18.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315] boot 2024-03-24T10:35:18.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: osdmap e46: 7 total, 7 up, 7 in 2024-03-24T10:35:18.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:17 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:18.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: purged_snaps scrub starts 2024-03-24T10:35:18.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: purged_snaps scrub ok 2024-03-24T10:35:18.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:18.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: Detected new or changed devices on smithi023 2024-03-24T10:35:18.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:18.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:18.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:18.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:18.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:18.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi023 to 5696M 2024-03-24T10:35:18.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:18.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:18.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:35:18.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:18.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315] boot 2024-03-24T10:35:18.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: osdmap e46: 7 total, 7 up, 7 in 2024-03-24T10:35:18.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:17 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:18.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: purged_snaps scrub starts 2024-03-24T10:35:18.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: purged_snaps scrub ok 2024-03-24T10:35:18.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-24T10:35:18.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: Detected new or changed devices on smithi023 2024-03-24T10:35:18.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:18.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:18.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:18.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:18.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:18.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi023 to 5696M 2024-03-24T10:35:18.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:18.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:18.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:35:18.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:18.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: osd.6 [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315] boot 2024-03-24T10:35:18.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: osdmap e46: 7 total, 7 up, 7 in 2024-03-24T10:35:18.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:35:18.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.850+0000 7feda9394700 1 -- 172.21.15.116:0/1684266645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda4074c60 msgr2=0x7feda40750e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:18.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.850+0000 7feda9394700 1 --2- 172.21.15.116:0/1684266645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda4074c60 0x7feda40750e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fed8c009990 tx=0x7fed8c009ca0 comp rx=0 tx=0).stop 2024-03-24T10:35:18.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.850+0000 7feda9394700 1 -- 172.21.15.116:0/1684266645 shutdown_connections 2024-03-24T10:35:18.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.850+0000 7feda9394700 1 --2- 172.21.15.116:0/1684266645 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feda4076230 0x7feda4074690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:18.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.850+0000 7feda9394700 1 --2- 172.21.15.116:0/1684266645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feda410a100 0x7feda410c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:18.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.850+0000 7feda9394700 1 --2- 172.21.15.116:0/1684266645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda4074c60 0x7feda40750e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:18.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.850+0000 7feda9394700 1 -- 172.21.15.116:0/1684266645 >> 172.21.15.116:0/1684266645 conn(0x7feda40fd810 msgr2=0x7feda40ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:18.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.850+0000 7feda9394700 1 -- 172.21.15.116:0/1684266645 shutdown_connections 2024-03-24T10:35:18.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.850+0000 7feda9394700 1 -- 172.21.15.116:0/1684266645 wait complete. 2024-03-24T10:35:18.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7feda9394700 1 Processor -- start 2024-03-24T10:35:18.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7feda9394700 1 -- start start 2024-03-24T10:35:18.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7feda9394700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feda4074c60 0x7feda419a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:18.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7feda9394700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda4076230 0x7feda419a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:18.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7feda9394700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feda410a100 0x7feda41a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:18.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7feda9394700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7feda41a1f40 con 0x7feda4074c60 2024-03-24T10:35:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7feda9394700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feda41a2080 con 0x7feda4076230 2024-03-24T10:35:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7feda9394700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feda41a2210 con 0x7feda410a100 2024-03-24T10:35:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7fed927fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda4076230 0x7feda419a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7fed927fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda4076230 0x7feda419a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40942/0 (socket says 172.21.15.116:40942) 2024-03-24T10:35:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7fed927fc700 1 -- 172.21.15.116:0/136089072 learned_addr learned my addr 172.21.15.116:0/136089072 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T10:35:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7feda2ffd700 1 --2- 172.21.15.116:0/136089072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feda4074c60 0x7feda419a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7feda37fe700 1 --2- 172.21.15.116:0/136089072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feda410a100 0x7feda41a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7fed927fc700 1 -- 172.21.15.116:0/136089072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feda410a100 msgr2=0x7feda41a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7fed927fc700 1 --2- 172.21.15.116:0/136089072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feda410a100 0x7feda41a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7fed927fc700 1 -- 172.21.15.116:0/136089072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feda4074c60 msgr2=0x7feda419a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7fed927fc700 1 --2- 172.21.15.116:0/136089072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feda4074c60 0x7feda419a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.851+0000 7fed927fc700 1 -- 172.21.15.116:0/136089072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8c0095f0 con 0x7feda4076230 2024-03-24T10:35:18.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.852+0000 7fed927fc700 1 --2- 172.21.15.116:0/136089072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda4076230 0x7feda419a8f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fed8c0095c0 tx=0x7fed8c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.852+0000 7feda0ff9700 1 -- 172.21.15.116:0/136089072 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8c01d070 con 0x7feda4076230 2024-03-24T10:35:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.852+0000 7feda9394700 1 -- 172.21.15.116:0/136089072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feda41a2490 con 0x7feda4076230 2024-03-24T10:35:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.852+0000 7feda0ff9700 1 -- 172.21.15.116:0/136089072 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed8c005530 con 0x7feda4076230 2024-03-24T10:35:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.852+0000 7feda0ff9700 1 -- 172.21.15.116:0/136089072 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8c004c40 con 0x7feda4076230 2024-03-24T10:35:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.852+0000 7feda9394700 1 -- 172.21.15.116:0/136089072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feda41a28b0 con 0x7feda4076230 2024-03-24T10:35:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.853+0000 7feda9394700 1 -- 172.21.15.116:0/136089072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feda4108ec0 con 0x7feda4076230 2024-03-24T10:35:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.854+0000 7feda0ff9700 1 -- 172.21.15.116:0/136089072 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fed8c00fc10 con 0x7feda4076230 2024-03-24T10:35:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.854+0000 7feda0ff9700 1 --2- 172.21.15.116:0/136089072 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fed840a2150 0x7fed840a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.855+0000 7feda0ff9700 1 -- 172.21.15.116:0/136089072 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fed8c0ce2c0 con 0x7feda4076230 2024-03-24T10:35:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.855+0000 7feda2ffd700 1 --2- 172.21.15.116:0/136089072 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fed840a2150 0x7fed840a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.855+0000 7feda2ffd700 1 --2- 172.21.15.116:0/136089072 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fed840a2150 0x7fed840a4610 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feda4076610 tx=0x7fed9400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:18.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:18.858+0000 7feda0ff9700 1 -- 172.21.15.116:0/136089072 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fed8c098030 con 0x7feda4076230 2024-03-24T10:35:19.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:19.063+0000 7feda9394700 1 -- 172.21.15.116:0/136089072 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7feda402dc00 con 0x7fed840a2150 2024-03-24T10:35:19.923 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:19 smithi116 ceph-mon[36248]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T10:35:19.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:19 smithi116 ceph-mon[36248]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:19.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:19 smithi116 ceph-mon[36248]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:35:19.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:19 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:35:19.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:19 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:35:19.924 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:19 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:20.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:19 smithi117 ceph-mon[36368]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T10:35:20.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:19 smithi117 ceph-mon[36368]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:20.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:19 smithi117 ceph-mon[36368]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:35:20.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:19 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:35:20.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:19 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:35:20.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:19 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:19 smithi023 ceph-mon[32466]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T10:35:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:19 smithi023 ceph-mon[32466]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:19 smithi023 ceph-mon[32466]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:35:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:35:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:35:20.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:22.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:21 smithi023 ceph-mon[32466]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:22.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:21 smithi023 ceph-mon[32466]: from='client.? 172.21.15.116:0/809375702' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dddddbc7-f15a-4844-8d5a-869421e3dead"}]: dispatch 2024-03-24T10:35:22.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:21 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dddddbc7-f15a-4844-8d5a-869421e3dead"}]: dispatch 2024-03-24T10:35:22.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:21 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dddddbc7-f15a-4844-8d5a-869421e3dead"}]': finished 2024-03-24T10:35:22.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:21 smithi023 ceph-mon[32466]: osdmap e48: 8 total, 7 up, 8 in 2024-03-24T10:35:22.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:21 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:35:22.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:21 smithi117 ceph-mon[36368]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:22.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:21 smithi117 ceph-mon[36368]: from='client.? 172.21.15.116:0/809375702' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dddddbc7-f15a-4844-8d5a-869421e3dead"}]: dispatch 2024-03-24T10:35:22.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:21 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dddddbc7-f15a-4844-8d5a-869421e3dead"}]: dispatch 2024-03-24T10:35:22.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:21 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dddddbc7-f15a-4844-8d5a-869421e3dead"}]': finished 2024-03-24T10:35:22.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:21 smithi117 ceph-mon[36368]: osdmap e48: 8 total, 7 up, 8 in 2024-03-24T10:35:22.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:21 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:35:22.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:21 smithi116 ceph-mon[36248]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:22.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:21 smithi116 ceph-mon[36248]: from='client.? 172.21.15.116:0/809375702' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dddddbc7-f15a-4844-8d5a-869421e3dead"}]: dispatch 2024-03-24T10:35:22.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:21 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dddddbc7-f15a-4844-8d5a-869421e3dead"}]: dispatch 2024-03-24T10:35:22.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:21 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dddddbc7-f15a-4844-8d5a-869421e3dead"}]': finished 2024-03-24T10:35:22.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:21 smithi116 ceph-mon[36248]: osdmap e48: 8 total, 7 up, 8 in 2024-03-24T10:35:22.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:21 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:35:23.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:22 smithi023 ceph-mon[32466]: from='client.? 172.21.15.116:0/3723664250' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:35:23.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:22 smithi117 ceph-mon[36368]: from='client.? 172.21.15.116:0/3723664250' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:35:23.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:22 smithi116 ceph-mon[36248]: from='client.? 172.21.15.116:0/3723664250' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:35:24.154 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:23 smithi117 ceph-mon[36368]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:24.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:23 smithi023 ceph-mon[32466]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:24.168 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:23 smithi116 ceph-mon[36248]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:26.106 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:25 smithi116 ceph-mon[36248]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:26.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:25 smithi023 ceph-mon[32466]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:26.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:25 smithi117 ceph-mon[36368]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:28.057 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:27 smithi116 ceph-mon[36248]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:28.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:27 smithi023 ceph-mon[32466]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:28.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:27 smithi117 ceph-mon[36368]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:29.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T10:35:29.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:29.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T10:35:29.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:29.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T10:35:29.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:30.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:29 smithi116 ceph-mon[36248]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:30.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:29 smithi116 ceph-mon[36248]: Deploying daemon osd.7 on smithi116 2024-03-24T10:35:30.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:29 smithi023 ceph-mon[32466]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:30.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:29 smithi023 ceph-mon[32466]: Deploying daemon osd.7 on smithi116 2024-03-24T10:35:30.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:29 smithi117 ceph-mon[36368]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:30.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:29 smithi117 ceph-mon[36368]: Deploying daemon osd.7 on smithi116 2024-03-24T10:35:32.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:31 smithi023 ceph-mon[32466]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:32.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:31 smithi117 ceph-mon[36368]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:32.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:31 smithi116 ceph-mon[36248]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:34.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:33 smithi023 ceph-mon[32466]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:34.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:33 smithi117 ceph-mon[36368]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:34.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:33 smithi116 ceph-mon[36248]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:35.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:35 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:35.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:35 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:35.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:35 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:35.458 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:35 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:35.458 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:35 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:35.458 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:35 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:35.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:35 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:35.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:35 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:35.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:35 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:36.353 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:36 smithi116 ceph-mon[36248]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:36.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:36 smithi023 ceph-mon[32466]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:36.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:36 smithi117 ceph-mon[36368]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:37.578 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 7 on host 'smithi116' 2024-03-24T10:35:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:37.578+0000 7feda0ff9700 1 -- 172.21.15.116:0/136089072 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7feda402dc00 con 0x7fed840a2150 2024-03-24T10:35:37.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:37.580+0000 7feda9394700 1 -- 172.21.15.116:0/136089072 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fed840a2150 msgr2=0x7fed840a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:37.580+0000 7feda9394700 1 --2- 172.21.15.116:0/136089072 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fed840a2150 0x7fed840a4610 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feda4076610 tx=0x7fed9400a380 comp rx=0 tx=0).stop 2024-03-24T10:35:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:37.580+0000 7feda9394700 1 -- 172.21.15.116:0/136089072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda4076230 msgr2=0x7feda419a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:37.580+0000 7feda9394700 1 --2- 172.21.15.116:0/136089072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda4076230 0x7feda419a8f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fed8c0095c0 tx=0x7fed8c00f690 comp rx=0 tx=0).stop 2024-03-24T10:35:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:37.580+0000 7feda9394700 1 -- 172.21.15.116:0/136089072 shutdown_connections 2024-03-24T10:35:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:37.580+0000 7feda9394700 1 --2- 172.21.15.116:0/136089072 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fed840a2150 0x7fed840a4610 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:37.580+0000 7feda9394700 1 --2- 172.21.15.116:0/136089072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feda410a100 0x7feda41a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:37.580+0000 7feda9394700 1 --2- 172.21.15.116:0/136089072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feda4074c60 0x7feda419a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:37.580+0000 7feda9394700 1 --2- 172.21.15.116:0/136089072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feda4076230 0x7feda419a8f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:37.580+0000 7feda9394700 1 -- 172.21.15.116:0/136089072 >> 172.21.15.116:0/136089072 conn(0x7feda40fd810 msgr2=0x7feda4106000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:37.580+0000 7feda9394700 1 -- 172.21.15.116:0/136089072 shutdown_connections 2024-03-24T10:35:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:35:37.580+0000 7feda9394700 1 -- 172.21.15.116:0/136089072 wait complete. 2024-03-24T10:35:37.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:37 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:37.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:37 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:37.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:37 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:37.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:37 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:35:37.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:37 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:37.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:37 smithi116 ceph-mon[36248]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:37.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:37 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:37.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:37 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:37.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:37 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:38.006 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:37 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:38.006 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:37 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:38.006 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:37 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:38.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:37 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:35:38.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:37 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:38.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:37 smithi117 ceph-mon[36368]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:38.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:37 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:38.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:37 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:38.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:37 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:38.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:37 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:38.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:37 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:38.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:37 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:38.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:37 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:35:38.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:37 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:38.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:37 smithi023 ceph-mon[32466]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:38.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:37 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:38.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:37 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:38.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:37 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:38.220 DEBUG:teuthology.orchestra.run.smithi116:osd.7> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.7.service 2024-03-24T10:35:38.222 INFO:tasks.cephadm:Deploying osd.8 on smithi117 with /dev/vg_nvme/lv_2... 2024-03-24T10:35:38.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T10:35:38.468 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:35:38.919 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:38 smithi117 ceph-mon[36368]: from='osd.7 [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T10:35:38.919 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:38 smithi117 ceph-mon[36368]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T10:35:39.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:38 smithi023 ceph-mon[32466]: from='osd.7 [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T10:35:39.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:38 smithi023 ceph-mon[32466]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T10:35:39.217 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:38 smithi116 ceph-mon[36248]: from='osd.7 [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T10:35:39.217 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:38 smithi116 ceph-mon[36248]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T10:35:39.839 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:39 smithi117 ceph-mon[36368]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:39.839 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:39 smithi117 ceph-mon[36368]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T10:35:39.839 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:39 smithi117 ceph-mon[36368]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T10:35:39.839 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:39 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:35:39.839 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:39 smithi117 ceph-mon[36368]: from='osd.7 [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:35:39.840 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:39 smithi117 ceph-mon[36368]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:35:40.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:39 smithi023 ceph-mon[32466]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:40.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:39 smithi023 ceph-mon[32466]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T10:35:40.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:39 smithi023 ceph-mon[32466]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T10:35:40.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:39 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:35:40.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:39 smithi023 ceph-mon[32466]: from='osd.7 [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:35:40.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:39 smithi023 ceph-mon[32466]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:35:40.236 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:39 smithi116 ceph-mon[36248]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:40.236 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:39 smithi116 ceph-mon[36248]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T10:35:40.236 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:39 smithi116 ceph-mon[36248]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T10:35:40.236 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:39 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:35:40.236 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:39 smithi116 ceph-mon[36248]: from='osd.7 [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:35:40.236 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:39 smithi116 ceph-mon[36248]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:35:40.236 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:35:39 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:35:39.785+0000 7fd048614700 -1 osd.7 0 waiting for initial osdmap 2024-03-24T10:35:40.236 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:35:39 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:35:39.795+0000 7fd044c08700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:35:40.405 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T10:35:40.405 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T10:35:40.405 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-24T10:35:40.405 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-24T10:35:40.406 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0118444 s, 885 MB/s 2024-03-24T10:35:40.406 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-24T10:35:41.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:40 smithi117 ceph-mon[36368]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T10:35:41.130 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:40 smithi117 ceph-mon[36368]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T10:35:41.130 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:35:41.130 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:40 smithi117 ceph-mon[36368]: osd.7 [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459] boot 2024-03-24T10:35:41.130 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:40 smithi117 ceph-mon[36368]: osdmap e51: 8 total, 8 up, 8 in 2024-03-24T10:35:41.130 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:40 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:35:41.156 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_2 2024-03-24T10:35:41.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:40 smithi023 ceph-mon[32466]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T10:35:41.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:40 smithi023 ceph-mon[32466]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T10:35:41.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:35:41.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:40 smithi023 ceph-mon[32466]: osd.7 [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459] boot 2024-03-24T10:35:41.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:40 smithi023 ceph-mon[32466]: osdmap e51: 8 total, 8 up, 8 in 2024-03-24T10:35:41.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:40 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:35:41.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:40 smithi116 ceph-mon[36248]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T10:35:41.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:40 smithi116 ceph-mon[36248]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T10:35:41.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:35:41.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:40 smithi116 ceph-mon[36248]: osd.7 [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459] boot 2024-03-24T10:35:41.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:40 smithi116 ceph-mon[36248]: osdmap e51: 8 total, 8 up, 8 in 2024-03-24T10:35:41.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:40 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:35:41.399 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:35:42.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: purged_snaps scrub starts 2024-03-24T10:35:42.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: purged_snaps scrub ok 2024-03-24T10:35:42.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:42.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: Detected new or changed devices on smithi116 2024-03-24T10:35:42.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:42.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:42.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:42.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:42.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:42.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi116 to 5693M 2024-03-24T10:35:42.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:42.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:42.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:35:42.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:42.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:41 smithi117 ceph-mon[36368]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T10:35:42.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: purged_snaps scrub starts 2024-03-24T10:35:42.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: purged_snaps scrub ok 2024-03-24T10:35:42.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:42.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: Detected new or changed devices on smithi116 2024-03-24T10:35:42.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:42.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:42.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:42.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:42.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:42.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi116 to 5693M 2024-03-24T10:35:42.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:42.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:42.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:35:42.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:42.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:41 smithi023 ceph-mon[32466]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T10:35:42.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: purged_snaps scrub starts 2024-03-24T10:35:42.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: purged_snaps scrub ok 2024-03-24T10:35:42.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-24T10:35:42.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: Detected new or changed devices on smithi116 2024-03-24T10:35:42.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:42.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:42.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:42.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:42.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:35:42.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi116 to 5693M 2024-03-24T10:35:42.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:42.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:42.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:35:42.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:42.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:41 smithi116 ceph-mon[36248]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T10:35:42.801 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.799+0000 7ffb24f83700 1 -- 172.21.15.117:0/3382474098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb20102cc0 msgr2=0x7ffb20109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:42.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.799+0000 7ffb24f83700 1 --2- 172.21.15.117:0/3382474098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb20102cc0 0x7ffb20109570 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009990 tx=0x7ffb10009ca0 comp rx=0 tx=0).stop 2024-03-24T10:35:42.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.799+0000 7ffb24f83700 1 -- 172.21.15.117:0/3382474098 shutdown_connections 2024-03-24T10:35:42.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.799+0000 7ffb24f83700 1 --2- 172.21.15.117:0/3382474098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb20102cc0 0x7ffb20109570 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:42.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.799+0000 7ffb24f83700 1 --2- 172.21.15.117:0/3382474098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb20101120 0x7ffb20101540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:42.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.799+0000 7ffb24f83700 1 --2- 172.21.15.117:0/3382474098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb20102320 0x7ffb20102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:42.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.799+0000 7ffb24f83700 1 -- 172.21.15.117:0/3382474098 >> 172.21.15.117:0/3382474098 conn(0x7ffb200fc6c0 msgr2=0x7ffb200feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:35:42.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.799+0000 7ffb24f83700 1 -- 172.21.15.117:0/3382474098 shutdown_connections 2024-03-24T10:35:42.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.799+0000 7ffb24f83700 1 -- 172.21.15.117:0/3382474098 wait complete. 2024-03-24T10:35:42.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.800+0000 7ffb24f83700 1 Processor -- start 2024-03-24T10:35:42.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.800+0000 7ffb24f83700 1 -- start start 2024-03-24T10:35:42.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.800+0000 7ffb24f83700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb20101120 0x7ffb2006b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:42.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.800+0000 7ffb24f83700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb20102320 0x7ffb2019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:42.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.800+0000 7ffb24f83700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb20102cc0 0x7ffb2019f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:42.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.800+0000 7ffb24f83700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2019fcb0 con 0x7ffb20101120 2024-03-24T10:35:42.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.800+0000 7ffb24f83700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2019fe20 con 0x7ffb20102320 2024-03-24T10:35:42.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.800+0000 7ffb24f83700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2019ffe0 con 0x7ffb20102cc0 2024-03-24T10:35:42.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.800+0000 7ffb1ed9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb20102cc0 0x7ffb2019f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:42.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.800+0000 7ffb1ed9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb20102cc0 0x7ffb2019f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:46294/0 (socket says 172.21.15.117:46294) 2024-03-24T10:35:42.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.800+0000 7ffb1ed9d700 1 -- 172.21.15.117:0/2270255055 learned_addr learned my addr 172.21.15.117:0/2270255055 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T10:35:42.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.800+0000 7ffb1dd9b700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb20102320 0x7ffb2019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:42.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.800+0000 7ffb1e59c700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb20101120 0x7ffb2006b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:42.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.801+0000 7ffb1ed9d700 1 -- 172.21.15.117:0/2270255055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb20102320 msgr2=0x7ffb2019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:42.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.801+0000 7ffb1ed9d700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb20102320 0x7ffb2019cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:42.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.801+0000 7ffb1ed9d700 1 -- 172.21.15.117:0/2270255055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb20101120 msgr2=0x7ffb2006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:35:42.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.801+0000 7ffb1ed9d700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb20101120 0x7ffb2006b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:35:42.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.801+0000 7ffb1ed9d700 1 -- 172.21.15.117:0/2270255055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb100095f0 con 0x7ffb20102cc0 2024-03-24T10:35:42.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.801+0000 7ffb1e59c700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb20101120 0x7ffb2006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:35:42.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.801+0000 7ffb1ed9d700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb20102cc0 0x7ffb2019f510 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffb100059d0 tx=0x7ffb1000f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:42.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.801+0000 7ffb0f7fe700 1 -- 172.21.15.117:0/2270255055 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb1001d070 con 0x7ffb20102cc0 2024-03-24T10:35:42.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.801+0000 7ffb24f83700 1 -- 172.21.15.117:0/2270255055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb201a0260 con 0x7ffb20102cc0 2024-03-24T10:35:42.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.801+0000 7ffb0f7fe700 1 -- 172.21.15.117:0/2270255055 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb1000fc30 con 0x7ffb20102cc0 2024-03-24T10:35:42.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.801+0000 7ffb0f7fe700 1 -- 172.21.15.117:0/2270255055 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb100173f0 con 0x7ffb20102cc0 2024-03-24T10:35:42.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.801+0000 7ffb24f83700 1 -- 172.21.15.117:0/2270255055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb201a06f0 con 0x7ffb20102cc0 2024-03-24T10:35:42.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.802+0000 7ffb24f83700 1 -- 172.21.15.117:0/2270255055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb20108480 con 0x7ffb20102cc0 2024-03-24T10:35:42.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.803+0000 7ffb0f7fe700 1 -- 172.21.15.117:0/2270255055 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ffb100039d0 con 0x7ffb20102cc0 2024-03-24T10:35:42.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.803+0000 7ffb0f7fe700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ffb080a21d0 0x7ffb080a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:35:42.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.804+0000 7ffb0f7fe700 1 -- 172.21.15.117:0/2270255055 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7ffb100ce410 con 0x7ffb20102cc0 2024-03-24T10:35:42.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.804+0000 7ffb1e59c700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ffb080a21d0 0x7ffb080a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:35:42.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.804+0000 7ffb1e59c700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ffb080a21d0 0x7ffb080a4690 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffb18005fd0 tx=0x7ffb18005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:35:42.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:42.807+0000 7ffb0f7fe700 1 -- 172.21.15.117:0/2270255055 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffb10097fb0 con 0x7ffb20102cc0 2024-03-24T10:35:43.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:35:43.008+0000 7ffb24f83700 1 -- 172.21.15.117:0/2270255055 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ffb2002dd20 con 0x7ffb080a21d0 2024-03-24T10:35:43.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:43 smithi023 ceph-mon[32466]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T10:35:43.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:35:43.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:35:43.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:43.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:43 smithi117 ceph-mon[36368]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T10:35:43.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:35:43.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:35:43.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:43.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:43 smithi116 ceph-mon[36248]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T10:35:43.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:35:43.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:35:43.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:44.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:44 smithi117 ceph-mon[36368]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:44.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:44 smithi117 ceph-mon[36368]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:35:44.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:44 smithi117 ceph-mon[36368]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T10:35:44.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:44 smithi023 ceph-mon[32466]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:44.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:44 smithi023 ceph-mon[32466]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:35:44.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:44 smithi023 ceph-mon[32466]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T10:35:44.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:44 smithi116 ceph-mon[36248]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:44.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:44 smithi116 ceph-mon[36248]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:35:44.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:44 smithi116 ceph-mon[36248]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T10:35:45.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:45 smithi117 ceph-mon[36368]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T10:35:45.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:45 smithi023 ceph-mon[32466]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T10:35:45.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:45 smithi116 ceph-mon[36248]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T10:35:46.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:46 smithi117 ceph-mon[36368]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:46.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:46 smithi117 ceph-mon[36368]: from='client.? 172.21.15.117:0/2357416416' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e84fd4bc-7173-410c-84f2-5c9dcf91b790"}]: dispatch 2024-03-24T10:35:46.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:46 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e84fd4bc-7173-410c-84f2-5c9dcf91b790"}]: dispatch 2024-03-24T10:35:46.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:46 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e84fd4bc-7173-410c-84f2-5c9dcf91b790"}]': finished 2024-03-24T10:35:46.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:46 smithi117 ceph-mon[36368]: osdmap e56: 9 total, 8 up, 9 in 2024-03-24T10:35:46.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:46 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:35:46.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:46 smithi023 ceph-mon[32466]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:46.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:46 smithi023 ceph-mon[32466]: from='client.? 172.21.15.117:0/2357416416' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e84fd4bc-7173-410c-84f2-5c9dcf91b790"}]: dispatch 2024-03-24T10:35:46.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:46 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e84fd4bc-7173-410c-84f2-5c9dcf91b790"}]: dispatch 2024-03-24T10:35:46.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:46 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e84fd4bc-7173-410c-84f2-5c9dcf91b790"}]': finished 2024-03-24T10:35:46.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:46 smithi023 ceph-mon[32466]: osdmap e56: 9 total, 8 up, 9 in 2024-03-24T10:35:46.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:46 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:35:46.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:46 smithi116 ceph-mon[36248]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:46.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:46 smithi116 ceph-mon[36248]: from='client.? 172.21.15.117:0/2357416416' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e84fd4bc-7173-410c-84f2-5c9dcf91b790"}]: dispatch 2024-03-24T10:35:46.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:46 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e84fd4bc-7173-410c-84f2-5c9dcf91b790"}]: dispatch 2024-03-24T10:35:46.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:46 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e84fd4bc-7173-410c-84f2-5c9dcf91b790"}]': finished 2024-03-24T10:35:46.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:46 smithi116 ceph-mon[36248]: osdmap e56: 9 total, 8 up, 9 in 2024-03-24T10:35:46.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:46 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:35:47.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:47 smithi023 ceph-mon[32466]: from='client.? 172.21.15.117:0/349694280' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:35:47.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:47 smithi117 ceph-mon[36368]: from='client.? 172.21.15.117:0/349694280' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:35:47.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:47 smithi116 ceph-mon[36248]: from='client.? 172.21.15.117:0/349694280' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:35:48.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:48 smithi023 ceph-mon[32466]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:48.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:48 smithi117 ceph-mon[36368]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:48.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:48 smithi116 ceph-mon[36248]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:50.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:50 smithi023 ceph-mon[32466]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:50.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:50 smithi117 ceph-mon[36368]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:50.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:50 smithi116 ceph-mon[36248]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:52.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:52 smithi023 ceph-mon[32466]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:52.421 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:52 smithi117 ceph-mon[36368]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:52.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:52 smithi116 ceph-mon[36248]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:53.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:53 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T10:35:53.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:53 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:53.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:53 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T10:35:53.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:53 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:53.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:53 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T10:35:53.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:53 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:35:54.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:54 smithi117 ceph-mon[36368]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:54.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:54 smithi117 ceph-mon[36368]: Deploying daemon osd.8 on smithi117 2024-03-24T10:35:54.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:54 smithi023 ceph-mon[32466]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:54.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:54 smithi023 ceph-mon[32466]: Deploying daemon osd.8 on smithi117 2024-03-24T10:35:54.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:54 smithi116 ceph-mon[36248]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:54.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:54 smithi116 ceph-mon[36248]: Deploying daemon osd.8 on smithi117 2024-03-24T10:35:56.413 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:56 smithi117 ceph-mon[36368]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:56.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:56 smithi023 ceph-mon[32466]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:56.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:56 smithi116 ceph-mon[36248]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:58.397 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:58 smithi117 ceph-mon[36368]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:58.463 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:58 smithi023 ceph-mon[32466]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:58.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:58 smithi116 ceph-mon[36248]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:35:59.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:59 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:59.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:59 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:59.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:35:59 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:59.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:59 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:59.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:59 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:59.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:35:59 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:35:59.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:59 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:59.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:59 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:35:59.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:35:59 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:00.593 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:00 smithi117 ceph-mon[36368]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:36:00.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:00 smithi116 ceph-mon[36248]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:36:00.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:00 smithi023 ceph-mon[32466]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:36:01.974 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 8 on host 'smithi117' 2024-03-24T10:36:01.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:01.974+0000 7ffb0f7fe700 1 -- 172.21.15.117:0/2270255055 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ffb2002dd20 con 0x7ffb080a21d0 2024-03-24T10:36:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:01.976+0000 7ffb24f83700 1 -- 172.21.15.117:0/2270255055 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ffb080a21d0 msgr2=0x7ffb080a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:01.976+0000 7ffb24f83700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ffb080a21d0 0x7ffb080a4690 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffb18005fd0 tx=0x7ffb18005f00 comp rx=0 tx=0).stop 2024-03-24T10:36:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:01.976+0000 7ffb24f83700 1 -- 172.21.15.117:0/2270255055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb20102cc0 msgr2=0x7ffb2019f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:01.976+0000 7ffb24f83700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb20102cc0 0x7ffb2019f510 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffb100059d0 tx=0x7ffb1000f740 comp rx=0 tx=0).stop 2024-03-24T10:36:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:01.976+0000 7ffb24f83700 1 -- 172.21.15.117:0/2270255055 shutdown_connections 2024-03-24T10:36:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:01.976+0000 7ffb24f83700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ffb080a21d0 0x7ffb080a4690 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:01.976+0000 7ffb24f83700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb20102cc0 0x7ffb2019f510 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:01.976+0000 7ffb24f83700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb20101120 0x7ffb2006b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:01.976+0000 7ffb24f83700 1 --2- 172.21.15.117:0/2270255055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb20102320 0x7ffb2019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:01.976+0000 7ffb24f83700 1 -- 172.21.15.117:0/2270255055 >> 172.21.15.117:0/2270255055 conn(0x7ffb200fc6c0 msgr2=0x7ffb201055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:01.976+0000 7ffb24f83700 1 -- 172.21.15.117:0/2270255055 shutdown_connections 2024-03-24T10:36:01.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:01.976+0000 7ffb24f83700 1 -- 172.21.15.117:0/2270255055 wait complete. 2024-03-24T10:36:02.559 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:02 smithi117 ceph-mon[36368]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:36:02.559 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:02 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:02.559 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:02 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:02.559 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:02 smithi117 ceph-mon[36368]: from='osd.8 [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T10:36:02.559 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:02 smithi117 ceph-mon[36368]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T10:36:02.561 DEBUG:teuthology.orchestra.run.smithi117:osd.8> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.8.service 2024-03-24T10:36:02.564 INFO:tasks.cephadm:Deploying osd.9 on smithi023 with /dev/vg_nvme/lv_1... 2024-03-24T10:36:02.564 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T10:36:02.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:02 smithi116 ceph-mon[36248]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:36:02.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:02 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:02.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:02 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:02.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:02 smithi116 ceph-mon[36248]: from='osd.8 [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T10:36:02.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:02 smithi116 ceph-mon[36248]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T10:36:02.818 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:36:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:02 smithi023 ceph-mon[32466]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:36:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:02 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:02 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:02 smithi023 ceph-mon[32466]: from='osd.8 [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T10:36:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:02 smithi023 ceph-mon[32466]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T10:36:03.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:03.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:03.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:03 smithi023 ceph-mon[32466]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T10:36:03.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:03 smithi023 ceph-mon[32466]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T10:36:03.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:36:03.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:03 smithi023 ceph-mon[32466]: from='osd.8 [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:36:03.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:03 smithi023 ceph-mon[32466]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:36:04.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:04.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:04.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:03 smithi117 ceph-mon[36368]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T10:36:04.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:03 smithi117 ceph-mon[36368]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T10:36:04.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:36:04.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:03 smithi117 ceph-mon[36368]: from='osd.8 [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:36:04.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:03 smithi117 ceph-mon[36368]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:36:04.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:03 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:04.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:03 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:04.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:03 smithi116 ceph-mon[36248]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T10:36:04.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:03 smithi116 ceph-mon[36248]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T10:36:04.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:03 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:36:04.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:03 smithi116 ceph-mon[36248]: from='osd.8 [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:36:04.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:03 smithi116 ceph-mon[36248]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:36:04.736 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:04 smithi117 ceph-mon[36368]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:36:04.736 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:04 smithi117 ceph-mon[36368]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T10:36:04.736 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:04 smithi117 ceph-mon[36368]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T10:36:04.736 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:04 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:36:04.736 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:04 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:36:04.879 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T10:36:04.879 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T10:36:04.880 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-24T10:36:04.880 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-24T10:36:04.880 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116553 s, 900 MB/s 2024-03-24T10:36:04.880 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-24T10:36:04.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:04 smithi023 ceph-mon[32466]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:36:04.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:04 smithi023 ceph-mon[32466]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T10:36:04.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:04 smithi023 ceph-mon[32466]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T10:36:04.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:04 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:36:04.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:04 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:36:05.003 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:36:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:36:04.734+0000 7f1a23d66700 -1 osd.8 0 waiting for initial osdmap 2024-03-24T10:36:05.003 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:36:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:36:04.747+0000 7f1a1c352700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:36:05.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:04 smithi116 ceph-mon[36248]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:36:05.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:04 smithi116 ceph-mon[36248]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T10:36:05.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:04 smithi116 ceph-mon[36248]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T10:36:05.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:04 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:36:05.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:04 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:36:05.565 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_1 2024-03-24T10:36:05.811 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:36:05.897 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:05 smithi117 ceph-mon[36368]: purged_snaps scrub starts 2024-03-24T10:36:05.897 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:05 smithi117 ceph-mon[36368]: purged_snaps scrub ok 2024-03-24T10:36:05.897 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:05 smithi117 ceph-mon[36368]: from='osd.8 ' entity='osd.8' 2024-03-24T10:36:05.897 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:05 smithi117 ceph-mon[36368]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:36:05.897 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:05 smithi117 ceph-mon[36368]: osd.8 [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563] boot 2024-03-24T10:36:05.897 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:05 smithi117 ceph-mon[36368]: osdmap e59: 9 total, 9 up, 9 in 2024-03-24T10:36:05.898 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:05 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:36:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:05 smithi023 ceph-mon[32466]: purged_snaps scrub starts 2024-03-24T10:36:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:05 smithi023 ceph-mon[32466]: purged_snaps scrub ok 2024-03-24T10:36:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:05 smithi023 ceph-mon[32466]: from='osd.8 ' entity='osd.8' 2024-03-24T10:36:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:05 smithi023 ceph-mon[32466]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:36:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:05 smithi023 ceph-mon[32466]: osd.8 [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563] boot 2024-03-24T10:36:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:05 smithi023 ceph-mon[32466]: osdmap e59: 9 total, 9 up, 9 in 2024-03-24T10:36:05.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:05 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:36:06.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:05 smithi116 ceph-mon[36248]: purged_snaps scrub starts 2024-03-24T10:36:06.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:05 smithi116 ceph-mon[36248]: purged_snaps scrub ok 2024-03-24T10:36:06.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:05 smithi116 ceph-mon[36248]: from='osd.8 ' entity='osd.8' 2024-03-24T10:36:06.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:05 smithi116 ceph-mon[36248]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-24T10:36:06.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:05 smithi116 ceph-mon[36248]: osd.8 [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563] boot 2024-03-24T10:36:06.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:05 smithi116 ceph-mon[36248]: osdmap e59: 9 total, 9 up, 9 in 2024-03-24T10:36:06.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:05 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:36:07.135 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: Detected new or changed devices on smithi117 2024-03-24T10:36:07.135 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.135 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.135 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:07.135 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:07.136 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:07.136 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi117 to 5693M 2024-03-24T10:36:07.136 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T10:36:07.136 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.136 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:07.136 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:07.136 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.136 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:07.136 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:07.136 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:07.136 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:06 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: Detected new or changed devices on smithi117 2024-03-24T10:36:07.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:07.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:07.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:07.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi117 to 5693M 2024-03-24T10:36:07.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T10:36:07.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:07.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:07.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:07.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:07.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:07.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:06 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: Detected new or changed devices on smithi117 2024-03-24T10:36:07.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:07.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:07.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:07.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi117 to 5693M 2024-03-24T10:36:07.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T10:36:07.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:07.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:07.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:07.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:07.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:07.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:06 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:07.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.359+0000 7fbec1198700 1 -- 172.21.15.23:0/3417720441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc101d80 msgr2=0x7fbebc109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:07.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.359+0000 7fbec1198700 1 --2- 172.21.15.23:0/3417720441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc101d80 0x7fbebc109620 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400b0a0 tx=0x7fbeb400b3b0 comp rx=0 tx=0).stop 2024-03-24T10:36:07.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.359+0000 7fbec1198700 1 -- 172.21.15.23:0/3417720441 shutdown_connections 2024-03-24T10:36:07.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.359+0000 7fbec1198700 1 --2- 172.21.15.23:0/3417720441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbebc102760 0x7fbebc102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:07.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.359+0000 7fbec1198700 1 --2- 172.21.15.23:0/3417720441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc101d80 0x7fbebc109620 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:07.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.359+0000 7fbec1198700 1 --2- 172.21.15.23:0/3417720441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc1013c0 0x7fbebc101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:07.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.359+0000 7fbec1198700 1 -- 172.21.15.23:0/3417720441 >> 172.21.15.23:0/3417720441 conn(0x7fbebc0fbe90 msgr2=0x7fbebc0fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:07.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.359+0000 7fbec1198700 1 -- 172.21.15.23:0/3417720441 shutdown_connections 2024-03-24T10:36:07.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.359+0000 7fbec1198700 1 -- 172.21.15.23:0/3417720441 wait complete. 2024-03-24T10:36:07.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.360+0000 7fbec1198700 1 Processor -- start 2024-03-24T10:36:07.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.360+0000 7fbec1198700 1 -- start start 2024-03-24T10:36:07.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.360+0000 7fbec1198700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc1013c0 0x7fbebc1982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:07.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.360+0000 7fbec1198700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc101d80 0x7fbebc1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:07.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.360+0000 7fbec1198700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbebc102760 0x7fbebc19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:07.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.360+0000 7fbec1198700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fbebc19fe30 con 0x7fbebc1013c0 2024-03-24T10:36:07.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.360+0000 7fbec1198700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbebc19ff70 con 0x7fbebc101d80 2024-03-24T10:36:07.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbec1198700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbebc1a0100 con 0x7fbebc102760 2024-03-24T10:36:07.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbebad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc1013c0 0x7fbebc1982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:07.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbebad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc1013c0 0x7fbebc1982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33924/0 (socket says 172.21.15.23:33924) 2024-03-24T10:36:07.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbebad9d700 1 -- 172.21.15.23:0/3661714737 learned_addr learned my addr 172.21.15.23:0/3661714737 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:36:07.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbebb59e700 1 --2- 172.21.15.23:0/3661714737 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbebc102760 0x7fbebc19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:07.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbeba59c700 1 --2- 172.21.15.23:0/3661714737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc101d80 0x7fbebc1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:07.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbebad9d700 1 -- 172.21.15.23:0/3661714737 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbebc102760 msgr2=0x7fbebc19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:07.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbebad9d700 1 --2- 172.21.15.23:0/3661714737 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbebc102760 0x7fbebc19f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:07.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbebad9d700 1 -- 172.21.15.23:0/3661714737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc101d80 msgr2=0x7fbebc1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:07.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbebad9d700 1 --2- 172.21.15.23:0/3661714737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc101d80 0x7fbebc1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:07.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbebad9d700 1 -- 172.21.15.23:0/3661714737 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb4009d00 con 0x7fbebc1013c0 2024-03-24T10:36:07.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbebad9d700 1 --2- 172.21.15.23:0/3661714737 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc1013c0 0x7fbebc1982a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbeac00d9b0 tx=0x7fbeac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:07.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbeb3fff700 1 -- 172.21.15.23:0/3661714737 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac004020 con 0x7fbebc1013c0 2024-03-24T10:36:07.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbeb3fff700 1 -- 172.21.15.23:0/3661714737 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbeac009730 con 0x7fbebc1013c0 2024-03-24T10:36:07.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbec1198700 1 -- 172.21.15.23:0/3661714737 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbebc1a03e0 con 0x7fbebc1013c0 2024-03-24T10:36:07.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbeb3fff700 1 -- 172.21.15.23:0/3661714737 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac003b40 con 0x7fbebc1013c0 2024-03-24T10:36:07.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.361+0000 7fbec1198700 1 -- 172.21.15.23:0/3661714737 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbebc1a08e0 con 0x7fbebc1013c0 2024-03-24T10:36:07.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.362+0000 7fbec1198700 1 -- 172.21.15.23:0/3661714737 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbebc108530 con 0x7fbebc1013c0 2024-03-24T10:36:07.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.364+0000 7fbeb3fff700 1 -- 172.21.15.23:0/3661714737 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fbeac009ba0 con 0x7fbebc1013c0 2024-03-24T10:36:07.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.364+0000 7fbeb3fff700 1 --2- 172.21.15.23:0/3661714737 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbea80a2150 0x7fbea80a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:07.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.364+0000 7fbeba59c700 1 --2- 172.21.15.23:0/3661714737 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbea80a2150 0x7fbea80a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:07.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.364+0000 7fbeb3fff700 1 -- 172.21.15.23:0/3661714737 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5858+0+0 (secure 0 0 0) 0x7fbeac0cd250 con 0x7fbebc1013c0 2024-03-24T10:36:07.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.365+0000 7fbeba59c700 1 --2- 172.21.15.23:0/3661714737 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbea80a2150 0x7fbea80a4610 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbea4005950 tx=0x7fbea400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:07.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.368+0000 7fbeb3fff700 1 -- 172.21.15.23:0/3661714737 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbeac096bb0 con 0x7fbebc1013c0 2024-03-24T10:36:07.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:07.580+0000 7fbec1198700 1 -- 172.21.15.23:0/3661714737 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fbebc02dca0 con 0x7fbea80a2150 2024-03-24T10:36:08.204 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:07 smithi023 ceph-mon[32466]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 660 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:08.204 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:07 smithi023 ceph-mon[32466]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T10:36:08.204 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:07 smithi023 ceph-mon[32466]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:36:08.204 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:07 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:36:08.204 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:07 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:36:08.204 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:07 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:07 smithi117 ceph-mon[36368]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 660 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:07 smithi117 ceph-mon[36368]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T10:36:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:07 smithi117 ceph-mon[36368]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:36:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:07 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:36:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:07 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:36:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:07 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:08.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:07 smithi116 ceph-mon[36248]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 660 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:08.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:07 smithi116 ceph-mon[36248]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T10:36:08.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:07 smithi116 ceph-mon[36248]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:36:08.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:36:08.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:36:08.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:09.228 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:08 smithi023 ceph-mon[32466]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T10:36:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:08 smithi117 ceph-mon[36368]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T10:36:09.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:08 smithi116 ceph-mon[36248]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T10:36:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:09 smithi117 ceph-mon[36368]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 660 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:09 smithi117 ceph-mon[36368]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T10:36:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:09 smithi116 ceph-mon[36248]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 660 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:09 smithi116 ceph-mon[36248]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T10:36:10.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:09 smithi023 ceph-mon[32466]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 660 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:10.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:09 smithi023 ceph-mon[32466]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T10:36:11.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:11 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1321504359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a1d9119-fd0f-45b4-a0f7-051b118e1c09"}]: dispatch 2024-03-24T10:36:11.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:11 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1321504359' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a1d9119-fd0f-45b4-a0f7-051b118e1c09"}]': finished 2024-03-24T10:36:11.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:11 smithi023 ceph-mon[32466]: osdmap e64: 10 total, 9 up, 10 in 2024-03-24T10:36:11.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:11 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:11.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:11 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/761941163' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:36:11.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:11 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1321504359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a1d9119-fd0f-45b4-a0f7-051b118e1c09"}]: dispatch 2024-03-24T10:36:11.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:11 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1321504359' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a1d9119-fd0f-45b4-a0f7-051b118e1c09"}]': finished 2024-03-24T10:36:11.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:11 smithi117 ceph-mon[36368]: osdmap e64: 10 total, 9 up, 10 in 2024-03-24T10:36:11.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:11 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:11.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:11 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/761941163' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:36:11.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:11 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1321504359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a1d9119-fd0f-45b4-a0f7-051b118e1c09"}]: dispatch 2024-03-24T10:36:11.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:11 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1321504359' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a1d9119-fd0f-45b4-a0f7-051b118e1c09"}]': finished 2024-03-24T10:36:11.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:11 smithi116 ceph-mon[36248]: osdmap e64: 10 total, 9 up, 10 in 2024-03-24T10:36:11.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:11 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:11.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:11 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/761941163' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:36:12.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:12 smithi117 ceph-mon[36368]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:12.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:12 smithi116 ceph-mon[36248]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:12.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:12 smithi023 ceph-mon[32466]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:14.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:14 smithi117 ceph-mon[36368]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-24T10:36:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:14 smithi116 ceph-mon[36248]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-24T10:36:14.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:14 smithi023 ceph-mon[32466]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-24T10:36:16.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:16 smithi023 ceph-mon[32466]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-24T10:36:16.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:16 smithi116 ceph-mon[36248]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-24T10:36:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:16 smithi117 ceph-mon[36368]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-24T10:36:18.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:18 smithi116 ceph-mon[36248]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T10:36:18.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:18 smithi023 ceph-mon[32466]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T10:36:19.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:18 smithi117 ceph-mon[36368]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T10:36:19.723 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T10:36:19.723 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:20.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:19 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T10:36:20.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:19 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:20.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:19 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T10:36:20.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:19 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:20.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:20 smithi023 ceph-mon[32466]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T10:36:20.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:20 smithi023 ceph-mon[32466]: Deploying daemon osd.9 on smithi023 2024-03-24T10:36:21.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:20 smithi117 ceph-mon[36368]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T10:36:21.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:20 smithi117 ceph-mon[36368]: Deploying daemon osd.9 on smithi023 2024-03-24T10:36:21.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:20 smithi116 ceph-mon[36248]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T10:36:21.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:20 smithi116 ceph-mon[36248]: Deploying daemon osd.9 on smithi023 2024-03-24T10:36:22.855 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:22 smithi023 ceph-mon[32466]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-24T10:36:23.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:22 smithi117 ceph-mon[36368]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-24T10:36:23.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:22 smithi116 ceph-mon[36248]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-24T10:36:24.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:24 smithi117 ceph-mon[36368]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-24T10:36:24.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:24.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:24.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:24 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:24.820 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:24 smithi116 ceph-mon[36248]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-24T10:36:24.820 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:24.820 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:24.820 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:24 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:24.880 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:24 smithi023 ceph-mon[32466]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-24T10:36:24.880 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:24.881 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:24.881 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:24 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:26.808 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:26 smithi023 ceph-mon[32466]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:27.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:26 smithi117 ceph-mon[36368]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:27.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:26 smithi116 ceph-mon[36248]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:28.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:28.238+0000 7fbeb3fff700 1 -- 172.21.15.23:0/3661714737 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbebc02dca0 con 0x7fbea80a2150 2024-03-24T10:36:28.239 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 9 on host 'smithi023' 2024-03-24T10:36:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:28.241+0000 7fbec1198700 1 -- 172.21.15.23:0/3661714737 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbea80a2150 msgr2=0x7fbea80a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:28.241+0000 7fbec1198700 1 --2- 172.21.15.23:0/3661714737 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbea80a2150 0x7fbea80a4610 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbea4005950 tx=0x7fbea400b410 comp rx=0 tx=0).stop 2024-03-24T10:36:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:28.241+0000 7fbec1198700 1 -- 172.21.15.23:0/3661714737 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc1013c0 msgr2=0x7fbebc1982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:28.241+0000 7fbec1198700 1 --2- 172.21.15.23:0/3661714737 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc1013c0 0x7fbebc1982a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbeac00d9b0 tx=0x7fbeac00dcc0 comp rx=0 tx=0).stop 2024-03-24T10:36:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:28.241+0000 7fbec1198700 1 -- 172.21.15.23:0/3661714737 shutdown_connections 2024-03-24T10:36:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:28.241+0000 7fbec1198700 1 --2- 172.21.15.23:0/3661714737 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbea80a2150 0x7fbea80a4610 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:28.241+0000 7fbec1198700 1 --2- 172.21.15.23:0/3661714737 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbebc102760 0x7fbebc19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:28.241+0000 7fbec1198700 1 --2- 172.21.15.23:0/3661714737 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbebc1013c0 0x7fbebc1982a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:28.241+0000 7fbec1198700 1 --2- 172.21.15.23:0/3661714737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc101d80 0x7fbebc1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:28.241+0000 7fbec1198700 1 -- 172.21.15.23:0/3661714737 >> 172.21.15.23:0/3661714737 conn(0x7fbebc0fbe90 msgr2=0x7fbebc105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:28.241+0000 7fbec1198700 1 -- 172.21.15.23:0/3661714737 shutdown_connections 2024-03-24T10:36:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:36:28.241+0000 7fbec1198700 1 -- 172.21.15.23:0/3661714737 wait complete. 2024-03-24T10:36:28.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:28 smithi023 ceph-mon[32466]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:28.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:28.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:28.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:28 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:28.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:28 smithi117 ceph-mon[36368]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:28.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:28.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:28.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:28 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:28 smithi116 ceph-mon[36248]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:28.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:28 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:28.991 DEBUG:teuthology.orchestra.run.smithi023:osd.9> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.9.service 2024-03-24T10:36:28.995 INFO:tasks.cephadm:Deploying osd.10 on smithi116 with /dev/vg_nvme/lv_1... 2024-03-24T10:36:28.995 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T10:36:29.240 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T10:36:29.666 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:29 smithi023 ceph-mon[32466]: from='osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T10:36:29.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:29 smithi116 ceph-mon[36248]: from='osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T10:36:29.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:29 smithi117 ceph-mon[36368]: from='osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T10:36:30.579 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:30 smithi116 ceph-mon[36248]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:30.579 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:30 smithi116 ceph-mon[36248]: from='osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T10:36:30.579 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:30 smithi116 ceph-mon[36248]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T10:36:30.579 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:30 smithi116 ceph-mon[36248]: from='osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T10:36:30.579 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:30 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:30.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:30 smithi023 ceph-mon[32466]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:30.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:30 smithi023 ceph-mon[32466]: from='osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T10:36:30.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:30 smithi023 ceph-mon[32466]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T10:36:30.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:30 smithi023 ceph-mon[32466]: from='osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T10:36:30.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:30 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:30.669 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:36:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:36:30.397+0000 7febcaa98700 -1 osd.9 0 waiting for initial osdmap 2024-03-24T10:36:30.669 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:36:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:36:30.409+0000 7febc708c700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:36:30.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:30 smithi117 ceph-mon[36368]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:30.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:30 smithi117 ceph-mon[36368]: from='osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T10:36:30.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:30 smithi117 ceph-mon[36368]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T10:36:30.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:30 smithi117 ceph-mon[36368]: from='osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-24T10:36:30.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:30 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T10:36:31.122 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T10:36:31.122 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-24T10:36:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-24T10:36:31.123 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111726 s, 939 MB/s 2024-03-24T10:36:31.123 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-24T10:36:31.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:31 smithi023 ceph-mon[32466]: purged_snaps scrub starts 2024-03-24T10:36:31.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:31 smithi023 ceph-mon[32466]: purged_snaps scrub ok 2024-03-24T10:36:31.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:31 smithi023 ceph-mon[32466]: from='osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T10:36:31.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:31 smithi023 ceph-mon[32466]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T10:36:31.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:31 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:31.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:31 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:31.691 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:31 smithi116 ceph-mon[36248]: purged_snaps scrub starts 2024-03-24T10:36:31.692 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:31 smithi116 ceph-mon[36248]: purged_snaps scrub ok 2024-03-24T10:36:31.692 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:31 smithi116 ceph-mon[36248]: from='osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T10:36:31.692 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:31 smithi116 ceph-mon[36248]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T10:36:31.692 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:31 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:31.692 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:31 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:31.719 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_1 2024-03-24T10:36:31.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:31 smithi117 ceph-mon[36368]: purged_snaps scrub starts 2024-03-24T10:36:31.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:31 smithi117 ceph-mon[36368]: purged_snaps scrub ok 2024-03-24T10:36:31.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:31 smithi117 ceph-mon[36368]: from='osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-24T10:36:31.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:31 smithi117 ceph-mon[36368]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T10:36:31.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:31 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:31.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:31 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:31.971 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T10:36:32.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:32 smithi023 ceph-mon[32466]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:32.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:32 smithi023 ceph-mon[32466]: osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530] boot 2024-03-24T10:36:32.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:32 smithi023 ceph-mon[32466]: osdmap e67: 10 total, 10 up, 10 in 2024-03-24T10:36:32.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:32.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:32.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:32.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:32.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:32.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:32.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:32.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:32.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:32.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:32.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:32.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:32 smithi117 ceph-mon[36368]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:32.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:32 smithi117 ceph-mon[36368]: osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530] boot 2024-03-24T10:36:32.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:32 smithi117 ceph-mon[36368]: osdmap e67: 10 total, 10 up, 10 in 2024-03-24T10:36:32.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:32.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:32.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:32.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:32.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:32.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:32.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:32.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:32.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:32.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:32.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:32.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:32 smithi116 ceph-mon[36248]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-24T10:36:32.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:32 smithi116 ceph-mon[36248]: osd.9 [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530] boot 2024-03-24T10:36:32.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:32 smithi116 ceph-mon[36248]: osdmap e67: 10 total, 10 up, 10 in 2024-03-24T10:36:32.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:36:32.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:32.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:32.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:32.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:32.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:32.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:32.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:32.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:32.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:32.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.489+0000 7f1bee770700 1 -- 172.21.15.116:0/1285178673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be8102320 msgr2=0x7f1be8102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.489+0000 7f1bee770700 1 --2- 172.21.15.116:0/1285178673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be8102320 0x7f1be8102780 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc0099c0 tx=0x7f1bdc009cd0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.489+0000 7f1bee770700 1 -- 172.21.15.116:0/1285178673 shutdown_connections 2024-03-24T10:36:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.489+0000 7f1bee770700 1 --2- 172.21.15.116:0/1285178673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1be8101120 0x7f1be8101540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.489+0000 7f1bee770700 1 --2- 172.21.15.116:0/1285178673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be8102cc0 0x7f1be8109570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.489+0000 7f1bee770700 1 --2- 172.21.15.116:0/1285178673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be8102320 0x7f1be8102780 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.489+0000 7f1bee770700 1 -- 172.21.15.116:0/1285178673 >> 172.21.15.116:0/1285178673 conn(0x7f1be80fc6c0 msgr2=0x7f1be80feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.489+0000 7f1bee770700 1 -- 172.21.15.116:0/1285178673 shutdown_connections 2024-03-24T10:36:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.489+0000 7f1bee770700 1 -- 172.21.15.116:0/1285178673 wait complete. 2024-03-24T10:36:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.489+0000 7f1bee770700 1 Processor -- start 2024-03-24T10:36:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1bee770700 1 -- start start 2024-03-24T10:36:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1bee770700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be8101120 0x7f1be81981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1bee770700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be8102320 0x7f1be81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1bee770700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1be8102cc0 0x7f1be819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1bee770700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f1be819fd40 con 0x7f1be8102320 2024-03-24T10:36:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1bee770700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1be819fe80 con 0x7f1be8101120 2024-03-24T10:36:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1bee770700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1be81a0010 con 0x7f1be8102cc0 2024-03-24T10:36:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1be7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be8101120 0x7f1be81981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1be7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be8101120 0x7f1be81981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34868/0 (socket says 172.21.15.116:34868) 2024-03-24T10:36:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1be7fff700 1 -- 172.21.15.116:0/3820673216 learned_addr learned my addr 172.21.15.116:0/3820673216 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T10:36:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1becd0d700 1 --2- 172.21.15.116:0/3820673216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1be8102cc0 0x7f1be819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1be77fe700 1 --2- 172.21.15.116:0/3820673216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be8102320 0x7f1be81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1be7fff700 1 -- 172.21.15.116:0/3820673216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1be8102cc0 msgr2=0x7f1be819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1be7fff700 1 --2- 172.21.15.116:0/3820673216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1be8102cc0 0x7f1be819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1be7fff700 1 -- 172.21.15.116:0/3820673216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be8102320 msgr2=0x7f1be81986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1be7fff700 1 --2- 172.21.15.116:0/3820673216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be8102320 0x7f1be81986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1be7fff700 1 -- 172.21.15.116:0/3820673216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bdc0095f0 con 0x7f1be8101120 2024-03-24T10:36:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.490+0000 7f1be7fff700 1 --2- 172.21.15.116:0/3820673216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be8101120 0x7f1be81981b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1bd800d980 tx=0x7f1bd800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.491+0000 7f1be57fa700 1 -- 172.21.15.116:0/3820673216 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd800b950 con 0x7f1be8101120 2024-03-24T10:36:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.491+0000 7f1bee770700 1 -- 172.21.15.116:0/3820673216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1be81a02f0 con 0x7f1be8101120 2024-03-24T10:36:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.491+0000 7f1be57fa700 1 -- 172.21.15.116:0/3820673216 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bd8007620 con 0x7f1be8101120 2024-03-24T10:36:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.491+0000 7f1be57fa700 1 -- 172.21.15.116:0/3820673216 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd800f620 con 0x7f1be8101120 2024-03-24T10:36:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.491+0000 7f1bee770700 1 -- 172.21.15.116:0/3820673216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1be81a0840 con 0x7f1be8101120 2024-03-24T10:36:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.492+0000 7f1bee770700 1 -- 172.21.15.116:0/3820673216 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1be81084e0 con 0x7f1be8101120 2024-03-24T10:36:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.493+0000 7f1be57fa700 1 -- 172.21.15.116:0/3820673216 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f1bd80106f0 con 0x7f1be8101120 2024-03-24T10:36:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.493+0000 7f1be57fa700 1 --2- 172.21.15.116:0/3820673216 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f1bd00a2150 0x7f1bd00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:33.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.493+0000 7f1be57fa700 1 -- 172.21.15.116:0/3820673216 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f1bd80cdb80 con 0x7f1be8101120 2024-03-24T10:36:33.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.494+0000 7f1be77fe700 1 --2- 172.21.15.116:0/3820673216 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f1bd00a2150 0x7f1bd00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:33.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.494+0000 7f1be77fe700 1 --2- 172.21.15.116:0/3820673216 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f1bd00a2150 0x7f1bd00a4610 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc000c00 tx=0x7f1bdc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:33.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.497+0000 7f1be57fa700 1 -- 172.21.15.116:0/3820673216 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1bd8097350 con 0x7f1be8101120 2024-03-24T10:36:33.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:33 smithi023 ceph-mon[32466]: Detected new or changed devices on smithi023 2024-03-24T10:36:33.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:33 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi023 to 4272M 2024-03-24T10:36:33.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:33 smithi023 ceph-mon[32466]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T10:36:33.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:33.698+0000 7f1bee770700 1 -- 172.21.15.116:0/3820673216 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f1be802db90 con 0x7f1bd00a2150 2024-03-24T10:36:33.699 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:33 smithi116 ceph-mon[36248]: Detected new or changed devices on smithi023 2024-03-24T10:36:33.700 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:33 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi023 to 4272M 2024-03-24T10:36:33.700 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:33 smithi116 ceph-mon[36248]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T10:36:33.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:33 smithi117 ceph-mon[36368]: Detected new or changed devices on smithi023 2024-03-24T10:36:33.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:33 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi023 to 4272M 2024-03-24T10:36:33.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:33 smithi117 ceph-mon[36368]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T10:36:34.588 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:34 smithi116 ceph-mon[36248]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:34.588 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:34 smithi116 ceph-mon[36248]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:36:34.588 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:34 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:36:34.588 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:34 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:36:34.588 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:34 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:34.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:34 smithi117 ceph-mon[36368]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:34.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:34 smithi117 ceph-mon[36368]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:36:34.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:34 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:36:34.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:34 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:36:34.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:34 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:34.864 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:34 smithi023 ceph-mon[32466]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:34.864 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:34 smithi023 ceph-mon[32466]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:36:34.864 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:34 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:36:34.864 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:34 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:36:34.864 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:34 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:36 smithi117 ceph-mon[36368]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:36 smithi117 ceph-mon[36368]: from='client.? 172.21.15.116:0/3862954326' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cef53e0f-9f79-44c7-b8ea-6849ade8bb39"}]: dispatch 2024-03-24T10:36:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:36 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cef53e0f-9f79-44c7-b8ea-6849ade8bb39"}]: dispatch 2024-03-24T10:36:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:36 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cef53e0f-9f79-44c7-b8ea-6849ade8bb39"}]': finished 2024-03-24T10:36:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:36 smithi117 ceph-mon[36368]: osdmap e69: 11 total, 10 up, 11 in 2024-03-24T10:36:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:36 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:36.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:36 smithi116 ceph-mon[36248]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:36.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:36 smithi116 ceph-mon[36248]: from='client.? 172.21.15.116:0/3862954326' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cef53e0f-9f79-44c7-b8ea-6849ade8bb39"}]: dispatch 2024-03-24T10:36:36.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:36 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cef53e0f-9f79-44c7-b8ea-6849ade8bb39"}]: dispatch 2024-03-24T10:36:36.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:36 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cef53e0f-9f79-44c7-b8ea-6849ade8bb39"}]': finished 2024-03-24T10:36:36.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:36 smithi116 ceph-mon[36248]: osdmap e69: 11 total, 10 up, 11 in 2024-03-24T10:36:36.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:36 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:36.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:36 smithi023 ceph-mon[32466]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:36.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:36 smithi023 ceph-mon[32466]: from='client.? 172.21.15.116:0/3862954326' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cef53e0f-9f79-44c7-b8ea-6849ade8bb39"}]: dispatch 2024-03-24T10:36:36.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:36 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cef53e0f-9f79-44c7-b8ea-6849ade8bb39"}]: dispatch 2024-03-24T10:36:36.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:36 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cef53e0f-9f79-44c7-b8ea-6849ade8bb39"}]': finished 2024-03-24T10:36:36.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:36 smithi023 ceph-mon[32466]: osdmap e69: 11 total, 10 up, 11 in 2024-03-24T10:36:36.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:37.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:37 smithi117 ceph-mon[36368]: from='client.? 172.21.15.116:0/4124687132' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:36:37.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:37 smithi116 ceph-mon[36248]: from='client.? 172.21.15.116:0/4124687132' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:36:37.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:37 smithi023 ceph-mon[32466]: from='client.? 172.21.15.116:0/4124687132' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:36:38.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:38 smithi117 ceph-mon[36368]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:38.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:38 smithi116 ceph-mon[36248]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:38.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:38 smithi023 ceph-mon[32466]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:40.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:40 smithi117 ceph-mon[36368]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:40.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:40 smithi116 ceph-mon[36248]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:40.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:40 smithi023 ceph-mon[32466]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:42.633 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:42 smithi116 ceph-mon[36248]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:42.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:42 smithi117 ceph-mon[36368]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:42.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:42 smithi023 ceph-mon[32466]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:44.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:44 smithi117 ceph-mon[36368]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:44.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:44 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T10:36:44.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:44 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:44.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:44 smithi117 ceph-mon[36368]: Deploying daemon osd.10 on smithi116 2024-03-24T10:36:44.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:44 smithi116 ceph-mon[36248]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:44.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:44 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T10:36:44.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:44 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:44.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:44 smithi116 ceph-mon[36248]: Deploying daemon osd.10 on smithi116 2024-03-24T10:36:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:44 smithi023 ceph-mon[32466]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:44 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T10:36:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:44 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:44 smithi023 ceph-mon[32466]: Deploying daemon osd.10 on smithi116 2024-03-24T10:36:46.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:46 smithi117 ceph-mon[36368]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:46.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:46 smithi116 ceph-mon[36248]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:46.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:46 smithi023 ceph-mon[32466]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:48.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:48 smithi117 ceph-mon[36368]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:48.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:48 smithi116 ceph-mon[36248]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:48 smithi023 ceph-mon[32466]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:50.000 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:49 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:50.000 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:49 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:50.000 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:49 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:50.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:49 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:49 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:49 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:50.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:49 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:50.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:49 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:50.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:49 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:50.892 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:50 smithi116 ceph-mon[36248]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:51.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:50 smithi117 ceph-mon[36368]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:51.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:50 smithi023 ceph-mon[32466]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:52.465 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 10 on host 'smithi116' 2024-03-24T10:36:52.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:52.464+0000 7f1be57fa700 1 -- 172.21.15.116:0/3820673216 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f1be802db90 con 0x7f1bd00a2150 2024-03-24T10:36:52.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:52.467+0000 7f1bee770700 1 -- 172.21.15.116:0/3820673216 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f1bd00a2150 msgr2=0x7f1bd00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:52.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:52.467+0000 7f1bee770700 1 --2- 172.21.15.116:0/3820673216 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f1bd00a2150 0x7f1bd00a4610 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc000c00 tx=0x7f1bdc005fd0 comp rx=0 tx=0).stop 2024-03-24T10:36:52.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:52.467+0000 7f1bee770700 1 -- 172.21.15.116:0/3820673216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be8101120 msgr2=0x7f1be81981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:52.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:52.467+0000 7f1bee770700 1 --2- 172.21.15.116:0/3820673216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be8101120 0x7f1be81981b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1bd800d980 tx=0x7f1bd800dc90 comp rx=0 tx=0).stop 2024-03-24T10:36:52.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:52.467+0000 7f1bee770700 1 -- 172.21.15.116:0/3820673216 shutdown_connections 2024-03-24T10:36:52.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:52.467+0000 7f1bee770700 1 --2- 172.21.15.116:0/3820673216 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f1bd00a2150 0x7f1bd00a4610 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:52.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:52.467+0000 7f1bee770700 1 --2- 172.21.15.116:0/3820673216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1be8102cc0 0x7f1be819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:52.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:52.467+0000 7f1bee770700 1 --2- 172.21.15.116:0/3820673216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be8101120 0x7f1be81981b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:52.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:52.467+0000 7f1bee770700 1 --2- 172.21.15.116:0/3820673216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be8102320 0x7f1be81986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:52.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:52.467+0000 7f1bee770700 1 -- 172.21.15.116:0/3820673216 >> 172.21.15.116:0/3820673216 conn(0x7f1be80fc6c0 msgr2=0x7f1be8105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:52.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:52.467+0000 7f1bee770700 1 -- 172.21.15.116:0/3820673216 shutdown_connections 2024-03-24T10:36:52.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T10:36:52.467+0000 7f1bee770700 1 -- 172.21.15.116:0/3820673216 wait complete. 2024-03-24T10:36:52.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:52 smithi023 ceph-mon[32466]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:52.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:52.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:52.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:52.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:52.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:52.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:52.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:52.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:52.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:52 smithi023 ceph-mon[32466]: from='osd.10 [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T10:36:52.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:52 smithi023 ceph-mon[32466]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T10:36:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:52 smithi117 ceph-mon[36368]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:53.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:53.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:53.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:52 smithi117 ceph-mon[36368]: from='osd.10 [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T10:36:53.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:52 smithi117 ceph-mon[36368]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T10:36:53.007 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:52 smithi116 ceph-mon[36248]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:53.007 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:53.008 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:53.008 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:53.008 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:53.008 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:53.008 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:53.008 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:53.008 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:36:53.008 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:52 smithi116 ceph-mon[36248]: from='osd.10 [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T10:36:53.008 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:52 smithi116 ceph-mon[36248]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T10:36:53.040 DEBUG:teuthology.orchestra.run.smithi116:osd.10> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.10.service 2024-03-24T10:36:53.042 INFO:tasks.cephadm:Deploying osd.11 on smithi117 with /dev/vg_nvme/lv_1... 2024-03-24T10:36:53.043 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T10:36:53.288 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:36:54.668 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:54 smithi117 ceph-mon[36368]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:54.668 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:54 smithi117 ceph-mon[36368]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T10:36:54.668 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:54 smithi117 ceph-mon[36368]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T10:36:54.668 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:54 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:54.669 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:54 smithi117 ceph-mon[36368]: from='osd.10 [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:36:54.669 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:54 smithi117 ceph-mon[36368]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:36:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:54 smithi116 ceph-mon[36248]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:54 smithi116 ceph-mon[36248]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T10:36:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:54 smithi116 ceph-mon[36248]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T10:36:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:54 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:54.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:54 smithi116 ceph-mon[36248]: from='osd.10 [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:36:54.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:54 smithi116 ceph-mon[36248]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:36:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:54 smithi023 ceph-mon[32466]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:54 smithi023 ceph-mon[32466]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T10:36:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:54 smithi023 ceph-mon[32466]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T10:36:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:54 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:54 smithi023 ceph-mon[32466]: from='osd.10 [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:36:54.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:54 smithi023 ceph-mon[32466]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T10:36:55.237 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T10:36:55.238 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T10:36:55.238 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-24T10:36:55.238 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-24T10:36:55.238 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131722 s, 796 MB/s 2024-03-24T10:36:55.238 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-24T10:36:55.544 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:55 smithi117 ceph-mon[36368]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T10:36:55.544 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:55 smithi117 ceph-mon[36368]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T10:36:55.544 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:55 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:55.545 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:55 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:55.545 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:55 smithi117 ceph-mon[36368]: from='osd.10 ' entity='osd.10' 2024-03-24T10:36:55.572 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:55 smithi116 ceph-mon[36248]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T10:36:55.572 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:55 smithi116 ceph-mon[36248]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T10:36:55.572 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:55 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:55.572 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:55 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:55.572 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:55 smithi116 ceph-mon[36248]: from='osd.10 ' entity='osd.10' 2024-03-24T10:36:55.572 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:36:55 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:36:55.285+0000 7f832a955700 -1 osd.10 0 waiting for initial osdmap 2024-03-24T10:36:55.573 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:36:55 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:36:55.302+0000 7f8322f41700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:36:55.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_1 2024-03-24T10:36:55.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:55 smithi023 ceph-mon[32466]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T10:36:55.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:55 smithi023 ceph-mon[32466]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T10:36:55.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:55 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:55.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:55 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:55.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:55 smithi023 ceph-mon[32466]: from='osd.10 ' entity='osd.10' 2024-03-24T10:36:56.168 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:36:56.540 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: purged_snaps scrub starts 2024-03-24T10:36:56.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: purged_snaps scrub ok 2024-03-24T10:36:56.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:56.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:56.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: osd.10 [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037] boot 2024-03-24T10:36:56.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: osdmap e72: 11 total, 11 up, 11 in 2024-03-24T10:36:56.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:56.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:56.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:56.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:56.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:56.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:56.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:56.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:56.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:56.542 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:56.542 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:56.542 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:56 smithi117 ceph-mon[36368]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T10:36:56.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: purged_snaps scrub starts 2024-03-24T10:36:56.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: purged_snaps scrub ok 2024-03-24T10:36:56.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:56.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:56.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: osd.10 [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037] boot 2024-03-24T10:36:56.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: osdmap e72: 11 total, 11 up, 11 in 2024-03-24T10:36:56.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:56.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:56.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:56.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:56.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:56.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:56.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:56.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:56.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:56.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:56.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:56.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:56 smithi116 ceph-mon[36248]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T10:36:56.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: purged_snaps scrub starts 2024-03-24T10:36:56.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: purged_snaps scrub ok 2024-03-24T10:36:56.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-24T10:36:56.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:56.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: osd.10 [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037] boot 2024-03-24T10:36:56.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: osdmap e72: 11 total, 11 up, 11 in 2024-03-24T10:36:56.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:36:56.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:56.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:56.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:56.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:56.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:56.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:36:56.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:56.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:56.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:36:56.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:36:56.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:56 smithi023 ceph-mon[32466]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T10:36:57.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.659+0000 7f7a4e44e700 1 -- 172.21.15.117:0/1293125811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a481055f0 msgr2=0x7f7a481079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:57.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.659+0000 7f7a4e44e700 1 --2- 172.21.15.117:0/1293125811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a481055f0 0x7f7a481079e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009990 tx=0x7f7a38009ca0 comp rx=0 tx=0).stop 2024-03-24T10:36:57.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.660+0000 7f7a4e44e700 1 -- 172.21.15.117:0/1293125811 shutdown_connections 2024-03-24T10:36:57.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.660+0000 7f7a4e44e700 1 --2- 172.21.15.117:0/1293125811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a48107f20 0x7f7a4810a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:57.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.660+0000 7f7a4e44e700 1 --2- 172.21.15.117:0/1293125811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a48102cc0 0x7f7a481050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:57.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.660+0000 7f7a4e44e700 1 --2- 172.21.15.117:0/1293125811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a481055f0 0x7f7a481079e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:57.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.660+0000 7f7a4e44e700 1 -- 172.21.15.117:0/1293125811 >> 172.21.15.117:0/1293125811 conn(0x7f7a480fc6f0 msgr2=0x7f7a480feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:36:57.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.660+0000 7f7a4e44e700 1 -- 172.21.15.117:0/1293125811 shutdown_connections 2024-03-24T10:36:57.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.660+0000 7f7a4e44e700 1 -- 172.21.15.117:0/1293125811 wait complete. 2024-03-24T10:36:57.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.660+0000 7f7a4e44e700 1 Processor -- start 2024-03-24T10:36:57.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.660+0000 7f7a4e44e700 1 -- start start 2024-03-24T10:36:57.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.660+0000 7f7a4e44e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a48102cc0 0x7f7a48198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:57.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a4e44e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a481055f0 0x7f7a48198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:57.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a4e44e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a48107f20 0x7f7a4819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:57.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a4e44e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4819fd50 con 0x7f7a48102cc0 2024-03-24T10:36:57.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a4e44e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4819fe90 con 0x7f7a48107f20 2024-03-24T10:36:57.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a4e44e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7a481a0020 con 0x7f7a481055f0 2024-03-24T10:36:57.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a477fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a481055f0 0x7f7a48198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:57.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a47fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a48102cc0 0x7f7a48198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:57.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a477fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a481055f0 0x7f7a48198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60280/0 (socket says 172.21.15.117:60280) 2024-03-24T10:36:57.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a477fe700 1 -- 172.21.15.117:0/3753178871 learned_addr learned my addr 172.21.15.117:0/3753178871 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T10:36:57.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a4c9eb700 1 --2- 172.21.15.117:0/3753178871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a48107f20 0x7f7a4819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:57.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a477fe700 1 -- 172.21.15.117:0/3753178871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a48107f20 msgr2=0x7f7a4819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:57.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a477fe700 1 --2- 172.21.15.117:0/3753178871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a48107f20 0x7f7a4819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:57.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a477fe700 1 -- 172.21.15.117:0/3753178871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a48102cc0 msgr2=0x7f7a48198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:36:57.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a477fe700 1 --2- 172.21.15.117:0/3753178871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a48102cc0 0x7f7a48198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:36:57.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a477fe700 1 -- 172.21.15.117:0/3753178871 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a380095f0 con 0x7f7a481055f0 2024-03-24T10:36:57.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a477fe700 1 --2- 172.21.15.117:0/3753178871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a481055f0 0x7f7a48198750 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7a38000c00 tx=0x7f7a3800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:57.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a457fa700 1 -- 172.21.15.117:0/3753178871 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3801d070 con 0x7f7a481055f0 2024-03-24T10:36:57.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a4e44e700 1 -- 172.21.15.117:0/3753178871 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a481a02a0 con 0x7f7a481055f0 2024-03-24T10:36:57.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a457fa700 1 -- 172.21.15.117:0/3753178871 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a38022470 con 0x7f7a481055f0 2024-03-24T10:36:57.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a457fa700 1 -- 172.21.15.117:0/3753178871 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38017650 con 0x7f7a481055f0 2024-03-24T10:36:57.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.661+0000 7f7a4e44e700 1 -- 172.21.15.117:0/3753178871 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a481a0710 con 0x7f7a481055f0 2024-03-24T10:36:57.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.662+0000 7f7a4e44e700 1 -- 172.21.15.117:0/3753178871 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a480fe270 con 0x7f7a481055f0 2024-03-24T10:36:57.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.663+0000 7f7a457fa700 1 -- 172.21.15.117:0/3753178871 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f7a380225e0 con 0x7f7a481055f0 2024-03-24T10:36:57.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.664+0000 7f7a457fa700 1 --2- 172.21.15.117:0/3753178871 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7a300a2150 0x7f7a300a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:36:57.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.664+0000 7f7a457fa700 1 -- 172.21.15.117:0/3753178871 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7f7a380cf000 con 0x7f7a481055f0 2024-03-24T10:36:57.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.664+0000 7f7a47fff700 1 --2- 172.21.15.117:0/3753178871 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7a300a2150 0x7f7a300a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:36:57.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.665+0000 7f7a47fff700 1 --2- 172.21.15.117:0/3753178871 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7a300a2150 0x7f7a300a4610 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c004630 tx=0x7f7a3c00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:36:57.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.668+0000 7f7a457fa700 1 -- 172.21.15.117:0/3753178871 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7a38098690 con 0x7f7a481055f0 2024-03-24T10:36:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:57 smithi117 ceph-mon[36368]: Detected new or changed devices on smithi116 2024-03-24T10:36:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:57 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi116 to 4270M 2024-03-24T10:36:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:57 smithi117 ceph-mon[36368]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T10:36:57.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:57 smithi116 ceph-mon[36248]: Detected new or changed devices on smithi116 2024-03-24T10:36:57.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:57 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi116 to 4270M 2024-03-24T10:36:57.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:57 smithi116 ceph-mon[36248]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T10:36:57.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:36:57.873+0000 7f7a4e44e700 1 -- 172.21.15.117:0/3753178871 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7a4802db60 con 0x7f7a300a2150 2024-03-24T10:36:57.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:57 smithi023 ceph-mon[32466]: Detected new or changed devices on smithi116 2024-03-24T10:36:57.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:57 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi116 to 4270M 2024-03-24T10:36:57.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:57 smithi023 ceph-mon[32466]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T10:36:58.719 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:58 smithi117 ceph-mon[36368]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:36:58.719 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:36:58.719 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:36:58.719 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:58 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:58.719 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:58 smithi117 ceph-mon[36368]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T10:36:58.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:58 smithi116 ceph-mon[36248]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:36:58.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:58 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:36:58.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:58 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:36:58.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:58 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:58.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:58 smithi116 ceph-mon[36248]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T10:36:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:58 smithi023 ceph-mon[32466]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:36:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:58 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T10:36:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:58 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T10:36:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:58 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:36:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:58 smithi023 ceph-mon[32466]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T10:36:59.744 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:59 smithi117 ceph-mon[36368]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:36:59.744 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:36:59 smithi117 ceph-mon[36368]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T10:36:59.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:59 smithi116 ceph-mon[36248]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:36:59.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:36:59 smithi116 ceph-mon[36248]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T10:36:59.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:59 smithi023 ceph-mon[32466]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:36:59.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:36:59 smithi023 ceph-mon[32466]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T10:37:00.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:00 smithi117 ceph-mon[36368]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:00.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:00 smithi116 ceph-mon[36248]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:00.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:00 smithi023 ceph-mon[32466]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:01.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:01 smithi023 ceph-mon[32466]: from='client.? 172.21.15.117:0/4088679228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d763fbde-fd35-4847-b8f1-8f22553ee1c3"}]: dispatch 2024-03-24T10:37:01.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:01 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d763fbde-fd35-4847-b8f1-8f22553ee1c3"}]: dispatch 2024-03-24T10:37:01.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:01 smithi023 ceph-mon[32466]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d763fbde-fd35-4847-b8f1-8f22553ee1c3"}]': finished 2024-03-24T10:37:01.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:01 smithi023 ceph-mon[32466]: osdmap e77: 12 total, 11 up, 12 in 2024-03-24T10:37:01.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:01 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:01.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:01 smithi023 ceph-mon[32466]: from='client.? 172.21.15.117:0/859373941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:37:02.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:01 smithi117 ceph-mon[36368]: from='client.? 172.21.15.117:0/4088679228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d763fbde-fd35-4847-b8f1-8f22553ee1c3"}]: dispatch 2024-03-24T10:37:02.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:01 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d763fbde-fd35-4847-b8f1-8f22553ee1c3"}]: dispatch 2024-03-24T10:37:02.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:01 smithi117 ceph-mon[36368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d763fbde-fd35-4847-b8f1-8f22553ee1c3"}]': finished 2024-03-24T10:37:02.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:01 smithi117 ceph-mon[36368]: osdmap e77: 12 total, 11 up, 12 in 2024-03-24T10:37:02.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:01 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:02.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:01 smithi117 ceph-mon[36368]: from='client.? 172.21.15.117:0/859373941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:37:02.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:01 smithi116 ceph-mon[36248]: from='client.? 172.21.15.117:0/4088679228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d763fbde-fd35-4847-b8f1-8f22553ee1c3"}]: dispatch 2024-03-24T10:37:02.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:01 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d763fbde-fd35-4847-b8f1-8f22553ee1c3"}]: dispatch 2024-03-24T10:37:02.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:01 smithi116 ceph-mon[36248]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d763fbde-fd35-4847-b8f1-8f22553ee1c3"}]': finished 2024-03-24T10:37:02.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:01 smithi116 ceph-mon[36248]: osdmap e77: 12 total, 11 up, 12 in 2024-03-24T10:37:02.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:01 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:02.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:01 smithi116 ceph-mon[36248]: from='client.? 172.21.15.117:0/859373941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T10:37:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:02 smithi023 ceph-mon[32466]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:03.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:02 smithi117 ceph-mon[36368]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:03.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:02 smithi116 ceph-mon[36248]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:04.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:04 smithi117 ceph-mon[36368]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-24T10:37:04.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:04 smithi023 ceph-mon[32466]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-24T10:37:05.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:04 smithi116 ceph-mon[36248]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-24T10:37:06.853 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:06 smithi023 ceph-mon[32466]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T10:37:06.995 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:06 smithi117 ceph-mon[36368]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T10:37:07.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:06 smithi116 ceph-mon[36248]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-24T10:37:08.878 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:08 smithi117 ceph-mon[36368]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T10:37:08.879 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:08 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T10:37:08.879 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:08 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:08.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:08 smithi023 ceph-mon[32466]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T10:37:08.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:08 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T10:37:08.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:08 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:09.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:08 smithi116 ceph-mon[36248]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T10:37:09.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:08 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T10:37:09.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:08 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:09.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:09 smithi117 ceph-mon[36368]: Deploying daemon osd.11 on smithi117 2024-03-24T10:37:09.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:09 smithi023 ceph-mon[32466]: Deploying daemon osd.11 on smithi117 2024-03-24T10:37:10.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:09 smithi116 ceph-mon[36248]: Deploying daemon osd.11 on smithi117 2024-03-24T10:37:10.898 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:10 smithi117 ceph-mon[36368]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-24T10:37:10.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:10 smithi023 ceph-mon[32466]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-24T10:37:11.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:10 smithi116 ceph-mon[36248]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-24T10:37:12.807 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:12 smithi117 ceph-mon[36368]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-24T10:37:12.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:12 smithi023 ceph-mon[32466]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-24T10:37:13.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:12 smithi116 ceph-mon[36248]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-24T10:37:15.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:14 smithi117 ceph-mon[36368]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-24T10:37:15.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:14 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:15.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:14 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:15.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:14 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:15.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:14 smithi116 ceph-mon[36248]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-24T10:37:15.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:15.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:15.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:14 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:14 smithi023 ceph-mon[32466]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-24T10:37:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:14 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:14 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:15.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:14 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:16.859 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:16 smithi117 ceph-mon[36368]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:16.937 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 11 on host 'smithi117' 2024-03-24T10:37:16.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:16.936+0000 7f7a457fa700 1 -- 172.21.15.117:0/3753178871 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f7a4802db60 con 0x7f7a300a2150 2024-03-24T10:37:16.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:16.938+0000 7f7a4e44e700 1 -- 172.21.15.117:0/3753178871 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7a300a2150 msgr2=0x7f7a300a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:16.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:16.938+0000 7f7a4e44e700 1 --2- 172.21.15.117:0/3753178871 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7a300a2150 0x7f7a300a4610 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c004630 tx=0x7f7a3c00b3c0 comp rx=0 tx=0).stop 2024-03-24T10:37:16.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:16.938+0000 7f7a4e44e700 1 -- 172.21.15.117:0/3753178871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a481055f0 msgr2=0x7f7a48198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:16.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:16.938+0000 7f7a4e44e700 1 --2- 172.21.15.117:0/3753178871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a481055f0 0x7f7a48198750 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7a38000c00 tx=0x7f7a3800f690 comp rx=0 tx=0).stop 2024-03-24T10:37:16.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:16.939+0000 7f7a4e44e700 1 -- 172.21.15.117:0/3753178871 shutdown_connections 2024-03-24T10:37:16.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:16.939+0000 7f7a4e44e700 1 --2- 172.21.15.117:0/3753178871 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7a300a2150 0x7f7a300a4610 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:16.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:16.939+0000 7f7a4e44e700 1 --2- 172.21.15.117:0/3753178871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a48107f20 0x7f7a4819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:16.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:16.939+0000 7f7a4e44e700 1 --2- 172.21.15.117:0/3753178871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a48102cc0 0x7f7a48198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:16.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:16.939+0000 7f7a4e44e700 1 --2- 172.21.15.117:0/3753178871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a481055f0 0x7f7a48198750 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:16.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:16.939+0000 7f7a4e44e700 1 -- 172.21.15.117:0/3753178871 >> 172.21.15.117:0/3753178871 conn(0x7f7a480fc6f0 msgr2=0x7f7a48103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:16.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:16.939+0000 7f7a4e44e700 1 -- 172.21.15.117:0/3753178871 shutdown_connections 2024-03-24T10:37:16.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:16.939+0000 7f7a4e44e700 1 -- 172.21.15.117:0/3753178871 wait complete. 2024-03-24T10:37:17.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:16 smithi116 ceph-mon[36248]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:17.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:16 smithi023 ceph-mon[32466]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:17.563 DEBUG:teuthology.orchestra.run.smithi117:osd.11> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.11.service 2024-03-24T10:37:17.565 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-24T10:37:17.566 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T10:37:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:37:17.902 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:37:17 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:37:17.646+0000 7f4a6b6dd7c0 -1 osd.11 0 log_to_monitors true 2024-03-24T10:37:18.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:17 smithi023 ceph-mon[32466]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:18.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:18.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:18.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:17 smithi023 ceph-mon[32466]: from='osd.11 [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T10:37:18.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:17 smithi023 ceph-mon[32466]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T10:37:18.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:18.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:17 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:18.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:17 smithi117 ceph-mon[36368]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:18.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:17 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:18.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:17 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:18.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:17 smithi117 ceph-mon[36368]: from='osd.11 [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T10:37:18.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:17 smithi117 ceph-mon[36368]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T10:37:18.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:17 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:18.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:17 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:18.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:17 smithi116 ceph-mon[36248]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:18.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:17 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:18.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:17 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:18.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:17 smithi116 ceph-mon[36248]: from='osd.11 [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T10:37:18.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:17 smithi116 ceph-mon[36248]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T10:37:18.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:17 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:18.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:17 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:19.189 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:18 smithi023 ceph-mon[32466]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T10:37:19.189 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:18 smithi023 ceph-mon[32466]: from='osd.11 [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:37:19.189 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:18 smithi023 ceph-mon[32466]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T10:37:19.189 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:18 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:19.190 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:18 smithi023 ceph-mon[32466]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:37:19.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:18 smithi117 ceph-mon[36368]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T10:37:19.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:18 smithi117 ceph-mon[36368]: from='osd.11 [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:37:19.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:18 smithi117 ceph-mon[36368]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T10:37:19.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:18 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:19.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:18 smithi117 ceph-mon[36368]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:37:19.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:18 smithi116 ceph-mon[36248]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T10:37:19.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:18 smithi116 ceph-mon[36248]: from='osd.11 [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:37:19.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:18 smithi116 ceph-mon[36248]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T10:37:19.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:18 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:19.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:18 smithi116 ceph-mon[36248]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T10:37:19.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.505+0000 7f039b9ba700 1 -- 172.21.15.23:0/1032026895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0394102d00 msgr2=0x7f03941095b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:19.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.505+0000 7f039b9ba700 1 --2- 172.21.15.23:0/1032026895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0394102d00 0x7f03941095b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f038c00b0d0 tx=0x7f038c00b3e0 comp rx=0 tx=0).stop 2024-03-24T10:37:19.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.505+0000 7f039b9ba700 1 -- 172.21.15.23:0/1032026895 shutdown_connections 2024-03-24T10:37:19.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.505+0000 7f039b9ba700 1 --2- 172.21.15.23:0/1032026895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0394101160 0x7f0394101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:19.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.505+0000 7f039b9ba700 1 --2- 172.21.15.23:0/1032026895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0394102d00 0x7f03941095b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:19.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.505+0000 7f039b9ba700 1 --2- 172.21.15.23:0/1032026895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0394102360 0x7f03941027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:19.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.505+0000 7f039b9ba700 1 -- 172.21.15.23:0/1032026895 >> 172.21.15.23:0/1032026895 conn(0x7f03940fc700 msgr2=0x7f03940feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:19.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.505+0000 7f039b9ba700 1 -- 172.21.15.23:0/1032026895 shutdown_connections 2024-03-24T10:37:19.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.505+0000 7f039b9ba700 1 -- 172.21.15.23:0/1032026895 wait complete. 2024-03-24T10:37:19.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f039b9ba700 1 Processor -- start 2024-03-24T10:37:19.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f039b9ba700 1 -- start start 2024-03-24T10:37:19.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f039b9ba700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0394101160 0x7f039406b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:19.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f039b9ba700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0394102360 0x7f039419cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:19.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f039b9ba700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0394102d00 0x7f039419f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:19.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f039b9ba700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f039419fdc0 con 0x7f0394102360 2024-03-24T10:37:19.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f039b9ba700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f039419ff30 con 0x7f0394101160 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f039b9ba700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f03941a00f0 con 0x7f0394102d00 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f0398f55700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0394102360 0x7f039419cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f0398f55700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0394102360 0x7f039419cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52518/0 (socket says 172.21.15.23:52518) 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f0398f55700 1 -- 172.21.15.23:0/4196959373 learned_addr learned my addr 172.21.15.23:0/4196959373 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f0399f57700 1 --2- 172.21.15.23:0/4196959373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0394102d00 0x7f039419f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f0399756700 1 --2- 172.21.15.23:0/4196959373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0394101160 0x7f039406b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f0398f55700 1 -- 172.21.15.23:0/4196959373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0394102d00 msgr2=0x7f039419f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f0398f55700 1 --2- 172.21.15.23:0/4196959373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0394102d00 0x7f039419f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f0398f55700 1 -- 172.21.15.23:0/4196959373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0394101160 msgr2=0x7f039406b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f0398f55700 1 --2- 172.21.15.23:0/4196959373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0394101160 0x7f039406b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.506+0000 7f0398f55700 1 -- 172.21.15.23:0/4196959373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038c009d00 con 0x7f0394102360 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.507+0000 7f0398f55700 1 --2- 172.21.15.23:0/4196959373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0394102360 0x7f039419cfe0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f039000d950 tx=0x7f039000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.507+0000 7f038a7fc700 1 -- 172.21.15.23:0/4196959373 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039000f890 con 0x7f0394102360 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.507+0000 7f039b9ba700 1 -- 172.21.15.23:0/4196959373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03941a03d0 con 0x7f0394102360 2024-03-24T10:37:19.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.507+0000 7f038a7fc700 1 -- 172.21.15.23:0/4196959373 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0390010460 con 0x7f0394102360 2024-03-24T10:37:19.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.507+0000 7f038a7fc700 1 -- 172.21.15.23:0/4196959373 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039000e610 con 0x7f0394102360 2024-03-24T10:37:19.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.507+0000 7f039b9ba700 1 -- 172.21.15.23:0/4196959373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03941a0920 con 0x7f0394102360 2024-03-24T10:37:19.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.508+0000 7f039b9ba700 1 -- 172.21.15.23:0/4196959373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0394108450 con 0x7f0394102360 2024-03-24T10:37:19.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.510+0000 7f038a7fc700 1 -- 172.21.15.23:0/4196959373 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f0390010960 con 0x7f0394102360 2024-03-24T10:37:19.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.510+0000 7f038a7fc700 1 --2- 172.21.15.23:0/4196959373 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f03800a2150 0x7f03800a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:19.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.510+0000 7f0399756700 1 --2- 172.21.15.23:0/4196959373 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f03800a2150 0x7f03800a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:19.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.510+0000 7f038a7fc700 1 -- 172.21.15.23:0/4196959373 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f03900cd7f0 con 0x7f0394102360 2024-03-24T10:37:19.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.511+0000 7f0399756700 1 --2- 172.21.15.23:0/4196959373 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f03800a2150 0x7f03800a4610 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f03941021c0 tx=0x7f0384009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:19.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.514+0000 7f038a7fc700 1 -- 172.21.15.23:0/4196959373 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0390096d80 con 0x7f0394102360 2024-03-24T10:37:19.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.721+0000 7f039b9ba700 1 -- 172.21.15.23:0/4196959373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f0394068ec0 con 0x7f0394102360 2024-03-24T10:37:19.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.721+0000 7f038a7fc700 1 -- 172.21.15.23:0/4196959373 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7f039009a440 con 0x7f0394102360 2024-03-24T10:37:19.722 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.723+0000 7f039b9ba700 1 -- 172.21.15.23:0/4196959373 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f03800a2150 msgr2=0x7f03800a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.723+0000 7f039b9ba700 1 --2- 172.21.15.23:0/4196959373 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f03800a2150 0x7f03800a4610 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f03941021c0 tx=0x7f0384009260 comp rx=0 tx=0).stop 2024-03-24T10:37:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.723+0000 7f039b9ba700 1 -- 172.21.15.23:0/4196959373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0394102360 msgr2=0x7f039419cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.723+0000 7f039b9ba700 1 --2- 172.21.15.23:0/4196959373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0394102360 0x7f039419cfe0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f039000d950 tx=0x7f039000dc60 comp rx=0 tx=0).stop 2024-03-24T10:37:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.724+0000 7f039b9ba700 1 -- 172.21.15.23:0/4196959373 shutdown_connections 2024-03-24T10:37:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.724+0000 7f039b9ba700 1 --2- 172.21.15.23:0/4196959373 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f03800a2150 0x7f03800a4610 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.724+0000 7f039b9ba700 1 --2- 172.21.15.23:0/4196959373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0394102d00 0x7f039419f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:19.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.724+0000 7f039b9ba700 1 --2- 172.21.15.23:0/4196959373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0394101160 0x7f039406b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:19.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.724+0000 7f039b9ba700 1 --2- 172.21.15.23:0/4196959373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0394102360 0x7f039419cfe0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:19.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.724+0000 7f039b9ba700 1 -- 172.21.15.23:0/4196959373 >> 172.21.15.23:0/4196959373 conn(0x7f03940fc700 msgr2=0x7f0394105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:19.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.724+0000 7f039b9ba700 1 -- 172.21.15.23:0/4196959373 shutdown_connections 2024-03-24T10:37:19.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:19.724+0000 7f039b9ba700 1 -- 172.21.15.23:0/4196959373 wait complete. 2024-03-24T10:37:19.927 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:37:19 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:37:19.711+0000 7f4a60541700 -1 osd.11 0 waiting for initial osdmap 2024-03-24T10:37:19.927 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:37:19 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:37:19.734+0000 7f4a58b2d700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:37:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:19 smithi023 ceph-mon[32466]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:19 smithi023 ceph-mon[32466]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T10:37:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:19 smithi023 ceph-mon[32466]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T10:37:20.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:20.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:20.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:19 smithi023 ceph-mon[32466]: from='osd.11 ' entity='osd.11' 2024-03-24T10:37:20.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:19 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4196959373' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T10:37:20.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:19 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:20.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:19 smithi117 ceph-mon[36368]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:20.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:19 smithi117 ceph-mon[36368]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T10:37:20.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:19 smithi117 ceph-mon[36368]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T10:37:20.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:19 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:20.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:19 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:20.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:19 smithi117 ceph-mon[36368]: from='osd.11 ' entity='osd.11' 2024-03-24T10:37:20.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:19 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4196959373' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T10:37:20.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:19 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:19 smithi116 ceph-mon[36248]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:19 smithi116 ceph-mon[36248]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T10:37:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:19 smithi116 ceph-mon[36248]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T10:37:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:19 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:19 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:19 smithi116 ceph-mon[36248]: from='osd.11 ' entity='osd.11' 2024-03-24T10:37:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:19 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4196959373' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T10:37:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:19 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:20.464 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1711276615,"num_in_osds":12,"osd_in_since":1711276620,"num_remapped_pgs":0} 2024-03-24T10:37:21.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:20 smithi023 ceph-mon[32466]: purged_snaps scrub starts 2024-03-24T10:37:21.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:20 smithi023 ceph-mon[32466]: purged_snaps scrub ok 2024-03-24T10:37:21.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:20 smithi023 ceph-mon[32466]: osd.11 [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153] boot 2024-03-24T10:37:21.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:20 smithi023 ceph-mon[32466]: osdmap e80: 12 total, 12 up, 12 in 2024-03-24T10:37:21.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:20 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:21.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:20 smithi117 ceph-mon[36368]: purged_snaps scrub starts 2024-03-24T10:37:21.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:20 smithi117 ceph-mon[36368]: purged_snaps scrub ok 2024-03-24T10:37:21.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:20 smithi117 ceph-mon[36368]: osd.11 [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153] boot 2024-03-24T10:37:21.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:20 smithi117 ceph-mon[36368]: osdmap e80: 12 total, 12 up, 12 in 2024-03-24T10:37:21.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:20 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:21.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:20 smithi116 ceph-mon[36248]: purged_snaps scrub starts 2024-03-24T10:37:21.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:20 smithi116 ceph-mon[36248]: purged_snaps scrub ok 2024-03-24T10:37:21.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:20 smithi116 ceph-mon[36248]: osd.11 [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153] boot 2024-03-24T10:37:21.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:20 smithi116 ceph-mon[36248]: osdmap e80: 12 total, 12 up, 12 in 2024-03-24T10:37:21.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:20 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:37:21.466 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T10:37:21.721 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:37:22.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:22.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: Detected new or changed devices on smithi117 2024-03-24T10:37:22.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:22.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:22.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:37:22.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:37:22.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:37:22.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:37:22.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi117 to 4270M 2024-03-24T10:37:22.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:22.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:22.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:37:22.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:22.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:22.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:22.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:37:22.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:22.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:22.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: Detected new or changed devices on smithi117 2024-03-24T10:37:22.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:22.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:22.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:37:22.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:37:22.504 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:37:22.504 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:37:22.504 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi117 to 4270M 2024-03-24T10:37:22.504 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:22.504 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:22.504 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:37:22.504 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:22.504 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:22.504 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:22.504 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:37:22.504 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:22.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-24T10:37:22.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: Detected new or changed devices on smithi117 2024-03-24T10:37:22.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:22.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:22.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:37:22.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:37:22.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:37:22.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:37:22.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi117 to 4270M 2024-03-24T10:37:22.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:22.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:22.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:37:22.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:22.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:22.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:22.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:37:22.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:23.401 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:23 smithi023 ceph-mon[32466]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T10:37:23.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.416+0000 7fb219cf0700 1 -- 172.21.15.23:0/442932834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2141056c0 msgr2=0x7fb214107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:23.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.416+0000 7fb219cf0700 1 --2- 172.21.15.23:0/442932834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2141056c0 0x7fb214107ab0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb208009990 tx=0x7fb208009ca0 comp rx=0 tx=0).stop 2024-03-24T10:37:23.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.416+0000 7fb219cf0700 1 -- 172.21.15.23:0/442932834 shutdown_connections 2024-03-24T10:37:23.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.416+0000 7fb219cf0700 1 --2- 172.21.15.23:0/442932834 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb214102d90 0x7fb214105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:23.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.416+0000 7fb219cf0700 1 --2- 172.21.15.23:0/442932834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb214107ff0 0x7fb21410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:23.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.416+0000 7fb219cf0700 1 --2- 172.21.15.23:0/442932834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2141056c0 0x7fb214107ab0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:23.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.416+0000 7fb219cf0700 1 -- 172.21.15.23:0/442932834 >> 172.21.15.23:0/442932834 conn(0x7fb2140fc760 msgr2=0x7fb2140feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:23.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.416+0000 7fb219cf0700 1 -- 172.21.15.23:0/442932834 shutdown_connections 2024-03-24T10:37:23.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.416+0000 7fb219cf0700 1 -- 172.21.15.23:0/442932834 wait complete. 2024-03-24T10:37:23.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.417+0000 7fb219cf0700 1 Processor -- start 2024-03-24T10:37:23.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.417+0000 7fb219cf0700 1 -- start start 2024-03-24T10:37:23.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.417+0000 7fb219cf0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb214102d90 0x7fb2141981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:23.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.417+0000 7fb219cf0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2141056c0 0x7fb214198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:23.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.417+0000 7fb219cf0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb214107ff0 0x7fb21419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:23.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.417+0000 7fb219cf0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb21419fd80 con 0x7fb214102d90 2024-03-24T10:37:23.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.417+0000 7fb219cf0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb21419fec0 con 0x7fb214107ff0 2024-03-24T10:37:23.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.417+0000 7fb219cf0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb2141a0050 con 0x7fb2141056c0 2024-03-24T10:37:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.417+0000 7fb2137fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb214102d90 0x7fb2141981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.417+0000 7fb2137fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb214102d90 0x7fb2141981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52538/0 (socket says 172.21.15.23:52538) 2024-03-24T10:37:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.417+0000 7fb2137fe700 1 -- 172.21.15.23:0/2650249358 learned_addr learned my addr 172.21.15.23:0/2650249358 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.417+0000 7fb212ffd700 1 --2- 172.21.15.23:0/2650249358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2141056c0 0x7fb214198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.417+0000 7fb213fff700 1 --2- 172.21.15.23:0/2650249358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb214107ff0 0x7fb21419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.418+0000 7fb2137fe700 1 -- 172.21.15.23:0/2650249358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2141056c0 msgr2=0x7fb214198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.418+0000 7fb2137fe700 1 --2- 172.21.15.23:0/2650249358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2141056c0 0x7fb214198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.418+0000 7fb2137fe700 1 -- 172.21.15.23:0/2650249358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb214107ff0 msgr2=0x7fb21419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.418+0000 7fb2137fe700 1 --2- 172.21.15.23:0/2650249358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb214107ff0 0x7fb21419f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.418+0000 7fb2137fe700 1 -- 172.21.15.23:0/2650249358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2080095f0 con 0x7fb214102d90 2024-03-24T10:37:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.418+0000 7fb2137fe700 1 --2- 172.21.15.23:0/2650249358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb214102d90 0x7fb2141981f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb20400d980 tx=0x7fb20400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.418+0000 7fb210ff9700 1 -- 172.21.15.23:0/2650249358 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2040098d0 con 0x7fb214102d90 2024-03-24T10:37:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.418+0000 7fb219cf0700 1 -- 172.21.15.23:0/2650249358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2141a0330 con 0x7fb214102d90 2024-03-24T10:37:23.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.418+0000 7fb210ff9700 1 -- 172.21.15.23:0/2650249358 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb204010460 con 0x7fb214102d90 2024-03-24T10:37:23.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.418+0000 7fb210ff9700 1 -- 172.21.15.23:0/2650249358 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20400f620 con 0x7fb214102d90 2024-03-24T10:37:23.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.418+0000 7fb219cf0700 1 -- 172.21.15.23:0/2650249358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2141a0880 con 0x7fb214102d90 2024-03-24T10:37:23.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.419+0000 7fb219cf0700 1 -- 172.21.15.23:0/2650249358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2140fe2e0 con 0x7fb214102d90 2024-03-24T10:37:23.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.420+0000 7fb210ff9700 1 -- 172.21.15.23:0/2650249358 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fb2040108f0 con 0x7fb214102d90 2024-03-24T10:37:23.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.421+0000 7fb210ff9700 1 --2- 172.21.15.23:0/2650249358 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb1fc0a2150 0x7fb1fc0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:23.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.421+0000 7fb212ffd700 1 --2- 172.21.15.23:0/2650249358 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb1fc0a2150 0x7fb1fc0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:23.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.421+0000 7fb210ff9700 1 -- 172.21.15.23:0/2650249358 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb2040cd560 con 0x7fb214102d90 2024-03-24T10:37:23.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.421+0000 7fb212ffd700 1 --2- 172.21.15.23:0/2650249358 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb1fc0a2150 0x7fb1fc0a4610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb208003790 tx=0x7fb208005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:23.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.425+0000 7fb210ff9700 1 -- 172.21.15.23:0/2650249358 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb2040969d0 con 0x7fb214102d90 2024-03-24T10:37:23.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.626+0000 7fb219cf0700 1 -- 172.21.15.23:0/2650249358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fb214068ec0 con 0x7fb214102d90 2024-03-24T10:37:23.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.627+0000 7fb210ff9700 1 -- 172.21.15.23:0/2650249358 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7fb20409a090 con 0x7fb214102d90 2024-03-24T10:37:23.627 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:23.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.629+0000 7fb219cf0700 1 -- 172.21.15.23:0/2650249358 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb1fc0a2150 msgr2=0x7fb1fc0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:23.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.629+0000 7fb219cf0700 1 --2- 172.21.15.23:0/2650249358 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb1fc0a2150 0x7fb1fc0a4610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb208003790 tx=0x7fb208005fd0 comp rx=0 tx=0).stop 2024-03-24T10:37:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.629+0000 7fb219cf0700 1 -- 172.21.15.23:0/2650249358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb214102d90 msgr2=0x7fb2141981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.629+0000 7fb219cf0700 1 --2- 172.21.15.23:0/2650249358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb214102d90 0x7fb2141981f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb20400d980 tx=0x7fb20400dc90 comp rx=0 tx=0).stop 2024-03-24T10:37:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.629+0000 7fb219cf0700 1 -- 172.21.15.23:0/2650249358 shutdown_connections 2024-03-24T10:37:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.629+0000 7fb219cf0700 1 --2- 172.21.15.23:0/2650249358 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb1fc0a2150 0x7fb1fc0a4610 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.629+0000 7fb219cf0700 1 --2- 172.21.15.23:0/2650249358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb214107ff0 0x7fb21419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.629+0000 7fb219cf0700 1 --2- 172.21.15.23:0/2650249358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb214102d90 0x7fb2141981f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.629+0000 7fb219cf0700 1 --2- 172.21.15.23:0/2650249358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2141056c0 0x7fb214198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.629+0000 7fb219cf0700 1 -- 172.21.15.23:0/2650249358 >> 172.21.15.23:0/2650249358 conn(0x7fb2140fc760 msgr2=0x7fb214103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.629+0000 7fb219cf0700 1 -- 172.21.15.23:0/2650249358 shutdown_connections 2024-03-24T10:37:23.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:23.629+0000 7fb219cf0700 1 -- 172.21.15.23:0/2650249358 wait complete. 2024-03-24T10:37:23.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:23 smithi117 ceph-mon[36368]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T10:37:23.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:23 smithi116 ceph-mon[36248]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T10:37:24.284 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1711276640,"num_in_osds":12,"osd_in_since":1711276620,"num_remapped_pgs":0} 2024-03-24T10:37:24.284 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T10:37:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:37:24.648 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:24 smithi023 ceph-mon[32466]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:24.649 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:24 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2650249358' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T10:37:24.654 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:24 smithi117 ceph-mon[36368]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:24.654 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:24 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2650249358' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T10:37:24.669 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:24 smithi116 ceph-mon[36248]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:24.669 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:24 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2650249358' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T10:37:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.108+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/1499858654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9d41010f0 msgr2=0x7fb9d4101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.108+0000 7fb9dc1c1700 1 --2- 172.21.15.23:0/1499858654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9d41010f0 0x7fb9d4101570 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009990 tx=0x7fb9c8009ca0 comp rx=0 tx=0).stop 2024-03-24T10:37:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.108+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/1499858654 shutdown_connections 2024-03-24T10:37:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.108+0000 7fb9dc1c1700 1 --2- 172.21.15.23:0/1499858654 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9d4101ab0 0x7fb9d410df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.108+0000 7fb9dc1c1700 1 --2- 172.21.15.23:0/1499858654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9d4107130 0x7fb9d4107510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.108+0000 7fb9dc1c1700 1 --2- 172.21.15.23:0/1499858654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9d41010f0 0x7fb9d4101570 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.108+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/1499858654 >> 172.21.15.23:0/1499858654 conn(0x7fb9d40780b0 msgr2=0x7fb9d40784c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.108+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/1499858654 shutdown_connections 2024-03-24T10:37:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.108+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/1499858654 wait complete. 2024-03-24T10:37:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9dc1c1700 1 Processor -- start 2024-03-24T10:37:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9dc1c1700 1 -- start start 2024-03-24T10:37:26.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9dc1c1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9d41010f0 0x7fb9d41975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:26.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9dc1c1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9d4101ab0 0x7fb9d4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:26.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9dc1c1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9d4107130 0x7fb9d419be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:26.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9dc1c1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d419c440 con 0x7fb9d41010f0 2024-03-24T10:37:26.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9dc1c1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d419c5b0 con 0x7fb9d4107130 2024-03-24T10:37:26.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9dc1c1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d419c770 con 0x7fb9d4101ab0 2024-03-24T10:37:26.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9d9f5d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9d41010f0 0x7fb9d41975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:26.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9d9f5d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9d41010f0 0x7fb9d41975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52552/0 (socket says 172.21.15.23:52552) 2024-03-24T10:37:26.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9d9f5d700 1 -- 172.21.15.23:0/4180346732 learned_addr learned my addr 172.21.15.23:0/4180346732 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:26.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9d975c700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9d4101ab0 0x7fb9d4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:26.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9da75e700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9d4107130 0x7fb9d419be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:26.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9d9f5d700 1 -- 172.21.15.23:0/4180346732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9d4101ab0 msgr2=0x7fb9d4197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:26.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9d9f5d700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9d4101ab0 0x7fb9d4197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:26.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9d9f5d700 1 -- 172.21.15.23:0/4180346732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9d4107130 msgr2=0x7fb9d419be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:26.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9d9f5d700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9d4107130 0x7fb9d419be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:26.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9d9f5d700 1 -- 172.21.15.23:0/4180346732 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c80095f0 con 0x7fb9d41010f0 2024-03-24T10:37:26.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.109+0000 7fb9da75e700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9d4107130 0x7fb9d419be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:37:26.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.110+0000 7fb9d9f5d700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9d41010f0 0x7fb9d41975b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40060b0 tx=0x7fb9c400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:26.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.110+0000 7fb9d2ffd700 1 -- 172.21.15.23:0/4180346732 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c400ddd0 con 0x7fb9d41010f0 2024-03-24T10:37:26.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.110+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/4180346732 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9d419ca50 con 0x7fb9d41010f0 2024-03-24T10:37:26.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.110+0000 7fb9d2ffd700 1 -- 172.21.15.23:0/4180346732 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9c400f040 con 0x7fb9d41010f0 2024-03-24T10:37:26.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.110+0000 7fb9d2ffd700 1 -- 172.21.15.23:0/4180346732 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4014720 con 0x7fb9d41010f0 2024-03-24T10:37:26.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.110+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/4180346732 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9d41a42b0 con 0x7fb9d41010f0 2024-03-24T10:37:26.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.111+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/4180346732 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9d4109650 con 0x7fb9d41010f0 2024-03-24T10:37:26.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.112+0000 7fb9d2ffd700 1 -- 172.21.15.23:0/4180346732 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fb9c4015400 con 0x7fb9d41010f0 2024-03-24T10:37:26.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.112+0000 7fb9d2ffd700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb9bc0a21d0 0x7fb9bc0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:26.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.113+0000 7fb9d975c700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb9bc0a21d0 0x7fb9bc0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:26.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.113+0000 7fb9d2ffd700 1 -- 172.21.15.23:0/4180346732 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb9c40cd160 con 0x7fb9d41010f0 2024-03-24T10:37:26.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.113+0000 7fb9d975c700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb9bc0a21d0 0x7fb9bc0a4690 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb9c800b5c0 tx=0x7fb9c8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:26.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.117+0000 7fb9d2ffd700 1 -- 172.21.15.23:0/4180346732 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb9c40965d0 con 0x7fb9d41010f0 2024-03-24T10:37:26.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.322+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/4180346732 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb9d41027e0 con 0x7fb9d41010f0 2024-03-24T10:37:26.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.323+0000 7fb9d2ffd700 1 -- 172.21.15.23:0/4180346732 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18470 (secure 0 0 0) 0x7fb9c4099c90 con 0x7fb9d41010f0 2024-03-24T10:37:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:26.324 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":81,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:37:22.316371+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:37:26.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.325+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/4180346732 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb9bc0a21d0 msgr2=0x7fb9bc0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:26.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.325+0000 7fb9dc1c1700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb9bc0a21d0 0x7fb9bc0a4690 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb9c800b5c0 tx=0x7fb9c8005960 comp rx=0 tx=0).stop 2024-03-24T10:37:26.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.325+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/4180346732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9d41010f0 msgr2=0x7fb9d41975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:26.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.325+0000 7fb9dc1c1700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9d41010f0 0x7fb9d41975b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40060b0 tx=0x7fb9c400d3b0 comp rx=0 tx=0).stop 2024-03-24T10:37:26.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.325+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/4180346732 shutdown_connections 2024-03-24T10:37:26.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.325+0000 7fb9dc1c1700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb9bc0a21d0 0x7fb9bc0a4690 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:26.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.325+0000 7fb9dc1c1700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9d4107130 0x7fb9d419be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:26.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.325+0000 7fb9dc1c1700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9d41010f0 0x7fb9d41975b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:26.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.325+0000 7fb9dc1c1700 1 --2- 172.21.15.23:0/4180346732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9d4101ab0 0x7fb9d4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:26.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.325+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/4180346732 >> 172.21.15.23:0/4180346732 conn(0x7fb9d40780b0 msgr2=0x7fb9d40ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:26.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.326+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/4180346732 shutdown_connections 2024-03-24T10:37:26.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:26.326+0000 7fb9dc1c1700 1 -- 172.21.15.23:0/4180346732 wait complete. 2024-03-24T10:37:26.612 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:26 smithi023 ceph-mon[32466]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:26.612 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:26 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4180346732' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:37:26.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:26 smithi117 ceph-mon[36368]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:26.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:26 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4180346732' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:37:26.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:26 smithi116 ceph-mon[36248]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:26.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:26 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4180346732' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:37:27.037 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:39.056452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:37:27.037 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T10:37:27.280 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:37:28.658 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:28 smithi023 ceph-mon[32466]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:28.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:28 smithi117 ceph-mon[36368]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:28.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:28 smithi116 ceph-mon[36248]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.915+0000 7f071a621700 1 -- 172.21.15.23:0/2106731659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0714102510 msgr2=0x7f0714105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.915+0000 7f071a621700 1 --2- 172.21.15.23:0/2106731659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0714102510 0x7f0714105da0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0704009990 tx=0x7f0704009ca0 comp rx=0 tx=0).stop 2024-03-24T10:37:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.915+0000 7f071a621700 1 -- 172.21.15.23:0/2106731659 shutdown_connections 2024-03-24T10:37:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.915+0000 7f071a621700 1 --2- 172.21.15.23:0/2106731659 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0714106480 0x7f071410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.915+0000 7f071a621700 1 --2- 172.21.15.23:0/2106731659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0714101b60 0x7f0714101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.915+0000 7f071a621700 1 --2- 172.21.15.23:0/2106731659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0714102510 0x7f0714105da0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.915+0000 7f071a621700 1 -- 172.21.15.23:0/2106731659 >> 172.21.15.23:0/2106731659 conn(0x7f0714077700 msgr2=0x7f07140feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.915+0000 7f071a621700 1 -- 172.21.15.23:0/2106731659 shutdown_connections 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.915+0000 7f071a621700 1 -- 172.21.15.23:0/2106731659 wait complete. 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.916+0000 7f071a621700 1 Processor -- start 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.916+0000 7f071a621700 1 -- start start 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.916+0000 7f071a621700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0714101b60 0x7f07140fcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.916+0000 7f071a621700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0714102510 0x7f07140fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.916+0000 7f071a621700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0714106480 0x7f07140fd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.916+0000 7f071a621700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f07141a6bb0 con 0x7f0714106480 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.916+0000 7f071a621700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f07141a6d20 con 0x7f0714101b60 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.916+0000 7f071a621700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f07141a6ee0 con 0x7f0714102510 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f0718bbe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0714106480 0x7f07140fd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f0718bbe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0714106480 0x7f07140fd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41386/0 (socket says 172.21.15.23:41386) 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f0718bbe700 1 -- 172.21.15.23:0/1583946798 learned_addr learned my addr 172.21.15.23:0/1583946798 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f07137fe700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0714102510 0x7f07140fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f0713fff700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0714101b60 0x7f07140fcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:28.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f0718bbe700 1 -- 172.21.15.23:0/1583946798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0714102510 msgr2=0x7f07140fd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:28.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f0718bbe700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0714102510 0x7f07140fd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:28.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f0718bbe700 1 -- 172.21.15.23:0/1583946798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0714101b60 msgr2=0x7f07140fcbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:28.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f0718bbe700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0714101b60 0x7f07140fcbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:28.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f0718bbe700 1 -- 172.21.15.23:0/1583946798 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07040095f0 con 0x7f0714106480 2024-03-24T10:37:28.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f0713fff700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0714101b60 0x7f07140fcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:37:28.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f0718bbe700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0714106480 0x7f07140fd660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f070c00ada0 tx=0x7f070c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:28.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f07117fa700 1 -- 172.21.15.23:0/1583946798 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c00f070 con 0x7f0714106480 2024-03-24T10:37:28.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f071a621700 1 -- 172.21.15.23:0/1583946798 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07141a71c0 con 0x7f0714106480 2024-03-24T10:37:28.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f07117fa700 1 -- 172.21.15.23:0/1583946798 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f070c007b30 con 0x7f0714106480 2024-03-24T10:37:28.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f07117fa700 1 -- 172.21.15.23:0/1583946798 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c0053c0 con 0x7f0714106480 2024-03-24T10:37:28.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.917+0000 7f071a621700 1 -- 172.21.15.23:0/1583946798 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07141a7690 con 0x7f0714106480 2024-03-24T10:37:28.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.918+0000 7f070affd700 1 -- 172.21.15.23:0/1583946798 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06f8005330 con 0x7f0714106480 2024-03-24T10:37:28.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.919+0000 7f07117fa700 1 -- 172.21.15.23:0/1583946798 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f070c005620 con 0x7f0714106480 2024-03-24T10:37:28.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.920+0000 7f07117fa700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f07000a21d0 0x7f07000a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:28.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.920+0000 7f0713fff700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f07000a21d0 0x7f07000a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:28.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.920+0000 7f07117fa700 1 -- 172.21.15.23:0/1583946798 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f070c0cd2f0 con 0x7f0714106480 2024-03-24T10:37:28.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.921+0000 7f0713fff700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f07000a21d0 0x7f07000a4690 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f06fc005950 tx=0x7f06fc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:28.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:28.923+0000 7f07117fa700 1 -- 172.21.15.23:0/1583946798 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f070c0967f0 con 0x7f0714106480 2024-03-24T10:37:29.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:29.125+0000 7f070affd700 1 -- 172.21.15.23:0/1583946798 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f06f8005b60 con 0x7f0714106480 2024-03-24T10:37:29.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:29.125+0000 7f07117fa700 1 -- 172.21.15.23:0/1583946798 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f070c01a310 con 0x7f0714106480 2024-03-24T10:37:29.126 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-24T10:37:29.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:29.127+0000 7f070affd700 1 -- 172.21.15.23:0/1583946798 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f07000a21d0 msgr2=0x7f07000a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:29.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:29.127+0000 7f070affd700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f07000a21d0 0x7f07000a4690 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f06fc005950 tx=0x7f06fc0058e0 comp rx=0 tx=0).stop 2024-03-24T10:37:29.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:29.127+0000 7f070affd700 1 -- 172.21.15.23:0/1583946798 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0714106480 msgr2=0x7f07140fd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:29.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:29.127+0000 7f070affd700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0714106480 0x7f07140fd660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f070c00ada0 tx=0x7f070c00c330 comp rx=0 tx=0).stop 2024-03-24T10:37:29.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:29.128+0000 7f070affd700 1 -- 172.21.15.23:0/1583946798 shutdown_connections 2024-03-24T10:37:29.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:29.128+0000 7f070affd700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f07000a21d0 0x7f07000a4690 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:29.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:29.128+0000 7f070affd700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0714106480 0x7f07140fd660 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:29.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:29.128+0000 7f070affd700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0714101b60 0x7f07140fcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:29.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:29.128+0000 7f070affd700 1 --2- 172.21.15.23:0/1583946798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0714102510 0x7f07140fd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:29.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:29.128+0000 7f070affd700 1 -- 172.21.15.23:0/1583946798 >> 172.21.15.23:0/1583946798 conn(0x7f0714077700 msgr2=0x7f0714104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:29.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:29.128+0000 7f070affd700 1 -- 172.21.15.23:0/1583946798 shutdown_connections 2024-03-24T10:37:29.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:29.128+0000 7f070affd700 1 -- 172.21.15.23:0/1583946798 wait complete. 2024-03-24T10:37:29.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:29 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1583946798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:37:29.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:29 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1583946798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:37:29.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:29 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1583946798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:37:29.789 INFO:tasks.cephadm:Adding mds.a on smithi023 2024-03-24T10:37:29.789 INFO:tasks.cephadm:Adding mds.d on smithi023 2024-03-24T10:37:29.790 INFO:tasks.cephadm:Adding mds.g on smithi023 2024-03-24T10:37:29.790 INFO:tasks.cephadm:Adding mds.j on smithi023 2024-03-24T10:37:29.790 INFO:tasks.cephadm:Adding mds.b on smithi116 2024-03-24T10:37:29.790 INFO:tasks.cephadm:Adding mds.e on smithi116 2024-03-24T10:37:29.790 INFO:tasks.cephadm:Adding mds.h on smithi116 2024-03-24T10:37:29.790 INFO:tasks.cephadm:Adding mds.k on smithi116 2024-03-24T10:37:29.790 INFO:tasks.cephadm:Adding mds.c on smithi117 2024-03-24T10:37:29.790 INFO:tasks.cephadm:Adding mds.f on smithi117 2024-03-24T10:37:29.790 INFO:tasks.cephadm:Adding mds.i on smithi117 2024-03-24T10:37:29.790 INFO:tasks.cephadm:Adding mds.l on smithi117 2024-03-24T10:37:29.790 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph orch apply mds all '12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi117=c;smithi117=f;smithi117=i;smithi117=l' 2024-03-24T10:37:30.038 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:37:30.445 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:30 smithi117 ceph-mon[36368]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:30.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:30 smithi023 ceph-mon[32466]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:30.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:30 smithi116 ceph-mon[36248]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:31.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.704+0000 7f513d867700 1 -- 172.21.15.117:0/1182617301 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138107f80 msgr2=0x7f513810a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:31.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.704+0000 7f513d867700 1 --2- 172.21.15.117:0/1182617301 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138107f80 0x7f513810a390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f513000b0a0 tx=0x7f513000b3b0 comp rx=0 tx=0).stop 2024-03-24T10:37:31.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.704+0000 7f513d867700 1 -- 172.21.15.117:0/1182617301 shutdown_connections 2024-03-24T10:37:31.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.704+0000 7f513d867700 1 --2- 172.21.15.117:0/1182617301 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138107f80 0x7f513810a390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:31.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.704+0000 7f513d867700 1 --2- 172.21.15.117:0/1182617301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5138102d20 0x7f5138105110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:31.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.704+0000 7f513d867700 1 --2- 172.21.15.117:0/1182617301 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138105650 0x7f5138107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:31.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.704+0000 7f513d867700 1 -- 172.21.15.117:0/1182617301 >> 172.21.15.117:0/1182617301 conn(0x7f51380fc6d0 msgr2=0x7f51380feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:31.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.704+0000 7f513d867700 1 -- 172.21.15.117:0/1182617301 shutdown_connections 2024-03-24T10:37:31.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.704+0000 7f513d867700 1 -- 172.21.15.117:0/1182617301 wait complete. 2024-03-24T10:37:31.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f513d867700 1 Processor -- start 2024-03-24T10:37:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f513d867700 1 -- start start 2024-03-24T10:37:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f513d867700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138102d20 0x7f5138102680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f513d867700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5138105650 0x7f5138100cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f513d867700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138107f80 0x7f5138101210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:31.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f513d867700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f5138101990 con 0x7f5138107f80 2024-03-24T10:37:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f513d867700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5138101b00 con 0x7f5138105650 2024-03-24T10:37:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f513d867700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f51381a23a0 con 0x7f5138102d20 2024-03-24T10:37:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f5136ffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138102d20 0x7f5138102680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f5136ffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138102d20 0x7f5138102680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:46294/0 (socket says 172.21.15.117:46294) 2024-03-24T10:37:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f5136ffd700 1 -- 172.21.15.117:0/2575847299 learned_addr learned my addr 172.21.15.117:0/2575847299 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T10:37:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f51367fc700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5138105650 0x7f5138100cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f51377fe700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138107f80 0x7f5138101210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f5136ffd700 1 -- 172.21.15.117:0/2575847299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5138105650 msgr2=0x7f5138100cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f5136ffd700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5138105650 0x7f5138100cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.705+0000 7f5136ffd700 1 -- 172.21.15.117:0/2575847299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138107f80 msgr2=0x7f5138101210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.706+0000 7f5136ffd700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138107f80 0x7f5138101210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.706+0000 7f5136ffd700 1 -- 172.21.15.117:0/2575847299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5130009d00 con 0x7f5138102d20 2024-03-24T10:37:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.706+0000 7f51367fc700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5138105650 0x7f5138100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:37:31.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.706+0000 7f5136ffd700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138102d20 0x7f5138102680 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f512c00b530 tx=0x7f512c00b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:31.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.706+0000 7f513c865700 1 -- 172.21.15.117:0/2575847299 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c011830 con 0x7f5138102d20 2024-03-24T10:37:31.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.706+0000 7f513d867700 1 -- 172.21.15.117:0/2575847299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51381a2680 con 0x7f5138102d20 2024-03-24T10:37:31.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.706+0000 7f513c865700 1 -- 172.21.15.117:0/2575847299 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f512c011e70 con 0x7f5138102d20 2024-03-24T10:37:31.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.706+0000 7f513c865700 1 -- 172.21.15.117:0/2575847299 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c00f2e0 con 0x7f5138102d20 2024-03-24T10:37:31.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.706+0000 7f513d867700 1 -- 172.21.15.117:0/2575847299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51381a2c50 con 0x7f5138102d20 2024-03-24T10:37:31.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.707+0000 7f513d867700 1 -- 172.21.15.117:0/2575847299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51380fe250 con 0x7f5138102d20 2024-03-24T10:37:31.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.708+0000 7f513c865700 1 -- 172.21.15.117:0/2575847299 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f512c0105f0 con 0x7f5138102d20 2024-03-24T10:37:31.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.708+0000 7f513c865700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f51200a2150 0x7f51200a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:31.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.709+0000 7f513c865700 1 -- 172.21.15.117:0/2575847299 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f512c0cd2f0 con 0x7f5138102d20 2024-03-24T10:37:31.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.709+0000 7f51367fc700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f51200a2150 0x7f51200a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:31.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.709+0000 7f51367fc700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f51200a2150 0x7f51200a4610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5128005950 tx=0x7f5128005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:31.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.713+0000 7f513c865700 1 -- 172.21.15.117:0/2575847299 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f512c096760 con 0x7f5138102d20 2024-03-24T10:37:31.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.918+0000 7f513d867700 1 -- 172.21.15.117:0/2575847299 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi117=c;smithi117=f;smithi117=i;smithi117=l", "target": ["mon-mgr", ""]}) -- 0x7f5138062740 con 0x7f51200a2150 2024-03-24T10:37:31.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.930+0000 7f513c865700 1 -- 172.21.15.117:0/2575847299 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f5138062740 con 0x7f51200a2150 2024-03-24T10:37:31.931 INFO:teuthology.orchestra.run.smithi117.stdout:Scheduled mds.all update... 2024-03-24T10:37:31.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.932+0000 7f513d867700 1 -- 172.21.15.117:0/2575847299 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f51200a2150 msgr2=0x7f51200a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:31.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.932+0000 7f513d867700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f51200a2150 0x7f51200a4610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5128005950 tx=0x7f5128005f50 comp rx=0 tx=0).stop 2024-03-24T10:37:31.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.932+0000 7f513d867700 1 -- 172.21.15.117:0/2575847299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138102d20 msgr2=0x7f5138102680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:31.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.932+0000 7f513d867700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138102d20 0x7f5138102680 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f512c00b530 tx=0x7f512c00b8f0 comp rx=0 tx=0).stop 2024-03-24T10:37:31.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.932+0000 7f513d867700 1 -- 172.21.15.117:0/2575847299 shutdown_connections 2024-03-24T10:37:31.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.932+0000 7f513d867700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f51200a2150 0x7f51200a4610 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:31.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.932+0000 7f513d867700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138102d20 0x7f5138102680 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:31.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.932+0000 7f513d867700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138107f80 0x7f5138101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:31.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.932+0000 7f513d867700 1 --2- 172.21.15.117:0/2575847299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5138105650 0x7f5138100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:31.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.932+0000 7f513d867700 1 -- 172.21.15.117:0/2575847299 >> 172.21.15.117:0/2575847299 conn(0x7f51380fc6d0 msgr2=0x7f5138103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:31.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.933+0000 7f513d867700 1 -- 172.21.15.117:0/2575847299 shutdown_connections 2024-03-24T10:37:31.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T10:37:31.933+0000 7f513d867700 1 -- 172.21.15.117:0/2575847299 wait complete. 2024-03-24T10:37:32.532 DEBUG:teuthology.orchestra.run.smithi023:mds.a> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.a.service 2024-03-24T10:37:32.535 DEBUG:teuthology.orchestra.run.smithi023:mds.d> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.d.service 2024-03-24T10:37:32.578 DEBUG:teuthology.orchestra.run.smithi023:mds.g> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.g.service 2024-03-24T10:37:32.623 DEBUG:teuthology.orchestra.run.smithi023:mds.j> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.j.service 2024-03-24T10:37:32.644 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:32 smithi023 ceph-mon[32466]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:32.644 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:32.644 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:32 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:32.645 DEBUG:teuthology.orchestra.run.smithi116:mds.b> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.b.service 2024-03-24T10:37:32.648 DEBUG:teuthology.orchestra.run.smithi116:mds.e> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.e.service 2024-03-24T10:37:32.670 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:32 smithi116 ceph-mon[36248]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:32.671 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:32.671 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:32 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:32.672 DEBUG:teuthology.orchestra.run.smithi116:mds.h> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.h.service 2024-03-24T10:37:32.716 DEBUG:teuthology.orchestra.run.smithi116:mds.k> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.k.service 2024-03-24T10:37:32.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:32 smithi117 ceph-mon[36368]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:32.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:32.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:32 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:37:32.759 DEBUG:teuthology.orchestra.run.smithi117:mds.c> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.c.service 2024-03-24T10:37:32.762 DEBUG:teuthology.orchestra.run.smithi117:mds.f> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.f.service 2024-03-24T10:37:32.805 DEBUG:teuthology.orchestra.run.smithi117:mds.i> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.i.service 2024-03-24T10:37:32.848 DEBUG:teuthology.orchestra.run.smithi117:mds.l> sudo journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.l.service 2024-03-24T10:37:32.891 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-24T10:37:32.891 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:37:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.002+0000 7f3f0f16b640 1 -- 172.21.15.23:0/1429488283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f08100630 msgr2=0x7f3f08100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.002+0000 7f3f0f16b640 1 --2- 172.21.15.23:0/1429488283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f08100630 0x7f3f08100a50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009900 tx=0x7f3efc02bc30 comp rx=0 tx=0).stop 2024-03-24T10:37:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.002+0000 7f3f0f16b640 1 -- 172.21.15.23:0/1429488283 shutdown_connections 2024-03-24T10:37:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.002+0000 7f3f0f16b640 1 --2- 172.21.15.23:0/1429488283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f081021e0 0x7f3f08108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.002+0000 7f3f0f16b640 1 --2- 172.21.15.23:0/1429488283 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f08101820 0x7f3f08101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.002+0000 7f3f0f16b640 1 --2- 172.21.15.23:0/1429488283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f08100630 0x7f3f08100a50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.002+0000 7f3f0f16b640 1 -- 172.21.15.23:0/1429488283 >> 172.21.15.23:0/1429488283 conn(0x7f3f080fbba0 msgr2=0x7f3f080fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.002+0000 7f3f0f16b640 1 -- 172.21.15.23:0/1429488283 shutdown_connections 2024-03-24T10:37:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.002+0000 7f3f0f16b640 1 -- 172.21.15.23:0/1429488283 wait complete. 2024-03-24T10:37:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0f16b640 1 Processor -- start 2024-03-24T10:37:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0f16b640 1 -- start start 2024-03-24T10:37:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0f16b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f08100630 0x7f3f08197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0f16b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f08101820 0x7f3f08197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0f16b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f081021e0 0x7f3f0819ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0f16b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0819f370 con 0x7f3f081021e0 2024-03-24T10:37:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0f16b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0819f4e0 con 0x7f3f08101820 2024-03-24T10:37:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0f16b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0819f6a0 con 0x7f3f08100630 2024-03-24T10:37:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0d708640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f081021e0 0x7f3f0819ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0d708640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f081021e0 0x7f3f0819ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41416/0 (socket says 172.21.15.23:41416) 2024-03-24T10:37:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0d708640 1 -- 172.21.15.23:0/823396501 learned_addr learned my addr 172.21.15.23:0/823396501 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0cf07640 1 --2- 172.21.15.23:0/823396501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f08100630 0x7f3f08197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f07fff640 1 --2- 172.21.15.23:0/823396501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f08101820 0x7f3f08197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0d708640 1 -- 172.21.15.23:0/823396501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f08100630 msgr2=0x7f3f08197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0d708640 1 --2- 172.21.15.23:0/823396501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f08100630 0x7f3f08197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0d708640 1 -- 172.21.15.23:0/823396501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f08101820 msgr2=0x7f3f08197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0d708640 1 --2- 172.21.15.23:0/823396501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f08101820 0x7f3f08197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.003+0000 7f3f0d708640 1 -- 172.21.15.23:0/823396501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f00009840 con 0x7f3f081021e0 2024-03-24T10:37:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.004+0000 7f3f0d708640 1 --2- 172.21.15.23:0/823396501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f081021e0 0x7f3f0819ee30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000efd0 tx=0x7f3f0000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.004+0000 7f3f05ffb640 1 -- 172.21.15.23:0/823396501 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f00009b40 con 0x7f3f081021e0 2024-03-24T10:37:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.004+0000 7f3f0f16b640 1 -- 172.21.15.23:0/823396501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3efc009560 con 0x7f3f081021e0 2024-03-24T10:37:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.004+0000 7f3f05ffb640 1 -- 172.21.15.23:0/823396501 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f00003b70 con 0x7f3f081021e0 2024-03-24T10:37:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.004+0000 7f3f05ffb640 1 -- 172.21.15.23:0/823396501 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f000108f0 con 0x7f3f081021e0 2024-03-24T10:37:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.004+0000 7f3f0f16b640 1 -- 172.21.15.23:0/823396501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f0819fcb0 con 0x7f3f081021e0 2024-03-24T10:37:33.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.005+0000 7f3f0f16b640 1 -- 172.21.15.23:0/823396501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f0810c040 con 0x7f3f081021e0 2024-03-24T10:37:33.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.006+0000 7f3f05ffb640 1 -- 172.21.15.23:0/823396501 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f3f00003ce0 con 0x7f3f081021e0 2024-03-24T10:37:33.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.006+0000 7f3f05ffb640 1 --2- 172.21.15.23:0/823396501 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3ef80a20c0 0x7f3ef80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:33.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.006+0000 7f3f0cf07640 1 --2- 172.21.15.23:0/823396501 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3ef80a20c0 0x7f3ef80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:33.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.006+0000 7f3f05ffb640 1 -- 172.21.15.23:0/823396501 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3f00014070 con 0x7f3f081021e0 2024-03-24T10:37:33.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.007+0000 7f3f0cf07640 1 --2- 172.21.15.23:0/823396501 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3ef80a20c0 0x7f3ef80a4580 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3efc005ab0 tx=0x7f3efc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:33.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.011+0000 7f3f05ffb640 1 -- 172.21.15.23:0/823396501 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3f000972c0 con 0x7f3f081021e0 2024-03-24T10:37:33.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.193+0000 7f3f0f16b640 1 -- 172.21.15.23:0/823396501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f081a00f0 con 0x7f3f081021e0 2024-03-24T10:37:33.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.194+0000 7f3f05ffb640 1 -- 172.21.15.23:0/823396501 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18470 (secure 0 0 0) 0x7f3f0009a980 con 0x7f3f081021e0 2024-03-24T10:37:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:33.195 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":81,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:37:22.316371+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:37:33.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.197+0000 7f3f0f16b640 1 -- 172.21.15.23:0/823396501 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3ef80a20c0 msgr2=0x7f3ef80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.197+0000 7f3f0f16b640 1 --2- 172.21.15.23:0/823396501 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3ef80a20c0 0x7f3ef80a4580 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3efc005ab0 tx=0x7f3efc005fd0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.197+0000 7f3f0f16b640 1 -- 172.21.15.23:0/823396501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f081021e0 msgr2=0x7f3f0819ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.197+0000 7f3f0f16b640 1 --2- 172.21.15.23:0/823396501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f081021e0 0x7f3f0819ee30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000efd0 tx=0x7f3f0000c6a0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.197+0000 7f3f0f16b640 1 -- 172.21.15.23:0/823396501 shutdown_connections 2024-03-24T10:37:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.197+0000 7f3f0f16b640 1 --2- 172.21.15.23:0/823396501 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3ef80a20c0 0x7f3ef80a4580 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.197+0000 7f3f0f16b640 1 --2- 172.21.15.23:0/823396501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f081021e0 0x7f3f0819ee30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.197+0000 7f3f0f16b640 1 --2- 172.21.15.23:0/823396501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f08101820 0x7f3f08197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.197+0000 7f3f0f16b640 1 --2- 172.21.15.23:0/823396501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f08100630 0x7f3f08197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.197+0000 7f3f0f16b640 1 -- 172.21.15.23:0/823396501 >> 172.21.15.23:0/823396501 conn(0x7f3f080fbba0 msgr2=0x7f3f08068960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.197+0000 7f3f0f16b640 1 -- 172.21.15.23:0/823396501 shutdown_connections 2024-03-24T10:37:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.197+0000 7f3f0f16b640 1 -- 172.21.15.23:0/823396501 wait complete. 2024-03-24T10:37:33.208 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:39.056452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:37:33.209 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:37:33.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2521505667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e41021c0 msgr2=0x7ff6e4108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 --2- 172.21.15.23:0/2521505667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e41021c0 0x7ff6e4108a50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc009ca0 tx=0x7ff6dc02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2521505667 shutdown_connections 2024-03-24T10:37:33.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 --2- 172.21.15.23:0/2521505667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e41021c0 0x7ff6e4108a50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 --2- 172.21.15.23:0/2521505667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6e4101800 0x7ff6e4101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 --2- 172.21.15.23:0/2521505667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e4100610 0x7ff6e4100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2521505667 >> 172.21.15.23:0/2521505667 conn(0x7ff6e40fbba0 msgr2=0x7ff6e40fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:33.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2521505667 shutdown_connections 2024-03-24T10:37:33.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2521505667 wait complete. 2024-03-24T10:37:33.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 Processor -- start 2024-03-24T10:37:33.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 -- start start 2024-03-24T10:37:33.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e4100610 0x7ff6e406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:33.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6e4101800 0x7ff6e419c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:33.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e41021c0 0x7ff6e419eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:33.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e419f270 con 0x7ff6e41021c0 2024-03-24T10:37:33.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e419f3e0 con 0x7ff6e4100610 2024-03-24T10:37:33.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.314+0000 7ff6ebfa0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e419f5a0 con 0x7ff6e4101800 2024-03-24T10:37:33.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6ea53d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e41021c0 0x7ff6e419eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:33.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6ea53d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e41021c0 0x7ff6e419eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41438/0 (socket says 172.21.15.23:41438) 2024-03-24T10:37:33.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6ea53d640 1 -- 172.21.15.23:0/2902673442 learned_addr learned my addr 172.21.15.23:0/2902673442 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:33.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6e953b640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6e4101800 0x7ff6e419c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:33.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6e9d3c640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e4100610 0x7ff6e406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:33.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6ea53d640 1 -- 172.21.15.23:0/2902673442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6e4101800 msgr2=0x7ff6e419c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6ea53d640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6e4101800 0x7ff6e419c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6ea53d640 1 -- 172.21.15.23:0/2902673442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e4100610 msgr2=0x7ff6e406a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6ea53d640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e4100610 0x7ff6e406a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6ea53d640 1 -- 172.21.15.23:0/2902673442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e0009490 con 0x7ff6e41021c0 2024-03-24T10:37:33.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6e953b640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6e4101800 0x7ff6e419c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:37:33.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6ea53d640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e41021c0 0x7ff6e419eae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc037040 tx=0x7ff6dc009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:33.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6daffd640 1 -- 172.21.15.23:0/2902673442 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0329b0 con 0x7ff6e41021c0 2024-03-24T10:37:33.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6daffd640 1 -- 172.21.15.23:0/2902673442 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6dc003e50 con 0x7ff6e41021c0 2024-03-24T10:37:33.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2902673442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6dc009900 con 0x7ff6e41021c0 2024-03-24T10:37:33.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6daffd640 1 -- 172.21.15.23:0/2902673442 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0048a0 con 0x7ff6e41021c0 2024-03-24T10:37:33.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.315+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2902673442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6e419fb90 con 0x7ff6e41021c0 2024-03-24T10:37:33.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.316+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2902673442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6e41078d0 con 0x7ff6e41021c0 2024-03-24T10:37:33.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.317+0000 7ff6daffd640 1 -- 172.21.15.23:0/2902673442 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ff6dc00a040 con 0x7ff6e41021c0 2024-03-24T10:37:33.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.318+0000 7ff6daffd640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff6d00a2140 0x7ff6d00a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:33.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.318+0000 7ff6e9d3c640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff6d00a2140 0x7ff6d00a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:33.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.318+0000 7ff6daffd640 1 -- 172.21.15.23:0/2902673442 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff6dc0f0a30 con 0x7ff6e41021c0 2024-03-24T10:37:33.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.318+0000 7ff6e9d3c640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff6d00a2140 0x7ff6d00a4600 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4101660 tx=0x7ff6e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:33.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.321+0000 7ff6daffd640 1 -- 172.21.15.23:0/2902673442 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff6dc0b9f30 con 0x7ff6e41021c0 2024-03-24T10:37:33.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.503+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2902673442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff6e410c270 con 0x7ff6e41021c0 2024-03-24T10:37:33.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.504+0000 7ff6daffd640 1 -- 172.21.15.23:0/2902673442 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7ff6dc034e60 con 0x7ff6e41021c0 2024-03-24T10:37:33.504 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-24T10:37:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.506+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2902673442 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff6d00a2140 msgr2=0x7ff6d00a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.506+0000 7ff6ebfa0640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff6d00a2140 0x7ff6d00a4600 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4101660 tx=0x7ff6e0005f90 comp rx=0 tx=0).stop 2024-03-24T10:37:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.506+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2902673442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e41021c0 msgr2=0x7ff6e419eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.506+0000 7ff6ebfa0640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e41021c0 0x7ff6e419eae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc037040 tx=0x7ff6dc009360 comp rx=0 tx=0).stop 2024-03-24T10:37:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.506+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2902673442 shutdown_connections 2024-03-24T10:37:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.506+0000 7ff6ebfa0640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff6d00a2140 0x7ff6d00a4600 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.506+0000 7ff6ebfa0640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e41021c0 0x7ff6e419eae0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.506+0000 7ff6ebfa0640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6e4101800 0x7ff6e419c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.506+0000 7ff6ebfa0640 1 --2- 172.21.15.23:0/2902673442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e4100610 0x7ff6e406a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.506+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2902673442 >> 172.21.15.23:0/2902673442 conn(0x7ff6e40fbba0 msgr2=0x7ff6e4104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.506+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2902673442 shutdown_connections 2024-03-24T10:37:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.506+0000 7ff6ebfa0640 1 -- 172.21.15.23:0/2902673442 wait complete. 2024-03-24T10:37:33.516 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:33 smithi023 ceph-mon[32466]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi117=c;smithi117=f;smithi117=i;smithi117=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:37:33.516 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:33 smithi023 ceph-mon[32466]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi117=c;smithi117=f;smithi117=i;smithi117=l;count:12 2024-03-24T10:37:33.516 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:33 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:33.516 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:33 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:37:33.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:33 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:33.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:33 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:33.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:33 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:33.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:33 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:33.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:33 smithi023 ceph-mon[32466]: Deploying daemon mds.l on smithi117 2024-03-24T10:37:33.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:33 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/823396501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:37:33.518 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:37:33.576 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:33 smithi117 ceph-mon[36368]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi117=c;smithi117=f;smithi117=i;smithi117=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:37:33.576 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:33 smithi117 ceph-mon[36368]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi117=c;smithi117=f;smithi117=i;smithi117=l;count:12 2024-03-24T10:37:33.576 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:33 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:33.576 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:33 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:37:33.576 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:33 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:33.576 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:33 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:33.576 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:33 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:33.576 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:33 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:33.576 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:33 smithi117 ceph-mon[36368]: Deploying daemon mds.l on smithi117 2024-03-24T10:37:33.576 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:33 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/823396501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:37:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.619+0000 7f1b89f02640 1 -- 172.21.15.23:0/3080490084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b84107410 msgr2=0x7f1b84109860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.619+0000 7f1b89f02640 1 --2- 172.21.15.23:0/3080490084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b84107410 0x7f1b84109860 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c009ca0 tx=0x7f1b7c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.619+0000 7f1b89f02640 1 -- 172.21.15.23:0/3080490084 shutdown_connections 2024-03-24T10:37:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.619+0000 7f1b89f02640 1 --2- 172.21.15.23:0/3080490084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b84107410 0x7f1b84109860 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.619+0000 7f1b89f02640 1 --2- 172.21.15.23:0/3080490084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b84104ae0 0x7f1b84106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.619+0000 7f1b89f02640 1 --2- 172.21.15.23:0/3080490084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b841021b0 0x7f1b841045a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.619+0000 7f1b89f02640 1 -- 172.21.15.23:0/3080490084 >> 172.21.15.23:0/3080490084 conn(0x7f1b840fbbc0 msgr2=0x7f1b840fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.619+0000 7f1b89f02640 1 -- 172.21.15.23:0/3080490084 shutdown_connections 2024-03-24T10:37:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.619+0000 7f1b89f02640 1 -- 172.21.15.23:0/3080490084 wait complete. 2024-03-24T10:37:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b89f02640 1 Processor -- start 2024-03-24T10:37:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b89f02640 1 -- start start 2024-03-24T10:37:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b89f02640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b841021b0 0x7f1b841977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b89f02640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b84104ae0 0x7f1b84197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b89f02640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b84107410 0x7f1b8419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b89f02640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8419f270 con 0x7f1b84104ae0 2024-03-24T10:37:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b89f02640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8419f3e0 con 0x7f1b841021b0 2024-03-24T10:37:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b89f02640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8419f5a0 con 0x7f1b84107410 2024-03-24T10:37:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b82ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b84104ae0 0x7f1b84197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b82ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b84104ae0 0x7f1b84197d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41454/0 (socket says 172.21.15.23:41454) 2024-03-24T10:37:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b82ffd640 1 -- 172.21.15.23:0/274376321 learned_addr learned my addr 172.21.15.23:0/274376321 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b82ffd640 1 -- 172.21.15.23:0/274376321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b84107410 msgr2=0x7f1b8419ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b82ffd640 1 --2- 172.21.15.23:0/274376321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b84107410 0x7f1b8419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b837fe640 1 --2- 172.21.15.23:0/274376321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b841021b0 0x7f1b841977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b82ffd640 1 -- 172.21.15.23:0/274376321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b841021b0 msgr2=0x7f1b841977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b82ffd640 1 --2- 172.21.15.23:0/274376321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b841021b0 0x7f1b841977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.620+0000 7f1b82ffd640 1 -- 172.21.15.23:0/274376321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7c009900 con 0x7f1b84104ae0 2024-03-24T10:37:33.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.621+0000 7f1b82ffd640 1 --2- 172.21.15.23:0/274376321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b84104ae0 0x7f1b84197d10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800e9c0 tx=0x7f1b7800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:33.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.621+0000 7f1b80ff9640 1 -- 172.21.15.23:0/274376321 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b78009700 con 0x7f1b84104ae0 2024-03-24T10:37:33.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.621+0000 7f1b80ff9640 1 -- 172.21.15.23:0/274376321 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b7800cd80 con 0x7f1b84104ae0 2024-03-24T10:37:33.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.621+0000 7f1b80ff9640 1 -- 172.21.15.23:0/274376321 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b78010430 con 0x7f1b84104ae0 2024-03-24T10:37:33.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.621+0000 7f1b89f02640 1 -- 172.21.15.23:0/274376321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b8419f880 con 0x7f1b84104ae0 2024-03-24T10:37:33.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.621+0000 7f1b89f02640 1 -- 172.21.15.23:0/274376321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b8419fd40 con 0x7f1b84104ae0 2024-03-24T10:37:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.622+0000 7f1b89f02640 1 -- 172.21.15.23:0/274376321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b840fd6b0 con 0x7f1b84104ae0 2024-03-24T10:37:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.623+0000 7f1b80ff9640 1 -- 172.21.15.23:0/274376321 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f1b78005000 con 0x7f1b84104ae0 2024-03-24T10:37:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.623+0000 7f1b80ff9640 1 --2- 172.21.15.23:0/274376321 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f1b6c0a20c0 0x7f1b6c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.623+0000 7f1b80ff9640 1 -- 172.21.15.23:0/274376321 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1b78014070 con 0x7f1b84104ae0 2024-03-24T10:37:33.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.624+0000 7f1b837fe640 1 --2- 172.21.15.23:0/274376321 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f1b6c0a20c0 0x7f1b6c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:33.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.624+0000 7f1b837fe640 1 --2- 172.21.15.23:0/274376321 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f1b6c0a20c0 0x7f1b6c0a4580 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1b74005fd0 tx=0x7f1b74005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:33.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.627+0000 7f1b80ff9640 1 -- 172.21.15.23:0/274376321 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1b78097380 con 0x7f1b84104ae0 2024-03-24T10:37:33.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:33 smithi116 ceph-mon[36248]: from='client.24409 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi117=c;smithi117=f;smithi117=i;smithi117=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:37:33.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:33 smithi116 ceph-mon[36248]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi117=c;smithi117=f;smithi117=i;smithi117=l;count:12 2024-03-24T10:37:33.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:33 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:33.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:33 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:37:33.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:33 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:33.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:33 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:33.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:33 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:33.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:33 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:33.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:33 smithi116 ceph-mon[36248]: Deploying daemon mds.l on smithi117 2024-03-24T10:37:33.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:33 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/823396501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:37:33.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.850+0000 7f1b89f02640 1 -- 172.21.15.23:0/274376321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b8410d120 con 0x7f1b84104ae0 2024-03-24T10:37:33.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.851+0000 7f1b80ff9640 1 -- 172.21.15.23:0/274376321 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f1b7809aa40 con 0x7f1b84104ae0 2024-03-24T10:37:33.852 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:33.852 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-24T10:37:33.852 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 1 2024-03-24T10:37:33.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.853+0000 7f1b89f02640 1 -- 172.21.15.23:0/274376321 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f1b6c0a20c0 msgr2=0x7f1b6c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.853+0000 7f1b89f02640 1 --2- 172.21.15.23:0/274376321 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f1b6c0a20c0 0x7f1b6c0a4580 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1b74005fd0 tx=0x7f1b74005c70 comp rx=0 tx=0).stop 2024-03-24T10:37:33.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.853+0000 7f1b89f02640 1 -- 172.21.15.23:0/274376321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b84104ae0 msgr2=0x7f1b84197d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:33.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.853+0000 7f1b89f02640 1 --2- 172.21.15.23:0/274376321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b84104ae0 0x7f1b84197d10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800e9c0 tx=0x7f1b7800ee80 comp rx=0 tx=0).stop 2024-03-24T10:37:33.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.853+0000 7f1b89f02640 1 -- 172.21.15.23:0/274376321 shutdown_connections 2024-03-24T10:37:33.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.853+0000 7f1b89f02640 1 --2- 172.21.15.23:0/274376321 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f1b6c0a20c0 0x7f1b6c0a4580 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.853+0000 7f1b89f02640 1 --2- 172.21.15.23:0/274376321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b84107410 0x7f1b8419ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.853+0000 7f1b89f02640 1 --2- 172.21.15.23:0/274376321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b84104ae0 0x7f1b84197d10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.853+0000 7f1b89f02640 1 --2- 172.21.15.23:0/274376321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b841021b0 0x7f1b841977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:33.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.853+0000 7f1b89f02640 1 -- 172.21.15.23:0/274376321 >> 172.21.15.23:0/274376321 conn(0x7f1b840fbbc0 msgr2=0x7f1b84102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:33.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.853+0000 7f1b89f02640 1 -- 172.21.15.23:0/274376321 shutdown_connections 2024-03-24T10:37:33.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:33.853+0000 7f1b89f02640 1 -- 172.21.15.23:0/274376321 wait complete. 2024-03-24T10:37:34.478 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:34 smithi117 ceph-mon[36368]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:34.478 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:34 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2902673442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:37:34.478 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:34 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/274376321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:34.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:34 smithi023 ceph-mon[32466]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:34.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:34 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2902673442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:37:34.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:34 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/274376321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:34.753 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:37:34 smithi117 systemd[1]: Starting Ceph mds.l for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:37:34.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:34 smithi116 ceph-mon[36248]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:34.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:34 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2902673442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:37:34.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:34 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/274376321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:35.158 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:37:34 smithi117 podman[66481]: 2024-03-24 10:37:34.872580381 +0000 UTC m=+0.112141623 container create f2de3d30da4abe3e1d0d157068914c39a1cb860389b661bc9cfacef1c733c8c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-l, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T10:37:35.158 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:37:34 smithi117 podman[66481]: 2024-03-24 10:37:34.778092133 +0000 UTC m=+0.017653374 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:37:35.158 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:37:35 smithi117 podman[66481]: 2024-03-24 10:37:35.083593001 +0000 UTC m=+0.323154263 container init f2de3d30da4abe3e1d0d157068914c39a1cb860389b661bc9cfacef1c733c8c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-l, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-24T10:37:35.159 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:37:35 smithi117 podman[66481]: 2024-03-24 10:37:35.086241769 +0000 UTC m=+0.325803007 container start f2de3d30da4abe3e1d0d157068914c39a1cb860389b661bc9cfacef1c733c8c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-l, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:37:35.159 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:37:35 smithi117 ceph-mds[66501]: -- 172.21.15.117:0/2115588941 <== mon.1 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eb8952c1a0 con 0x55eb89502800 2024-03-24T10:37:35.159 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:37:35 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-l[66496]: starting mds.l at 2024-03-24T10:37:35.503 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:37:35 smithi117 bash[66481]: f2de3d30da4abe3e1d0d157068914c39a1cb860389b661bc9cfacef1c733c8c9 2024-03-24T10:37:35.503 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:37:35 smithi117 systemd[1]: Started Ceph mds.l for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:37:35.865 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:37:35.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.970+0000 7fc24f60e640 1 -- 172.21.15.23:0/2340495957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc24810dd00 msgr2=0x7fc2481100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:35.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.970+0000 7fc24f60e640 1 --2- 172.21.15.23:0/2340495957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc24810dd00 0x7fc2481100f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc240009cd0 tx=0x7fc24002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:37:35.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.970+0000 7fc24f60e640 1 -- 172.21.15.23:0/2340495957 shutdown_connections 2024-03-24T10:37:35.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.970+0000 7fc24f60e640 1 --2- 172.21.15.23:0/2340495957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc24810dd00 0x7fc2481100f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:35.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.970+0000 7fc24f60e640 1 --2- 172.21.15.23:0/2340495957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc248074170 0x7fc2480745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:35.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.970+0000 7fc24f60e640 1 --2- 172.21.15.23:0/2340495957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248075750 0x7fc248073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:35.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.970+0000 7fc24f60e640 1 -- 172.21.15.23:0/2340495957 >> 172.21.15.23:0/2340495957 conn(0x7fc2480fccc0 msgr2=0x7fc2480ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:35.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.970+0000 7fc24f60e640 1 -- 172.21.15.23:0/2340495957 shutdown_connections 2024-03-24T10:37:35.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.970+0000 7fc24f60e640 1 -- 172.21.15.23:0/2340495957 wait complete. 2024-03-24T10:37:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24f60e640 1 Processor -- start 2024-03-24T10:37:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24f60e640 1 -- start start 2024-03-24T10:37:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24f60e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc248074170 0x7fc24819ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24f60e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248075750 0x7fc24819e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24f60e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24810dd00 0x7fc2481a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24f60e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc2481a5890 con 0x7fc248074170 2024-03-24T10:37:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24f60e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc2481a5a00 con 0x7fc24810dd00 2024-03-24T10:37:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24f60e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc2481a5bc0 con 0x7fc248075750 2024-03-24T10:37:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24d3aa640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc248074170 0x7fc24819ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24d3aa640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc248074170 0x7fc24819ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41478/0 (socket says 172.21.15.23:41478) 2024-03-24T10:37:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24d3aa640 1 -- 172.21.15.23:0/373548321 learned_addr learned my addr 172.21.15.23:0/373548321 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24dbab640 1 --2- 172.21.15.23:0/373548321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24810dd00 0x7fc2481a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24cba9640 1 --2- 172.21.15.23:0/373548321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248075750 0x7fc24819e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24d3aa640 1 -- 172.21.15.23:0/373548321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248075750 msgr2=0x7fc24819e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24d3aa640 1 --2- 172.21.15.23:0/373548321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248075750 0x7fc24819e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24d3aa640 1 -- 172.21.15.23:0/373548321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24810dd00 msgr2=0x7fc2481a5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24d3aa640 1 --2- 172.21.15.23:0/373548321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24810dd00 0x7fc2481a5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.971+0000 7fc24d3aa640 1 -- 172.21.15.23:0/373548321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc240009900 con 0x7fc248074170 2024-03-24T10:37:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.972+0000 7fc24d3aa640 1 --2- 172.21.15.23:0/373548321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc248074170 0x7fc24819ddb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc24400e9f0 tx=0x7fc24400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.972+0000 7fc23e7fc640 1 -- 172.21.15.23:0/373548321 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244009720 con 0x7fc248074170 2024-03-24T10:37:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.972+0000 7fc23e7fc640 1 -- 172.21.15.23:0/373548321 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc24400cd80 con 0x7fc248074170 2024-03-24T10:37:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.972+0000 7fc24f60e640 1 -- 172.21.15.23:0/373548321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2481a5ea0 con 0x7fc248074170 2024-03-24T10:37:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.972+0000 7fc23e7fc640 1 -- 172.21.15.23:0/373548321 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244010690 con 0x7fc248074170 2024-03-24T10:37:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.972+0000 7fc24f60e640 1 -- 172.21.15.23:0/373548321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2481a62c0 con 0x7fc248074170 2024-03-24T10:37:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.973+0000 7fc24f60e640 1 -- 172.21.15.23:0/373548321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc248103dc0 con 0x7fc248074170 2024-03-24T10:37:35.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.975+0000 7fc23e7fc640 1 -- 172.21.15.23:0/373548321 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fc2440046c0 con 0x7fc248074170 2024-03-24T10:37:35.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.975+0000 7fc23e7fc640 1 --2- 172.21.15.23:0/373548321 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc2340a2140 0x7fc2340a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:35.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.975+0000 7fc24cba9640 1 --2- 172.21.15.23:0/373548321 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc2340a2140 0x7fc2340a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:35.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.975+0000 7fc23e7fc640 1 -- 172.21.15.23:0/373548321 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc244014070 con 0x7fc248074170 2024-03-24T10:37:35.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.975+0000 7fc24cba9640 1 --2- 172.21.15.23:0/373548321 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc2340a2140 0x7fc2340a4600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc23800bbe0 tx=0x7fc23800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:35.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:35.979+0000 7fc23e7fc640 1 -- 172.21.15.23:0/373548321 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc2440970e0 con 0x7fc248074170 2024-03-24T10:37:36.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:36 smithi023 ceph-mon[32466]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:36.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:36 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.117:6832/134162564,v1:172.21.15.117:6833/134162564] up:boot 2024-03-24T10:37:36.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:36 smithi023 ceph-mon[32466]: fsmap 1 up:standby 2024-03-24T10:37:36.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T10:37:36.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:36.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:36.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:36.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:36.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:36.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:36 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:36.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:36 smithi023 ceph-mon[32466]: Deploying daemon mds.k on smithi116 2024-03-24T10:37:36.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:36.199+0000 7fc24f60e640 1 -- 172.21.15.23:0/373548321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc248113990 con 0x7fc248074170 2024-03-24T10:37:36.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:36.199+0000 7fc23e7fc640 1 -- 172.21.15.23:0/373548321 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7fc244099010 con 0x7fc248074170 2024-03-24T10:37:36.200 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:36.200 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-24T10:37:36.200 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 2 2024-03-24T10:37:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:36.201+0000 7fc24f60e640 1 -- 172.21.15.23:0/373548321 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc2340a2140 msgr2=0x7fc2340a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:36.201+0000 7fc24f60e640 1 --2- 172.21.15.23:0/373548321 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc2340a2140 0x7fc2340a4600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc23800bbe0 tx=0x7fc23800b390 comp rx=0 tx=0).stop 2024-03-24T10:37:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:36.201+0000 7fc24f60e640 1 -- 172.21.15.23:0/373548321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc248074170 msgr2=0x7fc24819ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:36.201+0000 7fc24f60e640 1 --2- 172.21.15.23:0/373548321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc248074170 0x7fc24819ddb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc24400e9f0 tx=0x7fc24400edf0 comp rx=0 tx=0).stop 2024-03-24T10:37:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:36.201+0000 7fc24f60e640 1 -- 172.21.15.23:0/373548321 shutdown_connections 2024-03-24T10:37:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:36.201+0000 7fc24f60e640 1 --2- 172.21.15.23:0/373548321 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc2340a2140 0x7fc2340a4600 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:36.201+0000 7fc24f60e640 1 --2- 172.21.15.23:0/373548321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24810dd00 0x7fc2481a5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:36.201+0000 7fc24f60e640 1 --2- 172.21.15.23:0/373548321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248075750 0x7fc24819e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:36.201+0000 7fc24f60e640 1 --2- 172.21.15.23:0/373548321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc248074170 0x7fc24819ddb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:36.201+0000 7fc24f60e640 1 -- 172.21.15.23:0/373548321 >> 172.21.15.23:0/373548321 conn(0x7fc2480fccc0 msgr2=0x7fc248100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:36.201+0000 7fc24f60e640 1 -- 172.21.15.23:0/373548321 shutdown_connections 2024-03-24T10:37:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:36.201+0000 7fc24f60e640 1 -- 172.21.15.23:0/373548321 wait complete. 2024-03-24T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:36 smithi117 ceph-mon[36368]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:36 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.117:6832/134162564,v1:172.21.15.117:6833/134162564] up:boot 2024-03-24T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:36 smithi117 ceph-mon[36368]: fsmap 1 up:standby 2024-03-24T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:36 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:36 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:36 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:36 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:36 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:36 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:36 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:36 smithi117 ceph-mon[36368]: Deploying daemon mds.k on smithi116 2024-03-24T10:37:36.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:36 smithi116 ceph-mon[36248]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:36.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:36 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.117:6832/134162564,v1:172.21.15.117:6833/134162564] up:boot 2024-03-24T10:37:36.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:36 smithi116 ceph-mon[36248]: fsmap 1 up:standby 2024-03-24T10:37:36.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:36 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T10:37:36.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:36 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:36.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:36 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:36.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:36 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:36.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:36 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:36.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:36 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:36.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:36 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:36.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:36 smithi116 ceph-mon[36248]: Deploying daemon mds.k on smithi116 2024-03-24T10:37:37.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:37 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/373548321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:37.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:37 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/373548321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:37.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:37 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/373548321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:37.518 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:37:37 smithi116 systemd[1]: Starting Ceph mds.k for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:37:37.773 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:37:37 smithi116 podman[65679]: 2024-03-24 10:37:37.422444666 +0000 UTC m=+0.018070353 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:37:37.773 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:37:37 smithi116 podman[65679]: 2024-03-24 10:37:37.54495667 +0000 UTC m=+0.140582348 container create 44f26d052fc5d85f41c0a73f9b89e63b182eca2c0452bcc6368439b3b9e54510 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-k, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:37:37.773 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:37:37 smithi116 podman[65679]: 2024-03-24 10:37:37.74723404 +0000 UTC m=+0.342859719 container init 44f26d052fc5d85f41c0a73f9b89e63b182eca2c0452bcc6368439b3b9e54510 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-k, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-24T10:37:37.773 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:37:37 smithi116 podman[65679]: 2024-03-24 10:37:37.749960561 +0000 UTC m=+0.345586242 container start 44f26d052fc5d85f41c0a73f9b89e63b182eca2c0452bcc6368439b3b9e54510 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T10:37:38.139 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:37:37 smithi116 ceph-mds[65698]: -- 172.21.15.116:0/88182191 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563e70cd81a0 con 0x563e70cae800 2024-03-24T10:37:38.139 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:37:37 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-k[65694]: starting mds.k at 2024-03-24T10:37:38.139 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:37:37 smithi116 bash[65679]: 44f26d052fc5d85f41c0a73f9b89e63b182eca2c0452bcc6368439b3b9e54510 2024-03-24T10:37:38.139 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:37:37 smithi116 systemd[1]: Started Ceph mds.k for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:37:38.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:37:38.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.316+0000 7fe77729d640 1 -- 172.21.15.23:0/1053477976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe770104ae0 msgr2=0x7fe770106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:38.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.316+0000 7fe77729d640 1 --2- 172.21.15.23:0/1053477976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe770104ae0 0x7fe770106ed0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe760009900 tx=0x7fe76002bc30 comp rx=0 tx=0).stop 2024-03-24T10:37:38.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.316+0000 7fe77729d640 1 -- 172.21.15.23:0/1053477976 shutdown_connections 2024-03-24T10:37:38.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.316+0000 7fe77729d640 1 --2- 172.21.15.23:0/1053477976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe770107410 0x7fe770109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:38.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.316+0000 7fe77729d640 1 --2- 172.21.15.23:0/1053477976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe770104ae0 0x7fe770106ed0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:38.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.316+0000 7fe77729d640 1 --2- 172.21.15.23:0/1053477976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7701021b0 0x7fe7701045a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:38.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.316+0000 7fe77729d640 1 -- 172.21.15.23:0/1053477976 >> 172.21.15.23:0/1053477976 conn(0x7fe7700fbba0 msgr2=0x7fe7700fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:38.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.316+0000 7fe77729d640 1 -- 172.21.15.23:0/1053477976 shutdown_connections 2024-03-24T10:37:38.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.316+0000 7fe77729d640 1 -- 172.21.15.23:0/1053477976 wait complete. 2024-03-24T10:37:38.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.317+0000 7fe77729d640 1 Processor -- start 2024-03-24T10:37:38.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.317+0000 7fe77729d640 1 -- start start 2024-03-24T10:37:38.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.317+0000 7fe77729d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7701021b0 0x7fe770101b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:38.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.317+0000 7fe77729d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe770104ae0 0x7fe770100160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:38.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.317+0000 7fe77729d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe770107410 0x7fe7701006a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:38.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.317+0000 7fe77729d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe770100e00 con 0x7fe770104ae0 2024-03-24T10:37:38.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.317+0000 7fe77729d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe770100f70 con 0x7fe770107410 2024-03-24T10:37:38.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.317+0000 7fe77729d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe7701a1870 con 0x7fe7701021b0 2024-03-24T10:37:38.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.317+0000 7fe774838640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe770104ae0 0x7fe770100160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:38.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.317+0000 7fe774838640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe770104ae0 0x7fe770100160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40790/0 (socket says 172.21.15.23:40790) 2024-03-24T10:37:38.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.317+0000 7fe774838640 1 -- 172.21.15.23:0/3459942188 learned_addr learned my addr 172.21.15.23:0/3459942188 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:38.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.317+0000 7fe77583a640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe770107410 0x7fe7701006a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:38.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.317+0000 7fe775039640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7701021b0 0x7fe770101b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:38.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.318+0000 7fe774838640 1 -- 172.21.15.23:0/3459942188 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7701021b0 msgr2=0x7fe770101b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:38.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.318+0000 7fe774838640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7701021b0 0x7fe770101b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:38.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.318+0000 7fe774838640 1 -- 172.21.15.23:0/3459942188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe770107410 msgr2=0x7fe7701006a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:38.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.318+0000 7fe774838640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe770107410 0x7fe7701006a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:38.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.318+0000 7fe774838640 1 -- 172.21.15.23:0/3459942188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe760009560 con 0x7fe770104ae0 2024-03-24T10:37:38.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.318+0000 7fe77583a640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe770107410 0x7fe7701006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:37:38.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.318+0000 7fe774838640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe770104ae0 0x7fe770100160 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe760000c00 tx=0x7fe760004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:38.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.318+0000 7fe7667fc640 1 -- 172.21.15.23:0/3459942188 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe760004aa0 con 0x7fe770104ae0 2024-03-24T10:37:38.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.318+0000 7fe77729d640 1 -- 172.21.15.23:0/3459942188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7701a1ac0 con 0x7fe770104ae0 2024-03-24T10:37:38.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.318+0000 7fe7667fc640 1 -- 172.21.15.23:0/3459942188 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7600052d0 con 0x7fe770104ae0 2024-03-24T10:37:38.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.318+0000 7fe7667fc640 1 -- 172.21.15.23:0/3459942188 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe760003e60 con 0x7fe770104ae0 2024-03-24T10:37:38.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.318+0000 7fe77729d640 1 -- 172.21.15.23:0/3459942188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7701a2020 con 0x7fe770104ae0 2024-03-24T10:37:38.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.319+0000 7fe77729d640 1 -- 172.21.15.23:0/3459942188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7700fd690 con 0x7fe770104ae0 2024-03-24T10:37:38.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.321+0000 7fe7667fc640 1 -- 172.21.15.23:0/3459942188 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fe760005440 con 0x7fe770104ae0 2024-03-24T10:37:38.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.321+0000 7fe7667fc640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe75c0a2140 0x7fe75c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:38.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.321+0000 7fe775039640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe75c0a2140 0x7fe75c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:38.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.321+0000 7fe7667fc640 1 -- 172.21.15.23:0/3459942188 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe7600f02a0 con 0x7fe770104ae0 2024-03-24T10:37:38.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.322+0000 7fe775039640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe75c0a2140 0x7fe75c0a4600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe76c005fd0 tx=0x7fe76c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:38.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.325+0000 7fe7667fc640 1 -- 172.21.15.23:0/3459942188 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe7600b9710 con 0x7fe770104ae0 2024-03-24T10:37:38.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:38 smithi023 ceph-mon[32466]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:38.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:38 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:38.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:38 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:38.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:38 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:38.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:38 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:38.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:38 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:38.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:38 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:38.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:38 smithi117 ceph-mon[36368]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:38.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:38 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:38.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:38 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:38.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:38 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:38.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:38 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:38.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:38 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:38.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:38 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:38.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:38 smithi116 ceph-mon[36248]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:38.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:38 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:38.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:38 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:38.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:38 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:38.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:38 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:38.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:38 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:38.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:38 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:38.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.556+0000 7fe77729d640 1 -- 172.21.15.23:0/3459942188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe77010d130 con 0x7fe770104ae0 2024-03-24T10:37:38.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.557+0000 7fe7667fc640 1 -- 172.21.15.23:0/3459942188 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7fe7600bcdd0 con 0x7fe770104ae0 2024-03-24T10:37:38.558 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:38.558 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-24T10:37:38.558 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 3 2024-03-24T10:37:38.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.558+0000 7fe77729d640 1 -- 172.21.15.23:0/3459942188 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe75c0a2140 msgr2=0x7fe75c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:38.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.558+0000 7fe77729d640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe75c0a2140 0x7fe75c0a4600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe76c005fd0 tx=0x7fe76c005850 comp rx=0 tx=0).stop 2024-03-24T10:37:38.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.558+0000 7fe77729d640 1 -- 172.21.15.23:0/3459942188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe770104ae0 msgr2=0x7fe770100160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:38.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.558+0000 7fe77729d640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe770104ae0 0x7fe770100160 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe760000c00 tx=0x7fe760004270 comp rx=0 tx=0).stop 2024-03-24T10:37:38.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.559+0000 7fe77729d640 1 -- 172.21.15.23:0/3459942188 shutdown_connections 2024-03-24T10:37:38.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.559+0000 7fe77729d640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe75c0a2140 0x7fe75c0a4600 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:38.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.559+0000 7fe77729d640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe770107410 0x7fe7701006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:38.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.559+0000 7fe77729d640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe770104ae0 0x7fe770100160 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:38.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.559+0000 7fe77729d640 1 --2- 172.21.15.23:0/3459942188 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7701021b0 0x7fe770101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:38.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.559+0000 7fe77729d640 1 -- 172.21.15.23:0/3459942188 >> 172.21.15.23:0/3459942188 conn(0x7fe7700fbba0 msgr2=0x7fe770102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:38.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.559+0000 7fe77729d640 1 -- 172.21.15.23:0/3459942188 shutdown_connections 2024-03-24T10:37:38.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:38.559+0000 7fe77729d640 1 -- 172.21.15.23:0/3459942188 wait complete. 2024-03-24T10:37:39.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:39 smithi023 ceph-mon[32466]: Deploying daemon mds.j on smithi023 2024-03-24T10:37:39.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:39 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.116:6832/566816400,v1:172.21.15.116:6833/566816400] up:boot 2024-03-24T10:37:39.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:39 smithi023 ceph-mon[32466]: fsmap 2 up:standby 2024-03-24T10:37:39.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:39 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T10:37:39.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:39 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3459942188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:39.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:39 smithi117 ceph-mon[36368]: Deploying daemon mds.j on smithi023 2024-03-24T10:37:39.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:39 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.116:6832/566816400,v1:172.21.15.116:6833/566816400] up:boot 2024-03-24T10:37:39.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:39 smithi117 ceph-mon[36368]: fsmap 2 up:standby 2024-03-24T10:37:39.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:39 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T10:37:39.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:39 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3459942188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:39.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:39 smithi116 ceph-mon[36248]: Deploying daemon mds.j on smithi023 2024-03-24T10:37:39.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:39 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.116:6832/566816400,v1:172.21.15.116:6833/566816400] up:boot 2024-03-24T10:37:39.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:39 smithi116 ceph-mon[36248]: fsmap 2 up:standby 2024-03-24T10:37:39.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:39 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T10:37:39.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:39 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3459942188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:40.152 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:37:39 smithi023 systemd[1]: Starting Ceph mds.j for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:37:40.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:40 smithi023 ceph-mon[32466]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:40.418 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:37:40 smithi023 podman[70371]: 2024-03-24 10:37:40.156390163 +0000 UTC m=+0.108510035 container create e83389afb056a2bc5dd5b5a169c8cd035a2cc4393eb555d0f627951fef1c611e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-24T10:37:40.419 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:37:40 smithi023 podman[70371]: 2024-03-24 10:37:40.065567588 +0000 UTC m=+0.017687461 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:37:40.419 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:37:40 smithi023 podman[70371]: 2024-03-24 10:37:40.375442146 +0000 UTC m=+0.327562027 container init e83389afb056a2bc5dd5b5a169c8cd035a2cc4393eb555d0f627951fef1c611e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T10:37:40.419 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:37:40 smithi023 podman[70371]: 2024-03-24 10:37:40.378115381 +0000 UTC m=+0.330235257 container start e83389afb056a2bc5dd5b5a169c8cd035a2cc4393eb555d0f627951fef1c611e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-j, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T10:37:40.419 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:37:40 smithi023 ceph-mds[70391]: -- 172.21.15.23:0/2156892714 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5572fb1cc1a0 con 0x5572fb1a3000 2024-03-24T10:37:40.419 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:37:40 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-j[70387]: starting mds.j at 2024-03-24T10:37:40.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:40 smithi117 ceph-mon[36368]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:40.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:40 smithi116 ceph-mon[36248]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:40.571 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:37:40.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.679+0000 7ff96538a640 1 -- 172.21.15.23:0/1313296243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff960100630 msgr2=0x7ff960100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:40.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.679+0000 7ff96538a640 1 --2- 172.21.15.23:0/1313296243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff960100630 0x7ff960100a50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff954009860 tx=0x7ff95402bb70 comp rx=0 tx=0).stop 2024-03-24T10:37:40.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.679+0000 7ff96538a640 1 -- 172.21.15.23:0/1313296243 shutdown_connections 2024-03-24T10:37:40.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.679+0000 7ff96538a640 1 --2- 172.21.15.23:0/1313296243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9601021e0 0x7ff960108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:40.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.679+0000 7ff96538a640 1 --2- 172.21.15.23:0/1313296243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff960101820 0x7ff960101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:40.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.679+0000 7ff96538a640 1 --2- 172.21.15.23:0/1313296243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff960100630 0x7ff960100a50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:40.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.679+0000 7ff96538a640 1 -- 172.21.15.23:0/1313296243 >> 172.21.15.23:0/1313296243 conn(0x7ff9600fbbc0 msgr2=0x7ff9600fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:40.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.679+0000 7ff96538a640 1 -- 172.21.15.23:0/1313296243 shutdown_connections 2024-03-24T10:37:40.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.679+0000 7ff96538a640 1 -- 172.21.15.23:0/1313296243 wait complete. 2024-03-24T10:37:40.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.679+0000 7ff96538a640 1 Processor -- start 2024-03-24T10:37:40.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.679+0000 7ff96538a640 1 -- start start 2024-03-24T10:37:40.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.679+0000 7ff96538a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff960100630 0x7ff9601978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:40.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.679+0000 7ff96538a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff960101820 0x7ff960197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:40.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff96538a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9601021e0 0x7ff96019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff96538a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff96019f310 con 0x7ff960101820 2024-03-24T10:37:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff96538a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff96019f480 con 0x7ff960100630 2024-03-24T10:37:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff96538a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff96019f640 con 0x7ff9601021e0 2024-03-24T10:37:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff95e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff960101820 0x7ff960197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff95effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff960100630 0x7ff9601978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff95e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff960101820 0x7ff960197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40836/0 (socket says 172.21.15.23:40836) 2024-03-24T10:37:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff95e7fc640 1 -- 172.21.15.23:0/2734795586 learned_addr learned my addr 172.21.15.23:0/2734795586 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff95f7fe640 1 --2- 172.21.15.23:0/2734795586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9601021e0 0x7ff96019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff95effd640 1 -- 172.21.15.23:0/2734795586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9601021e0 msgr2=0x7ff96019edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff95effd640 1 --2- 172.21.15.23:0/2734795586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9601021e0 0x7ff96019edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff95effd640 1 -- 172.21.15.23:0/2734795586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff960101820 msgr2=0x7ff960197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff95effd640 1 --2- 172.21.15.23:0/2734795586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff960101820 0x7ff960197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff95effd640 1 -- 172.21.15.23:0/2734795586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff950009560 con 0x7ff960100630 2024-03-24T10:37:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff95e7fc640 1 --2- 172.21.15.23:0/2734795586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff960101820 0x7ff960197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:37:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff95effd640 1 --2- 172.21.15.23:0/2734795586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff960100630 0x7ff9601978e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff95402c0b0 tx=0x7ff954034440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff947fff640 1 -- 172.21.15.23:0/2734795586 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95402c510 con 0x7ff960100630 2024-03-24T10:37:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff947fff640 1 -- 172.21.15.23:0/2734795586 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff95402cb50 con 0x7ff960100630 2024-03-24T10:37:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff96538a640 1 -- 172.21.15.23:0/2734795586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff954009490 con 0x7ff960100630 2024-03-24T10:37:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff947fff640 1 -- 172.21.15.23:0/2734795586 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff954003ea0 con 0x7ff960100630 2024-03-24T10:37:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.680+0000 7ff96538a640 1 -- 172.21.15.23:0/2734795586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff96019faa0 con 0x7ff960100630 2024-03-24T10:37:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.681+0000 7ff96538a640 1 -- 172.21.15.23:0/2734795586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff96010c040 con 0x7ff960100630 2024-03-24T10:37:40.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.682+0000 7ff947fff640 1 -- 172.21.15.23:0/2734795586 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ff95403b040 con 0x7ff960100630 2024-03-24T10:37:40.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.683+0000 7ff947fff640 1 --2- 172.21.15.23:0/2734795586 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff9480a20c0 0x7ff9480a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:40.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.683+0000 7ff95e7fc640 1 --2- 172.21.15.23:0/2734795586 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff9480a20c0 0x7ff9480a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:40.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.683+0000 7ff947fff640 1 -- 172.21.15.23:0/2734795586 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff9540ef560 con 0x7ff960100630 2024-03-24T10:37:40.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.683+0000 7ff95e7fc640 1 --2- 172.21.15.23:0/2734795586 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff9480a20c0 0x7ff9480a4580 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff960198d30 tx=0x7ff950009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:40.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.686+0000 7ff947fff640 1 -- 172.21.15.23:0/2734795586 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff9540b8930 con 0x7ff960100630 2024-03-24T10:37:40.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.908+0000 7ff96538a640 1 -- 172.21.15.23:0/2734795586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff96010c220 con 0x7ff960100630 2024-03-24T10:37:40.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.909+0000 7ff947fff640 1 -- 172.21.15.23:0/2734795586 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7ff9540418c0 con 0x7ff960100630 2024-03-24T10:37:40.909 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:40.909 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-24T10:37:40.910 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 3 2024-03-24T10:37:40.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.911+0000 7ff96538a640 1 -- 172.21.15.23:0/2734795586 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff9480a20c0 msgr2=0x7ff9480a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:40.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.911+0000 7ff96538a640 1 --2- 172.21.15.23:0/2734795586 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff9480a20c0 0x7ff9480a4580 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff960198d30 tx=0x7ff950009260 comp rx=0 tx=0).stop 2024-03-24T10:37:40.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.911+0000 7ff96538a640 1 -- 172.21.15.23:0/2734795586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff960100630 msgr2=0x7ff9601978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:40.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.911+0000 7ff96538a640 1 --2- 172.21.15.23:0/2734795586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff960100630 0x7ff9601978e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff95402c0b0 tx=0x7ff954034440 comp rx=0 tx=0).stop 2024-03-24T10:37:40.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.911+0000 7ff96538a640 1 -- 172.21.15.23:0/2734795586 shutdown_connections 2024-03-24T10:37:40.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.911+0000 7ff96538a640 1 --2- 172.21.15.23:0/2734795586 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff9480a20c0 0x7ff9480a4580 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:40.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.911+0000 7ff96538a640 1 --2- 172.21.15.23:0/2734795586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9601021e0 0x7ff96019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:40.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.911+0000 7ff96538a640 1 --2- 172.21.15.23:0/2734795586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff960101820 0x7ff960197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:40.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.911+0000 7ff96538a640 1 --2- 172.21.15.23:0/2734795586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff960100630 0x7ff9601978e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:40.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.911+0000 7ff96538a640 1 -- 172.21.15.23:0/2734795586 >> 172.21.15.23:0/2734795586 conn(0x7ff9600fbbc0 msgr2=0x7ff9600fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:40.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.911+0000 7ff96538a640 1 -- 172.21.15.23:0/2734795586 shutdown_connections 2024-03-24T10:37:40.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:40.911+0000 7ff96538a640 1 -- 172.21.15.23:0/2734795586 wait complete. 2024-03-24T10:37:40.918 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:37:40 smithi023 bash[70371]: e83389afb056a2bc5dd5b5a169c8cd035a2cc4393eb555d0f627951fef1c611e 2024-03-24T10:37:40.918 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:37:40 smithi023 systemd[1]: Started Ceph mds.j for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:37:41.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:41.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:41.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:41.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:41.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:41.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:41.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:41 smithi023 ceph-mon[32466]: Deploying daemon mds.d on smithi023 2024-03-24T10:37:41.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:41 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2734795586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:41.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:41 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.23:6834/3844921538,v1:172.21.15.23:6835/3844921538] up:boot 2024-03-24T10:37:41.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:41 smithi023 ceph-mon[32466]: fsmap 3 up:standby 2024-03-24T10:37:41.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:41 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T10:37:42.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:42.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:42.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:42.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:42.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:42.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:42.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:41 smithi117 ceph-mon[36368]: Deploying daemon mds.d on smithi023 2024-03-24T10:37:42.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:41 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2734795586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:42.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:41 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.23:6834/3844921538,v1:172.21.15.23:6835/3844921538] up:boot 2024-03-24T10:37:42.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:41 smithi117 ceph-mon[36368]: fsmap 3 up:standby 2024-03-24T10:37:42.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:41 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T10:37:42.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:42.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:42.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:42.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:42.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:42.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:42.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:41 smithi116 ceph-mon[36248]: Deploying daemon mds.d on smithi023 2024-03-24T10:37:42.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:41 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2734795586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:42.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:41 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.23:6834/3844921538,v1:172.21.15.23:6835/3844921538] up:boot 2024-03-24T10:37:42.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:41 smithi116 ceph-mon[36248]: fsmap 3 up:standby 2024-03-24T10:37:42.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:41 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T10:37:42.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:42 smithi023 ceph-mon[32466]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:42.918 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:37:42 smithi023 systemd[1]: Starting Ceph mds.d for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:37:42.924 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:37:43.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:42 smithi117 ceph-mon[36368]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:43.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:42 smithi116 ceph-mon[36248]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:43.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.030+0000 7f765e68f640 1 -- 172.21.15.23:0/2359989448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7658101820 msgr2=0x7f7658101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.030+0000 7f765e68f640 1 --2- 172.21.15.23:0/2359989448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7658101820 0x7f7658101ca0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f764c009900 tx=0x7f764c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:37:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.030+0000 7f765e68f640 1 -- 172.21.15.23:0/2359989448 shutdown_connections 2024-03-24T10:37:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.030+0000 7f765e68f640 1 --2- 172.21.15.23:0/2359989448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76581021e0 0x7f7658108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.030+0000 7f765e68f640 1 --2- 172.21.15.23:0/2359989448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7658101820 0x7f7658101ca0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.030+0000 7f765e68f640 1 --2- 172.21.15.23:0/2359989448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7658100630 0x7f7658100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.030+0000 7f765e68f640 1 -- 172.21.15.23:0/2359989448 >> 172.21.15.23:0/2359989448 conn(0x7f76580fbba0 msgr2=0x7f76580fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.030+0000 7f765e68f640 1 -- 172.21.15.23:0/2359989448 shutdown_connections 2024-03-24T10:37:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.030+0000 7f765e68f640 1 -- 172.21.15.23:0/2359989448 wait complete. 2024-03-24T10:37:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.030+0000 7f765e68f640 1 Processor -- start 2024-03-24T10:37:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.030+0000 7f765e68f640 1 -- start start 2024-03-24T10:37:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f765e68f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7658100630 0x7f76581966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f765e68f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7658101820 0x7f7658196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f765e68f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76581021e0 0x7f765819dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f765e68f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f765819e1a0 con 0x7f7658100630 2024-03-24T10:37:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f765e68f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f765819e310 con 0x7f76581021e0 2024-03-24T10:37:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f765e68f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f765819e4d0 con 0x7f7658101820 2024-03-24T10:37:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f7657fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7658100630 0x7f76581966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f7657fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7658100630 0x7f76581966b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40852/0 (socket says 172.21.15.23:40852) 2024-03-24T10:37:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f7657fff640 1 -- 172.21.15.23:0/1930241591 learned_addr learned my addr 172.21.15.23:0/1930241591 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f76577fe640 1 --2- 172.21.15.23:0/1930241591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7658101820 0x7f7658196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f7657fff640 1 -- 172.21.15.23:0/1930241591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7658101820 msgr2=0x7f7658196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f7657fff640 1 --2- 172.21.15.23:0/1930241591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7658101820 0x7f7658196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f7657fff640 1 -- 172.21.15.23:0/1930241591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76581021e0 msgr2=0x7f765819dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f765cc2c640 1 --2- 172.21.15.23:0/1930241591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76581021e0 0x7f765819dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f7657fff640 1 --2- 172.21.15.23:0/1930241591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76581021e0 0x7f765819dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f7657fff640 1 -- 172.21.15.23:0/1930241591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f764c009560 con 0x7f7658100630 2024-03-24T10:37:43.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f7657fff640 1 --2- 172.21.15.23:0/1930241591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7658100630 0x7f76581966b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f764800d950 tx=0x7f764800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:43.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f76557fa640 1 -- 172.21.15.23:0/1930241591 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648009c30 con 0x7f7658100630 2024-03-24T10:37:43.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f76557fa640 1 -- 172.21.15.23:0/1930241591 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7648009d90 con 0x7f7658100630 2024-03-24T10:37:43.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f765e68f640 1 -- 172.21.15.23:0/1930241591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f765819e7b0 con 0x7f7658100630 2024-03-24T10:37:43.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.031+0000 7f76557fa640 1 -- 172.21.15.23:0/1930241591 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764800f850 con 0x7f7658100630 2024-03-24T10:37:43.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.032+0000 7f765e68f640 1 -- 172.21.15.23:0/1930241591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f765819ece0 con 0x7f7658100630 2024-03-24T10:37:43.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.033+0000 7f765e68f640 1 -- 172.21.15.23:0/1930241591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76581079d0 con 0x7f7658100630 2024-03-24T10:37:43.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.034+0000 7f76557fa640 1 -- 172.21.15.23:0/1930241591 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f764800f9b0 con 0x7f7658100630 2024-03-24T10:37:43.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.034+0000 7f76557fa640 1 --2- 172.21.15.23:0/1930241591 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f76400a2140 0x7f76400a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:43.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.034+0000 7f76577fe640 1 --2- 172.21.15.23:0/1930241591 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f76400a2140 0x7f76400a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:43.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.034+0000 7f76557fa640 1 -- 172.21.15.23:0/1930241591 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f76480cd480 con 0x7f7658100630 2024-03-24T10:37:43.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.035+0000 7f76577fe640 1 --2- 172.21.15.23:0/1930241591 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f76400a2140 0x7f76400a4600 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f764c009230 tx=0x7f764c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:43.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.039+0000 7f76557fa640 1 -- 172.21.15.23:0/1930241591 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f76480968f0 con 0x7f7658100630 2024-03-24T10:37:43.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.263+0000 7f765e68f640 1 -- 172.21.15.23:0/1930241591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f765810c040 con 0x7f7658100630 2024-03-24T10:37:43.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.264+0000 7f76557fa640 1 -- 172.21.15.23:0/1930241591 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f7648099fb0 con 0x7f7658100630 2024-03-24T10:37:43.264 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:43.265 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-24T10:37:43.265 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 4 2024-03-24T10:37:43.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.266+0000 7f765e68f640 1 -- 172.21.15.23:0/1930241591 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f76400a2140 msgr2=0x7f76400a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:43.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.266+0000 7f765e68f640 1 --2- 172.21.15.23:0/1930241591 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f76400a2140 0x7f76400a4600 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f764c009230 tx=0x7f764c005fd0 comp rx=0 tx=0).stop 2024-03-24T10:37:43.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.266+0000 7f765e68f640 1 -- 172.21.15.23:0/1930241591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7658100630 msgr2=0x7f76581966b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:43.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.266+0000 7f765e68f640 1 --2- 172.21.15.23:0/1930241591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7658100630 0x7f76581966b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f764800d950 tx=0x7f764800de10 comp rx=0 tx=0).stop 2024-03-24T10:37:43.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.266+0000 7f765e68f640 1 -- 172.21.15.23:0/1930241591 shutdown_connections 2024-03-24T10:37:43.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.266+0000 7f765e68f640 1 --2- 172.21.15.23:0/1930241591 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f76400a2140 0x7f76400a4600 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:43.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.266+0000 7f765e68f640 1 --2- 172.21.15.23:0/1930241591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76581021e0 0x7f765819dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:43.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.266+0000 7f765e68f640 1 --2- 172.21.15.23:0/1930241591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7658101820 0x7f7658196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:43.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.266+0000 7f765e68f640 1 --2- 172.21.15.23:0/1930241591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7658100630 0x7f76581966b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:43.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.266+0000 7f765e68f640 1 -- 172.21.15.23:0/1930241591 >> 172.21.15.23:0/1930241591 conn(0x7f76580fbba0 msgr2=0x7f7658104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:43.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.266+0000 7f765e68f640 1 -- 172.21.15.23:0/1930241591 shutdown_connections 2024-03-24T10:37:43.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:43.266+0000 7f765e68f640 1 -- 172.21.15.23:0/1930241591 wait complete. 2024-03-24T10:37:43.276 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:37:43 smithi023 podman[70718]: 2024-03-24 10:37:42.930298188 +0000 UTC m=+0.018249478 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:37:43.276 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:37:43 smithi023 podman[70718]: 2024-03-24 10:37:43.043798913 +0000 UTC m=+0.131750215 container create 8b05d03599b867c52813141946a0912575e20904ba4443d2c9f8285ba42d1be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-d, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-24T10:37:43.576 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:37:43 smithi023 podman[70718]: 2024-03-24 10:37:43.29709862 +0000 UTC m=+0.385049910 container init 8b05d03599b867c52813141946a0912575e20904ba4443d2c9f8285ba42d1be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-24T10:37:43.576 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:37:43 smithi023 podman[70718]: 2024-03-24 10:37:43.299611682 +0000 UTC m=+0.387562977 container start 8b05d03599b867c52813141946a0912575e20904ba4443d2c9f8285ba42d1be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-24T10:37:43.576 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:37:43 smithi023 ceph-mds[70779]: -- 172.21.15.23:0/1212023244 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561ce510a1a0 con 0x561ce50e1000 2024-03-24T10:37:43.577 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:37:43 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-d[70773]: starting mds.d at 2024-03-24T10:37:43.577 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:37:43 smithi023 bash[70718]: 8b05d03599b867c52813141946a0912575e20904ba4443d2c9f8285ba42d1be0 2024-03-24T10:37:43.577 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:37:43 smithi023 systemd[1]: Started Ceph mds.d for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:37:43.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:43 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1930241591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:43.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:43.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:43.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:43.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:43.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:43.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:43 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:44.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:43 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1930241591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:44.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:44.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:44.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:44.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:44.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:44.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:43 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:43 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1930241591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:44.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:43 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:44 smithi023 ceph-mon[32466]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:44 smithi023 ceph-mon[32466]: Deploying daemon mds.e on smithi116 2024-03-24T10:37:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:44 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.23:6836/405041362,v1:172.21.15.23:6837/405041362] up:boot 2024-03-24T10:37:44.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:44 smithi023 ceph-mon[32466]: fsmap 4 up:standby 2024-03-24T10:37:44.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:44 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T10:37:45.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:44 smithi117 ceph-mon[36368]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:45.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:44 smithi117 ceph-mon[36368]: Deploying daemon mds.e on smithi116 2024-03-24T10:37:45.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:44 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.23:6836/405041362,v1:172.21.15.23:6837/405041362] up:boot 2024-03-24T10:37:45.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:44 smithi117 ceph-mon[36368]: fsmap 4 up:standby 2024-03-24T10:37:45.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:44 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T10:37:45.004 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:44 smithi116 ceph-mon[36248]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:45.004 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:44 smithi116 ceph-mon[36248]: Deploying daemon mds.e on smithi116 2024-03-24T10:37:45.004 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:44 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.23:6836/405041362,v1:172.21.15.23:6837/405041362] up:boot 2024-03-24T10:37:45.004 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:44 smithi116 ceph-mon[36248]: fsmap 4 up:standby 2024-03-24T10:37:45.004 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:44 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T10:37:45.278 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:37:45.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.401+0000 7fbf082d4640 1 -- 172.21.15.23:0/1717586747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf00101720 msgr2=0x7fbf00101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:45.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.401+0000 7fbf082d4640 1 --2- 172.21.15.23:0/1717586747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf00101720 0x7fbf00101b40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbef0009ca0 tx=0x7fbef002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:37:45.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.401+0000 7fbf082d4640 1 -- 172.21.15.23:0/1717586747 shutdown_connections 2024-03-24T10:37:45.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.401+0000 7fbf082d4640 1 --2- 172.21.15.23:0/1717586747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbf001032d0 0x7fbf00109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:45.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.401+0000 7fbf082d4640 1 --2- 172.21.15.23:0/1717586747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf00102910 0x7fbf00102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:45.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.401+0000 7fbf082d4640 1 --2- 172.21.15.23:0/1717586747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf00101720 0x7fbf00101b40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:45.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.401+0000 7fbf082d4640 1 -- 172.21.15.23:0/1717586747 >> 172.21.15.23:0/1717586747 conn(0x7fbf000fccf0 msgr2=0x7fbf000ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:45.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.401+0000 7fbf082d4640 1 -- 172.21.15.23:0/1717586747 shutdown_connections 2024-03-24T10:37:45.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.402+0000 7fbf082d4640 1 -- 172.21.15.23:0/1717586747 wait complete. 2024-03-24T10:37:45.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.402+0000 7fbf082d4640 1 Processor -- start 2024-03-24T10:37:45.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.402+0000 7fbf082d4640 1 -- start start 2024-03-24T10:37:45.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.402+0000 7fbf082d4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbf00101720 0x7fbf00199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:45.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.402+0000 7fbf082d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf00102910 0x7fbf00199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:45.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.402+0000 7fbf082d4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf001032d0 0x7fbf001a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:45.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.402+0000 7fbf082d4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fbf001a1410 con 0x7fbf001032d0 2024-03-24T10:37:45.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.402+0000 7fbf082d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbf001a1580 con 0x7fbf00102910 2024-03-24T10:37:45.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.402+0000 7fbf082d4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbf001a1740 con 0x7fbf00101720 2024-03-24T10:37:45.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.402+0000 7fbf06871640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf001032d0 0x7fbf001a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:45.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.402+0000 7fbf06871640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf001032d0 0x7fbf001a0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40892/0 (socket says 172.21.15.23:40892) 2024-03-24T10:37:45.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.402+0000 7fbf06871640 1 -- 172.21.15.23:0/1736886495 learned_addr learned my addr 172.21.15.23:0/1736886495 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:45.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.403+0000 7fbf0586f640 1 --2- 172.21.15.23:0/1736886495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf00102910 0x7fbf00199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:45.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.403+0000 7fbf06070640 1 --2- 172.21.15.23:0/1736886495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbf00101720 0x7fbf00199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:45.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.403+0000 7fbf06871640 1 -- 172.21.15.23:0/1736886495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbf00101720 msgr2=0x7fbf00199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:45.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.403+0000 7fbf06871640 1 --2- 172.21.15.23:0/1736886495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbf00101720 0x7fbf00199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:45.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.403+0000 7fbf06871640 1 -- 172.21.15.23:0/1736886495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf00102910 msgr2=0x7fbf00199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:45.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.403+0000 7fbf06871640 1 --2- 172.21.15.23:0/1736886495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf00102910 0x7fbf00199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:45.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.403+0000 7fbf06871640 1 -- 172.21.15.23:0/1736886495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef0009900 con 0x7fbf001032d0 2024-03-24T10:37:45.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.403+0000 7fbf06871640 1 --2- 172.21.15.23:0/1736886495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf001032d0 0x7fbf001a0ed0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbef800c970 tx=0x7fbef800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:45.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.403+0000 7fbeff7fe640 1 -- 172.21.15.23:0/1736886495 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef8007bb0 con 0x7fbf001032d0 2024-03-24T10:37:45.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.403+0000 7fbf082d4640 1 -- 172.21.15.23:0/1736886495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf001a1a20 con 0x7fbf001032d0 2024-03-24T10:37:45.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.403+0000 7fbeff7fe640 1 -- 172.21.15.23:0/1736886495 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbef8007d10 con 0x7fbf001032d0 2024-03-24T10:37:45.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.403+0000 7fbeff7fe640 1 -- 172.21.15.23:0/1736886495 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef80186c0 con 0x7fbf001032d0 2024-03-24T10:37:45.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.403+0000 7fbf082d4640 1 -- 172.21.15.23:0/1736886495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf001a1ee0 con 0x7fbf001032d0 2024-03-24T10:37:45.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.405+0000 7fbf082d4640 1 -- 172.21.15.23:0/1736886495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf0010d110 con 0x7fbf001032d0 2024-03-24T10:37:45.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.405+0000 7fbeff7fe640 1 -- 172.21.15.23:0/1736886495 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fbef800f450 con 0x7fbf001032d0 2024-03-24T10:37:45.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.406+0000 7fbeff7fe640 1 --2- 172.21.15.23:0/1736886495 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbee80a20c0 0x7fbee80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:45.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.406+0000 7fbf06070640 1 --2- 172.21.15.23:0/1736886495 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbee80a20c0 0x7fbee80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:45.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.406+0000 7fbeff7fe640 1 -- 172.21.15.23:0/1736886495 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbef80cd9d0 con 0x7fbf001032d0 2024-03-24T10:37:45.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.407+0000 7fbf06070640 1 --2- 172.21.15.23:0/1736886495 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbee80a20c0 0x7fbee80a4580 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbef002c700 tx=0x7fbef00093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:45.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.411+0000 7fbeff7fe640 1 -- 172.21.15.23:0/1736886495 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbef8096e40 con 0x7fbf001032d0 2024-03-24T10:37:45.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.632+0000 7fbf082d4640 1 -- 172.21.15.23:0/1736886495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf001a2380 con 0x7fbf001032d0 2024-03-24T10:37:45.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.632+0000 7fbeff7fe640 1 -- 172.21.15.23:0/1736886495 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3865 (secure 0 0 0) 0x7fbef809a500 con 0x7fbf001032d0 2024-03-24T10:37:45.633 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:45.633 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-24T10:37:45.633 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 5 2024-03-24T10:37:45.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.634+0000 7fbf082d4640 1 -- 172.21.15.23:0/1736886495 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbee80a20c0 msgr2=0x7fbee80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:45.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.634+0000 7fbf082d4640 1 --2- 172.21.15.23:0/1736886495 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbee80a20c0 0x7fbee80a4580 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbef002c700 tx=0x7fbef00093f0 comp rx=0 tx=0).stop 2024-03-24T10:37:45.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.634+0000 7fbf082d4640 1 -- 172.21.15.23:0/1736886495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf001032d0 msgr2=0x7fbf001a0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:45.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.634+0000 7fbf082d4640 1 --2- 172.21.15.23:0/1736886495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf001032d0 0x7fbf001a0ed0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbef800c970 tx=0x7fbef800cd70 comp rx=0 tx=0).stop 2024-03-24T10:37:45.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.634+0000 7fbf082d4640 1 -- 172.21.15.23:0/1736886495 shutdown_connections 2024-03-24T10:37:45.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.634+0000 7fbf082d4640 1 --2- 172.21.15.23:0/1736886495 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fbee80a20c0 0x7fbee80a4580 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:45.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.634+0000 7fbf082d4640 1 --2- 172.21.15.23:0/1736886495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf001032d0 0x7fbf001a0ed0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:45.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.634+0000 7fbf082d4640 1 --2- 172.21.15.23:0/1736886495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf00102910 0x7fbf00199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:45.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.634+0000 7fbf082d4640 1 --2- 172.21.15.23:0/1736886495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbf00101720 0x7fbf00199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:45.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.634+0000 7fbf082d4640 1 -- 172.21.15.23:0/1736886495 >> 172.21.15.23:0/1736886495 conn(0x7fbf000fccf0 msgr2=0x7fbf00073940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:45.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.634+0000 7fbf082d4640 1 -- 172.21.15.23:0/1736886495 shutdown_connections 2024-03-24T10:37:45.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:45.634+0000 7fbf082d4640 1 -- 172.21.15.23:0/1736886495 wait complete. 2024-03-24T10:37:45.699 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:37:45 smithi116 systemd[1]: Starting Ceph mds.e for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:37:45.967 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:37:45 smithi116 podman[65992]: 2024-03-24 10:37:45.602431525 +0000 UTC m=+0.019108189 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:37:45.967 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:37:45 smithi116 podman[65992]: 2024-03-24 10:37:45.722744924 +0000 UTC m=+0.139421567 container create dd163f521ebde1d39b93fa4749fb1db1d48bf701b1dc32a596aa9eda90b04ec2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-e, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T10:37:45.967 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:37:45 smithi116 podman[65992]: 2024-03-24 10:37:45.942113204 +0000 UTC m=+0.358789855 container init dd163f521ebde1d39b93fa4749fb1db1d48bf701b1dc32a596aa9eda90b04ec2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-24T10:37:45.968 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:37:45 smithi116 podman[65992]: 2024-03-24 10:37:45.944569014 +0000 UTC m=+0.361245665 container start dd163f521ebde1d39b93fa4749fb1db1d48bf701b1dc32a596aa9eda90b04ec2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-e, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:37:46.268 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:37:45 smithi116 ceph-mds[66011]: -- 172.21.15.116:0/3496594857 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56362eff81a0 con 0x56362efcec00 2024-03-24T10:37:46.268 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:37:45 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-e[66007]: starting mds.e at 2024-03-24T10:37:46.268 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:37:46 smithi116 bash[65992]: dd163f521ebde1d39b93fa4749fb1db1d48bf701b1dc32a596aa9eda90b04ec2 2024-03-24T10:37:46.268 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:37:46 smithi116 systemd[1]: Started Ceph mds.e for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:37:46.861 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:46 smithi117 ceph-mon[36368]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:46.861 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:46 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1736886495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:46.861 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:46 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:46.862 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:46 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:46.862 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:46 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:46.862 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:46 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:46.862 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:46 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:46.862 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:46 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:46.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:46 smithi023 ceph-mon[32466]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:46.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:46 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1736886495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:46.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:46 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:46.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:46 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:46.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:46 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:46.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:46 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:46.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:46 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:46.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:46 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:47.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:46 smithi116 ceph-mon[36248]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:47.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:46 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1736886495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:47.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:46 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:47.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:46 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:47.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:46 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:47.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:46 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:47.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:46 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:47.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:46 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:47.646 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:37:47.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.762+0000 7f7aef563640 1 -- 172.21.15.23:0/457800013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ae8102200 msgr2=0x7f7ae8108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.762+0000 7f7aef563640 1 --2- 172.21.15.23:0/457800013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ae8102200 0x7f7ae8108a90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0009ca0 tx=0x7f7ae002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:37:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.762+0000 7f7aef563640 1 -- 172.21.15.23:0/457800013 shutdown_connections 2024-03-24T10:37:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.762+0000 7f7aef563640 1 --2- 172.21.15.23:0/457800013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ae8102200 0x7f7ae8108a90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.762+0000 7f7aef563640 1 --2- 172.21.15.23:0/457800013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ae8101840 0x7f7ae8101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.762+0000 7f7aef563640 1 --2- 172.21.15.23:0/457800013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ae8100650 0x7f7ae8100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.762+0000 7f7aef563640 1 -- 172.21.15.23:0/457800013 >> 172.21.15.23:0/457800013 conn(0x7f7ae80fbbc0 msgr2=0x7f7ae80fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.762+0000 7f7aef563640 1 -- 172.21.15.23:0/457800013 shutdown_connections 2024-03-24T10:37:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.762+0000 7f7aef563640 1 -- 172.21.15.23:0/457800013 wait complete. 2024-03-24T10:37:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aef563640 1 Processor -- start 2024-03-24T10:37:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aef563640 1 -- start start 2024-03-24T10:37:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aef563640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ae8100650 0x7f7ae8199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aef563640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ae8101840 0x7f7ae8199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aef563640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ae8102200 0x7f7ae81a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aef563640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae81a1440 con 0x7f7ae8100650 2024-03-24T10:37:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aef563640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae81a15b0 con 0x7f7ae8101840 2024-03-24T10:37:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aef563640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae81a1770 con 0x7f7ae8102200 2024-03-24T10:37:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aed2ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ae8100650 0x7f7ae8199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aed2ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ae8100650 0x7f7ae8199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44420/0 (socket says 172.21.15.23:44420) 2024-03-24T10:37:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aed2ff640 1 -- 172.21.15.23:0/1179889727 learned_addr learned my addr 172.21.15.23:0/1179889727 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aedb00640 1 --2- 172.21.15.23:0/1179889727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ae8102200 0x7f7ae81a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aecafe640 1 --2- 172.21.15.23:0/1179889727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ae8101840 0x7f7ae8199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aed2ff640 1 -- 172.21.15.23:0/1179889727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ae8102200 msgr2=0x7f7ae81a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aed2ff640 1 --2- 172.21.15.23:0/1179889727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ae8102200 0x7f7ae81a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aed2ff640 1 -- 172.21.15.23:0/1179889727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ae8101840 msgr2=0x7f7ae8199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aed2ff640 1 --2- 172.21.15.23:0/1179889727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ae8101840 0x7f7ae8199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.763+0000 7f7aed2ff640 1 -- 172.21.15.23:0/1179889727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae0009900 con 0x7f7ae8100650 2024-03-24T10:37:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.764+0000 7f7aed2ff640 1 --2- 172.21.15.23:0/1179889727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ae8100650 0x7f7ae8199950 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400d770 tx=0x7f7ae400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.764+0000 7f7ade7fc640 1 -- 172.21.15.23:0/1179889727 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4009c30 con 0x7f7ae8100650 2024-03-24T10:37:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.764+0000 7f7aef563640 1 -- 172.21.15.23:0/1179889727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ae81a1a50 con 0x7f7ae8100650 2024-03-24T10:37:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.764+0000 7f7ade7fc640 1 -- 172.21.15.23:0/1179889727 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ae400de80 con 0x7f7ae8100650 2024-03-24T10:37:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.764+0000 7f7ade7fc640 1 -- 172.21.15.23:0/1179889727 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4010460 con 0x7f7ae8100650 2024-03-24T10:37:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.764+0000 7f7aef563640 1 -- 172.21.15.23:0/1179889727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ae81a1fb0 con 0x7f7ae8100650 2024-03-24T10:37:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.765+0000 7f7aef563640 1 -- 172.21.15.23:0/1179889727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ae810c060 con 0x7f7ae8100650 2024-03-24T10:37:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.766+0000 7f7ade7fc640 1 -- 172.21.15.23:0/1179889727 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f7ae4003f60 con 0x7f7ae8100650 2024-03-24T10:37:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.766+0000 7f7ade7fc640 1 --2- 172.21.15.23:0/1179889727 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7ad40a20c0 0x7f7ad40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.766+0000 7f7aecafe640 1 --2- 172.21.15.23:0/1179889727 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7ad40a20c0 0x7f7ad40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.766+0000 7f7ade7fc640 1 -- 172.21.15.23:0/1179889727 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7ae40cd510 con 0x7f7ae8100650 2024-03-24T10:37:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.767+0000 7f7aecafe640 1 --2- 172.21.15.23:0/1179889727 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7ad40a20c0 0x7f7ad40a4580 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7ae819ae60 tx=0x7f7ad8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.771+0000 7f7ade7fc640 1 -- 172.21.15.23:0/1179889727 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7ae4096980 con 0x7f7ae8100650 2024-03-24T10:37:47.853 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:47 smithi117 ceph-mon[36368]: Deploying daemon mds.f on smithi117 2024-03-24T10:37:47.854 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:47 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.116:6834/1535852767,v1:172.21.15.116:6835/1535852767] up:boot 2024-03-24T10:37:47.854 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:47 smithi117 ceph-mon[36368]: fsmap 5 up:standby 2024-03-24T10:37:47.854 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:47 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T10:37:47.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:47 smithi023 ceph-mon[32466]: Deploying daemon mds.f on smithi117 2024-03-24T10:37:47.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:47 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.116:6834/1535852767,v1:172.21.15.116:6835/1535852767] up:boot 2024-03-24T10:37:47.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:47 smithi023 ceph-mon[32466]: fsmap 5 up:standby 2024-03-24T10:37:47.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:47 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T10:37:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.994+0000 7f7aef563640 1 -- 172.21.15.23:0/1179889727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ae81a21a0 con 0x7f7ae8100650 2024-03-24T10:37:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.994+0000 7f7ade7fc640 1 -- 172.21.15.23:0/1179889727 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7f7ae409a040 con 0x7f7ae8100650 2024-03-24T10:37:47.995 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:47.995 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-24T10:37:47.995 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 6 2024-03-24T10:37:47.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.996+0000 7f7aef563640 1 -- 172.21.15.23:0/1179889727 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7ad40a20c0 msgr2=0x7f7ad40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:47.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.996+0000 7f7aef563640 1 --2- 172.21.15.23:0/1179889727 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7ad40a20c0 0x7f7ad40a4580 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7ae819ae60 tx=0x7f7ad8005f90 comp rx=0 tx=0).stop 2024-03-24T10:37:47.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.996+0000 7f7aef563640 1 -- 172.21.15.23:0/1179889727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ae8100650 msgr2=0x7f7ae8199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:47.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.996+0000 7f7aef563640 1 --2- 172.21.15.23:0/1179889727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ae8100650 0x7f7ae8199950 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400d770 tx=0x7f7ae400dc30 comp rx=0 tx=0).stop 2024-03-24T10:37:47.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.996+0000 7f7aef563640 1 -- 172.21.15.23:0/1179889727 shutdown_connections 2024-03-24T10:37:47.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.996+0000 7f7aef563640 1 --2- 172.21.15.23:0/1179889727 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7ad40a20c0 0x7f7ad40a4580 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:47.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.996+0000 7f7aef563640 1 --2- 172.21.15.23:0/1179889727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ae8102200 0x7f7ae81a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:47.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.996+0000 7f7aef563640 1 --2- 172.21.15.23:0/1179889727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ae8101840 0x7f7ae8199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:47.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.996+0000 7f7aef563640 1 --2- 172.21.15.23:0/1179889727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ae8100650 0x7f7ae8199950 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:47.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.996+0000 7f7aef563640 1 -- 172.21.15.23:0/1179889727 >> 172.21.15.23:0/1179889727 conn(0x7f7ae80fbbc0 msgr2=0x7f7ae8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:47.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.996+0000 7f7aef563640 1 -- 172.21.15.23:0/1179889727 shutdown_connections 2024-03-24T10:37:47.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:47.996+0000 7f7aef563640 1 -- 172.21.15.23:0/1179889727 wait complete. 2024-03-24T10:37:48.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:47 smithi116 ceph-mon[36248]: Deploying daemon mds.f on smithi117 2024-03-24T10:37:48.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:47 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.116:6834/1535852767,v1:172.21.15.116:6835/1535852767] up:boot 2024-03-24T10:37:48.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:47 smithi116 ceph-mon[36248]: fsmap 5 up:standby 2024-03-24T10:37:48.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:47 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T10:37:48.253 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:37:48 smithi117 systemd[1]: Starting Ceph mds.f for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:37:48.568 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:37:48 smithi117 podman[66797]: 2024-03-24 10:37:48.156926544 +0000 UTC m=+0.017643608 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:37:48.568 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:37:48 smithi117 podman[66797]: 2024-03-24 10:37:48.257162564 +0000 UTC m=+0.117879633 container create aac3988c0274261a20017ab628ba77e3de3844bd1bb684f5431f15e16a99dff8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-f, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T10:37:48.568 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:37:48 smithi117 podman[66797]: 2024-03-24 10:37:48.467783013 +0000 UTC m=+0.328500076 container init aac3988c0274261a20017ab628ba77e3de3844bd1bb684f5431f15e16a99dff8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-f, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD) 2024-03-24T10:37:48.568 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:37:48 smithi117 podman[66797]: 2024-03-24 10:37:48.47081084 +0000 UTC m=+0.331527883 container start aac3988c0274261a20017ab628ba77e3de3844bd1bb684f5431f15e16a99dff8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:37:48.568 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:37:48 smithi117 ceph-mds[66816]: -- 172.21.15.117:0/1519669251 <== mon.1 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cad096c1a0 con 0x55cad0942800 2024-03-24T10:37:48.568 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:37:48 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-f[66812]: starting mds.f at 2024-03-24T10:37:48.568 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:37:48 smithi117 bash[66797]: aac3988c0274261a20017ab628ba77e3de3844bd1bb684f5431f15e16a99dff8 2024-03-24T10:37:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:48 smithi023 ceph-mon[32466]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:48 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1179889727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:49.003 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:37:48 smithi117 systemd[1]: Started Ceph mds.f for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:37:49.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:48 smithi117 ceph-mon[36368]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:49.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:48 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1179889727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:49.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:48 smithi116 ceph-mon[36248]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:49.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:48 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1179889727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:49.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:49 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.117:6834/1664408853,v1:172.21.15.117:6835/1664408853] up:boot 2024-03-24T10:37:49.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:49 smithi023 ceph-mon[32466]: fsmap 6 up:standby 2024-03-24T10:37:49.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:49 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T10:37:49.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:49 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:49.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:49 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:49.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:49 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:49.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:49 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:49.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:49 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:49.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:49 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:49.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:49 smithi023 ceph-mon[32466]: Deploying daemon mds.c on smithi117 2024-03-24T10:37:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:49 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.117:6834/1664408853,v1:172.21.15.117:6835/1664408853] up:boot 2024-03-24T10:37:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:49 smithi117 ceph-mon[36368]: fsmap 6 up:standby 2024-03-24T10:37:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:49 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T10:37:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:49 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:49 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:50.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:49 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:50.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:49 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:50.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:49 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:50.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:49 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:50.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:49 smithi117 ceph-mon[36368]: Deploying daemon mds.c on smithi117 2024-03-24T10:37:50.009 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:37:50.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:49 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.117:6834/1664408853,v1:172.21.15.117:6835/1664408853] up:boot 2024-03-24T10:37:50.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:49 smithi116 ceph-mon[36248]: fsmap 6 up:standby 2024-03-24T10:37:50.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:49 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T10:37:50.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:49 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:50.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:49 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:50.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:49 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:50.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:49 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:50.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:49 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:50.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:49 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:50.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:49 smithi116 ceph-mon[36248]: Deploying daemon mds.c on smithi117 2024-03-24T10:37:50.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.123+0000 7fec622c3640 1 -- 172.21.15.23:0/2171346236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec5c1032f0 msgr2=0x7fec5c109b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:50.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.123+0000 7fec622c3640 1 --2- 172.21.15.23:0/2171346236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec5c1032f0 0x7fec5c109b80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fec54009ca0 tx=0x7fec5402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:37:50.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.123+0000 7fec622c3640 1 -- 172.21.15.23:0/2171346236 shutdown_connections 2024-03-24T10:37:50.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.123+0000 7fec622c3640 1 --2- 172.21.15.23:0/2171346236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec5c1032f0 0x7fec5c109b80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:50.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.123+0000 7fec622c3640 1 --2- 172.21.15.23:0/2171346236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec5c102930 0x7fec5c102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:50.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.123+0000 7fec622c3640 1 --2- 172.21.15.23:0/2171346236 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec5c101740 0x7fec5c101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:50.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.123+0000 7fec622c3640 1 -- 172.21.15.23:0/2171346236 >> 172.21.15.23:0/2171346236 conn(0x7fec5c0fccf0 msgr2=0x7fec5c0ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:50.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.123+0000 7fec622c3640 1 -- 172.21.15.23:0/2171346236 shutdown_connections 2024-03-24T10:37:50.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.123+0000 7fec622c3640 1 -- 172.21.15.23:0/2171346236 wait complete. 2024-03-24T10:37:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.123+0000 7fec622c3640 1 Processor -- start 2024-03-24T10:37:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec622c3640 1 -- start start 2024-03-24T10:37:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec622c3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec5c101740 0x7fec5c197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec622c3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec5c102930 0x7fec5c197cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec622c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec5c1032f0 0x7fec5c19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec622c3640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fec5c19f280 con 0x7fec5c101740 2024-03-24T10:37:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec622c3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fec5c19f3f0 con 0x7fec5c1032f0 2024-03-24T10:37:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec622c3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fec5c19f5b0 con 0x7fec5c102930 2024-03-24T10:37:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec5bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec5c101740 0x7fec5c197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec5bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec5c101740 0x7fec5c197790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44436/0 (socket says 172.21.15.23:44436) 2024-03-24T10:37:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec5bfff640 1 -- 172.21.15.23:0/4258324125 learned_addr learned my addr 172.21.15.23:0/4258324125 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec5b7fe640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec5c102930 0x7fec5c197cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec60860640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec5c1032f0 0x7fec5c19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec5bfff640 1 -- 172.21.15.23:0/4258324125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec5c102930 msgr2=0x7fec5c197cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec5bfff640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec5c102930 0x7fec5c197cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec5bfff640 1 -- 172.21.15.23:0/4258324125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec5c1032f0 msgr2=0x7fec5c19ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec5bfff640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec5c1032f0 0x7fec5c19ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec5bfff640 1 -- 172.21.15.23:0/4258324125 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec54009900 con 0x7fec5c101740 2024-03-24T10:37:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.124+0000 7fec60860640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec5c1032f0 0x7fec5c19ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:37:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.125+0000 7fec5bfff640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec5c101740 0x7fec5c197790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00e9c0 tx=0x7fec4c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.125+0000 7fec597fa640 1 -- 172.21.15.23:0/4258324125 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c0096e0 con 0x7fec5c101740 2024-03-24T10:37:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.125+0000 7fec597fa640 1 -- 172.21.15.23:0/4258324125 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec4c00cd80 con 0x7fec5c101740 2024-03-24T10:37:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.125+0000 7fec622c3640 1 -- 172.21.15.23:0/4258324125 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec5c19f890 con 0x7fec5c101740 2024-03-24T10:37:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.125+0000 7fec597fa640 1 -- 172.21.15.23:0/4258324125 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c010430 con 0x7fec5c101740 2024-03-24T10:37:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.125+0000 7fec622c3640 1 -- 172.21.15.23:0/4258324125 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec5c19fd50 con 0x7fec5c101740 2024-03-24T10:37:50.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.126+0000 7fec622c3640 1 -- 172.21.15.23:0/4258324125 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec5c108a80 con 0x7fec5c101740 2024-03-24T10:37:50.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.127+0000 7fec597fa640 1 -- 172.21.15.23:0/4258324125 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fec4c003680 con 0x7fec5c101740 2024-03-24T10:37:50.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.127+0000 7fec597fa640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fec440a20c0 0x7fec440a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:50.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.127+0000 7fec597fa640 1 -- 172.21.15.23:0/4258324125 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fec4c014070 con 0x7fec5c101740 2024-03-24T10:37:50.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.127+0000 7fec5b7fe640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fec440a20c0 0x7fec440a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:50.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.128+0000 7fec5b7fe640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fec440a20c0 0x7fec440a4580 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fec5c198ca0 tx=0x7fec5000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.132+0000 7fec597fa640 1 -- 172.21.15.23:0/4258324125 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fec4c0970b0 con 0x7fec5c101740 2024-03-24T10:37:50.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.353+0000 7fec622c3640 1 -- 172.21.15.23:0/4258324125 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec5c10d140 con 0x7fec5c101740 2024-03-24T10:37:50.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.353+0000 7fec597fa640 1 -- 172.21.15.23:0/4258324125 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7fec4c09a770 con 0x7fec5c101740 2024-03-24T10:37:50.354 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:50.354 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-24T10:37:50.354 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 7 2024-03-24T10:37:50.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.356+0000 7fec622c3640 1 -- 172.21.15.23:0/4258324125 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fec440a20c0 msgr2=0x7fec440a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:50.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.356+0000 7fec622c3640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fec440a20c0 0x7fec440a4580 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fec5c198ca0 tx=0x7fec5000b390 comp rx=0 tx=0).stop 2024-03-24T10:37:50.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.356+0000 7fec622c3640 1 -- 172.21.15.23:0/4258324125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec5c101740 msgr2=0x7fec5c197790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:50.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.356+0000 7fec622c3640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec5c101740 0x7fec5c197790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00e9c0 tx=0x7fec4c00ee80 comp rx=0 tx=0).stop 2024-03-24T10:37:50.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.356+0000 7fec622c3640 1 -- 172.21.15.23:0/4258324125 shutdown_connections 2024-03-24T10:37:50.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.356+0000 7fec622c3640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fec440a20c0 0x7fec440a4580 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:50.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.356+0000 7fec622c3640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec5c1032f0 0x7fec5c19ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:50.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.356+0000 7fec622c3640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec5c102930 0x7fec5c197cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:50.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.356+0000 7fec622c3640 1 --2- 172.21.15.23:0/4258324125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec5c101740 0x7fec5c197790 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:50.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.356+0000 7fec622c3640 1 -- 172.21.15.23:0/4258324125 >> 172.21.15.23:0/4258324125 conn(0x7fec5c0fccf0 msgr2=0x7fec5c105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:50.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.356+0000 7fec622c3640 1 -- 172.21.15.23:0/4258324125 shutdown_connections 2024-03-24T10:37:50.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:50.356+0000 7fec622c3640 1 -- 172.21.15.23:0/4258324125 wait complete. 2024-03-24T10:37:50.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:50 smithi023 ceph-mon[32466]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:50.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:50 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4258324125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:50.926 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:50 smithi117 ceph-mon[36368]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:50.926 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:50 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4258324125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:50.927 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:37:50 smithi117 systemd[1]: Starting Ceph mds.c for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:37:51.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:50 smithi116 ceph-mon[36248]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:51.071 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:50 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4258324125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:51.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:37:50 smithi117 podman[67106]: 2024-03-24 10:37:50.926310017 +0000 UTC m=+0.110069955 container create 9797d79d19738e49e516851e7d757974cf46528a64e728a4fda9afd1c6c8e9f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T10:37:51.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:37:50 smithi117 podman[67106]: 2024-03-24 10:37:50.833535187 +0000 UTC m=+0.017295118 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:37:51.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:37:51 smithi117 podman[67106]: 2024-03-24 10:37:51.136864259 +0000 UTC m=+0.320624192 container init 9797d79d19738e49e516851e7d757974cf46528a64e728a4fda9afd1c6c8e9f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:37:51.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:37:51 smithi117 podman[67106]: 2024-03-24 10:37:51.139508339 +0000 UTC m=+0.323268261 container start 9797d79d19738e49e516851e7d757974cf46528a64e728a4fda9afd1c6c8e9f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-03-24T10:37:51.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:37:51 smithi117 ceph-mds[67125]: -- 172.21.15.117:0/1482949454 <== mon.1 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ac502ca1a0 con 0x55ac502a0c00 2024-03-24T10:37:51.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:37:51 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: starting mds.c at 2024-03-24T10:37:51.213 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:37:51 smithi117 bash[67106]: 9797d79d19738e49e516851e7d757974cf46528a64e728a4fda9afd1c6c8e9f6 2024-03-24T10:37:51.503 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:37:51 smithi117 systemd[1]: Started Ceph mds.c for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:37:52.368 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:37:52.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.482+0000 7f89cfa0e640 1 -- 172.21.15.23:0/1535480890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89c81021e0 msgr2=0x7f89c8108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:52.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.482+0000 7f89cfa0e640 1 --2- 172.21.15.23:0/1535480890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89c81021e0 0x7f89c8108a70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009ca0 tx=0x7f89c002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:37:52.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.482+0000 7f89cfa0e640 1 -- 172.21.15.23:0/1535480890 shutdown_connections 2024-03-24T10:37:52.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.482+0000 7f89cfa0e640 1 --2- 172.21.15.23:0/1535480890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89c81021e0 0x7f89c8108a70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:52.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.482+0000 7f89cfa0e640 1 --2- 172.21.15.23:0/1535480890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c8101820 0x7f89c8101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:52.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.482+0000 7f89cfa0e640 1 --2- 172.21.15.23:0/1535480890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89c8100630 0x7f89c8100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:52.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.482+0000 7f89cfa0e640 1 -- 172.21.15.23:0/1535480890 >> 172.21.15.23:0/1535480890 conn(0x7f89c80fbba0 msgr2=0x7f89c80fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:52.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.483+0000 7f89cfa0e640 1 -- 172.21.15.23:0/1535480890 shutdown_connections 2024-03-24T10:37:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.483+0000 7f89cfa0e640 1 -- 172.21.15.23:0/1535480890 wait complete. 2024-03-24T10:37:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.483+0000 7f89cfa0e640 1 Processor -- start 2024-03-24T10:37:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.483+0000 7f89cfa0e640 1 -- start start 2024-03-24T10:37:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.483+0000 7f89cfa0e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c8100630 0x7f89c8197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.483+0000 7f89cfa0e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89c8101820 0x7f89c8197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.483+0000 7f89cfa0e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89c81021e0 0x7f89c819ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.483+0000 7f89cfa0e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f89c819f270 con 0x7f89c8101820 2024-03-24T10:37:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.483+0000 7f89cfa0e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f89c819f3e0 con 0x7f89c8100630 2024-03-24T10:37:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.483+0000 7f89cfa0e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f89c819f5a0 con 0x7f89c81021e0 2024-03-24T10:37:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.483+0000 7f89ccfa9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89c8101820 0x7f89c8197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.483+0000 7f89ccfa9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89c8101820 0x7f89c8197cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44452/0 (socket says 172.21.15.23:44452) 2024-03-24T10:37:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.483+0000 7f89ccfa9640 1 -- 172.21.15.23:0/646624149 learned_addr learned my addr 172.21.15.23:0/646624149 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89cd7aa640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c8100630 0x7f89c8197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:52.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89cdfab640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89c81021e0 0x7f89c819ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:52.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89ccfa9640 1 -- 172.21.15.23:0/646624149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89c81021e0 msgr2=0x7f89c819ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:52.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89ccfa9640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89c81021e0 0x7f89c819ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:52.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89ccfa9640 1 -- 172.21.15.23:0/646624149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c8100630 msgr2=0x7f89c8197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:52.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89ccfa9640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c8100630 0x7f89c8197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:52.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89ccfa9640 1 -- 172.21.15.23:0/646624149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c0009900 con 0x7f89c8101820 2024-03-24T10:37:52.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89cdfab640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89c81021e0 0x7f89c819ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:37:52.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89cd7aa640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c8100630 0x7f89c8197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:37:52.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89ccfa9640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89c8101820 0x7f89c8197cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f89b800d950 tx=0x7f89b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:52.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89be7fc640 1 -- 172.21.15.23:0/646624149 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8009c30 con 0x7f89c8101820 2024-03-24T10:37:52.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89cfa0e640 1 -- 172.21.15.23:0/646624149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c819f880 con 0x7f89c8101820 2024-03-24T10:37:52.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89be7fc640 1 -- 172.21.15.23:0/646624149 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89b8009d90 con 0x7f89c8101820 2024-03-24T10:37:52.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89be7fc640 1 -- 172.21.15.23:0/646624149 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b800f7e0 con 0x7f89c8101820 2024-03-24T10:37:52.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.484+0000 7f89cfa0e640 1 -- 172.21.15.23:0/646624149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c819fd90 con 0x7f89c8101820 2024-03-24T10:37:52.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.485+0000 7f89cfa0e640 1 -- 172.21.15.23:0/646624149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c810c010 con 0x7f89c8101820 2024-03-24T10:37:52.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.486+0000 7f89be7fc640 1 -- 172.21.15.23:0/646624149 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f89b8010ad0 con 0x7f89c8101820 2024-03-24T10:37:52.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.487+0000 7f89be7fc640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f89b40a20c0 0x7f89b40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:52.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.487+0000 7f89be7fc640 1 -- 172.21.15.23:0/646624149 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f89b80cd440 con 0x7f89c8101820 2024-03-24T10:37:52.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.487+0000 7f89cd7aa640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f89b40a20c0 0x7f89b40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:52.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.487+0000 7f89cd7aa640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f89b40a20c0 0x7f89b40a4580 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f89c4005d00 tx=0x7f89c4005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:52.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.491+0000 7f89be7fc640 1 -- 172.21.15.23:0/646624149 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f89b80968b0 con 0x7f89c8101820 2024-03-24T10:37:52.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:52 smithi116 ceph-mon[36248]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:52.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:52.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:52.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:52.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:52.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:52.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:52.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:52 smithi116 ceph-mon[36248]: Deploying daemon mds.h on smithi116 2024-03-24T10:37:52.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:52 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.117:6836/977076441,v1:172.21.15.117:6837/977076441] up:boot 2024-03-24T10:37:52.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:52 smithi116 ceph-mon[36248]: fsmap 7 up:standby 2024-03-24T10:37:52.519 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:52 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:37:52.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:52 smithi023 ceph-mon[32466]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:52.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:52.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:52.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:52.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:52.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:52.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:52.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:52 smithi023 ceph-mon[32466]: Deploying daemon mds.h on smithi116 2024-03-24T10:37:52.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:52 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.117:6836/977076441,v1:172.21.15.117:6837/977076441] up:boot 2024-03-24T10:37:52.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:52 smithi023 ceph-mon[32466]: fsmap 7 up:standby 2024-03-24T10:37:52.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:52 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:37:52.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.712+0000 7f89cfa0e640 1 -- 172.21.15.23:0/646624149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89c810c1d0 con 0x7f89c8101820 2024-03-24T10:37:52.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.713+0000 7f89be7fc640 1 -- 172.21.15.23:0/646624149 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6346 (secure 0 0 0) 0x7f89b8099f70 con 0x7f89c8101820 2024-03-24T10:37:52.713 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:52.714 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-24T10:37:52.714 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 8 2024-03-24T10:37:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.715+0000 7f89cfa0e640 1 -- 172.21.15.23:0/646624149 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f89b40a20c0 msgr2=0x7f89b40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.715+0000 7f89cfa0e640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f89b40a20c0 0x7f89b40a4580 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f89c4005d00 tx=0x7f89c4005c90 comp rx=0 tx=0).stop 2024-03-24T10:37:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.715+0000 7f89cfa0e640 1 -- 172.21.15.23:0/646624149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89c8101820 msgr2=0x7f89c8197cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.715+0000 7f89cfa0e640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89c8101820 0x7f89c8197cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f89b800d950 tx=0x7f89b800de10 comp rx=0 tx=0).stop 2024-03-24T10:37:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.715+0000 7f89cfa0e640 1 -- 172.21.15.23:0/646624149 shutdown_connections 2024-03-24T10:37:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.715+0000 7f89cfa0e640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f89b40a20c0 0x7f89b40a4580 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.715+0000 7f89cfa0e640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89c81021e0 0x7f89c819ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:52.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.715+0000 7f89cfa0e640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89c8101820 0x7f89c8197cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:52.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.715+0000 7f89cfa0e640 1 --2- 172.21.15.23:0/646624149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c8100630 0x7f89c8197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:52.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.715+0000 7f89cfa0e640 1 -- 172.21.15.23:0/646624149 >> 172.21.15.23:0/646624149 conn(0x7f89c80fbba0 msgr2=0x7f89c80fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:52.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.715+0000 7f89cfa0e640 1 -- 172.21.15.23:0/646624149 shutdown_connections 2024-03-24T10:37:52.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:52.715+0000 7f89cfa0e640 1 -- 172.21.15.23:0/646624149 wait complete. 2024-03-24T10:37:52.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:52 smithi117 ceph-mon[36368]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:52.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:52.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:52.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:52.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:52.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:52.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:52.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:52 smithi117 ceph-mon[36368]: Deploying daemon mds.h on smithi116 2024-03-24T10:37:52.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:52 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.117:6836/977076441,v1:172.21.15.117:6837/977076441] up:boot 2024-03-24T10:37:52.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:52 smithi117 ceph-mon[36368]: fsmap 7 up:standby 2024-03-24T10:37:52.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:52 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:37:53.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:53 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/646624149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:53.518 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:37:53 smithi116 systemd[1]: Starting Ceph mds.h for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:37:53.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:53 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/646624149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:53.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:53 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/646624149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:53.894 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:37:53 smithi116 podman[66302]: 2024-03-24 10:37:53.521289064 +0000 UTC m=+0.017526411 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:37:53.894 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:37:53 smithi116 podman[66302]: 2024-03-24 10:37:53.62562401 +0000 UTC m=+0.121861358 container create f886966040763359b713dd539b517e75eed6d5da42c730a8dffaca2c2b35a4bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-h, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:37:54.268 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:37:53 smithi116 podman[66302]: 2024-03-24 10:37:53.89472066 +0000 UTC m=+0.390958010 container init f886966040763359b713dd539b517e75eed6d5da42c730a8dffaca2c2b35a4bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-h, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T10:37:54.268 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:37:53 smithi116 podman[66302]: 2024-03-24 10:37:53.897430738 +0000 UTC m=+0.393668094 container start f886966040763359b713dd539b517e75eed6d5da42c730a8dffaca2c2b35a4bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-24T10:37:54.268 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:37:53 smithi116 ceph-mds[66322]: -- 172.21.15.116:0/45599461 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559ad29901a0 con 0x559ad2966800 2024-03-24T10:37:54.268 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:37:53 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-h[66318]: starting mds.h at 2024-03-24T10:37:54.268 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:37:53 smithi116 bash[66302]: f886966040763359b713dd539b517e75eed6d5da42c730a8dffaca2c2b35a4bc 2024-03-24T10:37:54.268 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:37:53 smithi116 systemd[1]: Started Ceph mds.h for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:37:54.654 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:54 smithi117 ceph-mon[36368]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:54.654 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:54 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:54.654 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:54 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:54.654 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:54 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:54.654 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:54 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:54.654 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:54 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:54.654 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:54 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:54.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:54 smithi023 ceph-mon[32466]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:54.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:54 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:54.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:54 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:54.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:54 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:54.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:54 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:54.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:54 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:54.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:54 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:54.727 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:37:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:54 smithi116 ceph-mon[36248]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:54 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:54 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:54 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:54 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:54 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:54 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:54.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.841+0000 7f81b9c4e640 1 -- 172.21.15.23:0/3097105603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81b4101820 msgr2=0x7f81b4101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:54.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.841+0000 7f81b9c4e640 1 --2- 172.21.15.23:0/3097105603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81b4101820 0x7f81b4101ca0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f81a8009900 tx=0x7f81a802bc30 comp rx=0 tx=0).stop 2024-03-24T10:37:54.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.841+0000 7f81b9c4e640 1 -- 172.21.15.23:0/3097105603 shutdown_connections 2024-03-24T10:37:54.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.841+0000 7f81b9c4e640 1 --2- 172.21.15.23:0/3097105603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b41021e0 0x7f81b4108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:54.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.841+0000 7f81b9c4e640 1 --2- 172.21.15.23:0/3097105603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81b4101820 0x7f81b4101ca0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.841+0000 7f81b9c4e640 1 --2- 172.21.15.23:0/3097105603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81b4100630 0x7f81b4100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.841+0000 7f81b9c4e640 1 -- 172.21.15.23:0/3097105603 >> 172.21.15.23:0/3097105603 conn(0x7f81b40fbba0 msgr2=0x7f81b40fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.841+0000 7f81b9c4e640 1 -- 172.21.15.23:0/3097105603 shutdown_connections 2024-03-24T10:37:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.841+0000 7f81b9c4e640 1 -- 172.21.15.23:0/3097105603 wait complete. 2024-03-24T10:37:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.842+0000 7f81b9c4e640 1 Processor -- start 2024-03-24T10:37:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.842+0000 7f81b9c4e640 1 -- start start 2024-03-24T10:37:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.842+0000 7f81b9c4e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b4100630 0x7f81b41977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.842+0000 7f81b9c4e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81b4101820 0x7f81b4197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.842+0000 7f81b9c4e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81b41021e0 0x7f81b419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.842+0000 7f81b9c4e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f81b419f2c0 con 0x7f81b4101820 2024-03-24T10:37:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.842+0000 7f81b9c4e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f81b419f430 con 0x7f81b4100630 2024-03-24T10:37:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.842+0000 7f81b9c4e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f81b419f5f0 con 0x7f81b41021e0 2024-03-24T10:37:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.842+0000 7f81b2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81b4101820 0x7f81b4197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.842+0000 7f81b2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81b4101820 0x7f81b4197d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44472/0 (socket says 172.21.15.23:44472) 2024-03-24T10:37:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.842+0000 7f81b2ffd640 1 -- 172.21.15.23:0/1535878430 learned_addr learned my addr 172.21.15.23:0/1535878430 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.842+0000 7f81b3fff640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81b41021e0 0x7f81b419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.843+0000 7f81b37fe640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b4100630 0x7f81b41977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.843+0000 7f81b2ffd640 1 -- 172.21.15.23:0/1535878430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81b41021e0 msgr2=0x7f81b419ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.843+0000 7f81b2ffd640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81b41021e0 0x7f81b419ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.843+0000 7f81b2ffd640 1 -- 172.21.15.23:0/1535878430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b4100630 msgr2=0x7f81b41977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.843+0000 7f81b2ffd640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b4100630 0x7f81b41977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.843+0000 7f81b2ffd640 1 -- 172.21.15.23:0/1535878430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a8009560 con 0x7f81b4101820 2024-03-24T10:37:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.843+0000 7f81b37fe640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b4100630 0x7f81b41977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:37:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.843+0000 7f81b2ffd640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81b4101820 0x7f81b4197d10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f81a8006010 tx=0x7f81a80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.843+0000 7f81b0ff9640 1 -- 172.21.15.23:0/1535878430 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a80044c0 con 0x7f81b4101820 2024-03-24T10:37:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.843+0000 7f81b0ff9640 1 -- 172.21.15.23:0/1535878430 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81a80052d0 con 0x7f81b4101820 2024-03-24T10:37:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.843+0000 7f81b9c4e640 1 -- 172.21.15.23:0/1535878430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81b419f870 con 0x7f81b4101820 2024-03-24T10:37:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.843+0000 7f81b0ff9640 1 -- 172.21.15.23:0/1535878430 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a8003e10 con 0x7f81b4101820 2024-03-24T10:37:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.843+0000 7f81b9c4e640 1 -- 172.21.15.23:0/1535878430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81b419fd80 con 0x7f81b4101820 2024-03-24T10:37:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.844+0000 7f81b9c4e640 1 -- 172.21.15.23:0/1535878430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81b4107950 con 0x7f81b4101820 2024-03-24T10:37:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.845+0000 7f81b0ff9640 1 -- 172.21.15.23:0/1535878430 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f81a8005440 con 0x7f81b4101820 2024-03-24T10:37:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.845+0000 7f81b0ff9640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f819c0a20c0 0x7f819c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.846+0000 7f81b37fe640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f819c0a20c0 0x7f819c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.846+0000 7f81b0ff9640 1 -- 172.21.15.23:0/1535878430 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f81a80efbc0 con 0x7f81b4101820 2024-03-24T10:37:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.846+0000 7f81b37fe640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f819c0a20c0 0x7f819c0a4580 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f81b4101680 tx=0x7f81a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:54.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:54.850+0000 7f81b0ff9640 1 -- 172.21.15.23:0/1535878430 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f81a80b9030 con 0x7f81b4101820 2024-03-24T10:37:55.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:55.071+0000 7f81b9c4e640 1 -- 172.21.15.23:0/1535878430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81b410d390 con 0x7f81b4101820 2024-03-24T10:37:55.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:55.071+0000 7f81b0ff9640 1 -- 172.21.15.23:0/1535878430 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7f81a80bc6f0 con 0x7f81b4101820 2024-03-24T10:37:55.072 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:55.072 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-24T10:37:55.072 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 9 2024-03-24T10:37:55.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:55.073+0000 7f81b9c4e640 1 -- 172.21.15.23:0/1535878430 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f819c0a20c0 msgr2=0x7f819c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:55.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:55.073+0000 7f81b9c4e640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f819c0a20c0 0x7f819c0a4580 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f81b4101680 tx=0x7f81a4005f90 comp rx=0 tx=0).stop 2024-03-24T10:37:55.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:55.073+0000 7f81b9c4e640 1 -- 172.21.15.23:0/1535878430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81b4101820 msgr2=0x7f81b4197d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:55.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:55.073+0000 7f81b9c4e640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81b4101820 0x7f81b4197d10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f81a8006010 tx=0x7f81a80040c0 comp rx=0 tx=0).stop 2024-03-24T10:37:55.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:55.073+0000 7f81b9c4e640 1 -- 172.21.15.23:0/1535878430 shutdown_connections 2024-03-24T10:37:55.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:55.073+0000 7f81b9c4e640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f819c0a20c0 0x7f819c0a4580 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:55.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:55.073+0000 7f81b9c4e640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81b41021e0 0x7f81b419ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:55.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:55.073+0000 7f81b9c4e640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81b4101820 0x7f81b4197d10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:55.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:55.073+0000 7f81b9c4e640 1 --2- 172.21.15.23:0/1535878430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81b4100630 0x7f81b41977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:55.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:55.073+0000 7f81b9c4e640 1 -- 172.21.15.23:0/1535878430 >> 172.21.15.23:0/1535878430 conn(0x7f81b40fbba0 msgr2=0x7f81b4104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:55.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:55.074+0000 7f81b9c4e640 1 -- 172.21.15.23:0/1535878430 shutdown_connections 2024-03-24T10:37:55.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:55.074+0000 7f81b9c4e640 1 -- 172.21.15.23:0/1535878430 wait complete. 2024-03-24T10:37:55.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:55 smithi023 ceph-mon[32466]: Deploying daemon mds.i on smithi117 2024-03-24T10:37:55.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:55 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] up:boot 2024-03-24T10:37:55.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:55 smithi023 ceph-mon[32466]: fsmap 8 up:standby 2024-03-24T10:37:55.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:55 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T10:37:55.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:55 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1535878430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:55.543 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:55 smithi116 ceph-mon[36248]: Deploying daemon mds.i on smithi117 2024-03-24T10:37:55.543 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:55 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] up:boot 2024-03-24T10:37:55.544 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:55 smithi116 ceph-mon[36248]: fsmap 8 up:standby 2024-03-24T10:37:55.544 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:55 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T10:37:55.544 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:55 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1535878430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:55.609 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:55 smithi117 ceph-mon[36368]: Deploying daemon mds.i on smithi117 2024-03-24T10:37:55.609 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:55 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] up:boot 2024-03-24T10:37:55.610 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:55 smithi117 ceph-mon[36368]: fsmap 8 up:standby 2024-03-24T10:37:55.610 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:55 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T10:37:55.610 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:55 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1535878430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:56.253 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:37:56 smithi117 systemd[1]: Starting Ceph mds.i for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:37:56.590 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:56 smithi117 ceph-mon[36368]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:56.590 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:37:56 smithi117 podman[67427]: 2024-03-24 10:37:56.217685893 +0000 UTC m=+0.018036160 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:37:56.590 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:37:56 smithi117 podman[67427]: 2024-03-24 10:37:56.354198519 +0000 UTC m=+0.154548777 container create 467f37d8aace28a632957e863c6e78cee9d21e3ffeda81f33e3907eca4aa4743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-i, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:37:56.590 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:37:56 smithi117 podman[67427]: 2024-03-24 10:37:56.59010696 +0000 UTC m=+0.390457253 container init 467f37d8aace28a632957e863c6e78cee9d21e3ffeda81f33e3907eca4aa4743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-i, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T10:37:56.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:56 smithi023 ceph-mon[32466]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:56.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:56 smithi116 ceph-mon[36248]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:57.003 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:37:56 smithi117 podman[67427]: 2024-03-24 10:37:56.59311828 +0000 UTC m=+0.393468547 container start 467f37d8aace28a632957e863c6e78cee9d21e3ffeda81f33e3907eca4aa4743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-i, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:37:57.003 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:37:56 smithi117 ceph-mds[67446]: -- 172.21.15.117:0/1736519609 <== mon.1 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5649179cc1a0 con 0x5649179a3000 2024-03-24T10:37:57.003 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:37:56 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-i[67442]: starting mds.i at 2024-03-24T10:37:57.003 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:37:56 smithi117 bash[67427]: 467f37d8aace28a632957e863c6e78cee9d21e3ffeda81f33e3907eca4aa4743 2024-03-24T10:37:57.003 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:37:56 smithi117 systemd[1]: Started Ceph mds.i for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:37:57.086 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:37:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.190+0000 7f75dd895640 1 -- 172.21.15.23:0/3775607601 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d8100650 msgr2=0x7f75d8100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.190+0000 7f75dd895640 1 --2- 172.21.15.23:0/3775607601 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d8100650 0x7f75d8100a70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009900 tx=0x7f75c802bc30 comp rx=0 tx=0).stop 2024-03-24T10:37:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.190+0000 7f75dd895640 1 -- 172.21.15.23:0/3775607601 shutdown_connections 2024-03-24T10:37:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.190+0000 7f75dd895640 1 --2- 172.21.15.23:0/3775607601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d8102200 0x7f75d8108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.190+0000 7f75dd895640 1 --2- 172.21.15.23:0/3775607601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d8101840 0x7f75d8101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.190+0000 7f75dd895640 1 --2- 172.21.15.23:0/3775607601 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d8100650 0x7f75d8100a70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.190+0000 7f75dd895640 1 -- 172.21.15.23:0/3775607601 >> 172.21.15.23:0/3775607601 conn(0x7f75d80fbbc0 msgr2=0x7f75d80fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.190+0000 7f75dd895640 1 -- 172.21.15.23:0/3775607601 shutdown_connections 2024-03-24T10:37:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.190+0000 7f75dd895640 1 -- 172.21.15.23:0/3775607601 wait complete. 2024-03-24T10:37:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.191+0000 7f75dd895640 1 Processor -- start 2024-03-24T10:37:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.191+0000 7f75dd895640 1 -- start start 2024-03-24T10:37:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.191+0000 7f75dd895640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d8100650 0x7f75d8196730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.191+0000 7f75dd895640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d8101840 0x7f75d8196c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.191+0000 7f75dd895640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d8102200 0x7f75d819dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.191+0000 7f75dd895640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f75d819e220 con 0x7f75d8102200 2024-03-24T10:37:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.191+0000 7f75dd895640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f75d819e390 con 0x7f75d8100650 2024-03-24T10:37:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.191+0000 7f75dd895640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f75d8069af0 con 0x7f75d8101840 2024-03-24T10:37:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.191+0000 7f75d77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d8102200 0x7f75d819dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.191+0000 7f75d77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d8102200 0x7f75d819dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44490/0 (socket says 172.21.15.23:44490) 2024-03-24T10:37:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.191+0000 7f75d77fe640 1 -- 172.21.15.23:0/3224536090 learned_addr learned my addr 172.21.15.23:0/3224536090 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.191+0000 7f75d67fc640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d8101840 0x7f75d8196c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.191+0000 7f75d6ffd640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d8100650 0x7f75d8196730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.192+0000 7f75d67fc640 1 -- 172.21.15.23:0/3224536090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d8100650 msgr2=0x7f75d8196730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.192+0000 7f75d67fc640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d8100650 0x7f75d8196730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.192+0000 7f75d67fc640 1 -- 172.21.15.23:0/3224536090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d8102200 msgr2=0x7f75d819dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.192+0000 7f75d67fc640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d8102200 0x7f75d819dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.192+0000 7f75d67fc640 1 -- 172.21.15.23:0/3224536090 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c8009560 con 0x7f75d8101840 2024-03-24T10:37:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.192+0000 7f75d77fe640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d8102200 0x7f75d819dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:37:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.192+0000 7f75d67fc640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d8101840 0x7f75d8196c70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f75cc005fd0 tx=0x7f75cc00dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.192+0000 7f75dc893640 1 -- 172.21.15.23:0/3224536090 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc009c30 con 0x7f75d8101840 2024-03-24T10:37:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.192+0000 7f75dc893640 1 -- 172.21.15.23:0/3224536090 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75cc004510 con 0x7f75d8101840 2024-03-24T10:37:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.192+0000 7f75dd895640 1 -- 172.21.15.23:0/3224536090 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75d8069dd0 con 0x7f75d8101840 2024-03-24T10:37:57.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.192+0000 7f75dc893640 1 -- 172.21.15.23:0/3224536090 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc003e60 con 0x7f75d8101840 2024-03-24T10:37:57.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.192+0000 7f75dd895640 1 -- 172.21.15.23:0/3224536090 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75d806a330 con 0x7f75d8101840 2024-03-24T10:37:57.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.193+0000 7f75dd895640 1 -- 172.21.15.23:0/3224536090 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75d8107a30 con 0x7f75d8101840 2024-03-24T10:37:57.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.195+0000 7f75dc893640 1 -- 172.21.15.23:0/3224536090 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f75cc004000 con 0x7f75d8101840 2024-03-24T10:37:57.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.195+0000 7f75dc893640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75c00a2140 0x7f75c00a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:57.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.195+0000 7f75d6ffd640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75c00a2140 0x7f75c00a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:57.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.196+0000 7f75dc893640 1 -- 172.21.15.23:0/3224536090 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f75cc0cd260 con 0x7f75d8101840 2024-03-24T10:37:57.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.196+0000 7f75d6ffd640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75c00a2140 0x7f75c00a4600 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f75c8005ab0 tx=0x7f75c8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:57.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.198+0000 7f75dc893640 1 -- 172.21.15.23:0/3224536090 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f75cc096800 con 0x7f75d8101840 2024-03-24T10:37:57.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.428+0000 7f75dd895640 1 -- 172.21.15.23:0/3224536090 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75d810c2b0 con 0x7f75d8101840 2024-03-24T10:37:57.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.429+0000 7f75dc893640 1 -- 172.21.15.23:0/3224536090 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7f75cc020090 con 0x7f75d8101840 2024-03-24T10:37:57.429 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:57.429 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-24T10:37:57.429 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 10 2024-03-24T10:37:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.431+0000 7f75dd895640 1 -- 172.21.15.23:0/3224536090 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75c00a2140 msgr2=0x7f75c00a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.431+0000 7f75dd895640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75c00a2140 0x7f75c00a4600 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f75c8005ab0 tx=0x7f75c8005fd0 comp rx=0 tx=0).stop 2024-03-24T10:37:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.431+0000 7f75dd895640 1 -- 172.21.15.23:0/3224536090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d8101840 msgr2=0x7f75d8196c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.431+0000 7f75dd895640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d8101840 0x7f75d8196c70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f75cc005fd0 tx=0x7f75cc00dde0 comp rx=0 tx=0).stop 2024-03-24T10:37:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.431+0000 7f75dd895640 1 -- 172.21.15.23:0/3224536090 shutdown_connections 2024-03-24T10:37:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.431+0000 7f75dd895640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75c00a2140 0x7f75c00a4600 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.431+0000 7f75dd895640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d8102200 0x7f75d819dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.431+0000 7f75dd895640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d8101840 0x7f75d8196c70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.431+0000 7f75dd895640 1 --2- 172.21.15.23:0/3224536090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75d8100650 0x7f75d8196730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.431+0000 7f75dd895640 1 -- 172.21.15.23:0/3224536090 >> 172.21.15.23:0/3224536090 conn(0x7f75d80fbbc0 msgr2=0x7f75d8104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.431+0000 7f75dd895640 1 -- 172.21.15.23:0/3224536090 shutdown_connections 2024-03-24T10:37:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:57.431+0000 7f75dd895640 1 -- 172.21.15.23:0/3224536090 wait complete. 2024-03-24T10:37:57.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:57 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:57.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:57 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:57.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:57 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:57.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:57 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:57.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:57 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:57.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:57 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:57.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:57 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] up:boot 2024-03-24T10:37:57.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:57 smithi023 ceph-mon[32466]: fsmap 9 up:standby 2024-03-24T10:37:57.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:57 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T10:37:57.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:57 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3224536090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:58.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:57 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:58.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:57 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:58.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:57 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:58.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:57 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:58.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:57 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:58.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:57 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:58.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:57 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] up:boot 2024-03-24T10:37:58.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:57 smithi117 ceph-mon[36368]: fsmap 9 up:standby 2024-03-24T10:37:58.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:57 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T10:37:58.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:57 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3224536090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:58.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:57 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:58.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:57 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:58.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:57 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:37:58.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:57 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:37:58.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:57 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:37:58.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:57 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:37:58.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:57 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] up:boot 2024-03-24T10:37:58.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:57 smithi116 ceph-mon[36248]: fsmap 9 up:standby 2024-03-24T10:37:58.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:57 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T10:37:58.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:57 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3224536090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:37:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:58 smithi023 ceph-mon[32466]: Deploying daemon mds.g on smithi023 2024-03-24T10:37:58.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:37:58 smithi023 ceph-mon[32466]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:58.919 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:37:58 smithi023 systemd[1]: Starting Ceph mds.g for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:37:59.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:58 smithi117 ceph-mon[36368]: Deploying daemon mds.g on smithi023 2024-03-24T10:37:59.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:37:58 smithi117 ceph-mon[36368]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:59.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:58 smithi116 ceph-mon[36248]: Deploying daemon mds.g on smithi023 2024-03-24T10:37:59.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:37:58 smithi116 ceph-mon[36248]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:37:59.318 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:37:59 smithi023 podman[71318]: 2024-03-24 10:37:59.056876366 +0000 UTC m=+0.106222011 container create 399d431e658793bea2efb19c8f2a47880a3fde663f484d1c54d831edfc952d70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-24T10:37:59.318 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:37:59 smithi023 podman[71318]: 2024-03-24 10:37:58.968373693 +0000 UTC m=+0.017719346 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:37:59.318 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:37:59 smithi023 podman[71318]: 2024-03-24 10:37:59.292626201 +0000 UTC m=+0.341971846 container init 399d431e658793bea2efb19c8f2a47880a3fde663f484d1c54d831edfc952d70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-g, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T10:37:59.319 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:37:59 smithi023 podman[71318]: 2024-03-24 10:37:59.295099295 +0000 UTC m=+0.344444947 container start 399d431e658793bea2efb19c8f2a47880a3fde663f484d1c54d831edfc952d70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-24T10:37:59.319 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:37:59 smithi023 ceph-mds[71337]: -- 172.21.15.23:0/659336675 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eb63c821a0 con 0x55eb63c58c00 2024-03-24T10:37:59.444 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:37:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.553+0000 7fcf76113640 1 -- 172.21.15.23:0/4234476512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf70101820 msgr2=0x7fcf70101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.553+0000 7fcf76113640 1 --2- 172.21.15.23:0/4234476512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf70101820 0x7fcf70101ca0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009900 tx=0x7fcf6002bc30 comp rx=0 tx=0).stop 2024-03-24T10:37:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.553+0000 7fcf76113640 1 -- 172.21.15.23:0/4234476512 shutdown_connections 2024-03-24T10:37:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.553+0000 7fcf76113640 1 --2- 172.21.15.23:0/4234476512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf701021e0 0x7fcf70108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.553+0000 7fcf76113640 1 --2- 172.21.15.23:0/4234476512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf70101820 0x7fcf70101ca0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.553+0000 7fcf76113640 1 --2- 172.21.15.23:0/4234476512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf70100630 0x7fcf70100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.553+0000 7fcf76113640 1 -- 172.21.15.23:0/4234476512 >> 172.21.15.23:0/4234476512 conn(0x7fcf700fbba0 msgr2=0x7fcf700fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.553+0000 7fcf76113640 1 -- 172.21.15.23:0/4234476512 shutdown_connections 2024-03-24T10:37:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.553+0000 7fcf76113640 1 -- 172.21.15.23:0/4234476512 wait complete. 2024-03-24T10:37:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf76113640 1 Processor -- start 2024-03-24T10:37:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf76113640 1 -- start start 2024-03-24T10:37:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf76113640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf70100630 0x7fcf701045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf76113640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf70101820 0x7fcf70102c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf76113640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf701021e0 0x7fcf70103140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf76113640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fcf701038a0 con 0x7fcf70100630 2024-03-24T10:37:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf76113640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcf70103a10 con 0x7fcf70101820 2024-03-24T10:37:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf76113640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcf701a1810 con 0x7fcf701021e0 2024-03-24T10:37:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf6f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf70100630 0x7fcf701045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf6f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf70100630 0x7fcf701045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59402/0 (socket says 172.21.15.23:59402) 2024-03-24T10:37:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf6f7fe640 1 -- 172.21.15.23:0/283935805 learned_addr learned my addr 172.21.15.23:0/283935805 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:37:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf6ffff640 1 --2- 172.21.15.23:0/283935805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf701021e0 0x7fcf70103140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf6effd640 1 --2- 172.21.15.23:0/283935805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf70101820 0x7fcf70102c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf6f7fe640 1 -- 172.21.15.23:0/283935805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf701021e0 msgr2=0x7fcf70103140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf6f7fe640 1 --2- 172.21.15.23:0/283935805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf701021e0 0x7fcf70103140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf6f7fe640 1 -- 172.21.15.23:0/283935805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf70101820 msgr2=0x7fcf70102c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf6f7fe640 1 --2- 172.21.15.23:0/283935805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf70101820 0x7fcf70102c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.554+0000 7fcf6f7fe640 1 -- 172.21.15.23:0/283935805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf60009560 con 0x7fcf70100630 2024-03-24T10:37:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.555+0000 7fcf6f7fe640 1 --2- 172.21.15.23:0/283935805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf70100630 0x7fcf701045c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcf6400e9f0 tx=0x7fcf6400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.555+0000 7fcf6cff9640 1 -- 172.21.15.23:0/283935805 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf640096e0 con 0x7fcf70100630 2024-03-24T10:37:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.555+0000 7fcf76113640 1 -- 172.21.15.23:0/283935805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf701a1ac0 con 0x7fcf70100630 2024-03-24T10:37:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.555+0000 7fcf6cff9640 1 -- 172.21.15.23:0/283935805 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf6400cd80 con 0x7fcf70100630 2024-03-24T10:37:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.555+0000 7fcf6cff9640 1 -- 172.21.15.23:0/283935805 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf64010430 con 0x7fcf70100630 2024-03-24T10:37:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.555+0000 7fcf76113640 1 -- 172.21.15.23:0/283935805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf701a1fa0 con 0x7fcf70100630 2024-03-24T10:37:59.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.556+0000 7fcf76113640 1 -- 172.21.15.23:0/283935805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf7010c040 con 0x7fcf70100630 2024-03-24T10:37:59.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.557+0000 7fcf6cff9640 1 -- 172.21.15.23:0/283935805 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fcf64003680 con 0x7fcf70100630 2024-03-24T10:37:59.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.557+0000 7fcf6cff9640 1 --2- 172.21.15.23:0/283935805 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcf580a20c0 0x7fcf580a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:37:59.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.558+0000 7fcf6effd640 1 --2- 172.21.15.23:0/283935805 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcf580a20c0 0x7fcf580a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:37:59.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.558+0000 7fcf6cff9640 1 -- 172.21.15.23:0/283935805 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcf64014070 con 0x7fcf70100630 2024-03-24T10:37:59.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.558+0000 7fcf6effd640 1 --2- 172.21.15.23:0/283935805 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcf580a20c0 0x7fcf580a4580 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009230 tx=0x7fcf60005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:37:59.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.562+0000 7fcf6cff9640 1 -- 172.21.15.23:0/283935805 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcf64096fd0 con 0x7fcf70100630 2024-03-24T10:37:59.668 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:37:59 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-g[71333]: starting mds.g at 2024-03-24T10:37:59.668 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:37:59 smithi023 bash[71318]: 399d431e658793bea2efb19c8f2a47880a3fde663f484d1c54d831edfc952d70 2024-03-24T10:37:59.668 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:37:59 smithi023 systemd[1]: Started Ceph mds.g for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:37:59.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.787+0000 7fcf76113640 1 -- 172.21.15.23:0/283935805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf7010c1d0 con 0x7fcf70100630 2024-03-24T10:37:59.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.788+0000 7fcf6cff9640 1 -- 172.21.15.23:0/283935805 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7fcf6409a690 con 0x7fcf70100630 2024-03-24T10:37:59.788 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:37:59.789 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-24T10:37:59.789 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 11 2024-03-24T10:37:59.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.790+0000 7fcf76113640 1 -- 172.21.15.23:0/283935805 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcf580a20c0 msgr2=0x7fcf580a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:59.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.790+0000 7fcf76113640 1 --2- 172.21.15.23:0/283935805 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcf580a20c0 0x7fcf580a4580 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009230 tx=0x7fcf60005fd0 comp rx=0 tx=0).stop 2024-03-24T10:37:59.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.790+0000 7fcf76113640 1 -- 172.21.15.23:0/283935805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf70100630 msgr2=0x7fcf701045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:37:59.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.790+0000 7fcf76113640 1 --2- 172.21.15.23:0/283935805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf70100630 0x7fcf701045c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcf6400e9f0 tx=0x7fcf6400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:37:59.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.790+0000 7fcf76113640 1 -- 172.21.15.23:0/283935805 shutdown_connections 2024-03-24T10:37:59.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.790+0000 7fcf76113640 1 --2- 172.21.15.23:0/283935805 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcf580a20c0 0x7fcf580a4580 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:59.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.790+0000 7fcf76113640 1 --2- 172.21.15.23:0/283935805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf701021e0 0x7fcf70103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:59.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.790+0000 7fcf76113640 1 --2- 172.21.15.23:0/283935805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf70101820 0x7fcf70102c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:59.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.790+0000 7fcf76113640 1 --2- 172.21.15.23:0/283935805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf70100630 0x7fcf701045c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:37:59.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.790+0000 7fcf76113640 1 -- 172.21.15.23:0/283935805 >> 172.21.15.23:0/283935805 conn(0x7fcf700fbba0 msgr2=0x7fcf700fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:37:59.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.790+0000 7fcf76113640 1 -- 172.21.15.23:0/283935805 shutdown_connections 2024-03-24T10:37:59.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:37:59.790+0000 7fcf76113640 1 -- 172.21.15.23:0/283935805 wait complete. 2024-03-24T10:38:00.613 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:00 smithi023 ceph-mon[32466]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:00.613 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:00.613 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:00.613 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:00.613 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:38:00.613 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:38:00.613 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:00.613 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:00 smithi023 ceph-mon[32466]: Deploying daemon mds.a on smithi023 2024-03-24T10:38:00.613 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:00 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.23:6838/4279444411,v1:172.21.15.23:6839/4279444411] up:boot 2024-03-24T10:38:00.613 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:00 smithi023 ceph-mon[32466]: fsmap 10 up:standby 2024-03-24T10:38:00.614 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:00 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T10:38:00.614 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:00 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/283935805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:00.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:00 smithi117 ceph-mon[36368]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:00.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:00.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:00.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:00.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:38:00.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:38:00.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:00.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:00 smithi117 ceph-mon[36368]: Deploying daemon mds.a on smithi023 2024-03-24T10:38:00.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:00 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.23:6838/4279444411,v1:172.21.15.23:6839/4279444411] up:boot 2024-03-24T10:38:00.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:00 smithi117 ceph-mon[36368]: fsmap 10 up:standby 2024-03-24T10:38:00.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:00 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T10:38:00.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:00 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/283935805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:00.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:00 smithi116 ceph-mon[36248]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:00.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:00.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:00.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:00.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:38:00.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:38:00.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:00.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:00 smithi116 ceph-mon[36248]: Deploying daemon mds.a on smithi023 2024-03-24T10:38:00.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:00 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.23:6838/4279444411,v1:172.21.15.23:6839/4279444411] up:boot 2024-03-24T10:38:00.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:00 smithi116 ceph-mon[36248]: fsmap 10 up:standby 2024-03-24T10:38:00.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:00 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T10:38:00.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:00 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/283935805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:01.668 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:38:01 smithi023 systemd[1]: Starting Ceph mds.a for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:38:01.804 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:38:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.916+0000 7f5516725640 1 -- 172.21.15.23:0/1992331031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5510100630 msgr2=0x7f5510100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.916+0000 7f5516725640 1 --2- 172.21.15.23:0/1992331031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5510100630 0x7f5510100a50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5500009900 tx=0x7f550002bc30 comp rx=0 tx=0).stop 2024-03-24T10:38:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.916+0000 7f5516725640 1 -- 172.21.15.23:0/1992331031 shutdown_connections 2024-03-24T10:38:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.916+0000 7f5516725640 1 --2- 172.21.15.23:0/1992331031 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55101021e0 0x7f5510108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.916+0000 7f5516725640 1 --2- 172.21.15.23:0/1992331031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5510101820 0x7f5510101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.916+0000 7f5516725640 1 --2- 172.21.15.23:0/1992331031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5510100630 0x7f5510100a50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.916+0000 7f5516725640 1 -- 172.21.15.23:0/1992331031 >> 172.21.15.23:0/1992331031 conn(0x7f55100fbba0 msgr2=0x7f55100fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.916+0000 7f5516725640 1 -- 172.21.15.23:0/1992331031 shutdown_connections 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.916+0000 7f5516725640 1 -- 172.21.15.23:0/1992331031 wait complete. 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.916+0000 7f5516725640 1 Processor -- start 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f5516725640 1 -- start start 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f5516725640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5510100630 0x7f5510197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f5516725640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5510101820 0x7f5510197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f5516725640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55101021e0 0x7f551019ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f5516725640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f551019f350 con 0x7f5510100630 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f5516725640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f551019f4c0 con 0x7f55101021e0 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f5516725640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f551019f680 con 0x7f5510101820 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f550ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5510100630 0x7f5510197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f550ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5510100630 0x7f5510197860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59418/0 (socket says 172.21.15.23:59418) 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f550ffff640 1 -- 172.21.15.23:0/2237387010 learned_addr learned my addr 172.21.15.23:0/2237387010 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f5514cc2640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55101021e0 0x7f551019ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f550f7fe640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5510101820 0x7f5510197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:01.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f550ffff640 1 -- 172.21.15.23:0/2237387010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5510101820 msgr2=0x7f5510197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:01.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f550ffff640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5510101820 0x7f5510197da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:01.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f550ffff640 1 -- 172.21.15.23:0/2237387010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55101021e0 msgr2=0x7f551019ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:01.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f550ffff640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55101021e0 0x7f551019ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:01.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f550ffff640 1 -- 172.21.15.23:0/2237387010 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5500009560 con 0x7f5510100630 2024-03-24T10:38:01.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.917+0000 7f5514cc2640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55101021e0 0x7f551019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:38:01.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.918+0000 7f550ffff640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5510100630 0x7f5510197860 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5500005ab0 tx=0x7f55000042d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:01.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.918+0000 7f550d7fa640 1 -- 172.21.15.23:0/2237387010 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55000044c0 con 0x7f5510100630 2024-03-24T10:38:01.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.918+0000 7f550d7fa640 1 -- 172.21.15.23:0/2237387010 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55000054e0 con 0x7f5510100630 2024-03-24T10:38:01.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.918+0000 7f550d7fa640 1 -- 172.21.15.23:0/2237387010 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550003e450 con 0x7f5510100630 2024-03-24T10:38:01.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.918+0000 7f5516725640 1 -- 172.21.15.23:0/2237387010 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f551019f900 con 0x7f5510100630 2024-03-24T10:38:01.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.918+0000 7f5516725640 1 -- 172.21.15.23:0/2237387010 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f551019fe10 con 0x7f5510100630 2024-03-24T10:38:01.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.919+0000 7f5516725640 1 -- 172.21.15.23:0/2237387010 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f551010c090 con 0x7f5510100630 2024-03-24T10:38:01.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.920+0000 7f550d7fa640 1 -- 172.21.15.23:0/2237387010 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f5500003780 con 0x7f5510100630 2024-03-24T10:38:01.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.920+0000 7f550d7fa640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f54f80a20c0 0x7f54f80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:01.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.920+0000 7f550f7fe640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f54f80a20c0 0x7f54f80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:01.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.920+0000 7f550d7fa640 1 -- 172.21.15.23:0/2237387010 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f55000f0330 con 0x7f5510100630 2024-03-24T10:38:01.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.921+0000 7f550f7fe640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f54f80a20c0 0x7f54f80a4580 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5510198d70 tx=0x7f550400a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:01.925+0000 7f550d7fa640 1 -- 172.21.15.23:0/2237387010 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f55000b97a0 con 0x7f5510100630 2024-03-24T10:38:01.973 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:38:01 smithi023 podman[71677]: 2024-03-24 10:38:01.615641074 +0000 UTC m=+0.017518068 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:38:01.973 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:38:01 smithi023 podman[71677]: 2024-03-24 10:38:01.752658916 +0000 UTC m=+0.154535911 container create 21230139e0cf1ca90d02679a06ac64724b8d21ce98b577d048dffc902b23d358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-a, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0) 2024-03-24T10:38:01.973 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:38:01 smithi023 podman[71677]: 2024-03-24 10:38:01.946997569 +0000 UTC m=+0.348874565 container init 21230139e0cf1ca90d02679a06ac64724b8d21ce98b577d048dffc902b23d358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-03-24T10:38:01.973 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:38:01 smithi023 podman[71677]: 2024-03-24 10:38:01.949649715 +0000 UTC m=+0.351526717 container start 21230139e0cf1ca90d02679a06ac64724b8d21ce98b577d048dffc902b23d358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:38:02.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:02.150+0000 7f5516725640 1 -- 172.21.15.23:0/2237387010 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f551010c270 con 0x7f5510100630 2024-03-24T10:38:02.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:02.150+0000 7f550d7fa640 1 -- 172.21.15.23:0/2237387010 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7f55000bce60 con 0x7f5510100630 2024-03-24T10:38:02.151 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:38:02.151 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-24T10:38:02.151 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 11 2024-03-24T10:38:02.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:02.152+0000 7f5516725640 1 -- 172.21.15.23:0/2237387010 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f54f80a20c0 msgr2=0x7f54f80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:02.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:02.152+0000 7f5516725640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f54f80a20c0 0x7f54f80a4580 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5510198d70 tx=0x7f550400a4c0 comp rx=0 tx=0).stop 2024-03-24T10:38:02.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:02.152+0000 7f5516725640 1 -- 172.21.15.23:0/2237387010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5510100630 msgr2=0x7f5510197860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:02.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:02.152+0000 7f5516725640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5510100630 0x7f5510197860 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5500005ab0 tx=0x7f55000042d0 comp rx=0 tx=0).stop 2024-03-24T10:38:02.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:02.152+0000 7f5516725640 1 -- 172.21.15.23:0/2237387010 shutdown_connections 2024-03-24T10:38:02.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:02.152+0000 7f5516725640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f54f80a20c0 0x7f54f80a4580 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:02.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:02.152+0000 7f5516725640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55101021e0 0x7f551019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:02.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:02.152+0000 7f5516725640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5510101820 0x7f5510197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:02.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:02.152+0000 7f5516725640 1 --2- 172.21.15.23:0/2237387010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5510100630 0x7f5510197860 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:02.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:02.152+0000 7f5516725640 1 -- 172.21.15.23:0/2237387010 >> 172.21.15.23:0/2237387010 conn(0x7f55100fbba0 msgr2=0x7f55100fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:02.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:02.152+0000 7f5516725640 1 -- 172.21.15.23:0/2237387010 shutdown_connections 2024-03-24T10:38:02.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:02.152+0000 7f5516725640 1 -- 172.21.15.23:0/2237387010 wait complete. 2024-03-24T10:38:02.418 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:38:01 smithi023 ceph-mds[71735]: -- 172.21.15.23:0/3812442149 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c726b441a0 con 0x55c726b1a800 2024-03-24T10:38:02.418 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:38:01 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-a[71711]: starting mds.a at 2024-03-24T10:38:02.418 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:38:02 smithi023 bash[71677]: 21230139e0cf1ca90d02679a06ac64724b8d21ce98b577d048dffc902b23d358 2024-03-24T10:38:02.418 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:38:02 smithi023 systemd[1]: Started Ceph mds.a for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:38:02.726 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:02 smithi116 ceph-mon[36248]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:02.726 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:02 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:02.726 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:02 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:02.726 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:02 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:02.727 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:02 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:38:02.727 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:02 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:38:02.727 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:02 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:02.727 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:02 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2237387010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:02.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:02 smithi117 ceph-mon[36368]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:02.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:02 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:02.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:02 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:02.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:02 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:02.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:02 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:38:02.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:02 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:38:02.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:02 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:02.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:02 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2237387010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:02 smithi023 ceph-mon[32466]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:02 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:02 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:02 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:02.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:02 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T10:38:02.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:02 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T10:38:02.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:02 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:02.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:02 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2237387010' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:03.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:03 smithi117 ceph-mon[36368]: Deploying daemon mds.b on smithi116 2024-03-24T10:38:03.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:03 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.23:6840/644733222,v1:172.21.15.23:6841/644733222] up:boot 2024-03-24T10:38:03.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:03 smithi117 ceph-mon[36368]: fsmap 11 up:standby 2024-03-24T10:38:03.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:03 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T10:38:03.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:03 smithi116 ceph-mon[36248]: Deploying daemon mds.b on smithi116 2024-03-24T10:38:03.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:03 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.23:6840/644733222,v1:172.21.15.23:6841/644733222] up:boot 2024-03-24T10:38:03.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:03 smithi116 ceph-mon[36248]: fsmap 11 up:standby 2024-03-24T10:38:03.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:03 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T10:38:03.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:03 smithi023 ceph-mon[32466]: Deploying daemon mds.b on smithi116 2024-03-24T10:38:03.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:03 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.23:6840/644733222,v1:172.21.15.23:6841/644733222] up:boot 2024-03-24T10:38:03.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:03 smithi023 ceph-mon[32466]: fsmap 11 up:standby 2024-03-24T10:38:03.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:03 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T10:38:04.165 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:38:04.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.280+0000 7f95b04c0640 1 -- 172.21.15.23:0/2812077332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95a80ffde0 msgr2=0x7f95a8100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:04.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.280+0000 7f95b04c0640 1 --2- 172.21.15.23:0/2812077332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95a80ffde0 0x7f95a8100200 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9598009900 tx=0x7f959802bc30 comp rx=0 tx=0).stop 2024-03-24T10:38:04.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.280+0000 7f95b04c0640 1 -- 172.21.15.23:0/2812077332 shutdown_connections 2024-03-24T10:38:04.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.280+0000 7f95b04c0640 1 --2- 172.21.15.23:0/2812077332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95a8101990 0x7f95a8108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:04.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.280+0000 7f95b04c0640 1 --2- 172.21.15.23:0/2812077332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8100fd0 0x7f95a8101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:04.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.280+0000 7f95b04c0640 1 --2- 172.21.15.23:0/2812077332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95a80ffde0 0x7f95a8100200 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:04.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.280+0000 7f95b04c0640 1 -- 172.21.15.23:0/2812077332 >> 172.21.15.23:0/2812077332 conn(0x7f95a80fb350 msgr2=0x7f95a80fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:04.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.280+0000 7f95b04c0640 1 -- 172.21.15.23:0/2812077332 shutdown_connections 2024-03-24T10:38:04.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.280+0000 7f95b04c0640 1 -- 172.21.15.23:0/2812077332 wait complete. 2024-03-24T10:38:04.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.280+0000 7f95b04c0640 1 Processor -- start 2024-03-24T10:38:04.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.280+0000 7f95b04c0640 1 -- start start 2024-03-24T10:38:04.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95b04c0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95a80ffde0 0x7f95a8197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:04.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95b04c0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95a8100fd0 0x7f95a8197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:04.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95b04c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8101990 0x7f95a819ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:04.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95b04c0640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f95a819f320 con 0x7f95a80ffde0 2024-03-24T10:38:04.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95b04c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f95a819f490 con 0x7f95a8101990 2024-03-24T10:38:04.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95b04c0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f95a819f650 con 0x7f95a8100fd0 2024-03-24T10:38:04.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95ae25c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95a80ffde0 0x7f95a8197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95ae25c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95a80ffde0 0x7f95a8197830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59456/0 (socket says 172.21.15.23:59456) 2024-03-24T10:38:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95ae25c640 1 -- 172.21.15.23:0/464701419 learned_addr learned my addr 172.21.15.23:0/464701419 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95aea5d640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8101990 0x7f95a819ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95ada5b640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95a8100fd0 0x7f95a8197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95ae25c640 1 -- 172.21.15.23:0/464701419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95a8100fd0 msgr2=0x7f95a8197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95ae25c640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95a8100fd0 0x7f95a8197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95ae25c640 1 -- 172.21.15.23:0/464701419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8101990 msgr2=0x7f95a819ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95ae25c640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8101990 0x7f95a819ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95ae25c640 1 -- 172.21.15.23:0/464701419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9598009560 con 0x7f95a80ffde0 2024-03-24T10:38:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95ada5b640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95a8100fd0 0x7f95a8197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:38:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95ae25c640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95a80ffde0 0x7f95a8197830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9598000c00 tx=0x7f95980042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f959f7fe640 1 -- 172.21.15.23:0/464701419 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95980044e0 con 0x7f95a80ffde0 2024-03-24T10:38:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f959f7fe640 1 -- 172.21.15.23:0/464701419 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95980054e0 con 0x7f95a80ffde0 2024-03-24T10:38:04.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f959f7fe640 1 -- 172.21.15.23:0/464701419 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959803e450 con 0x7f95a80ffde0 2024-03-24T10:38:04.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95b04c0640 1 -- 172.21.15.23:0/464701419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95a819f8d0 con 0x7f95a80ffde0 2024-03-24T10:38:04.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.281+0000 7f95b04c0640 1 -- 172.21.15.23:0/464701419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a819fde0 con 0x7f95a80ffde0 2024-03-24T10:38:04.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.282+0000 7f95b04c0640 1 -- 172.21.15.23:0/464701419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95a8107100 con 0x7f95a80ffde0 2024-03-24T10:38:04.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.283+0000 7f959f7fe640 1 -- 172.21.15.23:0/464701419 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f95980035f0 con 0x7f95a80ffde0 2024-03-24T10:38:04.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.284+0000 7f959f7fe640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f95940a20c0 0x7f95940a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:04.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.284+0000 7f95ada5b640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f95940a20c0 0x7f95940a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:04.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.284+0000 7f959f7fe640 1 -- 172.21.15.23:0/464701419 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f95980f00f0 con 0x7f95a80ffde0 2024-03-24T10:38:04.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.285+0000 7f95ada5b640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f95940a20c0 0x7f95940a4580 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f95a4005d00 tx=0x7f95a4005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:04.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.288+0000 7f959f7fe640 1 -- 172.21.15.23:0/464701419 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f95980b9560 con 0x7f95a80ffde0 2024-03-24T10:38:04.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.509+0000 7f95b04c0640 1 -- 172.21.15.23:0/464701419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95a8106b70 con 0x7f95a80ffde0 2024-03-24T10:38:04.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.510+0000 7f959f7fe640 1 -- 172.21.15.23:0/464701419 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7f95980bcc20 con 0x7f95a80ffde0 2024-03-24T10:38:04.511 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:38:04.511 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-24T10:38:04.511 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 12 2024-03-24T10:38:04.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.512+0000 7f95b04c0640 1 -- 172.21.15.23:0/464701419 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f95940a20c0 msgr2=0x7f95940a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:04.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.512+0000 7f95b04c0640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f95940a20c0 0x7f95940a4580 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f95a4005d00 tx=0x7f95a4005ed0 comp rx=0 tx=0).stop 2024-03-24T10:38:04.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.512+0000 7f95b04c0640 1 -- 172.21.15.23:0/464701419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95a80ffde0 msgr2=0x7f95a8197830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:04.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.512+0000 7f95b04c0640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95a80ffde0 0x7f95a8197830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9598000c00 tx=0x7f95980042f0 comp rx=0 tx=0).stop 2024-03-24T10:38:04.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.512+0000 7f95b04c0640 1 -- 172.21.15.23:0/464701419 shutdown_connections 2024-03-24T10:38:04.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.512+0000 7f95b04c0640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f95940a20c0 0x7f95940a4580 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:04.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.512+0000 7f95b04c0640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95a8101990 0x7f95a819ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:04.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.512+0000 7f95b04c0640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95a8100fd0 0x7f95a8197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:04.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.512+0000 7f95b04c0640 1 --2- 172.21.15.23:0/464701419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95a80ffde0 0x7f95a8197830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:04.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.512+0000 7f95b04c0640 1 -- 172.21.15.23:0/464701419 >> 172.21.15.23:0/464701419 conn(0x7f95a80fb350 msgr2=0x7f95a81042d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:04.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.512+0000 7f95b04c0640 1 -- 172.21.15.23:0/464701419 shutdown_connections 2024-03-24T10:38:04.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:04.512+0000 7f95b04c0640 1 -- 172.21.15.23:0/464701419 wait complete. 2024-03-24T10:38:04.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:04 smithi116 ceph-mon[36248]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:04 smithi023 ceph-mon[32466]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:04.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:04 smithi117 ceph-mon[36368]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:05.268 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:38:04 smithi116 systemd[1]: Started Ceph mds.b for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:38:05.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:05 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/464701419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:05.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:05 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:05 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:05 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:05 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:05 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:05 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:05.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:05 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/464701419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:05.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:05 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:05 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:05 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:05 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:05 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:05 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:05 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/464701419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:05 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:05 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:05 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:05 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:05 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:05.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:05 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:06.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:06 smithi117 ceph-mon[36368]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:06.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:06 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] up:boot 2024-03-24T10:38:06.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:06 smithi117 ceph-mon[36368]: fsmap 12 up:standby 2024-03-24T10:38:06.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:06 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T10:38:06.524 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:38:06.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.637+0000 7f3a1a02a640 1 -- 172.21.15.23:0/2123841904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a14101820 msgr2=0x7f3a14101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:06.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.637+0000 7f3a1a02a640 1 --2- 172.21.15.23:0/2123841904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a14101820 0x7f3a14101ca0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009900 tx=0x7f3a0802bc30 comp rx=0 tx=0).stop 2024-03-24T10:38:06.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.638+0000 7f3a1a02a640 1 -- 172.21.15.23:0/2123841904 shutdown_connections 2024-03-24T10:38:06.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.638+0000 7f3a1a02a640 1 --2- 172.21.15.23:0/2123841904 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a141021e0 0x7f3a14108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:06.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.638+0000 7f3a1a02a640 1 --2- 172.21.15.23:0/2123841904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a14101820 0x7f3a14101ca0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:06.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.638+0000 7f3a1a02a640 1 --2- 172.21.15.23:0/2123841904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a14100630 0x7f3a14100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:06.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.638+0000 7f3a1a02a640 1 -- 172.21.15.23:0/2123841904 >> 172.21.15.23:0/2123841904 conn(0x7f3a140fbba0 msgr2=0x7f3a140fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:06.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.638+0000 7f3a1a02a640 1 -- 172.21.15.23:0/2123841904 shutdown_connections 2024-03-24T10:38:06.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.638+0000 7f3a1a02a640 1 -- 172.21.15.23:0/2123841904 wait complete. 2024-03-24T10:38:06.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.638+0000 7f3a1a02a640 1 Processor -- start 2024-03-24T10:38:06.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.638+0000 7f3a1a02a640 1 -- start start 2024-03-24T10:38:06.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a1a02a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a14100630 0x7f3a141975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:06.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a1a02a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a14101820 0x7f3a14197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:06.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a1a02a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a141021e0 0x7f3a1419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:06.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a1a02a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1419f0e0 con 0x7f3a14101820 2024-03-24T10:38:06.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a1a02a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1419f250 con 0x7f3a14100630 2024-03-24T10:38:06.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a1a02a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1419f410 con 0x7f3a141021e0 2024-03-24T10:38:06.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a12ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a14101820 0x7f3a14197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:06.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a13fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a141021e0 0x7f3a1419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:06.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a12ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a14101820 0x7f3a14197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59464/0 (socket says 172.21.15.23:59464) 2024-03-24T10:38:06.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a12ffd640 1 -- 172.21.15.23:0/3547690640 learned_addr learned my addr 172.21.15.23:0/3547690640 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:06.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a137fe640 1 --2- 172.21.15.23:0/3547690640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a14100630 0x7f3a141975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:06.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a12ffd640 1 -- 172.21.15.23:0/3547690640 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a141021e0 msgr2=0x7f3a1419eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:06.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a12ffd640 1 --2- 172.21.15.23:0/3547690640 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a141021e0 0x7f3a1419eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:06.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a12ffd640 1 -- 172.21.15.23:0/3547690640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a14100630 msgr2=0x7f3a141975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:06.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a12ffd640 1 --2- 172.21.15.23:0/3547690640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a14100630 0x7f3a141975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:06.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a12ffd640 1 -- 172.21.15.23:0/3547690640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a08009560 con 0x7f3a14101820 2024-03-24T10:38:06.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a12ffd640 1 --2- 172.21.15.23:0/3547690640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a14101820 0x7f3a14197b30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009850 tx=0x7f3a080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:06.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a10ff9640 1 -- 172.21.15.23:0/3547690640 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a080044e0 con 0x7f3a14101820 2024-03-24T10:38:06.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.639+0000 7f3a1a02a640 1 -- 172.21.15.23:0/3547690640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a1419f690 con 0x7f3a14101820 2024-03-24T10:38:06.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.640+0000 7f3a10ff9640 1 -- 172.21.15.23:0/3547690640 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a080054e0 con 0x7f3a14101820 2024-03-24T10:38:06.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.640+0000 7f3a10ff9640 1 -- 172.21.15.23:0/3547690640 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0803e6b0 con 0x7f3a14101820 2024-03-24T10:38:06.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.640+0000 7f3a1a02a640 1 -- 172.21.15.23:0/3547690640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a1419fbf0 con 0x7f3a14101820 2024-03-24T10:38:06.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.641+0000 7f3a1a02a640 1 -- 172.21.15.23:0/3547690640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a1410c090 con 0x7f3a14101820 2024-03-24T10:38:06.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.642+0000 7f3a10ff9640 1 -- 172.21.15.23:0/3547690640 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f3a080037d0 con 0x7f3a14101820 2024-03-24T10:38:06.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.642+0000 7f3a10ff9640 1 --2- 172.21.15.23:0/3547690640 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f39fc0a20c0 0x7f39fc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:06.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.643+0000 7f3a137fe640 1 --2- 172.21.15.23:0/3547690640 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f39fc0a20c0 0x7f39fc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:06.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.643+0000 7f3a10ff9640 1 -- 172.21.15.23:0/3547690640 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3a080f0310 con 0x7f3a14101820 2024-03-24T10:38:06.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.643+0000 7f3a137fe640 1 --2- 172.21.15.23:0/3547690640 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f39fc0a20c0 0x7f39fc0a4580 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3a14101680 tx=0x7f3a0400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:06.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.647+0000 7f3a10ff9640 1 -- 172.21.15.23:0/3547690640 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3a080b9760 con 0x7f3a14101820 2024-03-24T10:38:06.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:06 smithi116 ceph-mon[36248]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:06.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:06 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] up:boot 2024-03-24T10:38:06.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:06 smithi116 ceph-mon[36248]: fsmap 12 up:standby 2024-03-24T10:38:06.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:06 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T10:38:06.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.899+0000 7f3a1a02a640 1 -- 172.21.15.23:0/3547690640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a1419ff40 con 0x7f3a14101820 2024-03-24T10:38:06.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.900+0000 7f3a10ff9640 1 -- 172.21.15.23:0/3547690640 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10479 (secure 0 0 0) 0x7f3a080bce20 con 0x7f3a14101820 2024-03-24T10:38:06.900 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:38:06.900 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24505,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-24T10:38:06.901 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 13 2024-03-24T10:38:06.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.902+0000 7f3a1a02a640 1 -- 172.21.15.23:0/3547690640 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f39fc0a20c0 msgr2=0x7f39fc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.902+0000 7f3a1a02a640 1 --2- 172.21.15.23:0/3547690640 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f39fc0a20c0 0x7f39fc0a4580 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3a14101680 tx=0x7f3a0400b390 comp rx=0 tx=0).stop 2024-03-24T10:38:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.902+0000 7f3a1a02a640 1 -- 172.21.15.23:0/3547690640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a14101820 msgr2=0x7f3a14197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.902+0000 7f3a1a02a640 1 --2- 172.21.15.23:0/3547690640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a14101820 0x7f3a14197b30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009850 tx=0x7f3a080042f0 comp rx=0 tx=0).stop 2024-03-24T10:38:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.902+0000 7f3a1a02a640 1 -- 172.21.15.23:0/3547690640 shutdown_connections 2024-03-24T10:38:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.902+0000 7f3a1a02a640 1 --2- 172.21.15.23:0/3547690640 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f39fc0a20c0 0x7f39fc0a4580 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.902+0000 7f3a1a02a640 1 --2- 172.21.15.23:0/3547690640 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a141021e0 0x7f3a1419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.902+0000 7f3a1a02a640 1 --2- 172.21.15.23:0/3547690640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a14101820 0x7f3a14197b30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.902+0000 7f3a1a02a640 1 --2- 172.21.15.23:0/3547690640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a14100630 0x7f3a141975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.902+0000 7f3a1a02a640 1 -- 172.21.15.23:0/3547690640 >> 172.21.15.23:0/3547690640 conn(0x7f3a140fbba0 msgr2=0x7f3a14068870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.902+0000 7f3a1a02a640 1 -- 172.21.15.23:0/3547690640 shutdown_connections 2024-03-24T10:38:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:06.902+0000 7f3a1a02a640 1 -- 172.21.15.23:0/3547690640 wait complete. 2024-03-24T10:38:06.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:06 smithi023 ceph-mon[32466]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:06.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:06 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] up:boot 2024-03-24T10:38:06.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:06 smithi023 ceph-mon[32466]: fsmap 12 up:standby 2024-03-24T10:38:06.913 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:06 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T10:38:06.914 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:38:07.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.021+0000 7feabc2df640 1 -- 172.21.15.23:0/3623507345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab4101990 msgr2=0x7feab4108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.021+0000 7feabc2df640 1 --2- 172.21.15.23:0/3623507345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab4101990 0x7feab4108220 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feaac009ca0 tx=0x7feaac02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.021+0000 7feabc2df640 1 -- 172.21.15.23:0/3623507345 shutdown_connections 2024-03-24T10:38:07.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.021+0000 7feabc2df640 1 --2- 172.21.15.23:0/3623507345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab4101990 0x7feab4108220 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.021+0000 7feabc2df640 1 --2- 172.21.15.23:0/3623507345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feab4100fd0 0x7feab4101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.021+0000 7feabc2df640 1 --2- 172.21.15.23:0/3623507345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab40ffde0 0x7feab4100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.021+0000 7feabc2df640 1 -- 172.21.15.23:0/3623507345 >> 172.21.15.23:0/3623507345 conn(0x7feab40fb350 msgr2=0x7feab40fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:07.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.021+0000 7feabc2df640 1 -- 172.21.15.23:0/3623507345 shutdown_connections 2024-03-24T10:38:07.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.021+0000 7feabc2df640 1 -- 172.21.15.23:0/3623507345 wait complete. 2024-03-24T10:38:07.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.021+0000 7feabc2df640 1 Processor -- start 2024-03-24T10:38:07.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.021+0000 7feabc2df640 1 -- start start 2024-03-24T10:38:07.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feabc2df640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab40ffde0 0x7feab4103d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feabc2df640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab4100fd0 0x7feab41023b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feabc2df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feab4101990 0x7feab41028f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feabc2df640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7feab4103050 con 0x7feab40ffde0 2024-03-24T10:38:07.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feabc2df640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feab41031c0 con 0x7feab4101990 2024-03-24T10:38:07.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feabc2df640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feab41a1830 con 0x7feab4100fd0 2024-03-24T10:38:07.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feaba07b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab40ffde0 0x7feab4103d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feaba07b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab40ffde0 0x7feab4103d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59474/0 (socket says 172.21.15.23:59474) 2024-03-24T10:38:07.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feaba07b640 1 -- 172.21.15.23:0/1204125419 learned_addr learned my addr 172.21.15.23:0/1204125419 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:07.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feab987a640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab4100fd0 0x7feab41023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feaba87c640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feab4101990 0x7feab41028f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feaba07b640 1 -- 172.21.15.23:0/1204125419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab4100fd0 msgr2=0x7feab41023b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feaba07b640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab4100fd0 0x7feab41023b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feaba07b640 1 -- 172.21.15.23:0/1204125419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feab4101990 msgr2=0x7feab41028f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feaba07b640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feab4101990 0x7feab41028f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feaba07b640 1 -- 172.21.15.23:0/1204125419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac009900 con 0x7feab40ffde0 2024-03-24T10:38:07.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feaba87c640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feab4101990 0x7feab41028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:38:07.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feaba07b640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab40ffde0 0x7feab4103d70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feab000e9f0 tx=0x7feab000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:07.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feaab7fe640 1 -- 172.21.15.23:0/1204125419 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab00096e0 con 0x7feab40ffde0 2024-03-24T10:38:07.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feabc2df640 1 -- 172.21.15.23:0/1204125419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab41a1ae0 con 0x7feab40ffde0 2024-03-24T10:38:07.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feaab7fe640 1 -- 172.21.15.23:0/1204125419 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feab000cd80 con 0x7feab40ffde0 2024-03-24T10:38:07.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feaab7fe640 1 -- 172.21.15.23:0/1204125419 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab0010430 con 0x7feab40ffde0 2024-03-24T10:38:07.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.022+0000 7feabc2df640 1 -- 172.21.15.23:0/1204125419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feab41a1ff0 con 0x7feab40ffde0 2024-03-24T10:38:07.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.024+0000 7feabc2df640 1 -- 172.21.15.23:0/1204125419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feab410c070 con 0x7feab40ffde0 2024-03-24T10:38:07.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.024+0000 7feaab7fe640 1 -- 172.21.15.23:0/1204125419 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7feab0003680 con 0x7feab40ffde0 2024-03-24T10:38:07.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.025+0000 7feaab7fe640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7feaa00a2140 0x7feaa00a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.025+0000 7feab987a640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7feaa00a2140 0x7feaa00a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.025+0000 7feaab7fe640 1 -- 172.21.15.23:0/1204125419 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7feab0014070 con 0x7feab40ffde0 2024-03-24T10:38:07.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.026+0000 7feab987a640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7feaa00a2140 0x7feaa00a4600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feab4103ae0 tx=0x7feaa4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:07.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.029+0000 7feaab7fe640 1 -- 172.21.15.23:0/1204125419 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7feab0097040 con 0x7feab40ffde0 2024-03-24T10:38:07.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.255+0000 7feabc2df640 1 -- 172.21.15.23:0/1204125419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feab41a2490 con 0x7feab40ffde0 2024-03-24T10:38:07.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.256+0000 7feaab7fe640 1 -- 172.21.15.23:0/1204125419 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18470 (secure 0 0 0) 0x7feab009a700 con 0x7feab40ffde0 2024-03-24T10:38:07.257 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:38:07.257 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":81,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:37:22.316371+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:38:07.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.258+0000 7feabc2df640 1 -- 172.21.15.23:0/1204125419 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7feaa00a2140 msgr2=0x7feaa00a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.258+0000 7feabc2df640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7feaa00a2140 0x7feaa00a4600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feab4103ae0 tx=0x7feaa4005f90 comp rx=0 tx=0).stop 2024-03-24T10:38:07.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.258+0000 7feabc2df640 1 -- 172.21.15.23:0/1204125419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab40ffde0 msgr2=0x7feab4103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.258+0000 7feabc2df640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab40ffde0 0x7feab4103d70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feab000e9f0 tx=0x7feab000eeb0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.259+0000 7feabc2df640 1 -- 172.21.15.23:0/1204125419 shutdown_connections 2024-03-24T10:38:07.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.259+0000 7feabc2df640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7feaa00a2140 0x7feaa00a4600 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.259+0000 7feabc2df640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feab4101990 0x7feab41028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.259+0000 7feabc2df640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab4100fd0 0x7feab41023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.259+0000 7feabc2df640 1 --2- 172.21.15.23:0/1204125419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab40ffde0 0x7feab4103d70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.259+0000 7feabc2df640 1 -- 172.21.15.23:0/1204125419 >> 172.21.15.23:0/1204125419 conn(0x7feab40fb350 msgr2=0x7feab40fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:07.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.259+0000 7feabc2df640 1 -- 172.21.15.23:0/1204125419 shutdown_connections 2024-03-24T10:38:07.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.259+0000 7feabc2df640 1 -- 172.21.15.23:0/1204125419 wait complete. 2024-03-24T10:38:07.270 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:39.056452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:38:07.271 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:38:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.377+0000 7fb479a2a640 1 -- 172.21.15.23:0/1448691259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4741021f0 msgr2=0x7fb4741045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.377+0000 7fb479a2a640 1 --2- 172.21.15.23:0/1448691259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4741021f0 0x7fb4741045e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb468009900 tx=0x7fb46802bc30 comp rx=0 tx=0).stop 2024-03-24T10:38:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.377+0000 7fb479a2a640 1 -- 172.21.15.23:0/1448691259 shutdown_connections 2024-03-24T10:38:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.377+0000 7fb479a2a640 1 --2- 172.21.15.23:0/1448691259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb474107450 0x7fb474109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.377+0000 7fb479a2a640 1 --2- 172.21.15.23:0/1448691259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb474104b20 0x7fb474106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.377+0000 7fb479a2a640 1 --2- 172.21.15.23:0/1448691259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4741021f0 0x7fb4741045e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.377+0000 7fb479a2a640 1 -- 172.21.15.23:0/1448691259 >> 172.21.15.23:0/1448691259 conn(0x7fb4740fbb80 msgr2=0x7fb4740fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.377+0000 7fb479a2a640 1 -- 172.21.15.23:0/1448691259 shutdown_connections 2024-03-24T10:38:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.377+0000 7fb479a2a640 1 -- 172.21.15.23:0/1448691259 wait complete. 2024-03-24T10:38:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.378+0000 7fb479a2a640 1 Processor -- start 2024-03-24T10:38:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.378+0000 7fb479a2a640 1 -- start start 2024-03-24T10:38:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.378+0000 7fb479a2a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4741021f0 0x7fb4741977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.378+0000 7fb479a2a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb474104b20 0x7fb474197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.378+0000 7fb479a2a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb474107450 0x7fb47419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.378+0000 7fb479a2a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb47419f2c0 con 0x7fb474104b20 2024-03-24T10:38:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.378+0000 7fb479a2a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb47419f430 con 0x7fb4741021f0 2024-03-24T10:38:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.378+0000 7fb479a2a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb47419f5f0 con 0x7fb474107450 2024-03-24T10:38:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.378+0000 7fb4727fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb474104b20 0x7fb474197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.378+0000 7fb4727fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb474104b20 0x7fb474197d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59492/0 (socket says 172.21.15.23:59492) 2024-03-24T10:38:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.378+0000 7fb4727fc640 1 -- 172.21.15.23:0/2980320968 learned_addr learned my addr 172.21.15.23:0/2980320968 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.378+0000 7fb4737fe640 1 --2- 172.21.15.23:0/2980320968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb474107450 0x7fb47419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.378+0000 7fb472ffd640 1 --2- 172.21.15.23:0/2980320968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4741021f0 0x7fb4741977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.379+0000 7fb4727fc640 1 -- 172.21.15.23:0/2980320968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb474107450 msgr2=0x7fb47419ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.379+0000 7fb4727fc640 1 --2- 172.21.15.23:0/2980320968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb474107450 0x7fb47419ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.379+0000 7fb4727fc640 1 -- 172.21.15.23:0/2980320968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4741021f0 msgr2=0x7fb4741977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.379+0000 7fb4727fc640 1 --2- 172.21.15.23:0/2980320968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4741021f0 0x7fb4741977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.379+0000 7fb4727fc640 1 -- 172.21.15.23:0/2980320968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb468009560 con 0x7fb474104b20 2024-03-24T10:38:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.379+0000 7fb4727fc640 1 --2- 172.21.15.23:0/2980320968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb474104b20 0x7fb474197d10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb46400e9c0 tx=0x7fb46400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.379+0000 7fb478a28640 1 -- 172.21.15.23:0/2980320968 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb464009700 con 0x7fb474104b20 2024-03-24T10:38:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.379+0000 7fb479a2a640 1 -- 172.21.15.23:0/2980320968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb47419f8d0 con 0x7fb474104b20 2024-03-24T10:38:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.379+0000 7fb478a28640 1 -- 172.21.15.23:0/2980320968 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb46400cd80 con 0x7fb474104b20 2024-03-24T10:38:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.379+0000 7fb478a28640 1 -- 172.21.15.23:0/2980320968 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb464010430 con 0x7fb474104b20 2024-03-24T10:38:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.379+0000 7fb479a2a640 1 -- 172.21.15.23:0/2980320968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb47419fde0 con 0x7fb474104b20 2024-03-24T10:38:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.380+0000 7fb479a2a640 1 -- 172.21.15.23:0/2980320968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4740fd670 con 0x7fb474104b20 2024-03-24T10:38:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.381+0000 7fb478a28640 1 -- 172.21.15.23:0/2980320968 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fb464005000 con 0x7fb474104b20 2024-03-24T10:38:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.382+0000 7fb478a28640 1 --2- 172.21.15.23:0/2980320968 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb45c0a20c0 0x7fb45c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.382+0000 7fb478a28640 1 -- 172.21.15.23:0/2980320968 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb464014070 con 0x7fb474104b20 2024-03-24T10:38:07.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.382+0000 7fb472ffd640 1 --2- 172.21.15.23:0/2980320968 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb45c0a20c0 0x7fb45c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.382+0000 7fb472ffd640 1 --2- 172.21.15.23:0/2980320968 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb45c0a20c0 0x7fb45c0a4580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb468009230 tx=0x7fb468005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:07.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.386+0000 7fb478a28640 1 -- 172.21.15.23:0/2980320968 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb4640972d0 con 0x7fb474104b20 2024-03-24T10:38:07.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.568+0000 7fb479a2a640 1 -- 172.21.15.23:0/2980320968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb47410d330 con 0x7fb474104b20 2024-03-24T10:38:07.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.569+0000 7fb478a28640 1 -- 172.21.15.23:0/2980320968 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fb46409a990 con 0x7fb474104b20 2024-03-24T10:38:07.569 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-24T10:38:07.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.571+0000 7fb479a2a640 1 -- 172.21.15.23:0/2980320968 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb45c0a20c0 msgr2=0x7fb45c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.571+0000 7fb479a2a640 1 --2- 172.21.15.23:0/2980320968 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb45c0a20c0 0x7fb45c0a4580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb468009230 tx=0x7fb468005fd0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.571+0000 7fb479a2a640 1 -- 172.21.15.23:0/2980320968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb474104b20 msgr2=0x7fb474197d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.571+0000 7fb479a2a640 1 --2- 172.21.15.23:0/2980320968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb474104b20 0x7fb474197d10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb46400e9c0 tx=0x7fb46400ee80 comp rx=0 tx=0).stop 2024-03-24T10:38:07.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.571+0000 7fb479a2a640 1 -- 172.21.15.23:0/2980320968 shutdown_connections 2024-03-24T10:38:07.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.571+0000 7fb479a2a640 1 --2- 172.21.15.23:0/2980320968 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb45c0a20c0 0x7fb45c0a4580 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.571+0000 7fb479a2a640 1 --2- 172.21.15.23:0/2980320968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb474107450 0x7fb47419ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.571+0000 7fb479a2a640 1 --2- 172.21.15.23:0/2980320968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb474104b20 0x7fb474197d10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.571+0000 7fb479a2a640 1 --2- 172.21.15.23:0/2980320968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4741021f0 0x7fb4741977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.571+0000 7fb479a2a640 1 -- 172.21.15.23:0/2980320968 >> 172.21.15.23:0/2980320968 conn(0x7fb4740fbb80 msgr2=0x7fb474102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:07.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.571+0000 7fb479a2a640 1 -- 172.21.15.23:0/2980320968 shutdown_connections 2024-03-24T10:38:07.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.571+0000 7fb479a2a640 1 -- 172.21.15.23:0/2980320968 wait complete. 2024-03-24T10:38:07.583 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T10:38:07.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.698+0000 7fa91d325640 1 -- 172.21.15.23:0/1849308355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa918100600 msgr2=0x7fa918100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.698+0000 7fa91d325640 1 --2- 172.21.15.23:0/1849308355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa918100600 0x7fa918100a20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa908009900 tx=0x7fa90802bc30 comp rx=0 tx=0).stop 2024-03-24T10:38:07.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.698+0000 7fa91d325640 1 -- 172.21.15.23:0/1849308355 shutdown_connections 2024-03-24T10:38:07.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.698+0000 7fa91d325640 1 --2- 172.21.15.23:0/1849308355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9181021b0 0x7fa918108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.698+0000 7fa91d325640 1 --2- 172.21.15.23:0/1849308355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9181017f0 0x7fa918101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.698+0000 7fa91d325640 1 --2- 172.21.15.23:0/1849308355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa918100600 0x7fa918100a20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.698+0000 7fa91d325640 1 -- 172.21.15.23:0/1849308355 >> 172.21.15.23:0/1849308355 conn(0x7fa9180fbb70 msgr2=0x7fa9180fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.698+0000 7fa91d325640 1 -- 172.21.15.23:0/1849308355 shutdown_connections 2024-03-24T10:38:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.698+0000 7fa91d325640 1 -- 172.21.15.23:0/1849308355 wait complete. 2024-03-24T10:38:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.698+0000 7fa91d325640 1 Processor -- start 2024-03-24T10:38:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.698+0000 7fa91d325640 1 -- start start 2024-03-24T10:38:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa91d325640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa918100600 0x7fa918069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa91d325640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9181017f0 0x7fa91819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa91d325640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9181021b0 0x7fa91819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa91d325640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fa91819f280 con 0x7fa918100600 2024-03-24T10:38:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa91d325640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa91819f3f0 con 0x7fa9181021b0 2024-03-24T10:38:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa91d325640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa91819f5b0 con 0x7fa9181017f0 2024-03-24T10:38:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa916ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa918100600 0x7fa918069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa916ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa918100600 0x7fa918069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59510/0 (socket says 172.21.15.23:59510) 2024-03-24T10:38:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa916ffd640 1 -- 172.21.15.23:0/3341814223 learned_addr learned my addr 172.21.15.23:0/3341814223 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa9167fc640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9181017f0 0x7fa91819c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa9177fe640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9181021b0 0x7fa91819eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa9167fc640 1 -- 172.21.15.23:0/3341814223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9181021b0 msgr2=0x7fa91819eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa9167fc640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9181021b0 0x7fa91819eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa9167fc640 1 -- 172.21.15.23:0/3341814223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa918100600 msgr2=0x7fa918069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa9167fc640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa918100600 0x7fa918069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa9167fc640 1 -- 172.21.15.23:0/3341814223 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa908009560 con 0x7fa9181017f0 2024-03-24T10:38:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa916ffd640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa918100600 0x7fa918069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa9177fe640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9181021b0 0x7fa91819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.699+0000 7fa9167fc640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9181017f0 0x7fa91819c480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa90c005fd0 tx=0x7fa90c00dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.700+0000 7fa8fffff640 1 -- 172.21.15.23:0/3341814223 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c009c30 con 0x7fa9181017f0 2024-03-24T10:38:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.700+0000 7fa8fffff640 1 -- 172.21.15.23:0/3341814223 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa90c004510 con 0x7fa9181017f0 2024-03-24T10:38:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.700+0000 7fa91d325640 1 -- 172.21.15.23:0/3341814223 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91819f890 con 0x7fa9181017f0 2024-03-24T10:38:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.700+0000 7fa8fffff640 1 -- 172.21.15.23:0/3341814223 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c003e70 con 0x7fa9181017f0 2024-03-24T10:38:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.700+0000 7fa91d325640 1 -- 172.21.15.23:0/3341814223 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa91819fda0 con 0x7fa9181017f0 2024-03-24T10:38:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.701+0000 7fa91d325640 1 -- 172.21.15.23:0/3341814223 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa918107920 con 0x7fa9181017f0 2024-03-24T10:38:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.702+0000 7fa8fffff640 1 -- 172.21.15.23:0/3341814223 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fa90c004000 con 0x7fa9181017f0 2024-03-24T10:38:07.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.702+0000 7fa8fffff640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa9000a20c0 0x7fa9000a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:07.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.703+0000 7fa8fffff640 1 -- 172.21.15.23:0/3341814223 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa90c0cd660 con 0x7fa9181017f0 2024-03-24T10:38:07.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.703+0000 7fa916ffd640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa9000a20c0 0x7fa9000a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:07.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.703+0000 7fa916ffd640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa9000a20c0 0x7fa9000a4580 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa90802c840 tx=0x7fa90802cd10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:07.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.706+0000 7fa8fffff640 1 -- 172.21.15.23:0/3341814223 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa90c096b60 con 0x7fa9181017f0 2024-03-24T10:38:07.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.906+0000 7fa91d325640 1 -- 172.21.15.23:0/3341814223 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa91810c260 con 0x7fa9181017f0 2024-03-24T10:38:07.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.907+0000 7fa8fffff640 1 -- 172.21.15.23:0/3341814223 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fa90c09a220 con 0x7fa9181017f0 2024-03-24T10:38:07.908 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:38:07.908 INFO:teuthology.orchestra.run.smithi023.stdout:[ 2024-03-24T10:38:07.908 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:38:07.908 INFO:teuthology.orchestra.run.smithi023.stdout: "poolnum": 1, 2024-03-24T10:38:07.908 INFO:teuthology.orchestra.run.smithi023.stdout: "poolname": ".mgr" 2024-03-24T10:38:07.908 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:38:07.908 INFO:teuthology.orchestra.run.smithi023.stdout:] 2024-03-24T10:38:07.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.909+0000 7fa91d325640 1 -- 172.21.15.23:0/3341814223 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa9000a20c0 msgr2=0x7fa9000a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.909+0000 7fa91d325640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa9000a20c0 0x7fa9000a4580 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa90802c840 tx=0x7fa90802cd10 comp rx=0 tx=0).stop 2024-03-24T10:38:07.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.909+0000 7fa91d325640 1 -- 172.21.15.23:0/3341814223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9181017f0 msgr2=0x7fa91819c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:07.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.909+0000 7fa91d325640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9181017f0 0x7fa91819c480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa90c005fd0 tx=0x7fa90c00dde0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.909+0000 7fa91d325640 1 -- 172.21.15.23:0/3341814223 shutdown_connections 2024-03-24T10:38:07.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.909+0000 7fa91d325640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa9000a20c0 0x7fa9000a4580 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.909+0000 7fa91d325640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9181021b0 0x7fa91819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.909+0000 7fa91d325640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9181017f0 0x7fa91819c480 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.909+0000 7fa91d325640 1 --2- 172.21.15.23:0/3341814223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa918100600 0x7fa918069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:07.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.909+0000 7fa91d325640 1 -- 172.21.15.23:0/3341814223 >> 172.21.15.23:0/3341814223 conn(0x7fa9180fbb70 msgr2=0x7fa918104af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:07.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.909+0000 7fa91d325640 1 -- 172.21.15.23:0/3341814223 shutdown_connections 2024-03-24T10:38:07.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:07.909+0000 7fa91d325640 1 -- 172.21.15.23:0/3341814223 wait complete. 2024-03-24T10:38:07.920 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T10:38:07.920 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T10:38:08.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.027+0000 7ffb2342c640 1 -- 172.21.15.23:0/3865956376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb1c102200 msgr2=0x7ffb1c108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:08.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.027+0000 7ffb2342c640 1 --2- 172.21.15.23:0/3865956376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb1c102200 0x7ffb1c108a90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009ca0 tx=0x7ffb1402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.027+0000 7ffb2342c640 1 -- 172.21.15.23:0/3865956376 shutdown_connections 2024-03-24T10:38:08.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.027+0000 7ffb2342c640 1 --2- 172.21.15.23:0/3865956376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb1c102200 0x7ffb1c108a90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.027+0000 7ffb2342c640 1 --2- 172.21.15.23:0/3865956376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb1c101840 0x7ffb1c101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.027+0000 7ffb2342c640 1 --2- 172.21.15.23:0/3865956376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1c100650 0x7ffb1c100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.027+0000 7ffb2342c640 1 -- 172.21.15.23:0/3865956376 >> 172.21.15.23:0/3865956376 conn(0x7ffb1c0fbbc0 msgr2=0x7ffb1c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:08.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.027+0000 7ffb2342c640 1 -- 172.21.15.23:0/3865956376 shutdown_connections 2024-03-24T10:38:08.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.027+0000 7ffb2342c640 1 -- 172.21.15.23:0/3865956376 wait complete. 2024-03-24T10:38:08.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb2342c640 1 Processor -- start 2024-03-24T10:38:08.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb2342c640 1 -- start start 2024-03-24T10:38:08.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb2342c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1c100650 0x7ffb1c1045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:08.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb2342c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb1c101840 0x7ffb1c102c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:08.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb2342c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb1c102200 0x7ffb1c103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:08.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb2342c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1c1038c0 con 0x7ffb1c101840 2024-03-24T10:38:08.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb2342c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1c103a30 con 0x7ffb1c102200 2024-03-24T10:38:08.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb2342c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1c1a1830 con 0x7ffb1c100650 2024-03-24T10:38:08.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb209c7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb1c101840 0x7ffb1c102c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:08.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb209c7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb1c101840 0x7ffb1c102c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35118/0 (socket says 172.21.15.23:35118) 2024-03-24T10:38:08.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb209c7640 1 -- 172.21.15.23:0/3223262386 learned_addr learned my addr 172.21.15.23:0/3223262386 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:08.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb211c8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1c100650 0x7ffb1c1045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:08.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb219c9640 1 --2- 172.21.15.23:0/3223262386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb1c102200 0x7ffb1c103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:08.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb209c7640 1 -- 172.21.15.23:0/3223262386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1c100650 msgr2=0x7ffb1c1045e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:08.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb209c7640 1 --2- 172.21.15.23:0/3223262386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1c100650 0x7ffb1c1045e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb209c7640 1 -- 172.21.15.23:0/3223262386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb1c102200 msgr2=0x7ffb1c103160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:08.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb209c7640 1 --2- 172.21.15.23:0/3223262386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb1c102200 0x7ffb1c103160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb209c7640 1 -- 172.21.15.23:0/3223262386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb14009900 con 0x7ffb1c101840 2024-03-24T10:38:08.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.028+0000 7ffb211c8640 1 --2- 172.21.15.23:0/3223262386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1c100650 0x7ffb1c1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:08.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.029+0000 7ffb209c7640 1 --2- 172.21.15.23:0/3223262386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb1c101840 0x7ffb1c102c20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffb1800ea20 tx=0x7ffb1800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:08.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.029+0000 7ffb0e7fc640 1 -- 172.21.15.23:0/3223262386 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb18009700 con 0x7ffb1c101840 2024-03-24T10:38:08.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.029+0000 7ffb0e7fc640 1 -- 172.21.15.23:0/3223262386 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb1800cd80 con 0x7ffb1c101840 2024-03-24T10:38:08.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.029+0000 7ffb2342c640 1 -- 172.21.15.23:0/3223262386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb1c1a1ae0 con 0x7ffb1c101840 2024-03-24T10:38:08.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.029+0000 7ffb0e7fc640 1 -- 172.21.15.23:0/3223262386 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb18018840 con 0x7ffb1c101840 2024-03-24T10:38:08.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.029+0000 7ffb2342c640 1 -- 172.21.15.23:0/3223262386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb1c1a1ff0 con 0x7ffb1c101840 2024-03-24T10:38:08.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.030+0000 7ffb2342c640 1 -- 172.21.15.23:0/3223262386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb1c10c060 con 0x7ffb1c101840 2024-03-24T10:38:08.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.032+0000 7ffb0e7fc640 1 -- 172.21.15.23:0/3223262386 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ffb18010a80 con 0x7ffb1c101840 2024-03-24T10:38:08.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.032+0000 7ffb0e7fc640 1 --2- 172.21.15.23:0/3223262386 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ffb080a20c0 0x7ffb080a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:08.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.032+0000 7ffb0e7fc640 1 -- 172.21.15.23:0/3223262386 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ffb18014070 con 0x7ffb1c101840 2024-03-24T10:38:08.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.032+0000 7ffb211c8640 1 --2- 172.21.15.23:0/3223262386 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ffb080a20c0 0x7ffb080a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:08.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.033+0000 7ffb211c8640 1 --2- 172.21.15.23:0/3223262386 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ffb080a20c0 0x7ffb080a4580 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c1016a0 tx=0x7ffb10005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:08.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.036+0000 7ffb0e7fc640 1 -- 172.21.15.23:0/3223262386 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffb180974b0 con 0x7ffb1c101840 2024-03-24T10:38:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:07 smithi117 ceph-mon[36368]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:08.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:07 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:07 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:07 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3547690640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:08.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:07 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:07 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:07 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:07 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:07 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:08.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:07 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:08.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:07 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:07 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:08.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:07 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1204125419' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:08.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:07 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2980320968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:38:08.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:07 smithi023 ceph-mon[32466]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:08.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:07 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:07 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:07 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3547690640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:08.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:07 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:07 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:07 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:07 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:07 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:08.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:07 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:08.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:07 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:07 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:08.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:07 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1204125419' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:08.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:07 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2980320968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:38:08.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.222+0000 7ffb2342c640 1 -- 172.21.15.23:0/3223262386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ffb1c10c1f0 con 0x7ffb1c101840 2024-03-24T10:38:08.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:07 smithi116 ceph-mon[36248]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:08.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:07 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3547690640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:08.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:08.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:08.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:08.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:07 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:38:08.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:07 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1204125419' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:08.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:07 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2980320968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:38:08.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.840+0000 7ffb0e7fc640 1 -- 172.21.15.23:0/3223262386 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7ffb1809ab70 con 0x7ffb1c101840 2024-03-24T10:38:08.841 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_metadata' created 2024-03-24T10:38:08.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.842+0000 7ffb2342c640 1 -- 172.21.15.23:0/3223262386 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ffb080a20c0 msgr2=0x7ffb080a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:08.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.842+0000 7ffb2342c640 1 --2- 172.21.15.23:0/3223262386 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ffb080a20c0 0x7ffb080a4580 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c1016a0 tx=0x7ffb10005f90 comp rx=0 tx=0).stop 2024-03-24T10:38:08.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.842+0000 7ffb2342c640 1 -- 172.21.15.23:0/3223262386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb1c101840 msgr2=0x7ffb1c102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:08.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.842+0000 7ffb2342c640 1 --2- 172.21.15.23:0/3223262386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb1c101840 0x7ffb1c102c20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffb1800ea20 tx=0x7ffb1800eee0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.843+0000 7ffb2342c640 1 -- 172.21.15.23:0/3223262386 shutdown_connections 2024-03-24T10:38:08.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.843+0000 7ffb2342c640 1 --2- 172.21.15.23:0/3223262386 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ffb080a20c0 0x7ffb080a4580 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.843+0000 7ffb2342c640 1 --2- 172.21.15.23:0/3223262386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb1c102200 0x7ffb1c103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.843+0000 7ffb2342c640 1 --2- 172.21.15.23:0/3223262386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb1c101840 0x7ffb1c102c20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.843+0000 7ffb2342c640 1 --2- 172.21.15.23:0/3223262386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb1c100650 0x7ffb1c1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.843+0000 7ffb2342c640 1 -- 172.21.15.23:0/3223262386 >> 172.21.15.23:0/3223262386 conn(0x7ffb1c0fbbc0 msgr2=0x7ffb1c0fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:08.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.843+0000 7ffb2342c640 1 -- 172.21.15.23:0/3223262386 shutdown_connections 2024-03-24T10:38:08.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.843+0000 7ffb2342c640 1 -- 172.21.15.23:0/3223262386 wait complete. 2024-03-24T10:38:08.857 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T10:38:08.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.970+0000 7f8ffd50c640 1 -- 172.21.15.23:0/432612122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ff81029c0 msgr2=0x7f8ff8102e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:08.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.970+0000 7f8ffd50c640 1 --2- 172.21.15.23:0/432612122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ff81029c0 0x7f8ff8102e40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009900 tx=0x7f8fec02bc30 comp rx=0 tx=0).stop 2024-03-24T10:38:08.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.970+0000 7f8ffd50c640 1 -- 172.21.15.23:0/432612122 shutdown_connections 2024-03-24T10:38:08.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.970+0000 7f8ffd50c640 1 --2- 172.21.15.23:0/432612122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ff8103380 0x7f8ff8109c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.970+0000 7f8ffd50c640 1 --2- 172.21.15.23:0/432612122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ff81029c0 0x7f8ff8102e40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.970+0000 7f8ffd50c640 1 --2- 172.21.15.23:0/432612122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ff80ff9f0 0x7f8ff80ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.970+0000 7f8ffd50c640 1 -- 172.21.15.23:0/432612122 >> 172.21.15.23:0/432612122 conn(0x7f8ff80fb320 msgr2=0x7f8ff80fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:08.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.970+0000 7f8ffd50c640 1 -- 172.21.15.23:0/432612122 shutdown_connections 2024-03-24T10:38:08.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.970+0000 7f8ffd50c640 1 -- 172.21.15.23:0/432612122 wait complete. 2024-03-24T10:38:08.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ffd50c640 1 Processor -- start 2024-03-24T10:38:08.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ffd50c640 1 -- start start 2024-03-24T10:38:08.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ffd50c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ff80ff9f0 0x7f8ff8197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:08.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ffd50c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ff81029c0 0x7f8ff8197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:08.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ffd50c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ff8103380 0x7f8ff819ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:08.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ffd50c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff819f260 con 0x7f8ff81029c0 2024-03-24T10:38:08.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ffd50c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff819f3d0 con 0x7f8ff8103380 2024-03-24T10:38:08.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ffd50c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff819f590 con 0x7f8ff80ff9f0 2024-03-24T10:38:08.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ff67fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ff81029c0 0x7f8ff8197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:08.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ff67fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ff81029c0 0x7f8ff8197cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35140/0 (socket says 172.21.15.23:35140) 2024-03-24T10:38:08.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ff67fc640 1 -- 172.21.15.23:0/2305713904 learned_addr learned my addr 172.21.15.23:0/2305713904 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:08.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ff77fe640 1 --2- 172.21.15.23:0/2305713904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ff8103380 0x7f8ff819ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:08.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ff6ffd640 1 --2- 172.21.15.23:0/2305713904 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ff80ff9f0 0x7f8ff8197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:08.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ff67fc640 1 -- 172.21.15.23:0/2305713904 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ff80ff9f0 msgr2=0x7f8ff8197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:08.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ff67fc640 1 --2- 172.21.15.23:0/2305713904 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ff80ff9f0 0x7f8ff8197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ff67fc640 1 -- 172.21.15.23:0/2305713904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ff8103380 msgr2=0x7f8ff819ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:08.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ff67fc640 1 --2- 172.21.15.23:0/2305713904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ff8103380 0x7f8ff819ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:08.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ff67fc640 1 -- 172.21.15.23:0/2305713904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fec009560 con 0x7f8ff81029c0 2024-03-24T10:38:08.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.971+0000 7f8ff67fc640 1 --2- 172.21.15.23:0/2305713904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ff81029c0 0x7f8ff8197cb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8fec006010 tx=0x7f8fec0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:08.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.972+0000 7f8fdffff640 1 -- 172.21.15.23:0/2305713904 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec0044e0 con 0x7f8ff81029c0 2024-03-24T10:38:08.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.972+0000 7f8fdffff640 1 -- 172.21.15.23:0/2305713904 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fec004aa0 con 0x7f8ff81029c0 2024-03-24T10:38:08.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.972+0000 7f8ffd50c640 1 -- 172.21.15.23:0/2305713904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff819f810 con 0x7f8ff81029c0 2024-03-24T10:38:08.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.972+0000 7f8fdffff640 1 -- 172.21.15.23:0/2305713904 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec003e80 con 0x7f8ff81029c0 2024-03-24T10:38:08.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.972+0000 7f8ffd50c640 1 -- 172.21.15.23:0/2305713904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff819fd20 con 0x7f8ff81029c0 2024-03-24T10:38:08.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.973+0000 7f8ffd50c640 1 -- 172.21.15.23:0/2305713904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff8108af0 con 0x7f8ff81029c0 2024-03-24T10:38:08.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.974+0000 7f8fdffff640 1 -- 172.21.15.23:0/2305713904 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f8fec0035f0 con 0x7f8ff81029c0 2024-03-24T10:38:08.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.974+0000 7f8fdffff640 1 --2- 172.21.15.23:0/2305713904 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8fe00a20c0 0x7f8fe00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:08.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.974+0000 7f8ff6ffd640 1 --2- 172.21.15.23:0/2305713904 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8fe00a20c0 0x7f8fe00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:08.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.974+0000 7f8fdffff640 1 -- 172.21.15.23:0/2305713904 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f8fec0efc50 con 0x7f8ff81029c0 2024-03-24T10:38:08.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.975+0000 7f8ff6ffd640 1 --2- 172.21.15.23:0/2305713904 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8fe00a20c0 0x7f8fe00a4580 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8069b00 tx=0x7f8fe8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:08.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:08.979+0000 7f8fdffff640 1 -- 172.21.15.23:0/2305713904 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8fec0f5020 con 0x7f8ff81029c0 2024-03-24T10:38:09.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.164+0000 7f8ffd50c640 1 -- 172.21.15.23:0/2305713904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f8ff810d140 con 0x7f8ff81029c0 2024-03-24T10:38:09.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:08 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3341814223' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T10:38:09.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:08 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3223262386' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T10:38:09.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:08 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:09.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:08 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:08 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3341814223' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T10:38:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:08 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3223262386' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T10:38:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:08 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:08 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:08 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3341814223' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T10:38:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:08 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3223262386' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T10:38:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:08 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:08 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:09.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.842+0000 7f8fdffff640 1 -- 172.21.15.23:0/2305713904 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) ==== 170+0+0 (secure 0 0 0) 0x7f8fec0b8f50 con 0x7f8ff81029c0 2024-03-24T10:38:09.843 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data' created 2024-03-24T10:38:09.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.845+0000 7f8ffd50c640 1 -- 172.21.15.23:0/2305713904 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8fe00a20c0 msgr2=0x7f8fe00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:09.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.845+0000 7f8ffd50c640 1 --2- 172.21.15.23:0/2305713904 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8fe00a20c0 0x7f8fe00a4580 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8069b00 tx=0x7f8fe8005f90 comp rx=0 tx=0).stop 2024-03-24T10:38:09.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.845+0000 7f8ffd50c640 1 -- 172.21.15.23:0/2305713904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ff81029c0 msgr2=0x7f8ff8197cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:09.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.845+0000 7f8ffd50c640 1 --2- 172.21.15.23:0/2305713904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ff81029c0 0x7f8ff8197cb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8fec006010 tx=0x7f8fec0042f0 comp rx=0 tx=0).stop 2024-03-24T10:38:09.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.845+0000 7f8ffd50c640 1 -- 172.21.15.23:0/2305713904 shutdown_connections 2024-03-24T10:38:09.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.845+0000 7f8ffd50c640 1 --2- 172.21.15.23:0/2305713904 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8fe00a20c0 0x7f8fe00a4580 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:09.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.845+0000 7f8ffd50c640 1 --2- 172.21.15.23:0/2305713904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ff8103380 0x7f8ff819ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:09.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.845+0000 7f8ffd50c640 1 --2- 172.21.15.23:0/2305713904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ff81029c0 0x7f8ff8197cb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:09.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.845+0000 7f8ffd50c640 1 --2- 172.21.15.23:0/2305713904 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ff80ff9f0 0x7f8ff8197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:09.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.845+0000 7f8ffd50c640 1 -- 172.21.15.23:0/2305713904 >> 172.21.15.23:0/2305713904 conn(0x7f8ff80fb320 msgr2=0x7f8ff8105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:09.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.845+0000 7f8ffd50c640 1 -- 172.21.15.23:0/2305713904 shutdown_connections 2024-03-24T10:38:09.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.845+0000 7f8ffd50c640 1 -- 172.21.15.23:0/2305713904 wait complete. 2024-03-24T10:38:09.864 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T10:38:09.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.970+0000 7f5745397640 1 -- 172.21.15.23:0/745069276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5740102190 msgr2=0x7f5740104580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:09.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.970+0000 7f5745397640 1 --2- 172.21.15.23:0/745069276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5740102190 0x7f5740104580 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5738009ca0 tx=0x7f573802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:38:09.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 -- 172.21.15.23:0/745069276 shutdown_connections 2024-03-24T10:38:09.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 --2- 172.21.15.23:0/745069276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57401073f0 0x7f5740109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:09.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 --2- 172.21.15.23:0/745069276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5740104ac0 0x7f5740106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:09.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 --2- 172.21.15.23:0/745069276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5740102190 0x7f5740104580 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:09.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 -- 172.21.15.23:0/745069276 >> 172.21.15.23:0/745069276 conn(0x7f57400fbba0 msgr2=0x7f57400fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:09.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 -- 172.21.15.23:0/745069276 shutdown_connections 2024-03-24T10:38:09.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 -- 172.21.15.23:0/745069276 wait complete. 2024-03-24T10:38:09.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 Processor -- start 2024-03-24T10:38:09.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 -- start start 2024-03-24T10:38:09.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5740102190 0x7f5740197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:09.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5740104ac0 0x7f5740197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:09.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57401073f0 0x7f574019edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:09.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f574019f300 con 0x7f5740102190 2024-03-24T10:38:09.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f574019f470 con 0x7f5740104ac0 2024-03-24T10:38:09.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.971+0000 7f5745397640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f574019f630 con 0x7f57401073f0 2024-03-24T10:38:09.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f573effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5740102190 0x7f5740197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:09.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f573effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5740102190 0x7f5740197810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35158/0 (socket says 172.21.15.23:35158) 2024-03-24T10:38:09.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f573effd640 1 -- 172.21.15.23:0/3803463327 learned_addr learned my addr 172.21.15.23:0/3803463327 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:09.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f573e7fc640 1 --2- 172.21.15.23:0/3803463327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5740104ac0 0x7f5740197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:09.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f573effd640 1 -- 172.21.15.23:0/3803463327 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57401073f0 msgr2=0x7f574019edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:09.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f573effd640 1 --2- 172.21.15.23:0/3803463327 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57401073f0 0x7f574019edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:09.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f573effd640 1 -- 172.21.15.23:0/3803463327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5740104ac0 msgr2=0x7f5740197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:09.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f573effd640 1 --2- 172.21.15.23:0/3803463327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5740104ac0 0x7f5740197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:09.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f573effd640 1 -- 172.21.15.23:0/3803463327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5738009900 con 0x7f5740102190 2024-03-24T10:38:09.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f573effd640 1 --2- 172.21.15.23:0/3803463327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5740102190 0x7f5740197810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f57380325e0 tx=0x7f5738034e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:09.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f5737fff640 1 -- 172.21.15.23:0/3803463327 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738003970 con 0x7f5740102190 2024-03-24T10:38:09.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f5745397640 1 -- 172.21.15.23:0/3803463327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f574019f8b0 con 0x7f5740102190 2024-03-24T10:38:09.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f5737fff640 1 -- 172.21.15.23:0/3803463327 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5738004040 con 0x7f5740102190 2024-03-24T10:38:09.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f5737fff640 1 -- 172.21.15.23:0/3803463327 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57380406b0 con 0x7f5740102190 2024-03-24T10:38:09.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.972+0000 7f5745397640 1 -- 172.21.15.23:0/3803463327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f574019fdb0 con 0x7f5740102190 2024-03-24T10:38:09.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.973+0000 7f5745397640 1 -- 172.21.15.23:0/3803463327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57400fd690 con 0x7f5740102190 2024-03-24T10:38:09.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.974+0000 7f5737fff640 1 -- 172.21.15.23:0/3803463327 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f573800a040 con 0x7f5740102190 2024-03-24T10:38:09.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.975+0000 7f5737fff640 1 --2- 172.21.15.23:0/3803463327 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f572c0a20c0 0x7f572c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:09.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.975+0000 7f573e7fc640 1 --2- 172.21.15.23:0/3803463327 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f572c0a20c0 0x7f572c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:09.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.975+0000 7f5737fff640 1 -- 172.21.15.23:0/3803463327 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f57380f0df0 con 0x7f5740102190 2024-03-24T10:38:09.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.975+0000 7f573e7fc640 1 --2- 172.21.15.23:0/3803463327 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f572c0a20c0 0x7f572c0a4580 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5740198d20 tx=0x7f5728008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:09.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:09.979+0000 7f5737fff640 1 -- 172.21.15.23:0/3803463327 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f57380ba000 con 0x7f5740102190 2024-03-24T10:38:10.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:09 smithi023 ceph-mon[32466]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:10.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:09 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3223262386' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T10:38:10.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:09 smithi023 ceph-mon[32466]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T10:38:10.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:09 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:09 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:09 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:09 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:09 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:10.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:09 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:10.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:09 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:09 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2305713904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T10:38:10.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:09 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:10.205+0000 7f5745397640 1 -- 172.21.15.23:0/3803463327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f574010d2a0 con 0x7f5740102190 2024-03-24T10:38:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:09 smithi117 ceph-mon[36368]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:09 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3223262386' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T10:38:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:09 smithi117 ceph-mon[36368]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T10:38:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:09 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:09 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:09 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:09 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:09 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:09 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:09 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:09 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2305713904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T10:38:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:09 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:09 smithi116 ceph-mon[36248]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:09 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3223262386' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T10:38:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:09 smithi116 ceph-mon[36248]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T10:38:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:09 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:09 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:09 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:09 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:09 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:38:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:09 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:38:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:09 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:09 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2305713904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T10:38:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:09 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:38:10.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:10.866+0000 7f5737fff640 1 -- 172.21.15.23:0/3803463327 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f57380bd6c0 con 0x7f5740102190 2024-03-24T10:38:10.867 INFO:teuthology.orchestra.run.smithi023.stderr:new fs with metadata pool 2 and data pool 3 2024-03-24T10:38:10.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:10.871+0000 7f5745397640 1 -- 172.21.15.23:0/3803463327 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f572c0a20c0 msgr2=0x7f572c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:10.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:10.871+0000 7f5745397640 1 --2- 172.21.15.23:0/3803463327 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f572c0a20c0 0x7f572c0a4580 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5740198d20 tx=0x7f5728008040 comp rx=0 tx=0).stop 2024-03-24T10:38:10.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:10.871+0000 7f5745397640 1 -- 172.21.15.23:0/3803463327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5740102190 msgr2=0x7f5740197810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:10.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:10.871+0000 7f5745397640 1 --2- 172.21.15.23:0/3803463327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5740102190 0x7f5740197810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f57380325e0 tx=0x7f5738034e80 comp rx=0 tx=0).stop 2024-03-24T10:38:10.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:10.872+0000 7f5745397640 1 -- 172.21.15.23:0/3803463327 shutdown_connections 2024-03-24T10:38:10.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:10.872+0000 7f5745397640 1 --2- 172.21.15.23:0/3803463327 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f572c0a20c0 0x7f572c0a4580 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:10.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:10.872+0000 7f5745397640 1 --2- 172.21.15.23:0/3803463327 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57401073f0 0x7f574019edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:10.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:10.872+0000 7f5745397640 1 --2- 172.21.15.23:0/3803463327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5740104ac0 0x7f5740197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:10.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:10.872+0000 7f5745397640 1 --2- 172.21.15.23:0/3803463327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5740102190 0x7f5740197810 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:10.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:10.872+0000 7f5745397640 1 -- 172.21.15.23:0/3803463327 >> 172.21.15.23:0/3803463327 conn(0x7f57400fbba0 msgr2=0x7f5740102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:10.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:10.874+0000 7f5745397640 1 -- 172.21.15.23:0/3803463327 shutdown_connections 2024-03-24T10:38:10.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:10.874+0000 7f5745397640 1 -- 172.21.15.23:0/3803463327 wait complete. 2024-03-24T10:38:10.896 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:38:11.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.003+0000 7fe7c5647640 1 -- 172.21.15.23:0/2673371897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c0100630 msgr2=0x7fe7c0100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:11.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.003+0000 7fe7c5647640 1 --2- 172.21.15.23:0/2673371897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c0100630 0x7fe7c0100a50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8009900 tx=0x7fe7b802bc30 comp rx=0 tx=0).stop 2024-03-24T10:38:11.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.003+0000 7fe7c5647640 1 -- 172.21.15.23:0/2673371897 shutdown_connections 2024-03-24T10:38:11.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.004+0000 7fe7c5647640 1 --2- 172.21.15.23:0/2673371897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7c01021e0 0x7fe7c0108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.004+0000 7fe7c5647640 1 --2- 172.21.15.23:0/2673371897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c0101820 0x7fe7c0101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.004+0000 7fe7c5647640 1 --2- 172.21.15.23:0/2673371897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c0100630 0x7fe7c0100a50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.004+0000 7fe7c5647640 1 -- 172.21.15.23:0/2673371897 >> 172.21.15.23:0/2673371897 conn(0x7fe7c00fbba0 msgr2=0x7fe7c00fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:11.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.004+0000 7fe7c5647640 1 -- 172.21.15.23:0/2673371897 shutdown_connections 2024-03-24T10:38:11.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.004+0000 7fe7c5647640 1 -- 172.21.15.23:0/2673371897 wait complete. 2024-03-24T10:38:11.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.004+0000 7fe7c5647640 1 Processor -- start 2024-03-24T10:38:11.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.004+0000 7fe7c5647640 1 -- start start 2024-03-24T10:38:11.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.004+0000 7fe7c5647640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c0100630 0x7fe7c0199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:11.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.004+0000 7fe7c5647640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7c0101820 0x7fe7c0199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:11.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.004+0000 7fe7c5647640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c01021e0 0x7fe7c01a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:11.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.004+0000 7fe7c5647640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c01a1560 con 0x7fe7c01021e0 2024-03-24T10:38:11.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.004+0000 7fe7c5647640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c01a16d0 con 0x7fe7c0100630 2024-03-24T10:38:11.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.004+0000 7fe7c5647640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c01a1890 con 0x7fe7c0101820 2024-03-24T10:38:11.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c01021e0 0x7fe7c01a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:11.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c01021e0 0x7fe7c01a1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35172/0 (socket says 172.21.15.23:35172) 2024-03-24T10:38:11.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7bf7fe640 1 -- 172.21.15.23:0/99214289 learned_addr learned my addr 172.21.15.23:0/99214289 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:11.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7beffd640 1 --2- 172.21.15.23:0/99214289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c0100630 0x7fe7c0199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:11.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7be7fc640 1 --2- 172.21.15.23:0/99214289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7c0101820 0x7fe7c0199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:11.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7bf7fe640 1 -- 172.21.15.23:0/99214289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7c0101820 msgr2=0x7fe7c0199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:11.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7bf7fe640 1 --2- 172.21.15.23:0/99214289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7c0101820 0x7fe7c0199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7bf7fe640 1 -- 172.21.15.23:0/99214289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c0100630 msgr2=0x7fe7c0199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:11.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7bf7fe640 1 --2- 172.21.15.23:0/99214289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c0100630 0x7fe7c0199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7bf7fe640 1 -- 172.21.15.23:0/99214289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b8009560 con 0x7fe7c01021e0 2024-03-24T10:38:11.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7bf7fe640 1 --2- 172.21.15.23:0/99214289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c01021e0 0x7fe7c01a1020 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000b970 tx=0x7fe7b000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:11.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7b7fff640 1 -- 172.21.15.23:0/99214289 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b000cb40 con 0x7fe7c01021e0 2024-03-24T10:38:11.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7b7fff640 1 -- 172.21.15.23:0/99214289 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7b000cca0 con 0x7fe7c01021e0 2024-03-24T10:38:11.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7c5647640 1 -- 172.21.15.23:0/99214289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7c01a1b70 con 0x7fe7c01021e0 2024-03-24T10:38:11.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7b7fff640 1 -- 172.21.15.23:0/99214289 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b00077d0 con 0x7fe7c01021e0 2024-03-24T10:38:11.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.005+0000 7fe7c5647640 1 -- 172.21.15.23:0/99214289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7c01a2080 con 0x7fe7c01021e0 2024-03-24T10:38:11.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.006+0000 7fe7c5647640 1 -- 172.21.15.23:0/99214289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7c0107950 con 0x7fe7c01021e0 2024-03-24T10:38:11.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.007+0000 7fe7b7fff640 1 -- 172.21.15.23:0/99214289 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fe7b0014440 con 0x7fe7c01021e0 2024-03-24T10:38:11.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.008+0000 7fe7b7fff640 1 --2- 172.21.15.23:0/99214289 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe7ac0a2140 0x7fe7ac0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:11.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.008+0000 7fe7beffd640 1 --2- 172.21.15.23:0/99214289 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe7ac0a2140 0x7fe7ac0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:11.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.008+0000 7fe7b7fff640 1 -- 172.21.15.23:0/99214289 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7fe7b00cd360 con 0x7fe7c01021e0 2024-03-24T10:38:11.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.008+0000 7fe7beffd640 1 --2- 172.21.15.23:0/99214289 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe7ac0a2140 0x7fe7ac0a4600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe7b802c240 tx=0x7fe7b802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:11.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.012+0000 7fe7b7fff640 1 -- 172.21.15.23:0/99214289 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe7b0096530 con 0x7fe7c01021e0 2024-03-24T10:38:11.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:10 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2305713904' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T10:38:11.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:10 smithi023 ceph-mon[32466]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T10:38:11.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:10 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3803463327' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T10:38:11.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:10 smithi023 ceph-mon[32466]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T10:38:11.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.197+0000 7fe7c5647640 1 -- 172.21.15.23:0/99214289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe7c002d190 con 0x7fe7c01021e0 2024-03-24T10:38:11.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.199+0000 7fe7b7fff640 1 -- 172.21.15.23:0/99214289 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22246 (secure 0 0 0) 0x7fe7b0099bf0 con 0x7fe7c01021e0 2024-03-24T10:38:11.199 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:38:11.200 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":84,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:38:10.206457+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:38:11.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.201+0000 7fe7c5647640 1 -- 172.21.15.23:0/99214289 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe7ac0a2140 msgr2=0x7fe7ac0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:11.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.201+0000 7fe7c5647640 1 --2- 172.21.15.23:0/99214289 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe7ac0a2140 0x7fe7ac0a4600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe7b802c240 tx=0x7fe7b802c7e0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.201+0000 7fe7c5647640 1 -- 172.21.15.23:0/99214289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c01021e0 msgr2=0x7fe7c01a1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.201+0000 7fe7c5647640 1 --2- 172.21.15.23:0/99214289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c01021e0 0x7fe7c01a1020 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000b970 tx=0x7fe7b000be30 comp rx=0 tx=0).stop 2024-03-24T10:38:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.201+0000 7fe7c5647640 1 -- 172.21.15.23:0/99214289 shutdown_connections 2024-03-24T10:38:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.201+0000 7fe7c5647640 1 --2- 172.21.15.23:0/99214289 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe7ac0a2140 0x7fe7ac0a4600 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.201+0000 7fe7c5647640 1 --2- 172.21.15.23:0/99214289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c01021e0 0x7fe7c01a1020 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.201+0000 7fe7c5647640 1 --2- 172.21.15.23:0/99214289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7c0101820 0x7fe7c0199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.201+0000 7fe7c5647640 1 --2- 172.21.15.23:0/99214289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7c0100630 0x7fe7c0199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.201+0000 7fe7c5647640 1 -- 172.21.15.23:0/99214289 >> 172.21.15.23:0/99214289 conn(0x7fe7c00fbba0 msgr2=0x7fe7c0104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.201+0000 7fe7c5647640 1 -- 172.21.15.23:0/99214289 shutdown_connections 2024-03-24T10:38:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.201+0000 7fe7c5647640 1 -- 172.21.15.23:0/99214289 wait complete. 2024-03-24T10:38:11.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:38:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:10 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2305713904' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T10:38:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:10 smithi117 ceph-mon[36368]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T10:38:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:10 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3803463327' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T10:38:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:10 smithi117 ceph-mon[36368]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T10:38:11.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:10 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2305713904' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T10:38:11.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:10 smithi116 ceph-mon[36248]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T10:38:11.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:10 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3803463327' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T10:38:11.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:10 smithi116 ceph-mon[36248]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T10:38:11.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.314+0000 7f7869ac0640 1 -- 172.21.15.23:0/3903105893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78641021e0 msgr2=0x7f7864108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:11.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.314+0000 7f7869ac0640 1 --2- 172.21.15.23:0/3903105893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78641021e0 0x7f7864108a70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f785c009900 tx=0x7f785c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:38:11.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.314+0000 7f7869ac0640 1 -- 172.21.15.23:0/3903105893 shutdown_connections 2024-03-24T10:38:11.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.314+0000 7f7869ac0640 1 --2- 172.21.15.23:0/3903105893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78641021e0 0x7f7864108a70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.314+0000 7f7869ac0640 1 --2- 172.21.15.23:0/3903105893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7864101820 0x7f7864101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.314+0000 7f7869ac0640 1 --2- 172.21.15.23:0/3903105893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7864100630 0x7f7864100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.314+0000 7f7869ac0640 1 -- 172.21.15.23:0/3903105893 >> 172.21.15.23:0/3903105893 conn(0x7f78640fbba0 msgr2=0x7f78640fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:11.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.315+0000 7f7869ac0640 1 -- 172.21.15.23:0/3903105893 shutdown_connections 2024-03-24T10:38:11.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.315+0000 7f7869ac0640 1 -- 172.21.15.23:0/3903105893 wait complete. 2024-03-24T10:38:11.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.315+0000 7f7869ac0640 1 Processor -- start 2024-03-24T10:38:11.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.315+0000 7f7869ac0640 1 -- start start 2024-03-24T10:38:11.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.315+0000 7f7869ac0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7864100630 0x7f7864197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:11.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.315+0000 7f7869ac0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7864101820 0x7f7864197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:11.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.315+0000 7f7869ac0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78641021e0 0x7f786419ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:11.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.315+0000 7f7869ac0640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f786419f240 con 0x7f7864101820 2024-03-24T10:38:11.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.315+0000 7f7869ac0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f786419f3b0 con 0x7f78641021e0 2024-03-24T10:38:11.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.315+0000 7f7869ac0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f786419f570 con 0x7f7864100630 2024-03-24T10:38:11.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.315+0000 7f7862ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7864101820 0x7f7864197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:11.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.315+0000 7f7862ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7864101820 0x7f7864197c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35186/0 (socket says 172.21.15.23:35186) 2024-03-24T10:38:11.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.315+0000 7f7862ffd640 1 -- 172.21.15.23:0/3323361748 learned_addr learned my addr 172.21.15.23:0/3323361748 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:11.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.316+0000 7f7863fff640 1 --2- 172.21.15.23:0/3323361748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78641021e0 0x7f786419ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:11.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.316+0000 7f7862ffd640 1 -- 172.21.15.23:0/3323361748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7864100630 msgr2=0x7f7864197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:11.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.316+0000 7f78637fe640 1 --2- 172.21.15.23:0/3323361748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7864100630 0x7f7864197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:11.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.316+0000 7f7862ffd640 1 --2- 172.21.15.23:0/3323361748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7864100630 0x7f7864197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.316+0000 7f7862ffd640 1 -- 172.21.15.23:0/3323361748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78641021e0 msgr2=0x7f786419ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:11.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.316+0000 7f7862ffd640 1 --2- 172.21.15.23:0/3323361748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78641021e0 0x7f786419ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.316+0000 7f7862ffd640 1 -- 172.21.15.23:0/3323361748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785c009560 con 0x7f7864101820 2024-03-24T10:38:11.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.316+0000 7f7862ffd640 1 --2- 172.21.15.23:0/3323361748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7864101820 0x7f7864197c90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f785800e9f0 tx=0x7f785800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:11.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.316+0000 7f7860ff9640 1 -- 172.21.15.23:0/3323361748 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7858009700 con 0x7f7864101820 2024-03-24T10:38:11.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.316+0000 7f7860ff9640 1 -- 172.21.15.23:0/3323361748 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f785800cd80 con 0x7f7864101820 2024-03-24T10:38:11.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.316+0000 7f7869ac0640 1 -- 172.21.15.23:0/3323361748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786419f850 con 0x7f7864101820 2024-03-24T10:38:11.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.316+0000 7f7860ff9640 1 -- 172.21.15.23:0/3323361748 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7858018920 con 0x7f7864101820 2024-03-24T10:38:11.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.316+0000 7f7869ac0640 1 -- 172.21.15.23:0/3323361748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786419fdb0 con 0x7f7864101820 2024-03-24T10:38:11.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.317+0000 7f7869ac0640 1 -- 172.21.15.23:0/3323361748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f786410c040 con 0x7f7864101820 2024-03-24T10:38:11.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.318+0000 7f7860ff9640 1 -- 172.21.15.23:0/3323361748 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f7858018a80 con 0x7f7864101820 2024-03-24T10:38:11.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.319+0000 7f7860ff9640 1 --2- 172.21.15.23:0/3323361748 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f784c0a2140 0x7f784c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:11.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.319+0000 7f78637fe640 1 --2- 172.21.15.23:0/3323361748 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f784c0a2140 0x7f784c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:11.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.319+0000 7f7860ff9640 1 -- 172.21.15.23:0/3323361748 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f7858014070 con 0x7f7864101820 2024-03-24T10:38:11.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.319+0000 7f78637fe640 1 --2- 172.21.15.23:0/3323361748 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f784c0a2140 0x7f784c0a4600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7854005c40 tx=0x7f7854005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:11.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.323+0000 7f7860ff9640 1 -- 172.21.15.23:0/3323361748 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7858053280 con 0x7f7864101820 2024-03-24T10:38:11.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.508+0000 7f7869ac0640 1 -- 172.21.15.23:0/3323361748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f786410c1d0 con 0x7f7864101820 2024-03-24T10:38:11.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.510+0000 7f7860ff9640 1 -- 172.21.15.23:0/3323361748 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22246 (secure 0 0 0) 0x7f7858097010 con 0x7f7864101820 2024-03-24T10:38:11.510 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:38:11.511 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":84,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:38:10.206457+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:38:11.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.512+0000 7f7869ac0640 1 -- 172.21.15.23:0/3323361748 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f784c0a2140 msgr2=0x7f784c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:11.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.512+0000 7f7869ac0640 1 --2- 172.21.15.23:0/3323361748 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f784c0a2140 0x7f784c0a4600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7854005c40 tx=0x7f7854005bd0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.512+0000 7f7869ac0640 1 -- 172.21.15.23:0/3323361748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7864101820 msgr2=0x7f7864197c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:11.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.512+0000 7f7869ac0640 1 --2- 172.21.15.23:0/3323361748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7864101820 0x7f7864197c90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f785800e9f0 tx=0x7f785800eeb0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.512+0000 7f7869ac0640 1 -- 172.21.15.23:0/3323361748 shutdown_connections 2024-03-24T10:38:11.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.512+0000 7f7869ac0640 1 --2- 172.21.15.23:0/3323361748 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f784c0a2140 0x7f784c0a4600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.512+0000 7f7869ac0640 1 --2- 172.21.15.23:0/3323361748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78641021e0 0x7f786419ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.512+0000 7f7869ac0640 1 --2- 172.21.15.23:0/3323361748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7864101820 0x7f7864197c90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.512+0000 7f7869ac0640 1 --2- 172.21.15.23:0/3323361748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7864100630 0x7f7864197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.512+0000 7f7869ac0640 1 -- 172.21.15.23:0/3323361748 >> 172.21.15.23:0/3323361748 conn(0x7f78640fbba0 msgr2=0x7f78640fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:11.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.512+0000 7f7869ac0640 1 -- 172.21.15.23:0/3323361748 shutdown_connections 2024-03-24T10:38:11.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.512+0000 7f7869ac0640 1 -- 172.21.15.23:0/3323361748 wait complete. 2024-03-24T10:38:11.524 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T10:38:11.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.628+0000 7f08939e2640 1 -- 172.21.15.23:0/3837479115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c1073f0 msgr2=0x7f088c109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:11.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.628+0000 7f08939e2640 1 --2- 172.21.15.23:0/3837479115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c1073f0 0x7f088c109840 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f087c009ca0 tx=0x7f087c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.628+0000 7f08939e2640 1 -- 172.21.15.23:0/3837479115 shutdown_connections 2024-03-24T10:38:11.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.628+0000 7f08939e2640 1 --2- 172.21.15.23:0/3837479115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c1073f0 0x7f088c109840 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.628+0000 7f08939e2640 1 --2- 172.21.15.23:0/3837479115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c104ac0 0x7f088c106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.628+0000 7f08939e2640 1 --2- 172.21.15.23:0/3837479115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f088c102190 0x7f088c104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.628+0000 7f08939e2640 1 -- 172.21.15.23:0/3837479115 >> 172.21.15.23:0/3837479115 conn(0x7f088c0fbba0 msgr2=0x7f088c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:11.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f08939e2640 1 -- 172.21.15.23:0/3837479115 shutdown_connections 2024-03-24T10:38:11.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f08939e2640 1 -- 172.21.15.23:0/3837479115 wait complete. 2024-03-24T10:38:11.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f08939e2640 1 Processor -- start 2024-03-24T10:38:11.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f08939e2640 1 -- start start 2024-03-24T10:38:11.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f08939e2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f088c102190 0x7f088c1977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:11.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f08939e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c104ac0 0x7f088c197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:11.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f08939e2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c1073f0 0x7f088c19ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:11.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f08939e2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f088c19f290 con 0x7f088c1073f0 2024-03-24T10:38:11.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f08939e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f088c19f400 con 0x7f088c104ac0 2024-03-24T10:38:11.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f08939e2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f088c19f5c0 con 0x7f088c102190 2024-03-24T10:38:11.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f0891f7f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c1073f0 0x7f088c19ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:11.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f0891f7f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c1073f0 0x7f088c19ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35202/0 (socket says 172.21.15.23:35202) 2024-03-24T10:38:11.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f0891f7f640 1 -- 172.21.15.23:0/2860961868 learned_addr learned my addr 172.21.15.23:0/2860961868 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:11.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f089177e640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f088c102190 0x7f088c1977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:11.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.629+0000 7f0890f7d640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c104ac0 0x7f088c197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:11.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.630+0000 7f0891f7f640 1 -- 172.21.15.23:0/2860961868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f088c102190 msgr2=0x7f088c1977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:11.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.630+0000 7f0891f7f640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f088c102190 0x7f088c1977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.630+0000 7f0891f7f640 1 -- 172.21.15.23:0/2860961868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c104ac0 msgr2=0x7f088c197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:11.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.630+0000 7f0891f7f640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c104ac0 0x7f088c197ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:11.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.630+0000 7f0891f7f640 1 -- 172.21.15.23:0/2860961868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f087c009900 con 0x7f088c1073f0 2024-03-24T10:38:11.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.630+0000 7f0890f7d640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c104ac0 0x7f088c197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:38:11.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.630+0000 7f0891f7f640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c1073f0 0x7f088c19ed50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f087c037040 tx=0x7f087c034a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:11.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.630+0000 7f088a7fc640 1 -- 172.21.15.23:0/2860961868 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c0329b0 con 0x7f088c1073f0 2024-03-24T10:38:11.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.630+0000 7f088a7fc640 1 -- 172.21.15.23:0/2860961868 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f087c003e50 con 0x7f088c1073f0 2024-03-24T10:38:11.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.630+0000 7f08939e2640 1 -- 172.21.15.23:0/2860961868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f088c19f840 con 0x7f088c1073f0 2024-03-24T10:38:11.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.630+0000 7f088a7fc640 1 -- 172.21.15.23:0/2860961868 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c040450 con 0x7f088c1073f0 2024-03-24T10:38:11.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.630+0000 7f08939e2640 1 -- 172.21.15.23:0/2860961868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f088c19fd40 con 0x7f088c1073f0 2024-03-24T10:38:11.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.631+0000 7f08939e2640 1 -- 172.21.15.23:0/2860961868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f088c0fd690 con 0x7f088c1073f0 2024-03-24T10:38:11.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.632+0000 7f088a7fc640 1 -- 172.21.15.23:0/2860961868 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f087c00a040 con 0x7f088c1073f0 2024-03-24T10:38:11.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.633+0000 7f088a7fc640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f08740a2140 0x7f08740a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:11.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.633+0000 7f089177e640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f08740a2140 0x7f08740a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:11.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.633+0000 7f088a7fc640 1 -- 172.21.15.23:0/2860961868 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f087c0f0640 con 0x7f088c1073f0 2024-03-24T10:38:11.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.633+0000 7f089177e640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f08740a2140 0x7f08740a4600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0884005fd0 tx=0x7f088400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:11.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.637+0000 7f088a7fc640 1 -- 172.21.15.23:0/2860961868 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f087c0b9760 con 0x7f088c1073f0 2024-03-24T10:38:11.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:11.860+0000 7f08939e2640 1 -- 172.21.15.23:0/2860961868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f088c02d130 con 0x7f088c1073f0 2024-03-24T10:38:12.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:11 smithi023 ceph-mon[32466]: pgmap v281: 97 pgs: 6 creating+peering, 7 active+clean, 84 unknown; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:12.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:11 smithi023 ceph-mon[32466]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T10:38:12.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:11 smithi023 ceph-mon[32466]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T10:38:12.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:11 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3803463327' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T10:38:12.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:11 smithi023 ceph-mon[32466]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T10:38:12.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:11 smithi023 ceph-mon[32466]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T10:38:12.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:11 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] up:active 2024-03-24T10:38:12.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:11 smithi023 ceph-mon[32466]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T10:38:12.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:11 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/99214289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:12.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:11 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3323361748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:12.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:11 smithi117 ceph-mon[36368]: pgmap v281: 97 pgs: 6 creating+peering, 7 active+clean, 84 unknown; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:12.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:11 smithi117 ceph-mon[36368]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T10:38:12.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:11 smithi117 ceph-mon[36368]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T10:38:12.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:11 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3803463327' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T10:38:12.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:11 smithi117 ceph-mon[36368]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T10:38:12.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:11 smithi117 ceph-mon[36368]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T10:38:12.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:11 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] up:active 2024-03-24T10:38:12.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:11 smithi117 ceph-mon[36368]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T10:38:12.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:11 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/99214289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:12.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:11 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3323361748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:12.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:11 smithi116 ceph-mon[36248]: pgmap v281: 97 pgs: 6 creating+peering, 7 active+clean, 84 unknown; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:38:12.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:11 smithi116 ceph-mon[36248]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T10:38:12.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:11 smithi116 ceph-mon[36248]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T10:38:12.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:11 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3803463327' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T10:38:12.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:11 smithi116 ceph-mon[36248]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T10:38:12.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:11 smithi116 ceph-mon[36248]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T10:38:12.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:11 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] up:active 2024-03-24T10:38:12.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:11 smithi116 ceph-mon[36248]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T10:38:12.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:11 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/99214289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:12.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:11 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3323361748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:12.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.866+0000 7f088a7fc640 1 -- 172.21.15.23:0/2860961868 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7f087c0bce20 con 0x7f088c1073f0 2024-03-24T10:38:12.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.869+0000 7f08939e2640 1 -- 172.21.15.23:0/2860961868 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f08740a2140 msgr2=0x7f08740a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:12.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.869+0000 7f08939e2640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f08740a2140 0x7f08740a4600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0884005fd0 tx=0x7f088400b390 comp rx=0 tx=0).stop 2024-03-24T10:38:12.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.869+0000 7f08939e2640 1 -- 172.21.15.23:0/2860961868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c1073f0 msgr2=0x7f088c19ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:12.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.869+0000 7f08939e2640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c1073f0 0x7f088c19ed50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f087c037040 tx=0x7f087c034a90 comp rx=0 tx=0).stop 2024-03-24T10:38:12.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.869+0000 7f08939e2640 1 -- 172.21.15.23:0/2860961868 shutdown_connections 2024-03-24T10:38:12.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.869+0000 7f08939e2640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f08740a2140 0x7f08740a4600 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:12.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.869+0000 7f08939e2640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f088c1073f0 0x7f088c19ed50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:12.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.869+0000 7f08939e2640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c104ac0 0x7f088c197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:12.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.869+0000 7f08939e2640 1 --2- 172.21.15.23:0/2860961868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f088c102190 0x7f088c1977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:12.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.870+0000 7f08939e2640 1 -- 172.21.15.23:0/2860961868 >> 172.21.15.23:0/2860961868 conn(0x7f088c0fbba0 msgr2=0x7f088c102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:12.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.870+0000 7f08939e2640 1 -- 172.21.15.23:0/2860961868 shutdown_connections 2024-03-24T10:38:12.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.870+0000 7f08939e2640 1 -- 172.21.15.23:0/2860961868 wait complete. 2024-03-24T10:38:12.884 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-24T10:38:12.884 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-24T10:38:12.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.995+0000 7fb813455640 1 -- 172.21.15.23:0/2874178153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb80c101720 msgr2=0x7fb80c101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:12.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.995+0000 7fb813455640 1 --2- 172.21.15.23:0/2874178153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb80c101720 0x7fb80c101b40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb808009900 tx=0x7fb80802bc30 comp rx=0 tx=0).stop 2024-03-24T10:38:12.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.995+0000 7fb813455640 1 -- 172.21.15.23:0/2874178153 shutdown_connections 2024-03-24T10:38:12.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.995+0000 7fb813455640 1 --2- 172.21.15.23:0/2874178153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb80c1032d0 0x7fb80c109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:12.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.995+0000 7fb813455640 1 --2- 172.21.15.23:0/2874178153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb80c102910 0x7fb80c102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:12.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.995+0000 7fb813455640 1 --2- 172.21.15.23:0/2874178153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb80c101720 0x7fb80c101b40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:12.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.995+0000 7fb813455640 1 -- 172.21.15.23:0/2874178153 >> 172.21.15.23:0/2874178153 conn(0x7fb80c0fccb0 msgr2=0x7fb80c0ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:12.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.995+0000 7fb813455640 1 -- 172.21.15.23:0/2874178153 shutdown_connections 2024-03-24T10:38:12.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.995+0000 7fb813455640 1 -- 172.21.15.23:0/2874178153 wait complete. 2024-03-24T10:38:12.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb813455640 1 Processor -- start 2024-03-24T10:38:12.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb813455640 1 -- start start 2024-03-24T10:38:12.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb813455640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb80c101720 0x7fb80c197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:12.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb813455640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb80c102910 0x7fb80c197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:12.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb813455640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb80c1032d0 0x7fb80c19edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:12.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb813455640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c19f300 con 0x7fb80c101720 2024-03-24T10:38:12.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb813455640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c19f470 con 0x7fb80c1032d0 2024-03-24T10:38:12.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb813455640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c19f630 con 0x7fb80c102910 2024-03-24T10:38:12.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb8111f1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb80c101720 0x7fb80c197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:12.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb8111f1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb80c101720 0x7fb80c197810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35222/0 (socket says 172.21.15.23:35222) 2024-03-24T10:38:12.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb8111f1640 1 -- 172.21.15.23:0/3790343003 learned_addr learned my addr 172.21.15.23:0/3790343003 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:12.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb8119f2640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb80c1032d0 0x7fb80c19edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:13.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb8109f0640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb80c102910 0x7fb80c197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:13.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb8111f1640 1 -- 172.21.15.23:0/3790343003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb80c102910 msgr2=0x7fb80c197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:13.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb8111f1640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb80c102910 0x7fb80c197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:13.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb8111f1640 1 -- 172.21.15.23:0/3790343003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb80c1032d0 msgr2=0x7fb80c19edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:13.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb8111f1640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb80c1032d0 0x7fb80c19edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:13.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb8111f1640 1 -- 172.21.15.23:0/3790343003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb808009560 con 0x7fb80c101720 2024-03-24T10:38:13.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.996+0000 7fb8109f0640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb80c102910 0x7fb80c197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:38:13.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.997+0000 7fb8111f1640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb80c101720 0x7fb80c197810 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb808005ab0 tx=0x7fb808004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:13.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.997+0000 7fb8027fc640 1 -- 172.21.15.23:0/3790343003 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb808004aa0 con 0x7fb80c101720 2024-03-24T10:38:13.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.997+0000 7fb8027fc640 1 -- 172.21.15.23:0/3790343003 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8080052d0 con 0x7fb80c101720 2024-03-24T10:38:13.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.997+0000 7fb813455640 1 -- 172.21.15.23:0/3790343003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb80c19f8b0 con 0x7fb80c101720 2024-03-24T10:38:13.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.997+0000 7fb8027fc640 1 -- 172.21.15.23:0/3790343003 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb808003e80 con 0x7fb80c101720 2024-03-24T10:38:13.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.997+0000 7fb813455640 1 -- 172.21.15.23:0/3790343003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb80c19fdc0 con 0x7fb80c101720 2024-03-24T10:38:13.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.998+0000 7fb813455640 1 -- 172.21.15.23:0/3790343003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb80c108ab0 con 0x7fb80c101720 2024-03-24T10:38:13.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.999+0000 7fb8027fc640 1 -- 172.21.15.23:0/3790343003 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fb808005440 con 0x7fb80c101720 2024-03-24T10:38:13.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.999+0000 7fb8027fc640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb7f80a20c0 0x7fb7f80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:13.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:12.999+0000 7fb8109f0640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb7f80a20c0 0x7fb7f80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:13.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.000+0000 7fb8027fc640 1 -- 172.21.15.23:0/3790343003 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fb8080f0050 con 0x7fb80c101720 2024-03-24T10:38:13.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.000+0000 7fb8109f0640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb7f80a20c0 0x7fb7f80a4580 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc005d00 tx=0x7fb7fc005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:13.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.004+0000 7fb8027fc640 1 -- 172.21.15.23:0/3790343003 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb8080b9170 con 0x7fb80c101720 2024-03-24T10:38:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:12 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2860961868' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T10:38:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:12 smithi023 ceph-mon[32466]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T10:38:13.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.225+0000 7fb813455640 1 -- 172.21.15.23:0/3790343003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fb80c02d1d0 con 0x7fb80c101720 2024-03-24T10:38:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:12 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2860961868' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T10:38:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:12 smithi117 ceph-mon[36368]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T10:38:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:12 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2860961868' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T10:38:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:12 smithi116 ceph-mon[36248]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T10:38:13.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.877+0000 7fb8027fc640 1 -- 172.21.15.23:0/3790343003 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v17) ==== 105+0+0 (secure 0 0 0) 0x7fb8080bc830 con 0x7fb80c101720 2024-03-24T10:38:13.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.879+0000 7fb813455640 1 -- 172.21.15.23:0/3790343003 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb7f80a20c0 msgr2=0x7fb7f80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:13.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.879+0000 7fb813455640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb7f80a20c0 0x7fb7f80a4580 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc005d00 tx=0x7fb7fc005c70 comp rx=0 tx=0).stop 2024-03-24T10:38:13.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.879+0000 7fb813455640 1 -- 172.21.15.23:0/3790343003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb80c101720 msgr2=0x7fb80c197810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:13.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.879+0000 7fb813455640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb80c101720 0x7fb80c197810 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb808005ab0 tx=0x7fb808004270 comp rx=0 tx=0).stop 2024-03-24T10:38:13.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.879+0000 7fb813455640 1 -- 172.21.15.23:0/3790343003 shutdown_connections 2024-03-24T10:38:13.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.879+0000 7fb813455640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb7f80a20c0 0x7fb7f80a4580 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:13.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.879+0000 7fb813455640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb80c1032d0 0x7fb80c19edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:13.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.879+0000 7fb813455640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb80c102910 0x7fb80c197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:13.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.879+0000 7fb813455640 1 --2- 172.21.15.23:0/3790343003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb80c101720 0x7fb80c197810 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:13.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.879+0000 7fb813455640 1 -- 172.21.15.23:0/3790343003 >> 172.21.15.23:0/3790343003 conn(0x7fb80c0fccb0 msgr2=0x7fb80c105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:13.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.879+0000 7fb813455640 1 -- 172.21.15.23:0/3790343003 shutdown_connections 2024-03-24T10:38:13.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:13.880+0000 7fb813455640 1 -- 172.21.15.23:0/3790343003 wait complete. 2024-03-24T10:38:13.896 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-24T10:38:14.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.006+0000 7f2c627b6640 1 -- 172.21.15.23:0/2302049969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c5c1021c0 msgr2=0x7f2c5c108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:14.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.006+0000 7f2c627b6640 1 --2- 172.21.15.23:0/2302049969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c5c1021c0 0x7f2c5c108a50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2c54009ca0 tx=0x7f2c5402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:38:14.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.006+0000 7f2c627b6640 1 -- 172.21.15.23:0/2302049969 shutdown_connections 2024-03-24T10:38:14.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.006+0000 7f2c627b6640 1 --2- 172.21.15.23:0/2302049969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c5c1021c0 0x7f2c5c108a50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:14.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.006+0000 7f2c627b6640 1 --2- 172.21.15.23:0/2302049969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c5c101800 0x7f2c5c101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:14.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.006+0000 7f2c627b6640 1 --2- 172.21.15.23:0/2302049969 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c5c100610 0x7f2c5c100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:14.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.006+0000 7f2c627b6640 1 -- 172.21.15.23:0/2302049969 >> 172.21.15.23:0/2302049969 conn(0x7f2c5c0fbba0 msgr2=0x7f2c5c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:14.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.006+0000 7f2c627b6640 1 -- 172.21.15.23:0/2302049969 shutdown_connections 2024-03-24T10:38:14.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.006+0000 7f2c627b6640 1 -- 172.21.15.23:0/2302049969 wait complete. 2024-03-24T10:38:14.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.007+0000 7f2c627b6640 1 Processor -- start 2024-03-24T10:38:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.007+0000 7f2c627b6640 1 -- start start 2024-03-24T10:38:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.007+0000 7f2c627b6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c5c100610 0x7f2c5c1977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.007+0000 7f2c627b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c5c101800 0x7f2c5c197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.007+0000 7f2c627b6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c5c1021c0 0x7f2c5c19ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.007+0000 7f2c627b6640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2c5c19f290 con 0x7f2c5c100610 2024-03-24T10:38:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.007+0000 7f2c627b6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c5c19f400 con 0x7f2c5c101800 2024-03-24T10:38:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.007+0000 7f2c627b6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2c5c19f5c0 con 0x7f2c5c1021c0 2024-03-24T10:38:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.007+0000 7f2c5bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c5c100610 0x7f2c5c1977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.007+0000 7f2c5bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c5c100610 0x7f2c5c1977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35244/0 (socket says 172.21.15.23:35244) 2024-03-24T10:38:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.007+0000 7f2c5bfff640 1 -- 172.21.15.23:0/2054791271 learned_addr learned my addr 172.21.15.23:0/2054791271 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.007+0000 7f2c5b7fe640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c5c101800 0x7f2c5c197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.007+0000 7f2c60d53640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c5c1021c0 0x7f2c5c19ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.008+0000 7f2c5bfff640 1 -- 172.21.15.23:0/2054791271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c5c1021c0 msgr2=0x7f2c5c19ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:14.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.008+0000 7f2c5bfff640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c5c1021c0 0x7f2c5c19ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:14.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.008+0000 7f2c5bfff640 1 -- 172.21.15.23:0/2054791271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c5c101800 msgr2=0x7f2c5c197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:14.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.008+0000 7f2c5bfff640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c5c101800 0x7f2c5c197ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:14.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.008+0000 7f2c5bfff640 1 -- 172.21.15.23:0/2054791271 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c54009900 con 0x7f2c5c100610 2024-03-24T10:38:14.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.008+0000 7f2c5b7fe640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c5c101800 0x7f2c5c197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:14.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.008+0000 7f2c5bfff640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c5c100610 0x7f2c5c1977a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00d950 tx=0x7f2c4c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:14.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.008+0000 7f2c597fa640 1 -- 172.21.15.23:0/2054791271 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c009c30 con 0x7f2c5c100610 2024-03-24T10:38:14.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.008+0000 7f2c597fa640 1 -- 172.21.15.23:0/2054791271 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c4c009d90 con 0x7f2c5c100610 2024-03-24T10:38:14.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.008+0000 7f2c627b6640 1 -- 172.21.15.23:0/2054791271 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c5c19f8a0 con 0x7f2c5c100610 2024-03-24T10:38:14.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.008+0000 7f2c597fa640 1 -- 172.21.15.23:0/2054791271 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c00f720 con 0x7f2c5c100610 2024-03-24T10:38:14.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.008+0000 7f2c627b6640 1 -- 172.21.15.23:0/2054791271 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c5c19fd60 con 0x7f2c5c100610 2024-03-24T10:38:14.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.009+0000 7f2c627b6640 1 -- 172.21.15.23:0/2054791271 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c5c10c020 con 0x7f2c5c100610 2024-03-24T10:38:14.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.010+0000 7f2c597fa640 1 -- 172.21.15.23:0/2054791271 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f2c4c010a60 con 0x7f2c5c100610 2024-03-24T10:38:14.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.010+0000 7f2c597fa640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2c440a20c0 0x7f2c440a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:14.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.011+0000 7f2c5b7fe640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2c440a20c0 0x7f2c440a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:14.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.011+0000 7f2c597fa640 1 -- 172.21.15.23:0/2054791271 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f2c4c0cd850 con 0x7f2c5c100610 2024-03-24T10:38:14.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.011+0000 7f2c5b7fe640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2c440a20c0 0x7f2c440a4580 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c198cb0 tx=0x7f2c50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:14.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.015+0000 7f2c597fa640 1 -- 172.21.15.23:0/2054791271 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2c4c096a20 con 0x7f2c5c100610 2024-03-24T10:38:14.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:13 smithi023 ceph-mon[32466]: pgmap v284: 97 pgs: 14 creating+peering, 63 active+clean, 20 unknown; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-24T10:38:14.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:13 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2860961868' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T10:38:14.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:13 smithi023 ceph-mon[32466]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T10:38:14.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:13 smithi023 ceph-mon[32466]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-24T10:38:14.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:13 smithi023 ceph-mon[32466]: Cluster is now healthy 2024-03-24T10:38:14.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:13 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3790343003' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-24T10:38:14.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.239+0000 7f2c627b6640 1 -- 172.21.15.23:0/2054791271 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f2c5c19ff80 con 0x7f2c5c100610 2024-03-24T10:38:14.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:13 smithi117 ceph-mon[36368]: pgmap v284: 97 pgs: 14 creating+peering, 63 active+clean, 20 unknown; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-24T10:38:14.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:13 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2860961868' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T10:38:14.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:13 smithi117 ceph-mon[36368]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T10:38:14.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:13 smithi117 ceph-mon[36368]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-24T10:38:14.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:13 smithi117 ceph-mon[36368]: Cluster is now healthy 2024-03-24T10:38:14.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:13 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3790343003' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-24T10:38:14.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:13 smithi116 ceph-mon[36248]: pgmap v284: 97 pgs: 14 creating+peering, 63 active+clean, 20 unknown; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-24T10:38:14.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:13 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2860961868' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T10:38:14.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:13 smithi116 ceph-mon[36248]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T10:38:14.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:13 smithi116 ceph-mon[36248]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-24T10:38:14.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:13 smithi116 ceph-mon[36248]: Cluster is now healthy 2024-03-24T10:38:14.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:13 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3790343003' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-24T10:38:14.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.892+0000 7f2c597fa640 1 -- 172.21.15.23:0/2054791271 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f2c5c19ff80 con 0x7f2c5c100610 2024-03-24T10:38:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.895+0000 7f2c627b6640 1 -- 172.21.15.23:0/2054791271 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2c440a20c0 msgr2=0x7f2c440a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.895+0000 7f2c627b6640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2c440a20c0 0x7f2c440a4580 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c198cb0 tx=0x7f2c50005f90 comp rx=0 tx=0).stop 2024-03-24T10:38:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.895+0000 7f2c627b6640 1 -- 172.21.15.23:0/2054791271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c5c100610 msgr2=0x7f2c5c1977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.895+0000 7f2c627b6640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c5c100610 0x7f2c5c1977a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00d950 tx=0x7f2c4c00de10 comp rx=0 tx=0).stop 2024-03-24T10:38:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.895+0000 7f2c627b6640 1 -- 172.21.15.23:0/2054791271 shutdown_connections 2024-03-24T10:38:14.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.895+0000 7f2c627b6640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2c440a20c0 0x7f2c440a4580 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:14.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.895+0000 7f2c627b6640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c5c1021c0 0x7f2c5c19ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:14.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.895+0000 7f2c627b6640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c5c101800 0x7f2c5c197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:14.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.895+0000 7f2c627b6640 1 --2- 172.21.15.23:0/2054791271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c5c100610 0x7f2c5c1977a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:14.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.895+0000 7f2c627b6640 1 -- 172.21.15.23:0/2054791271 >> 172.21.15.23:0/2054791271 conn(0x7f2c5c0fbba0 msgr2=0x7f2c5c0fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:14.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.895+0000 7f2c627b6640 1 -- 172.21.15.23:0/2054791271 shutdown_connections 2024-03-24T10:38:14.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:14.895+0000 7f2c627b6640 1 -- 172.21.15.23:0/2054791271 wait complete. 2024-03-24T10:38:14.909 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-24T10:38:15.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.014+0000 7f8fc64da640 1 -- 172.21.15.23:0/3433624024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fc01032d0 msgr2=0x7f8fc0109b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:15.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.014+0000 7f8fc64da640 1 --2- 172.21.15.23:0/3433624024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fc01032d0 0x7f8fc0109b60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009ca0 tx=0x7f8fb802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:38:15.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.014+0000 7f8fc64da640 1 -- 172.21.15.23:0/3433624024 shutdown_connections 2024-03-24T10:38:15.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.014+0000 7f8fc64da640 1 --2- 172.21.15.23:0/3433624024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fc01032d0 0x7f8fc0109b60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:15.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.014+0000 7f8fc64da640 1 --2- 172.21.15.23:0/3433624024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fc0102910 0x7f8fc0102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:15.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.014+0000 7f8fc64da640 1 --2- 172.21.15.23:0/3433624024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fc0101720 0x7f8fc0101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:15.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.014+0000 7f8fc64da640 1 -- 172.21.15.23:0/3433624024 >> 172.21.15.23:0/3433624024 conn(0x7f8fc00fccb0 msgr2=0x7f8fc00ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:15.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.014+0000 7f8fc64da640 1 -- 172.21.15.23:0/3433624024 shutdown_connections 2024-03-24T10:38:15.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.014+0000 7f8fc64da640 1 -- 172.21.15.23:0/3433624024 wait complete. 2024-03-24T10:38:15.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc64da640 1 Processor -- start 2024-03-24T10:38:15.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc64da640 1 -- start start 2024-03-24T10:38:15.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc64da640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fc0101720 0x7f8fc00754c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc64da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fc0102910 0x7f8fc0073b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc64da640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fc01032d0 0x7f8fc0074040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc64da640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc00747a0 con 0x7f8fc01032d0 2024-03-24T10:38:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc64da640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc0074910 con 0x7f8fc0102910 2024-03-24T10:38:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc64da640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc01a17d0 con 0x7f8fc0101720 2024-03-24T10:38:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc4a77640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fc01032d0 0x7f8fc0074040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc4a77640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fc01032d0 0x7f8fc0074040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35268/0 (socket says 172.21.15.23:35268) 2024-03-24T10:38:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fbffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fc0101720 0x7f8fc00754c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc4a77640 1 -- 172.21.15.23:0/4252520387 learned_addr learned my addr 172.21.15.23:0/4252520387 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fbf7fe640 1 --2- 172.21.15.23:0/4252520387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fc0102910 0x7f8fc0073b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc4a77640 1 -- 172.21.15.23:0/4252520387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fc0101720 msgr2=0x7f8fc00754c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc4a77640 1 --2- 172.21.15.23:0/4252520387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fc0101720 0x7f8fc00754c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc4a77640 1 -- 172.21.15.23:0/4252520387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fc0102910 msgr2=0x7f8fc0073b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:15.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc4a77640 1 --2- 172.21.15.23:0/4252520387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fc0102910 0x7f8fc0073b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:15.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.015+0000 7f8fc4a77640 1 -- 172.21.15.23:0/4252520387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb8009900 con 0x7f8fc01032d0 2024-03-24T10:38:15.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.016+0000 7f8fc4a77640 1 --2- 172.21.15.23:0/4252520387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fc01032d0 0x7f8fc0074040 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009bf0 tx=0x7f8fb8032f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:15.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.016+0000 7f8fbd7fa640 1 -- 172.21.15.23:0/4252520387 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8034e20 con 0x7f8fc01032d0 2024-03-24T10:38:15.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.016+0000 7f8fbd7fa640 1 -- 172.21.15.23:0/4252520387 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fb8003e50 con 0x7f8fc01032d0 2024-03-24T10:38:15.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.016+0000 7f8fc64da640 1 -- 172.21.15.23:0/4252520387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fc01a1a20 con 0x7f8fc01032d0 2024-03-24T10:38:15.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.016+0000 7f8fbd7fa640 1 -- 172.21.15.23:0/4252520387 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8040450 con 0x7f8fc01032d0 2024-03-24T10:38:15.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.016+0000 7f8fc64da640 1 -- 172.21.15.23:0/4252520387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fc01a1f20 con 0x7f8fc01032d0 2024-03-24T10:38:15.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.017+0000 7f8fc64da640 1 -- 172.21.15.23:0/4252520387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fc0108a90 con 0x7f8fc01032d0 2024-03-24T10:38:15.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.018+0000 7f8fbd7fa640 1 -- 172.21.15.23:0/4252520387 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f8fb800a040 con 0x7f8fc01032d0 2024-03-24T10:38:15.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.018+0000 7f8fbd7fa640 1 --2- 172.21.15.23:0/4252520387 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8fa80a2140 0x7f8fa80a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:15.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.018+0000 7f8fbffff640 1 --2- 172.21.15.23:0/4252520387 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8fa80a2140 0x7f8fa80a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:15.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.018+0000 7f8fbd7fa640 1 -- 172.21.15.23:0/4252520387 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f8fb80f0480 con 0x7f8fc01032d0 2024-03-24T10:38:15.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.019+0000 7f8fbffff640 1 --2- 172.21.15.23:0/4252520387 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8fa80a2140 0x7f8fa80a4600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0102770 tx=0x7f8fb0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.023+0000 7f8fbd7fa640 1 -- 172.21.15.23:0/4252520387 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8fb80b95a0 con 0x7f8fc01032d0 2024-03-24T10:38:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:14 smithi023 ceph-mon[32466]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T10:38:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:14 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3790343003' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-24T10:38:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:14 smithi023 ceph-mon[32466]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T10:38:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:14 smithi023 ceph-mon[32466]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T10:38:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:14 smithi023 ceph-mon[32466]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-24T10:38:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:14 smithi023 ceph-mon[32466]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T10:38:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:14 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2054791271' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T10:38:15.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.244+0000 7f8fc64da640 1 -- 172.21.15.23:0/4252520387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f8fc002d1d0 con 0x7f8fc01032d0 2024-03-24T10:38:15.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:14 smithi117 ceph-mon[36368]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T10:38:15.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:14 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3790343003' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-24T10:38:15.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:14 smithi117 ceph-mon[36368]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T10:38:15.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:14 smithi117 ceph-mon[36368]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T10:38:15.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:14 smithi117 ceph-mon[36368]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-24T10:38:15.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:14 smithi117 ceph-mon[36368]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T10:38:15.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:14 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2054791271' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T10:38:15.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:14 smithi116 ceph-mon[36248]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T10:38:15.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:14 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3790343003' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-24T10:38:15.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:14 smithi116 ceph-mon[36248]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T10:38:15.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:14 smithi116 ceph-mon[36248]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T10:38:15.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:14 smithi116 ceph-mon[36248]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-24T10:38:15.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:14 smithi116 ceph-mon[36248]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T10:38:15.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:14 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2054791271' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T10:38:15.768 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:38:15 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-b[66641]: 2024-03-24T10:38:15.447+0000 7f4a92cd3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T10:38:15.768 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:38:15 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-b[66641]: 2024-03-24T10:38:15.447+0000 7f4a92cd3700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T10:38:15.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.905+0000 7f8fbd7fa640 1 -- 172.21.15.23:0/4252520387 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7f8fb80bcc60 con 0x7f8fc01032d0 2024-03-24T10:38:15.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.909+0000 7f8fc64da640 1 -- 172.21.15.23:0/4252520387 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8fa80a2140 msgr2=0x7f8fa80a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:15.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.909+0000 7f8fc64da640 1 --2- 172.21.15.23:0/4252520387 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8fa80a2140 0x7f8fa80a4600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0102770 tx=0x7f8fb0005f90 comp rx=0 tx=0).stop 2024-03-24T10:38:15.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.909+0000 7f8fc64da640 1 -- 172.21.15.23:0/4252520387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fc01032d0 msgr2=0x7f8fc0074040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:15.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.909+0000 7f8fc64da640 1 --2- 172.21.15.23:0/4252520387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fc01032d0 0x7f8fc0074040 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009bf0 tx=0x7f8fb8032f80 comp rx=0 tx=0).stop 2024-03-24T10:38:15.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.909+0000 7f8fc64da640 1 -- 172.21.15.23:0/4252520387 shutdown_connections 2024-03-24T10:38:15.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.909+0000 7f8fc64da640 1 --2- 172.21.15.23:0/4252520387 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8fa80a2140 0x7f8fa80a4600 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:15.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.909+0000 7f8fc64da640 1 --2- 172.21.15.23:0/4252520387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fc01032d0 0x7f8fc0074040 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:15.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.909+0000 7f8fc64da640 1 --2- 172.21.15.23:0/4252520387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8fc0102910 0x7f8fc0073b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:15.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.909+0000 7f8fc64da640 1 --2- 172.21.15.23:0/4252520387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fc0101720 0x7f8fc00754c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:15.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.909+0000 7f8fc64da640 1 -- 172.21.15.23:0/4252520387 >> 172.21.15.23:0/4252520387 conn(0x7f8fc00fccb0 msgr2=0x7f8fc0105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:15.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.909+0000 7f8fc64da640 1 -- 172.21.15.23:0/4252520387 shutdown_connections 2024-03-24T10:38:15.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:15.909+0000 7f8fc64da640 1 -- 172.21.15.23:0/4252520387 wait complete. 2024-03-24T10:38:15.924 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-24T10:38:15.925 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-24T10:38:16.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.036+0000 7fcb3268d640 1 -- 172.21.15.23:0/3806960974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb2c1073f0 msgr2=0x7fcb2c109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.036+0000 7fcb3268d640 1 --2- 172.21.15.23:0/3806960974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb2c1073f0 0x7fcb2c109840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcb24009ca0 tx=0x7fcb2402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.036+0000 7fcb3268d640 1 -- 172.21.15.23:0/3806960974 shutdown_connections 2024-03-24T10:38:16.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.036+0000 7fcb3268d640 1 --2- 172.21.15.23:0/3806960974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb2c1073f0 0x7fcb2c109840 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.036+0000 7fcb3268d640 1 --2- 172.21.15.23:0/3806960974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb2c104ac0 0x7fcb2c106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.036+0000 7fcb3268d640 1 --2- 172.21.15.23:0/3806960974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c102190 0x7fcb2c104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.036+0000 7fcb3268d640 1 -- 172.21.15.23:0/3806960974 >> 172.21.15.23:0/3806960974 conn(0x7fcb2c0fbba0 msgr2=0x7fcb2c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:16.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.036+0000 7fcb3268d640 1 -- 172.21.15.23:0/3806960974 shutdown_connections 2024-03-24T10:38:16.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.036+0000 7fcb3268d640 1 -- 172.21.15.23:0/3806960974 wait complete. 2024-03-24T10:38:16.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.037+0000 7fcb3268d640 1 Processor -- start 2024-03-24T10:38:16.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.037+0000 7fcb3268d640 1 -- start start 2024-03-24T10:38:16.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.037+0000 7fcb3268d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb2c102190 0x7fcb2c1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.037+0000 7fcb3268d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c104ac0 0x7fcb2c199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.037+0000 7fcb3268d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb2c1073f0 0x7fcb2c1a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.037+0000 7fcb3268d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fcb2c1a14b0 con 0x7fcb2c1073f0 2024-03-24T10:38:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.037+0000 7fcb3268d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcb2c1a1620 con 0x7fcb2c104ac0 2024-03-24T10:38:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.037+0000 7fcb3268d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcb2c1a17e0 con 0x7fcb2c102190 2024-03-24T10:38:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.037+0000 7fcb30c2a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb2c1073f0 0x7fcb2c1a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.037+0000 7fcb30c2a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb2c1073f0 0x7fcb2c1a0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35284/0 (socket says 172.21.15.23:35284) 2024-03-24T10:38:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.037+0000 7fcb30c2a640 1 -- 172.21.15.23:0/164006661 learned_addr learned my addr 172.21.15.23:0/164006661 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.037+0000 7fcb2bfff640 1 --2- 172.21.15.23:0/164006661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb2c102190 0x7fcb2c1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.037+0000 7fcb2b7fe640 1 --2- 172.21.15.23:0/164006661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c104ac0 0x7fcb2c199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.038+0000 7fcb30c2a640 1 -- 172.21.15.23:0/164006661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb2c102190 msgr2=0x7fcb2c1999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.038+0000 7fcb30c2a640 1 --2- 172.21.15.23:0/164006661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb2c102190 0x7fcb2c1999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.038+0000 7fcb30c2a640 1 -- 172.21.15.23:0/164006661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c104ac0 msgr2=0x7fcb2c199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.038+0000 7fcb30c2a640 1 --2- 172.21.15.23:0/164006661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c104ac0 0x7fcb2c199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.038+0000 7fcb30c2a640 1 -- 172.21.15.23:0/164006661 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1c009490 con 0x7fcb2c1073f0 2024-03-24T10:38:16.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.038+0000 7fcb30c2a640 1 --2- 172.21.15.23:0/164006661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb2c1073f0 0x7fcb2c1a0f70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcb2402c630 tx=0x7fcb24032f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:16.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.038+0000 7fcb297fa640 1 -- 172.21.15.23:0/164006661 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb24034e20 con 0x7fcb2c1073f0 2024-03-24T10:38:16.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.038+0000 7fcb3268d640 1 -- 172.21.15.23:0/164006661 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb24009900 con 0x7fcb2c1073f0 2024-03-24T10:38:16.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.038+0000 7fcb297fa640 1 -- 172.21.15.23:0/164006661 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb24003e50 con 0x7fcb2c1073f0 2024-03-24T10:38:16.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.038+0000 7fcb297fa640 1 -- 172.21.15.23:0/164006661 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb24004880 con 0x7fcb2c1073f0 2024-03-24T10:38:16.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.038+0000 7fcb3268d640 1 -- 172.21.15.23:0/164006661 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb2c1a1d80 con 0x7fcb2c1073f0 2024-03-24T10:38:16.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.039+0000 7fcb3268d640 1 -- 172.21.15.23:0/164006661 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb2c108ee0 con 0x7fcb2c1073f0 2024-03-24T10:38:16.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.040+0000 7fcb297fa640 1 -- 172.21.15.23:0/164006661 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fcb2400a040 con 0x7fcb2c1073f0 2024-03-24T10:38:16.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.041+0000 7fcb297fa640 1 --2- 172.21.15.23:0/164006661 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcb140a20c0 0x7fcb140a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:16.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.041+0000 7fcb2bfff640 1 --2- 172.21.15.23:0/164006661 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcb140a20c0 0x7fcb140a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:16.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.041+0000 7fcb297fa640 1 -- 172.21.15.23:0/164006661 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fcb240f0cd0 con 0x7fcb2c1073f0 2024-03-24T10:38:16.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.041+0000 7fcb2bfff640 1 --2- 172.21.15.23:0/164006661 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcb140a20c0 0x7fcb140a4580 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c009d60 tx=0x7fcb1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:16.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.044+0000 7fcb297fa640 1 -- 172.21.15.23:0/164006661 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcb240b9df0 con 0x7fcb2c1073f0 2024-03-24T10:38:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:15 smithi023 ceph-mon[32466]: pgmap v285: 97 pgs: 12 creating+peering, 85 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 4 op/s 2024-03-24T10:38:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:15 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2054791271' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T10:38:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:15 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] up:active 2024-03-24T10:38:16.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:15 smithi023 ceph-mon[32466]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T10:38:16.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:15 smithi023 ceph-mon[32466]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T10:38:16.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:15 smithi023 ceph-mon[32466]: Cluster is now healthy 2024-03-24T10:38:16.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:15 smithi023 ceph-mon[32466]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-24T10:38:16.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:15 smithi023 ceph-mon[32466]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-24T10:38:16.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:15 smithi023 ceph-mon[32466]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-24T10:38:16.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:15 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4252520387' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T10:38:16.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:15 smithi117 ceph-mon[36368]: pgmap v285: 97 pgs: 12 creating+peering, 85 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 4 op/s 2024-03-24T10:38:16.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:15 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2054791271' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T10:38:16.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:15 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] up:active 2024-03-24T10:38:16.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:15 smithi117 ceph-mon[36368]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T10:38:16.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:15 smithi117 ceph-mon[36368]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T10:38:16.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:15 smithi117 ceph-mon[36368]: Cluster is now healthy 2024-03-24T10:38:16.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:15 smithi117 ceph-mon[36368]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-24T10:38:16.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:15 smithi117 ceph-mon[36368]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-24T10:38:16.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:15 smithi117 ceph-mon[36368]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-24T10:38:16.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:15 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4252520387' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T10:38:16.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.253+0000 7fcb3268d640 1 -- 172.21.15.23:0/164006661 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fcb2c02d240 con 0x7fcb140a20c0 2024-03-24T10:38:16.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:15 smithi116 ceph-mon[36248]: pgmap v285: 97 pgs: 12 creating+peering, 85 active+clean; 579 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 4 op/s 2024-03-24T10:38:16.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:15 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2054791271' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T10:38:16.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:15 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] up:active 2024-03-24T10:38:16.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:15 smithi116 ceph-mon[36248]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T10:38:16.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:15 smithi116 ceph-mon[36248]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T10:38:16.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:15 smithi116 ceph-mon[36248]: Cluster is now healthy 2024-03-24T10:38:16.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:15 smithi116 ceph-mon[36248]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-24T10:38:16.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:15 smithi116 ceph-mon[36248]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-24T10:38:16.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:15 smithi116 ceph-mon[36248]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-24T10:38:16.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:15 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4252520387' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T10:38:16.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.274+0000 7fcb297fa640 1 -- 172.21.15.23:0/164006661 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcb2c02d240 con 0x7fcb140a20c0 2024-03-24T10:38:16.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.277+0000 7fcb3268d640 1 -- 172.21.15.23:0/164006661 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcb140a20c0 msgr2=0x7fcb140a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.277+0000 7fcb3268d640 1 --2- 172.21.15.23:0/164006661 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcb140a20c0 0x7fcb140a4580 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c009d60 tx=0x7fcb1c005f90 comp rx=0 tx=0).stop 2024-03-24T10:38:16.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.277+0000 7fcb3268d640 1 -- 172.21.15.23:0/164006661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb2c1073f0 msgr2=0x7fcb2c1a0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.277+0000 7fcb3268d640 1 --2- 172.21.15.23:0/164006661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb2c1073f0 0x7fcb2c1a0f70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcb2402c630 tx=0x7fcb24032f80 comp rx=0 tx=0).stop 2024-03-24T10:38:16.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.277+0000 7fcb3268d640 1 -- 172.21.15.23:0/164006661 shutdown_connections 2024-03-24T10:38:16.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.277+0000 7fcb3268d640 1 --2- 172.21.15.23:0/164006661 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fcb140a20c0 0x7fcb140a4580 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.277+0000 7fcb3268d640 1 --2- 172.21.15.23:0/164006661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb2c1073f0 0x7fcb2c1a0f70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.277+0000 7fcb3268d640 1 --2- 172.21.15.23:0/164006661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb2c104ac0 0x7fcb2c199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.277+0000 7fcb3268d640 1 --2- 172.21.15.23:0/164006661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb2c102190 0x7fcb2c1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.277+0000 7fcb3268d640 1 -- 172.21.15.23:0/164006661 >> 172.21.15.23:0/164006661 conn(0x7fcb2c0fbba0 msgr2=0x7fcb2c1018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:16.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.277+0000 7fcb3268d640 1 -- 172.21.15.23:0/164006661 shutdown_connections 2024-03-24T10:38:16.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.277+0000 7fcb3268d640 1 -- 172.21.15.23:0/164006661 wait complete. 2024-03-24T10:38:16.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-24T10:38:16.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 -- 172.21.15.23:0/3254973865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f438c103380 msgr2=0x7f438c109c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 --2- 172.21.15.23:0/3254973865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f438c103380 0x7f438c109c10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f437c009900 tx=0x7f437c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:38:16.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 -- 172.21.15.23:0/3254973865 shutdown_connections 2024-03-24T10:38:16.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 --2- 172.21.15.23:0/3254973865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f438c103380 0x7f438c109c10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 --2- 172.21.15.23:0/3254973865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f438c1029c0 0x7f438c102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 --2- 172.21.15.23:0/3254973865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f438c0ff9f0 0x7f438c0ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 -- 172.21.15.23:0/3254973865 >> 172.21.15.23:0/3254973865 conn(0x7f438c0fb320 msgr2=0x7f438c0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:16.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 -- 172.21.15.23:0/3254973865 shutdown_connections 2024-03-24T10:38:16.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 -- 172.21.15.23:0/3254973865 wait complete. 2024-03-24T10:38:16.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 Processor -- start 2024-03-24T10:38:16.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 -- start start 2024-03-24T10:38:16.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f438c0ff9f0 0x7f438c197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:16.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f438c1029c0 0x7f438c197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:16.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f438c103380 0x7f438c19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:16.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f438c19f240 con 0x7f438c103380 2024-03-24T10:38:16.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f438c19f3b0 con 0x7f438c0ff9f0 2024-03-24T10:38:16.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.396+0000 7f4392ab5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f438c19f570 con 0x7f438c1029c0 2024-03-24T10:38:16.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4391052640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f438c103380 0x7f438c19ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:16.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f438bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f438c1029c0 0x7f438c197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:16.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4391052640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f438c103380 0x7f438c19ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35330/0 (socket says 172.21.15.23:35330) 2024-03-24T10:38:16.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4391052640 1 -- 172.21.15.23:0/4198105069 learned_addr learned my addr 172.21.15.23:0/4198105069 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:16.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f438bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f438c1029c0 0x7f438c197c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:48990/0 (socket says 172.21.15.23:48990) 2024-03-24T10:38:16.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4390851640 1 --2- 172.21.15.23:0/4198105069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f438c0ff9f0 0x7f438c197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:16.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4391052640 1 -- 172.21.15.23:0/4198105069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f438c1029c0 msgr2=0x7f438c197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4391052640 1 --2- 172.21.15.23:0/4198105069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f438c1029c0 0x7f438c197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4391052640 1 -- 172.21.15.23:0/4198105069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f438c0ff9f0 msgr2=0x7f438c197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4391052640 1 --2- 172.21.15.23:0/4198105069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f438c0ff9f0 0x7f438c197750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4391052640 1 -- 172.21.15.23:0/4198105069 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437c009560 con 0x7f438c103380 2024-03-24T10:38:16.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f438bfff640 1 --2- 172.21.15.23:0/4198105069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f438c1029c0 0x7f438c197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:16.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4391052640 1 --2- 172.21.15.23:0/4198105069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f438c103380 0x7f438c19ed00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f437c009230 tx=0x7f437c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:16.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4389ffb640 1 -- 172.21.15.23:0/4198105069 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c004440 con 0x7f438c103380 2024-03-24T10:38:16.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4389ffb640 1 -- 172.21.15.23:0/4198105069 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f437c0052d0 con 0x7f438c103380 2024-03-24T10:38:16.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4392ab5640 1 -- 172.21.15.23:0/4198105069 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f438c19f7f0 con 0x7f438c103380 2024-03-24T10:38:16.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4389ffb640 1 -- 172.21.15.23:0/4198105069 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c003c50 con 0x7f438c103380 2024-03-24T10:38:16.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.397+0000 7f4392ab5640 1 -- 172.21.15.23:0/4198105069 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f438c19fd50 con 0x7f438c103380 2024-03-24T10:38:16.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.398+0000 7f4392ab5640 1 -- 172.21.15.23:0/4198105069 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f438c108a90 con 0x7f438c103380 2024-03-24T10:38:16.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.400+0000 7f4389ffb640 1 -- 172.21.15.23:0/4198105069 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f437c005440 con 0x7f438c103380 2024-03-24T10:38:16.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.400+0000 7f4389ffb640 1 --2- 172.21.15.23:0/4198105069 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f43740a2140 0x7f43740a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:16.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.400+0000 7f4389ffb640 1 -- 172.21.15.23:0/4198105069 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f437c0efd20 con 0x7f438c103380 2024-03-24T10:38:16.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.400+0000 7f4390851640 1 --2- 172.21.15.23:0/4198105069 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f43740a2140 0x7f43740a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:16.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.401+0000 7f4390851640 1 --2- 172.21.15.23:0/4198105069 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f43740a2140 0x7f43740a4600 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f438c069b00 tx=0x7f4384009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:16.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.405+0000 7f4389ffb640 1 -- 172.21.15.23:0/4198105069 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f437c0b8e40 con 0x7f438c103380 2024-03-24T10:38:16.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.615+0000 7f4392ab5640 1 -- 172.21.15.23:0/4198105069 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f438c02d240 con 0x7f43740a2140 2024-03-24T10:38:16.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.638+0000 7f4389ffb640 1 -- 172.21.15.23:0/4198105069 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f438c02d240 con 0x7f43740a2140 2024-03-24T10:38:16.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.640+0000 7f4392ab5640 1 -- 172.21.15.23:0/4198105069 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f43740a2140 msgr2=0x7f43740a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.640+0000 7f4392ab5640 1 --2- 172.21.15.23:0/4198105069 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f43740a2140 0x7f43740a4600 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f438c069b00 tx=0x7f4384009500 comp rx=0 tx=0).stop 2024-03-24T10:38:16.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.640+0000 7f4392ab5640 1 -- 172.21.15.23:0/4198105069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f438c103380 msgr2=0x7f438c19ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.640+0000 7f4392ab5640 1 --2- 172.21.15.23:0/4198105069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f438c103380 0x7f438c19ed00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f437c009230 tx=0x7f437c004040 comp rx=0 tx=0).stop 2024-03-24T10:38:16.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.640+0000 7f4392ab5640 1 -- 172.21.15.23:0/4198105069 shutdown_connections 2024-03-24T10:38:16.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.640+0000 7f4392ab5640 1 --2- 172.21.15.23:0/4198105069 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f43740a2140 0x7f43740a4600 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.640+0000 7f4392ab5640 1 --2- 172.21.15.23:0/4198105069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f438c103380 0x7f438c19ed00 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.640+0000 7f4392ab5640 1 --2- 172.21.15.23:0/4198105069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f438c1029c0 0x7f438c197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.640+0000 7f4392ab5640 1 --2- 172.21.15.23:0/4198105069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f438c0ff9f0 0x7f438c197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.640+0000 7f4392ab5640 1 -- 172.21.15.23:0/4198105069 >> 172.21.15.23:0/4198105069 conn(0x7f438c0fb320 msgr2=0x7f438c105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:16.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.640+0000 7f4392ab5640 1 -- 172.21.15.23:0/4198105069 shutdown_connections 2024-03-24T10:38:16.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.640+0000 7f4392ab5640 1 -- 172.21.15.23:0/4198105069 wait complete. 2024-03-24T10:38:16.650 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:38:16 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:38:16.258+0000 7fb50ad6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:38:16.650 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:38:16 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:38:16.258+0000 7fb50ad6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:38:16.651 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:38:16 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:38:16.258+0000 7fb50ad6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:38:16.651 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:38:16 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:38:16.258+0000 7fb50ad6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:38:16.651 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:38:16 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:38:16.258+0000 7fb50ad6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:38:16.653 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-24T10:38:16.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.760+0000 7f4766d8a640 1 -- 172.21.15.23:0/347322228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47601021d0 msgr2=0x7f4760108a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.760+0000 7f4766d8a640 1 --2- 172.21.15.23:0/347322228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47601021d0 0x7f4760108a60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4748009900 tx=0x7f474802bc30 comp rx=0 tx=0).stop 2024-03-24T10:38:16.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.760+0000 7f4766d8a640 1 -- 172.21.15.23:0/347322228 shutdown_connections 2024-03-24T10:38:16.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.760+0000 7f4766d8a640 1 --2- 172.21.15.23:0/347322228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47601021d0 0x7f4760108a60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.760+0000 7f4766d8a640 1 --2- 172.21.15.23:0/347322228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4760101810 0x7f4760101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.760+0000 7f4766d8a640 1 --2- 172.21.15.23:0/347322228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4760100620 0x7f4760100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.760+0000 7f4766d8a640 1 -- 172.21.15.23:0/347322228 >> 172.21.15.23:0/347322228 conn(0x7f47600fbbb0 msgr2=0x7f47600fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.760+0000 7f4766d8a640 1 -- 172.21.15.23:0/347322228 shutdown_connections 2024-03-24T10:38:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.760+0000 7f4766d8a640 1 -- 172.21.15.23:0/347322228 wait complete. 2024-03-24T10:38:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4766d8a640 1 Processor -- start 2024-03-24T10:38:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4766d8a640 1 -- start start 2024-03-24T10:38:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4766d8a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4760100620 0x7f4760196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4766d8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4760101810 0x7f4760196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4766d8a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47601021d0 0x7f476019dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4766d8a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f476019e180 con 0x7f4760100620 2024-03-24T10:38:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4766d8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f476019e2f0 con 0x7f4760101810 2024-03-24T10:38:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4766d8a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f476019e4b0 con 0x7f47601021d0 2024-03-24T10:38:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4764b26640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4760100620 0x7f4760196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4764b26640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4760100620 0x7f4760196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35346/0 (socket says 172.21.15.23:35346) 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4764b26640 1 -- 172.21.15.23:0/3659740646 learned_addr learned my addr 172.21.15.23:0/3659740646 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f475ffff640 1 --2- 172.21.15.23:0/3659740646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4760101810 0x7f4760196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4765327640 1 --2- 172.21.15.23:0/3659740646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47601021d0 0x7f476019dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4764b26640 1 -- 172.21.15.23:0/3659740646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47601021d0 msgr2=0x7f476019dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4764b26640 1 --2- 172.21.15.23:0/3659740646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47601021d0 0x7f476019dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4764b26640 1 -- 172.21.15.23:0/3659740646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4760101810 msgr2=0x7f4760196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4764b26640 1 --2- 172.21.15.23:0/3659740646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4760101810 0x7f4760196bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.761+0000 7f4764b26640 1 -- 172.21.15.23:0/3659740646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4758009490 con 0x7f4760100620 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.762+0000 7f4764b26640 1 --2- 172.21.15.23:0/3659740646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4760100620 0x7f4760196690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f475800e9c0 tx=0x7f475800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.762+0000 7f475dffb640 1 -- 172.21.15.23:0/3659740646 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475800cde0 con 0x7f4760100620 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.762+0000 7f475dffb640 1 -- 172.21.15.23:0/3659740646 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4758010910 con 0x7f4760100620 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.762+0000 7f4766d8a640 1 -- 172.21.15.23:0/3659740646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4748009560 con 0x7f4760100620 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.762+0000 7f475dffb640 1 -- 172.21.15.23:0/3659740646 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4758018a10 con 0x7f4760100620 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.762+0000 7f4766d8a640 1 -- 172.21.15.23:0/3659740646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f476019eb10 con 0x7f4760100620 2024-03-24T10:38:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.763+0000 7f4766d8a640 1 -- 172.21.15.23:0/3659740646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47601079c0 con 0x7f4760100620 2024-03-24T10:38:16.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.764+0000 7f475dffb640 1 -- 172.21.15.23:0/3659740646 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f4758010a80 con 0x7f4760100620 2024-03-24T10:38:16.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.764+0000 7f475dffb640 1 --2- 172.21.15.23:0/3659740646 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f474c0a20c0 0x7f474c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:16.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.765+0000 7f475ffff640 1 --2- 172.21.15.23:0/3659740646 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f474c0a20c0 0x7f474c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:16.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.765+0000 7f475dffb640 1 -- 172.21.15.23:0/3659740646 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4758014070 con 0x7f4760100620 2024-03-24T10:38:16.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.765+0000 7f475ffff640 1 --2- 172.21.15.23:0/3659740646 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f474c0a20c0 0x7f474c0a4580 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4760197ba0 tx=0x7f4750009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:16.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.769+0000 7f475dffb640 1 -- 172.21.15.23:0/3659740646 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4758097340 con 0x7f4760100620 2024-03-24T10:38:16.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.982+0000 7f4766d8a640 1 -- 172.21.15.23:0/3659740646 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f476002d1a0 con 0x7f474c0a20c0 2024-03-24T10:38:16.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.987+0000 7f475dffb640 1 -- 172.21.15.23:0/3659740646 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f476002d1a0 con 0x7f474c0a20c0 2024-03-24T10:38:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/volumes/qa/sv_0/dd3c8986-5f72-43b0-9e51-1297dc9857cf 2024-03-24T10:38:16.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.989+0000 7f4766d8a640 1 -- 172.21.15.23:0/3659740646 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f474c0a20c0 msgr2=0x7f474c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.989+0000 7f4766d8a640 1 --2- 172.21.15.23:0/3659740646 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f474c0a20c0 0x7f474c0a4580 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4760197ba0 tx=0x7f4750009500 comp rx=0 tx=0).stop 2024-03-24T10:38:16.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.989+0000 7f4766d8a640 1 -- 172.21.15.23:0/3659740646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4760100620 msgr2=0x7f4760196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:16.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.989+0000 7f4766d8a640 1 --2- 172.21.15.23:0/3659740646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4760100620 0x7f4760196690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f475800e9c0 tx=0x7f475800ee80 comp rx=0 tx=0).stop 2024-03-24T10:38:16.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.990+0000 7f4766d8a640 1 -- 172.21.15.23:0/3659740646 shutdown_connections 2024-03-24T10:38:16.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.990+0000 7f4766d8a640 1 --2- 172.21.15.23:0/3659740646 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f474c0a20c0 0x7f474c0a4580 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.990+0000 7f4766d8a640 1 --2- 172.21.15.23:0/3659740646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47601021d0 0x7f476019dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.990+0000 7f4766d8a640 1 --2- 172.21.15.23:0/3659740646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4760101810 0x7f4760196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.990+0000 7f4766d8a640 1 --2- 172.21.15.23:0/3659740646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4760100620 0x7f4760196690 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:16.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.990+0000 7f4766d8a640 1 -- 172.21.15.23:0/3659740646 >> 172.21.15.23:0/3659740646 conn(0x7f47600fbbb0 msgr2=0x7f4760104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:16.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.990+0000 7f4766d8a640 1 -- 172.21.15.23:0/3659740646 shutdown_connections 2024-03-24T10:38:16.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:16.990+0000 7f4766d8a640 1 -- 172.21.15.23:0/3659740646 wait complete. 2024-03-24T10:38:17.001 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-24T10:38:17.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.108+0000 7f97d4b5c640 1 -- 172.21.15.23:0/296350141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d01021e0 msgr2=0x7f97d0108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:17.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.108+0000 7f97d4b5c640 1 --2- 172.21.15.23:0/296350141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d01021e0 0x7f97d0108a70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f97c8009ca0 tx=0x7f97c802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.108+0000 7f97d4b5c640 1 -- 172.21.15.23:0/296350141 shutdown_connections 2024-03-24T10:38:17.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.108+0000 7f97d4b5c640 1 --2- 172.21.15.23:0/296350141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d01021e0 0x7f97d0108a70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.108+0000 7f97d4b5c640 1 --2- 172.21.15.23:0/296350141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97d0101820 0x7f97d0101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.108+0000 7f97d4b5c640 1 --2- 172.21.15.23:0/296350141 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97d0100630 0x7f97d0100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.108+0000 7f97d4b5c640 1 -- 172.21.15.23:0/296350141 >> 172.21.15.23:0/296350141 conn(0x7f97d00fbbc0 msgr2=0x7f97d00fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.108+0000 7f97d4b5c640 1 -- 172.21.15.23:0/296350141 shutdown_connections 2024-03-24T10:38:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.108+0000 7f97d4b5c640 1 -- 172.21.15.23:0/296350141 wait complete. 2024-03-24T10:38:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.108+0000 7f97d4b5c640 1 Processor -- start 2024-03-24T10:38:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.108+0000 7f97d4b5c640 1 -- start start 2024-03-24T10:38:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97d4b5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d0100630 0x7f97d0197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97d4b5c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97d0101820 0x7f97d0197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97d4b5c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97d01021e0 0x7f97d019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97d4b5c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f97d019f330 con 0x7f97d0100630 2024-03-24T10:38:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97d4b5c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f97d019f4a0 con 0x7f97d01021e0 2024-03-24T10:38:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97d4b5c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f97d019f660 con 0x7f97d0101820 2024-03-24T10:38:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97ce59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d0100630 0x7f97d0197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97ce59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d0100630 0x7f97d0197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35368/0 (socket says 172.21.15.23:35368) 2024-03-24T10:38:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97ce59c640 1 -- 172.21.15.23:0/3137881603 learned_addr learned my addr 172.21.15.23:0/3137881603 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97c7fff640 1 --2- 172.21.15.23:0/3137881603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97d0101820 0x7f97d0197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:17.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97ced9d640 1 --2- 172.21.15.23:0/3137881603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97d01021e0 0x7f97d019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:17.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97ce59c640 1 -- 172.21.15.23:0/3137881603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97d0101820 msgr2=0x7f97d0197d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:17.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97ce59c640 1 --2- 172.21.15.23:0/3137881603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97d0101820 0x7f97d0197d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97ce59c640 1 -- 172.21.15.23:0/3137881603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97d01021e0 msgr2=0x7f97d019edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:17.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97ce59c640 1 --2- 172.21.15.23:0/3137881603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97d01021e0 0x7f97d019edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97ce59c640 1 -- 172.21.15.23:0/3137881603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b8009490 con 0x7f97d0100630 2024-03-24T10:38:17.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97ce59c640 1 --2- 172.21.15.23:0/3137881603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d0100630 0x7f97d0197840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f97c000e9c0 tx=0x7f97c000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:17.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97c77fe640 1 -- 172.21.15.23:0/3137881603 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c00096e0 con 0x7f97d0100630 2024-03-24T10:38:17.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97c77fe640 1 -- 172.21.15.23:0/3137881603 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97c000cd80 con 0x7f97d0100630 2024-03-24T10:38:17.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97d4b5c640 1 -- 172.21.15.23:0/3137881603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c8009900 con 0x7f97d0100630 2024-03-24T10:38:17.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97c77fe640 1 -- 172.21.15.23:0/3137881603 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0010430 con 0x7f97d0100630 2024-03-24T10:38:17.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.109+0000 7f97d4b5c640 1 -- 172.21.15.23:0/3137881603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d019fc70 con 0x7f97d0100630 2024-03-24T10:38:17.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.110+0000 7f97d4b5c640 1 -- 172.21.15.23:0/3137881603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97d010c040 con 0x7f97d0100630 2024-03-24T10:38:17.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.111+0000 7f97c77fe640 1 -- 172.21.15.23:0/3137881603 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f97c0003680 con 0x7f97d0100630 2024-03-24T10:38:17.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.112+0000 7f97c77fe640 1 --2- 172.21.15.23:0/3137881603 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f97bc0a20c0 0x7f97bc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:17.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.112+0000 7f97c7fff640 1 --2- 172.21.15.23:0/3137881603 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f97bc0a20c0 0x7f97bc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:17.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.112+0000 7f97c77fe640 1 -- 172.21.15.23:0/3137881603 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f97c0014070 con 0x7f97d0100630 2024-03-24T10:38:17.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.112+0000 7f97c7fff640 1 --2- 172.21.15.23:0/3137881603 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f97bc0a20c0 0x7f97bc0a4580 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f97d0198d50 tx=0x7f97b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:17.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.116+0000 7f97c77fe640 1 -- 172.21.15.23:0/3137881603 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f97c0097000 con 0x7f97d0100630 2024-03-24T10:38:17.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:16 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4252520387' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T10:38:17.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:16 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] up:active 2024-03-24T10:38:17.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:16 smithi023 ceph-mon[32466]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-24T10:38:17.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:16 smithi023 ceph-mon[32466]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:38:17.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:16 smithi023 ceph-mon[32466]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:17.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:16 smithi023 ceph-mon[32466]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:17.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:16 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4252520387' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T10:38:17.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:16 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] up:active 2024-03-24T10:38:17.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:16 smithi117 ceph-mon[36368]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-24T10:38:17.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:16 smithi117 ceph-mon[36368]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:38:17.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:16 smithi117 ceph-mon[36368]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:17.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:16 smithi117 ceph-mon[36368]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:17.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:16 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4252520387' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T10:38:17.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:16 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] up:active 2024-03-24T10:38:17.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:16 smithi116 ceph-mon[36248]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-24T10:38:17.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:16 smithi116 ceph-mon[36248]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:38:17.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:16 smithi116 ceph-mon[36248]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:17.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:16 smithi116 ceph-mon[36248]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:17.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.327+0000 7f97d4b5c640 1 -- 172.21.15.23:0/3137881603 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f97d00fdd50 con 0x7f97bc0a20c0 2024-03-24T10:38:17.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.353+0000 7f97c77fe640 1 -- 172.21.15.23:0/3137881603 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f97d00fdd50 con 0x7f97bc0a20c0 2024-03-24T10:38:17.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.355+0000 7f97d4b5c640 1 -- 172.21.15.23:0/3137881603 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f97bc0a20c0 msgr2=0x7f97bc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:17.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.355+0000 7f97d4b5c640 1 --2- 172.21.15.23:0/3137881603 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f97bc0a20c0 0x7f97bc0a4580 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f97d0198d50 tx=0x7f97b8005f90 comp rx=0 tx=0).stop 2024-03-24T10:38:17.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.355+0000 7f97d4b5c640 1 -- 172.21.15.23:0/3137881603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d0100630 msgr2=0x7f97d0197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:17.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.355+0000 7f97d4b5c640 1 --2- 172.21.15.23:0/3137881603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d0100630 0x7f97d0197840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f97c000e9c0 tx=0x7f97c000ee80 comp rx=0 tx=0).stop 2024-03-24T10:38:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.355+0000 7f97d4b5c640 1 -- 172.21.15.23:0/3137881603 shutdown_connections 2024-03-24T10:38:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.355+0000 7f97d4b5c640 1 --2- 172.21.15.23:0/3137881603 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f97bc0a20c0 0x7f97bc0a4580 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.355+0000 7f97d4b5c640 1 --2- 172.21.15.23:0/3137881603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97d01021e0 0x7f97d019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.355+0000 7f97d4b5c640 1 --2- 172.21.15.23:0/3137881603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97d0101820 0x7f97d0197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.355+0000 7f97d4b5c640 1 --2- 172.21.15.23:0/3137881603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d0100630 0x7f97d0197840 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.355+0000 7f97d4b5c640 1 -- 172.21.15.23:0/3137881603 >> 172.21.15.23:0/3137881603 conn(0x7f97d00fbbc0 msgr2=0x7f97d00fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.355+0000 7f97d4b5c640 1 -- 172.21.15.23:0/3137881603 shutdown_connections 2024-03-24T10:38:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.355+0000 7f97d4b5c640 1 -- 172.21.15.23:0/3137881603 wait complete. 2024-03-24T10:38:17.366 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-24T10:38:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.473+0000 7f4cca2ed640 1 -- 172.21.15.23:0/3497968229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4100620 msgr2=0x7f4cc4100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.473+0000 7f4cca2ed640 1 --2- 172.21.15.23:0/3497968229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4100620 0x7f4cc4100a40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009900 tx=0x7f4cb402bc30 comp rx=0 tx=0).stop 2024-03-24T10:38:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.473+0000 7f4cca2ed640 1 -- 172.21.15.23:0/3497968229 shutdown_connections 2024-03-24T10:38:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.473+0000 7f4cca2ed640 1 --2- 172.21.15.23:0/3497968229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cc41021d0 0x7f4cc4108a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.473+0000 7f4cca2ed640 1 --2- 172.21.15.23:0/3497968229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cc4101810 0x7f4cc4101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.473+0000 7f4cca2ed640 1 --2- 172.21.15.23:0/3497968229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4100620 0x7f4cc4100a40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.473+0000 7f4cca2ed640 1 -- 172.21.15.23:0/3497968229 >> 172.21.15.23:0/3497968229 conn(0x7f4cc40fbbb0 msgr2=0x7f4cc40fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.473+0000 7f4cca2ed640 1 -- 172.21.15.23:0/3497968229 shutdown_connections 2024-03-24T10:38:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.473+0000 7f4cca2ed640 1 -- 172.21.15.23:0/3497968229 wait complete. 2024-03-24T10:38:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.473+0000 7f4cca2ed640 1 Processor -- start 2024-03-24T10:38:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.473+0000 7f4cca2ed640 1 -- start start 2024-03-24T10:38:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cca2ed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cc4100620 0x7f4cc41045b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cca2ed640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4101810 0x7f4cc4102bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cca2ed640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cc41021d0 0x7f4cc4103130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cca2ed640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc4103890 con 0x7f4cc4101810 2024-03-24T10:38:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cca2ed640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc4103a00 con 0x7f4cc4100620 2024-03-24T10:38:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cca2ed640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc41a1880 con 0x7f4cc41021d0 2024-03-24T10:38:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4101810 0x7f4cc4102bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4101810 0x7f4cc4102bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35390/0 (socket says 172.21.15.23:35390) 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc37fe640 1 -- 172.21.15.23:0/1362086404 learned_addr learned my addr 172.21.15.23:0/1362086404 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc888a640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cc41021d0 0x7f4cc4103130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc3fff640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cc4100620 0x7f4cc41045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc37fe640 1 -- 172.21.15.23:0/1362086404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cc41021d0 msgr2=0x7f4cc4103130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc37fe640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cc41021d0 0x7f4cc4103130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc37fe640 1 -- 172.21.15.23:0/1362086404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cc4100620 msgr2=0x7f4cc41045b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc37fe640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cc4100620 0x7f4cc41045b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc37fe640 1 -- 172.21.15.23:0/1362086404 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb8009490 con 0x7f4cc4101810 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc3fff640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cc4100620 0x7f4cc41045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc37fe640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4101810 0x7f4cc4102bf0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800ea90 tx=0x7f4cb800ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc17fa640 1 -- 172.21.15.23:0/1362086404 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb800ce40 con 0x7f4cc4101810 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cca2ed640 1 -- 172.21.15.23:0/1362086404 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb4009560 con 0x7f4cc4101810 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc17fa640 1 -- 172.21.15.23:0/1362086404 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb80035f0 con 0x7f4cc4101810 2024-03-24T10:38:17.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.474+0000 7f4cc17fa640 1 -- 172.21.15.23:0/1362086404 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8010690 con 0x7f4cc4101810 2024-03-24T10:38:17.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.475+0000 7f4cca2ed640 1 -- 172.21.15.23:0/1362086404 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc41a1eb0 con 0x7f4cc4101810 2024-03-24T10:38:17.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.475+0000 7f4cca2ed640 1 -- 172.21.15.23:0/1362086404 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cc410c030 con 0x7f4cc4101810 2024-03-24T10:38:17.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.477+0000 7f4cc17fa640 1 -- 172.21.15.23:0/1362086404 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f4cb8003ad0 con 0x7f4cc4101810 2024-03-24T10:38:17.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.477+0000 7f4cc17fa640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4cac0a20c0 0x7f4cac0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:17.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.477+0000 7f4cc3fff640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4cac0a20c0 0x7f4cac0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:17.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.477+0000 7f4cc17fa640 1 -- 172.21.15.23:0/1362086404 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4cb8014070 con 0x7f4cc4101810 2024-03-24T10:38:17.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.477+0000 7f4cc3fff640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4cac0a20c0 0x7f4cac0a4580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4005ab0 tx=0x7f4cb4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:17.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.481+0000 7f4cc17fa640 1 -- 172.21.15.23:0/1362086404 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4cb8097590 con 0x7f4cc4101810 2024-03-24T10:38:17.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.691+0000 7f4cca2ed640 1 -- 172.21.15.23:0/1362086404 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f4cc40fddb0 con 0x7f4cac0a20c0 2024-03-24T10:38:17.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.696+0000 7f4cc17fa640 1 -- 172.21.15.23:0/1362086404 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f4cc40fddb0 con 0x7f4cac0a20c0 2024-03-24T10:38:17.696 INFO:teuthology.orchestra.run.smithi023.stdout:/volumes/qa/sv_1/b7fb1ade-3c33-4b9e-8a53-f368dfa57265 2024-03-24T10:38:17.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.698+0000 7f4cca2ed640 1 -- 172.21.15.23:0/1362086404 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4cac0a20c0 msgr2=0x7f4cac0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:17.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.698+0000 7f4cca2ed640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4cac0a20c0 0x7f4cac0a4580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4005ab0 tx=0x7f4cb4005fd0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.698+0000 7f4cca2ed640 1 -- 172.21.15.23:0/1362086404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4101810 msgr2=0x7f4cc4102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:17.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.698+0000 7f4cca2ed640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4101810 0x7f4cc4102bf0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800ea90 tx=0x7f4cb800ef50 comp rx=0 tx=0).stop 2024-03-24T10:38:17.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.698+0000 7f4cca2ed640 1 -- 172.21.15.23:0/1362086404 shutdown_connections 2024-03-24T10:38:17.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.698+0000 7f4cca2ed640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4cac0a20c0 0x7f4cac0a4580 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.698+0000 7f4cca2ed640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cc41021d0 0x7f4cc4103130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.698+0000 7f4cca2ed640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4101810 0x7f4cc4102bf0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.698+0000 7f4cca2ed640 1 --2- 172.21.15.23:0/1362086404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cc4100620 0x7f4cc41045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.698+0000 7f4cca2ed640 1 -- 172.21.15.23:0/1362086404 >> 172.21.15.23:0/1362086404 conn(0x7f4cc40fbbb0 msgr2=0x7f4cc40fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:17.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.698+0000 7f4cca2ed640 1 -- 172.21.15.23:0/1362086404 shutdown_connections 2024-03-24T10:38:17.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.698+0000 7f4cca2ed640 1 -- 172.21.15.23:0/1362086404 wait complete. 2024-03-24T10:38:17.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:38:17.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.811+0000 7f3c39d80640 1 -- 172.21.15.23:0/2357902448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c34100630 msgr2=0x7f3c34100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:17.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.811+0000 7f3c39d80640 1 --2- 172.21.15.23:0/2357902448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c34100630 0x7f3c34100a50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3c24009ca0 tx=0x7f3c2402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.811+0000 7f3c39d80640 1 -- 172.21.15.23:0/2357902448 shutdown_connections 2024-03-24T10:38:17.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.811+0000 7f3c39d80640 1 --2- 172.21.15.23:0/2357902448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c341021e0 0x7f3c34108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.811+0000 7f3c39d80640 1 --2- 172.21.15.23:0/2357902448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c34101820 0x7f3c34101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.811+0000 7f3c39d80640 1 --2- 172.21.15.23:0/2357902448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c34100630 0x7f3c34100a50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.811+0000 7f3c39d80640 1 -- 172.21.15.23:0/2357902448 >> 172.21.15.23:0/2357902448 conn(0x7f3c340fbba0 msgr2=0x7f3c340fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:17.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.811+0000 7f3c39d80640 1 -- 172.21.15.23:0/2357902448 shutdown_connections 2024-03-24T10:38:17.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.811+0000 7f3c39d80640 1 -- 172.21.15.23:0/2357902448 wait complete. 2024-03-24T10:38:17.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.812+0000 7f3c39d80640 1 Processor -- start 2024-03-24T10:38:17.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.812+0000 7f3c39d80640 1 -- start start 2024-03-24T10:38:17.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.812+0000 7f3c39d80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c34100630 0x7f3c34196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.812+0000 7f3c39d80640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c34101820 0x7f3c34196c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.812+0000 7f3c39d80640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c341021e0 0x7f3c3419dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.812+0000 7f3c39d80640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3419e210 con 0x7f3c341021e0 2024-03-24T10:38:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.812+0000 7f3c39d80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3419e380 con 0x7f3c34100630 2024-03-24T10:38:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.812+0000 7f3c39d80640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3419e540 con 0x7f3c34101820 2024-03-24T10:38:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.812+0000 7f3c33fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c341021e0 0x7f3c3419dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.812+0000 7f3c33fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c341021e0 0x7f3c3419dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44558/0 (socket says 172.21.15.23:44558) 2024-03-24T10:38:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.812+0000 7f3c32ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c34101820 0x7f3c34196c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.812+0000 7f3c33fff640 1 -- 172.21.15.23:0/1270059950 learned_addr learned my addr 172.21.15.23:0/1270059950 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.812+0000 7f3c337fe640 1 --2- 172.21.15.23:0/1270059950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c34100630 0x7f3c34196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.813+0000 7f3c33fff640 1 -- 172.21.15.23:0/1270059950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c34101820 msgr2=0x7f3c34196c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.813+0000 7f3c33fff640 1 --2- 172.21.15.23:0/1270059950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c34101820 0x7f3c34196c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.813+0000 7f3c33fff640 1 -- 172.21.15.23:0/1270059950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c34100630 msgr2=0x7f3c34196720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:17.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.813+0000 7f3c33fff640 1 --2- 172.21.15.23:0/1270059950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c34100630 0x7f3c34196720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:17.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.813+0000 7f3c33fff640 1 -- 172.21.15.23:0/1270059950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c2c009490 con 0x7f3c341021e0 2024-03-24T10:38:17.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.813+0000 7f3c33fff640 1 --2- 172.21.15.23:0/1270059950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c341021e0 0x7f3c3419dcd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00e9b0 tx=0x7f3c2c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:17.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.813+0000 7f3c30ff9640 1 -- 172.21.15.23:0/1270059950 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c00cd80 con 0x7f3c341021e0 2024-03-24T10:38:17.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.813+0000 7f3c30ff9640 1 -- 172.21.15.23:0/1270059950 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c2c0035f0 con 0x7f3c341021e0 2024-03-24T10:38:17.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.813+0000 7f3c39d80640 1 -- 172.21.15.23:0/1270059950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c24009900 con 0x7f3c341021e0 2024-03-24T10:38:17.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.813+0000 7f3c30ff9640 1 -- 172.21.15.23:0/1270059950 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c00cee0 con 0x7f3c341021e0 2024-03-24T10:38:17.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.813+0000 7f3c39d80640 1 -- 172.21.15.23:0/1270059950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c3419eba0 con 0x7f3c341021e0 2024-03-24T10:38:17.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.814+0000 7f3c39d80640 1 -- 172.21.15.23:0/1270059950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c341079d0 con 0x7f3c341021e0 2024-03-24T10:38:17.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.815+0000 7f3c30ff9640 1 -- 172.21.15.23:0/1270059950 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f3c2c003ad0 con 0x7f3c341021e0 2024-03-24T10:38:17.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.816+0000 7f3c30ff9640 1 --2- 172.21.15.23:0/1270059950 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3c200a2140 0x7f3c200a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:17.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.816+0000 7f3c337fe640 1 --2- 172.21.15.23:0/1270059950 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3c200a2140 0x7f3c200a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:17.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.816+0000 7f3c30ff9640 1 -- 172.21.15.23:0/1270059950 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f3c2c014070 con 0x7f3c341021e0 2024-03-24T10:38:17.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.816+0000 7f3c337fe640 1 --2- 172.21.15.23:0/1270059950 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3c200a2140 0x7f3c200a4600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3c2402c700 tx=0x7f3c240326b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:17.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.820+0000 7f3c30ff9640 1 -- 172.21.15.23:0/1270059950 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3c2c097070 con 0x7f3c341021e0 2024-03-24T10:38:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:17.936+0000 7f3c30ff9640 1 -- 172.21.15.23:0/1270059950 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f3c2c097440 con 0x7f3c341021e0 2024-03-24T10:38:18.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.046+0000 7f3c39d80640 1 -- 172.21.15.23:0/1270059950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c3410c010 con 0x7f3c341021e0 2024-03-24T10:38:18.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.047+0000 7f3c30ff9640 1 -- 172.21.15.23:0/1270059950 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11866 (secure 0 0 0) 0x7f3c2c045660 con 0x7f3c341021e0 2024-03-24T10:38:18.048 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:38:18.049 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:38:15.911086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:38:18.050 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 22 2024-03-24T10:38:18.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.049+0000 7f3c39d80640 1 -- 172.21.15.23:0/1270059950 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3c200a2140 msgr2=0x7f3c200a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.049+0000 7f3c39d80640 1 --2- 172.21.15.23:0/1270059950 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3c200a2140 0x7f3c200a4600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3c2402c700 tx=0x7f3c240326b0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.049+0000 7f3c39d80640 1 -- 172.21.15.23:0/1270059950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c341021e0 msgr2=0x7f3c3419dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.049+0000 7f3c39d80640 1 --2- 172.21.15.23:0/1270059950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c341021e0 0x7f3c3419dcd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00e9b0 tx=0x7f3c2c00ee70 comp rx=0 tx=0).stop 2024-03-24T10:38:18.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.049+0000 7f3c39d80640 1 -- 172.21.15.23:0/1270059950 shutdown_connections 2024-03-24T10:38:18.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.049+0000 7f3c39d80640 1 --2- 172.21.15.23:0/1270059950 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3c200a2140 0x7f3c200a4600 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.049+0000 7f3c39d80640 1 --2- 172.21.15.23:0/1270059950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c341021e0 0x7f3c3419dcd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.049+0000 7f3c39d80640 1 --2- 172.21.15.23:0/1270059950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c34101820 0x7f3c34196c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.049+0000 7f3c39d80640 1 --2- 172.21.15.23:0/1270059950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c34100630 0x7f3c34196720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.049+0000 7f3c39d80640 1 -- 172.21.15.23:0/1270059950 >> 172.21.15.23:0/1270059950 conn(0x7f3c340fbba0 msgr2=0x7f3c34104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:18.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.049+0000 7f3c39d80640 1 -- 172.21.15.23:0/1270059950 shutdown_connections 2024-03-24T10:38:18.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.049+0000 7f3c39d80640 1 -- 172.21.15.23:0/1270059950 wait complete. 2024-03-24T10:38:18.060 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:17 smithi023 ceph-mon[32466]: pgmap v286: 97 pgs: 97 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-03-24T10:38:18.060 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:17 smithi023 ceph-mon[32466]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:18.060 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:17 smithi023 ceph-mon[32466]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:18.060 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:17 smithi023 ceph-mon[32466]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:18.061 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:38:18.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.163+0000 7fa26b36c640 1 -- 172.21.15.23:0/1774966461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa264102d40 msgr2=0x7fa264103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.163+0000 7fa26b36c640 1 --2- 172.21.15.23:0/1774966461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa264102d40 0x7fa264103160 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa260009900 tx=0x7fa26002bc30 comp rx=0 tx=0).stop 2024-03-24T10:38:18.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.163+0000 7fa26b36c640 1 -- 172.21.15.23:0/1774966461 shutdown_connections 2024-03-24T10:38:18.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.163+0000 7fa26b36c640 1 --2- 172.21.15.23:0/1774966461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa264100170 0x7fa264109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.163+0000 7fa26b36c640 1 --2- 172.21.15.23:0/1774966461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2640ff7b0 0x7fa2640ffc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.163+0000 7fa26b36c640 1 --2- 172.21.15.23:0/1774966461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa264102d40 0x7fa264103160 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.163+0000 7fa26b36c640 1 -- 172.21.15.23:0/1774966461 >> 172.21.15.23:0/1774966461 conn(0x7fa2640fb330 msgr2=0x7fa2640fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:18.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.163+0000 7fa26b36c640 1 -- 172.21.15.23:0/1774966461 shutdown_connections 2024-03-24T10:38:18.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.163+0000 7fa26b36c640 1 -- 172.21.15.23:0/1774966461 wait complete. 2024-03-24T10:38:18.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa26b36c640 1 Processor -- start 2024-03-24T10:38:18.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa26b36c640 1 -- start start 2024-03-24T10:38:18.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa26b36c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2640ff7b0 0x7fa264102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:18.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa26b36c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa264100170 0x7fa264100a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:18.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa26b36c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa264102d40 0x7fa264100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:18.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa26b36c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fa264101720 con 0x7fa264100170 2024-03-24T10:38:18.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa26b36c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa264101890 con 0x7fa2640ff7b0 2024-03-24T10:38:18.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa26b36c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641a1840 con 0x7fa264102d40 2024-03-24T10:38:18.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa268907640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa264100170 0x7fa264100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:18.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa268907640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa264100170 0x7fa264100a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44578/0 (socket says 172.21.15.23:44578) 2024-03-24T10:38:18.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa268907640 1 -- 172.21.15.23:0/2920675742 learned_addr learned my addr 172.21.15.23:0/2920675742 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:18.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa269909640 1 --2- 172.21.15.23:0/2920675742 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa264102d40 0x7fa264100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:18.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa269108640 1 --2- 172.21.15.23:0/2920675742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2640ff7b0 0x7fa264102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:18.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa268907640 1 -- 172.21.15.23:0/2920675742 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa264102d40 msgr2=0x7fa264100fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa268907640 1 --2- 172.21.15.23:0/2920675742 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa264102d40 0x7fa264100fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa268907640 1 -- 172.21.15.23:0/2920675742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2640ff7b0 msgr2=0x7fa264102440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa268907640 1 --2- 172.21.15.23:0/2920675742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2640ff7b0 0x7fa264102440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa268907640 1 -- 172.21.15.23:0/2920675742 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa254009490 con 0x7fa264100170 2024-03-24T10:38:18.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.164+0000 7fa268907640 1 --2- 172.21.15.23:0/2920675742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa264100170 0x7fa264100a80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa25400ea90 tx=0x7fa25400ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:18.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.165+0000 7fa25a7fc640 1 -- 172.21.15.23:0/2920675742 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25400ce40 con 0x7fa264100170 2024-03-24T10:38:18.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.165+0000 7fa25a7fc640 1 -- 172.21.15.23:0/2920675742 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2540035f0 con 0x7fa264100170 2024-03-24T10:38:18.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.165+0000 7fa26b36c640 1 -- 172.21.15.23:0/2920675742 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa260009560 con 0x7fa264100170 2024-03-24T10:38:18.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.165+0000 7fa25a7fc640 1 -- 172.21.15.23:0/2920675742 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254010690 con 0x7fa264100170 2024-03-24T10:38:18.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.165+0000 7fa26b36c640 1 -- 172.21.15.23:0/2920675742 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2641a1ec0 con 0x7fa264100170 2024-03-24T10:38:18.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.166+0000 7fa26b36c640 1 -- 172.21.15.23:0/2920675742 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa264108c80 con 0x7fa264100170 2024-03-24T10:38:18.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.167+0000 7fa25a7fc640 1 -- 172.21.15.23:0/2920675742 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fa254003ad0 con 0x7fa264100170 2024-03-24T10:38:18.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.167+0000 7fa25a7fc640 1 --2- 172.21.15.23:0/2920675742 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa2500a21d0 0x7fa2500a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:18.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.167+0000 7fa269108640 1 --2- 172.21.15.23:0/2920675742 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa2500a21d0 0x7fa2500a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:18.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.167+0000 7fa25a7fc640 1 -- 172.21.15.23:0/2920675742 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fa254014070 con 0x7fa264100170 2024-03-24T10:38:18.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.168+0000 7fa269108640 1 --2- 172.21.15.23:0/2920675742 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa2500a21d0 0x7fa2500a4690 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa260005ab0 tx=0x7fa260005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:18.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.172+0000 7fa25a7fc640 1 -- 172.21.15.23:0/2920675742 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa2540970c0 con 0x7fa264100170 2024-03-24T10:38:18.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:17 smithi117 ceph-mon[36368]: pgmap v286: 97 pgs: 97 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-03-24T10:38:18.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:17 smithi117 ceph-mon[36368]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:18.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:17 smithi117 ceph-mon[36368]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:18.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:17 smithi117 ceph-mon[36368]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:18.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:17 smithi116 ceph-mon[36248]: pgmap v286: 97 pgs: 97 active+clean; 587 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-03-24T10:38:18.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:17 smithi116 ceph-mon[36248]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:18.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:17 smithi116 ceph-mon[36248]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:18.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:17 smithi116 ceph-mon[36248]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:38:18.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.353+0000 7fa26b36c640 1 -- 172.21.15.23:0/2920675742 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2641a1980 con 0x7fa264100170 2024-03-24T10:38:18.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.355+0000 7fa25a7fc640 1 -- 172.21.15.23:0/2920675742 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22233 (secure 0 0 0) 0x7fa254003760 con 0x7fa264100170 2024-03-24T10:38:18.355 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:38:18.356 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":85,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:38:11.864894+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:38:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.357+0000 7fa26b36c640 1 -- 172.21.15.23:0/2920675742 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa2500a21d0 msgr2=0x7fa2500a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.357+0000 7fa26b36c640 1 --2- 172.21.15.23:0/2920675742 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa2500a21d0 0x7fa2500a4690 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa260005ab0 tx=0x7fa260005fd0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.357+0000 7fa26b36c640 1 -- 172.21.15.23:0/2920675742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa264100170 msgr2=0x7fa264100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.357+0000 7fa26b36c640 1 --2- 172.21.15.23:0/2920675742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa264100170 0x7fa264100a80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa25400ea90 tx=0x7fa25400ef50 comp rx=0 tx=0).stop 2024-03-24T10:38:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.357+0000 7fa26b36c640 1 -- 172.21.15.23:0/2920675742 shutdown_connections 2024-03-24T10:38:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.357+0000 7fa26b36c640 1 --2- 172.21.15.23:0/2920675742 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa2500a21d0 0x7fa2500a4690 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.357+0000 7fa26b36c640 1 --2- 172.21.15.23:0/2920675742 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa264102d40 0x7fa264100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.357+0000 7fa26b36c640 1 --2- 172.21.15.23:0/2920675742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa264100170 0x7fa264100a80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.357+0000 7fa26b36c640 1 --2- 172.21.15.23:0/2920675742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2640ff7b0 0x7fa264102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.357+0000 7fa26b36c640 1 -- 172.21.15.23:0/2920675742 >> 172.21.15.23:0/2920675742 conn(0x7fa2640fb330 msgr2=0x7fa264105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.357+0000 7fa26b36c640 1 -- 172.21.15.23:0/2920675742 shutdown_connections 2024-03-24T10:38:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.357+0000 7fa26b36c640 1 -- 172.21.15.23:0/2920675742 wait complete. 2024-03-24T10:38:18.369 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T10:38:18.369 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:38:18.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.471+0000 7f852984e640 1 -- 172.21.15.23:0/2507129831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8524102200 msgr2=0x7f8524108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.471+0000 7f852984e640 1 --2- 172.21.15.23:0/2507129831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8524102200 0x7f8524108a90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f851c009ca0 tx=0x7f851c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.471+0000 7f852984e640 1 -- 172.21.15.23:0/2507129831 shutdown_connections 2024-03-24T10:38:18.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.471+0000 7f852984e640 1 --2- 172.21.15.23:0/2507129831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8524102200 0x7f8524108a90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.471+0000 7f852984e640 1 --2- 172.21.15.23:0/2507129831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8524101840 0x7f8524101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.471+0000 7f852984e640 1 --2- 172.21.15.23:0/2507129831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8524100650 0x7f8524100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.471+0000 7f852984e640 1 -- 172.21.15.23:0/2507129831 >> 172.21.15.23:0/2507129831 conn(0x7f85240fbbc0 msgr2=0x7f85240fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:18.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.471+0000 7f852984e640 1 -- 172.21.15.23:0/2507129831 shutdown_connections 2024-03-24T10:38:18.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.471+0000 7f852984e640 1 -- 172.21.15.23:0/2507129831 wait complete. 2024-03-24T10:38:18.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.471+0000 7f852984e640 1 Processor -- start 2024-03-24T10:38:18.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f852984e640 1 -- start start 2024-03-24T10:38:18.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f852984e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8524100650 0x7f8524197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:18.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f852984e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8524101840 0x7f8524197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:18.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f852984e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8524102200 0x7f852419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:18.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f852984e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f852419f270 con 0x7f8524100650 2024-03-24T10:38:18.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f852984e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f852419f3e0 con 0x7f8524101840 2024-03-24T10:38:18.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f852984e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f852419f5a0 con 0x7f8524102200 2024-03-24T10:38:18.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f8522ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8524100650 0x7f8524197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:18.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f8522ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8524100650 0x7f8524197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44596/0 (socket says 172.21.15.23:44596) 2024-03-24T10:38:18.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f8522ffd640 1 -- 172.21.15.23:0/1010859726 learned_addr learned my addr 172.21.15.23:0/1010859726 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:18.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f85227fc640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8524101840 0x7f8524197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:18.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f85237fe640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8524102200 0x7f852419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:18.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f8522ffd640 1 -- 172.21.15.23:0/1010859726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8524102200 msgr2=0x7f852419ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f8522ffd640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8524102200 0x7f852419ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f8522ffd640 1 -- 172.21.15.23:0/1010859726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8524101840 msgr2=0x7f8524197cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f8522ffd640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8524101840 0x7f8524197cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f8522ffd640 1 -- 172.21.15.23:0/1010859726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f851c009900 con 0x7f8524100650 2024-03-24T10:38:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f85227fc640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8524101840 0x7f8524197cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:38:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f8522ffd640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8524100650 0x7f8524197780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f851400e9c0 tx=0x7f851400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f852884c640 1 -- 172.21.15.23:0/1010859726 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85140096e0 con 0x7f8524100650 2024-03-24T10:38:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f852984e640 1 -- 172.21.15.23:0/1010859726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f852419f880 con 0x7f8524100650 2024-03-24T10:38:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f852884c640 1 -- 172.21.15.23:0/1010859726 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f851400cd80 con 0x7f8524100650 2024-03-24T10:38:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f852884c640 1 -- 172.21.15.23:0/1010859726 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8514010430 con 0x7f8524100650 2024-03-24T10:38:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.472+0000 7f852984e640 1 -- 172.21.15.23:0/1010859726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f852419fd40 con 0x7f8524100650 2024-03-24T10:38:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.473+0000 7f852984e640 1 -- 172.21.15.23:0/1010859726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8524107970 con 0x7f8524100650 2024-03-24T10:38:18.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.474+0000 7f852884c640 1 -- 172.21.15.23:0/1010859726 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f8514003680 con 0x7f8524100650 2024-03-24T10:38:18.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.475+0000 7f852884c640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f850c0a2150 0x7f850c0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:18.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.475+0000 7f85227fc640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f850c0a2150 0x7f850c0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:18.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.475+0000 7f852884c640 1 -- 172.21.15.23:0/1010859726 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f8514014070 con 0x7f8524100650 2024-03-24T10:38:18.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.475+0000 7f85227fc640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f850c0a2150 0x7f850c0a4610 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8524198c90 tx=0x7f8518005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:18.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.480+0000 7f852884c640 1 -- 172.21.15.23:0/1010859726 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f85140970e0 con 0x7f8524100650 2024-03-24T10:38:18.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.658+0000 7f852984e640 1 -- 172.21.15.23:0/1010859726 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f852402d240 con 0x7f850c0a2150 2024-03-24T10:38:18.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.666+0000 7f852884c640 1 -- 172.21.15.23:0/1010859726 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258726 (secure 0 0 0) 0x7f852402d240 con 0x7f850c0a2150 2024-03-24T10:38:18.669 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:38:18.671 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-24T10:38:18.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.669+0000 7f852984e640 1 -- 172.21.15.23:0/1010859726 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f850c0a2150 msgr2=0x7f850c0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.669+0000 7f852984e640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f850c0a2150 0x7f850c0a4610 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8524198c90 tx=0x7f8518005f90 comp rx=0 tx=0).stop 2024-03-24T10:38:18.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.669+0000 7f852984e640 1 -- 172.21.15.23:0/1010859726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8524100650 msgr2=0x7f8524197780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.669+0000 7f852984e640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8524100650 0x7f8524197780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f851400e9c0 tx=0x7f851400ee80 comp rx=0 tx=0).stop 2024-03-24T10:38:18.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.669+0000 7f852984e640 1 -- 172.21.15.23:0/1010859726 shutdown_connections 2024-03-24T10:38:18.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.669+0000 7f852984e640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f850c0a2150 0x7f850c0a4610 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.669+0000 7f852984e640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8524102200 0x7f852419ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.669+0000 7f852984e640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8524101840 0x7f8524197cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.669+0000 7f852984e640 1 --2- 172.21.15.23:0/1010859726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8524100650 0x7f8524197780 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.669+0000 7f852984e640 1 -- 172.21.15.23:0/1010859726 >> 172.21.15.23:0/1010859726 conn(0x7f85240fbbc0 msgr2=0x7f8524104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:18.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.669+0000 7f852984e640 1 -- 172.21.15.23:0/1010859726 shutdown_connections 2024-03-24T10:38:18.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.669+0000 7f852984e640 1 -- 172.21.15.23:0/1010859726 wait complete. 2024-03-24T10:38:18.681 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-03-24T10:38:16.794150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600896,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":272,"num_write_kb":2974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":218,"ondisk_log_size":218,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":277,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365940,"kb_used_data":12588,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575884,"statfs":{"total":1151940427776,"available":1151565705216,"internally_reserved":0,"allocated":12890112,"data_stored":7036598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10528,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":91,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.950146"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563024+0000","last_change":"2024-03-24T10:38:10.882385+0000","last_active":"2024-03-24T10:38:12.563024+0000","last_peered":"2024-03-24T10:38:12.563024+0000","last_clean":"2024-03-24T10:38:12.563024+0000","last_became_active":"2024-03-24T10:38:10.882062+0000","last_became_peered":"2024-03-24T10:38:10.882062+0000","last_unstale":"2024-03-24T10:38:12.563024+0000","last_undegraded":"2024-03-24T10:38:12.563024+0000","last_fullsized":"2024-03-24T10:38:12.563024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:25.151062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562813+0000","last_change":"2024-03-24T10:38:10.878001+0000","last_active":"2024-03-24T10:38:12.562813+0000","last_peered":"2024-03-24T10:38:12.562813+0000","last_clean":"2024-03-24T10:38:12.562813+0000","last_became_active":"2024-03-24T10:38:10.877661+0000","last_became_peered":"2024-03-24T10:38:10.877661+0000","last_unstale":"2024-03-24T10:38:12.562813+0000","last_undegraded":"2024-03-24T10:38:12.562813+0000","last_fullsized":"2024-03-24T10:38:12.562813+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:40:19.062969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562806+0000","last_change":"2024-03-24T10:38:10.891809+0000","last_active":"2024-03-24T10:38:12.562806+0000","last_peered":"2024-03-24T10:38:12.562806+0000","last_clean":"2024-03-24T10:38:12.562806+0000","last_became_active":"2024-03-24T10:38:10.890354+0000","last_became_peered":"2024-03-24T10:38:10.890354+0000","last_unstale":"2024-03-24T10:38:12.562806+0000","last_undegraded":"2024-03-24T10:38:12.562806+0000","last_fullsized":"2024-03-24T10:38:12.562806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:59:26.037914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.976421+0000","last_change":"2024-03-24T10:38:10.885097+0000","last_active":"2024-03-24T10:38:11.976421+0000","last_peered":"2024-03-24T10:38:11.976421+0000","last_clean":"2024-03-24T10:38:11.976421+0000","last_became_active":"2024-03-24T10:38:10.884842+0000","last_became_peered":"2024-03-24T10:38:10.884842+0000","last_unstale":"2024-03-24T10:38:11.976421+0000","last_undegraded":"2024-03-24T10:38:11.976421+0000","last_fullsized":"2024-03-24T10:38:11.976421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:39:44.085699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882410+0000","last_change":"2024-03-24T10:38:10.877653+0000","last_active":"2024-03-24T10:38:11.882410+0000","last_peered":"2024-03-24T10:38:11.882410+0000","last_clean":"2024-03-24T10:38:11.882410+0000","last_became_active":"2024-03-24T10:38:10.877249+0000","last_became_peered":"2024-03-24T10:38:10.877249+0000","last_unstale":"2024-03-24T10:38:11.882410+0000","last_undegraded":"2024-03-24T10:38:11.882410+0000","last_fullsized":"2024-03-24T10:38:11.882410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:17.504660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562542+0000","last_change":"2024-03-24T10:38:10.876584+0000","last_active":"2024-03-24T10:38:12.562542+0000","last_peered":"2024-03-24T10:38:12.562542+0000","last_clean":"2024-03-24T10:38:12.562542+0000","last_became_active":"2024-03-24T10:38:10.876173+0000","last_became_peered":"2024-03-24T10:38:10.876173+0000","last_unstale":"2024-03-24T10:38:12.562542+0000","last_undegraded":"2024-03-24T10:38:12.562542+0000","last_fullsized":"2024-03-24T10:38:12.562542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:33.899155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872500+0000","last_change":"2024-03-24T10:38:10.874206+0000","last_active":"2024-03-24T10:38:11.872500+0000","last_peered":"2024-03-24T10:38:11.872500+0000","last_clean":"2024-03-24T10:38:11.872500+0000","last_became_active":"2024-03-24T10:38:10.873779+0000","last_became_peered":"2024-03-24T10:38:10.873779+0000","last_unstale":"2024-03-24T10:38:11.872500+0000","last_undegraded":"2024-03-24T10:38:11.872500+0000","last_fullsized":"2024-03-24T10:38:11.872500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:08:19.928998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882170+0000","last_change":"2024-03-24T10:38:10.884219+0000","last_active":"2024-03-24T10:38:11.882170+0000","last_peered":"2024-03-24T10:38:11.882170+0000","last_clean":"2024-03-24T10:38:11.882170+0000","last_became_active":"2024-03-24T10:38:10.883863+0000","last_became_peered":"2024-03-24T10:38:10.883863+0000","last_unstale":"2024-03-24T10:38:11.882170+0000","last_undegraded":"2024-03-24T10:38:11.882170+0000","last_fullsized":"2024-03-24T10:38:11.882170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:26:34.932083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562469+0000","last_change":"2024-03-24T10:38:10.881297+0000","last_active":"2024-03-24T10:38:12.562469+0000","last_peered":"2024-03-24T10:38:12.562469+0000","last_clean":"2024-03-24T10:38:12.562469+0000","last_became_active":"2024-03-24T10:38:10.879871+0000","last_became_peered":"2024-03-24T10:38:10.879871+0000","last_unstale":"2024-03-24T10:38:12.562469+0000","last_undegraded":"2024-03-24T10:38:12.562469+0000","last_fullsized":"2024-03-24T10:38:12.562469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:05.132788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562873+0000","last_change":"2024-03-24T10:38:10.876648+0000","last_active":"2024-03-24T10:38:12.562873+0000","last_peered":"2024-03-24T10:38:12.562873+0000","last_clean":"2024-03-24T10:38:12.562873+0000","last_became_active":"2024-03-24T10:38:10.876113+0000","last_became_peered":"2024-03-24T10:38:10.876113+0000","last_unstale":"2024-03-24T10:38:12.562873+0000","last_undegraded":"2024-03-24T10:38:12.562873+0000","last_fullsized":"2024-03-24T10:38:12.562873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:29:18.282797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872901+0000","last_change":"2024-03-24T10:38:10.875836+0000","last_active":"2024-03-24T10:38:11.872901+0000","last_peered":"2024-03-24T10:38:11.872901+0000","last_clean":"2024-03-24T10:38:11.872901+0000","last_became_active":"2024-03-24T10:38:10.875522+0000","last_became_peered":"2024-03-24T10:38:10.875522+0000","last_unstale":"2024-03-24T10:38:11.872901+0000","last_undegraded":"2024-03-24T10:38:11.872901+0000","last_fullsized":"2024-03-24T10:38:11.872901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:41.282123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.976376+0000","last_change":"2024-03-24T10:38:10.884567+0000","last_active":"2024-03-24T10:38:11.976376+0000","last_peered":"2024-03-24T10:38:11.976376+0000","last_clean":"2024-03-24T10:38:11.976376+0000","last_became_active":"2024-03-24T10:38:10.884195+0000","last_became_peered":"2024-03-24T10:38:10.884195+0000","last_unstale":"2024-03-24T10:38:11.976376+0000","last_undegraded":"2024-03-24T10:38:11.976376+0000","last_fullsized":"2024-03-24T10:38:11.976376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:29.286631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.870005+0000","last_change":"2024-03-24T10:38:10.873464+0000","last_active":"2024-03-24T10:38:11.870005+0000","last_peered":"2024-03-24T10:38:11.870005+0000","last_clean":"2024-03-24T10:38:11.870005+0000","last_became_active":"2024-03-24T10:38:10.873128+0000","last_became_peered":"2024-03-24T10:38:10.873128+0000","last_unstale":"2024-03-24T10:38:11.870005+0000","last_undegraded":"2024-03-24T10:38:11.870005+0000","last_fullsized":"2024-03-24T10:38:11.870005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:21.857942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562588+0000","last_change":"2024-03-24T10:38:10.879942+0000","last_active":"2024-03-24T10:38:12.562588+0000","last_peered":"2024-03-24T10:38:12.562588+0000","last_clean":"2024-03-24T10:38:12.562588+0000","last_became_active":"2024-03-24T10:38:10.879396+0000","last_became_peered":"2024-03-24T10:38:10.879396+0000","last_unstale":"2024-03-24T10:38:12.562588+0000","last_undegraded":"2024-03-24T10:38:12.562588+0000","last_fullsized":"2024-03-24T10:38:12.562588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:41.477988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562540+0000","last_change":"2024-03-24T10:38:10.886667+0000","last_active":"2024-03-24T10:38:12.562540+0000","last_peered":"2024-03-24T10:38:12.562540+0000","last_clean":"2024-03-24T10:38:12.562540+0000","last_became_active":"2024-03-24T10:38:10.886293+0000","last_became_peered":"2024-03-24T10:38:10.886293+0000","last_unstale":"2024-03-24T10:38:12.562540+0000","last_undegraded":"2024-03-24T10:38:12.562540+0000","last_fullsized":"2024-03-24T10:38:12.562540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:43:43.414573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.561883+0000","last_change":"2024-03-24T10:38:10.883671+0000","last_active":"2024-03-24T10:38:12.561883+0000","last_peered":"2024-03-24T10:38:12.561883+0000","last_clean":"2024-03-24T10:38:12.561883+0000","last_became_active":"2024-03-24T10:38:10.883013+0000","last_became_peered":"2024-03-24T10:38:10.883013+0000","last_unstale":"2024-03-24T10:38:12.561883+0000","last_undegraded":"2024-03-24T10:38:12.561883+0000","last_fullsized":"2024-03-24T10:38:12.561883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:33.936543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563108+0000","last_change":"2024-03-24T10:38:10.887286+0000","last_active":"2024-03-24T10:38:12.563108+0000","last_peered":"2024-03-24T10:38:12.563108+0000","last_clean":"2024-03-24T10:38:12.563108+0000","last_became_active":"2024-03-24T10:38:10.886918+0000","last_became_peered":"2024-03-24T10:38:10.886918+0000","last_unstale":"2024-03-24T10:38:12.563108+0000","last_undegraded":"2024-03-24T10:38:12.563108+0000","last_fullsized":"2024-03-24T10:38:12.563108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:40.266589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.880480+0000","last_change":"2024-03-24T10:38:10.874476+0000","last_active":"2024-03-24T10:38:11.880480+0000","last_peered":"2024-03-24T10:38:11.880480+0000","last_clean":"2024-03-24T10:38:11.880480+0000","last_became_active":"2024-03-24T10:38:10.874123+0000","last_became_peered":"2024-03-24T10:38:10.874123+0000","last_unstale":"2024-03-24T10:38:11.880480+0000","last_undegraded":"2024-03-24T10:38:11.880480+0000","last_fullsized":"2024-03-24T10:38:11.880480+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:29.759320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563431+0000","last_change":"2024-03-24T10:38:10.878758+0000","last_active":"2024-03-24T10:38:12.563431+0000","last_peered":"2024-03-24T10:38:12.563431+0000","last_clean":"2024-03-24T10:38:12.563431+0000","last_became_active":"2024-03-24T10:38:10.878424+0000","last_became_peered":"2024-03-24T10:38:10.878424+0000","last_unstale":"2024-03-24T10:38:12.563431+0000","last_undegraded":"2024-03-24T10:38:12.563431+0000","last_fullsized":"2024-03-24T10:38:12.563431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:15:19.273108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.880406+0000","last_change":"2024-03-24T10:38:09.861751+0000","last_active":"2024-03-24T10:38:11.880406+0000","last_peered":"2024-03-24T10:38:11.880406+0000","last_clean":"2024-03-24T10:38:11.880406+0000","last_became_active":"2024-03-24T10:38:09.860889+0000","last_became_peered":"2024-03-24T10:38:09.860889+0000","last_unstale":"2024-03-24T10:38:11.880406+0000","last_undegraded":"2024-03-24T10:38:11.880406+0000","last_fullsized":"2024-03-24T10:38:11.880406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:22:53.447005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882538+0000","last_change":"2024-03-24T10:38:10.885198+0000","last_active":"2024-03-24T10:38:11.882538+0000","last_peered":"2024-03-24T10:38:11.882538+0000","last_clean":"2024-03-24T10:38:11.882538+0000","last_became_active":"2024-03-24T10:38:10.884123+0000","last_became_peered":"2024-03-24T10:38:10.884123+0000","last_unstale":"2024-03-24T10:38:11.882538+0000","last_undegraded":"2024-03-24T10:38:11.882538+0000","last_fullsized":"2024-03-24T10:38:11.882538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:37:12.510671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882500+0000","last_change":"2024-03-24T10:38:09.866223+0000","last_active":"2024-03-24T10:38:11.882500+0000","last_peered":"2024-03-24T10:38:11.882500+0000","last_clean":"2024-03-24T10:38:11.882500+0000","last_became_active":"2024-03-24T10:38:09.865711+0000","last_became_peered":"2024-03-24T10:38:09.865711+0000","last_unstale":"2024-03-24T10:38:11.882500+0000","last_undegraded":"2024-03-24T10:38:11.882500+0000","last_fullsized":"2024-03-24T10:38:11.882500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:26:08.380749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563059+0000","last_change":"2024-03-24T10:38:10.890557+0000","last_active":"2024-03-24T10:38:12.563059+0000","last_peered":"2024-03-24T10:38:12.563059+0000","last_clean":"2024-03-24T10:38:12.563059+0000","last_became_active":"2024-03-24T10:38:10.889883+0000","last_became_peered":"2024-03-24T10:38:10.889883+0000","last_unstale":"2024-03-24T10:38:12.563059+0000","last_undegraded":"2024-03-24T10:38:12.563059+0000","last_fullsized":"2024-03-24T10:38:12.563059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:34.731585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:14.906550+0000","last_change":"2024-03-24T10:38:09.870253+0000","last_active":"2024-03-24T10:38:14.906550+0000","last_peered":"2024-03-24T10:38:14.906550+0000","last_clean":"2024-03-24T10:38:14.906550+0000","last_became_active":"2024-03-24T10:38:09.869932+0000","last_became_peered":"2024-03-24T10:38:09.869932+0000","last_unstale":"2024-03-24T10:38:14.906550+0000","last_undegraded":"2024-03-24T10:38:14.906550+0000","last_fullsized":"2024-03-24T10:38:14.906550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:38:43.626390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562157+0000","last_change":"2024-03-24T10:38:10.881705+0000","last_active":"2024-03-24T10:38:12.562157+0000","last_peered":"2024-03-24T10:38:12.562157+0000","last_clean":"2024-03-24T10:38:12.562157+0000","last_became_active":"2024-03-24T10:38:10.881323+0000","last_became_peered":"2024-03-24T10:38:10.881323+0000","last_unstale":"2024-03-24T10:38:12.562157+0000","last_undegraded":"2024-03-24T10:38:12.562157+0000","last_fullsized":"2024-03-24T10:38:12.562157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:25.861184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882540+0000","last_change":"2024-03-24T10:38:09.870481+0000","last_active":"2024-03-24T10:38:11.882540+0000","last_peered":"2024-03-24T10:38:11.882540+0000","last_clean":"2024-03-24T10:38:11.882540+0000","last_became_active":"2024-03-24T10:38:09.870117+0000","last_became_peered":"2024-03-24T10:38:09.870117+0000","last_unstale":"2024-03-24T10:38:11.882540+0000","last_undegraded":"2024-03-24T10:38:11.882540+0000","last_fullsized":"2024-03-24T10:38:11.882540+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:37:51.871068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.870544+0000","last_change":"2024-03-24T10:38:10.884196+0000","last_active":"2024-03-24T10:38:11.870544+0000","last_peered":"2024-03-24T10:38:11.870544+0000","last_clean":"2024-03-24T10:38:11.870544+0000","last_became_active":"2024-03-24T10:38:10.882576+0000","last_became_peered":"2024-03-24T10:38:10.882576+0000","last_unstale":"2024-03-24T10:38:11.870544+0000","last_undegraded":"2024-03-24T10:38:11.870544+0000","last_fullsized":"2024-03-24T10:38:11.870544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:58:45.698875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:15.917729+0000","last_change":"2024-03-24T10:38:09.864160+0000","last_active":"2024-03-24T10:38:15.917729+0000","last_peered":"2024-03-24T10:38:15.917729+0000","last_clean":"2024-03-24T10:38:15.917729+0000","last_became_active":"2024-03-24T10:38:09.862066+0000","last_became_peered":"2024-03-24T10:38:09.862066+0000","last_unstale":"2024-03-24T10:38:15.917729+0000","last_undegraded":"2024-03-24T10:38:15.917729+0000","last_fullsized":"2024-03-24T10:38:15.917729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:59.003931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882427+0000","last_change":"2024-03-24T10:38:10.874433+0000","last_active":"2024-03-24T10:38:11.882427+0000","last_peered":"2024-03-24T10:38:11.882427+0000","last_clean":"2024-03-24T10:38:11.882427+0000","last_became_active":"2024-03-24T10:38:10.874090+0000","last_became_peered":"2024-03-24T10:38:10.874090+0000","last_unstale":"2024-03-24T10:38:11.882427+0000","last_undegraded":"2024-03-24T10:38:11.882427+0000","last_fullsized":"2024-03-24T10:38:11.882427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:25.979419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:14.907034+0000","last_change":"2024-03-24T10:38:09.863028+0000","last_active":"2024-03-24T10:38:14.907034+0000","last_peered":"2024-03-24T10:38:14.907034+0000","last_clean":"2024-03-24T10:38:14.907034+0000","last_became_active":"2024-03-24T10:38:09.862682+0000","last_became_peered":"2024-03-24T10:38:09.862682+0000","last_unstale":"2024-03-24T10:38:14.907034+0000","last_undegraded":"2024-03-24T10:38:14.907034+0000","last_fullsized":"2024-03-24T10:38:14.907034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:14.962115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.870486+0000","last_change":"2024-03-24T10:38:10.885950+0000","last_active":"2024-03-24T10:38:11.870486+0000","last_peered":"2024-03-24T10:38:11.870486+0000","last_clean":"2024-03-24T10:38:11.870486+0000","last_became_active":"2024-03-24T10:38:10.885685+0000","last_became_peered":"2024-03-24T10:38:10.885685+0000","last_unstale":"2024-03-24T10:38:11.870486+0000","last_undegraded":"2024-03-24T10:38:11.870486+0000","last_fullsized":"2024-03-24T10:38:11.870486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:42:09.912743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869974+0000","last_change":"2024-03-24T10:38:09.860376+0000","last_active":"2024-03-24T10:38:11.869974+0000","last_peered":"2024-03-24T10:38:11.869974+0000","last_clean":"2024-03-24T10:38:11.869974+0000","last_became_active":"2024-03-24T10:38:09.860062+0000","last_became_peered":"2024-03-24T10:38:09.860062+0000","last_unstale":"2024-03-24T10:38:11.869974+0000","last_undegraded":"2024-03-24T10:38:11.869974+0000","last_fullsized":"2024-03-24T10:38:11.869974+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:33.387953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882144+0000","last_change":"2024-03-24T10:38:10.883843+0000","last_active":"2024-03-24T10:38:11.882144+0000","last_peered":"2024-03-24T10:38:11.882144+0000","last_clean":"2024-03-24T10:38:11.882144+0000","last_became_active":"2024-03-24T10:38:10.883170+0000","last_became_peered":"2024-03-24T10:38:10.883170+0000","last_unstale":"2024-03-24T10:38:11.882144+0000","last_undegraded":"2024-03-24T10:38:11.882144+0000","last_fullsized":"2024-03-24T10:38:11.882144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:28:16.798546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869908+0000","last_change":"2024-03-24T10:38:09.862299+0000","last_active":"2024-03-24T10:38:11.869908+0000","last_peered":"2024-03-24T10:38:11.869908+0000","last_clean":"2024-03-24T10:38:11.869908+0000","last_became_active":"2024-03-24T10:38:09.861895+0000","last_became_peered":"2024-03-24T10:38:09.861895+0000","last_unstale":"2024-03-24T10:38:11.869908+0000","last_undegraded":"2024-03-24T10:38:11.869908+0000","last_fullsized":"2024-03-24T10:38:11.869908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:13:19.869511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.880311+0000","last_change":"2024-03-24T10:38:10.880507+0000","last_active":"2024-03-24T10:38:11.880311+0000","last_peered":"2024-03-24T10:38:11.880311+0000","last_clean":"2024-03-24T10:38:11.880311+0000","last_became_active":"2024-03-24T10:38:10.879790+0000","last_became_peered":"2024-03-24T10:38:10.879790+0000","last_unstale":"2024-03-24T10:38:11.880311+0000","last_undegraded":"2024-03-24T10:38:11.880311+0000","last_fullsized":"2024-03-24T10:38:11.880311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:11:02.650420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.897213+0000","last_change":"2024-03-24T10:38:09.860186+0000","last_active":"2024-03-24T10:38:13.897213+0000","last_peered":"2024-03-24T10:38:13.897213+0000","last_clean":"2024-03-24T10:38:13.897213+0000","last_became_active":"2024-03-24T10:38:09.859834+0000","last_became_peered":"2024-03-24T10:38:09.859834+0000","last_unstale":"2024-03-24T10:38:13.897213+0000","last_undegraded":"2024-03-24T10:38:13.897213+0000","last_fullsized":"2024-03-24T10:38:13.897213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:33:48.029053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562396+0000","last_change":"2024-03-24T10:38:10.875634+0000","last_active":"2024-03-24T10:38:12.562396+0000","last_peered":"2024-03-24T10:38:12.562396+0000","last_clean":"2024-03-24T10:38:12.562396+0000","last_became_active":"2024-03-24T10:38:10.875101+0000","last_became_peered":"2024-03-24T10:38:10.875101+0000","last_unstale":"2024-03-24T10:38:12.562396+0000","last_undegraded":"2024-03-24T10:38:12.562396+0000","last_fullsized":"2024-03-24T10:38:12.562396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:32:58.165087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.895392+0000","last_change":"2024-03-24T10:38:09.860412+0000","last_active":"2024-03-24T10:38:13.895392+0000","last_peered":"2024-03-24T10:38:13.895392+0000","last_clean":"2024-03-24T10:38:13.895392+0000","last_became_active":"2024-03-24T10:38:09.860084+0000","last_became_peered":"2024-03-24T10:38:09.860084+0000","last_unstale":"2024-03-24T10:38:13.895392+0000","last_undegraded":"2024-03-24T10:38:13.895392+0000","last_fullsized":"2024-03-24T10:38:13.895392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:44.585482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563060+0000","last_change":"2024-03-24T10:38:10.885489+0000","last_active":"2024-03-24T10:38:12.563060+0000","last_peered":"2024-03-24T10:38:12.563060+0000","last_clean":"2024-03-24T10:38:12.563060+0000","last_became_active":"2024-03-24T10:38:10.884810+0000","last_became_peered":"2024-03-24T10:38:10.884810+0000","last_unstale":"2024-03-24T10:38:12.563060+0000","last_undegraded":"2024-03-24T10:38:12.563060+0000","last_fullsized":"2024-03-24T10:38:12.563060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:12:15.922850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.897451+0000","last_change":"2024-03-24T10:38:09.865191+0000","last_active":"2024-03-24T10:38:13.897451+0000","last_peered":"2024-03-24T10:38:13.897451+0000","last_clean":"2024-03-24T10:38:13.897451+0000","last_became_active":"2024-03-24T10:38:09.864470+0000","last_became_peered":"2024-03-24T10:38:09.864470+0000","last_unstale":"2024-03-24T10:38:13.897451+0000","last_undegraded":"2024-03-24T10:38:13.897451+0000","last_fullsized":"2024-03-24T10:38:13.897451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:45:00.266470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562279+0000","last_change":"2024-03-24T10:38:10.889668+0000","last_active":"2024-03-24T10:38:12.562279+0000","last_peered":"2024-03-24T10:38:12.562279+0000","last_clean":"2024-03-24T10:38:12.562279+0000","last_became_active":"2024-03-24T10:38:10.889364+0000","last_became_peered":"2024-03-24T10:38:10.889364+0000","last_unstale":"2024-03-24T10:38:12.562279+0000","last_undegraded":"2024-03-24T10:38:12.562279+0000","last_fullsized":"2024-03-24T10:38:12.562279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:57.702719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'11","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:16.269532+0000","last_change":"2024-03-24T10:38:09.861809+0000","last_active":"2024-03-24T10:38:16.269532+0000","last_peered":"2024-03-24T10:38:16.269532+0000","last_clean":"2024-03-24T10:38:16.269532+0000","last_became_active":"2024-03-24T10:38:09.861444+0000","last_became_peered":"2024-03-24T10:38:09.861444+0000","last_unstale":"2024-03-24T10:38:16.269532+0000","last_undegraded":"2024-03-24T10:38:16.269532+0000","last_fullsized":"2024-03-24T10:38:16.269532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:19.652188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8212,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882691+0000","last_change":"2024-03-24T10:38:10.883090+0000","last_active":"2024-03-24T10:38:11.882691+0000","last_peered":"2024-03-24T10:38:11.882691+0000","last_clean":"2024-03-24T10:38:11.882691+0000","last_became_active":"2024-03-24T10:38:10.880507+0000","last_became_peered":"2024-03-24T10:38:10.880507+0000","last_unstale":"2024-03-24T10:38:11.882691+0000","last_undegraded":"2024-03-24T10:38:11.882691+0000","last_fullsized":"2024-03-24T10:38:11.882691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:01.422699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563326+0000","last_change":"2024-03-24T10:38:09.868072+0000","last_active":"2024-03-24T10:38:12.563326+0000","last_peered":"2024-03-24T10:38:12.563326+0000","last_clean":"2024-03-24T10:38:12.563326+0000","last_became_active":"2024-03-24T10:38:09.867704+0000","last_became_peered":"2024-03-24T10:38:09.867704+0000","last_unstale":"2024-03-24T10:38:12.563326+0000","last_undegraded":"2024-03-24T10:38:12.563326+0000","last_fullsized":"2024-03-24T10:38:12.563326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:09:28.726969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869956+0000","last_change":"2024-03-24T10:38:10.881882+0000","last_active":"2024-03-24T10:38:11.869956+0000","last_peered":"2024-03-24T10:38:11.869956+0000","last_clean":"2024-03-24T10:38:11.869956+0000","last_became_active":"2024-03-24T10:38:10.881495+0000","last_became_peered":"2024-03-24T10:38:10.881495+0000","last_unstale":"2024-03-24T10:38:11.869956+0000","last_undegraded":"2024-03-24T10:38:11.869956+0000","last_fullsized":"2024-03-24T10:38:11.869956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:31:43.392952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.897503+0000","last_change":"2024-03-24T10:38:09.867265+0000","last_active":"2024-03-24T10:38:13.897503+0000","last_peered":"2024-03-24T10:38:13.897503+0000","last_clean":"2024-03-24T10:38:13.897503+0000","last_became_active":"2024-03-24T10:38:09.866669+0000","last_became_peered":"2024-03-24T10:38:09.866669+0000","last_unstale":"2024-03-24T10:38:13.897503+0000","last_undegraded":"2024-03-24T10:38:13.897503+0000","last_fullsized":"2024-03-24T10:38:13.897503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:28.230967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872911+0000","last_change":"2024-03-24T10:38:10.886634+0000","last_active":"2024-03-24T10:38:11.872911+0000","last_peered":"2024-03-24T10:38:11.872911+0000","last_clean":"2024-03-24T10:38:11.872911+0000","last_became_active":"2024-03-24T10:38:10.886207+0000","last_became_peered":"2024-03-24T10:38:10.886207+0000","last_unstale":"2024-03-24T10:38:11.872911+0000","last_undegraded":"2024-03-24T10:38:11.872911+0000","last_fullsized":"2024-03-24T10:38:11.872911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:43:41.601693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.976423+0000","last_change":"2024-03-24T10:38:09.863262+0000","last_active":"2024-03-24T10:38:11.976423+0000","last_peered":"2024-03-24T10:38:11.976423+0000","last_clean":"2024-03-24T10:38:11.976423+0000","last_became_active":"2024-03-24T10:38:09.862665+0000","last_became_peered":"2024-03-24T10:38:09.862665+0000","last_unstale":"2024-03-24T10:38:11.976423+0000","last_undegraded":"2024-03-24T10:38:11.976423+0000","last_fullsized":"2024-03-24T10:38:11.976423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:12:29.117592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872423+0000","last_change":"2024-03-24T10:38:10.880793+0000","last_active":"2024-03-24T10:38:11.872423+0000","last_peered":"2024-03-24T10:38:11.872423+0000","last_clean":"2024-03-24T10:38:11.872423+0000","last_became_active":"2024-03-24T10:38:10.880449+0000","last_became_peered":"2024-03-24T10:38:10.880449+0000","last_unstale":"2024-03-24T10:38:11.872423+0000","last_undegraded":"2024-03-24T10:38:11.872423+0000","last_fullsized":"2024-03-24T10:38:11.872423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:32:57.184014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:14.909457+0000","last_change":"2024-03-24T10:38:09.867694+0000","last_active":"2024-03-24T10:38:14.909457+0000","last_peered":"2024-03-24T10:38:14.909457+0000","last_clean":"2024-03-24T10:38:14.909457+0000","last_became_active":"2024-03-24T10:38:09.867345+0000","last_became_peered":"2024-03-24T10:38:09.867345+0000","last_unstale":"2024-03-24T10:38:14.909457+0000","last_undegraded":"2024-03-24T10:38:14.909457+0000","last_fullsized":"2024-03-24T10:38:14.909457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:39.436352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.870014+0000","last_change":"2024-03-24T10:38:10.881952+0000","last_active":"2024-03-24T10:38:11.870014+0000","last_peered":"2024-03-24T10:38:11.870014+0000","last_clean":"2024-03-24T10:38:11.870014+0000","last_became_active":"2024-03-24T10:38:10.881628+0000","last_became_peered":"2024-03-24T10:38:10.881628+0000","last_unstale":"2024-03-24T10:38:11.870014+0000","last_undegraded":"2024-03-24T10:38:11.870014+0000","last_fullsized":"2024-03-24T10:38:11.870014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:24.777793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882682+0000","last_change":"2024-03-24T10:38:09.868572+0000","last_active":"2024-03-24T10:38:11.882682+0000","last_peered":"2024-03-24T10:38:11.882682+0000","last_clean":"2024-03-24T10:38:11.882682+0000","last_became_active":"2024-03-24T10:38:09.867702+0000","last_became_peered":"2024-03-24T10:38:09.867702+0000","last_unstale":"2024-03-24T10:38:11.882682+0000","last_undegraded":"2024-03-24T10:38:11.882682+0000","last_fullsized":"2024-03-24T10:38:11.882682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:27.890114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562843+0000","last_change":"2024-03-24T10:38:10.875342+0000","last_active":"2024-03-24T10:38:12.562843+0000","last_peered":"2024-03-24T10:38:12.562843+0000","last_clean":"2024-03-24T10:38:12.562843+0000","last_became_active":"2024-03-24T10:38:10.874933+0000","last_became_peered":"2024-03-24T10:38:10.874933+0000","last_unstale":"2024-03-24T10:38:12.562843+0000","last_undegraded":"2024-03-24T10:38:12.562843+0000","last_fullsized":"2024-03-24T10:38:12.562843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:56:09.139478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.880371+0000","last_change":"2024-03-24T10:38:09.857867+0000","last_active":"2024-03-24T10:38:11.880371+0000","last_peered":"2024-03-24T10:38:11.880371+0000","last_clean":"2024-03-24T10:38:11.880371+0000","last_became_active":"2024-03-24T10:38:09.857525+0000","last_became_peered":"2024-03-24T10:38:09.857525+0000","last_unstale":"2024-03-24T10:38:11.880371+0000","last_undegraded":"2024-03-24T10:38:11.880371+0000","last_fullsized":"2024-03-24T10:38:11.880371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:37.320245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562675+0000","last_change":"2024-03-24T10:38:10.878980+0000","last_active":"2024-03-24T10:38:12.562675+0000","last_peered":"2024-03-24T10:38:12.562675+0000","last_clean":"2024-03-24T10:38:12.562675+0000","last_became_active":"2024-03-24T10:38:10.878648+0000","last_became_peered":"2024-03-24T10:38:10.878648+0000","last_unstale":"2024-03-24T10:38:12.562675+0000","last_undegraded":"2024-03-24T10:38:12.562675+0000","last_fullsized":"2024-03-24T10:38:12.562675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:49.988403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872661+0000","last_change":"2024-03-24T10:38:09.863457+0000","last_active":"2024-03-24T10:38:11.872661+0000","last_peered":"2024-03-24T10:38:11.872661+0000","last_clean":"2024-03-24T10:38:11.872661+0000","last_became_active":"2024-03-24T10:38:09.863130+0000","last_became_peered":"2024-03-24T10:38:09.863130+0000","last_unstale":"2024-03-24T10:38:11.872661+0000","last_undegraded":"2024-03-24T10:38:11.872661+0000","last_fullsized":"2024-03-24T10:38:11.872661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:16.528703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562879+0000","last_change":"2024-03-24T10:38:10.887014+0000","last_active":"2024-03-24T10:38:12.562879+0000","last_peered":"2024-03-24T10:38:12.562879+0000","last_clean":"2024-03-24T10:38:12.562879+0000","last_became_active":"2024-03-24T10:38:10.886698+0000","last_became_peered":"2024-03-24T10:38:10.886698+0000","last_unstale":"2024-03-24T10:38:12.562879+0000","last_undegraded":"2024-03-24T10:38:12.562879+0000","last_fullsized":"2024-03-24T10:38:12.562879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:47:08.571886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.870336+0000","last_change":"2024-03-24T10:38:10.874811+0000","last_active":"2024-03-24T10:38:11.870336+0000","last_peered":"2024-03-24T10:38:11.870336+0000","last_clean":"2024-03-24T10:38:11.870336+0000","last_became_active":"2024-03-24T10:38:10.874483+0000","last_became_peered":"2024-03-24T10:38:10.874483+0000","last_unstale":"2024-03-24T10:38:11.870336+0000","last_undegraded":"2024-03-24T10:38:11.870336+0000","last_fullsized":"2024-03-24T10:38:11.870336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:57.382174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869601+0000","last_change":"2024-03-24T10:38:09.865614+0000","last_active":"2024-03-24T10:38:11.869601+0000","last_peered":"2024-03-24T10:38:11.869601+0000","last_clean":"2024-03-24T10:38:11.869601+0000","last_became_active":"2024-03-24T10:38:09.865317+0000","last_became_peered":"2024-03-24T10:38:09.865317+0000","last_unstale":"2024-03-24T10:38:11.869601+0000","last_undegraded":"2024-03-24T10:38:11.869601+0000","last_fullsized":"2024-03-24T10:38:11.869601+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:21:14.968552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882577+0000","last_change":"2024-03-24T10:38:10.884847+0000","last_active":"2024-03-24T10:38:11.882577+0000","last_peered":"2024-03-24T10:38:11.882577+0000","last_clean":"2024-03-24T10:38:11.882577+0000","last_became_active":"2024-03-24T10:38:10.884526+0000","last_became_peered":"2024-03-24T10:38:10.884526+0000","last_unstale":"2024-03-24T10:38:11.882577+0000","last_undegraded":"2024-03-24T10:38:11.882577+0000","last_fullsized":"2024-03-24T10:38:11.882577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:01:49.364083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.976264+0000","last_change":"2024-03-24T10:38:09.858893+0000","last_active":"2024-03-24T10:38:11.976264+0000","last_peered":"2024-03-24T10:38:11.976264+0000","last_clean":"2024-03-24T10:38:11.976264+0000","last_became_active":"2024-03-24T10:38:09.858474+0000","last_became_peered":"2024-03-24T10:38:09.858474+0000","last_unstale":"2024-03-24T10:38:11.976264+0000","last_undegraded":"2024-03-24T10:38:11.976264+0000","last_fullsized":"2024-03-24T10:38:11.976264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:03:36.709498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872412+0000","last_change":"2024-03-24T10:36:59.100728+0000","last_active":"2024-03-24T10:38:11.872412+0000","last_peered":"2024-03-24T10:38:11.872412+0000","last_clean":"2024-03-24T10:38:11.872412+0000","last_became_active":"2024-03-24T10:36:59.100159+0000","last_became_peered":"2024-03-24T10:36:59.100159+0000","last_unstale":"2024-03-24T10:38:11.872412+0000","last_undegraded":"2024-03-24T10:38:11.872412+0000","last_fullsized":"2024-03-24T10:38:11.872412+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:39.429553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:39.429553+0000","last_clean_scrub_stamp":"2024-03-24T10:33:39.429553+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:21:01.470585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869678+0000","last_change":"2024-03-24T10:38:10.885804+0000","last_active":"2024-03-24T10:38:11.869678+0000","last_peered":"2024-03-24T10:38:11.869678+0000","last_clean":"2024-03-24T10:38:11.869678+0000","last_became_active":"2024-03-24T10:38:10.885447+0000","last_became_peered":"2024-03-24T10:38:10.885447+0000","last_unstale":"2024-03-24T10:38:11.869678+0000","last_undegraded":"2024-03-24T10:38:11.869678+0000","last_fullsized":"2024-03-24T10:38:11.869678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:57.103205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:14.907674+0000","last_change":"2024-03-24T10:38:09.863187+0000","last_active":"2024-03-24T10:38:14.907674+0000","last_peered":"2024-03-24T10:38:14.907674+0000","last_clean":"2024-03-24T10:38:14.907674+0000","last_became_active":"2024-03-24T10:38:09.862859+0000","last_became_peered":"2024-03-24T10:38:09.862859+0000","last_unstale":"2024-03-24T10:38:14.907674+0000","last_undegraded":"2024-03-24T10:38:14.907674+0000","last_fullsized":"2024-03-24T10:38:14.907674+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:21:02.707127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882700+0000","last_change":"2024-03-24T10:38:09.867899+0000","last_active":"2024-03-24T10:38:11.882700+0000","last_peered":"2024-03-24T10:38:11.882700+0000","last_clean":"2024-03-24T10:38:11.882700+0000","last_became_active":"2024-03-24T10:38:09.867563+0000","last_became_peered":"2024-03-24T10:38:09.867563+0000","last_unstale":"2024-03-24T10:38:11.882700+0000","last_undegraded":"2024-03-24T10:38:11.882700+0000","last_fullsized":"2024-03-24T10:38:11.882700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:51.067981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869837+0000","last_change":"2024-03-24T10:38:10.870825+0000","last_active":"2024-03-24T10:38:11.869837+0000","last_peered":"2024-03-24T10:38:11.869837+0000","last_clean":"2024-03-24T10:38:11.869837+0000","last_became_active":"2024-03-24T10:38:10.870491+0000","last_became_peered":"2024-03-24T10:38:10.870491+0000","last_unstale":"2024-03-24T10:38:11.869837+0000","last_undegraded":"2024-03-24T10:38:11.869837+0000","last_fullsized":"2024-03-24T10:38:11.869837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:30:34.091299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882108+0000","last_change":"2024-03-24T10:38:09.863724+0000","last_active":"2024-03-24T10:38:11.882108+0000","last_peered":"2024-03-24T10:38:11.882108+0000","last_clean":"2024-03-24T10:38:11.882108+0000","last_became_active":"2024-03-24T10:38:09.862900+0000","last_became_peered":"2024-03-24T10:38:09.862900+0000","last_unstale":"2024-03-24T10:38:11.882108+0000","last_undegraded":"2024-03-24T10:38:11.882108+0000","last_fullsized":"2024-03-24T10:38:11.882108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:21.646074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882543+0000","last_change":"2024-03-24T10:38:10.884661+0000","last_active":"2024-03-24T10:38:11.882543+0000","last_peered":"2024-03-24T10:38:11.882543+0000","last_clean":"2024-03-24T10:38:11.882543+0000","last_became_active":"2024-03-24T10:38:10.882901+0000","last_became_peered":"2024-03-24T10:38:10.882901+0000","last_unstale":"2024-03-24T10:38:11.882543+0000","last_undegraded":"2024-03-24T10:38:11.882543+0000","last_fullsized":"2024-03-24T10:38:11.882543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:23:46.366567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.976386+0000","last_change":"2024-03-24T10:38:09.862122+0000","last_active":"2024-03-24T10:38:11.976386+0000","last_peered":"2024-03-24T10:38:11.976386+0000","last_clean":"2024-03-24T10:38:11.976386+0000","last_became_active":"2024-03-24T10:38:09.861765+0000","last_became_peered":"2024-03-24T10:38:09.861765+0000","last_unstale":"2024-03-24T10:38:11.976386+0000","last_undegraded":"2024-03-24T10:38:11.976386+0000","last_fullsized":"2024-03-24T10:38:11.976386+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:40:15.148349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562972+0000","last_change":"2024-03-24T10:38:10.886797+0000","last_active":"2024-03-24T10:38:12.562972+0000","last_peered":"2024-03-24T10:38:12.562972+0000","last_clean":"2024-03-24T10:38:12.562972+0000","last_became_active":"2024-03-24T10:38:10.886406+0000","last_became_peered":"2024-03-24T10:38:10.886406+0000","last_unstale":"2024-03-24T10:38:12.562972+0000","last_undegraded":"2024-03-24T10:38:12.562972+0000","last_fullsized":"2024-03-24T10:38:12.562972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:35.763623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.895552+0000","last_change":"2024-03-24T10:38:09.863542+0000","last_active":"2024-03-24T10:38:13.895552+0000","last_peered":"2024-03-24T10:38:13.895552+0000","last_clean":"2024-03-24T10:38:13.895552+0000","last_became_active":"2024-03-24T10:38:09.863103+0000","last_became_peered":"2024-03-24T10:38:09.863103+0000","last_unstale":"2024-03-24T10:38:13.895552+0000","last_undegraded":"2024-03-24T10:38:13.895552+0000","last_fullsized":"2024-03-24T10:38:13.895552+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:52:34.837100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882258+0000","last_change":"2024-03-24T10:38:10.873871+0000","last_active":"2024-03-24T10:38:11.882258+0000","last_peered":"2024-03-24T10:38:11.882258+0000","last_clean":"2024-03-24T10:38:11.882258+0000","last_became_active":"2024-03-24T10:38:10.873528+0000","last_became_peered":"2024-03-24T10:38:10.873528+0000","last_unstale":"2024-03-24T10:38:11.882258+0000","last_undegraded":"2024-03-24T10:38:11.882258+0000","last_fullsized":"2024-03-24T10:38:11.882258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:33.372195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.896094+0000","last_change":"2024-03-24T10:38:09.861548+0000","last_active":"2024-03-24T10:38:13.896094+0000","last_peered":"2024-03-24T10:38:13.896094+0000","last_clean":"2024-03-24T10:38:13.896094+0000","last_became_active":"2024-03-24T10:38:09.861219+0000","last_became_peered":"2024-03-24T10:38:09.861219+0000","last_unstale":"2024-03-24T10:38:13.896094+0000","last_undegraded":"2024-03-24T10:38:13.896094+0000","last_fullsized":"2024-03-24T10:38:13.896094+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:57:46.398179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562528+0000","last_change":"2024-03-24T10:38:10.880202+0000","last_active":"2024-03-24T10:38:12.562528+0000","last_peered":"2024-03-24T10:38:12.562528+0000","last_clean":"2024-03-24T10:38:12.562528+0000","last_became_active":"2024-03-24T10:38:10.879807+0000","last_became_peered":"2024-03-24T10:38:10.879807+0000","last_unstale":"2024-03-24T10:38:12.562528+0000","last_undegraded":"2024-03-24T10:38:12.562528+0000","last_fullsized":"2024-03-24T10:38:12.562528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:37.498728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563086+0000","last_change":"2024-03-24T10:38:09.866914+0000","last_active":"2024-03-24T10:38:12.563086+0000","last_peered":"2024-03-24T10:38:12.563086+0000","last_clean":"2024-03-24T10:38:12.563086+0000","last_became_active":"2024-03-24T10:38:09.866323+0000","last_became_peered":"2024-03-24T10:38:09.866323+0000","last_unstale":"2024-03-24T10:38:12.563086+0000","last_undegraded":"2024-03-24T10:38:12.563086+0000","last_fullsized":"2024-03-24T10:38:12.563086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:10:42.111336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882357+0000","last_change":"2024-03-24T10:38:10.884371+0000","last_active":"2024-03-24T10:38:11.882357+0000","last_peered":"2024-03-24T10:38:11.882357+0000","last_clean":"2024-03-24T10:38:11.882357+0000","last_became_active":"2024-03-24T10:38:10.884022+0000","last_became_peered":"2024-03-24T10:38:10.884022+0000","last_unstale":"2024-03-24T10:38:11.882357+0000","last_undegraded":"2024-03-24T10:38:11.882357+0000","last_fullsized":"2024-03-24T10:38:11.882357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:47:48.778727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.897322+0000","last_change":"2024-03-24T10:38:09.863231+0000","last_active":"2024-03-24T10:38:13.897322+0000","last_peered":"2024-03-24T10:38:13.897322+0000","last_clean":"2024-03-24T10:38:13.897322+0000","last_became_active":"2024-03-24T10:38:09.862848+0000","last_became_peered":"2024-03-24T10:38:09.862848+0000","last_unstale":"2024-03-24T10:38:13.897322+0000","last_undegraded":"2024-03-24T10:38:13.897322+0000","last_fullsized":"2024-03-24T10:38:13.897322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:36:57.532746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872857+0000","last_change":"2024-03-24T10:38:10.875187+0000","last_active":"2024-03-24T10:38:11.872857+0000","last_peered":"2024-03-24T10:38:11.872857+0000","last_clean":"2024-03-24T10:38:11.872857+0000","last_became_active":"2024-03-24T10:38:10.874682+0000","last_became_peered":"2024-03-24T10:38:10.874682+0000","last_unstale":"2024-03-24T10:38:11.872857+0000","last_undegraded":"2024-03-24T10:38:11.872857+0000","last_fullsized":"2024-03-24T10:38:11.872857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:53:56.915312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563392+0000","last_change":"2024-03-24T10:38:09.863430+0000","last_active":"2024-03-24T10:38:12.563392+0000","last_peered":"2024-03-24T10:38:12.563392+0000","last_clean":"2024-03-24T10:38:12.563392+0000","last_became_active":"2024-03-24T10:38:09.862877+0000","last_became_peered":"2024-03-24T10:38:09.862877+0000","last_unstale":"2024-03-24T10:38:12.563392+0000","last_undegraded":"2024-03-24T10:38:12.563392+0000","last_fullsized":"2024-03-24T10:38:12.563392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:33:29.039896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872851+0000","last_change":"2024-03-24T10:38:10.878446+0000","last_active":"2024-03-24T10:38:11.872851+0000","last_peered":"2024-03-24T10:38:11.872851+0000","last_clean":"2024-03-24T10:38:11.872851+0000","last_became_active":"2024-03-24T10:38:10.877944+0000","last_became_peered":"2024-03-24T10:38:10.877944+0000","last_unstale":"2024-03-24T10:38:11.872851+0000","last_undegraded":"2024-03-24T10:38:11.872851+0000","last_fullsized":"2024-03-24T10:38:11.872851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:00:22.568669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.870352+0000","last_change":"2024-03-24T10:38:09.867328+0000","last_active":"2024-03-24T10:38:11.870352+0000","last_peered":"2024-03-24T10:38:11.870352+0000","last_clean":"2024-03-24T10:38:11.870352+0000","last_became_active":"2024-03-24T10:38:09.866869+0000","last_became_peered":"2024-03-24T10:38:09.866869+0000","last_unstale":"2024-03-24T10:38:11.870352+0000","last_undegraded":"2024-03-24T10:38:11.870352+0000","last_fullsized":"2024-03-24T10:38:11.870352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:52:32.136151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872598+0000","last_change":"2024-03-24T10:38:10.878807+0000","last_active":"2024-03-24T10:38:11.872598+0000","last_peered":"2024-03-24T10:38:11.872598+0000","last_clean":"2024-03-24T10:38:11.872598+0000","last_became_active":"2024-03-24T10:38:10.878455+0000","last_became_peered":"2024-03-24T10:38:10.878455+0000","last_unstale":"2024-03-24T10:38:11.872598+0000","last_undegraded":"2024-03-24T10:38:11.872598+0000","last_fullsized":"2024-03-24T10:38:11.872598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:14.079275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869761+0000","last_change":"2024-03-24T10:38:09.865399+0000","last_active":"2024-03-24T10:38:11.869761+0000","last_peered":"2024-03-24T10:38:11.869761+0000","last_clean":"2024-03-24T10:38:11.869761+0000","last_became_active":"2024-03-24T10:38:09.864906+0000","last_became_peered":"2024-03-24T10:38:09.864906+0000","last_unstale":"2024-03-24T10:38:11.869761+0000","last_undegraded":"2024-03-24T10:38:11.869761+0000","last_fullsized":"2024-03-24T10:38:11.869761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:53:12.363018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872782+0000","last_change":"2024-03-24T10:38:10.876881+0000","last_active":"2024-03-24T10:38:11.872782+0000","last_peered":"2024-03-24T10:38:11.872782+0000","last_clean":"2024-03-24T10:38:11.872782+0000","last_became_active":"2024-03-24T10:38:10.876479+0000","last_became_peered":"2024-03-24T10:38:10.876479+0000","last_unstale":"2024-03-24T10:38:11.872782+0000","last_undegraded":"2024-03-24T10:38:11.872782+0000","last_fullsized":"2024-03-24T10:38:11.872782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:57:33.140451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562758+0000","last_change":"2024-03-24T10:38:10.872622+0000","last_active":"2024-03-24T10:38:12.562758+0000","last_peered":"2024-03-24T10:38:12.562758+0000","last_clean":"2024-03-24T10:38:12.562758+0000","last_became_active":"2024-03-24T10:38:10.872293+0000","last_became_peered":"2024-03-24T10:38:10.872293+0000","last_unstale":"2024-03-24T10:38:12.562758+0000","last_undegraded":"2024-03-24T10:38:12.562758+0000","last_fullsized":"2024-03-24T10:38:12.562758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:48.139589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563103+0000","last_change":"2024-03-24T10:38:10.888594+0000","last_active":"2024-03-24T10:38:12.563103+0000","last_peered":"2024-03-24T10:38:12.563103+0000","last_clean":"2024-03-24T10:38:12.563103+0000","last_became_active":"2024-03-24T10:38:10.888257+0000","last_became_peered":"2024-03-24T10:38:10.888257+0000","last_unstale":"2024-03-24T10:38:12.563103+0000","last_undegraded":"2024-03-24T10:38:12.563103+0000","last_fullsized":"2024-03-24T10:38:12.563103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:14:22.446011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563077+0000","last_change":"2024-03-24T10:38:10.879679+0000","last_active":"2024-03-24T10:38:12.563077+0000","last_peered":"2024-03-24T10:38:12.563077+0000","last_clean":"2024-03-24T10:38:12.563077+0000","last_became_active":"2024-03-24T10:38:10.879334+0000","last_became_peered":"2024-03-24T10:38:10.879334+0000","last_unstale":"2024-03-24T10:38:12.563077+0000","last_undegraded":"2024-03-24T10:38:12.563077+0000","last_fullsized":"2024-03-24T10:38:12.563077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:27:53.517368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562716+0000","last_change":"2024-03-24T10:38:10.886760+0000","last_active":"2024-03-24T10:38:12.562716+0000","last_peered":"2024-03-24T10:38:12.562716+0000","last_clean":"2024-03-24T10:38:12.562716+0000","last_became_active":"2024-03-24T10:38:10.886434+0000","last_became_peered":"2024-03-24T10:38:10.886434+0000","last_unstale":"2024-03-24T10:38:12.562716+0000","last_undegraded":"2024-03-24T10:38:12.562716+0000","last_fullsized":"2024-03-24T10:38:12.562716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:42:38.377683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869917+0000","last_change":"2024-03-24T10:38:10.887108+0000","last_active":"2024-03-24T10:38:11.869917+0000","last_peered":"2024-03-24T10:38:11.869917+0000","last_clean":"2024-03-24T10:38:11.869917+0000","last_became_active":"2024-03-24T10:38:10.886770+0000","last_became_peered":"2024-03-24T10:38:10.886770+0000","last_unstale":"2024-03-24T10:38:11.869917+0000","last_undegraded":"2024-03-24T10:38:11.869917+0000","last_fullsized":"2024-03-24T10:38:11.869917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:56.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562816+0000","last_change":"2024-03-24T10:38:10.873369+0000","last_active":"2024-03-24T10:38:12.562816+0000","last_peered":"2024-03-24T10:38:12.562816+0000","last_clean":"2024-03-24T10:38:12.562816+0000","last_became_active":"2024-03-24T10:38:10.873046+0000","last_became_peered":"2024-03-24T10:38:10.873046+0000","last_unstale":"2024-03-24T10:38:12.562816+0000","last_undegraded":"2024-03-24T10:38:12.562816+0000","last_fullsized":"2024-03-24T10:38:12.562816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:46.071084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563551+0000","last_change":"2024-03-24T10:38:10.878666+0000","last_active":"2024-03-24T10:38:12.563551+0000","last_peered":"2024-03-24T10:38:12.563551+0000","last_clean":"2024-03-24T10:38:12.563551+0000","last_became_active":"2024-03-24T10:38:10.877250+0000","last_became_peered":"2024-03-24T10:38:10.877250+0000","last_unstale":"2024-03-24T10:38:12.563551+0000","last_undegraded":"2024-03-24T10:38:12.563551+0000","last_fullsized":"2024-03-24T10:38:12.563551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:13:36.793584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882491+0000","last_change":"2024-03-24T10:38:10.889274+0000","last_active":"2024-03-24T10:38:11.882491+0000","last_peered":"2024-03-24T10:38:11.882491+0000","last_clean":"2024-03-24T10:38:11.882491+0000","last_became_active":"2024-03-24T10:38:10.888899+0000","last_became_peered":"2024-03-24T10:38:10.888899+0000","last_unstale":"2024-03-24T10:38:11.882491+0000","last_undegraded":"2024-03-24T10:38:11.882491+0000","last_fullsized":"2024-03-24T10:38:11.882491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:11.225562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.870025+0000","last_change":"2024-03-24T10:38:10.887135+0000","last_active":"2024-03-24T10:38:11.870025+0000","last_peered":"2024-03-24T10:38:11.870025+0000","last_clean":"2024-03-24T10:38:11.870025+0000","last_became_active":"2024-03-24T10:38:10.886812+0000","last_became_peered":"2024-03-24T10:38:10.886812+0000","last_unstale":"2024-03-24T10:38:11.870025+0000","last_undegraded":"2024-03-24T10:38:11.870025+0000","last_fullsized":"2024-03-24T10:38:11.870025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:51:53.959914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882277+0000","last_change":"2024-03-24T10:38:10.885417+0000","last_active":"2024-03-24T10:38:11.882277+0000","last_peered":"2024-03-24T10:38:11.882277+0000","last_clean":"2024-03-24T10:38:11.882277+0000","last_became_active":"2024-03-24T10:38:10.883209+0000","last_became_peered":"2024-03-24T10:38:10.883209+0000","last_unstale":"2024-03-24T10:38:11.882277+0000","last_undegraded":"2024-03-24T10:38:11.882277+0000","last_fullsized":"2024-03-24T10:38:11.882277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:52:52.535611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.561957+0000","last_change":"2024-03-24T10:38:10.885005+0000","last_active":"2024-03-24T10:38:12.561957+0000","last_peered":"2024-03-24T10:38:12.561957+0000","last_clean":"2024-03-24T10:38:12.561957+0000","last_became_active":"2024-03-24T10:38:10.884680+0000","last_became_peered":"2024-03-24T10:38:10.884680+0000","last_unstale":"2024-03-24T10:38:12.561957+0000","last_undegraded":"2024-03-24T10:38:12.561957+0000","last_fullsized":"2024-03-24T10:38:12.561957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:56.377777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869920+0000","last_change":"2024-03-24T10:38:10.875153+0000","last_active":"2024-03-24T10:38:11.869920+0000","last_peered":"2024-03-24T10:38:11.869920+0000","last_clean":"2024-03-24T10:38:11.869920+0000","last_became_active":"2024-03-24T10:38:10.874721+0000","last_became_peered":"2024-03-24T10:38:10.874721+0000","last_unstale":"2024-03-24T10:38:11.869920+0000","last_undegraded":"2024-03-24T10:38:11.869920+0000","last_fullsized":"2024-03-24T10:38:11.869920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:52.522797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562750+0000","last_change":"2024-03-24T10:38:10.887298+0000","last_active":"2024-03-24T10:38:12.562750+0000","last_peered":"2024-03-24T10:38:12.562750+0000","last_clean":"2024-03-24T10:38:12.562750+0000","last_became_active":"2024-03-24T10:38:10.886975+0000","last_became_peered":"2024-03-24T10:38:10.886975+0000","last_unstale":"2024-03-24T10:38:12.562750+0000","last_undegraded":"2024-03-24T10:38:12.562750+0000","last_fullsized":"2024-03-24T10:38:12.562750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:09.740303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10528,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":91,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":25515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32692,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712460,"statfs":{"total":95995035648,"available":95961559040,"internally_reserved":0,"allocated":1552384,"data_stored":1032079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070492,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":439147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332129,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1523712,"data_stored":1030681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495654,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":905216,"data_stored":430041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":440167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":438715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":917504,"data_stored":438543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1536000,"data_stored":1030365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":8212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":10181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:38:18.685 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:38:18.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.823+0000 7f2bd249d640 1 -- 172.21.15.23:0/2215042902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc100640 msgr2=0x7f2bcc100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.823+0000 7f2bd249d640 1 --2- 172.21.15.23:0/2215042902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc100640 0x7f2bcc100a60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009900 tx=0x7f2bbc02bc30 comp rx=0 tx=0).stop 2024-03-24T10:38:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.823+0000 7f2bd249d640 1 -- 172.21.15.23:0/2215042902 shutdown_connections 2024-03-24T10:38:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.823+0000 7f2bd249d640 1 --2- 172.21.15.23:0/2215042902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bcc1021f0 0x7f2bcc108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.823+0000 7f2bd249d640 1 --2- 172.21.15.23:0/2215042902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bcc101830 0x7f2bcc101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.823+0000 7f2bd249d640 1 --2- 172.21.15.23:0/2215042902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc100640 0x7f2bcc100a60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.823+0000 7f2bd249d640 1 -- 172.21.15.23:0/2215042902 >> 172.21.15.23:0/2215042902 conn(0x7f2bcc0fbbf0 msgr2=0x7f2bcc0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.823+0000 7f2bd249d640 1 -- 172.21.15.23:0/2215042902 shutdown_connections 2024-03-24T10:38:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.823+0000 7f2bd249d640 1 -- 172.21.15.23:0/2215042902 wait complete. 2024-03-24T10:38:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.823+0000 7f2bd249d640 1 Processor -- start 2024-03-24T10:38:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.823+0000 7f2bd249d640 1 -- start start 2024-03-24T10:38:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bd249d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bcc100640 0x7f2bcc1978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bd249d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bcc101830 0x7f2bcc197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:18.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bd249d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc1021f0 0x7f2bcc19ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bd249d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc19f3d0 con 0x7f2bcc100640 2024-03-24T10:38:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bd249d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc19f540 con 0x7f2bcc1021f0 2024-03-24T10:38:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bd249d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc19f700 con 0x7f2bcc101830 2024-03-24T10:38:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bcbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bcc100640 0x7f2bcc1978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bcb7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bcc101830 0x7f2bcc197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bcbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bcc100640 0x7f2bcc1978e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44614/0 (socket says 172.21.15.23:44614) 2024-03-24T10:38:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bcbfff640 1 -- 172.21.15.23:0/2904029594 learned_addr learned my addr 172.21.15.23:0/2904029594 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bd0a3a640 1 --2- 172.21.15.23:0/2904029594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc1021f0 0x7f2bcc19ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bcbfff640 1 -- 172.21.15.23:0/2904029594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bcc101830 msgr2=0x7f2bcc197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bcbfff640 1 --2- 172.21.15.23:0/2904029594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bcc101830 0x7f2bcc197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bcbfff640 1 -- 172.21.15.23:0/2904029594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc1021f0 msgr2=0x7f2bcc19ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bcbfff640 1 --2- 172.21.15.23:0/2904029594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc1021f0 0x7f2bcc19ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:18.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bcbfff640 1 -- 172.21.15.23:0/2904029594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc4009840 con 0x7f2bcc100640 2024-03-24T10:38:18.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bd0a3a640 1 --2- 172.21.15.23:0/2904029594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc1021f0 0x7f2bcc19ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:18.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bcbfff640 1 --2- 172.21.15.23:0/2904029594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bcc100640 0x7f2bcc1978e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc036040 tx=0x7f2bbc005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:18.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bc97fa640 1 -- 172.21.15.23:0/2904029594 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc004350 con 0x7f2bcc100640 2024-03-24T10:38:18.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bc97fa640 1 -- 172.21.15.23:0/2904029594 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bbc0044b0 con 0x7f2bcc100640 2024-03-24T10:38:18.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.824+0000 7f2bd249d640 1 -- 172.21.15.23:0/2904029594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bbc009560 con 0x7f2bcc100640 2024-03-24T10:38:18.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.825+0000 7f2bc97fa640 1 -- 172.21.15.23:0/2904029594 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc003d30 con 0x7f2bcc100640 2024-03-24T10:38:18.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.825+0000 7f2bd249d640 1 -- 172.21.15.23:0/2904029594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bcc19fd00 con 0x7f2bcc100640 2024-03-24T10:38:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.825+0000 7f2bd249d640 1 -- 172.21.15.23:0/2904029594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bcc10c050 con 0x7f2bcc100640 2024-03-24T10:38:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.826+0000 7f2bc97fa640 1 -- 172.21.15.23:0/2904029594 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f2bbc004df0 con 0x7f2bcc100640 2024-03-24T10:38:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.827+0000 7f2bc97fa640 1 --2- 172.21.15.23:0/2904029594 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2bb40a2150 0x7f2bb40a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.827+0000 7f2bcb7fe640 1 --2- 172.21.15.23:0/2904029594 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2bb40a2150 0x7f2bb40a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.827+0000 7f2bc97fa640 1 -- 172.21.15.23:0/2904029594 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f2bbc0efc20 con 0x7f2bcc100640 2024-03-24T10:38:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.827+0000 7f2bcb7fe640 1 --2- 172.21.15.23:0/2904029594 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2bb40a2150 0x7f2bb40a4610 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc198df0 tx=0x7f2bc0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:18.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:18.831+0000 7f2bc97fa640 1 -- 172.21.15.23:0/2904029594 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2bbc0b8d50 con 0x7f2bcc100640 2024-03-24T10:38:19.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:19.015+0000 7f2bd249d640 1 -- 172.21.15.23:0/2904029594 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2bcc0fde10 con 0x7f2bb40a2150 2024-03-24T10:38:19.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:19.022+0000 7f2bc97fa640 1 -- 172.21.15.23:0/2904029594 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258710 (secure 0 0 0) 0x7f2bcc0fde10 con 0x7f2bb40a2150 2024-03-24T10:38:19.025 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:38:19.026 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-24T10:38:19.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:19.025+0000 7f2bd249d640 1 -- 172.21.15.23:0/2904029594 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2bb40a2150 msgr2=0x7f2bb40a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:19.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:19.025+0000 7f2bd249d640 1 --2- 172.21.15.23:0/2904029594 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2bb40a2150 0x7f2bb40a4610 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc198df0 tx=0x7f2bc0005f90 comp rx=0 tx=0).stop 2024-03-24T10:38:19.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:19.025+0000 7f2bd249d640 1 -- 172.21.15.23:0/2904029594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bcc100640 msgr2=0x7f2bcc1978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:19.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:19.025+0000 7f2bd249d640 1 --2- 172.21.15.23:0/2904029594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bcc100640 0x7f2bcc1978e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc036040 tx=0x7f2bbc005dd0 comp rx=0 tx=0).stop 2024-03-24T10:38:19.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:19.025+0000 7f2bd249d640 1 -- 172.21.15.23:0/2904029594 shutdown_connections 2024-03-24T10:38:19.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:19.025+0000 7f2bd249d640 1 --2- 172.21.15.23:0/2904029594 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2bb40a2150 0x7f2bb40a4610 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:19.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:19.025+0000 7f2bd249d640 1 --2- 172.21.15.23:0/2904029594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc1021f0 0x7f2bcc19ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:19.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:19.025+0000 7f2bd249d640 1 --2- 172.21.15.23:0/2904029594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bcc101830 0x7f2bcc197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:19.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:19.025+0000 7f2bd249d640 1 --2- 172.21.15.23:0/2904029594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bcc100640 0x7f2bcc1978e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:19.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:19.025+0000 7f2bd249d640 1 -- 172.21.15.23:0/2904029594 >> 172.21.15.23:0/2904029594 conn(0x7f2bcc0fbbf0 msgr2=0x7f2bcc0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:19.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:19.025+0000 7f2bd249d640 1 -- 172.21.15.23:0/2904029594 shutdown_connections 2024-03-24T10:38:19.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:19.025+0000 7f2bd249d640 1 -- 172.21.15.23:0/2904029594 wait complete. 2024-03-24T10:38:19.036 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-03-24T10:38:18.794622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602581,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":161,"num_read_kb":139,"num_write":311,"num_write_kb":2986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":366028,"kb_used_data":12712,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575796,"statfs":{"total":1151940427776,"available":1151565615104,"internally_reserved":0,"allocated":13017088,"data_stored":7085909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12213,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":130,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001831"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563024+0000","last_change":"2024-03-24T10:38:10.882385+0000","last_active":"2024-03-24T10:38:12.563024+0000","last_peered":"2024-03-24T10:38:12.563024+0000","last_clean":"2024-03-24T10:38:12.563024+0000","last_became_active":"2024-03-24T10:38:10.882062+0000","last_became_peered":"2024-03-24T10:38:10.882062+0000","last_unstale":"2024-03-24T10:38:12.563024+0000","last_undegraded":"2024-03-24T10:38:12.563024+0000","last_fullsized":"2024-03-24T10:38:12.563024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:25.151062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562813+0000","last_change":"2024-03-24T10:38:10.878001+0000","last_active":"2024-03-24T10:38:12.562813+0000","last_peered":"2024-03-24T10:38:12.562813+0000","last_clean":"2024-03-24T10:38:12.562813+0000","last_became_active":"2024-03-24T10:38:10.877661+0000","last_became_peered":"2024-03-24T10:38:10.877661+0000","last_unstale":"2024-03-24T10:38:12.562813+0000","last_undegraded":"2024-03-24T10:38:12.562813+0000","last_fullsized":"2024-03-24T10:38:12.562813+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:40:19.062969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562806+0000","last_change":"2024-03-24T10:38:10.891809+0000","last_active":"2024-03-24T10:38:12.562806+0000","last_peered":"2024-03-24T10:38:12.562806+0000","last_clean":"2024-03-24T10:38:12.562806+0000","last_became_active":"2024-03-24T10:38:10.890354+0000","last_became_peered":"2024-03-24T10:38:10.890354+0000","last_unstale":"2024-03-24T10:38:12.562806+0000","last_undegraded":"2024-03-24T10:38:12.562806+0000","last_fullsized":"2024-03-24T10:38:12.562806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:59:26.037914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.976421+0000","last_change":"2024-03-24T10:38:10.885097+0000","last_active":"2024-03-24T10:38:11.976421+0000","last_peered":"2024-03-24T10:38:11.976421+0000","last_clean":"2024-03-24T10:38:11.976421+0000","last_became_active":"2024-03-24T10:38:10.884842+0000","last_became_peered":"2024-03-24T10:38:10.884842+0000","last_unstale":"2024-03-24T10:38:11.976421+0000","last_undegraded":"2024-03-24T10:38:11.976421+0000","last_fullsized":"2024-03-24T10:38:11.976421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:39:44.085699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882410+0000","last_change":"2024-03-24T10:38:10.877653+0000","last_active":"2024-03-24T10:38:11.882410+0000","last_peered":"2024-03-24T10:38:11.882410+0000","last_clean":"2024-03-24T10:38:11.882410+0000","last_became_active":"2024-03-24T10:38:10.877249+0000","last_became_peered":"2024-03-24T10:38:10.877249+0000","last_unstale":"2024-03-24T10:38:11.882410+0000","last_undegraded":"2024-03-24T10:38:11.882410+0000","last_fullsized":"2024-03-24T10:38:11.882410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:17.504660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562542+0000","last_change":"2024-03-24T10:38:10.876584+0000","last_active":"2024-03-24T10:38:12.562542+0000","last_peered":"2024-03-24T10:38:12.562542+0000","last_clean":"2024-03-24T10:38:12.562542+0000","last_became_active":"2024-03-24T10:38:10.876173+0000","last_became_peered":"2024-03-24T10:38:10.876173+0000","last_unstale":"2024-03-24T10:38:12.562542+0000","last_undegraded":"2024-03-24T10:38:12.562542+0000","last_fullsized":"2024-03-24T10:38:12.562542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:33.899155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872500+0000","last_change":"2024-03-24T10:38:10.874206+0000","last_active":"2024-03-24T10:38:11.872500+0000","last_peered":"2024-03-24T10:38:11.872500+0000","last_clean":"2024-03-24T10:38:11.872500+0000","last_became_active":"2024-03-24T10:38:10.873779+0000","last_became_peered":"2024-03-24T10:38:10.873779+0000","last_unstale":"2024-03-24T10:38:11.872500+0000","last_undegraded":"2024-03-24T10:38:11.872500+0000","last_fullsized":"2024-03-24T10:38:11.872500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:08:19.928998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882170+0000","last_change":"2024-03-24T10:38:10.884219+0000","last_active":"2024-03-24T10:38:11.882170+0000","last_peered":"2024-03-24T10:38:11.882170+0000","last_clean":"2024-03-24T10:38:11.882170+0000","last_became_active":"2024-03-24T10:38:10.883863+0000","last_became_peered":"2024-03-24T10:38:10.883863+0000","last_unstale":"2024-03-24T10:38:11.882170+0000","last_undegraded":"2024-03-24T10:38:11.882170+0000","last_fullsized":"2024-03-24T10:38:11.882170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:26:34.932083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562469+0000","last_change":"2024-03-24T10:38:10.881297+0000","last_active":"2024-03-24T10:38:12.562469+0000","last_peered":"2024-03-24T10:38:12.562469+0000","last_clean":"2024-03-24T10:38:12.562469+0000","last_became_active":"2024-03-24T10:38:10.879871+0000","last_became_peered":"2024-03-24T10:38:10.879871+0000","last_unstale":"2024-03-24T10:38:12.562469+0000","last_undegraded":"2024-03-24T10:38:12.562469+0000","last_fullsized":"2024-03-24T10:38:12.562469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:05.132788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562873+0000","last_change":"2024-03-24T10:38:10.876648+0000","last_active":"2024-03-24T10:38:12.562873+0000","last_peered":"2024-03-24T10:38:12.562873+0000","last_clean":"2024-03-24T10:38:12.562873+0000","last_became_active":"2024-03-24T10:38:10.876113+0000","last_became_peered":"2024-03-24T10:38:10.876113+0000","last_unstale":"2024-03-24T10:38:12.562873+0000","last_undegraded":"2024-03-24T10:38:12.562873+0000","last_fullsized":"2024-03-24T10:38:12.562873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:29:18.282797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872901+0000","last_change":"2024-03-24T10:38:10.875836+0000","last_active":"2024-03-24T10:38:11.872901+0000","last_peered":"2024-03-24T10:38:11.872901+0000","last_clean":"2024-03-24T10:38:11.872901+0000","last_became_active":"2024-03-24T10:38:10.875522+0000","last_became_peered":"2024-03-24T10:38:10.875522+0000","last_unstale":"2024-03-24T10:38:11.872901+0000","last_undegraded":"2024-03-24T10:38:11.872901+0000","last_fullsized":"2024-03-24T10:38:11.872901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:41.282123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.976376+0000","last_change":"2024-03-24T10:38:10.884567+0000","last_active":"2024-03-24T10:38:11.976376+0000","last_peered":"2024-03-24T10:38:11.976376+0000","last_clean":"2024-03-24T10:38:11.976376+0000","last_became_active":"2024-03-24T10:38:10.884195+0000","last_became_peered":"2024-03-24T10:38:10.884195+0000","last_unstale":"2024-03-24T10:38:11.976376+0000","last_undegraded":"2024-03-24T10:38:11.976376+0000","last_fullsized":"2024-03-24T10:38:11.976376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:29.286631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.870005+0000","last_change":"2024-03-24T10:38:10.873464+0000","last_active":"2024-03-24T10:38:11.870005+0000","last_peered":"2024-03-24T10:38:11.870005+0000","last_clean":"2024-03-24T10:38:11.870005+0000","last_became_active":"2024-03-24T10:38:10.873128+0000","last_became_peered":"2024-03-24T10:38:10.873128+0000","last_unstale":"2024-03-24T10:38:11.870005+0000","last_undegraded":"2024-03-24T10:38:11.870005+0000","last_fullsized":"2024-03-24T10:38:11.870005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:21.857942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562588+0000","last_change":"2024-03-24T10:38:10.879942+0000","last_active":"2024-03-24T10:38:12.562588+0000","last_peered":"2024-03-24T10:38:12.562588+0000","last_clean":"2024-03-24T10:38:12.562588+0000","last_became_active":"2024-03-24T10:38:10.879396+0000","last_became_peered":"2024-03-24T10:38:10.879396+0000","last_unstale":"2024-03-24T10:38:12.562588+0000","last_undegraded":"2024-03-24T10:38:12.562588+0000","last_fullsized":"2024-03-24T10:38:12.562588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:41.477988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562540+0000","last_change":"2024-03-24T10:38:10.886667+0000","last_active":"2024-03-24T10:38:12.562540+0000","last_peered":"2024-03-24T10:38:12.562540+0000","last_clean":"2024-03-24T10:38:12.562540+0000","last_became_active":"2024-03-24T10:38:10.886293+0000","last_became_peered":"2024-03-24T10:38:10.886293+0000","last_unstale":"2024-03-24T10:38:12.562540+0000","last_undegraded":"2024-03-24T10:38:12.562540+0000","last_fullsized":"2024-03-24T10:38:12.562540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:43:43.414573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.561883+0000","last_change":"2024-03-24T10:38:10.883671+0000","last_active":"2024-03-24T10:38:12.561883+0000","last_peered":"2024-03-24T10:38:12.561883+0000","last_clean":"2024-03-24T10:38:12.561883+0000","last_became_active":"2024-03-24T10:38:10.883013+0000","last_became_peered":"2024-03-24T10:38:10.883013+0000","last_unstale":"2024-03-24T10:38:12.561883+0000","last_undegraded":"2024-03-24T10:38:12.561883+0000","last_fullsized":"2024-03-24T10:38:12.561883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:33.936543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563108+0000","last_change":"2024-03-24T10:38:10.887286+0000","last_active":"2024-03-24T10:38:12.563108+0000","last_peered":"2024-03-24T10:38:12.563108+0000","last_clean":"2024-03-24T10:38:12.563108+0000","last_became_active":"2024-03-24T10:38:10.886918+0000","last_became_peered":"2024-03-24T10:38:10.886918+0000","last_unstale":"2024-03-24T10:38:12.563108+0000","last_undegraded":"2024-03-24T10:38:12.563108+0000","last_fullsized":"2024-03-24T10:38:12.563108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:40.266589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.880480+0000","last_change":"2024-03-24T10:38:10.874476+0000","last_active":"2024-03-24T10:38:11.880480+0000","last_peered":"2024-03-24T10:38:11.880480+0000","last_clean":"2024-03-24T10:38:11.880480+0000","last_became_active":"2024-03-24T10:38:10.874123+0000","last_became_peered":"2024-03-24T10:38:10.874123+0000","last_unstale":"2024-03-24T10:38:11.880480+0000","last_undegraded":"2024-03-24T10:38:11.880480+0000","last_fullsized":"2024-03-24T10:38:11.880480+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:29.759320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563431+0000","last_change":"2024-03-24T10:38:10.878758+0000","last_active":"2024-03-24T10:38:12.563431+0000","last_peered":"2024-03-24T10:38:12.563431+0000","last_clean":"2024-03-24T10:38:12.563431+0000","last_became_active":"2024-03-24T10:38:10.878424+0000","last_became_peered":"2024-03-24T10:38:10.878424+0000","last_unstale":"2024-03-24T10:38:12.563431+0000","last_undegraded":"2024-03-24T10:38:12.563431+0000","last_fullsized":"2024-03-24T10:38:12.563431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:15:19.273108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:14.908982+0000","last_change":"2024-03-24T10:38:09.861751+0000","last_active":"2024-03-24T10:38:14.908982+0000","last_peered":"2024-03-24T10:38:14.908982+0000","last_clean":"2024-03-24T10:38:14.908982+0000","last_became_active":"2024-03-24T10:38:09.860889+0000","last_became_peered":"2024-03-24T10:38:09.860889+0000","last_unstale":"2024-03-24T10:38:14.908982+0000","last_undegraded":"2024-03-24T10:38:14.908982+0000","last_fullsized":"2024-03-24T10:38:14.908982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:22:53.447005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882538+0000","last_change":"2024-03-24T10:38:10.885198+0000","last_active":"2024-03-24T10:38:11.882538+0000","last_peered":"2024-03-24T10:38:11.882538+0000","last_clean":"2024-03-24T10:38:11.882538+0000","last_became_active":"2024-03-24T10:38:10.884123+0000","last_became_peered":"2024-03-24T10:38:10.884123+0000","last_unstale":"2024-03-24T10:38:11.882538+0000","last_undegraded":"2024-03-24T10:38:11.882538+0000","last_fullsized":"2024-03-24T10:38:11.882538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:37:12.510671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882500+0000","last_change":"2024-03-24T10:38:09.866223+0000","last_active":"2024-03-24T10:38:11.882500+0000","last_peered":"2024-03-24T10:38:11.882500+0000","last_clean":"2024-03-24T10:38:11.882500+0000","last_became_active":"2024-03-24T10:38:09.865711+0000","last_became_peered":"2024-03-24T10:38:09.865711+0000","last_unstale":"2024-03-24T10:38:11.882500+0000","last_undegraded":"2024-03-24T10:38:11.882500+0000","last_fullsized":"2024-03-24T10:38:11.882500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:26:08.380749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563059+0000","last_change":"2024-03-24T10:38:10.890557+0000","last_active":"2024-03-24T10:38:12.563059+0000","last_peered":"2024-03-24T10:38:12.563059+0000","last_clean":"2024-03-24T10:38:12.563059+0000","last_became_active":"2024-03-24T10:38:10.889883+0000","last_became_peered":"2024-03-24T10:38:10.889883+0000","last_unstale":"2024-03-24T10:38:12.563059+0000","last_undegraded":"2024-03-24T10:38:12.563059+0000","last_fullsized":"2024-03-24T10:38:12.563059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:34.731585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:14.906550+0000","last_change":"2024-03-24T10:38:09.870253+0000","last_active":"2024-03-24T10:38:14.906550+0000","last_peered":"2024-03-24T10:38:14.906550+0000","last_clean":"2024-03-24T10:38:14.906550+0000","last_became_active":"2024-03-24T10:38:09.869932+0000","last_became_peered":"2024-03-24T10:38:09.869932+0000","last_unstale":"2024-03-24T10:38:14.906550+0000","last_undegraded":"2024-03-24T10:38:14.906550+0000","last_fullsized":"2024-03-24T10:38:14.906550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:38:43.626390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562157+0000","last_change":"2024-03-24T10:38:10.881705+0000","last_active":"2024-03-24T10:38:12.562157+0000","last_peered":"2024-03-24T10:38:12.562157+0000","last_clean":"2024-03-24T10:38:12.562157+0000","last_became_active":"2024-03-24T10:38:10.881323+0000","last_became_peered":"2024-03-24T10:38:10.881323+0000","last_unstale":"2024-03-24T10:38:12.562157+0000","last_undegraded":"2024-03-24T10:38:12.562157+0000","last_fullsized":"2024-03-24T10:38:12.562157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:25.861184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882540+0000","last_change":"2024-03-24T10:38:09.870481+0000","last_active":"2024-03-24T10:38:11.882540+0000","last_peered":"2024-03-24T10:38:11.882540+0000","last_clean":"2024-03-24T10:38:11.882540+0000","last_became_active":"2024-03-24T10:38:09.870117+0000","last_became_peered":"2024-03-24T10:38:09.870117+0000","last_unstale":"2024-03-24T10:38:11.882540+0000","last_undegraded":"2024-03-24T10:38:11.882540+0000","last_fullsized":"2024-03-24T10:38:11.882540+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:37:51.871068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.870544+0000","last_change":"2024-03-24T10:38:10.884196+0000","last_active":"2024-03-24T10:38:11.870544+0000","last_peered":"2024-03-24T10:38:11.870544+0000","last_clean":"2024-03-24T10:38:11.870544+0000","last_became_active":"2024-03-24T10:38:10.882576+0000","last_became_peered":"2024-03-24T10:38:10.882576+0000","last_unstale":"2024-03-24T10:38:11.870544+0000","last_undegraded":"2024-03-24T10:38:11.870544+0000","last_fullsized":"2024-03-24T10:38:11.870544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:58:45.698875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:15.917729+0000","last_change":"2024-03-24T10:38:09.864160+0000","last_active":"2024-03-24T10:38:15.917729+0000","last_peered":"2024-03-24T10:38:15.917729+0000","last_clean":"2024-03-24T10:38:15.917729+0000","last_became_active":"2024-03-24T10:38:09.862066+0000","last_became_peered":"2024-03-24T10:38:09.862066+0000","last_unstale":"2024-03-24T10:38:15.917729+0000","last_undegraded":"2024-03-24T10:38:15.917729+0000","last_fullsized":"2024-03-24T10:38:15.917729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:59.003931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882427+0000","last_change":"2024-03-24T10:38:10.874433+0000","last_active":"2024-03-24T10:38:11.882427+0000","last_peered":"2024-03-24T10:38:11.882427+0000","last_clean":"2024-03-24T10:38:11.882427+0000","last_became_active":"2024-03-24T10:38:10.874090+0000","last_became_peered":"2024-03-24T10:38:10.874090+0000","last_unstale":"2024-03-24T10:38:11.882427+0000","last_undegraded":"2024-03-24T10:38:11.882427+0000","last_fullsized":"2024-03-24T10:38:11.882427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:25.979419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:14.907034+0000","last_change":"2024-03-24T10:38:09.863028+0000","last_active":"2024-03-24T10:38:14.907034+0000","last_peered":"2024-03-24T10:38:14.907034+0000","last_clean":"2024-03-24T10:38:14.907034+0000","last_became_active":"2024-03-24T10:38:09.862682+0000","last_became_peered":"2024-03-24T10:38:09.862682+0000","last_unstale":"2024-03-24T10:38:14.907034+0000","last_undegraded":"2024-03-24T10:38:14.907034+0000","last_fullsized":"2024-03-24T10:38:14.907034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:14.962115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.870486+0000","last_change":"2024-03-24T10:38:10.885950+0000","last_active":"2024-03-24T10:38:11.870486+0000","last_peered":"2024-03-24T10:38:11.870486+0000","last_clean":"2024-03-24T10:38:11.870486+0000","last_became_active":"2024-03-24T10:38:10.885685+0000","last_became_peered":"2024-03-24T10:38:10.885685+0000","last_unstale":"2024-03-24T10:38:11.870486+0000","last_undegraded":"2024-03-24T10:38:11.870486+0000","last_fullsized":"2024-03-24T10:38:11.870486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:42:09.912743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:14.908918+0000","last_change":"2024-03-24T10:38:09.860376+0000","last_active":"2024-03-24T10:38:14.908918+0000","last_peered":"2024-03-24T10:38:14.908918+0000","last_clean":"2024-03-24T10:38:14.908918+0000","last_became_active":"2024-03-24T10:38:09.860062+0000","last_became_peered":"2024-03-24T10:38:09.860062+0000","last_unstale":"2024-03-24T10:38:14.908918+0000","last_undegraded":"2024-03-24T10:38:14.908918+0000","last_fullsized":"2024-03-24T10:38:14.908918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:33.387953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882144+0000","last_change":"2024-03-24T10:38:10.883843+0000","last_active":"2024-03-24T10:38:11.882144+0000","last_peered":"2024-03-24T10:38:11.882144+0000","last_clean":"2024-03-24T10:38:11.882144+0000","last_became_active":"2024-03-24T10:38:10.883170+0000","last_became_peered":"2024-03-24T10:38:10.883170+0000","last_unstale":"2024-03-24T10:38:11.882144+0000","last_undegraded":"2024-03-24T10:38:11.882144+0000","last_fullsized":"2024-03-24T10:38:11.882144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:28:16.798546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869908+0000","last_change":"2024-03-24T10:38:09.862299+0000","last_active":"2024-03-24T10:38:11.869908+0000","last_peered":"2024-03-24T10:38:11.869908+0000","last_clean":"2024-03-24T10:38:11.869908+0000","last_became_active":"2024-03-24T10:38:09.861895+0000","last_became_peered":"2024-03-24T10:38:09.861895+0000","last_unstale":"2024-03-24T10:38:11.869908+0000","last_undegraded":"2024-03-24T10:38:11.869908+0000","last_fullsized":"2024-03-24T10:38:11.869908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:13:19.869511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.880311+0000","last_change":"2024-03-24T10:38:10.880507+0000","last_active":"2024-03-24T10:38:11.880311+0000","last_peered":"2024-03-24T10:38:11.880311+0000","last_clean":"2024-03-24T10:38:11.880311+0000","last_became_active":"2024-03-24T10:38:10.879790+0000","last_became_peered":"2024-03-24T10:38:10.879790+0000","last_unstale":"2024-03-24T10:38:11.880311+0000","last_undegraded":"2024-03-24T10:38:11.880311+0000","last_fullsized":"2024-03-24T10:38:11.880311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:11:02.650420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.897213+0000","last_change":"2024-03-24T10:38:09.860186+0000","last_active":"2024-03-24T10:38:13.897213+0000","last_peered":"2024-03-24T10:38:13.897213+0000","last_clean":"2024-03-24T10:38:13.897213+0000","last_became_active":"2024-03-24T10:38:09.859834+0000","last_became_peered":"2024-03-24T10:38:09.859834+0000","last_unstale":"2024-03-24T10:38:13.897213+0000","last_undegraded":"2024-03-24T10:38:13.897213+0000","last_fullsized":"2024-03-24T10:38:13.897213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:33:48.029053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562396+0000","last_change":"2024-03-24T10:38:10.875634+0000","last_active":"2024-03-24T10:38:12.562396+0000","last_peered":"2024-03-24T10:38:12.562396+0000","last_clean":"2024-03-24T10:38:12.562396+0000","last_became_active":"2024-03-24T10:38:10.875101+0000","last_became_peered":"2024-03-24T10:38:10.875101+0000","last_unstale":"2024-03-24T10:38:12.562396+0000","last_undegraded":"2024-03-24T10:38:12.562396+0000","last_fullsized":"2024-03-24T10:38:12.562396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:32:58.165087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.895392+0000","last_change":"2024-03-24T10:38:09.860412+0000","last_active":"2024-03-24T10:38:13.895392+0000","last_peered":"2024-03-24T10:38:13.895392+0000","last_clean":"2024-03-24T10:38:13.895392+0000","last_became_active":"2024-03-24T10:38:09.860084+0000","last_became_peered":"2024-03-24T10:38:09.860084+0000","last_unstale":"2024-03-24T10:38:13.895392+0000","last_undegraded":"2024-03-24T10:38:13.895392+0000","last_fullsized":"2024-03-24T10:38:13.895392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:44.585482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563060+0000","last_change":"2024-03-24T10:38:10.885489+0000","last_active":"2024-03-24T10:38:12.563060+0000","last_peered":"2024-03-24T10:38:12.563060+0000","last_clean":"2024-03-24T10:38:12.563060+0000","last_became_active":"2024-03-24T10:38:10.884810+0000","last_became_peered":"2024-03-24T10:38:10.884810+0000","last_unstale":"2024-03-24T10:38:12.563060+0000","last_undegraded":"2024-03-24T10:38:12.563060+0000","last_fullsized":"2024-03-24T10:38:12.563060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:12:15.922850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.897451+0000","last_change":"2024-03-24T10:38:09.865191+0000","last_active":"2024-03-24T10:38:13.897451+0000","last_peered":"2024-03-24T10:38:13.897451+0000","last_clean":"2024-03-24T10:38:13.897451+0000","last_became_active":"2024-03-24T10:38:09.864470+0000","last_became_peered":"2024-03-24T10:38:09.864470+0000","last_unstale":"2024-03-24T10:38:13.897451+0000","last_undegraded":"2024-03-24T10:38:13.897451+0000","last_fullsized":"2024-03-24T10:38:13.897451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:45:00.266470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562279+0000","last_change":"2024-03-24T10:38:10.889668+0000","last_active":"2024-03-24T10:38:12.562279+0000","last_peered":"2024-03-24T10:38:12.562279+0000","last_clean":"2024-03-24T10:38:12.562279+0000","last_became_active":"2024-03-24T10:38:10.889364+0000","last_became_peered":"2024-03-24T10:38:10.889364+0000","last_unstale":"2024-03-24T10:38:12.562279+0000","last_undegraded":"2024-03-24T10:38:12.562279+0000","last_fullsized":"2024-03-24T10:38:12.562279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:57.702719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'11","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:16.269532+0000","last_change":"2024-03-24T10:38:09.861809+0000","last_active":"2024-03-24T10:38:16.269532+0000","last_peered":"2024-03-24T10:38:16.269532+0000","last_clean":"2024-03-24T10:38:16.269532+0000","last_became_active":"2024-03-24T10:38:09.861444+0000","last_became_peered":"2024-03-24T10:38:09.861444+0000","last_unstale":"2024-03-24T10:38:16.269532+0000","last_undegraded":"2024-03-24T10:38:16.269532+0000","last_fullsized":"2024-03-24T10:38:16.269532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:19.652188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8212,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882691+0000","last_change":"2024-03-24T10:38:10.883090+0000","last_active":"2024-03-24T10:38:11.882691+0000","last_peered":"2024-03-24T10:38:11.882691+0000","last_clean":"2024-03-24T10:38:11.882691+0000","last_became_active":"2024-03-24T10:38:10.880507+0000","last_became_peered":"2024-03-24T10:38:10.880507+0000","last_unstale":"2024-03-24T10:38:11.882691+0000","last_undegraded":"2024-03-24T10:38:11.882691+0000","last_fullsized":"2024-03-24T10:38:11.882691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:01.422699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:14.907607+0000","last_change":"2024-03-24T10:38:09.868072+0000","last_active":"2024-03-24T10:38:14.907607+0000","last_peered":"2024-03-24T10:38:14.907607+0000","last_clean":"2024-03-24T10:38:14.907607+0000","last_became_active":"2024-03-24T10:38:09.867704+0000","last_became_peered":"2024-03-24T10:38:09.867704+0000","last_unstale":"2024-03-24T10:38:14.907607+0000","last_undegraded":"2024-03-24T10:38:14.907607+0000","last_fullsized":"2024-03-24T10:38:14.907607+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:09:28.726969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869956+0000","last_change":"2024-03-24T10:38:10.881882+0000","last_active":"2024-03-24T10:38:11.869956+0000","last_peered":"2024-03-24T10:38:11.869956+0000","last_clean":"2024-03-24T10:38:11.869956+0000","last_became_active":"2024-03-24T10:38:10.881495+0000","last_became_peered":"2024-03-24T10:38:10.881495+0000","last_unstale":"2024-03-24T10:38:11.869956+0000","last_undegraded":"2024-03-24T10:38:11.869956+0000","last_fullsized":"2024-03-24T10:38:11.869956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:31:43.392952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.897503+0000","last_change":"2024-03-24T10:38:09.867265+0000","last_active":"2024-03-24T10:38:13.897503+0000","last_peered":"2024-03-24T10:38:13.897503+0000","last_clean":"2024-03-24T10:38:13.897503+0000","last_became_active":"2024-03-24T10:38:09.866669+0000","last_became_peered":"2024-03-24T10:38:09.866669+0000","last_unstale":"2024-03-24T10:38:13.897503+0000","last_undegraded":"2024-03-24T10:38:13.897503+0000","last_fullsized":"2024-03-24T10:38:13.897503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:28.230967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872911+0000","last_change":"2024-03-24T10:38:10.886634+0000","last_active":"2024-03-24T10:38:11.872911+0000","last_peered":"2024-03-24T10:38:11.872911+0000","last_clean":"2024-03-24T10:38:11.872911+0000","last_became_active":"2024-03-24T10:38:10.886207+0000","last_became_peered":"2024-03-24T10:38:10.886207+0000","last_unstale":"2024-03-24T10:38:11.872911+0000","last_undegraded":"2024-03-24T10:38:11.872911+0000","last_fullsized":"2024-03-24T10:38:11.872911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:43:41.601693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:16.920950+0000","last_change":"2024-03-24T10:38:09.863262+0000","last_active":"2024-03-24T10:38:16.920950+0000","last_peered":"2024-03-24T10:38:16.920950+0000","last_clean":"2024-03-24T10:38:16.920950+0000","last_became_active":"2024-03-24T10:38:09.862665+0000","last_became_peered":"2024-03-24T10:38:09.862665+0000","last_unstale":"2024-03-24T10:38:16.920950+0000","last_undegraded":"2024-03-24T10:38:16.920950+0000","last_fullsized":"2024-03-24T10:38:16.920950+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:12:29.117592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872423+0000","last_change":"2024-03-24T10:38:10.880793+0000","last_active":"2024-03-24T10:38:11.872423+0000","last_peered":"2024-03-24T10:38:11.872423+0000","last_clean":"2024-03-24T10:38:11.872423+0000","last_became_active":"2024-03-24T10:38:10.880449+0000","last_became_peered":"2024-03-24T10:38:10.880449+0000","last_unstale":"2024-03-24T10:38:11.872423+0000","last_undegraded":"2024-03-24T10:38:11.872423+0000","last_fullsized":"2024-03-24T10:38:11.872423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:32:57.184014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:14.909457+0000","last_change":"2024-03-24T10:38:09.867694+0000","last_active":"2024-03-24T10:38:14.909457+0000","last_peered":"2024-03-24T10:38:14.909457+0000","last_clean":"2024-03-24T10:38:14.909457+0000","last_became_active":"2024-03-24T10:38:09.867345+0000","last_became_peered":"2024-03-24T10:38:09.867345+0000","last_unstale":"2024-03-24T10:38:14.909457+0000","last_undegraded":"2024-03-24T10:38:14.909457+0000","last_fullsized":"2024-03-24T10:38:14.909457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:39.436352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.870014+0000","last_change":"2024-03-24T10:38:10.881952+0000","last_active":"2024-03-24T10:38:11.870014+0000","last_peered":"2024-03-24T10:38:11.870014+0000","last_clean":"2024-03-24T10:38:11.870014+0000","last_became_active":"2024-03-24T10:38:10.881628+0000","last_became_peered":"2024-03-24T10:38:10.881628+0000","last_unstale":"2024-03-24T10:38:11.870014+0000","last_undegraded":"2024-03-24T10:38:11.870014+0000","last_fullsized":"2024-03-24T10:38:11.870014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:24.777793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882682+0000","last_change":"2024-03-24T10:38:09.868572+0000","last_active":"2024-03-24T10:38:11.882682+0000","last_peered":"2024-03-24T10:38:11.882682+0000","last_clean":"2024-03-24T10:38:11.882682+0000","last_became_active":"2024-03-24T10:38:09.867702+0000","last_became_peered":"2024-03-24T10:38:09.867702+0000","last_unstale":"2024-03-24T10:38:11.882682+0000","last_undegraded":"2024-03-24T10:38:11.882682+0000","last_fullsized":"2024-03-24T10:38:11.882682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:27.890114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562843+0000","last_change":"2024-03-24T10:38:10.875342+0000","last_active":"2024-03-24T10:38:12.562843+0000","last_peered":"2024-03-24T10:38:12.562843+0000","last_clean":"2024-03-24T10:38:12.562843+0000","last_became_active":"2024-03-24T10:38:10.874933+0000","last_became_peered":"2024-03-24T10:38:10.874933+0000","last_unstale":"2024-03-24T10:38:12.562843+0000","last_undegraded":"2024-03-24T10:38:12.562843+0000","last_fullsized":"2024-03-24T10:38:12.562843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:56:09.139478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.880371+0000","last_change":"2024-03-24T10:38:09.857867+0000","last_active":"2024-03-24T10:38:11.880371+0000","last_peered":"2024-03-24T10:38:11.880371+0000","last_clean":"2024-03-24T10:38:11.880371+0000","last_became_active":"2024-03-24T10:38:09.857525+0000","last_became_peered":"2024-03-24T10:38:09.857525+0000","last_unstale":"2024-03-24T10:38:11.880371+0000","last_undegraded":"2024-03-24T10:38:11.880371+0000","last_fullsized":"2024-03-24T10:38:11.880371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:37.320245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562675+0000","last_change":"2024-03-24T10:38:10.878980+0000","last_active":"2024-03-24T10:38:12.562675+0000","last_peered":"2024-03-24T10:38:12.562675+0000","last_clean":"2024-03-24T10:38:12.562675+0000","last_became_active":"2024-03-24T10:38:10.878648+0000","last_became_peered":"2024-03-24T10:38:10.878648+0000","last_unstale":"2024-03-24T10:38:12.562675+0000","last_undegraded":"2024-03-24T10:38:12.562675+0000","last_fullsized":"2024-03-24T10:38:12.562675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:49.988403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872661+0000","last_change":"2024-03-24T10:38:09.863457+0000","last_active":"2024-03-24T10:38:11.872661+0000","last_peered":"2024-03-24T10:38:11.872661+0000","last_clean":"2024-03-24T10:38:11.872661+0000","last_became_active":"2024-03-24T10:38:09.863130+0000","last_became_peered":"2024-03-24T10:38:09.863130+0000","last_unstale":"2024-03-24T10:38:11.872661+0000","last_undegraded":"2024-03-24T10:38:11.872661+0000","last_fullsized":"2024-03-24T10:38:11.872661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:16.528703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562879+0000","last_change":"2024-03-24T10:38:10.887014+0000","last_active":"2024-03-24T10:38:12.562879+0000","last_peered":"2024-03-24T10:38:12.562879+0000","last_clean":"2024-03-24T10:38:12.562879+0000","last_became_active":"2024-03-24T10:38:10.886698+0000","last_became_peered":"2024-03-24T10:38:10.886698+0000","last_unstale":"2024-03-24T10:38:12.562879+0000","last_undegraded":"2024-03-24T10:38:12.562879+0000","last_fullsized":"2024-03-24T10:38:12.562879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:47:08.571886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.870336+0000","last_change":"2024-03-24T10:38:10.874811+0000","last_active":"2024-03-24T10:38:11.870336+0000","last_peered":"2024-03-24T10:38:11.870336+0000","last_clean":"2024-03-24T10:38:11.870336+0000","last_became_active":"2024-03-24T10:38:10.874483+0000","last_became_peered":"2024-03-24T10:38:10.874483+0000","last_unstale":"2024-03-24T10:38:11.870336+0000","last_undegraded":"2024-03-24T10:38:11.870336+0000","last_fullsized":"2024-03-24T10:38:11.870336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:57.382174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:15.917509+0000","last_change":"2024-03-24T10:38:09.865614+0000","last_active":"2024-03-24T10:38:15.917509+0000","last_peered":"2024-03-24T10:38:15.917509+0000","last_clean":"2024-03-24T10:38:15.917509+0000","last_became_active":"2024-03-24T10:38:09.865317+0000","last_became_peered":"2024-03-24T10:38:09.865317+0000","last_unstale":"2024-03-24T10:38:15.917509+0000","last_undegraded":"2024-03-24T10:38:15.917509+0000","last_fullsized":"2024-03-24T10:38:15.917509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:21:14.968552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882577+0000","last_change":"2024-03-24T10:38:10.884847+0000","last_active":"2024-03-24T10:38:11.882577+0000","last_peered":"2024-03-24T10:38:11.882577+0000","last_clean":"2024-03-24T10:38:11.882577+0000","last_became_active":"2024-03-24T10:38:10.884526+0000","last_became_peered":"2024-03-24T10:38:10.884526+0000","last_unstale":"2024-03-24T10:38:11.882577+0000","last_undegraded":"2024-03-24T10:38:11.882577+0000","last_fullsized":"2024-03-24T10:38:11.882577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:01:49.364083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:15.915524+0000","last_change":"2024-03-24T10:38:09.858893+0000","last_active":"2024-03-24T10:38:15.915524+0000","last_peered":"2024-03-24T10:38:15.915524+0000","last_clean":"2024-03-24T10:38:15.915524+0000","last_became_active":"2024-03-24T10:38:09.858474+0000","last_became_peered":"2024-03-24T10:38:09.858474+0000","last_unstale":"2024-03-24T10:38:15.915524+0000","last_undegraded":"2024-03-24T10:38:15.915524+0000","last_fullsized":"2024-03-24T10:38:15.915524+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:03:36.709498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872412+0000","last_change":"2024-03-24T10:36:59.100728+0000","last_active":"2024-03-24T10:38:11.872412+0000","last_peered":"2024-03-24T10:38:11.872412+0000","last_clean":"2024-03-24T10:38:11.872412+0000","last_became_active":"2024-03-24T10:36:59.100159+0000","last_became_peered":"2024-03-24T10:36:59.100159+0000","last_unstale":"2024-03-24T10:38:11.872412+0000","last_undegraded":"2024-03-24T10:38:11.872412+0000","last_fullsized":"2024-03-24T10:38:11.872412+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:39.429553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:39.429553+0000","last_clean_scrub_stamp":"2024-03-24T10:33:39.429553+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:21:01.470585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869678+0000","last_change":"2024-03-24T10:38:10.885804+0000","last_active":"2024-03-24T10:38:11.869678+0000","last_peered":"2024-03-24T10:38:11.869678+0000","last_clean":"2024-03-24T10:38:11.869678+0000","last_became_active":"2024-03-24T10:38:10.885447+0000","last_became_peered":"2024-03-24T10:38:10.885447+0000","last_unstale":"2024-03-24T10:38:11.869678+0000","last_undegraded":"2024-03-24T10:38:11.869678+0000","last_fullsized":"2024-03-24T10:38:11.869678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:57.103205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:14.907674+0000","last_change":"2024-03-24T10:38:09.863187+0000","last_active":"2024-03-24T10:38:14.907674+0000","last_peered":"2024-03-24T10:38:14.907674+0000","last_clean":"2024-03-24T10:38:14.907674+0000","last_became_active":"2024-03-24T10:38:09.862859+0000","last_became_peered":"2024-03-24T10:38:09.862859+0000","last_unstale":"2024-03-24T10:38:14.907674+0000","last_undegraded":"2024-03-24T10:38:14.907674+0000","last_fullsized":"2024-03-24T10:38:14.907674+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:21:02.707127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882700+0000","last_change":"2024-03-24T10:38:09.867899+0000","last_active":"2024-03-24T10:38:11.882700+0000","last_peered":"2024-03-24T10:38:11.882700+0000","last_clean":"2024-03-24T10:38:11.882700+0000","last_became_active":"2024-03-24T10:38:09.867563+0000","last_became_peered":"2024-03-24T10:38:09.867563+0000","last_unstale":"2024-03-24T10:38:11.882700+0000","last_undegraded":"2024-03-24T10:38:11.882700+0000","last_fullsized":"2024-03-24T10:38:11.882700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:51.067981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869837+0000","last_change":"2024-03-24T10:38:10.870825+0000","last_active":"2024-03-24T10:38:11.869837+0000","last_peered":"2024-03-24T10:38:11.869837+0000","last_clean":"2024-03-24T10:38:11.869837+0000","last_became_active":"2024-03-24T10:38:10.870491+0000","last_became_peered":"2024-03-24T10:38:10.870491+0000","last_unstale":"2024-03-24T10:38:11.869837+0000","last_undegraded":"2024-03-24T10:38:11.869837+0000","last_fullsized":"2024-03-24T10:38:11.869837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:30:34.091299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:14.908620+0000","last_change":"2024-03-24T10:38:09.863724+0000","last_active":"2024-03-24T10:38:14.908620+0000","last_peered":"2024-03-24T10:38:14.908620+0000","last_clean":"2024-03-24T10:38:14.908620+0000","last_became_active":"2024-03-24T10:38:09.862900+0000","last_became_peered":"2024-03-24T10:38:09.862900+0000","last_unstale":"2024-03-24T10:38:14.908620+0000","last_undegraded":"2024-03-24T10:38:14.908620+0000","last_fullsized":"2024-03-24T10:38:14.908620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:21.646074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882543+0000","last_change":"2024-03-24T10:38:10.884661+0000","last_active":"2024-03-24T10:38:11.882543+0000","last_peered":"2024-03-24T10:38:11.882543+0000","last_clean":"2024-03-24T10:38:11.882543+0000","last_became_active":"2024-03-24T10:38:10.882901+0000","last_became_peered":"2024-03-24T10:38:10.882901+0000","last_unstale":"2024-03-24T10:38:11.882543+0000","last_undegraded":"2024-03-24T10:38:11.882543+0000","last_fullsized":"2024-03-24T10:38:11.882543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:23:46.366567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.896003+0000","last_change":"2024-03-24T10:38:09.862122+0000","last_active":"2024-03-24T10:38:13.896003+0000","last_peered":"2024-03-24T10:38:13.896003+0000","last_clean":"2024-03-24T10:38:13.896003+0000","last_became_active":"2024-03-24T10:38:09.861765+0000","last_became_peered":"2024-03-24T10:38:09.861765+0000","last_unstale":"2024-03-24T10:38:13.896003+0000","last_undegraded":"2024-03-24T10:38:13.896003+0000","last_fullsized":"2024-03-24T10:38:13.896003+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:40:15.148349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562972+0000","last_change":"2024-03-24T10:38:10.886797+0000","last_active":"2024-03-24T10:38:12.562972+0000","last_peered":"2024-03-24T10:38:12.562972+0000","last_clean":"2024-03-24T10:38:12.562972+0000","last_became_active":"2024-03-24T10:38:10.886406+0000","last_became_peered":"2024-03-24T10:38:10.886406+0000","last_unstale":"2024-03-24T10:38:12.562972+0000","last_undegraded":"2024-03-24T10:38:12.562972+0000","last_fullsized":"2024-03-24T10:38:12.562972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:35.763623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.895552+0000","last_change":"2024-03-24T10:38:09.863542+0000","last_active":"2024-03-24T10:38:13.895552+0000","last_peered":"2024-03-24T10:38:13.895552+0000","last_clean":"2024-03-24T10:38:13.895552+0000","last_became_active":"2024-03-24T10:38:09.863103+0000","last_became_peered":"2024-03-24T10:38:09.863103+0000","last_unstale":"2024-03-24T10:38:13.895552+0000","last_undegraded":"2024-03-24T10:38:13.895552+0000","last_fullsized":"2024-03-24T10:38:13.895552+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:52:34.837100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882258+0000","last_change":"2024-03-24T10:38:10.873871+0000","last_active":"2024-03-24T10:38:11.882258+0000","last_peered":"2024-03-24T10:38:11.882258+0000","last_clean":"2024-03-24T10:38:11.882258+0000","last_became_active":"2024-03-24T10:38:10.873528+0000","last_became_peered":"2024-03-24T10:38:10.873528+0000","last_unstale":"2024-03-24T10:38:11.882258+0000","last_undegraded":"2024-03-24T10:38:11.882258+0000","last_fullsized":"2024-03-24T10:38:11.882258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:33.372195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.896094+0000","last_change":"2024-03-24T10:38:09.861548+0000","last_active":"2024-03-24T10:38:13.896094+0000","last_peered":"2024-03-24T10:38:13.896094+0000","last_clean":"2024-03-24T10:38:13.896094+0000","last_became_active":"2024-03-24T10:38:09.861219+0000","last_became_peered":"2024-03-24T10:38:09.861219+0000","last_unstale":"2024-03-24T10:38:13.896094+0000","last_undegraded":"2024-03-24T10:38:13.896094+0000","last_fullsized":"2024-03-24T10:38:13.896094+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:57:46.398179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562528+0000","last_change":"2024-03-24T10:38:10.880202+0000","last_active":"2024-03-24T10:38:12.562528+0000","last_peered":"2024-03-24T10:38:12.562528+0000","last_clean":"2024-03-24T10:38:12.562528+0000","last_became_active":"2024-03-24T10:38:10.879807+0000","last_became_peered":"2024-03-24T10:38:10.879807+0000","last_unstale":"2024-03-24T10:38:12.562528+0000","last_undegraded":"2024-03-24T10:38:12.562528+0000","last_fullsized":"2024-03-24T10:38:12.562528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:37.498728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.453867+0000","last_change":"2024-03-24T10:38:09.866914+0000","last_active":"2024-03-24T10:38:13.453867+0000","last_peered":"2024-03-24T10:38:13.453867+0000","last_clean":"2024-03-24T10:38:13.453867+0000","last_became_active":"2024-03-24T10:38:09.866323+0000","last_became_peered":"2024-03-24T10:38:09.866323+0000","last_unstale":"2024-03-24T10:38:13.453867+0000","last_undegraded":"2024-03-24T10:38:13.453867+0000","last_fullsized":"2024-03-24T10:38:13.453867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:10:42.111336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882357+0000","last_change":"2024-03-24T10:38:10.884371+0000","last_active":"2024-03-24T10:38:11.882357+0000","last_peered":"2024-03-24T10:38:11.882357+0000","last_clean":"2024-03-24T10:38:11.882357+0000","last_became_active":"2024-03-24T10:38:10.884022+0000","last_became_peered":"2024-03-24T10:38:10.884022+0000","last_unstale":"2024-03-24T10:38:11.882357+0000","last_undegraded":"2024-03-24T10:38:11.882357+0000","last_fullsized":"2024-03-24T10:38:11.882357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:47:48.778727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:13.897322+0000","last_change":"2024-03-24T10:38:09.863231+0000","last_active":"2024-03-24T10:38:13.897322+0000","last_peered":"2024-03-24T10:38:13.897322+0000","last_clean":"2024-03-24T10:38:13.897322+0000","last_became_active":"2024-03-24T10:38:09.862848+0000","last_became_peered":"2024-03-24T10:38:09.862848+0000","last_unstale":"2024-03-24T10:38:13.897322+0000","last_undegraded":"2024-03-24T10:38:13.897322+0000","last_fullsized":"2024-03-24T10:38:13.897322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:36:57.532746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872857+0000","last_change":"2024-03-24T10:38:10.875187+0000","last_active":"2024-03-24T10:38:11.872857+0000","last_peered":"2024-03-24T10:38:11.872857+0000","last_clean":"2024-03-24T10:38:11.872857+0000","last_became_active":"2024-03-24T10:38:10.874682+0000","last_became_peered":"2024-03-24T10:38:10.874682+0000","last_unstale":"2024-03-24T10:38:11.872857+0000","last_undegraded":"2024-03-24T10:38:11.872857+0000","last_fullsized":"2024-03-24T10:38:11.872857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:53:56.915312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:14.909133+0000","last_change":"2024-03-24T10:38:09.863430+0000","last_active":"2024-03-24T10:38:14.909133+0000","last_peered":"2024-03-24T10:38:14.909133+0000","last_clean":"2024-03-24T10:38:14.909133+0000","last_became_active":"2024-03-24T10:38:09.862877+0000","last_became_peered":"2024-03-24T10:38:09.862877+0000","last_unstale":"2024-03-24T10:38:14.909133+0000","last_undegraded":"2024-03-24T10:38:14.909133+0000","last_fullsized":"2024-03-24T10:38:14.909133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:33:29.039896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872851+0000","last_change":"2024-03-24T10:38:10.878446+0000","last_active":"2024-03-24T10:38:11.872851+0000","last_peered":"2024-03-24T10:38:11.872851+0000","last_clean":"2024-03-24T10:38:11.872851+0000","last_became_active":"2024-03-24T10:38:10.877944+0000","last_became_peered":"2024-03-24T10:38:10.877944+0000","last_unstale":"2024-03-24T10:38:11.872851+0000","last_undegraded":"2024-03-24T10:38:11.872851+0000","last_fullsized":"2024-03-24T10:38:11.872851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:00:22.568669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.870352+0000","last_change":"2024-03-24T10:38:09.867328+0000","last_active":"2024-03-24T10:38:11.870352+0000","last_peered":"2024-03-24T10:38:11.870352+0000","last_clean":"2024-03-24T10:38:11.870352+0000","last_became_active":"2024-03-24T10:38:09.866869+0000","last_became_peered":"2024-03-24T10:38:09.866869+0000","last_unstale":"2024-03-24T10:38:11.870352+0000","last_undegraded":"2024-03-24T10:38:11.870352+0000","last_fullsized":"2024-03-24T10:38:11.870352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:52:32.136151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872598+0000","last_change":"2024-03-24T10:38:10.878807+0000","last_active":"2024-03-24T10:38:11.872598+0000","last_peered":"2024-03-24T10:38:11.872598+0000","last_clean":"2024-03-24T10:38:11.872598+0000","last_became_active":"2024-03-24T10:38:10.878455+0000","last_became_peered":"2024-03-24T10:38:10.878455+0000","last_unstale":"2024-03-24T10:38:11.872598+0000","last_undegraded":"2024-03-24T10:38:11.872598+0000","last_fullsized":"2024-03-24T10:38:11.872598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:52:14.079275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869761+0000","last_change":"2024-03-24T10:38:09.865399+0000","last_active":"2024-03-24T10:38:11.869761+0000","last_peered":"2024-03-24T10:38:11.869761+0000","last_clean":"2024-03-24T10:38:11.869761+0000","last_became_active":"2024-03-24T10:38:09.864906+0000","last_became_peered":"2024-03-24T10:38:09.864906+0000","last_unstale":"2024-03-24T10:38:11.869761+0000","last_undegraded":"2024-03-24T10:38:11.869761+0000","last_fullsized":"2024-03-24T10:38:11.869761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:53:12.363018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.872782+0000","last_change":"2024-03-24T10:38:10.876881+0000","last_active":"2024-03-24T10:38:11.872782+0000","last_peered":"2024-03-24T10:38:11.872782+0000","last_clean":"2024-03-24T10:38:11.872782+0000","last_became_active":"2024-03-24T10:38:10.876479+0000","last_became_peered":"2024-03-24T10:38:10.876479+0000","last_unstale":"2024-03-24T10:38:11.872782+0000","last_undegraded":"2024-03-24T10:38:11.872782+0000","last_fullsized":"2024-03-24T10:38:11.872782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:57:33.140451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562758+0000","last_change":"2024-03-24T10:38:10.872622+0000","last_active":"2024-03-24T10:38:12.562758+0000","last_peered":"2024-03-24T10:38:12.562758+0000","last_clean":"2024-03-24T10:38:12.562758+0000","last_became_active":"2024-03-24T10:38:10.872293+0000","last_became_peered":"2024-03-24T10:38:10.872293+0000","last_unstale":"2024-03-24T10:38:12.562758+0000","last_undegraded":"2024-03-24T10:38:12.562758+0000","last_fullsized":"2024-03-24T10:38:12.562758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:48.139589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563103+0000","last_change":"2024-03-24T10:38:10.888594+0000","last_active":"2024-03-24T10:38:12.563103+0000","last_peered":"2024-03-24T10:38:12.563103+0000","last_clean":"2024-03-24T10:38:12.563103+0000","last_became_active":"2024-03-24T10:38:10.888257+0000","last_became_peered":"2024-03-24T10:38:10.888257+0000","last_unstale":"2024-03-24T10:38:12.563103+0000","last_undegraded":"2024-03-24T10:38:12.563103+0000","last_fullsized":"2024-03-24T10:38:12.563103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:14:22.446011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563077+0000","last_change":"2024-03-24T10:38:10.879679+0000","last_active":"2024-03-24T10:38:12.563077+0000","last_peered":"2024-03-24T10:38:12.563077+0000","last_clean":"2024-03-24T10:38:12.563077+0000","last_became_active":"2024-03-24T10:38:10.879334+0000","last_became_peered":"2024-03-24T10:38:10.879334+0000","last_unstale":"2024-03-24T10:38:12.563077+0000","last_undegraded":"2024-03-24T10:38:12.563077+0000","last_fullsized":"2024-03-24T10:38:12.563077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:27:53.517368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562716+0000","last_change":"2024-03-24T10:38:10.886760+0000","last_active":"2024-03-24T10:38:12.562716+0000","last_peered":"2024-03-24T10:38:12.562716+0000","last_clean":"2024-03-24T10:38:12.562716+0000","last_became_active":"2024-03-24T10:38:10.886434+0000","last_became_peered":"2024-03-24T10:38:10.886434+0000","last_unstale":"2024-03-24T10:38:12.562716+0000","last_undegraded":"2024-03-24T10:38:12.562716+0000","last_fullsized":"2024-03-24T10:38:12.562716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:42:38.377683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869917+0000","last_change":"2024-03-24T10:38:10.887108+0000","last_active":"2024-03-24T10:38:11.869917+0000","last_peered":"2024-03-24T10:38:11.869917+0000","last_clean":"2024-03-24T10:38:11.869917+0000","last_became_active":"2024-03-24T10:38:10.886770+0000","last_became_peered":"2024-03-24T10:38:10.886770+0000","last_unstale":"2024-03-24T10:38:11.869917+0000","last_undegraded":"2024-03-24T10:38:11.869917+0000","last_fullsized":"2024-03-24T10:38:11.869917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:56.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562816+0000","last_change":"2024-03-24T10:38:10.873369+0000","last_active":"2024-03-24T10:38:12.562816+0000","last_peered":"2024-03-24T10:38:12.562816+0000","last_clean":"2024-03-24T10:38:12.562816+0000","last_became_active":"2024-03-24T10:38:10.873046+0000","last_became_peered":"2024-03-24T10:38:10.873046+0000","last_unstale":"2024-03-24T10:38:12.562816+0000","last_undegraded":"2024-03-24T10:38:12.562816+0000","last_fullsized":"2024-03-24T10:38:12.562816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:46.071084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.563551+0000","last_change":"2024-03-24T10:38:10.878666+0000","last_active":"2024-03-24T10:38:12.563551+0000","last_peered":"2024-03-24T10:38:12.563551+0000","last_clean":"2024-03-24T10:38:12.563551+0000","last_became_active":"2024-03-24T10:38:10.877250+0000","last_became_peered":"2024-03-24T10:38:10.877250+0000","last_unstale":"2024-03-24T10:38:12.563551+0000","last_undegraded":"2024-03-24T10:38:12.563551+0000","last_fullsized":"2024-03-24T10:38:12.563551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:13:36.793584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882491+0000","last_change":"2024-03-24T10:38:10.889274+0000","last_active":"2024-03-24T10:38:11.882491+0000","last_peered":"2024-03-24T10:38:11.882491+0000","last_clean":"2024-03-24T10:38:11.882491+0000","last_became_active":"2024-03-24T10:38:10.888899+0000","last_became_peered":"2024-03-24T10:38:10.888899+0000","last_unstale":"2024-03-24T10:38:11.882491+0000","last_undegraded":"2024-03-24T10:38:11.882491+0000","last_fullsized":"2024-03-24T10:38:11.882491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:11.225562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.870025+0000","last_change":"2024-03-24T10:38:10.887135+0000","last_active":"2024-03-24T10:38:11.870025+0000","last_peered":"2024-03-24T10:38:11.870025+0000","last_clean":"2024-03-24T10:38:11.870025+0000","last_became_active":"2024-03-24T10:38:10.886812+0000","last_became_peered":"2024-03-24T10:38:10.886812+0000","last_unstale":"2024-03-24T10:38:11.870025+0000","last_undegraded":"2024-03-24T10:38:11.870025+0000","last_fullsized":"2024-03-24T10:38:11.870025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:51:53.959914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.882277+0000","last_change":"2024-03-24T10:38:10.885417+0000","last_active":"2024-03-24T10:38:11.882277+0000","last_peered":"2024-03-24T10:38:11.882277+0000","last_clean":"2024-03-24T10:38:11.882277+0000","last_became_active":"2024-03-24T10:38:10.883209+0000","last_became_peered":"2024-03-24T10:38:10.883209+0000","last_unstale":"2024-03-24T10:38:11.882277+0000","last_undegraded":"2024-03-24T10:38:11.882277+0000","last_fullsized":"2024-03-24T10:38:11.882277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:52:52.535611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.561957+0000","last_change":"2024-03-24T10:38:10.885005+0000","last_active":"2024-03-24T10:38:12.561957+0000","last_peered":"2024-03-24T10:38:12.561957+0000","last_clean":"2024-03-24T10:38:12.561957+0000","last_became_active":"2024-03-24T10:38:10.884680+0000","last_became_peered":"2024-03-24T10:38:10.884680+0000","last_unstale":"2024-03-24T10:38:12.561957+0000","last_undegraded":"2024-03-24T10:38:12.561957+0000","last_fullsized":"2024-03-24T10:38:12.561957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:56.377777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:11.869920+0000","last_change":"2024-03-24T10:38:10.875153+0000","last_active":"2024-03-24T10:38:11.869920+0000","last_peered":"2024-03-24T10:38:11.869920+0000","last_clean":"2024-03-24T10:38:11.869920+0000","last_became_active":"2024-03-24T10:38:10.874721+0000","last_became_peered":"2024-03-24T10:38:10.874721+0000","last_unstale":"2024-03-24T10:38:11.869920+0000","last_undegraded":"2024-03-24T10:38:11.869920+0000","last_fullsized":"2024-03-24T10:38:11.869920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:52.522797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-24T10:38:12.562750+0000","last_change":"2024-03-24T10:38:10.887298+0000","last_active":"2024-03-24T10:38:12.562750+0000","last_peered":"2024-03-24T10:38:12.562750+0000","last_clean":"2024-03-24T10:38:12.562750+0000","last_became_active":"2024-03-24T10:38:10.886975+0000","last_became_peered":"2024-03-24T10:38:10.886975+0000","last_unstale":"2024-03-24T10:38:12.562750+0000","last_undegraded":"2024-03-24T10:38:12.562750+0000","last_fullsized":"2024-03-24T10:38:12.562750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:09.740303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":12213,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":130,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":66139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32692,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712460,"statfs":{"total":95995035648,"available":95961559040,"internally_reserved":0,"allocated":1552384,"data_stored":1032079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808856,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":439339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070492,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":439147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332129,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1523712,"data_stored":1030681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":921600,"data_stored":438728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":439999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":478064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":438715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":439556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1536000,"data_stored":1030365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":8212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":39543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":10181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:38:19.042 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T10:38:19.043 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T10:38:19.043 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T10:38:19.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:18 smithi117 ceph-mon[36368]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-03-24T10:38:19.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:18 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1270059950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:19.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:18 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2920675742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:19.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:18 smithi117 ceph-mon[36368]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:38:19.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:18 smithi116 ceph-mon[36248]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-03-24T10:38:19.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:18 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1270059950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:19.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:18 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2920675742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:19.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:18 smithi116 ceph-mon[36248]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:38:19.317 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:19.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:18 smithi023 ceph-mon[32466]: mgrmap e17: x(active, since 7m), standbys: y, z 2024-03-24T10:38:19.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:18 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1270059950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:38:19.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:18 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2920675742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:19.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:18 smithi023 ceph-mon[32466]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:38:20.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:19 smithi023 ceph-mon[32466]: pgmap v287: 97 pgs: 97 active+clean; 588 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6.0 KiB/s wr, 19 op/s 2024-03-24T10:38:20.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:19 smithi023 ceph-mon[32466]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:38:20.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:19 smithi117 ceph-mon[36368]: pgmap v287: 97 pgs: 97 active+clean; 588 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6.0 KiB/s wr, 19 op/s 2024-03-24T10:38:20.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:19 smithi117 ceph-mon[36368]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:38:20.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:19 smithi116 ceph-mon[36248]: pgmap v287: 97 pgs: 97 active+clean; 588 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6.0 KiB/s wr, 19 op/s 2024-03-24T10:38:20.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:19 smithi116 ceph-mon[36248]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:38:20.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.887+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/526191707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5d0103450 msgr2=0x7fc5d01038b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:20.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.887+0000 7fc5d6c5c700 1 --2- 172.21.15.23:0/526191707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5d0103450 0x7fc5d01038b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40099c0 tx=0x7fc5c4009cd0 comp rx=0 tx=0).stop 2024-03-24T10:38:20.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.887+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/526191707 shutdown_connections 2024-03-24T10:38:20.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.887+0000 7fc5d6c5c700 1 --2- 172.21.15.23:0/526191707 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d0103df0 0x7fc5d010a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:20.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.887+0000 7fc5d6c5c700 1 --2- 172.21.15.23:0/526191707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0102250 0x7fc5d0102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:20.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.887+0000 7fc5d6c5c700 1 --2- 172.21.15.23:0/526191707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5d0103450 0x7fc5d01038b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:20.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.887+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/526191707 >> 172.21.15.23:0/526191707 conn(0x7fc5d00fd810 msgr2=0x7fc5d00ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:20.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.887+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/526191707 shutdown_connections 2024-03-24T10:38:20.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.887+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/526191707 wait complete. 2024-03-24T10:38:20.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.888+0000 7fc5d6c5c700 1 Processor -- start 2024-03-24T10:38:20.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.888+0000 7fc5d6c5c700 1 -- start start 2024-03-24T10:38:20.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.888+0000 7fc5d6c5c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5d0102250 0x7fc5d01980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.888+0000 7fc5d6c5c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d0103450 0x7fc5d0198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.888+0000 7fc5d6c5c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0103df0 0x7fc5d019f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.888+0000 7fc5d6c5c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d019fc70 con 0x7fc5d0102250 2024-03-24T10:38:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.888+0000 7fc5d6c5c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d019fdb0 con 0x7fc5d0103df0 2024-03-24T10:38:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.888+0000 7fc5d6c5c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d019ff40 con 0x7fc5d0103450 2024-03-24T10:38:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.888+0000 7fc5d49f8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5d0102250 0x7fc5d01980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.888+0000 7fc5d49f8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5d0102250 0x7fc5d01980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44632/0 (socket says 172.21.15.23:44632) 2024-03-24T10:38:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.888+0000 7fc5d49f8700 1 -- 172.21.15.23:0/3057474257 learned_addr learned my addr 172.21.15.23:0/3057474257 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.888+0000 7fc5d51f9700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0103df0 0x7fc5d019f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.888+0000 7fc5cffff700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d0103450 0x7fc5d0198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.888+0000 7fc5d49f8700 1 -- 172.21.15.23:0/3057474257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d0103450 msgr2=0x7fc5d0198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.889+0000 7fc5d49f8700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d0103450 0x7fc5d0198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.889+0000 7fc5d49f8700 1 -- 172.21.15.23:0/3057474257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0103df0 msgr2=0x7fc5d019f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.889+0000 7fc5d49f8700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0103df0 0x7fc5d019f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:20.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.889+0000 7fc5d49f8700 1 -- 172.21.15.23:0/3057474257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c40095f0 con 0x7fc5d0102250 2024-03-24T10:38:20.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.889+0000 7fc5d51f9700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0103df0 0x7fc5d019f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:20.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.889+0000 7fc5d49f8700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5d0102250 0x7fc5d01980e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc5c000e9a0 tx=0x7fc5c000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:20.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.889+0000 7fc5cdffb700 1 -- 172.21.15.23:0/3057474257 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0009790 con 0x7fc5d0102250 2024-03-24T10:38:20.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.889+0000 7fc5cdffb700 1 -- 172.21.15.23:0/3057474257 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5c000ccf0 con 0x7fc5d0102250 2024-03-24T10:38:20.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.889+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/3057474257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d0074490 con 0x7fc5d0102250 2024-03-24T10:38:20.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.889+0000 7fc5cdffb700 1 -- 172.21.15.23:0/3057474257 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0003cb0 con 0x7fc5d0102250 2024-03-24T10:38:20.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.889+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/3057474257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d00749e0 con 0x7fc5d0102250 2024-03-24T10:38:20.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.890+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/3057474257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5d0109610 con 0x7fc5d0102250 2024-03-24T10:38:20.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.892+0000 7fc5cdffb700 1 -- 172.21.15.23:0/3057474257 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fc5c0004320 con 0x7fc5d0102250 2024-03-24T10:38:20.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.892+0000 7fc5cdffb700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc5b80a2310 0x7fc5b80a47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:20.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.892+0000 7fc5cffff700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc5b80a2310 0x7fc5b80a47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:20.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.892+0000 7fc5cdffb700 1 -- 172.21.15.23:0/3057474257 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fc5c0014070 con 0x7fc5d0102250 2024-03-24T10:38:20.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.893+0000 7fc5cffff700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc5b80a2310 0x7fc5b80a47d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4009990 tx=0x7fc5c400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:20.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:20.895+0000 7fc5cdffb700 1 -- 172.21.15.23:0/3057474257 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc5c0096f70 con 0x7fc5d0102250 2024-03-24T10:38:21.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:21.187+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/3057474257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fc5d010dc20 con 0x7fc5d0102250 2024-03-24T10:38:21.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:21.189+0000 7fc5cdffb700 1 -- 172.21.15.23:0/3057474257 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fc5c009a630 con 0x7fc5d0102250 2024-03-24T10:38:21.190 INFO:teuthology.orchestra.run.smithi023.stdout:[client.0] 2024-03-24T10:38:21.190 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQCdAgBm0bRGCxAAZ7HXpCgkFQaZ2zWrPtn44Q== 2024-03-24T10:38:21.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:21.192+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/3057474257 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc5b80a2310 msgr2=0x7fc5b80a47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:21.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:21.192+0000 7fc5d6c5c700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc5b80a2310 0x7fc5b80a47d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4009990 tx=0x7fc5c400b540 comp rx=0 tx=0).stop 2024-03-24T10:38:21.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:21.192+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/3057474257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5d0102250 msgr2=0x7fc5d01980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:21.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:21.192+0000 7fc5d6c5c700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5d0102250 0x7fc5d01980e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc5c000e9a0 tx=0x7fc5c000ecb0 comp rx=0 tx=0).stop 2024-03-24T10:38:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:21.192+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/3057474257 shutdown_connections 2024-03-24T10:38:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:21.192+0000 7fc5d6c5c700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc5b80a2310 0x7fc5b80a47d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:21.192+0000 7fc5d6c5c700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5d0103df0 0x7fc5d019f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:21.192+0000 7fc5d6c5c700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5d0102250 0x7fc5d01980e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:21.192+0000 7fc5d6c5c700 1 --2- 172.21.15.23:0/3057474257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d0103450 0x7fc5d0198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:21.192+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/3057474257 >> 172.21.15.23:0/3057474257 conn(0x7fc5d00fd810 msgr2=0x7fc5d0106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:21.192+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/3057474257 shutdown_connections 2024-03-24T10:38:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:21.192+0000 7fc5d6c5c700 1 -- 172.21.15.23:0/3057474257 wait complete. 2024-03-24T10:38:21.722 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:38:21.722 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T10:38:21.722 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T10:38:21.765 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T10:38:21.804 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T10:38:21.804 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T10:38:22.045 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:22.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:22 smithi023 ceph-mon[32466]: pgmap v288: 97 pgs: 97 active+clean; 588 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.8 KiB/s wr, 16 op/s 2024-03-24T10:38:22.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:22 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3057474257' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T10:38:22.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:22 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3057474257' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T10:38:22.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:22 smithi117 ceph-mon[36368]: pgmap v288: 97 pgs: 97 active+clean; 588 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.8 KiB/s wr, 16 op/s 2024-03-24T10:38:22.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:22 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3057474257' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T10:38:22.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:22 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3057474257' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T10:38:22.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:22 smithi116 ceph-mon[36248]: pgmap v288: 97 pgs: 97 active+clean; 588 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.8 KiB/s wr, 16 op/s 2024-03-24T10:38:22.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:22 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3057474257' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T10:38:22.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:22 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3057474257' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T10:38:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.617+0000 7fc91d7c0700 1 -- 172.21.15.23:0/1582251139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc918101aa0 msgr2=0x7fc91810df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.617+0000 7fc91d7c0700 1 --2- 172.21.15.23:0/1582251139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc918101aa0 0x7fc91810df80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc908009990 tx=0x7fc908009ca0 comp rx=0 tx=0).stop 2024-03-24T10:38:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.617+0000 7fc91d7c0700 1 -- 172.21.15.23:0/1582251139 shutdown_connections 2024-03-24T10:38:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.617+0000 7fc91d7c0700 1 --2- 172.21.15.23:0/1582251139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc918101aa0 0x7fc91810df80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.617+0000 7fc91d7c0700 1 --2- 172.21.15.23:0/1582251139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc918107140 0x7fc918107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.617+0000 7fc91d7c0700 1 --2- 172.21.15.23:0/1582251139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc918101100 0x7fc918101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.617+0000 7fc91d7c0700 1 -- 172.21.15.23:0/1582251139 >> 172.21.15.23:0/1582251139 conn(0x7fc918078030 msgr2=0x7fc918078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.617+0000 7fc91d7c0700 1 -- 172.21.15.23:0/1582251139 shutdown_connections 2024-03-24T10:38:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.617+0000 7fc91d7c0700 1 -- 172.21.15.23:0/1582251139 wait complete. 2024-03-24T10:38:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.617+0000 7fc91d7c0700 1 Processor -- start 2024-03-24T10:38:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc91d7c0700 1 -- start start 2024-03-24T10:38:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc91d7c0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc918101100 0x7fc9181986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc91d7c0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc918101aa0 0x7fc918198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc91d7c0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc918107140 0x7fc91819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc91d7c0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc91819d5b0 con 0x7fc918101100 2024-03-24T10:38:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc91d7c0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc91819d720 con 0x7fc918101aa0 2024-03-24T10:38:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc91d7c0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc91819d8e0 con 0x7fc918107140 2024-03-24T10:38:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc916ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc918101100 0x7fc9181986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc9167fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc918101aa0 0x7fc918198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc916ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc918101100 0x7fc9181986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44640/0 (socket says 172.21.15.23:44640) 2024-03-24T10:38:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc916ffd700 1 -- 172.21.15.23:0/2885087316 learned_addr learned my addr 172.21.15.23:0/2885087316 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc9177fe700 1 --2- 172.21.15.23:0/2885087316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc918107140 0x7fc91819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc916ffd700 1 -- 172.21.15.23:0/2885087316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc918107140 msgr2=0x7fc91819cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc916ffd700 1 --2- 172.21.15.23:0/2885087316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc918107140 0x7fc91819cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc916ffd700 1 -- 172.21.15.23:0/2885087316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc918101aa0 msgr2=0x7fc918198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc916ffd700 1 --2- 172.21.15.23:0/2885087316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc918101aa0 0x7fc918198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc916ffd700 1 -- 172.21.15.23:0/2885087316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9080095f0 con 0x7fc918101100 2024-03-24T10:38:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc9167fc700 1 --2- 172.21.15.23:0/2885087316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc918101aa0 0x7fc918198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.618+0000 7fc916ffd700 1 --2- 172.21.15.23:0/2885087316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc918101100 0x7fc9181986d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc91000ef10 tx=0x7fc91000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.619+0000 7fc90ffff700 1 -- 172.21.15.23:0/2885087316 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91000c9c0 con 0x7fc918101100 2024-03-24T10:38:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.619+0000 7fc91d7c0700 1 -- 172.21.15.23:0/2885087316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc91819db60 con 0x7fc918101100 2024-03-24T10:38:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.619+0000 7fc90ffff700 1 -- 172.21.15.23:0/2885087316 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc91000f040 con 0x7fc918101100 2024-03-24T10:38:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.619+0000 7fc90ffff700 1 -- 172.21.15.23:0/2885087316 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc910014480 con 0x7fc918101100 2024-03-24T10:38:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.619+0000 7fc91d7c0700 1 -- 172.21.15.23:0/2885087316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9181a5420 con 0x7fc918101100 2024-03-24T10:38:23.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.620+0000 7fc91d7c0700 1 -- 172.21.15.23:0/2885087316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc918068ec0 con 0x7fc918101100 2024-03-24T10:38:23.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.621+0000 7fc90ffff700 1 -- 172.21.15.23:0/2885087316 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fc910007720 con 0x7fc918101100 2024-03-24T10:38:23.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.622+0000 7fc90ffff700 1 --2- 172.21.15.23:0/2885087316 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc9040a21e0 0x7fc9040a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:23.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.622+0000 7fc90ffff700 1 -- 172.21.15.23:0/2885087316 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fc9100ce550 con 0x7fc918101100 2024-03-24T10:38:23.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.622+0000 7fc9167fc700 1 --2- 172.21.15.23:0/2885087316 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc9040a21e0 0x7fc9040a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:23.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.622+0000 7fc9167fc700 1 --2- 172.21.15.23:0/2885087316 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc9040a21e0 0x7fc9040a46a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc91806af00 tx=0x7fc900009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:23.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.625+0000 7fc90ffff700 1 -- 172.21.15.23:0/2885087316 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc910097570 con 0x7fc918101100 2024-03-24T10:38:23.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.888+0000 7fc91d7c0700 1 -- 172.21.15.23:0/2885087316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fc918111110 con 0x7fc918101100 2024-03-24T10:38:23.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.892+0000 7fc90ffff700 1 -- 172.21.15.23:0/2885087316 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+261467 (secure 0 0 0) 0x7fc91009ac30 con 0x7fc918101100 2024-03-24T10:38:23.896 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:38:23.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.895+0000 7fc91d7c0700 1 -- 172.21.15.23:0/2885087316 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc9040a21e0 msgr2=0x7fc9040a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:23.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.895+0000 7fc91d7c0700 1 --2- 172.21.15.23:0/2885087316 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc9040a21e0 0x7fc9040a46a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc91806af00 tx=0x7fc900009260 comp rx=0 tx=0).stop 2024-03-24T10:38:23.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.895+0000 7fc91d7c0700 1 -- 172.21.15.23:0/2885087316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc918101100 msgr2=0x7fc9181986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:23.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.895+0000 7fc91d7c0700 1 --2- 172.21.15.23:0/2885087316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc918101100 0x7fc9181986d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc91000ef10 tx=0x7fc91000c3a0 comp rx=0 tx=0).stop 2024-03-24T10:38:23.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.895+0000 7fc91d7c0700 1 -- 172.21.15.23:0/2885087316 shutdown_connections 2024-03-24T10:38:23.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.895+0000 7fc91d7c0700 1 --2- 172.21.15.23:0/2885087316 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc9040a21e0 0x7fc9040a46a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:23.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.895+0000 7fc91d7c0700 1 --2- 172.21.15.23:0/2885087316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc918107140 0x7fc91819cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:23.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.895+0000 7fc91d7c0700 1 --2- 172.21.15.23:0/2885087316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc918101100 0x7fc9181986d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:23.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.895+0000 7fc91d7c0700 1 --2- 172.21.15.23:0/2885087316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc918101aa0 0x7fc918198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:23.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.895+0000 7fc91d7c0700 1 -- 172.21.15.23:0/2885087316 >> 172.21.15.23:0/2885087316 conn(0x7fc918078030 msgr2=0x7fc918105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:23.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.895+0000 7fc91d7c0700 1 -- 172.21.15.23:0/2885087316 shutdown_connections 2024-03-24T10:38:23.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:23.895+0000 7fc91d7c0700 1 -- 172.21.15.23:0/2885087316 wait complete. 2024-03-24T10:38:24.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:24 smithi023 ceph-mon[32466]: pgmap v289: 97 pgs: 97 active+clean; 623 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-24T10:38:24.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:24 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2885087316' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T10:38:24.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:24 smithi117 ceph-mon[36368]: pgmap v289: 97 pgs: 97 active+clean; 623 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-24T10:38:24.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:24 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2885087316' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T10:38:24.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:24 smithi116 ceph-mon[36248]: pgmap v289: 97 pgs: 97 active+clean; 623 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-24T10:38:24.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:24 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2885087316' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T10:38:24.598 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":2350021632},{"type":"v1","addr":"172.21.15.23:6801","nonce":2350021632}]},"active_addr":"172.21.15.23:6801/2350021632","active_change":"2024-03-24T10:31:14.712801+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.23:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":3614005933}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":725846898}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":256452041}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":352422381}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":1786662681}]}]} 2024-03-24T10:38:24.605 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T10:38:24.606 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T10:38:24.606 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T10:38:24.854 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:26.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.314+0000 7f891f178700 1 -- 172.21.15.23:0/594425536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89181011b0 msgr2=0x7f891810f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:26.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.314+0000 7f891f178700 1 --2- 172.21.15.23:0/594425536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89181011b0 0x7f891810f110 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8908009990 tx=0x7f8908009ca0 comp rx=0 tx=0).stop 2024-03-24T10:38:26.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.314+0000 7f891f178700 1 -- 172.21.15.23:0/594425536 shutdown_connections 2024-03-24T10:38:26.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.314+0000 7f891f178700 1 --2- 172.21.15.23:0/594425536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89181011b0 0x7f891810f110 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:26.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.314+0000 7f891f178700 1 --2- 172.21.15.23:0/594425536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8918107270 0x7f8918107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:26.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.314+0000 7f891f178700 1 --2- 172.21.15.23:0/594425536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89181007f0 0x7f8918100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:26.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.314+0000 7f891f178700 1 -- 172.21.15.23:0/594425536 >> 172.21.15.23:0/594425536 conn(0x7f8918077700 msgr2=0x7f89180feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:26.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.314+0000 7f891f178700 1 -- 172.21.15.23:0/594425536 shutdown_connections 2024-03-24T10:38:26.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.314+0000 7f891f178700 1 -- 172.21.15.23:0/594425536 wait complete. 2024-03-24T10:38:26.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.315+0000 7f891f178700 1 Processor -- start 2024-03-24T10:38:26.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.315+0000 7f891f178700 1 -- start start 2024-03-24T10:38:26.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.315+0000 7f891f178700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89181007f0 0x7f89181019d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:26.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.315+0000 7f891f178700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89181011b0 0x7f8918101f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:26.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.315+0000 7f891f178700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8918107270 0x7f8918102450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:26.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.315+0000 7f891f178700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f89181a6c50 con 0x7f89181011b0 2024-03-24T10:38:26.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.315+0000 7f891f178700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f89181a6dc0 con 0x7f89181007f0 2024-03-24T10:38:26.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.315+0000 7f891f178700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f89181a6f80 con 0x7f8918107270 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.315+0000 7f8917fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89181011b0 0x7f8918101f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.315+0000 7f8917fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89181011b0 0x7f8918101f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44650/0 (socket says 172.21.15.23:44650) 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.315+0000 7f8917fff700 1 -- 172.21.15.23:0/2970870458 learned_addr learned my addr 172.21.15.23:0/2970870458 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.315+0000 7f891cf14700 1 --2- 172.21.15.23:0/2970870458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89181007f0 0x7f89181019d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.315+0000 7f891d715700 1 --2- 172.21.15.23:0/2970870458 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8918107270 0x7f8918102450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.316+0000 7f8917fff700 1 -- 172.21.15.23:0/2970870458 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8918107270 msgr2=0x7f8918102450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.316+0000 7f8917fff700 1 --2- 172.21.15.23:0/2970870458 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8918107270 0x7f8918102450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.316+0000 7f8917fff700 1 -- 172.21.15.23:0/2970870458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89181007f0 msgr2=0x7f89181019d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.316+0000 7f8917fff700 1 --2- 172.21.15.23:0/2970870458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89181007f0 0x7f89181019d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.316+0000 7f8917fff700 1 -- 172.21.15.23:0/2970870458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89080095f0 con 0x7f89181011b0 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.316+0000 7f8917fff700 1 --2- 172.21.15.23:0/2970870458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89181011b0 0x7f8918101f10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f890c00e9a0 tx=0x7f890c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.316+0000 7f8915ffb700 1 -- 172.21.15.23:0/2970870458 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c009790 con 0x7f89181011b0 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.316+0000 7f8915ffb700 1 -- 172.21.15.23:0/2970870458 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f890c010430 con 0x7f89181011b0 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.316+0000 7f8915ffb700 1 -- 172.21.15.23:0/2970870458 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c018680 con 0x7f89181011b0 2024-03-24T10:38:26.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.316+0000 7f891f178700 1 -- 172.21.15.23:0/2970870458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89181a7260 con 0x7f89181011b0 2024-03-24T10:38:26.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.316+0000 7f891f178700 1 -- 172.21.15.23:0/2970870458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89181a77b0 con 0x7f89181011b0 2024-03-24T10:38:26.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.317+0000 7f891f178700 1 -- 172.21.15.23:0/2970870458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f891810a860 con 0x7f89181011b0 2024-03-24T10:38:26.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.318+0000 7f8915ffb700 1 -- 172.21.15.23:0/2970870458 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f890c010910 con 0x7f89181011b0 2024-03-24T10:38:26.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.319+0000 7f8915ffb700 1 --2- 172.21.15.23:0/2970870458 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f89000a2260 0x7f89000a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:26.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.319+0000 7f891cf14700 1 --2- 172.21.15.23:0/2970870458 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f89000a2260 0x7f89000a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:26.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.319+0000 7f8915ffb700 1 -- 172.21.15.23:0/2970870458 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f890c014070 con 0x7f89181011b0 2024-03-24T10:38:26.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.319+0000 7f891cf14700 1 --2- 172.21.15.23:0/2970870458 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f89000a2260 0x7f89000a4720 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f891000f540 tx=0x7f891000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:26.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.323+0000 7f8915ffb700 1 -- 172.21.15.23:0/2970870458 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f890c053280 con 0x7f89181011b0 2024-03-24T10:38:26.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:26 smithi023 ceph-mon[32466]: pgmap v290: 97 pgs: 97 active+clean; 623 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-03-24T10:38:26.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:26 smithi117 ceph-mon[36368]: pgmap v290: 97 pgs: 97 active+clean; 623 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-03-24T10:38:26.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:26 smithi116 ceph-mon[36248]: pgmap v290: 97 pgs: 97 active+clean; 623 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-03-24T10:38:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.526+0000 7f891f178700 1 -- 172.21.15.23:0/2970870458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8918102f10 con 0x7f89181011b0 2024-03-24T10:38:26.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.528+0000 7f8915ffb700 1 -- 172.21.15.23:0/2970870458 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22233 (secure 0 0 0) 0x7f890c096e30 con 0x7f89181011b0 2024-03-24T10:38:26.529 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:38:26.529 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":85,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:38:11.864894+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:38:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.530+0000 7f891f178700 1 -- 172.21.15.23:0/2970870458 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f89000a2260 msgr2=0x7f89000a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.530+0000 7f891f178700 1 --2- 172.21.15.23:0/2970870458 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f89000a2260 0x7f89000a4720 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f891000f540 tx=0x7f891000b040 comp rx=0 tx=0).stop 2024-03-24T10:38:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.530+0000 7f891f178700 1 -- 172.21.15.23:0/2970870458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89181011b0 msgr2=0x7f8918101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.530+0000 7f891f178700 1 --2- 172.21.15.23:0/2970870458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89181011b0 0x7f8918101f10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f890c00e9a0 tx=0x7f890c00ecb0 comp rx=0 tx=0).stop 2024-03-24T10:38:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.530+0000 7f891f178700 1 -- 172.21.15.23:0/2970870458 shutdown_connections 2024-03-24T10:38:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.530+0000 7f891f178700 1 --2- 172.21.15.23:0/2970870458 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f89000a2260 0x7f89000a4720 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.530+0000 7f891f178700 1 --2- 172.21.15.23:0/2970870458 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8918107270 0x7f8918102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.530+0000 7f891f178700 1 --2- 172.21.15.23:0/2970870458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89181007f0 0x7f89181019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.530+0000 7f891f178700 1 --2- 172.21.15.23:0/2970870458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89181011b0 0x7f8918101f10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.530+0000 7f891f178700 1 -- 172.21.15.23:0/2970870458 >> 172.21.15.23:0/2970870458 conn(0x7f8918077700 msgr2=0x7f8918105180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.530+0000 7f891f178700 1 -- 172.21.15.23:0/2970870458 shutdown_connections 2024-03-24T10:38:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:26.530+0000 7f891f178700 1 -- 172.21.15.23:0/2970870458 wait complete. 2024-03-24T10:38:27.121 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T10:38:27.121 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T10:38:27.366 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:27.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:27 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2970870458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:27.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:27 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2970870458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:27.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:27 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2970870458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:28.415 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:28 smithi023 ceph-mon[32466]: pgmap v291: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-24T10:38:28.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:28 smithi117 ceph-mon[36368]: pgmap v291: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-24T10:38:28.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:28 smithi116 ceph-mon[36248]: pgmap v291: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.6 KiB/s wr, 17 op/s 2024-03-24T10:38:28.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.989+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/3341126998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae4102610 msgr2=0x7f0ae410dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:28.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.989+0000 7f0ae9bbd700 1 --2- 172.21.15.23:0/3341126998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae4102610 0x7f0ae410dfe0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00b0a0 tx=0x7f0adc00b3b0 comp rx=0 tx=0).stop 2024-03-24T10:38:28.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.989+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/3341126998 shutdown_connections 2024-03-24T10:38:28.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.989+0000 7f0ae9bbd700 1 --2- 172.21.15.23:0/3341126998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae4106140 0x7f0ae4106520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:28.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.989+0000 7f0ae9bbd700 1 --2- 172.21.15.23:0/3341126998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae4102610 0x7f0ae410dfe0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:28.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.989+0000 7f0ae9bbd700 1 --2- 172.21.15.23:0/3341126998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae4101c50 0x7f0ae41020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:28.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.989+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/3341126998 >> 172.21.15.23:0/3341126998 conn(0x7f0ae4076460 msgr2=0x7f0ae4076870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:28.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.990+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/3341126998 shutdown_connections 2024-03-24T10:38:28.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.990+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/3341126998 wait complete. 2024-03-24T10:38:28.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.990+0000 7f0ae9bbd700 1 Processor -- start 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.990+0000 7f0ae9bbd700 1 -- start start 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae9bbd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae4101c50 0x7f0ae4073c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae9bbd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae4102610 0x7f0ae4100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae9bbd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae4106140 0x7f0ae4101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae9bbd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae4101710 con 0x7f0ae4101c50 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae9bbd700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae4101880 con 0x7f0ae4106140 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae9bbd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae41af870 con 0x7f0ae4102610 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae4101c50 0x7f0ae4073c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae4101c50 0x7f0ae4073c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57116/0 (socket says 172.21.15.23:57116) 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae2ffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae4102610 0x7f0ae4100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae37fe700 1 -- 172.21.15.23:0/1184333311 learned_addr learned my addr 172.21.15.23:0/1184333311 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae3fff700 1 --2- 172.21.15.23:0/1184333311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae4106140 0x7f0ae4101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae37fe700 1 -- 172.21.15.23:0/1184333311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae4102610 msgr2=0x7f0ae4100b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae37fe700 1 --2- 172.21.15.23:0/1184333311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae4102610 0x7f0ae4100b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae37fe700 1 -- 172.21.15.23:0/1184333311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae4106140 msgr2=0x7f0ae4101080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:28.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae37fe700 1 --2- 172.21.15.23:0/1184333311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae4106140 0x7f0ae4101080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:28.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae37fe700 1 -- 172.21.15.23:0/1184333311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0adc009d00 con 0x7f0ae4101c50 2024-03-24T10:38:28.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae2ffd700 1 --2- 172.21.15.23:0/1184333311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae4102610 0x7f0ae4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:28.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae37fe700 1 --2- 172.21.15.23:0/1184333311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae4101c50 0x7f0ae4073c00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400e9a0 tx=0x7f0ad400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:28.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.991+0000 7f0ae0ff9700 1 -- 172.21.15.23:0/1184333311 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4009790 con 0x7f0ae4101c50 2024-03-24T10:38:28.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.992+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/1184333311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae41afa70 con 0x7f0ae4101c50 2024-03-24T10:38:28.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.992+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/1184333311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae41aff90 con 0x7f0ae4101c50 2024-03-24T10:38:28.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.992+0000 7f0ae0ff9700 1 -- 172.21.15.23:0/1184333311 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ad4010430 con 0x7f0ae4101c50 2024-03-24T10:38:28.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.992+0000 7f0ae0ff9700 1 -- 172.21.15.23:0/1184333311 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4018680 con 0x7f0ae4101c50 2024-03-24T10:38:28.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.993+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/1184333311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae4109730 con 0x7f0ae4101c50 2024-03-24T10:38:28.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.994+0000 7f0ae0ff9700 1 -- 172.21.15.23:0/1184333311 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f0ad40105a0 con 0x7f0ae4101c50 2024-03-24T10:38:28.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.994+0000 7f0ae0ff9700 1 --2- 172.21.15.23:0/1184333311 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0acc0a21e0 0x7f0acc0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:28.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.994+0000 7f0ae2ffd700 1 --2- 172.21.15.23:0/1184333311 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0acc0a21e0 0x7f0acc0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:28.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.994+0000 7f0ae0ff9700 1 -- 172.21.15.23:0/1184333311 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f0ad4014070 con 0x7f0ae4101c50 2024-03-24T10:38:28.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.995+0000 7f0ae2ffd700 1 --2- 172.21.15.23:0/1184333311 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0acc0a21e0 0x7f0acc0a46a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8011490 tx=0x7f0ad8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:28.999+0000 7f0ae0ff9700 1 -- 172.21.15.23:0/1184333311 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0ad4053280 con 0x7f0ae4101c50 2024-03-24T10:38:29.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:29.203+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/1184333311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0ae4073310 con 0x7f0ae4101c50 2024-03-24T10:38:29.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:29.205+0000 7f0ae0ff9700 1 -- 172.21.15.23:0/1184333311 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22233 (secure 0 0 0) 0x7f0ad409a8e0 con 0x7f0ae4101c50 2024-03-24T10:38:29.205 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:38:29.205 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":85,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:38:11.864894+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:38:29.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:29.207+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/1184333311 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0acc0a21e0 msgr2=0x7f0acc0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:29.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:29.207+0000 7f0ae9bbd700 1 --2- 172.21.15.23:0/1184333311 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0acc0a21e0 0x7f0acc0a46a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8011490 tx=0x7f0ad8009260 comp rx=0 tx=0).stop 2024-03-24T10:38:29.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:29.207+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/1184333311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae4101c50 msgr2=0x7f0ae4073c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:29.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:29.207+0000 7f0ae9bbd700 1 --2- 172.21.15.23:0/1184333311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae4101c50 0x7f0ae4073c00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400e9a0 tx=0x7f0ad400ecb0 comp rx=0 tx=0).stop 2024-03-24T10:38:29.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:29.207+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/1184333311 shutdown_connections 2024-03-24T10:38:29.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:29.207+0000 7f0ae9bbd700 1 --2- 172.21.15.23:0/1184333311 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0acc0a21e0 0x7f0acc0a46a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:29.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:29.207+0000 7f0ae9bbd700 1 --2- 172.21.15.23:0/1184333311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae4106140 0x7f0ae4101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:29.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:29.207+0000 7f0ae9bbd700 1 --2- 172.21.15.23:0/1184333311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae4101c50 0x7f0ae4073c00 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:29.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:29.207+0000 7f0ae9bbd700 1 --2- 172.21.15.23:0/1184333311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae4102610 0x7f0ae4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:29.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:29.207+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/1184333311 >> 172.21.15.23:0/1184333311 conn(0x7f0ae4076460 msgr2=0x7f0ae40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:29.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:29.207+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/1184333311 shutdown_connections 2024-03-24T10:38:29.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:29.207+0000 7f0ae9bbd700 1 -- 172.21.15.23:0/1184333311 wait complete. 2024-03-24T10:38:29.751 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T10:38:29.752 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T10:38:29.752 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T10:38:29.752 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T10:38:29.753 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T10:38:29.753 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T10:38:29.754 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T10:38:29.754 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T10:38:29.755 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-24T10:38:29.755 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-24T10:38:29.755 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-24T10:38:29.756 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-24T10:38:30.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:30 smithi023 ceph-mon[32466]: pgmap v292: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-03-24T10:38:30.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:30 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1184333311' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:30.453 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:30.454 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:30.459 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:30.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:30 smithi117 ceph-mon[36368]: pgmap v292: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-03-24T10:38:30.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:30 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1184333311' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:30.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:30 smithi116 ceph-mon[36248]: pgmap v292: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-03-24T10:38:30.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:30 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1184333311' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:38:31.182 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:31.189 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:31.189 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:31.190 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:31.191 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:31.191 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:31.194 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:31.195 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:32.362 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:32 smithi023 ceph-mon[32466]: pgmap v293: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.7 KiB/s wr, 7 op/s 2024-03-24T10:38:32.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:32 smithi117 ceph-mon[36368]: pgmap v293: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.7 KiB/s wr, 7 op/s 2024-03-24T10:38:32.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:32 smithi116 ceph-mon[36248]: pgmap v293: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.7 KiB/s wr, 7 op/s 2024-03-24T10:38:34.377 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:34 smithi023 ceph-mon[32466]: pgmap v294: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-03-24T10:38:34.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:34 smithi117 ceph-mon[36368]: pgmap v294: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-03-24T10:38:34.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:34 smithi116 ceph-mon[36248]: pgmap v294: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.7 KiB/s wr, 9 op/s 2024-03-24T10:38:34.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.616+0000 7f0f06cfa700 1 -- 172.21.15.23:0/2670002284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f00101d40 msgr2=0x7f0f00105d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:34.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.616+0000 7f0f06cfa700 1 --2- 172.21.15.23:0/2670002284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f00101d40 0x7f0f00105d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009990 tx=0x7f0ef4009ca0 comp rx=0 tx=0).stop 2024-03-24T10:38:34.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.616+0000 7f0f06cfa700 1 -- 172.21.15.23:0/2670002284 shutdown_connections 2024-03-24T10:38:34.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.616+0000 7f0f06cfa700 1 --2- 172.21.15.23:0/2670002284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f00106420 0x7f0f0010dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:34.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.616+0000 7f0f06cfa700 1 --2- 172.21.15.23:0/2670002284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f00101390 0x7f0f00101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:34.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.616+0000 7f0f06cfa700 1 --2- 172.21.15.23:0/2670002284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f00101d40 0x7f0f00105d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:34.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.616+0000 7f0f06cfa700 1 -- 172.21.15.23:0/2670002284 >> 172.21.15.23:0/2670002284 conn(0x7f0f00077120 msgr2=0x7f0f00077530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:34.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.616+0000 7f0f06cfa700 1 -- 172.21.15.23:0/2670002284 shutdown_connections 2024-03-24T10:38:34.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.616+0000 7f0f06cfa700 1 -- 172.21.15.23:0/2670002284 wait complete. 2024-03-24T10:38:34.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.616+0000 7f0f06cfa700 1 Processor -- start 2024-03-24T10:38:34.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.616+0000 7f0f06cfa700 1 -- start start 2024-03-24T10:38:34.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0f06cfa700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f00101390 0x7f0f001a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:34.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0f06cfa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f00101d40 0x7f0f001a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:34.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0f06cfa700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f00106420 0x7f0f0019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:34.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0f06cfa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0019af20 con 0x7f0f00101d40 2024-03-24T10:38:34.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0f06cfa700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0019b090 con 0x7f0f00101390 2024-03-24T10:38:34.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0f06cfa700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0019b250 con 0x7f0f00106420 2024-03-24T10:38:34.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0efffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f00101d40 0x7f0f001a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:34.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0efffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f00101d40 0x7f0f001a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57138/0 (socket says 172.21.15.23:57138) 2024-03-24T10:38:34.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0efffff700 1 -- 172.21.15.23:0/1041143249 learned_addr learned my addr 172.21.15.23:0/1041143249 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:34.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0f04a96700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f00101390 0x7f0f001a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:34.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0f05297700 1 --2- 172.21.15.23:0/1041143249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f00106420 0x7f0f0019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:34.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0efffff700 1 -- 172.21.15.23:0/1041143249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f00106420 msgr2=0x7f0f0019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:34.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0efffff700 1 --2- 172.21.15.23:0/1041143249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f00106420 0x7f0f0019a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:34.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0efffff700 1 -- 172.21.15.23:0/1041143249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f00101390 msgr2=0x7f0f001a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:34.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0efffff700 1 --2- 172.21.15.23:0/1041143249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f00101390 0x7f0f001a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:34.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0efffff700 1 -- 172.21.15.23:0/1041143249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef40095f0 con 0x7f0f00101d40 2024-03-24T10:38:34.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0f04a96700 1 --2- 172.21.15.23:0/1041143249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f00101390 0x7f0f001a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:34.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0efffff700 1 --2- 172.21.15.23:0/1041143249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f00101d40 0x7f0f001a0be0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4000c00 tx=0x7f0ef4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:34.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0efdffb700 1 -- 172.21.15.23:0/1041143249 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef401d070 con 0x7f0f00101d40 2024-03-24T10:38:34.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0efdffb700 1 -- 172.21.15.23:0/1041143249 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ef400b610 con 0x7f0f00101d40 2024-03-24T10:38:34.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0f06cfa700 1 -- 172.21.15.23:0/1041143249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0019b4d0 con 0x7f0f00101d40 2024-03-24T10:38:34.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.617+0000 7f0efdffb700 1 -- 172.21.15.23:0/1041143249 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef40173f0 con 0x7f0f00101d40 2024-03-24T10:38:34.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.618+0000 7f0f06cfa700 1 -- 172.21.15.23:0/1041143249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f001a7620 con 0x7f0f00101d40 2024-03-24T10:38:34.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.618+0000 7f0f06cfa700 1 -- 172.21.15.23:0/1041143249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0f00102820 con 0x7f0f00101d40 2024-03-24T10:38:34.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.620+0000 7f0efdffb700 1 -- 172.21.15.23:0/1041143249 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f0ef4022670 con 0x7f0f00101d40 2024-03-24T10:38:34.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.620+0000 7f0efdffb700 1 --2- 172.21.15.23:0/1041143249 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ee80a21e0 0x7f0ee80a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:34.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.620+0000 7f0f04a96700 1 --2- 172.21.15.23:0/1041143249 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ee80a21e0 0x7f0ee80a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:34.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.620+0000 7f0efdffb700 1 -- 172.21.15.23:0/1041143249 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f0ef40cf3f0 con 0x7f0f00101d40 2024-03-24T10:38:34.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.620+0000 7f0efdffb700 1 --2- 172.21.15.23:0/1041143249 >> [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038] conn(0x7f0ee80ab9c0 0x7f0ee80adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.620+0000 7f0efdffb700 1 -- 172.21.15.23:0/1041143249 --> [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ee80ae490 con 0x7f0ee80ab9c0 2024-03-24T10:38:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.620+0000 7f0efdffb700 1 -- 172.21.15.23:0/1041143249 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f0ef40cf890 con 0x7f0f00101d40 2024-03-24T10:38:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.621+0000 7f0f05297700 1 --2- 172.21.15.23:0/1041143249 >> [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038] conn(0x7f0ee80ab9c0 0x7f0ee80adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.621+0000 7f0f04a96700 1 --2- 172.21.15.23:0/1041143249 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ee80a21e0 0x7f0ee80a46a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009480 tx=0x7f0ef800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.621+0000 7f0f05297700 1 --2- 172.21.15.23:0/1041143249 >> [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038] conn(0x7f0ee80ab9c0 0x7f0ee80adde0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.622+0000 7f0efdffb700 1 -- 172.21.15.23:0/1041143249 <== osd.0 v2:172.21.15.23:6802/65775038 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ee80ae490 con 0x7f0ee80ab9c0 2024-03-24T10:38:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.638+0000 7f0f06cfa700 1 -- 172.21.15.23:0/1041143249 --> [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0f00111450 con 0x7f0ee80ab9c0 2024-03-24T10:38:34.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.638+0000 7f0efdffb700 1 -- 172.21.15.23:0/1041143249 <== osd.0 v2:172.21.15.23:6802/65775038 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0f00111450 con 0x7f0ee80ab9c0 2024-03-24T10:38:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.639+0000 7f0f06cfa700 1 -- 172.21.15.23:0/1041143249 >> [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038] conn(0x7f0ee80ab9c0 msgr2=0x7f0ee80adde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.639+0000 7f0f06cfa700 1 --2- 172.21.15.23:0/1041143249 >> [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038] conn(0x7f0ee80ab9c0 0x7f0ee80adde0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.639+0000 7f0f06cfa700 1 -- 172.21.15.23:0/1041143249 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ee80a21e0 msgr2=0x7f0ee80a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.639+0000 7f0f06cfa700 1 --2- 172.21.15.23:0/1041143249 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ee80a21e0 0x7f0ee80a46a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009480 tx=0x7f0ef800b040 comp rx=0 tx=0).stop 2024-03-24T10:38:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.639+0000 7f0f06cfa700 1 -- 172.21.15.23:0/1041143249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f00101d40 msgr2=0x7f0f001a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.639+0000 7f0f06cfa700 1 --2- 172.21.15.23:0/1041143249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f00101d40 0x7f0f001a0be0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4000c00 tx=0x7f0ef4004730 comp rx=0 tx=0).stop 2024-03-24T10:38:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.639+0000 7f0f05297700 1 -- 172.21.15.23:0/1041143249 reap_dead start 2024-03-24T10:38:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.639+0000 7f0f06cfa700 1 -- 172.21.15.23:0/1041143249 shutdown_connections 2024-03-24T10:38:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.639+0000 7f0f06cfa700 1 -- 172.21.15.23:0/1041143249 >> 172.21.15.23:0/1041143249 conn(0x7f0f00077120 msgr2=0x7f0f0006ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.639+0000 7f0f06cfa700 1 -- 172.21.15.23:0/1041143249 shutdown_connections 2024-03-24T10:38:34.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:34.639+0000 7f0f06cfa700 1 -- 172.21.15.23:0/1041143249 wait complete. 2024-03-24T10:38:36.303 INFO:teuthology.orchestra.run.smithi023.stdout:34359738440 2024-03-24T10:38:36.303 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T10:38:36.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:36 smithi023 ceph-mon[32466]: pgmap v295: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-24T10:38:36.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:36 smithi117 ceph-mon[36368]: pgmap v295: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-24T10:38:36.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:36 smithi116 ceph-mon[36248]: pgmap v295: 97 pgs: 97 active+clean; 638 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-24T10:38:37.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.042+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/2493631768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dd0101100 msgr2=0x7f3dd0101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:37.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.042+0000 7f3dd4ea3700 1 --2- 172.21.15.23:0/2493631768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dd0101100 0x7f3dd0101560 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0009990 tx=0x7f3dc0009ca0 comp rx=0 tx=0).stop 2024-03-24T10:38:37.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.042+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/2493631768 shutdown_connections 2024-03-24T10:38:37.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.042+0000 7f3dd4ea3700 1 --2- 172.21.15.23:0/2493631768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd0107140 0x7f3dd0107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:37.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.042+0000 7f3dd4ea3700 1 --2- 172.21.15.23:0/2493631768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dd0101aa0 0x7f3dd010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:37.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.042+0000 7f3dd4ea3700 1 --2- 172.21.15.23:0/2493631768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dd0101100 0x7f3dd0101560 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:37.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.042+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/2493631768 >> 172.21.15.23:0/2493631768 conn(0x7f3dd0078030 msgr2=0x7f3dd0078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:37.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.042+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/2493631768 shutdown_connections 2024-03-24T10:38:37.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.042+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/2493631768 wait complete. 2024-03-24T10:38:37.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.043+0000 7f3dd4ea3700 1 Processor -- start 2024-03-24T10:38:37.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.043+0000 7f3dd4ea3700 1 -- start start 2024-03-24T10:38:37.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.043+0000 7f3dd4ea3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dd0101100 0x7f3dd01976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:37.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.043+0000 7f3dd4ea3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dd0101aa0 0x7f3dd0197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:37.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.043+0000 7f3dd4ea3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd0107140 0x7f3dd019bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:37.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.043+0000 7f3dd4ea3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd019c530 con 0x7f3dd0101100 2024-03-24T10:38:37.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.043+0000 7f3dd4ea3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd019c6a0 con 0x7f3dd0101aa0 2024-03-24T10:38:37.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.043+0000 7f3dd4ea3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd019c860 con 0x7f3dd0107140 2024-03-24T10:38:37.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.043+0000 7f3dce59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dd0101100 0x7f3dd01976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:37.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.043+0000 7f3dce59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dd0101100 0x7f3dd01976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57150/0 (socket says 172.21.15.23:57150) 2024-03-24T10:38:37.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.043+0000 7f3dce59c700 1 -- 172.21.15.23:0/1969104724 learned_addr learned my addr 172.21.15.23:0/1969104724 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:37.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.043+0000 7f3dcdd9b700 1 --2- 172.21.15.23:0/1969104724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dd0101aa0 0x7f3dd0197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:37.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.043+0000 7f3dced9d700 1 --2- 172.21.15.23:0/1969104724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd0107140 0x7f3dd019bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:37.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.044+0000 7f3dce59c700 1 -- 172.21.15.23:0/1969104724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd0107140 msgr2=0x7f3dd019bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:37.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.044+0000 7f3dce59c700 1 --2- 172.21.15.23:0/1969104724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd0107140 0x7f3dd019bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:37.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.044+0000 7f3dce59c700 1 -- 172.21.15.23:0/1969104724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dd0101aa0 msgr2=0x7f3dd0197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:37.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.044+0000 7f3dce59c700 1 --2- 172.21.15.23:0/1969104724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dd0101aa0 0x7f3dd0197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:37.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.044+0000 7f3dce59c700 1 -- 172.21.15.23:0/1969104724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc00095f0 con 0x7f3dd0101100 2024-03-24T10:38:37.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.044+0000 7f3dce59c700 1 --2- 172.21.15.23:0/1969104724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dd0101100 0x7f3dd01976a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3db800eab0 tx=0x7f3db800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:37.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.044+0000 7f3dc77fe700 1 -- 172.21.15.23:0/1969104724 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8013070 con 0x7f3dd0101100 2024-03-24T10:38:37.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.044+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/1969104724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dd019cb40 con 0x7f3dd0101100 2024-03-24T10:38:37.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.044+0000 7f3dc77fe700 1 -- 172.21.15.23:0/1969104724 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3db8017410 con 0x7f3dd0101100 2024-03-24T10:38:37.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.044+0000 7f3dc77fe700 1 -- 172.21.15.23:0/1969104724 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8009790 con 0x7f3dd0101100 2024-03-24T10:38:37.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.044+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/1969104724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dd006a080 con 0x7f3dd0101100 2024-03-24T10:38:37.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.045+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/1969104724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3dd01027f0 con 0x7f3dd0101100 2024-03-24T10:38:37.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.046+0000 7f3dc77fe700 1 -- 172.21.15.23:0/1969104724 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f3db8017910 con 0x7f3dd0101100 2024-03-24T10:38:37.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.046+0000 7f3dc77fe700 1 --2- 172.21.15.23:0/1969104724 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3dbc0a21e0 0x7f3dbc0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:37.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.047+0000 7f3dcdd9b700 1 --2- 172.21.15.23:0/1969104724 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3dbc0a21e0 0x7f3dbc0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:37.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.047+0000 7f3dc77fe700 1 -- 172.21.15.23:0/1969104724 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f3db80cfae0 con 0x7f3dd0101100 2024-03-24T10:38:37.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.047+0000 7f3dc77fe700 1 --2- 172.21.15.23:0/1969104724 >> [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315] conn(0x7f3dbc0ab9c0 0x7f3dbc0adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:37.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.047+0000 7f3dc77fe700 1 -- 172.21.15.23:0/1969104724 --> [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3dbc0ae490 con 0x7f3dbc0ab9c0 2024-03-24T10:38:37.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.047+0000 7f3dc77fe700 1 -- 172.21.15.23:0/1969104724 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f3db800bea0 con 0x7f3dd0101100 2024-03-24T10:38:37.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.047+0000 7f3dced9d700 1 --2- 172.21.15.23:0/1969104724 >> [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315] conn(0x7f3dbc0ab9c0 0x7f3dbc0adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:37.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.047+0000 7f3dcdd9b700 1 --2- 172.21.15.23:0/1969104724 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3dbc0a21e0 0x7f3dbc0a46a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3dd00786a0 tx=0x7f3dc0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:37.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.047+0000 7f3dced9d700 1 --2- 172.21.15.23:0/1969104724 >> [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315] conn(0x7f3dbc0ab9c0 0x7f3dbc0adde0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:37.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.048+0000 7f3dc77fe700 1 -- 172.21.15.23:0/1969104724 <== osd.6 v2:172.21.15.23:6818/1181430315 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3dbc0ae490 con 0x7f3dbc0ab9c0 2024-03-24T10:38:37.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.064+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/1969104724 --> [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3dd0111480 con 0x7f3dbc0ab9c0 2024-03-24T10:38:37.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.064+0000 7f3dc77fe700 1 -- 172.21.15.23:0/1969104724 <== osd.6 v2:172.21.15.23:6818/1181430315 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3dd0111480 con 0x7f3dbc0ab9c0 2024-03-24T10:38:37.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.065+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/1969104724 >> [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315] conn(0x7f3dbc0ab9c0 msgr2=0x7f3dbc0adde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:37.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.065+0000 7f3dd4ea3700 1 --2- 172.21.15.23:0/1969104724 >> [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315] conn(0x7f3dbc0ab9c0 0x7f3dbc0adde0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:37.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.065+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/1969104724 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3dbc0a21e0 msgr2=0x7f3dbc0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:37.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.065+0000 7f3dd4ea3700 1 --2- 172.21.15.23:0/1969104724 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f3dbc0a21e0 0x7f3dbc0a46a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3dd00786a0 tx=0x7f3dc0005fd0 comp rx=0 tx=0).stop 2024-03-24T10:38:37.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.065+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/1969104724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dd0101100 msgr2=0x7f3dd01976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:37.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.065+0000 7f3dd4ea3700 1 --2- 172.21.15.23:0/1969104724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dd0101100 0x7f3dd01976a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3db800eab0 tx=0x7f3db800edc0 comp rx=0 tx=0).stop 2024-03-24T10:38:37.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.065+0000 7f3dced9d700 1 -- 172.21.15.23:0/1969104724 reap_dead start 2024-03-24T10:38:37.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.065+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/1969104724 shutdown_connections 2024-03-24T10:38:37.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.065+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/1969104724 >> 172.21.15.23:0/1969104724 conn(0x7f3dd0078030 msgr2=0x7f3dd00ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:37.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.065+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/1969104724 shutdown_connections 2024-03-24T10:38:37.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:37.065+0000 7f3dd4ea3700 1 -- 172.21.15.23:0/1969104724 wait complete. 2024-03-24T10:38:38.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:38 smithi023 ceph-mon[32466]: pgmap v296: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-24T10:38:38.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:38 smithi117 ceph-mon[36368]: pgmap v296: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-24T10:38:38.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:38 smithi116 ceph-mon[36248]: pgmap v296: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-24T10:38:40.248 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:40 smithi023 ceph-mon[32466]: pgmap v297: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:38:40.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:40 smithi117 ceph-mon[36368]: pgmap v297: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:38:40.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:40 smithi116 ceph-mon[36248]: pgmap v297: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:38:41.321 INFO:teuthology.orchestra.run.smithi023.stdout:197568495659 2024-03-24T10:38:41.321 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T10:38:42.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:42 smithi023 ceph-mon[32466]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T10:38:42.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:42 smithi117 ceph-mon[36368]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T10:38:42.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:42 smithi116 ceph-mon[36248]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T10:38:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.664+0000 7f285b0e1700 1 -- 172.21.15.23:0/4016770681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2854108320 msgr2=0x7f2854108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.664+0000 7f285b0e1700 1 --2- 172.21.15.23:0/4016770681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2854108320 0x7f2854108700 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2848009990 tx=0x7f2848009ca0 comp rx=0 tx=0).stop 2024-03-24T10:38:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.664+0000 7f285b0e1700 1 -- 172.21.15.23:0/4016770681 shutdown_connections 2024-03-24T10:38:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.664+0000 7f285b0e1700 1 --2- 172.21.15.23:0/4016770681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2854102c70 0x7f285410f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.664+0000 7f285b0e1700 1 --2- 172.21.15.23:0/4016770681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2854108320 0x7f2854108700 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.664+0000 7f285b0e1700 1 --2- 172.21.15.23:0/4016770681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28541022b0 0x7f2854102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.664+0000 7f285b0e1700 1 -- 172.21.15.23:0/4016770681 >> 172.21.15.23:0/4016770681 conn(0x7f28540792d0 msgr2=0x7f28540796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.664+0000 7f285b0e1700 1 -- 172.21.15.23:0/4016770681 shutdown_connections 2024-03-24T10:38:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.664+0000 7f285b0e1700 1 -- 172.21.15.23:0/4016770681 wait complete. 2024-03-24T10:38:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285b0e1700 1 Processor -- start 2024-03-24T10:38:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285b0e1700 1 -- start start 2024-03-24T10:38:42.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285b0e1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28541022b0 0x7f285419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:42.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285b0e1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2854102c70 0x7f285419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:42.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285b0e1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2854108320 0x7f2854198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:42.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285b0e1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2854198c80 con 0x7f2854108320 2024-03-24T10:38:42.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285b0e1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2854198dc0 con 0x7f28541022b0 2024-03-24T10:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285b0e1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2854198f80 con 0x7f2854102c70 2024-03-24T10:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285967e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2854108320 0x7f2854198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285967e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2854108320 0x7f2854198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54652/0 (socket says 172.21.15.23:54652) 2024-03-24T10:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285967e700 1 -- 172.21.15.23:0/3292745634 learned_addr learned my addr 172.21.15.23:0/3292745634 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f2858e7d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28541022b0 0x7f285419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f2853fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2854102c70 0x7f285419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285967e700 1 -- 172.21.15.23:0/3292745634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2854102c70 msgr2=0x7f285419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285967e700 1 --2- 172.21.15.23:0/3292745634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2854102c70 0x7f285419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285967e700 1 -- 172.21.15.23:0/3292745634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28541022b0 msgr2=0x7f285419e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285967e700 1 --2- 172.21.15.23:0/3292745634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28541022b0 0x7f285419e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285967e700 1 -- 172.21.15.23:0/3292745634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28480095f0 con 0x7f2854108320 2024-03-24T10:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.665+0000 7f285967e700 1 --2- 172.21.15.23:0/3292745634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2854108320 0x7f2854198620 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f284c00ef30 tx=0x7f284c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.666+0000 7f2851ffb700 1 -- 172.21.15.23:0/3292745634 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c018070 con 0x7f2854108320 2024-03-24T10:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.666+0000 7f2851ffb700 1 -- 172.21.15.23:0/3292745634 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f284c00f040 con 0x7f2854108320 2024-03-24T10:38:42.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.666+0000 7f285b0e1700 1 -- 172.21.15.23:0/3292745634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2854199200 con 0x7f2854108320 2024-03-24T10:38:42.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.666+0000 7f2851ffb700 1 -- 172.21.15.23:0/3292745634 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c00cd50 con 0x7f2854108320 2024-03-24T10:38:42.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.666+0000 7f285b0e1700 1 -- 172.21.15.23:0/3292745634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2854074600 con 0x7f2854108320 2024-03-24T10:38:42.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.667+0000 7f285b0e1700 1 -- 172.21.15.23:0/3292745634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2854112590 con 0x7f2854108320 2024-03-24T10:38:42.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.668+0000 7f2851ffb700 1 -- 172.21.15.23:0/3292745634 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f284c007500 con 0x7f2854108320 2024-03-24T10:38:42.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.668+0000 7f2851ffb700 1 --2- 172.21.15.23:0/3292745634 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f28440a2260 0x7f28440a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:42.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.668+0000 7f2858e7d700 1 --2- 172.21.15.23:0/3292745634 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f28440a2260 0x7f28440a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:42.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.668+0000 7f2851ffb700 1 -- 172.21.15.23:0/3292745634 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f284c0ce1d0 con 0x7f2854108320 2024-03-24T10:38:42.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.668+0000 7f2851ffb700 1 --2- 172.21.15.23:0/3292745634 >> [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563] conn(0x7f28440aba40 0x7f28440ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:42.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.668+0000 7f2851ffb700 1 -- 172.21.15.23:0/3292745634 --> [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f28440ae510 con 0x7f28440aba40 2024-03-24T10:38:42.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.668+0000 7f2851ffb700 1 -- 172.21.15.23:0/3292745634 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f284c0ce540 con 0x7f2854108320 2024-03-24T10:38:42.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.669+0000 7f2858e7d700 1 --2- 172.21.15.23:0/3292745634 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f28440a2260 0x7f28440a4720 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2848006010 tx=0x7f284800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:42.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.669+0000 7f2853fff700 1 --2- 172.21.15.23:0/3292745634 >> [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563] conn(0x7f28440aba40 0x7f28440ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:42.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.669+0000 7f2853fff700 1 --2- 172.21.15.23:0/3292745634 >> [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563] conn(0x7f28440aba40 0x7f28440ade60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:42.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.670+0000 7f2851ffb700 1 -- 172.21.15.23:0/3292745634 <== osd.8 v2:172.21.15.117:6816/3633840563 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f28440ae510 con 0x7f28440aba40 2024-03-24T10:38:42.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.686+0000 7f285b0e1700 1 -- 172.21.15.23:0/3292745634 --> [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2854062910 con 0x7f28440aba40 2024-03-24T10:38:42.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.687+0000 7f2851ffb700 1 -- 172.21.15.23:0/3292745634 <== osd.8 v2:172.21.15.117:6816/3633840563 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2854062910 con 0x7f28440aba40 2024-03-24T10:38:42.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.687+0000 7f285b0e1700 1 -- 172.21.15.23:0/3292745634 >> [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563] conn(0x7f28440aba40 msgr2=0x7f28440ade60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:42.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.687+0000 7f285b0e1700 1 --2- 172.21.15.23:0/3292745634 >> [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563] conn(0x7f28440aba40 0x7f28440ade60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:42.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.687+0000 7f285b0e1700 1 -- 172.21.15.23:0/3292745634 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f28440a2260 msgr2=0x7f28440a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:42.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.687+0000 7f285b0e1700 1 --2- 172.21.15.23:0/3292745634 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f28440a2260 0x7f28440a4720 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2848006010 tx=0x7f284800b280 comp rx=0 tx=0).stop 2024-03-24T10:38:42.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.687+0000 7f285b0e1700 1 -- 172.21.15.23:0/3292745634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2854108320 msgr2=0x7f2854198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:42.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.687+0000 7f285b0e1700 1 --2- 172.21.15.23:0/3292745634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2854108320 0x7f2854198620 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f284c00ef30 tx=0x7f284c00c3a0 comp rx=0 tx=0).stop 2024-03-24T10:38:42.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.687+0000 7f285967e700 1 -- 172.21.15.23:0/3292745634 reap_dead start 2024-03-24T10:38:42.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.687+0000 7f285b0e1700 1 -- 172.21.15.23:0/3292745634 shutdown_connections 2024-03-24T10:38:42.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.687+0000 7f285b0e1700 1 -- 172.21.15.23:0/3292745634 >> 172.21.15.23:0/3292745634 conn(0x7f28540792d0 msgr2=0x7f2854100ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:42.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.687+0000 7f285b0e1700 1 -- 172.21.15.23:0/3292745634 shutdown_connections 2024-03-24T10:38:42.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.687+0000 7f285b0e1700 1 -- 172.21.15.23:0/3292745634 wait complete. 2024-03-24T10:38:43.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.852+0000 7f33233c1700 1 -- 172.21.15.23:0/3111176163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f331c102c10 msgr2=0x7f331c10f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:43.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.852+0000 7f33233c1700 1 --2- 172.21.15.23:0/3111176163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f331c102c10 0x7f331c10f050 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f331400b0d0 tx=0x7f331400b3e0 comp rx=0 tx=0).stop 2024-03-24T10:38:43.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.852+0000 7f33233c1700 1 -- 172.21.15.23:0/3111176163 shutdown_connections 2024-03-24T10:38:43.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.852+0000 7f33233c1700 1 --2- 172.21.15.23:0/3111176163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f331c102c10 0x7f331c10f050 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:43.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.852+0000 7f33233c1700 1 --2- 172.21.15.23:0/3111176163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f331c108230 0x7f331c108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:43.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.852+0000 7f33233c1700 1 --2- 172.21.15.23:0/3111176163 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f331c102250 0x7f331c1026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:43.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.852+0000 7f33233c1700 1 -- 172.21.15.23:0/3111176163 >> 172.21.15.23:0/3111176163 conn(0x7f331c079330 msgr2=0x7f331c079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:43.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.853+0000 7f33233c1700 1 -- 172.21.15.23:0/3111176163 shutdown_connections 2024-03-24T10:38:43.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.853+0000 7f33233c1700 1 -- 172.21.15.23:0/3111176163 wait complete. 2024-03-24T10:38:43.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.853+0000 7f33233c1700 1 Processor -- start 2024-03-24T10:38:43.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.853+0000 7f33233c1700 1 -- start start 2024-03-24T10:38:43.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.853+0000 7f33233c1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f331c102250 0x7f331c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:43.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.853+0000 7f33233c1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f331c102c10 0x7f331c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:43.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.853+0000 7f33233c1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f331c108230 0x7f331c19d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:43.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.853+0000 7f33233c1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f331c19d620 con 0x7f331c108230 2024-03-24T10:38:43.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.853+0000 7f33233c1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f331c19d790 con 0x7f331c102c10 2024-03-24T10:38:43.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.853+0000 7f33233c1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f331c19d950 con 0x7f331c102250 2024-03-24T10:38:43.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.853+0000 7f332195e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f331c108230 0x7f331c19d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:43.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f332195e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f331c108230 0x7f331c19d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54668/0 (socket says 172.21.15.23:54668) 2024-03-24T10:38:43.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f332095c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f331c102c10 0x7f331c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:43.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f332195e700 1 -- 172.21.15.23:0/2156674109 learned_addr learned my addr 172.21.15.23:0/2156674109 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:43.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f332115d700 1 --2- 172.21.15.23:0/2156674109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f331c102250 0x7f331c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:43.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f332195e700 1 -- 172.21.15.23:0/2156674109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f331c102250 msgr2=0x7f331c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:43.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f332195e700 1 --2- 172.21.15.23:0/2156674109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f331c102250 0x7f331c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:43.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f332195e700 1 -- 172.21.15.23:0/2156674109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f331c102c10 msgr2=0x7f331c198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:43.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f332195e700 1 --2- 172.21.15.23:0/2156674109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f331c102c10 0x7f331c198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:43.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f332195e700 1 -- 172.21.15.23:0/2156674109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3314009d00 con 0x7f331c108230 2024-03-24T10:38:43.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f332095c700 1 --2- 172.21.15.23:0/2156674109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f331c102c10 0x7f331c198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:43.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f332195e700 1 --2- 172.21.15.23:0/2156674109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f331c108230 0x7f331c19d010 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f331400d010 tx=0x7f331400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:43.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f33127fc700 1 -- 172.21.15.23:0/2156674109 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f331400e040 con 0x7f331c108230 2024-03-24T10:38:43.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f33233c1700 1 -- 172.21.15.23:0/2156674109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f331c19dbd0 con 0x7f331c108230 2024-03-24T10:38:43.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f33127fc700 1 -- 172.21.15.23:0/2156674109 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33140079d0 con 0x7f331c108230 2024-03-24T10:38:43.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f33127fc700 1 -- 172.21.15.23:0/2156674109 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f331401d6e0 con 0x7f331c108230 2024-03-24T10:38:43.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.854+0000 7f33233c1700 1 -- 172.21.15.23:0/2156674109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f331c1a5490 con 0x7f331c108230 2024-03-24T10:38:43.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.855+0000 7f33233c1700 1 -- 172.21.15.23:0/2156674109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f331c19dea0 con 0x7f331c108230 2024-03-24T10:38:43.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.856+0000 7f33127fc700 1 -- 172.21.15.23:0/2156674109 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f3314003a00 con 0x7f331c108230 2024-03-24T10:38:43.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.857+0000 7f33127fc700 1 --2- 172.21.15.23:0/2156674109 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f33080a21e0 0x7f33080a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:43.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.857+0000 7f332115d700 1 --2- 172.21.15.23:0/2156674109 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f33080a21e0 0x7f33080a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:43.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.857+0000 7f33127fc700 1 -- 172.21.15.23:0/2156674109 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f33140cec20 con 0x7f331c108230 2024-03-24T10:38:43.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.857+0000 7f33127fc700 1 --2- 172.21.15.23:0/2156674109 >> [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549] conn(0x7f33080ab9c0 0x7f33080adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:43.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.857+0000 7f33127fc700 1 -- 172.21.15.23:0/2156674109 --> [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f33080ae490 con 0x7f33080ab9c0 2024-03-24T10:38:43.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.857+0000 7f33127fc700 1 -- 172.21.15.23:0/2156674109 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f33140cf0c0 con 0x7f331c108230 2024-03-24T10:38:43.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.857+0000 7f332095c700 1 --2- 172.21.15.23:0/2156674109 >> [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549] conn(0x7f33080ab9c0 0x7f33080adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:43.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.857+0000 7f332115d700 1 --2- 172.21.15.23:0/2156674109 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f33080a21e0 0x7f33080a46a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3318009c60 tx=0x7f3318009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:43.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.857+0000 7f332095c700 1 --2- 172.21.15.23:0/2156674109 >> [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549] conn(0x7f33080ab9c0 0x7f33080adde0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:43.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.858+0000 7f33127fc700 1 -- 172.21.15.23:0/2156674109 <== osd.3 v2:172.21.15.23:6810/1780984549 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f33080ae490 con 0x7f33080ab9c0 2024-03-24T10:38:43.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.875+0000 7f33233c1700 1 -- 172.21.15.23:0/2156674109 --> [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f331c112540 con 0x7f33080ab9c0 2024-03-24T10:38:43.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.875+0000 7f33127fc700 1 -- 172.21.15.23:0/2156674109 <== osd.3 v2:172.21.15.23:6810/1780984549 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f331c112540 con 0x7f33080ab9c0 2024-03-24T10:38:43.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.875+0000 7f33233c1700 1 -- 172.21.15.23:0/2156674109 >> [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549] conn(0x7f33080ab9c0 msgr2=0x7f33080adde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:43.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.875+0000 7f33233c1700 1 --2- 172.21.15.23:0/2156674109 >> [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549] conn(0x7f33080ab9c0 0x7f33080adde0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:43.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.875+0000 7f33233c1700 1 -- 172.21.15.23:0/2156674109 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f33080a21e0 msgr2=0x7f33080a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:43.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.875+0000 7f33233c1700 1 --2- 172.21.15.23:0/2156674109 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f33080a21e0 0x7f33080a46a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3318009c60 tx=0x7f3318009260 comp rx=0 tx=0).stop 2024-03-24T10:38:43.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.875+0000 7f33233c1700 1 -- 172.21.15.23:0/2156674109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f331c108230 msgr2=0x7f331c19d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:43.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.875+0000 7f33233c1700 1 --2- 172.21.15.23:0/2156674109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f331c108230 0x7f331c19d010 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f331400d010 tx=0x7f331400bd50 comp rx=0 tx=0).stop 2024-03-24T10:38:43.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.875+0000 7f332195e700 1 -- 172.21.15.23:0/2156674109 reap_dead start 2024-03-24T10:38:43.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.875+0000 7f33233c1700 1 -- 172.21.15.23:0/2156674109 shutdown_connections 2024-03-24T10:38:43.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.875+0000 7f33233c1700 1 -- 172.21.15.23:0/2156674109 >> 172.21.15.23:0/2156674109 conn(0x7f331c079330 msgr2=0x7f331c100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:43.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.876+0000 7f33233c1700 1 -- 172.21.15.23:0/2156674109 shutdown_connections 2024-03-24T10:38:43.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:42.876+0000 7f33233c1700 1 -- 172.21.15.23:0/2156674109 wait complete. 2024-03-24T10:38:43.662 INFO:teuthology.orchestra.run.smithi023.stdout:253403070498 2024-03-24T10:38:43.663 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-24T10:38:44.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:44 smithi023 ceph-mon[32466]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-24T10:38:44.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:44 smithi117 ceph-mon[36368]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-24T10:38:44.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:44 smithi116 ceph-mon[36248]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-24T10:38:44.918 INFO:teuthology.orchestra.run.smithi023.stdout:107374182459 2024-03-24T10:38:44.918 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T10:38:46.291 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:46 smithi023 ceph-mon[32466]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T10:38:46.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:46 smithi117 ceph-mon[36368]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T10:38:46.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:46 smithi116 ceph-mon[36248]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T10:38:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.134+0000 7f4e4ff35700 1 -- 172.21.15.23:0/1657880296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e48101d80 msgr2=0x7f4e48105dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.134+0000 7f4e4ff35700 1 --2- 172.21.15.23:0/1657880296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e48101d80 0x7f4e48105dd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4e38009990 tx=0x7f4e38009ca0 comp rx=0 tx=0).stop 2024-03-24T10:38:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.134+0000 7f4e4ff35700 1 -- 172.21.15.23:0/1657880296 shutdown_connections 2024-03-24T10:38:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.134+0000 7f4e4ff35700 1 --2- 172.21.15.23:0/1657880296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e481013d0 0x7f4e481017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.134+0000 7f4e4ff35700 1 --2- 172.21.15.23:0/1657880296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e481064b0 0x7f4e4810e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.134+0000 7f4e4ff35700 1 --2- 172.21.15.23:0/1657880296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e48101d80 0x7f4e48105dd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.134+0000 7f4e4ff35700 1 -- 172.21.15.23:0/1657880296 >> 172.21.15.23:0/1657880296 conn(0x7f4e48078230 msgr2=0x7f4e48078640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.134+0000 7f4e4ff35700 1 -- 172.21.15.23:0/1657880296 shutdown_connections 2024-03-24T10:38:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.134+0000 7f4e4ff35700 1 -- 172.21.15.23:0/1657880296 wait complete. 2024-03-24T10:38:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.135+0000 7f4e4ff35700 1 Processor -- start 2024-03-24T10:38:46.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.135+0000 7f4e4ff35700 1 -- start start 2024-03-24T10:38:46.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.135+0000 7f4e4ff35700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e481013d0 0x7f4e481996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:46.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.135+0000 7f4e4ff35700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e48101d80 0x7f4e48199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:46.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.135+0000 7f4e4ff35700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e481064b0 0x7f4e481a4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:46.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.135+0000 7f4e4ff35700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4819a460 con 0x7f4e481013d0 2024-03-24T10:38:46.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.135+0000 7f4e4ff35700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4e481a4c90 con 0x7f4e48101d80 2024-03-24T10:38:46.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.135+0000 7f4e4ff35700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4e481a4dd0 con 0x7f4e481064b0 2024-03-24T10:38:46.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.135+0000 7f4e4dcd1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e481013d0 0x7f4e481996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:46.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.135+0000 7f4e4d4d0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e48101d80 0x7f4e48199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:46.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.135+0000 7f4e4dcd1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e481013d0 0x7f4e481996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54688/0 (socket says 172.21.15.23:54688) 2024-03-24T10:38:46.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.135+0000 7f4e4d4d0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e48101d80 0x7f4e48199c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:43324/0 (socket says 172.21.15.23:43324) 2024-03-24T10:38:46.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.135+0000 7f4e4dcd1700 1 -- 172.21.15.23:0/852567906 learned_addr learned my addr 172.21.15.23:0/852567906 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:46.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.135+0000 7f4e4e4d2700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e481064b0 0x7f4e481a4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:46.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.136+0000 7f4e4dcd1700 1 -- 172.21.15.23:0/852567906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e481064b0 msgr2=0x7f4e481a4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.136+0000 7f4e4dcd1700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e481064b0 0x7f4e481a4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.136+0000 7f4e4dcd1700 1 -- 172.21.15.23:0/852567906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e48101d80 msgr2=0x7f4e48199c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.136+0000 7f4e4dcd1700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e48101d80 0x7f4e48199c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.136+0000 7f4e4dcd1700 1 -- 172.21.15.23:0/852567906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e380095f0 con 0x7f4e481013d0 2024-03-24T10:38:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.136+0000 7f4e4d4d0700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e48101d80 0x7f4e48199c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.136+0000 7f4e4e4d2700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e481064b0 0x7f4e481a4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.136+0000 7f4e4dcd1700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e481013d0 0x7f4e481996d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400e9d0 tx=0x7f4e4400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:46.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.136+0000 7f4e3effd700 1 -- 172.21.15.23:0/852567906 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e44009790 con 0x7f4e481013d0 2024-03-24T10:38:46.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.136+0000 7f4e4ff35700 1 -- 172.21.15.23:0/852567906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e481a4fd0 con 0x7f4e481013d0 2024-03-24T10:38:46.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.136+0000 7f4e3effd700 1 -- 172.21.15.23:0/852567906 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e4400ee70 con 0x7f4e481013d0 2024-03-24T10:38:46.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.136+0000 7f4e3effd700 1 -- 172.21.15.23:0/852567906 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e44018680 con 0x7f4e481013d0 2024-03-24T10:38:46.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.136+0000 7f4e4ff35700 1 -- 172.21.15.23:0/852567906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e481a5470 con 0x7f4e481013d0 2024-03-24T10:38:46.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.137+0000 7f4e4ff35700 1 -- 172.21.15.23:0/852567906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4e48111500 con 0x7f4e481013d0 2024-03-24T10:38:46.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.138+0000 7f4e3effd700 1 -- 172.21.15.23:0/852567906 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f4e4400cbd0 con 0x7f4e481013d0 2024-03-24T10:38:46.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.138+0000 7f4e3effd700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4e340a21e0 0x7f4e340a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:46.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.139+0000 7f4e4d4d0700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4e340a21e0 0x7f4e340a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:46.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.139+0000 7f4e3effd700 1 -- 172.21.15.23:0/852567906 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4e44014070 con 0x7f4e481013d0 2024-03-24T10:38:46.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.139+0000 7f4e3effd700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153] conn(0x7f4e340ab9c0 0x7f4e340adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:46.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.139+0000 7f4e3effd700 1 -- 172.21.15.23:0/852567906 --> [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4e340ae490 con 0x7f4e340ab9c0 2024-03-24T10:38:46.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.139+0000 7f4e3effd700 1 -- 172.21.15.23:0/852567906 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f4e440ce5e0 con 0x7f4e481013d0 2024-03-24T10:38:46.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.139+0000 7f4e4e4d2700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153] conn(0x7f4e340ab9c0 0x7f4e340adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:46.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.139+0000 7f4e4d4d0700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4e340a21e0 0x7f4e340a46a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4e4806b440 tx=0x7f4e38005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:46.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.139+0000 7f4e4e4d2700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153] conn(0x7f4e340ab9c0 0x7f4e340adde0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:46.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.140+0000 7f4e3effd700 1 -- 172.21.15.23:0/852567906 <== osd.11 v2:172.21.15.117:6824/3654755153 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4e340ae490 con 0x7f4e340ab9c0 2024-03-24T10:38:46.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.156+0000 7f4e4ff35700 1 -- 172.21.15.23:0/852567906 --> [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4e48062890 con 0x7f4e340ab9c0 2024-03-24T10:38:46.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.156+0000 7f4e3effd700 1 -- 172.21.15.23:0/852567906 <== osd.11 v2:172.21.15.117:6824/3654755153 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4e48062890 con 0x7f4e340ab9c0 2024-03-24T10:38:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.157+0000 7f4e4ff35700 1 -- 172.21.15.23:0/852567906 >> [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153] conn(0x7f4e340ab9c0 msgr2=0x7f4e340adde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.157+0000 7f4e4ff35700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153] conn(0x7f4e340ab9c0 0x7f4e340adde0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.157+0000 7f4e4ff35700 1 -- 172.21.15.23:0/852567906 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4e340a21e0 msgr2=0x7f4e340a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.157+0000 7f4e4ff35700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f4e340a21e0 0x7f4e340a46a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4e4806b440 tx=0x7f4e38005fd0 comp rx=0 tx=0).stop 2024-03-24T10:38:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.157+0000 7f4e4ff35700 1 -- 172.21.15.23:0/852567906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e481013d0 msgr2=0x7f4e481996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.157+0000 7f4e4ff35700 1 --2- 172.21.15.23:0/852567906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e481013d0 0x7f4e481996d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400e9d0 tx=0x7f4e4400ece0 comp rx=0 tx=0).stop 2024-03-24T10:38:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.157+0000 7f4e4e4d2700 1 -- 172.21.15.23:0/852567906 reap_dead start 2024-03-24T10:38:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.157+0000 7f4e4ff35700 1 -- 172.21.15.23:0/852567906 shutdown_connections 2024-03-24T10:38:46.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.157+0000 7f4e4ff35700 1 -- 172.21.15.23:0/852567906 >> 172.21.15.23:0/852567906 conn(0x7f4e48078230 msgr2=0x7f4e480ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:46.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.157+0000 7f4e4ff35700 1 -- 172.21.15.23:0/852567906 shutdown_connections 2024-03-24T10:38:46.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:46.157+0000 7f4e4ff35700 1 -- 172.21.15.23:0/852567906 wait complete. 2024-03-24T10:38:47.424 INFO:teuthology.orchestra.run.smithi023.stdout:343597383701 2024-03-24T10:38:47.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-24T10:38:48.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.037+0000 7f9f471c7700 1 -- 172.21.15.23:0/1567214627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f40107580 msgr2=0x7f9f4010f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.037+0000 7f9f471c7700 1 --2- 172.21.15.23:0/1567214627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f40107580 0x7f9f4010f100 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800b0a0 tx=0x7f9f3800b3b0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.037+0000 7f9f471c7700 1 -- 172.21.15.23:0/1567214627 shutdown_connections 2024-03-24T10:38:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.037+0000 7f9f471c7700 1 --2- 172.21.15.23:0/1567214627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f40107580 0x7f9f4010f100 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.037+0000 7f9f471c7700 1 --2- 172.21.15.23:0/1567214627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f4006a9a0 0x7f9f4006ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.037+0000 7f9f471c7700 1 --2- 172.21.15.23:0/1567214627 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f4006b350 0x7f9f40106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.037+0000 7f9f471c7700 1 -- 172.21.15.23:0/1567214627 >> 172.21.15.23:0/1567214627 conn(0x7f9f40077150 msgr2=0x7f9f40077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.037+0000 7f9f471c7700 1 -- 172.21.15.23:0/1567214627 shutdown_connections 2024-03-24T10:38:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.037+0000 7f9f471c7700 1 -- 172.21.15.23:0/1567214627 wait complete. 2024-03-24T10:38:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.038+0000 7f9f471c7700 1 Processor -- start 2024-03-24T10:38:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.038+0000 7f9f471c7700 1 -- start start 2024-03-24T10:38:48.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.038+0000 7f9f471c7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f4006a9a0 0x7f9f4019e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:48.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.038+0000 7f9f471c7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f4006b350 0x7f9f4019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:48.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.038+0000 7f9f471c7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f40107580 0x7f9f401986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:48.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.038+0000 7f9f471c7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f9f40198e80 con 0x7f9f4006b350 2024-03-24T10:38:48.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.038+0000 7f9f471c7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9f40198ff0 con 0x7f9f40107580 2024-03-24T10:38:48.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.038+0000 7f9f471c7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9f401991b0 con 0x7f9f4006a9a0 2024-03-24T10:38:48.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.038+0000 7f9f3ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f4006b350 0x7f9f4019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:48.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.038+0000 7f9f44f63700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f4006a9a0 0x7f9f4019e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:48.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.038+0000 7f9f45764700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f40107580 0x7f9f401986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:48.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.038+0000 7f9f3ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f4006b350 0x7f9f4019eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51012/0 (socket says 172.21.15.23:51012) 2024-03-24T10:38:48.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.038+0000 7f9f3ffff700 1 -- 172.21.15.23:0/1823265132 learned_addr learned my addr 172.21.15.23:0/1823265132 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:48.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.039+0000 7f9f44f63700 1 -- 172.21.15.23:0/1823265132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f40107580 msgr2=0x7f9f401986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:48.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.039+0000 7f9f44f63700 1 --2- 172.21.15.23:0/1823265132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f40107580 0x7f9f401986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.039+0000 7f9f44f63700 1 -- 172.21.15.23:0/1823265132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f4006b350 msgr2=0x7f9f4019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:48.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.039+0000 7f9f44f63700 1 --2- 172.21.15.23:0/1823265132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f4006b350 0x7f9f4019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.039+0000 7f9f44f63700 1 -- 172.21.15.23:0/1823265132 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f38009d00 con 0x7f9f4006a9a0 2024-03-24T10:38:48.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.039+0000 7f9f3ffff700 1 --2- 172.21.15.23:0/1823265132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f4006b350 0x7f9f4019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:48.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.039+0000 7f9f44f63700 1 --2- 172.21.15.23:0/1823265132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f4006a9a0 0x7f9f4019e5b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009c50 tx=0x7f9f34009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:48.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.039+0000 7f9f3dffb700 1 -- 172.21.15.23:0/1823265132 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f34015070 con 0x7f9f4006a9a0 2024-03-24T10:38:48.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.039+0000 7f9f3dffb700 1 -- 172.21.15.23:0/1823265132 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f34019410 con 0x7f9f4006a9a0 2024-03-24T10:38:48.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.039+0000 7f9f471c7700 1 -- 172.21.15.23:0/1823265132 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f40199430 con 0x7f9f4006a9a0 2024-03-24T10:38:48.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.039+0000 7f9f3dffb700 1 -- 172.21.15.23:0/1823265132 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f34018610 con 0x7f9f4006a9a0 2024-03-24T10:38:48.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.039+0000 7f9f471c7700 1 -- 172.21.15.23:0/1823265132 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f40101350 con 0x7f9f4006a9a0 2024-03-24T10:38:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.040+0000 7f9f471c7700 1 -- 172.21.15.23:0/1823265132 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9f40103930 con 0x7f9f4006a9a0 2024-03-24T10:38:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.041+0000 7f9f3dffb700 1 -- 172.21.15.23:0/1823265132 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f9f34019910 con 0x7f9f4006a9a0 2024-03-24T10:38:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.042+0000 7f9f3dffb700 1 --2- 172.21.15.23:0/1823265132 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9f300a2260 0x7f9f300a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.042+0000 7f9f3ffff700 1 --2- 172.21.15.23:0/1823265132 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9f300a2260 0x7f9f300a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.042+0000 7f9f3dffb700 1 -- 172.21.15.23:0/1823265132 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f9f340d1b80 con 0x7f9f4006a9a0 2024-03-24T10:38:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.042+0000 7f9f3dffb700 1 --2- 172.21.15.23:0/1823265132 >> [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037] conn(0x7f9f300aba40 0x7f9f300ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.042+0000 7f9f3dffb700 1 -- 172.21.15.23:0/1823265132 --> [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f300ae510 con 0x7f9f300aba40 2024-03-24T10:38:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.042+0000 7f9f3dffb700 1 -- 172.21.15.23:0/1823265132 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f9f340d1f30 con 0x7f9f4006a9a0 2024-03-24T10:38:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.043+0000 7f9f3ffff700 1 --2- 172.21.15.23:0/1823265132 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9f300a2260 0x7f9f300a4720 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9f40199e40 tx=0x7f9f2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.043+0000 7f9f45764700 1 --2- 172.21.15.23:0/1823265132 >> [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037] conn(0x7f9f300aba40 0x7f9f300ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.043+0000 7f9f45764700 1 --2- 172.21.15.23:0/1823265132 >> [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037] conn(0x7f9f300aba40 0x7f9f300ade60 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.044+0000 7f9f3dffb700 1 -- 172.21.15.23:0/1823265132 <== osd.10 v2:172.21.15.116:6824/2413381037 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9f300ae510 con 0x7f9f300aba40 2024-03-24T10:38:48.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.060+0000 7f9f471c7700 1 -- 172.21.15.23:0/1823265132 --> [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9f401125a0 con 0x7f9f300aba40 2024-03-24T10:38:48.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.061+0000 7f9f3dffb700 1 -- 172.21.15.23:0/1823265132 <== osd.10 v2:172.21.15.116:6824/2413381037 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9f401125a0 con 0x7f9f300aba40 2024-03-24T10:38:48.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.061+0000 7f9f471c7700 1 -- 172.21.15.23:0/1823265132 >> [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037] conn(0x7f9f300aba40 msgr2=0x7f9f300ade60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:48.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.061+0000 7f9f471c7700 1 --2- 172.21.15.23:0/1823265132 >> [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037] conn(0x7f9f300aba40 0x7f9f300ade60 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.061+0000 7f9f471c7700 1 -- 172.21.15.23:0/1823265132 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9f300a2260 msgr2=0x7f9f300a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:48.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.061+0000 7f9f471c7700 1 --2- 172.21.15.23:0/1823265132 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9f300a2260 0x7f9f300a4720 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9f40199e40 tx=0x7f9f2c009260 comp rx=0 tx=0).stop 2024-03-24T10:38:48.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.061+0000 7f9f471c7700 1 -- 172.21.15.23:0/1823265132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f4006a9a0 msgr2=0x7f9f4019e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:48.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.061+0000 7f9f471c7700 1 --2- 172.21.15.23:0/1823265132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f4006a9a0 0x7f9f4019e5b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009c50 tx=0x7f9f34009f60 comp rx=0 tx=0).stop 2024-03-24T10:38:48.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.061+0000 7f9f45764700 1 -- 172.21.15.23:0/1823265132 reap_dead start 2024-03-24T10:38:48.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.061+0000 7f9f471c7700 1 -- 172.21.15.23:0/1823265132 shutdown_connections 2024-03-24T10:38:48.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.061+0000 7f9f471c7700 1 -- 172.21.15.23:0/1823265132 >> 172.21.15.23:0/1823265132 conn(0x7f9f40077150 msgr2=0x7f9f400ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:48.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.061+0000 7f9f471c7700 1 -- 172.21.15.23:0/1823265132 shutdown_connections 2024-03-24T10:38:48.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.061+0000 7f9f471c7700 1 -- 172.21.15.23:0/1823265132 wait complete. 2024-03-24T10:38:48.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:48 smithi023 ceph-mon[32466]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:38:48.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:48 smithi117 ceph-mon[36368]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:38:48.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:48 smithi116 ceph-mon[36248]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:38:48.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.806+0000 7ff32d931700 1 -- 172.21.15.23:0/1647677527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff328100b30 msgr2=0x7ff328100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:48.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.806+0000 7ff32d931700 1 --2- 172.21.15.23:0/1647677527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff328100b30 0x7ff328100fb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff31c0099c0 tx=0x7ff31c009cd0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.806+0000 7ff32d931700 1 -- 172.21.15.23:0/1647677527 shutdown_connections 2024-03-24T10:38:48.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.806+0000 7ff32d931700 1 --2- 172.21.15.23:0/1647677527 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3281014f0 0x7ff32810d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.806+0000 7ff32d931700 1 --2- 172.21.15.23:0/1647677527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff328106910 0x7ff328106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.806+0000 7ff32d931700 1 --2- 172.21.15.23:0/1647677527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff328100b30 0x7ff328100fb0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.806+0000 7ff32d931700 1 -- 172.21.15.23:0/1647677527 >> 172.21.15.23:0/1647677527 conn(0x7ff328076430 msgr2=0x7ff328076840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:48.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.806+0000 7ff32d931700 1 -- 172.21.15.23:0/1647677527 shutdown_connections 2024-03-24T10:38:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.806+0000 7ff32d931700 1 -- 172.21.15.23:0/1647677527 wait complete. 2024-03-24T10:38:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.806+0000 7ff32d931700 1 Processor -- start 2024-03-24T10:38:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.806+0000 7ff32d931700 1 -- start start 2024-03-24T10:38:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff32d931700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff328100b30 0x7ff328198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff32d931700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3281014f0 0x7ff328198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff32d931700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff328106910 0x7ff32819cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff32d931700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff32819d500 con 0x7ff328106910 2024-03-24T10:38:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff32d931700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff32819d670 con 0x7ff3281014f0 2024-03-24T10:38:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff32d931700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff32819d830 con 0x7ff328100b30 2024-03-24T10:38:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff3277fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff328106910 0x7ff32819cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff326ffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff328100b30 0x7ff328198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff326ffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff328100b30 0x7ff328198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:54974/0 (socket says 172.21.15.23:54974) 2024-03-24T10:38:48.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff3277fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff328106910 0x7ff32819cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51026/0 (socket says 172.21.15.23:51026) 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff326ffd700 1 -- 172.21.15.23:0/2852439670 learned_addr learned my addr 172.21.15.23:0/2852439670 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff3267fc700 1 --2- 172.21.15.23:0/2852439670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3281014f0 0x7ff328198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff326ffd700 1 -- 172.21.15.23:0/2852439670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3281014f0 msgr2=0x7ff328198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff326ffd700 1 --2- 172.21.15.23:0/2852439670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3281014f0 0x7ff328198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff326ffd700 1 -- 172.21.15.23:0/2852439670 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff328106910 msgr2=0x7ff32819cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff326ffd700 1 --2- 172.21.15.23:0/2852439670 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff328106910 0x7ff32819cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff326ffd700 1 -- 172.21.15.23:0/2852439670 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff31c0095f0 con 0x7ff328100b30 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff3277fe700 1 --2- 172.21.15.23:0/2852439670 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff328106910 0x7ff32819cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff326ffd700 1 --2- 172.21.15.23:0/2852439670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff328100b30 0x7ff328198620 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff31800d980 tx=0x7ff31800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.807+0000 7ff32c92f700 1 -- 172.21.15.23:0/2852439670 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff318004220 con 0x7ff328100b30 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.808+0000 7ff32c92f700 1 -- 172.21.15.23:0/2852439670 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff318007640 con 0x7ff328100b30 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.808+0000 7ff32d931700 1 -- 172.21.15.23:0/2852439670 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff32819db10 con 0x7ff328100b30 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.808+0000 7ff32c92f700 1 -- 172.21.15.23:0/2852439670 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31800f690 con 0x7ff328100b30 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.808+0000 7ff32d931700 1 -- 172.21.15.23:0/2852439670 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3281a5370 con 0x7ff328100b30 2024-03-24T10:38:48.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.808+0000 7ff32d931700 1 -- 172.21.15.23:0/2852439670 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff328101fa0 con 0x7ff328100b30 2024-03-24T10:38:48.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.810+0000 7ff32c92f700 1 -- 172.21.15.23:0/2852439670 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff318004410 con 0x7ff328100b30 2024-03-24T10:38:48.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.810+0000 7ff32c92f700 1 --2- 172.21.15.23:0/2852439670 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff3100a21e0 0x7ff3100a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:48.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.810+0000 7ff3267fc700 1 --2- 172.21.15.23:0/2852439670 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff3100a21e0 0x7ff3100a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:48.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.810+0000 7ff32c92f700 1 -- 172.21.15.23:0/2852439670 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7ff3180ce060 con 0x7ff328100b30 2024-03-24T10:38:48.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.810+0000 7ff32c92f700 1 --2- 172.21.15.23:0/2852439670 >> [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459] conn(0x7ff3100ab9c0 0x7ff3100adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:48.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.810+0000 7ff32c92f700 1 -- 172.21.15.23:0/2852439670 --> [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3100ae490 con 0x7ff3100ab9c0 2024-03-24T10:38:48.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.810+0000 7ff32c92f700 1 -- 172.21.15.23:0/2852439670 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7ff3180ce500 con 0x7ff328100b30 2024-03-24T10:38:48.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.811+0000 7ff3267fc700 1 --2- 172.21.15.23:0/2852439670 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff3100a21e0 0x7ff3100a46a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009990 tx=0x7ff31c005bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:48.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.811+0000 7ff3277fe700 1 --2- 172.21.15.23:0/2852439670 >> [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459] conn(0x7ff3100ab9c0 0x7ff3100adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:48.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.811+0000 7ff3277fe700 1 --2- 172.21.15.23:0/2852439670 >> [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459] conn(0x7ff3100ab9c0 0x7ff3100adde0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:48.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.812+0000 7ff32c92f700 1 -- 172.21.15.23:0/2852439670 <== osd.7 v2:172.21.15.116:6816/2721290459 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3100ae490 con 0x7ff3100ab9c0 2024-03-24T10:38:48.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.829+0000 7ff32d931700 1 -- 172.21.15.23:0/2852439670 --> [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff328111490 con 0x7ff3100ab9c0 2024-03-24T10:38:48.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.829+0000 7ff32c92f700 1 -- 172.21.15.23:0/2852439670 <== osd.7 v2:172.21.15.116:6816/2721290459 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff328111490 con 0x7ff3100ab9c0 2024-03-24T10:38:48.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.830+0000 7ff32d931700 1 -- 172.21.15.23:0/2852439670 >> [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459] conn(0x7ff3100ab9c0 msgr2=0x7ff3100adde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:48.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.830+0000 7ff32d931700 1 --2- 172.21.15.23:0/2852439670 >> [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459] conn(0x7ff3100ab9c0 0x7ff3100adde0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.830+0000 7ff32d931700 1 -- 172.21.15.23:0/2852439670 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff3100a21e0 msgr2=0x7ff3100a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:48.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.830+0000 7ff32d931700 1 --2- 172.21.15.23:0/2852439670 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff3100a21e0 0x7ff3100a46a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff31c009990 tx=0x7ff31c005bc0 comp rx=0 tx=0).stop 2024-03-24T10:38:48.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.830+0000 7ff32d931700 1 -- 172.21.15.23:0/2852439670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff328100b30 msgr2=0x7ff328198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:48.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.830+0000 7ff32d931700 1 --2- 172.21.15.23:0/2852439670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff328100b30 0x7ff328198620 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff31800d980 tx=0x7ff31800dc90 comp rx=0 tx=0).stop 2024-03-24T10:38:48.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.830+0000 7ff3277fe700 1 -- 172.21.15.23:0/2852439670 reap_dead start 2024-03-24T10:38:48.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.830+0000 7ff32d931700 1 -- 172.21.15.23:0/2852439670 shutdown_connections 2024-03-24T10:38:48.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.830+0000 7ff32d931700 1 -- 172.21.15.23:0/2852439670 >> 172.21.15.23:0/2852439670 conn(0x7ff328076430 msgr2=0x7ff32810e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:48.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.830+0000 7ff32d931700 1 -- 172.21.15.23:0/2852439670 shutdown_connections 2024-03-24T10:38:48.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:48.830+0000 7ff32d931700 1 -- 172.21.15.23:0/2852439670 wait complete. 2024-03-24T10:38:49.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.421+0000 7fa83303a700 1 -- 172.21.15.23:0/57044416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa82c102c10 msgr2=0x7fa82c10f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:49.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.421+0000 7fa83303a700 1 --2- 172.21.15.23:0/57044416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa82c102c10 0x7fa82c10f050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa82400b0d0 tx=0x7fa82400b3e0 comp rx=0 tx=0).stop 2024-03-24T10:38:49.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.421+0000 7fa83303a700 1 -- 172.21.15.23:0/57044416 shutdown_connections 2024-03-24T10:38:49.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.421+0000 7fa83303a700 1 --2- 172.21.15.23:0/57044416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa82c108230 0x7fa82c108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:49.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.421+0000 7fa83303a700 1 --2- 172.21.15.23:0/57044416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa82c102c10 0x7fa82c10f050 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:49.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.421+0000 7fa83303a700 1 --2- 172.21.15.23:0/57044416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa82c102250 0x7fa82c1026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:49.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.421+0000 7fa83303a700 1 -- 172.21.15.23:0/57044416 >> 172.21.15.23:0/57044416 conn(0x7fa82c079330 msgr2=0x7fa82c079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:49.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.421+0000 7fa83303a700 1 -- 172.21.15.23:0/57044416 shutdown_connections 2024-03-24T10:38:49.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.421+0000 7fa83303a700 1 -- 172.21.15.23:0/57044416 wait complete. 2024-03-24T10:38:49.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.421+0000 7fa83303a700 1 Processor -- start 2024-03-24T10:38:49.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa83303a700 1 -- start start 2024-03-24T10:38:49.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa83303a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa82c102250 0x7fa82c19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:49.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa83303a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa82c102c10 0x7fa82c19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:49.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa83303a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa82c108230 0x7fa82c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:49.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa83303a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c198c60 con 0x7fa82c102c10 2024-03-24T10:38:49.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa83303a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c198da0 con 0x7fa82c102250 2024-03-24T10:38:49.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa83303a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c198f60 con 0x7fa82c108230 2024-03-24T10:38:49.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa82bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa82c102c10 0x7fa82c19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:49.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa830dd6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa82c102250 0x7fa82c19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:49.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa82bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa82c102c10 0x7fa82c19e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51032/0 (socket says 172.21.15.23:51032) 2024-03-24T10:38:49.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa82bfff700 1 -- 172.21.15.23:0/3986966757 learned_addr learned my addr 172.21.15.23:0/3986966757 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:49.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa8315d7700 1 --2- 172.21.15.23:0/3986966757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa82c108230 0x7fa82c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:49.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa82bfff700 1 -- 172.21.15.23:0/3986966757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa82c108230 msgr2=0x7fa82c198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:49.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa82bfff700 1 --2- 172.21.15.23:0/3986966757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa82c108230 0x7fa82c198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:49.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa82bfff700 1 -- 172.21.15.23:0/3986966757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa82c102250 msgr2=0x7fa82c19e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:49.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa82bfff700 1 --2- 172.21.15.23:0/3986966757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa82c102250 0x7fa82c19e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:49.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa82bfff700 1 -- 172.21.15.23:0/3986966757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa824009d00 con 0x7fa82c102c10 2024-03-24T10:38:49.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa830dd6700 1 --2- 172.21.15.23:0/3986966757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa82c102250 0x7fa82c19e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:38:49.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.422+0000 7fa82bfff700 1 --2- 172.21.15.23:0/3986966757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa82c102c10 0x7fa82c19e980 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa81800e9a0 tx=0x7fa81800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:49.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.423+0000 7fa829ffb700 1 -- 172.21.15.23:0/3986966757 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818009790 con 0x7fa82c102c10 2024-03-24T10:38:49.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.423+0000 7fa829ffb700 1 -- 172.21.15.23:0/3986966757 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa818010430 con 0x7fa82c102c10 2024-03-24T10:38:49.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.423+0000 7fa83303a700 1 -- 172.21.15.23:0/3986966757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa82c199240 con 0x7fa82c102c10 2024-03-24T10:38:49.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.423+0000 7fa829ffb700 1 -- 172.21.15.23:0/3986966757 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818018680 con 0x7fa82c102c10 2024-03-24T10:38:49.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.423+0000 7fa83303a700 1 -- 172.21.15.23:0/3986966757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa82c1a54e0 con 0x7fa82c102c10 2024-03-24T10:38:49.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.424+0000 7fa83303a700 1 -- 172.21.15.23:0/3986966757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa82c1038e0 con 0x7fa82c102c10 2024-03-24T10:38:49.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.425+0000 7fa829ffb700 1 -- 172.21.15.23:0/3986966757 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fa818010910 con 0x7fa82c102c10 2024-03-24T10:38:49.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.425+0000 7fa829ffb700 1 --2- 172.21.15.23:0/3986966757 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa81c0a2260 0x7fa81c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:49.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.425+0000 7fa830dd6700 1 --2- 172.21.15.23:0/3986966757 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa81c0a2260 0x7fa81c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:49.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.425+0000 7fa829ffb700 1 -- 172.21.15.23:0/3986966757 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fa818014070 con 0x7fa82c102c10 2024-03-24T10:38:49.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.426+0000 7fa829ffb700 1 --2- 172.21.15.23:0/3986966757 >> [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126] conn(0x7fa81c0aba40 0x7fa81c0ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:49.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.426+0000 7fa829ffb700 1 -- 172.21.15.23:0/3986966757 --> [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa81c0ae510 con 0x7fa81c0aba40 2024-03-24T10:38:49.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.426+0000 7fa829ffb700 1 -- 172.21.15.23:0/3986966757 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fa8180ce8d0 con 0x7fa82c102c10 2024-03-24T10:38:49.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.426+0000 7fa830dd6700 1 --2- 172.21.15.23:0/3986966757 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa81c0a2260 0x7fa81c0a4720 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa820011490 tx=0x7fa820009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:49.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.426+0000 7fa8315d7700 1 --2- 172.21.15.23:0/3986966757 >> [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126] conn(0x7fa81c0aba40 0x7fa81c0ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:49.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.426+0000 7fa8315d7700 1 --2- 172.21.15.23:0/3986966757 >> [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126] conn(0x7fa81c0aba40 0x7fa81c0ade60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:49.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.427+0000 7fa829ffb700 1 -- 172.21.15.23:0/3986966757 <== osd.1 v2:172.21.15.116:6800/1598773126 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa81c0ae510 con 0x7fa81c0aba40 2024-03-24T10:38:49.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.443+0000 7fa83303a700 1 -- 172.21.15.23:0/3986966757 --> [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa82c112590 con 0x7fa81c0aba40 2024-03-24T10:38:49.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.444+0000 7fa829ffb700 1 -- 172.21.15.23:0/3986966757 <== osd.1 v2:172.21.15.116:6800/1598773126 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa82c112590 con 0x7fa81c0aba40 2024-03-24T10:38:49.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.444+0000 7fa83303a700 1 -- 172.21.15.23:0/3986966757 >> [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126] conn(0x7fa81c0aba40 msgr2=0x7fa81c0ade60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:49.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.444+0000 7fa83303a700 1 --2- 172.21.15.23:0/3986966757 >> [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126] conn(0x7fa81c0aba40 0x7fa81c0ade60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:49.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.444+0000 7fa83303a700 1 -- 172.21.15.23:0/3986966757 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa81c0a2260 msgr2=0x7fa81c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:49.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.444+0000 7fa83303a700 1 --2- 172.21.15.23:0/3986966757 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa81c0a2260 0x7fa81c0a4720 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa820011490 tx=0x7fa820009260 comp rx=0 tx=0).stop 2024-03-24T10:38:49.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.444+0000 7fa83303a700 1 -- 172.21.15.23:0/3986966757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa82c102c10 msgr2=0x7fa82c19e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:49.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.444+0000 7fa83303a700 1 --2- 172.21.15.23:0/3986966757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa82c102c10 0x7fa82c19e980 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa81800e9a0 tx=0x7fa81800ecb0 comp rx=0 tx=0).stop 2024-03-24T10:38:49.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.444+0000 7fa8315d7700 1 -- 172.21.15.23:0/3986966757 reap_dead start 2024-03-24T10:38:49.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.444+0000 7fa83303a700 1 -- 172.21.15.23:0/3986966757 shutdown_connections 2024-03-24T10:38:49.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.444+0000 7fa83303a700 1 -- 172.21.15.23:0/3986966757 >> 172.21.15.23:0/3986966757 conn(0x7fa82c079330 msgr2=0x7fa82c100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:49.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.444+0000 7fa83303a700 1 -- 172.21.15.23:0/3986966757 shutdown_connections 2024-03-24T10:38:49.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:49.444+0000 7fa83303a700 1 -- 172.21.15.23:0/3986966757 wait complete. 2024-03-24T10:38:50.297 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:50 smithi023 ceph-mon[32466]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T10:38:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.399+0000 7f0cfda52700 1 -- 172.21.15.23:0/2233095005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf8102e90 msgr2=0x7f0cf8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.399+0000 7f0cfda52700 1 --2- 172.21.15.23:0/2233095005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf8102e90 0x7f0cf8106e60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009990 tx=0x7f0ce0009ca0 comp rx=0 tx=0).stop 2024-03-24T10:38:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.399+0000 7f0cfda52700 1 -- 172.21.15.23:0/2233095005 shutdown_connections 2024-03-24T10:38:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.399+0000 7f0cfda52700 1 --2- 172.21.15.23:0/2233095005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cf8107590 0x7f0cf810f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.399+0000 7f0cfda52700 1 --2- 172.21.15.23:0/2233095005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf81024e0 0x7f0cf81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.399+0000 7f0cfda52700 1 --2- 172.21.15.23:0/2233095005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf8102e90 0x7f0cf8106e60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.399+0000 7f0cfda52700 1 -- 172.21.15.23:0/2233095005 >> 172.21.15.23:0/2233095005 conn(0x7f0cf8079320 msgr2=0x7f0cf8079730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.399+0000 7f0cfda52700 1 -- 172.21.15.23:0/2233095005 shutdown_connections 2024-03-24T10:38:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.399+0000 7f0cfda52700 1 -- 172.21.15.23:0/2233095005 wait complete. 2024-03-24T10:38:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.400+0000 7f0cfda52700 1 Processor -- start 2024-03-24T10:38:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.400+0000 7f0cfda52700 1 -- start start 2024-03-24T10:38:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.400+0000 7f0cfda52700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf81024e0 0x7f0cf8198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.400+0000 7f0cfda52700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cf8102e90 0x7f0cf8198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.400+0000 7f0cfda52700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf8107590 0x7f0cf819cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.400+0000 7f0cfda52700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf819d5e0 con 0x7f0cf81024e0 2024-03-24T10:38:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.400+0000 7f0cfda52700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf819d750 con 0x7f0cf8107590 2024-03-24T10:38:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.400+0000 7f0cfda52700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf819d910 con 0x7f0cf8102e90 2024-03-24T10:38:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.400+0000 7f0cf6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf81024e0 0x7f0cf8198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.400+0000 7f0cf6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf81024e0 0x7f0cf8198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51040/0 (socket says 172.21.15.23:51040) 2024-03-24T10:38:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.400+0000 7f0cf6ffd700 1 -- 172.21.15.23:0/3126536653 learned_addr learned my addr 172.21.15.23:0/3126536653 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.400+0000 7f0cf67fc700 1 --2- 172.21.15.23:0/3126536653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cf8102e90 0x7f0cf8198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.400+0000 7f0cf77fe700 1 --2- 172.21.15.23:0/3126536653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf8107590 0x7f0cf819cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.401+0000 7f0cf6ffd700 1 -- 172.21.15.23:0/3126536653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cf8102e90 msgr2=0x7f0cf8198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.401+0000 7f0cf6ffd700 1 --2- 172.21.15.23:0/3126536653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cf8102e90 0x7f0cf8198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.401+0000 7f0cf6ffd700 1 -- 172.21.15.23:0/3126536653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf8107590 msgr2=0x7f0cf819cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.401+0000 7f0cf6ffd700 1 --2- 172.21.15.23:0/3126536653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf8107590 0x7f0cf819cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.401+0000 7f0cf6ffd700 1 -- 172.21.15.23:0/3126536653 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce00095f0 con 0x7f0cf81024e0 2024-03-24T10:38:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.401+0000 7f0cf6ffd700 1 --2- 172.21.15.23:0/3126536653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf81024e0 0x7f0cf8198700 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000ef10 tx=0x7f0cf000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.401+0000 7f0cfca50700 1 -- 172.21.15.23:0/3126536653 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf000c9c0 con 0x7f0cf81024e0 2024-03-24T10:38:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.401+0000 7f0cfca50700 1 -- 172.21.15.23:0/3126536653 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cf000f040 con 0x7f0cf81024e0 2024-03-24T10:38:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.401+0000 7f0cfda52700 1 -- 172.21.15.23:0/3126536653 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cf819db90 con 0x7f0cf81024e0 2024-03-24T10:38:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.401+0000 7f0cfca50700 1 -- 172.21.15.23:0/3126536653 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0014480 con 0x7f0cf81024e0 2024-03-24T10:38:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.401+0000 7f0cfda52700 1 -- 172.21.15.23:0/3126536653 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cf81a5450 con 0x7f0cf81024e0 2024-03-24T10:38:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.402+0000 7f0cfda52700 1 -- 172.21.15.23:0/3126536653 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0cf8103940 con 0x7f0cf81024e0 2024-03-24T10:38:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.403+0000 7f0cfca50700 1 -- 172.21.15.23:0/3126536653 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f0cf0007720 con 0x7f0cf81024e0 2024-03-24T10:38:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.404+0000 7f0cfca50700 1 --2- 172.21.15.23:0/3126536653 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ce40a21e0 0x7f0ce40a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.404+0000 7f0cf67fc700 1 --2- 172.21.15.23:0/3126536653 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ce40a21e0 0x7f0ce40a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.404+0000 7f0cfca50700 1 -- 172.21.15.23:0/3126536653 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f0cf00cea00 con 0x7f0cf81024e0 2024-03-24T10:38:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.404+0000 7f0cfca50700 1 --2- 172.21.15.23:0/3126536653 >> [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530] conn(0x7f0ce40ab9c0 0x7f0ce40adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.404+0000 7f0cfca50700 1 -- 172.21.15.23:0/3126536653 --> [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ce40ae490 con 0x7f0ce40ab9c0 2024-03-24T10:38:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.404+0000 7f0cfca50700 1 -- 172.21.15.23:0/3126536653 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f0cf00ceea0 con 0x7f0cf81024e0 2024-03-24T10:38:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.404+0000 7f0cf77fe700 1 --2- 172.21.15.23:0/3126536653 >> [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530] conn(0x7f0ce40ab9c0 0x7f0ce40adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.404+0000 7f0cf67fc700 1 --2- 172.21.15.23:0/3126536653 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ce40a21e0 0x7f0ce40a46a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0ce00059d0 tx=0x7f0ce0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.404+0000 7f0cf77fe700 1 --2- 172.21.15.23:0/3126536653 >> [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530] conn(0x7f0ce40ab9c0 0x7f0ce40adde0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.405+0000 7f0cfca50700 1 -- 172.21.15.23:0/3126536653 <== osd.9 v2:172.21.15.23:6826/3239156530 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ce40ae490 con 0x7f0ce40ab9c0 2024-03-24T10:38:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.421+0000 7f0cfda52700 1 -- 172.21.15.23:0/3126536653 --> [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0cf8112560 con 0x7f0ce40ab9c0 2024-03-24T10:38:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.422+0000 7f0cfca50700 1 -- 172.21.15.23:0/3126536653 <== osd.9 v2:172.21.15.23:6826/3239156530 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0cf8112560 con 0x7f0ce40ab9c0 2024-03-24T10:38:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.422+0000 7f0cfda52700 1 -- 172.21.15.23:0/3126536653 >> [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530] conn(0x7f0ce40ab9c0 msgr2=0x7f0ce40adde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.422+0000 7f0cfda52700 1 --2- 172.21.15.23:0/3126536653 >> [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530] conn(0x7f0ce40ab9c0 0x7f0ce40adde0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.422+0000 7f0cfda52700 1 -- 172.21.15.23:0/3126536653 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ce40a21e0 msgr2=0x7f0ce40a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.422+0000 7f0cfda52700 1 --2- 172.21.15.23:0/3126536653 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ce40a21e0 0x7f0ce40a46a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0ce00059d0 tx=0x7f0ce0005960 comp rx=0 tx=0).stop 2024-03-24T10:38:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.422+0000 7f0cfda52700 1 -- 172.21.15.23:0/3126536653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf81024e0 msgr2=0x7f0cf8198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.422+0000 7f0cfda52700 1 --2- 172.21.15.23:0/3126536653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf81024e0 0x7f0cf8198700 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000ef10 tx=0x7f0cf000c3a0 comp rx=0 tx=0).stop 2024-03-24T10:38:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.422+0000 7f0cf77fe700 1 -- 172.21.15.23:0/3126536653 reap_dead start 2024-03-24T10:38:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.422+0000 7f0cfda52700 1 -- 172.21.15.23:0/3126536653 shutdown_connections 2024-03-24T10:38:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.422+0000 7f0cfda52700 1 -- 172.21.15.23:0/3126536653 >> 172.21.15.23:0/3126536653 conn(0x7f0cf8079320 msgr2=0x7f0cf8100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.422+0000 7f0cfda52700 1 -- 172.21.15.23:0/3126536653 shutdown_connections 2024-03-24T10:38:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:50.422+0000 7f0cfda52700 1 -- 172.21.15.23:0/3126536653 wait complete. 2024-03-24T10:38:50.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:50 smithi117 ceph-mon[36368]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T10:38:50.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:50 smithi116 ceph-mon[36248]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T10:38:51.326 INFO:teuthology.orchestra.run.smithi023.stdout:287762808863 2024-03-24T10:38:51.326 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-24T10:38:51.682 INFO:teuthology.orchestra.run.smithi023.stdout:309237645338 2024-03-24T10:38:51.682 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-24T10:38:52.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:52 smithi117 ceph-mon[36368]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T10:38:52.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:52 smithi116 ceph-mon[36248]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T10:38:52.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:52 smithi023 ceph-mon[32466]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T10:38:52.730 INFO:teuthology.orchestra.run.smithi023.stdout:219043332136 2024-03-24T10:38:52.730 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T10:38:52.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.927+0000 7f705bfae700 1 -- 172.21.15.23:0/3214545389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7054101390 msgr2=0x7f7054101770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:52.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.927+0000 7f705bfae700 1 --2- 172.21.15.23:0/3214545389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7054101390 0x7f7054101770 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7044009990 tx=0x7f7044009ca0 comp rx=0 tx=0).stop 2024-03-24T10:38:52.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.927+0000 7f705bfae700 1 -- 172.21.15.23:0/3214545389 shutdown_connections 2024-03-24T10:38:52.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.927+0000 7f705bfae700 1 --2- 172.21.15.23:0/3214545389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70541063e0 0x7f705410dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:52.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.927+0000 7f705bfae700 1 --2- 172.21.15.23:0/3214545389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7054101390 0x7f7054101770 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.927+0000 7f705bfae700 1 --2- 172.21.15.23:0/3214545389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7054101d40 0x7f7054105d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.927+0000 7f705bfae700 1 -- 172.21.15.23:0/3214545389 >> 172.21.15.23:0/3214545389 conn(0x7f7054077150 msgr2=0x7f7054077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.927+0000 7f705bfae700 1 -- 172.21.15.23:0/3214545389 shutdown_connections 2024-03-24T10:38:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.927+0000 7f705bfae700 1 -- 172.21.15.23:0/3214545389 wait complete. 2024-03-24T10:38:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f705bfae700 1 Processor -- start 2024-03-24T10:38:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f705bfae700 1 -- start start 2024-03-24T10:38:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f705bfae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7054101390 0x7f7054198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f705bfae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7054101d40 0x7f7054198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f705bfae700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70541063e0 0x7f705419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f705bfae700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f705419d610 con 0x7f7054101d40 2024-03-24T10:38:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f705bfae700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f705419d780 con 0x7f7054101390 2024-03-24T10:38:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f705bfae700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f705419d940 con 0x7f70541063e0 2024-03-24T10:38:52.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f7059549700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7054101d40 0x7f7054198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:52.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f7059549700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7054101d40 0x7f7054198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51056/0 (socket says 172.21.15.23:51056) 2024-03-24T10:38:52.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f7059549700 1 -- 172.21.15.23:0/2263973643 learned_addr learned my addr 172.21.15.23:0/2263973643 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:52.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f7059d4a700 1 --2- 172.21.15.23:0/2263973643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7054101390 0x7f7054198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:52.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f705a54b700 1 --2- 172.21.15.23:0/2263973643 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70541063e0 0x7f705419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:52.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f7059549700 1 -- 172.21.15.23:0/2263973643 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70541063e0 msgr2=0x7f705419d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:52.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f7059549700 1 --2- 172.21.15.23:0/2263973643 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70541063e0 0x7f705419d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:52.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f7059549700 1 -- 172.21.15.23:0/2263973643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7054101390 msgr2=0x7f7054198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:52.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f7059549700 1 --2- 172.21.15.23:0/2263973643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7054101390 0x7f7054198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:52.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.928+0000 7f7059549700 1 -- 172.21.15.23:0/2263973643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70440095f0 con 0x7f7054101d40 2024-03-24T10:38:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.929+0000 7f7059549700 1 --2- 172.21.15.23:0/2263973643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7054101d40 0x7f7054198c70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f705000e970 tx=0x7f705000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.929+0000 7f704affd700 1 -- 172.21.15.23:0/2263973643 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7050009790 con 0x7f7054101d40 2024-03-24T10:38:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.929+0000 7f705bfae700 1 -- 172.21.15.23:0/2263973643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f705419dc20 con 0x7f7054101d40 2024-03-24T10:38:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.929+0000 7f704affd700 1 -- 172.21.15.23:0/2263973643 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f705000c7e0 con 0x7f7054101d40 2024-03-24T10:38:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.929+0000 7f704affd700 1 -- 172.21.15.23:0/2263973643 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7050005580 con 0x7f7054101d40 2024-03-24T10:38:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.929+0000 7f705bfae700 1 -- 172.21.15.23:0/2263973643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70541a5480 con 0x7f7054101d40 2024-03-24T10:38:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.930+0000 7f705bfae700 1 -- 172.21.15.23:0/2263973643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7054111440 con 0x7f7054101d40 2024-03-24T10:38:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.931+0000 7f704affd700 1 -- 172.21.15.23:0/2263973643 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f705000cce0 con 0x7f7054101d40 2024-03-24T10:38:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.931+0000 7f704affd700 1 --2- 172.21.15.23:0/2263973643 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f70400a21e0 0x7f70400a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.931+0000 7f7059d4a700 1 --2- 172.21.15.23:0/2263973643 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f70400a21e0 0x7f70400a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.932+0000 7f704affd700 1 -- 172.21.15.23:0/2263973643 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7050014070 con 0x7f7054101d40 2024-03-24T10:38:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.932+0000 7f704affd700 1 --2- 172.21.15.23:0/2263973643 >> [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953] conn(0x7f70400ab9c0 0x7f70400adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.932+0000 7f704affd700 1 -- 172.21.15.23:0/2263973643 --> [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70400ae490 con 0x7f70400ab9c0 2024-03-24T10:38:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.932+0000 7f704affd700 1 -- 172.21.15.23:0/2263973643 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f70500ce610 con 0x7f7054101d40 2024-03-24T10:38:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.932+0000 7f7059d4a700 1 --2- 172.21.15.23:0/2263973643 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f70400a21e0 0x7f70400a46a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f704400b2e0 tx=0x7f7044005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.932+0000 7f705a54b700 1 --2- 172.21.15.23:0/2263973643 >> [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953] conn(0x7f70400ab9c0 0x7f70400adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.932+0000 7f705a54b700 1 --2- 172.21.15.23:0/2263973643 >> [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953] conn(0x7f70400ab9c0 0x7f70400adde0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.933+0000 7f704affd700 1 -- 172.21.15.23:0/2263973643 <== osd.4 v2:172.21.15.116:6808/3074814953 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f70400ae490 con 0x7f70400ab9c0 2024-03-24T10:38:52.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.949+0000 7f705bfae700 1 -- 172.21.15.23:0/2263973643 --> [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7054062890 con 0x7f70400ab9c0 2024-03-24T10:38:52.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.950+0000 7f704affd700 1 -- 172.21.15.23:0/2263973643 <== osd.4 v2:172.21.15.116:6808/3074814953 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7054062890 con 0x7f70400ab9c0 2024-03-24T10:38:52.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.950+0000 7f705bfae700 1 -- 172.21.15.23:0/2263973643 >> [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953] conn(0x7f70400ab9c0 msgr2=0x7f70400adde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:52.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.950+0000 7f705bfae700 1 --2- 172.21.15.23:0/2263973643 >> [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953] conn(0x7f70400ab9c0 0x7f70400adde0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:52.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.950+0000 7f705bfae700 1 -- 172.21.15.23:0/2263973643 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f70400a21e0 msgr2=0x7f70400a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:52.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.950+0000 7f705bfae700 1 --2- 172.21.15.23:0/2263973643 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f70400a21e0 0x7f70400a46a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f704400b2e0 tx=0x7f7044005960 comp rx=0 tx=0).stop 2024-03-24T10:38:52.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.950+0000 7f705bfae700 1 -- 172.21.15.23:0/2263973643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7054101d40 msgr2=0x7f7054198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:52.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.950+0000 7f705bfae700 1 --2- 172.21.15.23:0/2263973643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7054101d40 0x7f7054198c70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f705000e970 tx=0x7f705000ec80 comp rx=0 tx=0).stop 2024-03-24T10:38:52.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.950+0000 7f705a54b700 1 -- 172.21.15.23:0/2263973643 reap_dead start 2024-03-24T10:38:52.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.950+0000 7f705bfae700 1 -- 172.21.15.23:0/2263973643 shutdown_connections 2024-03-24T10:38:52.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.950+0000 7f705bfae700 1 -- 172.21.15.23:0/2263973643 >> 172.21.15.23:0/2263973643 conn(0x7f7054077150 msgr2=0x7f705406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:52.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.950+0000 7f705bfae700 1 -- 172.21.15.23:0/2263973643 shutdown_connections 2024-03-24T10:38:52.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:52.950+0000 7f705bfae700 1 -- 172.21.15.23:0/2263973643 wait complete. 2024-03-24T10:38:53.888 INFO:teuthology.orchestra.run.smithi023.stdout:141733920824 2024-03-24T10:38:53.889 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T10:38:54.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.061+0000 7f2935dfb700 1 -- 172.21.15.23:0/1760273985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2930108360 msgr2=0x7f2930108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:54.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.061+0000 7f2935dfb700 1 --2- 172.21.15.23:0/1760273985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2930108360 0x7f2930108740 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2920009990 tx=0x7f2920009ca0 comp rx=0 tx=0).stop 2024-03-24T10:38:54.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.061+0000 7f2935dfb700 1 -- 172.21.15.23:0/1760273985 shutdown_connections 2024-03-24T10:38:54.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.061+0000 7f2935dfb700 1 --2- 172.21.15.23:0/1760273985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2930102c40 0x7f293010f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:54.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.061+0000 7f2935dfb700 1 --2- 172.21.15.23:0/1760273985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2930108360 0x7f2930108740 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:54.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.061+0000 7f2935dfb700 1 --2- 172.21.15.23:0/1760273985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2930102280 0x7f2930102700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:54.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.061+0000 7f2935dfb700 1 -- 172.21.15.23:0/1760273985 >> 172.21.15.23:0/1760273985 conn(0x7f29300792c0 msgr2=0x7f29300796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:54.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.061+0000 7f2935dfb700 1 -- 172.21.15.23:0/1760273985 shutdown_connections 2024-03-24T10:38:54.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.061+0000 7f2935dfb700 1 -- 172.21.15.23:0/1760273985 wait complete. 2024-03-24T10:38:54.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.062+0000 7f2935dfb700 1 Processor -- start 2024-03-24T10:38:54.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.062+0000 7f2935dfb700 1 -- start start 2024-03-24T10:38:54.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.062+0000 7f2935dfb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2930102280 0x7f29301a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:54.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.062+0000 7f2935dfb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2930102c40 0x7f29301a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:54.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.062+0000 7f2935dfb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2930108360 0x7f293019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:54.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.062+0000 7f2935dfb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f293019ae80 con 0x7f2930102280 2024-03-24T10:38:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.062+0000 7f2935dfb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f293019aff0 con 0x7f2930108360 2024-03-24T10:38:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.062+0000 7f2935dfb700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f293019b1b0 con 0x7f2930102c40 2024-03-24T10:38:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.062+0000 7f292f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2930102280 0x7f29301a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.062+0000 7f292effd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2930102c40 0x7f29301a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.062+0000 7f292f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2930102280 0x7f29301a0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51082/0 (socket says 172.21.15.23:51082) 2024-03-24T10:38:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.062+0000 7f292f7fe700 1 -- 172.21.15.23:0/115595083 learned_addr learned my addr 172.21.15.23:0/115595083 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.062+0000 7f292ffff700 1 --2- 172.21.15.23:0/115595083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2930108360 0x7f293019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.063+0000 7f292effd700 1 -- 172.21.15.23:0/115595083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2930108360 msgr2=0x7f293019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.063+0000 7f292effd700 1 --2- 172.21.15.23:0/115595083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2930108360 0x7f293019a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.063+0000 7f292effd700 1 -- 172.21.15.23:0/115595083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2930102280 msgr2=0x7f29301a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.063+0000 7f292effd700 1 --2- 172.21.15.23:0/115595083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2930102280 0x7f29301a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:54.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.063+0000 7f292effd700 1 -- 172.21.15.23:0/115595083 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29200095f0 con 0x7f2930102c40 2024-03-24T10:38:54.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.063+0000 7f292effd700 1 --2- 172.21.15.23:0/115595083 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2930102c40 0x7f29301a0b40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f292400e9f0 tx=0x7f292400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:54.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.063+0000 7f292cff9700 1 -- 172.21.15.23:0/115595083 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292400c9c0 con 0x7f2930102c40 2024-03-24T10:38:54.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.063+0000 7f292cff9700 1 -- 172.21.15.23:0/115595083 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2924010430 con 0x7f2930102c40 2024-03-24T10:38:54.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.064+0000 7f292cff9700 1 -- 172.21.15.23:0/115595083 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2924018660 con 0x7f2930102c40 2024-03-24T10:38:54.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.064+0000 7f2935dfb700 1 -- 172.21.15.23:0/115595083 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f293019b490 con 0x7f2930102c40 2024-03-24T10:38:54.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.064+0000 7f2935dfb700 1 -- 172.21.15.23:0/115595083 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29301a7580 con 0x7f2930102c40 2024-03-24T10:38:54.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.064+0000 7f2935dfb700 1 -- 172.21.15.23:0/115595083 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2930103940 con 0x7f2930102c40 2024-03-24T10:38:54.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.066+0000 7f292cff9700 1 -- 172.21.15.23:0/115595083 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f2924010910 con 0x7f2930102c40 2024-03-24T10:38:54.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.066+0000 7f292cff9700 1 --2- 172.21.15.23:0/115595083 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f29180a21e0 0x7f29180a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:54.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.066+0000 7f292f7fe700 1 --2- 172.21.15.23:0/115595083 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f29180a21e0 0x7f29180a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:54.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.067+0000 7f292cff9700 1 -- 172.21.15.23:0/115595083 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f2924014070 con 0x7f2930102c40 2024-03-24T10:38:54.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.067+0000 7f292cff9700 1 --2- 172.21.15.23:0/115595083 >> [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608] conn(0x7f29180ab9c0 0x7f29180adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:54.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.067+0000 7f292cff9700 1 -- 172.21.15.23:0/115595083 --> [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29180ae490 con 0x7f29180ab9c0 2024-03-24T10:38:54.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.067+0000 7f292cff9700 1 -- 172.21.15.23:0/115595083 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f29240ce890 con 0x7f2930102c40 2024-03-24T10:38:54.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.067+0000 7f292ffff700 1 --2- 172.21.15.23:0/115595083 >> [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608] conn(0x7f29180ab9c0 0x7f29180adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:54.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.067+0000 7f292f7fe700 1 --2- 172.21.15.23:0/115595083 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f29180a21e0 0x7f29180a46a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2920006010 tx=0x7f292000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:54.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.067+0000 7f292ffff700 1 --2- 172.21.15.23:0/115595083 >> [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608] conn(0x7f29180ab9c0 0x7f29180adde0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:54.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.068+0000 7f292cff9700 1 -- 172.21.15.23:0/115595083 <== osd.5 v2:172.21.15.117:6808/1549866608 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f29180ae490 con 0x7f29180ab9c0 2024-03-24T10:38:54.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.085+0000 7f2935dfb700 1 -- 172.21.15.23:0/115595083 --> [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2930112560 con 0x7f29180ab9c0 2024-03-24T10:38:54.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.086+0000 7f292cff9700 1 -- 172.21.15.23:0/115595083 <== osd.5 v2:172.21.15.117:6808/1549866608 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2930112560 con 0x7f29180ab9c0 2024-03-24T10:38:54.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.086+0000 7f2935dfb700 1 -- 172.21.15.23:0/115595083 >> [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608] conn(0x7f29180ab9c0 msgr2=0x7f29180adde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:54.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.086+0000 7f2935dfb700 1 --2- 172.21.15.23:0/115595083 >> [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608] conn(0x7f29180ab9c0 0x7f29180adde0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:54.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.086+0000 7f2935dfb700 1 -- 172.21.15.23:0/115595083 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f29180a21e0 msgr2=0x7f29180a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:54.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.086+0000 7f2935dfb700 1 --2- 172.21.15.23:0/115595083 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f29180a21e0 0x7f29180a46a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2920006010 tx=0x7f292000b260 comp rx=0 tx=0).stop 2024-03-24T10:38:54.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.086+0000 7f2935dfb700 1 -- 172.21.15.23:0/115595083 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2930102c40 msgr2=0x7f29301a0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:54.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.086+0000 7f2935dfb700 1 --2- 172.21.15.23:0/115595083 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2930102c40 0x7f29301a0b40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f292400e9f0 tx=0x7f292400ed00 comp rx=0 tx=0).stop 2024-03-24T10:38:54.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.086+0000 7f292ffff700 1 -- 172.21.15.23:0/115595083 reap_dead start 2024-03-24T10:38:54.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.086+0000 7f2935dfb700 1 -- 172.21.15.23:0/115595083 shutdown_connections 2024-03-24T10:38:54.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.086+0000 7f2935dfb700 1 -- 172.21.15.23:0/115595083 >> 172.21.15.23:0/115595083 conn(0x7f29300792c0 msgr2=0x7f293010d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:54.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.087+0000 7f2935dfb700 1 -- 172.21.15.23:0/115595083 shutdown_connections 2024-03-24T10:38:54.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.087+0000 7f2935dfb700 1 -- 172.21.15.23:0/115595083 wait complete. 2024-03-24T10:38:54.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:54 smithi117 ceph-mon[36368]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:38:54.516 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:54 smithi023 ceph-mon[32466]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:38:54.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:54 smithi116 ceph-mon[36248]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:38:54.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.529+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/1475752586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc107190 msgr2=0x7fc8bc107570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:54.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.529+0000 7fc8c3bc2700 1 --2- 172.21.15.23:0/1475752586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc107190 0x7fc8bc107570 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8009990 tx=0x7fc8b8009ca0 comp rx=0 tx=0).stop 2024-03-24T10:38:54.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.529+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/1475752586 shutdown_connections 2024-03-24T10:38:54.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.529+0000 7fc8c3bc2700 1 --2- 172.21.15.23:0/1475752586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8bc101aa0 0x7fc8bc10df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:54.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.529+0000 7fc8c3bc2700 1 --2- 172.21.15.23:0/1475752586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc107190 0x7fc8bc107570 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:54.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.529+0000 7fc8c3bc2700 1 --2- 172.21.15.23:0/1475752586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8bc1010e0 0x7fc8bc101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:54.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.529+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/1475752586 >> 172.21.15.23:0/1475752586 conn(0x7fc8bc078110 msgr2=0x7fc8bc078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:54.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.529+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/1475752586 shutdown_connections 2024-03-24T10:38:54.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.529+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/1475752586 wait complete. 2024-03-24T10:38:54.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.529+0000 7fc8c3bc2700 1 Processor -- start 2024-03-24T10:38:54.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.529+0000 7fc8c3bc2700 1 -- start start 2024-03-24T10:38:54.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c3bc2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc1010e0 0x7fc8bc198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:54.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c3bc2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8bc101aa0 0x7fc8bc198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:54.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c3bc2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8bc107190 0x7fc8bc19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:54.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c3bc2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc8bc19d5b0 con 0x7fc8bc1010e0 2024-03-24T10:38:54.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c3bc2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8bc19d720 con 0x7fc8bc101aa0 2024-03-24T10:38:54.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c3bc2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc8bc19d8e0 con 0x7fc8bc107190 2024-03-24T10:38:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c195e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc1010e0 0x7fc8bc198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c195e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc1010e0 0x7fc8bc198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51096/0 (socket says 172.21.15.23:51096) 2024-03-24T10:38:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c195e700 1 -- 172.21.15.23:0/4156339123 learned_addr learned my addr 172.21.15.23:0/4156339123 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:38:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c215f700 1 --2- 172.21.15.23:0/4156339123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8bc107190 0x7fc8bc19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c115d700 1 --2- 172.21.15.23:0/4156339123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8bc101aa0 0x7fc8bc198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c195e700 1 -- 172.21.15.23:0/4156339123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8bc107190 msgr2=0x7fc8bc19cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c195e700 1 --2- 172.21.15.23:0/4156339123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8bc107190 0x7fc8bc19cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c195e700 1 -- 172.21.15.23:0/4156339123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8bc101aa0 msgr2=0x7fc8bc198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c195e700 1 --2- 172.21.15.23:0/4156339123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8bc101aa0 0x7fc8bc198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c195e700 1 -- 172.21.15.23:0/4156339123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b80095f0 con 0x7fc8bc1010e0 2024-03-24T10:38:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c195e700 1 --2- 172.21.15.23:0/4156339123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc1010e0 0x7fc8bc198720 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc8b800b2e0 tx=0x7fc8b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8b2ffd700 1 -- 172.21.15.23:0/4156339123 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b801d070 con 0x7fc8bc1010e0 2024-03-24T10:38:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8b2ffd700 1 -- 172.21.15.23:0/4156339123 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8b800b610 con 0x7fc8bc1010e0 2024-03-24T10:38:54.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/4156339123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8bc19db60 con 0x7fc8bc1010e0 2024-03-24T10:38:54.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8b2ffd700 1 -- 172.21.15.23:0/4156339123 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b80173f0 con 0x7fc8bc1010e0 2024-03-24T10:38:54.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.530+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/4156339123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8bc102e90 con 0x7fc8bc1010e0 2024-03-24T10:38:54.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.531+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/4156339123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc8bc19de30 con 0x7fc8bc1010e0 2024-03-24T10:38:54.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.532+0000 7fc8b2ffd700 1 -- 172.21.15.23:0/4156339123 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fc8b80216a0 con 0x7fc8bc1010e0 2024-03-24T10:38:54.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.533+0000 7fc8b2ffd700 1 --2- 172.21.15.23:0/4156339123 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc8a80a21e0 0x7fc8a80a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:54.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.533+0000 7fc8c115d700 1 --2- 172.21.15.23:0/4156339123 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc8a80a21e0 0x7fc8a80a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:54.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.533+0000 7fc8b2ffd700 1 -- 172.21.15.23:0/4156339123 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fc8b80cf810 con 0x7fc8bc1010e0 2024-03-24T10:38:54.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.533+0000 7fc8b2ffd700 1 --2- 172.21.15.23:0/4156339123 >> [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351] conn(0x7fc8a80ab9c0 0x7fc8a80adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:38:54.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.533+0000 7fc8b2ffd700 1 -- 172.21.15.23:0/4156339123 --> [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc8a80ae490 con 0x7fc8a80ab9c0 2024-03-24T10:38:54.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.533+0000 7fc8b2ffd700 1 -- 172.21.15.23:0/4156339123 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fc8b80cfcb0 con 0x7fc8bc1010e0 2024-03-24T10:38:54.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.533+0000 7fc8c115d700 1 --2- 172.21.15.23:0/4156339123 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc8a80a21e0 0x7fc8a80a46a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc078710 tx=0x7fc8ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:54.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.533+0000 7fc8c215f700 1 --2- 172.21.15.23:0/4156339123 >> [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351] conn(0x7fc8a80ab9c0 0x7fc8a80adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:38:54.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.534+0000 7fc8c215f700 1 --2- 172.21.15.23:0/4156339123 >> [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351] conn(0x7fc8a80ab9c0 0x7fc8a80adde0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:38:54.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.535+0000 7fc8b2ffd700 1 -- 172.21.15.23:0/4156339123 <== osd.2 v2:172.21.15.117:6800/3979033351 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc8a80ae490 con 0x7fc8a80ab9c0 2024-03-24T10:38:54.563 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:54.563 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:54.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.555+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/4156339123 --> [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc8bc111430 con 0x7fc8a80ab9c0 2024-03-24T10:38:54.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.556+0000 7fc8b2ffd700 1 -- 172.21.15.23:0/4156339123 <== osd.2 v2:172.21.15.117:6800/3979033351 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc8bc111430 con 0x7fc8a80ab9c0 2024-03-24T10:38:54.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.560+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/4156339123 >> [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351] conn(0x7fc8a80ab9c0 msgr2=0x7fc8a80adde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:54.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.560+0000 7fc8c3bc2700 1 --2- 172.21.15.23:0/4156339123 >> [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351] conn(0x7fc8a80ab9c0 0x7fc8a80adde0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:38:54.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.562+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/4156339123 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc8a80a21e0 msgr2=0x7fc8a80a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:54.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.562+0000 7fc8c3bc2700 1 --2- 172.21.15.23:0/4156339123 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fc8a80a21e0 0x7fc8a80a46a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc078710 tx=0x7fc8ac009260 comp rx=0 tx=0).stop 2024-03-24T10:38:54.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.562+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/4156339123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc1010e0 msgr2=0x7fc8bc198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:38:54.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.562+0000 7fc8c3bc2700 1 --2- 172.21.15.23:0/4156339123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc1010e0 0x7fc8bc198720 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc8b800b2e0 tx=0x7fc8b8004730 comp rx=0 tx=0).stop 2024-03-24T10:38:54.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.562+0000 7fc8c215f700 1 -- 172.21.15.23:0/4156339123 reap_dead start 2024-03-24T10:38:54.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.563+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/4156339123 shutdown_connections 2024-03-24T10:38:54.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.563+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/4156339123 >> 172.21.15.23:0/4156339123 conn(0x7fc8bc078110 msgr2=0x7fc8bc06aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:38:54.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.565+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/4156339123 shutdown_connections 2024-03-24T10:38:54.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:38:54.565+0000 7fc8c3bc2700 1 -- 172.21.15.23:0/4156339123 wait complete. 2024-03-24T10:38:54.576 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:54.938 INFO:teuthology.orchestra.run.smithi023.stdout:55834574918 2024-03-24T10:38:54.939 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T10:38:55.869 INFO:teuthology.orchestra.run.smithi023.stdout:163208757299 2024-03-24T10:38:55.869 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T10:38:56.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:56 smithi023 ceph-mon[32466]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:38:56.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:56 smithi117 ceph-mon[36368]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:38:56.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:56 smithi116 ceph-mon[36248]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:38:56.888 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:56.917 INFO:teuthology.orchestra.run.smithi023.stdout:77309411394 2024-03-24T10:38:56.917 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T10:38:58.053 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:58.053 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:58.054 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:58.054 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:38:58.293 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:38:58 smithi023 ceph-mon[32466]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T10:38:58.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:38:58 smithi117 ceph-mon[36368]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T10:38:58.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:38:58 smithi116 ceph-mon[36248]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T10:38:59.948 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:00.492 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:00 smithi023 ceph-mon[32466]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T10:39:00.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:00 smithi117 ceph-mon[36368]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T10:39:00.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:00 smithi116 ceph-mon[36248]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T10:39:02.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.295+0000 7f83117bc700 1 -- 172.21.15.23:0/4101090876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830c102250 msgr2=0x7f830c1026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:02.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.295+0000 7f83117bc700 1 --2- 172.21.15.23:0/4101090876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830c102250 0x7f830c1026d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f83000099c0 tx=0x7f8300009cd0 comp rx=0 tx=0).stop 2024-03-24T10:39:02.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.295+0000 7f83117bc700 1 -- 172.21.15.23:0/4101090876 shutdown_connections 2024-03-24T10:39:02.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.295+0000 7f83117bc700 1 --2- 172.21.15.23:0/4101090876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f830c108230 0x7f830c108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:02.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.295+0000 7f83117bc700 1 --2- 172.21.15.23:0/4101090876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f830c102c10 0x7f830c10f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:02.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.295+0000 7f83117bc700 1 --2- 172.21.15.23:0/4101090876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830c102250 0x7f830c1026d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:02.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.295+0000 7f83117bc700 1 -- 172.21.15.23:0/4101090876 >> 172.21.15.23:0/4101090876 conn(0x7f830c079330 msgr2=0x7f830c079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:02.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.295+0000 7f83117bc700 1 -- 172.21.15.23:0/4101090876 shutdown_connections 2024-03-24T10:39:02.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.295+0000 7f83117bc700 1 -- 172.21.15.23:0/4101090876 wait complete. 2024-03-24T10:39:02.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.296+0000 7f83117bc700 1 Processor -- start 2024-03-24T10:39:02.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.296+0000 7f83117bc700 1 -- start start 2024-03-24T10:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f83117bc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f830c102250 0x7f830c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f83117bc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f830c102c10 0x7f830c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f83117bc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830c108230 0x7f830c19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f83117bc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f830c19d560 con 0x7f830c108230 2024-03-24T10:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f83117bc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f830c19d6d0 con 0x7f830c102c10 2024-03-24T10:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f83117bc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f830c19d890 con 0x7f830c102250 2024-03-24T10:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f830b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830c108230 0x7f830c19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f830b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830c108230 0x7f830c19cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45716/0 (socket says 172.21.15.23:45716) 2024-03-24T10:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f830b7fe700 1 -- 172.21.15.23:0/1430757735 learned_addr learned my addr 172.21.15.23:0/1430757735 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f830a7fc700 1 --2- 172.21.15.23:0/1430757735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f830c102c10 0x7f830c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f830affd700 1 --2- 172.21.15.23:0/1430757735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f830c102250 0x7f830c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f830b7fe700 1 -- 172.21.15.23:0/1430757735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f830c102250 msgr2=0x7f830c198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f830b7fe700 1 --2- 172.21.15.23:0/1430757735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f830c102250 0x7f830c198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f830b7fe700 1 -- 172.21.15.23:0/1430757735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f830c102c10 msgr2=0x7f830c198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f830b7fe700 1 --2- 172.21.15.23:0/1430757735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f830c102c10 0x7f830c198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f830b7fe700 1 -- 172.21.15.23:0/1430757735 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83000095f0 con 0x7f830c108230 2024-03-24T10:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f830b7fe700 1 --2- 172.21.15.23:0/1430757735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830c108230 0x7f830c19cf50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f830400ef30 tx=0x7f830400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f82f3fff700 1 -- 172.21.15.23:0/1430757735 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8304018070 con 0x7f830c108230 2024-03-24T10:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f83117bc700 1 -- 172.21.15.23:0/1430757735 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f830c19db10 con 0x7f830c108230 2024-03-24T10:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f82f3fff700 1 -- 172.21.15.23:0/1430757735 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f830400f040 con 0x7f830c108230 2024-03-24T10:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f82f3fff700 1 -- 172.21.15.23:0/1430757735 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830400cd50 con 0x7f830c108230 2024-03-24T10:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.297+0000 7f83117bc700 1 -- 172.21.15.23:0/1430757735 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f830c19ddc0 con 0x7f830c108230 2024-03-24T10:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.298+0000 7f83117bc700 1 -- 172.21.15.23:0/1430757735 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f830c10a710 con 0x7f830c108230 2024-03-24T10:39:02.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.300+0000 7f82f3fff700 1 -- 172.21.15.23:0/1430757735 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f8304003a00 con 0x7f830c108230 2024-03-24T10:39:02.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.300+0000 7f82f3fff700 1 --2- 172.21.15.23:0/1430757735 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f82f40a21e0 0x7f82f40a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:02.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.300+0000 7f830affd700 1 --2- 172.21.15.23:0/1430757735 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f82f40a21e0 0x7f82f40a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:02.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.300+0000 7f82f3fff700 1 -- 172.21.15.23:0/1430757735 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f83040cdc70 con 0x7f830c108230 2024-03-24T10:39:02.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.301+0000 7f830affd700 1 --2- 172.21.15.23:0/1430757735 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f82f40a21e0 0x7f82f40a46a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f82fc009c60 tx=0x7f82fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:02.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.304+0000 7f82f3fff700 1 -- 172.21.15.23:0/1430757735 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8304096d20 con 0x7f830c108230 2024-03-24T10:39:02.506 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:02 smithi117 ceph-mon[36368]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T10:39:02.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.514+0000 7f83117bc700 1 -- 172.21.15.23:0/1430757735 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f830c19ddc0 con 0x7f830c108230 2024-03-24T10:39:02.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:02 smithi023 ceph-mon[32466]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T10:39:02.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.514+0000 7f82f3fff700 1 -- 172.21.15.23:0/1430757735 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f830c19ddc0 con 0x7f830c108230 2024-03-24T10:39:02.515 INFO:teuthology.orchestra.run.smithi023.stdout:197568495664 2024-03-24T10:39:02.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.516+0000 7f83117bc700 1 -- 172.21.15.23:0/1430757735 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f82f40a21e0 msgr2=0x7f82f40a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:02.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.517+0000 7f83117bc700 1 --2- 172.21.15.23:0/1430757735 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f82f40a21e0 0x7f82f40a46a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f82fc009c60 tx=0x7f82fc009260 comp rx=0 tx=0).stop 2024-03-24T10:39:02.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.517+0000 7f83117bc700 1 -- 172.21.15.23:0/1430757735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830c108230 msgr2=0x7f830c19cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:02.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.517+0000 7f83117bc700 1 --2- 172.21.15.23:0/1430757735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830c108230 0x7f830c19cf50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f830400ef30 tx=0x7f830400c3a0 comp rx=0 tx=0).stop 2024-03-24T10:39:02.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.517+0000 7f83117bc700 1 -- 172.21.15.23:0/1430757735 shutdown_connections 2024-03-24T10:39:02.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.517+0000 7f83117bc700 1 --2- 172.21.15.23:0/1430757735 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f82f40a21e0 0x7f82f40a46a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:02.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.517+0000 7f83117bc700 1 --2- 172.21.15.23:0/1430757735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f830c102250 0x7f830c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:02.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.517+0000 7f83117bc700 1 --2- 172.21.15.23:0/1430757735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830c108230 0x7f830c19cf50 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:02.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.517+0000 7f83117bc700 1 --2- 172.21.15.23:0/1430757735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f830c102c10 0x7f830c198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:02.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.517+0000 7f83117bc700 1 -- 172.21.15.23:0/1430757735 >> 172.21.15.23:0/1430757735 conn(0x7f830c079330 msgr2=0x7f830c100e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:02.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.517+0000 7f83117bc700 1 -- 172.21.15.23:0/1430757735 shutdown_connections 2024-03-24T10:39:02.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:02.517+0000 7f83117bc700 1 -- 172.21.15.23:0/1430757735 wait complete. 2024-03-24T10:39:02.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:02 smithi116 ceph-mon[36248]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T10:39:03.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.125+0000 7f68104cf700 1 -- 172.21.15.23:0/2175676194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68081064b0 msgr2=0x7f680810e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:03.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.125+0000 7f68104cf700 1 --2- 172.21.15.23:0/2175676194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68081064b0 0x7f680810e080 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f680000b0a0 tx=0x7f680000b3b0 comp rx=0 tx=0).stop 2024-03-24T10:39:03.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.125+0000 7f68104cf700 1 -- 172.21.15.23:0/2175676194 shutdown_connections 2024-03-24T10:39:03.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.125+0000 7f68104cf700 1 --2- 172.21.15.23:0/2175676194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68081013d0 0x7f68081017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:03.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.125+0000 7f68104cf700 1 --2- 172.21.15.23:0/2175676194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68081064b0 0x7f680810e080 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:03.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.125+0000 7f68104cf700 1 --2- 172.21.15.23:0/2175676194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6808101d80 0x7f6808105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:03.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.125+0000 7f68104cf700 1 -- 172.21.15.23:0/2175676194 >> 172.21.15.23:0/2175676194 conn(0x7f6808078230 msgr2=0x7f6808078640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:03.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.126+0000 7f68104cf700 1 -- 172.21.15.23:0/2175676194 shutdown_connections 2024-03-24T10:39:03.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.126+0000 7f68104cf700 1 -- 172.21.15.23:0/2175676194 wait complete. 2024-03-24T10:39:03.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.126+0000 7f68104cf700 1 Processor -- start 2024-03-24T10:39:03.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.126+0000 7f68104cf700 1 -- start start 2024-03-24T10:39:03.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.126+0000 7f68104cf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68081013d0 0x7f68081a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:03.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.126+0000 7f68104cf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6808101d80 0x7f68081a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:03.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.126+0000 7f68104cf700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68081064b0 0x7f680819a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:03.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.126+0000 7f68104cf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f680819afe0 con 0x7f68081013d0 2024-03-24T10:39:03.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.126+0000 7f68104cf700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f680819b120 con 0x7f6808101d80 2024-03-24T10:39:03.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.126+0000 7f68104cf700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f680819b2e0 con 0x7f68081064b0 2024-03-24T10:39:03.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f680e26b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68081013d0 0x7f68081a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:03.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f680da6a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6808101d80 0x7f68081a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:03.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f680e26b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68081013d0 0x7f68081a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45738/0 (socket says 172.21.15.23:45738) 2024-03-24T10:39:03.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f680ea6c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68081064b0 0x7f680819a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:03.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f680e26b700 1 -- 172.21.15.23:0/114096595 learned_addr learned my addr 172.21.15.23:0/114096595 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:03.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f680e26b700 1 -- 172.21.15.23:0/114096595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68081064b0 msgr2=0x7f680819a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:03.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f680e26b700 1 --2- 172.21.15.23:0/114096595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68081064b0 0x7f680819a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:03.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f680e26b700 1 -- 172.21.15.23:0/114096595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6808101d80 msgr2=0x7f68081a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:03.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f680e26b700 1 --2- 172.21.15.23:0/114096595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6808101d80 0x7f68081a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:03.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f680e26b700 1 -- 172.21.15.23:0/114096595 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6800009d00 con 0x7f68081013d0 2024-03-24T10:39:03.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f680da6a700 1 --2- 172.21.15.23:0/114096595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6808101d80 0x7f68081a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:03.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f680e26b700 1 --2- 172.21.15.23:0/114096595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68081013d0 0x7f68081a07c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f680400e9a0 tx=0x7f680400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:03.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f67ff7fe700 1 -- 172.21.15.23:0/114096595 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6804009790 con 0x7f68081013d0 2024-03-24T10:39:03.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f67ff7fe700 1 -- 172.21.15.23:0/114096595 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6804010430 con 0x7f68081013d0 2024-03-24T10:39:03.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f68104cf700 1 -- 172.21.15.23:0/114096595 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f680819b5c0 con 0x7f68081013d0 2024-03-24T10:39:03.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f67ff7fe700 1 -- 172.21.15.23:0/114096595 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6804018680 con 0x7f68081013d0 2024-03-24T10:39:03.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.127+0000 7f68104cf700 1 -- 172.21.15.23:0/114096595 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68081a7860 con 0x7f68081013d0 2024-03-24T10:39:03.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.128+0000 7f68104cf700 1 -- 172.21.15.23:0/114096595 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6808104540 con 0x7f68081013d0 2024-03-24T10:39:03.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.129+0000 7f67ff7fe700 1 -- 172.21.15.23:0/114096595 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f6804010910 con 0x7f68081013d0 2024-03-24T10:39:03.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.130+0000 7f67ff7fe700 1 --2- 172.21.15.23:0/114096595 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f67f40a21e0 0x7f67f40a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:03.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.130+0000 7f680da6a700 1 --2- 172.21.15.23:0/114096595 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f67f40a21e0 0x7f67f40a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:03.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.130+0000 7f67ff7fe700 1 -- 172.21.15.23:0/114096595 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f6804014070 con 0x7f68081013d0 2024-03-24T10:39:03.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.130+0000 7f680da6a700 1 --2- 172.21.15.23:0/114096595 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f67f40a21e0 0x7f67f40a46a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6808107b80 tx=0x7f67f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:03.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.133+0000 7f67ff7fe700 1 -- 172.21.15.23:0/114096595 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f68040973a0 con 0x7f68081013d0 2024-03-24T10:39:03.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.340+0000 7f68104cf700 1 -- 172.21.15.23:0/114096595 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f680819beb0 con 0x7f68081013d0 2024-03-24T10:39:03.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.341+0000 7f67ff7fe700 1 -- 172.21.15.23:0/114096595 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f6804026090 con 0x7f68081013d0 2024-03-24T10:39:03.341 INFO:teuthology.orchestra.run.smithi023.stdout:343597383704 2024-03-24T10:39:03.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.343+0000 7f68104cf700 1 -- 172.21.15.23:0/114096595 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f67f40a21e0 msgr2=0x7f67f40a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:03.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.343+0000 7f68104cf700 1 --2- 172.21.15.23:0/114096595 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f67f40a21e0 0x7f67f40a46a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6808107b80 tx=0x7f67f8009260 comp rx=0 tx=0).stop 2024-03-24T10:39:03.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.343+0000 7f68104cf700 1 -- 172.21.15.23:0/114096595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68081013d0 msgr2=0x7f68081a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:03.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.343+0000 7f68104cf700 1 --2- 172.21.15.23:0/114096595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68081013d0 0x7f68081a07c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f680400e9a0 tx=0x7f680400ecb0 comp rx=0 tx=0).stop 2024-03-24T10:39:03.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.343+0000 7f68104cf700 1 -- 172.21.15.23:0/114096595 shutdown_connections 2024-03-24T10:39:03.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.343+0000 7f68104cf700 1 --2- 172.21.15.23:0/114096595 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f67f40a21e0 0x7f67f40a46a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:03.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.343+0000 7f68104cf700 1 --2- 172.21.15.23:0/114096595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68081064b0 0x7f680819a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:03.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.343+0000 7f68104cf700 1 --2- 172.21.15.23:0/114096595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68081013d0 0x7f68081a07c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:03.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.343+0000 7f68104cf700 1 --2- 172.21.15.23:0/114096595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6808101d80 0x7f68081a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:03.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.343+0000 7f68104cf700 1 -- 172.21.15.23:0/114096595 >> 172.21.15.23:0/114096595 conn(0x7f6808078230 msgr2=0x7f680806adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:03.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.343+0000 7f68104cf700 1 -- 172.21.15.23:0/114096595 shutdown_connections 2024-03-24T10:39:03.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:03.343+0000 7f68104cf700 1 -- 172.21.15.23:0/114096595 wait complete. 2024-03-24T10:39:03.354 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:03 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1430757735' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T10:39:03.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:03 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1430757735' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T10:39:03.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:03 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1430757735' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T10:39:04.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:04 smithi023 ceph-mon[32466]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T10:39:04.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:04 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/114096595' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T10:39:04.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:04 smithi117 ceph-mon[36368]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T10:39:04.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:04 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/114096595' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T10:39:04.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:04 smithi116 ceph-mon[36248]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T10:39:04.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:04 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/114096595' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T10:39:04.833 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383701 got 343597383704 for osd.11 2024-03-24T10:39:04.833 DEBUG:teuthology.parallel:result is None 2024-03-24T10:39:05.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.248+0000 7f779424e700 1 -- 172.21.15.23:0/1951284004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f778c074d50 msgr2=0x7f778c10e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:05.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.248+0000 7f779424e700 1 --2- 172.21.15.23:0/1951284004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f778c074d50 0x7f778c10e720 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f777c009990 tx=0x7f777c009ca0 comp rx=0 tx=0).stop 2024-03-24T10:39:05.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.248+0000 7f779424e700 1 -- 172.21.15.23:0/1951284004 shutdown_connections 2024-03-24T10:39:05.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.248+0000 7f779424e700 1 --2- 172.21.15.23:0/1951284004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f778c074430 0x7f778c074810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:05.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.248+0000 7f779424e700 1 --2- 172.21.15.23:0/1951284004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f778c10ec60 0x7f778c111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:05.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.248+0000 7f779424e700 1 --2- 172.21.15.23:0/1951284004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f778c074d50 0x7f778c10e720 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:05.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.248+0000 7f779424e700 1 -- 172.21.15.23:0/1951284004 >> 172.21.15.23:0/1951284004 conn(0x7f778c0792d0 msgr2=0x7f778c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:05.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.248+0000 7f779424e700 1 -- 172.21.15.23:0/1951284004 shutdown_connections 2024-03-24T10:39:05.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.248+0000 7f779424e700 1 -- 172.21.15.23:0/1951284004 wait complete. 2024-03-24T10:39:05.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f779424e700 1 Processor -- start 2024-03-24T10:39:05.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f779424e700 1 -- start start 2024-03-24T10:39:05.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f779424e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f778c074430 0x7f778c1a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:05.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f779424e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f778c074d50 0x7f778c1a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:05.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f779424e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f778c10ec60 0x7f778c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:05.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f779424e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f778c19ae40 con 0x7f778c10ec60 2024-03-24T10:39:05.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f779424e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f778c19afb0 con 0x7f778c074d50 2024-03-24T10:39:05.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f779424e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f778c19b170 con 0x7f778c074430 2024-03-24T10:39:05.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f77927eb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f778c10ec60 0x7f778c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:05.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f77917e9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f778c074d50 0x7f778c1a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:05.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f77927eb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f778c10ec60 0x7f778c19a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45754/0 (socket says 172.21.15.23:45754) 2024-03-24T10:39:05.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f7791fea700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f778c074430 0x7f778c1a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:05.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f77927eb700 1 -- 172.21.15.23:0/2153502137 learned_addr learned my addr 172.21.15.23:0/2153502137 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:05.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f77927eb700 1 -- 172.21.15.23:0/2153502137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f778c074430 msgr2=0x7f778c1a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:05.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f77927eb700 1 --2- 172.21.15.23:0/2153502137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f778c074430 0x7f778c1a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:05.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f77927eb700 1 -- 172.21.15.23:0/2153502137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f778c074d50 msgr2=0x7f778c1a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:05.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f77927eb700 1 --2- 172.21.15.23:0/2153502137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f778c074d50 0x7f778c1a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:05.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.249+0000 7f77927eb700 1 -- 172.21.15.23:0/2153502137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777c0095f0 con 0x7f778c10ec60 2024-03-24T10:39:05.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.250+0000 7f77927eb700 1 --2- 172.21.15.23:0/2153502137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f778c10ec60 0x7f778c19a790 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f778400ef90 tx=0x7f778400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:05.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.250+0000 7f7782ffd700 1 -- 172.21.15.23:0/2153502137 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7784018070 con 0x7f778c10ec60 2024-03-24T10:39:05.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.250+0000 7f7782ffd700 1 -- 172.21.15.23:0/2153502137 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f778400f040 con 0x7f778c10ec60 2024-03-24T10:39:05.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.250+0000 7f7782ffd700 1 -- 172.21.15.23:0/2153502137 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778400cdf0 con 0x7f778c10ec60 2024-03-24T10:39:05.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.250+0000 7f779424e700 1 -- 172.21.15.23:0/2153502137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f778c19b3f0 con 0x7f778c10ec60 2024-03-24T10:39:05.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.250+0000 7f779424e700 1 -- 172.21.15.23:0/2153502137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f778c1a7640 con 0x7f778c10ec60 2024-03-24T10:39:05.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.251+0000 7f779424e700 1 -- 172.21.15.23:0/2153502137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f778c109e70 con 0x7f778c10ec60 2024-03-24T10:39:05.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.252+0000 7f7782ffd700 1 -- 172.21.15.23:0/2153502137 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f7784004ad0 con 0x7f778c10ec60 2024-03-24T10:39:05.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.253+0000 7f7782ffd700 1 --2- 172.21.15.23:0/2153502137 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f77780a21e0 0x7f77780a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:05.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.253+0000 7f7791fea700 1 --2- 172.21.15.23:0/2153502137 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f77780a21e0 0x7f77780a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:05.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.253+0000 7f7782ffd700 1 -- 172.21.15.23:0/2153502137 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f77840cd760 con 0x7f778c10ec60 2024-03-24T10:39:05.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.253+0000 7f7791fea700 1 --2- 172.21.15.23:0/2153502137 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f77780a21e0 0x7f77780a46a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7788009c60 tx=0x7f7788009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:05.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.257+0000 7f7782ffd700 1 -- 172.21.15.23:0/2153502137 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7784096810 con 0x7f778c10ec60 2024-03-24T10:39:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.471+0000 7f779424e700 1 -- 172.21.15.23:0/2153502137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f778c19bcc0 con 0x7f778c10ec60 2024-03-24T10:39:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.471+0000 7f7782ffd700 1 -- 172.21.15.23:0/2153502137 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7784099ed0 con 0x7f778c10ec60 2024-03-24T10:39:05.472 INFO:teuthology.orchestra.run.smithi023.stdout:34359738446 2024-03-24T10:39:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.474+0000 7f779424e700 1 -- 172.21.15.23:0/2153502137 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f77780a21e0 msgr2=0x7f77780a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.474+0000 7f779424e700 1 --2- 172.21.15.23:0/2153502137 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f77780a21e0 0x7f77780a46a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7788009c60 tx=0x7f7788009260 comp rx=0 tx=0).stop 2024-03-24T10:39:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.474+0000 7f779424e700 1 -- 172.21.15.23:0/2153502137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f778c10ec60 msgr2=0x7f778c19a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.474+0000 7f779424e700 1 --2- 172.21.15.23:0/2153502137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f778c10ec60 0x7f778c19a790 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f778400ef90 tx=0x7f778400c3a0 comp rx=0 tx=0).stop 2024-03-24T10:39:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.474+0000 7f779424e700 1 -- 172.21.15.23:0/2153502137 shutdown_connections 2024-03-24T10:39:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.474+0000 7f779424e700 1 --2- 172.21.15.23:0/2153502137 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f77780a21e0 0x7f77780a46a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:05.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.474+0000 7f779424e700 1 --2- 172.21.15.23:0/2153502137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f778c074430 0x7f778c1a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:05.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.474+0000 7f779424e700 1 --2- 172.21.15.23:0/2153502137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f778c10ec60 0x7f778c19a790 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:05.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.474+0000 7f779424e700 1 --2- 172.21.15.23:0/2153502137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f778c074d50 0x7f778c1a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:05.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.474+0000 7f779424e700 1 -- 172.21.15.23:0/2153502137 >> 172.21.15.23:0/2153502137 conn(0x7f778c0792d0 msgr2=0x7f778c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:05.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.474+0000 7f779424e700 1 -- 172.21.15.23:0/2153502137 shutdown_connections 2024-03-24T10:39:05.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:05.474+0000 7f779424e700 1 -- 172.21.15.23:0/2153502137 wait complete. 2024-03-24T10:39:06.086 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:06.086 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:06.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:06 smithi023 ceph-mon[32466]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:39:06.285 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:06 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2153502137' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T10:39:06.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:06 smithi117 ceph-mon[36368]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:39:06.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:06 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2153502137' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T10:39:06.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:06 smithi116 ceph-mon[36248]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:39:06.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:06 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2153502137' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T10:39:07.198 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495659 got 197568495664 for osd.6 2024-03-24T10:39:07.198 DEBUG:teuthology.parallel:result is None 2024-03-24T10:39:07.215 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:07.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.750+0000 7f9c8eb71700 1 -- 172.21.15.23:0/757144866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88106400 msgr2=0x7f9c8810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:07.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.750+0000 7f9c8eb71700 1 --2- 172.21.15.23:0/757144866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88106400 0x7f9c8810dfd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000b0a0 tx=0x7f9c8000b3b0 comp rx=0 tx=0).stop 2024-03-24T10:39:07.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.750+0000 7f9c8eb71700 1 -- 172.21.15.23:0/757144866 shutdown_connections 2024-03-24T10:39:07.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.750+0000 7f9c8eb71700 1 --2- 172.21.15.23:0/757144866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c88101340 0x7f9c88101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:07.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.750+0000 7f9c8eb71700 1 --2- 172.21.15.23:0/757144866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88106400 0x7f9c8810dfd0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:07.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.750+0000 7f9c8eb71700 1 --2- 172.21.15.23:0/757144866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c88101cf0 0x7f9c88105d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:07.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.750+0000 7f9c8eb71700 1 -- 172.21.15.23:0/757144866 >> 172.21.15.23:0/757144866 conn(0x7f9c880780a0 msgr2=0x7f9c880784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:07.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.750+0000 7f9c8eb71700 1 -- 172.21.15.23:0/757144866 shutdown_connections 2024-03-24T10:39:07.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.751+0000 7f9c8eb71700 1 -- 172.21.15.23:0/757144866 wait complete. 2024-03-24T10:39:07.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.751+0000 7f9c8eb71700 1 Processor -- start 2024-03-24T10:39:07.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.751+0000 7f9c8eb71700 1 -- start start 2024-03-24T10:39:07.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.751+0000 7f9c8eb71700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88101340 0x7f9c8819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:07.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.751+0000 7f9c8eb71700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c88101cf0 0x7f9c8819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:07.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.751+0000 7f9c8eb71700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c88106400 0x7f9c88198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:07.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.751+0000 7f9c8eb71700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f9c88198e10 con 0x7f9c88101340 2024-03-24T10:39:07.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.751+0000 7f9c8eb71700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9c88198f80 con 0x7f9c88101cf0 2024-03-24T10:39:07.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.751+0000 7f9c8eb71700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9c88199140 con 0x7f9c88106400 2024-03-24T10:39:07.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.751+0000 7f9c8c90d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88101340 0x7f9c8819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:07.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.751+0000 7f9c8c90d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88101340 0x7f9c8819e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33108/0 (socket says 172.21.15.23:33108) 2024-03-24T10:39:07.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.751+0000 7f9c87fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c88101cf0 0x7f9c8819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:07.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.751+0000 7f9c8c90d700 1 -- 172.21.15.23:0/1956431950 learned_addr learned my addr 172.21.15.23:0/1956431950 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:07.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.752+0000 7f9c8d10e700 1 --2- 172.21.15.23:0/1956431950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c88106400 0x7f9c88198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:07.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.752+0000 7f9c8c90d700 1 -- 172.21.15.23:0/1956431950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c88106400 msgr2=0x7f9c88198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:07.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.752+0000 7f9c8c90d700 1 --2- 172.21.15.23:0/1956431950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c88106400 0x7f9c88198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:07.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.752+0000 7f9c8c90d700 1 -- 172.21.15.23:0/1956431950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c88101cf0 msgr2=0x7f9c8819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:07.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.752+0000 7f9c8c90d700 1 --2- 172.21.15.23:0/1956431950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c88101cf0 0x7f9c8819ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:07.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.752+0000 7f9c8c90d700 1 -- 172.21.15.23:0/1956431950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c80009d00 con 0x7f9c88101340 2024-03-24T10:39:07.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.752+0000 7f9c87fff700 1 --2- 172.21.15.23:0/1956431950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c88101cf0 0x7f9c8819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:07.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.752+0000 7f9c8c90d700 1 --2- 172.21.15.23:0/1956431950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88101340 0x7f9c8819e590 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c00e9a0 tx=0x7f9c7c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:07.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.752+0000 7f9c85ffb700 1 -- 172.21.15.23:0/1956431950 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c009790 con 0x7f9c88101340 2024-03-24T10:39:07.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.752+0000 7f9c85ffb700 1 -- 172.21.15.23:0/1956431950 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c7c010430 con 0x7f9c88101340 2024-03-24T10:39:07.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.752+0000 7f9c8eb71700 1 -- 172.21.15.23:0/1956431950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c88199420 con 0x7f9c88101340 2024-03-24T10:39:07.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.752+0000 7f9c85ffb700 1 -- 172.21.15.23:0/1956431950 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c018680 con 0x7f9c88101340 2024-03-24T10:39:07.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.752+0000 7f9c8eb71700 1 -- 172.21.15.23:0/1956431950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c881a5510 con 0x7f9c88101340 2024-03-24T10:39:07.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.753+0000 7f9c8eb71700 1 -- 172.21.15.23:0/1956431950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c8810a6c0 con 0x7f9c88101340 2024-03-24T10:39:07.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.754+0000 7f9c85ffb700 1 -- 172.21.15.23:0/1956431950 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f9c7c010910 con 0x7f9c88101340 2024-03-24T10:39:07.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.755+0000 7f9c85ffb700 1 --2- 172.21.15.23:0/1956431950 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9c780a21e0 0x7f9c780a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:07.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.755+0000 7f9c85ffb700 1 -- 172.21.15.23:0/1956431950 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f9c7c014070 con 0x7f9c88101340 2024-03-24T10:39:07.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.755+0000 7f9c87fff700 1 --2- 172.21.15.23:0/1956431950 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9c780a21e0 0x7f9c780a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:07.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.756+0000 7f9c87fff700 1 --2- 172.21.15.23:0/1956431950 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9c780a21e0 0x7f9c780a46a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9c88199e20 tx=0x7f9c74009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:07.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.760+0000 7f9c85ffb700 1 -- 172.21.15.23:0/1956431950 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9c7c0973a0 con 0x7f9c88101340 2024-03-24T10:39:07.975 INFO:teuthology.orchestra.run.smithi023.stdout:253403070503 2024-03-24T10:39:07.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.975+0000 7f9c8eb71700 1 -- 172.21.15.23:0/1956431950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f9c88102800 con 0x7f9c88101340 2024-03-24T10:39:07.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.975+0000 7f9c85ffb700 1 -- 172.21.15.23:0/1956431950 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9c7c026020 con 0x7f9c88101340 2024-03-24T10:39:07.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.977+0000 7f9c8eb71700 1 -- 172.21.15.23:0/1956431950 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9c780a21e0 msgr2=0x7f9c780a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.977+0000 7f9c8eb71700 1 --2- 172.21.15.23:0/1956431950 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9c780a21e0 0x7f9c780a46a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9c88199e20 tx=0x7f9c74009260 comp rx=0 tx=0).stop 2024-03-24T10:39:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.977+0000 7f9c8eb71700 1 -- 172.21.15.23:0/1956431950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88101340 msgr2=0x7f9c8819e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.977+0000 7f9c8eb71700 1 --2- 172.21.15.23:0/1956431950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88101340 0x7f9c8819e590 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c00e9a0 tx=0x7f9c7c00ecb0 comp rx=0 tx=0).stop 2024-03-24T10:39:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.977+0000 7f9c8eb71700 1 -- 172.21.15.23:0/1956431950 shutdown_connections 2024-03-24T10:39:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.977+0000 7f9c8eb71700 1 --2- 172.21.15.23:0/1956431950 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9c780a21e0 0x7f9c780a46a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.977+0000 7f9c8eb71700 1 --2- 172.21.15.23:0/1956431950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c88106400 0x7f9c88198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.977+0000 7f9c8eb71700 1 --2- 172.21.15.23:0/1956431950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88101340 0x7f9c8819e590 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.977+0000 7f9c8eb71700 1 --2- 172.21.15.23:0/1956431950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c88101cf0 0x7f9c8819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.977+0000 7f9c8eb71700 1 -- 172.21.15.23:0/1956431950 >> 172.21.15.23:0/1956431950 conn(0x7f9c880780a0 msgr2=0x7f9c8806ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.977+0000 7f9c8eb71700 1 -- 172.21.15.23:0/1956431950 shutdown_connections 2024-03-24T10:39:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:07.977+0000 7f9c8eb71700 1 -- 172.21.15.23:0/1956431950 wait complete. 2024-03-24T10:39:08.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:08 smithi023 ceph-mon[32466]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T10:39:08.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:08 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1956431950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T10:39:08.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:08 smithi117 ceph-mon[36368]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T10:39:08.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:08 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1956431950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T10:39:08.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:08 smithi116 ceph-mon[36248]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T10:39:08.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:08 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1956431950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T10:39:08.795 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738440 got 34359738446 for osd.0 2024-03-24T10:39:08.795 DEBUG:teuthology.parallel:result is None 2024-03-24T10:39:09.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:09 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:39:09.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:09 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:39:09.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:09 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:39:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.735+0000 7fd31bccf700 1 -- 172.21.15.23:0/2686522980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31406a920 msgr2=0x7fd31406ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.735+0000 7fd31bccf700 1 --2- 172.21.15.23:0/2686522980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31406a920 0x7fd31406ad00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd304009990 tx=0x7fd304009ca0 comp rx=0 tx=0).stop 2024-03-24T10:39:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.735+0000 7fd31bccf700 1 -- 172.21.15.23:0/2686522980 shutdown_connections 2024-03-24T10:39:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.735+0000 7fd31bccf700 1 --2- 172.21.15.23:0/2686522980 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd31410ec90 0x7fd314111080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.735+0000 7fd31bccf700 1 --2- 172.21.15.23:0/2686522980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31406a920 0x7fd31406ad00 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.735+0000 7fd31bccf700 1 --2- 172.21.15.23:0/2686522980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd31406b240 0x7fd31410e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.735+0000 7fd31bccf700 1 -- 172.21.15.23:0/2686522980 >> 172.21.15.23:0/2686522980 conn(0x7fd314078110 msgr2=0x7fd314078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd31bccf700 1 -- 172.21.15.23:0/2686522980 shutdown_connections 2024-03-24T10:39:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd31bccf700 1 -- 172.21.15.23:0/2686522980 wait complete. 2024-03-24T10:39:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd31bccf700 1 Processor -- start 2024-03-24T10:39:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd31bccf700 1 -- start start 2024-03-24T10:39:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd31bccf700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd31406a920 0x7fd3141a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd31bccf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31406b240 0x7fd3141a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd31bccf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd31410ec90 0x7fd31419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd31bccf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd31419af10 con 0x7fd31406b240 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd31bccf700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd31419b080 con 0x7fd31410ec90 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd31bccf700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd31419b240 con 0x7fd31406a920 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd31926a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31406b240 0x7fd3141a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd31926a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31406b240 0x7fd3141a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33128/0 (socket says 172.21.15.23:33128) 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd31926a700 1 -- 172.21.15.23:0/4233197011 learned_addr learned my addr 172.21.15.23:0/4233197011 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd31a26c700 1 --2- 172.21.15.23:0/4233197011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd31410ec90 0x7fd31419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.736+0000 7fd319a6b700 1 --2- 172.21.15.23:0/4233197011 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd31406a920 0x7fd3141a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.737+0000 7fd31926a700 1 -- 172.21.15.23:0/4233197011 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd31406a920 msgr2=0x7fd3141a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.737+0000 7fd31926a700 1 --2- 172.21.15.23:0/4233197011 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd31406a920 0x7fd3141a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.737+0000 7fd31926a700 1 -- 172.21.15.23:0/4233197011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd31410ec90 msgr2=0x7fd31419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.737+0000 7fd31926a700 1 --2- 172.21.15.23:0/4233197011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd31410ec90 0x7fd31419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.737+0000 7fd31926a700 1 -- 172.21.15.23:0/4233197011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3040095f0 con 0x7fd31406b240 2024-03-24T10:39:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.737+0000 7fd31926a700 1 --2- 172.21.15.23:0/4233197011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31406b240 0x7fd3141a0bd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd31000e9a0 tx=0x7fd31000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.737+0000 7fd30affd700 1 -- 172.21.15.23:0/4233197011 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd310009790 con 0x7fd31406b240 2024-03-24T10:39:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.737+0000 7fd30affd700 1 -- 172.21.15.23:0/4233197011 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd310010430 con 0x7fd31406b240 2024-03-24T10:39:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.737+0000 7fd31bccf700 1 -- 172.21.15.23:0/4233197011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31419b520 con 0x7fd31406b240 2024-03-24T10:39:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.737+0000 7fd30affd700 1 -- 172.21.15.23:0/4233197011 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd310018680 con 0x7fd31406b240 2024-03-24T10:39:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.737+0000 7fd31bccf700 1 -- 172.21.15.23:0/4233197011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3141a7660 con 0x7fd31406b240 2024-03-24T10:39:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.738+0000 7fd31bccf700 1 -- 172.21.15.23:0/4233197011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd314109eb0 con 0x7fd31406b240 2024-03-24T10:39:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.740+0000 7fd30affd700 1 -- 172.21.15.23:0/4233197011 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fd310010910 con 0x7fd31406b240 2024-03-24T10:39:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.740+0000 7fd30affd700 1 --2- 172.21.15.23:0/4233197011 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd3000a2260 0x7fd3000a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.741+0000 7fd319a6b700 1 --2- 172.21.15.23:0/4233197011 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd3000a2260 0x7fd3000a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.741+0000 7fd30affd700 1 -- 172.21.15.23:0/4233197011 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fd310014070 con 0x7fd31406b240 2024-03-24T10:39:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.741+0000 7fd319a6b700 1 --2- 172.21.15.23:0/4233197011 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd3000a2260 0x7fd3000a4720 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd304006010 tx=0x7fd30400b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.744+0000 7fd30affd700 1 -- 172.21.15.23:0/4233197011 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd310097320 con 0x7fd31406b240 2024-03-24T10:39:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.948+0000 7fd31bccf700 1 -- 172.21.15.23:0/4233197011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd3141016c0 con 0x7fd31406b240 2024-03-24T10:39:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.949+0000 7fd30affd700 1 -- 172.21.15.23:0/4233197011 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd310026090 con 0x7fd31406b240 2024-03-24T10:39:10.274 INFO:teuthology.orchestra.run.smithi023.stdout:141733920828 2024-03-24T10:39:10.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.951+0000 7fd31bccf700 1 -- 172.21.15.23:0/4233197011 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd3000a2260 msgr2=0x7fd3000a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:10.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.951+0000 7fd31bccf700 1 --2- 172.21.15.23:0/4233197011 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd3000a2260 0x7fd3000a4720 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd304006010 tx=0x7fd30400b280 comp rx=0 tx=0).stop 2024-03-24T10:39:10.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.951+0000 7fd31bccf700 1 -- 172.21.15.23:0/4233197011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31406b240 msgr2=0x7fd3141a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:10.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.951+0000 7fd31bccf700 1 --2- 172.21.15.23:0/4233197011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31406b240 0x7fd3141a0bd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd31000e9a0 tx=0x7fd31000ecb0 comp rx=0 tx=0).stop 2024-03-24T10:39:10.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.951+0000 7fd31bccf700 1 -- 172.21.15.23:0/4233197011 shutdown_connections 2024-03-24T10:39:10.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.951+0000 7fd31bccf700 1 --2- 172.21.15.23:0/4233197011 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd3000a2260 0x7fd3000a4720 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:10.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.951+0000 7fd31bccf700 1 --2- 172.21.15.23:0/4233197011 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd31406a920 0x7fd3141a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:10.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.951+0000 7fd31bccf700 1 --2- 172.21.15.23:0/4233197011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd31410ec90 0x7fd31419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:10.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.951+0000 7fd31bccf700 1 --2- 172.21.15.23:0/4233197011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31406b240 0x7fd3141a0bd0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:10.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.951+0000 7fd31bccf700 1 -- 172.21.15.23:0/4233197011 >> 172.21.15.23:0/4233197011 conn(0x7fd314078110 msgr2=0x7fd3141100e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:10.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.951+0000 7fd31bccf700 1 -- 172.21.15.23:0/4233197011 shutdown_connections 2024-03-24T10:39:10.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:09.951+0000 7fd31bccf700 1 -- 172.21.15.23:0/4233197011 wait complete. 2024-03-24T10:39:10.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:10 smithi117 ceph-mon[36368]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T10:39:10.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:10 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4233197011' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T10:39:10.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:10 smithi116 ceph-mon[36248]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T10:39:10.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:10 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4233197011' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T10:39:10.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:10 smithi023 ceph-mon[32466]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T10:39:10.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:10 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4233197011' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T10:39:10.837 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070498 got 253403070503 for osd.8 2024-03-24T10:39:10.837 DEBUG:teuthology.parallel:result is None 2024-03-24T10:39:11.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.465+0000 7f7168fb9700 1 -- 172.21.15.23:0/4189650741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7164101390 msgr2=0x7f7164101770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:11.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.465+0000 7f7168fb9700 1 --2- 172.21.15.23:0/4189650741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7164101390 0x7f7164101770 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7154009990 tx=0x7f7154009ca0 comp rx=0 tx=0).stop 2024-03-24T10:39:11.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.466+0000 7f7168fb9700 1 -- 172.21.15.23:0/4189650741 shutdown_connections 2024-03-24T10:39:11.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.466+0000 7f7168fb9700 1 --2- 172.21.15.23:0/4189650741 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7164106410 0x7f716410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:11.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.466+0000 7f7168fb9700 1 --2- 172.21.15.23:0/4189650741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7164101390 0x7f7164101770 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:11.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.466+0000 7f7168fb9700 1 --2- 172.21.15.23:0/4189650741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7164101d40 0x7f7164105d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:11.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.466+0000 7f7168fb9700 1 -- 172.21.15.23:0/4189650741 >> 172.21.15.23:0/4189650741 conn(0x7f7164077110 msgr2=0x7f7164077520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:11.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.466+0000 7f7168fb9700 1 -- 172.21.15.23:0/4189650741 shutdown_connections 2024-03-24T10:39:11.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.466+0000 7f7168fb9700 1 -- 172.21.15.23:0/4189650741 wait complete. 2024-03-24T10:39:11.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.466+0000 7f7168fb9700 1 Processor -- start 2024-03-24T10:39:11.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7168fb9700 1 -- start start 2024-03-24T10:39:11.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7168fb9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7164101390 0x7f7164102dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:11.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7168fb9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7164101d40 0x7f7164103310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:11.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7168fb9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7164106410 0x7f7164103850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:11.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7168fb9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f71641a6b30 con 0x7f7164101390 2024-03-24T10:39:11.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7168fb9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f71641a6ca0 con 0x7f7164101d40 2024-03-24T10:39:11.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7168fb9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f71641a6e60 con 0x7f7164106410 2024-03-24T10:39:11.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f716259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7164101390 0x7f7164102dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:11.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7161d9b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7164101d40 0x7f7164103310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:11.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7162d9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7164106410 0x7f7164103850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:11.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f716259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7164101390 0x7f7164102dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33150/0 (socket says 172.21.15.23:33150) 2024-03-24T10:39:11.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f716259c700 1 -- 172.21.15.23:0/1594033738 learned_addr learned my addr 172.21.15.23:0/1594033738 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:11.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7161d9b700 1 -- 172.21.15.23:0/1594033738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7164106410 msgr2=0x7f7164103850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:11.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7161d9b700 1 --2- 172.21.15.23:0/1594033738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7164106410 0x7f7164103850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:11.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7161d9b700 1 -- 172.21.15.23:0/1594033738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7164101390 msgr2=0x7f7164102dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:11.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7161d9b700 1 --2- 172.21.15.23:0/1594033738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7164101390 0x7f7164102dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:11.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7161d9b700 1 -- 172.21.15.23:0/1594033738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71540095f0 con 0x7f7164101d40 2024-03-24T10:39:11.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.467+0000 7f7162d9d700 1 --2- 172.21.15.23:0/1594033738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7164106410 0x7f7164103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:11.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.468+0000 7f7161d9b700 1 --2- 172.21.15.23:0/1594033738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7164101d40 0x7f7164103310 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f715800e9c0 tx=0x7f715800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:11.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.468+0000 7f71537fe700 1 -- 172.21.15.23:0/1594033738 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715800c9c0 con 0x7f7164101d40 2024-03-24T10:39:11.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.468+0000 7f71537fe700 1 -- 172.21.15.23:0/1594033738 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7158010430 con 0x7f7164101d40 2024-03-24T10:39:11.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.468+0000 7f7168fb9700 1 -- 172.21.15.23:0/1594033738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71641a7140 con 0x7f7164101d40 2024-03-24T10:39:11.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.468+0000 7f71537fe700 1 -- 172.21.15.23:0/1594033738 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7158018640 con 0x7f7164101d40 2024-03-24T10:39:11.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.468+0000 7f7168fb9700 1 -- 172.21.15.23:0/1594033738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71641a7640 con 0x7f7164101d40 2024-03-24T10:39:11.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.470+0000 7f7168fb9700 1 -- 172.21.15.23:0/1594033738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f716410a6b0 con 0x7f7164101d40 2024-03-24T10:39:11.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.471+0000 7f71537fe700 1 -- 172.21.15.23:0/1594033738 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f71580105a0 con 0x7f7164101d40 2024-03-24T10:39:11.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.471+0000 7f71537fe700 1 --2- 172.21.15.23:0/1594033738 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f714c0a21e0 0x7f714c0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:11.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.471+0000 7f716259c700 1 --2- 172.21.15.23:0/1594033738 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f714c0a21e0 0x7f714c0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:11.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.471+0000 7f71537fe700 1 -- 172.21.15.23:0/1594033738 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7158014070 con 0x7f7164101d40 2024-03-24T10:39:11.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.472+0000 7f716259c700 1 --2- 172.21.15.23:0/1594033738 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f714c0a21e0 0x7f714c0a46a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f715400b2e0 tx=0x7f715400b610 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:11.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.476+0000 7f71537fe700 1 -- 172.21.15.23:0/1594033738 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7158053280 con 0x7f7164101d40 2024-03-24T10:39:11.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.685+0000 7f7168fb9700 1 -- 172.21.15.23:0/1594033738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f7164104300 con 0x7f7164101d40 2024-03-24T10:39:11.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.686+0000 7f71537fe700 1 -- 172.21.15.23:0/1594033738 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7158096da0 con 0x7f7164101d40 2024-03-24T10:39:11.687 INFO:teuthology.orchestra.run.smithi023.stdout:287762808867 2024-03-24T10:39:11.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.688+0000 7f7168fb9700 1 -- 172.21.15.23:0/1594033738 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f714c0a21e0 msgr2=0x7f714c0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:11.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.688+0000 7f7168fb9700 1 --2- 172.21.15.23:0/1594033738 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f714c0a21e0 0x7f714c0a46a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f715400b2e0 tx=0x7f715400b610 comp rx=0 tx=0).stop 2024-03-24T10:39:11.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.688+0000 7f7168fb9700 1 -- 172.21.15.23:0/1594033738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7164101d40 msgr2=0x7f7164103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:11.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.688+0000 7f7168fb9700 1 --2- 172.21.15.23:0/1594033738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7164101d40 0x7f7164103310 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f715800e9c0 tx=0x7f715800ecd0 comp rx=0 tx=0).stop 2024-03-24T10:39:11.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.688+0000 7f7168fb9700 1 -- 172.21.15.23:0/1594033738 shutdown_connections 2024-03-24T10:39:11.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.688+0000 7f7168fb9700 1 --2- 172.21.15.23:0/1594033738 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f714c0a21e0 0x7f714c0a46a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:11.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.688+0000 7f7168fb9700 1 --2- 172.21.15.23:0/1594033738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7164106410 0x7f7164103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:11.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.688+0000 7f7168fb9700 1 --2- 172.21.15.23:0/1594033738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7164101390 0x7f7164102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:11.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.688+0000 7f7168fb9700 1 --2- 172.21.15.23:0/1594033738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7164101d40 0x7f7164103310 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:11.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.688+0000 7f7168fb9700 1 -- 172.21.15.23:0/1594033738 >> 172.21.15.23:0/1594033738 conn(0x7f7164077110 msgr2=0x7f716406acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:11.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.688+0000 7f7168fb9700 1 -- 172.21.15.23:0/1594033738 shutdown_connections 2024-03-24T10:39:11.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:11.688+0000 7f7168fb9700 1 -- 172.21.15.23:0/1594033738 wait complete. 2024-03-24T10:39:11.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:11 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:11.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:11 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:11.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:11 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:11.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:11 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:12.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:11 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:12.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:11 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:12.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:11 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:12.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:11 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:12.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:11 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:12.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:11 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:12.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:11 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:12.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:11 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:12.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.559+0000 7fb54357b700 1 -- 172.21.15.23:0/1510998563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb53c106410 msgr2=0x7fb53c10dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.559+0000 7fb54357b700 1 --2- 172.21.15.23:0/1510998563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb53c106410 0x7fb53c10dfe0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb534009990 tx=0x7fb534009ca0 comp rx=0 tx=0).stop 2024-03-24T10:39:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.559+0000 7fb54357b700 1 -- 172.21.15.23:0/1510998563 shutdown_connections 2024-03-24T10:39:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.559+0000 7fb54357b700 1 --2- 172.21.15.23:0/1510998563 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb53c101350 0x7fb53c101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.559+0000 7fb54357b700 1 --2- 172.21.15.23:0/1510998563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb53c106410 0x7fb53c10dfe0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.559+0000 7fb54357b700 1 --2- 172.21.15.23:0/1510998563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb53c101d00 0x7fb53c105d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.559+0000 7fb54357b700 1 -- 172.21.15.23:0/1510998563 >> 172.21.15.23:0/1510998563 conn(0x7fb53c078070 msgr2=0x7fb53c078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.559+0000 7fb54357b700 1 -- 172.21.15.23:0/1510998563 shutdown_connections 2024-03-24T10:39:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.559+0000 7fb54357b700 1 -- 172.21.15.23:0/1510998563 wait complete. 2024-03-24T10:39:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.560+0000 7fb54357b700 1 Processor -- start 2024-03-24T10:39:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.560+0000 7fb54357b700 1 -- start start 2024-03-24T10:39:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.560+0000 7fb54357b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb53c101350 0x7fb53c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.560+0000 7fb54357b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb53c101d00 0x7fb53c198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.560+0000 7fb54357b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb53c106410 0x7fb53c19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.560+0000 7fb54357b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c19d530 con 0x7fb53c106410 2024-03-24T10:39:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.560+0000 7fb54357b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c19d6a0 con 0x7fb53c101350 2024-03-24T10:39:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.560+0000 7fb54357b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c19d860 con 0x7fb53c101d00 2024-03-24T10:39:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.560+0000 7fb541b18700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb53c106410 0x7fb53c19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.560+0000 7fb541b18700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb53c106410 0x7fb53c19cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33158/0 (socket says 172.21.15.23:33158) 2024-03-24T10:39:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.560+0000 7fb541b18700 1 -- 172.21.15.23:0/3898960921 learned_addr learned my addr 172.21.15.23:0/3898960921 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.560+0000 7fb540b16700 1 --2- 172.21.15.23:0/3898960921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb53c101d00 0x7fb53c198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.560+0000 7fb541317700 1 --2- 172.21.15.23:0/3898960921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb53c101350 0x7fb53c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.561+0000 7fb541b18700 1 -- 172.21.15.23:0/3898960921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb53c101d00 msgr2=0x7fb53c198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.561+0000 7fb541b18700 1 --2- 172.21.15.23:0/3898960921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb53c101d00 0x7fb53c198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.561+0000 7fb541b18700 1 -- 172.21.15.23:0/3898960921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb53c101350 msgr2=0x7fb53c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.561+0000 7fb541b18700 1 --2- 172.21.15.23:0/3898960921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb53c101350 0x7fb53c198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.561+0000 7fb541b18700 1 -- 172.21.15.23:0/3898960921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5340095f0 con 0x7fb53c106410 2024-03-24T10:39:12.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.561+0000 7fb541b18700 1 --2- 172.21.15.23:0/3898960921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb53c106410 0x7fb53c19cf20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb534004730 tx=0x7fb534004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:12.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.561+0000 7fb53a7fc700 1 -- 172.21.15.23:0/3898960921 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53401d070 con 0x7fb53c106410 2024-03-24T10:39:12.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.561+0000 7fb53a7fc700 1 -- 172.21.15.23:0/3898960921 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb53400b610 con 0x7fb53c106410 2024-03-24T10:39:12.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.561+0000 7fb53a7fc700 1 -- 172.21.15.23:0/3898960921 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb534022ea0 con 0x7fb53c106410 2024-03-24T10:39:12.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.561+0000 7fb54357b700 1 -- 172.21.15.23:0/3898960921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb53c19dae0 con 0x7fb53c106410 2024-03-24T10:39:12.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.561+0000 7fb54357b700 1 -- 172.21.15.23:0/3898960921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb53c1a53a0 con 0x7fb53c106410 2024-03-24T10:39:12.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.562+0000 7fb54357b700 1 -- 172.21.15.23:0/3898960921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb53c10a6d0 con 0x7fb53c106410 2024-03-24T10:39:12.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.564+0000 7fb53a7fc700 1 -- 172.21.15.23:0/3898960921 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fb53400fb00 con 0x7fb53c106410 2024-03-24T10:39:12.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.564+0000 7fb53a7fc700 1 --2- 172.21.15.23:0/3898960921 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb5240a21e0 0x7fb5240a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:12.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.564+0000 7fb541317700 1 --2- 172.21.15.23:0/3898960921 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb5240a21e0 0x7fb5240a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:12.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.564+0000 7fb53a7fc700 1 -- 172.21.15.23:0/3898960921 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fb5340cf210 con 0x7fb53c106410 2024-03-24T10:39:12.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.565+0000 7fb541317700 1 --2- 172.21.15.23:0/3898960921 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb5240a21e0 0x7fb5240a46a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb52c00f540 tx=0x7fb52c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:12.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.569+0000 7fb53a7fc700 1 -- 172.21.15.23:0/3898960921 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb5340982c0 con 0x7fb53c106410 2024-03-24T10:39:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.777+0000 7fb54357b700 1 -- 172.21.15.23:0/3898960921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb53c06a150 con 0x7fb53c106410 2024-03-24T10:39:12.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:12 smithi023 ceph-mon[32466]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:39:12.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:12 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1594033738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T10:39:12.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.778+0000 7fb53a7fc700 1 -- 172.21.15.23:0/3898960921 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb53409b980 con 0x7fb53c106410 2024-03-24T10:39:12.779 INFO:teuthology.orchestra.run.smithi023.stdout:107374182465 2024-03-24T10:39:12.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.780+0000 7fb54357b700 1 -- 172.21.15.23:0/3898960921 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb5240a21e0 msgr2=0x7fb5240a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:12.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.780+0000 7fb54357b700 1 --2- 172.21.15.23:0/3898960921 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb5240a21e0 0x7fb5240a46a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb52c00f540 tx=0x7fb52c00b040 comp rx=0 tx=0).stop 2024-03-24T10:39:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.780+0000 7fb54357b700 1 -- 172.21.15.23:0/3898960921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb53c106410 msgr2=0x7fb53c19cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.780+0000 7fb54357b700 1 --2- 172.21.15.23:0/3898960921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb53c106410 0x7fb53c19cf20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb534004730 tx=0x7fb534004760 comp rx=0 tx=0).stop 2024-03-24T10:39:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.780+0000 7fb54357b700 1 -- 172.21.15.23:0/3898960921 shutdown_connections 2024-03-24T10:39:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.780+0000 7fb54357b700 1 --2- 172.21.15.23:0/3898960921 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb5240a21e0 0x7fb5240a46a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.780+0000 7fb54357b700 1 --2- 172.21.15.23:0/3898960921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb53c106410 0x7fb53c19cf20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.780+0000 7fb54357b700 1 --2- 172.21.15.23:0/3898960921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb53c101350 0x7fb53c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.780+0000 7fb54357b700 1 --2- 172.21.15.23:0/3898960921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb53c101d00 0x7fb53c198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.780+0000 7fb54357b700 1 -- 172.21.15.23:0/3898960921 >> 172.21.15.23:0/3898960921 conn(0x7fb53c078070 msgr2=0x7fb53c06ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.780+0000 7fb54357b700 1 -- 172.21.15.23:0/3898960921 shutdown_connections 2024-03-24T10:39:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:12.780+0000 7fb54357b700 1 -- 172.21.15.23:0/3898960921 wait complete. 2024-03-24T10:39:13.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:12 smithi117 ceph-mon[36368]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:39:13.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:12 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1594033738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T10:39:13.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:12 smithi116 ceph-mon[36248]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:39:13.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:12 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1594033738' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T10:39:13.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.127+0000 7f2e250af700 1 -- 172.21.15.23:0/1030096258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e20101aa0 msgr2=0x7f2e2010df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:13.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.127+0000 7f2e250af700 1 --2- 172.21.15.23:0/1030096258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e20101aa0 0x7f2e2010df80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800b0a0 tx=0x7f2e1800b3b0 comp rx=0 tx=0).stop 2024-03-24T10:39:13.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.127+0000 7f2e250af700 1 -- 172.21.15.23:0/1030096258 shutdown_connections 2024-03-24T10:39:13.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.127+0000 7f2e250af700 1 --2- 172.21.15.23:0/1030096258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e20107140 0x7f2e20107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:13.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.127+0000 7f2e250af700 1 --2- 172.21.15.23:0/1030096258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e20101aa0 0x7f2e2010df80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:13.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.127+0000 7f2e250af700 1 --2- 172.21.15.23:0/1030096258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20101100 0x7f2e20101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:13.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.127+0000 7f2e250af700 1 -- 172.21.15.23:0/1030096258 >> 172.21.15.23:0/1030096258 conn(0x7f2e20078030 msgr2=0x7f2e20078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:13.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.127+0000 7f2e250af700 1 -- 172.21.15.23:0/1030096258 shutdown_connections 2024-03-24T10:39:13.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.127+0000 7f2e250af700 1 -- 172.21.15.23:0/1030096258 wait complete. 2024-03-24T10:39:13.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.128+0000 7f2e250af700 1 Processor -- start 2024-03-24T10:39:13.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.128+0000 7f2e250af700 1 -- start start 2024-03-24T10:39:13.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.128+0000 7f2e250af700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e20101100 0x7f2e20199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:13.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.128+0000 7f2e250af700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e20101aa0 0x7f2e20199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:13.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.128+0000 7f2e250af700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20107140 0x7f2e201a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:13.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.128+0000 7f2e250af700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2019a470 con 0x7f2e20101100 2024-03-24T10:39:13.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.128+0000 7f2e250af700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2e201a4c50 con 0x7f2e20107140 2024-03-24T10:39:13.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.128+0000 7f2e250af700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2e201a4d90 con 0x7f2e20101aa0 2024-03-24T10:39:13.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.128+0000 7f2e1ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e20101100 0x7f2e20199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:13.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.128+0000 7f2e1ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e20101100 0x7f2e20199690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33164/0 (socket says 172.21.15.23:33164) 2024-03-24T10:39:13.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.128+0000 7f2e1ed9d700 1 -- 172.21.15.23:0/1903454785 learned_addr learned my addr 172.21.15.23:0/1903454785 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:13.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.128+0000 7f2e1e59c700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e20101aa0 0x7f2e20199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:13.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.128+0000 7f2e1f59e700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20107140 0x7f2e201a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:13.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.129+0000 7f2e1ed9d700 1 -- 172.21.15.23:0/1903454785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e20101aa0 msgr2=0x7f2e20199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:13.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.129+0000 7f2e1ed9d700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e20101aa0 0x7f2e20199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:13.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.129+0000 7f2e1ed9d700 1 -- 172.21.15.23:0/1903454785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20107140 msgr2=0x7f2e201a4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:13.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.129+0000 7f2e1ed9d700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20107140 0x7f2e201a4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:13.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.129+0000 7f2e1ed9d700 1 -- 172.21.15.23:0/1903454785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e18009d00 con 0x7f2e20101100 2024-03-24T10:39:13.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.129+0000 7f2e1e59c700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e20101aa0 0x7f2e20199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:13.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.129+0000 7f2e1ed9d700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e20101100 0x7f2e20199690 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2e1000e9d0 tx=0x7f2e1000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:13.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.129+0000 7f2e07fff700 1 -- 172.21.15.23:0/1903454785 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e10009790 con 0x7f2e20101100 2024-03-24T10:39:13.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.129+0000 7f2e07fff700 1 -- 172.21.15.23:0/1903454785 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e1000ee70 con 0x7f2e20101100 2024-03-24T10:39:13.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.129+0000 7f2e250af700 1 -- 172.21.15.23:0/1903454785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e201a4fe0 con 0x7f2e20101100 2024-03-24T10:39:13.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.129+0000 7f2e07fff700 1 -- 172.21.15.23:0/1903454785 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e10018680 con 0x7f2e20101100 2024-03-24T10:39:13.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.129+0000 7f2e250af700 1 -- 172.21.15.23:0/1903454785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e201a54b0 con 0x7f2e20101100 2024-03-24T10:39:13.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.130+0000 7f2e250af700 1 -- 172.21.15.23:0/1903454785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e20109660 con 0x7f2e20101100 2024-03-24T10:39:13.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.131+0000 7f2e07fff700 1 -- 172.21.15.23:0/1903454785 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f2e1000cbd0 con 0x7f2e20101100 2024-03-24T10:39:13.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.131+0000 7f2e07fff700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2e080a21e0 0x7f2e080a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:13.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.132+0000 7f2e1e59c700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2e080a21e0 0x7f2e080a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:13.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.132+0000 7f2e07fff700 1 -- 172.21.15.23:0/1903454785 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f2e10014070 con 0x7f2e20101100 2024-03-24T10:39:13.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.132+0000 7f2e1e59c700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2e080a21e0 0x7f2e080a46a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2e200786a0 tx=0x7f2e14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:13.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.135+0000 7f2e07fff700 1 -- 172.21.15.23:0/1903454785 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2e10096f80 con 0x7f2e20101100 2024-03-24T10:39:13.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.342+0000 7f2e250af700 1 -- 172.21.15.23:0/1903454785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f2e20111110 con 0x7f2e20101100 2024-03-24T10:39:13.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.343+0000 7f2e07fff700 1 -- 172.21.15.23:0/1903454785 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2e1009a640 con 0x7f2e20101100 2024-03-24T10:39:13.344 INFO:teuthology.orchestra.run.smithi023.stdout:219043332141 2024-03-24T10:39:13.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.345+0000 7f2e250af700 1 -- 172.21.15.23:0/1903454785 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2e080a21e0 msgr2=0x7f2e080a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:13.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.345+0000 7f2e250af700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2e080a21e0 0x7f2e080a46a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2e200786a0 tx=0x7f2e14009260 comp rx=0 tx=0).stop 2024-03-24T10:39:13.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.345+0000 7f2e250af700 1 -- 172.21.15.23:0/1903454785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e20101100 msgr2=0x7f2e20199690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:13.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.345+0000 7f2e250af700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e20101100 0x7f2e20199690 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2e1000e9d0 tx=0x7f2e1000ece0 comp rx=0 tx=0).stop 2024-03-24T10:39:13.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.345+0000 7f2e250af700 1 -- 172.21.15.23:0/1903454785 shutdown_connections 2024-03-24T10:39:13.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.345+0000 7f2e250af700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2e080a21e0 0x7f2e080a46a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:13.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.345+0000 7f2e250af700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e20107140 0x7f2e201a4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:13.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.345+0000 7f2e250af700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e20101100 0x7f2e20199690 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:13.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.345+0000 7f2e250af700 1 --2- 172.21.15.23:0/1903454785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e20101aa0 0x7f2e20199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:13.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.345+0000 7f2e250af700 1 -- 172.21.15.23:0/1903454785 >> 172.21.15.23:0/1903454785 conn(0x7f2e20078030 msgr2=0x7f2e200ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:13.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.345+0000 7f2e250af700 1 -- 172.21.15.23:0/1903454785 shutdown_connections 2024-03-24T10:39:13.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:13.345+0000 7f2e250af700 1 -- 172.21.15.23:0/1903454785 wait complete. 2024-03-24T10:39:14.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:13 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3898960921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T10:39:14.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:13 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1903454785' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T10:39:14.005 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:13 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3898960921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T10:39:14.005 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:13 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1903454785' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T10:39:14.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:13 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3898960921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T10:39:14.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:13 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1903454785' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T10:39:14.653 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920824 got 141733920828 for osd.4 2024-03-24T10:39:14.653 DEBUG:teuthology.parallel:result is None 2024-03-24T10:39:14.784 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:14 smithi023 ceph-mon[32466]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:39:15.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:14 smithi117 ceph-mon[36368]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:39:15.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:14 smithi116 ceph-mon[36248]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:39:15.274 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182459 got 107374182465 for osd.3 2024-03-24T10:39:15.274 DEBUG:teuthology.parallel:result is None 2024-03-24T10:39:15.368 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808863 got 287762808867 for osd.9 2024-03-24T10:39:15.368 DEBUG:teuthology.parallel:result is None 2024-03-24T10:39:15.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-24T10:39:15.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-24T10:39:15.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-24T10:39:15.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-03-24T10:39:15.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-24T10:39:15.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-03-24T10:39:15.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-03-24T10:39:15.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 7]}]: dispatch 2024-03-24T10:39:15.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-03-24T10:39:15.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:15 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-24T10:39:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-24T10:39:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-24T10:39:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-24T10:39:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-03-24T10:39:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-24T10:39:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-03-24T10:39:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-03-24T10:39:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 7]}]: dispatch 2024-03-24T10:39:16.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-03-24T10:39:16.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:15 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-24T10:39:16.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-24T10:39:16.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-24T10:39:16.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-24T10:39:16.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-03-24T10:39:16.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-24T10:39:16.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-03-24T10:39:16.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-03-24T10:39:16.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 7]}]: dispatch 2024-03-24T10:39:16.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-03-24T10:39:16.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:15 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-03-24T10:39:16.347 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332136 got 219043332141 for osd.7 2024-03-24T10:39:16.347 DEBUG:teuthology.parallel:result is None 2024-03-24T10:39:16.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.718+0000 7fb29c42a700 1 -- 172.21.15.23:0/1557983286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb294107b30 msgr2=0x7fb2941007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:16.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.718+0000 7fb29c42a700 1 --2- 172.21.15.23:0/1557983286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb294107b30 0x7fb2941007f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb284009990 tx=0x7fb284009ca0 comp rx=0 tx=0).stop 2024-03-24T10:39:16.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.718+0000 7fb29c42a700 1 -- 172.21.15.23:0/1557983286 shutdown_connections 2024-03-24T10:39:16.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.718+0000 7fb29c42a700 1 --2- 172.21.15.23:0/1557983286 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb294107180 0x7fb294107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:16.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.718+0000 7fb29c42a700 1 --2- 172.21.15.23:0/1557983286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb294100f20 0x7fb294104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:16.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.718+0000 7fb29c42a700 1 --2- 172.21.15.23:0/1557983286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb294107b30 0x7fb2941007f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:16.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.718+0000 7fb29c42a700 1 -- 172.21.15.23:0/1557983286 >> 172.21.15.23:0/1557983286 conn(0x7fb294077700 msgr2=0x7fb2940feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.718+0000 7fb29c42a700 1 -- 172.21.15.23:0/1557983286 shutdown_connections 2024-03-24T10:39:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.718+0000 7fb29c42a700 1 -- 172.21.15.23:0/1557983286 wait complete. 2024-03-24T10:39:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29c42a700 1 Processor -- start 2024-03-24T10:39:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29c42a700 1 -- start start 2024-03-24T10:39:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29c42a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb294100f20 0x7fb29419edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29c42a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb294107180 0x7fb29419f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29c42a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2941a3680 0x7fb2941a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29c42a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb2941a4070 con 0x7fb294100f20 2024-03-24T10:39:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29c42a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb2941a41e0 con 0x7fb2941a3680 2024-03-24T10:39:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29c42a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb2941a43a0 con 0x7fb294107180 2024-03-24T10:39:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29a1c6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb294100f20 0x7fb29419edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29a1c6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb294100f20 0x7fb29419edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33178/0 (socket says 172.21.15.23:33178) 2024-03-24T10:39:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29a1c6700 1 -- 172.21.15.23:0/2607114062 learned_addr learned my addr 172.21.15.23:0/2607114062 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb2999c5700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb294107180 0x7fb29419f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29a9c7700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2941a3680 0x7fb2941a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29a1c6700 1 -- 172.21.15.23:0/2607114062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb294107180 msgr2=0x7fb29419f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29a1c6700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb294107180 0x7fb29419f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29a1c6700 1 -- 172.21.15.23:0/2607114062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2941a3680 msgr2=0x7fb2941a3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29a1c6700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2941a3680 0x7fb2941a3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29a1c6700 1 -- 172.21.15.23:0/2607114062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2840095f0 con 0x7fb294100f20 2024-03-24T10:39:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.719+0000 7fb29a9c7700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2941a3680 0x7fb2941a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:39:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.720+0000 7fb29a1c6700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb294100f20 0x7fb29419edb0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00ef10 tx=0x7fb28c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.720+0000 7fb28b7fe700 1 -- 172.21.15.23:0/2607114062 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c00ca70 con 0x7fb294100f20 2024-03-24T10:39:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.720+0000 7fb28b7fe700 1 -- 172.21.15.23:0/2607114062 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb28c00f040 con 0x7fb294100f20 2024-03-24T10:39:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.720+0000 7fb29c42a700 1 -- 172.21.15.23:0/2607114062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2941ab930 con 0x7fb294100f20 2024-03-24T10:39:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.720+0000 7fb28b7fe700 1 -- 172.21.15.23:0/2607114062 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c0146e0 con 0x7fb294100f20 2024-03-24T10:39:16.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.720+0000 7fb29c42a700 1 -- 172.21.15.23:0/2607114062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2941abda0 con 0x7fb294100f20 2024-03-24T10:39:16.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.721+0000 7fb29c42a700 1 -- 172.21.15.23:0/2607114062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb294101800 con 0x7fb294100f20 2024-03-24T10:39:16.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.722+0000 7fb28b7fe700 1 -- 172.21.15.23:0/2607114062 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fb28c007500 con 0x7fb294100f20 2024-03-24T10:39:16.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.723+0000 7fb28b7fe700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb2800a21e0 0x7fb2800a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:16.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.723+0000 7fb2999c5700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb2800a21e0 0x7fb2800a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:16.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.723+0000 7fb28b7fe700 1 -- 172.21.15.23:0/2607114062 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb28c0ce0c0 con 0x7fb294100f20 2024-03-24T10:39:16.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.723+0000 7fb2999c5700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb2800a21e0 0x7fb2800a46a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb284006010 tx=0x7fb284005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:16.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.728+0000 7fb28b7fe700 1 -- 172.21.15.23:0/2607114062 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb28c096f10 con 0x7fb294100f20 2024-03-24T10:39:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:16 smithi023 ceph-mon[32466]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:39:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-24T10:39:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-24T10:39:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-24T10:39:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-03-24T10:39:16.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-24T10:39:16.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-03-24T10:39:16.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-03-24T10:39:16.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 7]}]': finished 2024-03-24T10:39:16.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-03-24T10:39:16.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:16 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-24T10:39:16.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:16 smithi023 ceph-mon[32466]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T10:39:16.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.943+0000 7fb29c42a700 1 -- 172.21.15.23:0/2607114062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fb294107bf0 con 0x7fb294100f20 2024-03-24T10:39:16.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.943+0000 7fb28b7fe700 1 -- 172.21.15.23:0/2607114062 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fb28c09a5d0 con 0x7fb294100f20 2024-03-24T10:39:16.945 INFO:teuthology.orchestra.run.smithi023.stdout:309237645343 2024-03-24T10:39:16.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.945+0000 7fb29c42a700 1 -- 172.21.15.23:0/2607114062 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb2800a21e0 msgr2=0x7fb2800a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:16.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.945+0000 7fb29c42a700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb2800a21e0 0x7fb2800a46a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb284006010 tx=0x7fb284005960 comp rx=0 tx=0).stop 2024-03-24T10:39:16.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.945+0000 7fb29c42a700 1 -- 172.21.15.23:0/2607114062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb294100f20 msgr2=0x7fb29419edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:16.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.945+0000 7fb29c42a700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb294100f20 0x7fb29419edb0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00ef10 tx=0x7fb28c00c3a0 comp rx=0 tx=0).stop 2024-03-24T10:39:16.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.946+0000 7fb29c42a700 1 -- 172.21.15.23:0/2607114062 shutdown_connections 2024-03-24T10:39:16.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.946+0000 7fb29c42a700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb2800a21e0 0x7fb2800a46a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:16.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.946+0000 7fb29c42a700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2941a3680 0x7fb2941a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:16.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.946+0000 7fb29c42a700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb294100f20 0x7fb29419edb0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:16.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.946+0000 7fb29c42a700 1 --2- 172.21.15.23:0/2607114062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb294107180 0x7fb29419f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:16.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.946+0000 7fb29c42a700 1 -- 172.21.15.23:0/2607114062 >> 172.21.15.23:0/2607114062 conn(0x7fb294077700 msgr2=0x7fb294102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:16.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.946+0000 7fb29c42a700 1 -- 172.21.15.23:0/2607114062 shutdown_connections 2024-03-24T10:39:16.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.946+0000 7fb29c42a700 1 -- 172.21.15.23:0/2607114062 wait complete. 2024-03-24T10:39:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:16 smithi117 ceph-mon[36368]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:39:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-24T10:39:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-24T10:39:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-24T10:39:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-03-24T10:39:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-24T10:39:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-03-24T10:39:17.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-03-24T10:39:17.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 7]}]': finished 2024-03-24T10:39:17.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-03-24T10:39:17.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:16 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-24T10:39:17.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:16 smithi117 ceph-mon[36368]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T10:39:17.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:16 smithi116 ceph-mon[36248]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:39:17.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-24T10:39:17.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-24T10:39:17.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-24T10:39:17.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-03-24T10:39:17.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-24T10:39:17.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-03-24T10:39:17.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-03-24T10:39:17.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 7]}]': finished 2024-03-24T10:39:17.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-03-24T10:39:17.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:16 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-03-24T10:39:17.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:16 smithi116 ceph-mon[36248]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T10:39:17.785 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645338 got 309237645343 for osd.10 2024-03-24T10:39:17.786 DEBUG:teuthology.parallel:result is None 2024-03-24T10:39:17.987 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:17 smithi023 ceph-mon[32466]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T10:39:17.987 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:17 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2607114062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T10:39:17.987 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:17 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2026855174' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T10:39:18.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:17 smithi117 ceph-mon[36368]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T10:39:18.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:17 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2607114062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T10:39:18.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:17 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2026855174' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T10:39:18.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:17 smithi116 ceph-mon[36248]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T10:39:18.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:17 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2607114062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T10:39:18.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:17 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2026855174' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T10:39:18.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.229+0000 7f678a8b3700 1 -- 172.21.15.23:0/1313781432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67841013e0 msgr2=0x7f6784101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:18.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.229+0000 7f678a8b3700 1 --2- 172.21.15.23:0/1313781432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67841013e0 0x7f6784101860 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f676c0098f0 tx=0x7f676c009c00 comp rx=0 tx=0).stop 2024-03-24T10:39:18.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.229+0000 7f678a8b3700 1 -- 172.21.15.23:0/1313781432 shutdown_connections 2024-03-24T10:39:18.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.229+0000 7f678a8b3700 1 --2- 172.21.15.23:0/1313781432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6784101da0 0x7f678410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.229+0000 7f678a8b3700 1 --2- 172.21.15.23:0/1313781432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f67841072f0 0x7f67841076d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.229+0000 7f678a8b3700 1 --2- 172.21.15.23:0/1313781432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67841013e0 0x7f6784101860 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.229+0000 7f678a8b3700 1 -- 172.21.15.23:0/1313781432 >> 172.21.15.23:0/1313781432 conn(0x7f6784077f60 msgr2=0x7f6784078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:18.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.229+0000 7f678a8b3700 1 -- 172.21.15.23:0/1313781432 shutdown_connections 2024-03-24T10:39:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.229+0000 7f678a8b3700 1 -- 172.21.15.23:0/1313781432 wait complete. 2024-03-24T10:39:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f678a8b3700 1 Processor -- start 2024-03-24T10:39:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f678a8b3700 1 -- start start 2024-03-24T10:39:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f678a8b3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67841013e0 0x7f678419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f678a8b3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6784101da0 0x7f678419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f678a8b3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f67841072f0 0x7f6784198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f678a8b3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f6784198c80 con 0x7f67841072f0 2024-03-24T10:39:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f678a8b3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6784198dc0 con 0x7f6784101da0 2024-03-24T10:39:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f678a8b3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6784198f80 con 0x7f67841013e0 2024-03-24T10:39:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f6788e50700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f67841072f0 0x7f6784198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f67837fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6784101da0 0x7f678419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f6788e50700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f67841072f0 0x7f6784198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50086/0 (socket says 172.21.15.23:50086) 2024-03-24T10:39:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f6783fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67841013e0 0x7f678419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f6788e50700 1 -- 172.21.15.23:0/3145219197 learned_addr learned my addr 172.21.15.23:0/3145219197 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f6788e50700 1 -- 172.21.15.23:0/3145219197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67841013e0 msgr2=0x7f678419e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f6788e50700 1 --2- 172.21.15.23:0/3145219197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67841013e0 0x7f678419e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f6788e50700 1 -- 172.21.15.23:0/3145219197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6784101da0 msgr2=0x7f678419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f6788e50700 1 --2- 172.21.15.23:0/3145219197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6784101da0 0x7f678419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f6788e50700 1 -- 172.21.15.23:0/3145219197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f676c009520 con 0x7f67841072f0 2024-03-24T10:39:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.230+0000 7f6783fff700 1 --2- 172.21.15.23:0/3145219197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67841013e0 0x7f678419e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.231+0000 7f6788e50700 1 --2- 172.21.15.23:0/3145219197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f67841072f0 0x7f6784198620 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f677c00ef90 tx=0x7f677c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.231+0000 7f67817fa700 1 -- 172.21.15.23:0/3145219197 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c018070 con 0x7f67841072f0 2024-03-24T10:39:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.231+0000 7f678a8b3700 1 -- 172.21.15.23:0/3145219197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6784199200 con 0x7f67841072f0 2024-03-24T10:39:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.231+0000 7f67817fa700 1 -- 172.21.15.23:0/3145219197 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f677c00f040 con 0x7f67841072f0 2024-03-24T10:39:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.231+0000 7f67817fa700 1 -- 172.21.15.23:0/3145219197 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c00cd00 con 0x7f67841072f0 2024-03-24T10:39:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.231+0000 7f678a8b3700 1 -- 172.21.15.23:0/3145219197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67841a54b0 con 0x7f67841072f0 2024-03-24T10:39:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.232+0000 7f678a8b3700 1 -- 172.21.15.23:0/3145219197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6784068ec0 con 0x7f67841072f0 2024-03-24T10:39:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.233+0000 7f67817fa700 1 -- 172.21.15.23:0/3145219197 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f677c0075c0 con 0x7f67841072f0 2024-03-24T10:39:18.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.233+0000 7f67817fa700 1 --2- 172.21.15.23:0/3145219197 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f67700a21e0 0x7f67700a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:18.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.233+0000 7f6783fff700 1 --2- 172.21.15.23:0/3145219197 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f67700a21e0 0x7f67700a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:18.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.234+0000 7f67817fa700 1 -- 172.21.15.23:0/3145219197 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f677c0cd960 con 0x7f67841072f0 2024-03-24T10:39:18.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.234+0000 7f6783fff700 1 --2- 172.21.15.23:0/3145219197 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f67700a21e0 0x7f67700a46a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6774009fd0 tx=0x7f6774009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:18.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.238+0000 7f67817fa700 1 -- 172.21.15.23:0/3145219197 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f677c096840 con 0x7f67841072f0 2024-03-24T10:39:18.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.821+0000 7f8a881c5700 1 -- 172.21.15.23:0/3758221968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a80101c50 msgr2=0x7f8a801020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:18.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.821+0000 7f8a881c5700 1 --2- 172.21.15.23:0/3758221968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a80101c50 0x7f8a801020d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009990 tx=0x7f8a70009ca0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.821+0000 7f8a881c5700 1 -- 172.21.15.23:0/3758221968 shutdown_connections 2024-03-24T10:39:18.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.821+0000 7f8a881c5700 1 --2- 172.21.15.23:0/3758221968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a80102610 0x7f8a8010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.821+0000 7f8a881c5700 1 --2- 172.21.15.23:0/3758221968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80106140 0x7f8a80106520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.821+0000 7f8a881c5700 1 --2- 172.21.15.23:0/3758221968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a80101c50 0x7f8a801020d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.821+0000 7f8a881c5700 1 -- 172.21.15.23:0/3758221968 >> 172.21.15.23:0/3758221968 conn(0x7f8a80076440 msgr2=0x7f8a80076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:18.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.822+0000 7f8a881c5700 1 -- 172.21.15.23:0/3758221968 shutdown_connections 2024-03-24T10:39:18.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.822+0000 7f8a881c5700 1 -- 172.21.15.23:0/3758221968 wait complete. 2024-03-24T10:39:18.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.822+0000 7f8a881c5700 1 Processor -- start 2024-03-24T10:39:18.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.822+0000 7f8a881c5700 1 -- start start 2024-03-24T10:39:18.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a881c5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80101c50 0x7f8a801976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:18.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a881c5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a80102610 0x7f8a80197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:18.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a881c5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a80106140 0x7f8a8019bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:18.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a881c5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8019c560 con 0x7f8a80106140 2024-03-24T10:39:18.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a881c5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8019c6d0 con 0x7f8a80101c50 2024-03-24T10:39:18.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a881c5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8019c890 con 0x7f8a80102610 2024-03-24T10:39:18.354 INFO:teuthology.orchestra.run.smithi023.stdout:77309411398 2024-03-24T10:39:18.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a86762700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a80106140 0x7f8a8019bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:18.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a86762700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a80106140 0x7f8a8019bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33198/0 (socket says 172.21.15.23:33198) 2024-03-24T10:39:18.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a85f61700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80101c50 0x7f8a801976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:18.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a86762700 1 -- 172.21.15.23:0/2026855174 learned_addr learned my addr 172.21.15.23:0/2026855174 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:18.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a85760700 1 --2- 172.21.15.23:0/2026855174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a80102610 0x7f8a80197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:18.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a85f61700 1 -- 172.21.15.23:0/2026855174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a80102610 msgr2=0x7f8a80197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:18.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a85f61700 1 --2- 172.21.15.23:0/2026855174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a80102610 0x7f8a80197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a85f61700 1 -- 172.21.15.23:0/2026855174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a80106140 msgr2=0x7f8a8019bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a85f61700 1 --2- 172.21.15.23:0/2026855174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a80106140 0x7f8a8019bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a85f61700 1 -- 172.21.15.23:0/2026855174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a700095f0 con 0x7f8a80101c50 2024-03-24T10:39:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a86762700 1 --2- 172.21.15.23:0/2026855174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a80106140 0x7f8a8019bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.823+0000 7f8a85f61700 1 --2- 172.21.15.23:0/2026855174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80101c50 0x7f8a801976d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c00e9f0 tx=0x7f8a7c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.824+0000 7f8a76ffd700 1 -- 172.21.15.23:0/2026855174 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c00c9c0 con 0x7f8a80101c50 2024-03-24T10:39:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.824+0000 7f8a76ffd700 1 -- 172.21.15.23:0/2026855174 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a7c010430 con 0x7f8a80101c50 2024-03-24T10:39:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.824+0000 7f8a76ffd700 1 -- 172.21.15.23:0/2026855174 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c018680 con 0x7f8a80101c50 2024-03-24T10:39:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.824+0000 7f8a881c5700 1 -- 172.21.15.23:0/2026855174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a8019cb70 con 0x7f8a80101c50 2024-03-24T10:39:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.824+0000 7f8a881c5700 1 -- 172.21.15.23:0/2026855174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a801a43d0 con 0x7f8a80101c50 2024-03-24T10:39:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.825+0000 7f8a881c5700 1 -- 172.21.15.23:0/2026855174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a80109730 con 0x7f8a80101c50 2024-03-24T10:39:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.827+0000 7f8a76ffd700 1 -- 172.21.15.23:0/2026855174 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f8a7c010910 con 0x7f8a80101c50 2024-03-24T10:39:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.828+0000 7f8a76ffd700 1 --2- 172.21.15.23:0/2026855174 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8a6c0a2260 0x7f8a6c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.828+0000 7f8a85760700 1 --2- 172.21.15.23:0/2026855174 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8a6c0a2260 0x7f8a6c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.828+0000 7f8a76ffd700 1 -- 172.21.15.23:0/2026855174 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f8a7c014070 con 0x7f8a80101c50 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.828+0000 7f8a85760700 1 --2- 172.21.15.23:0/2026855174 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8a6c0a2260 0x7f8a6c0a4720 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000b2e0 tx=0x7f8a70005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:16.830+0000 7f8a76ffd700 1 -- 172.21.15.23:0/2026855174 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8a7c097180 con 0x7f8a80101c50 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:17.044+0000 7f8a881c5700 1 -- 172.21.15.23:0/2026855174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8a80068ec0 con 0x7f8a80101c50 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:17.044+0000 7f8a76ffd700 1 -- 172.21.15.23:0/2026855174 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8a7c026090 con 0x7f8a80101c50 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:17.046+0000 7f8a881c5700 1 -- 172.21.15.23:0/2026855174 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8a6c0a2260 msgr2=0x7f8a6c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:17.046+0000 7f8a881c5700 1 --2- 172.21.15.23:0/2026855174 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8a6c0a2260 0x7f8a6c0a4720 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000b2e0 tx=0x7f8a70005960 comp rx=0 tx=0).stop 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:17.046+0000 7f8a881c5700 1 -- 172.21.15.23:0/2026855174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80101c50 msgr2=0x7f8a801976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:17.046+0000 7f8a881c5700 1 --2- 172.21.15.23:0/2026855174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80101c50 0x7f8a801976d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c00e9f0 tx=0x7f8a7c00ed00 comp rx=0 tx=0).stop 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:17.046+0000 7f8a881c5700 1 -- 172.21.15.23:0/2026855174 shutdown_connections 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:17.046+0000 7f8a881c5700 1 --2- 172.21.15.23:0/2026855174 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8a6c0a2260 0x7f8a6c0a4720 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:17.046+0000 7f8a881c5700 1 --2- 172.21.15.23:0/2026855174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a80106140 0x7f8a8019bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:17.046+0000 7f8a881c5700 1 --2- 172.21.15.23:0/2026855174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80101c50 0x7f8a801976d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:17.046+0000 7f8a881c5700 1 --2- 172.21.15.23:0/2026855174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a80102610 0x7f8a80197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:17.046+0000 7f8a881c5700 1 -- 172.21.15.23:0/2026855174 >> 172.21.15.23:0/2026855174 conn(0x7f8a80076440 msgr2=0x7f8a800feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:17.046+0000 7f8a881c5700 1 -- 172.21.15.23:0/2026855174 shutdown_connections 2024-03-24T10:39:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:17.046+0000 7f8a881c5700 1 -- 172.21.15.23:0/2026855174 wait complete. 2024-03-24T10:39:18.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.444+0000 7f678a8b3700 1 -- 172.21.15.23:0/3145219197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f67841111d0 con 0x7f67841072f0 2024-03-24T10:39:18.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.445+0000 7f67817fa700 1 -- 172.21.15.23:0/3145219197 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f677c00cea0 con 0x7f67841072f0 2024-03-24T10:39:18.447 INFO:teuthology.orchestra.run.smithi023.stdout:55834574924 2024-03-24T10:39:18.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.446+0000 7f678a8b3700 1 -- 172.21.15.23:0/3145219197 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f67700a21e0 msgr2=0x7f67700a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:18.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.446+0000 7f678a8b3700 1 --2- 172.21.15.23:0/3145219197 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f67700a21e0 0x7f67700a46a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6774009fd0 tx=0x7f6774009260 comp rx=0 tx=0).stop 2024-03-24T10:39:18.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.446+0000 7f678a8b3700 1 -- 172.21.15.23:0/3145219197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f67841072f0 msgr2=0x7f6784198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:18.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.446+0000 7f678a8b3700 1 --2- 172.21.15.23:0/3145219197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f67841072f0 0x7f6784198620 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f677c00ef90 tx=0x7f677c00c3a0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.447+0000 7f678a8b3700 1 -- 172.21.15.23:0/3145219197 shutdown_connections 2024-03-24T10:39:18.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.447+0000 7f678a8b3700 1 --2- 172.21.15.23:0/3145219197 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f67700a21e0 0x7f67700a46a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.447+0000 7f678a8b3700 1 --2- 172.21.15.23:0/3145219197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67841013e0 0x7f678419e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.447+0000 7f678a8b3700 1 --2- 172.21.15.23:0/3145219197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f67841072f0 0x7f6784198620 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.447+0000 7f678a8b3700 1 --2- 172.21.15.23:0/3145219197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6784101da0 0x7f678419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:18.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.447+0000 7f678a8b3700 1 -- 172.21.15.23:0/3145219197 >> 172.21.15.23:0/3145219197 conn(0x7f6784077f60 msgr2=0x7f67841006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:18.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.447+0000 7f678a8b3700 1 -- 172.21.15.23:0/3145219197 shutdown_connections 2024-03-24T10:39:18.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:18.447+0000 7f678a8b3700 1 -- 172.21.15.23:0/3145219197 wait complete. 2024-03-24T10:39:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:18 smithi023 ceph-mon[32466]: pgmap v318: 97 pgs: 3 peering, 94 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T10:39:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:18 smithi023 ceph-mon[32466]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-24T10:39:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:18 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3145219197' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T10:39:19.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:18 smithi117 ceph-mon[36368]: pgmap v318: 97 pgs: 3 peering, 94 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T10:39:19.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:18 smithi117 ceph-mon[36368]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-24T10:39:19.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:18 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3145219197' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T10:39:19.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:18 smithi116 ceph-mon[36248]: pgmap v318: 97 pgs: 3 peering, 94 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T10:39:19.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:18 smithi116 ceph-mon[36248]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-24T10:39:19.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:18 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3145219197' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T10:39:19.569 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411394 got 77309411398 for osd.2 2024-03-24T10:39:19.569 DEBUG:teuthology.parallel:result is None 2024-03-24T10:39:20.030 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574918 got 55834574924 for osd.1 2024-03-24T10:39:20.030 DEBUG:teuthology.parallel:result is None 2024-03-24T10:39:20.871 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:20 smithi023 ceph-mon[32466]: pgmap v319: 97 pgs: 3 peering, 94 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 72 B/s, 0 objects/s recovering 2024-03-24T10:39:21.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:20 smithi117 ceph-mon[36368]: pgmap v319: 97 pgs: 3 peering, 94 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 72 B/s, 0 objects/s recovering 2024-03-24T10:39:21.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:20 smithi116 ceph-mon[36248]: pgmap v319: 97 pgs: 3 peering, 94 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 72 B/s, 0 objects/s recovering 2024-03-24T10:39:21.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.870+0000 7fa81f529700 1 -- 172.21.15.23:0/1607508211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa818108290 msgr2=0x7fa818108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:21.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.870+0000 7fa81f529700 1 --2- 172.21.15.23:0/1607508211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa818108290 0x7fa818108670 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa814009990 tx=0x7fa814009ca0 comp rx=0 tx=0).stop 2024-03-24T10:39:21.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.870+0000 7fa81f529700 1 -- 172.21.15.23:0/1607508211 shutdown_connections 2024-03-24T10:39:21.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.870+0000 7fa81f529700 1 --2- 172.21.15.23:0/1607508211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818102c20 0x7fa81810f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:21.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.870+0000 7fa81f529700 1 --2- 172.21.15.23:0/1607508211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa818108290 0x7fa818108670 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:21.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.870+0000 7fa81f529700 1 --2- 172.21.15.23:0/1607508211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa818102280 0x7fa8181026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:21.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.870+0000 7fa81f529700 1 -- 172.21.15.23:0/1607508211 >> 172.21.15.23:0/1607508211 conn(0x7fa8180792d0 msgr2=0x7fa8180796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:21.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.870+0000 7fa81f529700 1 -- 172.21.15.23:0/1607508211 shutdown_connections 2024-03-24T10:39:21.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.870+0000 7fa81f529700 1 -- 172.21.15.23:0/1607508211 wait complete. 2024-03-24T10:39:21.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.870+0000 7fa81f529700 1 Processor -- start 2024-03-24T10:39:21.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.870+0000 7fa81f529700 1 -- start start 2024-03-24T10:39:21.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81f529700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818102280 0x7fa818198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:21.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81f529700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa818102c20 0x7fa818198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:21.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81f529700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa818108290 0x7fa81819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:21.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81f529700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fa81819d5b0 con 0x7fa818102c20 2024-03-24T10:39:21.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81f529700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa81819d720 con 0x7fa818102280 2024-03-24T10:39:21.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81f529700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa81819d8e0 con 0x7fa818108290 2024-03-24T10:39:21.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81cac4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa818102c20 0x7fa818198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:21.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81cac4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa818102c20 0x7fa818198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50102/0 (socket says 172.21.15.23:50102) 2024-03-24T10:39:21.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81cac4700 1 -- 172.21.15.23:0/2464802791 learned_addr learned my addr 172.21.15.23:0/2464802791 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:21.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81dac6700 1 --2- 172.21.15.23:0/2464802791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa818108290 0x7fa81819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:21.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81d2c5700 1 --2- 172.21.15.23:0/2464802791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818102280 0x7fa818198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:21.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81cac4700 1 -- 172.21.15.23:0/2464802791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa818108290 msgr2=0x7fa81819cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:21.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81cac4700 1 --2- 172.21.15.23:0/2464802791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa818108290 0x7fa81819cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:21.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81cac4700 1 -- 172.21.15.23:0/2464802791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818102280 msgr2=0x7fa818198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:21.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81cac4700 1 --2- 172.21.15.23:0/2464802791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818102280 0x7fa818198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:21.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81cac4700 1 -- 172.21.15.23:0/2464802791 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8140095f0 con 0x7fa818102c20 2024-03-24T10:39:21.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81cac4700 1 --2- 172.21.15.23:0/2464802791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa818102c20 0x7fa818198c60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa80800e9a0 tx=0x7fa80800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:21.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa80e7fc700 1 -- 172.21.15.23:0/2464802791 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808009790 con 0x7fa818102c20 2024-03-24T10:39:21.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81f529700 1 -- 172.21.15.23:0/2464802791 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81819dbc0 con 0x7fa818102c20 2024-03-24T10:39:21.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa80e7fc700 1 -- 172.21.15.23:0/2464802791 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa808010430 con 0x7fa818102c20 2024-03-24T10:39:21.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa80e7fc700 1 -- 172.21.15.23:0/2464802791 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808018680 con 0x7fa818102c20 2024-03-24T10:39:21.207 INFO:teuthology.orchestra.run.smithi023.stdout:163208757304 2024-03-24T10:39:21.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.871+0000 7fa81f529700 1 -- 172.21.15.23:0/2464802791 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8181a5420 con 0x7fa818102c20 2024-03-24T10:39:21.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.872+0000 7fa81f529700 1 -- 172.21.15.23:0/2464802791 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa81810a7b0 con 0x7fa818102c20 2024-03-24T10:39:21.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.873+0000 7fa80e7fc700 1 -- 172.21.15.23:0/2464802791 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fa808010910 con 0x7fa818102c20 2024-03-24T10:39:21.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.874+0000 7fa80e7fc700 1 --2- 172.21.15.23:0/2464802791 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa8040a21e0 0x7fa8040a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:21.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.874+0000 7fa80e7fc700 1 -- 172.21.15.23:0/2464802791 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa808014070 con 0x7fa818102c20 2024-03-24T10:39:21.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.874+0000 7fa81d2c5700 1 --2- 172.21.15.23:0/2464802791 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa8040a21e0 0x7fa8040a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.874+0000 7fa81d2c5700 1 --2- 172.21.15.23:0/2464802791 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa8040a21e0 0x7fa8040a46a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa81400b2e0 tx=0x7fa814005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:20.877+0000 7fa80e7fc700 1 -- 172.21.15.23:0/2464802791 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa8080973a0 con 0x7fa818102c20 2024-03-24T10:39:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:21.083+0000 7fa81f529700 1 -- 172.21.15.23:0/2464802791 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa818068ec0 con 0x7fa818102c20 2024-03-24T10:39:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:21.084+0000 7fa80e7fc700 1 -- 172.21.15.23:0/2464802791 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa808026090 con 0x7fa818102c20 2024-03-24T10:39:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:21.086+0000 7fa81f529700 1 -- 172.21.15.23:0/2464802791 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa8040a21e0 msgr2=0x7fa8040a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:21.086+0000 7fa81f529700 1 --2- 172.21.15.23:0/2464802791 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa8040a21e0 0x7fa8040a46a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa81400b2e0 tx=0x7fa814005960 comp rx=0 tx=0).stop 2024-03-24T10:39:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:21.086+0000 7fa81f529700 1 -- 172.21.15.23:0/2464802791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa818102c20 msgr2=0x7fa818198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:21.086+0000 7fa81f529700 1 --2- 172.21.15.23:0/2464802791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa818102c20 0x7fa818198c60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa80800e9a0 tx=0x7fa80800ecb0 comp rx=0 tx=0).stop 2024-03-24T10:39:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:21.086+0000 7fa81f529700 1 -- 172.21.15.23:0/2464802791 shutdown_connections 2024-03-24T10:39:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:21.086+0000 7fa81f529700 1 --2- 172.21.15.23:0/2464802791 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa8040a21e0 0x7fa8040a46a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:21.086+0000 7fa81f529700 1 --2- 172.21.15.23:0/2464802791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa818108290 0x7fa81819cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:21.086+0000 7fa81f529700 1 --2- 172.21.15.23:0/2464802791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa818102280 0x7fa818198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:21.086+0000 7fa81f529700 1 --2- 172.21.15.23:0/2464802791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa818102c20 0x7fa818198c60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:21.086+0000 7fa81f529700 1 -- 172.21.15.23:0/2464802791 >> 172.21.15.23:0/2464802791 conn(0x7fa8180792d0 msgr2=0x7fa818100f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:21.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:21.086+0000 7fa81f529700 1 -- 172.21.15.23:0/2464802791 shutdown_connections 2024-03-24T10:39:21.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:21.086+0000 7fa81f529700 1 -- 172.21.15.23:0/2464802791 wait complete. 2024-03-24T10:39:21.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:21 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2464802791' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T10:39:21.849 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757299 got 163208757304 for osd.5 2024-03-24T10:39:21.850 DEBUG:teuthology.parallel:result is None 2024-03-24T10:39:21.850 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T10:39:21.850 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T10:39:22.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:21 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2464802791' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T10:39:22.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:21 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2464802791' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T10:39:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:22.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:22 smithi023 ceph-mon[32466]: pgmap v320: 97 pgs: 2 peering, 95 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 101 B/s, 1 objects/s recovering 2024-03-24T10:39:22.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:22.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:22 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:23.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:22 smithi117 ceph-mon[36368]: pgmap v320: 97 pgs: 2 peering, 95 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 101 B/s, 1 objects/s recovering 2024-03-24T10:39:23.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:23.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:22 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:23.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:22 smithi116 ceph-mon[36248]: pgmap v320: 97 pgs: 2 peering, 95 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 101 B/s, 1 objects/s recovering 2024-03-24T10:39:23.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:23.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:22 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:23.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.720+0000 7f8264f29700 1 -- 172.21.15.23:0/2415905774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82601063e0 msgr2=0x7f826010dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:23.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.720+0000 7f8264f29700 1 --2- 172.21.15.23:0/2415905774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82601063e0 0x7f826010dfb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8250009990 tx=0x7f8250009ca0 comp rx=0 tx=0).stop 2024-03-24T10:39:23.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.720+0000 7f8264f29700 1 -- 172.21.15.23:0/2415905774 shutdown_connections 2024-03-24T10:39:23.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.720+0000 7f8264f29700 1 --2- 172.21.15.23:0/2415905774 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8260101390 0x7f8260101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:23.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.720+0000 7f8264f29700 1 --2- 172.21.15.23:0/2415905774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82601063e0 0x7f826010dfb0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:23.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.720+0000 7f8264f29700 1 --2- 172.21.15.23:0/2415905774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8260101d40 0x7f8260105d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:23.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.720+0000 7f8264f29700 1 -- 172.21.15.23:0/2415905774 >> 172.21.15.23:0/2415905774 conn(0x7f8260077150 msgr2=0x7f8260077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:23.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.720+0000 7f8264f29700 1 -- 172.21.15.23:0/2415905774 shutdown_connections 2024-03-24T10:39:23.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.720+0000 7f8264f29700 1 -- 172.21.15.23:0/2415905774 wait complete. 2024-03-24T10:39:23.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.720+0000 7f8264f29700 1 Processor -- start 2024-03-24T10:39:23.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.720+0000 7f8264f29700 1 -- start start 2024-03-24T10:39:23.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.720+0000 7f8264f29700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8260101390 0x7f8260102da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:23.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.720+0000 7f8264f29700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8260101d40 0x7f82601032e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:23.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f8264f29700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82601063e0 0x7f8260103820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:23.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f8264f29700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f82601a6b00 con 0x7f8260101d40 2024-03-24T10:39:23.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f8264f29700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f82601a6c70 con 0x7f82601063e0 2024-03-24T10:39:23.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f8264f29700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f82601a6e30 con 0x7f8260101390 2024-03-24T10:39:23.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f825e59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8260101390 0x7f8260102da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:23.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f825e59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8260101390 0x7f8260102da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:36630/0 (socket says 172.21.15.23:36630) 2024-03-24T10:39:23.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f825ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82601063e0 0x7f8260103820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:23.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f825e59c700 1 -- 172.21.15.23:0/954454662 learned_addr learned my addr 172.21.15.23:0/954454662 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:23.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f825dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8260101d40 0x7f82601032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:23.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f825e59c700 1 -- 172.21.15.23:0/954454662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82601063e0 msgr2=0x7f8260103820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:23.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f825e59c700 1 --2- 172.21.15.23:0/954454662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82601063e0 0x7f8260103820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:23.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f825e59c700 1 -- 172.21.15.23:0/954454662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8260101d40 msgr2=0x7f82601032e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:23.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f825e59c700 1 --2- 172.21.15.23:0/954454662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8260101d40 0x7f82601032e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:23.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f825e59c700 1 -- 172.21.15.23:0/954454662 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82500095f0 con 0x7f8260101390 2024-03-24T10:39:23.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.722+0000 7f825e59c700 1 --2- 172.21.15.23:0/954454662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8260101390 0x7f8260102da0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f825800ef10 tx=0x7f825800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:23.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.725+0000 7f82577fe700 1 -- 172.21.15.23:0/954454662 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825800c9c0 con 0x7f8260101390 2024-03-24T10:39:23.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.725+0000 7f82577fe700 1 -- 172.21.15.23:0/954454662 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f825800f040 con 0x7f8260101390 2024-03-24T10:39:23.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.725+0000 7f8264f29700 1 -- 172.21.15.23:0/954454662 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82601a70b0 con 0x7f8260101390 2024-03-24T10:39:23.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.725+0000 7f82577fe700 1 -- 172.21.15.23:0/954454662 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258014480 con 0x7f8260101390 2024-03-24T10:39:23.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.725+0000 7f8264f29700 1 -- 172.21.15.23:0/954454662 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82601a75b0 con 0x7f8260101390 2024-03-24T10:39:23.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.727+0000 7f8264f29700 1 -- 172.21.15.23:0/954454662 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f826010a710 con 0x7f8260101390 2024-03-24T10:39:23.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.727+0000 7f82577fe700 1 -- 172.21.15.23:0/954454662 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f8258007720 con 0x7f8260101390 2024-03-24T10:39:23.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.727+0000 7f82577fe700 1 --2- 172.21.15.23:0/954454662 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f824c0a21e0 0x7f824c0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:23.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.727+0000 7f825dd9b700 1 --2- 172.21.15.23:0/954454662 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f824c0a21e0 0x7f824c0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:23.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.728+0000 7f825dd9b700 1 --2- 172.21.15.23:0/954454662 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f824c0a21e0 0x7f824c0a46a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8260107910 tx=0x7f8248009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:23.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.729+0000 7f82577fe700 1 -- 172.21.15.23:0/954454662 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f825809b3c0 con 0x7f8260101390 2024-03-24T10:39:23.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.737+0000 7f82577fe700 1 -- 172.21.15.23:0/954454662 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8258096f60 con 0x7f8260101390 2024-03-24T10:39:23.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:23 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:23.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:23 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:39:23.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:23 smithi023 ceph-mon[32466]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:23.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.935+0000 7f8264f29700 1 -- 172.21.15.23:0/954454662 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8260111140 con 0x7f824c0a21e0 2024-03-24T10:39:23.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.943+0000 7f82577fe700 1 -- 172.21.15.23:0/954454662 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258729 (secure 0 0 0) 0x7f8260111140 con 0x7f824c0a21e0 2024-03-24T10:39:23.947 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:23.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.946+0000 7f8264f29700 1 -- 172.21.15.23:0/954454662 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f824c0a21e0 msgr2=0x7f824c0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:23.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.946+0000 7f8264f29700 1 --2- 172.21.15.23:0/954454662 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f824c0a21e0 0x7f824c0a46a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8260107910 tx=0x7f8248009260 comp rx=0 tx=0).stop 2024-03-24T10:39:23.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.946+0000 7f8264f29700 1 -- 172.21.15.23:0/954454662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8260101390 msgr2=0x7f8260102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:23.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.946+0000 7f8264f29700 1 --2- 172.21.15.23:0/954454662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8260101390 0x7f8260102da0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f825800ef10 tx=0x7f825800c3a0 comp rx=0 tx=0).stop 2024-03-24T10:39:23.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.947+0000 7f8264f29700 1 -- 172.21.15.23:0/954454662 shutdown_connections 2024-03-24T10:39:23.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.947+0000 7f8264f29700 1 --2- 172.21.15.23:0/954454662 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f824c0a21e0 0x7f824c0a46a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:23.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.947+0000 7f8264f29700 1 --2- 172.21.15.23:0/954454662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8260101390 0x7f8260102da0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:23.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.947+0000 7f8264f29700 1 --2- 172.21.15.23:0/954454662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82601063e0 0x7f8260103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:23.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.947+0000 7f8264f29700 1 --2- 172.21.15.23:0/954454662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8260101d40 0x7f82601032e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:23.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.947+0000 7f8264f29700 1 -- 172.21.15.23:0/954454662 >> 172.21.15.23:0/954454662 conn(0x7f8260077150 msgr2=0x7f826006aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:23.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.947+0000 7f8264f29700 1 -- 172.21.15.23:0/954454662 shutdown_connections 2024-03-24T10:39:23.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:23.947+0000 7f8264f29700 1 -- 172.21.15.23:0/954454662 wait complete. 2024-03-24T10:39:23.950 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-24T10:39:24.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:23 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:24.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:23 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:39:24.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:23 smithi117 ceph-mon[36368]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:24.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:23 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:39:24.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:23 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:39:24.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:23 smithi116 ceph-mon[36248]: from='mgr.14152 172.21.15.23:0/4134698612' entity='mgr.x' 2024-03-24T10:39:24.541 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-03-24T10:39:22.810915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654068,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":556,"num_read_kb":398,"num_write":338,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1181547,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":295,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":366636,"kb_used_data":13260,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575188,"statfs":{"total":1151940427776,"available":1151564992512,"internally_reserved":0,"allocated":13578240,"data_stored":7431960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001931"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698168+0000","last_change":"2024-03-24T10:38:10.882385+0000","last_active":"2024-03-24T10:39:16.698168+0000","last_peered":"2024-03-24T10:39:16.698168+0000","last_clean":"2024-03-24T10:39:16.698168+0000","last_became_active":"2024-03-24T10:38:10.882062+0000","last_became_peered":"2024-03-24T10:38:10.882062+0000","last_unstale":"2024-03-24T10:39:16.698168+0000","last_undegraded":"2024-03-24T10:39:16.698168+0000","last_fullsized":"2024-03-24T10:39:16.698168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:25.151062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695985+0000","last_change":"2024-03-24T10:38:10.878001+0000","last_active":"2024-03-24T10:39:16.695985+0000","last_peered":"2024-03-24T10:39:16.695985+0000","last_clean":"2024-03-24T10:39:16.695985+0000","last_became_active":"2024-03-24T10:38:10.877661+0000","last_became_peered":"2024-03-24T10:38:10.877661+0000","last_unstale":"2024-03-24T10:39:16.695985+0000","last_undegraded":"2024-03-24T10:39:16.695985+0000","last_fullsized":"2024-03-24T10:39:16.695985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:40:19.062969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696467+0000","last_change":"2024-03-24T10:38:10.891809+0000","last_active":"2024-03-24T10:39:16.696467+0000","last_peered":"2024-03-24T10:39:16.696467+0000","last_clean":"2024-03-24T10:39:16.696467+0000","last_became_active":"2024-03-24T10:38:10.890354+0000","last_became_peered":"2024-03-24T10:38:10.890354+0000","last_unstale":"2024-03-24T10:39:16.696467+0000","last_undegraded":"2024-03-24T10:39:16.696467+0000","last_fullsized":"2024-03-24T10:39:16.696467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:59:26.037914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.905236+0000","last_change":"2024-03-24T10:38:10.885097+0000","last_active":"2024-03-24T10:39:16.905236+0000","last_peered":"2024-03-24T10:39:16.905236+0000","last_clean":"2024-03-24T10:39:16.905236+0000","last_became_active":"2024-03-24T10:38:10.884842+0000","last_became_peered":"2024-03-24T10:38:10.884842+0000","last_unstale":"2024-03-24T10:39:16.905236+0000","last_undegraded":"2024-03-24T10:39:16.905236+0000","last_fullsized":"2024-03-24T10:39:16.905236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:39:44.085699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695563+0000","last_change":"2024-03-24T10:38:10.877653+0000","last_active":"2024-03-24T10:39:16.695563+0000","last_peered":"2024-03-24T10:39:16.695563+0000","last_clean":"2024-03-24T10:39:16.695563+0000","last_became_active":"2024-03-24T10:38:10.877249+0000","last_became_peered":"2024-03-24T10:38:10.877249+0000","last_unstale":"2024-03-24T10:39:16.695563+0000","last_undegraded":"2024-03-24T10:39:16.695563+0000","last_fullsized":"2024-03-24T10:39:16.695563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:17.504660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.725073+0000","last_change":"2024-03-24T10:38:10.876584+0000","last_active":"2024-03-24T10:39:16.725073+0000","last_peered":"2024-03-24T10:39:16.725073+0000","last_clean":"2024-03-24T10:39:16.725073+0000","last_became_active":"2024-03-24T10:38:10.876173+0000","last_became_peered":"2024-03-24T10:38:10.876173+0000","last_unstale":"2024-03-24T10:39:16.725073+0000","last_undegraded":"2024-03-24T10:39:16.725073+0000","last_fullsized":"2024-03-24T10:39:16.725073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:33.899155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695666+0000","last_change":"2024-03-24T10:38:10.874206+0000","last_active":"2024-03-24T10:39:16.695666+0000","last_peered":"2024-03-24T10:39:16.695666+0000","last_clean":"2024-03-24T10:39:16.695666+0000","last_became_active":"2024-03-24T10:38:10.873779+0000","last_became_peered":"2024-03-24T10:38:10.873779+0000","last_unstale":"2024-03-24T10:39:16.695666+0000","last_undegraded":"2024-03-24T10:39:16.695666+0000","last_fullsized":"2024-03-24T10:39:16.695666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:08:19.928998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699354+0000","last_change":"2024-03-24T10:38:10.884219+0000","last_active":"2024-03-24T10:39:16.699354+0000","last_peered":"2024-03-24T10:39:16.699354+0000","last_clean":"2024-03-24T10:39:16.699354+0000","last_became_active":"2024-03-24T10:38:10.883863+0000","last_became_peered":"2024-03-24T10:38:10.883863+0000","last_unstale":"2024-03-24T10:39:16.699354+0000","last_undegraded":"2024-03-24T10:39:16.699354+0000","last_fullsized":"2024-03-24T10:39:16.699354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:26:34.932083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698407+0000","last_change":"2024-03-24T10:38:10.881297+0000","last_active":"2024-03-24T10:39:16.698407+0000","last_peered":"2024-03-24T10:39:16.698407+0000","last_clean":"2024-03-24T10:39:16.698407+0000","last_became_active":"2024-03-24T10:38:10.879871+0000","last_became_peered":"2024-03-24T10:38:10.879871+0000","last_unstale":"2024-03-24T10:39:16.698407+0000","last_undegraded":"2024-03-24T10:39:16.698407+0000","last_fullsized":"2024-03-24T10:39:16.698407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:05.132788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.725363+0000","last_change":"2024-03-24T10:38:10.876648+0000","last_active":"2024-03-24T10:39:16.725363+0000","last_peered":"2024-03-24T10:39:16.725363+0000","last_clean":"2024-03-24T10:39:16.725363+0000","last_became_active":"2024-03-24T10:38:10.876113+0000","last_became_peered":"2024-03-24T10:38:10.876113+0000","last_unstale":"2024-03-24T10:39:16.725363+0000","last_undegraded":"2024-03-24T10:39:16.725363+0000","last_fullsized":"2024-03-24T10:39:16.725363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:29:18.282797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695901+0000","last_change":"2024-03-24T10:38:10.875836+0000","last_active":"2024-03-24T10:39:16.695901+0000","last_peered":"2024-03-24T10:39:16.695901+0000","last_clean":"2024-03-24T10:39:16.695901+0000","last_became_active":"2024-03-24T10:38:10.875522+0000","last_became_peered":"2024-03-24T10:38:10.875522+0000","last_unstale":"2024-03-24T10:39:16.695901+0000","last_undegraded":"2024-03-24T10:39:16.695901+0000","last_fullsized":"2024-03-24T10:39:16.695901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:41.282123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.905099+0000","last_change":"2024-03-24T10:38:10.884567+0000","last_active":"2024-03-24T10:39:16.905099+0000","last_peered":"2024-03-24T10:39:16.905099+0000","last_clean":"2024-03-24T10:39:16.905099+0000","last_became_active":"2024-03-24T10:38:10.884195+0000","last_became_peered":"2024-03-24T10:38:10.884195+0000","last_unstale":"2024-03-24T10:39:16.905099+0000","last_undegraded":"2024-03-24T10:39:16.905099+0000","last_fullsized":"2024-03-24T10:39:16.905099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:29.286631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.726561+0000","last_change":"2024-03-24T10:39:16.726561+0000","last_active":"2024-03-24T10:39:16.726561+0000","last_peered":"2024-03-24T10:39:16.726561+0000","last_clean":"2024-03-24T10:39:16.726561+0000","last_became_active":"2024-03-24T10:39:16.726088+0000","last_became_peered":"2024-03-24T10:39:16.726088+0000","last_unstale":"2024-03-24T10:39:16.726561+0000","last_undegraded":"2024-03-24T10:39:16.726561+0000","last_fullsized":"2024-03-24T10:39:16.726561+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:08:46.638610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695788+0000","last_change":"2024-03-24T10:38:10.879942+0000","last_active":"2024-03-24T10:39:16.695788+0000","last_peered":"2024-03-24T10:39:16.695788+0000","last_clean":"2024-03-24T10:39:16.695788+0000","last_became_active":"2024-03-24T10:38:10.879396+0000","last_became_peered":"2024-03-24T10:38:10.879396+0000","last_unstale":"2024-03-24T10:39:16.695788+0000","last_undegraded":"2024-03-24T10:39:16.695788+0000","last_fullsized":"2024-03-24T10:39:16.695788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:41.477988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695815+0000","last_change":"2024-03-24T10:38:10.886667+0000","last_active":"2024-03-24T10:39:16.695815+0000","last_peered":"2024-03-24T10:39:16.695815+0000","last_clean":"2024-03-24T10:39:16.695815+0000","last_became_active":"2024-03-24T10:38:10.886293+0000","last_became_peered":"2024-03-24T10:38:10.886293+0000","last_unstale":"2024-03-24T10:39:16.695815+0000","last_undegraded":"2024-03-24T10:39:16.695815+0000","last_fullsized":"2024-03-24T10:39:16.695815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:43:43.414573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697524+0000","last_change":"2024-03-24T10:38:10.883671+0000","last_active":"2024-03-24T10:39:16.697524+0000","last_peered":"2024-03-24T10:39:16.697524+0000","last_clean":"2024-03-24T10:39:16.697524+0000","last_became_active":"2024-03-24T10:38:10.883013+0000","last_became_peered":"2024-03-24T10:38:10.883013+0000","last_unstale":"2024-03-24T10:39:16.697524+0000","last_undegraded":"2024-03-24T10:39:16.697524+0000","last_fullsized":"2024-03-24T10:39:16.697524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:33.936543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698261+0000","last_change":"2024-03-24T10:38:10.887286+0000","last_active":"2024-03-24T10:39:16.698261+0000","last_peered":"2024-03-24T10:39:16.698261+0000","last_clean":"2024-03-24T10:39:16.698261+0000","last_became_active":"2024-03-24T10:38:10.886918+0000","last_became_peered":"2024-03-24T10:38:10.886918+0000","last_unstale":"2024-03-24T10:39:16.698261+0000","last_undegraded":"2024-03-24T10:39:16.698261+0000","last_fullsized":"2024-03-24T10:39:16.698261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:40.266589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698568+0000","last_change":"2024-03-24T10:38:10.874476+0000","last_active":"2024-03-24T10:39:16.698568+0000","last_peered":"2024-03-24T10:39:16.698568+0000","last_clean":"2024-03-24T10:39:16.698568+0000","last_became_active":"2024-03-24T10:38:10.874123+0000","last_became_peered":"2024-03-24T10:38:10.874123+0000","last_unstale":"2024-03-24T10:39:16.698568+0000","last_undegraded":"2024-03-24T10:39:16.698568+0000","last_fullsized":"2024-03-24T10:39:16.698568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:29.759320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699586+0000","last_change":"2024-03-24T10:38:10.878758+0000","last_active":"2024-03-24T10:39:16.699586+0000","last_peered":"2024-03-24T10:39:16.699586+0000","last_clean":"2024-03-24T10:39:16.699586+0000","last_became_active":"2024-03-24T10:38:10.878424+0000","last_became_peered":"2024-03-24T10:38:10.878424+0000","last_unstale":"2024-03-24T10:39:16.699586+0000","last_undegraded":"2024-03-24T10:39:16.699586+0000","last_fullsized":"2024-03-24T10:39:16.699586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:15:19.273108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.709304+0000","last_change":"2024-03-24T10:39:16.709304+0000","last_active":"2024-03-24T10:39:16.709304+0000","last_peered":"2024-03-24T10:39:16.709304+0000","last_clean":"2024-03-24T10:39:16.709304+0000","last_became_active":"2024-03-24T10:39:16.700325+0000","last_became_peered":"2024-03-24T10:39:16.700325+0000","last_unstale":"2024-03-24T10:39:16.709304+0000","last_undegraded":"2024-03-24T10:39:16.709304+0000","last_fullsized":"2024-03-24T10:39:16.709304+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:57.633288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699762+0000","last_change":"2024-03-24T10:38:10.885198+0000","last_active":"2024-03-24T10:39:16.699762+0000","last_peered":"2024-03-24T10:39:16.699762+0000","last_clean":"2024-03-24T10:39:16.699762+0000","last_became_active":"2024-03-24T10:38:10.884123+0000","last_became_peered":"2024-03-24T10:38:10.884123+0000","last_unstale":"2024-03-24T10:39:16.699762+0000","last_undegraded":"2024-03-24T10:39:16.699762+0000","last_fullsized":"2024-03-24T10:39:16.699762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:37:12.510671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699892+0000","last_change":"2024-03-24T10:38:09.866223+0000","last_active":"2024-03-24T10:39:16.699892+0000","last_peered":"2024-03-24T10:39:16.699892+0000","last_clean":"2024-03-24T10:39:16.699892+0000","last_became_active":"2024-03-24T10:38:09.865711+0000","last_became_peered":"2024-03-24T10:38:09.865711+0000","last_unstale":"2024-03-24T10:39:16.699892+0000","last_undegraded":"2024-03-24T10:39:16.699892+0000","last_fullsized":"2024-03-24T10:39:16.699892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:26:08.380749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696084+0000","last_change":"2024-03-24T10:38:10.890557+0000","last_active":"2024-03-24T10:39:16.696084+0000","last_peered":"2024-03-24T10:39:16.696084+0000","last_clean":"2024-03-24T10:39:16.696084+0000","last_became_active":"2024-03-24T10:38:10.889883+0000","last_became_peered":"2024-03-24T10:38:10.889883+0000","last_unstale":"2024-03-24T10:39:16.696084+0000","last_undegraded":"2024-03-24T10:39:16.696084+0000","last_fullsized":"2024-03-24T10:39:16.696084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:34.731585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695880+0000","last_change":"2024-03-24T10:38:09.870253+0000","last_active":"2024-03-24T10:39:16.695880+0000","last_peered":"2024-03-24T10:39:16.695880+0000","last_clean":"2024-03-24T10:39:16.695880+0000","last_became_active":"2024-03-24T10:38:09.869932+0000","last_became_peered":"2024-03-24T10:38:09.869932+0000","last_unstale":"2024-03-24T10:39:16.695880+0000","last_undegraded":"2024-03-24T10:39:16.695880+0000","last_fullsized":"2024-03-24T10:39:16.695880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:38:43.626390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697788+0000","last_change":"2024-03-24T10:38:10.881705+0000","last_active":"2024-03-24T10:39:16.697788+0000","last_peered":"2024-03-24T10:39:16.697788+0000","last_clean":"2024-03-24T10:39:16.697788+0000","last_became_active":"2024-03-24T10:38:10.881323+0000","last_became_peered":"2024-03-24T10:38:10.881323+0000","last_unstale":"2024-03-24T10:39:16.697788+0000","last_undegraded":"2024-03-24T10:39:16.697788+0000","last_fullsized":"2024-03-24T10:39:16.697788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:25.861184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696090+0000","last_change":"2024-03-24T10:38:09.870481+0000","last_active":"2024-03-24T10:39:16.696090+0000","last_peered":"2024-03-24T10:39:16.696090+0000","last_clean":"2024-03-24T10:39:16.696090+0000","last_became_active":"2024-03-24T10:38:09.870117+0000","last_became_peered":"2024-03-24T10:38:09.870117+0000","last_unstale":"2024-03-24T10:39:16.696090+0000","last_undegraded":"2024-03-24T10:39:16.696090+0000","last_fullsized":"2024-03-24T10:39:16.696090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:37:51.871068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.907072+0000","last_change":"2024-03-24T10:39:16.906560+0000","last_active":"2024-03-24T10:39:16.907072+0000","last_peered":"2024-03-24T10:39:16.907072+0000","last_clean":"2024-03-24T10:39:16.907072+0000","last_became_active":"2024-03-24T10:39:16.906009+0000","last_became_peered":"2024-03-24T10:39:16.906009+0000","last_unstale":"2024-03-24T10:39:16.907072+0000","last_undegraded":"2024-03-24T10:39:16.907072+0000","last_fullsized":"2024-03-24T10:39:16.907072+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:45:56.513953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":92,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:19.984716+0000","last_change":"2024-03-24T10:38:09.864160+0000","last_active":"2024-03-24T10:39:19.984716+0000","last_peered":"2024-03-24T10:39:19.984716+0000","last_clean":"2024-03-24T10:39:19.984716+0000","last_became_active":"2024-03-24T10:38:09.862066+0000","last_became_peered":"2024-03-24T10:38:09.862066+0000","last_unstale":"2024-03-24T10:39:19.984716+0000","last_undegraded":"2024-03-24T10:39:19.984716+0000","last_fullsized":"2024-03-24T10:39:19.984716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:59.003931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695578+0000","last_change":"2024-03-24T10:38:10.874433+0000","last_active":"2024-03-24T10:39:16.695578+0000","last_peered":"2024-03-24T10:39:16.695578+0000","last_clean":"2024-03-24T10:39:16.695578+0000","last_became_active":"2024-03-24T10:38:10.874090+0000","last_became_peered":"2024-03-24T10:38:10.874090+0000","last_unstale":"2024-03-24T10:39:16.695578+0000","last_undegraded":"2024-03-24T10:39:16.695578+0000","last_fullsized":"2024-03-24T10:39:16.695578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:25.979419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.725049+0000","last_change":"2024-03-24T10:38:09.863028+0000","last_active":"2024-03-24T10:39:16.725049+0000","last_peered":"2024-03-24T10:39:16.725049+0000","last_clean":"2024-03-24T10:39:16.725049+0000","last_became_active":"2024-03-24T10:38:09.862682+0000","last_became_peered":"2024-03-24T10:38:09.862682+0000","last_unstale":"2024-03-24T10:39:16.725049+0000","last_undegraded":"2024-03-24T10:39:16.725049+0000","last_fullsized":"2024-03-24T10:39:16.725049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:14.962115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699089+0000","last_change":"2024-03-24T10:38:10.885950+0000","last_active":"2024-03-24T10:39:16.699089+0000","last_peered":"2024-03-24T10:39:16.699089+0000","last_clean":"2024-03-24T10:39:16.699089+0000","last_became_active":"2024-03-24T10:38:10.885685+0000","last_became_peered":"2024-03-24T10:38:10.885685+0000","last_unstale":"2024-03-24T10:39:16.699089+0000","last_undegraded":"2024-03-24T10:39:16.699089+0000","last_fullsized":"2024-03-24T10:39:16.699089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:42:09.912743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.700938+0000","last_change":"2024-03-24T10:38:09.860376+0000","last_active":"2024-03-24T10:39:16.700938+0000","last_peered":"2024-03-24T10:39:16.700938+0000","last_clean":"2024-03-24T10:39:16.700938+0000","last_became_active":"2024-03-24T10:38:09.860062+0000","last_became_peered":"2024-03-24T10:38:09.860062+0000","last_unstale":"2024-03-24T10:39:16.700938+0000","last_undegraded":"2024-03-24T10:39:16.700938+0000","last_fullsized":"2024-03-24T10:39:16.700938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:33.387953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699208+0000","last_change":"2024-03-24T10:38:10.883843+0000","last_active":"2024-03-24T10:39:16.699208+0000","last_peered":"2024-03-24T10:39:16.699208+0000","last_clean":"2024-03-24T10:39:16.699208+0000","last_became_active":"2024-03-24T10:38:10.883170+0000","last_became_peered":"2024-03-24T10:38:10.883170+0000","last_unstale":"2024-03-24T10:39:16.699208+0000","last_undegraded":"2024-03-24T10:39:16.699208+0000","last_fullsized":"2024-03-24T10:39:16.699208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:28:16.798546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698585+0000","last_change":"2024-03-24T10:38:09.862299+0000","last_active":"2024-03-24T10:39:16.698585+0000","last_peered":"2024-03-24T10:39:16.698585+0000","last_clean":"2024-03-24T10:39:16.698585+0000","last_became_active":"2024-03-24T10:38:09.861895+0000","last_became_peered":"2024-03-24T10:38:09.861895+0000","last_unstale":"2024-03-24T10:39:16.698585+0000","last_undegraded":"2024-03-24T10:39:16.698585+0000","last_fullsized":"2024-03-24T10:39:16.698585+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:13:19.869511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698225+0000","last_change":"2024-03-24T10:38:10.880507+0000","last_active":"2024-03-24T10:39:16.698225+0000","last_peered":"2024-03-24T10:39:16.698225+0000","last_clean":"2024-03-24T10:39:16.698225+0000","last_became_active":"2024-03-24T10:38:10.879790+0000","last_became_peered":"2024-03-24T10:38:10.879790+0000","last_unstale":"2024-03-24T10:39:16.698225+0000","last_undegraded":"2024-03-24T10:39:16.698225+0000","last_fullsized":"2024-03-24T10:39:16.698225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:11:02.650420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":88,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:19.984086+0000","last_change":"2024-03-24T10:38:09.860186+0000","last_active":"2024-03-24T10:39:19.984086+0000","last_peered":"2024-03-24T10:39:19.984086+0000","last_clean":"2024-03-24T10:39:19.984086+0000","last_became_active":"2024-03-24T10:38:09.859834+0000","last_became_peered":"2024-03-24T10:38:09.859834+0000","last_unstale":"2024-03-24T10:39:19.984086+0000","last_undegraded":"2024-03-24T10:39:19.984086+0000","last_fullsized":"2024-03-24T10:39:19.984086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:33:48.029053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698120+0000","last_change":"2024-03-24T10:38:10.875634+0000","last_active":"2024-03-24T10:39:16.698120+0000","last_peered":"2024-03-24T10:39:16.698120+0000","last_clean":"2024-03-24T10:39:16.698120+0000","last_became_active":"2024-03-24T10:38:10.875101+0000","last_became_peered":"2024-03-24T10:38:10.875101+0000","last_unstale":"2024-03-24T10:39:16.698120+0000","last_undegraded":"2024-03-24T10:39:16.698120+0000","last_fullsized":"2024-03-24T10:39:16.698120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:32:58.165087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695826+0000","last_change":"2024-03-24T10:38:09.860412+0000","last_active":"2024-03-24T10:39:16.695826+0000","last_peered":"2024-03-24T10:39:16.695826+0000","last_clean":"2024-03-24T10:39:16.695826+0000","last_became_active":"2024-03-24T10:38:09.860084+0000","last_became_peered":"2024-03-24T10:38:09.860084+0000","last_unstale":"2024-03-24T10:39:16.695826+0000","last_undegraded":"2024-03-24T10:39:16.695826+0000","last_fullsized":"2024-03-24T10:39:16.695826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:44.585482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.703566+0000","last_change":"2024-03-24T10:39:16.703566+0000","last_active":"2024-03-24T10:39:16.703566+0000","last_peered":"2024-03-24T10:39:16.703566+0000","last_clean":"2024-03-24T10:39:16.703566+0000","last_became_active":"2024-03-24T10:39:16.703213+0000","last_became_peered":"2024-03-24T10:39:16.703213+0000","last_unstale":"2024-03-24T10:39:16.703566+0000","last_undegraded":"2024-03-24T10:39:16.703566+0000","last_fullsized":"2024-03-24T10:39:16.703566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:10.888996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698568+0000","last_change":"2024-03-24T10:38:09.865191+0000","last_active":"2024-03-24T10:39:16.698568+0000","last_peered":"2024-03-24T10:39:16.698568+0000","last_clean":"2024-03-24T10:39:16.698568+0000","last_became_active":"2024-03-24T10:38:09.864470+0000","last_became_peered":"2024-03-24T10:38:09.864470+0000","last_unstale":"2024-03-24T10:39:16.698568+0000","last_undegraded":"2024-03-24T10:39:16.698568+0000","last_fullsized":"2024-03-24T10:39:16.698568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:45:00.266470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698350+0000","last_change":"2024-03-24T10:38:10.889668+0000","last_active":"2024-03-24T10:39:16.698350+0000","last_peered":"2024-03-24T10:39:16.698350+0000","last_clean":"2024-03-24T10:39:16.698350+0000","last_became_active":"2024-03-24T10:38:10.889364+0000","last_became_peered":"2024-03-24T10:38:10.889364+0000","last_unstale":"2024-03-24T10:39:16.698350+0000","last_undegraded":"2024-03-24T10:39:16.698350+0000","last_fullsized":"2024-03-24T10:39:16.698350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:57.702719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'27","reported_seq":185,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:20.985182+0000","last_change":"2024-03-24T10:38:09.861809+0000","last_active":"2024-03-24T10:39:20.985182+0000","last_peered":"2024-03-24T10:39:20.985182+0000","last_clean":"2024-03-24T10:39:20.985182+0000","last_became_active":"2024-03-24T10:38:09.861444+0000","last_became_peered":"2024-03-24T10:38:09.861444+0000","last_unstale":"2024-03-24T10:39:20.985182+0000","last_undegraded":"2024-03-24T10:39:20.985182+0000","last_fullsized":"2024-03-24T10:39:20.985182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:19.652188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59465,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":139,"num_read_kb":127,"num_write":28,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695671+0000","last_change":"2024-03-24T10:38:10.883090+0000","last_active":"2024-03-24T10:39:16.695671+0000","last_peered":"2024-03-24T10:39:16.695671+0000","last_clean":"2024-03-24T10:39:16.695671+0000","last_became_active":"2024-03-24T10:38:10.880507+0000","last_became_peered":"2024-03-24T10:38:10.880507+0000","last_unstale":"2024-03-24T10:39:16.695671+0000","last_undegraded":"2024-03-24T10:39:16.695671+0000","last_fullsized":"2024-03-24T10:39:16.695671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:01.422699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697835+0000","last_change":"2024-03-24T10:38:09.868072+0000","last_active":"2024-03-24T10:39:16.697835+0000","last_peered":"2024-03-24T10:39:16.697835+0000","last_clean":"2024-03-24T10:39:16.697835+0000","last_became_active":"2024-03-24T10:38:09.867704+0000","last_became_peered":"2024-03-24T10:38:09.867704+0000","last_unstale":"2024-03-24T10:39:16.697835+0000","last_undegraded":"2024-03-24T10:39:16.697835+0000","last_fullsized":"2024-03-24T10:39:16.697835+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:09:28.726969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.701952+0000","last_change":"2024-03-24T10:38:10.881882+0000","last_active":"2024-03-24T10:39:16.701952+0000","last_peered":"2024-03-24T10:39:16.701952+0000","last_clean":"2024-03-24T10:39:16.701952+0000","last_became_active":"2024-03-24T10:38:10.881495+0000","last_became_peered":"2024-03-24T10:38:10.881495+0000","last_unstale":"2024-03-24T10:39:16.701952+0000","last_undegraded":"2024-03-24T10:39:16.701952+0000","last_fullsized":"2024-03-24T10:39:16.701952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:31:43.392952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698414+0000","last_change":"2024-03-24T10:38:09.867265+0000","last_active":"2024-03-24T10:39:16.698414+0000","last_peered":"2024-03-24T10:39:16.698414+0000","last_clean":"2024-03-24T10:39:16.698414+0000","last_became_active":"2024-03-24T10:38:09.866669+0000","last_became_peered":"2024-03-24T10:38:09.866669+0000","last_unstale":"2024-03-24T10:39:16.698414+0000","last_undegraded":"2024-03-24T10:39:16.698414+0000","last_fullsized":"2024-03-24T10:39:16.698414+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:28.230967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696333+0000","last_change":"2024-03-24T10:38:10.886634+0000","last_active":"2024-03-24T10:39:16.696333+0000","last_peered":"2024-03-24T10:39:16.696333+0000","last_clean":"2024-03-24T10:39:16.696333+0000","last_became_active":"2024-03-24T10:38:10.886207+0000","last_became_peered":"2024-03-24T10:38:10.886207+0000","last_unstale":"2024-03-24T10:39:16.696333+0000","last_undegraded":"2024-03-24T10:39:16.696333+0000","last_fullsized":"2024-03-24T10:39:16.696333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:43:41.601693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":89,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.983839+0000","last_change":"2024-03-24T10:38:09.863262+0000","last_active":"2024-03-24T10:39:16.983839+0000","last_peered":"2024-03-24T10:39:16.983839+0000","last_clean":"2024-03-24T10:39:16.983839+0000","last_became_active":"2024-03-24T10:38:09.862665+0000","last_became_peered":"2024-03-24T10:38:09.862665+0000","last_unstale":"2024-03-24T10:39:16.983839+0000","last_undegraded":"2024-03-24T10:39:16.983839+0000","last_fullsized":"2024-03-24T10:39:16.983839+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:12:29.117592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695642+0000","last_change":"2024-03-24T10:38:10.880793+0000","last_active":"2024-03-24T10:39:16.695642+0000","last_peered":"2024-03-24T10:39:16.695642+0000","last_clean":"2024-03-24T10:39:16.695642+0000","last_became_active":"2024-03-24T10:38:10.880449+0000","last_became_peered":"2024-03-24T10:38:10.880449+0000","last_unstale":"2024-03-24T10:39:16.695642+0000","last_undegraded":"2024-03-24T10:39:16.695642+0000","last_fullsized":"2024-03-24T10:39:16.695642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:32:57.184014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.915438+0000","last_change":"2024-03-24T10:39:16.915438+0000","last_active":"2024-03-24T10:39:16.915438+0000","last_peered":"2024-03-24T10:39:16.915438+0000","last_clean":"2024-03-24T10:39:16.915438+0000","last_became_active":"2024-03-24T10:39:16.906155+0000","last_became_peered":"2024-03-24T10:39:16.906155+0000","last_unstale":"2024-03-24T10:39:16.915438+0000","last_undegraded":"2024-03-24T10:39:16.915438+0000","last_fullsized":"2024-03-24T10:39:16.915438+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:36:12.356069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698214+0000","last_change":"2024-03-24T10:38:10.881952+0000","last_active":"2024-03-24T10:39:16.698214+0000","last_peered":"2024-03-24T10:39:16.698214+0000","last_clean":"2024-03-24T10:39:16.698214+0000","last_became_active":"2024-03-24T10:38:10.881628+0000","last_became_peered":"2024-03-24T10:38:10.881628+0000","last_unstale":"2024-03-24T10:39:16.698214+0000","last_undegraded":"2024-03-24T10:39:16.698214+0000","last_fullsized":"2024-03-24T10:39:16.698214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:24.777793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695669+0000","last_change":"2024-03-24T10:38:09.868572+0000","last_active":"2024-03-24T10:39:16.695669+0000","last_peered":"2024-03-24T10:39:16.695669+0000","last_clean":"2024-03-24T10:39:16.695669+0000","last_became_active":"2024-03-24T10:38:09.867702+0000","last_became_peered":"2024-03-24T10:38:09.867702+0000","last_unstale":"2024-03-24T10:39:16.695669+0000","last_undegraded":"2024-03-24T10:39:16.695669+0000","last_fullsized":"2024-03-24T10:39:16.695669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:27.890114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697936+0000","last_change":"2024-03-24T10:38:10.875342+0000","last_active":"2024-03-24T10:39:16.697936+0000","last_peered":"2024-03-24T10:39:16.697936+0000","last_clean":"2024-03-24T10:39:16.697936+0000","last_became_active":"2024-03-24T10:38:10.874933+0000","last_became_peered":"2024-03-24T10:38:10.874933+0000","last_unstale":"2024-03-24T10:39:16.697936+0000","last_undegraded":"2024-03-24T10:39:16.697936+0000","last_fullsized":"2024-03-24T10:39:16.697936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:56:09.139478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697917+0000","last_change":"2024-03-24T10:38:09.857867+0000","last_active":"2024-03-24T10:39:16.697917+0000","last_peered":"2024-03-24T10:39:16.697917+0000","last_clean":"2024-03-24T10:39:16.697917+0000","last_became_active":"2024-03-24T10:38:09.857525+0000","last_became_peered":"2024-03-24T10:38:09.857525+0000","last_unstale":"2024-03-24T10:39:16.697917+0000","last_undegraded":"2024-03-24T10:39:16.697917+0000","last_fullsized":"2024-03-24T10:39:16.697917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:37.320245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.725224+0000","last_change":"2024-03-24T10:38:10.878980+0000","last_active":"2024-03-24T10:39:16.725224+0000","last_peered":"2024-03-24T10:39:16.725224+0000","last_clean":"2024-03-24T10:39:16.725224+0000","last_became_active":"2024-03-24T10:38:10.878648+0000","last_became_peered":"2024-03-24T10:38:10.878648+0000","last_unstale":"2024-03-24T10:39:16.725224+0000","last_undegraded":"2024-03-24T10:39:16.725224+0000","last_fullsized":"2024-03-24T10:39:16.725224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:49.988403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695848+0000","last_change":"2024-03-24T10:38:09.863457+0000","last_active":"2024-03-24T10:39:16.695848+0000","last_peered":"2024-03-24T10:39:16.695848+0000","last_clean":"2024-03-24T10:39:16.695848+0000","last_became_active":"2024-03-24T10:38:09.863130+0000","last_became_peered":"2024-03-24T10:38:09.863130+0000","last_unstale":"2024-03-24T10:39:16.695848+0000","last_undegraded":"2024-03-24T10:39:16.695848+0000","last_fullsized":"2024-03-24T10:39:16.695848+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:16.528703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.725232+0000","last_change":"2024-03-24T10:38:10.887014+0000","last_active":"2024-03-24T10:39:16.725232+0000","last_peered":"2024-03-24T10:39:16.725232+0000","last_clean":"2024-03-24T10:39:16.725232+0000","last_became_active":"2024-03-24T10:38:10.886698+0000","last_became_peered":"2024-03-24T10:38:10.886698+0000","last_unstale":"2024-03-24T10:39:16.725232+0000","last_undegraded":"2024-03-24T10:39:16.725232+0000","last_fullsized":"2024-03-24T10:39:16.725232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:47:08.571886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698893+0000","last_change":"2024-03-24T10:38:10.874811+0000","last_active":"2024-03-24T10:39:16.698893+0000","last_peered":"2024-03-24T10:39:16.698893+0000","last_clean":"2024-03-24T10:39:16.698893+0000","last_became_active":"2024-03-24T10:38:10.874483+0000","last_became_peered":"2024-03-24T10:38:10.874483+0000","last_unstale":"2024-03-24T10:39:16.698893+0000","last_undegraded":"2024-03-24T10:39:16.698893+0000","last_fullsized":"2024-03-24T10:39:16.698893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:57.382174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":90,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:20.987529+0000","last_change":"2024-03-24T10:38:09.865614+0000","last_active":"2024-03-24T10:39:20.987529+0000","last_peered":"2024-03-24T10:39:20.987529+0000","last_clean":"2024-03-24T10:39:20.987529+0000","last_became_active":"2024-03-24T10:38:09.865317+0000","last_became_peered":"2024-03-24T10:38:09.865317+0000","last_unstale":"2024-03-24T10:39:20.987529+0000","last_undegraded":"2024-03-24T10:39:20.987529+0000","last_fullsized":"2024-03-24T10:39:20.987529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:21:14.968552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":67,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.700602+0000","last_change":"2024-03-24T10:38:10.884847+0000","last_active":"2024-03-24T10:39:16.700602+0000","last_peered":"2024-03-24T10:39:16.700602+0000","last_clean":"2024-03-24T10:39:16.700602+0000","last_became_active":"2024-03-24T10:38:10.884526+0000","last_became_peered":"2024-03-24T10:38:10.884526+0000","last_unstale":"2024-03-24T10:39:16.700602+0000","last_undegraded":"2024-03-24T10:39:16.700602+0000","last_fullsized":"2024-03-24T10:39:16.700602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:01:49.364083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.905135+0000","last_change":"2024-03-24T10:38:09.858893+0000","last_active":"2024-03-24T10:39:16.905135+0000","last_peered":"2024-03-24T10:39:16.905135+0000","last_clean":"2024-03-24T10:39:16.905135+0000","last_became_active":"2024-03-24T10:38:09.858474+0000","last_became_peered":"2024-03-24T10:38:09.858474+0000","last_unstale":"2024-03-24T10:39:16.905135+0000","last_undegraded":"2024-03-24T10:39:16.905135+0000","last_fullsized":"2024-03-24T10:39:16.905135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:03:36.709498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":351,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695719+0000","last_change":"2024-03-24T10:36:59.100728+0000","last_active":"2024-03-24T10:39:16.695719+0000","last_peered":"2024-03-24T10:39:16.695719+0000","last_clean":"2024-03-24T10:39:16.695719+0000","last_became_active":"2024-03-24T10:36:59.100159+0000","last_became_peered":"2024-03-24T10:36:59.100159+0000","last_unstale":"2024-03-24T10:39:16.695719+0000","last_undegraded":"2024-03-24T10:39:16.695719+0000","last_fullsized":"2024-03-24T10:39:16.695719+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:39.429553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:39.429553+0000","last_clean_scrub_stamp":"2024-03-24T10:33:39.429553+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:21:01.470585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.700727+0000","last_change":"2024-03-24T10:38:10.885804+0000","last_active":"2024-03-24T10:39:16.700727+0000","last_peered":"2024-03-24T10:39:16.700727+0000","last_clean":"2024-03-24T10:39:16.700727+0000","last_became_active":"2024-03-24T10:38:10.885447+0000","last_became_peered":"2024-03-24T10:38:10.885447+0000","last_unstale":"2024-03-24T10:39:16.700727+0000","last_undegraded":"2024-03-24T10:39:16.700727+0000","last_fullsized":"2024-03-24T10:39:16.700727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:57.103205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696024+0000","last_change":"2024-03-24T10:38:09.863187+0000","last_active":"2024-03-24T10:39:16.696024+0000","last_peered":"2024-03-24T10:39:16.696024+0000","last_clean":"2024-03-24T10:39:16.696024+0000","last_became_active":"2024-03-24T10:38:09.862859+0000","last_became_peered":"2024-03-24T10:38:09.862859+0000","last_unstale":"2024-03-24T10:39:16.696024+0000","last_undegraded":"2024-03-24T10:39:16.696024+0000","last_fullsized":"2024-03-24T10:39:16.696024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:21:02.707127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695813+0000","last_change":"2024-03-24T10:38:09.867899+0000","last_active":"2024-03-24T10:39:16.695813+0000","last_peered":"2024-03-24T10:39:16.695813+0000","last_clean":"2024-03-24T10:39:16.695813+0000","last_became_active":"2024-03-24T10:38:09.867563+0000","last_became_peered":"2024-03-24T10:38:09.867563+0000","last_unstale":"2024-03-24T10:39:16.695813+0000","last_undegraded":"2024-03-24T10:39:16.695813+0000","last_fullsized":"2024-03-24T10:39:16.695813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:51.067981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.701363+0000","last_change":"2024-03-24T10:38:10.870825+0000","last_active":"2024-03-24T10:39:16.701363+0000","last_peered":"2024-03-24T10:39:16.701363+0000","last_clean":"2024-03-24T10:39:16.701363+0000","last_became_active":"2024-03-24T10:38:10.870491+0000","last_became_peered":"2024-03-24T10:38:10.870491+0000","last_unstale":"2024-03-24T10:39:16.701363+0000","last_undegraded":"2024-03-24T10:39:16.701363+0000","last_fullsized":"2024-03-24T10:39:16.701363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:30:34.091299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698460+0000","last_change":"2024-03-24T10:38:09.863724+0000","last_active":"2024-03-24T10:39:16.698460+0000","last_peered":"2024-03-24T10:39:16.698460+0000","last_clean":"2024-03-24T10:39:16.698460+0000","last_became_active":"2024-03-24T10:38:09.862900+0000","last_became_peered":"2024-03-24T10:38:09.862900+0000","last_unstale":"2024-03-24T10:39:16.698460+0000","last_undegraded":"2024-03-24T10:39:16.698460+0000","last_fullsized":"2024-03-24T10:39:16.698460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:21.646074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699470+0000","last_change":"2024-03-24T10:38:10.884661+0000","last_active":"2024-03-24T10:39:16.699470+0000","last_peered":"2024-03-24T10:39:16.699470+0000","last_clean":"2024-03-24T10:39:16.699470+0000","last_became_active":"2024-03-24T10:38:10.882901+0000","last_became_peered":"2024-03-24T10:38:10.882901+0000","last_unstale":"2024-03-24T10:39:16.699470+0000","last_undegraded":"2024-03-24T10:39:16.699470+0000","last_fullsized":"2024-03-24T10:39:16.699470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:23:46.366567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.905272+0000","last_change":"2024-03-24T10:38:09.862122+0000","last_active":"2024-03-24T10:39:16.905272+0000","last_peered":"2024-03-24T10:39:16.905272+0000","last_clean":"2024-03-24T10:39:16.905272+0000","last_became_active":"2024-03-24T10:38:09.861765+0000","last_became_peered":"2024-03-24T10:38:09.861765+0000","last_unstale":"2024-03-24T10:39:16.905272+0000","last_undegraded":"2024-03-24T10:39:16.905272+0000","last_fullsized":"2024-03-24T10:39:16.905272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:40:15.148349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696018+0000","last_change":"2024-03-24T10:38:10.886797+0000","last_active":"2024-03-24T10:39:16.696018+0000","last_peered":"2024-03-24T10:39:16.696018+0000","last_clean":"2024-03-24T10:39:16.696018+0000","last_became_active":"2024-03-24T10:38:10.886406+0000","last_became_peered":"2024-03-24T10:38:10.886406+0000","last_unstale":"2024-03-24T10:39:16.696018+0000","last_undegraded":"2024-03-24T10:39:16.696018+0000","last_fullsized":"2024-03-24T10:39:16.696018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:35.763623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.707034+0000","last_change":"2024-03-24T10:39:16.707034+0000","last_active":"2024-03-24T10:39:16.707034+0000","last_peered":"2024-03-24T10:39:16.707034+0000","last_clean":"2024-03-24T10:39:16.707034+0000","last_became_active":"2024-03-24T10:39:16.701995+0000","last_became_peered":"2024-03-24T10:39:16.701995+0000","last_unstale":"2024-03-24T10:39:16.707034+0000","last_undegraded":"2024-03-24T10:39:16.707034+0000","last_fullsized":"2024-03-24T10:39:16.707034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:18:41.362740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695469+0000","last_change":"2024-03-24T10:38:10.873871+0000","last_active":"2024-03-24T10:39:16.695469+0000","last_peered":"2024-03-24T10:39:16.695469+0000","last_clean":"2024-03-24T10:39:16.695469+0000","last_became_active":"2024-03-24T10:38:10.873528+0000","last_became_peered":"2024-03-24T10:38:10.873528+0000","last_unstale":"2024-03-24T10:39:16.695469+0000","last_undegraded":"2024-03-24T10:39:16.695469+0000","last_fullsized":"2024-03-24T10:39:16.695469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:33.372195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699787+0000","last_change":"2024-03-24T10:38:09.861548+0000","last_active":"2024-03-24T10:39:16.699787+0000","last_peered":"2024-03-24T10:39:16.699787+0000","last_clean":"2024-03-24T10:39:16.699787+0000","last_became_active":"2024-03-24T10:38:09.861219+0000","last_became_peered":"2024-03-24T10:38:09.861219+0000","last_unstale":"2024-03-24T10:39:16.699787+0000","last_undegraded":"2024-03-24T10:39:16.699787+0000","last_fullsized":"2024-03-24T10:39:16.699787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:57:46.398179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697418+0000","last_change":"2024-03-24T10:38:10.880202+0000","last_active":"2024-03-24T10:39:16.697418+0000","last_peered":"2024-03-24T10:39:16.697418+0000","last_clean":"2024-03-24T10:39:16.697418+0000","last_became_active":"2024-03-24T10:38:10.879807+0000","last_became_peered":"2024-03-24T10:38:10.879807+0000","last_unstale":"2024-03-24T10:39:16.697418+0000","last_undegraded":"2024-03-24T10:39:16.697418+0000","last_fullsized":"2024-03-24T10:39:16.697418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:37.498728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.704470+0000","last_change":"2024-03-24T10:39:16.704470+0000","last_active":"2024-03-24T10:39:16.704470+0000","last_peered":"2024-03-24T10:39:16.704470+0000","last_clean":"2024-03-24T10:39:16.704470+0000","last_became_active":"2024-03-24T10:39:16.701168+0000","last_became_peered":"2024-03-24T10:39:16.701168+0000","last_unstale":"2024-03-24T10:39:16.704470+0000","last_undegraded":"2024-03-24T10:39:16.704470+0000","last_fullsized":"2024-03-24T10:39:16.704470+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:20:34.781782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699419+0000","last_change":"2024-03-24T10:38:10.884371+0000","last_active":"2024-03-24T10:39:16.699419+0000","last_peered":"2024-03-24T10:39:16.699419+0000","last_clean":"2024-03-24T10:39:16.699419+0000","last_became_active":"2024-03-24T10:38:10.884022+0000","last_became_peered":"2024-03-24T10:38:10.884022+0000","last_unstale":"2024-03-24T10:39:16.699419+0000","last_undegraded":"2024-03-24T10:39:16.699419+0000","last_fullsized":"2024-03-24T10:39:16.699419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:47:48.778727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695865+0000","last_change":"2024-03-24T10:38:09.863231+0000","last_active":"2024-03-24T10:39:16.695865+0000","last_peered":"2024-03-24T10:39:16.695865+0000","last_clean":"2024-03-24T10:39:16.695865+0000","last_became_active":"2024-03-24T10:38:09.862848+0000","last_became_peered":"2024-03-24T10:38:09.862848+0000","last_unstale":"2024-03-24T10:39:16.695865+0000","last_undegraded":"2024-03-24T10:39:16.695865+0000","last_fullsized":"2024-03-24T10:39:16.695865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:36:57.532746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696011+0000","last_change":"2024-03-24T10:38:10.875187+0000","last_active":"2024-03-24T10:39:16.696011+0000","last_peered":"2024-03-24T10:39:16.696011+0000","last_clean":"2024-03-24T10:39:16.696011+0000","last_became_active":"2024-03-24T10:38:10.874682+0000","last_became_peered":"2024-03-24T10:38:10.874682+0000","last_unstale":"2024-03-24T10:39:16.696011+0000","last_undegraded":"2024-03-24T10:39:16.696011+0000","last_fullsized":"2024-03-24T10:39:16.696011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:53:56.915312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699445+0000","last_change":"2024-03-24T10:38:09.863430+0000","last_active":"2024-03-24T10:39:16.699445+0000","last_peered":"2024-03-24T10:39:16.699445+0000","last_clean":"2024-03-24T10:39:16.699445+0000","last_became_active":"2024-03-24T10:38:09.862877+0000","last_became_peered":"2024-03-24T10:38:09.862877+0000","last_unstale":"2024-03-24T10:39:16.699445+0000","last_undegraded":"2024-03-24T10:39:16.699445+0000","last_fullsized":"2024-03-24T10:39:16.699445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:33:29.039896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695959+0000","last_change":"2024-03-24T10:38:10.878446+0000","last_active":"2024-03-24T10:39:16.695959+0000","last_peered":"2024-03-24T10:39:16.695959+0000","last_clean":"2024-03-24T10:39:16.695959+0000","last_became_active":"2024-03-24T10:38:10.877944+0000","last_became_peered":"2024-03-24T10:38:10.877944+0000","last_unstale":"2024-03-24T10:39:16.695959+0000","last_undegraded":"2024-03-24T10:39:16.695959+0000","last_fullsized":"2024-03-24T10:39:16.695959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:00:22.568669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698328+0000","last_change":"2024-03-24T10:38:09.867328+0000","last_active":"2024-03-24T10:39:16.698328+0000","last_peered":"2024-03-24T10:39:16.698328+0000","last_clean":"2024-03-24T10:39:16.698328+0000","last_became_active":"2024-03-24T10:38:09.866869+0000","last_became_peered":"2024-03-24T10:38:09.866869+0000","last_unstale":"2024-03-24T10:39:16.698328+0000","last_undegraded":"2024-03-24T10:39:16.698328+0000","last_fullsized":"2024-03-24T10:39:16.698328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:52:32.136151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.705912+0000","last_change":"2024-03-24T10:39:16.705912+0000","last_active":"2024-03-24T10:39:16.705912+0000","last_peered":"2024-03-24T10:39:16.705912+0000","last_clean":"2024-03-24T10:39:16.705912+0000","last_became_active":"2024-03-24T10:39:16.699809+0000","last_became_peered":"2024-03-24T10:39:16.699809+0000","last_unstale":"2024-03-24T10:39:16.705912+0000","last_undegraded":"2024-03-24T10:39:16.705912+0000","last_fullsized":"2024-03-24T10:39:16.705912+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:31.858976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.701719+0000","last_change":"2024-03-24T10:38:09.865399+0000","last_active":"2024-03-24T10:39:16.701719+0000","last_peered":"2024-03-24T10:39:16.701719+0000","last_clean":"2024-03-24T10:39:16.701719+0000","last_became_active":"2024-03-24T10:38:09.864906+0000","last_became_peered":"2024-03-24T10:38:09.864906+0000","last_unstale":"2024-03-24T10:39:16.701719+0000","last_undegraded":"2024-03-24T10:39:16.701719+0000","last_fullsized":"2024-03-24T10:39:16.701719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:53:12.363018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696203+0000","last_change":"2024-03-24T10:38:10.876881+0000","last_active":"2024-03-24T10:39:16.696203+0000","last_peered":"2024-03-24T10:39:16.696203+0000","last_clean":"2024-03-24T10:39:16.696203+0000","last_became_active":"2024-03-24T10:38:10.876479+0000","last_became_peered":"2024-03-24T10:38:10.876479+0000","last_unstale":"2024-03-24T10:39:16.696203+0000","last_undegraded":"2024-03-24T10:39:16.696203+0000","last_fullsized":"2024-03-24T10:39:16.696203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:57:33.140451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697521+0000","last_change":"2024-03-24T10:38:10.872622+0000","last_active":"2024-03-24T10:39:16.697521+0000","last_peered":"2024-03-24T10:39:16.697521+0000","last_clean":"2024-03-24T10:39:16.697521+0000","last_became_active":"2024-03-24T10:38:10.872293+0000","last_became_peered":"2024-03-24T10:38:10.872293+0000","last_unstale":"2024-03-24T10:39:16.697521+0000","last_undegraded":"2024-03-24T10:39:16.697521+0000","last_fullsized":"2024-03-24T10:39:16.697521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:48.139589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.701548+0000","last_change":"2024-03-24T10:39:16.701548+0000","last_active":"2024-03-24T10:39:16.701548+0000","last_peered":"2024-03-24T10:39:16.701548+0000","last_clean":"2024-03-24T10:39:16.701548+0000","last_became_active":"2024-03-24T10:39:16.701213+0000","last_became_peered":"2024-03-24T10:39:16.701213+0000","last_unstale":"2024-03-24T10:39:16.701548+0000","last_undegraded":"2024-03-24T10:39:16.701548+0000","last_fullsized":"2024-03-24T10:39:16.701548+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:50:28.131754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697504+0000","last_change":"2024-03-24T10:38:10.879679+0000","last_active":"2024-03-24T10:39:16.697504+0000","last_peered":"2024-03-24T10:39:16.697504+0000","last_clean":"2024-03-24T10:39:16.697504+0000","last_became_active":"2024-03-24T10:38:10.879334+0000","last_became_peered":"2024-03-24T10:38:10.879334+0000","last_unstale":"2024-03-24T10:39:16.697504+0000","last_undegraded":"2024-03-24T10:39:16.697504+0000","last_fullsized":"2024-03-24T10:39:16.697504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:27:53.517368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.704444+0000","last_change":"2024-03-24T10:39:16.704444+0000","last_active":"2024-03-24T10:39:16.704444+0000","last_peered":"2024-03-24T10:39:16.704444+0000","last_clean":"2024-03-24T10:39:16.704444+0000","last_became_active":"2024-03-24T10:39:16.704085+0000","last_became_peered":"2024-03-24T10:39:16.704085+0000","last_unstale":"2024-03-24T10:39:16.704444+0000","last_undegraded":"2024-03-24T10:39:16.704444+0000","last_fullsized":"2024-03-24T10:39:16.704444+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:04:49.643195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.701459+0000","last_change":"2024-03-24T10:38:10.887108+0000","last_active":"2024-03-24T10:39:16.701459+0000","last_peered":"2024-03-24T10:39:16.701459+0000","last_clean":"2024-03-24T10:39:16.701459+0000","last_became_active":"2024-03-24T10:38:10.886770+0000","last_became_peered":"2024-03-24T10:38:10.886770+0000","last_unstale":"2024-03-24T10:39:16.701459+0000","last_undegraded":"2024-03-24T10:39:16.701459+0000","last_fullsized":"2024-03-24T10:39:16.701459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:56.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697741+0000","last_change":"2024-03-24T10:38:10.873369+0000","last_active":"2024-03-24T10:39:16.697741+0000","last_peered":"2024-03-24T10:39:16.697741+0000","last_clean":"2024-03-24T10:39:16.697741+0000","last_became_active":"2024-03-24T10:38:10.873046+0000","last_became_peered":"2024-03-24T10:38:10.873046+0000","last_unstale":"2024-03-24T10:39:16.697741+0000","last_undegraded":"2024-03-24T10:39:16.697741+0000","last_fullsized":"2024-03-24T10:39:16.697741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:46.071084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699744+0000","last_change":"2024-03-24T10:38:10.878666+0000","last_active":"2024-03-24T10:39:16.699744+0000","last_peered":"2024-03-24T10:39:16.699744+0000","last_clean":"2024-03-24T10:39:16.699744+0000","last_became_active":"2024-03-24T10:38:10.877250+0000","last_became_peered":"2024-03-24T10:38:10.877250+0000","last_unstale":"2024-03-24T10:39:16.699744+0000","last_undegraded":"2024-03-24T10:39:16.699744+0000","last_fullsized":"2024-03-24T10:39:16.699744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:13:36.793584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699422+0000","last_change":"2024-03-24T10:38:10.889274+0000","last_active":"2024-03-24T10:39:16.699422+0000","last_peered":"2024-03-24T10:39:16.699422+0000","last_clean":"2024-03-24T10:39:16.699422+0000","last_became_active":"2024-03-24T10:38:10.888899+0000","last_became_peered":"2024-03-24T10:38:10.888899+0000","last_unstale":"2024-03-24T10:39:16.699422+0000","last_undegraded":"2024-03-24T10:39:16.699422+0000","last_fullsized":"2024-03-24T10:39:16.699422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:11.225562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698707+0000","last_change":"2024-03-24T10:38:10.887135+0000","last_active":"2024-03-24T10:39:16.698707+0000","last_peered":"2024-03-24T10:39:16.698707+0000","last_clean":"2024-03-24T10:39:16.698707+0000","last_became_active":"2024-03-24T10:38:10.886812+0000","last_became_peered":"2024-03-24T10:38:10.886812+0000","last_unstale":"2024-03-24T10:39:16.698707+0000","last_undegraded":"2024-03-24T10:39:16.698707+0000","last_fullsized":"2024-03-24T10:39:16.698707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:51:53.959914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699624+0000","last_change":"2024-03-24T10:38:10.885417+0000","last_active":"2024-03-24T10:39:16.699624+0000","last_peered":"2024-03-24T10:39:16.699624+0000","last_clean":"2024-03-24T10:39:16.699624+0000","last_became_active":"2024-03-24T10:38:10.883209+0000","last_became_peered":"2024-03-24T10:38:10.883209+0000","last_unstale":"2024-03-24T10:39:16.699624+0000","last_undegraded":"2024-03-24T10:39:16.699624+0000","last_fullsized":"2024-03-24T10:39:16.699624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:52:52.535611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697650+0000","last_change":"2024-03-24T10:38:10.885005+0000","last_active":"2024-03-24T10:39:16.697650+0000","last_peered":"2024-03-24T10:39:16.697650+0000","last_clean":"2024-03-24T10:39:16.697650+0000","last_became_active":"2024-03-24T10:38:10.884680+0000","last_became_peered":"2024-03-24T10:38:10.884680+0000","last_unstale":"2024-03-24T10:39:16.697650+0000","last_undegraded":"2024-03-24T10:39:16.697650+0000","last_fullsized":"2024-03-24T10:39:16.697650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:56.377777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.702195+0000","last_change":"2024-03-24T10:38:10.875153+0000","last_active":"2024-03-24T10:39:16.702195+0000","last_peered":"2024-03-24T10:39:16.702195+0000","last_clean":"2024-03-24T10:39:16.702195+0000","last_became_active":"2024-03-24T10:38:10.874721+0000","last_became_peered":"2024-03-24T10:38:10.874721+0000","last_unstale":"2024-03-24T10:39:16.702195+0000","last_undegraded":"2024-03-24T10:39:16.702195+0000","last_fullsized":"2024-03-24T10:39:16.702195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:52.522797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697574+0000","last_change":"2024-03-24T10:38:10.887298+0000","last_active":"2024-03-24T10:39:16.697574+0000","last_peered":"2024-03-24T10:39:16.697574+0000","last_clean":"2024-03-24T10:39:16.697574+0000","last_became_active":"2024-03-24T10:38:10.886975+0000","last_became_peered":"2024-03-24T10:38:10.886975+0000","last_unstale":"2024-03-24T10:39:16.697574+0000","last_undegraded":"2024-03-24T10:39:16.697574+0000","last_fullsized":"2024-03-24T10:39:16.697574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:09.740303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63466,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":420,"num_read_kb":279,"num_write":155,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":190398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383708,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":950272,"data_stored":456395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645345,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32788,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712364,"statfs":{"total":95995035648,"available":95961460736,"internally_reserved":0,"allocated":1646592,"data_stored":1107732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808870,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":456748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070506,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":455969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332143,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1560576,"data_stored":1048481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495669,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":456020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1019904,"data_stored":515860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":457291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":516516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182467,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":456214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920830,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":456960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757305,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1572864,"data_stored":1047774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":60047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":60703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:39:24.546 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T10:39:24.793 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:24.906 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:24 smithi023 ceph-mon[32466]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 101 B/s, 1 objects/s recovering 2024-03-24T10:39:24.906 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:24 smithi023 ceph-mon[32466]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-24T10:39:24.906 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:24 smithi023 ceph-mon[32466]: Cluster is now healthy 2024-03-24T10:39:25.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:24 smithi117 ceph-mon[36368]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 101 B/s, 1 objects/s recovering 2024-03-24T10:39:25.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:24 smithi117 ceph-mon[36368]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-24T10:39:25.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:24 smithi117 ceph-mon[36368]: Cluster is now healthy 2024-03-24T10:39:25.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:24 smithi116 ceph-mon[36248]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 101 B/s, 1 objects/s recovering 2024-03-24T10:39:25.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:24 smithi116 ceph-mon[36248]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-24T10:39:25.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:24 smithi116 ceph-mon[36248]: Cluster is now healthy 2024-03-24T10:39:25.812 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:25 smithi023 ceph-mon[32466]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:39:26.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:25 smithi117 ceph-mon[36368]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:39:26.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:25 smithi116 ceph-mon[36248]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:39:26.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.388+0000 7f89b5e32700 1 -- 172.21.15.23:0/1767256926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89b0102250 msgr2=0x7f89b01026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:26.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.388+0000 7f89b5e32700 1 --2- 172.21.15.23:0/1767256926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89b0102250 0x7f89b01026d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8998009990 tx=0x7f8998009ca0 comp rx=0 tx=0).stop 2024-03-24T10:39:26.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.388+0000 7f89b5e32700 1 -- 172.21.15.23:0/1767256926 shutdown_connections 2024-03-24T10:39:26.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.388+0000 7f89b5e32700 1 --2- 172.21.15.23:0/1767256926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89b0108300 0x7f89b01086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:26.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.388+0000 7f89b5e32700 1 --2- 172.21.15.23:0/1767256926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89b0102c10 0x7f89b010f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:26.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.388+0000 7f89b5e32700 1 --2- 172.21.15.23:0/1767256926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89b0102250 0x7f89b01026d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:26.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.388+0000 7f89b5e32700 1 -- 172.21.15.23:0/1767256926 >> 172.21.15.23:0/1767256926 conn(0x7f89b0079330 msgr2=0x7f89b0079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:26.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.388+0000 7f89b5e32700 1 -- 172.21.15.23:0/1767256926 shutdown_connections 2024-03-24T10:39:26.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.388+0000 7f89b5e32700 1 -- 172.21.15.23:0/1767256926 wait complete. 2024-03-24T10:39:26.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.389+0000 7f89b5e32700 1 Processor -- start 2024-03-24T10:39:26.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.389+0000 7f89b5e32700 1 -- start start 2024-03-24T10:39:26.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.389+0000 7f89b5e32700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89b0102250 0x7f89b01986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:26.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.389+0000 7f89b5e32700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89b0102c10 0x7f89b0198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:26.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.389+0000 7f89b5e32700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89b0108300 0x7f89b019cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:26.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.389+0000 7f89b5e32700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f89b019d5a0 con 0x7f89b0108300 2024-03-24T10:39:26.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.389+0000 7f89b5e32700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f89b019d710 con 0x7f89b0102c10 2024-03-24T10:39:26.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.389+0000 7f89b5e32700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f89b019d8d0 con 0x7f89b0102250 2024-03-24T10:39:26.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.389+0000 7f89affff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89b0108300 0x7f89b019cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:26.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.389+0000 7f89af7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89b0102250 0x7f89b01986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:26.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.389+0000 7f89aeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89b0102c10 0x7f89b0198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:26.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.389+0000 7f89affff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89b0108300 0x7f89b019cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50132/0 (socket says 172.21.15.23:50132) 2024-03-24T10:39:26.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.389+0000 7f89affff700 1 -- 172.21.15.23:0/3516620796 learned_addr learned my addr 172.21.15.23:0/3516620796 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:26.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.390+0000 7f89af7fe700 1 -- 172.21.15.23:0/3516620796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89b0102c10 msgr2=0x7f89b0198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:26.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.390+0000 7f89af7fe700 1 --2- 172.21.15.23:0/3516620796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89b0102c10 0x7f89b0198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:26.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.390+0000 7f89af7fe700 1 -- 172.21.15.23:0/3516620796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89b0108300 msgr2=0x7f89b019cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:26.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.390+0000 7f89af7fe700 1 --2- 172.21.15.23:0/3516620796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89b0108300 0x7f89b019cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:26.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.390+0000 7f89af7fe700 1 -- 172.21.15.23:0/3516620796 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89980095f0 con 0x7f89b0102250 2024-03-24T10:39:26.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.390+0000 7f89aeffd700 1 --2- 172.21.15.23:0/3516620796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89b0102c10 0x7f89b0198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:26.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.390+0000 7f89af7fe700 1 --2- 172.21.15.23:0/3516620796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89b0102250 0x7f89b01986c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f89a000ef10 tx=0x7f89a000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:26.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.390+0000 7f89acff9700 1 -- 172.21.15.23:0/3516620796 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a000c9c0 con 0x7f89b0102250 2024-03-24T10:39:26.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.390+0000 7f89acff9700 1 -- 172.21.15.23:0/3516620796 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89a000f040 con 0x7f89b0102250 2024-03-24T10:39:26.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.390+0000 7f89b5e32700 1 -- 172.21.15.23:0/3516620796 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b019db50 con 0x7f89b0102250 2024-03-24T10:39:26.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.390+0000 7f89acff9700 1 -- 172.21.15.23:0/3516620796 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a00146e0 con 0x7f89b0102250 2024-03-24T10:39:26.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.390+0000 7f89b5e32700 1 -- 172.21.15.23:0/3516620796 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b01a5410 con 0x7f89b0102250 2024-03-24T10:39:26.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.391+0000 7f89b5e32700 1 -- 172.21.15.23:0/3516620796 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89b010a770 con 0x7f89b0102250 2024-03-24T10:39:26.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.392+0000 7f89acff9700 1 -- 172.21.15.23:0/3516620796 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f89a0007720 con 0x7f89b0102250 2024-03-24T10:39:26.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.393+0000 7f89acff9700 1 --2- 172.21.15.23:0/3516620796 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f899c0a21e0 0x7f899c0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:26.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.393+0000 7f89aeffd700 1 --2- 172.21.15.23:0/3516620796 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f899c0a21e0 0x7f899c0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:26.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.393+0000 7f89acff9700 1 -- 172.21.15.23:0/3516620796 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f89a00ce1a0 con 0x7f89b0102250 2024-03-24T10:39:26.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.393+0000 7f89aeffd700 1 --2- 172.21.15.23:0/3516620796 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f899c0a21e0 0x7f899c0a46a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f899800b2e0 tx=0x7f8998005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:26.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.396+0000 7f89acff9700 1 -- 172.21.15.23:0/3516620796 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f89a0096f90 con 0x7f89b0102250 2024-03-24T10:39:26.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.594+0000 7f89b5e32700 1 -- 172.21.15.23:0/3516620796 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f89b0199a20 con 0x7f899c0a21e0 2024-03-24T10:39:26.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.601+0000 7f89acff9700 1 -- 172.21.15.23:0/3516620796 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258727 (secure 0 0 0) 0x7f89b0199a20 con 0x7f899c0a21e0 2024-03-24T10:39:26.605 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:26.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.604+0000 7f89b5e32700 1 -- 172.21.15.23:0/3516620796 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f899c0a21e0 msgr2=0x7f899c0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:26.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.604+0000 7f89b5e32700 1 --2- 172.21.15.23:0/3516620796 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f899c0a21e0 0x7f899c0a46a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f899800b2e0 tx=0x7f8998005960 comp rx=0 tx=0).stop 2024-03-24T10:39:26.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.604+0000 7f89b5e32700 1 -- 172.21.15.23:0/3516620796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89b0102250 msgr2=0x7f89b01986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:26.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.604+0000 7f89b5e32700 1 --2- 172.21.15.23:0/3516620796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89b0102250 0x7f89b01986c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f89a000ef10 tx=0x7f89a000c3a0 comp rx=0 tx=0).stop 2024-03-24T10:39:26.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.604+0000 7f89b5e32700 1 -- 172.21.15.23:0/3516620796 shutdown_connections 2024-03-24T10:39:26.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.604+0000 7f89b5e32700 1 --2- 172.21.15.23:0/3516620796 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f899c0a21e0 0x7f899c0a46a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:26.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.604+0000 7f89b5e32700 1 --2- 172.21.15.23:0/3516620796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89b0102250 0x7f89b01986c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:26.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.604+0000 7f89b5e32700 1 --2- 172.21.15.23:0/3516620796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89b0108300 0x7f89b019cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:26.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.604+0000 7f89b5e32700 1 --2- 172.21.15.23:0/3516620796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89b0102c10 0x7f89b0198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:26.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.604+0000 7f89b5e32700 1 -- 172.21.15.23:0/3516620796 >> 172.21.15.23:0/3516620796 conn(0x7f89b0079330 msgr2=0x7f89b0100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:26.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.605+0000 7f89b5e32700 1 -- 172.21.15.23:0/3516620796 shutdown_connections 2024-03-24T10:39:26.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:26.605+0000 7f89b5e32700 1 -- 172.21.15.23:0/3516620796 wait complete. 2024-03-24T10:39:26.608 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-24T10:39:26.969 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:26 smithi023 ceph-mon[32466]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 88 B/s, 1 objects/s recovering 2024-03-24T10:39:27.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:26 smithi117 ceph-mon[36368]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 88 B/s, 1 objects/s recovering 2024-03-24T10:39:27.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:26 smithi116 ceph-mon[36248]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 88 B/s, 1 objects/s recovering 2024-03-24T10:39:27.281 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-03-24T10:39:24.811407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654068,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":561,"num_read_kb":398,"num_write":338,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1181547,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":289,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":366636,"kb_used_data":13260,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575188,"statfs":{"total":1151940427776,"available":1151564992512,"internally_reserved":0,"allocated":13578240,"data_stored":7431960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.116076"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698168+0000","last_change":"2024-03-24T10:38:10.882385+0000","last_active":"2024-03-24T10:39:16.698168+0000","last_peered":"2024-03-24T10:39:16.698168+0000","last_clean":"2024-03-24T10:39:16.698168+0000","last_became_active":"2024-03-24T10:38:10.882062+0000","last_became_peered":"2024-03-24T10:38:10.882062+0000","last_unstale":"2024-03-24T10:39:16.698168+0000","last_undegraded":"2024-03-24T10:39:16.698168+0000","last_fullsized":"2024-03-24T10:39:16.698168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:25.151062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695985+0000","last_change":"2024-03-24T10:38:10.878001+0000","last_active":"2024-03-24T10:39:16.695985+0000","last_peered":"2024-03-24T10:39:16.695985+0000","last_clean":"2024-03-24T10:39:16.695985+0000","last_became_active":"2024-03-24T10:38:10.877661+0000","last_became_peered":"2024-03-24T10:38:10.877661+0000","last_unstale":"2024-03-24T10:39:16.695985+0000","last_undegraded":"2024-03-24T10:39:16.695985+0000","last_fullsized":"2024-03-24T10:39:16.695985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:40:19.062969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696467+0000","last_change":"2024-03-24T10:38:10.891809+0000","last_active":"2024-03-24T10:39:16.696467+0000","last_peered":"2024-03-24T10:39:16.696467+0000","last_clean":"2024-03-24T10:39:16.696467+0000","last_became_active":"2024-03-24T10:38:10.890354+0000","last_became_peered":"2024-03-24T10:38:10.890354+0000","last_unstale":"2024-03-24T10:39:16.696467+0000","last_undegraded":"2024-03-24T10:39:16.696467+0000","last_fullsized":"2024-03-24T10:39:16.696467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:59:26.037914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.905236+0000","last_change":"2024-03-24T10:38:10.885097+0000","last_active":"2024-03-24T10:39:16.905236+0000","last_peered":"2024-03-24T10:39:16.905236+0000","last_clean":"2024-03-24T10:39:16.905236+0000","last_became_active":"2024-03-24T10:38:10.884842+0000","last_became_peered":"2024-03-24T10:38:10.884842+0000","last_unstale":"2024-03-24T10:39:16.905236+0000","last_undegraded":"2024-03-24T10:39:16.905236+0000","last_fullsized":"2024-03-24T10:39:16.905236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:39:44.085699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695563+0000","last_change":"2024-03-24T10:38:10.877653+0000","last_active":"2024-03-24T10:39:16.695563+0000","last_peered":"2024-03-24T10:39:16.695563+0000","last_clean":"2024-03-24T10:39:16.695563+0000","last_became_active":"2024-03-24T10:38:10.877249+0000","last_became_peered":"2024-03-24T10:38:10.877249+0000","last_unstale":"2024-03-24T10:39:16.695563+0000","last_undegraded":"2024-03-24T10:39:16.695563+0000","last_fullsized":"2024-03-24T10:39:16.695563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:17.504660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.725073+0000","last_change":"2024-03-24T10:38:10.876584+0000","last_active":"2024-03-24T10:39:16.725073+0000","last_peered":"2024-03-24T10:39:16.725073+0000","last_clean":"2024-03-24T10:39:16.725073+0000","last_became_active":"2024-03-24T10:38:10.876173+0000","last_became_peered":"2024-03-24T10:38:10.876173+0000","last_unstale":"2024-03-24T10:39:16.725073+0000","last_undegraded":"2024-03-24T10:39:16.725073+0000","last_fullsized":"2024-03-24T10:39:16.725073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:33.899155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695666+0000","last_change":"2024-03-24T10:38:10.874206+0000","last_active":"2024-03-24T10:39:16.695666+0000","last_peered":"2024-03-24T10:39:16.695666+0000","last_clean":"2024-03-24T10:39:16.695666+0000","last_became_active":"2024-03-24T10:38:10.873779+0000","last_became_peered":"2024-03-24T10:38:10.873779+0000","last_unstale":"2024-03-24T10:39:16.695666+0000","last_undegraded":"2024-03-24T10:39:16.695666+0000","last_fullsized":"2024-03-24T10:39:16.695666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:08:19.928998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699354+0000","last_change":"2024-03-24T10:38:10.884219+0000","last_active":"2024-03-24T10:39:16.699354+0000","last_peered":"2024-03-24T10:39:16.699354+0000","last_clean":"2024-03-24T10:39:16.699354+0000","last_became_active":"2024-03-24T10:38:10.883863+0000","last_became_peered":"2024-03-24T10:38:10.883863+0000","last_unstale":"2024-03-24T10:39:16.699354+0000","last_undegraded":"2024-03-24T10:39:16.699354+0000","last_fullsized":"2024-03-24T10:39:16.699354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:26:34.932083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698407+0000","last_change":"2024-03-24T10:38:10.881297+0000","last_active":"2024-03-24T10:39:16.698407+0000","last_peered":"2024-03-24T10:39:16.698407+0000","last_clean":"2024-03-24T10:39:16.698407+0000","last_became_active":"2024-03-24T10:38:10.879871+0000","last_became_peered":"2024-03-24T10:38:10.879871+0000","last_unstale":"2024-03-24T10:39:16.698407+0000","last_undegraded":"2024-03-24T10:39:16.698407+0000","last_fullsized":"2024-03-24T10:39:16.698407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:05.132788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.725363+0000","last_change":"2024-03-24T10:38:10.876648+0000","last_active":"2024-03-24T10:39:16.725363+0000","last_peered":"2024-03-24T10:39:16.725363+0000","last_clean":"2024-03-24T10:39:16.725363+0000","last_became_active":"2024-03-24T10:38:10.876113+0000","last_became_peered":"2024-03-24T10:38:10.876113+0000","last_unstale":"2024-03-24T10:39:16.725363+0000","last_undegraded":"2024-03-24T10:39:16.725363+0000","last_fullsized":"2024-03-24T10:39:16.725363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:29:18.282797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695901+0000","last_change":"2024-03-24T10:38:10.875836+0000","last_active":"2024-03-24T10:39:16.695901+0000","last_peered":"2024-03-24T10:39:16.695901+0000","last_clean":"2024-03-24T10:39:16.695901+0000","last_became_active":"2024-03-24T10:38:10.875522+0000","last_became_peered":"2024-03-24T10:38:10.875522+0000","last_unstale":"2024-03-24T10:39:16.695901+0000","last_undegraded":"2024-03-24T10:39:16.695901+0000","last_fullsized":"2024-03-24T10:39:16.695901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:41.282123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.905099+0000","last_change":"2024-03-24T10:38:10.884567+0000","last_active":"2024-03-24T10:39:16.905099+0000","last_peered":"2024-03-24T10:39:16.905099+0000","last_clean":"2024-03-24T10:39:16.905099+0000","last_became_active":"2024-03-24T10:38:10.884195+0000","last_became_peered":"2024-03-24T10:38:10.884195+0000","last_unstale":"2024-03-24T10:39:16.905099+0000","last_undegraded":"2024-03-24T10:39:16.905099+0000","last_fullsized":"2024-03-24T10:39:16.905099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:29.286631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.726561+0000","last_change":"2024-03-24T10:39:16.726561+0000","last_active":"2024-03-24T10:39:16.726561+0000","last_peered":"2024-03-24T10:39:16.726561+0000","last_clean":"2024-03-24T10:39:16.726561+0000","last_became_active":"2024-03-24T10:39:16.726088+0000","last_became_peered":"2024-03-24T10:39:16.726088+0000","last_unstale":"2024-03-24T10:39:16.726561+0000","last_undegraded":"2024-03-24T10:39:16.726561+0000","last_fullsized":"2024-03-24T10:39:16.726561+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:08:46.638610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695788+0000","last_change":"2024-03-24T10:38:10.879942+0000","last_active":"2024-03-24T10:39:16.695788+0000","last_peered":"2024-03-24T10:39:16.695788+0000","last_clean":"2024-03-24T10:39:16.695788+0000","last_became_active":"2024-03-24T10:38:10.879396+0000","last_became_peered":"2024-03-24T10:38:10.879396+0000","last_unstale":"2024-03-24T10:39:16.695788+0000","last_undegraded":"2024-03-24T10:39:16.695788+0000","last_fullsized":"2024-03-24T10:39:16.695788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:41.477988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695815+0000","last_change":"2024-03-24T10:38:10.886667+0000","last_active":"2024-03-24T10:39:16.695815+0000","last_peered":"2024-03-24T10:39:16.695815+0000","last_clean":"2024-03-24T10:39:16.695815+0000","last_became_active":"2024-03-24T10:38:10.886293+0000","last_became_peered":"2024-03-24T10:38:10.886293+0000","last_unstale":"2024-03-24T10:39:16.695815+0000","last_undegraded":"2024-03-24T10:39:16.695815+0000","last_fullsized":"2024-03-24T10:39:16.695815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:43:43.414573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697524+0000","last_change":"2024-03-24T10:38:10.883671+0000","last_active":"2024-03-24T10:39:16.697524+0000","last_peered":"2024-03-24T10:39:16.697524+0000","last_clean":"2024-03-24T10:39:16.697524+0000","last_became_active":"2024-03-24T10:38:10.883013+0000","last_became_peered":"2024-03-24T10:38:10.883013+0000","last_unstale":"2024-03-24T10:39:16.697524+0000","last_undegraded":"2024-03-24T10:39:16.697524+0000","last_fullsized":"2024-03-24T10:39:16.697524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:33.936543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698261+0000","last_change":"2024-03-24T10:38:10.887286+0000","last_active":"2024-03-24T10:39:16.698261+0000","last_peered":"2024-03-24T10:39:16.698261+0000","last_clean":"2024-03-24T10:39:16.698261+0000","last_became_active":"2024-03-24T10:38:10.886918+0000","last_became_peered":"2024-03-24T10:38:10.886918+0000","last_unstale":"2024-03-24T10:39:16.698261+0000","last_undegraded":"2024-03-24T10:39:16.698261+0000","last_fullsized":"2024-03-24T10:39:16.698261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:40.266589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698568+0000","last_change":"2024-03-24T10:38:10.874476+0000","last_active":"2024-03-24T10:39:16.698568+0000","last_peered":"2024-03-24T10:39:16.698568+0000","last_clean":"2024-03-24T10:39:16.698568+0000","last_became_active":"2024-03-24T10:38:10.874123+0000","last_became_peered":"2024-03-24T10:38:10.874123+0000","last_unstale":"2024-03-24T10:39:16.698568+0000","last_undegraded":"2024-03-24T10:39:16.698568+0000","last_fullsized":"2024-03-24T10:39:16.698568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:29.759320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699586+0000","last_change":"2024-03-24T10:38:10.878758+0000","last_active":"2024-03-24T10:39:16.699586+0000","last_peered":"2024-03-24T10:39:16.699586+0000","last_clean":"2024-03-24T10:39:16.699586+0000","last_became_active":"2024-03-24T10:38:10.878424+0000","last_became_peered":"2024-03-24T10:38:10.878424+0000","last_unstale":"2024-03-24T10:39:16.699586+0000","last_undegraded":"2024-03-24T10:39:16.699586+0000","last_fullsized":"2024-03-24T10:39:16.699586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:15:19.273108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.709304+0000","last_change":"2024-03-24T10:39:16.709304+0000","last_active":"2024-03-24T10:39:16.709304+0000","last_peered":"2024-03-24T10:39:16.709304+0000","last_clean":"2024-03-24T10:39:16.709304+0000","last_became_active":"2024-03-24T10:39:16.700325+0000","last_became_peered":"2024-03-24T10:39:16.700325+0000","last_unstale":"2024-03-24T10:39:16.709304+0000","last_undegraded":"2024-03-24T10:39:16.709304+0000","last_fullsized":"2024-03-24T10:39:16.709304+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:57.633288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699762+0000","last_change":"2024-03-24T10:38:10.885198+0000","last_active":"2024-03-24T10:39:16.699762+0000","last_peered":"2024-03-24T10:39:16.699762+0000","last_clean":"2024-03-24T10:39:16.699762+0000","last_became_active":"2024-03-24T10:38:10.884123+0000","last_became_peered":"2024-03-24T10:38:10.884123+0000","last_unstale":"2024-03-24T10:39:16.699762+0000","last_undegraded":"2024-03-24T10:39:16.699762+0000","last_fullsized":"2024-03-24T10:39:16.699762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:37:12.510671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699892+0000","last_change":"2024-03-24T10:38:09.866223+0000","last_active":"2024-03-24T10:39:16.699892+0000","last_peered":"2024-03-24T10:39:16.699892+0000","last_clean":"2024-03-24T10:39:16.699892+0000","last_became_active":"2024-03-24T10:38:09.865711+0000","last_became_peered":"2024-03-24T10:38:09.865711+0000","last_unstale":"2024-03-24T10:39:16.699892+0000","last_undegraded":"2024-03-24T10:39:16.699892+0000","last_fullsized":"2024-03-24T10:39:16.699892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:26:08.380749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696084+0000","last_change":"2024-03-24T10:38:10.890557+0000","last_active":"2024-03-24T10:39:16.696084+0000","last_peered":"2024-03-24T10:39:16.696084+0000","last_clean":"2024-03-24T10:39:16.696084+0000","last_became_active":"2024-03-24T10:38:10.889883+0000","last_became_peered":"2024-03-24T10:38:10.889883+0000","last_unstale":"2024-03-24T10:39:16.696084+0000","last_undegraded":"2024-03-24T10:39:16.696084+0000","last_fullsized":"2024-03-24T10:39:16.696084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:34.731585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695880+0000","last_change":"2024-03-24T10:38:09.870253+0000","last_active":"2024-03-24T10:39:16.695880+0000","last_peered":"2024-03-24T10:39:16.695880+0000","last_clean":"2024-03-24T10:39:16.695880+0000","last_became_active":"2024-03-24T10:38:09.869932+0000","last_became_peered":"2024-03-24T10:38:09.869932+0000","last_unstale":"2024-03-24T10:39:16.695880+0000","last_undegraded":"2024-03-24T10:39:16.695880+0000","last_fullsized":"2024-03-24T10:39:16.695880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:38:43.626390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697788+0000","last_change":"2024-03-24T10:38:10.881705+0000","last_active":"2024-03-24T10:39:16.697788+0000","last_peered":"2024-03-24T10:39:16.697788+0000","last_clean":"2024-03-24T10:39:16.697788+0000","last_became_active":"2024-03-24T10:38:10.881323+0000","last_became_peered":"2024-03-24T10:38:10.881323+0000","last_unstale":"2024-03-24T10:39:16.697788+0000","last_undegraded":"2024-03-24T10:39:16.697788+0000","last_fullsized":"2024-03-24T10:39:16.697788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:25.861184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696090+0000","last_change":"2024-03-24T10:38:09.870481+0000","last_active":"2024-03-24T10:39:16.696090+0000","last_peered":"2024-03-24T10:39:16.696090+0000","last_clean":"2024-03-24T10:39:16.696090+0000","last_became_active":"2024-03-24T10:38:09.870117+0000","last_became_peered":"2024-03-24T10:38:09.870117+0000","last_unstale":"2024-03-24T10:39:16.696090+0000","last_undegraded":"2024-03-24T10:39:16.696090+0000","last_fullsized":"2024-03-24T10:39:16.696090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:37:51.871068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.907072+0000","last_change":"2024-03-24T10:39:16.906560+0000","last_active":"2024-03-24T10:39:16.907072+0000","last_peered":"2024-03-24T10:39:16.907072+0000","last_clean":"2024-03-24T10:39:16.907072+0000","last_became_active":"2024-03-24T10:39:16.906009+0000","last_became_peered":"2024-03-24T10:39:16.906009+0000","last_unstale":"2024-03-24T10:39:16.907072+0000","last_undegraded":"2024-03-24T10:39:16.907072+0000","last_fullsized":"2024-03-24T10:39:16.907072+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:45:56.513953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":92,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:19.984716+0000","last_change":"2024-03-24T10:38:09.864160+0000","last_active":"2024-03-24T10:39:19.984716+0000","last_peered":"2024-03-24T10:39:19.984716+0000","last_clean":"2024-03-24T10:39:19.984716+0000","last_became_active":"2024-03-24T10:38:09.862066+0000","last_became_peered":"2024-03-24T10:38:09.862066+0000","last_unstale":"2024-03-24T10:39:19.984716+0000","last_undegraded":"2024-03-24T10:39:19.984716+0000","last_fullsized":"2024-03-24T10:39:19.984716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:59.003931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695578+0000","last_change":"2024-03-24T10:38:10.874433+0000","last_active":"2024-03-24T10:39:16.695578+0000","last_peered":"2024-03-24T10:39:16.695578+0000","last_clean":"2024-03-24T10:39:16.695578+0000","last_became_active":"2024-03-24T10:38:10.874090+0000","last_became_peered":"2024-03-24T10:38:10.874090+0000","last_unstale":"2024-03-24T10:39:16.695578+0000","last_undegraded":"2024-03-24T10:39:16.695578+0000","last_fullsized":"2024-03-24T10:39:16.695578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:25.979419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.725049+0000","last_change":"2024-03-24T10:38:09.863028+0000","last_active":"2024-03-24T10:39:16.725049+0000","last_peered":"2024-03-24T10:39:16.725049+0000","last_clean":"2024-03-24T10:39:16.725049+0000","last_became_active":"2024-03-24T10:38:09.862682+0000","last_became_peered":"2024-03-24T10:38:09.862682+0000","last_unstale":"2024-03-24T10:39:16.725049+0000","last_undegraded":"2024-03-24T10:39:16.725049+0000","last_fullsized":"2024-03-24T10:39:16.725049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:14.962115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699089+0000","last_change":"2024-03-24T10:38:10.885950+0000","last_active":"2024-03-24T10:39:16.699089+0000","last_peered":"2024-03-24T10:39:16.699089+0000","last_clean":"2024-03-24T10:39:16.699089+0000","last_became_active":"2024-03-24T10:38:10.885685+0000","last_became_peered":"2024-03-24T10:38:10.885685+0000","last_unstale":"2024-03-24T10:39:16.699089+0000","last_undegraded":"2024-03-24T10:39:16.699089+0000","last_fullsized":"2024-03-24T10:39:16.699089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:42:09.912743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.700938+0000","last_change":"2024-03-24T10:38:09.860376+0000","last_active":"2024-03-24T10:39:16.700938+0000","last_peered":"2024-03-24T10:39:16.700938+0000","last_clean":"2024-03-24T10:39:16.700938+0000","last_became_active":"2024-03-24T10:38:09.860062+0000","last_became_peered":"2024-03-24T10:38:09.860062+0000","last_unstale":"2024-03-24T10:39:16.700938+0000","last_undegraded":"2024-03-24T10:39:16.700938+0000","last_fullsized":"2024-03-24T10:39:16.700938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:33.387953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699208+0000","last_change":"2024-03-24T10:38:10.883843+0000","last_active":"2024-03-24T10:39:16.699208+0000","last_peered":"2024-03-24T10:39:16.699208+0000","last_clean":"2024-03-24T10:39:16.699208+0000","last_became_active":"2024-03-24T10:38:10.883170+0000","last_became_peered":"2024-03-24T10:38:10.883170+0000","last_unstale":"2024-03-24T10:39:16.699208+0000","last_undegraded":"2024-03-24T10:39:16.699208+0000","last_fullsized":"2024-03-24T10:39:16.699208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:28:16.798546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698585+0000","last_change":"2024-03-24T10:38:09.862299+0000","last_active":"2024-03-24T10:39:16.698585+0000","last_peered":"2024-03-24T10:39:16.698585+0000","last_clean":"2024-03-24T10:39:16.698585+0000","last_became_active":"2024-03-24T10:38:09.861895+0000","last_became_peered":"2024-03-24T10:38:09.861895+0000","last_unstale":"2024-03-24T10:39:16.698585+0000","last_undegraded":"2024-03-24T10:39:16.698585+0000","last_fullsized":"2024-03-24T10:39:16.698585+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:13:19.869511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698225+0000","last_change":"2024-03-24T10:38:10.880507+0000","last_active":"2024-03-24T10:39:16.698225+0000","last_peered":"2024-03-24T10:39:16.698225+0000","last_clean":"2024-03-24T10:39:16.698225+0000","last_became_active":"2024-03-24T10:38:10.879790+0000","last_became_peered":"2024-03-24T10:38:10.879790+0000","last_unstale":"2024-03-24T10:39:16.698225+0000","last_undegraded":"2024-03-24T10:39:16.698225+0000","last_fullsized":"2024-03-24T10:39:16.698225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:11:02.650420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":88,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:19.984086+0000","last_change":"2024-03-24T10:38:09.860186+0000","last_active":"2024-03-24T10:39:19.984086+0000","last_peered":"2024-03-24T10:39:19.984086+0000","last_clean":"2024-03-24T10:39:19.984086+0000","last_became_active":"2024-03-24T10:38:09.859834+0000","last_became_peered":"2024-03-24T10:38:09.859834+0000","last_unstale":"2024-03-24T10:39:19.984086+0000","last_undegraded":"2024-03-24T10:39:19.984086+0000","last_fullsized":"2024-03-24T10:39:19.984086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:33:48.029053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698120+0000","last_change":"2024-03-24T10:38:10.875634+0000","last_active":"2024-03-24T10:39:16.698120+0000","last_peered":"2024-03-24T10:39:16.698120+0000","last_clean":"2024-03-24T10:39:16.698120+0000","last_became_active":"2024-03-24T10:38:10.875101+0000","last_became_peered":"2024-03-24T10:38:10.875101+0000","last_unstale":"2024-03-24T10:39:16.698120+0000","last_undegraded":"2024-03-24T10:39:16.698120+0000","last_fullsized":"2024-03-24T10:39:16.698120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:32:58.165087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695826+0000","last_change":"2024-03-24T10:38:09.860412+0000","last_active":"2024-03-24T10:39:16.695826+0000","last_peered":"2024-03-24T10:39:16.695826+0000","last_clean":"2024-03-24T10:39:16.695826+0000","last_became_active":"2024-03-24T10:38:09.860084+0000","last_became_peered":"2024-03-24T10:38:09.860084+0000","last_unstale":"2024-03-24T10:39:16.695826+0000","last_undegraded":"2024-03-24T10:39:16.695826+0000","last_fullsized":"2024-03-24T10:39:16.695826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:44.585482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.703566+0000","last_change":"2024-03-24T10:39:16.703566+0000","last_active":"2024-03-24T10:39:16.703566+0000","last_peered":"2024-03-24T10:39:16.703566+0000","last_clean":"2024-03-24T10:39:16.703566+0000","last_became_active":"2024-03-24T10:39:16.703213+0000","last_became_peered":"2024-03-24T10:39:16.703213+0000","last_unstale":"2024-03-24T10:39:16.703566+0000","last_undegraded":"2024-03-24T10:39:16.703566+0000","last_fullsized":"2024-03-24T10:39:16.703566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:10.888996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698568+0000","last_change":"2024-03-24T10:38:09.865191+0000","last_active":"2024-03-24T10:39:16.698568+0000","last_peered":"2024-03-24T10:39:16.698568+0000","last_clean":"2024-03-24T10:39:16.698568+0000","last_became_active":"2024-03-24T10:38:09.864470+0000","last_became_peered":"2024-03-24T10:38:09.864470+0000","last_unstale":"2024-03-24T10:39:16.698568+0000","last_undegraded":"2024-03-24T10:39:16.698568+0000","last_fullsized":"2024-03-24T10:39:16.698568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:45:00.266470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698350+0000","last_change":"2024-03-24T10:38:10.889668+0000","last_active":"2024-03-24T10:39:16.698350+0000","last_peered":"2024-03-24T10:39:16.698350+0000","last_clean":"2024-03-24T10:39:16.698350+0000","last_became_active":"2024-03-24T10:38:10.889364+0000","last_became_peered":"2024-03-24T10:38:10.889364+0000","last_unstale":"2024-03-24T10:39:16.698350+0000","last_undegraded":"2024-03-24T10:39:16.698350+0000","last_fullsized":"2024-03-24T10:39:16.698350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:57.702719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'27","reported_seq":185,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:20.985182+0000","last_change":"2024-03-24T10:38:09.861809+0000","last_active":"2024-03-24T10:39:20.985182+0000","last_peered":"2024-03-24T10:39:20.985182+0000","last_clean":"2024-03-24T10:39:20.985182+0000","last_became_active":"2024-03-24T10:38:09.861444+0000","last_became_peered":"2024-03-24T10:38:09.861444+0000","last_unstale":"2024-03-24T10:39:20.985182+0000","last_undegraded":"2024-03-24T10:39:20.985182+0000","last_fullsized":"2024-03-24T10:39:20.985182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:19.652188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59465,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":139,"num_read_kb":127,"num_write":28,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695671+0000","last_change":"2024-03-24T10:38:10.883090+0000","last_active":"2024-03-24T10:39:16.695671+0000","last_peered":"2024-03-24T10:39:16.695671+0000","last_clean":"2024-03-24T10:39:16.695671+0000","last_became_active":"2024-03-24T10:38:10.880507+0000","last_became_peered":"2024-03-24T10:38:10.880507+0000","last_unstale":"2024-03-24T10:39:16.695671+0000","last_undegraded":"2024-03-24T10:39:16.695671+0000","last_fullsized":"2024-03-24T10:39:16.695671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:01.422699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697835+0000","last_change":"2024-03-24T10:38:09.868072+0000","last_active":"2024-03-24T10:39:16.697835+0000","last_peered":"2024-03-24T10:39:16.697835+0000","last_clean":"2024-03-24T10:39:16.697835+0000","last_became_active":"2024-03-24T10:38:09.867704+0000","last_became_peered":"2024-03-24T10:38:09.867704+0000","last_unstale":"2024-03-24T10:39:16.697835+0000","last_undegraded":"2024-03-24T10:39:16.697835+0000","last_fullsized":"2024-03-24T10:39:16.697835+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:09:28.726969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.701952+0000","last_change":"2024-03-24T10:38:10.881882+0000","last_active":"2024-03-24T10:39:16.701952+0000","last_peered":"2024-03-24T10:39:16.701952+0000","last_clean":"2024-03-24T10:39:16.701952+0000","last_became_active":"2024-03-24T10:38:10.881495+0000","last_became_peered":"2024-03-24T10:38:10.881495+0000","last_unstale":"2024-03-24T10:39:16.701952+0000","last_undegraded":"2024-03-24T10:39:16.701952+0000","last_fullsized":"2024-03-24T10:39:16.701952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:31:43.392952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698414+0000","last_change":"2024-03-24T10:38:09.867265+0000","last_active":"2024-03-24T10:39:16.698414+0000","last_peered":"2024-03-24T10:39:16.698414+0000","last_clean":"2024-03-24T10:39:16.698414+0000","last_became_active":"2024-03-24T10:38:09.866669+0000","last_became_peered":"2024-03-24T10:38:09.866669+0000","last_unstale":"2024-03-24T10:39:16.698414+0000","last_undegraded":"2024-03-24T10:39:16.698414+0000","last_fullsized":"2024-03-24T10:39:16.698414+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:28.230967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696333+0000","last_change":"2024-03-24T10:38:10.886634+0000","last_active":"2024-03-24T10:39:16.696333+0000","last_peered":"2024-03-24T10:39:16.696333+0000","last_clean":"2024-03-24T10:39:16.696333+0000","last_became_active":"2024-03-24T10:38:10.886207+0000","last_became_peered":"2024-03-24T10:38:10.886207+0000","last_unstale":"2024-03-24T10:39:16.696333+0000","last_undegraded":"2024-03-24T10:39:16.696333+0000","last_fullsized":"2024-03-24T10:39:16.696333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:43:41.601693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":94,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:21.988902+0000","last_change":"2024-03-24T10:38:09.863262+0000","last_active":"2024-03-24T10:39:21.988902+0000","last_peered":"2024-03-24T10:39:21.988902+0000","last_clean":"2024-03-24T10:39:21.988902+0000","last_became_active":"2024-03-24T10:38:09.862665+0000","last_became_peered":"2024-03-24T10:38:09.862665+0000","last_unstale":"2024-03-24T10:39:21.988902+0000","last_undegraded":"2024-03-24T10:39:21.988902+0000","last_fullsized":"2024-03-24T10:39:21.988902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:12:29.117592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695642+0000","last_change":"2024-03-24T10:38:10.880793+0000","last_active":"2024-03-24T10:39:16.695642+0000","last_peered":"2024-03-24T10:39:16.695642+0000","last_clean":"2024-03-24T10:39:16.695642+0000","last_became_active":"2024-03-24T10:38:10.880449+0000","last_became_peered":"2024-03-24T10:38:10.880449+0000","last_unstale":"2024-03-24T10:39:16.695642+0000","last_undegraded":"2024-03-24T10:39:16.695642+0000","last_fullsized":"2024-03-24T10:39:16.695642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:32:57.184014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.915438+0000","last_change":"2024-03-24T10:39:16.915438+0000","last_active":"2024-03-24T10:39:16.915438+0000","last_peered":"2024-03-24T10:39:16.915438+0000","last_clean":"2024-03-24T10:39:16.915438+0000","last_became_active":"2024-03-24T10:39:16.906155+0000","last_became_peered":"2024-03-24T10:39:16.906155+0000","last_unstale":"2024-03-24T10:39:16.915438+0000","last_undegraded":"2024-03-24T10:39:16.915438+0000","last_fullsized":"2024-03-24T10:39:16.915438+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:36:12.356069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698214+0000","last_change":"2024-03-24T10:38:10.881952+0000","last_active":"2024-03-24T10:39:16.698214+0000","last_peered":"2024-03-24T10:39:16.698214+0000","last_clean":"2024-03-24T10:39:16.698214+0000","last_became_active":"2024-03-24T10:38:10.881628+0000","last_became_peered":"2024-03-24T10:38:10.881628+0000","last_unstale":"2024-03-24T10:39:16.698214+0000","last_undegraded":"2024-03-24T10:39:16.698214+0000","last_fullsized":"2024-03-24T10:39:16.698214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:24.777793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695669+0000","last_change":"2024-03-24T10:38:09.868572+0000","last_active":"2024-03-24T10:39:16.695669+0000","last_peered":"2024-03-24T10:39:16.695669+0000","last_clean":"2024-03-24T10:39:16.695669+0000","last_became_active":"2024-03-24T10:38:09.867702+0000","last_became_peered":"2024-03-24T10:38:09.867702+0000","last_unstale":"2024-03-24T10:39:16.695669+0000","last_undegraded":"2024-03-24T10:39:16.695669+0000","last_fullsized":"2024-03-24T10:39:16.695669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:27.890114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697936+0000","last_change":"2024-03-24T10:38:10.875342+0000","last_active":"2024-03-24T10:39:16.697936+0000","last_peered":"2024-03-24T10:39:16.697936+0000","last_clean":"2024-03-24T10:39:16.697936+0000","last_became_active":"2024-03-24T10:38:10.874933+0000","last_became_peered":"2024-03-24T10:38:10.874933+0000","last_unstale":"2024-03-24T10:39:16.697936+0000","last_undegraded":"2024-03-24T10:39:16.697936+0000","last_fullsized":"2024-03-24T10:39:16.697936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:56:09.139478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697917+0000","last_change":"2024-03-24T10:38:09.857867+0000","last_active":"2024-03-24T10:39:16.697917+0000","last_peered":"2024-03-24T10:39:16.697917+0000","last_clean":"2024-03-24T10:39:16.697917+0000","last_became_active":"2024-03-24T10:38:09.857525+0000","last_became_peered":"2024-03-24T10:38:09.857525+0000","last_unstale":"2024-03-24T10:39:16.697917+0000","last_undegraded":"2024-03-24T10:39:16.697917+0000","last_fullsized":"2024-03-24T10:39:16.697917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:37.320245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.725224+0000","last_change":"2024-03-24T10:38:10.878980+0000","last_active":"2024-03-24T10:39:16.725224+0000","last_peered":"2024-03-24T10:39:16.725224+0000","last_clean":"2024-03-24T10:39:16.725224+0000","last_became_active":"2024-03-24T10:38:10.878648+0000","last_became_peered":"2024-03-24T10:38:10.878648+0000","last_unstale":"2024-03-24T10:39:16.725224+0000","last_undegraded":"2024-03-24T10:39:16.725224+0000","last_fullsized":"2024-03-24T10:39:16.725224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:49.988403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695848+0000","last_change":"2024-03-24T10:38:09.863457+0000","last_active":"2024-03-24T10:39:16.695848+0000","last_peered":"2024-03-24T10:39:16.695848+0000","last_clean":"2024-03-24T10:39:16.695848+0000","last_became_active":"2024-03-24T10:38:09.863130+0000","last_became_peered":"2024-03-24T10:38:09.863130+0000","last_unstale":"2024-03-24T10:39:16.695848+0000","last_undegraded":"2024-03-24T10:39:16.695848+0000","last_fullsized":"2024-03-24T10:39:16.695848+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:16.528703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.725232+0000","last_change":"2024-03-24T10:38:10.887014+0000","last_active":"2024-03-24T10:39:16.725232+0000","last_peered":"2024-03-24T10:39:16.725232+0000","last_clean":"2024-03-24T10:39:16.725232+0000","last_became_active":"2024-03-24T10:38:10.886698+0000","last_became_peered":"2024-03-24T10:38:10.886698+0000","last_unstale":"2024-03-24T10:39:16.725232+0000","last_undegraded":"2024-03-24T10:39:16.725232+0000","last_fullsized":"2024-03-24T10:39:16.725232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:47:08.571886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698893+0000","last_change":"2024-03-24T10:38:10.874811+0000","last_active":"2024-03-24T10:39:16.698893+0000","last_peered":"2024-03-24T10:39:16.698893+0000","last_clean":"2024-03-24T10:39:16.698893+0000","last_became_active":"2024-03-24T10:38:10.874483+0000","last_became_peered":"2024-03-24T10:38:10.874483+0000","last_unstale":"2024-03-24T10:39:16.698893+0000","last_undegraded":"2024-03-24T10:39:16.698893+0000","last_fullsized":"2024-03-24T10:39:16.698893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:57.382174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":90,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:20.987529+0000","last_change":"2024-03-24T10:38:09.865614+0000","last_active":"2024-03-24T10:39:20.987529+0000","last_peered":"2024-03-24T10:39:20.987529+0000","last_clean":"2024-03-24T10:39:20.987529+0000","last_became_active":"2024-03-24T10:38:09.865317+0000","last_became_peered":"2024-03-24T10:38:09.865317+0000","last_unstale":"2024-03-24T10:39:20.987529+0000","last_undegraded":"2024-03-24T10:39:20.987529+0000","last_fullsized":"2024-03-24T10:39:20.987529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:21:14.968552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":67,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.700602+0000","last_change":"2024-03-24T10:38:10.884847+0000","last_active":"2024-03-24T10:39:16.700602+0000","last_peered":"2024-03-24T10:39:16.700602+0000","last_clean":"2024-03-24T10:39:16.700602+0000","last_became_active":"2024-03-24T10:38:10.884526+0000","last_became_peered":"2024-03-24T10:38:10.884526+0000","last_unstale":"2024-03-24T10:39:16.700602+0000","last_undegraded":"2024-03-24T10:39:16.700602+0000","last_fullsized":"2024-03-24T10:39:16.700602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:01:49.364083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.905135+0000","last_change":"2024-03-24T10:38:09.858893+0000","last_active":"2024-03-24T10:39:16.905135+0000","last_peered":"2024-03-24T10:39:16.905135+0000","last_clean":"2024-03-24T10:39:16.905135+0000","last_became_active":"2024-03-24T10:38:09.858474+0000","last_became_peered":"2024-03-24T10:38:09.858474+0000","last_unstale":"2024-03-24T10:39:16.905135+0000","last_undegraded":"2024-03-24T10:39:16.905135+0000","last_fullsized":"2024-03-24T10:39:16.905135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:03:36.709498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":351,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695719+0000","last_change":"2024-03-24T10:36:59.100728+0000","last_active":"2024-03-24T10:39:16.695719+0000","last_peered":"2024-03-24T10:39:16.695719+0000","last_clean":"2024-03-24T10:39:16.695719+0000","last_became_active":"2024-03-24T10:36:59.100159+0000","last_became_peered":"2024-03-24T10:36:59.100159+0000","last_unstale":"2024-03-24T10:39:16.695719+0000","last_undegraded":"2024-03-24T10:39:16.695719+0000","last_fullsized":"2024-03-24T10:39:16.695719+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:39.429553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:39.429553+0000","last_clean_scrub_stamp":"2024-03-24T10:33:39.429553+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:21:01.470585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.700727+0000","last_change":"2024-03-24T10:38:10.885804+0000","last_active":"2024-03-24T10:39:16.700727+0000","last_peered":"2024-03-24T10:39:16.700727+0000","last_clean":"2024-03-24T10:39:16.700727+0000","last_became_active":"2024-03-24T10:38:10.885447+0000","last_became_peered":"2024-03-24T10:38:10.885447+0000","last_unstale":"2024-03-24T10:39:16.700727+0000","last_undegraded":"2024-03-24T10:39:16.700727+0000","last_fullsized":"2024-03-24T10:39:16.700727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:57.103205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696024+0000","last_change":"2024-03-24T10:38:09.863187+0000","last_active":"2024-03-24T10:39:16.696024+0000","last_peered":"2024-03-24T10:39:16.696024+0000","last_clean":"2024-03-24T10:39:16.696024+0000","last_became_active":"2024-03-24T10:38:09.862859+0000","last_became_peered":"2024-03-24T10:38:09.862859+0000","last_unstale":"2024-03-24T10:39:16.696024+0000","last_undegraded":"2024-03-24T10:39:16.696024+0000","last_fullsized":"2024-03-24T10:39:16.696024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:21:02.707127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695813+0000","last_change":"2024-03-24T10:38:09.867899+0000","last_active":"2024-03-24T10:39:16.695813+0000","last_peered":"2024-03-24T10:39:16.695813+0000","last_clean":"2024-03-24T10:39:16.695813+0000","last_became_active":"2024-03-24T10:38:09.867563+0000","last_became_peered":"2024-03-24T10:38:09.867563+0000","last_unstale":"2024-03-24T10:39:16.695813+0000","last_undegraded":"2024-03-24T10:39:16.695813+0000","last_fullsized":"2024-03-24T10:39:16.695813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:51.067981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.701363+0000","last_change":"2024-03-24T10:38:10.870825+0000","last_active":"2024-03-24T10:39:16.701363+0000","last_peered":"2024-03-24T10:39:16.701363+0000","last_clean":"2024-03-24T10:39:16.701363+0000","last_became_active":"2024-03-24T10:38:10.870491+0000","last_became_peered":"2024-03-24T10:38:10.870491+0000","last_unstale":"2024-03-24T10:39:16.701363+0000","last_undegraded":"2024-03-24T10:39:16.701363+0000","last_fullsized":"2024-03-24T10:39:16.701363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:30:34.091299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698460+0000","last_change":"2024-03-24T10:38:09.863724+0000","last_active":"2024-03-24T10:39:16.698460+0000","last_peered":"2024-03-24T10:39:16.698460+0000","last_clean":"2024-03-24T10:39:16.698460+0000","last_became_active":"2024-03-24T10:38:09.862900+0000","last_became_peered":"2024-03-24T10:38:09.862900+0000","last_unstale":"2024-03-24T10:39:16.698460+0000","last_undegraded":"2024-03-24T10:39:16.698460+0000","last_fullsized":"2024-03-24T10:39:16.698460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:21.646074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699470+0000","last_change":"2024-03-24T10:38:10.884661+0000","last_active":"2024-03-24T10:39:16.699470+0000","last_peered":"2024-03-24T10:39:16.699470+0000","last_clean":"2024-03-24T10:39:16.699470+0000","last_became_active":"2024-03-24T10:38:10.882901+0000","last_became_peered":"2024-03-24T10:38:10.882901+0000","last_unstale":"2024-03-24T10:39:16.699470+0000","last_undegraded":"2024-03-24T10:39:16.699470+0000","last_fullsized":"2024-03-24T10:39:16.699470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:23:46.366567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.905272+0000","last_change":"2024-03-24T10:38:09.862122+0000","last_active":"2024-03-24T10:39:16.905272+0000","last_peered":"2024-03-24T10:39:16.905272+0000","last_clean":"2024-03-24T10:39:16.905272+0000","last_became_active":"2024-03-24T10:38:09.861765+0000","last_became_peered":"2024-03-24T10:38:09.861765+0000","last_unstale":"2024-03-24T10:39:16.905272+0000","last_undegraded":"2024-03-24T10:39:16.905272+0000","last_fullsized":"2024-03-24T10:39:16.905272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:40:15.148349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696018+0000","last_change":"2024-03-24T10:38:10.886797+0000","last_active":"2024-03-24T10:39:16.696018+0000","last_peered":"2024-03-24T10:39:16.696018+0000","last_clean":"2024-03-24T10:39:16.696018+0000","last_became_active":"2024-03-24T10:38:10.886406+0000","last_became_peered":"2024-03-24T10:38:10.886406+0000","last_unstale":"2024-03-24T10:39:16.696018+0000","last_undegraded":"2024-03-24T10:39:16.696018+0000","last_fullsized":"2024-03-24T10:39:16.696018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:35.763623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.707034+0000","last_change":"2024-03-24T10:39:16.707034+0000","last_active":"2024-03-24T10:39:16.707034+0000","last_peered":"2024-03-24T10:39:16.707034+0000","last_clean":"2024-03-24T10:39:16.707034+0000","last_became_active":"2024-03-24T10:39:16.701995+0000","last_became_peered":"2024-03-24T10:39:16.701995+0000","last_unstale":"2024-03-24T10:39:16.707034+0000","last_undegraded":"2024-03-24T10:39:16.707034+0000","last_fullsized":"2024-03-24T10:39:16.707034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:18:41.362740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695469+0000","last_change":"2024-03-24T10:38:10.873871+0000","last_active":"2024-03-24T10:39:16.695469+0000","last_peered":"2024-03-24T10:39:16.695469+0000","last_clean":"2024-03-24T10:39:16.695469+0000","last_became_active":"2024-03-24T10:38:10.873528+0000","last_became_peered":"2024-03-24T10:38:10.873528+0000","last_unstale":"2024-03-24T10:39:16.695469+0000","last_undegraded":"2024-03-24T10:39:16.695469+0000","last_fullsized":"2024-03-24T10:39:16.695469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:33.372195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699787+0000","last_change":"2024-03-24T10:38:09.861548+0000","last_active":"2024-03-24T10:39:16.699787+0000","last_peered":"2024-03-24T10:39:16.699787+0000","last_clean":"2024-03-24T10:39:16.699787+0000","last_became_active":"2024-03-24T10:38:09.861219+0000","last_became_peered":"2024-03-24T10:38:09.861219+0000","last_unstale":"2024-03-24T10:39:16.699787+0000","last_undegraded":"2024-03-24T10:39:16.699787+0000","last_fullsized":"2024-03-24T10:39:16.699787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:57:46.398179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697418+0000","last_change":"2024-03-24T10:38:10.880202+0000","last_active":"2024-03-24T10:39:16.697418+0000","last_peered":"2024-03-24T10:39:16.697418+0000","last_clean":"2024-03-24T10:39:16.697418+0000","last_became_active":"2024-03-24T10:38:10.879807+0000","last_became_peered":"2024-03-24T10:38:10.879807+0000","last_unstale":"2024-03-24T10:39:16.697418+0000","last_undegraded":"2024-03-24T10:39:16.697418+0000","last_fullsized":"2024-03-24T10:39:16.697418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:37.498728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.704470+0000","last_change":"2024-03-24T10:39:16.704470+0000","last_active":"2024-03-24T10:39:16.704470+0000","last_peered":"2024-03-24T10:39:16.704470+0000","last_clean":"2024-03-24T10:39:16.704470+0000","last_became_active":"2024-03-24T10:39:16.701168+0000","last_became_peered":"2024-03-24T10:39:16.701168+0000","last_unstale":"2024-03-24T10:39:16.704470+0000","last_undegraded":"2024-03-24T10:39:16.704470+0000","last_fullsized":"2024-03-24T10:39:16.704470+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:20:34.781782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699419+0000","last_change":"2024-03-24T10:38:10.884371+0000","last_active":"2024-03-24T10:39:16.699419+0000","last_peered":"2024-03-24T10:39:16.699419+0000","last_clean":"2024-03-24T10:39:16.699419+0000","last_became_active":"2024-03-24T10:38:10.884022+0000","last_became_peered":"2024-03-24T10:38:10.884022+0000","last_unstale":"2024-03-24T10:39:16.699419+0000","last_undegraded":"2024-03-24T10:39:16.699419+0000","last_fullsized":"2024-03-24T10:39:16.699419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:47:48.778727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695865+0000","last_change":"2024-03-24T10:38:09.863231+0000","last_active":"2024-03-24T10:39:16.695865+0000","last_peered":"2024-03-24T10:39:16.695865+0000","last_clean":"2024-03-24T10:39:16.695865+0000","last_became_active":"2024-03-24T10:38:09.862848+0000","last_became_peered":"2024-03-24T10:38:09.862848+0000","last_unstale":"2024-03-24T10:39:16.695865+0000","last_undegraded":"2024-03-24T10:39:16.695865+0000","last_fullsized":"2024-03-24T10:39:16.695865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:36:57.532746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696011+0000","last_change":"2024-03-24T10:38:10.875187+0000","last_active":"2024-03-24T10:39:16.696011+0000","last_peered":"2024-03-24T10:39:16.696011+0000","last_clean":"2024-03-24T10:39:16.696011+0000","last_became_active":"2024-03-24T10:38:10.874682+0000","last_became_peered":"2024-03-24T10:38:10.874682+0000","last_unstale":"2024-03-24T10:39:16.696011+0000","last_undegraded":"2024-03-24T10:39:16.696011+0000","last_fullsized":"2024-03-24T10:39:16.696011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:53:56.915312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699445+0000","last_change":"2024-03-24T10:38:09.863430+0000","last_active":"2024-03-24T10:39:16.699445+0000","last_peered":"2024-03-24T10:39:16.699445+0000","last_clean":"2024-03-24T10:39:16.699445+0000","last_became_active":"2024-03-24T10:38:09.862877+0000","last_became_peered":"2024-03-24T10:38:09.862877+0000","last_unstale":"2024-03-24T10:39:16.699445+0000","last_undegraded":"2024-03-24T10:39:16.699445+0000","last_fullsized":"2024-03-24T10:39:16.699445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:33:29.039896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.695959+0000","last_change":"2024-03-24T10:38:10.878446+0000","last_active":"2024-03-24T10:39:16.695959+0000","last_peered":"2024-03-24T10:39:16.695959+0000","last_clean":"2024-03-24T10:39:16.695959+0000","last_became_active":"2024-03-24T10:38:10.877944+0000","last_became_peered":"2024-03-24T10:38:10.877944+0000","last_unstale":"2024-03-24T10:39:16.695959+0000","last_undegraded":"2024-03-24T10:39:16.695959+0000","last_fullsized":"2024-03-24T10:39:16.695959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:00:22.568669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698328+0000","last_change":"2024-03-24T10:38:09.867328+0000","last_active":"2024-03-24T10:39:16.698328+0000","last_peered":"2024-03-24T10:39:16.698328+0000","last_clean":"2024-03-24T10:39:16.698328+0000","last_became_active":"2024-03-24T10:38:09.866869+0000","last_became_peered":"2024-03-24T10:38:09.866869+0000","last_unstale":"2024-03-24T10:39:16.698328+0000","last_undegraded":"2024-03-24T10:39:16.698328+0000","last_fullsized":"2024-03-24T10:39:16.698328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:52:32.136151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.705912+0000","last_change":"2024-03-24T10:39:16.705912+0000","last_active":"2024-03-24T10:39:16.705912+0000","last_peered":"2024-03-24T10:39:16.705912+0000","last_clean":"2024-03-24T10:39:16.705912+0000","last_became_active":"2024-03-24T10:39:16.699809+0000","last_became_peered":"2024-03-24T10:39:16.699809+0000","last_unstale":"2024-03-24T10:39:16.705912+0000","last_undegraded":"2024-03-24T10:39:16.705912+0000","last_fullsized":"2024-03-24T10:39:16.705912+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:31.858976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.701719+0000","last_change":"2024-03-24T10:38:09.865399+0000","last_active":"2024-03-24T10:39:16.701719+0000","last_peered":"2024-03-24T10:39:16.701719+0000","last_clean":"2024-03-24T10:39:16.701719+0000","last_became_active":"2024-03-24T10:38:09.864906+0000","last_became_peered":"2024-03-24T10:38:09.864906+0000","last_unstale":"2024-03-24T10:39:16.701719+0000","last_undegraded":"2024-03-24T10:39:16.701719+0000","last_fullsized":"2024-03-24T10:39:16.701719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:53:12.363018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.696203+0000","last_change":"2024-03-24T10:38:10.876881+0000","last_active":"2024-03-24T10:39:16.696203+0000","last_peered":"2024-03-24T10:39:16.696203+0000","last_clean":"2024-03-24T10:39:16.696203+0000","last_became_active":"2024-03-24T10:38:10.876479+0000","last_became_peered":"2024-03-24T10:38:10.876479+0000","last_unstale":"2024-03-24T10:39:16.696203+0000","last_undegraded":"2024-03-24T10:39:16.696203+0000","last_fullsized":"2024-03-24T10:39:16.696203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:57:33.140451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697521+0000","last_change":"2024-03-24T10:38:10.872622+0000","last_active":"2024-03-24T10:39:16.697521+0000","last_peered":"2024-03-24T10:39:16.697521+0000","last_clean":"2024-03-24T10:39:16.697521+0000","last_became_active":"2024-03-24T10:38:10.872293+0000","last_became_peered":"2024-03-24T10:38:10.872293+0000","last_unstale":"2024-03-24T10:39:16.697521+0000","last_undegraded":"2024-03-24T10:39:16.697521+0000","last_fullsized":"2024-03-24T10:39:16.697521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:48.139589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.701548+0000","last_change":"2024-03-24T10:39:16.701548+0000","last_active":"2024-03-24T10:39:16.701548+0000","last_peered":"2024-03-24T10:39:16.701548+0000","last_clean":"2024-03-24T10:39:16.701548+0000","last_became_active":"2024-03-24T10:39:16.701213+0000","last_became_peered":"2024-03-24T10:39:16.701213+0000","last_unstale":"2024-03-24T10:39:16.701548+0000","last_undegraded":"2024-03-24T10:39:16.701548+0000","last_fullsized":"2024-03-24T10:39:16.701548+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:50:28.131754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697504+0000","last_change":"2024-03-24T10:38:10.879679+0000","last_active":"2024-03-24T10:39:16.697504+0000","last_peered":"2024-03-24T10:39:16.697504+0000","last_clean":"2024-03-24T10:39:16.697504+0000","last_became_active":"2024-03-24T10:38:10.879334+0000","last_became_peered":"2024-03-24T10:38:10.879334+0000","last_unstale":"2024-03-24T10:39:16.697504+0000","last_undegraded":"2024-03-24T10:39:16.697504+0000","last_fullsized":"2024-03-24T10:39:16.697504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:27:53.517368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.704444+0000","last_change":"2024-03-24T10:39:16.704444+0000","last_active":"2024-03-24T10:39:16.704444+0000","last_peered":"2024-03-24T10:39:16.704444+0000","last_clean":"2024-03-24T10:39:16.704444+0000","last_became_active":"2024-03-24T10:39:16.704085+0000","last_became_peered":"2024-03-24T10:39:16.704085+0000","last_unstale":"2024-03-24T10:39:16.704444+0000","last_undegraded":"2024-03-24T10:39:16.704444+0000","last_fullsized":"2024-03-24T10:39:16.704444+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:04:49.643195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.701459+0000","last_change":"2024-03-24T10:38:10.887108+0000","last_active":"2024-03-24T10:39:16.701459+0000","last_peered":"2024-03-24T10:39:16.701459+0000","last_clean":"2024-03-24T10:39:16.701459+0000","last_became_active":"2024-03-24T10:38:10.886770+0000","last_became_peered":"2024-03-24T10:38:10.886770+0000","last_unstale":"2024-03-24T10:39:16.701459+0000","last_undegraded":"2024-03-24T10:39:16.701459+0000","last_fullsized":"2024-03-24T10:39:16.701459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:56.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697741+0000","last_change":"2024-03-24T10:38:10.873369+0000","last_active":"2024-03-24T10:39:16.697741+0000","last_peered":"2024-03-24T10:39:16.697741+0000","last_clean":"2024-03-24T10:39:16.697741+0000","last_became_active":"2024-03-24T10:38:10.873046+0000","last_became_peered":"2024-03-24T10:38:10.873046+0000","last_unstale":"2024-03-24T10:39:16.697741+0000","last_undegraded":"2024-03-24T10:39:16.697741+0000","last_fullsized":"2024-03-24T10:39:16.697741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:46.071084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699744+0000","last_change":"2024-03-24T10:38:10.878666+0000","last_active":"2024-03-24T10:39:16.699744+0000","last_peered":"2024-03-24T10:39:16.699744+0000","last_clean":"2024-03-24T10:39:16.699744+0000","last_became_active":"2024-03-24T10:38:10.877250+0000","last_became_peered":"2024-03-24T10:38:10.877250+0000","last_unstale":"2024-03-24T10:39:16.699744+0000","last_undegraded":"2024-03-24T10:39:16.699744+0000","last_fullsized":"2024-03-24T10:39:16.699744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:13:36.793584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699422+0000","last_change":"2024-03-24T10:38:10.889274+0000","last_active":"2024-03-24T10:39:16.699422+0000","last_peered":"2024-03-24T10:39:16.699422+0000","last_clean":"2024-03-24T10:39:16.699422+0000","last_became_active":"2024-03-24T10:38:10.888899+0000","last_became_peered":"2024-03-24T10:38:10.888899+0000","last_unstale":"2024-03-24T10:39:16.699422+0000","last_undegraded":"2024-03-24T10:39:16.699422+0000","last_fullsized":"2024-03-24T10:39:16.699422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:11.225562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.698707+0000","last_change":"2024-03-24T10:38:10.887135+0000","last_active":"2024-03-24T10:39:16.698707+0000","last_peered":"2024-03-24T10:39:16.698707+0000","last_clean":"2024-03-24T10:39:16.698707+0000","last_became_active":"2024-03-24T10:38:10.886812+0000","last_became_peered":"2024-03-24T10:38:10.886812+0000","last_unstale":"2024-03-24T10:39:16.698707+0000","last_undegraded":"2024-03-24T10:39:16.698707+0000","last_fullsized":"2024-03-24T10:39:16.698707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:51:53.959914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.699624+0000","last_change":"2024-03-24T10:38:10.885417+0000","last_active":"2024-03-24T10:39:16.699624+0000","last_peered":"2024-03-24T10:39:16.699624+0000","last_clean":"2024-03-24T10:39:16.699624+0000","last_became_active":"2024-03-24T10:38:10.883209+0000","last_became_peered":"2024-03-24T10:38:10.883209+0000","last_unstale":"2024-03-24T10:39:16.699624+0000","last_undegraded":"2024-03-24T10:39:16.699624+0000","last_fullsized":"2024-03-24T10:39:16.699624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:52:52.535611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697650+0000","last_change":"2024-03-24T10:38:10.885005+0000","last_active":"2024-03-24T10:39:16.697650+0000","last_peered":"2024-03-24T10:39:16.697650+0000","last_clean":"2024-03-24T10:39:16.697650+0000","last_became_active":"2024-03-24T10:38:10.884680+0000","last_became_peered":"2024-03-24T10:38:10.884680+0000","last_unstale":"2024-03-24T10:39:16.697650+0000","last_undegraded":"2024-03-24T10:39:16.697650+0000","last_fullsized":"2024-03-24T10:39:16.697650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:56.377777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.702195+0000","last_change":"2024-03-24T10:38:10.875153+0000","last_active":"2024-03-24T10:39:16.702195+0000","last_peered":"2024-03-24T10:39:16.702195+0000","last_clean":"2024-03-24T10:39:16.702195+0000","last_became_active":"2024-03-24T10:38:10.874721+0000","last_became_peered":"2024-03-24T10:38:10.874721+0000","last_unstale":"2024-03-24T10:39:16.702195+0000","last_undegraded":"2024-03-24T10:39:16.702195+0000","last_fullsized":"2024-03-24T10:39:16.702195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:52.522797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-24T10:39:16.697574+0000","last_change":"2024-03-24T10:38:10.887298+0000","last_active":"2024-03-24T10:39:16.697574+0000","last_peered":"2024-03-24T10:39:16.697574+0000","last_clean":"2024-03-24T10:39:16.697574+0000","last_became_active":"2024-03-24T10:38:10.886975+0000","last_became_peered":"2024-03-24T10:38:10.886975+0000","last_unstale":"2024-03-24T10:39:16.697574+0000","last_undegraded":"2024-03-24T10:39:16.697574+0000","last_fullsized":"2024-03-24T10:39:16.697574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:09.740303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63466,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":425,"num_read_kb":279,"num_write":155,"num_write_kb":106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":190398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383708,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":950272,"data_stored":456395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645345,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32788,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712364,"statfs":{"total":95995035648,"available":95961460736,"internally_reserved":0,"allocated":1646592,"data_stored":1107732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808870,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":456748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070506,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":455969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332143,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1560576,"data_stored":1048481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495669,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":456020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1019904,"data_stored":515860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":457291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":516516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182468,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":456214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920831,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":456960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757305,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1572864,"data_stored":1047774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":60047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":60703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:39:27.285 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T10:39:27.285 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T10:39:27.285 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T10:39:27.285 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T10:39:27.528 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:27.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:27 smithi023 ceph-mon[32466]: from='client.24838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:39:28.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:27 smithi117 ceph-mon[36368]: from='client.24838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:39:28.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:27 smithi116 ceph-mon[36248]: from='client.24838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:39:28.832 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:28 smithi023 ceph-mon[32466]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 80 B/s, 1 objects/s recovering 2024-03-24T10:39:29.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:28 smithi117 ceph-mon[36368]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 80 B/s, 1 objects/s recovering 2024-03-24T10:39:29.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:28 smithi116 ceph-mon[36248]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 80 B/s, 1 objects/s recovering 2024-03-24T10:39:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.065+0000 7f6d1b4da700 1 -- 172.21.15.23:0/4044463167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d141013c0 msgr2=0x7f6d141017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.065+0000 7f6d1b4da700 1 --2- 172.21.15.23:0/4044463167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d141013c0 0x7f6d141017a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400b0a0 tx=0x7f6d0400b3b0 comp rx=0 tx=0).stop 2024-03-24T10:39:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.065+0000 7f6d1b4da700 1 -- 172.21.15.23:0/4044463167 shutdown_connections 2024-03-24T10:39:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.065+0000 7f6d1b4da700 1 --2- 172.21.15.23:0/4044463167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d14106410 0x7f6d1410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.065+0000 7f6d1b4da700 1 --2- 172.21.15.23:0/4044463167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d141013c0 0x7f6d141017a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.065+0000 7f6d1b4da700 1 --2- 172.21.15.23:0/4044463167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d14101d70 0x7f6d14105d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.065+0000 7f6d1b4da700 1 -- 172.21.15.23:0/4044463167 >> 172.21.15.23:0/4044463167 conn(0x7f6d14077150 msgr2=0x7f6d14077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.065+0000 7f6d1b4da700 1 -- 172.21.15.23:0/4044463167 shutdown_connections 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.065+0000 7f6d1b4da700 1 -- 172.21.15.23:0/4044463167 wait complete. 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.065+0000 7f6d1b4da700 1 Processor -- start 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.065+0000 7f6d1b4da700 1 -- start start 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d1b4da700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d141013c0 0x7f6d14102dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d1b4da700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d14101d70 0x7f6d14103310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d1b4da700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d14106410 0x7f6d14103850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d1b4da700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f6d141a6c80 con 0x7f6d14101d70 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d1b4da700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6d141a6df0 con 0x7f6d141013c0 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d1b4da700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6d141a6fb0 con 0x7f6d14106410 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d18a75700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d14101d70 0x7f6d14103310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d18a75700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d14101d70 0x7f6d14103310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38060/0 (socket says 172.21.15.23:38060) 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d18a75700 1 -- 172.21.15.23:0/2973316072 learned_addr learned my addr 172.21.15.23:0/2973316072 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d19a77700 1 --2- 172.21.15.23:0/2973316072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d14106410 0x7f6d14103850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d19276700 1 --2- 172.21.15.23:0/2973316072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d141013c0 0x7f6d14102dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d18a75700 1 -- 172.21.15.23:0/2973316072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d14106410 msgr2=0x7f6d14103850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:29.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d18a75700 1 --2- 172.21.15.23:0/2973316072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d14106410 0x7f6d14103850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:29.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d18a75700 1 -- 172.21.15.23:0/2973316072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d141013c0 msgr2=0x7f6d14102dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:29.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d18a75700 1 --2- 172.21.15.23:0/2973316072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d141013c0 0x7f6d14102dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:29.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d18a75700 1 -- 172.21.15.23:0/2973316072 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d04009d00 con 0x7f6d14101d70 2024-03-24T10:39:29.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d18a75700 1 --2- 172.21.15.23:0/2973316072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d14101d70 0x7f6d14103310 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800e970 tx=0x7f6d0800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:29.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d127fc700 1 -- 172.21.15.23:0/2973316072 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08009790 con 0x7f6d14101d70 2024-03-24T10:39:29.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d127fc700 1 -- 172.21.15.23:0/2973316072 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d0800c7e0 con 0x7f6d14101d70 2024-03-24T10:39:29.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.066+0000 7f6d1b4da700 1 -- 172.21.15.23:0/2973316072 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d141a7290 con 0x7f6d14101d70 2024-03-24T10:39:29.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.067+0000 7f6d127fc700 1 -- 172.21.15.23:0/2973316072 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08005580 con 0x7f6d14101d70 2024-03-24T10:39:29.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.067+0000 7f6d1b4da700 1 -- 172.21.15.23:0/2973316072 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d141a7790 con 0x7f6d14101d70 2024-03-24T10:39:29.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.067+0000 7f6d1b4da700 1 -- 172.21.15.23:0/2973316072 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d1410a6e0 con 0x7f6d14101d70 2024-03-24T10:39:29.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.069+0000 7f6d127fc700 1 -- 172.21.15.23:0/2973316072 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f6d0800cce0 con 0x7f6d14101d70 2024-03-24T10:39:29.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.069+0000 7f6d127fc700 1 --2- 172.21.15.23:0/2973316072 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6cfc0a21e0 0x7f6cfc0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:29.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.069+0000 7f6d19276700 1 --2- 172.21.15.23:0/2973316072 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6cfc0a21e0 0x7f6cfc0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:29.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.069+0000 7f6d127fc700 1 -- 172.21.15.23:0/2973316072 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6d08014070 con 0x7f6d14101d70 2024-03-24T10:39:29.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.070+0000 7f6d19276700 1 --2- 172.21.15.23:0/2973316072 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6cfc0a21e0 0x7f6cfc0a46a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009fd0 tx=0x7f6d0400b720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:29.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.074+0000 7f6d127fc700 1 -- 172.21.15.23:0/2973316072 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6d08096e80 con 0x7f6d14101d70 2024-03-24T10:39:29.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.321+0000 7f6d1b4da700 1 -- 172.21.15.23:0/2973316072 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f6d14068ec0 con 0x7f6d14101d70 2024-03-24T10:39:29.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.321+0000 7f6d127fc700 1 -- 172.21.15.23:0/2973316072 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f6d08085020 con 0x7f6d14101d70 2024-03-24T10:39:29.322 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:29.322 INFO:teuthology.orchestra.run.smithi023.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T10:39:29.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.323+0000 7f6d1b4da700 1 -- 172.21.15.23:0/2973316072 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6cfc0a21e0 msgr2=0x7f6cfc0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:29.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.323+0000 7f6d1b4da700 1 --2- 172.21.15.23:0/2973316072 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6cfc0a21e0 0x7f6cfc0a46a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009fd0 tx=0x7f6d0400b720 comp rx=0 tx=0).stop 2024-03-24T10:39:29.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.323+0000 7f6d1b4da700 1 -- 172.21.15.23:0/2973316072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d14101d70 msgr2=0x7f6d14103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:29.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.323+0000 7f6d1b4da700 1 --2- 172.21.15.23:0/2973316072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d14101d70 0x7f6d14103310 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800e970 tx=0x7f6d0800ec80 comp rx=0 tx=0).stop 2024-03-24T10:39:29.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.323+0000 7f6d1b4da700 1 -- 172.21.15.23:0/2973316072 shutdown_connections 2024-03-24T10:39:29.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.324+0000 7f6d1b4da700 1 --2- 172.21.15.23:0/2973316072 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6cfc0a21e0 0x7f6cfc0a46a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:29.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.324+0000 7f6d1b4da700 1 --2- 172.21.15.23:0/2973316072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d14106410 0x7f6d14103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:29.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.324+0000 7f6d1b4da700 1 --2- 172.21.15.23:0/2973316072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d141013c0 0x7f6d14102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:29.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.324+0000 7f6d1b4da700 1 --2- 172.21.15.23:0/2973316072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d14101d70 0x7f6d14103310 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:29.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.324+0000 7f6d1b4da700 1 -- 172.21.15.23:0/2973316072 >> 172.21.15.23:0/2973316072 conn(0x7f6d14077150 msgr2=0x7f6d1406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:29.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.324+0000 7f6d1b4da700 1 -- 172.21.15.23:0/2973316072 shutdown_connections 2024-03-24T10:39:29.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:29.324+0000 7f6d1b4da700 1 -- 172.21.15.23:0/2973316072 wait complete. 2024-03-24T10:39:29.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:29 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2973316072' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T10:39:29.970 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T10:39:29.970 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:39:30.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:29 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2973316072' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T10:39:30.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:29 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2973316072' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T10:39:30.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.077+0000 7fcef3288640 1 -- 172.21.15.23:0/1295856686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fceec1021e0 msgr2=0x7fceec108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.077+0000 7fcef3288640 1 --2- 172.21.15.23:0/1295856686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fceec1021e0 0x7fceec108a70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcee4009ca0 tx=0x7fcee402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.077+0000 7fcef3288640 1 -- 172.21.15.23:0/1295856686 shutdown_connections 2024-03-24T10:39:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.077+0000 7fcef3288640 1 --2- 172.21.15.23:0/1295856686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fceec1021e0 0x7fceec108a70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.077+0000 7fcef3288640 1 --2- 172.21.15.23:0/1295856686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fceec101820 0x7fceec101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.077+0000 7fcef3288640 1 --2- 172.21.15.23:0/1295856686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fceec100630 0x7fceec100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.077+0000 7fcef3288640 1 -- 172.21.15.23:0/1295856686 >> 172.21.15.23:0/1295856686 conn(0x7fceec0fbba0 msgr2=0x7fceec0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.077+0000 7fcef3288640 1 -- 172.21.15.23:0/1295856686 shutdown_connections 2024-03-24T10:39:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.077+0000 7fcef3288640 1 -- 172.21.15.23:0/1295856686 wait complete. 2024-03-24T10:39:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef3288640 1 Processor -- start 2024-03-24T10:39:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef3288640 1 -- start start 2024-03-24T10:39:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef3288640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fceec100630 0x7fceec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef3288640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fceec101820 0x7fceec19e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef3288640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fceec1021e0 0x7fceec1a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef3288640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fceec1a14f0 con 0x7fceec101820 2024-03-24T10:39:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef3288640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fceec1a1660 con 0x7fceec1021e0 2024-03-24T10:39:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef3288640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fceec1a1820 con 0x7fceec100630 2024-03-24T10:39:30.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef0823640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fceec101820 0x7fceec19e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:30.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef1024640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fceec100630 0x7fceec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:30.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef0823640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fceec101820 0x7fceec19e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38068/0 (socket says 172.21.15.23:38068) 2024-03-24T10:39:30.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef0823640 1 -- 172.21.15.23:0/354642959 learned_addr learned my addr 172.21.15.23:0/354642959 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:30.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef1825640 1 --2- 172.21.15.23:0/354642959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fceec1021e0 0x7fceec1a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:30.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef0823640 1 -- 172.21.15.23:0/354642959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fceec100630 msgr2=0x7fceec0693e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef0823640 1 --2- 172.21.15.23:0/354642959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fceec100630 0x7fceec0693e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef0823640 1 -- 172.21.15.23:0/354642959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fceec1021e0 msgr2=0x7fceec1a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef0823640 1 --2- 172.21.15.23:0/354642959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fceec1021e0 0x7fceec1a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef0823640 1 -- 172.21.15.23:0/354642959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee4009900 con 0x7fceec101820 2024-03-24T10:39:30.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef1024640 1 --2- 172.21.15.23:0/354642959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fceec100630 0x7fceec0693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:30.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef0823640 1 --2- 172.21.15.23:0/354642959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fceec101820 0x7fceec19e8e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcee800e9f0 tx=0x7fcee800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:30.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcee27fc640 1 -- 172.21.15.23:0/354642959 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee8009720 con 0x7fceec101820 2024-03-24T10:39:30.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcee27fc640 1 -- 172.21.15.23:0/354642959 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcee800cd80 con 0x7fceec101820 2024-03-24T10:39:30.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.078+0000 7fcef3288640 1 -- 172.21.15.23:0/354642959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceec1a1aa0 con 0x7fceec101820 2024-03-24T10:39:30.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.079+0000 7fcee27fc640 1 -- 172.21.15.23:0/354642959 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee8010690 con 0x7fceec101820 2024-03-24T10:39:30.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.079+0000 7fcef3288640 1 -- 172.21.15.23:0/354642959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceec1a1f60 con 0x7fceec101820 2024-03-24T10:39:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.080+0000 7fcef3288640 1 -- 172.21.15.23:0/354642959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceec107950 con 0x7fceec101820 2024-03-24T10:39:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.080+0000 7fcee27fc640 1 -- 172.21.15.23:0/354642959 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fcee80046c0 con 0x7fceec101820 2024-03-24T10:39:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.081+0000 7fcee27fc640 1 --2- 172.21.15.23:0/354642959 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fced80a2180 0x7fced80a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.081+0000 7fcef1024640 1 --2- 172.21.15.23:0/354642959 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fced80a2180 0x7fced80a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.081+0000 7fcee27fc640 1 -- 172.21.15.23:0/354642959 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fcee8014070 con 0x7fceec101820 2024-03-24T10:39:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.081+0000 7fcef1024640 1 --2- 172.21.15.23:0/354642959 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fced80a2180 0x7fced80a4640 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fceec101680 tx=0x7fcedc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:30.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.085+0000 7fcee27fc640 1 -- 172.21.15.23:0/354642959 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcee8096fd0 con 0x7fceec101820 2024-03-24T10:39:30.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.271+0000 7fcef3288640 1 -- 172.21.15.23:0/354642959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fceec10c250 con 0x7fceec101820 2024-03-24T10:39:30.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.272+0000 7fcee27fc640 1 -- 172.21.15.23:0/354642959 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22704 (secure 0 0 0) 0x7fcee809a690 con 0x7fceec101820 2024-03-24T10:39:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:30.273 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:39:16.692832+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:39:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.274+0000 7fcef3288640 1 -- 172.21.15.23:0/354642959 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fced80a2180 msgr2=0x7fced80a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.274+0000 7fcef3288640 1 --2- 172.21.15.23:0/354642959 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fced80a2180 0x7fced80a4640 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fceec101680 tx=0x7fcedc005f90 comp rx=0 tx=0).stop 2024-03-24T10:39:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.274+0000 7fcef3288640 1 -- 172.21.15.23:0/354642959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fceec101820 msgr2=0x7fceec19e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.274+0000 7fcef3288640 1 --2- 172.21.15.23:0/354642959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fceec101820 0x7fceec19e8e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcee800e9f0 tx=0x7fcee800edf0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.274+0000 7fcef3288640 1 -- 172.21.15.23:0/354642959 shutdown_connections 2024-03-24T10:39:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.274+0000 7fcef3288640 1 --2- 172.21.15.23:0/354642959 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fced80a2180 0x7fced80a4640 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.274+0000 7fcef3288640 1 --2- 172.21.15.23:0/354642959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fceec1021e0 0x7fceec1a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.274+0000 7fcef3288640 1 --2- 172.21.15.23:0/354642959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fceec101820 0x7fceec19e8e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.274+0000 7fcef3288640 1 --2- 172.21.15.23:0/354642959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fceec100630 0x7fceec0693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.274+0000 7fcef3288640 1 -- 172.21.15.23:0/354642959 >> 172.21.15.23:0/354642959 conn(0x7fceec0fbba0 msgr2=0x7fceec104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:30.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.275+0000 7fcef3288640 1 -- 172.21.15.23:0/354642959 shutdown_connections 2024-03-24T10:39:30.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.275+0000 7fcef3288640 1 -- 172.21.15.23:0/354642959 wait complete. 2024-03-24T10:39:30.286 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:39.056452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:38:08.473472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:38:09.414667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:39:30.287 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:39:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.391+0000 7f6d5c215640 1 -- 172.21.15.23:0/2685275719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d541073f0 msgr2=0x7f6d54109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.391+0000 7f6d5c215640 1 --2- 172.21.15.23:0/2685275719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d541073f0 0x7f6d54109840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c009be0 tx=0x7f6d4c02bef0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.391+0000 7f6d5c215640 1 -- 172.21.15.23:0/2685275719 shutdown_connections 2024-03-24T10:39:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.391+0000 7f6d5c215640 1 --2- 172.21.15.23:0/2685275719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d541073f0 0x7f6d54109840 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.391+0000 7f6d5c215640 1 --2- 172.21.15.23:0/2685275719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d54104ac0 0x7f6d54106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.391+0000 7f6d5c215640 1 --2- 172.21.15.23:0/2685275719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d54102190 0x7f6d54104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.391+0000 7f6d5c215640 1 -- 172.21.15.23:0/2685275719 >> 172.21.15.23:0/2685275719 conn(0x7f6d540fbb80 msgr2=0x7f6d540fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d5c215640 1 -- 172.21.15.23:0/2685275719 shutdown_connections 2024-03-24T10:39:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d5c215640 1 -- 172.21.15.23:0/2685275719 wait complete. 2024-03-24T10:39:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d5c215640 1 Processor -- start 2024-03-24T10:39:30.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d5c215640 1 -- start start 2024-03-24T10:39:30.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d5c215640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d54102190 0x7f6d54197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:30.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d5c215640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d54104ac0 0x7f6d54197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:30.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d5c215640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d541073f0 0x7f6d5419ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:30.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d5c215640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5419f230 con 0x7f6d54104ac0 2024-03-24T10:39:30.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d5c215640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5419f3a0 con 0x7f6d541073f0 2024-03-24T10:39:30.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d5c215640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5419f560 con 0x7f6d54102190 2024-03-24T10:39:30.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d597b0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d54104ac0 0x7f6d54197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:30.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d597b0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d54104ac0 0x7f6d54197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38094/0 (socket says 172.21.15.23:38094) 2024-03-24T10:39:30.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d597b0640 1 -- 172.21.15.23:0/3373006496 learned_addr learned my addr 172.21.15.23:0/3373006496 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:30.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d59fb1640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d54102190 0x7f6d54197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:30.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.392+0000 7f6d5a7b2640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d541073f0 0x7f6d5419ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:30.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.393+0000 7f6d597b0640 1 -- 172.21.15.23:0/3373006496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d54102190 msgr2=0x7f6d54197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.393+0000 7f6d597b0640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d54102190 0x7f6d54197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.393+0000 7f6d597b0640 1 -- 172.21.15.23:0/3373006496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d541073f0 msgr2=0x7f6d5419ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.393+0000 7f6d597b0640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d541073f0 0x7f6d5419ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.393+0000 7f6d597b0640 1 -- 172.21.15.23:0/3373006496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d44009560 con 0x7f6d54104ac0 2024-03-24T10:39:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.393+0000 7f6d59fb1640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d54102190 0x7f6d54197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.393+0000 7f6d597b0640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d54104ac0 0x7f6d54197c80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800e9c0 tx=0x7f6d4800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.393+0000 7f6d52ffd640 1 -- 172.21.15.23:0/3373006496 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48009700 con 0x7f6d54104ac0 2024-03-24T10:39:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.393+0000 7f6d5c215640 1 -- 172.21.15.23:0/3373006496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4c009840 con 0x7f6d54104ac0 2024-03-24T10:39:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.393+0000 7f6d52ffd640 1 -- 172.21.15.23:0/3373006496 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d4800cd80 con 0x7f6d54104ac0 2024-03-24T10:39:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.393+0000 7f6d52ffd640 1 -- 172.21.15.23:0/3373006496 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48010430 con 0x7f6d54104ac0 2024-03-24T10:39:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.393+0000 7f6d5c215640 1 -- 172.21.15.23:0/3373006496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d5419fbc0 con 0x7f6d54104ac0 2024-03-24T10:39:30.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.394+0000 7f6d5c215640 1 -- 172.21.15.23:0/3373006496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d540fd670 con 0x7f6d54104ac0 2024-03-24T10:39:30.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.395+0000 7f6d52ffd640 1 -- 172.21.15.23:0/3373006496 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f6d48005000 con 0x7f6d54104ac0 2024-03-24T10:39:30.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.396+0000 7f6d52ffd640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6d3c0a2150 0x7f6d3c0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:30.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.396+0000 7f6d59fb1640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6d3c0a2150 0x7f6d3c0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:30.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.396+0000 7f6d52ffd640 1 -- 172.21.15.23:0/3373006496 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6d48014070 con 0x7f6d54104ac0 2024-03-24T10:39:30.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.396+0000 7f6d59fb1640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6d3c0a2150 0x7f6d3c0a4610 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009530 tx=0x7f6d44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:30.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.399+0000 7f6d52ffd640 1 -- 172.21.15.23:0/3373006496 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6d48097300 con 0x7f6d54104ac0 2024-03-24T10:39:30.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.583+0000 7f6d5c215640 1 -- 172.21.15.23:0/3373006496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6d5410d290 con 0x7f6d54104ac0 2024-03-24T10:39:30.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.584+0000 7f6d52ffd640 1 -- 172.21.15.23:0/3373006496 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f6d5410d290 con 0x7f6d54104ac0 2024-03-24T10:39:30.584 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-24T10:39:30.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.586+0000 7f6d5c215640 1 -- 172.21.15.23:0/3373006496 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6d3c0a2150 msgr2=0x7f6d3c0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.586+0000 7f6d5c215640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6d3c0a2150 0x7f6d3c0a4610 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009530 tx=0x7f6d44009260 comp rx=0 tx=0).stop 2024-03-24T10:39:30.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.586+0000 7f6d5c215640 1 -- 172.21.15.23:0/3373006496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d54104ac0 msgr2=0x7f6d54197c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.586+0000 7f6d5c215640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d54104ac0 0x7f6d54197c80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800e9c0 tx=0x7f6d4800ee80 comp rx=0 tx=0).stop 2024-03-24T10:39:30.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.586+0000 7f6d5c215640 1 -- 172.21.15.23:0/3373006496 shutdown_connections 2024-03-24T10:39:30.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.586+0000 7f6d5c215640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6d3c0a2150 0x7f6d3c0a4610 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.586+0000 7f6d5c215640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d541073f0 0x7f6d5419ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.586+0000 7f6d5c215640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d54104ac0 0x7f6d54197c80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.586+0000 7f6d5c215640 1 --2- 172.21.15.23:0/3373006496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d54102190 0x7f6d54197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.586+0000 7f6d5c215640 1 -- 172.21.15.23:0/3373006496 >> 172.21.15.23:0/3373006496 conn(0x7f6d540fbb80 msgr2=0x7f6d54102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:30.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.586+0000 7f6d5c215640 1 -- 172.21.15.23:0/3373006496 shutdown_connections 2024-03-24T10:39:30.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.586+0000 7f6d5c215640 1 -- 172.21.15.23:0/3373006496 wait complete. 2024-03-24T10:39:30.597 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:39:30.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.702+0000 7f7fa68ce640 1 -- 172.21.15.23:0/2712969205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fa0102910 msgr2=0x7f7fa0102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.702+0000 7f7fa68ce640 1 --2- 172.21.15.23:0/2712969205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fa0102910 0x7f7fa0102d90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7f94009860 tx=0x7f7f9402bb70 comp rx=0 tx=0).stop 2024-03-24T10:39:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.702+0000 7f7fa68ce640 1 -- 172.21.15.23:0/2712969205 shutdown_connections 2024-03-24T10:39:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.702+0000 7f7fa68ce640 1 --2- 172.21.15.23:0/2712969205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fa01032d0 0x7f7fa0109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.702+0000 7f7fa68ce640 1 --2- 172.21.15.23:0/2712969205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fa0102910 0x7f7fa0102d90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.702+0000 7f7fa68ce640 1 --2- 172.21.15.23:0/2712969205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fa0101720 0x7f7fa0101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.702+0000 7f7fa68ce640 1 -- 172.21.15.23:0/2712969205 >> 172.21.15.23:0/2712969205 conn(0x7f7fa00fccb0 msgr2=0x7f7fa00ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.702+0000 7f7fa68ce640 1 -- 172.21.15.23:0/2712969205 shutdown_connections 2024-03-24T10:39:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.702+0000 7f7fa68ce640 1 -- 172.21.15.23:0/2712969205 wait complete. 2024-03-24T10:39:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.702+0000 7f7fa68ce640 1 Processor -- start 2024-03-24T10:39:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.702+0000 7f7fa68ce640 1 -- start start 2024-03-24T10:39:30.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7fa68ce640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fa0101720 0x7f7fa01977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:30.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7fa68ce640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fa0102910 0x7f7fa0197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:30.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7fa68ce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fa01032d0 0x7f7fa019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:30.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7fa68ce640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa019f2b0 con 0x7f7fa0101720 2024-03-24T10:39:30.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7fa68ce640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa019f420 con 0x7f7fa01032d0 2024-03-24T10:39:30.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7fa68ce640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa019f5e0 con 0x7f7fa0102910 2024-03-24T10:39:30.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7f9ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fa0101720 0x7f7fa01977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:30.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7f9ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fa0101720 0x7f7fa01977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38102/0 (socket says 172.21.15.23:38102) 2024-03-24T10:39:30.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7f9ffff640 1 -- 172.21.15.23:0/16886079 learned_addr learned my addr 172.21.15.23:0/16886079 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:30.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7f9f7fe640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fa0102910 0x7f7fa0197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:30.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7fa4e6b640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fa01032d0 0x7f7fa019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:30.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7f9ffff640 1 -- 172.21.15.23:0/16886079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fa0102910 msgr2=0x7f7fa0197d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7f9ffff640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fa0102910 0x7f7fa0197d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7f9ffff640 1 -- 172.21.15.23:0/16886079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fa01032d0 msgr2=0x7f7fa019ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7f9ffff640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fa01032d0 0x7f7fa019ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7f9ffff640 1 -- 172.21.15.23:0/16886079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f90009560 con 0x7f7fa0101720 2024-03-24T10:39:30.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7f9f7fe640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fa0102910 0x7f7fa0197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:30.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7f9ffff640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fa0101720 0x7f7fa01977c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000e9a0 tx=0x7f7f9000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:30.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.703+0000 7f7f9d7fa640 1 -- 172.21.15.23:0/16886079 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9000cd80 con 0x7f7fa0101720 2024-03-24T10:39:30.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.704+0000 7f7f9d7fa640 1 -- 172.21.15.23:0/16886079 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f90010910 con 0x7f7fa0101720 2024-03-24T10:39:30.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.704+0000 7f7fa68ce640 1 -- 172.21.15.23:0/16886079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f94009490 con 0x7f7fa0101720 2024-03-24T10:39:30.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.704+0000 7f7f9d7fa640 1 -- 172.21.15.23:0/16886079 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90018980 con 0x7f7fa0101720 2024-03-24T10:39:30.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.704+0000 7f7fa68ce640 1 -- 172.21.15.23:0/16886079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fa019fc10 con 0x7f7fa0101720 2024-03-24T10:39:30.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.705+0000 7f7fa68ce640 1 -- 172.21.15.23:0/16886079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fa0108ac0 con 0x7f7fa0101720 2024-03-24T10:39:30.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.706+0000 7f7f9d7fa640 1 -- 172.21.15.23:0/16886079 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f7f90010a80 con 0x7f7fa0101720 2024-03-24T10:39:30.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.706+0000 7f7f9d7fa640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f880a2150 0x7f7f880a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:30.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.706+0000 7f7f9f7fe640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f880a2150 0x7f7f880a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:30.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.706+0000 7f7f9d7fa640 1 -- 172.21.15.23:0/16886079 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7f90014070 con 0x7f7fa0101720 2024-03-24T10:39:30.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.707+0000 7f7f9f7fe640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f880a2150 0x7f7f880a4610 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7f9402c180 tx=0x7f7f94034c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:30.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.710+0000 7f7f9d7fa640 1 -- 172.21.15.23:0/16886079 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7f900977d0 con 0x7f7fa0101720 2024-03-24T10:39:30.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.895+0000 7f7fa68ce640 1 -- 172.21.15.23:0/16886079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7fa010d2c0 con 0x7f7fa0101720 2024-03-24T10:39:30.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.895+0000 7f7f9d7fa640 1 -- 172.21.15.23:0/16886079 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f7f9009ae90 con 0x7f7fa0101720 2024-03-24T10:39:30.896 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-24T10:39:30.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.897+0000 7f7fa68ce640 1 -- 172.21.15.23:0/16886079 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f880a2150 msgr2=0x7f7f880a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.897+0000 7f7fa68ce640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f880a2150 0x7f7f880a4610 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7f9402c180 tx=0x7f7f94034c00 comp rx=0 tx=0).stop 2024-03-24T10:39:30.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.897+0000 7f7fa68ce640 1 -- 172.21.15.23:0/16886079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fa0101720 msgr2=0x7f7fa01977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:30.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.897+0000 7f7fa68ce640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fa0101720 0x7f7fa01977c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000e9a0 tx=0x7f7f9000ee60 comp rx=0 tx=0).stop 2024-03-24T10:39:30.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.897+0000 7f7fa68ce640 1 -- 172.21.15.23:0/16886079 shutdown_connections 2024-03-24T10:39:30.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.897+0000 7f7fa68ce640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f880a2150 0x7f7f880a4610 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.897+0000 7f7fa68ce640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fa01032d0 0x7f7fa019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.897+0000 7f7fa68ce640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fa0102910 0x7f7fa0197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.897+0000 7f7fa68ce640 1 --2- 172.21.15.23:0/16886079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fa0101720 0x7f7fa01977c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:30.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.897+0000 7f7fa68ce640 1 -- 172.21.15.23:0/16886079 >> 172.21.15.23:0/16886079 conn(0x7f7fa00fccb0 msgr2=0x7f7fa0105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:30.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.898+0000 7f7fa68ce640 1 -- 172.21.15.23:0/16886079 shutdown_connections 2024-03-24T10:39:30.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:30.898+0000 7f7fa68ce640 1 -- 172.21.15.23:0/16886079 wait complete. 2024-03-24T10:39:30.908 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:30 smithi023 ceph-mon[32466]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 67 B/s, 1 objects/s recovering 2024-03-24T10:39:30.908 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:30 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/354642959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:30.908 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:30 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3373006496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:39:30.909 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:39:31.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:30 smithi117 ceph-mon[36368]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 67 B/s, 1 objects/s recovering 2024-03-24T10:39:31.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:30 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/354642959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:31.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:30 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3373006496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:39:31.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.014+0000 7fa30422a640 1 -- 172.21.15.23:0/3087161759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc102900 msgr2=0x7fa2fc102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.014+0000 7fa30422a640 1 --2- 172.21.15.23:0/3087161759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc102900 0x7fa2fc102d80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8009900 tx=0x7fa2f802bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:31.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.014+0000 7fa30422a640 1 -- 172.21.15.23:0/3087161759 shutdown_connections 2024-03-24T10:39:31.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.014+0000 7fa30422a640 1 --2- 172.21.15.23:0/3087161759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2fc1032c0 0x7fa2fc109b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.014+0000 7fa30422a640 1 --2- 172.21.15.23:0/3087161759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc102900 0x7fa2fc102d80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.014+0000 7fa30422a640 1 --2- 172.21.15.23:0/3087161759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2fc101710 0x7fa2fc101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.014+0000 7fa30422a640 1 -- 172.21.15.23:0/3087161759 >> 172.21.15.23:0/3087161759 conn(0x7fa2fc0fcc80 msgr2=0x7fa2fc0ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.014+0000 7fa30422a640 1 -- 172.21.15.23:0/3087161759 shutdown_connections 2024-03-24T10:39:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.014+0000 7fa30422a640 1 -- 172.21.15.23:0/3087161759 wait complete. 2024-03-24T10:39:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.014+0000 7fa30422a640 1 Processor -- start 2024-03-24T10:39:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.014+0000 7fa30422a640 1 -- start start 2024-03-24T10:39:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.014+0000 7fa30422a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2fc101710 0x7fa2fc197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.014+0000 7fa30422a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc102900 0x7fa2fc197cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.014+0000 7fa30422a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2fc1032c0 0x7fa2fc19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa30422a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc19f280 con 0x7fa2fc102900 2024-03-24T10:39:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa30422a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc19f3f0 con 0x7fa2fc1032c0 2024-03-24T10:39:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa30422a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc19f5b0 con 0x7fa2fc101710 2024-03-24T10:39:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa3017c5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc102900 0x7fa2fc197cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa3017c5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc102900 0x7fa2fc197cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38116/0 (socket says 172.21.15.23:38116) 2024-03-24T10:39:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa3017c5640 1 -- 172.21.15.23:0/2814415958 learned_addr learned my addr 172.21.15.23:0/2814415958 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:31.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa301fc6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2fc101710 0x7fa2fc197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa3027c7640 1 --2- 172.21.15.23:0/2814415958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2fc1032c0 0x7fa2fc19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa3017c5640 1 -- 172.21.15.23:0/2814415958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2fc101710 msgr2=0x7fa2fc197790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa3017c5640 1 --2- 172.21.15.23:0/2814415958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2fc101710 0x7fa2fc197790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa3017c5640 1 -- 172.21.15.23:0/2814415958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2fc1032c0 msgr2=0x7fa2fc19ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa3017c5640 1 --2- 172.21.15.23:0/2814415958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2fc1032c0 0x7fa2fc19ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa3017c5640 1 -- 172.21.15.23:0/2814415958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f8009560 con 0x7fa2fc102900 2024-03-24T10:39:31.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa3017c5640 1 --2- 172.21.15.23:0/2814415958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc102900 0x7fa2fc197cd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8006010 tx=0x7fa2f8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:31.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa2f2ffd640 1 -- 172.21.15.23:0/2814415958 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f80043d0 con 0x7fa2fc102900 2024-03-24T10:39:31.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa2f2ffd640 1 -- 172.21.15.23:0/2814415958 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2f80052d0 con 0x7fa2fc102900 2024-03-24T10:39:31.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa30422a640 1 -- 172.21.15.23:0/2814415958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2fc19f830 con 0x7fa2fc102900 2024-03-24T10:39:31.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa2f2ffd640 1 -- 172.21.15.23:0/2814415958 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8003cc0 con 0x7fa2fc102900 2024-03-24T10:39:31.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.015+0000 7fa30422a640 1 -- 172.21.15.23:0/2814415958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2fc19fd90 con 0x7fa2fc102900 2024-03-24T10:39:31.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:30 smithi116 ceph-mon[36248]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 67 B/s, 1 objects/s recovering 2024-03-24T10:39:31.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:30 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/354642959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:31.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:30 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3373006496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:39:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.016+0000 7fa30422a640 1 -- 172.21.15.23:0/2814415958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2fc108a30 con 0x7fa2fc102900 2024-03-24T10:39:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.017+0000 7fa2f2ffd640 1 -- 172.21.15.23:0/2814415958 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fa2f8005440 con 0x7fa2fc102900 2024-03-24T10:39:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.018+0000 7fa2f2ffd640 1 --2- 172.21.15.23:0/2814415958 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa2e80a21d0 0x7fa2e80a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.018+0000 7fa301fc6640 1 --2- 172.21.15.23:0/2814415958 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa2e80a21d0 0x7fa2e80a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.018+0000 7fa2f2ffd640 1 -- 172.21.15.23:0/2814415958 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa2f80efed0 con 0x7fa2fc102900 2024-03-24T10:39:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.018+0000 7fa301fc6640 1 --2- 172.21.15.23:0/2814415958 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa2e80a21d0 0x7fa2e80a4690 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc102760 tx=0x7fa2ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:31.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.022+0000 7fa2f2ffd640 1 -- 172.21.15.23:0/2814415958 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa2f80b8e30 con 0x7fa2fc102900 2024-03-24T10:39:31.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.203+0000 7fa30422a640 1 -- 172.21.15.23:0/2814415958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa2fc02d130 con 0x7fa2fc102900 2024-03-24T10:39:31.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.203+0000 7fa2f2ffd640 1 -- 172.21.15.23:0/2814415958 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fa2f80bc4f0 con 0x7fa2fc102900 2024-03-24T10:39:31.204 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-24T10:39:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.205+0000 7fa30422a640 1 -- 172.21.15.23:0/2814415958 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa2e80a21d0 msgr2=0x7fa2e80a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.205+0000 7fa30422a640 1 --2- 172.21.15.23:0/2814415958 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa2e80a21d0 0x7fa2e80a4690 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc102760 tx=0x7fa2ec005f90 comp rx=0 tx=0).stop 2024-03-24T10:39:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.205+0000 7fa30422a640 1 -- 172.21.15.23:0/2814415958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc102900 msgr2=0x7fa2fc197cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.205+0000 7fa30422a640 1 --2- 172.21.15.23:0/2814415958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc102900 0x7fa2fc197cd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8006010 tx=0x7fa2f8005e80 comp rx=0 tx=0).stop 2024-03-24T10:39:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.205+0000 7fa30422a640 1 -- 172.21.15.23:0/2814415958 shutdown_connections 2024-03-24T10:39:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.205+0000 7fa30422a640 1 --2- 172.21.15.23:0/2814415958 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fa2e80a21d0 0x7fa2e80a4690 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.205+0000 7fa30422a640 1 --2- 172.21.15.23:0/2814415958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2fc1032c0 0x7fa2fc19ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.205+0000 7fa30422a640 1 --2- 172.21.15.23:0/2814415958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc102900 0x7fa2fc197cd0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.205+0000 7fa30422a640 1 --2- 172.21.15.23:0/2814415958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2fc101710 0x7fa2fc197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.205+0000 7fa30422a640 1 -- 172.21.15.23:0/2814415958 >> 172.21.15.23:0/2814415958 conn(0x7fa2fc0fcc80 msgr2=0x7fa2fc105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:31.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.205+0000 7fa30422a640 1 -- 172.21.15.23:0/2814415958 shutdown_connections 2024-03-24T10:39:31.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.205+0000 7fa30422a640 1 -- 172.21.15.23:0/2814415958 wait complete. 2024-03-24T10:39:31.217 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:39:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.319+0000 7f52f1e8f640 1 -- 172.21.15.23:0/1853654895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52ec101820 msgr2=0x7f52ec101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.319+0000 7f52f1e8f640 1 --2- 172.21.15.23:0/1853654895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52ec101820 0x7f52ec101ca0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f52d4009900 tx=0x7f52d402bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.319+0000 7f52f1e8f640 1 -- 172.21.15.23:0/1853654895 shutdown_connections 2024-03-24T10:39:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.319+0000 7f52f1e8f640 1 --2- 172.21.15.23:0/1853654895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ec1021e0 0x7f52ec108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.319+0000 7f52f1e8f640 1 --2- 172.21.15.23:0/1853654895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52ec101820 0x7f52ec101ca0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.319+0000 7f52f1e8f640 1 --2- 172.21.15.23:0/1853654895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52ec100630 0x7f52ec100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.319+0000 7f52f1e8f640 1 -- 172.21.15.23:0/1853654895 >> 172.21.15.23:0/1853654895 conn(0x7f52ec0fbba0 msgr2=0x7f52ec0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.319+0000 7f52f1e8f640 1 -- 172.21.15.23:0/1853654895 shutdown_connections 2024-03-24T10:39:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.319+0000 7f52f1e8f640 1 -- 172.21.15.23:0/1853654895 wait complete. 2024-03-24T10:39:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.320+0000 7f52f1e8f640 1 Processor -- start 2024-03-24T10:39:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.320+0000 7f52f1e8f640 1 -- start start 2024-03-24T10:39:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.320+0000 7f52f1e8f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52ec100630 0x7f52ec069400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.320+0000 7f52f1e8f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52ec101820 0x7f52ec19c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.320+0000 7f52f1e8f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ec1021e0 0x7f52ec19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.320+0000 7f52f1e8f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec19f2c0 con 0x7f52ec100630 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.320+0000 7f52f1e8f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec19f430 con 0x7f52ec1021e0 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.320+0000 7f52f1e8f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec19f5f0 con 0x7f52ec101820 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.320+0000 7f52eb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52ec100630 0x7f52ec069400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.320+0000 7f52eb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52ec100630 0x7f52ec069400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38136/0 (socket says 172.21.15.23:38136) 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.320+0000 7f52eb7fe640 1 -- 172.21.15.23:0/2903411418 learned_addr learned my addr 172.21.15.23:0/2903411418 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.320+0000 7f52eaffd640 1 --2- 172.21.15.23:0/2903411418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52ec101820 0x7f52ec19c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.320+0000 7f52ebfff640 1 --2- 172.21.15.23:0/2903411418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ec1021e0 0x7f52ec19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.321+0000 7f52eb7fe640 1 -- 172.21.15.23:0/2903411418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52ec101820 msgr2=0x7f52ec19c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.321+0000 7f52eb7fe640 1 --2- 172.21.15.23:0/2903411418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52ec101820 0x7f52ec19c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.321+0000 7f52eb7fe640 1 -- 172.21.15.23:0/2903411418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ec1021e0 msgr2=0x7f52ec19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.321+0000 7f52eb7fe640 1 --2- 172.21.15.23:0/2903411418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ec1021e0 0x7f52ec19eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.321+0000 7f52eb7fe640 1 -- 172.21.15.23:0/2903411418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d4009560 con 0x7f52ec100630 2024-03-24T10:39:31.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.321+0000 7f52eb7fe640 1 --2- 172.21.15.23:0/2903411418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52ec100630 0x7f52ec069400 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00eb70 tx=0x7f52dc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.321+0000 7f52e8ff9640 1 -- 172.21.15.23:0/2903411418 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc003b70 con 0x7f52ec100630 2024-03-24T10:39:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.321+0000 7f52f1e8f640 1 -- 172.21.15.23:0/2903411418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52ec19f8d0 con 0x7f52ec100630 2024-03-24T10:39:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.321+0000 7f52e8ff9640 1 -- 172.21.15.23:0/2903411418 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52dc00ee70 con 0x7f52ec100630 2024-03-24T10:39:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.321+0000 7f52e8ff9640 1 -- 172.21.15.23:0/2903411418 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc0107f0 con 0x7f52ec100630 2024-03-24T10:39:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.321+0000 7f52f1e8f640 1 -- 172.21.15.23:0/2903411418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52ec19fdb0 con 0x7f52ec100630 2024-03-24T10:39:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.322+0000 7f52f1e8f640 1 -- 172.21.15.23:0/2903411418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52ec107a20 con 0x7f52ec100630 2024-03-24T10:39:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.323+0000 7f52e8ff9640 1 -- 172.21.15.23:0/2903411418 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f52dc004020 con 0x7f52ec100630 2024-03-24T10:39:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.324+0000 7f52e8ff9640 1 --2- 172.21.15.23:0/2903411418 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f52d80a2150 0x7f52d80a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.324+0000 7f52eaffd640 1 --2- 172.21.15.23:0/2903411418 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f52d80a2150 0x7f52d80a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.324+0000 7f52e8ff9640 1 -- 172.21.15.23:0/2903411418 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f52dc014070 con 0x7f52ec100630 2024-03-24T10:39:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.324+0000 7f52eaffd640 1 --2- 172.21.15.23:0/2903411418 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f52d80a2150 0x7f52d80a4610 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f52d4009230 tx=0x7f52d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:31.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.328+0000 7f52e8ff9640 1 -- 172.21.15.23:0/2903411418 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f52dc0972a0 con 0x7f52ec100630 2024-03-24T10:39:31.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.551+0000 7f52f1e8f640 1 -- 172.21.15.23:0/2903411418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52ec10c040 con 0x7f52ec100630 2024-03-24T10:39:31.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.552+0000 7f52e8ff9640 1 -- 172.21.15.23:0/2903411418 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11866 (secure 0 0 0) 0x7f52dc09a960 con 0x7f52ec100630 2024-03-24T10:39:31.554 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:31.554 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:38:15.911086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:39:31.555 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 22 2024-03-24T10:39:31.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.554+0000 7f52f1e8f640 1 -- 172.21.15.23:0/2903411418 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f52d80a2150 msgr2=0x7f52d80a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.554+0000 7f52f1e8f640 1 --2- 172.21.15.23:0/2903411418 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f52d80a2150 0x7f52d80a4610 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f52d4009230 tx=0x7f52d4005fd0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.554+0000 7f52f1e8f640 1 -- 172.21.15.23:0/2903411418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52ec100630 msgr2=0x7f52ec069400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.554+0000 7f52f1e8f640 1 --2- 172.21.15.23:0/2903411418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52ec100630 0x7f52ec069400 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00eb70 tx=0x7f52dc00c6a0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.554+0000 7f52f1e8f640 1 -- 172.21.15.23:0/2903411418 shutdown_connections 2024-03-24T10:39:31.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.554+0000 7f52f1e8f640 1 --2- 172.21.15.23:0/2903411418 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f52d80a2150 0x7f52d80a4610 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.554+0000 7f52f1e8f640 1 --2- 172.21.15.23:0/2903411418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ec1021e0 0x7f52ec19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.554+0000 7f52f1e8f640 1 --2- 172.21.15.23:0/2903411418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52ec101820 0x7f52ec19c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.554+0000 7f52f1e8f640 1 --2- 172.21.15.23:0/2903411418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52ec100630 0x7f52ec069400 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.554+0000 7f52f1e8f640 1 -- 172.21.15.23:0/2903411418 >> 172.21.15.23:0/2903411418 conn(0x7f52ec0fbba0 msgr2=0x7f52ec104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:31.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.554+0000 7f52f1e8f640 1 -- 172.21.15.23:0/2903411418 shutdown_connections 2024-03-24T10:39:31.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.554+0000 7f52f1e8f640 1 -- 172.21.15.23:0/2903411418 wait complete. 2024-03-24T10:39:31.566 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:39:31.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.667+0000 7f216b883640 1 -- 172.21.15.23:0/1249452859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2164100630 msgr2=0x7f2164100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.667+0000 7f216b883640 1 --2- 172.21.15.23:0/1249452859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2164100630 0x7f2164100a50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2160009900 tx=0x7f216002bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:31.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.667+0000 7f216b883640 1 -- 172.21.15.23:0/1249452859 shutdown_connections 2024-03-24T10:39:31.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.667+0000 7f216b883640 1 --2- 172.21.15.23:0/1249452859 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21641021e0 0x7f2164108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f216b883640 1 --2- 172.21.15.23:0/1249452859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2164101820 0x7f2164101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f216b883640 1 --2- 172.21.15.23:0/1249452859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2164100630 0x7f2164100a50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f216b883640 1 -- 172.21.15.23:0/1249452859 >> 172.21.15.23:0/1249452859 conn(0x7f21640fbba0 msgr2=0x7f21640fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:31.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f216b883640 1 -- 172.21.15.23:0/1249452859 shutdown_connections 2024-03-24T10:39:31.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f216b883640 1 -- 172.21.15.23:0/1249452859 wait complete. 2024-03-24T10:39:31.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f216b883640 1 Processor -- start 2024-03-24T10:39:31.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f216b883640 1 -- start start 2024-03-24T10:39:31.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f216b883640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2164100630 0x7f21641977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f216b883640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2164101820 0x7f2164197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f216b883640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21641021e0 0x7f216419ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f216b883640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f216419f2b0 con 0x7f21641021e0 2024-03-24T10:39:31.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f216b883640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f216419f420 con 0x7f2164101820 2024-03-24T10:39:31.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f216b883640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f216419f5e0 con 0x7f2164100630 2024-03-24T10:39:31.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f2169e20640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21641021e0 0x7f216419ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f2169e20640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21641021e0 0x7f216419ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38142/0 (socket says 172.21.15.23:38142) 2024-03-24T10:39:31.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f2169e20640 1 -- 172.21.15.23:0/3823684575 learned_addr learned my addr 172.21.15.23:0/3823684575 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:31.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f2168e1e640 1 --2- 172.21.15.23:0/3823684575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2164101820 0x7f2164197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.668+0000 7f216961f640 1 --2- 172.21.15.23:0/3823684575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2164100630 0x7f21641977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.669+0000 7f2169e20640 1 -- 172.21.15.23:0/3823684575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2164100630 msgr2=0x7f21641977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.669+0000 7f2169e20640 1 --2- 172.21.15.23:0/3823684575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2164100630 0x7f21641977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.669+0000 7f2169e20640 1 -- 172.21.15.23:0/3823684575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2164101820 msgr2=0x7f2164197d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.669+0000 7f2169e20640 1 --2- 172.21.15.23:0/3823684575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2164101820 0x7f2164197d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.669+0000 7f2169e20640 1 -- 172.21.15.23:0/3823684575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2154009490 con 0x7f21641021e0 2024-03-24T10:39:31.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.669+0000 7f2169e20640 1 --2- 172.21.15.23:0/3823684575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21641021e0 0x7f216419ed70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f215c00eb10 tx=0x7f215c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:31.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.669+0000 7f215a7fc640 1 -- 172.21.15.23:0/3823684575 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c004b20 con 0x7f21641021e0 2024-03-24T10:39:31.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.669+0000 7f215a7fc640 1 -- 172.21.15.23:0/3823684575 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f215c004c80 con 0x7f21641021e0 2024-03-24T10:39:31.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.669+0000 7f215a7fc640 1 -- 172.21.15.23:0/3823684575 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c010650 con 0x7f21641021e0 2024-03-24T10:39:31.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.669+0000 7f216b883640 1 -- 172.21.15.23:0/3823684575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2160009560 con 0x7f21641021e0 2024-03-24T10:39:31.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.669+0000 7f216b883640 1 -- 172.21.15.23:0/3823684575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f216419fc40 con 0x7f21641021e0 2024-03-24T10:39:31.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.670+0000 7f216b883640 1 -- 172.21.15.23:0/3823684575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f216410c040 con 0x7f21641021e0 2024-03-24T10:39:31.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.671+0000 7f215a7fc640 1 -- 172.21.15.23:0/3823684575 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f215c005020 con 0x7f21641021e0 2024-03-24T10:39:31.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.671+0000 7f215a7fc640 1 --2- 172.21.15.23:0/3823684575 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f21500a2150 0x7f21500a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.672+0000 7f216961f640 1 --2- 172.21.15.23:0/3823684575 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f21500a2150 0x7f21500a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.672+0000 7f215a7fc640 1 -- 172.21.15.23:0/3823684575 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f215c014070 con 0x7f21641021e0 2024-03-24T10:39:31.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.672+0000 7f216961f640 1 --2- 172.21.15.23:0/3823684575 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f21500a2150 0x7f21500a4610 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2160005ab0 tx=0x7f2160005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:31.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.676+0000 7f215a7fc640 1 -- 172.21.15.23:0/3823684575 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f215c0971f0 con 0x7f21641021e0 2024-03-24T10:39:31.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.862+0000 7f216b883640 1 -- 172.21.15.23:0/3823684575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f21641a0140 con 0x7f21641021e0 2024-03-24T10:39:31.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.863+0000 7f215a7fc640 1 -- 172.21.15.23:0/3823684575 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22704 (secure 0 0 0) 0x7f215c09a8b0 con 0x7f21641021e0 2024-03-24T10:39:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:31.864 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:39:16.692832+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:39:31.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.866+0000 7f216b883640 1 -- 172.21.15.23:0/3823684575 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f21500a2150 msgr2=0x7f21500a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.866+0000 7f216b883640 1 --2- 172.21.15.23:0/3823684575 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f21500a2150 0x7f21500a4610 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2160005ab0 tx=0x7f2160005fd0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.866+0000 7f216b883640 1 -- 172.21.15.23:0/3823684575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21641021e0 msgr2=0x7f216419ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.866+0000 7f216b883640 1 --2- 172.21.15.23:0/3823684575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21641021e0 0x7f216419ed70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f215c00eb10 tx=0x7f215c00c2d0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.866+0000 7f216b883640 1 -- 172.21.15.23:0/3823684575 shutdown_connections 2024-03-24T10:39:31.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.866+0000 7f216b883640 1 --2- 172.21.15.23:0/3823684575 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f21500a2150 0x7f21500a4610 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.866+0000 7f216b883640 1 --2- 172.21.15.23:0/3823684575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21641021e0 0x7f216419ed70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.866+0000 7f216b883640 1 --2- 172.21.15.23:0/3823684575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2164101820 0x7f2164197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.866+0000 7f216b883640 1 --2- 172.21.15.23:0/3823684575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2164100630 0x7f21641977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.866+0000 7f216b883640 1 -- 172.21.15.23:0/3823684575 >> 172.21.15.23:0/3823684575 conn(0x7f21640fbba0 msgr2=0x7f2164068870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:31.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.866+0000 7f216b883640 1 -- 172.21.15.23:0/3823684575 shutdown_connections 2024-03-24T10:39:31.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.866+0000 7f216b883640 1 -- 172.21.15.23:0/3823684575 wait complete. 2024-03-24T10:39:31.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:31 smithi023 ceph-mon[32466]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 19 B/s, 0 objects/s recovering 2024-03-24T10:39:31.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:31 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/16886079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:39:31.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:31 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2814415958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:39:31.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:31 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2903411418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:31.878 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:39.056452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:38:08.473472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:38:09.414667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:39:31.878 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:39:31.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.983+0000 7f76cd18a640 1 -- 172.21.15.23:0/4277485487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76c8101800 msgr2=0x7f76c8101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.983+0000 7f76cd18a640 1 --2- 172.21.15.23:0/4277485487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76c8101800 0x7f76c8101c80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f76bc009900 tx=0x7f76bc02bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:31.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.983+0000 7f76cd18a640 1 -- 172.21.15.23:0/4277485487 shutdown_connections 2024-03-24T10:39:31.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.983+0000 7f76cd18a640 1 --2- 172.21.15.23:0/4277485487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76c81021c0 0x7f76c8108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.983+0000 7f76cd18a640 1 --2- 172.21.15.23:0/4277485487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76c8101800 0x7f76c8101c80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.983+0000 7f76cd18a640 1 --2- 172.21.15.23:0/4277485487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c8100610 0x7f76c8100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.983+0000 7f76cd18a640 1 -- 172.21.15.23:0/4277485487 >> 172.21.15.23:0/4277485487 conn(0x7f76c80fbba0 msgr2=0x7f76c80fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:31.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.983+0000 7f76cd18a640 1 -- 172.21.15.23:0/4277485487 shutdown_connections 2024-03-24T10:39:31.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.983+0000 7f76cd18a640 1 -- 172.21.15.23:0/4277485487 wait complete. 2024-03-24T10:39:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.983+0000 7f76cd18a640 1 Processor -- start 2024-03-24T10:39:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76cd18a640 1 -- start start 2024-03-24T10:39:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76cd18a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76c8100610 0x7f76c806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76cd18a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76c8101800 0x7f76c819c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76cd18a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c81021c0 0x7f76c819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76cd18a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f76c819f260 con 0x7f76c8101800 2024-03-24T10:39:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76cd18a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f76c819f3d0 con 0x7f76c81021c0 2024-03-24T10:39:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76cd18a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f76c819f590 con 0x7f76c8100610 2024-03-24T10:39:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76c659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76c8101800 0x7f76c819c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76c659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76c8101800 0x7f76c819c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38166/0 (socket says 172.21.15.23:38166) 2024-03-24T10:39:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76c659c640 1 -- 172.21.15.23:0/547895623 learned_addr learned my addr 172.21.15.23:0/547895623 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76c6d9d640 1 --2- 172.21.15.23:0/547895623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76c8100610 0x7f76c806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76c759e640 1 --2- 172.21.15.23:0/547895623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c81021c0 0x7f76c819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76c659c640 1 -- 172.21.15.23:0/547895623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76c8100610 msgr2=0x7f76c806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76c659c640 1 --2- 172.21.15.23:0/547895623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76c8100610 0x7f76c806a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76c659c640 1 -- 172.21.15.23:0/547895623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c81021c0 msgr2=0x7f76c819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76c659c640 1 --2- 172.21.15.23:0/547895623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c81021c0 0x7f76c819ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76c659c640 1 -- 172.21.15.23:0/547895623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c0009840 con 0x7f76c8101800 2024-03-24T10:39:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76c659c640 1 --2- 172.21.15.23:0/547895623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76c8101800 0x7f76c819c490 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f76bc006010 tx=0x7f76bc005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.984+0000 7f76affff640 1 -- 172.21.15.23:0/547895623 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc0043d0 con 0x7f76c8101800 2024-03-24T10:39:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.985+0000 7f76cd18a640 1 -- 172.21.15.23:0/547895623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76bc009560 con 0x7f76c8101800 2024-03-24T10:39:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.985+0000 7f76affff640 1 -- 172.21.15.23:0/547895623 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76bc0052d0 con 0x7f76c8101800 2024-03-24T10:39:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.985+0000 7f76affff640 1 -- 172.21.15.23:0/547895623 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc003bc0 con 0x7f76c8101800 2024-03-24T10:39:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.985+0000 7f76cd18a640 1 -- 172.21.15.23:0/547895623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76c819fbe0 con 0x7f76c8101800 2024-03-24T10:39:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.986+0000 7f76cd18a640 1 -- 172.21.15.23:0/547895623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76c8068950 con 0x7f76c8101800 2024-03-24T10:39:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.987+0000 7f76affff640 1 -- 172.21.15.23:0/547895623 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f76bc003d40 con 0x7f76c8101800 2024-03-24T10:39:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.987+0000 7f76affff640 1 --2- 172.21.15.23:0/547895623 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f76b00a2150 0x7f76b00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.987+0000 7f76c6d9d640 1 --2- 172.21.15.23:0/547895623 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f76b00a2150 0x7f76b00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.988+0000 7f76affff640 1 -- 172.21.15.23:0/547895623 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f76bc0eff60 con 0x7f76c8101800 2024-03-24T10:39:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.988+0000 7f76c6d9d640 1 --2- 172.21.15.23:0/547895623 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f76b00a2150 0x7f76b00a4610 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f76c8101660 tx=0x7f76b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:31.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:31.992+0000 7f76affff640 1 -- 172.21.15.23:0/547895623 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f76bc0b8ec0 con 0x7f76c8101800 2024-03-24T10:39:32.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:31 smithi116 ceph-mon[36248]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 19 B/s, 0 objects/s recovering 2024-03-24T10:39:32.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:31 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/16886079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:39:32.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:31 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2814415958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:39:32.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:31 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2903411418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:32.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.173+0000 7f76cd18a640 1 -- 172.21.15.23:0/547895623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f76c802d130 con 0x7f76c8101800 2024-03-24T10:39:32.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.173+0000 7f76affff640 1 -- 172.21.15.23:0/547895623 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f76bc0bc580 con 0x7f76c8101800 2024-03-24T10:39:32.174 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-24T10:39:32.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.175+0000 7f76cd18a640 1 -- 172.21.15.23:0/547895623 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f76b00a2150 msgr2=0x7f76b00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.175+0000 7f76cd18a640 1 --2- 172.21.15.23:0/547895623 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f76b00a2150 0x7f76b00a4610 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f76c8101660 tx=0x7f76b8005f90 comp rx=0 tx=0).stop 2024-03-24T10:39:32.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.175+0000 7f76cd18a640 1 -- 172.21.15.23:0/547895623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76c8101800 msgr2=0x7f76c819c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.175+0000 7f76cd18a640 1 --2- 172.21.15.23:0/547895623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76c8101800 0x7f76c819c490 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f76bc006010 tx=0x7f76bc005e80 comp rx=0 tx=0).stop 2024-03-24T10:39:32.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.175+0000 7f76cd18a640 1 -- 172.21.15.23:0/547895623 shutdown_connections 2024-03-24T10:39:32.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.175+0000 7f76cd18a640 1 --2- 172.21.15.23:0/547895623 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f76b00a2150 0x7f76b00a4610 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.175+0000 7f76cd18a640 1 --2- 172.21.15.23:0/547895623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76c81021c0 0x7f76c819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.175+0000 7f76cd18a640 1 --2- 172.21.15.23:0/547895623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76c8101800 0x7f76c819c490 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.175+0000 7f76cd18a640 1 --2- 172.21.15.23:0/547895623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76c8100610 0x7f76c806a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.175+0000 7f76cd18a640 1 -- 172.21.15.23:0/547895623 >> 172.21.15.23:0/547895623 conn(0x7f76c80fbba0 msgr2=0x7f76c8104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:32.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.175+0000 7f76cd18a640 1 -- 172.21.15.23:0/547895623 shutdown_connections 2024-03-24T10:39:32.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.175+0000 7f76cd18a640 1 -- 172.21.15.23:0/547895623 wait complete. 2024-03-24T10:39:32.187 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:39:32.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:31 smithi117 ceph-mon[36368]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 19 B/s, 0 objects/s recovering 2024-03-24T10:39:32.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:31 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/16886079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:39:32.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:31 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2814415958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:39:32.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:31 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2903411418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:32.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.292+0000 7f81e9e57640 1 -- 172.21.15.23:0/3254130233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e4101720 msgr2=0x7f81e4101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.292+0000 7f81e9e57640 1 --2- 172.21.15.23:0/3254130233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e4101720 0x7f81e4101b40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f81d4009be0 tx=0x7f81d402bef0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.292+0000 7f81e9e57640 1 -- 172.21.15.23:0/3254130233 shutdown_connections 2024-03-24T10:39:32.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.292+0000 7f81e9e57640 1 --2- 172.21.15.23:0/3254130233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81e41032d0 0x7f81e4109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.292+0000 7f81e9e57640 1 --2- 172.21.15.23:0/3254130233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81e4102910 0x7f81e4102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.292+0000 7f81e9e57640 1 --2- 172.21.15.23:0/3254130233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e4101720 0x7f81e4101b40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.292+0000 7f81e9e57640 1 -- 172.21.15.23:0/3254130233 >> 172.21.15.23:0/3254130233 conn(0x7f81e40fccb0 msgr2=0x7f81e40ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:32.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.292+0000 7f81e9e57640 1 -- 172.21.15.23:0/3254130233 shutdown_connections 2024-03-24T10:39:32.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.292+0000 7f81e9e57640 1 -- 172.21.15.23:0/3254130233 wait complete. 2024-03-24T10:39:32.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.292+0000 7f81e9e57640 1 Processor -- start 2024-03-24T10:39:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e9e57640 1 -- start start 2024-03-24T10:39:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e9e57640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81e4101720 0x7f81e4197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e9e57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e4102910 0x7f81e4197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e9e57640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81e41032d0 0x7f81e419edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e9e57640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f81e419f330 con 0x7f81e41032d0 2024-03-24T10:39:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e9e57640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f81e419f4a0 con 0x7f81e4102910 2024-03-24T10:39:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e9e57640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f81e419f660 con 0x7f81e4101720 2024-03-24T10:39:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81e41032d0 0x7f81e419edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e37fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81e4101720 0x7f81e4197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81e41032d0 0x7f81e419edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38188/0 (socket says 172.21.15.23:38188) 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e37fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81e4101720 0x7f81e4197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:60266/0 (socket says 172.21.15.23:60266) 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e3fff640 1 -- 172.21.15.23:0/3943818747 learned_addr learned my addr 172.21.15.23:0/3943818747 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e2ffd640 1 --2- 172.21.15.23:0/3943818747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e4102910 0x7f81e4197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e3fff640 1 -- 172.21.15.23:0/3943818747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81e4101720 msgr2=0x7f81e4197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e3fff640 1 --2- 172.21.15.23:0/3943818747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81e4101720 0x7f81e4197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e3fff640 1 -- 172.21.15.23:0/3943818747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e4102910 msgr2=0x7f81e4197d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e3fff640 1 --2- 172.21.15.23:0/3943818747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e4102910 0x7f81e4197d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e3fff640 1 -- 172.21.15.23:0/3943818747 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d4009840 con 0x7f81e41032d0 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e3fff640 1 --2- 172.21.15.23:0/3943818747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81e41032d0 0x7f81e419edf0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00e970 tx=0x7f81dc00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.293+0000 7f81e0ff9640 1 -- 172.21.15.23:0/3943818747 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc00ccc0 con 0x7f81e41032d0 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.294+0000 7f81e9e57640 1 -- 172.21.15.23:0/3943818747 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e419f940 con 0x7f81e41032d0 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.294+0000 7f81e0ff9640 1 -- 172.21.15.23:0/3943818747 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81dc00ce20 con 0x7f81e41032d0 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.294+0000 7f81e0ff9640 1 -- 172.21.15.23:0/3943818747 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc010430 con 0x7f81e41032d0 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.294+0000 7f81e9e57640 1 -- 172.21.15.23:0/3943818747 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e419fe00 con 0x7f81e41032d0 2024-03-24T10:39:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.295+0000 7f81e9e57640 1 -- 172.21.15.23:0/3943818747 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e4108ae0 con 0x7f81e41032d0 2024-03-24T10:39:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.296+0000 7f81e0ff9640 1 -- 172.21.15.23:0/3943818747 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f81dc005000 con 0x7f81e41032d0 2024-03-24T10:39:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.296+0000 7f81e0ff9640 1 --2- 172.21.15.23:0/3943818747 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f81d00a2150 0x7f81d00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.296+0000 7f81e37fe640 1 --2- 172.21.15.23:0/3943818747 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f81d00a2150 0x7f81d00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.296+0000 7f81e0ff9640 1 -- 172.21.15.23:0/3943818747 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f81dc014070 con 0x7f81e41032d0 2024-03-24T10:39:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.297+0000 7f81e37fe640 1 --2- 172.21.15.23:0/3943818747 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f81d00a2150 0x7f81d00a4610 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f81d402c500 tx=0x7f81d40333e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:32.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.300+0000 7f81e0ff9640 1 -- 172.21.15.23:0/3943818747 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f81dc097240 con 0x7f81e41032d0 2024-03-24T10:39:32.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.481+0000 7f81e9e57640 1 -- 172.21.15.23:0/3943818747 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f81e402d1d0 con 0x7f81e41032d0 2024-03-24T10:39:32.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.482+0000 7f81e0ff9640 1 -- 172.21.15.23:0/3943818747 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f81dc0052b0 con 0x7f81e41032d0 2024-03-24T10:39:32.482 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-24T10:39:32.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.484+0000 7f81e9e57640 1 -- 172.21.15.23:0/3943818747 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f81d00a2150 msgr2=0x7f81d00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.484+0000 7f81e9e57640 1 --2- 172.21.15.23:0/3943818747 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f81d00a2150 0x7f81d00a4610 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f81d402c500 tx=0x7f81d40333e0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.484+0000 7f81e9e57640 1 -- 172.21.15.23:0/3943818747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81e41032d0 msgr2=0x7f81e419edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.484+0000 7f81e9e57640 1 --2- 172.21.15.23:0/3943818747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81e41032d0 0x7f81e419edf0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00e970 tx=0x7f81dc00ee30 comp rx=0 tx=0).stop 2024-03-24T10:39:32.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.484+0000 7f81e9e57640 1 -- 172.21.15.23:0/3943818747 shutdown_connections 2024-03-24T10:39:32.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.484+0000 7f81e9e57640 1 --2- 172.21.15.23:0/3943818747 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f81d00a2150 0x7f81d00a4610 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.484+0000 7f81e9e57640 1 --2- 172.21.15.23:0/3943818747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81e41032d0 0x7f81e419edf0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.484+0000 7f81e9e57640 1 --2- 172.21.15.23:0/3943818747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e4102910 0x7f81e4197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.484+0000 7f81e9e57640 1 --2- 172.21.15.23:0/3943818747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81e4101720 0x7f81e4197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.484+0000 7f81e9e57640 1 -- 172.21.15.23:0/3943818747 >> 172.21.15.23:0/3943818747 conn(0x7f81e40fccb0 msgr2=0x7f81e4105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:32.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.484+0000 7f81e9e57640 1 -- 172.21.15.23:0/3943818747 shutdown_connections 2024-03-24T10:39:32.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.484+0000 7f81e9e57640 1 -- 172.21.15.23:0/3943818747 wait complete. 2024-03-24T10:39:32.495 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:39:32.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.600+0000 7ff834c40640 1 -- 172.21.15.23:0/2830065777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff830100610 msgr2=0x7ff830100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.600+0000 7ff834c40640 1 --2- 172.21.15.23:0/2830065777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff830100610 0x7ff830100a30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff828009900 tx=0x7ff82802bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:32.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.600+0000 7ff834c40640 1 -- 172.21.15.23:0/2830065777 shutdown_connections 2024-03-24T10:39:32.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.600+0000 7ff834c40640 1 --2- 172.21.15.23:0/2830065777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8301021c0 0x7ff830108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.600+0000 7ff834c40640 1 --2- 172.21.15.23:0/2830065777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff830101800 0x7ff830101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.600+0000 7ff834c40640 1 --2- 172.21.15.23:0/2830065777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff830100610 0x7ff830100a30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.600+0000 7ff834c40640 1 -- 172.21.15.23:0/2830065777 >> 172.21.15.23:0/2830065777 conn(0x7ff8300fbba0 msgr2=0x7ff8300fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:32.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.600+0000 7ff834c40640 1 -- 172.21.15.23:0/2830065777 shutdown_connections 2024-03-24T10:39:32.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.600+0000 7ff834c40640 1 -- 172.21.15.23:0/2830065777 wait complete. 2024-03-24T10:39:32.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff834c40640 1 Processor -- start 2024-03-24T10:39:32.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff834c40640 1 -- start start 2024-03-24T10:39:32.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff834c40640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff830100610 0x7ff830197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:32.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff834c40640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff830101800 0x7ff830197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:32.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff834c40640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8301021c0 0x7ff83019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:32.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff834c40640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff83019f2f0 con 0x7ff8301021c0 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff834c40640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff83019f460 con 0x7ff830101800 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff834c40640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff83019f620 con 0x7ff830100610 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff82ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8301021c0 0x7ff83019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff82ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8301021c0 0x7ff83019edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38214/0 (socket says 172.21.15.23:38214) 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff82ed9d640 1 -- 172.21.15.23:0/3525050772 learned_addr learned my addr 172.21.15.23:0/3525050772 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff82e59c640 1 --2- 172.21.15.23:0/3525050772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff830100610 0x7ff830197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff82dd9b640 1 --2- 172.21.15.23:0/3525050772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff830101800 0x7ff830197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff82ed9d640 1 -- 172.21.15.23:0/3525050772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff830100610 msgr2=0x7ff830197800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff82ed9d640 1 --2- 172.21.15.23:0/3525050772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff830100610 0x7ff830197800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff82ed9d640 1 -- 172.21.15.23:0/3525050772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff830101800 msgr2=0x7ff830197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff82ed9d640 1 --2- 172.21.15.23:0/3525050772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff830101800 0x7ff830197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff82ed9d640 1 -- 172.21.15.23:0/3525050772 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff818009490 con 0x7ff8301021c0 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.601+0000 7ff82ed9d640 1 --2- 172.21.15.23:0/3525050772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8301021c0 0x7ff83019edb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff82000efd0 tx=0x7ff82000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.602+0000 7ff8277fe640 1 -- 172.21.15.23:0/3525050772 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff820009b40 con 0x7ff8301021c0 2024-03-24T10:39:32.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.602+0000 7ff8277fe640 1 -- 172.21.15.23:0/3525050772 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff820003b70 con 0x7ff8301021c0 2024-03-24T10:39:32.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.602+0000 7ff834c40640 1 -- 172.21.15.23:0/3525050772 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff820009840 con 0x7ff8301021c0 2024-03-24T10:39:32.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.602+0000 7ff8277fe640 1 -- 172.21.15.23:0/3525050772 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8200108f0 con 0x7ff8301021c0 2024-03-24T10:39:32.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.602+0000 7ff834c40640 1 -- 172.21.15.23:0/3525050772 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff828009560 con 0x7ff8301021c0 2024-03-24T10:39:32.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.603+0000 7ff834c40640 1 -- 172.21.15.23:0/3525050772 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff830107a00 con 0x7ff8301021c0 2024-03-24T10:39:32.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.604+0000 7ff8277fe640 1 -- 172.21.15.23:0/3525050772 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff820003ce0 con 0x7ff8301021c0 2024-03-24T10:39:32.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.604+0000 7ff8277fe640 1 --2- 172.21.15.23:0/3525050772 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff81c0a21d0 0x7ff81c0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:32.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.604+0000 7ff82e59c640 1 --2- 172.21.15.23:0/3525050772 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff81c0a21d0 0x7ff81c0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:32.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.604+0000 7ff8277fe640 1 -- 172.21.15.23:0/3525050772 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7ff820014070 con 0x7ff8301021c0 2024-03-24T10:39:32.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.605+0000 7ff82e59c640 1 --2- 172.21.15.23:0/3525050772 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff81c0a21d0 0x7ff81c0a4690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff828005ab0 tx=0x7ff828005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:32.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.608+0000 7ff8277fe640 1 -- 172.21.15.23:0/3525050772 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff820097570 con 0x7ff8301021c0 2024-03-24T10:39:32.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.794+0000 7ff834c40640 1 -- 172.21.15.23:0/3525050772 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff83002d160 con 0x7ff8301021c0 2024-03-24T10:39:32.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.794+0000 7ff8277fe640 1 -- 172.21.15.23:0/3525050772 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7ff82009ac30 con 0x7ff8301021c0 2024-03-24T10:39:32.795 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-24T10:39:32.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.796+0000 7ff834c40640 1 -- 172.21.15.23:0/3525050772 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff81c0a21d0 msgr2=0x7ff81c0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.796+0000 7ff834c40640 1 --2- 172.21.15.23:0/3525050772 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff81c0a21d0 0x7ff81c0a4690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff828005ab0 tx=0x7ff828005fd0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.796+0000 7ff834c40640 1 -- 172.21.15.23:0/3525050772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8301021c0 msgr2=0x7ff83019edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.796+0000 7ff834c40640 1 --2- 172.21.15.23:0/3525050772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8301021c0 0x7ff83019edb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff82000efd0 tx=0x7ff82000c6a0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.796+0000 7ff834c40640 1 -- 172.21.15.23:0/3525050772 shutdown_connections 2024-03-24T10:39:32.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.796+0000 7ff834c40640 1 --2- 172.21.15.23:0/3525050772 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff81c0a21d0 0x7ff81c0a4690 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.796+0000 7ff834c40640 1 --2- 172.21.15.23:0/3525050772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8301021c0 0x7ff83019edb0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.796+0000 7ff834c40640 1 --2- 172.21.15.23:0/3525050772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff830101800 0x7ff830197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.796+0000 7ff834c40640 1 --2- 172.21.15.23:0/3525050772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff830100610 0x7ff830197800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.796+0000 7ff834c40640 1 -- 172.21.15.23:0/3525050772 >> 172.21.15.23:0/3525050772 conn(0x7ff8300fbba0 msgr2=0x7ff830104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:32.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.796+0000 7ff834c40640 1 -- 172.21.15.23:0/3525050772 shutdown_connections 2024-03-24T10:39:32.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.797+0000 7ff834c40640 1 -- 172.21.15.23:0/3525050772 wait complete. 2024-03-24T10:39:32.808 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:39:32.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.910+0000 7f215cdc9640 1 -- 172.21.15.23:0/1255930872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21581032e0 msgr2=0x7f2158109b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.910+0000 7f215cdc9640 1 --2- 172.21.15.23:0/1255930872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21581032e0 0x7f2158109b70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2150009ca0 tx=0x7f215002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.910+0000 7f215cdc9640 1 -- 172.21.15.23:0/1255930872 shutdown_connections 2024-03-24T10:39:32.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.910+0000 7f215cdc9640 1 --2- 172.21.15.23:0/1255930872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21581032e0 0x7f2158109b70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.910+0000 7f215cdc9640 1 --2- 172.21.15.23:0/1255930872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2158102920 0x7f2158102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.910+0000 7f215cdc9640 1 --2- 172.21.15.23:0/1255930872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2158101730 0x7f2158101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.910+0000 7f215cdc9640 1 -- 172.21.15.23:0/1255930872 >> 172.21.15.23:0/1255930872 conn(0x7f21580fccc0 msgr2=0x7f21580ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.910+0000 7f215cdc9640 1 -- 172.21.15.23:0/1255930872 shutdown_connections 2024-03-24T10:39:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.910+0000 7f215cdc9640 1 -- 172.21.15.23:0/1255930872 wait complete. 2024-03-24T10:39:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.910+0000 7f215cdc9640 1 Processor -- start 2024-03-24T10:39:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215cdc9640 1 -- start start 2024-03-24T10:39:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215cdc9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2158101730 0x7f21581977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215cdc9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2158102920 0x7f2158197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215cdc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21581032e0 0x7f215819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215cdc9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f215819f2b0 con 0x7f2158101730 2024-03-24T10:39:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215cdc9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f215819f420 con 0x7f21581032e0 2024-03-24T10:39:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215cdc9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f215819f5e0 con 0x7f2158102920 2024-03-24T10:39:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2158101730 0x7f21581977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2158101730 0x7f21581977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38222/0 (socket says 172.21.15.23:38222) 2024-03-24T10:39:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215659c640 1 -- 172.21.15.23:0/2988870182 learned_addr learned my addr 172.21.15.23:0/2988870182 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f2155d9b640 1 --2- 172.21.15.23:0/2988870182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2158102920 0x7f2158197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f2156d9d640 1 --2- 172.21.15.23:0/2988870182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21581032e0 0x7f215819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215659c640 1 -- 172.21.15.23:0/2988870182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2158102920 msgr2=0x7f2158197d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215659c640 1 --2- 172.21.15.23:0/2988870182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2158102920 0x7f2158197d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215659c640 1 -- 172.21.15.23:0/2988870182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21581032e0 msgr2=0x7f215819ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215659c640 1 --2- 172.21.15.23:0/2988870182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21581032e0 0x7f215819ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215659c640 1 -- 172.21.15.23:0/2988870182 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2150009900 con 0x7f2158101730 2024-03-24T10:39:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215659c640 1 --2- 172.21.15.23:0/2988870182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2158101730 0x7f21581977c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f214000e9c0 tx=0x7f214000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f214f7fe640 1 -- 172.21.15.23:0/2988870182 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2140009700 con 0x7f2158101730 2024-03-24T10:39:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f215cdc9640 1 -- 172.21.15.23:0/2988870182 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f215819f8c0 con 0x7f2158101730 2024-03-24T10:39:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.911+0000 7f214f7fe640 1 -- 172.21.15.23:0/2988870182 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f214000cd80 con 0x7f2158101730 2024-03-24T10:39:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.912+0000 7f214f7fe640 1 -- 172.21.15.23:0/2988870182 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2140010430 con 0x7f2158101730 2024-03-24T10:39:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.912+0000 7f215cdc9640 1 -- 172.21.15.23:0/2988870182 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f215819fe20 con 0x7f2158101730 2024-03-24T10:39:32.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.912+0000 7f215cdc9640 1 -- 172.21.15.23:0/2988870182 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21581089f0 con 0x7f2158101730 2024-03-24T10:39:32.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.913+0000 7f214f7fe640 1 -- 172.21.15.23:0/2988870182 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f2140003680 con 0x7f2158101730 2024-03-24T10:39:32.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.914+0000 7f214f7fe640 1 --2- 172.21.15.23:0/2988870182 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f21440a2150 0x7f21440a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:32.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.914+0000 7f2155d9b640 1 --2- 172.21.15.23:0/2988870182 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f21440a2150 0x7f21440a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:32.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.914+0000 7f214f7fe640 1 -- 172.21.15.23:0/2988870182 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2140014070 con 0x7f2158101730 2024-03-24T10:39:32.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.914+0000 7f2155d9b640 1 --2- 172.21.15.23:0/2988870182 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f21440a2150 0x7f21440a4610 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2158198cd0 tx=0x7f2148005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:32.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:32.918+0000 7f214f7fe640 1 -- 172.21.15.23:0/2988870182 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2140097050 con 0x7f2158101730 2024-03-24T10:39:33.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:32 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3823684575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:33.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:32 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/547895623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:39:33.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:32 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3943818747' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:39:33.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.138+0000 7f215cdc9640 1 -- 172.21.15.23:0/2988870182 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f215810d110 con 0x7f2158101730 2024-03-24T10:39:33.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.139+0000 7f214f7fe640 1 -- 172.21.15.23:0/2988870182 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11866 (secure 0 0 0) 0x7f214009a710 con 0x7f2158101730 2024-03-24T10:39:33.139 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:33.140 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:38:15.911086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:39:33.140 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 22 2024-03-24T10:39:33.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.141+0000 7f215cdc9640 1 -- 172.21.15.23:0/2988870182 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f21440a2150 msgr2=0x7f21440a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.141+0000 7f215cdc9640 1 --2- 172.21.15.23:0/2988870182 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f21440a2150 0x7f21440a4610 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2158198cd0 tx=0x7f2148005f90 comp rx=0 tx=0).stop 2024-03-24T10:39:33.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.141+0000 7f215cdc9640 1 -- 172.21.15.23:0/2988870182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2158101730 msgr2=0x7f21581977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.141+0000 7f215cdc9640 1 --2- 172.21.15.23:0/2988870182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2158101730 0x7f21581977c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f214000e9c0 tx=0x7f214000ee80 comp rx=0 tx=0).stop 2024-03-24T10:39:33.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.141+0000 7f215cdc9640 1 -- 172.21.15.23:0/2988870182 shutdown_connections 2024-03-24T10:39:33.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.141+0000 7f215cdc9640 1 --2- 172.21.15.23:0/2988870182 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f21440a2150 0x7f21440a4610 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.141+0000 7f215cdc9640 1 --2- 172.21.15.23:0/2988870182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21581032e0 0x7f215819ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.141+0000 7f215cdc9640 1 --2- 172.21.15.23:0/2988870182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2158102920 0x7f2158197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.141+0000 7f215cdc9640 1 --2- 172.21.15.23:0/2988870182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2158101730 0x7f21581977c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.141+0000 7f215cdc9640 1 -- 172.21.15.23:0/2988870182 >> 172.21.15.23:0/2988870182 conn(0x7f21580fccc0 msgr2=0x7f2158105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:33.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.141+0000 7f215cdc9640 1 -- 172.21.15.23:0/2988870182 shutdown_connections 2024-03-24T10:39:33.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.141+0000 7f215cdc9640 1 -- 172.21.15.23:0/2988870182 wait complete. 2024-03-24T10:39:33.152 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:32 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3823684575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:33.152 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:32 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/547895623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:39:33.152 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:32 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3943818747' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:39:33.153 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:39:33.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:32 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3823684575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:33.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:32 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/547895623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:39:33.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:32 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3943818747' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:39:33.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.255+0000 7f88f655e640 1 -- 172.21.15.23:0/1916063434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f0100630 msgr2=0x7f88f0100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.255+0000 7f88f655e640 1 --2- 172.21.15.23:0/1916063434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f0100630 0x7f88f0100a50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f88e0009900 tx=0x7f88e002bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:33.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.255+0000 7f88f655e640 1 -- 172.21.15.23:0/1916063434 shutdown_connections 2024-03-24T10:39:33.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.255+0000 7f88f655e640 1 --2- 172.21.15.23:0/1916063434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f01021e0 0x7f88f0108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.255+0000 7f88f655e640 1 --2- 172.21.15.23:0/1916063434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0101820 0x7f88f0101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.255+0000 7f88f655e640 1 --2- 172.21.15.23:0/1916063434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f0100630 0x7f88f0100a50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.255+0000 7f88f655e640 1 -- 172.21.15.23:0/1916063434 >> 172.21.15.23:0/1916063434 conn(0x7f88f00fbba0 msgr2=0x7f88f00fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.255+0000 7f88f655e640 1 -- 172.21.15.23:0/1916063434 shutdown_connections 2024-03-24T10:39:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.255+0000 7f88f655e640 1 -- 172.21.15.23:0/1916063434 wait complete. 2024-03-24T10:39:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.255+0000 7f88f655e640 1 Processor -- start 2024-03-24T10:39:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.255+0000 7f88f655e640 1 -- start start 2024-03-24T10:39:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88f655e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f0100630 0x7f88f0069400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88f655e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f0101820 0x7f88f019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88f655e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f01021e0 0x7f88f019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88f655e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f88f019f2e0 con 0x7f88f0101820 2024-03-24T10:39:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88f655e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f88f019f450 con 0x7f88f0100630 2024-03-24T10:39:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88f655e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88f019f610 con 0x7f88f01021e0 2024-03-24T10:39:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88ef7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f0101820 0x7f88f019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88ef7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f0101820 0x7f88f019c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38234/0 (socket says 172.21.15.23:38234) 2024-03-24T10:39:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88ef7fe640 1 -- 172.21.15.23:0/4251830600 learned_addr learned my addr 172.21.15.23:0/4251830600 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88effff640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f0100630 0x7f88f0069400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88f4afb640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f01021e0 0x7f88f019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88ef7fe640 1 -- 172.21.15.23:0/4251830600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f01021e0 msgr2=0x7f88f019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88ef7fe640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f01021e0 0x7f88f019eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88ef7fe640 1 -- 172.21.15.23:0/4251830600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f0100630 msgr2=0x7f88f0069400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88ef7fe640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f0100630 0x7f88f0069400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88ef7fe640 1 -- 172.21.15.23:0/4251830600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e0009560 con 0x7f88f0101820 2024-03-24T10:39:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88effff640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f0100630 0x7f88f0069400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88ef7fe640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f0101820 0x7f88f019c510 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f88e400e9f0 tx=0x7f88e400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.256+0000 7f88ed7fa640 1 -- 172.21.15.23:0/4251830600 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4009720 con 0x7f88f0101820 2024-03-24T10:39:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.257+0000 7f88ed7fa640 1 -- 172.21.15.23:0/4251830600 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e400cd80 con 0x7f88f0101820 2024-03-24T10:39:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.257+0000 7f88f655e640 1 -- 172.21.15.23:0/4251830600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f019f8f0 con 0x7f88f0101820 2024-03-24T10:39:33.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.257+0000 7f88ed7fa640 1 -- 172.21.15.23:0/4251830600 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4010690 con 0x7f88f0101820 2024-03-24T10:39:33.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.257+0000 7f88f655e640 1 -- 172.21.15.23:0/4251830600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f019fdb0 con 0x7f88f0101820 2024-03-24T10:39:33.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.257+0000 7f88f655e640 1 -- 172.21.15.23:0/4251830600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f01079a0 con 0x7f88f0101820 2024-03-24T10:39:33.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.258+0000 7f88ed7fa640 1 -- 172.21.15.23:0/4251830600 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f88e40046c0 con 0x7f88f0101820 2024-03-24T10:39:33.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.259+0000 7f88ed7fa640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f88d80a2150 0x7f88d80a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.259+0000 7f88effff640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f88d80a2150 0x7f88d80a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.259+0000 7f88ed7fa640 1 -- 172.21.15.23:0/4251830600 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f88e4014070 con 0x7f88f0101820 2024-03-24T10:39:33.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.260+0000 7f88effff640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f88d80a2150 0x7f88d80a4610 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f88e0005ab0 tx=0x7f88e0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:33.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.264+0000 7f88ed7fa640 1 -- 172.21.15.23:0/4251830600 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f88e4097010 con 0x7f88f0101820 2024-03-24T10:39:33.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.446+0000 7f88f655e640 1 -- 172.21.15.23:0/4251830600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88f010c290 con 0x7f88f0101820 2024-03-24T10:39:33.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.447+0000 7f88ed7fa640 1 -- 172.21.15.23:0/4251830600 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22704 (secure 0 0 0) 0x7f88e409a6d0 con 0x7f88f0101820 2024-03-24T10:39:33.448 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:33.448 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:39:16.692832+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:39:33.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.449+0000 7f88f655e640 1 -- 172.21.15.23:0/4251830600 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f88d80a2150 msgr2=0x7f88d80a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.449+0000 7f88f655e640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f88d80a2150 0x7f88d80a4610 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f88e0005ab0 tx=0x7f88e0005fd0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.449+0000 7f88f655e640 1 -- 172.21.15.23:0/4251830600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f0101820 msgr2=0x7f88f019c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.449+0000 7f88f655e640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f0101820 0x7f88f019c510 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f88e400e9f0 tx=0x7f88e400edf0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.450+0000 7f88f655e640 1 -- 172.21.15.23:0/4251830600 shutdown_connections 2024-03-24T10:39:33.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.450+0000 7f88f655e640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f88d80a2150 0x7f88d80a4610 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.450+0000 7f88f655e640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f01021e0 0x7f88f019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.450+0000 7f88f655e640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f0101820 0x7f88f019c510 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.450+0000 7f88f655e640 1 --2- 172.21.15.23:0/4251830600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f0100630 0x7f88f0069400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.450+0000 7f88f655e640 1 -- 172.21.15.23:0/4251830600 >> 172.21.15.23:0/4251830600 conn(0x7f88f00fbba0 msgr2=0x7f88f0104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:33.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.450+0000 7f88f655e640 1 -- 172.21.15.23:0/4251830600 shutdown_connections 2024-03-24T10:39:33.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.450+0000 7f88f655e640 1 -- 172.21.15.23:0/4251830600 wait complete. 2024-03-24T10:39:33.462 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:39:33.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.564+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2562324190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70e4102260 msgr2=0x7f70e4108af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.564+0000 7f70e9b0c640 1 --2- 172.21.15.23:0/2562324190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70e4102260 0x7f70e4108af0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009ca0 tx=0x7f70dc02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.564+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2562324190 shutdown_connections 2024-03-24T10:39:33.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.564+0000 7f70e9b0c640 1 --2- 172.21.15.23:0/2562324190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70e4102260 0x7f70e4108af0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.564+0000 7f70e9b0c640 1 --2- 172.21.15.23:0/2562324190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e41018a0 0x7f70e4101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.564+0000 7f70e9b0c640 1 --2- 172.21.15.23:0/2562324190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70e41006b0 0x7f70e4100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.564+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2562324190 >> 172.21.15.23:0/2562324190 conn(0x7f70e40fbc40 msgr2=0x7f70e40fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:33.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.564+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2562324190 shutdown_connections 2024-03-24T10:39:33.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.564+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2562324190 wait complete. 2024-03-24T10:39:33.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e9b0c640 1 Processor -- start 2024-03-24T10:39:33.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e9b0c640 1 -- start start 2024-03-24T10:39:33.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e9b0c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70e41006b0 0x7f70e410af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e9b0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e41018a0 0x7f70e41095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e9b0c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70e4102260 0x7f70e4109af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e9b0c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f70e410a250 con 0x7f70e4102260 2024-03-24T10:39:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e9b0c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f70e410a3c0 con 0x7f70e41018a0 2024-03-24T10:39:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e9b0c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f70e41a17b0 con 0x7f70e41006b0 2024-03-24T10:39:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70e4102260 0x7f70e4109af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e41018a0 0x7f70e41095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70e4102260 0x7f70e4109af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38242/0 (socket says 172.21.15.23:38242) 2024-03-24T10:39:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e3fff640 1 -- 172.21.15.23:0/2296015724 learned_addr learned my addr 172.21.15.23:0/2296015724 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e37fe640 1 --2- 172.21.15.23:0/2296015724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70e41006b0 0x7f70e410af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e3fff640 1 -- 172.21.15.23:0/2296015724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70e41006b0 msgr2=0x7f70e410af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e3fff640 1 --2- 172.21.15.23:0/2296015724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70e41006b0 0x7f70e410af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e3fff640 1 -- 172.21.15.23:0/2296015724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e41018a0 msgr2=0x7f70e41095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e3fff640 1 --2- 172.21.15.23:0/2296015724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e41018a0 0x7f70e41095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e3fff640 1 -- 172.21.15.23:0/2296015724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70dc009900 con 0x7f70e4102260 2024-03-24T10:39:33.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e2ffd640 1 --2- 172.21.15.23:0/2296015724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e41018a0 0x7f70e41095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:33.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.565+0000 7f70e3fff640 1 --2- 172.21.15.23:0/2296015724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70e4102260 0x7f70e4109af0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009bf0 tx=0x7f70dc032f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:33.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.566+0000 7f70e0ff9640 1 -- 172.21.15.23:0/2296015724 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70dc034e20 con 0x7f70e4102260 2024-03-24T10:39:33.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.566+0000 7f70e0ff9640 1 -- 172.21.15.23:0/2296015724 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70dc003e50 con 0x7f70e4102260 2024-03-24T10:39:33.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.566+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2296015724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70e41a1a00 con 0x7f70e4102260 2024-03-24T10:39:33.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.566+0000 7f70e0ff9640 1 -- 172.21.15.23:0/2296015724 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70dc004840 con 0x7f70e4102260 2024-03-24T10:39:33.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.566+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2296015724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70e41a1eb0 con 0x7f70e4102260 2024-03-24T10:39:33.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.567+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2296015724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70e4107970 con 0x7f70e4102260 2024-03-24T10:39:33.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.569+0000 7f70e0ff9640 1 -- 172.21.15.23:0/2296015724 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f70dc00a040 con 0x7f70e4102260 2024-03-24T10:39:33.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.569+0000 7f70e0ff9640 1 --2- 172.21.15.23:0/2296015724 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f70cc0a2150 0x7f70cc0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.569+0000 7f70e37fe640 1 --2- 172.21.15.23:0/2296015724 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f70cc0a2150 0x7f70cc0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.569+0000 7f70e0ff9640 1 -- 172.21.15.23:0/2296015724 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f70dc0f0f10 con 0x7f70e4102260 2024-03-24T10:39:33.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.569+0000 7f70e37fe640 1 --2- 172.21.15.23:0/2296015724 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f70cc0a2150 0x7f70cc0a4610 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f70e4101700 tx=0x7f70d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:33.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.573+0000 7f70e0ff9640 1 -- 172.21.15.23:0/2296015724 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f70dc0b9f90 con 0x7f70e4102260 2024-03-24T10:39:33.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.796+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2296015724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70e410d150 con 0x7f70e4102260 2024-03-24T10:39:33.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.797+0000 7f70e0ff9640 1 -- 172.21.15.23:0/2296015724 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11866 (secure 0 0 0) 0x7f70dc044090 con 0x7f70e4102260 2024-03-24T10:39:33.798 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:33.798 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:38:15.911086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:39:33.798 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 22 2024-03-24T10:39:33.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.799+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2296015724 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f70cc0a2150 msgr2=0x7f70cc0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.799+0000 7f70e9b0c640 1 --2- 172.21.15.23:0/2296015724 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f70cc0a2150 0x7f70cc0a4610 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f70e4101700 tx=0x7f70d8005f90 comp rx=0 tx=0).stop 2024-03-24T10:39:33.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.799+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2296015724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70e4102260 msgr2=0x7f70e4109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.799+0000 7f70e9b0c640 1 --2- 172.21.15.23:0/2296015724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70e4102260 0x7f70e4109af0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009bf0 tx=0x7f70dc032f80 comp rx=0 tx=0).stop 2024-03-24T10:39:33.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.799+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2296015724 shutdown_connections 2024-03-24T10:39:33.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.799+0000 7f70e9b0c640 1 --2- 172.21.15.23:0/2296015724 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f70cc0a2150 0x7f70cc0a4610 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.799+0000 7f70e9b0c640 1 --2- 172.21.15.23:0/2296015724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70e4102260 0x7f70e4109af0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.799+0000 7f70e9b0c640 1 --2- 172.21.15.23:0/2296015724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e41018a0 0x7f70e41095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.799+0000 7f70e9b0c640 1 --2- 172.21.15.23:0/2296015724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70e41006b0 0x7f70e410af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.799+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2296015724 >> 172.21.15.23:0/2296015724 conn(0x7f70e40fbc40 msgr2=0x7f70e4104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:33.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.799+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2296015724 shutdown_connections 2024-03-24T10:39:33.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.799+0000 7f70e9b0c640 1 -- 172.21.15.23:0/2296015724 wait complete. 2024-03-24T10:39:33.812 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-24T10:38:10.206899+0000', 'modified': '2024-03-24T10:38:15.911086+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24539, 'mds_1': 24505, 'mds_2': 24503}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24470': {'gid': 24470, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6835/1535852767', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6834', 'nonce': 1535852767}, {'type': 'v1', 'addr': '172.21.15.116:6835', 'nonce': 1535852767}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.117:6835/1664408853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 1664408853}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 1664408853}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.117:6837/977076441', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6836', 'nonce': 977076441}, {'type': 'v1', 'addr': '172.21.15.117:6837', 'nonce': 977076441}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.116:6837/3576031724', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6836', 'nonce': 3576031724}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 3576031724}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.117:6839/263724286', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6838', 'nonce': 263724286}, {'type': 'v1', 'addr': '172.21.15.117:6839', 'nonce': 263724286}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24539': {'gid': 24539, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.116:6839/2435082224', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 2435082224}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 2435082224}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24539, 'qdb_cluster': [24503, 24505, 24539]} 2024-03-24T10:39:33.812 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-24T10:39:33.812 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-24T10:39:33.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.915+0000 7f224995e640 1 -- 172.21.15.23:0/1668554161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2244102200 msgr2=0x7f22441045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.915+0000 7f224995e640 1 --2- 172.21.15.23:0/1668554161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2244102200 0x7f22441045f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2234009900 tx=0x7f223402bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:33.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.915+0000 7f224995e640 1 -- 172.21.15.23:0/1668554161 shutdown_connections 2024-03-24T10:39:33.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.915+0000 7f224995e640 1 --2- 172.21.15.23:0/1668554161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2244107460 0x7f2244109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.915+0000 7f224995e640 1 --2- 172.21.15.23:0/1668554161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2244104b30 0x7f2244106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.915+0000 7f224995e640 1 --2- 172.21.15.23:0/1668554161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2244102200 0x7f22441045f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.915+0000 7f224995e640 1 -- 172.21.15.23:0/1668554161 >> 172.21.15.23:0/1668554161 conn(0x7f22440fbbb0 msgr2=0x7f22440fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:33.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.915+0000 7f224995e640 1 -- 172.21.15.23:0/1668554161 shutdown_connections 2024-03-24T10:39:33.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.915+0000 7f224995e640 1 -- 172.21.15.23:0/1668554161 wait complete. 2024-03-24T10:39:33.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f224995e640 1 Processor -- start 2024-03-24T10:39:33.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f224995e640 1 -- start start 2024-03-24T10:39:33.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f224995e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2244102200 0x7f2244197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f224995e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2244104b30 0x7f2244197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f224995e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2244107460 0x7f224419edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f224995e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f224419f300 con 0x7f2244107460 2024-03-24T10:39:33.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f224995e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f224419f470 con 0x7f2244104b30 2024-03-24T10:39:33.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f224995e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f224419f630 con 0x7f2244102200 2024-03-24T10:39:33.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f22437fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2244107460 0x7f224419edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f2242ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2244102200 0x7f2244197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f223bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2244104b30 0x7f2244197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f22437fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2244107460 0x7f224419edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38258/0 (socket says 172.21.15.23:38258) 2024-03-24T10:39:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f2242ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2244102200 0x7f2244197810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:60372/0 (socket says 172.21.15.23:60372) 2024-03-24T10:39:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f22437fe640 1 -- 172.21.15.23:0/1812033471 learned_addr learned my addr 172.21.15.23:0/1812033471 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f22437fe640 1 -- 172.21.15.23:0/1812033471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2244102200 msgr2=0x7f2244197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f22437fe640 1 --2- 172.21.15.23:0/1812033471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2244102200 0x7f2244197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f22437fe640 1 -- 172.21.15.23:0/1812033471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2244104b30 msgr2=0x7f2244197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f22437fe640 1 --2- 172.21.15.23:0/1812033471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2244104b30 0x7f2244197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f22437fe640 1 -- 172.21.15.23:0/1812033471 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2234009560 con 0x7f2244107460 2024-03-24T10:39:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f223bfff640 1 --2- 172.21.15.23:0/1812033471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2244104b30 0x7f2244197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f22437fe640 1 --2- 172.21.15.23:0/1812033471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2244107460 0x7f224419edc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f223c00eb10 tx=0x7f223c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.916+0000 7f224895c640 1 -- 172.21.15.23:0/1812033471 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223c004b20 con 0x7f2244107460 2024-03-24T10:39:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.917+0000 7f224995e640 1 -- 172.21.15.23:0/1812033471 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f224419f910 con 0x7f2244107460 2024-03-24T10:39:33.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.917+0000 7f224895c640 1 -- 172.21.15.23:0/1812033471 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f223c004c80 con 0x7f2244107460 2024-03-24T10:39:33.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.917+0000 7f224895c640 1 -- 172.21.15.23:0/1812033471 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223c010650 con 0x7f2244107460 2024-03-24T10:39:33.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.917+0000 7f224995e640 1 -- 172.21.15.23:0/1812033471 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f224419fe20 con 0x7f2244107460 2024-03-24T10:39:33.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.919+0000 7f224895c640 1 -- 172.21.15.23:0/1812033471 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f223c005020 con 0x7f2244107460 2024-03-24T10:39:33.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.919+0000 7f224c61d740 1 Processor -- start 2024-03-24T10:39:33.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.919+0000 7f224c61d740 1 -- start start 2024-03-24T10:39:33.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.919+0000 7f224895c640 1 --2- 172.21.15.23:0/1812033471 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f22300a2150 0x7f22300a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.919+0000 7f224c61d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a5e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.919+0000 7f224c61d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a63a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.919+0000 7f224c61d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a66550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.919+0000 7f224c61d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f0919cd180 con 0x55f091a5ef50 2024-03-24T10:39:33.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.919+0000 7f224c61d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f0919ca6b0 con 0x55f091a640c0 2024-03-24T10:39:33.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.919+0000 7f224c61d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f0919c5e90 con 0x55f091a5e560 2024-03-24T10:39:33.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.919+0000 7f223bfff640 1 --2- 172.21.15.23:0/1812033471 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f22300a2150 0x7f22300a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.919+0000 7f224895c640 1 -- 172.21.15.23:0/1812033471 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f223c014070 con 0x7f2244107460 2024-03-24T10:39:33.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f223bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a63a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f2242ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a5e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f223bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a63a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38260/0 (socket says 172.21.15.23:38260) 2024-03-24T10:39:33.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f223bfff640 1 -- 172.21.15.23:0/2321498824 learned_addr learned my addr 172.21.15.23:0/2321498824 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:33.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f22437fe640 1 --2- 172.21.15.23:0/2321498824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a66550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f2242ffd640 1 -- 172.21.15.23:0/2321498824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 msgr2=0x55f091a66550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:33.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f2242ffd640 1 --2- 172.21.15.23:0/2321498824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a66550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f2242ffd640 1 -- 172.21.15.23:0/2321498824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 msgr2=0x55f091a63a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:33.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f2242ffd640 1 --2- 172.21.15.23:0/2321498824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a63a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f2242ffd640 1 -- 172.21.15.23:0/2321498824 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f091982d80 con 0x55f091a5e560 2024-03-24T10:39:33.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f223bfff640 1 --2- 172.21.15.23:0/2321498824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a63a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:33.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f223bfff640 1 --2- 172.21.15.23:0/1812033471 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f22300a2150 0x7f22300a4610 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f222c009fd0 tx=0x7f222c009f20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:33.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f2242ffd640 1 --2- 172.21.15.23:0/2321498824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a5e980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2234000c00 tx=0x7f2234005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdd16747a664d4c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:33.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f2239ffb640 1 -- 172.21.15.23:0/2321498824 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223402ca20 con 0x55f091a5e560 2024-03-24T10:39:33.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f2239ffb640 1 -- 172.21.15.23:0/2321498824 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f223402cb80 con 0x55f091a5e560 2024-03-24T10:39:33.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.920+0000 7f2239ffb640 1 -- 172.21.15.23:0/2321498824 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234005050 con 0x55f091a5e560 2024-03-24T10:39:33.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.921+0000 7f224c61d740 1 -- 172.21.15.23:0/2321498824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 msgr2=0x55f091a5e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.921+0000 7f224c61d740 1 --2- 172.21.15.23:0/2321498824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a5e980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2234000c00 tx=0x7f2234005d50 comp rx=0 tx=0).stop 2024-03-24T10:39:33.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.921+0000 7f224c61d740 1 -- 172.21.15.23:0/2321498824 shutdown_connections 2024-03-24T10:39:33.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.921+0000 7f224c61d740 1 --2- 172.21.15.23:0/2321498824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a66550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.921+0000 7f224c61d740 1 --2- 172.21.15.23:0/2321498824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a63a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.921+0000 7f224c61d740 1 --2- 172.21.15.23:0/2321498824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a5e980 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.921+0000 7f224c61d740 1 -- 172.21.15.23:0/2321498824 >> 172.21.15.23:0/2321498824 conn(0x55f091a59ad0 msgr2=0x55f091a59ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:33.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.921+0000 7f224c61d740 1 -- 172.21.15.23:0/2321498824 shutdown_connections 2024-03-24T10:39:33.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.921+0000 7f224c61d740 1 -- 172.21.15.23:0/2321498824 wait complete. 2024-03-24T10:39:33.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.921+0000 7f224c61d740 1 -- start start 2024-03-24T10:39:33.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.921+0000 7f224c61d740 1 Processor -- start 2024-03-24T10:39:33.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f224c61d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a684b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f224c61d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a689f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f224c61d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a6fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f224c61d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f09196ad50 con 0x55f091a5ef50 2024-03-24T10:39:33.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f224c61d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f091699310 con 0x55f091a640c0 2024-03-24T10:39:33.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f224c61d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f09194b8f0 con 0x55f091a5e560 2024-03-24T10:39:33.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f224c61d740 10 client.0 _ll_register_callbacks cb 0x55f0916d6ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:39:33.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f223bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a689f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f223bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a689f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38266/0 (socket says 172.21.15.23:38266) 2024-03-24T10:39:33.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f223bfff640 1 -- 172.21.15.23:0/2947851013 learned_addr learned my addr 172.21.15.23:0/2947851013 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:33.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f2242ffd640 1 --2- 172.21.15.23:0/2947851013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a684b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f22437fe640 1 --2- 172.21.15.23:0/2947851013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a6fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f223bfff640 1 -- 172.21.15.23:0/2947851013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 msgr2=0x55f091a684b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:33.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f223bfff640 1 --2- 172.21.15.23:0/2947851013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a684b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f223bfff640 1 -- 172.21.15.23:0/2947851013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 msgr2=0x55f091a6fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:33.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f223bfff640 1 --2- 172.21.15.23:0/2947851013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a6fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f223bfff640 1 -- 172.21.15.23:0/2947851013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f224410a430 con 0x55f091a5ef50 2024-03-24T10:39:33.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f223bfff640 1 --2- 172.21.15.23:0/2947851013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a689f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f222c01e930 tx=0x7f222c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac36030d6b496ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:33.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f2222ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:39:33.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f2222ffd640 1 -- 172.21.15.23:0/2947851013 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c016070 con 0x55f091a5ef50 2024-03-24T10:39:33.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f2222ffd640 1 -- 172.21.15.23:0/2947851013 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f222c009b50 con 0x55f091a5ef50 2024-03-24T10:39:33.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f224c61d740 1 -- 172.21.15.23:0/2947851013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f0916c56c0 con 0x55f091a5ef50 2024-03-24T10:39:33.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.922+0000 7f2222ffd640 1 -- 172.21.15.23:0/2947851013 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f222c0208f0 con 0x55f091a5ef50 2024-03-24T10:39:33.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.923+0000 7f224c61d740 10 client.15060 fetch_fsmap learned FSMap version 22 2024-03-24T10:39:33.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.923+0000 7f224c61d740 1 -- 172.21.15.23:0/2947851013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f0919c4510 con 0x55f091a5ef50 2024-03-24T10:39:33.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.923+0000 7f2222ffd640 1 -- 172.21.15.23:0/2947851013 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f0916c56c0 con 0x55f091a5ef50 2024-03-24T10:39:33.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.923+0000 7f2222ffd640 1 -- 172.21.15.23:0/2947851013 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f222c02aa60 con 0x55f091a5ef50 2024-03-24T10:39:33.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.923+0000 7f224c61d740 10 client.15060 fetch_fsmap finished waiting for FSMap version 22 2024-03-24T10:39:33.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.923+0000 7f224c61d740 10 client.15060 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-24T10:39:33.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.923+0000 7f224c61d740 20 client.15060 populate_metadata read hostname 'smithi023' 2024-03-24T10:39:33.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.923+0000 7f224c61d740 1 --2- 172.21.15.23:0/2947851013 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f091a74d60 0x55f091a761d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.923+0000 7f224c61d740 4 client.15060 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:39:33.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.923+0000 7f224c61d740 1 -- 172.21.15.23:0/2947851013 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f091716270 con 0x55f091a74d60 2024-03-24T10:39:33.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.923+0000 7f2242ffd640 1 --2- 172.21.15.23:0/2947851013 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f091a74d60 0x55f091a761d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.923+0000 7f2242ffd640 1 --2- 172.21.15.23:0/2947851013 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f091a74d60 0x55f091a761d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f88da87bd62df8ef server_cookie=14c85bbd88dcd43f in_seq=0 out_seq=0 2024-03-24T10:39:33.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.923+0000 7f2222ffd640 10 client.15060 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:39:33.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f2222ffd640 1 -- 172.21.15.23:0/2947851013 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f091716270 con 0x55f091a74d60 2024-03-24T10:39:33.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f2222ffd640 10 client.15060 handle_command_reply: tid=0 2024-03-24T10:39:33.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f224c61d740 1 client.15060 shutdown 2024-03-24T10:39:33.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f224c61d740 1 -- 172.21.15.23:0/2947851013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 msgr2=0x55f091a689f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f224c61d740 1 --2- 172.21.15.23:0/2947851013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a689f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f222c01e930 tx=0x7f222c005f90 comp rx=0 tx=0).stop 2024-03-24T10:39:33.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f224c61d740 1 -- 172.21.15.23:0/2947851013 shutdown_connections 2024-03-24T10:39:33.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f224c61d740 1 --2- 172.21.15.23:0/2947851013 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f091a74d60 0x55f091a761d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f224c61d740 1 --2- 172.21.15.23:0/2947851013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a6fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f224c61d740 1 --2- 172.21.15.23:0/2947851013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a689f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f224c61d740 1 --2- 172.21.15.23:0/2947851013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a684b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f2222ffd640 0 client.15060 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:39:33.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f224c61d740 1 -- 172.21.15.23:0/2947851013 >> 172.21.15.23:0/2947851013 conn(0x55f091a59ad0 msgr2=0x55f091a58ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:33.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f224c61d740 1 -- 172.21.15.23:0/2947851013 shutdown_connections 2024-03-24T10:39:33.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f224c61d740 1 -- 172.21.15.23:0/2947851013 wait complete. 2024-03-24T10:39:33.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.925+0000 7f224c61d740 20 client.15060 trim_cache size 0 max 16384 2024-03-24T10:39:33.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.943+0000 7f224c61d740 1 Processor -- start 2024-03-24T10:39:33.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.943+0000 7f224c61d740 1 -- start start 2024-03-24T10:39:33.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.943+0000 7f224c61d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a73d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.943+0000 7f224c61d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a6f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.943+0000 7f224c61d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a59eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.943+0000 7f224c61d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f091a75140 con 0x55f091a5ef50 2024-03-24T10:39:33.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.943+0000 7f224c61d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f091716270 con 0x55f091a640c0 2024-03-24T10:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.943+0000 7f224c61d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f0916c56c0 con 0x55f091a5e560 2024-03-24T10:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.943+0000 7f223bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a6f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.943+0000 7f223bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a6f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38268/0 (socket says 172.21.15.23:38268) 2024-03-24T10:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.943+0000 7f223bfff640 1 -- 172.21.15.23:0/272172626 learned_addr learned my addr 172.21.15.23:0/272172626 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.943+0000 7f2242ffd640 1 --2- 172.21.15.23:0/272172626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a73d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.943+0000 7f22437fe640 1 --2- 172.21.15.23:0/272172626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a59eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.943+0000 7f223bfff640 1 -- 172.21.15.23:0/272172626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 msgr2=0x55f091a73d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.944+0000 7f223bfff640 1 --2- 172.21.15.23:0/272172626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a73d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.944+0000 7f223bfff640 1 -- 172.21.15.23:0/272172626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 msgr2=0x55f091a59eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.944+0000 7f223bfff640 1 --2- 172.21.15.23:0/272172626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a59eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.944+0000 7f223bfff640 1 -- 172.21.15.23:0/272172626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f091960a80 con 0x55f091a5ef50 2024-03-24T10:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.944+0000 7f22437fe640 1 --2- 172.21.15.23:0/272172626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a59eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.944+0000 7f2242ffd640 1 --2- 172.21.15.23:0/272172626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a73d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.944+0000 7f223bfff640 1 --2- 172.21.15.23:0/272172626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a6f9f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f222c00fc90 tx=0x7f222c011990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc71118ad4c357b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.944+0000 7f22397fa640 1 -- 172.21.15.23:0/272172626 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c016040 con 0x55f091a5ef50 2024-03-24T10:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.944+0000 7f22397fa640 1 -- 172.21.15.23:0/272172626 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f222c01e710 con 0x55f091a5ef50 2024-03-24T10:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.944+0000 7f22397fa640 1 -- 172.21.15.23:0/272172626 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c0207c0 con 0x55f091a5ef50 2024-03-24T10:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.944+0000 7f224c61d740 1 -- 172.21.15.23:0/272172626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 msgr2=0x55f091a6f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.944+0000 7f224c61d740 1 --2- 172.21.15.23:0/272172626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a6f9f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f222c00fc90 tx=0x7f222c011990 comp rx=0 tx=0).stop 2024-03-24T10:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 -- 172.21.15.23:0/272172626 shutdown_connections 2024-03-24T10:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 --2- 172.21.15.23:0/272172626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a59eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 --2- 172.21.15.23:0/272172626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a6f9f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 --2- 172.21.15.23:0/272172626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a73d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 -- 172.21.15.23:0/272172626 >> 172.21.15.23:0/272172626 conn(0x55f091a59ad0 msgr2=0x55f091a58ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 -- 172.21.15.23:0/272172626 shutdown_connections 2024-03-24T10:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 -- 172.21.15.23:0/272172626 wait complete. 2024-03-24T10:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 -- start start 2024-03-24T10:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 Processor -- start 2024-03-24T10:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a6b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a6bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a6ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f091a6f120 con 0x55f091a5ef50 2024-03-24T10:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f091a6f290 con 0x55f091a640c0 2024-03-24T10:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f091a6f450 con 0x55f091a5e560 2024-03-24T10:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f224c61d740 10 client.0 _ll_register_callbacks cb 0x7f222c011720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f223bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a6bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f223bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a6bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38284/0 (socket says 172.21.15.23:38284) 2024-03-24T10:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f223bfff640 1 -- 172.21.15.23:0/2015475885 learned_addr learned my addr 172.21.15.23:0/2015475885 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f2242ffd640 1 --2- 172.21.15.23:0/2015475885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a6b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.945+0000 7f22437fe640 1 --2- 172.21.15.23:0/2015475885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a6ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f223bfff640 1 -- 172.21.15.23:0/2015475885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 msgr2=0x55f091a6b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f223bfff640 1 --2- 172.21.15.23:0/2015475885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a6b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f223bfff640 1 -- 172.21.15.23:0/2015475885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 msgr2=0x55f091a6ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f223bfff640 1 --2- 172.21.15.23:0/2015475885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a6ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f223bfff640 1 -- 172.21.15.23:0/2015475885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f091982d80 con 0x55f091a5ef50 2024-03-24T10:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f223bfff640 1 --2- 172.21.15.23:0/2015475885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a6bbc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f222c010010 tx=0x7f222c005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5eaddcf2568c8cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f2222ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f2222ffd640 1 -- 172.21.15.23:0/2015475885 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c045070 con 0x55f091a5ef50 2024-03-24T10:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f2222ffd640 1 -- 172.21.15.23:0/2015475885 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f222c005250 con 0x55f091a5ef50 2024-03-24T10:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f224c61d740 1 -- 172.21.15.23:0/2015475885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f091a54c60 con 0x55f091a5ef50 2024-03-24T10:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f2222ffd640 1 -- 172.21.15.23:0/2015475885 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f222c011b00 con 0x55f091a5ef50 2024-03-24T10:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f224c61d740 10 client.15066 fetch_fsmap learned FSMap version 22 2024-03-24T10:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f224c61d740 1 -- 172.21.15.23:0/2015475885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f0919c4510 con 0x55f091a5ef50 2024-03-24T10:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f2222ffd640 1 -- 172.21.15.23:0/2015475885 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f091a54c60 con 0x55f091a5ef50 2024-03-24T10:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.946+0000 7f2222ffd640 1 -- 172.21.15.23:0/2015475885 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f222c048030 con 0x55f091a5ef50 2024-03-24T10:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 10 client.15066 fetch_fsmap finished waiting for FSMap version 22 2024-03-24T10:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 10 client.15066 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-24T10:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 20 client.15066 populate_metadata read hostname 'smithi023' 2024-03-24T10:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 1 --2- 172.21.15.23:0/2015475885 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f091adf2e0 0x55f091addff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 4 client.15066 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T10:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 1 -- 172.21.15.23:0/2015475885 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "status"}) -- 0x7f223402cb80 con 0x55f091adf2e0 2024-03-24T10:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f2242ffd640 1 --2- 172.21.15.23:0/2015475885 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f091adf2e0 0x55f091addff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f2242ffd640 1 --2- 172.21.15.23:0/2015475885 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f091adf2e0 0x55f091addff0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=141c0c0e2a46fc53 server_cookie=1fff835f00163a95 in_seq=0 out_seq=0 2024-03-24T10:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f2222ffd640 10 client.15066 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f2222ffd640 1 -- 172.21.15.23:0/2015475885 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x7f223402cb80 con 0x55f091adf2e0 2024-03-24T10:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f2222ffd640 10 client.15066 handle_command_reply: tid=0 2024-03-24T10:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 1 client.15066 shutdown 2024-03-24T10:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 1 -- 172.21.15.23:0/2015475885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 msgr2=0x55f091a6bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 1 --2- 172.21.15.23:0/2015475885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a6bbc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f222c010010 tx=0x7f222c005db0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 1 -- 172.21.15.23:0/2015475885 shutdown_connections 2024-03-24T10:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 1 --2- 172.21.15.23:0/2015475885 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f091adf2e0 0x55f091addff0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 1 --2- 172.21.15.23:0/2015475885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f091a640c0 0x55f091a6ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f2222ffd640 0 client.15066 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 1 --2- 172.21.15.23:0/2015475885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f091a5ef50 0x55f091a6bbc0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 1 --2- 172.21.15.23:0/2015475885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f091a5e560 0x55f091a6b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.947+0000 7f224c61d740 1 -- 172.21.15.23:0/2015475885 >> 172.21.15.23:0/2015475885 conn(0x55f091a59ad0 msgr2=0x55f091a58ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224c61d740 1 -- 172.21.15.23:0/2015475885 shutdown_connections 2024-03-24T10:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224c61d740 1 -- 172.21.15.23:0/2015475885 wait complete. 2024-03-24T10:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224c61d740 20 client.15066 trim_cache size 0 max 16384 2024-03-24T10:39:33.955 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:39:33.956 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "4f8fb8f6-e9c9-11ee-95cd-87774f69a715", 2024-03-24T10:39:33.956 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 2, 2024-03-24T10:39:33.956 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24503, 2024-03-24T10:39:33.956 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-24T10:39:33.956 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-24T10:39:33.956 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-24T10:39:33.956 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 79.048077366000001, 2024-03-24T10:39:33.956 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 22, 2024-03-24T10:39:33.956 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 87, 2024-03-24T10:39:33.956 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 85, 2024-03-24T10:39:33.956 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 100.025654139 2024-03-24T10:39:33.956 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:39:33.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224995e640 1 -- 172.21.15.23:0/1812033471 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f22300a2150 msgr2=0x7f22300a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224995e640 1 --2- 172.21.15.23:0/1812033471 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f22300a2150 0x7f22300a4610 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f222c009fd0 tx=0x7f222c009f20 comp rx=0 tx=0).stop 2024-03-24T10:39:33.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224995e640 1 -- 172.21.15.23:0/1812033471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2244107460 msgr2=0x7f224419edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:33.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224995e640 1 --2- 172.21.15.23:0/1812033471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2244107460 0x7f224419edc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f223c00eb10 tx=0x7f223c00c2d0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224995e640 1 -- 172.21.15.23:0/1812033471 shutdown_connections 2024-03-24T10:39:33.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224995e640 1 --2- 172.21.15.23:0/1812033471 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f22300a2150 0x7f22300a4610 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224995e640 1 --2- 172.21.15.23:0/1812033471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2244107460 0x7f224419edc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224995e640 1 --2- 172.21.15.23:0/1812033471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2244104b30 0x7f2244197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224995e640 1 --2- 172.21.15.23:0/1812033471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2244102200 0x7f2244197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:33.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224995e640 1 -- 172.21.15.23:0/1812033471 >> 172.21.15.23:0/1812033471 conn(0x7f22440fbbb0 msgr2=0x7f2244102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:33.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224995e640 1 -- 172.21.15.23:0/1812033471 shutdown_connections 2024-03-24T10:39:33.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:33.948+0000 7f224995e640 1 -- 172.21.15.23:0/1812033471 wait complete. 2024-03-24T10:39:33.959 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-24T10:39:34.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:33 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3525050772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:39:34.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:33 smithi116 ceph-mon[36248]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T10:39:34.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:33 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2988870182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:34.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:33 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4251830600' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:34.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.063+0000 7f7f58c15640 1 -- 172.21.15.23:0/4072823604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f541021d0 msgr2=0x7f7f54108a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.063+0000 7f7f58c15640 1 --2- 172.21.15.23:0/4072823604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f541021d0 0x7f7f54108a60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7f44009ca0 tx=0x7f7f4402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.063+0000 7f7f58c15640 1 -- 172.21.15.23:0/4072823604 shutdown_connections 2024-03-24T10:39:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.063+0000 7f7f58c15640 1 --2- 172.21.15.23:0/4072823604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f541021d0 0x7f7f54108a60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.063+0000 7f7f58c15640 1 --2- 172.21.15.23:0/4072823604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f54101810 0x7f7f54101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.063+0000 7f7f58c15640 1 --2- 172.21.15.23:0/4072823604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f54100620 0x7f7f54100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.063+0000 7f7f58c15640 1 -- 172.21.15.23:0/4072823604 >> 172.21.15.23:0/4072823604 conn(0x7f7f540fbbb0 msgr2=0x7f7f540fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.063+0000 7f7f58c15640 1 -- 172.21.15.23:0/4072823604 shutdown_connections 2024-03-24T10:39:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.063+0000 7f7f58c15640 1 -- 172.21.15.23:0/4072823604 wait complete. 2024-03-24T10:39:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.063+0000 7f7f58c15640 1 Processor -- start 2024-03-24T10:39:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f58c15640 1 -- start start 2024-03-24T10:39:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f58c15640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f54100620 0x7f7f54197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f58c15640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f54101810 0x7f7f54197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f58c15640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f541021d0 0x7f7f5419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f58c15640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5419f270 con 0x7f7f541021d0 2024-03-24T10:39:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f58c15640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5419f3e0 con 0x7f7f54100620 2024-03-24T10:39:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f58c15640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5419f5a0 con 0x7f7f54101810 2024-03-24T10:39:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f52d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f541021d0 0x7f7f5419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f52d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f541021d0 0x7f7f5419ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38292/0 (socket says 172.21.15.23:38292) 2024-03-24T10:39:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f52d9d640 1 -- 172.21.15.23:0/2383094974 learned_addr learned my addr 172.21.15.23:0/2383094974 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f5259c640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f54100620 0x7f7f54197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f54101810 0x7f7f54197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f52d9d640 1 -- 172.21.15.23:0/2383094974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f54101810 msgr2=0x7f7f54197cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f52d9d640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f54101810 0x7f7f54197cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f52d9d640 1 -- 172.21.15.23:0/2383094974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f54100620 msgr2=0x7f7f54197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f52d9d640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f54100620 0x7f7f54197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f52d9d640 1 -- 172.21.15.23:0/2383094974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f3c009490 con 0x7f7f541021d0 2024-03-24T10:39:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f5259c640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f54100620 0x7f7f54197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f52d9d640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f541021d0 0x7f7f5419ed30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7f44037040 tx=0x7f7f44009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f4b7fe640 1 -- 172.21.15.23:0/2383094974 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f440329b0 con 0x7f7f541021d0 2024-03-24T10:39:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f58c15640 1 -- 172.21.15.23:0/2383094974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f44009900 con 0x7f7f541021d0 2024-03-24T10:39:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f4b7fe640 1 -- 172.21.15.23:0/2383094974 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f44003e50 con 0x7f7f541021d0 2024-03-24T10:39:34.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.064+0000 7f7f4b7fe640 1 -- 172.21.15.23:0/2383094974 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f440048a0 con 0x7f7f541021d0 2024-03-24T10:39:34.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.065+0000 7f7f58c15640 1 -- 172.21.15.23:0/2383094974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f5419fb90 con 0x7f7f541021d0 2024-03-24T10:39:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.067+0000 7f7f4b7fe640 1 -- 172.21.15.23:0/2383094974 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f7f4400a040 con 0x7f7f541021d0 2024-03-24T10:39:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.067+0000 7f7f5b7fa740 1 Processor -- start 2024-03-24T10:39:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.067+0000 7f7f5b7fa740 1 -- start start 2024-03-24T10:39:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f5b7fa740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccdd560 0x55f23ccdd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f5b7fa740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccddf50 0x55f23cce2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f5b7fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23cce5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f5b7fa740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f23cc4c180 con 0x55f23ccddf50 2024-03-24T10:39:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f5b7fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f23cc496b0 con 0x55f23cce30c0 2024-03-24T10:39:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f5b7fa740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f23cc44e90 con 0x55f23ccdd560 2024-03-24T10:39:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f4b7fe640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f400a2150 0x7f7f400a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f51d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccdd560 0x55f23ccdd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f5259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccddf50 0x55f23cce2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f51d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccdd560 0x55f23ccdd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:60424/0 (socket says 172.21.15.23:60424) 2024-03-24T10:39:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f5259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccddf50 0x55f23cce2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38304/0 (socket says 172.21.15.23:38304) 2024-03-24T10:39:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f51d9b640 1 -- 172.21.15.23:0/1216716994 learned_addr learned my addr 172.21.15.23:0/1216716994 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/1216716994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23cce5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f52d9d640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f400a2150 0x7f7f400a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f4b7fe640 1 -- 172.21.15.23:0/2383094974 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7f440f0ab0 con 0x7f7f541021d0 2024-03-24T10:39:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f51d9b640 1 -- 172.21.15.23:0/1216716994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 msgr2=0x55f23cce5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/1216716994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23cce5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f51d9b640 1 -- 172.21.15.23:0/1216716994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccddf50 msgr2=0x55f23cce2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/1216716994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccddf50 0x55f23cce2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f51d9b640 1 -- 172.21.15.23:0/1216716994 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f23cc01d80 con 0x55f23ccdd560 2024-03-24T10:39:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f5259c640 1 --2- 172.21.15.23:0/1216716994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccddf50 0x55f23cce2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/1216716994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccdd560 0x55f23ccdd980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c011990 tx=0x7f7f3c009370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a78a3df02baa706b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.068+0000 7f7f52d9d640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f400a2150 0x7f7f400a4610 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7f440bd790 tx=0x7f7f440e7e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.069+0000 7f7f48ff9640 1 -- 172.21.15.23:0/1216716994 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c011020 con 0x55f23ccdd560 2024-03-24T10:39:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.069+0000 7f7f48ff9640 1 -- 172.21.15.23:0/1216716994 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f3c011180 con 0x55f23ccdd560 2024-03-24T10:39:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.069+0000 7f7f48ff9640 1 -- 172.21.15.23:0/1216716994 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c0035f0 con 0x55f23ccdd560 2024-03-24T10:39:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.069+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/1216716994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccdd560 msgr2=0x55f23ccdd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.069+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/1216716994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccdd560 0x55f23ccdd980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c011990 tx=0x7f7f3c009370 comp rx=0 tx=0).stop 2024-03-24T10:39:34.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.069+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/1216716994 shutdown_connections 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.069+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/1216716994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23cce5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.069+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/1216716994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccddf50 0x55f23cce2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.069+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/1216716994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccdd560 0x55f23ccdd980 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.069+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/1216716994 >> 172.21.15.23:0/1216716994 conn(0x55f23ccd8ad0 msgr2=0x55f23ccd8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.069+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/1216716994 shutdown_connections 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.069+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/1216716994 wait complete. 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5b7fa740 1 -- start start 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5b7fa740 1 Processor -- start 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5b7fa740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccdd560 0x55f23cce75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5b7fa740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccddf50 0x55f23cce7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5b7fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23cceeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5b7fa740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f23cbe9d50 con 0x55f23ccdd560 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5b7fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f23c918310 con 0x55f23cce30c0 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5b7fa740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f23cbca8f0 con 0x55f23ccddf50 2024-03-24T10:39:34.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5b7fa740 10 client.0 _ll_register_callbacks cb 0x55f23c955ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:39:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f51d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccdd560 0x55f23cce75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f51d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccdd560 0x55f23cce75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38320/0 (socket says 172.21.15.23:38320) 2024-03-24T10:39:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f51d9b640 1 -- 172.21.15.23:0/3607439211 learned_addr learned my addr 172.21.15.23:0/3607439211 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5259c640 1 --2- 172.21.15.23:0/3607439211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccddf50 0x55f23cce7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/3607439211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23cceeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5259c640 1 -- 172.21.15.23:0/3607439211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 msgr2=0x55f23cceeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5259c640 1 --2- 172.21.15.23:0/3607439211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23cceeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5259c640 1 -- 172.21.15.23:0/3607439211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccdd560 msgr2=0x55f23cce75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5259c640 1 --2- 172.21.15.23:0/3607439211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccdd560 0x55f23cce75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f5259c640 1 -- 172.21.15.23:0/3607439211 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f3c019040 con 0x55f23ccddf50 2024-03-24T10:39:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.070+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/3607439211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccdd560 0x55f23cce75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f5259c640 1 --2- 172.21.15.23:0/3607439211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccddf50 0x55f23cce7b30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x55f23ccde350 tx=0x7f7f4c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7ae60ea74036099 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:39:34.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f39ffb640 1 -- 172.21.15.23:0/3607439211 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c017070 con 0x55f23ccddf50 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/3607439211 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f23c9446c0 con 0x55f23ccddf50 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f39ffb640 1 -- 172.21.15.23:0/3607439211 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7f4c012110 con 0x55f23ccddf50 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f39ffb640 1 -- 172.21.15.23:0/3607439211 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7f4c0122d0 con 0x55f23ccddf50 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f5b7fa740 10 client.24946 fetch_fsmap learned FSMap version 22 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/3607439211 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f23cc43510 con 0x55f23ccddf50 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f39ffb640 1 -- 172.21.15.23:0/3607439211 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f23c9446c0 con 0x55f23ccddf50 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f39ffb640 1 -- 172.21.15.23:0/3607439211 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7f4c025050 con 0x55f23ccddf50 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f5b7fa740 10 client.24946 fetch_fsmap finished waiting for FSMap version 22 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f5b7fa740 10 client.24946 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f5b7fa740 20 client.24946 populate_metadata read hostname 'smithi023' 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/3607439211 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f23ccf3ef0 0x55f23ccf5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f5b7fa740 4 client.24946 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/3607439211 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f23c995270 con 0x55f23ccf3ef0 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.071+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/3607439211 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f23ccf3ef0 0x55f23ccf5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.072+0000 7f7f39ffb640 10 client.24946 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:39:34.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.072+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/3607439211 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f23ccf3ef0 0x55f23ccf5360 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e52964fe6ca92a44 server_cookie=9d8e6c366c8d9869 in_seq=0 out_seq=0 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.072+0000 7f7f39ffb640 1 -- 172.21.15.23:0/3607439211 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f23c995270 con 0x55f23ccf3ef0 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.072+0000 7f7f39ffb640 10 client.24946 handle_command_reply: tid=0 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.073+0000 7f7f5b7fa740 1 client.24946 shutdown 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.073+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/3607439211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccddf50 msgr2=0x55f23cce7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.073+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/3607439211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccddf50 0x55f23cce7b30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x55f23ccde350 tx=0x7f7f4c00f790 comp rx=0 tx=0).stop 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.073+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/3607439211 shutdown_connections 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.073+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/3607439211 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f23ccf3ef0 0x55f23ccf5360 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.073+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/3607439211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23cceeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.073+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/3607439211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccddf50 0x55f23cce7b30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.073+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/3607439211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccdd560 0x55f23cce75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.073+0000 7f7f39ffb640 0 client.24946 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.073+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/3607439211 >> 172.21.15.23:0/3607439211 conn(0x55f23ccd8ad0 msgr2=0x55f23ccd7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.073+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/3607439211 shutdown_connections 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.073+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/3607439211 wait complete. 2024-03-24T10:39:34.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.073+0000 7f7f5b7fa740 20 client.24946 trim_cache size 0 max 16384 2024-03-24T10:39:34.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.091+0000 7f7f5b7fa740 1 Processor -- start 2024-03-24T10:39:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.091+0000 7f7f5b7fa740 1 -- start start 2024-03-24T10:39:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.091+0000 7f7f5b7fa740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccdd560 0x55f23cce54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.091+0000 7f7f5b7fa740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccddf50 0x55f23cceeb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.091+0000 7f7f5b7fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23ccd8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.091+0000 7f7f5b7fa740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f23ccf47f0 con 0x55f23ccdd560 2024-03-24T10:39:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.091+0000 7f7f5b7fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f23cce8200 con 0x55f23cce30c0 2024-03-24T10:39:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.091+0000 7f7f5b7fa740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f23ccf4300 con 0x55f23ccddf50 2024-03-24T10:39:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.091+0000 7f7f51d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccdd560 0x55f23cce54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.091+0000 7f7f51d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccdd560 0x55f23cce54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38334/0 (socket says 172.21.15.23:38334) 2024-03-24T10:39:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.091+0000 7f7f51d9b640 1 -- 172.21.15.23:0/3898629452 learned_addr learned my addr 172.21.15.23:0/3898629452 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.091+0000 7f7f5259c640 1 --2- 172.21.15.23:0/3898629452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccddf50 0x55f23cceeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.091+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/3898629452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23ccd8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f51d9b640 1 -- 172.21.15.23:0/3898629452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccddf50 msgr2=0x55f23cceeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/3898629452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccddf50 0x55f23cceeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f51d9b640 1 -- 172.21.15.23:0/3898629452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 msgr2=0x55f23ccd8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/3898629452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23ccd8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f51d9b640 1 -- 172.21.15.23:0/3898629452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f23cbdfa80 con 0x55f23ccdd560 2024-03-24T10:39:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/3898629452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccdd560 0x55f23cce54b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c0097e0 tx=0x7f7f3c009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a1a6d8371c1389 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f3bfff640 1 -- 172.21.15.23:0/3898629452 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c00fe40 con 0x55f23ccdd560 2024-03-24T10:39:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f3bfff640 1 -- 172.21.15.23:0/3898629452 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f3c030850 con 0x55f23ccdd560 2024-03-24T10:39:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f3bfff640 1 -- 172.21.15.23:0/3898629452 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c0035f0 con 0x55f23ccdd560 2024-03-24T10:39:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/3898629452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccdd560 msgr2=0x55f23cce54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/3898629452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccdd560 0x55f23cce54b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c0097e0 tx=0x7f7f3c009f30 comp rx=0 tx=0).stop 2024-03-24T10:39:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/3898629452 shutdown_connections 2024-03-24T10:39:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/3898629452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23ccd8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/3898629452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccddf50 0x55f23cceeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/3898629452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccdd560 0x55f23cce54b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/3898629452 >> 172.21.15.23:0/3898629452 conn(0x55f23ccd8ad0 msgr2=0x55f23ccdc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:34.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/3898629452 shutdown_connections 2024-03-24T10:39:34.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.092+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/3898629452 wait complete. 2024-03-24T10:39:34.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5b7fa740 1 -- start start 2024-03-24T10:39:34.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5b7fa740 1 Processor -- start 2024-03-24T10:39:34.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5b7fa740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccdd560 0x55f23cce9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5b7fa740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccddf50 0x55f23ccea510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5b7fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23cced4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5b7fa740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f23cceda00 con 0x55f23ccddf50 2024-03-24T10:39:34.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5b7fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f23ccedb70 con 0x55f23cce30c0 2024-03-24T10:39:34.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5b7fa740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f23ccedd30 con 0x55f23ccdd560 2024-03-24T10:39:34.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5b7fa740 10 client.0 _ll_register_callbacks cb 0x7f7f4c009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccddf50 0x55f23ccea510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccddf50 0x55f23ccea510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38346/0 (socket says 172.21.15.23:38346) 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5259c640 1 -- 172.21.15.23:0/1998641260 learned_addr learned my addr 172.21.15.23:0/1998641260 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/1998641260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccdd560 0x55f23cce9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/1998641260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23cced4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5259c640 1 -- 172.21.15.23:0/1998641260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccdd560 msgr2=0x55f23cce9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5259c640 1 --2- 172.21.15.23:0/1998641260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccdd560 0x55f23cce9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5259c640 1 -- 172.21.15.23:0/1998641260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 msgr2=0x55f23cced4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5259c640 1 --2- 172.21.15.23:0/1998641260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23cced4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5259c640 1 -- 172.21.15.23:0/1998641260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f3c028040 con 0x55f23ccddf50 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/1998641260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccdd560 0x55f23cce9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f5259c640 1 --2- 172.21.15.23:0/1998641260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccddf50 0x55f23ccea510 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c012a20 tx=0x7f7f4c003890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c0c34e45ad07d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.093+0000 7f7f39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f39ffb640 1 -- 172.21.15.23:0/1998641260 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c017040 con 0x55f23ccddf50 2024-03-24T10:39:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f39ffb640 1 -- 172.21.15.23:0/1998641260 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7f4c000e40 con 0x55f23ccddf50 2024-03-24T10:39:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/1998641260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f3c011180 con 0x55f23ccddf50 2024-03-24T10:39:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f39ffb640 1 -- 172.21.15.23:0/1998641260 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7f4c0052d0 con 0x55f23ccddf50 2024-03-24T10:39:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f5b7fa740 10 client.15084 fetch_fsmap learned FSMap version 22 2024-03-24T10:39:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/1998641260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55f23cc057e0 con 0x55f23ccddf50 2024-03-24T10:39:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f39ffb640 1 -- 172.21.15.23:0/1998641260 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f3c011180 con 0x55f23ccddf50 2024-03-24T10:39:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f39ffb640 1 -- 172.21.15.23:0/1998641260 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7f4c025040 con 0x55f23ccddf50 2024-03-24T10:39:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f5b7fa740 10 client.15084 fetch_fsmap finished waiting for FSMap version 22 2024-03-24T10:39:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f5b7fa740 10 client.15084 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T10:39:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f5b7fa740 20 client.15084 populate_metadata read hostname 'smithi023' 2024-03-24T10:39:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/1998641260 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f23cd5e0c0 0x55f23cd5cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f5b7fa740 4 client.15084 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T10:39:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/1998641260 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "status"}) -- 0x55f23c981610 con 0x55f23cd5e0c0 2024-03-24T10:39:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.094+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/1998641260 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f23cd5e0c0 0x55f23cd5cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f39ffb640 10 client.15084 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:39:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f51d9b640 1 --2- 172.21.15.23:0/1998641260 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f23cd5e0c0 0x55f23cd5cdd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a9d85d19d031b147 server_cookie=12f39cb7239d46fd in_seq=0 out_seq=0 2024-03-24T10:39:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f39ffb640 1 -- 172.21.15.23:0/1998641260 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55f23c981610 con 0x55f23cd5e0c0 2024-03-24T10:39:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f39ffb640 10 client.15084 handle_command_reply: tid=0 2024-03-24T10:39:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f5b7fa740 1 client.15084 shutdown 2024-03-24T10:39:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/1998641260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccddf50 msgr2=0x55f23ccea510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/1998641260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccddf50 0x55f23ccea510 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c012a20 tx=0x7f7f4c003890 comp rx=0 tx=0).stop 2024-03-24T10:39:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/1998641260 shutdown_connections 2024-03-24T10:39:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/1998641260 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f23cd5e0c0 0x55f23cd5cdd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/1998641260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f23cce30c0 0x55f23cced4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/1998641260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f23ccddf50 0x55f23ccea510 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f39ffb640 0 client.15084 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:39:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f5b7fa740 1 --2- 172.21.15.23:0/1998641260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f23ccdd560 0x55f23cce9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/1998641260 >> 172.21.15.23:0/1998641260 conn(0x55f23ccd8ad0 msgr2=0x55f23ccdc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:34.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/1998641260 shutdown_connections 2024-03-24T10:39:34.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f5b7fa740 1 -- 172.21.15.23:0/1998641260 wait complete. 2024-03-24T10:39:34.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.095+0000 7f7f5b7fa740 20 client.15084 trim_cache size 0 max 16384 2024-03-24T10:39:34.103 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:39:34.103 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "4f8fb8f6-e9c9-11ee-95cd-87774f69a715", 2024-03-24T10:39:34.103 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 1, 2024-03-24T10:39:34.103 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24505, 2024-03-24T10:39:34.103 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-24T10:39:34.103 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-24T10:39:34.103 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-24T10:39:34.103 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 80.208038923999993, 2024-03-24T10:39:34.103 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 22, 2024-03-24T10:39:34.104 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 87, 2024-03-24T10:39:34.104 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 85, 2024-03-24T10:39:34.104 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 97.477642431000007 2024-03-24T10:39:34.104 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:39:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.096+0000 7f7f58c15640 1 -- 172.21.15.23:0/2383094974 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f400a2150 msgr2=0x7f7f400a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.096+0000 7f7f58c15640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f400a2150 0x7f7f400a4610 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7f440bd790 tx=0x7f7f440e7e30 comp rx=0 tx=0).stop 2024-03-24T10:39:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.096+0000 7f7f58c15640 1 -- 172.21.15.23:0/2383094974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f541021d0 msgr2=0x7f7f5419ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.096+0000 7f7f58c15640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f541021d0 0x7f7f5419ed30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7f44037040 tx=0x7f7f44009360 comp rx=0 tx=0).stop 2024-03-24T10:39:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.096+0000 7f7f58c15640 1 -- 172.21.15.23:0/2383094974 shutdown_connections 2024-03-24T10:39:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.096+0000 7f7f58c15640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f400a2150 0x7f7f400a4610 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.096+0000 7f7f58c15640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f541021d0 0x7f7f5419ed30 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.096+0000 7f7f58c15640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f54101810 0x7f7f54197cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.096+0000 7f7f58c15640 1 --2- 172.21.15.23:0/2383094974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f54100620 0x7f7f54197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.096+0000 7f7f58c15640 1 -- 172.21.15.23:0/2383094974 >> 172.21.15.23:0/2383094974 conn(0x7f7f540fbbb0 msgr2=0x7f7f54069d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.096+0000 7f7f58c15640 1 -- 172.21.15.23:0/2383094974 shutdown_connections 2024-03-24T10:39:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.096+0000 7f7f58c15640 1 -- 172.21.15.23:0/2383094974 wait complete. 2024-03-24T10:39:34.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:33 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3525050772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:39:34.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:33 smithi023 ceph-mon[32466]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T10:39:34.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:33 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2988870182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:34.106 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:33 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4251830600' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:34.107 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T10:39:34.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.212+0000 7f70784df640 1 -- 172.21.15.23:0/2416135589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7070104ac0 msgr2=0x7f7070106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.212+0000 7f70784df640 1 --2- 172.21.15.23:0/2416135589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7070104ac0 0x7f7070106eb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7060009860 tx=0x7f706002bb70 comp rx=0 tx=0).stop 2024-03-24T10:39:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.212+0000 7f70784df640 1 -- 172.21.15.23:0/2416135589 shutdown_connections 2024-03-24T10:39:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.212+0000 7f70784df640 1 --2- 172.21.15.23:0/2416135589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70701073f0 0x7f7070109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.212+0000 7f70784df640 1 --2- 172.21.15.23:0/2416135589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7070104ac0 0x7f7070106eb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.212+0000 7f70784df640 1 --2- 172.21.15.23:0/2416135589 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7070102190 0x7f7070104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.212+0000 7f70784df640 1 -- 172.21.15.23:0/2416135589 >> 172.21.15.23:0/2416135589 conn(0x7f70700fbba0 msgr2=0x7f70700fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.212+0000 7f70784df640 1 -- 172.21.15.23:0/2416135589 shutdown_connections 2024-03-24T10:39:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.212+0000 7f70784df640 1 -- 172.21.15.23:0/2416135589 wait complete. 2024-03-24T10:39:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.212+0000 7f70784df640 1 Processor -- start 2024-03-24T10:39:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.212+0000 7f70784df640 1 -- start start 2024-03-24T10:39:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f70784df640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7070102190 0x7f7070197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f70784df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7070104ac0 0x7f7070197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f70784df640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70701073f0 0x7f707019ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f70784df640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f707019f320 con 0x7f70701073f0 2024-03-24T10:39:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f70784df640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f707019f490 con 0x7f7070104ac0 2024-03-24T10:39:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f70784df640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f707019f650 con 0x7f7070102190 2024-03-24T10:39:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f7076a7c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70701073f0 0x7f707019ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f7076a7c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70701073f0 0x7f707019ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38370/0 (socket says 172.21.15.23:38370) 2024-03-24T10:39:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f707627b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7070102190 0x7f7070197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f7076a7c640 1 -- 172.21.15.23:0/3563962674 learned_addr learned my addr 172.21.15.23:0/3563962674 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f7075a7a640 1 --2- 172.21.15.23:0/3563962674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7070104ac0 0x7f7070197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f7076a7c640 1 -- 172.21.15.23:0/3563962674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7070102190 msgr2=0x7f7070197830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f7076a7c640 1 --2- 172.21.15.23:0/3563962674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7070102190 0x7f7070197830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f7076a7c640 1 -- 172.21.15.23:0/3563962674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7070104ac0 msgr2=0x7f7070197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f7076a7c640 1 --2- 172.21.15.23:0/3563962674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7070104ac0 0x7f7070197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f7076a7c640 1 -- 172.21.15.23:0/3563962674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7068009900 con 0x7f70701073f0 2024-03-24T10:39:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f7076a7c640 1 --2- 172.21.15.23:0/3563962674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70701073f0 0x7f707019ede0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f706800efc0 tx=0x7f706800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f70677fe640 1 -- 172.21.15.23:0/3563962674 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068003970 con 0x7f70701073f0 2024-03-24T10:39:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f70677fe640 1 -- 172.21.15.23:0/3563962674 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7068004020 con 0x7f70701073f0 2024-03-24T10:39:34.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f70784df640 1 -- 172.21.15.23:0/3563962674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7060009490 con 0x7f70701073f0 2024-03-24T10:39:34.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.213+0000 7f70677fe640 1 -- 172.21.15.23:0/3563962674 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706800b9b0 con 0x7f70701073f0 2024-03-24T10:39:34.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.214+0000 7f70784df640 1 -- 172.21.15.23:0/3563962674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70701002e0 con 0x7f70701073f0 2024-03-24T10:39:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.216+0000 7f70677fe640 1 -- 172.21.15.23:0/3563962674 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f706800a040 con 0x7f70701073f0 2024-03-24T10:39:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.216+0000 7f707b198740 1 Processor -- start 2024-03-24T10:39:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707b198740 1 -- start start 2024-03-24T10:39:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f70677fe640 1 --2- 172.21.15.23:0/3563962674 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f705c0a2150 0x7f705c0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707b198740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9ccd560 0x5620b9ccd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707b198740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccdf50 0x5620b9cd2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707b198740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9cd30c0 0x5620b9cd5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707b198740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5620b9c3c180 con 0x5620b9ccdf50 2024-03-24T10:39:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707b198740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5620b9c396b0 con 0x5620b9ccd560 2024-03-24T10:39:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707b198740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5620b9c34e90 con 0x5620b9cd30c0 2024-03-24T10:39:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707627b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9ccd560 0x5620b9ccd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f7075a7a640 1 --2- 172.21.15.23:0/3563962674 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f705c0a2150 0x7f705c0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707627b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9ccd560 0x5620b9ccd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:47180/0 (socket says 172.21.15.23:47180) 2024-03-24T10:39:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707627b640 1 -- 172.21.15.23:0/4521607 learned_addr learned my addr 172.21.15.23:0/4521607 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f7076a7c640 1 --2- 172.21.15.23:0/4521607 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9cd30c0 0x5620b9cd5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f7075a7a640 1 --2- 172.21.15.23:0/4521607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccdf50 0x5620b9cd2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f70677fe640 1 -- 172.21.15.23:0/3563962674 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f70680cde60 con 0x7f70701073f0 2024-03-24T10:39:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707627b640 1 -- 172.21.15.23:0/4521607 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9cd30c0 msgr2=0x5620b9cd5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707627b640 1 --2- 172.21.15.23:0/4521607 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9cd30c0 0x5620b9cd5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707627b640 1 -- 172.21.15.23:0/4521607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccdf50 msgr2=0x5620b9cd2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707627b640 1 --2- 172.21.15.23:0/4521607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccdf50 0x5620b9cd2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f707627b640 1 -- 172.21.15.23:0/4521607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620b9bf1d80 con 0x5620b9ccd560 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.217+0000 7f7076a7c640 1 --2- 172.21.15.23:0/4521607 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9cd30c0 0x5620b9cd5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.218+0000 7f707627b640 1 --2- 172.21.15.23:0/4521607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9ccd560 0x5620b9ccd980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f706c004610 tx=0x7f706c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3d8aa583bd00149 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.218+0000 7f7075a7a640 1 --2- 172.21.15.23:0/3563962674 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f705c0a2150 0x7f705c0a4610 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7060009830 tx=0x7f7060032450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.218+0000 7f7064bfe640 1 -- 172.21.15.23:0/4521607 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c017040 con 0x5620b9ccd560 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.218+0000 7f7064bfe640 1 -- 172.21.15.23:0/4521607 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f706c009e60 con 0x5620b9ccd560 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.218+0000 7f7064bfe640 1 -- 172.21.15.23:0/4521607 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c01b920 con 0x5620b9ccd560 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 -- 172.21.15.23:0/4521607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9ccd560 msgr2=0x5620b9ccd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 --2- 172.21.15.23:0/4521607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9ccd560 0x5620b9ccd980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f706c004610 tx=0x7f706c005f90 comp rx=0 tx=0).stop 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 -- 172.21.15.23:0/4521607 shutdown_connections 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 --2- 172.21.15.23:0/4521607 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9cd30c0 0x5620b9cd5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 --2- 172.21.15.23:0/4521607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccdf50 0x5620b9cd2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 --2- 172.21.15.23:0/4521607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9ccd560 0x5620b9ccd980 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 -- 172.21.15.23:0/4521607 >> 172.21.15.23:0/4521607 conn(0x5620b9cc8ad0 msgr2=0x5620b9cc8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 -- 172.21.15.23:0/4521607 shutdown_connections 2024-03-24T10:39:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 -- 172.21.15.23:0/4521607 wait complete. 2024-03-24T10:39:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 -- start start 2024-03-24T10:39:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 Processor -- start 2024-03-24T10:39:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccd560 0x5620b9cd75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccdf50 0x5620b9cd7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 0x5620b9cdead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5620b9bd9d50 con 0x5620b9ccd560 2024-03-24T10:39:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5620b9908310 con 0x5620b9cd30c0 2024-03-24T10:39:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5620b9bba8f0 con 0x5620b9ccdf50 2024-03-24T10:39:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707b198740 10 client.0 _ll_register_callbacks cb 0x5620b9945ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:39:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707627b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccd560 0x5620b9cd75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707627b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccd560 0x5620b9cd75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38382/0 (socket says 172.21.15.23:38382) 2024-03-24T10:39:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f707627b640 1 -- 172.21.15.23:0/3460266612 learned_addr learned my addr 172.21.15.23:0/3460266612 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f7075a7a640 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccdf50 0x5620b9cd7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.219+0000 7f7076a7c640 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 0x5620b9cdead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f707627b640 1 -- 172.21.15.23:0/3460266612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccdf50 msgr2=0x5620b9cd7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f707627b640 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccdf50 0x5620b9cd7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f707627b640 1 -- 172.21.15.23:0/3460266612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 msgr2=0x5620b9cdead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f707627b640 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 0x5620b9cdead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f707627b640 1 -- 172.21.15.23:0/3460266612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f706c004270 con 0x5620b9ccd560 2024-03-24T10:39:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f7076a7c640 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 0x5620b9cdead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f7075a7a640 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccdf50 0x5620b9cd7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f707627b640 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccd560 0x5620b9cd75c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f706c009380 tx=0x7f706c01b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3217c134a54c25 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:34.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f7059ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:39:34.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f7059ffb640 1 -- 172.21.15.23:0/3460266612 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c017070 con 0x5620b9ccd560 2024-03-24T10:39:34.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f707b198740 1 -- 172.21.15.23:0/3460266612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5620b99346c0 con 0x5620b9ccd560 2024-03-24T10:39:34.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f7059ffb640 1 -- 172.21.15.23:0/3460266612 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f706c005070 con 0x5620b9ccd560 2024-03-24T10:39:34.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f7059ffb640 1 -- 172.21.15.23:0/3460266612 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f706c005230 con 0x5620b9ccd560 2024-03-24T10:39:34.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f707b198740 10 client.15096 fetch_fsmap learned FSMap version 22 2024-03-24T10:39:34.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f707b198740 1 -- 172.21.15.23:0/3460266612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5620b9c33510 con 0x5620b9ccd560 2024-03-24T10:39:34.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.220+0000 7f7059ffb640 1 -- 172.21.15.23:0/3460266612 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5620b99346c0 con 0x5620b9ccd560 2024-03-24T10:39:34.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.221+0000 7f7059ffb640 1 -- 172.21.15.23:0/3460266612 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f706c011850 con 0x5620b9ccd560 2024-03-24T10:39:34.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.221+0000 7f707b198740 10 client.15096 fetch_fsmap finished waiting for FSMap version 22 2024-03-24T10:39:34.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.221+0000 7f707b198740 10 client.15096 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T10:39:34.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.221+0000 7f707b198740 20 client.15096 populate_metadata read hostname 'smithi023' 2024-03-24T10:39:34.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.221+0000 7f707b198740 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5620b9ce3d50 0x5620b9ce51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.221+0000 7f707b198740 4 client.15096 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.221+0000 7f707b198740 1 -- 172.21.15.23:0/3460266612 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620b9985270 con 0x5620b9ce3d50 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.221+0000 7f7075a7a640 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5620b9ce3d50 0x5620b9ce51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.221+0000 7f7075a7a640 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5620b9ce3d50 0x5620b9ce51c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c89b41d36f939fe6 server_cookie=f56b143fbc546860 in_seq=0 out_seq=0 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.221+0000 7f7059ffb640 10 client.15096 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f7059ffb640 1 -- 172.21.15.23:0/3460266612 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5620b9985270 con 0x5620b9ce3d50 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f7059ffb640 10 client.15096 handle_command_reply: tid=0 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f707b198740 1 client.15096 shutdown 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f707b198740 1 -- 172.21.15.23:0/3460266612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccd560 msgr2=0x5620b9cd75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f707b198740 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccd560 0x5620b9cd75c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f706c009380 tx=0x7f706c01b910 comp rx=0 tx=0).stop 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f707b198740 1 -- 172.21.15.23:0/3460266612 shutdown_connections 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f707b198740 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 0x5620b9cdead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f707b198740 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5620b9ce3d50 0x5620b9ce51c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f707b198740 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccdf50 0x5620b9cd7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f707b198740 1 --2- 172.21.15.23:0/3460266612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccd560 0x5620b9cd75c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f7059ffb640 0 client.15096 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:39:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f707b198740 1 -- 172.21.15.23:0/3460266612 >> 172.21.15.23:0/3460266612 conn(0x5620b9cc8ad0 msgr2=0x5620b9cc7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f707b198740 1 -- 172.21.15.23:0/3460266612 shutdown_connections 2024-03-24T10:39:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f707b198740 1 -- 172.21.15.23:0/3460266612 wait complete. 2024-03-24T10:39:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.222+0000 7f707b198740 20 client.15096 trim_cache size 0 max 16384 2024-03-24T10:39:34.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.240+0000 7f707b198740 1 Processor -- start 2024-03-24T10:39:34.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.240+0000 7f707b198740 1 -- start start 2024-03-24T10:39:34.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707b198740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccd560 0x5620b9cc8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707b198740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccdf50 0x5620b9cdea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707b198740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 0x5620b9ccf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707b198740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5620b9985270 con 0x5620b9ccd560 2024-03-24T10:39:34.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707b198740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5620b99346c0 con 0x5620b9cd30c0 2024-03-24T10:39:34.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707b198740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5620b9ce4130 con 0x5620b9ccdf50 2024-03-24T10:39:34.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707627b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccd560 0x5620b9cc8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707627b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccd560 0x5620b9cc8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38394/0 (socket says 172.21.15.23:38394) 2024-03-24T10:39:34.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707627b640 1 -- 172.21.15.23:0/4191327105 learned_addr learned my addr 172.21.15.23:0/4191327105 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:34.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f7075a7a640 1 --2- 172.21.15.23:0/4191327105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccdf50 0x5620b9cdea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f7076a7c640 1 --2- 172.21.15.23:0/4191327105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 0x5620b9ccf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707627b640 1 -- 172.21.15.23:0/4191327105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccdf50 msgr2=0x5620b9cdea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707627b640 1 --2- 172.21.15.23:0/4191327105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccdf50 0x5620b9cdea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707627b640 1 -- 172.21.15.23:0/4191327105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 msgr2=0x5620b9ccf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707627b640 1 --2- 172.21.15.23:0/4191327105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 0x5620b9ccf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707627b640 1 -- 172.21.15.23:0/4191327105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620b9bcfa80 con 0x5620b9ccd560 2024-03-24T10:39:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f707627b640 1 --2- 172.21.15.23:0/4191327105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccd560 0x5620b9cc8eb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f706c011e70 tx=0x7f706c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc18b00101492b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f705bfff640 1 -- 172.21.15.23:0/4191327105 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c017070 con 0x5620b9ccd560 2024-03-24T10:39:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f705bfff640 1 -- 172.21.15.23:0/4191327105 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f706c01b4e0 con 0x5620b9ccd560 2024-03-24T10:39:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.241+0000 7f705bfff640 1 -- 172.21.15.23:0/4191327105 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c0035f0 con 0x5620b9ccd560 2024-03-24T10:39:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.242+0000 7f707b198740 1 -- 172.21.15.23:0/4191327105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccd560 msgr2=0x5620b9cc8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.242+0000 7f707b198740 1 --2- 172.21.15.23:0/4191327105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccd560 0x5620b9cc8eb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f706c011e70 tx=0x7f706c009da0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.242+0000 7f707b198740 1 -- 172.21.15.23:0/4191327105 shutdown_connections 2024-03-24T10:39:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.242+0000 7f707b198740 1 --2- 172.21.15.23:0/4191327105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 0x5620b9ccf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.242+0000 7f707b198740 1 --2- 172.21.15.23:0/4191327105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccdf50 0x5620b9cdea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.242+0000 7f707b198740 1 --2- 172.21.15.23:0/4191327105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccd560 0x5620b9cc8eb0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.242+0000 7f707b198740 1 -- 172.21.15.23:0/4191327105 >> 172.21.15.23:0/4191327105 conn(0x5620b9cc8ad0 msgr2=0x5620b9ce2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.242+0000 7f707b198740 1 -- 172.21.15.23:0/4191327105 shutdown_connections 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.242+0000 7f707b198740 1 -- 172.21.15.23:0/4191327105 wait complete. 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.242+0000 7f707b198740 1 -- start start 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.242+0000 7f707b198740 1 Processor -- start 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707b198740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccd560 0x5620b9cd9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707b198740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccdf50 0x5620b9cd9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707b198740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 0x5620b9d47ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707b198740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5620b9cdec10 con 0x5620b9ccdf50 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707b198740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5620b9cded80 con 0x5620b9cd30c0 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707b198740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5620b9d47ff0 con 0x5620b9ccd560 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707b198740 10 client.0 _ll_register_callbacks cb 0x7f7068009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f7075a7a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccdf50 0x5620b9cd9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f7075a7a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccdf50 0x5620b9cd9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38398/0 (socket says 172.21.15.23:38398) 2024-03-24T10:39:34.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f7075a7a640 1 -- 172.21.15.23:0/663709053 learned_addr learned my addr 172.21.15.23:0/663709053 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707627b640 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccd560 0x5620b9cd9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f7076a7c640 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 0x5620b9d47ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707627b640 1 -- 172.21.15.23:0/663709053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 msgr2=0x5620b9d47ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707627b640 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 0x5620b9d47ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707627b640 1 -- 172.21.15.23:0/663709053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccdf50 msgr2=0x5620b9cd9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707627b640 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccdf50 0x5620b9cd9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707627b640 1 -- 172.21.15.23:0/663709053 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620b9c33510 con 0x5620b9ccd560 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f7076a7c640 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 0x5620b9d47ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f7075a7a640 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccdf50 0x5620b9cd9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707627b640 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccd560 0x5620b9cd9710 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f706c011d00 tx=0x7f706c0092b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db78ddf64a8b0d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f7059ffb640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f7059ffb640 1 -- 172.21.15.23:0/663709053 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c017070 con 0x5620b9ccd560 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f7059ffb640 1 -- 172.21.15.23:0/663709053 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f706c012100 con 0x5620b9ccd560 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.243+0000 7f707b198740 1 -- 172.21.15.23:0/663709053 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5620b9971610 con 0x5620b9ccd560 2024-03-24T10:39:34.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.244+0000 7f7059ffb640 1 -- 172.21.15.23:0/663709053 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f706c0126e0 con 0x5620b9ccd560 2024-03-24T10:39:34.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.244+0000 7f707b198740 10 client.24970 fetch_fsmap learned FSMap version 22 2024-03-24T10:39:34.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.244+0000 7f707b198740 1 -- 172.21.15.23:0/663709053 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5620b9bf57e0 con 0x5620b9ccd560 2024-03-24T10:39:34.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.244+0000 7f7059ffb640 1 -- 172.21.15.23:0/663709053 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5620b9971610 con 0x5620b9ccd560 2024-03-24T10:39:34.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.244+0000 7f7059ffb640 1 -- 172.21.15.23:0/663709053 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f706c004490 con 0x5620b9ccd560 2024-03-24T10:39:34.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.244+0000 7f707b198740 10 client.24970 fetch_fsmap finished waiting for FSMap version 22 2024-03-24T10:39:34.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.244+0000 7f707b198740 10 client.24970 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T10:39:34.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.244+0000 7f707b198740 20 client.24970 populate_metadata read hostname 'smithi023' 2024-03-24T10:39:34.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.244+0000 7f707b198740 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5620b9d4d410 0x5620b9d4c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:34.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.244+0000 7f707b198740 4 client.24970 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T10:39:34.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.244+0000 7f707b198740 1 -- 172.21.15.23:0/663709053 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "status"}) -- 0x5620b98e05e0 con 0x5620b9d4d410 2024-03-24T10:39:34.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.244+0000 7f7075a7a640 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5620b9d4d410 0x5620b9d4c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:34.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f7075a7a640 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5620b9d4d410 0x5620b9d4c120 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71517d93f4d2c543 server_cookie=45ebaf64d0e26ede in_seq=0 out_seq=0 2024-03-24T10:39:34.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f7059ffb640 10 client.24970 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f7059ffb640 1 -- 172.21.15.23:0/663709053 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5620b98e05e0 con 0x5620b9d4d410 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f7059ffb640 10 client.24970 handle_command_reply: tid=0 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f707b198740 1 client.24970 shutdown 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f707b198740 1 -- 172.21.15.23:0/663709053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccd560 msgr2=0x5620b9cd9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f707b198740 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccd560 0x5620b9cd9710 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f706c011d00 tx=0x7f706c0092b0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f707b198740 1 -- 172.21.15.23:0/663709053 shutdown_connections 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f707b198740 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5620b9cd30c0 0x5620b9d47ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f707b198740 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5620b9ccdf50 0x5620b9cd9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f707b198740 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5620b9d4d410 0x5620b9d4c120 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f707b198740 1 --2- 172.21.15.23:0/663709053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5620b9ccd560 0x5620b9cd9710 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f7059ffb640 0 client.24970 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f707b198740 1 -- 172.21.15.23:0/663709053 >> 172.21.15.23:0/663709053 conn(0x5620b9cc8ad0 msgr2=0x5620b9cc8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f707b198740 1 -- 172.21.15.23:0/663709053 shutdown_connections 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f707b198740 1 -- 172.21.15.23:0/663709053 wait complete. 2024-03-24T10:39:34.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.245+0000 7f707b198740 20 client.24970 trim_cache size 0 max 16384 2024-03-24T10:39:34.253 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:39:34.253 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "4f8fb8f6-e9c9-11ee-95cd-87774f69a715", 2024-03-24T10:39:34.253 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2024-03-24T10:39:34.253 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24539, 2024-03-24T10:39:34.253 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-24T10:39:34.253 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-24T10:39:34.253 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-24T10:39:34.253 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 83.377602272000004, 2024-03-24T10:39:34.253 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 22, 2024-03-24T10:39:34.253 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 87, 2024-03-24T10:39:34.253 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 85, 2024-03-24T10:39:34.253 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 89.328494728999999 2024-03-24T10:39:34.253 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:39:34.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.246+0000 7f70784df640 1 -- 172.21.15.23:0/3563962674 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f705c0a2150 msgr2=0x7f705c0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.246+0000 7f70784df640 1 --2- 172.21.15.23:0/3563962674 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f705c0a2150 0x7f705c0a4610 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7060009830 tx=0x7f7060032450 comp rx=0 tx=0).stop 2024-03-24T10:39:34.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.246+0000 7f70784df640 1 -- 172.21.15.23:0/3563962674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70701073f0 msgr2=0x7f707019ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:34.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.246+0000 7f70784df640 1 --2- 172.21.15.23:0/3563962674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70701073f0 0x7f707019ede0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f706800efc0 tx=0x7f706800c800 comp rx=0 tx=0).stop 2024-03-24T10:39:34.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.246+0000 7f70784df640 1 -- 172.21.15.23:0/3563962674 shutdown_connections 2024-03-24T10:39:34.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.246+0000 7f70784df640 1 --2- 172.21.15.23:0/3563962674 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f705c0a2150 0x7f705c0a4610 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.246+0000 7f70784df640 1 --2- 172.21.15.23:0/3563962674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70701073f0 0x7f707019ede0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.246+0000 7f70784df640 1 --2- 172.21.15.23:0/3563962674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7070104ac0 0x7f7070197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.246+0000 7f70784df640 1 --2- 172.21.15.23:0/3563962674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7070102190 0x7f7070197830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:34.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.246+0000 7f70784df640 1 -- 172.21.15.23:0/3563962674 >> 172.21.15.23:0/3563962674 conn(0x7f70700fbba0 msgr2=0x7f7070102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:34.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.246+0000 7f70784df640 1 -- 172.21.15.23:0/3563962674 shutdown_connections 2024-03-24T10:39:34.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:34.246+0000 7f70784df640 1 -- 172.21.15.23:0/3563962674 wait complete. 2024-03-24T10:39:34.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:33 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3525050772' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:39:34.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:33 smithi117 ceph-mon[36368]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T10:39:34.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:33 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2988870182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:34.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:33 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4251830600' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:34.257 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T10:39:34.257 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T10:39:34.267 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T10:39:34.268 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-24T10:39:34.519 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:34.893 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:34 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2296015724' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:35.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:34 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2296015724' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:35.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:34 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2296015724' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:36.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:35 smithi117 ceph-mon[36368]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:39:36.052 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:35 smithi023 ceph-mon[32466]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:39:36.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.157+0000 7f9a81e5f700 1 -- 172.21.15.23:0/3020916617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a7c103dd0 msgr2=0x7f9a7c10a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:36.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.157+0000 7f9a81e5f700 1 --2- 172.21.15.23:0/3020916617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a7c103dd0 0x7f9a7c10a680 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9a7400b0d0 tx=0x7f9a7400b3e0 comp rx=0 tx=0).stop 2024-03-24T10:39:36.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.157+0000 7f9a81e5f700 1 -- 172.21.15.23:0/3020916617 shutdown_connections 2024-03-24T10:39:36.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.157+0000 7f9a81e5f700 1 --2- 172.21.15.23:0/3020916617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a7c103dd0 0x7f9a7c10a680 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:36.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.157+0000 7f9a81e5f700 1 --2- 172.21.15.23:0/3020916617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c102230 0x7f9a7c102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:36.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.157+0000 7f9a81e5f700 1 --2- 172.21.15.23:0/3020916617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a7c103430 0x7f9a7c103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:36.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.157+0000 7f9a81e5f700 1 -- 172.21.15.23:0/3020916617 >> 172.21.15.23:0/3020916617 conn(0x7f9a7c0fd810 msgr2=0x7f9a7c0ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:36.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.157+0000 7f9a81e5f700 1 -- 172.21.15.23:0/3020916617 shutdown_connections 2024-03-24T10:39:36.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.157+0000 7f9a81e5f700 1 -- 172.21.15.23:0/3020916617 wait complete. 2024-03-24T10:39:36.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.158+0000 7f9a81e5f700 1 Processor -- start 2024-03-24T10:39:36.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.158+0000 7f9a81e5f700 1 -- start start 2024-03-24T10:39:36.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.158+0000 7f9a81e5f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a7c102230 0x7f9a7c075fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:36.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.158+0000 7f9a81e5f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c103430 0x7f9a7c0745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:36.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.158+0000 7f9a81e5f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a7c103dd0 0x7f9a7c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:36.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.158+0000 7f9a81e5f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c0752b0 con 0x7f9a7c102230 2024-03-24T10:39:36.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.158+0000 7f9a81e5f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c075420 con 0x7f9a7c103430 2024-03-24T10:39:36.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.158+0000 7f9a81e5f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c1a2340 con 0x7f9a7c103dd0 2024-03-24T10:39:36.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.158+0000 7f9a7b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a7c102230 0x7f9a7c075fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:36.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.158+0000 7f9a7b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a7c102230 0x7f9a7c075fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38402/0 (socket says 172.21.15.23:38402) 2024-03-24T10:39:36.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.158+0000 7f9a7b7fe700 1 -- 172.21.15.23:0/909196335 learned_addr learned my addr 172.21.15.23:0/909196335 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:36.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.159+0000 7f9a7affd700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c103430 0x7f9a7c0745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:36.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.159+0000 7f9a7bfff700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a7c103dd0 0x7f9a7c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:36.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.159+0000 7f9a7b7fe700 1 -- 172.21.15.23:0/909196335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a7c103dd0 msgr2=0x7f9a7c074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:36.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.159+0000 7f9a7b7fe700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a7c103dd0 0x7f9a7c074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:36.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.159+0000 7f9a7b7fe700 1 -- 172.21.15.23:0/909196335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c103430 msgr2=0x7f9a7c0745f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:36.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.159+0000 7f9a7b7fe700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c103430 0x7f9a7c0745f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:36.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.159+0000 7f9a7b7fe700 1 -- 172.21.15.23:0/909196335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a74009d00 con 0x7f9a7c102230 2024-03-24T10:39:36.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.159+0000 7f9a7bfff700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a7c103dd0 0x7f9a7c074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:39:36.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.159+0000 7f9a7b7fe700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a7c102230 0x7f9a7c075fa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000e970 tx=0x7f9a7000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:36.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.159+0000 7f9a78ff9700 1 -- 172.21.15.23:0/909196335 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a70009790 con 0x7f9a7c102230 2024-03-24T10:39:36.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.159+0000 7f9a78ff9700 1 -- 172.21.15.23:0/909196335 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a7000eec0 con 0x7f9a7c102230 2024-03-24T10:39:36.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.159+0000 7f9a81e5f700 1 -- 172.21.15.23:0/909196335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a7c1a2620 con 0x7f9a7c102230 2024-03-24T10:39:36.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.159+0000 7f9a78ff9700 1 -- 172.21.15.23:0/909196335 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a70003e00 con 0x7f9a7c102230 2024-03-24T10:39:36.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.159+0000 7f9a81e5f700 1 -- 172.21.15.23:0/909196335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a7c1a2b70 con 0x7f9a7c102230 2024-03-24T10:39:36.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.160+0000 7f9a81e5f700 1 -- 172.21.15.23:0/909196335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a7c1095f0 con 0x7f9a7c102230 2024-03-24T10:39:36.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.162+0000 7f9a78ff9700 1 -- 172.21.15.23:0/909196335 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f9a70003f60 con 0x7f9a7c102230 2024-03-24T10:39:36.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.163+0000 7f9a78ff9700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9a640a2260 0x7f9a640a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:36.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.163+0000 7f9a7affd700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9a640a2260 0x7f9a640a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:36.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.163+0000 7f9a78ff9700 1 -- 172.21.15.23:0/909196335 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f9a70014070 con 0x7f9a7c102230 2024-03-24T10:39:36.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.163+0000 7f9a7affd700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9a640a2260 0x7f9a640a4720 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c005950 tx=0x7f9a6c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:36.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.166+0000 7f9a78ff9700 1 -- 172.21.15.23:0/909196335 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9a700970a0 con 0x7f9a7c102230 2024-03-24T10:39:36.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:35 smithi116 ceph-mon[36248]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:39:36.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.386+0000 7f9a81e5f700 1 -- 172.21.15.23:0/909196335 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f9a7c02dc60 con 0x7f9a640a2260 2024-03-24T10:39:36.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.387+0000 7f9a78ff9700 1 -- 172.21.15.23:0/909196335 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f9a7c02dc60 con 0x7f9a640a2260 2024-03-24T10:39:36.387 INFO:teuthology.orchestra.run.smithi023.stdout:Backend: cephadm 2024-03-24T10:39:36.387 INFO:teuthology.orchestra.run.smithi023.stdout:Available: Yes 2024-03-24T10:39:36.388 INFO:teuthology.orchestra.run.smithi023.stdout:Paused: No 2024-03-24T10:39:36.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.389+0000 7f9a81e5f700 1 -- 172.21.15.23:0/909196335 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9a640a2260 msgr2=0x7f9a640a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:36.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.389+0000 7f9a81e5f700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9a640a2260 0x7f9a640a4720 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c005950 tx=0x7f9a6c00b410 comp rx=0 tx=0).stop 2024-03-24T10:39:36.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.389+0000 7f9a81e5f700 1 -- 172.21.15.23:0/909196335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a7c102230 msgr2=0x7f9a7c075fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:36.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.389+0000 7f9a81e5f700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a7c102230 0x7f9a7c075fa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000e970 tx=0x7f9a7000ec80 comp rx=0 tx=0).stop 2024-03-24T10:39:36.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.389+0000 7f9a81e5f700 1 -- 172.21.15.23:0/909196335 shutdown_connections 2024-03-24T10:39:36.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.389+0000 7f9a81e5f700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f9a640a2260 0x7f9a640a4720 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:36.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.389+0000 7f9a81e5f700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a7c103dd0 0x7f9a7c074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:36.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.389+0000 7f9a81e5f700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a7c102230 0x7f9a7c075fa0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:36.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.389+0000 7f9a81e5f700 1 --2- 172.21.15.23:0/909196335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a7c103430 0x7f9a7c0745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:36.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.389+0000 7f9a81e5f700 1 -- 172.21.15.23:0/909196335 >> 172.21.15.23:0/909196335 conn(0x7f9a7c0fd810 msgr2=0x7f9a7c106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:36.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.389+0000 7f9a81e5f700 1 -- 172.21.15.23:0/909196335 shutdown_connections 2024-03-24T10:39:36.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:36.389+0000 7f9a81e5f700 1 -- 172.21.15.23:0/909196335 wait complete. 2024-03-24T10:39:36.947 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:36 smithi023 ceph-mon[32466]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:36.973 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-24T10:39:37.219 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:37.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:36 smithi117 ceph-mon[36368]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:37.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:36 smithi116 ceph-mon[36248]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:37.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:37 smithi023 ceph-mon[32466]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T10:39:38.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:37 smithi117 ceph-mon[36368]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T10:39:38.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:37 smithi116 ceph-mon[36248]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T10:39:38.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.824+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2695649683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b84103e70 msgr2=0x7f2b8410a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:38.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.824+0000 7f2b8b8c1700 1 --2- 172.21.15.23:0/2695649683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b84103e70 0x7f2b8410a720 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00b0a0 tx=0x7f2b7c00b3b0 comp rx=0 tx=0).stop 2024-03-24T10:39:38.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.824+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2695649683 shutdown_connections 2024-03-24T10:39:38.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.824+0000 7f2b8b8c1700 1 --2- 172.21.15.23:0/2695649683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b841004f0 0x7f2b84100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.824+0000 7f2b8b8c1700 1 --2- 172.21.15.23:0/2695649683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b84103e70 0x7f2b8410a720 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.824+0000 7f2b8b8c1700 1 --2- 172.21.15.23:0/2695649683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b841034d0 0x7f2b84103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.824+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2695649683 >> 172.21.15.23:0/2695649683 conn(0x7f2b840fbe10 msgr2=0x7f2b840fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.824+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2695649683 shutdown_connections 2024-03-24T10:39:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.824+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2695649683 wait complete. 2024-03-24T10:39:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.824+0000 7f2b8b8c1700 1 Processor -- start 2024-03-24T10:39:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.824+0000 7f2b8b8c1700 1 -- start start 2024-03-24T10:39:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b8b8c1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b841004f0 0x7f2b8419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b8b8c1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b841034d0 0x7f2b8419aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b8b8c1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b84103e70 0x7f2b841a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b8b8c1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2b841a2050 con 0x7f2b841034d0 2024-03-24T10:39:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b8b8c1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2b841a2190 con 0x7f2b84103e70 2024-03-24T10:39:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b8b8c1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2b841a2320 con 0x7f2b841004f0 2024-03-24T10:39:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b88e5c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b841034d0 0x7f2b8419aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:38.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b88e5c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b841034d0 0x7f2b8419aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60180/0 (socket says 172.21.15.23:60180) 2024-03-24T10:39:38.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b88e5c700 1 -- 172.21.15.23:0/2059258427 learned_addr learned my addr 172.21.15.23:0/2059258427 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:38.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b8965d700 1 --2- 172.21.15.23:0/2059258427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b841004f0 0x7f2b8419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:38.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b89e5e700 1 --2- 172.21.15.23:0/2059258427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b84103e70 0x7f2b841a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:38.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b88e5c700 1 -- 172.21.15.23:0/2059258427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b841004f0 msgr2=0x7f2b8419a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:38.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b88e5c700 1 --2- 172.21.15.23:0/2059258427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b841004f0 0x7f2b8419a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:38.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b88e5c700 1 -- 172.21.15.23:0/2059258427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b84103e70 msgr2=0x7f2b841a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:38.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b88e5c700 1 --2- 172.21.15.23:0/2059258427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b84103e70 0x7f2b841a1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:38.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b88e5c700 1 -- 172.21.15.23:0/2059258427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b7c009d00 con 0x7f2b841034d0 2024-03-24T10:39:38.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b88e5c700 1 --2- 172.21.15.23:0/2059258427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b841034d0 0x7f2b8419aa00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400e9d0 tx=0x7f2b7400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b7a7fc700 1 -- 172.21.15.23:0/2059258427 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74009790 con 0x7f2b841034d0 2024-03-24T10:39:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b7a7fc700 1 -- 172.21.15.23:0/2059258427 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b7400ee70 con 0x7f2b841034d0 2024-03-24T10:39:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2059258427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b841a2600 con 0x7f2b841034d0 2024-03-24T10:39:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b7a7fc700 1 -- 172.21.15.23:0/2059258427 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74018680 con 0x7f2b841034d0 2024-03-24T10:39:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.825+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2059258427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b841a2b50 con 0x7f2b841034d0 2024-03-24T10:39:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.826+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2059258427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b84068ec0 con 0x7f2b841034d0 2024-03-24T10:39:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.827+0000 7f2b7a7fc700 1 -- 172.21.15.23:0/2059258427 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f2b7400cbd0 con 0x7f2b841034d0 2024-03-24T10:39:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.828+0000 7f2b7a7fc700 1 --2- 172.21.15.23:0/2059258427 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2b700a2260 0x7f2b700a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.828+0000 7f2b8965d700 1 --2- 172.21.15.23:0/2059258427 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2b700a2260 0x7f2b700a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.828+0000 7f2b7a7fc700 1 -- 172.21.15.23:0/2059258427 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2b74014070 con 0x7f2b841034d0 2024-03-24T10:39:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.828+0000 7f2b8965d700 1 --2- 172.21.15.23:0/2059258427 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2b700a2260 0x7f2b700a4720 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2b8406a7a0 tx=0x7f2b80009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:38.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:38.833+0000 7f2b7a7fc700 1 -- 172.21.15.23:0/2059258427 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2b74053280 con 0x7f2b841034d0 2024-03-24T10:39:39.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:39.053+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2059258427 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f2b841029e0 con 0x7f2b700a2260 2024-03-24T10:39:39.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:39.066+0000 7f2b7a7fc700 1 -- 172.21.15.23:0/2059258427 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f2b841029e0 con 0x7f2b700a2260 2024-03-24T10:39:39.067 INFO:teuthology.orchestra.run.smithi023.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T10:39:39.067 INFO:teuthology.orchestra.run.smithi023.stdout:mds.a smithi023 running (97s) 16s ago 97s 15.1M - 19.0.0-1855-g21b3bd67 553463a90fd5 21230139e0cf 2024-03-24T10:39:39.067 INFO:teuthology.orchestra.run.smithi023.stdout:mds.b smithi116 running (94s) 28s ago 94s 26.0M - 19.0.0-1855-g21b3bd67 553463a90fd5 db10c050566f 2024-03-24T10:39:39.067 INFO:teuthology.orchestra.run.smithi023.stdout:mds.c smithi117 running (108s) 28s ago 107s 19.2M - 19.0.0-1855-g21b3bd67 553463a90fd5 9797d79d1973 2024-03-24T10:39:39.067 INFO:teuthology.orchestra.run.smithi023.stdout:mds.d smithi023 running (116s) 16s ago 115s 13.0M - 19.0.0-1855-g21b3bd67 553463a90fd5 8b05d03599b8 2024-03-24T10:39:39.067 INFO:teuthology.orchestra.run.smithi023.stdout:mds.e smithi116 running (113s) 28s ago 113s 19.1M - 19.0.0-1855-g21b3bd67 553463a90fd5 dd163f521ebd 2024-03-24T10:39:39.068 INFO:teuthology.orchestra.run.smithi023.stdout:mds.f smithi117 running (110s) 28s ago 110s 16.8M - 19.0.0-1855-g21b3bd67 553463a90fd5 aac3988c0274 2024-03-24T10:39:39.068 INFO:teuthology.orchestra.run.smithi023.stdout:mds.g smithi023 running (100s) 16s ago 99s 15.2M - 19.0.0-1855-g21b3bd67 553463a90fd5 399d431e6587 2024-03-24T10:39:39.068 INFO:teuthology.orchestra.run.smithi023.stdout:mds.h smithi116 running (105s) 28s ago 105s 17.6M - 19.0.0-1855-g21b3bd67 553463a90fd5 f88696604076 2024-03-24T10:39:39.068 INFO:teuthology.orchestra.run.smithi023.stdout:mds.i smithi117 running (102s) 28s ago 102s 19.8M - 19.0.0-1855-g21b3bd67 553463a90fd5 467f37d8aace 2024-03-24T10:39:39.068 INFO:teuthology.orchestra.run.smithi023.stdout:mds.j smithi023 running (118s) 16s ago 118s 16.2M - 19.0.0-1855-g21b3bd67 553463a90fd5 e83389afb056 2024-03-24T10:39:39.068 INFO:teuthology.orchestra.run.smithi023.stdout:mds.k smithi116 running (2m) 28s ago 2m 15.3M - 19.0.0-1855-g21b3bd67 553463a90fd5 44f26d052fc5 2024-03-24T10:39:39.068 INFO:teuthology.orchestra.run.smithi023.stdout:mds.l smithi117 running (2m) 28s ago 2m 12.9M - 19.0.0-1855-g21b3bd67 553463a90fd5 f2de3d30da4a 2024-03-24T10:39:39.068 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.x smithi023 *:9283,8765 running (9m) 16s ago 9m 512M - 19.0.0-1855-g21b3bd67 553463a90fd5 52b2cf3da6ce 2024-03-24T10:39:39.068 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.y smithi116 *:8443,8765 running (7m) 28s ago 7m 451M - 19.0.0-1855-g21b3bd67 553463a90fd5 b69d43a495ff 2024-03-24T10:39:39.068 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.z smithi117 *:8443,8765 running (7m) 28s ago 7m 453M - 19.0.0-1855-g21b3bd67 553463a90fd5 574d0e6e20f7 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:mon.a smithi023 running (9m) 16s ago 9m 50.1M 2048M 19.0.0-1855-g21b3bd67 553463a90fd5 fc7c0651aa5d 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:mon.b smithi116 running (7m) 28s ago 7m 40.9M 2048M 19.0.0-1855-g21b3bd67 553463a90fd5 c9b7a171dc13 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:mon.c smithi117 running (7m) 28s ago 7m 39.5M 2048M 19.0.0-1855-g21b3bd67 553463a90fd5 08756fdf33ea 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 smithi023 running (6m) 16s ago 6m 55.3M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 a71253101cd5 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 smithi116 running (6m) 28s ago 6m 56.9M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 9424386b0c33 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 smithi117 running (6m) 28s ago 6m 49.3M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 4e9e5e415fa3 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3 smithi023 running (5m) 16s ago 5m 49.3M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 caadd884494e 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4 smithi116 running (5m) 28s ago 5m 51.7M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 bbb20a66ceae 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5 smithi117 running (4m) 28s ago 4m 44.5M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 0cf35610b52a 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6 smithi023 running (4m) 16s ago 4m 50.9M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 029b4e00ff5e 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7 smithi116 running (4m) 28s ago 4m 47.2M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 dccc619f7d34 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:osd.8 smithi117 running (3m) 28s ago 3m 52.0M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 8b8176871a14 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:osd.9 smithi023 running (3m) 16s ago 3m 43.9M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 515edaca3bf3 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:osd.10 smithi116 running (2m) 28s ago 2m 53.3M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 3def853a5cac 2024-03-24T10:39:39.069 INFO:teuthology.orchestra.run.smithi023.stdout:osd.11 smithi117 running (2m) 28s ago 2m 49.6M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 38e3f907c64c 2024-03-24T10:39:39.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:39.068+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2059258427 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2b700a2260 msgr2=0x7f2b700a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:39.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:39.068+0000 7f2b8b8c1700 1 --2- 172.21.15.23:0/2059258427 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2b700a2260 0x7f2b700a4720 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2b8406a7a0 tx=0x7f2b80009260 comp rx=0 tx=0).stop 2024-03-24T10:39:39.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:39.068+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2059258427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b841034d0 msgr2=0x7f2b8419aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:39.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:39.068+0000 7f2b8b8c1700 1 --2- 172.21.15.23:0/2059258427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b841034d0 0x7f2b8419aa00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400e9d0 tx=0x7f2b7400ece0 comp rx=0 tx=0).stop 2024-03-24T10:39:39.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:39.068+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2059258427 shutdown_connections 2024-03-24T10:39:39.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:39.069+0000 7f2b8b8c1700 1 --2- 172.21.15.23:0/2059258427 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2b700a2260 0x7f2b700a4720 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:39.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:39.069+0000 7f2b8b8c1700 1 --2- 172.21.15.23:0/2059258427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b841004f0 0x7f2b8419a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:39.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:39.069+0000 7f2b8b8c1700 1 --2- 172.21.15.23:0/2059258427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b84103e70 0x7f2b841a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:39.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:39.069+0000 7f2b8b8c1700 1 --2- 172.21.15.23:0/2059258427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b841034d0 0x7f2b8419aa00 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:39.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:39.069+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2059258427 >> 172.21.15.23:0/2059258427 conn(0x7f2b840fbe10 msgr2=0x7f2b841012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:39.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:39.069+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2059258427 shutdown_connections 2024-03-24T10:39:39.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:39.069+0000 7f2b8b8c1700 1 -- 172.21.15.23:0/2059258427 wait complete. 2024-03-24T10:39:39.785 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-24T10:39:40.032 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:40.154 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:39 smithi023 ceph-mon[32466]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T10:39:40.154 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:39 smithi023 ceph-mon[32466]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:40.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:39 smithi117 ceph-mon[36368]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T10:39:40.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:39 smithi117 ceph-mon[36368]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:40.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:39 smithi116 ceph-mon[36248]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T10:39:40.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:39 smithi116 ceph-mon[36248]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:41.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.593+0000 7f6cf4dab700 1 -- 172.21.15.23:0/1794534123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6cf0102cd0 msgr2=0x7f6cf0109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:41.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.593+0000 7f6cf4dab700 1 --2- 172.21.15.23:0/1794534123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6cf0102cd0 0x7f6cf0109580 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6ce000b0d0 tx=0x7f6ce000b3e0 comp rx=0 tx=0).stop 2024-03-24T10:39:41.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.593+0000 7f6cf4dab700 1 -- 172.21.15.23:0/1794534123 shutdown_connections 2024-03-24T10:39:41.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.593+0000 7f6cf4dab700 1 --2- 172.21.15.23:0/1794534123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cf0101130 0x7f6cf0101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:41.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.593+0000 7f6cf4dab700 1 --2- 172.21.15.23:0/1794534123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6cf0102cd0 0x7f6cf0109580 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:41.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.593+0000 7f6cf4dab700 1 --2- 172.21.15.23:0/1794534123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cf0102330 0x7f6cf0102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:41.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.593+0000 7f6cf4dab700 1 -- 172.21.15.23:0/1794534123 >> 172.21.15.23:0/1794534123 conn(0x7f6cf00fc6d0 msgr2=0x7f6cf00feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:41.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.593+0000 7f6cf4dab700 1 -- 172.21.15.23:0/1794534123 shutdown_connections 2024-03-24T10:39:41.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.593+0000 7f6cf4dab700 1 -- 172.21.15.23:0/1794534123 wait complete. 2024-03-24T10:39:41.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cf4dab700 1 Processor -- start 2024-03-24T10:39:41.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cf4dab700 1 -- start start 2024-03-24T10:39:41.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cf4dab700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cf0101130 0x7f6cf019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:41.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cf4dab700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cf0102330 0x7f6cf019aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:41.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cf4dab700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6cf0102cd0 0x7f6cf01a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:41.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cf4dab700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf01a2050 con 0x7f6cf0102cd0 2024-03-24T10:39:41.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cf4dab700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf01a2190 con 0x7f6cf0102330 2024-03-24T10:39:41.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cf4dab700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf01a2320 con 0x7f6cf0101130 2024-03-24T10:39:41.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cee59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cf0101130 0x7f6cf019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:41.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cee59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cf0101130 0x7f6cf019a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:36212/0 (socket says 172.21.15.23:36212) 2024-03-24T10:39:41.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cee59c700 1 -- 172.21.15.23:0/4102035507 learned_addr learned my addr 172.21.15.23:0/4102035507 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:41.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6ceed9d700 1 --2- 172.21.15.23:0/4102035507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6cf0102cd0 0x7f6cf01a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:41.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cedd9b700 1 --2- 172.21.15.23:0/4102035507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cf0102330 0x7f6cf019aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:41.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cee59c700 1 -- 172.21.15.23:0/4102035507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cf0102330 msgr2=0x7f6cf019aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:41.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cee59c700 1 --2- 172.21.15.23:0/4102035507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cf0102330 0x7f6cf019aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:41.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cee59c700 1 -- 172.21.15.23:0/4102035507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6cf0102cd0 msgr2=0x7f6cf01a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:41.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cee59c700 1 --2- 172.21.15.23:0/4102035507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6cf0102cd0 0x7f6cf01a1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:41.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.594+0000 7f6cee59c700 1 -- 172.21.15.23:0/4102035507 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce0009d00 con 0x7f6cf0101130 2024-03-24T10:39:41.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.595+0000 7f6cee59c700 1 --2- 172.21.15.23:0/4102035507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cf0101130 0x7f6cf019a4c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800e9a0 tx=0x7f6ce800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:41.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.595+0000 7f6ce77fe700 1 -- 172.21.15.23:0/4102035507 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8009790 con 0x7f6cf0101130 2024-03-24T10:39:41.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.595+0000 7f6cf4dab700 1 -- 172.21.15.23:0/4102035507 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf01a2600 con 0x7f6cf0101130 2024-03-24T10:39:41.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.595+0000 7f6ce77fe700 1 -- 172.21.15.23:0/4102035507 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ce800ccf0 con 0x7f6cf0101130 2024-03-24T10:39:41.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.595+0000 7f6ce77fe700 1 -- 172.21.15.23:0/4102035507 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8003d20 con 0x7f6cf0101130 2024-03-24T10:39:41.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.595+0000 7f6cf4dab700 1 -- 172.21.15.23:0/4102035507 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf01a2b00 con 0x7f6cf0101130 2024-03-24T10:39:41.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.596+0000 7f6cf4dab700 1 -- 172.21.15.23:0/4102035507 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf0068ec0 con 0x7f6cf0101130 2024-03-24T10:39:41.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.597+0000 7f6ce77fe700 1 -- 172.21.15.23:0/4102035507 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f6ce8004390 con 0x7f6cf0101130 2024-03-24T10:39:41.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.597+0000 7f6ce77fe700 1 --2- 172.21.15.23:0/4102035507 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6cdc0a21e0 0x7f6cdc0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:41.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.597+0000 7f6cedd9b700 1 --2- 172.21.15.23:0/4102035507 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6cdc0a21e0 0x7f6cdc0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:41.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.597+0000 7f6ce77fe700 1 -- 172.21.15.23:0/4102035507 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6ce8014070 con 0x7f6cf0101130 2024-03-24T10:39:41.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.598+0000 7f6cedd9b700 1 --2- 172.21.15.23:0/4102035507 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6cdc0a21e0 0x7f6cdc0a46a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8009c60 tx=0x7f6cd8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:41.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.601+0000 7f6ce77fe700 1 -- 172.21.15.23:0/4102035507 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6ce8096e40 con 0x7f6cf0101130 2024-03-24T10:39:41.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.814+0000 7f6cf4dab700 1 -- 172.21.15.23:0/4102035507 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f6cf006afb0 con 0x7f6cdc0a21e0 2024-03-24T10:39:41.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.820+0000 7f6ce77fe700 1 -- 172.21.15.23:0/4102035507 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f6cf006afb0 con 0x7f6cdc0a21e0 2024-03-24T10:39:41.820 INFO:teuthology.orchestra.run.smithi023.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-24T10:39:41.821 INFO:teuthology.orchestra.run.smithi023.stdout:mds.all 12/12 31s ago 2m smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi117=c;smithi117=f;smithi117=i;smithi117=l;count:12 2024-03-24T10:39:41.821 INFO:teuthology.orchestra.run.smithi023.stdout:mgr 3/3 31s ago 7m smithi023=x;smithi116=y;smithi117=z;count:3 2024-03-24T10:39:41.821 INFO:teuthology.orchestra.run.smithi023.stdout:mon 3/3 31s ago 7m smithi023:172.21.15.23=a;smithi116:172.21.15.116=b;smithi117:172.21.15.117=c;count:3 2024-03-24T10:39:41.821 INFO:teuthology.orchestra.run.smithi023.stdout:osd 12 31s ago - 2024-03-24T10:39:41.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.822+0000 7f6cf4dab700 1 -- 172.21.15.23:0/4102035507 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6cdc0a21e0 msgr2=0x7f6cdc0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:41.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.822+0000 7f6cf4dab700 1 --2- 172.21.15.23:0/4102035507 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6cdc0a21e0 0x7f6cdc0a46a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8009c60 tx=0x7f6cd8009260 comp rx=0 tx=0).stop 2024-03-24T10:39:41.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.822+0000 7f6cf4dab700 1 -- 172.21.15.23:0/4102035507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cf0101130 msgr2=0x7f6cf019a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:41.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.822+0000 7f6cf4dab700 1 --2- 172.21.15.23:0/4102035507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cf0101130 0x7f6cf019a4c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800e9a0 tx=0x7f6ce800ecb0 comp rx=0 tx=0).stop 2024-03-24T10:39:41.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.822+0000 7f6cf4dab700 1 -- 172.21.15.23:0/4102035507 shutdown_connections 2024-03-24T10:39:41.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.822+0000 7f6cf4dab700 1 --2- 172.21.15.23:0/4102035507 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f6cdc0a21e0 0x7f6cdc0a46a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:41.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.822+0000 7f6cf4dab700 1 --2- 172.21.15.23:0/4102035507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cf0101130 0x7f6cf019a4c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:41.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.822+0000 7f6cf4dab700 1 --2- 172.21.15.23:0/4102035507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6cf0102cd0 0x7f6cf01a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:41.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.822+0000 7f6cf4dab700 1 --2- 172.21.15.23:0/4102035507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cf0102330 0x7f6cf019aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:41.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.822+0000 7f6cf4dab700 1 -- 172.21.15.23:0/4102035507 >> 172.21.15.23:0/4102035507 conn(0x7f6cf00fc6d0 msgr2=0x7f6cf006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:41.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.822+0000 7f6cf4dab700 1 -- 172.21.15.23:0/4102035507 shutdown_connections 2024-03-24T10:39:41.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:41.822+0000 7f6cf4dab700 1 -- 172.21.15.23:0/4102035507 wait complete. 2024-03-24T10:39:42.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:41 smithi023 ceph-mon[32466]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:39:42.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:41 smithi117 ceph-mon[36368]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:39:42.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:41 smithi116 ceph-mon[36248]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:39:42.439 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-24T10:39:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:43.114 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:42 smithi023 ceph-mon[32466]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:43.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:42 smithi117 ceph-mon[36368]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:43.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:42 smithi116 ceph-mon[36248]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:44.138 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:43 smithi023 ceph-mon[32466]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:39:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.241+0000 7fb3a3107700 1 -- 172.21.15.23:0/2696896523 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb39c102230 msgr2=0x7fb39c102650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.241+0000 7fb3a3107700 1 --2- 172.21.15.23:0/2696896523 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb39c102230 0x7fb39c102650 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb3900099c0 tx=0x7fb390009cd0 comp rx=0 tx=0).stop 2024-03-24T10:39:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.241+0000 7fb3a3107700 1 -- 172.21.15.23:0/2696896523 shutdown_connections 2024-03-24T10:39:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.241+0000 7fb3a3107700 1 --2- 172.21.15.23:0/2696896523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb39c103dd0 0x7fb39c10a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.241+0000 7fb3a3107700 1 --2- 172.21.15.23:0/2696896523 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb39c102230 0x7fb39c102650 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.241+0000 7fb3a3107700 1 --2- 172.21.15.23:0/2696896523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb39c103430 0x7fb39c103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.241+0000 7fb3a3107700 1 -- 172.21.15.23:0/2696896523 >> 172.21.15.23:0/2696896523 conn(0x7fb39c0fd810 msgr2=0x7fb39c0ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.241+0000 7fb3a3107700 1 -- 172.21.15.23:0/2696896523 shutdown_connections 2024-03-24T10:39:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.241+0000 7fb3a3107700 1 -- 172.21.15.23:0/2696896523 wait complete. 2024-03-24T10:39:44.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.242+0000 7fb3a3107700 1 Processor -- start 2024-03-24T10:39:44.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.242+0000 7fb3a3107700 1 -- start start 2024-03-24T10:39:44.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.242+0000 7fb3a3107700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb39c102230 0x7fb39c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:44.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.242+0000 7fb3a3107700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb39c103430 0x7fb39c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:44.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.242+0000 7fb3a3107700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb39c103dd0 0x7fb39c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:44.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.242+0000 7fb3a3107700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c19fd80 con 0x7fb39c102230 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.242+0000 7fb3a3107700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c19fec0 con 0x7fb39c103430 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.242+0000 7fb3a3107700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c1a0050 con 0x7fb39c103dd0 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.242+0000 7fb3a0ea3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb39c102230 0x7fb39c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.242+0000 7fb3a0ea3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb39c102230 0x7fb39c1981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60222/0 (socket says 172.21.15.23:60222) 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.242+0000 7fb3a0ea3700 1 -- 172.21.15.23:0/4029584056 learned_addr learned my addr 172.21.15.23:0/4029584056 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.242+0000 7fb39bfff700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb39c103430 0x7fb39c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.242+0000 7fb3a16a4700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb39c103dd0 0x7fb39c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.243+0000 7fb3a0ea3700 1 -- 172.21.15.23:0/4029584056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb39c103dd0 msgr2=0x7fb39c19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.243+0000 7fb3a0ea3700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb39c103dd0 0x7fb39c19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.243+0000 7fb3a0ea3700 1 -- 172.21.15.23:0/4029584056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb39c103430 msgr2=0x7fb39c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.243+0000 7fb3a0ea3700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb39c103430 0x7fb39c198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.243+0000 7fb3a0ea3700 1 -- 172.21.15.23:0/4029584056 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3900095f0 con 0x7fb39c102230 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.243+0000 7fb39bfff700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb39c103430 0x7fb39c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.243+0000 7fb3a0ea3700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb39c102230 0x7fb39c1981f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb390009fd0 tx=0x7fb39000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.243+0000 7fb399ffb700 1 -- 172.21.15.23:0/4029584056 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39001d070 con 0x7fb39c102230 2024-03-24T10:39:44.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.243+0000 7fb399ffb700 1 -- 172.21.15.23:0/4029584056 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb390005940 con 0x7fb39c102230 2024-03-24T10:39:44.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.243+0000 7fb3a3107700 1 -- 172.21.15.23:0/4029584056 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb39c1a02d0 con 0x7fb39c102230 2024-03-24T10:39:44.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.243+0000 7fb399ffb700 1 -- 172.21.15.23:0/4029584056 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb390022e90 con 0x7fb39c102230 2024-03-24T10:39:44.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.243+0000 7fb3a3107700 1 -- 172.21.15.23:0/4029584056 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb39c1a07c0 con 0x7fb39c102230 2024-03-24T10:39:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.244+0000 7fb3a3107700 1 -- 172.21.15.23:0/4029584056 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb39c109670 con 0x7fb39c102230 2024-03-24T10:39:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.245+0000 7fb399ffb700 1 -- 172.21.15.23:0/4029584056 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fb390022470 con 0x7fb39c102230 2024-03-24T10:39:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.246+0000 7fb399ffb700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb38c0a2260 0x7fb38c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.246+0000 7fb39bfff700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb38c0a2260 0x7fb38c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.246+0000 7fb399ffb700 1 -- 172.21.15.23:0/4029584056 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb3900cec70 con 0x7fb39c102230 2024-03-24T10:39:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.246+0000 7fb39bfff700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb38c0a2260 0x7fb38c0a4720 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb388011490 tx=0x7fb388009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:44.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.250+0000 7fb399ffb700 1 -- 172.21.15.23:0/4029584056 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb39005e280 con 0x7fb39c102230 2024-03-24T10:39:44.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:43 smithi117 ceph-mon[36368]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:39:44.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:43 smithi116 ceph-mon[36248]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:39:44.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.459+0000 7fb3a3107700 1 -- 172.21.15.23:0/4029584056 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fb39c02dc00 con 0x7fb38c0a2260 2024-03-24T10:39:44.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.460+0000 7fb399ffb700 1 -- 172.21.15.23:0/4029584056 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fb39c02dc00 con 0x7fb38c0a2260 2024-03-24T10:39:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:HOST ADDR LABELS STATUS 2024-03-24T10:39:44.460 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 172.21.15.23 2024-03-24T10:39:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:smithi116 172.21.15.116 2024-03-24T10:39:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:smithi117 172.21.15.117 2024-03-24T10:39:44.461 INFO:teuthology.orchestra.run.smithi023.stdout:3 hosts in cluster 2024-03-24T10:39:44.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.462+0000 7fb3a3107700 1 -- 172.21.15.23:0/4029584056 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb38c0a2260 msgr2=0x7fb38c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:44.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.462+0000 7fb3a3107700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb38c0a2260 0x7fb38c0a4720 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb388011490 tx=0x7fb388009260 comp rx=0 tx=0).stop 2024-03-24T10:39:44.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.462+0000 7fb3a3107700 1 -- 172.21.15.23:0/4029584056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb39c102230 msgr2=0x7fb39c1981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:44.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.462+0000 7fb3a3107700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb39c102230 0x7fb39c1981f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb390009fd0 tx=0x7fb39000f690 comp rx=0 tx=0).stop 2024-03-24T10:39:44.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.462+0000 7fb3a3107700 1 -- 172.21.15.23:0/4029584056 shutdown_connections 2024-03-24T10:39:44.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.462+0000 7fb3a3107700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fb38c0a2260 0x7fb38c0a4720 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:44.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.462+0000 7fb3a3107700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb39c103dd0 0x7fb39c19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:44.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.462+0000 7fb3a3107700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb39c102230 0x7fb39c1981f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:44.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.462+0000 7fb3a3107700 1 --2- 172.21.15.23:0/4029584056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb39c103430 0x7fb39c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:44.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.462+0000 7fb3a3107700 1 -- 172.21.15.23:0/4029584056 >> 172.21.15.23:0/4029584056 conn(0x7fb39c0fd810 msgr2=0x7fb39c1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:44.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.462+0000 7fb3a3107700 1 -- 172.21.15.23:0/4029584056 shutdown_connections 2024-03-24T10:39:44.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:44.462+0000 7fb3a3107700 1 -- 172.21.15.23:0/4029584056 wait complete. 2024-03-24T10:39:45.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:44 smithi023 ceph-mon[32466]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:45.051 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-24T10:39:45.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:44 smithi117 ceph-mon[36368]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:45.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:44 smithi116 ceph-mon[36248]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:45.292 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:46.065 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:45 smithi023 ceph-mon[32466]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:39:46.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:45 smithi117 ceph-mon[36368]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:39:46.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:45 smithi116 ceph-mon[36248]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T10:39:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.891+0000 7febe9eb7700 1 -- 172.21.15.23:0/2251568661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febe4100540 msgr2=0x7febe4100960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.891+0000 7febe9eb7700 1 --2- 172.21.15.23:0/2251568661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febe4100540 0x7febe4100960 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7febd4009990 tx=0x7febd4009ca0 comp rx=0 tx=0).stop 2024-03-24T10:39:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.891+0000 7febe9eb7700 1 -- 172.21.15.23:0/2251568661 shutdown_connections 2024-03-24T10:39:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.891+0000 7febe9eb7700 1 --2- 172.21.15.23:0/2251568661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febe4103ec0 0x7febe410a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.891+0000 7febe9eb7700 1 --2- 172.21.15.23:0/2251568661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febe4100540 0x7febe4100960 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.891+0000 7febe9eb7700 1 --2- 172.21.15.23:0/2251568661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4103520 0x7febe4103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.891+0000 7febe9eb7700 1 -- 172.21.15.23:0/2251568661 >> 172.21.15.23:0/2251568661 conn(0x7febe40fbea0 msgr2=0x7febe40fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.891+0000 7febe9eb7700 1 -- 172.21.15.23:0/2251568661 shutdown_connections 2024-03-24T10:39:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.891+0000 7febe9eb7700 1 -- 172.21.15.23:0/2251568661 wait complete. 2024-03-24T10:39:46.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.892+0000 7febe9eb7700 1 Processor -- start 2024-03-24T10:39:46.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.892+0000 7febe9eb7700 1 -- start start 2024-03-24T10:39:46.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.892+0000 7febe9eb7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febe4100540 0x7febe41981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:46.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.892+0000 7febe9eb7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4103520 0x7febe41986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:46.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.892+0000 7febe9eb7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febe4103ec0 0x7febe419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:46.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.892+0000 7febe9eb7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7febe419fd40 con 0x7febe4103ec0 2024-03-24T10:39:46.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.892+0000 7febe9eb7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7febe419fe80 con 0x7febe4103520 2024-03-24T10:39:46.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.892+0000 7febe9eb7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7febe41a0010 con 0x7febe4100540 2024-03-24T10:39:46.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.892+0000 7febe37fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febe4100540 0x7febe41981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:46.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.892+0000 7febe3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febe4103ec0 0x7febe419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:46.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe37fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febe4100540 0x7febe41981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:36242/0 (socket says 172.21.15.23:36242) 2024-03-24T10:39:46.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe37fe700 1 -- 172.21.15.23:0/536671831 learned_addr learned my addr 172.21.15.23:0/536671831 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febe4103ec0 0x7febe419f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60244/0 (socket says 172.21.15.23:60244) 2024-03-24T10:39:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe2ffd700 1 --2- 172.21.15.23:0/536671831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4103520 0x7febe41986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe37fe700 1 -- 172.21.15.23:0/536671831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4103520 msgr2=0x7febe41986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe37fe700 1 --2- 172.21.15.23:0/536671831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4103520 0x7febe41986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe37fe700 1 -- 172.21.15.23:0/536671831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febe4103ec0 msgr2=0x7febe419f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe37fe700 1 --2- 172.21.15.23:0/536671831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febe4103ec0 0x7febe419f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe37fe700 1 -- 172.21.15.23:0/536671831 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd40095f0 con 0x7febe4100540 2024-03-24T10:39:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe3fff700 1 --2- 172.21.15.23:0/536671831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febe4103ec0 0x7febe419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe37fe700 1 --2- 172.21.15.23:0/536671831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febe4100540 0x7febe41981b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7febd40098e0 tx=0x7febd4005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe0ff9700 1 -- 172.21.15.23:0/536671831 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd401d070 con 0x7febe4100540 2024-03-24T10:39:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe9eb7700 1 -- 172.21.15.23:0/536671831 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe41a0290 con 0x7febe4100540 2024-03-24T10:39:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe0ff9700 1 -- 172.21.15.23:0/536671831 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febd4021410 con 0x7febe4100540 2024-03-24T10:39:46.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe0ff9700 1 -- 172.21.15.23:0/536671831 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd4017650 con 0x7febe4100540 2024-03-24T10:39:46.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.893+0000 7febe9eb7700 1 -- 172.21.15.23:0/536671831 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe41a0700 con 0x7febe4100540 2024-03-24T10:39:46.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.894+0000 7febe9eb7700 1 -- 172.21.15.23:0/536671831 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febe41096e0 con 0x7febe4100540 2024-03-24T10:39:46.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.895+0000 7febe0ff9700 1 -- 172.21.15.23:0/536671831 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7febd4021580 con 0x7febe4100540 2024-03-24T10:39:46.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.896+0000 7febe0ff9700 1 --2- 172.21.15.23:0/536671831 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7febcc0a2150 0x7febcc0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:46.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.896+0000 7febe2ffd700 1 --2- 172.21.15.23:0/536671831 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7febcc0a2150 0x7febcc0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:46.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.896+0000 7febe0ff9700 1 -- 172.21.15.23:0/536671831 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7febd40cef80 con 0x7febe4100540 2024-03-24T10:39:46.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.896+0000 7febe2ffd700 1 --2- 172.21.15.23:0/536671831 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7febcc0a2150 0x7febcc0a4610 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7febd8009c60 tx=0x7febd8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:46.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:46.900+0000 7febe0ff9700 1 -- 172.21.15.23:0/536671831 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7febd4097f60 con 0x7febe4100540 2024-03-24T10:39:47.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:47.113+0000 7febe9eb7700 1 -- 172.21.15.23:0/536671831 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7febe402db60 con 0x7febcc0a2150 2024-03-24T10:39:47.114 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:46 smithi023 ceph-mon[32466]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-03-24T10:39:47.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:47.115+0000 7febe0ff9700 1 -- 172.21.15.23:0/536671831 <== mgr.14152 v2:172.21.15.23:6800/2350021632 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7febe402db60 con 0x7febcc0a2150 2024-03-24T10:39:47.116 INFO:teuthology.orchestra.run.smithi023.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-24T10:39:47.116 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN 3726G No 3m ago Has a FileSystem, LVM detected 2024-03-24T10:39:47.116 INFO:teuthology.orchestra.run.smithi023.stdout:smithi116 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VJ400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-24T10:39:47.116 INFO:teuthology.orchestra.run.smithi023.stdout:smithi117 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400V6400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-24T10:39:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:47.117+0000 7febe9eb7700 1 -- 172.21.15.23:0/536671831 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7febcc0a2150 msgr2=0x7febcc0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:47.117+0000 7febe9eb7700 1 --2- 172.21.15.23:0/536671831 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7febcc0a2150 0x7febcc0a4610 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7febd8009c60 tx=0x7febd8009260 comp rx=0 tx=0).stop 2024-03-24T10:39:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:47.117+0000 7febe9eb7700 1 -- 172.21.15.23:0/536671831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febe4100540 msgr2=0x7febe41981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:47.117+0000 7febe9eb7700 1 --2- 172.21.15.23:0/536671831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febe4100540 0x7febe41981b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7febd40098e0 tx=0x7febd4005740 comp rx=0 tx=0).stop 2024-03-24T10:39:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:47.118+0000 7febe9eb7700 1 -- 172.21.15.23:0/536671831 shutdown_connections 2024-03-24T10:39:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:47.118+0000 7febe9eb7700 1 --2- 172.21.15.23:0/536671831 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7febcc0a2150 0x7febcc0a4610 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:47.118+0000 7febe9eb7700 1 --2- 172.21.15.23:0/536671831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febe4100540 0x7febe41981b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:47.118+0000 7febe9eb7700 1 --2- 172.21.15.23:0/536671831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febe4103ec0 0x7febe419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:47.118+0000 7febe9eb7700 1 --2- 172.21.15.23:0/536671831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4103520 0x7febe41986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:47.118+0000 7febe9eb7700 1 -- 172.21.15.23:0/536671831 >> 172.21.15.23:0/536671831 conn(0x7febe40fbea0 msgr2=0x7febe4106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:47.118+0000 7febe9eb7700 1 -- 172.21.15.23:0/536671831 shutdown_connections 2024-03-24T10:39:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:47.118+0000 7febe9eb7700 1 -- 172.21.15.23:0/536671831 wait complete. 2024-03-24T10:39:47.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:46 smithi117 ceph-mon[36368]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-03-24T10:39:47.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:46 smithi116 ceph-mon[36248]: mgrmap e18: x(active, since 8m), standbys: y, z 2024-03-24T10:39:47.695 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T10:39:47.704 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T10:39:47.704 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-24T10:39:47.954 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:48.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:47 smithi117 ceph-mon[36368]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:39:48.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:47 smithi117 ceph-mon[36368]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:48.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:47 smithi116 ceph-mon[36248]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:39:48.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:47 smithi116 ceph-mon[36248]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:48.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:47 smithi023 ceph-mon[32466]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:39:48.268 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:47 smithi023 ceph-mon[32466]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:39:49.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.529+0000 7f8b61e30700 1 -- 172.21.15.23:0/655624170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5c102d00 msgr2=0x7f8b5c1095b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:49.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.529+0000 7f8b61e30700 1 --2- 172.21.15.23:0/655624170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5c102d00 0x7f8b5c1095b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8b540099c0 tx=0x7f8b54009cd0 comp rx=0 tx=0).stop 2024-03-24T10:39:49.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.529+0000 7f8b61e30700 1 -- 172.21.15.23:0/655624170 shutdown_connections 2024-03-24T10:39:49.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.529+0000 7f8b61e30700 1 --2- 172.21.15.23:0/655624170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b5c101160 0x7f8b5c101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:49.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.529+0000 7f8b61e30700 1 --2- 172.21.15.23:0/655624170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5c102d00 0x7f8b5c1095b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:49.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.529+0000 7f8b61e30700 1 --2- 172.21.15.23:0/655624170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b5c102360 0x7f8b5c1027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:49.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.529+0000 7f8b61e30700 1 -- 172.21.15.23:0/655624170 >> 172.21.15.23:0/655624170 conn(0x7f8b5c0fc720 msgr2=0x7f8b5c0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:49.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.529+0000 7f8b61e30700 1 -- 172.21.15.23:0/655624170 shutdown_connections 2024-03-24T10:39:49.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.529+0000 7f8b61e30700 1 -- 172.21.15.23:0/655624170 wait complete. 2024-03-24T10:39:49.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b61e30700 1 Processor -- start 2024-03-24T10:39:49.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b61e30700 1 -- start start 2024-03-24T10:39:49.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b61e30700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b5c101160 0x7f8b5c10ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:49.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b61e30700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5c102360 0x7f8b5c10a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:49.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b61e30700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b5c102d00 0x7f8b5c10a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:49.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b61e30700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5c10ad80 con 0x7f8b5c102360 2024-03-24T10:39:49.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b61e30700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5c10aef0 con 0x7f8b5c102d00 2024-03-24T10:39:49.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b61e30700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5c1a21a0 con 0x7f8b5c101160 2024-03-24T10:39:49.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b5affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5c102360 0x7f8b5c10a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:49.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b5b7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b5c101160 0x7f8b5c10ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:49.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b5affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5c102360 0x7f8b5c10a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53184/0 (socket says 172.21.15.23:53184) 2024-03-24T10:39:49.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b5affd700 1 -- 172.21.15.23:0/883159819 learned_addr learned my addr 172.21.15.23:0/883159819 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:49.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b5bfff700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b5c102d00 0x7f8b5c10a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:49.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b5affd700 1 -- 172.21.15.23:0/883159819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b5c101160 msgr2=0x7f8b5c10ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:49.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b5affd700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b5c101160 0x7f8b5c10ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:49.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b5affd700 1 -- 172.21.15.23:0/883159819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b5c102d00 msgr2=0x7f8b5c10a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:49.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.530+0000 7f8b5affd700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b5c102d00 0x7f8b5c10a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:49.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.531+0000 7f8b5affd700 1 -- 172.21.15.23:0/883159819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b540095f0 con 0x7f8b5c102360 2024-03-24T10:39:49.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.531+0000 7f8b5b7fe700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b5c101160 0x7f8b5c10ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:49.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.531+0000 7f8b5bfff700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b5c102d00 0x7f8b5c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:49.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.531+0000 7f8b5affd700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5c102360 0x7f8b5c10a0c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8b4400e9a0 tx=0x7f8b4400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:49.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.531+0000 7f8b58ff9700 1 -- 172.21.15.23:0/883159819 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b44009790 con 0x7f8b5c102360 2024-03-24T10:39:49.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.531+0000 7f8b58ff9700 1 -- 172.21.15.23:0/883159819 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b44010430 con 0x7f8b5c102360 2024-03-24T10:39:49.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.531+0000 7f8b61e30700 1 -- 172.21.15.23:0/883159819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b5c1a2480 con 0x7f8b5c102360 2024-03-24T10:39:49.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.531+0000 7f8b58ff9700 1 -- 172.21.15.23:0/883159819 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b44018680 con 0x7f8b5c102360 2024-03-24T10:39:49.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.531+0000 7f8b61e30700 1 -- 172.21.15.23:0/883159819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b5c1a29d0 con 0x7f8b5c102360 2024-03-24T10:39:49.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.532+0000 7f8b61e30700 1 -- 172.21.15.23:0/883159819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b5c108450 con 0x7f8b5c102360 2024-03-24T10:39:49.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.533+0000 7f8b58ff9700 1 -- 172.21.15.23:0/883159819 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f8b44010910 con 0x7f8b5c102360 2024-03-24T10:39:49.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.534+0000 7f8b58ff9700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8b480a21d0 0x7f8b480a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:49.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.534+0000 7f8b5b7fe700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8b480a21d0 0x7f8b480a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:49.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.534+0000 7f8b58ff9700 1 -- 172.21.15.23:0/883159819 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f8b44014070 con 0x7f8b5c102360 2024-03-24T10:39:49.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.534+0000 7f8b5b7fe700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8b480a21d0 0x7f8b480a4690 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c1021c0 tx=0x7f8b4c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:49.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.538+0000 7f8b58ff9700 1 -- 172.21.15.23:0/883159819 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8b44053280 con 0x7f8b5c102360 2024-03-24T10:39:49.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.792+0000 7f8b61e30700 1 -- 172.21.15.23:0/883159819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f8b5c02daf0 con 0x7f8b5c102360 2024-03-24T10:39:49.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.793+0000 7f8b58ff9700 1 -- 172.21.15.23:0/883159819 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) ==== 76+0+3046 (secure 0 0 0) 0x7f8b44096e00 con 0x7f8b5c102360 2024-03-24T10:39:49.793 INFO:teuthology.orchestra.run.smithi023.stdout:e22 2024-03-24T10:39:49.794 INFO:teuthology.orchestra.run.smithi023.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T10:39:49.794 INFO:teuthology.orchestra.run.smithi023.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-24T10:39:49.794 INFO:teuthology.orchestra.run.smithi023.stdout:legacy client fscid: 1 2024-03-24T10:39:49.794 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:49.794 INFO:teuthology.orchestra.run.smithi023.stdout:Filesystem 'cephfs' (1) 2024-03-24T10:39:49.794 INFO:teuthology.orchestra.run.smithi023.stdout:fs_name cephfs 2024-03-24T10:39:49.794 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 22 2024-03-24T10:39:49.794 INFO:teuthology.orchestra.run.smithi023.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-24T10:39:49.794 INFO:teuthology.orchestra.run.smithi023.stdout:created 2024-03-24T10:38:10.206899+0000 2024-03-24T10:39:49.794 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2024-03-24T10:38:15.911086+0000 2024-03-24T10:39:49.794 INFO:teuthology.orchestra.run.smithi023.stdout:tableserver 0 2024-03-24T10:39:49.794 INFO:teuthology.orchestra.run.smithi023.stdout:root 0 2024-03-24T10:39:49.794 INFO:teuthology.orchestra.run.smithi023.stdout:session_timeout 300 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:session_autoclose 300 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:max_file_size 1099511627776 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:max_xattr_size 65536 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:required_client_features {} 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure 0 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure_osd_epoch 0 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:max_mds 3 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:in 0,1,2 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:up {0=24539,1=24505,2=24503} 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:failed 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:damaged 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:stopped 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:data_pools [3] 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:metadata_pool 2 2024-03-24T10:39:49.795 INFO:teuthology.orchestra.run.smithi023.stdout:inline_data disabled 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout:balancer 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout:bal_rank_mask -1 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout:standby_count_wanted 0 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout:qdb_cluster leader: 24539 members: 24503,24505,24539 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.b{0:24539} state up:active seq 3 addr [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.c{0:24487} state up:standby-replay seq 1 addr [v2:172.21.15.117:6836/977076441,v1:172.21.15.117:6837/977076441] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.i{1:24505} state up:active seq 6 addr [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.f{1:24475} state up:standby-replay seq 1 addr [v2:172.21.15.117:6834/1664408853,v1:172.21.15.117:6835/1664408853] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.h{2:24503} state up:active seq 7 addr [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.e{2:24470} state up:standby-replay seq 1 addr [v2:172.21.15.116:6834/1535852767,v1:172.21.15.116:6835/1535852767] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout:Standby daemons: 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.j{-1:14544} state up:standby seq 1 addr [v2:172.21.15.23:6834/3844921538,v1:172.21.15.23:6835/3844921538] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.d{-1:14562} state up:standby seq 1 addr [v2:172.21.15.23:6836/405041362,v1:172.21.15.23:6837/405041362] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T10:39:49.796 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.g{-1:14622} state up:standby seq 1 addr [v2:172.21.15.23:6838/4279444411,v1:172.21.15.23:6839/4279444411] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T10:39:49.797 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.a{-1:14640} state up:standby seq 1 addr [v2:172.21.15.23:6840/644733222,v1:172.21.15.23:6841/644733222] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T10:39:49.797 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.l{-1:24427} state up:standby seq 1 addr [v2:172.21.15.117:6832/134162564,v1:172.21.15.117:6833/134162564] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T10:39:49.797 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.k{-1:24440} state up:standby seq 1 addr [v2:172.21.15.116:6832/566816400,v1:172.21.15.116:6833/566816400] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T10:39:49.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.795+0000 7f8b61e30700 1 -- 172.21.15.23:0/883159819 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8b480a21d0 msgr2=0x7f8b480a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:49.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.795+0000 7f8b61e30700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8b480a21d0 0x7f8b480a4690 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c1021c0 tx=0x7f8b4c00b040 comp rx=0 tx=0).stop 2024-03-24T10:39:49.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.795+0000 7f8b61e30700 1 -- 172.21.15.23:0/883159819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5c102360 msgr2=0x7f8b5c10a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:49.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.795+0000 7f8b61e30700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5c102360 0x7f8b5c10a0c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8b4400e9a0 tx=0x7f8b4400ecb0 comp rx=0 tx=0).stop 2024-03-24T10:39:49.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.795+0000 7f8b61e30700 1 -- 172.21.15.23:0/883159819 shutdown_connections 2024-03-24T10:39:49.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.795+0000 7f8b61e30700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f8b480a21d0 0x7f8b480a4690 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:49.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.795+0000 7f8b61e30700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b5c101160 0x7f8b5c10ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:49.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.795+0000 7f8b61e30700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b5c102d00 0x7f8b5c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:49.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.795+0000 7f8b61e30700 1 --2- 172.21.15.23:0/883159819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5c102360 0x7f8b5c10a0c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:49.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.795+0000 7f8b61e30700 1 -- 172.21.15.23:0/883159819 >> 172.21.15.23:0/883159819 conn(0x7f8b5c0fc720 msgr2=0x7f8b5c105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:49.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.795+0000 7f8b61e30700 1 -- 172.21.15.23:0/883159819 shutdown_connections 2024-03-24T10:39:49.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:49.795+0000 7f8b61e30700 1 -- 172.21.15.23:0/883159819 wait complete. 2024-03-24T10:39:49.798 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 22 2024-03-24T10:39:50.085 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:49 smithi023 ceph-mon[32466]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-24T10:39:50.086 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:49 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/883159819' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T10:39:50.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:49 smithi117 ceph-mon[36368]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-24T10:39:50.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:49 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/883159819' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T10:39:50.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:49 smithi116 ceph-mon[36248]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-24T10:39:50.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:49 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/883159819' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T10:39:50.490 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-24T10:39:50.731 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:52.231 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:51 smithi023 ceph-mon[32466]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-24T10:39:52.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:51 smithi117 ceph-mon[36368]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-24T10:39:52.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:51 smithi116 ceph-mon[36248]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-24T10:39:52.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.338+0000 7f7760423700 1 -- 172.21.15.23:0/610847289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7758102330 msgr2=0x7f7758102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:52.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.338+0000 7f7760423700 1 --2- 172.21.15.23:0/610847289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7758102330 0x7f7758102790 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f77540099c0 tx=0x7f7754009cd0 comp rx=0 tx=0).stop 2024-03-24T10:39:52.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.338+0000 7f7760423700 1 -- 172.21.15.23:0/610847289 shutdown_connections 2024-03-24T10:39:52.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.338+0000 7f7760423700 1 --2- 172.21.15.23:0/610847289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7758101130 0x7f7758101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:52.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.338+0000 7f7760423700 1 --2- 172.21.15.23:0/610847289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7758102cd0 0x7f7758109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:52.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.338+0000 7f7760423700 1 --2- 172.21.15.23:0/610847289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7758102330 0x7f7758102790 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:52.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.338+0000 7f7760423700 1 -- 172.21.15.23:0/610847289 >> 172.21.15.23:0/610847289 conn(0x7f77580fc6f0 msgr2=0x7f77580feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:52.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.338+0000 7f7760423700 1 -- 172.21.15.23:0/610847289 shutdown_connections 2024-03-24T10:39:52.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.338+0000 7f7760423700 1 -- 172.21.15.23:0/610847289 wait complete. 2024-03-24T10:39:52.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.339+0000 7f7760423700 1 Processor -- start 2024-03-24T10:39:52.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.339+0000 7f7760423700 1 -- start start 2024-03-24T10:39:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.339+0000 7f7760423700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7758101130 0x7f7758197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.339+0000 7f7760423700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7758102330 0x7f7758197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.339+0000 7f7760423700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7758102cd0 0x7f775819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.339+0000 7f7760423700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f775819eca0 con 0x7f7758102330 2024-03-24T10:39:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.339+0000 7f7760423700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f775806a920 con 0x7f7758102cd0 2024-03-24T10:39:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.339+0000 7f7760423700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f775806aae0 con 0x7f7758101130 2024-03-24T10:39:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.339+0000 7f775d9be700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7758102330 0x7f7758197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.339+0000 7f775d9be700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7758102330 0x7f7758197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53202/0 (socket says 172.21.15.23:53202) 2024-03-24T10:39:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.339+0000 7f775d9be700 1 -- 172.21.15.23:0/2326101990 learned_addr learned my addr 172.21.15.23:0/2326101990 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.339+0000 7f775e1bf700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7758101130 0x7f7758197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.339+0000 7f775e9c0700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7758102cd0 0x7f775819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.340+0000 7f775d9be700 1 -- 172.21.15.23:0/2326101990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7758101130 msgr2=0x7f7758197110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.340+0000 7f775d9be700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7758101130 0x7f7758197110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.340+0000 7f775d9be700 1 -- 172.21.15.23:0/2326101990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7758102cd0 msgr2=0x7f775819e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.340+0000 7f775d9be700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7758102cd0 0x7f775819e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.340+0000 7f775d9be700 1 -- 172.21.15.23:0/2326101990 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77540095f0 con 0x7f7758102330 2024-03-24T10:39:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.340+0000 7f775e9c0700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7758102cd0 0x7f775819e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.340+0000 7f775e1bf700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7758101130 0x7f7758197110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.340+0000 7f775d9be700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7758102330 0x7f7758197650 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f77540036c0 tx=0x7f775400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.340+0000 7f774f7fe700 1 -- 172.21.15.23:0/2326101990 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775401d070 con 0x7f7758102330 2024-03-24T10:39:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.340+0000 7f774f7fe700 1 -- 172.21.15.23:0/2326101990 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7754022470 con 0x7f7758102330 2024-03-24T10:39:52.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.340+0000 7f7760423700 1 -- 172.21.15.23:0/2326101990 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f775806ad60 con 0x7f7758102330 2024-03-24T10:39:52.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.340+0000 7f774f7fe700 1 -- 172.21.15.23:0/2326101990 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7754017650 con 0x7f7758102330 2024-03-24T10:39:52.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.340+0000 7f7760423700 1 -- 172.21.15.23:0/2326101990 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f775806b200 con 0x7f7758102330 2024-03-24T10:39:52.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.341+0000 7f7760423700 1 -- 172.21.15.23:0/2326101990 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77581084f0 con 0x7f7758102330 2024-03-24T10:39:52.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.342+0000 7f774f7fe700 1 -- 172.21.15.23:0/2326101990 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f77540225e0 con 0x7f7758102330 2024-03-24T10:39:52.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.343+0000 7f774f7fe700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f77440a2150 0x7f77440a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:52.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.343+0000 7f775e1bf700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f77440a2150 0x7f77440a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:52.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.343+0000 7f774f7fe700 1 -- 172.21.15.23:0/2326101990 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f77540cec70 con 0x7f7758102330 2024-03-24T10:39:52.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.343+0000 7f775e1bf700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f77440a2150 0x7f77440a4610 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7758102190 tx=0x7f7748009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:52.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.347+0000 7f774f7fe700 1 -- 172.21.15.23:0/2326101990 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7754097cb0 con 0x7f7758102330 2024-03-24T10:39:52.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.552+0000 7f7760423700 1 -- 172.21.15.23:0/2326101990 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f7758068ec0 con 0x7f7758102330 2024-03-24T10:39:52.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.553+0000 7f774f7fe700 1 -- 172.21.15.23:0/2326101990 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) ==== 56+0+5341 (secure 0 0 0) 0x7f775409b370 con 0x7f7758102330 2024-03-24T10:39:52.553 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 87 2024-03-24T10:39:52.554 INFO:teuthology.orchestra.run.smithi023.stdout:fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:39:52.554 INFO:teuthology.orchestra.run.smithi023.stdout:created 2024-03-24T10:30:20.456703+0000 2024-03-24T10:39:52.554 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2024-03-24T10:39:16.692832+0000 2024-03-24T10:39:52.554 INFO:teuthology.orchestra.run.smithi023.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-24T10:39:52.554 INFO:teuthology.orchestra.run.smithi023.stdout:crush_version 26 2024-03-24T10:39:52.554 INFO:teuthology.orchestra.run.smithi023.stdout:full_ratio 0.95 2024-03-24T10:39:52.554 INFO:teuthology.orchestra.run.smithi023.stdout:backfillfull_ratio 0.9 2024-03-24T10:39:52.554 INFO:teuthology.orchestra.run.smithi023.stdout:nearfull_ratio 0.85 2024-03-24T10:39:52.554 INFO:teuthology.orchestra.run.smithi023.stdout:require_min_compat_client luminous 2024-03-24T10:39:52.554 INFO:teuthology.orchestra.run.smithi023.stdout:min_compat_client luminous 2024-03-24T10:39:52.554 INFO:teuthology.orchestra.run.smithi023.stdout:require_osd_release squid 2024-03-24T10:39:52.554 INFO:teuthology.orchestra.run.smithi023.stdout:stretch_mode_enabled false 2024-03-24T10:39:52.555 INFO:teuthology.orchestra.run.smithi023.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-24T10:39:52.555 INFO:teuthology.orchestra.run.smithi023.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-03-24T10:39:52.555 INFO:teuthology.orchestra.run.smithi023.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-24T10:39:52.555 INFO:teuthology.orchestra.run.smithi023.stdout:max_osd 12 2024-03-24T10:39:52.555 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 up in weight 1 up_from 8 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6802/65775038,v1:172.21.15.23:6803/65775038] [v2:172.21.15.23:6804/65775038,v1:172.21.15.23:6805/65775038] exists,up e6c1781a-5af4-4191-b299-02371336d02a 2024-03-24T10:39:52.555 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6800/1598773126,v1:172.21.15.116:6801/1598773126] [v2:172.21.15.116:6802/1598773126,v1:172.21.15.116:6803/1598773126] exists,up 8fd17c0c-e834-46bf-bb82-b247e4d5bb01 2024-03-24T10:39:52.555 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 up in weight 1 up_from 18 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6800/3979033351,v1:172.21.15.117:6801/3979033351] [v2:172.21.15.117:6802/3979033351,v1:172.21.15.117:6803/3979033351] exists,up 8ec590e7-eda9-4938-94c9-b1f61f8dfc6f 2024-03-24T10:39:52.555 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3 up in weight 1 up_from 25 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6810/1780984549,v1:172.21.15.23:6811/1780984549] [v2:172.21.15.23:6812/1780984549,v1:172.21.15.23:6813/1780984549] exists,up 61dcad05-5af6-4faf-860c-54f5e3566b84 2024-03-24T10:39:52.555 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4 up in weight 1 up_from 33 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6808/3074814953,v1:172.21.15.116:6809/3074814953] [v2:172.21.15.116:6810/3074814953,v1:172.21.15.116:6811/3074814953] exists,up 7d56de8f-6140-477e-b851-42928411277d 2024-03-24T10:39:52.555 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5 up in weight 1 up_from 38 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6808/1549866608,v1:172.21.15.117:6809/1549866608] [v2:172.21.15.117:6810/1549866608,v1:172.21.15.117:6811/1549866608] exists,up 9f52e187-a40f-4360-8e5e-b665e241657e 2024-03-24T10:39:52.555 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6 up in weight 1 up_from 46 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6818/1181430315,v1:172.21.15.23:6819/1181430315] [v2:172.21.15.23:6820/1181430315,v1:172.21.15.23:6821/1181430315] exists,up 32e3081d-af77-4879-ab4c-e77a0303b741 2024-03-24T10:39:52.555 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7 up in weight 1 up_from 51 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6816/2721290459,v1:172.21.15.116:6817/2721290459] [v2:172.21.15.116:6818/2721290459,v1:172.21.15.116:6819/2721290459] exists,up dddddbc7-f15a-4844-8d5a-869421e3dead 2024-03-24T10:39:52.555 INFO:teuthology.orchestra.run.smithi023.stdout:osd.8 up in weight 1 up_from 59 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6816/3633840563,v1:172.21.15.117:6817/3633840563] [v2:172.21.15.117:6818/3633840563,v1:172.21.15.117:6819/3633840563] exists,up e84fd4bc-7173-410c-84f2-5c9dcf91b790 2024-03-24T10:39:52.556 INFO:teuthology.orchestra.run.smithi023.stdout:osd.9 up in weight 1 up_from 67 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6826/3239156530,v1:172.21.15.23:6827/3239156530] [v2:172.21.15.23:6828/3239156530,v1:172.21.15.23:6829/3239156530] exists,up 7a1d9119-fd0f-45b4-a0f7-051b118e1c09 2024-03-24T10:39:52.556 INFO:teuthology.orchestra.run.smithi023.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6824/2413381037,v1:172.21.15.116:6825/2413381037] [v2:172.21.15.116:6826/2413381037,v1:172.21.15.116:6827/2413381037] exists,up cef53e0f-9f79-44c7-b8ea-6849ade8bb39 2024-03-24T10:39:52.556 INFO:teuthology.orchestra.run.smithi023.stdout:osd.11 up in weight 1 up_from 80 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6824/3654755153,v1:172.21.15.117:6825/3654755153] [v2:172.21.15.117:6826/3654755153,v1:172.21.15.117:6827/3654755153] exists,up d763fbde-fd35-4847-b8f1-8f22553ee1c3 2024-03-24T10:39:52.556 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.8 [2,11] 2024-03-24T10:39:52.556 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.a [2,11] 2024-03-24T10:39:52.556 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.10 [2,4] 2024-03-24T10:39:52.556 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.1f [8,1] 2024-03-24T10:39:52.556 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.c [3,9] 2024-03-24T10:39:52.556 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.17 [4,6] 2024-03-24T10:39:52.556 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.1d [5,9] 2024-03-24T10:39:52.556 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.21 [4,7] 2024-03-24T10:39:52.556 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.23 [4,8] 2024-03-24T10:39:52.556 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.33 [5,9] 2024-03-24T10:39:52.556 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/3980246761 expires 2024-03-25T10:31:14.712690+0000 2024-03-24T10:39:52.557 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/1376242368 expires 2024-03-25T10:31:14.712690+0000 2024-03-24T10:39:52.557 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/3663301421 expires 2024-03-25T10:31:14.712690+0000 2024-03-24T10:39:52.557 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/1774258590 expires 2024-03-25T10:30:48.096659+0000 2024-03-24T10:39:52.557 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/1794809128 expires 2024-03-25T10:30:48.096659+0000 2024-03-24T10:39:52.557 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/2159398914 expires 2024-03-25T10:31:14.712690+0000 2024-03-24T10:39:52.557 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/1774258590 expires 2024-03-25T10:30:48.096659+0000 2024-03-24T10:39:52.557 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/3434856686 expires 2024-03-25T10:30:48.096659+0000 2024-03-24T10:39:52.557 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/3666340600 expires 2024-03-25T10:30:48.096659+0000 2024-03-24T10:39:52.557 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/2159398914 expires 2024-03-25T10:31:14.712690+0000 2024-03-24T10:39:52.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.556+0000 7f7760423700 1 -- 172.21.15.23:0/2326101990 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f77440a2150 msgr2=0x7f77440a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:52.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.556+0000 7f7760423700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f77440a2150 0x7f77440a4610 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7758102190 tx=0x7f7748009260 comp rx=0 tx=0).stop 2024-03-24T10:39:52.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.556+0000 7f7760423700 1 -- 172.21.15.23:0/2326101990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7758102330 msgr2=0x7f7758197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:52.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.556+0000 7f7760423700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7758102330 0x7f7758197650 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f77540036c0 tx=0x7f775400f690 comp rx=0 tx=0).stop 2024-03-24T10:39:52.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.556+0000 7f7760423700 1 -- 172.21.15.23:0/2326101990 shutdown_connections 2024-03-24T10:39:52.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.556+0000 7f7760423700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f77440a2150 0x7f77440a4610 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:52.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.556+0000 7f7760423700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7758101130 0x7f7758197110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:52.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.556+0000 7f7760423700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7758102cd0 0x7f775819e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:52.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.556+0000 7f7760423700 1 --2- 172.21.15.23:0/2326101990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7758102330 0x7f7758197650 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:52.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.556+0000 7f7760423700 1 -- 172.21.15.23:0/2326101990 >> 172.21.15.23:0/2326101990 conn(0x7f77580fc6f0 msgr2=0x7f7758105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:52.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.556+0000 7f7760423700 1 -- 172.21.15.23:0/2326101990 shutdown_connections 2024-03-24T10:39:52.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:52.556+0000 7f7760423700 1 -- 172.21.15.23:0/2326101990 wait complete. 2024-03-24T10:39:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:52 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2326101990' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T10:39:53.199 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-24T10:39:53.209 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:39:53.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:52 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2326101990' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T10:39:53.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:52 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2326101990' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T10:39:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.318+0000 7fea78fe9640 1 -- 172.21.15.23:0/3658901288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea741018a0 msgr2=0x7fea74101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.318+0000 7fea78fe9640 1 --2- 172.21.15.23:0/3658901288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea741018a0 0x7fea74101d20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fea68009900 tx=0x7fea6802bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.318+0000 7fea78fe9640 1 -- 172.21.15.23:0/3658901288 shutdown_connections 2024-03-24T10:39:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.318+0000 7fea78fe9640 1 --2- 172.21.15.23:0/3658901288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea74102260 0x7fea74108af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.318+0000 7fea78fe9640 1 --2- 172.21.15.23:0/3658901288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea741018a0 0x7fea74101d20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.318+0000 7fea78fe9640 1 --2- 172.21.15.23:0/3658901288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea741006b0 0x7fea74100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.318+0000 7fea78fe9640 1 -- 172.21.15.23:0/3658901288 >> 172.21.15.23:0/3658901288 conn(0x7fea740fbc40 msgr2=0x7fea740fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.318+0000 7fea78fe9640 1 -- 172.21.15.23:0/3658901288 shutdown_connections 2024-03-24T10:39:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.318+0000 7fea78fe9640 1 -- 172.21.15.23:0/3658901288 wait complete. 2024-03-24T10:39:53.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.319+0000 7fea78fe9640 1 Processor -- start 2024-03-24T10:39:53.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.319+0000 7fea78fe9640 1 -- start start 2024-03-24T10:39:53.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.319+0000 7fea78fe9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea741006b0 0x7fea741977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:53.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.319+0000 7fea78fe9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea741018a0 0x7fea74197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.319+0000 7fea78fe9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea74102260 0x7fea7419ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.319+0000 7fea78fe9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fea7419f2a0 con 0x7fea74102260 2024-03-24T10:39:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.319+0000 7fea78fe9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fea7419f410 con 0x7fea741018a0 2024-03-24T10:39:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.319+0000 7fea78fe9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fea7419f5d0 con 0x7fea741006b0 2024-03-24T10:39:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.319+0000 7fea72d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea74102260 0x7fea7419ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.319+0000 7fea72d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea74102260 0x7fea7419ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53212/0 (socket says 172.21.15.23:53212) 2024-03-24T10:39:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.319+0000 7fea7259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea741006b0 0x7fea741977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.319+0000 7fea72d9d640 1 -- 172.21.15.23:0/2716528667 learned_addr learned my addr 172.21.15.23:0/2716528667 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.319+0000 7fea71d9b640 1 --2- 172.21.15.23:0/2716528667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea741018a0 0x7fea74197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.320+0000 7fea72d9d640 1 -- 172.21.15.23:0/2716528667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea741006b0 msgr2=0x7fea741977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.320+0000 7fea72d9d640 1 --2- 172.21.15.23:0/2716528667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea741006b0 0x7fea741977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.320+0000 7fea72d9d640 1 -- 172.21.15.23:0/2716528667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea741018a0 msgr2=0x7fea74197cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.320+0000 7fea72d9d640 1 --2- 172.21.15.23:0/2716528667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea741018a0 0x7fea74197cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.320+0000 7fea72d9d640 1 -- 172.21.15.23:0/2716528667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea68009560 con 0x7fea74102260 2024-03-24T10:39:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.320+0000 7fea7259c640 1 --2- 172.21.15.23:0/2716528667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea741006b0 0x7fea741977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.320+0000 7fea72d9d640 1 --2- 172.21.15.23:0/2716528667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea74102260 0x7fea7419ed60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fea6400eb10 tx=0x7fea6400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.320+0000 7fea637fe640 1 -- 172.21.15.23:0/2716528667 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea64004b20 con 0x7fea74102260 2024-03-24T10:39:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.320+0000 7fea78fe9640 1 -- 172.21.15.23:0/2716528667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea7419f8b0 con 0x7fea74102260 2024-03-24T10:39:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.320+0000 7fea637fe640 1 -- 172.21.15.23:0/2716528667 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea64004c80 con 0x7fea74102260 2024-03-24T10:39:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.320+0000 7fea637fe640 1 -- 172.21.15.23:0/2716528667 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea64010670 con 0x7fea74102260 2024-03-24T10:39:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.320+0000 7fea78fe9640 1 -- 172.21.15.23:0/2716528667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea7419fde0 con 0x7fea74102260 2024-03-24T10:39:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.321+0000 7fea78fe9640 1 -- 172.21.15.23:0/2716528667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea74107a20 con 0x7fea74102260 2024-03-24T10:39:53.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.322+0000 7fea637fe640 1 -- 172.21.15.23:0/2716528667 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fea64005020 con 0x7fea74102260 2024-03-24T10:39:53.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.322+0000 7fea637fe640 1 --2- 172.21.15.23:0/2716528667 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fea5c0a20c0 0x7fea5c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:53.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.323+0000 7fea7259c640 1 --2- 172.21.15.23:0/2716528667 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fea5c0a20c0 0x7fea5c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:53.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.323+0000 7fea637fe640 1 -- 172.21.15.23:0/2716528667 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fea64014070 con 0x7fea74102260 2024-03-24T10:39:53.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.323+0000 7fea7259c640 1 --2- 172.21.15.23:0/2716528667 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fea5c0a20c0 0x7fea5c0a4580 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fea74101700 tx=0x7fea6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:53.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.327+0000 7fea637fe640 1 -- 172.21.15.23:0/2716528667 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fea640971b0 con 0x7fea74102260 2024-03-24T10:39:53.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.511+0000 7fea78fe9640 1 -- 172.21.15.23:0/2716528667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea7402d1d0 con 0x7fea74102260 2024-03-24T10:39:53.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.513+0000 7fea637fe640 1 -- 172.21.15.23:0/2716528667 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22704 (secure 0 0 0) 0x7fea6409a870 con 0x7fea74102260 2024-03-24T10:39:53.513 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:53.514 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:39:16.692832+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:39:53.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.515+0000 7fea78fe9640 1 -- 172.21.15.23:0/2716528667 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fea5c0a20c0 msgr2=0x7fea5c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:53.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.515+0000 7fea78fe9640 1 --2- 172.21.15.23:0/2716528667 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fea5c0a20c0 0x7fea5c0a4580 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fea74101700 tx=0x7fea6c005f90 comp rx=0 tx=0).stop 2024-03-24T10:39:53.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.515+0000 7fea78fe9640 1 -- 172.21.15.23:0/2716528667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea74102260 msgr2=0x7fea7419ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:53.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.515+0000 7fea78fe9640 1 --2- 172.21.15.23:0/2716528667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea74102260 0x7fea7419ed60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fea6400eb10 tx=0x7fea6400c2d0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.515+0000 7fea78fe9640 1 -- 172.21.15.23:0/2716528667 shutdown_connections 2024-03-24T10:39:53.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.515+0000 7fea78fe9640 1 --2- 172.21.15.23:0/2716528667 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fea5c0a20c0 0x7fea5c0a4580 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.515+0000 7fea78fe9640 1 --2- 172.21.15.23:0/2716528667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea74102260 0x7fea7419ed60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.515+0000 7fea78fe9640 1 --2- 172.21.15.23:0/2716528667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea741018a0 0x7fea74197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.515+0000 7fea78fe9640 1 --2- 172.21.15.23:0/2716528667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea741006b0 0x7fea741977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.515+0000 7fea78fe9640 1 -- 172.21.15.23:0/2716528667 >> 172.21.15.23:0/2716528667 conn(0x7fea740fbc40 msgr2=0x7fea74104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:53.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.515+0000 7fea78fe9640 1 -- 172.21.15.23:0/2716528667 shutdown_connections 2024-03-24T10:39:53.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.515+0000 7fea78fe9640 1 -- 172.21.15.23:0/2716528667 wait complete. 2024-03-24T10:39:53.527 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:39.056452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:38:08.473472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:38:09.414667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:39:53.527 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:39:53.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.632+0000 7ff47d82a640 1 -- 172.21.15.23:0/213618087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4781008f0 msgr2=0x7ff478100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:53.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.632+0000 7ff47d82a640 1 --2- 172.21.15.23:0/213618087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4781008f0 0x7ff478100d70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff468009930 tx=0x7ff46802bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:53.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.632+0000 7ff47d82a640 1 -- 172.21.15.23:0/213618087 shutdown_connections 2024-03-24T10:39:53.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.632+0000 7ff47d82a640 1 --2- 172.21.15.23:0/213618087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4781012b0 0x7ff478108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.632+0000 7ff47d82a640 1 --2- 172.21.15.23:0/213618087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4781008f0 0x7ff478100d70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.632+0000 7ff47d82a640 1 --2- 172.21.15.23:0/213618087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff478101bc0 0x7ff478101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.632+0000 7ff47d82a640 1 -- 172.21.15.23:0/213618087 >> 172.21.15.23:0/213618087 conn(0x7ff4780fb320 msgr2=0x7ff4780fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.632+0000 7ff47d82a640 1 -- 172.21.15.23:0/213618087 shutdown_connections 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.632+0000 7ff47d82a640 1 -- 172.21.15.23:0/213618087 wait complete. 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.632+0000 7ff47d82a640 1 Processor -- start 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff47d82a640 1 -- start start 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff47d82a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4781008f0 0x7ff478197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff47d82a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4781012b0 0x7ff478197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff47d82a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff478101bc0 0x7ff47819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff47d82a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff47819f2f0 con 0x7ff4781008f0 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff47d82a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff47819f460 con 0x7ff4781012b0 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff47d82a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff47819f620 con 0x7ff478101bc0 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff476ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4781008f0 0x7ff478197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff476ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4781008f0 0x7ff478197850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53236/0 (socket says 172.21.15.23:53236) 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff476ffd640 1 -- 172.21.15.23:0/3111578290 learned_addr learned my addr 172.21.15.23:0/3111578290 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:53.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff4767fc640 1 --2- 172.21.15.23:0/3111578290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4781012b0 0x7ff478197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:53.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff4777fe640 1 --2- 172.21.15.23:0/3111578290 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff478101bc0 0x7ff47819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:53.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff476ffd640 1 -- 172.21.15.23:0/3111578290 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff478101bc0 msgr2=0x7ff47819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:53.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff476ffd640 1 --2- 172.21.15.23:0/3111578290 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff478101bc0 0x7ff47819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff476ffd640 1 -- 172.21.15.23:0/3111578290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4781012b0 msgr2=0x7ff478197d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:53.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff476ffd640 1 --2- 172.21.15.23:0/3111578290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4781012b0 0x7ff478197d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff476ffd640 1 -- 172.21.15.23:0/3111578290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff470009840 con 0x7ff4781008f0 2024-03-24T10:39:53.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff476ffd640 1 --2- 172.21.15.23:0/3111578290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4781008f0 0x7ff478197850 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00e9c0 tx=0x7ff46c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:53.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff47c828640 1 -- 172.21.15.23:0/3111578290 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c009700 con 0x7ff4781008f0 2024-03-24T10:39:53.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff47d82a640 1 -- 172.21.15.23:0/3111578290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff468009560 con 0x7ff4781008f0 2024-03-24T10:39:53.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff47c828640 1 -- 172.21.15.23:0/3111578290 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff46c00ccc0 con 0x7ff4781008f0 2024-03-24T10:39:53.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff47c828640 1 -- 172.21.15.23:0/3111578290 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c010430 con 0x7ff4781008f0 2024-03-24T10:39:53.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.633+0000 7ff47d82a640 1 -- 172.21.15.23:0/3111578290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff47819fca0 con 0x7ff4781008f0 2024-03-24T10:39:53.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.634+0000 7ff47d82a640 1 -- 172.21.15.23:0/3111578290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff478107b00 con 0x7ff4781008f0 2024-03-24T10:39:53.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.635+0000 7ff47c828640 1 -- 172.21.15.23:0/3111578290 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7ff46c003f60 con 0x7ff4781008f0 2024-03-24T10:39:53.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.636+0000 7ff47c828640 1 --2- 172.21.15.23:0/3111578290 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff4600a20c0 0x7ff4600a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:53.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.636+0000 7ff4767fc640 1 --2- 172.21.15.23:0/3111578290 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff4600a20c0 0x7ff4600a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:53.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.636+0000 7ff47c828640 1 -- 172.21.15.23:0/3111578290 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7ff46c014070 con 0x7ff4781008f0 2024-03-24T10:39:53.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.637+0000 7ff4767fc640 1 --2- 172.21.15.23:0/3111578290 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff4600a20c0 0x7ff4600a4580 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff468009900 tx=0x7ff468005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:53.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.639+0000 7ff47c828640 1 -- 172.21.15.23:0/3111578290 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff46c097230 con 0x7ff4781008f0 2024-03-24T10:39:53.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.823+0000 7ff47d82a640 1 -- 172.21.15.23:0/3111578290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff47810d180 con 0x7ff4781008f0 2024-03-24T10:39:53.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.823+0000 7ff47c828640 1 -- 172.21.15.23:0/3111578290 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7ff46c09a8f0 con 0x7ff4781008f0 2024-03-24T10:39:53.824 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-24T10:39:53.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.826+0000 7ff47d82a640 1 -- 172.21.15.23:0/3111578290 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff4600a20c0 msgr2=0x7ff4600a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:53.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.826+0000 7ff47d82a640 1 --2- 172.21.15.23:0/3111578290 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff4600a20c0 0x7ff4600a4580 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff468009900 tx=0x7ff468005fd0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.826+0000 7ff47d82a640 1 -- 172.21.15.23:0/3111578290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4781008f0 msgr2=0x7ff478197850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:53.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.826+0000 7ff47d82a640 1 --2- 172.21.15.23:0/3111578290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4781008f0 0x7ff478197850 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00e9c0 tx=0x7ff46c00edc0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.826+0000 7ff47d82a640 1 -- 172.21.15.23:0/3111578290 shutdown_connections 2024-03-24T10:39:53.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.826+0000 7ff47d82a640 1 --2- 172.21.15.23:0/3111578290 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7ff4600a20c0 0x7ff4600a4580 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.826+0000 7ff47d82a640 1 --2- 172.21.15.23:0/3111578290 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff478101bc0 0x7ff47819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.826+0000 7ff47d82a640 1 --2- 172.21.15.23:0/3111578290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4781012b0 0x7ff478197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.826+0000 7ff47d82a640 1 --2- 172.21.15.23:0/3111578290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4781008f0 0x7ff478197850 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.826+0000 7ff47d82a640 1 -- 172.21.15.23:0/3111578290 >> 172.21.15.23:0/3111578290 conn(0x7ff4780fb320 msgr2=0x7ff478104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:53.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.826+0000 7ff47d82a640 1 -- 172.21.15.23:0/3111578290 shutdown_connections 2024-03-24T10:39:53.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.826+0000 7ff47d82a640 1 -- 172.21.15.23:0/3111578290 wait complete. 2024-03-24T10:39:53.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:39:53.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.943+0000 7f0aedc39640 1 -- 172.21.15.23:0/1944911153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8104ac0 msgr2=0x7f0ae8106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:53.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.943+0000 7f0aedc39640 1 --2- 172.21.15.23:0/1944911153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8104ac0 0x7f0ae8106eb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009900 tx=0x7f0adc02bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:53.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.943+0000 7f0aedc39640 1 -- 172.21.15.23:0/1944911153 shutdown_connections 2024-03-24T10:39:53.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.943+0000 7f0aedc39640 1 --2- 172.21.15.23:0/1944911153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae81073f0 0x7f0ae8109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.943+0000 7f0aedc39640 1 --2- 172.21.15.23:0/1944911153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8104ac0 0x7f0ae8106eb0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.943+0000 7f0aedc39640 1 --2- 172.21.15.23:0/1944911153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae8102190 0x7f0ae8104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.943+0000 7f0aedc39640 1 -- 172.21.15.23:0/1944911153 >> 172.21.15.23:0/1944911153 conn(0x7f0ae80fbba0 msgr2=0x7f0ae80fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:53.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.943+0000 7f0aedc39640 1 -- 172.21.15.23:0/1944911153 shutdown_connections 2024-03-24T10:39:53.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.943+0000 7f0aedc39640 1 -- 172.21.15.23:0/1944911153 wait complete. 2024-03-24T10:39:53.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.944+0000 7f0aedc39640 1 Processor -- start 2024-03-24T10:39:53.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.944+0000 7f0aedc39640 1 -- start start 2024-03-24T10:39:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.944+0000 7f0aedc39640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8102190 0x7f0ae8197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.944+0000 7f0aedc39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae8104ac0 0x7f0ae8197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.944+0000 7f0aedc39640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae81073f0 0x7f0ae819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.944+0000 7f0aedc39640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae819f310 con 0x7f0ae8102190 2024-03-24T10:39:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.944+0000 7f0aedc39640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae819f480 con 0x7f0ae8104ac0 2024-03-24T10:39:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.944+0000 7f0aedc39640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae819f640 con 0x7f0ae81073f0 2024-03-24T10:39:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.944+0000 7f0ae77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8102190 0x7f0ae8197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.944+0000 7f0ae77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8102190 0x7f0ae8197820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53260/0 (socket says 172.21.15.23:53260) 2024-03-24T10:39:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.944+0000 7f0ae77fe640 1 -- 172.21.15.23:0/2778226536 learned_addr learned my addr 172.21.15.23:0/2778226536 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.944+0000 7f0ae6ffd640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae8104ac0 0x7f0ae8197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.944+0000 7f0ae7fff640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae81073f0 0x7f0ae819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.945+0000 7f0ae77fe640 1 -- 172.21.15.23:0/2778226536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae81073f0 msgr2=0x7f0ae819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.945+0000 7f0ae77fe640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae81073f0 0x7f0ae819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.945+0000 7f0ae77fe640 1 -- 172.21.15.23:0/2778226536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae8104ac0 msgr2=0x7f0ae8197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.945+0000 7f0ae77fe640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae8104ac0 0x7f0ae8197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.945+0000 7f0ae77fe640 1 -- 172.21.15.23:0/2778226536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae0009840 con 0x7f0ae8102190 2024-03-24T10:39:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.945+0000 7f0ae6ffd640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae8104ac0 0x7f0ae8197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.945+0000 7f0ae77fe640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8102190 0x7f0ae8197820 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0ad800e9f0 tx=0x7f0ad800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.945+0000 7f0ae4ff9640 1 -- 172.21.15.23:0/2778226536 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8009700 con 0x7f0ae8102190 2024-03-24T10:39:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.945+0000 7f0ae4ff9640 1 -- 172.21.15.23:0/2778226536 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ad800ccc0 con 0x7f0ae8102190 2024-03-24T10:39:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.945+0000 7f0ae4ff9640 1 -- 172.21.15.23:0/2778226536 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8018820 con 0x7f0ae8102190 2024-03-24T10:39:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.945+0000 7f0aedc39640 1 -- 172.21.15.23:0/2778226536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0adc009560 con 0x7f0ae8102190 2024-03-24T10:39:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.945+0000 7f0aedc39640 1 -- 172.21.15.23:0/2778226536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae819fc70 con 0x7f0ae8102190 2024-03-24T10:39:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.946+0000 7f0aedc39640 1 -- 172.21.15.23:0/2778226536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae80fd690 con 0x7f0ae8102190 2024-03-24T10:39:53.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.947+0000 7f0ae4ff9640 1 -- 172.21.15.23:0/2778226536 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f0ad8010430 con 0x7f0ae8102190 2024-03-24T10:39:53.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.947+0000 7f0ae4ff9640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ad00a20c0 0x7f0ad00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:53.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.948+0000 7f0ae6ffd640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ad00a20c0 0x7f0ad00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:53.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.948+0000 7f0ae4ff9640 1 -- 172.21.15.23:0/2778226536 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0ad8014070 con 0x7f0ae8102190 2024-03-24T10:39:53.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.948+0000 7f0ae6ffd640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ad00a20c0 0x7f0ad00a4580 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0adc02c240 tx=0x7f0adc02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:53.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:53.951+0000 7f0ae4ff9640 1 -- 172.21.15.23:0/2778226536 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0ad8097530 con 0x7f0ae8102190 2024-03-24T10:39:54.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.131+0000 7f0aedc39640 1 -- 172.21.15.23:0/2778226536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0ae802d1d0 con 0x7f0ae8102190 2024-03-24T10:39:54.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.132+0000 7f0ae4ff9640 1 -- 172.21.15.23:0/2778226536 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f0ad809abf0 con 0x7f0ae8102190 2024-03-24T10:39:54.132 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-24T10:39:54.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.134+0000 7f0aedc39640 1 -- 172.21.15.23:0/2778226536 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ad00a20c0 msgr2=0x7f0ad00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.134+0000 7f0aedc39640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ad00a20c0 0x7f0ad00a4580 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0adc02c240 tx=0x7f0adc02c7e0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.134+0000 7f0aedc39640 1 -- 172.21.15.23:0/2778226536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8102190 msgr2=0x7f0ae8197820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.134+0000 7f0aedc39640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8102190 0x7f0ae8197820 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0ad800e9f0 tx=0x7f0ad800edf0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.134+0000 7f0aedc39640 1 -- 172.21.15.23:0/2778226536 shutdown_connections 2024-03-24T10:39:54.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.134+0000 7f0aedc39640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0ad00a20c0 0x7f0ad00a4580 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.134+0000 7f0aedc39640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae81073f0 0x7f0ae819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.134+0000 7f0aedc39640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae8104ac0 0x7f0ae8197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.134+0000 7f0aedc39640 1 --2- 172.21.15.23:0/2778226536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8102190 0x7f0ae8197820 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.134+0000 7f0aedc39640 1 -- 172.21.15.23:0/2778226536 >> 172.21.15.23:0/2778226536 conn(0x7f0ae80fbba0 msgr2=0x7f0ae8102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:54.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.134+0000 7f0aedc39640 1 -- 172.21.15.23:0/2778226536 shutdown_connections 2024-03-24T10:39:54.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.134+0000 7f0aedc39640 1 -- 172.21.15.23:0/2778226536 wait complete. 2024-03-24T10:39:54.144 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:53 smithi023 ceph-mon[32466]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-24T10:39:54.144 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:53 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2716528667' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:54.144 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:53 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3111578290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:39:54.145 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:39:54.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.250+0000 7f7725bf4640 1 -- 172.21.15.23:0/1050299504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7720101800 msgr2=0x7f7720101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.250+0000 7f7725bf4640 1 --2- 172.21.15.23:0/1050299504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7720101800 0x7f7720101c80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7708009900 tx=0x7f770802bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:54.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.250+0000 7f7725bf4640 1 -- 172.21.15.23:0/1050299504 shutdown_connections 2024-03-24T10:39:54.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.250+0000 7f7725bf4640 1 --2- 172.21.15.23:0/1050299504 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77201021c0 0x7f7720108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.250+0000 7f7725bf4640 1 --2- 172.21.15.23:0/1050299504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7720101800 0x7f7720101c80 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.250+0000 7f7725bf4640 1 --2- 172.21.15.23:0/1050299504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7720100610 0x7f7720100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.250+0000 7f7725bf4640 1 -- 172.21.15.23:0/1050299504 >> 172.21.15.23:0/1050299504 conn(0x7f77200fbba0 msgr2=0x7f77200fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:54.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.250+0000 7f7725bf4640 1 -- 172.21.15.23:0/1050299504 shutdown_connections 2024-03-24T10:39:54.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.250+0000 7f7725bf4640 1 -- 172.21.15.23:0/1050299504 wait complete. 2024-03-24T10:39:54.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.250+0000 7f7725bf4640 1 Processor -- start 2024-03-24T10:39:54.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f7725bf4640 1 -- start start 2024-03-24T10:39:54.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f7725bf4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7720100610 0x7f772006a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f7725bf4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7720101800 0x7f772019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f7725bf4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77201021c0 0x7f772019eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f7725bf4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f772019f2c0 con 0x7f7720100610 2024-03-24T10:39:54.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f7725bf4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f772019f430 con 0x7f7720101800 2024-03-24T10:39:54.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f7725bf4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f772019f5f0 con 0x7f77201021c0 2024-03-24T10:39:54.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7720100610 0x7f772006a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7720100610 0x7f772006a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53272/0 (socket says 172.21.15.23:53272) 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771f7fe640 1 -- 172.21.15.23:0/2932599963 learned_addr learned my addr 172.21.15.23:0/2932599963 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771effd640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7720101800 0x7f772019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771ffff640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77201021c0 0x7f772019eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771f7fe640 1 -- 172.21.15.23:0/2932599963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77201021c0 msgr2=0x7f772019eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771f7fe640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77201021c0 0x7f772019eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771f7fe640 1 -- 172.21.15.23:0/2932599963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7720101800 msgr2=0x7f772019c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771f7fe640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7720101800 0x7f772019c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771f7fe640 1 -- 172.21.15.23:0/2932599963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7708009560 con 0x7f7720100610 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771ffff640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77201021c0 0x7f772019eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771f7fe640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7720100610 0x7f772006a8b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f771000e9c0 tx=0x7f771000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771cff9640 1 -- 172.21.15.23:0/2932599963 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7710009700 con 0x7f7720100610 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771cff9640 1 -- 172.21.15.23:0/2932599963 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f771000cd80 con 0x7f7720100610 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f7725bf4640 1 -- 172.21.15.23:0/2932599963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f772019f870 con 0x7f7720100610 2024-03-24T10:39:54.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f771cff9640 1 -- 172.21.15.23:0/2932599963 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7710010430 con 0x7f7720100610 2024-03-24T10:39:54.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.251+0000 7f7725bf4640 1 -- 172.21.15.23:0/2932599963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f772019fd50 con 0x7f7720100610 2024-03-24T10:39:54.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.252+0000 7f7725bf4640 1 -- 172.21.15.23:0/2932599963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7720107930 con 0x7f7720100610 2024-03-24T10:39:54.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:53 smithi117 ceph-mon[36368]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-24T10:39:54.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:53 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2716528667' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:54.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:53 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3111578290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:39:54.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.253+0000 7f771cff9640 1 -- 172.21.15.23:0/2932599963 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f7710003680 con 0x7f7720100610 2024-03-24T10:39:54.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.254+0000 7f771cff9640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f770c0a20c0 0x7f770c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.254+0000 7f771effd640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f770c0a20c0 0x7f770c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.254+0000 7f771cff9640 1 -- 172.21.15.23:0/2932599963 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7710014070 con 0x7f7720100610 2024-03-24T10:39:54.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.254+0000 7f771effd640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f770c0a20c0 0x7f770c0a4580 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7708009230 tx=0x7f7708005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:54.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.258+0000 7f771cff9640 1 -- 172.21.15.23:0/2932599963 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f77100970d0 con 0x7f7720100610 2024-03-24T10:39:54.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:53 smithi116 ceph-mon[36248]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-24T10:39:54.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:53 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2716528667' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:54.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:53 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3111578290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:39:54.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.442+0000 7f7725bf4640 1 -- 172.21.15.23:0/2932599963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f772002d200 con 0x7f7720100610 2024-03-24T10:39:54.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.443+0000 7f771cff9640 1 -- 172.21.15.23:0/2932599963 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f771009a790 con 0x7f7720100610 2024-03-24T10:39:54.443 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-24T10:39:54.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.445+0000 7f7725bf4640 1 -- 172.21.15.23:0/2932599963 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f770c0a20c0 msgr2=0x7f770c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.445+0000 7f7725bf4640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f770c0a20c0 0x7f770c0a4580 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7708009230 tx=0x7f7708005fd0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.445+0000 7f7725bf4640 1 -- 172.21.15.23:0/2932599963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7720100610 msgr2=0x7f772006a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.445+0000 7f7725bf4640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7720100610 0x7f772006a8b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f771000e9c0 tx=0x7f771000ee80 comp rx=0 tx=0).stop 2024-03-24T10:39:54.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.445+0000 7f7725bf4640 1 -- 172.21.15.23:0/2932599963 shutdown_connections 2024-03-24T10:39:54.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.445+0000 7f7725bf4640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f770c0a20c0 0x7f770c0a4580 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.445+0000 7f7725bf4640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77201021c0 0x7f772019eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.445+0000 7f7725bf4640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7720101800 0x7f772019c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.445+0000 7f7725bf4640 1 --2- 172.21.15.23:0/2932599963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7720100610 0x7f772006a8b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.445+0000 7f7725bf4640 1 -- 172.21.15.23:0/2932599963 >> 172.21.15.23:0/2932599963 conn(0x7f77200fbba0 msgr2=0x7f7720104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:54.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.445+0000 7f7725bf4640 1 -- 172.21.15.23:0/2932599963 shutdown_connections 2024-03-24T10:39:54.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.445+0000 7f7725bf4640 1 -- 172.21.15.23:0/2932599963 wait complete. 2024-03-24T10:39:54.457 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:39:54.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.558+0000 7fd763f56640 1 -- 172.21.15.23:0/709816446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd75c101840 msgr2=0x7fd75c101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.558+0000 7fd763f56640 1 --2- 172.21.15.23:0/709816446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd75c101840 0x7fd75c101cc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009900 tx=0x7fd74c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:54.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.558+0000 7fd763f56640 1 -- 172.21.15.23:0/709816446 shutdown_connections 2024-03-24T10:39:54.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.558+0000 7fd763f56640 1 --2- 172.21.15.23:0/709816446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd75c102200 0x7fd75c108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.558+0000 7fd763f56640 1 --2- 172.21.15.23:0/709816446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd75c101840 0x7fd75c101cc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.558+0000 7fd763f56640 1 --2- 172.21.15.23:0/709816446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c100650 0x7fd75c100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.558+0000 7fd763f56640 1 -- 172.21.15.23:0/709816446 >> 172.21.15.23:0/709816446 conn(0x7fd75c0fbbc0 msgr2=0x7fd75c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:54.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.558+0000 7fd763f56640 1 -- 172.21.15.23:0/709816446 shutdown_connections 2024-03-24T10:39:54.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.558+0000 7fd763f56640 1 -- 172.21.15.23:0/709816446 wait complete. 2024-03-24T10:39:54.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd763f56640 1 Processor -- start 2024-03-24T10:39:54.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd763f56640 1 -- start start 2024-03-24T10:39:54.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd763f56640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd75c100650 0x7fd75c1966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd763f56640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd75c101840 0x7fd75c196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd763f56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c102200 0x7fd75c19dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd763f56640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd75c19e1b0 con 0x7fd75c101840 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd763f56640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd75c19e320 con 0x7fd75c102200 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd763f56640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd75c19e4e0 con 0x7fd75c100650 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd7614f1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd75c101840 0x7fd75c196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd7614f1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd75c101840 0x7fd75c196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53280/0 (socket says 172.21.15.23:53280) 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd7614f1640 1 -- 172.21.15.23:0/3915099311 learned_addr learned my addr 172.21.15.23:0/3915099311 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd761cf2640 1 --2- 172.21.15.23:0/3915099311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd75c100650 0x7fd75c1966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd7624f3640 1 --2- 172.21.15.23:0/3915099311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c102200 0x7fd75c19dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd7614f1640 1 -- 172.21.15.23:0/3915099311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd75c100650 msgr2=0x7fd75c1966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd7614f1640 1 --2- 172.21.15.23:0/3915099311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd75c100650 0x7fd75c1966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd7614f1640 1 -- 172.21.15.23:0/3915099311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c102200 msgr2=0x7fd75c19dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd7614f1640 1 --2- 172.21.15.23:0/3915099311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c102200 0x7fd75c19dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.559+0000 7fd7614f1640 1 -- 172.21.15.23:0/3915099311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd74c009560 con 0x7fd75c101840 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.560+0000 7fd7614f1640 1 --2- 172.21.15.23:0/3915099311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd75c101840 0x7fd75c196c00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd74c006010 tx=0x7fd74c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.560+0000 7fd752ffd640 1 -- 172.21.15.23:0/3915099311 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0043d0 con 0x7fd75c101840 2024-03-24T10:39:54.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.560+0000 7fd752ffd640 1 -- 172.21.15.23:0/3915099311 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd74c0052d0 con 0x7fd75c101840 2024-03-24T10:39:54.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.560+0000 7fd763f56640 1 -- 172.21.15.23:0/3915099311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd75c19e760 con 0x7fd75c101840 2024-03-24T10:39:54.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.560+0000 7fd752ffd640 1 -- 172.21.15.23:0/3915099311 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74c003cc0 con 0x7fd75c101840 2024-03-24T10:39:54.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.560+0000 7fd763f56640 1 -- 172.21.15.23:0/3915099311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd75c19ec70 con 0x7fd75c101840 2024-03-24T10:39:54.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.561+0000 7fd763f56640 1 -- 172.21.15.23:0/3915099311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd75c107970 con 0x7fd75c101840 2024-03-24T10:39:54.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.562+0000 7fd752ffd640 1 -- 172.21.15.23:0/3915099311 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fd74c005440 con 0x7fd75c101840 2024-03-24T10:39:54.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.562+0000 7fd752ffd640 1 --2- 172.21.15.23:0/3915099311 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd7480a20c0 0x7fd7480a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.562+0000 7fd761cf2640 1 --2- 172.21.15.23:0/3915099311 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd7480a20c0 0x7fd7480a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.562+0000 7fd752ffd640 1 -- 172.21.15.23:0/3915099311 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd74c0f0490 con 0x7fd75c101840 2024-03-24T10:39:54.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.563+0000 7fd761cf2640 1 --2- 172.21.15.23:0/3915099311 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd7480a20c0 0x7fd7480a4580 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd75c1016a0 tx=0x7fd758005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:54.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.566+0000 7fd752ffd640 1 -- 172.21.15.23:0/3915099311 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd74c0b9440 con 0x7fd75c101840 2024-03-24T10:39:54.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.787+0000 7fd763f56640 1 -- 172.21.15.23:0/3915099311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd75c10c060 con 0x7fd75c101840 2024-03-24T10:39:54.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.787+0000 7fd752ffd640 1 -- 172.21.15.23:0/3915099311 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11866 (secure 0 0 0) 0x7fd74c0bcb00 con 0x7fd75c101840 2024-03-24T10:39:54.788 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:54.788 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:38:15.911086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:39:54.788 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 22 2024-03-24T10:39:54.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.790+0000 7fd763f56640 1 -- 172.21.15.23:0/3915099311 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd7480a20c0 msgr2=0x7fd7480a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.790+0000 7fd763f56640 1 --2- 172.21.15.23:0/3915099311 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd7480a20c0 0x7fd7480a4580 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd75c1016a0 tx=0x7fd758005f90 comp rx=0 tx=0).stop 2024-03-24T10:39:54.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.790+0000 7fd763f56640 1 -- 172.21.15.23:0/3915099311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd75c101840 msgr2=0x7fd75c196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.790+0000 7fd763f56640 1 --2- 172.21.15.23:0/3915099311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd75c101840 0x7fd75c196c00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd74c006010 tx=0x7fd74c005e80 comp rx=0 tx=0).stop 2024-03-24T10:39:54.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.790+0000 7fd763f56640 1 -- 172.21.15.23:0/3915099311 shutdown_connections 2024-03-24T10:39:54.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.790+0000 7fd763f56640 1 --2- 172.21.15.23:0/3915099311 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fd7480a20c0 0x7fd7480a4580 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.790+0000 7fd763f56640 1 --2- 172.21.15.23:0/3915099311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c102200 0x7fd75c19dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.790+0000 7fd763f56640 1 --2- 172.21.15.23:0/3915099311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd75c101840 0x7fd75c196c00 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.790+0000 7fd763f56640 1 --2- 172.21.15.23:0/3915099311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd75c100650 0x7fd75c1966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.790+0000 7fd763f56640 1 -- 172.21.15.23:0/3915099311 >> 172.21.15.23:0/3915099311 conn(0x7fd75c0fbbc0 msgr2=0x7fd75c104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:54.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.790+0000 7fd763f56640 1 -- 172.21.15.23:0/3915099311 shutdown_connections 2024-03-24T10:39:54.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.790+0000 7fd763f56640 1 -- 172.21.15.23:0/3915099311 wait complete. 2024-03-24T10:39:54.802 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:39:54.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.905+0000 7f2c9edc7640 1 -- 172.21.15.23:0/926052247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c98101730 msgr2=0x7f2c98101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.905+0000 7f2c9edc7640 1 --2- 172.21.15.23:0/926052247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c98101730 0x7f2c98101b50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009900 tx=0x7f2c8c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:54.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.905+0000 7f2c9edc7640 1 -- 172.21.15.23:0/926052247 shutdown_connections 2024-03-24T10:39:54.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.905+0000 7f2c9edc7640 1 --2- 172.21.15.23:0/926052247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c981032e0 0x7f2c98109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.905+0000 7f2c9edc7640 1 --2- 172.21.15.23:0/926052247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c98102920 0x7f2c98102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.905+0000 7f2c9edc7640 1 --2- 172.21.15.23:0/926052247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c98101730 0x7f2c98101b50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.905+0000 7f2c9edc7640 1 -- 172.21.15.23:0/926052247 >> 172.21.15.23:0/926052247 conn(0x7f2c980fccc0 msgr2=0x7f2c980ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:54.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.905+0000 7f2c9edc7640 1 -- 172.21.15.23:0/926052247 shutdown_connections 2024-03-24T10:39:54.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.905+0000 7f2c9edc7640 1 -- 172.21.15.23:0/926052247 wait complete. 2024-03-24T10:39:54.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c9edc7640 1 Processor -- start 2024-03-24T10:39:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c9edc7640 1 -- start start 2024-03-24T10:39:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c9edc7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c98101730 0x7f2c98197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c9edc7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c98102920 0x7f2c98197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c9edc7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c981032e0 0x7f2c9819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c9edc7640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9819f2f0 con 0x7f2c98102920 2024-03-24T10:39:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c9edc7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9819f460 con 0x7f2c981032e0 2024-03-24T10:39:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c9edc7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9819f620 con 0x7f2c98101730 2024-03-24T10:39:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c97fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c98102920 0x7f2c98197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c97fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c98102920 0x7f2c98197d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53288/0 (socket says 172.21.15.23:53288) 2024-03-24T10:39:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c97fff640 1 -- 172.21.15.23:0/2045819304 learned_addr learned my addr 172.21.15.23:0/2045819304 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c9cb63640 1 --2- 172.21.15.23:0/2045819304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c98101730 0x7f2c98197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c9d364640 1 --2- 172.21.15.23:0/2045819304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c981032e0 0x7f2c9819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c97fff640 1 -- 172.21.15.23:0/2045819304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c98101730 msgr2=0x7f2c98197800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c97fff640 1 --2- 172.21.15.23:0/2045819304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c98101730 0x7f2c98197800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c97fff640 1 -- 172.21.15.23:0/2045819304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c981032e0 msgr2=0x7f2c9819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c97fff640 1 --2- 172.21.15.23:0/2045819304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c981032e0 0x7f2c9819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c97fff640 1 -- 172.21.15.23:0/2045819304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c8c009560 con 0x7f2c98102920 2024-03-24T10:39:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.906+0000 7f2c97fff640 1 --2- 172.21.15.23:0/2045819304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c98102920 0x7f2c98197d40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2c8400e9f0 tx=0x7f2c8400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.907+0000 7f2c95ffb640 1 -- 172.21.15.23:0/2045819304 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c84009700 con 0x7f2c98102920 2024-03-24T10:39:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.907+0000 7f2c95ffb640 1 -- 172.21.15.23:0/2045819304 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c8400cd80 con 0x7f2c98102920 2024-03-24T10:39:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.907+0000 7f2c9edc7640 1 -- 172.21.15.23:0/2045819304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c9819f900 con 0x7f2c98102920 2024-03-24T10:39:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.907+0000 7f2c95ffb640 1 -- 172.21.15.23:0/2045819304 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c84018920 con 0x7f2c98102920 2024-03-24T10:39:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.907+0000 7f2c9edc7640 1 -- 172.21.15.23:0/2045819304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c9819fe10 con 0x7f2c98102920 2024-03-24T10:39:54.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.908+0000 7f2c9edc7640 1 -- 172.21.15.23:0/2045819304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c98108a70 con 0x7f2c98102920 2024-03-24T10:39:54.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.909+0000 7f2c95ffb640 1 -- 172.21.15.23:0/2045819304 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f2c84018a80 con 0x7f2c98102920 2024-03-24T10:39:54.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.910+0000 7f2c95ffb640 1 --2- 172.21.15.23:0/2045819304 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2c880a20c0 0x7f2c880a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:54.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.910+0000 7f2c9cb63640 1 --2- 172.21.15.23:0/2045819304 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2c880a20c0 0x7f2c880a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:54.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.910+0000 7f2c95ffb640 1 -- 172.21.15.23:0/2045819304 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2c84014070 con 0x7f2c98102920 2024-03-24T10:39:54.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.910+0000 7f2c9cb63640 1 --2- 172.21.15.23:0/2045819304 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2c880a20c0 0x7f2c880a4580 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c005ab0 tx=0x7f2c8c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:54.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:54.913+0000 7f2c95ffb640 1 -- 172.21.15.23:0/2045819304 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2c84097530 con 0x7f2c98102920 2024-03-24T10:39:55.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.097+0000 7f2c9edc7640 1 -- 172.21.15.23:0/2045819304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c9810d370 con 0x7f2c98102920 2024-03-24T10:39:55.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.098+0000 7f2c95ffb640 1 -- 172.21.15.23:0/2045819304 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22704 (secure 0 0 0) 0x7f2c8409abf0 con 0x7f2c98102920 2024-03-24T10:39:55.099 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:55.099 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:39:16.692832+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:39:55.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.100+0000 7f2c9edc7640 1 -- 172.21.15.23:0/2045819304 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2c880a20c0 msgr2=0x7f2c880a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.100+0000 7f2c9edc7640 1 --2- 172.21.15.23:0/2045819304 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2c880a20c0 0x7f2c880a4580 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c005ab0 tx=0x7f2c8c005fd0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.100+0000 7f2c9edc7640 1 -- 172.21.15.23:0/2045819304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c98102920 msgr2=0x7f2c98197d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.100+0000 7f2c9edc7640 1 --2- 172.21.15.23:0/2045819304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c98102920 0x7f2c98197d40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2c8400e9f0 tx=0x7f2c8400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.100+0000 7f2c9edc7640 1 -- 172.21.15.23:0/2045819304 shutdown_connections 2024-03-24T10:39:55.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.100+0000 7f2c9edc7640 1 --2- 172.21.15.23:0/2045819304 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f2c880a20c0 0x7f2c880a4580 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.100+0000 7f2c9edc7640 1 --2- 172.21.15.23:0/2045819304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c981032e0 0x7f2c9819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.100+0000 7f2c9edc7640 1 --2- 172.21.15.23:0/2045819304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c98102920 0x7f2c98197d40 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.100+0000 7f2c9edc7640 1 --2- 172.21.15.23:0/2045819304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c98101730 0x7f2c98197800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.100+0000 7f2c9edc7640 1 -- 172.21.15.23:0/2045819304 >> 172.21.15.23:0/2045819304 conn(0x7f2c980fccc0 msgr2=0x7f2c98105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.101+0000 7f2c9edc7640 1 -- 172.21.15.23:0/2045819304 shutdown_connections 2024-03-24T10:39:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.101+0000 7f2c9edc7640 1 -- 172.21.15.23:0/2045819304 wait complete. 2024-03-24T10:39:55.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:54 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2778226536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:39:55.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:54 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2932599963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:39:55.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:54 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3915099311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:55.112 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:39.056452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:38:08.473472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:38:09.414667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:39:55.113 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:39:55.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.218+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/1122657678 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eb8102200 msgr2=0x7f5eb81045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.218+0000 7f5ebfc9d640 1 --2- 172.21.15.23:0/1122657678 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eb8102200 0x7f5eb81045f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0009900 tx=0x7f5eb002bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.218+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/1122657678 shutdown_connections 2024-03-24T10:39:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.218+0000 7f5ebfc9d640 1 --2- 172.21.15.23:0/1122657678 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5eb8107460 0x7f5eb8109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.218+0000 7f5ebfc9d640 1 --2- 172.21.15.23:0/1122657678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eb8104b30 0x7f5eb8106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.218+0000 7f5ebfc9d640 1 --2- 172.21.15.23:0/1122657678 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eb8102200 0x7f5eb81045f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.218+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/1122657678 >> 172.21.15.23:0/1122657678 conn(0x7f5eb80fbbb0 msgr2=0x7f5eb80fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.218+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/1122657678 shutdown_connections 2024-03-24T10:39:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.218+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/1122657678 wait complete. 2024-03-24T10:39:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.218+0000 7f5ebfc9d640 1 Processor -- start 2024-03-24T10:39:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.218+0000 7f5ebfc9d640 1 -- start start 2024-03-24T10:39:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.218+0000 7f5ebfc9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eb8102200 0x7f5eb81977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebfc9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eb8104b30 0x7f5eb8197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebfc9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5eb8107460 0x7f5eb819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebfc9d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb819f2a0 con 0x7f5eb8104b30 2024-03-24T10:39:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebfc9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb819f410 con 0x7f5eb8102200 2024-03-24T10:39:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebfc9d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb819f5d0 con 0x7f5eb8107460 2024-03-24T10:39:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebd238640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eb8104b30 0x7f5eb8197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebd238640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eb8104b30 0x7f5eb8197cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53310/0 (socket says 172.21.15.23:53310) 2024-03-24T10:39:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebd238640 1 -- 172.21.15.23:0/4174417542 learned_addr learned my addr 172.21.15.23:0/4174417542 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebda39640 1 --2- 172.21.15.23:0/4174417542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eb8102200 0x7f5eb81977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebe23a640 1 --2- 172.21.15.23:0/4174417542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5eb8107460 0x7f5eb819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebd238640 1 -- 172.21.15.23:0/4174417542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5eb8107460 msgr2=0x7f5eb819ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebd238640 1 --2- 172.21.15.23:0/4174417542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5eb8107460 0x7f5eb819ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebd238640 1 -- 172.21.15.23:0/4174417542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eb8102200 msgr2=0x7f5eb81977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebd238640 1 --2- 172.21.15.23:0/4174417542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eb8102200 0x7f5eb81977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebd238640 1 -- 172.21.15.23:0/4174417542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eb0009560 con 0x7f5eb8104b30 2024-03-24T10:39:55.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebd238640 1 --2- 172.21.15.23:0/4174417542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eb8104b30 0x7f5eb8197cf0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5eac005fd0 tx=0x7f5eac00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:55.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5eb6ffd640 1 -- 172.21.15.23:0/4174417542 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eac009c30 con 0x7f5eb8104b30 2024-03-24T10:39:55.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5eb6ffd640 1 -- 172.21.15.23:0/4174417542 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5eac009d90 con 0x7f5eb8104b30 2024-03-24T10:39:55.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/4174417542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eb819f8b0 con 0x7f5eb8104b30 2024-03-24T10:39:55.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5eb6ffd640 1 -- 172.21.15.23:0/4174417542 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eac010460 con 0x7f5eb8104b30 2024-03-24T10:39:55.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.219+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/4174417542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eb819fe10 con 0x7f5eb8104b30 2024-03-24T10:39:55.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.220+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/4174417542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5eb80fd6a0 con 0x7f5eb8104b30 2024-03-24T10:39:55.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.221+0000 7f5eb6ffd640 1 -- 172.21.15.23:0/4174417542 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f5eac003f20 con 0x7f5eb8104b30 2024-03-24T10:39:55.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.222+0000 7f5eb6ffd640 1 --2- 172.21.15.23:0/4174417542 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f5ea00a2140 0x7f5ea00a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:55.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.222+0000 7f5ebda39640 1 --2- 172.21.15.23:0/4174417542 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f5ea00a2140 0x7f5ea00a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:55.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.222+0000 7f5eb6ffd640 1 -- 172.21.15.23:0/4174417542 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f5eac0cd8e0 con 0x7f5eb8104b30 2024-03-24T10:39:55.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.222+0000 7f5ebda39640 1 --2- 172.21.15.23:0/4174417542 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f5ea00a2140 0x7f5ea00a4600 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0000c00 tx=0x7f5eb002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:55.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.226+0000 7f5eb6ffd640 1 -- 172.21.15.23:0/4174417542 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5eac096890 con 0x7f5eb8104b30 2024-03-24T10:39:55.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:54 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2778226536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:39:55.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:54 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2932599963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:39:55.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:54 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3915099311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:55.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:54 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2778226536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:39:55.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:54 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2932599963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:39:55.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:54 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3915099311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:55.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.407+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/4174417542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5eb802d1d0 con 0x7f5eb8104b30 2024-03-24T10:39:55.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.407+0000 7f5eb6ffd640 1 -- 172.21.15.23:0/4174417542 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f5eac099f50 con 0x7f5eb8104b30 2024-03-24T10:39:55.408 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-24T10:39:55.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.409+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/4174417542 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f5ea00a2140 msgr2=0x7f5ea00a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.409+0000 7f5ebfc9d640 1 --2- 172.21.15.23:0/4174417542 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f5ea00a2140 0x7f5ea00a4600 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0000c00 tx=0x7f5eb002c7e0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.409+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/4174417542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eb8104b30 msgr2=0x7f5eb8197cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.409+0000 7f5ebfc9d640 1 --2- 172.21.15.23:0/4174417542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eb8104b30 0x7f5eb8197cf0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5eac005fd0 tx=0x7f5eac00dde0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.409+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/4174417542 shutdown_connections 2024-03-24T10:39:55.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.409+0000 7f5ebfc9d640 1 --2- 172.21.15.23:0/4174417542 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f5ea00a2140 0x7f5ea00a4600 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.409+0000 7f5ebfc9d640 1 --2- 172.21.15.23:0/4174417542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5eb8107460 0x7f5eb819ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.409+0000 7f5ebfc9d640 1 --2- 172.21.15.23:0/4174417542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eb8104b30 0x7f5eb8197cf0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.409+0000 7f5ebfc9d640 1 --2- 172.21.15.23:0/4174417542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eb8102200 0x7f5eb81977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.409+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/4174417542 >> 172.21.15.23:0/4174417542 conn(0x7f5eb80fbbb0 msgr2=0x7f5eb8102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:55.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.409+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/4174417542 shutdown_connections 2024-03-24T10:39:55.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.409+0000 7f5ebfc9d640 1 -- 172.21.15.23:0/4174417542 wait complete. 2024-03-24T10:39:55.422 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:39:55.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.526+0000 7f9196007640 1 -- 172.21.15.23:0/1546397393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9190100610 msgr2=0x7f9190100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.526+0000 7f9196007640 1 --2- 172.21.15.23:0/1546397393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9190100610 0x7f9190100a30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9180009900 tx=0x7f918002bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:55.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.526+0000 7f9196007640 1 -- 172.21.15.23:0/1546397393 shutdown_connections 2024-03-24T10:39:55.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.526+0000 7f9196007640 1 --2- 172.21.15.23:0/1546397393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91901021c0 0x7f9190108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.526+0000 7f9196007640 1 --2- 172.21.15.23:0/1546397393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9190101800 0x7f9190101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.526+0000 7f9196007640 1 --2- 172.21.15.23:0/1546397393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9190100610 0x7f9190100a30 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.526+0000 7f9196007640 1 -- 172.21.15.23:0/1546397393 >> 172.21.15.23:0/1546397393 conn(0x7f91900fbba0 msgr2=0x7f91900fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:55.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.526+0000 7f9196007640 1 -- 172.21.15.23:0/1546397393 shutdown_connections 2024-03-24T10:39:55.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.526+0000 7f9196007640 1 -- 172.21.15.23:0/1546397393 wait complete. 2024-03-24T10:39:55.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.527+0000 7f9196007640 1 Processor -- start 2024-03-24T10:39:55.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.527+0000 7f9196007640 1 -- start start 2024-03-24T10:39:55.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.527+0000 7f9196007640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9190100610 0x7f91901966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:55.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.527+0000 7f9196007640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9190101800 0x7f9190196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:55.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.527+0000 7f9196007640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91901021c0 0x7f919019dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:55.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.527+0000 7f9196007640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f919019e1b0 con 0x7f91901021c0 2024-03-24T10:39:55.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.527+0000 7f9196007640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f919019e320 con 0x7f9190101800 2024-03-24T10:39:55.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.527+0000 7f9196007640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f919019e4e0 con 0x7f9190100610 2024-03-24T10:39:55.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.527+0000 7f918ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91901021c0 0x7f919019dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:55.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.527+0000 7f918ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91901021c0 0x7f919019dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53340/0 (socket says 172.21.15.23:53340) 2024-03-24T10:39:55.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.527+0000 7f918ffff640 1 -- 172.21.15.23:0/2028203985 learned_addr learned my addr 172.21.15.23:0/2028203985 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:55.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.527+0000 7f918f7fe640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9190100610 0x7f91901966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.527+0000 7f918effd640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9190101800 0x7f9190196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.528+0000 7f918ffff640 1 -- 172.21.15.23:0/2028203985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9190100610 msgr2=0x7f91901966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.528+0000 7f918ffff640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9190100610 0x7f91901966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.528+0000 7f918ffff640 1 -- 172.21.15.23:0/2028203985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9190101800 msgr2=0x7f9190196c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.528+0000 7f918ffff640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9190101800 0x7f9190196c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.528+0000 7f918ffff640 1 -- 172.21.15.23:0/2028203985 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9180009560 con 0x7f91901021c0 2024-03-24T10:39:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.528+0000 7f918f7fe640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9190100610 0x7f91901966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.528+0000 7f918ffff640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91901021c0 0x7f919019dc70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f918800eb10 tx=0x7f918800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.528+0000 7f918cff9640 1 -- 172.21.15.23:0/2028203985 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188004b20 con 0x7f91901021c0 2024-03-24T10:39:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.528+0000 7f9196007640 1 -- 172.21.15.23:0/2028203985 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f919019e7c0 con 0x7f91901021c0 2024-03-24T10:39:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.528+0000 7f918cff9640 1 -- 172.21.15.23:0/2028203985 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9188004c80 con 0x7f91901021c0 2024-03-24T10:39:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.528+0000 7f918cff9640 1 -- 172.21.15.23:0/2028203985 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188010650 con 0x7f91901021c0 2024-03-24T10:39:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.528+0000 7f9196007640 1 -- 172.21.15.23:0/2028203985 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f919019ec80 con 0x7f91901021c0 2024-03-24T10:39:55.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.529+0000 7f9196007640 1 -- 172.21.15.23:0/2028203985 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9190107a50 con 0x7f91901021c0 2024-03-24T10:39:55.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.530+0000 7f918cff9640 1 -- 172.21.15.23:0/2028203985 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f9188005020 con 0x7f91901021c0 2024-03-24T10:39:55.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.531+0000 7f918cff9640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f91780a20c0 0x7f91780a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:55.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.531+0000 7f918f7fe640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f91780a20c0 0x7f91780a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:55.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.531+0000 7f918cff9640 1 -- 172.21.15.23:0/2028203985 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f9188014070 con 0x7f91901021c0 2024-03-24T10:39:55.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.531+0000 7f918f7fe640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f91780a20c0 0x7f91780a4580 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9180005ab0 tx=0x7f9180005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:55.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.535+0000 7f918cff9640 1 -- 172.21.15.23:0/2028203985 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9188097200 con 0x7f91901021c0 2024-03-24T10:39:55.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.718+0000 7f9196007640 1 -- 172.21.15.23:0/2028203985 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f919002d1d0 con 0x7f91901021c0 2024-03-24T10:39:55.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.719+0000 7f918cff9640 1 -- 172.21.15.23:0/2028203985 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f918809a8c0 con 0x7f91901021c0 2024-03-24T10:39:55.719 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-24T10:39:55.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.721+0000 7f9196007640 1 -- 172.21.15.23:0/2028203985 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f91780a20c0 msgr2=0x7f91780a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.721+0000 7f9196007640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f91780a20c0 0x7f91780a4580 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9180005ab0 tx=0x7f9180005fd0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.721+0000 7f9196007640 1 -- 172.21.15.23:0/2028203985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91901021c0 msgr2=0x7f919019dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.721+0000 7f9196007640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91901021c0 0x7f919019dc70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f918800eb10 tx=0x7f918800c2d0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.721+0000 7f9196007640 1 -- 172.21.15.23:0/2028203985 shutdown_connections 2024-03-24T10:39:55.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.721+0000 7f9196007640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f91780a20c0 0x7f91780a4580 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.721+0000 7f9196007640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91901021c0 0x7f919019dc70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.721+0000 7f9196007640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9190101800 0x7f9190196c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.721+0000 7f9196007640 1 --2- 172.21.15.23:0/2028203985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9190100610 0x7f91901966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.721+0000 7f9196007640 1 -- 172.21.15.23:0/2028203985 >> 172.21.15.23:0/2028203985 conn(0x7f91900fbba0 msgr2=0x7f9190104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:55.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.721+0000 7f9196007640 1 -- 172.21.15.23:0/2028203985 shutdown_connections 2024-03-24T10:39:55.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.721+0000 7f9196007640 1 -- 172.21.15.23:0/2028203985 wait complete. 2024-03-24T10:39:55.732 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:39:55.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.837+0000 7f7563db5640 1 -- 172.21.15.23:0/4066931821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755c101800 msgr2=0x7f755c101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.837+0000 7f7563db5640 1 --2- 172.21.15.23:0/4066931821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755c101800 0x7f755c101c80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f754c009900 tx=0x7f754c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:55.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.837+0000 7f7563db5640 1 -- 172.21.15.23:0/4066931821 shutdown_connections 2024-03-24T10:39:55.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.837+0000 7f7563db5640 1 --2- 172.21.15.23:0/4066931821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f755c1021c0 0x7f755c108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.837+0000 7f7563db5640 1 --2- 172.21.15.23:0/4066931821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755c101800 0x7f755c101c80 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.837+0000 7f7563db5640 1 --2- 172.21.15.23:0/4066931821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f755c100610 0x7f755c100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.837+0000 7f7563db5640 1 -- 172.21.15.23:0/4066931821 >> 172.21.15.23:0/4066931821 conn(0x7f755c0fbb80 msgr2=0x7f755c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:55.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.838+0000 7f7563db5640 1 -- 172.21.15.23:0/4066931821 shutdown_connections 2024-03-24T10:39:55.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.838+0000 7f7563db5640 1 -- 172.21.15.23:0/4066931821 wait complete. 2024-03-24T10:39:55.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.838+0000 7f7563db5640 1 Processor -- start 2024-03-24T10:39:55.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.838+0000 7f7563db5640 1 -- start start 2024-03-24T10:39:55.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.838+0000 7f7563db5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f755c100610 0x7f755c1977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:55.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.838+0000 7f7563db5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755c101800 0x7f755c197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:55.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.838+0000 7f7563db5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f755c1021c0 0x7f755c19ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:55.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.838+0000 7f7563db5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f755c19f2c0 con 0x7f755c101800 2024-03-24T10:39:55.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.838+0000 7f7563db5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f755c19f430 con 0x7f755c100610 2024-03-24T10:39:55.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.838+0000 7f7563db5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f755c19f5f0 con 0x7f755c1021c0 2024-03-24T10:39:55.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.838+0000 7f7561350640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755c101800 0x7f755c197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:55.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.838+0000 7f7561350640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755c101800 0x7f755c197d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53350/0 (socket says 172.21.15.23:53350) 2024-03-24T10:39:55.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.838+0000 7f7561350640 1 -- 172.21.15.23:0/1423139444 learned_addr learned my addr 172.21.15.23:0/1423139444 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:55.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.838+0000 7f7561b51640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f755c100610 0x7f755c1977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:55.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.839+0000 7f7562352640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f755c1021c0 0x7f755c19ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:55.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.839+0000 7f7561350640 1 -- 172.21.15.23:0/1423139444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f755c1021c0 msgr2=0x7f755c19ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.839+0000 7f7561350640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f755c1021c0 0x7f755c19ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.839+0000 7f7561350640 1 -- 172.21.15.23:0/1423139444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f755c100610 msgr2=0x7f755c1977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:55.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.839+0000 7f7561350640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f755c100610 0x7f755c1977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:55.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.839+0000 7f7561350640 1 -- 172.21.15.23:0/1423139444 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754c009560 con 0x7f755c101800 2024-03-24T10:39:55.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.839+0000 7f7561b51640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f755c100610 0x7f755c1977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:55.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.839+0000 7f7561350640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755c101800 0x7f755c197d10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f754c006010 tx=0x7f754c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:55.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.839+0000 7f7552ffd640 1 -- 172.21.15.23:0/1423139444 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754c0043d0 con 0x7f755c101800 2024-03-24T10:39:55.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.839+0000 7f7563db5640 1 -- 172.21.15.23:0/1423139444 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f755c19f870 con 0x7f755c101800 2024-03-24T10:39:55.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.839+0000 7f7552ffd640 1 -- 172.21.15.23:0/1423139444 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f754c0052d0 con 0x7f755c101800 2024-03-24T10:39:55.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.839+0000 7f7552ffd640 1 -- 172.21.15.23:0/1423139444 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754c003cc0 con 0x7f755c101800 2024-03-24T10:39:55.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.839+0000 7f7563db5640 1 -- 172.21.15.23:0/1423139444 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f755c19fd80 con 0x7f755c101800 2024-03-24T10:39:55.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.840+0000 7f7563db5640 1 -- 172.21.15.23:0/1423139444 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f755c10c020 con 0x7f755c101800 2024-03-24T10:39:55.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.841+0000 7f7552ffd640 1 -- 172.21.15.23:0/1423139444 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f754c005440 con 0x7f755c101800 2024-03-24T10:39:55.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.842+0000 7f7552ffd640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75480a20c0 0x7f75480a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:55.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.842+0000 7f7561b51640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75480a20c0 0x7f75480a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:55.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.842+0000 7f7552ffd640 1 -- 172.21.15.23:0/1423139444 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f754c0efee0 con 0x7f755c101800 2024-03-24T10:39:55.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.842+0000 7f7561b51640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75480a20c0 0x7f75480a4580 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f755c101660 tx=0x7f7558005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:55.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:55.846+0000 7f7552ffd640 1 -- 172.21.15.23:0/1423139444 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f754c0b8e90 con 0x7f755c101800 2024-03-24T10:39:56.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.030+0000 7f7563db5640 1 -- 172.21.15.23:0/1423139444 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f755c1a01f0 con 0x7f755c101800 2024-03-24T10:39:56.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.031+0000 7f7552ffd640 1 -- 172.21.15.23:0/1423139444 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f754c0bc550 con 0x7f755c101800 2024-03-24T10:39:56.031 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-24T10:39:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.032+0000 7f7563db5640 1 -- 172.21.15.23:0/1423139444 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75480a20c0 msgr2=0x7f75480a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.032+0000 7f7563db5640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75480a20c0 0x7f75480a4580 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f755c101660 tx=0x7f7558005f90 comp rx=0 tx=0).stop 2024-03-24T10:39:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.033+0000 7f7563db5640 1 -- 172.21.15.23:0/1423139444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755c101800 msgr2=0x7f755c197d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.033+0000 7f7563db5640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755c101800 0x7f755c197d10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f754c006010 tx=0x7f754c005e80 comp rx=0 tx=0).stop 2024-03-24T10:39:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.033+0000 7f7563db5640 1 -- 172.21.15.23:0/1423139444 shutdown_connections 2024-03-24T10:39:56.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.033+0000 7f7563db5640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75480a20c0 0x7f75480a4580 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.033+0000 7f7563db5640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f755c1021c0 0x7f755c19ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.033+0000 7f7563db5640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755c101800 0x7f755c197d10 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.033+0000 7f7563db5640 1 --2- 172.21.15.23:0/1423139444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f755c100610 0x7f755c1977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.033+0000 7f7563db5640 1 -- 172.21.15.23:0/1423139444 >> 172.21.15.23:0/1423139444 conn(0x7f755c0fbb80 msgr2=0x7f755c068870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.033+0000 7f7563db5640 1 -- 172.21.15.23:0/1423139444 shutdown_connections 2024-03-24T10:39:56.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.033+0000 7f7563db5640 1 -- 172.21.15.23:0/1423139444 wait complete. 2024-03-24T10:39:56.043 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:55 smithi023 ceph-mon[32466]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-24T10:39:56.043 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:55 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2045819304' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:56.043 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:55 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4174417542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:39:56.043 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:55 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2028203985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:39:56.044 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:39:56.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.146+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2650979066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3781033b0 msgr2=0x7fe378109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.146+0000 7fe37f6f4640 1 --2- 172.21.15.23:0/2650979066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3781033b0 0x7fe378109c40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe370009ca0 tx=0x7fe37002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.146+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2650979066 shutdown_connections 2024-03-24T10:39:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.146+0000 7fe37f6f4640 1 --2- 172.21.15.23:0/2650979066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3781033b0 0x7fe378109c40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.146+0000 7fe37f6f4640 1 --2- 172.21.15.23:0/2650979066 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3781029f0 0x7fe378102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.146+0000 7fe37f6f4640 1 --2- 172.21.15.23:0/2650979066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3780ffa20 0x7fe3780ffe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.146+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2650979066 >> 172.21.15.23:0/2650979066 conn(0x7fe3780fb350 msgr2=0x7fe3780fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.146+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2650979066 shutdown_connections 2024-03-24T10:39:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.146+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2650979066 wait complete. 2024-03-24T10:39:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.146+0000 7fe37f6f4640 1 Processor -- start 2024-03-24T10:39:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37f6f4640 1 -- start start 2024-03-24T10:39:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37f6f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3780ffa20 0x7fe3781977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37f6f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3781029f0 0x7fe378197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37f6f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3781033b0 0x7fe37819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37f6f4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe37819f2b0 con 0x7fe3780ffa20 2024-03-24T10:39:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37f6f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe37819f420 con 0x7fe3781029f0 2024-03-24T10:39:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37f6f4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe37819f5e0 con 0x7fe3781033b0 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37d490640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3780ffa20 0x7fe3781977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37d490640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3780ffa20 0x7fe3781977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53366/0 (socket says 172.21.15.23:53366) 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37d490640 1 -- 172.21.15.23:0/2168182447 learned_addr learned my addr 172.21.15.23:0/2168182447 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37cc8f640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3781029f0 0x7fe378197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37dc91640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3781033b0 0x7fe37819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37d490640 1 -- 172.21.15.23:0/2168182447 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3781033b0 msgr2=0x7fe37819ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37d490640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3781033b0 0x7fe37819ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37d490640 1 -- 172.21.15.23:0/2168182447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3781029f0 msgr2=0x7fe378197d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37d490640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3781029f0 0x7fe378197d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37d490640 1 -- 172.21.15.23:0/2168182447 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe370009900 con 0x7fe3780ffa20 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37cc8f640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3781029f0 0x7fe378197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37d490640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3780ffa20 0x7fe3781977c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe37400e9c0 tx=0x7fe37400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe36e7fc640 1 -- 172.21.15.23:0/2168182447 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe374009700 con 0x7fe3780ffa20 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe36e7fc640 1 -- 172.21.15.23:0/2168182447 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe37400cd80 con 0x7fe3780ffa20 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2168182447 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe37819f8c0 con 0x7fe3780ffa20 2024-03-24T10:39:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.147+0000 7fe36e7fc640 1 -- 172.21.15.23:0/2168182447 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe374010430 con 0x7fe3780ffa20 2024-03-24T10:39:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.148+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2168182447 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe37819fe20 con 0x7fe3780ffa20 2024-03-24T10:39:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.148+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2168182447 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe378108b20 con 0x7fe3780ffa20 2024-03-24T10:39:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.150+0000 7fe36e7fc640 1 -- 172.21.15.23:0/2168182447 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fe374003680 con 0x7fe3780ffa20 2024-03-24T10:39:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.151+0000 7fe36e7fc640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe3640a2140 0x7fe3640a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.151+0000 7fe37cc8f640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe3640a2140 0x7fe3640a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.151+0000 7fe36e7fc640 1 -- 172.21.15.23:0/2168182447 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe374014070 con 0x7fe3780ffa20 2024-03-24T10:39:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.151+0000 7fe37cc8f640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe3640a2140 0x7fe3640a4600 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe378198cd0 tx=0x7fe368005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.155+0000 7fe36e7fc640 1 -- 172.21.15.23:0/2168182447 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe3740970d0 con 0x7fe3780ffa20 2024-03-24T10:39:56.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:55 smithi117 ceph-mon[36368]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-24T10:39:56.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:55 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2045819304' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:56.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:55 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4174417542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:39:56.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:55 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2028203985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:39:56.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:55 smithi116 ceph-mon[36248]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-24T10:39:56.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:55 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2045819304' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:56.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:55 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4174417542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:39:56.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:55 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2028203985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:39:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.376+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2168182447 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe37810d170 con 0x7fe3780ffa20 2024-03-24T10:39:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.377+0000 7fe36e7fc640 1 -- 172.21.15.23:0/2168182447 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11866 (secure 0 0 0) 0x7fe37409a790 con 0x7fe3780ffa20 2024-03-24T10:39:56.377 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:56.378 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:38:15.911086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:39:56.378 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 22 2024-03-24T10:39:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.379+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2168182447 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe3640a2140 msgr2=0x7fe3640a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.379+0000 7fe37f6f4640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe3640a2140 0x7fe3640a4600 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe378198cd0 tx=0x7fe368005f90 comp rx=0 tx=0).stop 2024-03-24T10:39:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.379+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2168182447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3780ffa20 msgr2=0x7fe3781977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.379+0000 7fe37f6f4640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3780ffa20 0x7fe3781977c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe37400e9c0 tx=0x7fe37400ee80 comp rx=0 tx=0).stop 2024-03-24T10:39:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.379+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2168182447 shutdown_connections 2024-03-24T10:39:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.379+0000 7fe37f6f4640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7fe3640a2140 0x7fe3640a4600 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.379+0000 7fe37f6f4640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3781033b0 0x7fe37819ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.379+0000 7fe37f6f4640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3781029f0 0x7fe378197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.379+0000 7fe37f6f4640 1 --2- 172.21.15.23:0/2168182447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3780ffa20 0x7fe3781977c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.379+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2168182447 >> 172.21.15.23:0/2168182447 conn(0x7fe3780fb350 msgr2=0x7fe378105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.379+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2168182447 shutdown_connections 2024-03-24T10:39:56.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.379+0000 7fe37f6f4640 1 -- 172.21.15.23:0/2168182447 wait complete. 2024-03-24T10:39:56.391 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:39:56.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.493+0000 7f033d062640 1 -- 172.21.15.23:0/2847970869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03380ff7b0 msgr2=0x7f03380ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.493+0000 7f033d062640 1 --2- 172.21.15.23:0/2847970869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03380ff7b0 0x7f03380ffc30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f032c009900 tx=0x7f032c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:56.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.493+0000 7f033d062640 1 -- 172.21.15.23:0/2847970869 shutdown_connections 2024-03-24T10:39:56.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.493+0000 7f033d062640 1 --2- 172.21.15.23:0/2847970869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0338100170 0x7f0338109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.493+0000 7f033d062640 1 --2- 172.21.15.23:0/2847970869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03380ff7b0 0x7f03380ffc30 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.493+0000 7f033d062640 1 --2- 172.21.15.23:0/2847970869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0338102d40 0x7f0338103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.493+0000 7f033d062640 1 -- 172.21.15.23:0/2847970869 >> 172.21.15.23:0/2847970869 conn(0x7f03380fb2f0 msgr2=0x7f03380fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.493+0000 7f033d062640 1 -- 172.21.15.23:0/2847970869 shutdown_connections 2024-03-24T10:39:56.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.493+0000 7f033d062640 1 -- 172.21.15.23:0/2847970869 wait complete. 2024-03-24T10:39:56.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f033d062640 1 Processor -- start 2024-03-24T10:39:56.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f033d062640 1 -- start start 2024-03-24T10:39:56.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f033d062640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03380ff7b0 0x7f0338197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f033d062640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0338100170 0x7f0338197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f033d062640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0338102d40 0x7f033819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f033d062640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f033819f310 con 0x7f03380ff7b0 2024-03-24T10:39:56.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f033d062640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f033819f480 con 0x7f0338102d40 2024-03-24T10:39:56.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f033d062640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f033819f640 con 0x7f0338100170 2024-03-24T10:39:56.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f0336d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03380ff7b0 0x7f0338197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f0336d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03380ff7b0 0x7f0338197820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53388/0 (socket says 172.21.15.23:53388) 2024-03-24T10:39:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f0336d9d640 1 -- 172.21.15.23:0/3240903240 learned_addr learned my addr 172.21.15.23:0/3240903240 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f033659c640 1 --2- 172.21.15.23:0/3240903240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0338100170 0x7f0338197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f033759e640 1 --2- 172.21.15.23:0/3240903240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0338102d40 0x7f033819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f0336d9d640 1 -- 172.21.15.23:0/3240903240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0338100170 msgr2=0x7f0338197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f0336d9d640 1 --2- 172.21.15.23:0/3240903240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0338100170 0x7f0338197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f0336d9d640 1 -- 172.21.15.23:0/3240903240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0338102d40 msgr2=0x7f033819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f0336d9d640 1 --2- 172.21.15.23:0/3240903240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0338102d40 0x7f033819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.494+0000 7f0336d9d640 1 -- 172.21.15.23:0/3240903240 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0330009840 con 0x7f03380ff7b0 2024-03-24T10:39:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.495+0000 7f0336d9d640 1 --2- 172.21.15.23:0/3240903240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03380ff7b0 0x7f0338197820 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f032800e9c0 tx=0x7f032800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.495+0000 7f031ffff640 1 -- 172.21.15.23:0/3240903240 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03280096e0 con 0x7f03380ff7b0 2024-03-24T10:39:56.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.495+0000 7f033d062640 1 -- 172.21.15.23:0/3240903240 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f032c009560 con 0x7f03380ff7b0 2024-03-24T10:39:56.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.495+0000 7f031ffff640 1 -- 172.21.15.23:0/3240903240 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f032800ccc0 con 0x7f03380ff7b0 2024-03-24T10:39:56.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.495+0000 7f031ffff640 1 -- 172.21.15.23:0/3240903240 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0328010430 con 0x7f03380ff7b0 2024-03-24T10:39:56.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.495+0000 7f033d062640 1 -- 172.21.15.23:0/3240903240 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f033819fc20 con 0x7f03380ff7b0 2024-03-24T10:39:56.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.496+0000 7f033d062640 1 -- 172.21.15.23:0/3240903240 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0338108c00 con 0x7f03380ff7b0 2024-03-24T10:39:56.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.497+0000 7f031ffff640 1 -- 172.21.15.23:0/3240903240 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f0328003f80 con 0x7f03380ff7b0 2024-03-24T10:39:56.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.497+0000 7f031ffff640 1 --2- 172.21.15.23:0/3240903240 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f03200a20c0 0x7f03200a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.497+0000 7f033659c640 1 --2- 172.21.15.23:0/3240903240 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f03200a20c0 0x7f03200a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.497+0000 7f031ffff640 1 -- 172.21.15.23:0/3240903240 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0328014070 con 0x7f03380ff7b0 2024-03-24T10:39:56.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.498+0000 7f033659c640 1 --2- 172.21.15.23:0/3240903240 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f03200a20c0 0x7f03200a4580 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f032c009230 tx=0x7f032c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.502+0000 7f031ffff640 1 -- 172.21.15.23:0/3240903240 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0328097060 con 0x7f03380ff7b0 2024-03-24T10:39:56.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.685+0000 7f033d062640 1 -- 172.21.15.23:0/3240903240 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f033810d150 con 0x7f03380ff7b0 2024-03-24T10:39:56.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.687+0000 7f031ffff640 1 -- 172.21.15.23:0/3240903240 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22704 (secure 0 0 0) 0x7f032809a720 con 0x7f03380ff7b0 2024-03-24T10:39:56.687 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:39:56.687 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:39:16.692832+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:39:56.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.689+0000 7f033d062640 1 -- 172.21.15.23:0/3240903240 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f03200a20c0 msgr2=0x7f03200a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.689+0000 7f033d062640 1 --2- 172.21.15.23:0/3240903240 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f03200a20c0 0x7f03200a4580 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f032c009230 tx=0x7f032c005fd0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.689+0000 7f033d062640 1 -- 172.21.15.23:0/3240903240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03380ff7b0 msgr2=0x7f0338197820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.689+0000 7f033d062640 1 --2- 172.21.15.23:0/3240903240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03380ff7b0 0x7f0338197820 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f032800e9c0 tx=0x7f032800edc0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.689+0000 7f033d062640 1 -- 172.21.15.23:0/3240903240 shutdown_connections 2024-03-24T10:39:56.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.689+0000 7f033d062640 1 --2- 172.21.15.23:0/3240903240 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f03200a20c0 0x7f03200a4580 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.689+0000 7f033d062640 1 --2- 172.21.15.23:0/3240903240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0338102d40 0x7f033819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.689+0000 7f033d062640 1 --2- 172.21.15.23:0/3240903240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0338100170 0x7f0338197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.689+0000 7f033d062640 1 --2- 172.21.15.23:0/3240903240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03380ff7b0 0x7f0338197820 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.689+0000 7f033d062640 1 -- 172.21.15.23:0/3240903240 >> 172.21.15.23:0/3240903240 conn(0x7f03380fb2f0 msgr2=0x7f0338105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.689+0000 7f033d062640 1 -- 172.21.15.23:0/3240903240 shutdown_connections 2024-03-24T10:39:56.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.689+0000 7f033d062640 1 -- 172.21.15.23:0/3240903240 wait complete. 2024-03-24T10:39:56.701 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-24T10:38:10.206899+0000', 'modified': '2024-03-24T10:38:15.911086+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24539, 'mds_1': 24505, 'mds_2': 24503}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24470': {'gid': 24470, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6835/1535852767', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6834', 'nonce': 1535852767}, {'type': 'v1', 'addr': '172.21.15.116:6835', 'nonce': 1535852767}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.117:6835/1664408853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 1664408853}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 1664408853}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.117:6837/977076441', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6836', 'nonce': 977076441}, {'type': 'v1', 'addr': '172.21.15.117:6837', 'nonce': 977076441}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.116:6837/3576031724', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6836', 'nonce': 3576031724}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 3576031724}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.117:6839/263724286', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6838', 'nonce': 263724286}, {'type': 'v1', 'addr': '172.21.15.117:6839', 'nonce': 263724286}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24539': {'gid': 24539, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.116:6839/2435082224', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 2435082224}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 2435082224}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24539, 'qdb_cluster': [24503, 24505, 24539]} 2024-03-24T10:39:56.701 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-24T10:39:56.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-24T10:39:56.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.804+0000 7f75b3fc7640 1 -- 172.21.15.23:0/1166614986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ac1021c0 msgr2=0x7f75ac108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.804+0000 7f75b3fc7640 1 --2- 172.21.15.23:0/1166614986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ac1021c0 0x7f75ac108a50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009900 tx=0x7f75a402bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:56.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.804+0000 7f75b3fc7640 1 -- 172.21.15.23:0/1166614986 shutdown_connections 2024-03-24T10:39:56.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.804+0000 7f75b3fc7640 1 --2- 172.21.15.23:0/1166614986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ac1021c0 0x7f75ac108a50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.804+0000 7f75b3fc7640 1 --2- 172.21.15.23:0/1166614986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac101800 0x7f75ac101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.804+0000 7f75b3fc7640 1 --2- 172.21.15.23:0/1166614986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75ac100610 0x7f75ac100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.804+0000 7f75b3fc7640 1 -- 172.21.15.23:0/1166614986 >> 172.21.15.23:0/1166614986 conn(0x7f75ac0fbba0 msgr2=0x7f75ac0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.804+0000 7f75b3fc7640 1 -- 172.21.15.23:0/1166614986 shutdown_connections 2024-03-24T10:39:56.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.804+0000 7f75b3fc7640 1 -- 172.21.15.23:0/1166614986 wait complete. 2024-03-24T10:39:56.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b3fc7640 1 Processor -- start 2024-03-24T10:39:56.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b3fc7640 1 -- start start 2024-03-24T10:39:56.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b3fc7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ac100610 0x7f75ac069400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b3fc7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75ac101800 0x7f75ac19c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b3fc7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac1021c0 0x7f75ac19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b3fc7640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac19f270 con 0x7f75ac100610 2024-03-24T10:39:56.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b3fc7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac19f3e0 con 0x7f75ac1021c0 2024-03-24T10:39:56.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b3fc7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac19f5a0 con 0x7f75ac101800 2024-03-24T10:39:56.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b1d63640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ac100610 0x7f75ac069400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b1d63640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ac100610 0x7f75ac069400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53400/0 (socket says 172.21.15.23:53400) 2024-03-24T10:39:56.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b1d63640 1 -- 172.21.15.23:0/3952341110 learned_addr learned my addr 172.21.15.23:0/3952341110 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:56.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b1562640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75ac101800 0x7f75ac19c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b2564640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac1021c0 0x7f75ac19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b1d63640 1 -- 172.21.15.23:0/3952341110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75ac101800 msgr2=0x7f75ac19c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b1d63640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75ac101800 0x7f75ac19c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b1d63640 1 -- 172.21.15.23:0/3952341110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac1021c0 msgr2=0x7f75ac19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b1d63640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac1021c0 0x7f75ac19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b1d63640 1 -- 172.21.15.23:0/3952341110 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a4009560 con 0x7f75ac100610 2024-03-24T10:39:56.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b2564640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac1021c0 0x7f75ac19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:39:56.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.805+0000 7f75b1562640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75ac101800 0x7f75ac19c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:39:56.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.806+0000 7f75b1d63640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ac100610 0x7f75ac069400 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f759c00eb70 tx=0x7f759c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.806+0000 7f75aaffd640 1 -- 172.21.15.23:0/3952341110 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c003b70 con 0x7f75ac100610 2024-03-24T10:39:56.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.806+0000 7f75b3fc7640 1 -- 172.21.15.23:0/3952341110 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ac19f880 con 0x7f75ac100610 2024-03-24T10:39:56.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.806+0000 7f75aaffd640 1 -- 172.21.15.23:0/3952341110 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f759c00ee70 con 0x7f75ac100610 2024-03-24T10:39:56.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.806+0000 7f75aaffd640 1 -- 172.21.15.23:0/3952341110 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c0107f0 con 0x7f75ac100610 2024-03-24T10:39:56.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.806+0000 7f75b3fc7640 1 -- 172.21.15.23:0/3952341110 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ac19fd40 con 0x7f75ac100610 2024-03-24T10:39:56.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b6ba4740 1 Processor -- start 2024-03-24T10:39:56.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b6ba4740 1 -- start start 2024-03-24T10:39:56.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75aaffd640 1 -- 172.21.15.23:0/3952341110 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f759c004020 con 0x7f75ac100610 2024-03-24T10:39:56.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b6ba4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758560 0x5633d6758980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b6ba4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d6758f50 0x5633d675da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b6ba4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d675e0c0 0x5633d6760550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b6ba4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5633d66c7180 con 0x5633d675e0c0 2024-03-24T10:39:56.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b6ba4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633d66c46b0 con 0x5633d6758560 2024-03-24T10:39:56.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b6ba4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5633d66bfe90 con 0x5633d6758f50 2024-03-24T10:39:56.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b1562640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d675e0c0 0x5633d6760550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b2564640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758560 0x5633d6758980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b1562640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d675e0c0 0x5633d6760550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53410/0 (socket says 172.21.15.23:53410) 2024-03-24T10:39:56.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b1562640 1 -- 172.21.15.23:0/757538788 learned_addr learned my addr 172.21.15.23:0/757538788 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:56.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b1d63640 1 --2- 172.21.15.23:0/757538788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d6758f50 0x5633d675da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b1562640 1 -- 172.21.15.23:0/757538788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d6758f50 msgr2=0x5633d675da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b1562640 1 --2- 172.21.15.23:0/757538788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d6758f50 0x5633d675da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b1562640 1 -- 172.21.15.23:0/757538788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758560 msgr2=0x5633d6758980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b1562640 1 --2- 172.21.15.23:0/757538788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758560 0x5633d6758980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.809+0000 7f75b1562640 1 -- 172.21.15.23:0/757538788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633d667cd80 con 0x5633d675e0c0 2024-03-24T10:39:56.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.810+0000 7f75b2564640 1 --2- 172.21.15.23:0/757538788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758560 0x5633d6758980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:56.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.810+0000 7f75aaffd640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75940a20c0 0x7f75940a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.810+0000 7f75b1562640 1 --2- 172.21.15.23:0/757538788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d675e0c0 0x5633d6760550 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009cf0 tx=0x7f75a00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e714b728663db7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.810+0000 7f75b2564640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75940a20c0 0x7f75940a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.810+0000 7f75aaffd640 1 -- 172.21.15.23:0/3952341110 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f759c014070 con 0x7f75ac100610 2024-03-24T10:39:56.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.810+0000 7f7593fff640 1 -- 172.21.15.23:0/757538788 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0012450 con 0x5633d675e0c0 2024-03-24T10:39:56.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.810+0000 7f7593fff640 1 -- 172.21.15.23:0/757538788 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a0012a90 con 0x5633d675e0c0 2024-03-24T10:39:56.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.810+0000 7f7593fff640 1 -- 172.21.15.23:0/757538788 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a000fb80 con 0x5633d675e0c0 2024-03-24T10:39:56.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.811+0000 7f75b2564640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75940a20c0 0x7f75940a4580 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f75a402c840 tx=0x7f75a4005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.811+0000 7f75b6ba4740 1 -- 172.21.15.23:0/757538788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d675e0c0 msgr2=0x5633d6760550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.811+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/757538788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d675e0c0 0x5633d6760550 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009cf0 tx=0x7f75a00117b0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.811+0000 7f75b6ba4740 1 -- 172.21.15.23:0/757538788 shutdown_connections 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.811+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/757538788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d675e0c0 0x5633d6760550 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.811+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/757538788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d6758f50 0x5633d675da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.811+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/757538788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758560 0x5633d6758980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.811+0000 7f75b6ba4740 1 -- 172.21.15.23:0/757538788 >> 172.21.15.23:0/757538788 conn(0x5633d6753ad0 msgr2=0x5633d6753ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.811+0000 7f75b6ba4740 1 -- 172.21.15.23:0/757538788 shutdown_connections 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.811+0000 7f75b6ba4740 1 -- 172.21.15.23:0/757538788 wait complete. 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.811+0000 7f75b6ba4740 1 -- start start 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.811+0000 7f75b6ba4740 1 Processor -- start 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b6ba4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758560 0x5633d6762510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b6ba4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758f50 0x5633d6762a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b6ba4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d675e0c0 0x5633d6769a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b6ba4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5633d6664d50 con 0x5633d6758f50 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b6ba4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633d6393310 con 0x5633d6758560 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b6ba4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5633d66458f0 con 0x5633d675e0c0 2024-03-24T10:39:56.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b6ba4740 10 client.0 _ll_register_callbacks cb 0x5633d63d0ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:39:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b1562640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758f50 0x5633d6762a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b1562640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758f50 0x5633d6762a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53424/0 (socket says 172.21.15.23:53424) 2024-03-24T10:39:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b1562640 1 -- 172.21.15.23:0/3405806568 learned_addr learned my addr 172.21.15.23:0/3405806568 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b1d63640 1 --2- 172.21.15.23:0/3405806568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758560 0x5633d6762510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b2564640 1 --2- 172.21.15.23:0/3405806568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d675e0c0 0x5633d6769a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b1562640 1 -- 172.21.15.23:0/3405806568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d675e0c0 msgr2=0x5633d6769a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b1562640 1 --2- 172.21.15.23:0/3405806568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d675e0c0 0x5633d6769a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b1562640 1 -- 172.21.15.23:0/3405806568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758560 msgr2=0x5633d6762510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b1562640 1 --2- 172.21.15.23:0/3405806568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758560 0x5633d6762510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b1562640 1 -- 172.21.15.23:0/3405806568 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a0009920 con 0x5633d6758f50 2024-03-24T10:39:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b1562640 1 --2- 172.21.15.23:0/3405806568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758f50 0x5633d6762a50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009cc0 tx=0x7f75a0009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fd30dc0f30d8826 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75917fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:39:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75917fa640 1 -- 172.21.15.23:0/3405806568 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0004250 con 0x5633d6758f50 2024-03-24T10:39:56.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75917fa640 1 -- 172.21.15.23:0/3405806568 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f75a00043b0 con 0x5633d6758f50 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3405806568 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5633d63bf6c0 con 0x5633d6758f50 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75917fa640 1 -- 172.21.15.23:0/3405806568 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f75a0012450 con 0x5633d6758f50 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b6ba4740 10 client.15222 fetch_fsmap learned FSMap version 22 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.812+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3405806568 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5633d66be510 con 0x5633d6758f50 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.813+0000 7f75917fa640 1 -- 172.21.15.23:0/3405806568 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5633d63bf6c0 con 0x5633d6758f50 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.813+0000 7f75917fa640 1 -- 172.21.15.23:0/3405806568 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f75a0022070 con 0x5633d6758f50 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.813+0000 7f75b6ba4740 10 client.15222 fetch_fsmap finished waiting for FSMap version 22 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.813+0000 7f75b6ba4740 10 client.15222 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.813+0000 7f75b6ba4740 20 client.15222 populate_metadata read hostname 'smithi023' 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.813+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/3405806568 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5633d676ecd0 0x5633d6770140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.813+0000 7f75b6ba4740 4 client.15222 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.813+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3405806568 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633d6410270 con 0x5633d676ecd0 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.813+0000 7f75b1d63640 1 --2- 172.21.15.23:0/3405806568 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5633d676ecd0 0x5633d6770140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.813+0000 7f75917fa640 10 client.15222 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:39:56.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.813+0000 7f75b1d63640 1 --2- 172.21.15.23:0/3405806568 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5633d676ecd0 0x5633d6770140 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2fbbf3a47a024f50 server_cookie=5f211819eee7adb1 in_seq=0 out_seq=0 2024-03-24T10:39:56.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75917fa640 1 -- 172.21.15.23:0/3405806568 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5633d6410270 con 0x5633d676ecd0 2024-03-24T10:39:56.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75917fa640 10 client.15222 handle_command_reply: tid=0 2024-03-24T10:39:56.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75b6ba4740 1 client.15222 shutdown 2024-03-24T10:39:56.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3405806568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758f50 msgr2=0x5633d6762a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/3405806568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758f50 0x5633d6762a50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009cc0 tx=0x7f75a0009de0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3405806568 shutdown_connections 2024-03-24T10:39:56.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/3405806568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d675e0c0 0x5633d6769a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/3405806568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758f50 0x5633d6762a50 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/3405806568 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5633d676ecd0 0x5633d6770140 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/3405806568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758560 0x5633d6762510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75917fa640 0 client.15222 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:39:56.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3405806568 >> 172.21.15.23:0/3405806568 conn(0x5633d6753ad0 msgr2=0x5633d6752ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3405806568 shutdown_connections 2024-03-24T10:39:56.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3405806568 wait complete. 2024-03-24T10:39:56.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.814+0000 7f75b6ba4740 20 client.15222 trim_cache size 0 max 16384 2024-03-24T10:39:56.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.832+0000 7f75b6ba4740 1 Processor -- start 2024-03-24T10:39:56.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b6ba4740 1 -- start start 2024-03-24T10:39:56.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b6ba4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d6758560 0x5633d6761b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b6ba4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758f50 0x5633d6769a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b6ba4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d675e0c0 0x5633d6762070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b6ba4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5633d6410270 con 0x5633d6758f50 2024-03-24T10:39:56.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b6ba4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633d63bf6c0 con 0x5633d675e0c0 2024-03-24T10:39:56.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b6ba4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5633d6678f70 con 0x5633d6758560 2024-03-24T10:39:56.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b1562640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758f50 0x5633d6769a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b1562640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758f50 0x5633d6769a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53438/0 (socket says 172.21.15.23:53438) 2024-03-24T10:39:56.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b1562640 1 -- 172.21.15.23:0/2320004925 learned_addr learned my addr 172.21.15.23:0/2320004925 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:56.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b1d63640 1 --2- 172.21.15.23:0/2320004925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d6758560 0x5633d6761b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b2564640 1 --2- 172.21.15.23:0/2320004925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d675e0c0 0x5633d6762070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b1562640 1 -- 172.21.15.23:0/2320004925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d6758560 msgr2=0x5633d6761b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b1562640 1 --2- 172.21.15.23:0/2320004925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d6758560 0x5633d6761b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b1562640 1 -- 172.21.15.23:0/2320004925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d675e0c0 msgr2=0x5633d6762070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b1562640 1 --2- 172.21.15.23:0/2320004925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d675e0c0 0x5633d6762070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b1562640 1 -- 172.21.15.23:0/2320004925 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633d665aa80 con 0x5633d6758f50 2024-03-24T10:39:56.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75b1562640 1 --2- 172.21.15.23:0/2320004925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758f50 0x5633d6769a50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f75a0012f50 tx=0x7f75a0009bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72d3f197a61028b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75937fe640 1 -- 172.21.15.23:0/2320004925 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a000fd50 con 0x5633d6758f50 2024-03-24T10:39:56.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75937fe640 1 -- 172.21.15.23:0/2320004925 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a000f650 con 0x5633d6758f50 2024-03-24T10:39:56.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.833+0000 7f75937fe640 1 -- 172.21.15.23:0/2320004925 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0003c20 con 0x5633d6758f50 2024-03-24T10:39:56.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.834+0000 7f75b6ba4740 1 -- 172.21.15.23:0/2320004925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758f50 msgr2=0x5633d6769a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.834+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/2320004925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758f50 0x5633d6769a50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f75a0012f50 tx=0x7f75a0009bc0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.834+0000 7f75b6ba4740 1 -- 172.21.15.23:0/2320004925 shutdown_connections 2024-03-24T10:39:56.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.834+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/2320004925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d675e0c0 0x5633d6762070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.834+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/2320004925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758f50 0x5633d6769a50 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.834+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/2320004925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d6758560 0x5633d6761b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.834+0000 7f75b6ba4740 1 -- 172.21.15.23:0/2320004925 >> 172.21.15.23:0/2320004925 conn(0x5633d6753ad0 msgr2=0x5633d6753eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.834+0000 7f75b6ba4740 1 -- 172.21.15.23:0/2320004925 shutdown_connections 2024-03-24T10:39:56.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.834+0000 7f75b6ba4740 1 -- 172.21.15.23:0/2320004925 wait complete. 2024-03-24T10:39:56.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.834+0000 7f75b6ba4740 1 -- start start 2024-03-24T10:39:56.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.834+0000 7f75b6ba4740 1 Processor -- start 2024-03-24T10:39:56.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b6ba4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758560 0x5633d6765220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b6ba4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758f50 0x5633d6765760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b6ba4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d675e0c0 0x5633d6768670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b6ba4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5633d6768bb0 con 0x5633d6758560 2024-03-24T10:39:56.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b6ba4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633d6768d20 con 0x5633d6758f50 2024-03-24T10:39:56.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b6ba4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5633d6768ee0 con 0x5633d675e0c0 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b6ba4740 10 client.0 _ll_register_callbacks cb 0x7f75a0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b1d63640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758560 0x5633d6765220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b1d63640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758560 0x5633d6765220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53446/0 (socket says 172.21.15.23:53446) 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b1d63640 1 -- 172.21.15.23:0/3955442634 learned_addr learned my addr 172.21.15.23:0/3955442634 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b1562640 1 --2- 172.21.15.23:0/3955442634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758f50 0x5633d6765760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b2564640 1 --2- 172.21.15.23:0/3955442634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d675e0c0 0x5633d6768670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b1d63640 1 -- 172.21.15.23:0/3955442634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d675e0c0 msgr2=0x5633d6768670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b1d63640 1 --2- 172.21.15.23:0/3955442634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d675e0c0 0x5633d6768670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b1d63640 1 -- 172.21.15.23:0/3955442634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758f50 msgr2=0x5633d6765760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b1d63640 1 --2- 172.21.15.23:0/3955442634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758f50 0x5633d6765760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b1d63640 1 -- 172.21.15.23:0/3955442634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633d66be510 con 0x5633d6758560 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b1d63640 1 --2- 172.21.15.23:0/3955442634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758560 0x5633d6765220 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f759c097b40 tx=0x7f759c0974d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf719eeefb2f0fa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75917fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75917fa640 1 -- 172.21.15.23:0/3955442634 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c09acb0 con 0x5633d6758560 2024-03-24T10:39:56.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75917fa640 1 -- 172.21.15.23:0/3955442634 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f759c003e60 con 0x5633d6758560 2024-03-24T10:39:56.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.835+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3955442634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75a0012a90 con 0x5633d6758560 2024-03-24T10:39:56.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.836+0000 7f75917fa640 1 -- 172.21.15.23:0/3955442634 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f759c0b3ea0 con 0x5633d6758560 2024-03-24T10:39:56.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.836+0000 7f75b6ba4740 10 client.15228 fetch_fsmap learned FSMap version 22 2024-03-24T10:39:56.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.836+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3955442634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5633d66807e0 con 0x5633d6758560 2024-03-24T10:39:56.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.836+0000 7f75917fa640 1 -- 172.21.15.23:0/3955442634 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f759c0b7080 con 0x5633d6758560 2024-03-24T10:39:56.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.836+0000 7f75917fa640 1 -- 172.21.15.23:0/3955442634 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f759c01f060 con 0x5633d6758560 2024-03-24T10:39:56.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.836+0000 7f75b6ba4740 10 client.15228 fetch_fsmap finished waiting for FSMap version 22 2024-03-24T10:39:56.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.836+0000 7f75b6ba4740 10 client.15228 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-24T10:39:56.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.836+0000 7f75b6ba4740 20 client.15228 populate_metadata read hostname 'smithi023' 2024-03-24T10:39:56.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.836+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/3955442634 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5633d67d92b0 0x5633d67d7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.836+0000 7f75b6ba4740 4 client.15228 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T10:39:56.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.836+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3955442634 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "status"}) -- 0x5633d63fc610 con 0x5633d67d92b0 2024-03-24T10:39:56.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.836+0000 7f75b1562640 1 --2- 172.21.15.23:0/3955442634 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5633d67d92b0 0x5633d67d7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.836+0000 7f75b1562640 1 --2- 172.21.15.23:0/3955442634 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5633d67d92b0 0x5633d67d7fc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3c6447b645afdf80 server_cookie=23259396e363545a in_seq=0 out_seq=0 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.836+0000 7f75917fa640 10 client.15228 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75917fa640 1 -- 172.21.15.23:0/3955442634 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5633d63fc610 con 0x5633d67d92b0 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75917fa640 10 client.15228 handle_command_reply: tid=0 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b6ba4740 1 client.15228 shutdown 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3955442634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758560 msgr2=0x5633d6765220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/3955442634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758560 0x5633d6765220 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f759c097b40 tx=0x7f759c0974d0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3955442634 shutdown_connections 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/3955442634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d675e0c0 0x5633d6768670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/3955442634 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5633d67d92b0 0x5633d67d7fc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/3955442634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d6758f50 0x5633d6765760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b6ba4740 1 --2- 172.21.15.23:0/3955442634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633d6758560 0x5633d6765220 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75917fa640 0 client.15228 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3955442634 >> 172.21.15.23:0/3955442634 conn(0x5633d6753ad0 msgr2=0x5633d6756d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3955442634 shutdown_connections 2024-03-24T10:39:56.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b6ba4740 1 -- 172.21.15.23:0/3955442634 wait complete. 2024-03-24T10:39:56.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b6ba4740 20 client.15228 trim_cache size 0 max 16384 2024-03-24T10:39:56.846 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:39:56.846 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "4f8fb8f6-e9c9-11ee-95cd-87774f69a715", 2024-03-24T10:39:56.846 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 2, 2024-03-24T10:39:56.846 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24503, 2024-03-24T10:39:56.846 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-24T10:39:56.846 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-24T10:39:56.846 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-24T10:39:56.846 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 101.93747045400001, 2024-03-24T10:39:56.846 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 22, 2024-03-24T10:39:56.847 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 87, 2024-03-24T10:39:56.847 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 85, 2024-03-24T10:39:56.847 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 122.91504917 2024-03-24T10:39:56.847 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:39:56.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b3fc7640 1 -- 172.21.15.23:0/3952341110 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75940a20c0 msgr2=0x7f75940a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b3fc7640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75940a20c0 0x7f75940a4580 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f75a402c840 tx=0x7f75a4005a20 comp rx=0 tx=0).stop 2024-03-24T10:39:56.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b3fc7640 1 -- 172.21.15.23:0/3952341110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ac100610 msgr2=0x7f75ac069400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b3fc7640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ac100610 0x7f75ac069400 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f759c00eb70 tx=0x7f759c00c6a0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.837+0000 7f75b3fc7640 1 -- 172.21.15.23:0/3952341110 shutdown_connections 2024-03-24T10:39:56.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.838+0000 7f75b3fc7640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f75940a20c0 0x7f75940a4580 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.838+0000 7f75b3fc7640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac1021c0 0x7f75ac19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.838+0000 7f75b3fc7640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75ac101800 0x7f75ac19c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.838+0000 7f75b3fc7640 1 --2- 172.21.15.23:0/3952341110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ac100610 0x7f75ac069400 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.838+0000 7f75b3fc7640 1 -- 172.21.15.23:0/3952341110 >> 172.21.15.23:0/3952341110 conn(0x7f75ac0fbba0 msgr2=0x7f75ac104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.838+0000 7f75b3fc7640 1 -- 172.21.15.23:0/3952341110 shutdown_connections 2024-03-24T10:39:56.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.838+0000 7f75b3fc7640 1 -- 172.21.15.23:0/3952341110 wait complete. 2024-03-24T10:39:56.849 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-24T10:39:56.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 -- 172.21.15.23:0/4056117227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0600103380 msgr2=0x7f0600109c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 --2- 172.21.15.23:0/4056117227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0600103380 0x7f0600109c10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009ca0 tx=0x7f05f802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 -- 172.21.15.23:0/4056117227 shutdown_connections 2024-03-24T10:39:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 --2- 172.21.15.23:0/4056117227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0600103380 0x7f0600109c10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 --2- 172.21.15.23:0/4056117227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06001029c0 0x7f0600102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 --2- 172.21.15.23:0/4056117227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06000ff9f0 0x7f06000ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 -- 172.21.15.23:0/4056117227 >> 172.21.15.23:0/4056117227 conn(0x7f06000fb320 msgr2=0x7f06000fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 -- 172.21.15.23:0/4056117227 shutdown_connections 2024-03-24T10:39:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 -- 172.21.15.23:0/4056117227 wait complete. 2024-03-24T10:39:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 Processor -- start 2024-03-24T10:39:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 -- start start 2024-03-24T10:39:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06000ff9f0 0x7f0600197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06001029c0 0x7f0600197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0600103380 0x7f060019ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f060019f220 con 0x7f06000ff9f0 2024-03-24T10:39:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f060019f390 con 0x7f0600103380 2024-03-24T10:39:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f06080aa640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f060019f550 con 0x7f06001029c0 2024-03-24T10:39:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.952+0000 7f0605e46640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06000ff9f0 0x7f0600197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f0605e46640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06000ff9f0 0x7f0600197730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53474/0 (socket says 172.21.15.23:53474) 2024-03-24T10:39:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f0605e46640 1 -- 172.21.15.23:0/2412305223 learned_addr learned my addr 172.21.15.23:0/2412305223 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f0605645640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06001029c0 0x7f0600197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f0606647640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0600103380 0x7f060019ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f0605e46640 1 -- 172.21.15.23:0/2412305223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06001029c0 msgr2=0x7f0600197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f0605e46640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06001029c0 0x7f0600197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f0605e46640 1 -- 172.21.15.23:0/2412305223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0600103380 msgr2=0x7f060019ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f0605e46640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0600103380 0x7f060019ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f0605e46640 1 -- 172.21.15.23:0/2412305223 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05fc009490 con 0x7f06000ff9f0 2024-03-24T10:39:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f0605645640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06001029c0 0x7f0600197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:56.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f0605e46640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06000ff9f0 0x7f0600197730 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00e9c0 tx=0x7f05fc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f05f6ffd640 1 -- 172.21.15.23:0/2412305223 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc00cde0 con 0x7f06000ff9f0 2024-03-24T10:39:56.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f05f6ffd640 1 -- 172.21.15.23:0/2412305223 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05fc010910 con 0x7f06000ff9f0 2024-03-24T10:39:56.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f06080aa640 1 -- 172.21.15.23:0/2412305223 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05f8009900 con 0x7f06000ff9f0 2024-03-24T10:39:56.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f05f6ffd640 1 -- 172.21.15.23:0/2412305223 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc018a10 con 0x7f06000ff9f0 2024-03-24T10:39:56.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.953+0000 7f06080aa640 1 -- 172.21.15.23:0/2412305223 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f060019fc00 con 0x7f06000ff9f0 2024-03-24T10:39:56.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.955+0000 7f05f6ffd640 1 -- 172.21.15.23:0/2412305223 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f05fc010a80 con 0x7f06000ff9f0 2024-03-24T10:39:56.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f060ac85740 1 Processor -- start 2024-03-24T10:39:56.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f060ac85740 1 -- start start 2024-03-24T10:39:56.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f060ac85740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0560 0x55ca4afa0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f060ac85740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0f50 0x55ca4afa5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f060ac85740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa60c0 0x55ca4afa8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f060ac85740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ca4af0f180 con 0x55ca4afa0560 2024-03-24T10:39:56.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f060ac85740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ca4af0c6b0 con 0x55ca4afa0f50 2024-03-24T10:39:56.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f060ac85740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ca4af07e90 con 0x55ca4afa60c0 2024-03-24T10:39:56.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f05f6ffd640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f05ec0a2140 0x7f05ec0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f0605645640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa60c0 0x55ca4afa8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f0605e46640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0f50 0x55ca4afa5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f0606647640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0560 0x55ca4afa0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f0605645640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa60c0 0x55ca4afa8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:54304/0 (socket says 172.21.15.23:54304) 2024-03-24T10:39:56.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f0605645640 1 -- 172.21.15.23:0/782962587 learned_addr learned my addr 172.21.15.23:0/782962587 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:56.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f0606647640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0560 0x55ca4afa0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53482/0 (socket says 172.21.15.23:53482) 2024-03-24T10:39:56.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f0606647640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f05ec0a2140 0x7f05ec0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.956+0000 7f05f6ffd640 1 -- 172.21.15.23:0/2412305223 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f05fc014070 con 0x7f06000ff9f0 2024-03-24T10:39:56.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.957+0000 7f0605645640 1 -- 172.21.15.23:0/782962587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0f50 msgr2=0x55ca4afa5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.957+0000 7f0605645640 1 --2- 172.21.15.23:0/782962587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0f50 0x55ca4afa5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.957+0000 7f0605645640 1 -- 172.21.15.23:0/782962587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0560 msgr2=0x55ca4afa0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.957+0000 7f0605645640 1 --2- 172.21.15.23:0/782962587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0560 0x55ca4afa0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.957+0000 7f0605645640 1 -- 172.21.15.23:0/782962587 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca4aec4d80 con 0x55ca4afa60c0 2024-03-24T10:39:56.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.957+0000 7f0606647640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f05ec0a2140 0x7f05ec0a4600 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f05f8005e30 tx=0x7f05f80326b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.957+0000 7f0605645640 1 --2- 172.21.15.23:0/782962587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa60c0 0x55ca4afa8550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f05f00114f0 tx=0x7f05f00118f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2de9240992d5f9a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.957+0000 7f05ebfff640 1 -- 172.21.15.23:0/782962587 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f0004030 con 0x55ca4afa60c0 2024-03-24T10:39:56.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.957+0000 7f05ebfff640 1 -- 172.21.15.23:0/782962587 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05f00035f0 con 0x55ca4afa60c0 2024-03-24T10:39:56.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.957+0000 7f05ebfff640 1 -- 172.21.15.23:0/782962587 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f000f850 con 0x55ca4afa60c0 2024-03-24T10:39:56.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 -- 172.21.15.23:0/782962587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa60c0 msgr2=0x55ca4afa8550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 --2- 172.21.15.23:0/782962587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa60c0 0x55ca4afa8550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f05f00114f0 tx=0x7f05f00118f0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 -- 172.21.15.23:0/782962587 shutdown_connections 2024-03-24T10:39:56.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 --2- 172.21.15.23:0/782962587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa60c0 0x55ca4afa8550 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 --2- 172.21.15.23:0/782962587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0f50 0x55ca4afa5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 --2- 172.21.15.23:0/782962587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0560 0x55ca4afa0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 -- 172.21.15.23:0/782962587 >> 172.21.15.23:0/782962587 conn(0x55ca4af9bad0 msgr2=0x55ca4af9bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 -- 172.21.15.23:0/782962587 shutdown_connections 2024-03-24T10:39:56.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 -- 172.21.15.23:0/782962587 wait complete. 2024-03-24T10:39:56.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 -- start start 2024-03-24T10:39:56.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 Processor -- start 2024-03-24T10:39:56.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0560 0x55ca4afaa510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa0f50 0x55ca4afaaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa60c0 0x55ca4afb1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ca4aeacd50 con 0x55ca4afa60c0 2024-03-24T10:39:56.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.958+0000 7f060ac85740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ca4abdb310 con 0x55ca4afa0560 2024-03-24T10:39:56.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f060ac85740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ca4ae8d8f0 con 0x55ca4afa0f50 2024-03-24T10:39:56.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f060ac85740 10 client.0 _ll_register_callbacks cb 0x55ca4ac18ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:39:56.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f0606647640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa60c0 0x55ca4afb1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f0605645640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa0f50 0x55ca4afaaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f0606647640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa60c0 0x55ca4afb1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53496/0 (socket says 172.21.15.23:53496) 2024-03-24T10:39:56.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f0606647640 1 -- 172.21.15.23:0/875187040 learned_addr learned my addr 172.21.15.23:0/875187040 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f0605e46640 1 --2- 172.21.15.23:0/875187040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0560 0x55ca4afaa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f0606647640 1 -- 172.21.15.23:0/875187040 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa0f50 msgr2=0x55ca4afaaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f0606647640 1 --2- 172.21.15.23:0/875187040 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa0f50 0x55ca4afaaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f0606647640 1 -- 172.21.15.23:0/875187040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0560 msgr2=0x55ca4afaa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f0606647640 1 --2- 172.21.15.23:0/875187040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0560 0x55ca4afaa510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f0606647640 1 -- 172.21.15.23:0/875187040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05f0009cc0 con 0x55ca4afa60c0 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f0605645640 1 --2- 172.21.15.23:0/875187040 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa0f50 0x55ca4afaaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f0606647640 1 --2- 172.21.15.23:0/875187040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa60c0 0x55ca4afb1a90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009ca0 tx=0x7f05f80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591a9d1660154e00 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f05e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f05e97fa640 1 -- 172.21.15.23:0/875187040 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f8003b80 con 0x55ca4afa60c0 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f05e97fa640 1 -- 172.21.15.23:0/875187040 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f05f8003ce0 con 0x55ca4afa60c0 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f060ac85740 1 -- 172.21.15.23:0/875187040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ca4ac076c0 con 0x55ca4afa60c0 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f05e97fa640 1 -- 172.21.15.23:0/875187040 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f05f8003ea0 con 0x55ca4afa60c0 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f060ac85740 10 client.15240 fetch_fsmap learned FSMap version 22 2024-03-24T10:39:56.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.959+0000 7f060ac85740 1 -- 172.21.15.23:0/875187040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ca4af06510 con 0x55ca4afa60c0 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.960+0000 7f05e97fa640 1 -- 172.21.15.23:0/875187040 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ca4ac076c0 con 0x55ca4afa60c0 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.960+0000 7f05e97fa640 1 -- 172.21.15.23:0/875187040 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f05f8050460 con 0x55ca4afa60c0 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.960+0000 7f060ac85740 10 client.15240 fetch_fsmap finished waiting for FSMap version 22 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.960+0000 7f060ac85740 10 client.15240 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.960+0000 7f060ac85740 20 client.15240 populate_metadata read hostname 'smithi023' 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.960+0000 7f060ac85740 1 --2- 172.21.15.23:0/875187040 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ca4afb6e50 0x55ca4afb82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.960+0000 7f060ac85740 4 client.15240 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.960+0000 7f060ac85740 1 -- 172.21.15.23:0/875187040 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca4ac58270 con 0x55ca4afb6e50 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.960+0000 7f0605e46640 1 --2- 172.21.15.23:0/875187040 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ca4afb6e50 0x55ca4afb82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.960+0000 7f0605e46640 1 --2- 172.21.15.23:0/875187040 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ca4afb6e50 0x55ca4afb82c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f61142b7f6253fba server_cookie=234988f5d457def5 in_seq=0 out_seq=0 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.960+0000 7f05e97fa640 10 client.15240 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f05e97fa640 1 -- 172.21.15.23:0/875187040 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ca4ac58270 con 0x55ca4afb6e50 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f05e97fa640 10 client.15240 handle_command_reply: tid=0 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f060ac85740 1 client.15240 shutdown 2024-03-24T10:39:56.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f060ac85740 1 -- 172.21.15.23:0/875187040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa60c0 msgr2=0x55ca4afb1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f060ac85740 1 --2- 172.21.15.23:0/875187040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa60c0 0x55ca4afb1a90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009ca0 tx=0x7f05f80095c0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f060ac85740 1 -- 172.21.15.23:0/875187040 shutdown_connections 2024-03-24T10:39:56.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f060ac85740 1 --2- 172.21.15.23:0/875187040 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ca4afb6e50 0x55ca4afb82c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f060ac85740 1 --2- 172.21.15.23:0/875187040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa60c0 0x55ca4afb1a90 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f060ac85740 1 --2- 172.21.15.23:0/875187040 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa0f50 0x55ca4afaaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f05e97fa640 0 client.15240 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:39:56.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f060ac85740 1 --2- 172.21.15.23:0/875187040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0560 0x55ca4afaa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f060ac85740 1 -- 172.21.15.23:0/875187040 >> 172.21.15.23:0/875187040 conn(0x55ca4af9bad0 msgr2=0x55ca4af9ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f060ac85740 1 -- 172.21.15.23:0/875187040 shutdown_connections 2024-03-24T10:39:56.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f060ac85740 1 -- 172.21.15.23:0/875187040 wait complete. 2024-03-24T10:39:56.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.961+0000 7f060ac85740 20 client.15240 trim_cache size 0 max 16384 2024-03-24T10:39:56.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f060ac85740 1 Processor -- start 2024-03-24T10:39:56.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f060ac85740 1 -- start start 2024-03-24T10:39:56.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f060ac85740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa0560 0x55ca4afa5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f060ac85740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0f50 0x55ca4afb19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f060ac85740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa60c0 0x55ca4afb5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f060ac85740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ca4afa5ea0 con 0x55ca4afa0f50 2024-03-24T10:39:56.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f060ac85740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ca4afb2350 con 0x55ca4afa60c0 2024-03-24T10:39:56.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f060ac85740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ca4af9c520 con 0x55ca4afa0560 2024-03-24T10:39:56.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f0605645640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0f50 0x55ca4afb19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f0605645640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0f50 0x55ca4afb19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53504/0 (socket says 172.21.15.23:53504) 2024-03-24T10:39:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f0605645640 1 -- 172.21.15.23:0/1255230214 learned_addr learned my addr 172.21.15.23:0/1255230214 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f0605e46640 1 --2- 172.21.15.23:0/1255230214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa0560 0x55ca4afa5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f0606647640 1 --2- 172.21.15.23:0/1255230214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa60c0 0x55ca4afb5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f0605645640 1 -- 172.21.15.23:0/1255230214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa0560 msgr2=0x55ca4afa5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f0605645640 1 --2- 172.21.15.23:0/1255230214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa0560 0x55ca4afa5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f0605645640 1 -- 172.21.15.23:0/1255230214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa60c0 msgr2=0x55ca4afb5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f0605645640 1 --2- 172.21.15.23:0/1255230214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa60c0 0x55ca4afb5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f0605645640 1 -- 172.21.15.23:0/1255230214 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca4aea2a80 con 0x55ca4afa0f50 2024-03-24T10:39:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f0605e46640 1 --2- 172.21.15.23:0/1255230214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa0560 0x55ca4afa5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.981+0000 7f0605645640 1 --2- 172.21.15.23:0/1255230214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0f50 0x55ca4afb19b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f05f00114f0 tx=0x7f05f0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc9b9ff768041cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.982+0000 7f05eb7fe640 1 -- 172.21.15.23:0/1255230214 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f0003d80 con 0x55ca4afa0f50 2024-03-24T10:39:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.982+0000 7f05eb7fe640 1 -- 172.21.15.23:0/1255230214 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05f0005590 con 0x55ca4afa0f50 2024-03-24T10:39:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.982+0000 7f05eb7fe640 1 -- 172.21.15.23:0/1255230214 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f000fb20 con 0x55ca4afa0f50 2024-03-24T10:39:56.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.982+0000 7f060ac85740 1 -- 172.21.15.23:0/1255230214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0f50 msgr2=0x55ca4afb19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.982+0000 7f060ac85740 1 --2- 172.21.15.23:0/1255230214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0f50 0x55ca4afb19b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f05f00114f0 tx=0x7f05f0009f90 comp rx=0 tx=0).stop 2024-03-24T10:39:56.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.982+0000 7f060ac85740 1 -- 172.21.15.23:0/1255230214 shutdown_connections 2024-03-24T10:39:56.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.982+0000 7f060ac85740 1 --2- 172.21.15.23:0/1255230214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa60c0 0x55ca4afb5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.982+0000 7f060ac85740 1 --2- 172.21.15.23:0/1255230214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0f50 0x55ca4afb19b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.982+0000 7f060ac85740 1 --2- 172.21.15.23:0/1255230214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa0560 0x55ca4afa5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.982+0000 7f060ac85740 1 -- 172.21.15.23:0/1255230214 >> 172.21.15.23:0/1255230214 conn(0x55ca4af9bad0 msgr2=0x55ca4af96810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.982+0000 7f060ac85740 1 -- 172.21.15.23:0/1255230214 shutdown_connections 2024-03-24T10:39:56.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.982+0000 7f060ac85740 1 -- 172.21.15.23:0/1255230214 wait complete. 2024-03-24T10:39:56.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f060ac85740 1 -- start start 2024-03-24T10:39:56.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f060ac85740 1 Processor -- start 2024-03-24T10:39:56.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f060ac85740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0560 0x55ca4afad470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f060ac85740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0f50 0x55ca4afad9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f060ac85740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa60c0 0x55ca4afb09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f060ac85740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ca4afa86e0 con 0x55ca4afa0f50 2024-03-24T10:39:56.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f060ac85740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ca4afb0f10 con 0x55ca4afa0560 2024-03-24T10:39:56.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f060ac85740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ca4afb10d0 con 0x55ca4afa60c0 2024-03-24T10:39:56.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f060ac85740 10 client.0 _ll_register_callbacks cb 0x7f05fc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:39:56.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f0605645640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0f50 0x55ca4afad9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f0605645640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0f50 0x55ca4afad9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53516/0 (socket says 172.21.15.23:53516) 2024-03-24T10:39:56.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f0605645640 1 -- 172.21.15.23:0/2027934692 learned_addr learned my addr 172.21.15.23:0/2027934692 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:56.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f0605e46640 1 --2- 172.21.15.23:0/2027934692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0560 0x55ca4afad470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f0606647640 1 --2- 172.21.15.23:0/2027934692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa60c0 0x55ca4afb09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f0605645640 1 -- 172.21.15.23:0/2027934692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa60c0 msgr2=0x55ca4afb09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f0605645640 1 --2- 172.21.15.23:0/2027934692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa60c0 0x55ca4afb09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f0605645640 1 -- 172.21.15.23:0/2027934692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0560 msgr2=0x55ca4afad470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:56.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f0605645640 1 --2- 172.21.15.23:0/2027934692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0560 0x55ca4afad470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.983+0000 7f0605645640 1 -- 172.21.15.23:0/2027934692 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05fc079e70 con 0x55ca4afa0f50 2024-03-24T10:39:56.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f0605645640 1 --2- 172.21.15.23:0/2027934692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0f50 0x55ca4afad9b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f05f00114c0 tx=0x7f05f000fb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b222bce744b7d555 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:56.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f05e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:39:56.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f05e97fa640 1 -- 172.21.15.23:0/2027934692 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f0034670 con 0x55ca4afa0f50 2024-03-24T10:39:56.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f05e97fa640 1 -- 172.21.15.23:0/2027934692 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f05f00347d0 con 0x55ca4afa0f50 2024-03-24T10:39:56.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f060ac85740 1 -- 172.21.15.23:0/2027934692 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05f00035f0 con 0x55ca4afa0f50 2024-03-24T10:39:56.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f05e97fa640 1 -- 172.21.15.23:0/2027934692 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f05f0034990 con 0x55ca4afa0f50 2024-03-24T10:39:56.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f060ac85740 10 client.15246 fetch_fsmap learned FSMap version 22 2024-03-24T10:39:56.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f060ac85740 1 -- 172.21.15.23:0/2027934692 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ca4aec4d80 con 0x55ca4afa0f50 2024-03-24T10:39:56.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f05e97fa640 1 -- 172.21.15.23:0/2027934692 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05f00035f0 con 0x55ca4afa0f50 2024-03-24T10:39:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f05e97fa640 1 -- 172.21.15.23:0/2027934692 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f05f00236d0 con 0x55ca4afa0f50 2024-03-24T10:39:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f060ac85740 10 client.15246 fetch_fsmap finished waiting for FSMap version 22 2024-03-24T10:39:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f060ac85740 10 client.15246 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T10:39:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f060ac85740 20 client.15246 populate_metadata read hostname 'smithi023' 2024-03-24T10:39:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f060ac85740 1 --2- 172.21.15.23:0/2027934692 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ca4b021010 0x55ca4b01fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f060ac85740 4 client.15246 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T10:39:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f060ac85740 1 -- 172.21.15.23:0/2027934692 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "status"}) -- 0x55ca4ac44610 con 0x55ca4b021010 2024-03-24T10:39:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.984+0000 7f0605e46640 1 --2- 172.21.15.23:0/2027934692 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ca4b021010 0x55ca4b01fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f05e97fa640 10 client.15246 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:39:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f0605e46640 1 --2- 172.21.15.23:0/2027934692 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ca4b021010 0x55ca4b01fd20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3bccda8f1d0f5c9f server_cookie=43a0073e6ff81df in_seq=0 out_seq=0 2024-03-24T10:39:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f05e97fa640 1 -- 172.21.15.23:0/2027934692 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ca4ac44610 con 0x55ca4b021010 2024-03-24T10:39:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f05e97fa640 10 client.15246 handle_command_reply: tid=0 2024-03-24T10:39:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f060ac85740 1 client.15246 shutdown 2024-03-24T10:39:56.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f060ac85740 1 -- 172.21.15.23:0/2027934692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0f50 msgr2=0x55ca4afad9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f060ac85740 1 --2- 172.21.15.23:0/2027934692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0f50 0x55ca4afad9b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f05f00114c0 tx=0x7f05f000fb50 comp rx=0 tx=0).stop 2024-03-24T10:39:56.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f060ac85740 1 -- 172.21.15.23:0/2027934692 shutdown_connections 2024-03-24T10:39:56.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f060ac85740 1 --2- 172.21.15.23:0/2027934692 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ca4b021010 0x55ca4b01fd20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f060ac85740 1 --2- 172.21.15.23:0/2027934692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca4afa60c0 0x55ca4afb09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f060ac85740 1 --2- 172.21.15.23:0/2027934692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca4afa0f50 0x55ca4afad9b0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f060ac85740 1 --2- 172.21.15.23:0/2027934692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ca4afa0560 0x55ca4afad470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f05e97fa640 0 client.15246 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:39:56.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f060ac85740 1 -- 172.21.15.23:0/2027934692 >> 172.21.15.23:0/2027934692 conn(0x55ca4af9bad0 msgr2=0x55ca4afa56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f060ac85740 1 -- 172.21.15.23:0/2027934692 shutdown_connections 2024-03-24T10:39:56.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f060ac85740 1 -- 172.21.15.23:0/2027934692 wait complete. 2024-03-24T10:39:56.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.985+0000 7f060ac85740 20 client.15246 trim_cache size 0 max 16384 2024-03-24T10:39:56.993 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:39:56.993 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "4f8fb8f6-e9c9-11ee-95cd-87774f69a715", 2024-03-24T10:39:56.993 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 1, 2024-03-24T10:39:56.993 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24505, 2024-03-24T10:39:56.993 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-24T10:39:56.993 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-24T10:39:56.993 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-24T10:39:56.993 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 103.09816049299999, 2024-03-24T10:39:56.993 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 22, 2024-03-24T10:39:56.994 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 87, 2024-03-24T10:39:56.994 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 85, 2024-03-24T10:39:56.994 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 120.367766665 2024-03-24T10:39:56.994 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:39:56.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.986+0000 7f06080aa640 1 -- 172.21.15.23:0/2412305223 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f05ec0a2140 msgr2=0x7f05ec0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.986+0000 7f06080aa640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f05ec0a2140 0x7f05ec0a4600 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f05f8005e30 tx=0x7f05f80326b0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.986+0000 7f06080aa640 1 -- 172.21.15.23:0/2412305223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06000ff9f0 msgr2=0x7f0600197730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:56.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.986+0000 7f06080aa640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06000ff9f0 0x7f0600197730 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00e9c0 tx=0x7f05fc00ee80 comp rx=0 tx=0).stop 2024-03-24T10:39:56.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.986+0000 7f06080aa640 1 -- 172.21.15.23:0/2412305223 shutdown_connections 2024-03-24T10:39:56.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.986+0000 7f06080aa640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f05ec0a2140 0x7f05ec0a4600 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.986+0000 7f06080aa640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0600103380 0x7f060019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.986+0000 7f06080aa640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06001029c0 0x7f0600197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.986+0000 7f06080aa640 1 --2- 172.21.15.23:0/2412305223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06000ff9f0 0x7f0600197730 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:56.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.986+0000 7f06080aa640 1 -- 172.21.15.23:0/2412305223 >> 172.21.15.23:0/2412305223 conn(0x7f06000fb320 msgr2=0x7f0600105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:56.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.986+0000 7f06080aa640 1 -- 172.21.15.23:0/2412305223 shutdown_connections 2024-03-24T10:39:56.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:56.986+0000 7f06080aa640 1 -- 172.21.15.23:0/2412305223 wait complete. 2024-03-24T10:39:56.997 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T10:39:57.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.100+0000 7f0a3eaf1640 1 -- 172.21.15.23:0/3458989054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a381021c0 msgr2=0x7f0a38108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.100+0000 7f0a3eaf1640 1 --2- 172.21.15.23:0/3458989054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a381021c0 0x7f0a38108a50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0a30009ca0 tx=0x7f0a3002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.100+0000 7f0a3eaf1640 1 -- 172.21.15.23:0/3458989054 shutdown_connections 2024-03-24T10:39:57.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.100+0000 7f0a3eaf1640 1 --2- 172.21.15.23:0/3458989054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a381021c0 0x7f0a38108a50 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.100+0000 7f0a3eaf1640 1 --2- 172.21.15.23:0/3458989054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a38101800 0x7f0a38101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.100+0000 7f0a3eaf1640 1 --2- 172.21.15.23:0/3458989054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a38100610 0x7f0a38100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.100+0000 7f0a3eaf1640 1 -- 172.21.15.23:0/3458989054 >> 172.21.15.23:0/3458989054 conn(0x7f0a380fbba0 msgr2=0x7f0a380fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:57.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.100+0000 7f0a3eaf1640 1 -- 172.21.15.23:0/3458989054 shutdown_connections 2024-03-24T10:39:57.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.100+0000 7f0a3eaf1640 1 -- 172.21.15.23:0/3458989054 wait complete. 2024-03-24T10:39:57.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.101+0000 7f0a3eaf1640 1 Processor -- start 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.101+0000 7f0a3eaf1640 1 -- start start 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.101+0000 7f0a3eaf1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a38100610 0x7f0a38197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.101+0000 7f0a3eaf1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a38101800 0x7f0a38197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.101+0000 7f0a3eaf1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a381021c0 0x7f0a3819ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.101+0000 7f0a3eaf1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3819f260 con 0x7f0a38100610 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.101+0000 7f0a3eaf1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3819f3d0 con 0x7f0a381021c0 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.101+0000 7f0a3eaf1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3819f590 con 0x7f0a38101800 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.101+0000 7f0a3c88d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a38100610 0x7f0a38197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.101+0000 7f0a3c88d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a38100610 0x7f0a38197770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53536/0 (socket says 172.21.15.23:53536) 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.101+0000 7f0a3c88d640 1 -- 172.21.15.23:0/655207230 learned_addr learned my addr 172.21.15.23:0/655207230 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.101+0000 7f0a37fff640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a38101800 0x7f0a38197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.101+0000 7f0a3d08e640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a381021c0 0x7f0a3819ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.102+0000 7f0a3c88d640 1 -- 172.21.15.23:0/655207230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a38101800 msgr2=0x7f0a38197cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.102+0000 7f0a3c88d640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a38101800 0x7f0a38197cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.102+0000 7f0a3c88d640 1 -- 172.21.15.23:0/655207230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a381021c0 msgr2=0x7f0a3819ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.102+0000 7f0a3c88d640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a381021c0 0x7f0a3819ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.102+0000 7f0a3c88d640 1 -- 172.21.15.23:0/655207230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a30009900 con 0x7f0a38100610 2024-03-24T10:39:57.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.102+0000 7f0a3d08e640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a381021c0 0x7f0a3819ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:39:57.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.102+0000 7f0a3c88d640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a38100610 0x7f0a38197770 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c00e9c0 tx=0x7f0a2c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:57.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.102+0000 7f0a35ffb640 1 -- 172.21.15.23:0/655207230 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c0096e0 con 0x7f0a38100610 2024-03-24T10:39:57.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.102+0000 7f0a3eaf1640 1 -- 172.21.15.23:0/655207230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a3819f870 con 0x7f0a38100610 2024-03-24T10:39:57.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.102+0000 7f0a35ffb640 1 -- 172.21.15.23:0/655207230 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a2c00cd80 con 0x7f0a38100610 2024-03-24T10:39:57.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.102+0000 7f0a35ffb640 1 -- 172.21.15.23:0/655207230 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c010430 con 0x7f0a38100610 2024-03-24T10:39:57.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.102+0000 7f0a3eaf1640 1 -- 172.21.15.23:0/655207230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a3819fd80 con 0x7f0a38100610 2024-03-24T10:39:57.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.104+0000 7f0a35ffb640 1 -- 172.21.15.23:0/655207230 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f0a2c003680 con 0x7f0a38100610 2024-03-24T10:39:57.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.104+0000 7f0a417a9740 1 Processor -- start 2024-03-24T10:39:57.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.104+0000 7f0a35ffb640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0a280a20c0 0x7f0a280a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.104+0000 7f0a417a9740 1 -- start start 2024-03-24T10:39:57.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a35ffb640 1 -- 172.21.15.23:0/655207230 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0a2c014070 con 0x7f0a38100610 2024-03-24T10:39:57.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a3d08e640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0a280a20c0 0x7f0a280a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a417a9740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c4692560 0x5616c4692980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a417a9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692f50 0x5616c4697a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a417a9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c46980c0 0x5616c469a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a417a9740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5616c4601180 con 0x5616c4692560 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a417a9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5616c45fe6b0 con 0x5616c4692f50 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a417a9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5616c45f9e90 con 0x5616c46980c0 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a3c88d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c4692560 0x5616c4692980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a3c88d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c4692560 0x5616c4692980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53550/0 (socket says 172.21.15.23:53550) 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a3c88d640 1 -- 172.21.15.23:0/1658232140 learned_addr learned my addr 172.21.15.23:0/1658232140 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a37fff640 1 --2- 172.21.15.23:0/1658232140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692f50 0x5616c4697a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a3d08e640 1 --2- 172.21.15.23:0/1658232140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c46980c0 0x5616c469a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a3c88d640 1 -- 172.21.15.23:0/1658232140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c46980c0 msgr2=0x5616c469a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a3c88d640 1 --2- 172.21.15.23:0/1658232140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c46980c0 0x5616c469a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a3c88d640 1 -- 172.21.15.23:0/1658232140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692f50 msgr2=0x5616c4697a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a3c88d640 1 --2- 172.21.15.23:0/1658232140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692f50 0x5616c4697a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a3c88d640 1 -- 172.21.15.23:0/1658232140 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616c45b6d80 con 0x5616c4692560 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a3d08e640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0a280a20c0 0x7f0a280a4580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0a3002c630 tx=0x7f0a30034490 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a3c88d640 1 --2- 172.21.15.23:0/1658232140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c4692560 0x5616c4692980 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c09ada0 tx=0x7f0a2c09aa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eef06ef41a6fe4d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:57.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a22ffd640 1 -- 172.21.15.23:0/1658232140 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c09ef60 con 0x5616c4692560 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a22ffd640 1 -- 172.21.15.23:0/1658232140 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a2c09d070 con 0x5616c4692560 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.105+0000 7f0a22ffd640 1 -- 172.21.15.23:0/1658232140 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c09b070 con 0x5616c4692560 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.106+0000 7f0a417a9740 1 -- 172.21.15.23:0/1658232140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c4692560 msgr2=0x5616c4692980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.106+0000 7f0a417a9740 1 --2- 172.21.15.23:0/1658232140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c4692560 0x5616c4692980 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c09ada0 tx=0x7f0a2c09aa10 comp rx=0 tx=0).stop 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.106+0000 7f0a417a9740 1 -- 172.21.15.23:0/1658232140 shutdown_connections 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.106+0000 7f0a417a9740 1 --2- 172.21.15.23:0/1658232140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c46980c0 0x5616c469a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.106+0000 7f0a417a9740 1 --2- 172.21.15.23:0/1658232140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692f50 0x5616c4697a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.106+0000 7f0a417a9740 1 --2- 172.21.15.23:0/1658232140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c4692560 0x5616c4692980 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.106+0000 7f0a417a9740 1 -- 172.21.15.23:0/1658232140 >> 172.21.15.23:0/1658232140 conn(0x5616c468dad0 msgr2=0x5616c468ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.106+0000 7f0a417a9740 1 -- 172.21.15.23:0/1658232140 shutdown_connections 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.106+0000 7f0a417a9740 1 -- 172.21.15.23:0/1658232140 wait complete. 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.106+0000 7f0a417a9740 1 -- start start 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.106+0000 7f0a417a9740 1 Processor -- start 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a417a9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c469c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a417a9740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c4692f50 0x5616c469cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a417a9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c46980c0 0x5616c46a3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a417a9740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5616c459ed50 con 0x5616c4692f50 2024-03-24T10:39:57.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a417a9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5616c42cd310 con 0x5616c4692560 2024-03-24T10:39:57.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a417a9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5616c457f8f0 con 0x5616c46980c0 2024-03-24T10:39:57.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a417a9740 10 client.0 _ll_register_callbacks cb 0x5616c430aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:39:57.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a37fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c4692f50 0x5616c469cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a37fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c4692f50 0x5616c469cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53556/0 (socket says 172.21.15.23:53556) 2024-03-24T10:39:57.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a37fff640 1 -- 172.21.15.23:0/2329546493 learned_addr learned my addr 172.21.15.23:0/2329546493 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:57.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a3c88d640 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c469c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a3d08e640 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c46980c0 0x5616c46a3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a37fff640 1 -- 172.21.15.23:0/2329546493 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c46980c0 msgr2=0x5616c46a3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:57.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a37fff640 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c46980c0 0x5616c46a3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a37fff640 1 -- 172.21.15.23:0/2329546493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 msgr2=0x5616c469c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:57.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a37fff640 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c469c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a37fff640 1 -- 172.21.15.23:0/2329546493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a3819fd80 con 0x5616c4692f50 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a3d08e640 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c46980c0 0x5616c46a3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a3c88d640 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c469c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a37fff640 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c4692f50 0x5616c469cab0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0a24012700 tx=0x7f0a24012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b088e98b172e734 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a17fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.107+0000 7f0a17fff640 1 -- 172.21.15.23:0/2329546493 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a24017070 con 0x5616c4692f50 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.108+0000 7f0a417a9740 1 -- 172.21.15.23:0/2329546493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5616c42f96c0 con 0x5616c4692f50 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.108+0000 7f0a17fff640 1 -- 172.21.15.23:0/2329546493 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0a2400f8c0 con 0x5616c4692f50 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.108+0000 7f0a17fff640 1 -- 172.21.15.23:0/2329546493 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0a2400fa80 con 0x5616c4692f50 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.108+0000 7f0a417a9740 10 client.15258 fetch_fsmap learned FSMap version 22 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.108+0000 7f0a417a9740 1 -- 172.21.15.23:0/2329546493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5616c45f8510 con 0x5616c4692f50 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.108+0000 7f0a17fff640 1 -- 172.21.15.23:0/2329546493 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5616c42f96c0 con 0x5616c4692f50 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.108+0000 7f0a17fff640 1 -- 172.21.15.23:0/2329546493 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f0a24024020 con 0x5616c4692f50 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.108+0000 7f0a417a9740 10 client.15258 fetch_fsmap finished waiting for FSMap version 22 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.108+0000 7f0a417a9740 10 client.15258 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T10:39:57.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.108+0000 7f0a417a9740 20 client.15258 populate_metadata read hostname 'smithi023' 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.108+0000 7f0a417a9740 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616c46a8eb0 0x5616c46aa320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.108+0000 7f0a417a9740 4 client.15258 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.108+0000 7f0a417a9740 1 -- 172.21.15.23:0/2329546493 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616c434a270 con 0x5616c46a8eb0 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.108+0000 7f0a3c88d640 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616c46a8eb0 0x5616c46aa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.109+0000 7f0a17fff640 10 client.15258 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.109+0000 7f0a3c88d640 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616c46a8eb0 0x5616c46aa320 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=172728383ae6e33c server_cookie=1371754d2dd2a4da in_seq=0 out_seq=0 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.109+0000 7f0a17fff640 1 -- 172.21.15.23:0/2329546493 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5616c434a270 con 0x5616c46a8eb0 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.109+0000 7f0a17fff640 10 client.15258 handle_command_reply: tid=0 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.109+0000 7f0a417a9740 1 client.15258 shutdown 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.109+0000 7f0a417a9740 1 -- 172.21.15.23:0/2329546493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c4692f50 msgr2=0x5616c469cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.109+0000 7f0a417a9740 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c4692f50 0x5616c469cab0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0a24012700 tx=0x7f0a24012bc0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.109+0000 7f0a417a9740 1 -- 172.21.15.23:0/2329546493 shutdown_connections 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.109+0000 7f0a417a9740 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616c46a8eb0 0x5616c46aa320 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.109+0000 7f0a417a9740 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c46980c0 0x5616c46a3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.109+0000 7f0a417a9740 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c4692f50 0x5616c469cab0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.109+0000 7f0a17fff640 0 client.15258 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:39:57.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.109+0000 7f0a417a9740 1 --2- 172.21.15.23:0/2329546493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c469c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.109+0000 7f0a417a9740 1 -- 172.21.15.23:0/2329546493 >> 172.21.15.23:0/2329546493 conn(0x5616c468dad0 msgr2=0x5616c468cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:57.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.110+0000 7f0a417a9740 1 -- 172.21.15.23:0/2329546493 shutdown_connections 2024-03-24T10:39:57.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.110+0000 7f0a417a9740 1 -- 172.21.15.23:0/2329546493 wait complete. 2024-03-24T10:39:57.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.110+0000 7f0a417a9740 20 client.15258 trim_cache size 0 max 16384 2024-03-24T10:39:57.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a417a9740 1 Processor -- start 2024-03-24T10:39:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a417a9740 1 -- start start 2024-03-24T10:39:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a417a9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c469a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a417a9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c4692f50 0x5616c46a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a417a9740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c46980c0 0x5616c46937e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a417a9740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5616c46a97b0 con 0x5616c46980c0 2024-03-24T10:39:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a417a9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5616c46a9290 con 0x5616c4692560 2024-03-24T10:39:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a417a9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5616c434a270 con 0x5616c4692f50 2024-03-24T10:39:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a3d08e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c46980c0 0x5616c46937e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a3d08e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c46980c0 0x5616c46937e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53558/0 (socket says 172.21.15.23:53558) 2024-03-24T10:39:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a3d08e640 1 -- 172.21.15.23:0/1789069070 learned_addr learned my addr 172.21.15.23:0/1789069070 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a3c88d640 1 --2- 172.21.15.23:0/1789069070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c469a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a37fff640 1 --2- 172.21.15.23:0/1789069070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c4692f50 0x5616c46a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a3d08e640 1 -- 172.21.15.23:0/1789069070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c4692f50 msgr2=0x5616c46a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a3d08e640 1 --2- 172.21.15.23:0/1789069070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c4692f50 0x5616c46a3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a3d08e640 1 -- 172.21.15.23:0/1789069070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 msgr2=0x5616c469a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a3d08e640 1 --2- 172.21.15.23:0/1789069070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c469a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a3d08e640 1 -- 172.21.15.23:0/1789069070 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616c4594a80 con 0x5616c46980c0 2024-03-24T10:39:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a3d08e640 1 --2- 172.21.15.23:0/1789069070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c46980c0 0x5616c46937e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0a3002c700 tx=0x7f0a30045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f04cc2d8bdf012e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.128+0000 7f0a227fc640 1 -- 172.21.15.23:0/1789069070 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a30034c90 con 0x5616c46980c0 2024-03-24T10:39:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.129+0000 7f0a227fc640 1 -- 172.21.15.23:0/1789069070 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a30003e50 con 0x5616c46980c0 2024-03-24T10:39:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.129+0000 7f0a227fc640 1 -- 172.21.15.23:0/1789069070 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a30041440 con 0x5616c46980c0 2024-03-24T10:39:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.129+0000 7f0a417a9740 1 -- 172.21.15.23:0/1789069070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c46980c0 msgr2=0x5616c46937e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.129+0000 7f0a417a9740 1 --2- 172.21.15.23:0/1789069070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c46980c0 0x5616c46937e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0a3002c700 tx=0x7f0a30045000 comp rx=0 tx=0).stop 2024-03-24T10:39:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.129+0000 7f0a417a9740 1 -- 172.21.15.23:0/1789069070 shutdown_connections 2024-03-24T10:39:57.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.129+0000 7f0a417a9740 1 --2- 172.21.15.23:0/1789069070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c46980c0 0x5616c46937e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.129+0000 7f0a417a9740 1 --2- 172.21.15.23:0/1789069070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c4692f50 0x5616c46a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.129+0000 7f0a417a9740 1 --2- 172.21.15.23:0/1789069070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c469a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.129+0000 7f0a417a9740 1 -- 172.21.15.23:0/1789069070 >> 172.21.15.23:0/1789069070 conn(0x5616c468dad0 msgr2=0x5616c468cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.129+0000 7f0a417a9740 1 -- 172.21.15.23:0/1789069070 shutdown_connections 2024-03-24T10:39:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.129+0000 7f0a417a9740 1 -- 172.21.15.23:0/1789069070 wait complete. 2024-03-24T10:39:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.129+0000 7f0a417a9740 1 -- start start 2024-03-24T10:39:57.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.129+0000 7f0a417a9740 1 Processor -- start 2024-03-24T10:39:57.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a417a9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c46963d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a417a9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c4692f50 0x5616c4696910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a417a9740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c46980c0 0x5616c46a2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a417a9740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5616c46a2690 con 0x5616c46980c0 2024-03-24T10:39:57.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a417a9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5616c46a2800 con 0x5616c4692560 2024-03-24T10:39:57.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a417a9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5616c46a29c0 con 0x5616c4692f50 2024-03-24T10:39:57.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a417a9740 10 client.0 _ll_register_callbacks cb 0x7f0a24009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:39:57.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a3d08e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c46980c0 0x5616c46a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a3c88d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c46963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a37fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c4692f50 0x5616c4696910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a3d08e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c46980c0 0x5616c46a2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53572/0 (socket says 172.21.15.23:53572) 2024-03-24T10:39:57.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a3c88d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c46963d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:33160/0 (socket says 172.21.15.23:33160) 2024-03-24T10:39:57.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a3d08e640 1 -- 172.21.15.23:0/3339998103 learned_addr learned my addr 172.21.15.23:0/3339998103 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:57.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a37fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c4692f50 0x5616c4696910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:54388/0 (socket says 172.21.15.23:54388) 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a3d08e640 1 -- 172.21.15.23:0/3339998103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c4692f50 msgr2=0x5616c4696910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a3d08e640 1 --2- 172.21.15.23:0/3339998103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c4692f50 0x5616c4696910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a3d08e640 1 -- 172.21.15.23:0/3339998103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 msgr2=0x5616c46963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a3d08e640 1 --2- 172.21.15.23:0/3339998103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c46963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a3d08e640 1 -- 172.21.15.23:0/3339998103 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a38109350 con 0x5616c46980c0 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a37fff640 1 --2- 172.21.15.23:0/3339998103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c4692f50 0x5616c4696910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a3c88d640 1 --2- 172.21.15.23:0/3339998103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c46963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a3d08e640 1 --2- 172.21.15.23:0/3339998103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c46980c0 0x5616c46a2150 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0a3002c6d0 tx=0x7f0a300041f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f674140c23d3be67 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.130+0000 7f0a17fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a17fff640 1 -- 172.21.15.23:0/3339998103 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a30003b70 con 0x5616c46980c0 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a17fff640 1 -- 172.21.15.23:0/3339998103 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0a30034a90 con 0x5616c46980c0 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a417a9740 1 -- 172.21.15.23:0/3339998103 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a2c09d070 con 0x5616c46980c0 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a17fff640 1 -- 172.21.15.23:0/3339998103 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0a30003cd0 con 0x5616c46980c0 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a417a9740 10 client.15264 fetch_fsmap learned FSMap version 22 2024-03-24T10:39:57.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a417a9740 1 -- 172.21.15.23:0/3339998103 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5616c45ba7e0 con 0x5616c46980c0 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a17fff640 1 -- 172.21.15.23:0/3339998103 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a2c09d070 con 0x5616c46980c0 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a17fff640 1 -- 172.21.15.23:0/3339998103 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f0a30043430 con 0x5616c46980c0 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a417a9740 10 client.15264 fetch_fsmap finished waiting for FSMap version 22 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a417a9740 10 client.15264 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a417a9740 20 client.15264 populate_metadata read hostname 'smithi023' 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a417a9740 1 --2- 172.21.15.23:0/3339998103 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616c4712a80 0x5616c4711790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a417a9740 4 client.15264 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a417a9740 1 -- 172.21.15.23:0/3339998103 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "status"}) -- 0x5616c4336610 con 0x5616c4712a80 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.131+0000 7f0a3c88d640 1 --2- 172.21.15.23:0/3339998103 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616c4712a80 0x5616c4711790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a3c88d640 1 --2- 172.21.15.23:0/3339998103 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616c4712a80 0x5616c4711790 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=237749d34b42f7ab server_cookie=ef13ff1b575697d2 in_seq=0 out_seq=0 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a17fff640 10 client.15264 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a17fff640 1 -- 172.21.15.23:0/3339998103 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5616c4336610 con 0x5616c4712a80 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a17fff640 10 client.15264 handle_command_reply: tid=0 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a417a9740 1 client.15264 shutdown 2024-03-24T10:39:57.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a417a9740 1 -- 172.21.15.23:0/3339998103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c46980c0 msgr2=0x5616c46a2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a417a9740 1 --2- 172.21.15.23:0/3339998103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c46980c0 0x5616c46a2150 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0a3002c6d0 tx=0x7f0a300041f0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a417a9740 1 -- 172.21.15.23:0/3339998103 shutdown_connections 2024-03-24T10:39:57.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a417a9740 1 --2- 172.21.15.23:0/3339998103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616c46980c0 0x5616c46a2150 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a417a9740 1 --2- 172.21.15.23:0/3339998103 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616c4712a80 0x5616c4711790 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a417a9740 1 --2- 172.21.15.23:0/3339998103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616c4692f50 0x5616c4696910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a417a9740 1 --2- 172.21.15.23:0/3339998103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616c4692560 0x5616c46963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a17fff640 0 client.15264 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:39:57.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a417a9740 1 -- 172.21.15.23:0/3339998103 >> 172.21.15.23:0/3339998103 conn(0x5616c468dad0 msgr2=0x5616c468cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:57.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a417a9740 1 -- 172.21.15.23:0/3339998103 shutdown_connections 2024-03-24T10:39:57.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a417a9740 1 -- 172.21.15.23:0/3339998103 wait complete. 2024-03-24T10:39:57.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.132+0000 7f0a417a9740 20 client.15264 trim_cache size 0 max 16384 2024-03-24T10:39:57.139 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:39:57.140 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "4f8fb8f6-e9c9-11ee-95cd-87774f69a715", 2024-03-24T10:39:57.140 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2024-03-24T10:39:57.140 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24539, 2024-03-24T10:39:57.140 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-24T10:39:57.140 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-24T10:39:57.140 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-24T10:39:57.140 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 106.26463468999999, 2024-03-24T10:39:57.140 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 22, 2024-03-24T10:39:57.140 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 87, 2024-03-24T10:39:57.140 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 85, 2024-03-24T10:39:57.140 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 112.215527446 2024-03-24T10:39:57.140 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:39:57.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.133+0000 7f0a3eaf1640 1 -- 172.21.15.23:0/655207230 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0a280a20c0 msgr2=0x7f0a280a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.133+0000 7f0a3eaf1640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0a280a20c0 0x7f0a280a4580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0a3002c630 tx=0x7f0a30034490 comp rx=0 tx=0).stop 2024-03-24T10:39:57.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.133+0000 7f0a3eaf1640 1 -- 172.21.15.23:0/655207230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a38100610 msgr2=0x7f0a38197770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.133+0000 7f0a3eaf1640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a38100610 0x7f0a38197770 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c00e9c0 tx=0x7f0a2c00ee80 comp rx=0 tx=0).stop 2024-03-24T10:39:57.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.133+0000 7f0a3eaf1640 1 -- 172.21.15.23:0/655207230 shutdown_connections 2024-03-24T10:39:57.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.133+0000 7f0a3eaf1640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f0a280a20c0 0x7f0a280a4580 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.133+0000 7f0a3eaf1640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a381021c0 0x7f0a3819ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.133+0000 7f0a3eaf1640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a38101800 0x7f0a38197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.133+0000 7f0a3eaf1640 1 --2- 172.21.15.23:0/655207230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a38100610 0x7f0a38197770 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.133+0000 7f0a3eaf1640 1 -- 172.21.15.23:0/655207230 >> 172.21.15.23:0/655207230 conn(0x7f0a380fbba0 msgr2=0x7f0a380fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:57.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.133+0000 7f0a3eaf1640 1 -- 172.21.15.23:0/655207230 shutdown_connections 2024-03-24T10:39:57.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.133+0000 7f0a3eaf1640 1 -- 172.21.15.23:0/655207230 wait complete. 2024-03-24T10:39:57.144 INFO:teuthology.run_tasks:Running task sequential... 2024-03-24T10:39:57.157 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-03-24T10:39:57.157 INFO:teuthology.task.sequential:In sequential, running task print... 2024-03-24T10:39:57.157 INFO:teuthology.task.print:Enabling mgr modules 2024-03-24T10:39:57.158 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-03-24T10:39:57.158 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T10:39:57.158 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T10:39:57.159 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-24T10:39:57.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:56 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1423139444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:39:57.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:56 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2168182447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:57.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:56 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3240903240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:57.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.260+0000 7f7f75f9a640 1 -- 172.21.15.23:0/2232891762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f70100630 msgr2=0x7f7f70100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.260+0000 7f7f75f9a640 1 --2- 172.21.15.23:0/2232891762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f70100630 0x7f7f70100a50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7f64009900 tx=0x7f7f6402bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:57.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.260+0000 7f7f75f9a640 1 -- 172.21.15.23:0/2232891762 shutdown_connections 2024-03-24T10:39:57.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.260+0000 7f7f75f9a640 1 --2- 172.21.15.23:0/2232891762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f701021e0 0x7f7f70108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.260+0000 7f7f75f9a640 1 --2- 172.21.15.23:0/2232891762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f70101820 0x7f7f70101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.260+0000 7f7f75f9a640 1 --2- 172.21.15.23:0/2232891762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f70100630 0x7f7f70100a50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.260+0000 7f7f75f9a640 1 -- 172.21.15.23:0/2232891762 >> 172.21.15.23:0/2232891762 conn(0x7f7f700fbba0 msgr2=0x7f7f700fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:57.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.260+0000 7f7f75f9a640 1 -- 172.21.15.23:0/2232891762 shutdown_connections 2024-03-24T10:39:57.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.260+0000 7f7f75f9a640 1 -- 172.21.15.23:0/2232891762 wait complete. 2024-03-24T10:39:57.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.260+0000 7f7f75f9a640 1 Processor -- start 2024-03-24T10:39:57.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f75f9a640 1 -- start start 2024-03-24T10:39:57.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f75f9a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f70100630 0x7f7f701999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f75f9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f70101820 0x7f7f70199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f75f9a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f701021e0 0x7f7f701a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f75f9a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701a1460 con 0x7f7f701021e0 2024-03-24T10:39:57.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f75f9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701a15d0 con 0x7f7f70101820 2024-03-24T10:39:57.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f75f9a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701a1790 con 0x7f7f70100630 2024-03-24T10:39:57.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f6ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f701021e0 0x7f7f701a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f6effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f70101820 0x7f7f70199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f6ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f701021e0 0x7f7f701a0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53586/0 (socket says 172.21.15.23:53586) 2024-03-24T10:39:57.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f6ffff640 1 -- 172.21.15.23:0/1523661121 learned_addr learned my addr 172.21.15.23:0/1523661121 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:57.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f6f7fe640 1 --2- 172.21.15.23:0/1523661121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f70100630 0x7f7f701999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f6ffff640 1 -- 172.21.15.23:0/1523661121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f70100630 msgr2=0x7f7f701999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f6ffff640 1 --2- 172.21.15.23:0/1523661121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f70100630 0x7f7f701999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f6ffff640 1 -- 172.21.15.23:0/1523661121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f70101820 msgr2=0x7f7f70199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f6ffff640 1 --2- 172.21.15.23:0/1523661121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f70101820 0x7f7f70199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f6ffff640 1 -- 172.21.15.23:0/1523661121 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f60009490 con 0x7f7f701021e0 2024-03-24T10:39:57.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f6ffff640 1 --2- 172.21.15.23:0/1523661121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f701021e0 0x7f7f701a0f20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7f6800eb10 tx=0x7f7f6800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:57.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.261+0000 7f7f6cff9640 1 -- 172.21.15.23:0/1523661121 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f68004b20 con 0x7f7f701021e0 2024-03-24T10:39:57.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.262+0000 7f7f6cff9640 1 -- 172.21.15.23:0/1523661121 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f68004c80 con 0x7f7f701021e0 2024-03-24T10:39:57.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.262+0000 7f7f75f9a640 1 -- 172.21.15.23:0/1523661121 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f64009560 con 0x7f7f701021e0 2024-03-24T10:39:57.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.262+0000 7f7f6cff9640 1 -- 172.21.15.23:0/1523661121 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f68010650 con 0x7f7f701021e0 2024-03-24T10:39:57.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.262+0000 7f7f75f9a640 1 -- 172.21.15.23:0/1523661121 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f701a1df0 con 0x7f7f701021e0 2024-03-24T10:39:57.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.263+0000 7f7f75f9a640 1 -- 172.21.15.23:0/1523661121 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f7010c040 con 0x7f7f701021e0 2024-03-24T10:39:57.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.264+0000 7f7f6cff9640 1 -- 172.21.15.23:0/1523661121 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f7f68004df0 con 0x7f7f701021e0 2024-03-24T10:39:57.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.264+0000 7f7f6cff9640 1 --2- 172.21.15.23:0/1523661121 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f580a2160 0x7f7f580a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:57.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.264+0000 7f7f6f7fe640 1 --2- 172.21.15.23:0/1523661121 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f580a2160 0x7f7f580a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:57.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.264+0000 7f7f6cff9640 1 -- 172.21.15.23:0/1523661121 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7f68014070 con 0x7f7f701021e0 2024-03-24T10:39:57.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.265+0000 7f7f6f7fe640 1 --2- 172.21.15.23:0/1523661121 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f580a2160 0x7f7f580a4620 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7f64005ab0 tx=0x7f7f64005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:57.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:56 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1423139444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:39:57.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:56 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2168182447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:57.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:56 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3240903240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:57.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.268+0000 7f7f6cff9640 1 -- 172.21.15.23:0/1523661121 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7f680973c0 con 0x7f7f701021e0 2024-03-24T10:39:57.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:56 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1423139444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:39:57.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:56 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2168182447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:39:57.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:56 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3240903240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:39:57.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.499+0000 7f7f75f9a640 1 -- 172.21.15.23:0/1523661121 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f7f7010c1d0 con 0x7f7f701021e0 2024-03-24T10:39:57.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.982+0000 7f7f6cff9640 1 -- 172.21.15.23:0/1523661121 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f7f6809aa80 con 0x7f7f701021e0 2024-03-24T10:39:57.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.986+0000 7f7f75f9a640 1 -- 172.21.15.23:0/1523661121 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f580a2160 msgr2=0x7f7f580a4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.986+0000 7f7f75f9a640 1 --2- 172.21.15.23:0/1523661121 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f580a2160 0x7f7f580a4620 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7f64005ab0 tx=0x7f7f64005fd0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.986+0000 7f7f75f9a640 1 -- 172.21.15.23:0/1523661121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f701021e0 msgr2=0x7f7f701a0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:57.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.986+0000 7f7f75f9a640 1 --2- 172.21.15.23:0/1523661121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f701021e0 0x7f7f701a0f20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7f6800eb10 tx=0x7f7f6800c2d0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.986+0000 7f7f75f9a640 1 -- 172.21.15.23:0/1523661121 shutdown_connections 2024-03-24T10:39:57.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.986+0000 7f7f75f9a640 1 --2- 172.21.15.23:0/1523661121 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f7f580a2160 0x7f7f580a4620 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.986+0000 7f7f75f9a640 1 --2- 172.21.15.23:0/1523661121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f701021e0 0x7f7f701a0f20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.986+0000 7f7f75f9a640 1 --2- 172.21.15.23:0/1523661121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f70101820 0x7f7f70199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.986+0000 7f7f75f9a640 1 --2- 172.21.15.23:0/1523661121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f70100630 0x7f7f701999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:57.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.986+0000 7f7f75f9a640 1 -- 172.21.15.23:0/1523661121 >> 172.21.15.23:0/1523661121 conn(0x7f7f700fbba0 msgr2=0x7f7f700fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:57.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.986+0000 7f7f75f9a640 1 -- 172.21.15.23:0/1523661121 shutdown_connections 2024-03-24T10:39:57.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:57.988+0000 7f7f75f9a640 1 -- 172.21.15.23:0/1523661121 wait complete. 2024-03-24T10:39:58.006 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-24T10:39:58.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.116+0000 7f86728cd640 1 -- 172.21.15.23:0/4274735552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c101820 msgr2=0x7f866c101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:58.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.116+0000 7f86728cd640 1 --2- 172.21.15.23:0/4274735552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c101820 0x7f866c101ca0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8660009900 tx=0x7f866002bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:58.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.117+0000 7f86728cd640 1 -- 172.21.15.23:0/4274735552 shutdown_connections 2024-03-24T10:39:58.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.117+0000 7f86728cd640 1 --2- 172.21.15.23:0/4274735552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866c1021e0 0x7f866c108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.117+0000 7f86728cd640 1 --2- 172.21.15.23:0/4274735552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c101820 0x7f866c101ca0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.117+0000 7f86728cd640 1 --2- 172.21.15.23:0/4274735552 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f866c100630 0x7f866c100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.117+0000 7f86728cd640 1 -- 172.21.15.23:0/4274735552 >> 172.21.15.23:0/4274735552 conn(0x7f866c0fbba0 msgr2=0x7f866c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:58.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.117+0000 7f86728cd640 1 -- 172.21.15.23:0/4274735552 shutdown_connections 2024-03-24T10:39:58.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.117+0000 7f86728cd640 1 -- 172.21.15.23:0/4274735552 wait complete. 2024-03-24T10:39:58.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.117+0000 7f86728cd640 1 Processor -- start 2024-03-24T10:39:58.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.117+0000 7f86728cd640 1 -- start start 2024-03-24T10:39:58.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.117+0000 7f86728cd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c100630 0x7f866c196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:58.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.117+0000 7f86728cd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f866c101820 0x7f866c196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:58.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f86728cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866c1021e0 0x7f866c19dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:58.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f86728cd640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f866c19e150 con 0x7f866c100630 2024-03-24T10:39:58.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f86728cd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f866c19e2c0 con 0x7f866c1021e0 2024-03-24T10:39:58.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f86728cd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f866c19e480 con 0x7f866c101820 2024-03-24T10:39:58.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f866bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c100630 0x7f866c196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:58.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f866bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c100630 0x7f866c196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38570/0 (socket says 172.21.15.23:38570) 2024-03-24T10:39:58.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f866bfff640 1 -- 172.21.15.23:0/48706298 learned_addr learned my addr 172.21.15.23:0/48706298 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:58.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f8670e6a640 1 --2- 172.21.15.23:0/48706298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866c1021e0 0x7f866c19dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:58.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f866b7fe640 1 --2- 172.21.15.23:0/48706298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f866c101820 0x7f866c196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:58.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f866bfff640 1 -- 172.21.15.23:0/48706298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f866c101820 msgr2=0x7f866c196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:58.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f866bfff640 1 --2- 172.21.15.23:0/48706298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f866c101820 0x7f866c196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f866bfff640 1 -- 172.21.15.23:0/48706298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866c1021e0 msgr2=0x7f866c19dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:58.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f866bfff640 1 --2- 172.21.15.23:0/48706298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866c1021e0 0x7f866c19dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f866bfff640 1 -- 172.21.15.23:0/48706298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8660009560 con 0x7f866c100630 2024-03-24T10:39:58.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f866bfff640 1 --2- 172.21.15.23:0/48706298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c100630 0x7f866c196660 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f865c00e9c0 tx=0x7f865c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:58.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f86697fa640 1 -- 172.21.15.23:0/48706298 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c0096e0 con 0x7f866c100630 2024-03-24T10:39:58.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f86697fa640 1 -- 172.21.15.23:0/48706298 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f865c00cd80 con 0x7f866c100630 2024-03-24T10:39:58.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f86728cd640 1 -- 172.21.15.23:0/48706298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f866c19e760 con 0x7f866c100630 2024-03-24T10:39:58.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f86697fa640 1 -- 172.21.15.23:0/48706298 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c010430 con 0x7f866c100630 2024-03-24T10:39:58.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.118+0000 7f86728cd640 1 -- 172.21.15.23:0/48706298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f866c19ec40 con 0x7f866c100630 2024-03-24T10:39:58.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.119+0000 7f86728cd640 1 -- 172.21.15.23:0/48706298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f866c1078f0 con 0x7f866c100630 2024-03-24T10:39:58.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.121+0000 7f86697fa640 1 -- 172.21.15.23:0/48706298 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) ==== 136773+0+0 (secure 0 0 0) 0x7f865c003680 con 0x7f866c100630 2024-03-24T10:39:58.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.121+0000 7f86697fa640 1 --2- 172.21.15.23:0/48706298 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f86540a2230 0x7f86540a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:58.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.121+0000 7f866b7fe640 1 -- 172.21.15.23:0/48706298 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f86540a2230 msgr2=0x7f86540a46f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2350021632 2024-03-24T10:39:58.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.121+0000 7f86697fa640 1 -- 172.21.15.23:0/48706298 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f865c014070 con 0x7f866c100630 2024-03-24T10:39:58.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.121+0000 7f866b7fe640 1 --2- 172.21.15.23:0/48706298 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f86540a2230 0x7f86540a46f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:39:58.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.129+0000 7f86697fa640 1 -- 172.21.15.23:0/48706298 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f865c097080 con 0x7f866c100630 2024-03-24T10:39:58.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:57 smithi117 ceph-mon[36368]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-24T10:39:58.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:57 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1523661121' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T10:39:58.253 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:39:58 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: ignoring --setuser ceph since I am not root 2024-03-24T10:39:58.253 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:39:58 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: ignoring --setgroup ceph since I am not root 2024-03-24T10:39:58.254 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:39:58 smithi117 ceph-mgr[38309]: -- 172.21.15.117:0/2808685422 <== mon.1 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563d6c4f61a0 con 0x563d6c4cd000 2024-03-24T10:39:58.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:57 smithi116 ceph-mon[36248]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-24T10:39:58.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:57 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1523661121' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T10:39:58.268 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:39:58 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: ignoring --setuser ceph since I am not root 2024-03-24T10:39:58.268 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:39:58 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: ignoring --setgroup ceph since I am not root 2024-03-24T10:39:58.268 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:39:58 smithi116 ceph-mgr[37647]: -- 172.21.15.116:0/2825937284 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ce2f5aa1a0 con 0x55ce2f580c00 2024-03-24T10:39:58.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.313+0000 7f86728cd640 1 -- 172.21.15.23:0/48706298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f866c10c040 con 0x7f866c100630 2024-03-24T10:39:58.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.318+0000 7f86697fa640 1 -- 172.21.15.23:0/48706298 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) ==== 138+0+0 (secure 0 0 0) 0x7f865c09a740 con 0x7f866c100630 2024-03-24T10:39:58.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.320+0000 7f86728cd640 1 -- 172.21.15.23:0/48706298 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f86540a2230 msgr2=0x7f86540a46f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:39:58.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.320+0000 7f86728cd640 1 --2- 172.21.15.23:0/48706298 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f86540a2230 0x7f86540a46f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.320+0000 7f86728cd640 1 -- 172.21.15.23:0/48706298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c100630 msgr2=0x7f866c196660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:58.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.320+0000 7f86728cd640 1 --2- 172.21.15.23:0/48706298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c100630 0x7f866c196660 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f865c00e9c0 tx=0x7f865c00ee80 comp rx=0 tx=0).stop 2024-03-24T10:39:58.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.321+0000 7f86728cd640 1 -- 172.21.15.23:0/48706298 shutdown_connections 2024-03-24T10:39:58.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.321+0000 7f86728cd640 1 --2- 172.21.15.23:0/48706298 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f86540a2230 0x7f86540a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.321+0000 7f86728cd640 1 --2- 172.21.15.23:0/48706298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866c1021e0 0x7f866c19dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.321+0000 7f86728cd640 1 --2- 172.21.15.23:0/48706298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f866c101820 0x7f866c196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.321+0000 7f86728cd640 1 --2- 172.21.15.23:0/48706298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c100630 0x7f866c196660 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.321+0000 7f86728cd640 1 -- 172.21.15.23:0/48706298 >> 172.21.15.23:0/48706298 conn(0x7f866c0fbba0 msgr2=0x7f866c104ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:58.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.321+0000 7f86728cd640 1 -- 172.21.15.23:0/48706298 shutdown_connections 2024-03-24T10:39:58.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.321+0000 7f86728cd640 1 -- 172.21.15.23:0/48706298 wait complete. 2024-03-24T10:39:58.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:57 smithi023 ceph-mon[32466]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-24T10:39:58.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:57 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1523661121' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T10:39:58.331 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:39:58 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: ignoring --setuser ceph since I am not root 2024-03-24T10:39:58.331 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:39:58 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: ignoring --setgroup ceph since I am not root 2024-03-24T10:39:58.332 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-24T10:39:58.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.438+0000 7f61b7943640 1 -- 172.21.15.23:0/3840171170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b0100630 msgr2=0x7f61b0100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:58.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.438+0000 7f61b7943640 1 --2- 172.21.15.23:0/3840171170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b0100630 0x7f61b0100a50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009900 tx=0x7f61ac02bc30 comp rx=0 tx=0).stop 2024-03-24T10:39:58.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.438+0000 7f61b7943640 1 -- 172.21.15.23:0/3840171170 shutdown_connections 2024-03-24T10:39:58.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.438+0000 7f61b7943640 1 --2- 172.21.15.23:0/3840171170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b01021e0 0x7f61b0108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.438+0000 7f61b7943640 1 --2- 172.21.15.23:0/3840171170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61b0101820 0x7f61b0101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.438+0000 7f61b7943640 1 --2- 172.21.15.23:0/3840171170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b0100630 0x7f61b0100a50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.438+0000 7f61b7943640 1 -- 172.21.15.23:0/3840171170 >> 172.21.15.23:0/3840171170 conn(0x7f61b00fbba0 msgr2=0x7f61b00fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:58.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.438+0000 7f61b7943640 1 -- 172.21.15.23:0/3840171170 shutdown_connections 2024-03-24T10:39:58.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.438+0000 7f61b7943640 1 -- 172.21.15.23:0/3840171170 wait complete. 2024-03-24T10:39:58.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.438+0000 7f61b7943640 1 Processor -- start 2024-03-24T10:39:58.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b7943640 1 -- start start 2024-03-24T10:39:58.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b7943640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61b0100630 0x7f61b0199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:58.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b7943640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b0101820 0x7f61b0199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:58.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b7943640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b01021e0 0x7f61b01a0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:58.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b7943640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f61b01a1480 con 0x7f61b0101820 2024-03-24T10:39:58.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b7943640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f61b01a15f0 con 0x7f61b01021e0 2024-03-24T10:39:58.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b7943640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f61b01a17b0 con 0x7f61b0100630 2024-03-24T10:39:58.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b4ede640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b0101820 0x7f61b0199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:58.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b4ede640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b0101820 0x7f61b0199ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38578/0 (socket says 172.21.15.23:38578) 2024-03-24T10:39:58.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b4ede640 1 -- 172.21.15.23:0/1365530083 learned_addr learned my addr 172.21.15.23:0/1365530083 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:39:58.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b56df640 1 --2- 172.21.15.23:0/1365530083 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61b0100630 0x7f61b0199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:58.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b5ee0640 1 --2- 172.21.15.23:0/1365530083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b01021e0 0x7f61b01a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:39:58.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b4ede640 1 -- 172.21.15.23:0/1365530083 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61b0100630 msgr2=0x7f61b0199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:58.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b4ede640 1 --2- 172.21.15.23:0/1365530083 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61b0100630 0x7f61b0199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b4ede640 1 -- 172.21.15.23:0/1365530083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b01021e0 msgr2=0x7f61b01a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:58.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b4ede640 1 --2- 172.21.15.23:0/1365530083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b01021e0 0x7f61b01a0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b4ede640 1 -- 172.21.15.23:0/1365530083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61ac009560 con 0x7f61b0101820 2024-03-24T10:39:58.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b4ede640 1 --2- 172.21.15.23:0/1365530083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b0101820 0x7f61b0199ed0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f61a000e9f0 tx=0x7f61a000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:39:58.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61a67fc640 1 -- 172.21.15.23:0/1365530083 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a0009720 con 0x7f61b0101820 2024-03-24T10:39:58.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61a67fc640 1 -- 172.21.15.23:0/1365530083 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61a000cd80 con 0x7f61b0101820 2024-03-24T10:39:58.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61b7943640 1 -- 172.21.15.23:0/1365530083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b01a1a90 con 0x7f61b0101820 2024-03-24T10:39:58.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.439+0000 7f61a67fc640 1 -- 172.21.15.23:0/1365530083 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a0010690 con 0x7f61b0101820 2024-03-24T10:39:58.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.440+0000 7f61b7943640 1 -- 172.21.15.23:0/1365530083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b01a1fa0 con 0x7f61b0101820 2024-03-24T10:39:58.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.440+0000 7f61b7943640 1 -- 172.21.15.23:0/1365530083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b010c040 con 0x7f61b0101820 2024-03-24T10:39:58.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.442+0000 7f61a67fc640 1 -- 172.21.15.23:0/1365530083 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) ==== 136773+0+0 (secure 0 0 0) 0x7f61a00046c0 con 0x7f61b0101820 2024-03-24T10:39:58.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.443+0000 7f61a67fc640 1 --2- 172.21.15.23:0/1365530083 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f619c0a2230 0x7f619c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:39:58.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.443+0000 7f61b56df640 1 -- 172.21.15.23:0/1365530083 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f619c0a2230 msgr2=0x7f619c0a46f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2350021632 2024-03-24T10:39:58.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.443+0000 7f61a67fc640 1 -- 172.21.15.23:0/1365530083 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f61a0014070 con 0x7f61b0101820 2024-03-24T10:39:58.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.443+0000 7f61b56df640 1 --2- 172.21.15.23:0/1365530083 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f619c0a2230 0x7f619c0a46f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:39:58.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.446+0000 7f61a67fc640 1 -- 172.21.15.23:0/1365530083 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f61a0097150 con 0x7f61b0101820 2024-03-24T10:39:58.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.630+0000 7f61b7943640 1 -- 172.21.15.23:0/1365530083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f61b01a21f0 con 0x7f61b0101820 2024-03-24T10:39:58.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.634+0000 7f61a67fc640 1 -- 172.21.15.23:0/1365530083 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) ==== 133+0+0 (secure 0 0 0) 0x7f61a009a810 con 0x7f61b0101820 2024-03-24T10:39:58.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.636+0000 7f61b7943640 1 -- 172.21.15.23:0/1365530083 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f619c0a2230 msgr2=0x7f619c0a46f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:39:58.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.636+0000 7f61b7943640 1 --2- 172.21.15.23:0/1365530083 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f619c0a2230 0x7f619c0a46f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.636+0000 7f61b7943640 1 -- 172.21.15.23:0/1365530083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b0101820 msgr2=0x7f61b0199ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:39:58.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.636+0000 7f61b7943640 1 --2- 172.21.15.23:0/1365530083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b0101820 0x7f61b0199ed0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f61a000e9f0 tx=0x7f61a000edf0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.636+0000 7f61b7943640 1 -- 172.21.15.23:0/1365530083 shutdown_connections 2024-03-24T10:39:58.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.636+0000 7f61b7943640 1 --2- 172.21.15.23:0/1365530083 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f619c0a2230 0x7f619c0a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.636+0000 7f61b7943640 1 --2- 172.21.15.23:0/1365530083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b01021e0 0x7f61b01a0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.636+0000 7f61b7943640 1 --2- 172.21.15.23:0/1365530083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b0101820 0x7f61b0199ed0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.636+0000 7f61b7943640 1 --2- 172.21.15.23:0/1365530083 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61b0100630 0x7f61b0199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:39:58.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.636+0000 7f61b7943640 1 -- 172.21.15.23:0/1365530083 >> 172.21.15.23:0/1365530083 conn(0x7f61b00fbba0 msgr2=0x7f61b00fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:39:58.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.636+0000 7f61b7943640 1 -- 172.21.15.23:0/1365530083 shutdown_connections 2024-03-24T10:39:58.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:39:58.636+0000 7f61b7943640 1 -- 172.21.15.23:0/1365530083 wait complete. 2024-03-24T10:39:58.648 INFO:teuthology.run_tasks:Running task kclient... 2024-03-24T10:39:58.671 INFO:tasks.kclient:Mounting kernel clients... 2024-03-24T10:39:58.671 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-24T10:39:58.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:39:58.671 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-24T10:39:58.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:39:58.686 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2024-03-24T10:39:58.755 INFO:teuthology.orchestra.run.smithi023.stderr:Cannot find device "ceph-brx" 2024-03-24T10:39:58.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:39:58.757 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-24T10:39:58.757 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T10:39:58.757 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T10:39:58.757 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T10:39:58.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:39:58 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:39:58.589+0000 7f5cf47e2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T10:39:58.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:39:58 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:39:58.688+0000 7f5cf47e2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T10:39:58.985 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:39:58 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:39:58.640+0000 7f5617284200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T10:39:58.985 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:39:58 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:39:58.736+0000 7f5617284200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T10:39:58.986 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:39:58 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:39:58.552+0000 7fdb43faa200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T10:39:58.986 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:39:58 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:39:58.648+0000 7fdb43faa200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T10:39:59.057 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:39:59.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:58 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1523661121' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-24T10:39:59.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:58 smithi117 ceph-mon[36368]: mgrmap e19: x(active, since 8m), standbys: y, z 2024-03-24T10:39:59.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:58 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/48706298' entity='client.admin' 2024-03-24T10:39:59.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:39:58 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1365530083' entity='client.admin' 2024-03-24T10:39:59.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:58 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1523661121' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-24T10:39:59.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:58 smithi116 ceph-mon[36248]: mgrmap e19: x(active, since 8m), standbys: y, z 2024-03-24T10:39:59.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:58 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/48706298' entity='client.admin' 2024-03-24T10:39:59.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:39:58 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1365530083' entity='client.admin' 2024-03-24T10:39:59.296 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:58 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1523661121' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-24T10:39:59.296 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:58 smithi023 ceph-mon[32466]: mgrmap e19: x(active, since 8m), standbys: y, z 2024-03-24T10:39:59.296 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:58 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/48706298' entity='client.admin' 2024-03-24T10:39:59.296 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:39:58 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1365530083' entity='client.admin' 2024-03-24T10:39:59.630 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:39:59 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:39:59.384+0000 7f5cf47e2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T10:39:59.630 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:39:59 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:39:59.483+0000 7f5cf47e2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T10:39:59.630 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:39:59 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:39:59.629+0000 7f5cf47e2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T10:39:59.687 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:39:59 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:39:59.350+0000 7fdb43faa200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T10:39:59.687 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:39:59 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:39:59.444+0000 7fdb43faa200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T10:39:59.687 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:39:59 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:39:59.587+0000 7fdb43faa200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T10:39:59.687 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:39:59 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:39:59.686+0000 7fdb43faa200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T10:39:59.746 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:39:59 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:39:59.410+0000 7f5617284200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T10:39:59.746 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:39:59 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:39:59.504+0000 7f5617284200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T10:39:59.746 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:39:59 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:39:59.647+0000 7f5617284200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T10:39:59.746 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:39:59 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:39:59.745+0000 7f5617284200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T10:39:59.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:39:59 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:39:59.733+0000 7f5cf47e2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T10:39:59.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:39:59 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:39:59.820+0000 7f5cf47e2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T10:40:00.003 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:39:59 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:39:59.770+0000 7fdb43faa200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T10:40:00.018 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:39:59 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:39:59.830+0000 7f5617284200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T10:40:00.302 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:00 smithi023 ceph-mon[32466]: overall HEALTH_OK 2024-03-24T10:40:00.302 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:00 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:00.092+0000 7f5cf47e2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T10:40:00.302 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:00 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:00.210+0000 7f5cf47e2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T10:40:00.302 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:00 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:00.301+0000 7f5cf47e2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T10:40:00.312 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:00 smithi116 ceph-mon[36248]: overall HEALTH_OK 2024-03-24T10:40:00.312 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:00 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:00.099+0000 7f5617284200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T10:40:00.312 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:00 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:00.211+0000 7f5617284200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T10:40:00.331 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:00 smithi117 ceph-mon[36368]: overall HEALTH_OK 2024-03-24T10:40:00.331 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:00 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:00.035+0000 7fdb43faa200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T10:40:00.331 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:00 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:00.146+0000 7fdb43faa200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T10:40:00.332 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:00 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:00.235+0000 7fdb43faa200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T10:40:00.332 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:00 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:00.330+0000 7fdb43faa200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T10:40:00.554 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:00 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:00.400+0000 7f5cf47e2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T10:40:00.657 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:00 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:00.311+0000 7f5617284200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T10:40:00.657 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:00 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:00.434+0000 7f5617284200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T10:40:00.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.669+0000 7f52324e4700 1 -- 172.21.15.23:0/1378837047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c101d40 msgr2=0x7f522c105d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:00.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.669+0000 7f52324e4700 1 --2- 172.21.15.23:0/1378837047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c101d40 0x7f522c105d00 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f521c009990 tx=0x7f521c009ca0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.669+0000 7f52324e4700 1 -- 172.21.15.23:0/1378837047 shutdown_connections 2024-03-24T10:40:00.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.669+0000 7f52324e4700 1 --2- 172.21.15.23:0/1378837047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f522c1063e0 0x7f522c10dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.669+0000 7f52324e4700 1 --2- 172.21.15.23:0/1378837047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522c101390 0x7f522c101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.669+0000 7f52324e4700 1 --2- 172.21.15.23:0/1378837047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c101d40 0x7f522c105d00 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.669+0000 7f52324e4700 1 -- 172.21.15.23:0/1378837047 >> 172.21.15.23:0/1378837047 conn(0x7f522c077150 msgr2=0x7f522c077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:00.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.669+0000 7f52324e4700 1 -- 172.21.15.23:0/1378837047 shutdown_connections 2024-03-24T10:40:00.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.669+0000 7f52324e4700 1 -- 172.21.15.23:0/1378837047 wait complete. 2024-03-24T10:40:00.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.669+0000 7f52324e4700 1 Processor -- start 2024-03-24T10:40:00.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f52324e4700 1 -- start start 2024-03-24T10:40:00.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f52324e4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c101390 0x7f522c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:00.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f52324e4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f522c101d40 0x7f522c198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:00.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f52324e4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522c1063e0 0x7f522c19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:00.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f52324e4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f522c19d5d0 con 0x7f522c101390 2024-03-24T10:40:00.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f52324e4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f522c19d740 con 0x7f522c1063e0 2024-03-24T10:40:00.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f52324e4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f522c19d900 con 0x7f522c101d40 2024-03-24T10:40:00.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f522bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c101390 0x7f522c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:00.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f522b7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f522c101d40 0x7f522c198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f522bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c101390 0x7f522c1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38604/0 (socket says 172.21.15.23:38604) 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f522bfff700 1 -- 172.21.15.23:0/4050298490 learned_addr learned my addr 172.21.15.23:0/4050298490 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f5230a81700 1 --2- 172.21.15.23:0/4050298490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522c1063e0 0x7f522c19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f522bfff700 1 -- 172.21.15.23:0/4050298490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f522c101d40 msgr2=0x7f522c198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f522bfff700 1 --2- 172.21.15.23:0/4050298490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f522c101d40 0x7f522c198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f522bfff700 1 -- 172.21.15.23:0/4050298490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522c1063e0 msgr2=0x7f522c19cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f522bfff700 1 --2- 172.21.15.23:0/4050298490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522c1063e0 0x7f522c19cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f522bfff700 1 -- 172.21.15.23:0/4050298490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f521c0095f0 con 0x7f522c101390 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.670+0000 7f522b7fe700 1 --2- 172.21.15.23:0/4050298490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f522c101d40 0x7f522c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.671+0000 7f522bfff700 1 --2- 172.21.15.23:0/4050298490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c101390 0x7f522c1986f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f522000e9a0 tx=0x7f522000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.671+0000 7f52297fa700 1 -- 172.21.15.23:0/4050298490 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5220009790 con 0x7f522c101390 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.671+0000 7f52297fa700 1 -- 172.21.15.23:0/4050298490 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5220010430 con 0x7f522c101390 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.671+0000 7f52297fa700 1 -- 172.21.15.23:0/4050298490 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5220018680 con 0x7f522c101390 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.671+0000 7f52324e4700 1 -- 172.21.15.23:0/4050298490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522c19dbe0 con 0x7f522c101390 2024-03-24T10:40:00.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.671+0000 7f52324e4700 1 -- 172.21.15.23:0/4050298490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f522c102ea0 con 0x7f522c101390 2024-03-24T10:40:00.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.672+0000 7f52324e4700 1 -- 172.21.15.23:0/4050298490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f522c10a710 con 0x7f522c101390 2024-03-24T10:40:00.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.674+0000 7f52297fa700 1 -- 172.21.15.23:0/4050298490 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) ==== 136773+0+0 (secure 0 0 0) 0x7f5220010910 con 0x7f522c101390 2024-03-24T10:40:00.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.674+0000 7f52297fa700 1 --2- 172.21.15.23:0/4050298490 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f52140a21a0 0x7f52140a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:00.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.674+0000 7f522b7fe700 1 -- 172.21.15.23:0/4050298490 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f52140a21a0 msgr2=0x7f52140a4660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2350021632 2024-03-24T10:40:00.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.674+0000 7f52297fa700 1 -- 172.21.15.23:0/4050298490 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f5220014070 con 0x7f522c101390 2024-03-24T10:40:00.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.674+0000 7f522b7fe700 1 --2- 172.21.15.23:0/4050298490 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f52140a21a0 0x7f52140a4660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:40:00.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.677+0000 7f52297fa700 1 -- 172.21.15.23:0/4050298490 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f52200973d0 con 0x7f522c101390 2024-03-24T10:40:00.753 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:00 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:00.510+0000 7fdb43faa200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T10:40:00.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.875+0000 7f522b7fe700 1 -- 172.21.15.23:0/4050298490 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f52140a21a0 msgr2=0x7f52140a4660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2350021632 2024-03-24T10:40:00.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.875+0000 7f522b7fe700 1 --2- 172.21.15.23:0/4050298490 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f52140a21a0 0x7f52140a4660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:40:00.876 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:00 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:00.596+0000 7f5cf47e2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T10:40:00.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.927+0000 7f52324e4700 1 -- 172.21.15.23:0/4050298490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f522c199a50 con 0x7f522c101390 2024-03-24T10:40:00.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.928+0000 7f52297fa700 1 -- 172.21.15.23:0/4050298490 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) ==== 53+0+73 (secure 0 0 0) 0x7f5220026060 con 0x7f522c101390 2024-03-24T10:40:00.928 INFO:teuthology.orchestra.run.smithi023.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T10:40:00.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.931+0000 7f52324e4700 1 -- 172.21.15.23:0/4050298490 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f52140a21a0 msgr2=0x7f52140a4660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:40:00.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.931+0000 7f52324e4700 1 --2- 172.21.15.23:0/4050298490 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f52140a21a0 0x7f52140a4660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.931+0000 7f52324e4700 1 -- 172.21.15.23:0/4050298490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c101390 msgr2=0x7f522c1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:00.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.931+0000 7f52324e4700 1 --2- 172.21.15.23:0/4050298490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c101390 0x7f522c1986f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f522000e9a0 tx=0x7f522000ecb0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.931+0000 7f52324e4700 1 -- 172.21.15.23:0/4050298490 shutdown_connections 2024-03-24T10:40:00.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.931+0000 7f52324e4700 1 --2- 172.21.15.23:0/4050298490 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f52140a21a0 0x7f52140a4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.931+0000 7f52324e4700 1 --2- 172.21.15.23:0/4050298490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522c1063e0 0x7f522c19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.931+0000 7f52324e4700 1 --2- 172.21.15.23:0/4050298490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c101390 0x7f522c1986f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.931+0000 7f52324e4700 1 --2- 172.21.15.23:0/4050298490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f522c101d40 0x7f522c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:00.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.931+0000 7f52324e4700 1 -- 172.21.15.23:0/4050298490 >> 172.21.15.23:0/4050298490 conn(0x7f522c077150 msgr2=0x7f522c06aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:00.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.931+0000 7f52324e4700 1 -- 172.21.15.23:0/4050298490 shutdown_connections 2024-03-24T10:40:00.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:00.931+0000 7f52324e4700 1 -- 172.21.15.23:0/4050298490 wait complete. 2024-03-24T10:40:01.018 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:00 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:00.656+0000 7f5617284200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T10:40:01.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:01 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4050298490' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T10:40:01.485 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:01 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4050298490' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T10:40:01.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:01 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4050298490' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T10:40:01.619 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T10:40:01.619 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T10:40:01.619 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2024-03-24T10:40:01.619 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T10:40:01.619 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T10:40:01.619 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T10:40:01.619 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T10:40:01.619 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T10:40:01.619 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:40:01.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:40:01.620 DEBUG:teuthology.orchestra.run.smithi023:> ip addr 2024-03-24T10:40:01.637 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T10:40:01.637 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T10:40:01.637 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T10:40:01.637 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:40:01.637 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2024-03-24T10:40:01.637 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:40:01.637 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:40:01.637 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2024-03-24T10:40:01.637 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f0 2024-03-24T10:40:01.637 INFO:teuthology.orchestra.run.smithi023.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:40:01.637 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2024-03-24T10:40:01.637 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f1 2024-03-24T10:40:01.637 INFO:teuthology.orchestra.run.smithi023.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T10:40:01.638 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:40:01.638 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f0 2024-03-24T10:40:01.638 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T10:40:01.638 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 42134sec preferred_lft 42134sec 2024-03-24T10:40:01.638 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link noprefixroute 2024-03-24T10:40:01.638 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-24T10:40:01.638 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T10:40:01.638 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2024-03-24T10:40:01.638 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f1 2024-03-24T10:40:01.641 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T10:40:01.641 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:40:01.642 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-24T10:40:01.642 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2024-03-24T10:40:01.642 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2024-03-24T10:40:01.642 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2024-03-24T10:40:01.642 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T10:40:01.642 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-24T10:40:01.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:40:01.753 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:01 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:01.484+0000 7fdb43faa200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T10:40:01.753 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:01 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:01.617+0000 7fdb43faa200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T10:40:01.834 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:01 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:01.630+0000 7f5cf47e2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T10:40:01.834 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:01 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:01.769+0000 7f5cf47e2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T10:40:01.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:40:01.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:40:01.842 DEBUG:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T10:40:01.871 INFO:teuthology.orchestra.run.smithi023.stdout:1 2024-03-24T10:40:01.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:40:01.872 DEBUG:teuthology.orchestra.run.smithi023:> route 2024-03-24T10:40:01.940 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2024-03-24T10:40:01.940 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T10:40:01.940 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T10:40:01.940 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T10:40:01.940 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T10:40:01.941 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:40:01.941 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-24T10:40:01.941 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T10:40:01.941 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T10:40:01.941 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T10:40:01.941 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-24T10:40:02.018 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:01 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:01.748+0000 7f5617284200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T10:40:02.018 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:01 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:01.915+0000 7f5617284200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T10:40:02.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:40:02.248 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:01 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:01.956+0000 7fdb43faa200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T10:40:02.248 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:02 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:02.048+0000 7fdb43faa200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T10:40:02.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:40:02.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:40:02.256 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-24T10:40:02.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:40:02.271 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list-id 2024-03-24T10:40:02.326 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:40:02.326 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-24T10:40:02.326 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:40:02.326 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-24T10:40:02.326 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-24T10:40:02.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:40:02.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:02 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:02.125+0000 7f5cf47e2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T10:40:02.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:02 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:02.221+0000 7f5cf47e2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T10:40:02.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:40:02.449 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-24T10:40:02.450 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:40:02.450 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-24T10:40:02.450 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-24T10:40:02.450 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T10:40:02.450 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-24T10:40:02.450 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-24T10:40:02.450 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-24T10:40:02.450 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-24T10:40:02.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:40:02.503 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:02 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:02.247+0000 7fdb43faa200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:40:02.503 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:02 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:02.435+0000 7fdb43faa200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T10:40:02.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:40:02.588 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:02 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:02.299+0000 7f5617284200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T10:40:02.588 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:02 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:02.389+0000 7f5617284200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T10:40:02.592 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T10:40:02.592 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-24T10:40:02.592 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2024-03-24T10:40:02.592 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T10:40:02.592 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-24T10:40:02.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:40:02.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:40:02.677 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:40:02.677 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T10:40:02.692 INFO:teuthology.orchestra.run.smithi023.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T10:40:02.693 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T10:40:02.693 DEBUG:teuthology.orchestra.run.smithi023:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:40:02.748 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-24T10:40:02.748 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-24T10:40:02.748 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/b7fb1ade-3c33-4b9e-8a53-f368dfa57265 2024-03-24T10:40:02.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:40:02.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/b7fb1ade-3c33-4b9e-8a53-f368dfa57265 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-24T10:40:02.813 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:02 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:02.432+0000 7f5cf47e2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:40:02.813 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:02 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:02.641+0000 7f5cf47e2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T10:40:02.833 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-24T10:40:02.834 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-03-24T10:40:02.834 INFO:teuthology.orchestra.run.smithi023.stdout:invalid new device string format 2024-03-24T10:40:02.961 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-24T10:40:02.961 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-03-24T10:40:02.961 INFO:teuthology.orchestra.run.smithi023.stdout:invalid new device string format 2024-03-24T10:40:02.961 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: resolved to: "172.21.15.23:3300,172.21.15.116:3300,172.21.15.117:3300" 2024-03-24T10:40:02.961 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.23:3300,172.21.15.116:3300,172.21.15.117:3300:/volumes/qa/sv_1/b7fb1ade-3c33-4b9e-8a53-f368dfa57265 2024-03-24T10:40:02.961 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=4f8fb8f6-e9c9-11ee-95cd-87774f69a715" will pass to kernel 2024-03-24T10:40:02.968 INFO:teuthology.orchestra.run.smithi023.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T10:40:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: You just mounted a file system that supports labels which does not 2024-03-24T10:40:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T10:40:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T10:40:02.968 INFO:teuthology.orchestra.run.smithi023.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T10:40:02.969 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T10:40:02.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:40:02.969 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T10:40:03.003 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:02 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:02.743+0000 7fdb43faa200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:40:03.017 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:02 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:02.587+0000 7f5617284200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:40:03.018 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:02 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:02.813+0000 7f5617284200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T10:40:03.042 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T10:40:03.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T10:40:03.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:40:03.141 INFO:teuthology.orchestra.run.smithi023.stdout:client25174 2024-03-24T10:40:03.141 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:40:03.142 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:40:03.157 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T10:40:03.168 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:02 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:02.953+0000 7f5cf47e2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:40:03.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T10:40:03.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:40:03.223 INFO:teuthology.orchestra.run.smithi023.stdout:client25174 2024-03-24T10:40:03.224 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T10:40:03.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T10:40:03.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:40:03.280 INFO:teuthology.orchestra.run.smithi023.stdout:client25174 2024-03-24T10:40:03.281 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715.client25174/status') 2024-03-24T10:40:03.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T10:40:03.342 INFO:teuthology.orchestra.run.smithi023.stdout:instance: client.25174 (3)172.21.15.23:0/2849745419 2024-03-24T10:40:03.342 INFO:teuthology.orchestra.run.smithi023.stdout:blocklisted: false 2024-03-24T10:40:03.342 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2024-03-24T10:40:03.343 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2024-03-24T10:40:03.343 INFO:teuthology.orchestra.run.smithi023.stderr:71 bytes copied, 2.1425e-05 s, 3.3 MB/s 2024-03-24T10:40:03.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T10:40:03.349 INFO:teuthology.run_tasks:Running task exec... 2024-03-24T10:40:03.359 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T10:40:03.359 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T10:40:03.359 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-24T10:40:03.431 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:03 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:03.159+0000 7f5617284200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:40:03.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.464+0000 7f99cf9dd640 1 -- 172.21.15.23:0/3038883991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c81021e0 msgr2=0x7f99c8108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:03.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.464+0000 7f99cf9dd640 1 --2- 172.21.15.23:0/3038883991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c81021e0 0x7f99c8108a70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f99c0009ca0 tx=0x7f99c002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.464+0000 7f99cf9dd640 1 -- 172.21.15.23:0/3038883991 shutdown_connections 2024-03-24T10:40:03.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.464+0000 7f99cf9dd640 1 --2- 172.21.15.23:0/3038883991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c81021e0 0x7f99c8108a70 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.464+0000 7f99cf9dd640 1 --2- 172.21.15.23:0/3038883991 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99c8101820 0x7f99c8101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.464+0000 7f99cf9dd640 1 --2- 172.21.15.23:0/3038883991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c8100630 0x7f99c8100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.464+0000 7f99cf9dd640 1 -- 172.21.15.23:0/3038883991 >> 172.21.15.23:0/3038883991 conn(0x7f99c80fbba0 msgr2=0x7f99c80fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:03.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.464+0000 7f99cf9dd640 1 -- 172.21.15.23:0/3038883991 shutdown_connections 2024-03-24T10:40:03.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.464+0000 7f99cf9dd640 1 -- 172.21.15.23:0/3038883991 wait complete. 2024-03-24T10:40:03.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99cf9dd640 1 Processor -- start 2024-03-24T10:40:03.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99cf9dd640 1 -- start start 2024-03-24T10:40:03.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99cf9dd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99c8100630 0x7f99c8069960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:03.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99cf9dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c8101820 0x7f99c819c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:03.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99cf9dd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c81021e0 0x7f99c819ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:03.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99cf9dd640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f99c819f1a0 con 0x7f99c81021e0 2024-03-24T10:40:03.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99cf9dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99c819f310 con 0x7f99c8101820 2024-03-24T10:40:03.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99cf9dd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f99c819f4d0 con 0x7f99c8100630 2024-03-24T10:40:03.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99cdf7a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c81021e0 0x7f99c819ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:03.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99ccf78640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c8101820 0x7f99c819c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:03.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99ccf78640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c8101820 0x7f99c819c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:46688/0 (socket says 172.21.15.23:46688) 2024-03-24T10:40:03.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99cdf7a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c81021e0 0x7f99c819ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38616/0 (socket says 172.21.15.23:38616) 2024-03-24T10:40:03.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99cd779640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99c8100630 0x7f99c8069960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:03.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99ccf78640 1 -- 172.21.15.23:0/1763567078 learned_addr learned my addr 172.21.15.23:0/1763567078 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:03.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99cdf7a640 1 -- 172.21.15.23:0/1763567078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99c8100630 msgr2=0x7f99c8069960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:03.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.465+0000 7f99cdf7a640 1 --2- 172.21.15.23:0/1763567078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99c8100630 0x7f99c8069960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.466+0000 7f99cdf7a640 1 -- 172.21.15.23:0/1763567078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c8101820 msgr2=0x7f99c819c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:03.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.466+0000 7f99cdf7a640 1 --2- 172.21.15.23:0/1763567078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c8101820 0x7f99c819c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.466+0000 7f99cdf7a640 1 -- 172.21.15.23:0/1763567078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c0009900 con 0x7f99c81021e0 2024-03-24T10:40:03.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.466+0000 7f99ccf78640 1 --2- 172.21.15.23:0/1763567078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c8101820 0x7f99c819c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:03.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.466+0000 7f99cdf7a640 1 --2- 172.21.15.23:0/1763567078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c81021e0 0x7f99c819ea10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f99c002c630 tx=0x7f99c0034a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:03.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.466+0000 7f99be7fc640 1 -- 172.21.15.23:0/1763567078 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c00329b0 con 0x7f99c81021e0 2024-03-24T10:40:03.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.466+0000 7f99be7fc640 1 -- 172.21.15.23:0/1763567078 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99c0003e50 con 0x7f99c81021e0 2024-03-24T10:40:03.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.466+0000 7f99cf9dd640 1 -- 172.21.15.23:0/1763567078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99c819f750 con 0x7f99c81021e0 2024-03-24T10:40:03.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.466+0000 7f99be7fc640 1 -- 172.21.15.23:0/1763567078 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c0040450 con 0x7f99c81021e0 2024-03-24T10:40:03.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.466+0000 7f99cf9dd640 1 -- 172.21.15.23:0/1763567078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99c819fbd0 con 0x7f99c81021e0 2024-03-24T10:40:03.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.467+0000 7f99cf9dd640 1 -- 172.21.15.23:0/1763567078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99c8107950 con 0x7f99c81021e0 2024-03-24T10:40:03.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.468+0000 7f99be7fc640 1 -- 172.21.15.23:0/1763567078 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) ==== 136773+0+0 (secure 0 0 0) 0x7f99c000a040 con 0x7f99c81021e0 2024-03-24T10:40:03.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.469+0000 7f99be7fc640 1 --2- 172.21.15.23:0/1763567078 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f99b40a21b0 0x7f99b40a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:03.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.469+0000 7f99cd779640 1 -- 172.21.15.23:0/1763567078 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f99b40a21b0 msgr2=0x7f99b40a4670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2350021632 2024-03-24T10:40:03.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.469+0000 7f99be7fc640 1 -- 172.21.15.23:0/1763567078 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f99c00f09b0 con 0x7f99c81021e0 2024-03-24T10:40:03.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.469+0000 7f99cd779640 1 --2- 172.21.15.23:0/1763567078 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f99b40a21b0 0x7f99b40a4670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:40:03.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.473+0000 7f99be7fc640 1 -- 172.21.15.23:0/1763567078 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f99c00b9960 con 0x7f99c81021e0 2024-03-24T10:40:03.482 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:03 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:03.225+0000 7f5cf47e2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T10:40:03.483 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:03 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:03.482+0000 7f5cf47e2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T10:40:03.505 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:03 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:03.040+0000 7fdb43faa200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T10:40:03.506 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:03 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:03.280+0000 7fdb43faa200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T10:40:03.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.670+0000 7f99cd779640 1 -- 172.21.15.23:0/1763567078 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f99b40a21b0 msgr2=0x7f99b40a4670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2350021632 2024-03-24T10:40:03.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.670+0000 7f99cd779640 1 --2- 172.21.15.23:0/1763567078 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f99b40a21b0 0x7f99b40a4670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:40:03.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.695+0000 7f99cf9dd640 1 -- 172.21.15.23:0/1763567078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f99c802d1d0 con 0x7f99c81021e0 2024-03-24T10:40:03.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.748+0000 7f99be7fc640 1 -- 172.21.15.23:0/1763567078 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v23) ==== 118+0+0 (secure 0 0 0) 0x7f99c00bd020 con 0x7f99c81021e0 2024-03-24T10:40:03.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.750+0000 7f99cf9dd640 1 -- 172.21.15.23:0/1763567078 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f99b40a21b0 msgr2=0x7f99b40a4670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:40:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.750+0000 7f99cf9dd640 1 --2- 172.21.15.23:0/1763567078 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f99b40a21b0 0x7f99b40a4670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.750+0000 7f99cf9dd640 1 -- 172.21.15.23:0/1763567078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c81021e0 msgr2=0x7f99c819ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.750+0000 7f99cf9dd640 1 --2- 172.21.15.23:0/1763567078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c81021e0 0x7f99c819ea10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f99c002c630 tx=0x7f99c0034a90 comp rx=0 tx=0).stop 2024-03-24T10:40:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.750+0000 7f99cf9dd640 1 -- 172.21.15.23:0/1763567078 shutdown_connections 2024-03-24T10:40:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.750+0000 7f99cf9dd640 1 --2- 172.21.15.23:0/1763567078 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f99b40a21b0 0x7f99b40a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.750+0000 7f99cf9dd640 1 --2- 172.21.15.23:0/1763567078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c81021e0 0x7f99c819ea10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.750+0000 7f99cf9dd640 1 --2- 172.21.15.23:0/1763567078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c8101820 0x7f99c819c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.750+0000 7f99cf9dd640 1 --2- 172.21.15.23:0/1763567078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99c8100630 0x7f99c8069960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.750+0000 7f99cf9dd640 1 -- 172.21.15.23:0/1763567078 >> 172.21.15.23:0/1763567078 conn(0x7f99c80fbba0 msgr2=0x7f99c8104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.750+0000 7f99cf9dd640 1 -- 172.21.15.23:0/1763567078 shutdown_connections 2024-03-24T10:40:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.750+0000 7f99cf9dd640 1 -- 172.21.15.23:0/1763567078 wait complete. 2024-03-24T10:40:03.753 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:03 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:03.430+0000 7f5617284200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T10:40:03.754 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:03 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:03.678+0000 7f5617284200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T10:40:03.762 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-24T10:40:03.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.868+0000 7f01e1855640 1 -- 172.21.15.23:0/3653283598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01dc100640 msgr2=0x7f01dc100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:03.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.868+0000 7f01e1855640 1 --2- 172.21.15.23:0/3653283598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01dc100640 0x7f01dc100a60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f01d0009900 tx=0x7f01d002bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:03.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.868+0000 7f01e1855640 1 -- 172.21.15.23:0/3653283598 shutdown_connections 2024-03-24T10:40:03.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.868+0000 7f01e1855640 1 --2- 172.21.15.23:0/3653283598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01dc1021f0 0x7f01dc108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.868+0000 7f01e1855640 1 --2- 172.21.15.23:0/3653283598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01dc101830 0x7f01dc101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.868+0000 7f01e1855640 1 --2- 172.21.15.23:0/3653283598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01dc100640 0x7f01dc100a60 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.868+0000 7f01e1855640 1 -- 172.21.15.23:0/3653283598 >> 172.21.15.23:0/3653283598 conn(0x7f01dc0fbbd0 msgr2=0x7f01dc0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.868+0000 7f01e1855640 1 -- 172.21.15.23:0/3653283598 shutdown_connections 2024-03-24T10:40:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.868+0000 7f01e1855640 1 -- 172.21.15.23:0/3653283598 wait complete. 2024-03-24T10:40:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.868+0000 7f01e1855640 1 Processor -- start 2024-03-24T10:40:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01e1855640 1 -- start start 2024-03-24T10:40:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01e1855640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01dc100640 0x7f01dc06aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01e1855640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01dc101830 0x7f01dc19e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01e1855640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01dc1021f0 0x7f01dc1a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01e1855640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc1a14a0 con 0x7f01dc101830 2024-03-24T10:40:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01e1855640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc1a1610 con 0x7f01dc100640 2024-03-24T10:40:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01e1855640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc1a17d0 con 0x7f01dc1021f0 2024-03-24T10:40:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01da7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01dc101830 0x7f01dc19e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01daffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01dc100640 0x7f01dc06aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:03.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01da7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01dc101830 0x7f01dc19e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38638/0 (socket says 172.21.15.23:38638) 2024-03-24T10:40:03.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01db7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01dc1021f0 0x7f01dc1a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:03.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01daffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01dc100640 0x7f01dc06aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:46710/0 (socket says 172.21.15.23:46710) 2024-03-24T10:40:03.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01da7fc640 1 -- 172.21.15.23:0/2248275989 learned_addr learned my addr 172.21.15.23:0/2248275989 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:03.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01da7fc640 1 -- 172.21.15.23:0/2248275989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01dc1021f0 msgr2=0x7f01dc1a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:03.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01da7fc640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01dc1021f0 0x7f01dc1a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01da7fc640 1 -- 172.21.15.23:0/2248275989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01dc100640 msgr2=0x7f01dc06aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:03.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01da7fc640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01dc100640 0x7f01dc06aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:03.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01da7fc640 1 -- 172.21.15.23:0/2248275989 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01cc009490 con 0x7f01dc101830 2024-03-24T10:40:03.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01daffd640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01dc100640 0x7f01dc06aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:03.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.869+0000 7f01da7fc640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01dc101830 0x7f01dc19e890 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00ea60 tx=0x7f01cc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:03.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.870+0000 7f01e0853640 1 -- 172.21.15.23:0/2248275989 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc00cde0 con 0x7f01dc101830 2024-03-24T10:40:03.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.870+0000 7f01e0853640 1 -- 172.21.15.23:0/2248275989 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01cc0035f0 con 0x7f01dc101830 2024-03-24T10:40:03.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.870+0000 7f01e1855640 1 -- 172.21.15.23:0/2248275989 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d0009560 con 0x7f01dc101830 2024-03-24T10:40:03.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.870+0000 7f01e0853640 1 -- 172.21.15.23:0/2248275989 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc010690 con 0x7f01dc101830 2024-03-24T10:40:03.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.870+0000 7f01e1855640 1 -- 172.21.15.23:0/2248275989 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01dc1a1dd0 con 0x7f01dc101830 2024-03-24T10:40:03.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.871+0000 7f01e1855640 1 -- 172.21.15.23:0/2248275989 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01dc107900 con 0x7f01dc101830 2024-03-24T10:40:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.872+0000 7f01e0853640 1 -- 172.21.15.23:0/2248275989 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) ==== 136773+0+0 (secure 0 0 0) 0x7f01cc003af0 con 0x7f01dc101830 2024-03-24T10:40:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.872+0000 7f01e0853640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f01c40a21e0 0x7f01c40a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.873+0000 7f01daffd640 1 -- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f01c40a21e0 msgr2=0x7f01c40a46a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2350021632 2024-03-24T10:40:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.873+0000 7f01e0853640 1 -- 172.21.15.23:0/2248275989 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f01cc014070 con 0x7f01dc101830 2024-03-24T10:40:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.873+0000 7f01daffd640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f01c40a21e0 0x7f01c40a46a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:40:03.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:03.876+0000 7f01e0853640 1 -- 172.21.15.23:0/2248275989 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f01cc097210 con 0x7f01dc101830 2024-03-24T10:40:04.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:03 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1763567078' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-24T10:40:04.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:04.074+0000 7f01daffd640 1 -- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f01c40a21e0 msgr2=0x7f01c40a46a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2350021632 2024-03-24T10:40:04.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:04.074+0000 7f01daffd640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f01c40a21e0 0x7f01c40a46a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:40:04.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:04.088+0000 7f01e1855640 1 -- 172.21.15.23:0/2248275989 --> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f01dc02d2e0 con 0x7f01c40a21e0 2024-03-24T10:40:04.116 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:03 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1763567078' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-24T10:40:04.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:03 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1763567078' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-24T10:40:04.456 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:04 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:04.345+0000 7f5cf47e2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T10:40:04.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:04.475+0000 7f01daffd640 1 -- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f01c40a21e0 msgr2=0x7f01c40a46a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2350021632 2024-03-24T10:40:04.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:04.475+0000 7f01daffd640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f01c40a21e0 0x7f01c40a46a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:40:04.503 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:04.116+0000 7fdb43faa200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T10:40:05.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:04 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1763567078' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-24T10:40:05.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:04 smithi116 ceph-mon[36248]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:40:05.018 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:04 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:04.567+0000 7f5617284200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T10:40:05.049 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:04 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1763567078' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-24T10:40:05.049 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:04 smithi023 ceph-mon[32466]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:40:05.049 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:04 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:04.942+0000 7f5cf47e2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T10:40:05.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:04 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1763567078' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-24T10:40:05.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:04 smithi117 ceph-mon[36368]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:40:05.108 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:04.763+0000 7fdb43faa200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T10:40:05.108 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:04.884+0000 7fdb43faa200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T10:40:05.108 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z[38272]: 2024-03-24T10:40:05.106+0000 7fdb43faa200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T10:40:05.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:05.276+0000 7f01daffd640 1 -- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f01c40a21e0 msgr2=0x7f01c40a46a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2350021632 2024-03-24T10:40:05.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:05.276+0000 7f01daffd640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f01c40a21e0 0x7f01c40a46a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:40:05.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:05.304+0000 7f01e0853640 1 -- 172.21.15.23:0/2248275989 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 20) ==== 136441+0+0 (secure 0 0 0) 0x7f01cc0975e0 con 0x7f01dc101830 2024-03-24T10:40:05.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:05.305+0000 7f01e0853640 1 -- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f01c40a21e0 msgr2=0x7f01c40a46a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T10:40:05.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:05.305+0000 7f01e0853640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/2350021632,v1:172.21.15.23:6801/2350021632] conn(0x7f01c40a21e0 0x7f01c40a46a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:05.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:05.048+0000 7f5cf47e2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T10:40:05.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:05.278+0000 7f5cf47e2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T10:40:05.481 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:05.174+0000 7f5617284200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T10:40:05.481 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:05.274+0000 7f5617284200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T10:40:05.757 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y[37643]: 2024-03-24T10:40:05.504+0000 7f5617284200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T10:40:05.757 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:05.591+0000 7f5c571bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:05.757 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:05.591+0000 7f5c571bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:05.757 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:05.591+0000 7f5c571bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:05.757 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:05.591+0000 7f5c571bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:05.758 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:05.591+0000 7f5c571bf700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:05.758 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:05.596+0000 7f5c541b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:05.758 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:05.596+0000 7f5c541b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:05.758 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:05.596+0000 7f5c541b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:05.758 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:05.596+0000 7f5c541b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:05.758 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:05.596+0000 7f5c541b9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:06.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: Standby manager daemon z restarted 2024-03-24T10:40:06.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: Standby manager daemon z started 2024-03-24T10:40:06.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.117:0/4129920998' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T10:40:06.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.117:0/4129920998' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T10:40:06.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.117:0/4129920998' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T10:40:06.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.117:0/4129920998' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T10:40:06.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: Active manager daemon x restarted 2024-03-24T10:40:06.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: Activating manager daemon x 2024-03-24T10:40:06.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T10:40:06.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: mgrmap e20: x(active, starting, since 0.0144233s), standbys: y, z 2024-03-24T10:40:06.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:40:06.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:40:06.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:40:06.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T10:40:06.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T10:40:06.020 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T10:40:06.020 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T10:40:06.020 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T10:40:06.020 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T10:40:06.020 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T10:40:06.020 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T10:40:06.020 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:40:06.020 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T10:40:06.020 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T10:40:06.020 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T10:40:06.020 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T10:40:06.020 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T10:40:06.020 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T10:40:06.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:40:06.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:40:06.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:40:06.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:40:06.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:40:06.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:40:06.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:40:06.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:40:06.021 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:40:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:40:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:40:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:40:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T10:40:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T10:40:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T10:40:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: Manager daemon x is now available 2024-03-24T10:40:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:40:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:40:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T10:40:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T10:40:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: Standby manager daemon y restarted 2024-03-24T10:40:06.022 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: Standby manager daemon y started 2024-03-24T10:40:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.116:0/4032755372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T10:40:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.116:0/4032755372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T10:40:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.116:0/4032755372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T10:40:06.023 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:05 smithi116 ceph-mon[36248]: from='mgr.? 172.21.15.116:0/4032755372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T10:40:06.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: Standby manager daemon z restarted 2024-03-24T10:40:06.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: Standby manager daemon z started 2024-03-24T10:40:06.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.117:0/4129920998' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T10:40:06.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.117:0/4129920998' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T10:40:06.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.117:0/4129920998' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T10:40:06.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.117:0/4129920998' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T10:40:06.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: Active manager daemon x restarted 2024-03-24T10:40:06.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: Activating manager daemon x 2024-03-24T10:40:06.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T10:40:06.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: mgrmap e20: x(active, starting, since 0.0144233s), standbys: y, z 2024-03-24T10:40:06.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:40:06.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:40:06.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:40:06.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T10:40:06.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T10:40:06.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T10:40:06.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T10:40:06.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T10:40:06.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T10:40:06.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:40:06.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: Manager daemon x is now available 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: Standby manager daemon y restarted 2024-03-24T10:40:06.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: Standby manager daemon y started 2024-03-24T10:40:06.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.116:0/4032755372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T10:40:06.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.116:0/4032755372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T10:40:06.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.116:0/4032755372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T10:40:06.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:05 smithi023 ceph-mon[32466]: from='mgr.? 172.21.15.116:0/4032755372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T10:40:06.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: Standby manager daemon z restarted 2024-03-24T10:40:06.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: Standby manager daemon z started 2024-03-24T10:40:06.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.117:0/4129920998' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T10:40:06.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.117:0/4129920998' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T10:40:06.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.117:0/4129920998' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T10:40:06.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.117:0/4129920998' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T10:40:06.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: Active manager daemon x restarted 2024-03-24T10:40:06.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: Activating manager daemon x 2024-03-24T10:40:06.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T10:40:06.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: mgrmap e20: x(active, starting, since 0.0144233s), standbys: y, z 2024-03-24T10:40:06.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T10:40:06.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T10:40:06.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T10:40:06.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T10:40:06.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T10:40:06.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T10:40:06.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T10:40:06.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T10:40:06.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T10:40:06.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T10:40:06.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T10:40:06.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:40:06.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T10:40:06.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T10:40:06.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T10:40:06.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T10:40:06.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T10:40:06.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T10:40:06.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T10:40:06.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T10:40:06.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T10:40:06.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T10:40:06.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T10:40:06.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T10:40:06.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T10:40:06.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T10:40:06.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T10:40:06.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T10:40:06.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T10:40:06.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T10:40:06.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T10:40:06.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T10:40:06.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T10:40:06.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: Manager daemon x is now available 2024-03-24T10:40:06.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:40:06.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:40:06.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T10:40:06.258 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T10:40:06.258 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: Standby manager daemon y restarted 2024-03-24T10:40:06.258 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: Standby manager daemon y started 2024-03-24T10:40:06.258 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.116:0/4032755372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T10:40:06.258 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.116:0/4032755372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T10:40:06.258 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.116:0/4032755372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T10:40:06.258 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:05 smithi117 ceph-mon[36368]: from='mgr.? 172.21.15.116:0/4032755372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T10:40:06.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.309+0000 7f01e0853640 1 -- 172.21.15.23:0/2248275989 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 21) ==== 136568+0+0 (secure 0 0 0) 0x7f01cc0cbd00 con 0x7f01dc101830 2024-03-24T10:40:06.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.310+0000 7f01e0853640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f01c4051080 0x7f01c4051460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:06.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.310+0000 7f01e0853640 1 -- 172.21.15.23:0/2248275989 --> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f01dc02d2e0 con 0x7f01c4051080 2024-03-24T10:40:06.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.313+0000 7f01daffd640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f01c4051080 0x7f01c4051460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:06.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.315+0000 7f01daffd640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f01c4051080 0x7f01c4051460 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f01d0005ab0 tx=0x7f01d0005fd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:06.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.337+0000 7f01e0853640 1 -- 172.21.15.23:0/2248275989 <== mgr.15288 v2:172.21.15.23:6800/581236369 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f01dc02d2e0 con 0x7f01c4051080 2024-03-24T10:40:06.338 INFO:teuthology.orchestra.run.smithi023.stdout:{} 2024-03-24T10:40:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.339+0000 7f01e1855640 1 -- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f01c4051080 msgr2=0x7f01c4051460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.339+0000 7f01e1855640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f01c4051080 0x7f01c4051460 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f01d0005ab0 tx=0x7f01d0005fd0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.339+0000 7f01e1855640 1 -- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01dc101830 msgr2=0x7f01dc19e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.339+0000 7f01e1855640 1 --2- 172.21.15.23:0/2248275989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01dc101830 0x7f01dc19e890 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00ea60 tx=0x7f01cc00ef20 comp rx=0 tx=0).stop 2024-03-24T10:40:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.339+0000 7f01db7fe640 1 -- 172.21.15.23:0/2248275989 reap_dead start 2024-03-24T10:40:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.339+0000 7f01e1855640 1 -- 172.21.15.23:0/2248275989 shutdown_connections 2024-03-24T10:40:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.339+0000 7f01e1855640 1 -- 172.21.15.23:0/2248275989 >> 172.21.15.23:0/2248275989 conn(0x7f01dc0fbbd0 msgr2=0x7f01dc104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.339+0000 7f01e1855640 1 -- 172.21.15.23:0/2248275989 shutdown_connections 2024-03-24T10:40:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.339+0000 7f01e1855640 1 -- 172.21.15.23:0/2248275989 wait complete. 2024-03-24T10:40:06.352 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-24T10:40:06.363 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-24T10:40:06.383 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:40:06.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.488+0000 7f831d03b640 1 -- 172.21.15.23:0/1461775728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8318100630 msgr2=0x7f8318100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:06.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.488+0000 7f831d03b640 1 --2- 172.21.15.23:0/1461775728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8318100630 0x7f8318100a50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8308009ca0 tx=0x7f830802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.488+0000 7f831d03b640 1 -- 172.21.15.23:0/1461775728 shutdown_connections 2024-03-24T10:40:06.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.488+0000 7f831d03b640 1 --2- 172.21.15.23:0/1461775728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83181021e0 0x7f8318108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.488+0000 7f831d03b640 1 --2- 172.21.15.23:0/1461775728 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8318101820 0x7f8318101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.488+0000 7f831d03b640 1 --2- 172.21.15.23:0/1461775728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8318100630 0x7f8318100a50 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.488+0000 7f831d03b640 1 -- 172.21.15.23:0/1461775728 >> 172.21.15.23:0/1461775728 conn(0x7f83180fbba0 msgr2=0x7f83180fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:06.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.488+0000 7f831d03b640 1 -- 172.21.15.23:0/1461775728 shutdown_connections 2024-03-24T10:40:06.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.488+0000 7f831d03b640 1 -- 172.21.15.23:0/1461775728 wait complete. 2024-03-24T10:40:06.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.489+0000 7f831d03b640 1 Processor -- start 2024-03-24T10:40:06.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.489+0000 7f831d03b640 1 -- start start 2024-03-24T10:40:06.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.489+0000 7f831d03b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8318100630 0x7f83180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:06.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.489+0000 7f831d03b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8318101820 0x7f831819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:06.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.489+0000 7f831d03b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83181021e0 0x7f831819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:06.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.489+0000 7f831d03b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f831819f330 con 0x7f83181021e0 2024-03-24T10:40:06.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.489+0000 7f831d03b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f831819f4a0 con 0x7f8318101820 2024-03-24T10:40:06.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.489+0000 7f831d03b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f831819f660 con 0x7f8318100630 2024-03-24T10:40:06.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f831759e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83181021e0 0x7f831819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:06.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f831759e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83181021e0 0x7f831819eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38772/0 (socket says 172.21.15.23:38772) 2024-03-24T10:40:06.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f8316d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8318100630 0x7f83180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:06.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f831759e640 1 -- 172.21.15.23:0/3003658155 learned_addr learned my addr 172.21.15.23:0/3003658155 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:06.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f831659c640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8318101820 0x7f831819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:06.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f831759e640 1 -- 172.21.15.23:0/3003658155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8318100630 msgr2=0x7f83180693e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:06.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f831759e640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8318100630 0x7f83180693e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f831759e640 1 -- 172.21.15.23:0/3003658155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8318101820 msgr2=0x7f831819c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:06.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f831759e640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8318101820 0x7f831819c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f831759e640 1 -- 172.21.15.23:0/3003658155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8308009900 con 0x7f83181021e0 2024-03-24T10:40:06.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f8316d9d640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8318100630 0x7f83180693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:06.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f831659c640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8318101820 0x7f831819c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:06.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f831759e640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83181021e0 0x7f831819eb50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f831000e970 tx=0x7f831000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:06.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f830ffff640 1 -- 172.21.15.23:0/3003658155 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310009720 con 0x7f83181021e0 2024-03-24T10:40:06.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f830ffff640 1 -- 172.21.15.23:0/3003658155 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8310005500 con 0x7f83181021e0 2024-03-24T10:40:06.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f831d03b640 1 -- 172.21.15.23:0/3003658155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f831819f8e0 con 0x7f83181021e0 2024-03-24T10:40:06.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f830ffff640 1 -- 172.21.15.23:0/3003658155 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310010430 con 0x7f83181021e0 2024-03-24T10:40:06.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.490+0000 7f831d03b640 1 -- 172.21.15.23:0/3003658155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f831819fda0 con 0x7f83181021e0 2024-03-24T10:40:06.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.491+0000 7f831d03b640 1 -- 172.21.15.23:0/3003658155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83181079a0 con 0x7f83181021e0 2024-03-24T10:40:06.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.492+0000 7f830ffff640 1 -- 172.21.15.23:0/3003658155 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 21) ==== 136568+0+0 (secure 0 0 0) 0x7f8310005020 con 0x7f83181021e0 2024-03-24T10:40:06.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.493+0000 7f830ffff640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f83040a1ec0 0x7f83040a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:06.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.493+0000 7f8316d9d640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f83040a1ec0 0x7f83040a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:06.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.493+0000 7f830ffff640 1 -- 172.21.15.23:0/3003658155 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f8310014070 con 0x7f83181021e0 2024-03-24T10:40:06.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.494+0000 7f8316d9d640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f83040a1ec0 0x7f83040a4380 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f830802c630 tx=0x7f8308009800 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:06.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.498+0000 7f830ffff640 1 -- 172.21.15.23:0/3003658155 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8310096ef0 con 0x7f83181021e0 2024-03-24T10:40:06.668 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:06 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:06.322+0000 7f5c511b3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:06.669 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:06 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:06.322+0000 7f5c511b3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:06.669 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:06 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:06.322+0000 7f5c511b3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:06.669 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:06 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:06.322+0000 7f5c511b3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:06.669 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:40:06 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:40:06.322+0000 7f5c511b3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:40:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.683+0000 7f831d03b640 1 -- 172.21.15.23:0/3003658155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f831802d230 con 0x7f83181021e0 2024-03-24T10:40:06.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.685+0000 7f830ffff640 1 -- 172.21.15.23:0/3003658155 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23079 (secure 0 0 0) 0x7f831009a5b0 con 0x7f83181021e0 2024-03-24T10:40:06.686 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:40:06.686 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:40:05.291943+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2350021632":"2024-03-25T10:40:05.291912+0000","172.21.15.23:6800/2350021632":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3614005933":"2024-03-25T10:40:05.291912+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/256452041":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/352422381":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/725846898":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:40:06.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.687+0000 7f831d03b640 1 -- 172.21.15.23:0/3003658155 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f83040a1ec0 msgr2=0x7f83040a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:06.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.687+0000 7f831d03b640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f83040a1ec0 0x7f83040a4380 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f830802c630 tx=0x7f8308009800 comp rx=0 tx=0).stop 2024-03-24T10:40:06.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.687+0000 7f831d03b640 1 -- 172.21.15.23:0/3003658155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83181021e0 msgr2=0x7f831819eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:06.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.687+0000 7f831d03b640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83181021e0 0x7f831819eb50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f831000e970 tx=0x7f831000ed70 comp rx=0 tx=0).stop 2024-03-24T10:40:06.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.687+0000 7f831d03b640 1 -- 172.21.15.23:0/3003658155 shutdown_connections 2024-03-24T10:40:06.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.687+0000 7f831d03b640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f83040a1ec0 0x7f83040a4380 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.687+0000 7f831d03b640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83181021e0 0x7f831819eb50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.687+0000 7f831d03b640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8318101820 0x7f831819c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.687+0000 7f831d03b640 1 --2- 172.21.15.23:0/3003658155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8318100630 0x7f83180693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.687+0000 7f831d03b640 1 -- 172.21.15.23:0/3003658155 >> 172.21.15.23:0/3003658155 conn(0x7f83180fbba0 msgr2=0x7f8318104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:06.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.687+0000 7f831d03b640 1 -- 172.21.15.23:0/3003658155 shutdown_connections 2024-03-24T10:40:06.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.687+0000 7f831d03b640 1 -- 172.21.15.23:0/3003658155 wait complete. 2024-03-24T10:40:06.698 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:39.056452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:38:08.473472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:38:09.414667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:40:06.699 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:40:06.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.806+0000 7efe8766f640 1 -- 172.21.15.23:0/3613516240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe801021c0 msgr2=0x7efe80108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:06.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.806+0000 7efe8766f640 1 --2- 172.21.15.23:0/3613516240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe801021c0 0x7efe80108a50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7efe78009ca0 tx=0x7efe7802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.806+0000 7efe8766f640 1 -- 172.21.15.23:0/3613516240 shutdown_connections 2024-03-24T10:40:06.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.806+0000 7efe8766f640 1 --2- 172.21.15.23:0/3613516240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe801021c0 0x7efe80108a50 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.806+0000 7efe8766f640 1 --2- 172.21.15.23:0/3613516240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe80101800 0x7efe80101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.806+0000 7efe8766f640 1 --2- 172.21.15.23:0/3613516240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe80100610 0x7efe80100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.806+0000 7efe8766f640 1 -- 172.21.15.23:0/3613516240 >> 172.21.15.23:0/3613516240 conn(0x7efe800fbb80 msgr2=0x7efe800fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:06.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.806+0000 7efe8766f640 1 -- 172.21.15.23:0/3613516240 shutdown_connections 2024-03-24T10:40:06.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.806+0000 7efe8766f640 1 -- 172.21.15.23:0/3613516240 wait complete. 2024-03-24T10:40:06.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.807+0000 7efe8766f640 1 Processor -- start 2024-03-24T10:40:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.807+0000 7efe8766f640 1 -- start start 2024-03-24T10:40:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.807+0000 7efe8766f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe80100610 0x7efe80197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.807+0000 7efe8766f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe80101800 0x7efe80197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.807+0000 7efe8766f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe801021c0 0x7efe8019ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.807+0000 7efe8766f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7efe8019f250 con 0x7efe801021c0 2024-03-24T10:40:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.807+0000 7efe8766f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efe8019f3c0 con 0x7efe80100610 2024-03-24T10:40:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.807+0000 7efe8766f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efe8019f580 con 0x7efe80101800 2024-03-24T10:40:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.807+0000 7efe85c0c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe801021c0 0x7efe8019ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.807+0000 7efe85c0c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe801021c0 0x7efe8019ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38796/0 (socket says 172.21.15.23:38796) 2024-03-24T10:40:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.807+0000 7efe85c0c640 1 -- 172.21.15.23:0/2633097987 learned_addr learned my addr 172.21.15.23:0/2633097987 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.807+0000 7efe8540b640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe80100610 0x7efe80197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.807+0000 7efe84c0a640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe80101800 0x7efe80197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:06.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.808+0000 7efe85c0c640 1 -- 172.21.15.23:0/2633097987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe80101800 msgr2=0x7efe80197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.808+0000 7efe85c0c640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe80101800 0x7efe80197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.808+0000 7efe85c0c640 1 -- 172.21.15.23:0/2633097987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe80100610 msgr2=0x7efe80197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.808+0000 7efe85c0c640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe80100610 0x7efe80197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.808+0000 7efe85c0c640 1 -- 172.21.15.23:0/2633097987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe78009900 con 0x7efe801021c0 2024-03-24T10:40:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.808+0000 7efe8540b640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe80100610 0x7efe80197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.808+0000 7efe85c0c640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe801021c0 0x7efe8019ed10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efe78037040 tx=0x7efe78009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.808+0000 7efe767fc640 1 -- 172.21.15.23:0/2633097987 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe780329b0 con 0x7efe801021c0 2024-03-24T10:40:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.808+0000 7efe8766f640 1 -- 172.21.15.23:0/2633097987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe8019f800 con 0x7efe801021c0 2024-03-24T10:40:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.808+0000 7efe767fc640 1 -- 172.21.15.23:0/2633097987 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe78003e50 con 0x7efe801021c0 2024-03-24T10:40:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.808+0000 7efe767fc640 1 -- 172.21.15.23:0/2633097987 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe780048a0 con 0x7efe801021c0 2024-03-24T10:40:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.808+0000 7efe8766f640 1 -- 172.21.15.23:0/2633097987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe8019fcb0 con 0x7efe801021c0 2024-03-24T10:40:06.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.809+0000 7efe8766f640 1 -- 172.21.15.23:0/2633097987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe8010c020 con 0x7efe801021c0 2024-03-24T10:40:06.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.810+0000 7efe767fc640 1 -- 172.21.15.23:0/2633097987 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 21) ==== 136568+0+0 (secure 0 0 0) 0x7efe7800a040 con 0x7efe801021c0 2024-03-24T10:40:06.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.811+0000 7efe767fc640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7efe6c0a1ec0 0x7efe6c0a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:06.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.811+0000 7efe8540b640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7efe6c0a1ec0 0x7efe6c0a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:06.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.811+0000 7efe767fc640 1 -- 172.21.15.23:0/2633097987 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7efe780f0730 con 0x7efe801021c0 2024-03-24T10:40:06.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.811+0000 7efe8540b640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7efe6c0a1ec0 0x7efe6c0a4380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe80101660 tx=0x7efe70005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:06.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:06.816+0000 7efe767fc640 1 -- 172.21.15.23:0/2633097987 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efe780b9960 con 0x7efe801021c0 2024-03-24T10:40:07.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:06 smithi117 ceph-mon[36368]: [24/Mar/2024:10:40:05] ENGINE Bus STARTING 2024-03-24T10:40:07.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:06 smithi117 ceph-mon[36368]: [24/Mar/2024:10:40:05] ENGINE Serving on http://172.21.15.23:8765 2024-03-24T10:40:07.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:06 smithi117 ceph-mon[36368]: [24/Mar/2024:10:40:05] ENGINE Serving on https://172.21.15.23:7150 2024-03-24T10:40:07.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:06 smithi117 ceph-mon[36368]: [24/Mar/2024:10:40:05] ENGINE Bus STARTED 2024-03-24T10:40:07.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:06 smithi117 ceph-mon[36368]: mgrmap e21: x(active, since 1.02018s), standbys: y, z 2024-03-24T10:40:07.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:06 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3003658155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:07.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.017+0000 7efe8766f640 1 -- 172.21.15.23:0/2633097987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efe801a0150 con 0x7efe801021c0 2024-03-24T10:40:07.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:06 smithi116 ceph-mon[36248]: [24/Mar/2024:10:40:05] ENGINE Bus STARTING 2024-03-24T10:40:07.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:06 smithi116 ceph-mon[36248]: [24/Mar/2024:10:40:05] ENGINE Serving on http://172.21.15.23:8765 2024-03-24T10:40:07.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:06 smithi116 ceph-mon[36248]: [24/Mar/2024:10:40:05] ENGINE Serving on https://172.21.15.23:7150 2024-03-24T10:40:07.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:06 smithi116 ceph-mon[36248]: [24/Mar/2024:10:40:05] ENGINE Bus STARTED 2024-03-24T10:40:07.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:06 smithi116 ceph-mon[36248]: mgrmap e21: x(active, since 1.02018s), standbys: y, z 2024-03-24T10:40:07.019 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:06 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3003658155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:07.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.018+0000 7efe767fc640 1 -- 172.21.15.23:0/2633097987 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7efe78034e60 con 0x7efe801021c0 2024-03-24T10:40:07.019 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-24T10:40:07.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.020+0000 7efe8766f640 1 -- 172.21.15.23:0/2633097987 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7efe6c0a1ec0 msgr2=0x7efe6c0a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.020+0000 7efe8766f640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7efe6c0a1ec0 0x7efe6c0a4380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe80101660 tx=0x7efe70005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:07.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.020+0000 7efe8766f640 1 -- 172.21.15.23:0/2633097987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe801021c0 msgr2=0x7efe8019ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.020+0000 7efe8766f640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe801021c0 0x7efe8019ed10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efe78037040 tx=0x7efe78009360 comp rx=0 tx=0).stop 2024-03-24T10:40:07.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.020+0000 7efe8766f640 1 -- 172.21.15.23:0/2633097987 shutdown_connections 2024-03-24T10:40:07.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.020+0000 7efe8766f640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7efe6c0a1ec0 0x7efe6c0a4380 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.020+0000 7efe8766f640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe801021c0 0x7efe8019ed10 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.020+0000 7efe8766f640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe80101800 0x7efe80197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.020+0000 7efe8766f640 1 --2- 172.21.15.23:0/2633097987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe80100610 0x7efe80197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.020+0000 7efe8766f640 1 -- 172.21.15.23:0/2633097987 >> 172.21.15.23:0/2633097987 conn(0x7efe800fbb80 msgr2=0x7efe800fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:07.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.020+0000 7efe8766f640 1 -- 172.21.15.23:0/2633097987 shutdown_connections 2024-03-24T10:40:07.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.020+0000 7efe8766f640 1 -- 172.21.15.23:0/2633097987 wait complete. 2024-03-24T10:40:07.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:06 smithi023 ceph-mon[32466]: [24/Mar/2024:10:40:05] ENGINE Bus STARTING 2024-03-24T10:40:07.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:06 smithi023 ceph-mon[32466]: [24/Mar/2024:10:40:05] ENGINE Serving on http://172.21.15.23:8765 2024-03-24T10:40:07.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:06 smithi023 ceph-mon[32466]: [24/Mar/2024:10:40:05] ENGINE Serving on https://172.21.15.23:7150 2024-03-24T10:40:07.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:06 smithi023 ceph-mon[32466]: [24/Mar/2024:10:40:05] ENGINE Bus STARTED 2024-03-24T10:40:07.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:06 smithi023 ceph-mon[32466]: mgrmap e21: x(active, since 1.02018s), standbys: y, z 2024-03-24T10:40:07.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:06 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3003658155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:07.033 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:40:07.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.157+0000 7f88edb0c640 1 -- 172.21.15.23:0/2937739746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e8103440 msgr2=0x7f88e8109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.157+0000 7f88edb0c640 1 --2- 172.21.15.23:0/2937739746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e8103440 0x7f88e8109cd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f88e0009ca0 tx=0x7f88e002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.157+0000 7f88edb0c640 1 -- 172.21.15.23:0/2937739746 shutdown_connections 2024-03-24T10:40:07.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.157+0000 7f88edb0c640 1 --2- 172.21.15.23:0/2937739746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e8103440 0x7f88e8109cd0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.157+0000 7f88edb0c640 1 --2- 172.21.15.23:0/2937739746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88e8102a80 0x7f88e8102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.157+0000 7f88edb0c640 1 --2- 172.21.15.23:0/2937739746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e80ffa10 0x7f88e80ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.157+0000 7f88edb0c640 1 -- 172.21.15.23:0/2937739746 >> 172.21.15.23:0/2937739746 conn(0x7f88e80fb300 msgr2=0x7f88e80fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:07.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.157+0000 7f88edb0c640 1 -- 172.21.15.23:0/2937739746 shutdown_connections 2024-03-24T10:40:07.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.157+0000 7f88edb0c640 1 -- 172.21.15.23:0/2937739746 wait complete. 2024-03-24T10:40:07.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.157+0000 7f88edb0c640 1 Processor -- start 2024-03-24T10:40:07.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88edb0c640 1 -- start start 2024-03-24T10:40:07.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88edb0c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e80ffa10 0x7f88e8197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:07.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88edb0c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88e8102a80 0x7f88e8197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:07.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88edb0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e8103440 0x7f88e819ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:07.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88edb0c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f88e819f270 con 0x7f88e80ffa10 2024-03-24T10:40:07.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88edb0c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f88e819f3e0 con 0x7f88e8103440 2024-03-24T10:40:07.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88edb0c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88e819f5a0 con 0x7f88e8102a80 2024-03-24T10:40:07.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88e77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e80ffa10 0x7f88e8197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:07.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88e77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e80ffa10 0x7f88e8197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38820/0 (socket says 172.21.15.23:38820) 2024-03-24T10:40:07.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88e77fe640 1 -- 172.21.15.23:0/26301998 learned_addr learned my addr 172.21.15.23:0/26301998 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:07.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88e6ffd640 1 --2- 172.21.15.23:0/26301998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88e8102a80 0x7f88e8197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:07.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88e7fff640 1 --2- 172.21.15.23:0/26301998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e8103440 0x7f88e819ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:07.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88e77fe640 1 -- 172.21.15.23:0/26301998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88e8102a80 msgr2=0x7f88e8197cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88e77fe640 1 --2- 172.21.15.23:0/26301998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88e8102a80 0x7f88e8197cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88e77fe640 1 -- 172.21.15.23:0/26301998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e8103440 msgr2=0x7f88e819ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88e77fe640 1 --2- 172.21.15.23:0/26301998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e8103440 0x7f88e819ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88e77fe640 1 -- 172.21.15.23:0/26301998 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e0009900 con 0x7f88e80ffa10 2024-03-24T10:40:07.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.158+0000 7f88e77fe640 1 --2- 172.21.15.23:0/26301998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e80ffa10 0x7f88e8197780 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f88d800d950 tx=0x7f88d800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:07.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.159+0000 7f88e4ff9640 1 -- 172.21.15.23:0/26301998 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d8009c30 con 0x7f88e80ffa10 2024-03-24T10:40:07.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.159+0000 7f88edb0c640 1 -- 172.21.15.23:0/26301998 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88e819f880 con 0x7f88e80ffa10 2024-03-24T10:40:07.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.159+0000 7f88e4ff9640 1 -- 172.21.15.23:0/26301998 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88d8009d90 con 0x7f88e80ffa10 2024-03-24T10:40:07.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.159+0000 7f88e4ff9640 1 -- 172.21.15.23:0/26301998 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d800f850 con 0x7f88e80ffa10 2024-03-24T10:40:07.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.159+0000 7f88edb0c640 1 -- 172.21.15.23:0/26301998 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88e819fd90 con 0x7f88e80ffa10 2024-03-24T10:40:07.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.160+0000 7f88edb0c640 1 -- 172.21.15.23:0/26301998 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88e8108bb0 con 0x7f88e80ffa10 2024-03-24T10:40:07.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.161+0000 7f88e4ff9640 1 -- 172.21.15.23:0/26301998 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 21) ==== 136568+0+0 (secure 0 0 0) 0x7f88d800f9b0 con 0x7f88e80ffa10 2024-03-24T10:40:07.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.162+0000 7f88e4ff9640 1 --2- 172.21.15.23:0/26301998 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f88d00a1ec0 0x7f88d00a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:07.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.162+0000 7f88e6ffd640 1 --2- 172.21.15.23:0/26301998 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f88d00a1ec0 0x7f88d00a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:07.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.162+0000 7f88e4ff9640 1 -- 172.21.15.23:0/26301998 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f88d80cd920 con 0x7f88e80ffa10 2024-03-24T10:40:07.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.163+0000 7f88e6ffd640 1 --2- 172.21.15.23:0/26301998 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f88d00a1ec0 0x7f88d00a4380 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f88e8198c90 tx=0x7f88dc005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:07.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.166+0000 7f88e4ff9640 1 -- 172.21.15.23:0/26301998 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f88d8096ac0 con 0x7f88e80ffa10 2024-03-24T10:40:07.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.376+0000 7f88edb0c640 1 -- 172.21.15.23:0/26301998 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f88e810d140 con 0x7f88e80ffa10 2024-03-24T10:40:07.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.376+0000 7f88e4ff9640 1 -- 172.21.15.23:0/26301998 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f88d809a180 con 0x7f88e80ffa10 2024-03-24T10:40:07.377 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-24T10:40:07.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.378+0000 7f88edb0c640 1 -- 172.21.15.23:0/26301998 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f88d00a1ec0 msgr2=0x7f88d00a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.378+0000 7f88edb0c640 1 --2- 172.21.15.23:0/26301998 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f88d00a1ec0 0x7f88d00a4380 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f88e8198c90 tx=0x7f88dc005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:07.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.378+0000 7f88edb0c640 1 -- 172.21.15.23:0/26301998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e80ffa10 msgr2=0x7f88e8197780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.378+0000 7f88edb0c640 1 --2- 172.21.15.23:0/26301998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e80ffa10 0x7f88e8197780 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f88d800d950 tx=0x7f88d800de10 comp rx=0 tx=0).stop 2024-03-24T10:40:07.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.379+0000 7f88edb0c640 1 -- 172.21.15.23:0/26301998 shutdown_connections 2024-03-24T10:40:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.379+0000 7f88edb0c640 1 --2- 172.21.15.23:0/26301998 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f88d00a1ec0 0x7f88d00a4380 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.379+0000 7f88edb0c640 1 --2- 172.21.15.23:0/26301998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88e8103440 0x7f88e819ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.379+0000 7f88edb0c640 1 --2- 172.21.15.23:0/26301998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88e8102a80 0x7f88e8197cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.379+0000 7f88edb0c640 1 --2- 172.21.15.23:0/26301998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e80ffa10 0x7f88e8197780 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.379+0000 7f88edb0c640 1 -- 172.21.15.23:0/26301998 >> 172.21.15.23:0/26301998 conn(0x7f88e80fb300 msgr2=0x7f88e8105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.379+0000 7f88edb0c640 1 -- 172.21.15.23:0/26301998 shutdown_connections 2024-03-24T10:40:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.379+0000 7f88edb0c640 1 -- 172.21.15.23:0/26301998 wait complete. 2024-03-24T10:40:07.393 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:40:07.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.508+0000 7f811ddc9640 1 -- 172.21.15.23:0/3091885639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8118101800 msgr2=0x7f8118101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.508+0000 7f811ddc9640 1 --2- 172.21.15.23:0/3091885639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8118101800 0x7f8118101c80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f810c009900 tx=0x7f810c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:07.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.508+0000 7f811ddc9640 1 -- 172.21.15.23:0/3091885639 shutdown_connections 2024-03-24T10:40:07.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.508+0000 7f811ddc9640 1 --2- 172.21.15.23:0/3091885639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81181021c0 0x7f8118108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.508+0000 7f811ddc9640 1 --2- 172.21.15.23:0/3091885639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8118101800 0x7f8118101c80 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.508+0000 7f811ddc9640 1 --2- 172.21.15.23:0/3091885639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8118100610 0x7f8118100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.508+0000 7f811ddc9640 1 -- 172.21.15.23:0/3091885639 >> 172.21.15.23:0/3091885639 conn(0x7f81180fbb80 msgr2=0x7f81180fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:07.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.508+0000 7f811ddc9640 1 -- 172.21.15.23:0/3091885639 shutdown_connections 2024-03-24T10:40:07.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.508+0000 7f811ddc9640 1 -- 172.21.15.23:0/3091885639 wait complete. 2024-03-24T10:40:07.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.508+0000 7f811ddc9640 1 Processor -- start 2024-03-24T10:40:07.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.508+0000 7f811ddc9640 1 -- start start 2024-03-24T10:40:07.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f811ddc9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8118100610 0x7f8118197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:07.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f811ddc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8118101800 0x7f8118197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:07.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f811ddc9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81181021c0 0x7f811819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:07.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f811ddc9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f811819f2b0 con 0x7f81181021c0 2024-03-24T10:40:07.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f811ddc9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f811819f420 con 0x7f8118101800 2024-03-24T10:40:07.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f811ddc9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f811819f5e0 con 0x7f8118100610 2024-03-24T10:40:07.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f8117fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81181021c0 0x7f811819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:07.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f8117fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81181021c0 0x7f811819ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38862/0 (socket says 172.21.15.23:38862) 2024-03-24T10:40:07.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f8117fff640 1 -- 172.21.15.23:0/2389529450 learned_addr learned my addr 172.21.15.23:0/2389529450 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:07.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f8116ffd640 1 --2- 172.21.15.23:0/2389529450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8118101800 0x7f8118197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:07.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f81177fe640 1 --2- 172.21.15.23:0/2389529450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8118100610 0x7f8118197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:07.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f8117fff640 1 -- 172.21.15.23:0/2389529450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8118100610 msgr2=0x7f8118197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f8117fff640 1 --2- 172.21.15.23:0/2389529450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8118100610 0x7f8118197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f8117fff640 1 -- 172.21.15.23:0/2389529450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8118101800 msgr2=0x7f8118197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f8117fff640 1 --2- 172.21.15.23:0/2389529450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8118101800 0x7f8118197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f8117fff640 1 -- 172.21.15.23:0/2389529450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f810c009560 con 0x7f81181021c0 2024-03-24T10:40:07.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f8117fff640 1 --2- 172.21.15.23:0/2389529450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81181021c0 0x7f811819ed70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f811000eb10 tx=0x7f811000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:07.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f8114ff9640 1 -- 172.21.15.23:0/2389529450 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110004b20 con 0x7f81181021c0 2024-03-24T10:40:07.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f811ddc9640 1 -- 172.21.15.23:0/2389529450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811819f8c0 con 0x7f81181021c0 2024-03-24T10:40:07.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f8114ff9640 1 -- 172.21.15.23:0/2389529450 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8110004c80 con 0x7f81181021c0 2024-03-24T10:40:07.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.509+0000 7f8114ff9640 1 -- 172.21.15.23:0/2389529450 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110010650 con 0x7f81181021c0 2024-03-24T10:40:07.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.510+0000 7f811ddc9640 1 -- 172.21.15.23:0/2389529450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f811819fdf0 con 0x7f81181021c0 2024-03-24T10:40:07.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.511+0000 7f811ddc9640 1 -- 172.21.15.23:0/2389529450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f811810c070 con 0x7f81181021c0 2024-03-24T10:40:07.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.512+0000 7f8114ff9640 1 -- 172.21.15.23:0/2389529450 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 21) ==== 136568+0+0 (secure 0 0 0) 0x7f8110004df0 con 0x7f81181021c0 2024-03-24T10:40:07.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.513+0000 7f8114ff9640 1 --2- 172.21.15.23:0/2389529450 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f81000a1ec0 0x7f81000a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:07.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.513+0000 7f81177fe640 1 --2- 172.21.15.23:0/2389529450 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f81000a1ec0 0x7f81000a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:07.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.513+0000 7f8114ff9640 1 -- 172.21.15.23:0/2389529450 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f8110014070 con 0x7f81181021c0 2024-03-24T10:40:07.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.513+0000 7f81177fe640 1 --2- 172.21.15.23:0/2389529450 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f81000a1ec0 0x7f81000a4380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8118101660 tx=0x7f8108005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:07.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.516+0000 7f8114ff9640 1 -- 172.21.15.23:0/2389529450 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8110097290 con 0x7f81181021c0 2024-03-24T10:40:07.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.700+0000 7f811ddc9640 1 -- 172.21.15.23:0/2389529450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f811810c200 con 0x7f81181021c0 2024-03-24T10:40:07.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.701+0000 7f8114ff9640 1 -- 172.21.15.23:0/2389529450 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f811009a950 con 0x7f81181021c0 2024-03-24T10:40:07.702 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-24T10:40:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.703+0000 7f811ddc9640 1 -- 172.21.15.23:0/2389529450 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f81000a1ec0 msgr2=0x7f81000a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.703+0000 7f811ddc9640 1 --2- 172.21.15.23:0/2389529450 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f81000a1ec0 0x7f81000a4380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8118101660 tx=0x7f8108005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.703+0000 7f811ddc9640 1 -- 172.21.15.23:0/2389529450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81181021c0 msgr2=0x7f811819ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.703+0000 7f811ddc9640 1 --2- 172.21.15.23:0/2389529450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81181021c0 0x7f811819ed70 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f811000eb10 tx=0x7f811000c2d0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.703+0000 7f811ddc9640 1 -- 172.21.15.23:0/2389529450 shutdown_connections 2024-03-24T10:40:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.703+0000 7f811ddc9640 1 --2- 172.21.15.23:0/2389529450 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f81000a1ec0 0x7f81000a4380 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.703+0000 7f811ddc9640 1 --2- 172.21.15.23:0/2389529450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81181021c0 0x7f811819ed70 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.703+0000 7f811ddc9640 1 --2- 172.21.15.23:0/2389529450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8118101800 0x7f8118197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.703+0000 7f811ddc9640 1 --2- 172.21.15.23:0/2389529450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8118100610 0x7f8118197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.703+0000 7f811ddc9640 1 -- 172.21.15.23:0/2389529450 >> 172.21.15.23:0/2389529450 conn(0x7f81180fbb80 msgr2=0x7f81180fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.703+0000 7f811ddc9640 1 -- 172.21.15.23:0/2389529450 shutdown_connections 2024-03-24T10:40:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.703+0000 7f811ddc9640 1 -- 172.21.15.23:0/2389529450 wait complete. 2024-03-24T10:40:07.714 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:40:07.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.820+0000 7f0af0747640 1 -- 172.21.15.23:0/2813280037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8100980 msgr2=0x7f0ae8100de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.820+0000 7f0af0747640 1 --2- 172.21.15.23:0/2813280037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8100980 0x7f0ae8100de0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8009900 tx=0x7f0ad802bcd0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.820+0000 7f0af0747640 1 -- 172.21.15.23:0/2813280037 shutdown_connections 2024-03-24T10:40:07.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.820+0000 7f0af0747640 1 --2- 172.21.15.23:0/2813280037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae8101320 0x7f0ae81017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.820+0000 7f0af0747640 1 --2- 172.21.15.23:0/2813280037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8100980 0x7f0ae8100de0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.820+0000 7f0af0747640 1 --2- 172.21.15.23:0/2813280037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae8100020 0x7f0ae8100440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.820+0000 7f0af0747640 1 -- 172.21.15.23:0/2813280037 >> 172.21.15.23:0/2813280037 conn(0x7f0ae80fbba0 msgr2=0x7f0ae80fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:07.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.821+0000 7f0af0747640 1 -- 172.21.15.23:0/2813280037 shutdown_connections 2024-03-24T10:40:07.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.821+0000 7f0af0747640 1 -- 172.21.15.23:0/2813280037 wait complete. 2024-03-24T10:40:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.821+0000 7f0af0747640 1 Processor -- start 2024-03-24T10:40:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.821+0000 7f0af0747640 1 -- start start 2024-03-24T10:40:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0af0747640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8100020 0x7f0ae81999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0af0747640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae8100980 0x7f0ae8199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0af0747640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae8101320 0x7f0ae81a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0af0747640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae81a14d0 con 0x7f0ae8100020 2024-03-24T10:40:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0af0747640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae81a1640 con 0x7f0ae8101320 2024-03-24T10:40:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0af0747640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae81a1800 con 0x7f0ae8100980 2024-03-24T10:40:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0aee4e3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8100020 0x7f0ae81999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0aee4e3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8100020 0x7f0ae81999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33386/0 (socket says 172.21.15.23:33386) 2024-03-24T10:40:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0aee4e3640 1 -- 172.21.15.23:0/582031112 learned_addr learned my addr 172.21.15.23:0/582031112 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0aeece4640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae8101320 0x7f0ae81a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0aedce2640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae8100980 0x7f0ae8199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0aee4e3640 1 -- 172.21.15.23:0/582031112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae8100980 msgr2=0x7f0ae8199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0aee4e3640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae8100980 0x7f0ae8199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0aee4e3640 1 -- 172.21.15.23:0/582031112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae8101320 msgr2=0x7f0ae81a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0aee4e3640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae8101320 0x7f0ae81a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0aee4e3640 1 -- 172.21.15.23:0/582031112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae4009490 con 0x7f0ae8100020 2024-03-24T10:40:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0aedce2640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae8100980 0x7f0ae8199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:40:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0aee4e3640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8100020 0x7f0ae81999e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400ea60 tx=0x7f0ae400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0adf7fe640 1 -- 172.21.15.23:0/582031112 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae400cde0 con 0x7f0ae8100020 2024-03-24T10:40:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0adf7fe640 1 -- 172.21.15.23:0/582031112 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ae40035f0 con 0x7f0ae8100020 2024-03-24T10:40:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0adf7fe640 1 -- 172.21.15.23:0/582031112 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4010690 con 0x7f0ae8100020 2024-03-24T10:40:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0af0747640 1 -- 172.21.15.23:0/582031112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ad8009560 con 0x7f0ae8100020 2024-03-24T10:40:07.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.822+0000 7f0af0747640 1 -- 172.21.15.23:0/582031112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae81a1e60 con 0x7f0ae8100020 2024-03-24T10:40:07.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.823+0000 7f0af0747640 1 -- 172.21.15.23:0/582031112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae810af60 con 0x7f0ae8100020 2024-03-24T10:40:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.825+0000 7f0adf7fe640 1 -- 172.21.15.23:0/582031112 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 21) ==== 136568+0+0 (secure 0 0 0) 0x7f0ae4003af0 con 0x7f0ae8100020 2024-03-24T10:40:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.825+0000 7f0adf7fe640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0ad40a1ec0 0x7f0ad40a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.825+0000 7f0aedce2640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0ad40a1ec0 0x7f0ad40a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.826+0000 7f0adf7fe640 1 -- 172.21.15.23:0/582031112 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0ae4014070 con 0x7f0ae8100020 2024-03-24T10:40:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.826+0000 7f0aedce2640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0ad40a1ec0 0x7f0ad40a4380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0ae819aef0 tx=0x7f0ad802c880 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:07.830+0000 7f0adf7fe640 1 -- 172.21.15.23:0/582031112 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0ae4097070 con 0x7f0ae8100020 2024-03-24T10:40:08.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.016+0000 7f0af0747640 1 -- 172.21.15.23:0/582031112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0ae810f310 con 0x7f0ae8100020 2024-03-24T10:40:08.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:07 smithi116 ceph-mon[36248]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:08.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:07 smithi116 ceph-mon[36248]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:40:08.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:07 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2633097987' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:40:08.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:07 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/26301998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:40:08.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:07 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:07 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:07 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:07 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:07 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:07 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:07 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2389529450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:40:08.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.018+0000 7f0adf7fe640 1 -- 172.21.15.23:0/582031112 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23079 (secure 0 0 0) 0x7f0ae409a730 con 0x7f0ae8100020 2024-03-24T10:40:08.019 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:40:08.019 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:40:05.291943+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2350021632":"2024-03-25T10:40:05.291912+0000","172.21.15.23:6800/2350021632":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3614005933":"2024-03-25T10:40:05.291912+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/256452041":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/352422381":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/725846898":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:40:08.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.020+0000 7f0af0747640 1 -- 172.21.15.23:0/582031112 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0ad40a1ec0 msgr2=0x7f0ad40a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.020+0000 7f0af0747640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0ad40a1ec0 0x7f0ad40a4380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0ae819aef0 tx=0x7f0ad802c880 comp rx=0 tx=0).stop 2024-03-24T10:40:08.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.020+0000 7f0af0747640 1 -- 172.21.15.23:0/582031112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8100020 msgr2=0x7f0ae81999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.020+0000 7f0af0747640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8100020 0x7f0ae81999e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400ea60 tx=0x7f0ae400ef20 comp rx=0 tx=0).stop 2024-03-24T10:40:08.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.020+0000 7f0af0747640 1 -- 172.21.15.23:0/582031112 shutdown_connections 2024-03-24T10:40:08.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.020+0000 7f0af0747640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0ad40a1ec0 0x7f0ad40a4380 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.020+0000 7f0af0747640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ae8101320 0x7f0ae81a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.020+0000 7f0af0747640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae8100980 0x7f0ae8199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.020+0000 7f0af0747640 1 --2- 172.21.15.23:0/582031112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae8100020 0x7f0ae81999e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.020+0000 7f0af0747640 1 -- 172.21.15.23:0/582031112 >> 172.21.15.23:0/582031112 conn(0x7f0ae80fbba0 msgr2=0x7f0ae80fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:08.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.020+0000 7f0af0747640 1 -- 172.21.15.23:0/582031112 shutdown_connections 2024-03-24T10:40:08.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.020+0000 7f0af0747640 1 -- 172.21.15.23:0/582031112 wait complete. 2024-03-24T10:40:08.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:07 smithi023 ceph-mon[32466]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:08.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:07 smithi023 ceph-mon[32466]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:40:08.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:07 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2633097987' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:40:08.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:07 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/26301998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:40:08.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:07 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:07 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:07 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:07 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:07 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:07 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:07 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2389529450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:40:08.033 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:39.056452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:38:08.473472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:38:09.414667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:40:08.034 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:40:08.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.141+0000 7fd712c6f640 1 -- 172.21.15.23:0/3159669083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd70c104b20 msgr2=0x7fd70c106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.141+0000 7fd712c6f640 1 --2- 172.21.15.23:0/3159669083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd70c104b20 0x7fd70c106f10 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009900 tx=0x7fd6fc02bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:08.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.141+0000 7fd712c6f640 1 -- 172.21.15.23:0/3159669083 shutdown_connections 2024-03-24T10:40:08.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.141+0000 7fd712c6f640 1 --2- 172.21.15.23:0/3159669083 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd70c107450 0x7fd70c109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.141+0000 7fd712c6f640 1 --2- 172.21.15.23:0/3159669083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd70c104b20 0x7fd70c106f10 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.141+0000 7fd712c6f640 1 --2- 172.21.15.23:0/3159669083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd70c1021f0 0x7fd70c1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.141+0000 7fd712c6f640 1 -- 172.21.15.23:0/3159669083 >> 172.21.15.23:0/3159669083 conn(0x7fd70c0fbb80 msgr2=0x7fd70c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:08.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.141+0000 7fd712c6f640 1 -- 172.21.15.23:0/3159669083 shutdown_connections 2024-03-24T10:40:08.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.141+0000 7fd712c6f640 1 -- 172.21.15.23:0/3159669083 wait complete. 2024-03-24T10:40:08.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.142+0000 7fd712c6f640 1 Processor -- start 2024-03-24T10:40:08.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.142+0000 7fd712c6f640 1 -- start start 2024-03-24T10:40:08.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.142+0000 7fd712c6f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd70c1021f0 0x7fd70c1999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.142+0000 7fd712c6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd70c104b20 0x7fd70c199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.142+0000 7fd712c6f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd70c107450 0x7fd70c1a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.142+0000 7fd712c6f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c1a14c0 con 0x7fd70c1021f0 2024-03-24T10:40:08.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.142+0000 7fd712c6f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c1a1630 con 0x7fd70c104b20 2024-03-24T10:40:08.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.142+0000 7fd712c6f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c1a17f0 con 0x7fd70c107450 2024-03-24T10:40:08.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.142+0000 7fd710a0b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd70c1021f0 0x7fd70c1999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.142+0000 7fd710a0b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd70c1021f0 0x7fd70c1999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33392/0 (socket says 172.21.15.23:33392) 2024-03-24T10:40:08.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.142+0000 7fd710a0b640 1 -- 172.21.15.23:0/907604594 learned_addr learned my addr 172.21.15.23:0/907604594 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:08.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.142+0000 7fd703fff640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd70c104b20 0x7fd70c199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.142+0000 7fd71120c640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd70c107450 0x7fd70c1a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.143+0000 7fd710a0b640 1 -- 172.21.15.23:0/907604594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd70c107450 msgr2=0x7fd70c1a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.143+0000 7fd710a0b640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd70c107450 0x7fd70c1a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.143+0000 7fd710a0b640 1 -- 172.21.15.23:0/907604594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd70c104b20 msgr2=0x7fd70c199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.143+0000 7fd710a0b640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd70c104b20 0x7fd70c199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.143+0000 7fd710a0b640 1 -- 172.21.15.23:0/907604594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6fc009560 con 0x7fd70c1021f0 2024-03-24T10:40:08.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.143+0000 7fd703fff640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd70c104b20 0x7fd70c199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:08.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.143+0000 7fd710a0b640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd70c1021f0 0x7fd70c1999d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd70800e9f0 tx=0x7fd70800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:08.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.143+0000 7fd701ffb640 1 -- 172.21.15.23:0/907604594 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd708009700 con 0x7fd70c1021f0 2024-03-24T10:40:08.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.143+0000 7fd701ffb640 1 -- 172.21.15.23:0/907604594 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd70800cd80 con 0x7fd70c1021f0 2024-03-24T10:40:08.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.143+0000 7fd712c6f640 1 -- 172.21.15.23:0/907604594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd70c1a1ad0 con 0x7fd70c1021f0 2024-03-24T10:40:08.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.143+0000 7fd701ffb640 1 -- 172.21.15.23:0/907604594 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd708018920 con 0x7fd70c1021f0 2024-03-24T10:40:08.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.143+0000 7fd712c6f640 1 -- 172.21.15.23:0/907604594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd70c1a1fb0 con 0x7fd70c1021f0 2024-03-24T10:40:08.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.144+0000 7fd712c6f640 1 -- 172.21.15.23:0/907604594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd70c108f40 con 0x7fd70c1021f0 2024-03-24T10:40:08.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.146+0000 7fd701ffb640 1 -- 172.21.15.23:0/907604594 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 21) ==== 136568+0+0 (secure 0 0 0) 0x7fd708018a80 con 0x7fd70c1021f0 2024-03-24T10:40:08.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.146+0000 7fd701ffb640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd6f40a1ec0 0x7fd6f40a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.147+0000 7fd701ffb640 1 -- 172.21.15.23:0/907604594 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fd708014070 con 0x7fd70c1021f0 2024-03-24T10:40:08.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.147+0000 7fd703fff640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd6f40a1ec0 0x7fd6f40a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.147+0000 7fd703fff640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd6f40a1ec0 0x7fd6f40a4380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc005ab0 tx=0x7fd6fc005fd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:08.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.150+0000 7fd701ffb640 1 -- 172.21.15.23:0/907604594 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd708097420 con 0x7fd70c1021f0 2024-03-24T10:40:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:07 smithi117 ceph-mon[36368]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:07 smithi117 ceph-mon[36368]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:40:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:07 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2633097987' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:40:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:07 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/26301998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:40:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:07 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:07 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:07 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:07 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:07 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:07 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:08.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:07 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2389529450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:40:08.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.317+0000 7fd701ffb640 1 -- 172.21.15.23:0/907604594 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 22) ==== 136927+0+0 (secure 0 0 0) 0x7fd7080977f0 con 0x7fd70c1021f0 2024-03-24T10:40:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.339+0000 7fd712c6f640 1 -- 172.21.15.23:0/907604594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd70c02d270 con 0x7fd70c1021f0 2024-03-24T10:40:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.340+0000 7fd701ffb640 1 -- 172.21.15.23:0/907604594 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fd708090e60 con 0x7fd70c1021f0 2024-03-24T10:40:08.340 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-24T10:40:08.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.342+0000 7fd712c6f640 1 -- 172.21.15.23:0/907604594 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd6f40a1ec0 msgr2=0x7fd6f40a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.342+0000 7fd712c6f640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd6f40a1ec0 0x7fd6f40a4380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc005ab0 tx=0x7fd6fc005fd0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.342+0000 7fd712c6f640 1 -- 172.21.15.23:0/907604594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd70c1021f0 msgr2=0x7fd70c1999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.342+0000 7fd712c6f640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd70c1021f0 0x7fd70c1999d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd70800e9f0 tx=0x7fd70800eeb0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.342+0000 7fd712c6f640 1 -- 172.21.15.23:0/907604594 shutdown_connections 2024-03-24T10:40:08.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.342+0000 7fd712c6f640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd6f40a1ec0 0x7fd6f40a4380 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.342+0000 7fd712c6f640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd70c107450 0x7fd70c1a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.342+0000 7fd712c6f640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd70c104b20 0x7fd70c199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.342+0000 7fd712c6f640 1 --2- 172.21.15.23:0/907604594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd70c1021f0 0x7fd70c1999d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.342+0000 7fd712c6f640 1 -- 172.21.15.23:0/907604594 >> 172.21.15.23:0/907604594 conn(0x7fd70c0fbb80 msgr2=0x7fd70c101970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:08.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.342+0000 7fd712c6f640 1 -- 172.21.15.23:0/907604594 shutdown_connections 2024-03-24T10:40:08.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.342+0000 7fd712c6f640 1 -- 172.21.15.23:0/907604594 wait complete. 2024-03-24T10:40:08.354 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:40:08.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.461+0000 7fd5e4d13640 1 -- 172.21.15.23:0/1652467898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e01021c0 msgr2=0x7fd5e0108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.461+0000 7fd5e4d13640 1 --2- 172.21.15.23:0/1652467898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e01021c0 0x7fd5e0108a50 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8009ca0 tx=0x7fd5d802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.461+0000 7fd5e4d13640 1 -- 172.21.15.23:0/1652467898 shutdown_connections 2024-03-24T10:40:08.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.461+0000 7fd5e4d13640 1 --2- 172.21.15.23:0/1652467898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e01021c0 0x7fd5e0108a50 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.461+0000 7fd5e4d13640 1 --2- 172.21.15.23:0/1652467898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5e0101800 0x7fd5e0101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.461+0000 7fd5e4d13640 1 --2- 172.21.15.23:0/1652467898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5e0100610 0x7fd5e0100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.461+0000 7fd5e4d13640 1 -- 172.21.15.23:0/1652467898 >> 172.21.15.23:0/1652467898 conn(0x7fd5e00fbba0 msgr2=0x7fd5e00fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.461+0000 7fd5e4d13640 1 -- 172.21.15.23:0/1652467898 shutdown_connections 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.461+0000 7fd5e4d13640 1 -- 172.21.15.23:0/1652467898 wait complete. 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.461+0000 7fd5e4d13640 1 Processor -- start 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.461+0000 7fd5e4d13640 1 -- start start 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5e4d13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5e0100610 0x7fd5e01999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5e4d13640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e0101800 0x7fd5e0199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5e4d13640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5e01021c0 0x7fd5e01a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5e4d13640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e01a14e0 con 0x7fd5e0101800 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5e4d13640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e01a1650 con 0x7fd5e0100610 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5e4d13640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e01a1810 con 0x7fd5e01021c0 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5ddd9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e0101800 0x7fd5e0199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5ddd9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e0101800 0x7fd5e0199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33406/0 (socket says 172.21.15.23:33406) 2024-03-24T10:40:08.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5ddd9b640 1 -- 172.21.15.23:0/4191044927 learned_addr learned my addr 172.21.15.23:0/4191044927 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:08.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5de59c640 1 --2- 172.21.15.23:0/4191044927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5e0100610 0x7fd5e01999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5ded9d640 1 --2- 172.21.15.23:0/4191044927 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5e01021c0 0x7fd5e01a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5ddd9b640 1 -- 172.21.15.23:0/4191044927 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5e01021c0 msgr2=0x7fd5e01a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5ddd9b640 1 --2- 172.21.15.23:0/4191044927 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5e01021c0 0x7fd5e01a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5ddd9b640 1 -- 172.21.15.23:0/4191044927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5e0100610 msgr2=0x7fd5e01999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5ddd9b640 1 --2- 172.21.15.23:0/4191044927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5e0100610 0x7fd5e01999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5ddd9b640 1 -- 172.21.15.23:0/4191044927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d8009900 con 0x7fd5e0101800 2024-03-24T10:40:08.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5ddd9b640 1 --2- 172.21.15.23:0/4191044927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e0101800 0x7fd5e0199f30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000e9f0 tx=0x7fd5d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:08.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5cf7fe640 1 -- 172.21.15.23:0/4191044927 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0009720 con 0x7fd5e0101800 2024-03-24T10:40:08.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5e4d13640 1 -- 172.21.15.23:0/4191044927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e01a1af0 con 0x7fd5e0101800 2024-03-24T10:40:08.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5cf7fe640 1 -- 172.21.15.23:0/4191044927 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5d000cd80 con 0x7fd5e0101800 2024-03-24T10:40:08.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.462+0000 7fd5cf7fe640 1 -- 172.21.15.23:0/4191044927 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0010690 con 0x7fd5e0101800 2024-03-24T10:40:08.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.463+0000 7fd5e4d13640 1 -- 172.21.15.23:0/4191044927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e01a2000 con 0x7fd5e0101800 2024-03-24T10:40:08.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.463+0000 7fd5e4d13640 1 -- 172.21.15.23:0/4191044927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5e010c020 con 0x7fd5e0101800 2024-03-24T10:40:08.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.465+0000 7fd5cf7fe640 1 -- 172.21.15.23:0/4191044927 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 22) ==== 136927+0+0 (secure 0 0 0) 0x7fd5d00046c0 con 0x7fd5e0101800 2024-03-24T10:40:08.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.465+0000 7fd5cf7fe640 1 --2- 172.21.15.23:0/4191044927 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd5c80a22c0 0x7fd5c80a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.465+0000 7fd5de59c640 1 --2- 172.21.15.23:0/4191044927 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd5c80a22c0 0x7fd5c80a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.466+0000 7fd5cf7fe640 1 -- 172.21.15.23:0/4191044927 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fd5d0014070 con 0x7fd5e0101800 2024-03-24T10:40:08.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.466+0000 7fd5de59c640 1 --2- 172.21.15.23:0/4191044927 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd5c80a22c0 0x7fd5c80a4780 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0101660 tx=0x7fd5d4005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:08.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.473+0000 7fd5cf7fe640 1 -- 172.21.15.23:0/4191044927 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd5d00972a0 con 0x7fd5e0101800 2024-03-24T10:40:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.664+0000 7fd5e4d13640 1 -- 172.21.15.23:0/4191044927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd5e010c1b0 con 0x7fd5e0101800 2024-03-24T10:40:08.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.664+0000 7fd5cf7fe640 1 -- 172.21.15.23:0/4191044927 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fd5d009a960 con 0x7fd5e0101800 2024-03-24T10:40:08.665 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-24T10:40:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.668+0000 7fd5e4d13640 1 -- 172.21.15.23:0/4191044927 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd5c80a22c0 msgr2=0x7fd5c80a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.668+0000 7fd5e4d13640 1 --2- 172.21.15.23:0/4191044927 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd5c80a22c0 0x7fd5c80a4780 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0101660 tx=0x7fd5d4005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.668+0000 7fd5e4d13640 1 -- 172.21.15.23:0/4191044927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e0101800 msgr2=0x7fd5e0199f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.668+0000 7fd5e4d13640 1 --2- 172.21.15.23:0/4191044927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e0101800 0x7fd5e0199f30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000e9f0 tx=0x7fd5d000edf0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.668+0000 7fd5e4d13640 1 -- 172.21.15.23:0/4191044927 shutdown_connections 2024-03-24T10:40:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.668+0000 7fd5e4d13640 1 --2- 172.21.15.23:0/4191044927 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd5c80a22c0 0x7fd5c80a4780 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.668+0000 7fd5e4d13640 1 --2- 172.21.15.23:0/4191044927 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5e01021c0 0x7fd5e01a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.668+0000 7fd5e4d13640 1 --2- 172.21.15.23:0/4191044927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e0101800 0x7fd5e0199f30 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.668+0000 7fd5e4d13640 1 --2- 172.21.15.23:0/4191044927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5e0100610 0x7fd5e01999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.668+0000 7fd5e4d13640 1 -- 172.21.15.23:0/4191044927 >> 172.21.15.23:0/4191044927 conn(0x7fd5e00fbba0 msgr2=0x7fd5e00fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.668+0000 7fd5e4d13640 1 -- 172.21.15.23:0/4191044927 shutdown_connections 2024-03-24T10:40:08.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.668+0000 7fd5e4d13640 1 -- 172.21.15.23:0/4191044927 wait complete. 2024-03-24T10:40:08.682 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:40:08.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.794+0000 7fa95baf1640 1 -- 172.21.15.23:0/1202292774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa954103340 msgr2=0x7fa954109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.794+0000 7fa95baf1640 1 --2- 172.21.15.23:0/1202292774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa954103340 0x7fa954109bd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa94c009ca0 tx=0x7fa94c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.794+0000 7fa95baf1640 1 -- 172.21.15.23:0/1202292774 shutdown_connections 2024-03-24T10:40:08.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.794+0000 7fa95baf1640 1 --2- 172.21.15.23:0/1202292774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa954103340 0x7fa954109bd0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.794+0000 7fa95baf1640 1 --2- 172.21.15.23:0/1202292774 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa954102980 0x7fa954102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.794+0000 7fa95baf1640 1 --2- 172.21.15.23:0/1202292774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa954101790 0x7fa954101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.794+0000 7fa95baf1640 1 -- 172.21.15.23:0/1202292774 >> 172.21.15.23:0/1202292774 conn(0x7fa9540fcce0 msgr2=0x7fa9540ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.794+0000 7fa95baf1640 1 -- 172.21.15.23:0/1202292774 shutdown_connections 2024-03-24T10:40:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.794+0000 7fa95baf1640 1 -- 172.21.15.23:0/1202292774 wait complete. 2024-03-24T10:40:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.794+0000 7fa95baf1640 1 Processor -- start 2024-03-24T10:40:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95baf1640 1 -- start start 2024-03-24T10:40:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95baf1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa954101790 0x7fa9541976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95baf1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa954102980 0x7fa954197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95baf1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa954103340 0x7fa95419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95baf1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fa95419f1e0 con 0x7fa954101790 2024-03-24T10:40:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95baf1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa95419f350 con 0x7fa954103340 2024-03-24T10:40:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95baf1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa95419f510 con 0x7fa954102980 2024-03-24T10:40:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95988d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa954101790 0x7fa9541976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95988d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa954101790 0x7fa9541976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33418/0 (socket says 172.21.15.23:33418) 2024-03-24T10:40:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95988d640 1 -- 172.21.15.23:0/2823891709 learned_addr learned my addr 172.21.15.23:0/2823891709 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95908c640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa954102980 0x7fa954197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95a08e640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa954103340 0x7fa95419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95908c640 1 -- 172.21.15.23:0/2823891709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa954103340 msgr2=0x7fa95419eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95908c640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa954103340 0x7fa95419eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95908c640 1 -- 172.21.15.23:0/2823891709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa954101790 msgr2=0x7fa9541976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95908c640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa954101790 0x7fa9541976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95908c640 1 -- 172.21.15.23:0/2823891709 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c009900 con 0x7fa954102980 2024-03-24T10:40:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95988d640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa954101790 0x7fa9541976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.795+0000 7fa95908c640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa954102980 0x7fa954197c30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa94400e9f0 tx=0x7fa94400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.796+0000 7fa94affd640 1 -- 172.21.15.23:0/2823891709 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94400ccc0 con 0x7fa954102980 2024-03-24T10:40:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.796+0000 7fa94affd640 1 -- 172.21.15.23:0/2823891709 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa94400ce20 con 0x7fa954102980 2024-03-24T10:40:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.796+0000 7fa95baf1640 1 -- 172.21.15.23:0/2823891709 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa95419f7f0 con 0x7fa954102980 2024-03-24T10:40:08.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.796+0000 7fa94affd640 1 -- 172.21.15.23:0/2823891709 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944018920 con 0x7fa954102980 2024-03-24T10:40:08.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.796+0000 7fa95baf1640 1 -- 172.21.15.23:0/2823891709 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa95419fd00 con 0x7fa954102980 2024-03-24T10:40:08.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.797+0000 7fa95baf1640 1 -- 172.21.15.23:0/2823891709 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa954108ab0 con 0x7fa954102980 2024-03-24T10:40:08.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.798+0000 7fa94affd640 1 -- 172.21.15.23:0/2823891709 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 22) ==== 136927+0+0 (secure 0 0 0) 0x7fa944018a80 con 0x7fa954102980 2024-03-24T10:40:08.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.798+0000 7fa94affd640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa9400a22c0 0x7fa9400a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:08.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.799+0000 7fa94affd640 1 -- 172.21.15.23:0/2823891709 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fa944014070 con 0x7fa954102980 2024-03-24T10:40:08.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.799+0000 7fa95988d640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa9400a22c0 0x7fa9400a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:08.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.799+0000 7fa95988d640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa9400a22c0 0x7fa9400a4780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa9500058c0 tx=0x7fa950005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:08.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.802+0000 7fa94affd640 1 -- 172.21.15.23:0/2823891709 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa9440970c0 con 0x7fa954102980 2024-03-24T10:40:08.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.988+0000 7fa95baf1640 1 -- 172.21.15.23:0/2823891709 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa95410d290 con 0x7fa954102980 2024-03-24T10:40:08.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.988+0000 7fa94affd640 1 -- 172.21.15.23:0/2823891709 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fa94409a780 con 0x7fa954102980 2024-03-24T10:40:08.989 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-24T10:40:08.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.990+0000 7fa95baf1640 1 -- 172.21.15.23:0/2823891709 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa9400a22c0 msgr2=0x7fa9400a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.990+0000 7fa95baf1640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa9400a22c0 0x7fa9400a4780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa9500058c0 tx=0x7fa950005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:08.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.990+0000 7fa95baf1640 1 -- 172.21.15.23:0/2823891709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa954102980 msgr2=0x7fa954197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:08.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.990+0000 7fa95baf1640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa954102980 0x7fa954197c30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa94400e9f0 tx=0x7fa94400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.991+0000 7fa95baf1640 1 -- 172.21.15.23:0/2823891709 shutdown_connections 2024-03-24T10:40:08.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.991+0000 7fa95baf1640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa9400a22c0 0x7fa9400a4780 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.991+0000 7fa95baf1640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa954103340 0x7fa95419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.991+0000 7fa95baf1640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa954102980 0x7fa954197c30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.991+0000 7fa95baf1640 1 --2- 172.21.15.23:0/2823891709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa954101790 0x7fa9541976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:08.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.991+0000 7fa95baf1640 1 -- 172.21.15.23:0/2823891709 >> 172.21.15.23:0/2823891709 conn(0x7fa9540fcce0 msgr2=0x7fa954105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:08.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.991+0000 7fa95baf1640 1 -- 172.21.15.23:0/2823891709 shutdown_connections 2024-03-24T10:40:08.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:08.991+0000 7fa95baf1640 1 -- 172.21.15.23:0/2823891709 wait complete. 2024-03-24T10:40:09.002 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:40:09.003 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/582031112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:09.003 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: mgrmap e22: x(active, since 3s), standbys: y, z 2024-03-24T10:40:09.003 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/907604594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:40:09.003 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.003 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.003 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.003 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.003 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.003 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.003 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.003 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:40:09.005 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:40:09.005 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:08 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4191044927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:40:09.005 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-24T10:40:09.006 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:40:09.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.112+0000 7fd91fd7a640 1 -- 172.21.15.23:0/3032136581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd918100630 msgr2=0x7fd918100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:09.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.112+0000 7fd91fd7a640 1 --2- 172.21.15.23:0/3032136581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd918100630 0x7fd918100a50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd914009900 tx=0x7fd91402bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:09.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.112+0000 7fd91fd7a640 1 -- 172.21.15.23:0/3032136581 shutdown_connections 2024-03-24T10:40:09.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.112+0000 7fd91fd7a640 1 --2- 172.21.15.23:0/3032136581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9181021e0 0x7fd918108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.112+0000 7fd91fd7a640 1 --2- 172.21.15.23:0/3032136581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd918101820 0x7fd918101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.112+0000 7fd91fd7a640 1 --2- 172.21.15.23:0/3032136581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd918100630 0x7fd918100a50 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.112+0000 7fd91fd7a640 1 -- 172.21.15.23:0/3032136581 >> 172.21.15.23:0/3032136581 conn(0x7fd9180fbba0 msgr2=0x7fd9180fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:09.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.112+0000 7fd91fd7a640 1 -- 172.21.15.23:0/3032136581 shutdown_connections 2024-03-24T10:40:09.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.112+0000 7fd91fd7a640 1 -- 172.21.15.23:0/3032136581 wait complete. 2024-03-24T10:40:09.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91fd7a640 1 Processor -- start 2024-03-24T10:40:09.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91fd7a640 1 -- start start 2024-03-24T10:40:09.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91fd7a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd918100630 0x7fd9181966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:09.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91fd7a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd918101820 0x7fd918196c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:09.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91fd7a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9181021e0 0x7fd91819dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:09.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91fd7a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd91819e1d0 con 0x7fd918101820 2024-03-24T10:40:09.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91fd7a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd91819e340 con 0x7fd9181021e0 2024-03-24T10:40:09.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91fd7a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd91819e500 con 0x7fd918100630 2024-03-24T10:40:09.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91d315640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd918101820 0x7fd918196c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:09.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91db16640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd918100630 0x7fd9181966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:09.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91d315640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd918101820 0x7fd918196c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33444/0 (socket says 172.21.15.23:33444) 2024-03-24T10:40:09.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91d315640 1 -- 172.21.15.23:0/4158847352 learned_addr learned my addr 172.21.15.23:0/4158847352 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:09.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91e317640 1 --2- 172.21.15.23:0/4158847352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9181021e0 0x7fd91819dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:09.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91d315640 1 -- 172.21.15.23:0/4158847352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd918100630 msgr2=0x7fd9181966e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:09.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91d315640 1 --2- 172.21.15.23:0/4158847352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd918100630 0x7fd9181966e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91d315640 1 -- 172.21.15.23:0/4158847352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9181021e0 msgr2=0x7fd91819dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:09.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91d315640 1 --2- 172.21.15.23:0/4158847352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9181021e0 0x7fd91819dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.113+0000 7fd91d315640 1 -- 172.21.15.23:0/4158847352 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd914009560 con 0x7fd918101820 2024-03-24T10:40:09.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.114+0000 7fd91d315640 1 --2- 172.21.15.23:0/4158847352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd918101820 0x7fd918196c20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd90800e9f0 tx=0x7fd90800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:09.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.114+0000 7fd90effd640 1 -- 172.21.15.23:0/4158847352 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd908009720 con 0x7fd918101820 2024-03-24T10:40:09.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.114+0000 7fd91fd7a640 1 -- 172.21.15.23:0/4158847352 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd91819e7e0 con 0x7fd918101820 2024-03-24T10:40:09.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.114+0000 7fd91fd7a640 1 -- 172.21.15.23:0/4158847352 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd91819ecf0 con 0x7fd918101820 2024-03-24T10:40:09.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.114+0000 7fd90effd640 1 -- 172.21.15.23:0/4158847352 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd90800cd80 con 0x7fd918101820 2024-03-24T10:40:09.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.114+0000 7fd90effd640 1 -- 172.21.15.23:0/4158847352 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd908010690 con 0x7fd918101820 2024-03-24T10:40:09.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.115+0000 7fd91fd7a640 1 -- 172.21.15.23:0/4158847352 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9181079a0 con 0x7fd918101820 2024-03-24T10:40:09.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.117+0000 7fd90effd640 1 -- 172.21.15.23:0/4158847352 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 22) ==== 136927+0+0 (secure 0 0 0) 0x7fd9080046c0 con 0x7fd918101820 2024-03-24T10:40:09.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.117+0000 7fd90effd640 1 --2- 172.21.15.23:0/4158847352 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9040a22c0 0x7fd9040a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:09.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.117+0000 7fd90effd640 1 -- 172.21.15.23:0/4158847352 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fd908014070 con 0x7fd918101820 2024-03-24T10:40:09.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.117+0000 7fd91db16640 1 --2- 172.21.15.23:0/4158847352 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9040a22c0 0x7fd9040a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:09.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.118+0000 7fd91db16640 1 --2- 172.21.15.23:0/4158847352 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9040a22c0 0x7fd9040a4780 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd914005ab0 tx=0x7fd914005fd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:09.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.122+0000 7fd90effd640 1 -- 172.21.15.23:0/4158847352 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd9080970e0 con 0x7fd918101820 2024-03-24T10:40:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:40:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/582031112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: mgrmap e22: x(active, since 3s), standbys: y, z 2024-03-24T10:40:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/907604594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:40:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:40:09.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:40:09.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:08 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4191044927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:40:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:40:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/582031112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: mgrmap e22: x(active, since 3s), standbys: y, z 2024-03-24T10:40:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/907604594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:40:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:09.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:40:09.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:40:09.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:40:09.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:08 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4191044927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:40:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.349+0000 7fd91fd7a640 1 -- 172.21.15.23:0/4158847352 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd91810c1f0 con 0x7fd918101820 2024-03-24T10:40:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.350+0000 7fd90effd640 1 -- 172.21.15.23:0/4158847352 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11866 (secure 0 0 0) 0x7fd90809a7a0 con 0x7fd918101820 2024-03-24T10:40:09.351 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:40:09.351 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:40:03.697229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24539,24503,24505]},"id":1}]} 2024-03-24T10:40:09.351 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 23 2024-03-24T10:40:09.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.352+0000 7fd91fd7a640 1 -- 172.21.15.23:0/4158847352 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9040a22c0 msgr2=0x7fd9040a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:09.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.352+0000 7fd91fd7a640 1 --2- 172.21.15.23:0/4158847352 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9040a22c0 0x7fd9040a4780 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd914005ab0 tx=0x7fd914005fd0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.352+0000 7fd91fd7a640 1 -- 172.21.15.23:0/4158847352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd918101820 msgr2=0x7fd918196c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:09.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.352+0000 7fd91fd7a640 1 --2- 172.21.15.23:0/4158847352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd918101820 0x7fd918196c20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd90800e9f0 tx=0x7fd90800edf0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.352+0000 7fd91fd7a640 1 -- 172.21.15.23:0/4158847352 shutdown_connections 2024-03-24T10:40:09.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.352+0000 7fd91fd7a640 1 --2- 172.21.15.23:0/4158847352 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9040a22c0 0x7fd9040a4780 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.352+0000 7fd91fd7a640 1 --2- 172.21.15.23:0/4158847352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9181021e0 0x7fd91819dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.352+0000 7fd91fd7a640 1 --2- 172.21.15.23:0/4158847352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd918101820 0x7fd918196c20 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.352+0000 7fd91fd7a640 1 --2- 172.21.15.23:0/4158847352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd918100630 0x7fd9181966e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.352+0000 7fd91fd7a640 1 -- 172.21.15.23:0/4158847352 >> 172.21.15.23:0/4158847352 conn(0x7fd9180fbba0 msgr2=0x7fd918104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:09.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.352+0000 7fd91fd7a640 1 -- 172.21.15.23:0/4158847352 shutdown_connections 2024-03-24T10:40:09.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.352+0000 7fd91fd7a640 1 -- 172.21.15.23:0/4158847352 wait complete. 2024-03-24T10:40:09.364 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-24T10:40:09.365 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-24T10:40:09.365 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:40:09.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.469+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/3169633975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4cc100f90 msgr2=0x7fc4cc10d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:09.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.469+0000 7fc4d1dfd640 1 --2- 172.21.15.23:0/3169633975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4cc100f90 0x7fc4cc10d450 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009ca0 tx=0x7fc4c402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.469+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/3169633975 shutdown_connections 2024-03-24T10:40:09.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.469+0000 7fc4d1dfd640 1 --2- 172.21.15.23:0/3169633975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4cc100f90 0x7fc4cc10d450 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.469+0000 7fc4d1dfd640 1 --2- 172.21.15.23:0/3169633975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4cc1005d0 0x7fc4cc100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.469+0000 7fc4d1dfd640 1 --2- 172.21.15.23:0/3169633975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4cc106690 0x7fc4cc106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.469+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/3169633975 >> 172.21.15.23:0/3169633975 conn(0x7fc4cc076990 msgr2=0x7fc4cc076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:09.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.469+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/3169633975 shutdown_connections 2024-03-24T10:40:09.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.469+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/3169633975 wait complete. 2024-03-24T10:40:09.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.470+0000 7fc4d1dfd640 1 Processor -- start 2024-03-24T10:40:09.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.470+0000 7fc4d1dfd640 1 -- start start 2024-03-24T10:40:09.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.470+0000 7fc4d1dfd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4cc1005d0 0x7fc4cc198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:09.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.470+0000 7fc4d1dfd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4cc100f90 0x7fc4cc199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:09.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.470+0000 7fc4d1dfd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4cc106690 0x7fc4cc1a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:09.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.470+0000 7fc4d1dfd640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc199910 con 0x7fc4cc106690 2024-03-24T10:40:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.470+0000 7fc4d1dfd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc1a4110 con 0x7fc4cc100f90 2024-03-24T10:40:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.470+0000 7fc4d1dfd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc1a4250 con 0x7fc4cc1005d0 2024-03-24T10:40:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4cbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4cc106690 0x7fc4cc1a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4cbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4cc106690 0x7fc4cc1a3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33464/0 (socket says 172.21.15.23:33464) 2024-03-24T10:40:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4caffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4cc100f90 0x7fc4cc199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4cbfff640 1 -- 172.21.15.23:0/4027297028 learned_addr learned my addr 172.21.15.23:0/4027297028 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4cb7fe640 1 --2- 172.21.15.23:0/4027297028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4cc1005d0 0x7fc4cc198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4cbfff640 1 -- 172.21.15.23:0/4027297028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4cc1005d0 msgr2=0x7fc4cc198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4cbfff640 1 --2- 172.21.15.23:0/4027297028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4cc1005d0 0x7fc4cc198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4cbfff640 1 -- 172.21.15.23:0/4027297028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4cc100f90 msgr2=0x7fc4cc199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4cbfff640 1 --2- 172.21.15.23:0/4027297028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4cc100f90 0x7fc4cc199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4cbfff640 1 -- 172.21.15.23:0/4027297028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4c4009900 con 0x7fc4cc106690 2024-03-24T10:40:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4cbfff640 1 --2- 172.21.15.23:0/4027297028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4cc106690 0x7fc4cc1a3bd0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc4c402c700 tx=0x7fc4c4035aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:09.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4c8ff9640 1 -- 172.21.15.23:0/4027297028 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4035ea0 con 0x7fc4cc106690 2024-03-24T10:40:09.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4c8ff9640 1 -- 172.21.15.23:0/4027297028 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4c4003e50 con 0x7fc4cc106690 2024-03-24T10:40:09.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4c8ff9640 1 -- 172.21.15.23:0/4027297028 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4004860 con 0x7fc4cc106690 2024-03-24T10:40:09.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/4027297028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4cc1a4450 con 0x7fc4cc106690 2024-03-24T10:40:09.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.471+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/4027297028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4cc1a4850 con 0x7fc4cc106690 2024-03-24T10:40:09.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.473+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/4027297028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4cc101cf0 con 0x7fc4cc106690 2024-03-24T10:40:09.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.474+0000 7fc4c8ff9640 1 -- 172.21.15.23:0/4027297028 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 22) ==== 136927+0+0 (secure 0 0 0) 0x7fc4c403b040 con 0x7fc4cc106690 2024-03-24T10:40:09.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.474+0000 7fc4c8ff9640 1 --2- 172.21.15.23:0/4027297028 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc4b40a22c0 0x7fc4b40a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:09.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.475+0000 7fc4cb7fe640 1 --2- 172.21.15.23:0/4027297028 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc4b40a22c0 0x7fc4b40a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:09.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.475+0000 7fc4c8ff9640 1 -- 172.21.15.23:0/4027297028 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fc4c40f0a80 con 0x7fc4cc106690 2024-03-24T10:40:09.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.475+0000 7fc4cb7fe640 1 --2- 172.21.15.23:0/4027297028 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc4b40a22c0 0x7fc4b40a4780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc009b50 tx=0x7fc4bc005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:09.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.479+0000 7fc4c8ff9640 1 -- 172.21.15.23:0/4027297028 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc4c40b96b0 con 0x7fc4cc106690 2024-03-24T10:40:09.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.670+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/4027297028 --> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc4cc1106f0 con 0x7fc4b40a22c0 2024-03-24T10:40:09.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.680+0000 7fc4c8ff9640 1 -- 172.21.15.23:0/4027297028 <== mgr.15288 v2:172.21.15.23:6800/581236369 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257744 (secure 0 0 0) 0x7fc4cc1106f0 con 0x7fc4b40a22c0 2024-03-24T10:40:09.683 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:40:09.685 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-24T10:40:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.683+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/4027297028 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc4b40a22c0 msgr2=0x7fc4b40a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.683+0000 7fc4d1dfd640 1 --2- 172.21.15.23:0/4027297028 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc4b40a22c0 0x7fc4b40a4780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc009b50 tx=0x7fc4bc005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.683+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/4027297028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4cc106690 msgr2=0x7fc4cc1a3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.683+0000 7fc4d1dfd640 1 --2- 172.21.15.23:0/4027297028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4cc106690 0x7fc4cc1a3bd0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc4c402c700 tx=0x7fc4c4035aa0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.684+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/4027297028 shutdown_connections 2024-03-24T10:40:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.684+0000 7fc4d1dfd640 1 --2- 172.21.15.23:0/4027297028 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc4b40a22c0 0x7fc4b40a4780 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.684+0000 7fc4d1dfd640 1 --2- 172.21.15.23:0/4027297028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4cc106690 0x7fc4cc1a3bd0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.684+0000 7fc4d1dfd640 1 --2- 172.21.15.23:0/4027297028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4cc100f90 0x7fc4cc199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.684+0000 7fc4d1dfd640 1 --2- 172.21.15.23:0/4027297028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4cc1005d0 0x7fc4cc198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.684+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/4027297028 >> 172.21.15.23:0/4027297028 conn(0x7fc4cc076990 msgr2=0x7fc4cc0fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.684+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/4027297028 shutdown_connections 2024-03-24T10:40:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.684+0000 7fc4d1dfd640 1 -- 172.21.15.23:0/4027297028 wait complete. 2024-03-24T10:40:09.695 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-03-24T10:40:09.312178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185768,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":927,"num_read_kb":622,"num_write":468,"num_write_kb":4373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1181547,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":863,"ondisk_log_size":863,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368356,"kb_used_data":15012,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124573468,"statfs":{"total":1151940427776,"available":1151563231232,"internally_reserved":0,"allocated":15372288,"data_stored":9134098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.992192"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575582+0000","last_change":"2024-03-24T10:38:10.882385+0000","last_active":"2024-03-24T10:40:05.575582+0000","last_peered":"2024-03-24T10:40:05.575582+0000","last_clean":"2024-03-24T10:40:05.575582+0000","last_became_active":"2024-03-24T10:38:10.882062+0000","last_became_peered":"2024-03-24T10:38:10.882062+0000","last_unstale":"2024-03-24T10:40:05.575582+0000","last_undegraded":"2024-03-24T10:40:05.575582+0000","last_fullsized":"2024-03-24T10:40:05.575582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:25.151062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575927+0000","last_change":"2024-03-24T10:38:10.878001+0000","last_active":"2024-03-24T10:40:05.575927+0000","last_peered":"2024-03-24T10:40:05.575927+0000","last_clean":"2024-03-24T10:40:05.575927+0000","last_became_active":"2024-03-24T10:38:10.877661+0000","last_became_peered":"2024-03-24T10:38:10.877661+0000","last_unstale":"2024-03-24T10:40:05.575927+0000","last_undegraded":"2024-03-24T10:40:05.575927+0000","last_fullsized":"2024-03-24T10:40:05.575927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:40:19.062969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576049+0000","last_change":"2024-03-24T10:38:10.891809+0000","last_active":"2024-03-24T10:40:05.576049+0000","last_peered":"2024-03-24T10:40:05.576049+0000","last_clean":"2024-03-24T10:40:05.576049+0000","last_became_active":"2024-03-24T10:38:10.890354+0000","last_became_peered":"2024-03-24T10:38:10.890354+0000","last_unstale":"2024-03-24T10:40:05.576049+0000","last_undegraded":"2024-03-24T10:40:05.576049+0000","last_fullsized":"2024-03-24T10:40:05.576049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:59:26.037914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575921+0000","last_change":"2024-03-24T10:38:10.885097+0000","last_active":"2024-03-24T10:40:05.575921+0000","last_peered":"2024-03-24T10:40:05.575921+0000","last_clean":"2024-03-24T10:40:05.575921+0000","last_became_active":"2024-03-24T10:38:10.884842+0000","last_became_peered":"2024-03-24T10:38:10.884842+0000","last_unstale":"2024-03-24T10:40:05.575921+0000","last_undegraded":"2024-03-24T10:40:05.575921+0000","last_fullsized":"2024-03-24T10:40:05.575921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:39:44.085699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412906+0000","last_change":"2024-03-24T10:38:10.877653+0000","last_active":"2024-03-24T10:40:05.412906+0000","last_peered":"2024-03-24T10:40:05.412906+0000","last_clean":"2024-03-24T10:40:05.412906+0000","last_became_active":"2024-03-24T10:38:10.877249+0000","last_became_peered":"2024-03-24T10:38:10.877249+0000","last_unstale":"2024-03-24T10:40:05.412906+0000","last_undegraded":"2024-03-24T10:40:05.412906+0000","last_fullsized":"2024-03-24T10:40:05.412906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:17.504660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422853+0000","last_change":"2024-03-24T10:38:10.876584+0000","last_active":"2024-03-24T10:40:05.422853+0000","last_peered":"2024-03-24T10:40:05.422853+0000","last_clean":"2024-03-24T10:40:05.422853+0000","last_became_active":"2024-03-24T10:38:10.876173+0000","last_became_peered":"2024-03-24T10:38:10.876173+0000","last_unstale":"2024-03-24T10:40:05.422853+0000","last_undegraded":"2024-03-24T10:40:05.422853+0000","last_fullsized":"2024-03-24T10:40:05.422853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:33.899155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299791+0000","last_change":"2024-03-24T10:38:10.874206+0000","last_active":"2024-03-24T10:40:05.299791+0000","last_peered":"2024-03-24T10:40:05.299791+0000","last_clean":"2024-03-24T10:40:05.299791+0000","last_became_active":"2024-03-24T10:38:10.873779+0000","last_became_peered":"2024-03-24T10:38:10.873779+0000","last_unstale":"2024-03-24T10:40:05.299791+0000","last_undegraded":"2024-03-24T10:40:05.299791+0000","last_fullsized":"2024-03-24T10:40:05.299791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:08:19.928998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413254+0000","last_change":"2024-03-24T10:38:10.884219+0000","last_active":"2024-03-24T10:40:05.413254+0000","last_peered":"2024-03-24T10:40:05.413254+0000","last_clean":"2024-03-24T10:40:05.413254+0000","last_became_active":"2024-03-24T10:38:10.883863+0000","last_became_peered":"2024-03-24T10:38:10.883863+0000","last_unstale":"2024-03-24T10:40:05.413254+0000","last_undegraded":"2024-03-24T10:40:05.413254+0000","last_fullsized":"2024-03-24T10:40:05.413254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:26:34.932083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575665+0000","last_change":"2024-03-24T10:38:10.881297+0000","last_active":"2024-03-24T10:40:05.575665+0000","last_peered":"2024-03-24T10:40:05.575665+0000","last_clean":"2024-03-24T10:40:05.575665+0000","last_became_active":"2024-03-24T10:38:10.879871+0000","last_became_peered":"2024-03-24T10:38:10.879871+0000","last_unstale":"2024-03-24T10:40:05.575665+0000","last_undegraded":"2024-03-24T10:40:05.575665+0000","last_fullsized":"2024-03-24T10:40:05.575665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:05.132788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422954+0000","last_change":"2024-03-24T10:38:10.876648+0000","last_active":"2024-03-24T10:40:05.422954+0000","last_peered":"2024-03-24T10:40:05.422954+0000","last_clean":"2024-03-24T10:40:05.422954+0000","last_became_active":"2024-03-24T10:38:10.876113+0000","last_became_peered":"2024-03-24T10:38:10.876113+0000","last_unstale":"2024-03-24T10:40:05.422954+0000","last_undegraded":"2024-03-24T10:40:05.422954+0000","last_fullsized":"2024-03-24T10:40:05.422954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:29:18.282797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300282+0000","last_change":"2024-03-24T10:38:10.875836+0000","last_active":"2024-03-24T10:40:05.300282+0000","last_peered":"2024-03-24T10:40:05.300282+0000","last_clean":"2024-03-24T10:40:05.300282+0000","last_became_active":"2024-03-24T10:38:10.875522+0000","last_became_peered":"2024-03-24T10:38:10.875522+0000","last_unstale":"2024-03-24T10:40:05.300282+0000","last_undegraded":"2024-03-24T10:40:05.300282+0000","last_fullsized":"2024-03-24T10:40:05.300282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:41.282123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575678+0000","last_change":"2024-03-24T10:38:10.884567+0000","last_active":"2024-03-24T10:40:05.575678+0000","last_peered":"2024-03-24T10:40:05.575678+0000","last_clean":"2024-03-24T10:40:05.575678+0000","last_became_active":"2024-03-24T10:38:10.884195+0000","last_became_peered":"2024-03-24T10:38:10.884195+0000","last_unstale":"2024-03-24T10:40:05.575678+0000","last_undegraded":"2024-03-24T10:40:05.575678+0000","last_fullsized":"2024-03-24T10:40:05.575678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:29.286631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575744+0000","last_change":"2024-03-24T10:39:16.726561+0000","last_active":"2024-03-24T10:40:05.575744+0000","last_peered":"2024-03-24T10:40:05.575744+0000","last_clean":"2024-03-24T10:40:05.575744+0000","last_became_active":"2024-03-24T10:39:16.726088+0000","last_became_peered":"2024-03-24T10:39:16.726088+0000","last_unstale":"2024-03-24T10:40:05.575744+0000","last_undegraded":"2024-03-24T10:40:05.575744+0000","last_fullsized":"2024-03-24T10:40:05.575744+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:08:46.638610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575643+0000","last_change":"2024-03-24T10:38:10.879942+0000","last_active":"2024-03-24T10:40:05.575643+0000","last_peered":"2024-03-24T10:40:05.575643+0000","last_clean":"2024-03-24T10:40:05.575643+0000","last_became_active":"2024-03-24T10:38:10.879396+0000","last_became_peered":"2024-03-24T10:38:10.879396+0000","last_unstale":"2024-03-24T10:40:05.575643+0000","last_undegraded":"2024-03-24T10:40:05.575643+0000","last_fullsized":"2024-03-24T10:40:05.575643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:41.477988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575389+0000","last_change":"2024-03-24T10:38:10.886667+0000","last_active":"2024-03-24T10:40:05.575389+0000","last_peered":"2024-03-24T10:40:05.575389+0000","last_clean":"2024-03-24T10:40:05.575389+0000","last_became_active":"2024-03-24T10:38:10.886293+0000","last_became_peered":"2024-03-24T10:38:10.886293+0000","last_unstale":"2024-03-24T10:40:05.575389+0000","last_undegraded":"2024-03-24T10:40:05.575389+0000","last_fullsized":"2024-03-24T10:40:05.575389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:43:43.414573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574995+0000","last_change":"2024-03-24T10:38:10.883671+0000","last_active":"2024-03-24T10:40:05.574995+0000","last_peered":"2024-03-24T10:40:05.574995+0000","last_clean":"2024-03-24T10:40:05.574995+0000","last_became_active":"2024-03-24T10:38:10.883013+0000","last_became_peered":"2024-03-24T10:38:10.883013+0000","last_unstale":"2024-03-24T10:40:05.574995+0000","last_undegraded":"2024-03-24T10:40:05.574995+0000","last_fullsized":"2024-03-24T10:40:05.574995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:33.936543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575587+0000","last_change":"2024-03-24T10:38:10.887286+0000","last_active":"2024-03-24T10:40:05.575587+0000","last_peered":"2024-03-24T10:40:05.575587+0000","last_clean":"2024-03-24T10:40:05.575587+0000","last_became_active":"2024-03-24T10:38:10.886918+0000","last_became_peered":"2024-03-24T10:38:10.886918+0000","last_unstale":"2024-03-24T10:40:05.575587+0000","last_undegraded":"2024-03-24T10:40:05.575587+0000","last_fullsized":"2024-03-24T10:40:05.575587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:40.266589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.298339+0000","last_change":"2024-03-24T10:38:10.874476+0000","last_active":"2024-03-24T10:40:05.298339+0000","last_peered":"2024-03-24T10:40:05.298339+0000","last_clean":"2024-03-24T10:40:05.298339+0000","last_became_active":"2024-03-24T10:38:10.874123+0000","last_became_peered":"2024-03-24T10:38:10.874123+0000","last_unstale":"2024-03-24T10:40:05.298339+0000","last_undegraded":"2024-03-24T10:40:05.298339+0000","last_fullsized":"2024-03-24T10:40:05.298339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:29.759320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575455+0000","last_change":"2024-03-24T10:38:10.878758+0000","last_active":"2024-03-24T10:40:05.575455+0000","last_peered":"2024-03-24T10:40:05.575455+0000","last_clean":"2024-03-24T10:40:05.575455+0000","last_became_active":"2024-03-24T10:38:10.878424+0000","last_became_peered":"2024-03-24T10:38:10.878424+0000","last_unstale":"2024-03-24T10:40:05.575455+0000","last_undegraded":"2024-03-24T10:40:05.575455+0000","last_fullsized":"2024-03-24T10:40:05.575455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:15:19.273108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.298282+0000","last_change":"2024-03-24T10:39:16.709304+0000","last_active":"2024-03-24T10:40:05.298282+0000","last_peered":"2024-03-24T10:40:05.298282+0000","last_clean":"2024-03-24T10:40:05.298282+0000","last_became_active":"2024-03-24T10:39:16.700325+0000","last_became_peered":"2024-03-24T10:39:16.700325+0000","last_unstale":"2024-03-24T10:40:05.298282+0000","last_undegraded":"2024-03-24T10:40:05.298282+0000","last_fullsized":"2024-03-24T10:40:05.298282+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:57.633288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.414021+0000","last_change":"2024-03-24T10:38:10.885198+0000","last_active":"2024-03-24T10:40:05.414021+0000","last_peered":"2024-03-24T10:40:05.414021+0000","last_clean":"2024-03-24T10:40:05.414021+0000","last_became_active":"2024-03-24T10:38:10.884123+0000","last_became_peered":"2024-03-24T10:38:10.884123+0000","last_unstale":"2024-03-24T10:40:05.414021+0000","last_undegraded":"2024-03-24T10:40:05.414021+0000","last_fullsized":"2024-03-24T10:40:05.414021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:37:12.510671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.414153+0000","last_change":"2024-03-24T10:38:09.866223+0000","last_active":"2024-03-24T10:40:05.414153+0000","last_peered":"2024-03-24T10:40:05.414153+0000","last_clean":"2024-03-24T10:40:05.414153+0000","last_became_active":"2024-03-24T10:38:09.865711+0000","last_became_peered":"2024-03-24T10:38:09.865711+0000","last_unstale":"2024-03-24T10:40:05.414153+0000","last_undegraded":"2024-03-24T10:40:05.414153+0000","last_fullsized":"2024-03-24T10:40:05.414153+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:26:08.380749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576281+0000","last_change":"2024-03-24T10:38:10.890557+0000","last_active":"2024-03-24T10:40:05.576281+0000","last_peered":"2024-03-24T10:40:05.576281+0000","last_clean":"2024-03-24T10:40:05.576281+0000","last_became_active":"2024-03-24T10:38:10.889883+0000","last_became_peered":"2024-03-24T10:38:10.889883+0000","last_unstale":"2024-03-24T10:40:05.576281+0000","last_undegraded":"2024-03-24T10:40:05.576281+0000","last_fullsized":"2024-03-24T10:40:05.576281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:34.731585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413316+0000","last_change":"2024-03-24T10:38:09.870253+0000","last_active":"2024-03-24T10:40:05.413316+0000","last_peered":"2024-03-24T10:40:05.413316+0000","last_clean":"2024-03-24T10:40:05.413316+0000","last_became_active":"2024-03-24T10:38:09.869932+0000","last_became_peered":"2024-03-24T10:38:09.869932+0000","last_unstale":"2024-03-24T10:40:05.413316+0000","last_undegraded":"2024-03-24T10:40:05.413316+0000","last_fullsized":"2024-03-24T10:40:05.413316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:38:43.626390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575343+0000","last_change":"2024-03-24T10:38:10.881705+0000","last_active":"2024-03-24T10:40:05.575343+0000","last_peered":"2024-03-24T10:40:05.575343+0000","last_clean":"2024-03-24T10:40:05.575343+0000","last_became_active":"2024-03-24T10:38:10.881323+0000","last_became_peered":"2024-03-24T10:38:10.881323+0000","last_unstale":"2024-03-24T10:40:05.575343+0000","last_undegraded":"2024-03-24T10:40:05.575343+0000","last_fullsized":"2024-03-24T10:40:05.575343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:25.861184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413086+0000","last_change":"2024-03-24T10:38:09.870481+0000","last_active":"2024-03-24T10:40:05.413086+0000","last_peered":"2024-03-24T10:40:05.413086+0000","last_clean":"2024-03-24T10:40:05.413086+0000","last_became_active":"2024-03-24T10:38:09.870117+0000","last_became_peered":"2024-03-24T10:38:09.870117+0000","last_unstale":"2024-03-24T10:40:05.413086+0000","last_undegraded":"2024-03-24T10:40:05.413086+0000","last_fullsized":"2024-03-24T10:40:05.413086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:37:51.871068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.297975+0000","last_change":"2024-03-24T10:39:16.906560+0000","last_active":"2024-03-24T10:40:05.297975+0000","last_peered":"2024-03-24T10:40:05.297975+0000","last_clean":"2024-03-24T10:40:05.297975+0000","last_became_active":"2024-03-24T10:39:16.906009+0000","last_became_peered":"2024-03-24T10:39:16.906009+0000","last_unstale":"2024-03-24T10:40:05.297975+0000","last_undegraded":"2024-03-24T10:40:05.297975+0000","last_fullsized":"2024-03-24T10:40:05.297975+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:45:56.513953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"88'8","reported_seq":144,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:06.037385+0000","last_change":"2024-03-24T10:38:09.864160+0000","last_active":"2024-03-24T10:40:06.037385+0000","last_peered":"2024-03-24T10:40:06.037385+0000","last_clean":"2024-03-24T10:40:06.037385+0000","last_became_active":"2024-03-24T10:38:09.862066+0000","last_became_peered":"2024-03-24T10:38:09.862066+0000","last_unstale":"2024-03-24T10:40:06.037385+0000","last_undegraded":"2024-03-24T10:40:06.037385+0000","last_fullsized":"2024-03-24T10:40:06.037385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:59.003931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412874+0000","last_change":"2024-03-24T10:38:10.874433+0000","last_active":"2024-03-24T10:40:05.412874+0000","last_peered":"2024-03-24T10:40:05.412874+0000","last_clean":"2024-03-24T10:40:05.412874+0000","last_became_active":"2024-03-24T10:38:10.874090+0000","last_became_peered":"2024-03-24T10:38:10.874090+0000","last_unstale":"2024-03-24T10:40:05.412874+0000","last_undegraded":"2024-03-24T10:40:05.412874+0000","last_fullsized":"2024-03-24T10:40:05.412874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:25.979419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422847+0000","last_change":"2024-03-24T10:38:09.863028+0000","last_active":"2024-03-24T10:40:05.422847+0000","last_peered":"2024-03-24T10:40:05.422847+0000","last_clean":"2024-03-24T10:40:05.422847+0000","last_became_active":"2024-03-24T10:38:09.862682+0000","last_became_peered":"2024-03-24T10:38:09.862682+0000","last_unstale":"2024-03-24T10:40:05.422847+0000","last_undegraded":"2024-03-24T10:40:05.422847+0000","last_fullsized":"2024-03-24T10:40:05.422847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:14.962115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300267+0000","last_change":"2024-03-24T10:38:10.885950+0000","last_active":"2024-03-24T10:40:05.300267+0000","last_peered":"2024-03-24T10:40:05.300267+0000","last_clean":"2024-03-24T10:40:05.300267+0000","last_became_active":"2024-03-24T10:38:10.885685+0000","last_became_peered":"2024-03-24T10:38:10.885685+0000","last_unstale":"2024-03-24T10:40:05.300267+0000","last_undegraded":"2024-03-24T10:40:05.300267+0000","last_fullsized":"2024-03-24T10:40:05.300267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:42:09.912743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575251+0000","last_change":"2024-03-24T10:38:09.860376+0000","last_active":"2024-03-24T10:40:05.575251+0000","last_peered":"2024-03-24T10:40:05.575251+0000","last_clean":"2024-03-24T10:40:05.575251+0000","last_became_active":"2024-03-24T10:38:09.860062+0000","last_became_peered":"2024-03-24T10:38:09.860062+0000","last_unstale":"2024-03-24T10:40:05.575251+0000","last_undegraded":"2024-03-24T10:40:05.575251+0000","last_fullsized":"2024-03-24T10:40:05.575251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:33.387953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413177+0000","last_change":"2024-03-24T10:38:10.883843+0000","last_active":"2024-03-24T10:40:05.413177+0000","last_peered":"2024-03-24T10:40:05.413177+0000","last_clean":"2024-03-24T10:40:05.413177+0000","last_became_active":"2024-03-24T10:38:10.883170+0000","last_became_peered":"2024-03-24T10:38:10.883170+0000","last_unstale":"2024-03-24T10:40:05.413177+0000","last_undegraded":"2024-03-24T10:40:05.413177+0000","last_fullsized":"2024-03-24T10:40:05.413177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:28:16.798546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299912+0000","last_change":"2024-03-24T10:38:09.862299+0000","last_active":"2024-03-24T10:40:05.299912+0000","last_peered":"2024-03-24T10:40:05.299912+0000","last_clean":"2024-03-24T10:40:05.299912+0000","last_became_active":"2024-03-24T10:38:09.861895+0000","last_became_peered":"2024-03-24T10:38:09.861895+0000","last_unstale":"2024-03-24T10:40:05.299912+0000","last_undegraded":"2024-03-24T10:40:05.299912+0000","last_fullsized":"2024-03-24T10:40:05.299912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:13:19.869511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.297954+0000","last_change":"2024-03-24T10:38:10.880507+0000","last_active":"2024-03-24T10:40:05.297954+0000","last_peered":"2024-03-24T10:40:05.297954+0000","last_clean":"2024-03-24T10:40:05.297954+0000","last_became_active":"2024-03-24T10:38:10.879790+0000","last_became_peered":"2024-03-24T10:38:10.879790+0000","last_unstale":"2024-03-24T10:40:05.297954+0000","last_undegraded":"2024-03-24T10:40:05.297954+0000","last_fullsized":"2024-03-24T10:40:05.297954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:11:02.650420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.423038+0000","last_change":"2024-03-24T10:38:09.860186+0000","last_active":"2024-03-24T10:40:05.423038+0000","last_peered":"2024-03-24T10:40:05.423038+0000","last_clean":"2024-03-24T10:40:05.423038+0000","last_became_active":"2024-03-24T10:38:09.859834+0000","last_became_peered":"2024-03-24T10:38:09.859834+0000","last_unstale":"2024-03-24T10:40:05.423038+0000","last_undegraded":"2024-03-24T10:40:05.423038+0000","last_fullsized":"2024-03-24T10:40:05.423038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:33:48.029053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575833+0000","last_change":"2024-03-24T10:38:10.875634+0000","last_active":"2024-03-24T10:40:05.575833+0000","last_peered":"2024-03-24T10:40:05.575833+0000","last_clean":"2024-03-24T10:40:05.575833+0000","last_became_active":"2024-03-24T10:38:10.875101+0000","last_became_peered":"2024-03-24T10:38:10.875101+0000","last_unstale":"2024-03-24T10:40:05.575833+0000","last_undegraded":"2024-03-24T10:40:05.575833+0000","last_fullsized":"2024-03-24T10:40:05.575833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:32:58.165087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300091+0000","last_change":"2024-03-24T10:38:09.860412+0000","last_active":"2024-03-24T10:40:05.300091+0000","last_peered":"2024-03-24T10:40:05.300091+0000","last_clean":"2024-03-24T10:40:05.300091+0000","last_became_active":"2024-03-24T10:38:09.860084+0000","last_became_peered":"2024-03-24T10:38:09.860084+0000","last_unstale":"2024-03-24T10:40:05.300091+0000","last_undegraded":"2024-03-24T10:40:05.300091+0000","last_fullsized":"2024-03-24T10:40:05.300091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:44.585482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576331+0000","last_change":"2024-03-24T10:39:16.703566+0000","last_active":"2024-03-24T10:40:05.576331+0000","last_peered":"2024-03-24T10:40:05.576331+0000","last_clean":"2024-03-24T10:40:05.576331+0000","last_became_active":"2024-03-24T10:39:16.703213+0000","last_became_peered":"2024-03-24T10:39:16.703213+0000","last_unstale":"2024-03-24T10:40:05.576331+0000","last_undegraded":"2024-03-24T10:40:05.576331+0000","last_fullsized":"2024-03-24T10:40:05.576331+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:10.888996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300419+0000","last_change":"2024-03-24T10:38:09.865191+0000","last_active":"2024-03-24T10:40:05.300419+0000","last_peered":"2024-03-24T10:40:05.300419+0000","last_clean":"2024-03-24T10:40:05.300419+0000","last_became_active":"2024-03-24T10:38:09.864470+0000","last_became_peered":"2024-03-24T10:38:09.864470+0000","last_unstale":"2024-03-24T10:40:05.300419+0000","last_undegraded":"2024-03-24T10:40:05.300419+0000","last_fullsized":"2024-03-24T10:40:05.300419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:45:00.266470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575474+0000","last_change":"2024-03-24T10:38:10.889668+0000","last_active":"2024-03-24T10:40:05.575474+0000","last_peered":"2024-03-24T10:40:05.575474+0000","last_clean":"2024-03-24T10:40:05.575474+0000","last_became_active":"2024-03-24T10:38:10.889364+0000","last_became_peered":"2024-03-24T10:38:10.889364+0000","last_unstale":"2024-03-24T10:40:05.575474+0000","last_undegraded":"2024-03-24T10:40:05.575474+0000","last_fullsized":"2024-03-24T10:40:05.575474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:57.702719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'35","reported_seq":289,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:06.037599+0000","last_change":"2024-03-24T10:38:09.861809+0000","last_active":"2024-03-24T10:40:06.037599+0000","last_peered":"2024-03-24T10:40:06.037599+0000","last_clean":"2024-03-24T10:40:06.037599+0000","last_became_active":"2024-03-24T10:38:09.861444+0000","last_became_peered":"2024-03-24T10:38:09.861444+0000","last_unstale":"2024-03-24T10:40:06.037599+0000","last_undegraded":"2024-03-24T10:40:06.037599+0000","last_fullsized":"2024-03-24T10:40:06.037599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:19.652188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65118,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":180,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413060+0000","last_change":"2024-03-24T10:38:10.883090+0000","last_active":"2024-03-24T10:40:05.413060+0000","last_peered":"2024-03-24T10:40:05.413060+0000","last_clean":"2024-03-24T10:40:05.413060+0000","last_became_active":"2024-03-24T10:38:10.880507+0000","last_became_peered":"2024-03-24T10:38:10.880507+0000","last_unstale":"2024-03-24T10:40:05.413060+0000","last_undegraded":"2024-03-24T10:40:05.413060+0000","last_fullsized":"2024-03-24T10:40:05.413060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:01.422699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575220+0000","last_change":"2024-03-24T10:38:09.868072+0000","last_active":"2024-03-24T10:40:05.575220+0000","last_peered":"2024-03-24T10:40:05.575220+0000","last_clean":"2024-03-24T10:40:05.575220+0000","last_became_active":"2024-03-24T10:38:09.867704+0000","last_became_peered":"2024-03-24T10:38:09.867704+0000","last_unstale":"2024-03-24T10:40:05.575220+0000","last_undegraded":"2024-03-24T10:40:05.575220+0000","last_fullsized":"2024-03-24T10:40:05.575220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:09:28.726969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575265+0000","last_change":"2024-03-24T10:38:10.881882+0000","last_active":"2024-03-24T10:40:05.575265+0000","last_peered":"2024-03-24T10:40:05.575265+0000","last_clean":"2024-03-24T10:40:05.575265+0000","last_became_active":"2024-03-24T10:38:10.881495+0000","last_became_peered":"2024-03-24T10:38:10.881495+0000","last_unstale":"2024-03-24T10:40:05.575265+0000","last_undegraded":"2024-03-24T10:40:05.575265+0000","last_fullsized":"2024-03-24T10:40:05.575265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:31:43.392952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300200+0000","last_change":"2024-03-24T10:38:09.867265+0000","last_active":"2024-03-24T10:40:05.300200+0000","last_peered":"2024-03-24T10:40:05.300200+0000","last_clean":"2024-03-24T10:40:05.300200+0000","last_became_active":"2024-03-24T10:38:09.866669+0000","last_became_peered":"2024-03-24T10:38:09.866669+0000","last_unstale":"2024-03-24T10:40:05.300200+0000","last_undegraded":"2024-03-24T10:40:05.300200+0000","last_fullsized":"2024-03-24T10:40:05.300200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:28.230967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300253+0000","last_change":"2024-03-24T10:38:10.886634+0000","last_active":"2024-03-24T10:40:05.300253+0000","last_peered":"2024-03-24T10:40:05.300253+0000","last_clean":"2024-03-24T10:40:05.300253+0000","last_became_active":"2024-03-24T10:38:10.886207+0000","last_became_peered":"2024-03-24T10:38:10.886207+0000","last_unstale":"2024-03-24T10:40:05.300253+0000","last_undegraded":"2024-03-24T10:40:05.300253+0000","last_fullsized":"2024-03-24T10:40:05.300253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:43:41.601693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'5","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575133+0000","last_change":"2024-03-24T10:38:09.863262+0000","last_active":"2024-03-24T10:40:05.575133+0000","last_peered":"2024-03-24T10:40:05.575133+0000","last_clean":"2024-03-24T10:40:05.575133+0000","last_became_active":"2024-03-24T10:38:09.862665+0000","last_became_peered":"2024-03-24T10:38:09.862665+0000","last_unstale":"2024-03-24T10:40:05.575133+0000","last_undegraded":"2024-03-24T10:40:05.575133+0000","last_fullsized":"2024-03-24T10:40:05.575133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:12:29.117592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299650+0000","last_change":"2024-03-24T10:38:10.880793+0000","last_active":"2024-03-24T10:40:05.299650+0000","last_peered":"2024-03-24T10:40:05.299650+0000","last_clean":"2024-03-24T10:40:05.299650+0000","last_became_active":"2024-03-24T10:38:10.880449+0000","last_became_peered":"2024-03-24T10:38:10.880449+0000","last_unstale":"2024-03-24T10:40:05.299650+0000","last_undegraded":"2024-03-24T10:40:05.299650+0000","last_fullsized":"2024-03-24T10:40:05.299650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:32:57.184014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412771+0000","last_change":"2024-03-24T10:39:16.915438+0000","last_active":"2024-03-24T10:40:05.412771+0000","last_peered":"2024-03-24T10:40:05.412771+0000","last_clean":"2024-03-24T10:40:05.412771+0000","last_became_active":"2024-03-24T10:39:16.906155+0000","last_became_peered":"2024-03-24T10:39:16.906155+0000","last_unstale":"2024-03-24T10:40:05.412771+0000","last_undegraded":"2024-03-24T10:40:05.412771+0000","last_fullsized":"2024-03-24T10:40:05.412771+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:36:12.356069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299997+0000","last_change":"2024-03-24T10:38:10.881952+0000","last_active":"2024-03-24T10:40:05.299997+0000","last_peered":"2024-03-24T10:40:05.299997+0000","last_clean":"2024-03-24T10:40:05.299997+0000","last_became_active":"2024-03-24T10:38:10.881628+0000","last_became_peered":"2024-03-24T10:38:10.881628+0000","last_unstale":"2024-03-24T10:40:05.299997+0000","last_undegraded":"2024-03-24T10:40:05.299997+0000","last_fullsized":"2024-03-24T10:40:05.299997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:24.777793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413319+0000","last_change":"2024-03-24T10:38:09.868572+0000","last_active":"2024-03-24T10:40:05.413319+0000","last_peered":"2024-03-24T10:40:05.413319+0000","last_clean":"2024-03-24T10:40:05.413319+0000","last_became_active":"2024-03-24T10:38:09.867702+0000","last_became_peered":"2024-03-24T10:38:09.867702+0000","last_unstale":"2024-03-24T10:40:05.413319+0000","last_undegraded":"2024-03-24T10:40:05.413319+0000","last_fullsized":"2024-03-24T10:40:05.413319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:27.890114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575267+0000","last_change":"2024-03-24T10:38:10.875342+0000","last_active":"2024-03-24T10:40:05.575267+0000","last_peered":"2024-03-24T10:40:05.575267+0000","last_clean":"2024-03-24T10:40:05.575267+0000","last_became_active":"2024-03-24T10:38:10.874933+0000","last_became_peered":"2024-03-24T10:38:10.874933+0000","last_unstale":"2024-03-24T10:40:05.575267+0000","last_undegraded":"2024-03-24T10:40:05.575267+0000","last_fullsized":"2024-03-24T10:40:05.575267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:56:09.139478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.298000+0000","last_change":"2024-03-24T10:38:09.857867+0000","last_active":"2024-03-24T10:40:05.298000+0000","last_peered":"2024-03-24T10:40:05.298000+0000","last_clean":"2024-03-24T10:40:05.298000+0000","last_became_active":"2024-03-24T10:38:09.857525+0000","last_became_peered":"2024-03-24T10:38:09.857525+0000","last_unstale":"2024-03-24T10:40:05.298000+0000","last_undegraded":"2024-03-24T10:40:05.298000+0000","last_fullsized":"2024-03-24T10:40:05.298000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:37.320245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422684+0000","last_change":"2024-03-24T10:38:10.878980+0000","last_active":"2024-03-24T10:40:05.422684+0000","last_peered":"2024-03-24T10:40:05.422684+0000","last_clean":"2024-03-24T10:40:05.422684+0000","last_became_active":"2024-03-24T10:38:10.878648+0000","last_became_peered":"2024-03-24T10:38:10.878648+0000","last_unstale":"2024-03-24T10:40:05.422684+0000","last_undegraded":"2024-03-24T10:40:05.422684+0000","last_fullsized":"2024-03-24T10:40:05.422684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:49.988403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299957+0000","last_change":"2024-03-24T10:38:09.863457+0000","last_active":"2024-03-24T10:40:05.299957+0000","last_peered":"2024-03-24T10:40:05.299957+0000","last_clean":"2024-03-24T10:40:05.299957+0000","last_became_active":"2024-03-24T10:38:09.863130+0000","last_became_peered":"2024-03-24T10:38:09.863130+0000","last_unstale":"2024-03-24T10:40:05.299957+0000","last_undegraded":"2024-03-24T10:40:05.299957+0000","last_fullsized":"2024-03-24T10:40:05.299957+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:16.528703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422825+0000","last_change":"2024-03-24T10:38:10.887014+0000","last_active":"2024-03-24T10:40:05.422825+0000","last_peered":"2024-03-24T10:40:05.422825+0000","last_clean":"2024-03-24T10:40:05.422825+0000","last_became_active":"2024-03-24T10:38:10.886698+0000","last_became_peered":"2024-03-24T10:38:10.886698+0000","last_unstale":"2024-03-24T10:40:05.422825+0000","last_undegraded":"2024-03-24T10:40:05.422825+0000","last_fullsized":"2024-03-24T10:40:05.422825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:47:08.571886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300143+0000","last_change":"2024-03-24T10:38:10.874811+0000","last_active":"2024-03-24T10:40:05.300143+0000","last_peered":"2024-03-24T10:40:05.300143+0000","last_clean":"2024-03-24T10:40:05.300143+0000","last_became_active":"2024-03-24T10:38:10.874483+0000","last_became_peered":"2024-03-24T10:38:10.874483+0000","last_unstale":"2024-03-24T10:40:05.300143+0000","last_undegraded":"2024-03-24T10:40:05.300143+0000","last_fullsized":"2024-03-24T10:40:05.300143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:57.382174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":137,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575176+0000","last_change":"2024-03-24T10:38:09.865614+0000","last_active":"2024-03-24T10:40:05.575176+0000","last_peered":"2024-03-24T10:40:05.575176+0000","last_clean":"2024-03-24T10:40:05.575176+0000","last_became_active":"2024-03-24T10:38:09.865317+0000","last_became_peered":"2024-03-24T10:38:09.865317+0000","last_unstale":"2024-03-24T10:40:05.575176+0000","last_undegraded":"2024-03-24T10:40:05.575176+0000","last_fullsized":"2024-03-24T10:40:05.575176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:21:14.968552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413502+0000","last_change":"2024-03-24T10:38:10.884847+0000","last_active":"2024-03-24T10:40:05.413502+0000","last_peered":"2024-03-24T10:40:05.413502+0000","last_clean":"2024-03-24T10:40:05.413502+0000","last_became_active":"2024-03-24T10:38:10.884526+0000","last_became_peered":"2024-03-24T10:38:10.884526+0000","last_unstale":"2024-03-24T10:40:05.413502+0000","last_undegraded":"2024-03-24T10:40:05.413502+0000","last_fullsized":"2024-03-24T10:40:05.413502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:01:49.364083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575339+0000","last_change":"2024-03-24T10:38:09.858893+0000","last_active":"2024-03-24T10:40:05.575339+0000","last_peered":"2024-03-24T10:40:05.575339+0000","last_clean":"2024-03-24T10:40:05.575339+0000","last_became_active":"2024-03-24T10:38:09.858474+0000","last_became_peered":"2024-03-24T10:38:09.858474+0000","last_unstale":"2024-03-24T10:40:05.575339+0000","last_undegraded":"2024-03-24T10:40:05.575339+0000","last_fullsized":"2024-03-24T10:40:05.575339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:03:36.709498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299715+0000","last_change":"2024-03-24T10:36:59.100728+0000","last_active":"2024-03-24T10:40:05.299715+0000","last_peered":"2024-03-24T10:40:05.299715+0000","last_clean":"2024-03-24T10:40:05.299715+0000","last_became_active":"2024-03-24T10:36:59.100159+0000","last_became_peered":"2024-03-24T10:36:59.100159+0000","last_unstale":"2024-03-24T10:40:05.299715+0000","last_undegraded":"2024-03-24T10:40:05.299715+0000","last_fullsized":"2024-03-24T10:40:05.299715+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:39.429553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:39.429553+0000","last_clean_scrub_stamp":"2024-03-24T10:33:39.429553+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:21:01.470585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574970+0000","last_change":"2024-03-24T10:38:10.885804+0000","last_active":"2024-03-24T10:40:05.574970+0000","last_peered":"2024-03-24T10:40:05.574970+0000","last_clean":"2024-03-24T10:40:05.574970+0000","last_became_active":"2024-03-24T10:38:10.885447+0000","last_became_peered":"2024-03-24T10:38:10.885447+0000","last_unstale":"2024-03-24T10:40:05.574970+0000","last_undegraded":"2024-03-24T10:40:05.574970+0000","last_fullsized":"2024-03-24T10:40:05.574970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:57.103205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299830+0000","last_change":"2024-03-24T10:38:09.863187+0000","last_active":"2024-03-24T10:40:05.299830+0000","last_peered":"2024-03-24T10:40:05.299830+0000","last_clean":"2024-03-24T10:40:05.299830+0000","last_became_active":"2024-03-24T10:38:09.862859+0000","last_became_peered":"2024-03-24T10:38:09.862859+0000","last_unstale":"2024-03-24T10:40:05.299830+0000","last_undegraded":"2024-03-24T10:40:05.299830+0000","last_fullsized":"2024-03-24T10:40:05.299830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:21:02.707127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413190+0000","last_change":"2024-03-24T10:38:09.867899+0000","last_active":"2024-03-24T10:40:05.413190+0000","last_peered":"2024-03-24T10:40:05.413190+0000","last_clean":"2024-03-24T10:40:05.413190+0000","last_became_active":"2024-03-24T10:38:09.867563+0000","last_became_peered":"2024-03-24T10:38:09.867563+0000","last_unstale":"2024-03-24T10:40:05.413190+0000","last_undegraded":"2024-03-24T10:40:05.413190+0000","last_fullsized":"2024-03-24T10:40:05.413190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:51.067981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575138+0000","last_change":"2024-03-24T10:38:10.870825+0000","last_active":"2024-03-24T10:40:05.575138+0000","last_peered":"2024-03-24T10:40:05.575138+0000","last_clean":"2024-03-24T10:40:05.575138+0000","last_became_active":"2024-03-24T10:38:10.870491+0000","last_became_peered":"2024-03-24T10:38:10.870491+0000","last_unstale":"2024-03-24T10:40:05.575138+0000","last_undegraded":"2024-03-24T10:40:05.575138+0000","last_fullsized":"2024-03-24T10:40:05.575138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:30:34.091299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'44","reported_seq":88,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.527925+0000","last_change":"2024-03-24T10:38:09.863724+0000","last_active":"2024-03-24T10:40:05.527925+0000","last_peered":"2024-03-24T10:40:05.527925+0000","last_clean":"2024-03-24T10:40:05.527925+0000","last_became_active":"2024-03-24T10:38:09.862900+0000","last_became_peered":"2024-03-24T10:38:09.862900+0000","last_unstale":"2024-03-24T10:40:05.527925+0000","last_undegraded":"2024-03-24T10:40:05.527925+0000","last_fullsized":"2024-03-24T10:40:05.527925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:21.646074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413886+0000","last_change":"2024-03-24T10:38:10.884661+0000","last_active":"2024-03-24T10:40:05.413886+0000","last_peered":"2024-03-24T10:40:05.413886+0000","last_clean":"2024-03-24T10:40:05.413886+0000","last_became_active":"2024-03-24T10:38:10.882901+0000","last_became_peered":"2024-03-24T10:38:10.882901+0000","last_unstale":"2024-03-24T10:40:05.413886+0000","last_undegraded":"2024-03-24T10:40:05.413886+0000","last_fullsized":"2024-03-24T10:40:05.413886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:23:46.366567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576002+0000","last_change":"2024-03-24T10:38:09.862122+0000","last_active":"2024-03-24T10:40:05.576002+0000","last_peered":"2024-03-24T10:40:05.576002+0000","last_clean":"2024-03-24T10:40:05.576002+0000","last_became_active":"2024-03-24T10:38:09.861765+0000","last_became_peered":"2024-03-24T10:38:09.861765+0000","last_unstale":"2024-03-24T10:40:05.576002+0000","last_undegraded":"2024-03-24T10:40:05.576002+0000","last_fullsized":"2024-03-24T10:40:05.576002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:40:15.148349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575995+0000","last_change":"2024-03-24T10:38:10.886797+0000","last_active":"2024-03-24T10:40:05.575995+0000","last_peered":"2024-03-24T10:40:05.575995+0000","last_clean":"2024-03-24T10:40:05.575995+0000","last_became_active":"2024-03-24T10:38:10.886406+0000","last_became_peered":"2024-03-24T10:38:10.886406+0000","last_unstale":"2024-03-24T10:40:05.575995+0000","last_undegraded":"2024-03-24T10:40:05.575995+0000","last_fullsized":"2024-03-24T10:40:05.575995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:35.763623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575166+0000","last_change":"2024-03-24T10:39:16.707034+0000","last_active":"2024-03-24T10:40:05.575166+0000","last_peered":"2024-03-24T10:40:05.575166+0000","last_clean":"2024-03-24T10:40:05.575166+0000","last_became_active":"2024-03-24T10:39:16.701995+0000","last_became_peered":"2024-03-24T10:39:16.701995+0000","last_unstale":"2024-03-24T10:40:05.575166+0000","last_undegraded":"2024-03-24T10:40:05.575166+0000","last_fullsized":"2024-03-24T10:40:05.575166+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:18:41.362740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412756+0000","last_change":"2024-03-24T10:38:10.873871+0000","last_active":"2024-03-24T10:40:05.412756+0000","last_peered":"2024-03-24T10:40:05.412756+0000","last_clean":"2024-03-24T10:40:05.412756+0000","last_became_active":"2024-03-24T10:38:10.873528+0000","last_became_peered":"2024-03-24T10:38:10.873528+0000","last_unstale":"2024-03-24T10:40:05.412756+0000","last_undegraded":"2024-03-24T10:40:05.412756+0000","last_fullsized":"2024-03-24T10:40:05.412756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:33.372195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413424+0000","last_change":"2024-03-24T10:38:09.861548+0000","last_active":"2024-03-24T10:40:05.413424+0000","last_peered":"2024-03-24T10:40:05.413424+0000","last_clean":"2024-03-24T10:40:05.413424+0000","last_became_active":"2024-03-24T10:38:09.861219+0000","last_became_peered":"2024-03-24T10:38:09.861219+0000","last_unstale":"2024-03-24T10:40:05.413424+0000","last_undegraded":"2024-03-24T10:40:05.413424+0000","last_fullsized":"2024-03-24T10:40:05.413424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:57:46.398179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574856+0000","last_change":"2024-03-24T10:38:10.880202+0000","last_active":"2024-03-24T10:40:05.574856+0000","last_peered":"2024-03-24T10:40:05.574856+0000","last_clean":"2024-03-24T10:40:05.574856+0000","last_became_active":"2024-03-24T10:38:10.879807+0000","last_became_peered":"2024-03-24T10:38:10.879807+0000","last_unstale":"2024-03-24T10:40:05.574856+0000","last_undegraded":"2024-03-24T10:40:05.574856+0000","last_fullsized":"2024-03-24T10:40:05.574856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:37.498728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575135+0000","last_change":"2024-03-24T10:39:16.704470+0000","last_active":"2024-03-24T10:40:05.575135+0000","last_peered":"2024-03-24T10:40:05.575135+0000","last_clean":"2024-03-24T10:40:05.575135+0000","last_became_active":"2024-03-24T10:39:16.701168+0000","last_became_peered":"2024-03-24T10:39:16.701168+0000","last_unstale":"2024-03-24T10:40:05.575135+0000","last_undegraded":"2024-03-24T10:40:05.575135+0000","last_fullsized":"2024-03-24T10:40:05.575135+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:20:34.781782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413602+0000","last_change":"2024-03-24T10:38:10.884371+0000","last_active":"2024-03-24T10:40:05.413602+0000","last_peered":"2024-03-24T10:40:05.413602+0000","last_clean":"2024-03-24T10:40:05.413602+0000","last_became_active":"2024-03-24T10:38:10.884022+0000","last_became_peered":"2024-03-24T10:38:10.884022+0000","last_unstale":"2024-03-24T10:40:05.413602+0000","last_undegraded":"2024-03-24T10:40:05.413602+0000","last_fullsized":"2024-03-24T10:40:05.413602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:47:48.778727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412949+0000","last_change":"2024-03-24T10:38:09.863231+0000","last_active":"2024-03-24T10:40:05.412949+0000","last_peered":"2024-03-24T10:40:05.412949+0000","last_clean":"2024-03-24T10:40:05.412949+0000","last_became_active":"2024-03-24T10:38:09.862848+0000","last_became_peered":"2024-03-24T10:38:09.862848+0000","last_unstale":"2024-03-24T10:40:05.412949+0000","last_undegraded":"2024-03-24T10:40:05.412949+0000","last_fullsized":"2024-03-24T10:40:05.412949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:36:57.532746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300075+0000","last_change":"2024-03-24T10:38:10.875187+0000","last_active":"2024-03-24T10:40:05.300075+0000","last_peered":"2024-03-24T10:40:05.300075+0000","last_clean":"2024-03-24T10:40:05.300075+0000","last_became_active":"2024-03-24T10:38:10.874682+0000","last_became_peered":"2024-03-24T10:38:10.874682+0000","last_unstale":"2024-03-24T10:40:05.300075+0000","last_undegraded":"2024-03-24T10:40:05.300075+0000","last_fullsized":"2024-03-24T10:40:05.300075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:53:56.915312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575456+0000","last_change":"2024-03-24T10:38:09.863430+0000","last_active":"2024-03-24T10:40:05.575456+0000","last_peered":"2024-03-24T10:40:05.575456+0000","last_clean":"2024-03-24T10:40:05.575456+0000","last_became_active":"2024-03-24T10:38:09.862877+0000","last_became_peered":"2024-03-24T10:38:09.862877+0000","last_unstale":"2024-03-24T10:40:05.575456+0000","last_undegraded":"2024-03-24T10:40:05.575456+0000","last_fullsized":"2024-03-24T10:40:05.575456+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:33:29.039896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300427+0000","last_change":"2024-03-24T10:38:10.878446+0000","last_active":"2024-03-24T10:40:05.300427+0000","last_peered":"2024-03-24T10:40:05.300427+0000","last_clean":"2024-03-24T10:40:05.300427+0000","last_became_active":"2024-03-24T10:38:10.877944+0000","last_became_peered":"2024-03-24T10:38:10.877944+0000","last_unstale":"2024-03-24T10:40:05.300427+0000","last_undegraded":"2024-03-24T10:40:05.300427+0000","last_fullsized":"2024-03-24T10:40:05.300427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:00:22.568669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.525634+0000","last_change":"2024-03-24T10:38:09.867328+0000","last_active":"2024-03-24T10:40:05.525634+0000","last_peered":"2024-03-24T10:40:05.525634+0000","last_clean":"2024-03-24T10:40:05.525634+0000","last_became_active":"2024-03-24T10:38:09.866869+0000","last_became_peered":"2024-03-24T10:38:09.866869+0000","last_unstale":"2024-03-24T10:40:05.525634+0000","last_undegraded":"2024-03-24T10:40:05.525634+0000","last_fullsized":"2024-03-24T10:40:05.525634+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:52:32.136151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300060+0000","last_change":"2024-03-24T10:39:16.705912+0000","last_active":"2024-03-24T10:40:05.300060+0000","last_peered":"2024-03-24T10:40:05.300060+0000","last_clean":"2024-03-24T10:40:05.300060+0000","last_became_active":"2024-03-24T10:39:16.699809+0000","last_became_peered":"2024-03-24T10:39:16.699809+0000","last_unstale":"2024-03-24T10:40:05.300060+0000","last_undegraded":"2024-03-24T10:40:05.300060+0000","last_fullsized":"2024-03-24T10:40:05.300060+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:31.858976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575611+0000","last_change":"2024-03-24T10:38:09.865399+0000","last_active":"2024-03-24T10:40:05.575611+0000","last_peered":"2024-03-24T10:40:05.575611+0000","last_clean":"2024-03-24T10:40:05.575611+0000","last_became_active":"2024-03-24T10:38:09.864906+0000","last_became_peered":"2024-03-24T10:38:09.864906+0000","last_unstale":"2024-03-24T10:40:05.575611+0000","last_undegraded":"2024-03-24T10:40:05.575611+0000","last_fullsized":"2024-03-24T10:40:05.575611+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:53:12.363018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300349+0000","last_change":"2024-03-24T10:38:10.876881+0000","last_active":"2024-03-24T10:40:05.300349+0000","last_peered":"2024-03-24T10:40:05.300349+0000","last_clean":"2024-03-24T10:40:05.300349+0000","last_became_active":"2024-03-24T10:38:10.876479+0000","last_became_peered":"2024-03-24T10:38:10.876479+0000","last_unstale":"2024-03-24T10:40:05.300349+0000","last_undegraded":"2024-03-24T10:40:05.300349+0000","last_fullsized":"2024-03-24T10:40:05.300349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:57:33.140451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574900+0000","last_change":"2024-03-24T10:38:10.872622+0000","last_active":"2024-03-24T10:40:05.574900+0000","last_peered":"2024-03-24T10:40:05.574900+0000","last_clean":"2024-03-24T10:40:05.574900+0000","last_became_active":"2024-03-24T10:38:10.872293+0000","last_became_peered":"2024-03-24T10:38:10.872293+0000","last_unstale":"2024-03-24T10:40:05.574900+0000","last_undegraded":"2024-03-24T10:40:05.574900+0000","last_fullsized":"2024-03-24T10:40:05.574900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:48.139589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575020+0000","last_change":"2024-03-24T10:39:16.701548+0000","last_active":"2024-03-24T10:40:05.575020+0000","last_peered":"2024-03-24T10:40:05.575020+0000","last_clean":"2024-03-24T10:40:05.575020+0000","last_became_active":"2024-03-24T10:39:16.701213+0000","last_became_peered":"2024-03-24T10:39:16.701213+0000","last_unstale":"2024-03-24T10:40:05.575020+0000","last_undegraded":"2024-03-24T10:40:05.575020+0000","last_fullsized":"2024-03-24T10:40:05.575020+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:50:28.131754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574960+0000","last_change":"2024-03-24T10:38:10.879679+0000","last_active":"2024-03-24T10:40:05.574960+0000","last_peered":"2024-03-24T10:40:05.574960+0000","last_clean":"2024-03-24T10:40:05.574960+0000","last_became_active":"2024-03-24T10:38:10.879334+0000","last_became_peered":"2024-03-24T10:38:10.879334+0000","last_unstale":"2024-03-24T10:40:05.574960+0000","last_undegraded":"2024-03-24T10:40:05.574960+0000","last_fullsized":"2024-03-24T10:40:05.574960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:27:53.517368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576067+0000","last_change":"2024-03-24T10:39:16.704444+0000","last_active":"2024-03-24T10:40:05.576067+0000","last_peered":"2024-03-24T10:40:05.576067+0000","last_clean":"2024-03-24T10:40:05.576067+0000","last_became_active":"2024-03-24T10:39:16.704085+0000","last_became_peered":"2024-03-24T10:39:16.704085+0000","last_unstale":"2024-03-24T10:40:05.576067+0000","last_undegraded":"2024-03-24T10:40:05.576067+0000","last_fullsized":"2024-03-24T10:40:05.576067+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:04:49.643195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575450+0000","last_change":"2024-03-24T10:38:10.887108+0000","last_active":"2024-03-24T10:40:05.575450+0000","last_peered":"2024-03-24T10:40:05.575450+0000","last_clean":"2024-03-24T10:40:05.575450+0000","last_became_active":"2024-03-24T10:38:10.886770+0000","last_became_peered":"2024-03-24T10:38:10.886770+0000","last_unstale":"2024-03-24T10:40:05.575450+0000","last_undegraded":"2024-03-24T10:40:05.575450+0000","last_fullsized":"2024-03-24T10:40:05.575450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:56.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575209+0000","last_change":"2024-03-24T10:38:10.873369+0000","last_active":"2024-03-24T10:40:05.575209+0000","last_peered":"2024-03-24T10:40:05.575209+0000","last_clean":"2024-03-24T10:40:05.575209+0000","last_became_active":"2024-03-24T10:38:10.873046+0000","last_became_peered":"2024-03-24T10:38:10.873046+0000","last_unstale":"2024-03-24T10:40:05.575209+0000","last_undegraded":"2024-03-24T10:40:05.575209+0000","last_fullsized":"2024-03-24T10:40:05.575209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:46.071084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575565+0000","last_change":"2024-03-24T10:38:10.878666+0000","last_active":"2024-03-24T10:40:05.575565+0000","last_peered":"2024-03-24T10:40:05.575565+0000","last_clean":"2024-03-24T10:40:05.575565+0000","last_became_active":"2024-03-24T10:38:10.877250+0000","last_became_peered":"2024-03-24T10:38:10.877250+0000","last_unstale":"2024-03-24T10:40:05.575565+0000","last_undegraded":"2024-03-24T10:40:05.575565+0000","last_fullsized":"2024-03-24T10:40:05.575565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:13:36.793584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413854+0000","last_change":"2024-03-24T10:38:10.889274+0000","last_active":"2024-03-24T10:40:05.413854+0000","last_peered":"2024-03-24T10:40:05.413854+0000","last_clean":"2024-03-24T10:40:05.413854+0000","last_became_active":"2024-03-24T10:38:10.888899+0000","last_became_peered":"2024-03-24T10:38:10.888899+0000","last_unstale":"2024-03-24T10:40:05.413854+0000","last_undegraded":"2024-03-24T10:40:05.413854+0000","last_fullsized":"2024-03-24T10:40:05.413854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:11.225562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299977+0000","last_change":"2024-03-24T10:38:10.887135+0000","last_active":"2024-03-24T10:40:05.299977+0000","last_peered":"2024-03-24T10:40:05.299977+0000","last_clean":"2024-03-24T10:40:05.299977+0000","last_became_active":"2024-03-24T10:38:10.886812+0000","last_became_peered":"2024-03-24T10:38:10.886812+0000","last_unstale":"2024-03-24T10:40:05.299977+0000","last_undegraded":"2024-03-24T10:40:05.299977+0000","last_fullsized":"2024-03-24T10:40:05.299977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:51:53.959914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413259+0000","last_change":"2024-03-24T10:38:10.885417+0000","last_active":"2024-03-24T10:40:05.413259+0000","last_peered":"2024-03-24T10:40:05.413259+0000","last_clean":"2024-03-24T10:40:05.413259+0000","last_became_active":"2024-03-24T10:38:10.883209+0000","last_became_peered":"2024-03-24T10:38:10.883209+0000","last_unstale":"2024-03-24T10:40:05.413259+0000","last_undegraded":"2024-03-24T10:40:05.413259+0000","last_fullsized":"2024-03-24T10:40:05.413259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:52:52.535611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575193+0000","last_change":"2024-03-24T10:38:10.885005+0000","last_active":"2024-03-24T10:40:05.575193+0000","last_peered":"2024-03-24T10:40:05.575193+0000","last_clean":"2024-03-24T10:40:05.575193+0000","last_became_active":"2024-03-24T10:38:10.884680+0000","last_became_peered":"2024-03-24T10:38:10.884680+0000","last_unstale":"2024-03-24T10:40:05.575193+0000","last_undegraded":"2024-03-24T10:40:05.575193+0000","last_fullsized":"2024-03-24T10:40:05.575193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:56.377777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575580+0000","last_change":"2024-03-24T10:38:10.875153+0000","last_active":"2024-03-24T10:40:05.575580+0000","last_peered":"2024-03-24T10:40:05.575580+0000","last_clean":"2024-03-24T10:40:05.575580+0000","last_became_active":"2024-03-24T10:38:10.874721+0000","last_became_peered":"2024-03-24T10:38:10.874721+0000","last_unstale":"2024-03-24T10:40:05.575580+0000","last_undegraded":"2024-03-24T10:40:05.575580+0000","last_fullsized":"2024-03-24T10:40:05.575580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:52.522797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575025+0000","last_change":"2024-03-24T10:38:10.887298+0000","last_active":"2024-03-24T10:40:05.575025+0000","last_peered":"2024-03-24T10:40:05.575025+0000","last_clean":"2024-03-24T10:40:05.575025+0000","last_became_active":"2024-03-24T10:38:10.886975+0000","last_became_peered":"2024-03-24T10:38:10.886975+0000","last_unstale":"2024-03-24T10:40:05.575025+0000","last_undegraded":"2024-03-24T10:40:05.575025+0000","last_fullsized":"2024-03-24T10:40:05.575025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:09.740303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595166,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":791,"num_read_kb":503,"num_write":285,"num_write_kb":1433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2015232,"data_stored":1785498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383716,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":465380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645353,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32936,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712216,"statfs":{"total":95995035648,"available":95961309184,"internally_reserved":0,"allocated":1802240,"data_stored":1253329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808878,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1380352,"data_stored":858949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32520,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712632,"statfs":{"total":95995035648,"available":95961735168,"internally_reserved":0,"allocated":1372160,"data_stored":858170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1576960,"data_stored":1058697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":465005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":530498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":466276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":532244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28492,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716660,"statfs":{"total":95995035648,"available":95965859840,"internally_reserved":0,"allocated":1511424,"data_stored":990015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":983040,"data_stored":466086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28700,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716452,"statfs":{"total":95995035648,"available":95965646848,"internally_reserved":0,"allocated":1724416,"data_stored":1189449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":65700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":67446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":393372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":198828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:40:09.699 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:40:09.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.845+0000 7fe7824d4640 1 -- 172.21.15.23:0/2705921291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe77c1016b0 msgr2=0x7fe77c101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:09.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.845+0000 7fe7824d4640 1 --2- 172.21.15.23:0/2705921291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe77c1016b0 0x7fe77c101b30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe770009930 tx=0x7fe77002bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:09.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.845+0000 7fe7824d4640 1 -- 172.21.15.23:0/2705921291 shutdown_connections 2024-03-24T10:40:09.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.845+0000 7fe7824d4640 1 --2- 172.21.15.23:0/2705921291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe77c102070 0x7fe77c10e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.845+0000 7fe7824d4640 1 --2- 172.21.15.23:0/2705921291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe77c1016b0 0x7fe77c101b30 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.845+0000 7fe7824d4640 1 --2- 172.21.15.23:0/2705921291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe77c1076c0 0x7fe77c107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.845+0000 7fe7824d4640 1 -- 172.21.15.23:0/2705921291 >> 172.21.15.23:0/2705921291 conn(0x7fe77c0fd1a0 msgr2=0x7fe77c0ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:09.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.845+0000 7fe7824d4640 1 -- 172.21.15.23:0/2705921291 shutdown_connections 2024-03-24T10:40:09.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.845+0000 7fe7824d4640 1 -- 172.21.15.23:0/2705921291 wait complete. 2024-03-24T10:40:09.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.846+0000 7fe7824d4640 1 Processor -- start 2024-03-24T10:40:09.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.846+0000 7fe7824d4640 1 -- start start 2024-03-24T10:40:09.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.846+0000 7fe7824d4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe77c1016b0 0x7fe77c197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:09.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.846+0000 7fe7824d4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe77c102070 0x7fe77c1982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:09.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.846+0000 7fe7824d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe77c1076c0 0x7fe77c19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:09.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.846+0000 7fe7824d4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c19cc20 con 0x7fe77c1016b0 2024-03-24T10:40:09.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.846+0000 7fe7824d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c19cd90 con 0x7fe77c1076c0 2024-03-24T10:40:09.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.846+0000 7fe7824d4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c19cf50 con 0x7fe77c102070 2024-03-24T10:40:09.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.846+0000 7fe77bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe77c1016b0 0x7fe77c197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:09.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.846+0000 7fe77bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe77c1016b0 0x7fe77c197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33478/0 (socket says 172.21.15.23:33478) 2024-03-24T10:40:09.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.847+0000 7fe77b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe77c102070 0x7fe77c1982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:09.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.846+0000 7fe780a71640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe77c1076c0 0x7fe77c19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:09.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.847+0000 7fe77bfff640 1 -- 172.21.15.23:0/2930031076 learned_addr learned my addr 172.21.15.23:0/2930031076 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:09.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.847+0000 7fe77bfff640 1 -- 172.21.15.23:0/2930031076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe77c102070 msgr2=0x7fe77c1982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:09.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.847+0000 7fe77bfff640 1 --2- 172.21.15.23:0/2930031076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe77c102070 0x7fe77c1982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.847+0000 7fe77bfff640 1 -- 172.21.15.23:0/2930031076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe77c1076c0 msgr2=0x7fe77c19c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:09.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.847+0000 7fe77bfff640 1 --2- 172.21.15.23:0/2930031076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe77c1076c0 0x7fe77c19c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:09.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.847+0000 7fe77bfff640 1 -- 172.21.15.23:0/2930031076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe770009560 con 0x7fe77c1016b0 2024-03-24T10:40:09.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.847+0000 7fe77bfff640 1 --2- 172.21.15.23:0/2930031076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe77c1016b0 0x7fe77c197d60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe76c00e9f0 tx=0x7fe76c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:09.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.847+0000 7fe7797fa640 1 -- 172.21.15.23:0/2930031076 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c009700 con 0x7fe77c1016b0 2024-03-24T10:40:09.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.847+0000 7fe7797fa640 1 -- 172.21.15.23:0/2930031076 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe76c00cd80 con 0x7fe77c1016b0 2024-03-24T10:40:09.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.847+0000 7fe7824d4640 1 -- 172.21.15.23:0/2930031076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe77c19d230 con 0x7fe77c1016b0 2024-03-24T10:40:09.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.847+0000 7fe7797fa640 1 -- 172.21.15.23:0/2930031076 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c018920 con 0x7fe77c1016b0 2024-03-24T10:40:09.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.847+0000 7fe7824d4640 1 -- 172.21.15.23:0/2930031076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe77c1a4ab0 con 0x7fe77c1016b0 2024-03-24T10:40:09.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.848+0000 7fe7824d4640 1 -- 172.21.15.23:0/2930031076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe77c068500 con 0x7fe77c1016b0 2024-03-24T10:40:09.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.850+0000 7fe7797fa640 1 -- 172.21.15.23:0/2930031076 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7fe76c018a80 con 0x7fe77c1016b0 2024-03-24T10:40:09.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.850+0000 7fe7797fa640 1 --2- 172.21.15.23:0/2930031076 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe7640a2350 0x7fe7640a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:09.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.850+0000 7fe7797fa640 1 -- 172.21.15.23:0/2930031076 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe76c014070 con 0x7fe77c1016b0 2024-03-24T10:40:09.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.850+0000 7fe77b7fe640 1 --2- 172.21.15.23:0/2930031076 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe7640a2350 0x7fe7640a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:09.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.851+0000 7fe77b7fe640 1 --2- 172.21.15.23:0/2930031076 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe7640a2350 0x7fe7640a4810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe770009900 tx=0x7fe770005a20 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:09.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:09.854+0000 7fe7797fa640 1 -- 172.21.15.23:0/2930031076 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe76c097560 con 0x7fe77c1016b0 2024-03-24T10:40:10.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.049+0000 7fe7824d4640 1 -- 172.21.15.23:0/2930031076 --> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe77c1118a0 con 0x7fe7640a2350 2024-03-24T10:40:10.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.056+0000 7fe7797fa640 1 -- 172.21.15.23:0/2930031076 <== mgr.15288 v2:172.21.15.23:6800/581236369 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257744 (secure 0 0 0) 0x7fe77c1118a0 con 0x7fe7640a2350 2024-03-24T10:40:10.060 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:40:10.061 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-24T10:40:10.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.060+0000 7fe7824d4640 1 -- 172.21.15.23:0/2930031076 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe7640a2350 msgr2=0x7fe7640a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.060+0000 7fe7824d4640 1 --2- 172.21.15.23:0/2930031076 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe7640a2350 0x7fe7640a4810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe770009900 tx=0x7fe770005a20 comp rx=0 tx=0).stop 2024-03-24T10:40:10.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.060+0000 7fe7824d4640 1 -- 172.21.15.23:0/2930031076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe77c1016b0 msgr2=0x7fe77c197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.060+0000 7fe7824d4640 1 --2- 172.21.15.23:0/2930031076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe77c1016b0 0x7fe77c197d60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe76c00e9f0 tx=0x7fe76c00eeb0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.060+0000 7fe7824d4640 1 -- 172.21.15.23:0/2930031076 shutdown_connections 2024-03-24T10:40:10.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.060+0000 7fe7824d4640 1 --2- 172.21.15.23:0/2930031076 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe7640a2350 0x7fe7640a4810 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.060+0000 7fe7824d4640 1 --2- 172.21.15.23:0/2930031076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe77c1076c0 0x7fe77c19c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.060+0000 7fe7824d4640 1 --2- 172.21.15.23:0/2930031076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe77c102070 0x7fe77c1982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.060+0000 7fe7824d4640 1 --2- 172.21.15.23:0/2930031076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe77c1016b0 0x7fe77c197d60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.060+0000 7fe7824d4640 1 -- 172.21.15.23:0/2930031076 >> 172.21.15.23:0/2930031076 conn(0x7fe77c0fd1a0 msgr2=0x7fe77c0fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:10.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.060+0000 7fe7824d4640 1 -- 172.21.15.23:0/2930031076 shutdown_connections 2024-03-24T10:40:10.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.060+0000 7fe7824d4640 1 -- 172.21.15.23:0/2930031076 wait complete. 2024-03-24T10:40:10.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi117 to 174.1M 2024-03-24T10:40:10.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: Unable to set osd_memory_target on smithi117 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T10:40:10.070 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-24T10:40:10.071 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: Unable to set osd_memory_target on smithi023 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-03-24T10:40:10.071 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi116 to 174.1M 2024-03-24T10:40:10.071 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: Unable to set osd_memory_target on smithi116 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-24T10:40:10.071 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-24T10:40:10.071 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T10:40:10.071 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T10:40:10.071 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:40:10.071 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: Updating smithi023:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:40:10.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:40:10.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2823891709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:40:10.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4158847352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:40:10.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.072 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:09 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.073 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-03-24T10:40:09.312178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185768,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":927,"num_read_kb":622,"num_write":468,"num_write_kb":4373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1181547,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":863,"ondisk_log_size":863,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368356,"kb_used_data":15012,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124573468,"statfs":{"total":1151940427776,"available":1151563231232,"internally_reserved":0,"allocated":15372288,"data_stored":9134098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.992192"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575582+0000","last_change":"2024-03-24T10:38:10.882385+0000","last_active":"2024-03-24T10:40:05.575582+0000","last_peered":"2024-03-24T10:40:05.575582+0000","last_clean":"2024-03-24T10:40:05.575582+0000","last_became_active":"2024-03-24T10:38:10.882062+0000","last_became_peered":"2024-03-24T10:38:10.882062+0000","last_unstale":"2024-03-24T10:40:05.575582+0000","last_undegraded":"2024-03-24T10:40:05.575582+0000","last_fullsized":"2024-03-24T10:40:05.575582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:25.151062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575927+0000","last_change":"2024-03-24T10:38:10.878001+0000","last_active":"2024-03-24T10:40:05.575927+0000","last_peered":"2024-03-24T10:40:05.575927+0000","last_clean":"2024-03-24T10:40:05.575927+0000","last_became_active":"2024-03-24T10:38:10.877661+0000","last_became_peered":"2024-03-24T10:38:10.877661+0000","last_unstale":"2024-03-24T10:40:05.575927+0000","last_undegraded":"2024-03-24T10:40:05.575927+0000","last_fullsized":"2024-03-24T10:40:05.575927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:40:19.062969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576049+0000","last_change":"2024-03-24T10:38:10.891809+0000","last_active":"2024-03-24T10:40:05.576049+0000","last_peered":"2024-03-24T10:40:05.576049+0000","last_clean":"2024-03-24T10:40:05.576049+0000","last_became_active":"2024-03-24T10:38:10.890354+0000","last_became_peered":"2024-03-24T10:38:10.890354+0000","last_unstale":"2024-03-24T10:40:05.576049+0000","last_undegraded":"2024-03-24T10:40:05.576049+0000","last_fullsized":"2024-03-24T10:40:05.576049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:59:26.037914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575921+0000","last_change":"2024-03-24T10:38:10.885097+0000","last_active":"2024-03-24T10:40:05.575921+0000","last_peered":"2024-03-24T10:40:05.575921+0000","last_clean":"2024-03-24T10:40:05.575921+0000","last_became_active":"2024-03-24T10:38:10.884842+0000","last_became_peered":"2024-03-24T10:38:10.884842+0000","last_unstale":"2024-03-24T10:40:05.575921+0000","last_undegraded":"2024-03-24T10:40:05.575921+0000","last_fullsized":"2024-03-24T10:40:05.575921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:39:44.085699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412906+0000","last_change":"2024-03-24T10:38:10.877653+0000","last_active":"2024-03-24T10:40:05.412906+0000","last_peered":"2024-03-24T10:40:05.412906+0000","last_clean":"2024-03-24T10:40:05.412906+0000","last_became_active":"2024-03-24T10:38:10.877249+0000","last_became_peered":"2024-03-24T10:38:10.877249+0000","last_unstale":"2024-03-24T10:40:05.412906+0000","last_undegraded":"2024-03-24T10:40:05.412906+0000","last_fullsized":"2024-03-24T10:40:05.412906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:17.504660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422853+0000","last_change":"2024-03-24T10:38:10.876584+0000","last_active":"2024-03-24T10:40:05.422853+0000","last_peered":"2024-03-24T10:40:05.422853+0000","last_clean":"2024-03-24T10:40:05.422853+0000","last_became_active":"2024-03-24T10:38:10.876173+0000","last_became_peered":"2024-03-24T10:38:10.876173+0000","last_unstale":"2024-03-24T10:40:05.422853+0000","last_undegraded":"2024-03-24T10:40:05.422853+0000","last_fullsized":"2024-03-24T10:40:05.422853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:33.899155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299791+0000","last_change":"2024-03-24T10:38:10.874206+0000","last_active":"2024-03-24T10:40:05.299791+0000","last_peered":"2024-03-24T10:40:05.299791+0000","last_clean":"2024-03-24T10:40:05.299791+0000","last_became_active":"2024-03-24T10:38:10.873779+0000","last_became_peered":"2024-03-24T10:38:10.873779+0000","last_unstale":"2024-03-24T10:40:05.299791+0000","last_undegraded":"2024-03-24T10:40:05.299791+0000","last_fullsized":"2024-03-24T10:40:05.299791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:08:19.928998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413254+0000","last_change":"2024-03-24T10:38:10.884219+0000","last_active":"2024-03-24T10:40:05.413254+0000","last_peered":"2024-03-24T10:40:05.413254+0000","last_clean":"2024-03-24T10:40:05.413254+0000","last_became_active":"2024-03-24T10:38:10.883863+0000","last_became_peered":"2024-03-24T10:38:10.883863+0000","last_unstale":"2024-03-24T10:40:05.413254+0000","last_undegraded":"2024-03-24T10:40:05.413254+0000","last_fullsized":"2024-03-24T10:40:05.413254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:26:34.932083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575665+0000","last_change":"2024-03-24T10:38:10.881297+0000","last_active":"2024-03-24T10:40:05.575665+0000","last_peered":"2024-03-24T10:40:05.575665+0000","last_clean":"2024-03-24T10:40:05.575665+0000","last_became_active":"2024-03-24T10:38:10.879871+0000","last_became_peered":"2024-03-24T10:38:10.879871+0000","last_unstale":"2024-03-24T10:40:05.575665+0000","last_undegraded":"2024-03-24T10:40:05.575665+0000","last_fullsized":"2024-03-24T10:40:05.575665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:05.132788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422954+0000","last_change":"2024-03-24T10:38:10.876648+0000","last_active":"2024-03-24T10:40:05.422954+0000","last_peered":"2024-03-24T10:40:05.422954+0000","last_clean":"2024-03-24T10:40:05.422954+0000","last_became_active":"2024-03-24T10:38:10.876113+0000","last_became_peered":"2024-03-24T10:38:10.876113+0000","last_unstale":"2024-03-24T10:40:05.422954+0000","last_undegraded":"2024-03-24T10:40:05.422954+0000","last_fullsized":"2024-03-24T10:40:05.422954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:29:18.282797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300282+0000","last_change":"2024-03-24T10:38:10.875836+0000","last_active":"2024-03-24T10:40:05.300282+0000","last_peered":"2024-03-24T10:40:05.300282+0000","last_clean":"2024-03-24T10:40:05.300282+0000","last_became_active":"2024-03-24T10:38:10.875522+0000","last_became_peered":"2024-03-24T10:38:10.875522+0000","last_unstale":"2024-03-24T10:40:05.300282+0000","last_undegraded":"2024-03-24T10:40:05.300282+0000","last_fullsized":"2024-03-24T10:40:05.300282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:41.282123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575678+0000","last_change":"2024-03-24T10:38:10.884567+0000","last_active":"2024-03-24T10:40:05.575678+0000","last_peered":"2024-03-24T10:40:05.575678+0000","last_clean":"2024-03-24T10:40:05.575678+0000","last_became_active":"2024-03-24T10:38:10.884195+0000","last_became_peered":"2024-03-24T10:38:10.884195+0000","last_unstale":"2024-03-24T10:40:05.575678+0000","last_undegraded":"2024-03-24T10:40:05.575678+0000","last_fullsized":"2024-03-24T10:40:05.575678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:29.286631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575744+0000","last_change":"2024-03-24T10:39:16.726561+0000","last_active":"2024-03-24T10:40:05.575744+0000","last_peered":"2024-03-24T10:40:05.575744+0000","last_clean":"2024-03-24T10:40:05.575744+0000","last_became_active":"2024-03-24T10:39:16.726088+0000","last_became_peered":"2024-03-24T10:39:16.726088+0000","last_unstale":"2024-03-24T10:40:05.575744+0000","last_undegraded":"2024-03-24T10:40:05.575744+0000","last_fullsized":"2024-03-24T10:40:05.575744+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:08:46.638610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575643+0000","last_change":"2024-03-24T10:38:10.879942+0000","last_active":"2024-03-24T10:40:05.575643+0000","last_peered":"2024-03-24T10:40:05.575643+0000","last_clean":"2024-03-24T10:40:05.575643+0000","last_became_active":"2024-03-24T10:38:10.879396+0000","last_became_peered":"2024-03-24T10:38:10.879396+0000","last_unstale":"2024-03-24T10:40:05.575643+0000","last_undegraded":"2024-03-24T10:40:05.575643+0000","last_fullsized":"2024-03-24T10:40:05.575643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:41.477988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575389+0000","last_change":"2024-03-24T10:38:10.886667+0000","last_active":"2024-03-24T10:40:05.575389+0000","last_peered":"2024-03-24T10:40:05.575389+0000","last_clean":"2024-03-24T10:40:05.575389+0000","last_became_active":"2024-03-24T10:38:10.886293+0000","last_became_peered":"2024-03-24T10:38:10.886293+0000","last_unstale":"2024-03-24T10:40:05.575389+0000","last_undegraded":"2024-03-24T10:40:05.575389+0000","last_fullsized":"2024-03-24T10:40:05.575389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:43:43.414573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574995+0000","last_change":"2024-03-24T10:38:10.883671+0000","last_active":"2024-03-24T10:40:05.574995+0000","last_peered":"2024-03-24T10:40:05.574995+0000","last_clean":"2024-03-24T10:40:05.574995+0000","last_became_active":"2024-03-24T10:38:10.883013+0000","last_became_peered":"2024-03-24T10:38:10.883013+0000","last_unstale":"2024-03-24T10:40:05.574995+0000","last_undegraded":"2024-03-24T10:40:05.574995+0000","last_fullsized":"2024-03-24T10:40:05.574995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:33.936543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575587+0000","last_change":"2024-03-24T10:38:10.887286+0000","last_active":"2024-03-24T10:40:05.575587+0000","last_peered":"2024-03-24T10:40:05.575587+0000","last_clean":"2024-03-24T10:40:05.575587+0000","last_became_active":"2024-03-24T10:38:10.886918+0000","last_became_peered":"2024-03-24T10:38:10.886918+0000","last_unstale":"2024-03-24T10:40:05.575587+0000","last_undegraded":"2024-03-24T10:40:05.575587+0000","last_fullsized":"2024-03-24T10:40:05.575587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:40.266589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.298339+0000","last_change":"2024-03-24T10:38:10.874476+0000","last_active":"2024-03-24T10:40:05.298339+0000","last_peered":"2024-03-24T10:40:05.298339+0000","last_clean":"2024-03-24T10:40:05.298339+0000","last_became_active":"2024-03-24T10:38:10.874123+0000","last_became_peered":"2024-03-24T10:38:10.874123+0000","last_unstale":"2024-03-24T10:40:05.298339+0000","last_undegraded":"2024-03-24T10:40:05.298339+0000","last_fullsized":"2024-03-24T10:40:05.298339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:29.759320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575455+0000","last_change":"2024-03-24T10:38:10.878758+0000","last_active":"2024-03-24T10:40:05.575455+0000","last_peered":"2024-03-24T10:40:05.575455+0000","last_clean":"2024-03-24T10:40:05.575455+0000","last_became_active":"2024-03-24T10:38:10.878424+0000","last_became_peered":"2024-03-24T10:38:10.878424+0000","last_unstale":"2024-03-24T10:40:05.575455+0000","last_undegraded":"2024-03-24T10:40:05.575455+0000","last_fullsized":"2024-03-24T10:40:05.575455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:15:19.273108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.298282+0000","last_change":"2024-03-24T10:39:16.709304+0000","last_active":"2024-03-24T10:40:05.298282+0000","last_peered":"2024-03-24T10:40:05.298282+0000","last_clean":"2024-03-24T10:40:05.298282+0000","last_became_active":"2024-03-24T10:39:16.700325+0000","last_became_peered":"2024-03-24T10:39:16.700325+0000","last_unstale":"2024-03-24T10:40:05.298282+0000","last_undegraded":"2024-03-24T10:40:05.298282+0000","last_fullsized":"2024-03-24T10:40:05.298282+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:57.633288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.414021+0000","last_change":"2024-03-24T10:38:10.885198+0000","last_active":"2024-03-24T10:40:05.414021+0000","last_peered":"2024-03-24T10:40:05.414021+0000","last_clean":"2024-03-24T10:40:05.414021+0000","last_became_active":"2024-03-24T10:38:10.884123+0000","last_became_peered":"2024-03-24T10:38:10.884123+0000","last_unstale":"2024-03-24T10:40:05.414021+0000","last_undegraded":"2024-03-24T10:40:05.414021+0000","last_fullsized":"2024-03-24T10:40:05.414021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:37:12.510671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.414153+0000","last_change":"2024-03-24T10:38:09.866223+0000","last_active":"2024-03-24T10:40:05.414153+0000","last_peered":"2024-03-24T10:40:05.414153+0000","last_clean":"2024-03-24T10:40:05.414153+0000","last_became_active":"2024-03-24T10:38:09.865711+0000","last_became_peered":"2024-03-24T10:38:09.865711+0000","last_unstale":"2024-03-24T10:40:05.414153+0000","last_undegraded":"2024-03-24T10:40:05.414153+0000","last_fullsized":"2024-03-24T10:40:05.414153+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:26:08.380749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576281+0000","last_change":"2024-03-24T10:38:10.890557+0000","last_active":"2024-03-24T10:40:05.576281+0000","last_peered":"2024-03-24T10:40:05.576281+0000","last_clean":"2024-03-24T10:40:05.576281+0000","last_became_active":"2024-03-24T10:38:10.889883+0000","last_became_peered":"2024-03-24T10:38:10.889883+0000","last_unstale":"2024-03-24T10:40:05.576281+0000","last_undegraded":"2024-03-24T10:40:05.576281+0000","last_fullsized":"2024-03-24T10:40:05.576281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:34.731585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413316+0000","last_change":"2024-03-24T10:38:09.870253+0000","last_active":"2024-03-24T10:40:05.413316+0000","last_peered":"2024-03-24T10:40:05.413316+0000","last_clean":"2024-03-24T10:40:05.413316+0000","last_became_active":"2024-03-24T10:38:09.869932+0000","last_became_peered":"2024-03-24T10:38:09.869932+0000","last_unstale":"2024-03-24T10:40:05.413316+0000","last_undegraded":"2024-03-24T10:40:05.413316+0000","last_fullsized":"2024-03-24T10:40:05.413316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:38:43.626390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575343+0000","last_change":"2024-03-24T10:38:10.881705+0000","last_active":"2024-03-24T10:40:05.575343+0000","last_peered":"2024-03-24T10:40:05.575343+0000","last_clean":"2024-03-24T10:40:05.575343+0000","last_became_active":"2024-03-24T10:38:10.881323+0000","last_became_peered":"2024-03-24T10:38:10.881323+0000","last_unstale":"2024-03-24T10:40:05.575343+0000","last_undegraded":"2024-03-24T10:40:05.575343+0000","last_fullsized":"2024-03-24T10:40:05.575343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:25.861184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413086+0000","last_change":"2024-03-24T10:38:09.870481+0000","last_active":"2024-03-24T10:40:05.413086+0000","last_peered":"2024-03-24T10:40:05.413086+0000","last_clean":"2024-03-24T10:40:05.413086+0000","last_became_active":"2024-03-24T10:38:09.870117+0000","last_became_peered":"2024-03-24T10:38:09.870117+0000","last_unstale":"2024-03-24T10:40:05.413086+0000","last_undegraded":"2024-03-24T10:40:05.413086+0000","last_fullsized":"2024-03-24T10:40:05.413086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:37:51.871068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.297975+0000","last_change":"2024-03-24T10:39:16.906560+0000","last_active":"2024-03-24T10:40:05.297975+0000","last_peered":"2024-03-24T10:40:05.297975+0000","last_clean":"2024-03-24T10:40:05.297975+0000","last_became_active":"2024-03-24T10:39:16.906009+0000","last_became_peered":"2024-03-24T10:39:16.906009+0000","last_unstale":"2024-03-24T10:40:05.297975+0000","last_undegraded":"2024-03-24T10:40:05.297975+0000","last_fullsized":"2024-03-24T10:40:05.297975+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:45:56.513953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"88'8","reported_seq":144,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:06.037385+0000","last_change":"2024-03-24T10:38:09.864160+0000","last_active":"2024-03-24T10:40:06.037385+0000","last_peered":"2024-03-24T10:40:06.037385+0000","last_clean":"2024-03-24T10:40:06.037385+0000","last_became_active":"2024-03-24T10:38:09.862066+0000","last_became_peered":"2024-03-24T10:38:09.862066+0000","last_unstale":"2024-03-24T10:40:06.037385+0000","last_undegraded":"2024-03-24T10:40:06.037385+0000","last_fullsized":"2024-03-24T10:40:06.037385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:59.003931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412874+0000","last_change":"2024-03-24T10:38:10.874433+0000","last_active":"2024-03-24T10:40:05.412874+0000","last_peered":"2024-03-24T10:40:05.412874+0000","last_clean":"2024-03-24T10:40:05.412874+0000","last_became_active":"2024-03-24T10:38:10.874090+0000","last_became_peered":"2024-03-24T10:38:10.874090+0000","last_unstale":"2024-03-24T10:40:05.412874+0000","last_undegraded":"2024-03-24T10:40:05.412874+0000","last_fullsized":"2024-03-24T10:40:05.412874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:25.979419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422847+0000","last_change":"2024-03-24T10:38:09.863028+0000","last_active":"2024-03-24T10:40:05.422847+0000","last_peered":"2024-03-24T10:40:05.422847+0000","last_clean":"2024-03-24T10:40:05.422847+0000","last_became_active":"2024-03-24T10:38:09.862682+0000","last_became_peered":"2024-03-24T10:38:09.862682+0000","last_unstale":"2024-03-24T10:40:05.422847+0000","last_undegraded":"2024-03-24T10:40:05.422847+0000","last_fullsized":"2024-03-24T10:40:05.422847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:14.962115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300267+0000","last_change":"2024-03-24T10:38:10.885950+0000","last_active":"2024-03-24T10:40:05.300267+0000","last_peered":"2024-03-24T10:40:05.300267+0000","last_clean":"2024-03-24T10:40:05.300267+0000","last_became_active":"2024-03-24T10:38:10.885685+0000","last_became_peered":"2024-03-24T10:38:10.885685+0000","last_unstale":"2024-03-24T10:40:05.300267+0000","last_undegraded":"2024-03-24T10:40:05.300267+0000","last_fullsized":"2024-03-24T10:40:05.300267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:42:09.912743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575251+0000","last_change":"2024-03-24T10:38:09.860376+0000","last_active":"2024-03-24T10:40:05.575251+0000","last_peered":"2024-03-24T10:40:05.575251+0000","last_clean":"2024-03-24T10:40:05.575251+0000","last_became_active":"2024-03-24T10:38:09.860062+0000","last_became_peered":"2024-03-24T10:38:09.860062+0000","last_unstale":"2024-03-24T10:40:05.575251+0000","last_undegraded":"2024-03-24T10:40:05.575251+0000","last_fullsized":"2024-03-24T10:40:05.575251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:33.387953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413177+0000","last_change":"2024-03-24T10:38:10.883843+0000","last_active":"2024-03-24T10:40:05.413177+0000","last_peered":"2024-03-24T10:40:05.413177+0000","last_clean":"2024-03-24T10:40:05.413177+0000","last_became_active":"2024-03-24T10:38:10.883170+0000","last_became_peered":"2024-03-24T10:38:10.883170+0000","last_unstale":"2024-03-24T10:40:05.413177+0000","last_undegraded":"2024-03-24T10:40:05.413177+0000","last_fullsized":"2024-03-24T10:40:05.413177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:28:16.798546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299912+0000","last_change":"2024-03-24T10:38:09.862299+0000","last_active":"2024-03-24T10:40:05.299912+0000","last_peered":"2024-03-24T10:40:05.299912+0000","last_clean":"2024-03-24T10:40:05.299912+0000","last_became_active":"2024-03-24T10:38:09.861895+0000","last_became_peered":"2024-03-24T10:38:09.861895+0000","last_unstale":"2024-03-24T10:40:05.299912+0000","last_undegraded":"2024-03-24T10:40:05.299912+0000","last_fullsized":"2024-03-24T10:40:05.299912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:13:19.869511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.297954+0000","last_change":"2024-03-24T10:38:10.880507+0000","last_active":"2024-03-24T10:40:05.297954+0000","last_peered":"2024-03-24T10:40:05.297954+0000","last_clean":"2024-03-24T10:40:05.297954+0000","last_became_active":"2024-03-24T10:38:10.879790+0000","last_became_peered":"2024-03-24T10:38:10.879790+0000","last_unstale":"2024-03-24T10:40:05.297954+0000","last_undegraded":"2024-03-24T10:40:05.297954+0000","last_fullsized":"2024-03-24T10:40:05.297954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:11:02.650420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.423038+0000","last_change":"2024-03-24T10:38:09.860186+0000","last_active":"2024-03-24T10:40:05.423038+0000","last_peered":"2024-03-24T10:40:05.423038+0000","last_clean":"2024-03-24T10:40:05.423038+0000","last_became_active":"2024-03-24T10:38:09.859834+0000","last_became_peered":"2024-03-24T10:38:09.859834+0000","last_unstale":"2024-03-24T10:40:05.423038+0000","last_undegraded":"2024-03-24T10:40:05.423038+0000","last_fullsized":"2024-03-24T10:40:05.423038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:33:48.029053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575833+0000","last_change":"2024-03-24T10:38:10.875634+0000","last_active":"2024-03-24T10:40:05.575833+0000","last_peered":"2024-03-24T10:40:05.575833+0000","last_clean":"2024-03-24T10:40:05.575833+0000","last_became_active":"2024-03-24T10:38:10.875101+0000","last_became_peered":"2024-03-24T10:38:10.875101+0000","last_unstale":"2024-03-24T10:40:05.575833+0000","last_undegraded":"2024-03-24T10:40:05.575833+0000","last_fullsized":"2024-03-24T10:40:05.575833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:32:58.165087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300091+0000","last_change":"2024-03-24T10:38:09.860412+0000","last_active":"2024-03-24T10:40:05.300091+0000","last_peered":"2024-03-24T10:40:05.300091+0000","last_clean":"2024-03-24T10:40:05.300091+0000","last_became_active":"2024-03-24T10:38:09.860084+0000","last_became_peered":"2024-03-24T10:38:09.860084+0000","last_unstale":"2024-03-24T10:40:05.300091+0000","last_undegraded":"2024-03-24T10:40:05.300091+0000","last_fullsized":"2024-03-24T10:40:05.300091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:44.585482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576331+0000","last_change":"2024-03-24T10:39:16.703566+0000","last_active":"2024-03-24T10:40:05.576331+0000","last_peered":"2024-03-24T10:40:05.576331+0000","last_clean":"2024-03-24T10:40:05.576331+0000","last_became_active":"2024-03-24T10:39:16.703213+0000","last_became_peered":"2024-03-24T10:39:16.703213+0000","last_unstale":"2024-03-24T10:40:05.576331+0000","last_undegraded":"2024-03-24T10:40:05.576331+0000","last_fullsized":"2024-03-24T10:40:05.576331+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:10.888996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300419+0000","last_change":"2024-03-24T10:38:09.865191+0000","last_active":"2024-03-24T10:40:05.300419+0000","last_peered":"2024-03-24T10:40:05.300419+0000","last_clean":"2024-03-24T10:40:05.300419+0000","last_became_active":"2024-03-24T10:38:09.864470+0000","last_became_peered":"2024-03-24T10:38:09.864470+0000","last_unstale":"2024-03-24T10:40:05.300419+0000","last_undegraded":"2024-03-24T10:40:05.300419+0000","last_fullsized":"2024-03-24T10:40:05.300419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:45:00.266470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575474+0000","last_change":"2024-03-24T10:38:10.889668+0000","last_active":"2024-03-24T10:40:05.575474+0000","last_peered":"2024-03-24T10:40:05.575474+0000","last_clean":"2024-03-24T10:40:05.575474+0000","last_became_active":"2024-03-24T10:38:10.889364+0000","last_became_peered":"2024-03-24T10:38:10.889364+0000","last_unstale":"2024-03-24T10:40:05.575474+0000","last_undegraded":"2024-03-24T10:40:05.575474+0000","last_fullsized":"2024-03-24T10:40:05.575474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:57.702719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'35","reported_seq":289,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:06.037599+0000","last_change":"2024-03-24T10:38:09.861809+0000","last_active":"2024-03-24T10:40:06.037599+0000","last_peered":"2024-03-24T10:40:06.037599+0000","last_clean":"2024-03-24T10:40:06.037599+0000","last_became_active":"2024-03-24T10:38:09.861444+0000","last_became_peered":"2024-03-24T10:38:09.861444+0000","last_unstale":"2024-03-24T10:40:06.037599+0000","last_undegraded":"2024-03-24T10:40:06.037599+0000","last_fullsized":"2024-03-24T10:40:06.037599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:19.652188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65118,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":180,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413060+0000","last_change":"2024-03-24T10:38:10.883090+0000","last_active":"2024-03-24T10:40:05.413060+0000","last_peered":"2024-03-24T10:40:05.413060+0000","last_clean":"2024-03-24T10:40:05.413060+0000","last_became_active":"2024-03-24T10:38:10.880507+0000","last_became_peered":"2024-03-24T10:38:10.880507+0000","last_unstale":"2024-03-24T10:40:05.413060+0000","last_undegraded":"2024-03-24T10:40:05.413060+0000","last_fullsized":"2024-03-24T10:40:05.413060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:01.422699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575220+0000","last_change":"2024-03-24T10:38:09.868072+0000","last_active":"2024-03-24T10:40:05.575220+0000","last_peered":"2024-03-24T10:40:05.575220+0000","last_clean":"2024-03-24T10:40:05.575220+0000","last_became_active":"2024-03-24T10:38:09.867704+0000","last_became_peered":"2024-03-24T10:38:09.867704+0000","last_unstale":"2024-03-24T10:40:05.575220+0000","last_undegraded":"2024-03-24T10:40:05.575220+0000","last_fullsized":"2024-03-24T10:40:05.575220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:09:28.726969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575265+0000","last_change":"2024-03-24T10:38:10.881882+0000","last_active":"2024-03-24T10:40:05.575265+0000","last_peered":"2024-03-24T10:40:05.575265+0000","last_clean":"2024-03-24T10:40:05.575265+0000","last_became_active":"2024-03-24T10:38:10.881495+0000","last_became_peered":"2024-03-24T10:38:10.881495+0000","last_unstale":"2024-03-24T10:40:05.575265+0000","last_undegraded":"2024-03-24T10:40:05.575265+0000","last_fullsized":"2024-03-24T10:40:05.575265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:31:43.392952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300200+0000","last_change":"2024-03-24T10:38:09.867265+0000","last_active":"2024-03-24T10:40:05.300200+0000","last_peered":"2024-03-24T10:40:05.300200+0000","last_clean":"2024-03-24T10:40:05.300200+0000","last_became_active":"2024-03-24T10:38:09.866669+0000","last_became_peered":"2024-03-24T10:38:09.866669+0000","last_unstale":"2024-03-24T10:40:05.300200+0000","last_undegraded":"2024-03-24T10:40:05.300200+0000","last_fullsized":"2024-03-24T10:40:05.300200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:28.230967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300253+0000","last_change":"2024-03-24T10:38:10.886634+0000","last_active":"2024-03-24T10:40:05.300253+0000","last_peered":"2024-03-24T10:40:05.300253+0000","last_clean":"2024-03-24T10:40:05.300253+0000","last_became_active":"2024-03-24T10:38:10.886207+0000","last_became_peered":"2024-03-24T10:38:10.886207+0000","last_unstale":"2024-03-24T10:40:05.300253+0000","last_undegraded":"2024-03-24T10:40:05.300253+0000","last_fullsized":"2024-03-24T10:40:05.300253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:43:41.601693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'5","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575133+0000","last_change":"2024-03-24T10:38:09.863262+0000","last_active":"2024-03-24T10:40:05.575133+0000","last_peered":"2024-03-24T10:40:05.575133+0000","last_clean":"2024-03-24T10:40:05.575133+0000","last_became_active":"2024-03-24T10:38:09.862665+0000","last_became_peered":"2024-03-24T10:38:09.862665+0000","last_unstale":"2024-03-24T10:40:05.575133+0000","last_undegraded":"2024-03-24T10:40:05.575133+0000","last_fullsized":"2024-03-24T10:40:05.575133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:12:29.117592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299650+0000","last_change":"2024-03-24T10:38:10.880793+0000","last_active":"2024-03-24T10:40:05.299650+0000","last_peered":"2024-03-24T10:40:05.299650+0000","last_clean":"2024-03-24T10:40:05.299650+0000","last_became_active":"2024-03-24T10:38:10.880449+0000","last_became_peered":"2024-03-24T10:38:10.880449+0000","last_unstale":"2024-03-24T10:40:05.299650+0000","last_undegraded":"2024-03-24T10:40:05.299650+0000","last_fullsized":"2024-03-24T10:40:05.299650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:32:57.184014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412771+0000","last_change":"2024-03-24T10:39:16.915438+0000","last_active":"2024-03-24T10:40:05.412771+0000","last_peered":"2024-03-24T10:40:05.412771+0000","last_clean":"2024-03-24T10:40:05.412771+0000","last_became_active":"2024-03-24T10:39:16.906155+0000","last_became_peered":"2024-03-24T10:39:16.906155+0000","last_unstale":"2024-03-24T10:40:05.412771+0000","last_undegraded":"2024-03-24T10:40:05.412771+0000","last_fullsized":"2024-03-24T10:40:05.412771+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:36:12.356069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299997+0000","last_change":"2024-03-24T10:38:10.881952+0000","last_active":"2024-03-24T10:40:05.299997+0000","last_peered":"2024-03-24T10:40:05.299997+0000","last_clean":"2024-03-24T10:40:05.299997+0000","last_became_active":"2024-03-24T10:38:10.881628+0000","last_became_peered":"2024-03-24T10:38:10.881628+0000","last_unstale":"2024-03-24T10:40:05.299997+0000","last_undegraded":"2024-03-24T10:40:05.299997+0000","last_fullsized":"2024-03-24T10:40:05.299997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:24.777793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413319+0000","last_change":"2024-03-24T10:38:09.868572+0000","last_active":"2024-03-24T10:40:05.413319+0000","last_peered":"2024-03-24T10:40:05.413319+0000","last_clean":"2024-03-24T10:40:05.413319+0000","last_became_active":"2024-03-24T10:38:09.867702+0000","last_became_peered":"2024-03-24T10:38:09.867702+0000","last_unstale":"2024-03-24T10:40:05.413319+0000","last_undegraded":"2024-03-24T10:40:05.413319+0000","last_fullsized":"2024-03-24T10:40:05.413319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:27.890114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575267+0000","last_change":"2024-03-24T10:38:10.875342+0000","last_active":"2024-03-24T10:40:05.575267+0000","last_peered":"2024-03-24T10:40:05.575267+0000","last_clean":"2024-03-24T10:40:05.575267+0000","last_became_active":"2024-03-24T10:38:10.874933+0000","last_became_peered":"2024-03-24T10:38:10.874933+0000","last_unstale":"2024-03-24T10:40:05.575267+0000","last_undegraded":"2024-03-24T10:40:05.575267+0000","last_fullsized":"2024-03-24T10:40:05.575267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:56:09.139478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.298000+0000","last_change":"2024-03-24T10:38:09.857867+0000","last_active":"2024-03-24T10:40:05.298000+0000","last_peered":"2024-03-24T10:40:05.298000+0000","last_clean":"2024-03-24T10:40:05.298000+0000","last_became_active":"2024-03-24T10:38:09.857525+0000","last_became_peered":"2024-03-24T10:38:09.857525+0000","last_unstale":"2024-03-24T10:40:05.298000+0000","last_undegraded":"2024-03-24T10:40:05.298000+0000","last_fullsized":"2024-03-24T10:40:05.298000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:37.320245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422684+0000","last_change":"2024-03-24T10:38:10.878980+0000","last_active":"2024-03-24T10:40:05.422684+0000","last_peered":"2024-03-24T10:40:05.422684+0000","last_clean":"2024-03-24T10:40:05.422684+0000","last_became_active":"2024-03-24T10:38:10.878648+0000","last_became_peered":"2024-03-24T10:38:10.878648+0000","last_unstale":"2024-03-24T10:40:05.422684+0000","last_undegraded":"2024-03-24T10:40:05.422684+0000","last_fullsized":"2024-03-24T10:40:05.422684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:49.988403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299957+0000","last_change":"2024-03-24T10:38:09.863457+0000","last_active":"2024-03-24T10:40:05.299957+0000","last_peered":"2024-03-24T10:40:05.299957+0000","last_clean":"2024-03-24T10:40:05.299957+0000","last_became_active":"2024-03-24T10:38:09.863130+0000","last_became_peered":"2024-03-24T10:38:09.863130+0000","last_unstale":"2024-03-24T10:40:05.299957+0000","last_undegraded":"2024-03-24T10:40:05.299957+0000","last_fullsized":"2024-03-24T10:40:05.299957+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:16.528703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422825+0000","last_change":"2024-03-24T10:38:10.887014+0000","last_active":"2024-03-24T10:40:05.422825+0000","last_peered":"2024-03-24T10:40:05.422825+0000","last_clean":"2024-03-24T10:40:05.422825+0000","last_became_active":"2024-03-24T10:38:10.886698+0000","last_became_peered":"2024-03-24T10:38:10.886698+0000","last_unstale":"2024-03-24T10:40:05.422825+0000","last_undegraded":"2024-03-24T10:40:05.422825+0000","last_fullsized":"2024-03-24T10:40:05.422825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:47:08.571886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300143+0000","last_change":"2024-03-24T10:38:10.874811+0000","last_active":"2024-03-24T10:40:05.300143+0000","last_peered":"2024-03-24T10:40:05.300143+0000","last_clean":"2024-03-24T10:40:05.300143+0000","last_became_active":"2024-03-24T10:38:10.874483+0000","last_became_peered":"2024-03-24T10:38:10.874483+0000","last_unstale":"2024-03-24T10:40:05.300143+0000","last_undegraded":"2024-03-24T10:40:05.300143+0000","last_fullsized":"2024-03-24T10:40:05.300143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:57.382174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":137,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575176+0000","last_change":"2024-03-24T10:38:09.865614+0000","last_active":"2024-03-24T10:40:05.575176+0000","last_peered":"2024-03-24T10:40:05.575176+0000","last_clean":"2024-03-24T10:40:05.575176+0000","last_became_active":"2024-03-24T10:38:09.865317+0000","last_became_peered":"2024-03-24T10:38:09.865317+0000","last_unstale":"2024-03-24T10:40:05.575176+0000","last_undegraded":"2024-03-24T10:40:05.575176+0000","last_fullsized":"2024-03-24T10:40:05.575176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:21:14.968552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413502+0000","last_change":"2024-03-24T10:38:10.884847+0000","last_active":"2024-03-24T10:40:05.413502+0000","last_peered":"2024-03-24T10:40:05.413502+0000","last_clean":"2024-03-24T10:40:05.413502+0000","last_became_active":"2024-03-24T10:38:10.884526+0000","last_became_peered":"2024-03-24T10:38:10.884526+0000","last_unstale":"2024-03-24T10:40:05.413502+0000","last_undegraded":"2024-03-24T10:40:05.413502+0000","last_fullsized":"2024-03-24T10:40:05.413502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:01:49.364083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575339+0000","last_change":"2024-03-24T10:38:09.858893+0000","last_active":"2024-03-24T10:40:05.575339+0000","last_peered":"2024-03-24T10:40:05.575339+0000","last_clean":"2024-03-24T10:40:05.575339+0000","last_became_active":"2024-03-24T10:38:09.858474+0000","last_became_peered":"2024-03-24T10:38:09.858474+0000","last_unstale":"2024-03-24T10:40:05.575339+0000","last_undegraded":"2024-03-24T10:40:05.575339+0000","last_fullsized":"2024-03-24T10:40:05.575339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:03:36.709498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299715+0000","last_change":"2024-03-24T10:36:59.100728+0000","last_active":"2024-03-24T10:40:05.299715+0000","last_peered":"2024-03-24T10:40:05.299715+0000","last_clean":"2024-03-24T10:40:05.299715+0000","last_became_active":"2024-03-24T10:36:59.100159+0000","last_became_peered":"2024-03-24T10:36:59.100159+0000","last_unstale":"2024-03-24T10:40:05.299715+0000","last_undegraded":"2024-03-24T10:40:05.299715+0000","last_fullsized":"2024-03-24T10:40:05.299715+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:39.429553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:39.429553+0000","last_clean_scrub_stamp":"2024-03-24T10:33:39.429553+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:21:01.470585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574970+0000","last_change":"2024-03-24T10:38:10.885804+0000","last_active":"2024-03-24T10:40:05.574970+0000","last_peered":"2024-03-24T10:40:05.574970+0000","last_clean":"2024-03-24T10:40:05.574970+0000","last_became_active":"2024-03-24T10:38:10.885447+0000","last_became_peered":"2024-03-24T10:38:10.885447+0000","last_unstale":"2024-03-24T10:40:05.574970+0000","last_undegraded":"2024-03-24T10:40:05.574970+0000","last_fullsized":"2024-03-24T10:40:05.574970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:57.103205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299830+0000","last_change":"2024-03-24T10:38:09.863187+0000","last_active":"2024-03-24T10:40:05.299830+0000","last_peered":"2024-03-24T10:40:05.299830+0000","last_clean":"2024-03-24T10:40:05.299830+0000","last_became_active":"2024-03-24T10:38:09.862859+0000","last_became_peered":"2024-03-24T10:38:09.862859+0000","last_unstale":"2024-03-24T10:40:05.299830+0000","last_undegraded":"2024-03-24T10:40:05.299830+0000","last_fullsized":"2024-03-24T10:40:05.299830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:21:02.707127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413190+0000","last_change":"2024-03-24T10:38:09.867899+0000","last_active":"2024-03-24T10:40:05.413190+0000","last_peered":"2024-03-24T10:40:05.413190+0000","last_clean":"2024-03-24T10:40:05.413190+0000","last_became_active":"2024-03-24T10:38:09.867563+0000","last_became_peered":"2024-03-24T10:38:09.867563+0000","last_unstale":"2024-03-24T10:40:05.413190+0000","last_undegraded":"2024-03-24T10:40:05.413190+0000","last_fullsized":"2024-03-24T10:40:05.413190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:51.067981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575138+0000","last_change":"2024-03-24T10:38:10.870825+0000","last_active":"2024-03-24T10:40:05.575138+0000","last_peered":"2024-03-24T10:40:05.575138+0000","last_clean":"2024-03-24T10:40:05.575138+0000","last_became_active":"2024-03-24T10:38:10.870491+0000","last_became_peered":"2024-03-24T10:38:10.870491+0000","last_unstale":"2024-03-24T10:40:05.575138+0000","last_undegraded":"2024-03-24T10:40:05.575138+0000","last_fullsized":"2024-03-24T10:40:05.575138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:30:34.091299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'44","reported_seq":88,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.527925+0000","last_change":"2024-03-24T10:38:09.863724+0000","last_active":"2024-03-24T10:40:05.527925+0000","last_peered":"2024-03-24T10:40:05.527925+0000","last_clean":"2024-03-24T10:40:05.527925+0000","last_became_active":"2024-03-24T10:38:09.862900+0000","last_became_peered":"2024-03-24T10:38:09.862900+0000","last_unstale":"2024-03-24T10:40:05.527925+0000","last_undegraded":"2024-03-24T10:40:05.527925+0000","last_fullsized":"2024-03-24T10:40:05.527925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:21.646074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413886+0000","last_change":"2024-03-24T10:38:10.884661+0000","last_active":"2024-03-24T10:40:05.413886+0000","last_peered":"2024-03-24T10:40:05.413886+0000","last_clean":"2024-03-24T10:40:05.413886+0000","last_became_active":"2024-03-24T10:38:10.882901+0000","last_became_peered":"2024-03-24T10:38:10.882901+0000","last_unstale":"2024-03-24T10:40:05.413886+0000","last_undegraded":"2024-03-24T10:40:05.413886+0000","last_fullsized":"2024-03-24T10:40:05.413886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:23:46.366567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576002+0000","last_change":"2024-03-24T10:38:09.862122+0000","last_active":"2024-03-24T10:40:05.576002+0000","last_peered":"2024-03-24T10:40:05.576002+0000","last_clean":"2024-03-24T10:40:05.576002+0000","last_became_active":"2024-03-24T10:38:09.861765+0000","last_became_peered":"2024-03-24T10:38:09.861765+0000","last_unstale":"2024-03-24T10:40:05.576002+0000","last_undegraded":"2024-03-24T10:40:05.576002+0000","last_fullsized":"2024-03-24T10:40:05.576002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:40:15.148349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575995+0000","last_change":"2024-03-24T10:38:10.886797+0000","last_active":"2024-03-24T10:40:05.575995+0000","last_peered":"2024-03-24T10:40:05.575995+0000","last_clean":"2024-03-24T10:40:05.575995+0000","last_became_active":"2024-03-24T10:38:10.886406+0000","last_became_peered":"2024-03-24T10:38:10.886406+0000","last_unstale":"2024-03-24T10:40:05.575995+0000","last_undegraded":"2024-03-24T10:40:05.575995+0000","last_fullsized":"2024-03-24T10:40:05.575995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:35.763623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575166+0000","last_change":"2024-03-24T10:39:16.707034+0000","last_active":"2024-03-24T10:40:05.575166+0000","last_peered":"2024-03-24T10:40:05.575166+0000","last_clean":"2024-03-24T10:40:05.575166+0000","last_became_active":"2024-03-24T10:39:16.701995+0000","last_became_peered":"2024-03-24T10:39:16.701995+0000","last_unstale":"2024-03-24T10:40:05.575166+0000","last_undegraded":"2024-03-24T10:40:05.575166+0000","last_fullsized":"2024-03-24T10:40:05.575166+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:18:41.362740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412756+0000","last_change":"2024-03-24T10:38:10.873871+0000","last_active":"2024-03-24T10:40:05.412756+0000","last_peered":"2024-03-24T10:40:05.412756+0000","last_clean":"2024-03-24T10:40:05.412756+0000","last_became_active":"2024-03-24T10:38:10.873528+0000","last_became_peered":"2024-03-24T10:38:10.873528+0000","last_unstale":"2024-03-24T10:40:05.412756+0000","last_undegraded":"2024-03-24T10:40:05.412756+0000","last_fullsized":"2024-03-24T10:40:05.412756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:33.372195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413424+0000","last_change":"2024-03-24T10:38:09.861548+0000","last_active":"2024-03-24T10:40:05.413424+0000","last_peered":"2024-03-24T10:40:05.413424+0000","last_clean":"2024-03-24T10:40:05.413424+0000","last_became_active":"2024-03-24T10:38:09.861219+0000","last_became_peered":"2024-03-24T10:38:09.861219+0000","last_unstale":"2024-03-24T10:40:05.413424+0000","last_undegraded":"2024-03-24T10:40:05.413424+0000","last_fullsized":"2024-03-24T10:40:05.413424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:57:46.398179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574856+0000","last_change":"2024-03-24T10:38:10.880202+0000","last_active":"2024-03-24T10:40:05.574856+0000","last_peered":"2024-03-24T10:40:05.574856+0000","last_clean":"2024-03-24T10:40:05.574856+0000","last_became_active":"2024-03-24T10:38:10.879807+0000","last_became_peered":"2024-03-24T10:38:10.879807+0000","last_unstale":"2024-03-24T10:40:05.574856+0000","last_undegraded":"2024-03-24T10:40:05.574856+0000","last_fullsized":"2024-03-24T10:40:05.574856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:37.498728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575135+0000","last_change":"2024-03-24T10:39:16.704470+0000","last_active":"2024-03-24T10:40:05.575135+0000","last_peered":"2024-03-24T10:40:05.575135+0000","last_clean":"2024-03-24T10:40:05.575135+0000","last_became_active":"2024-03-24T10:39:16.701168+0000","last_became_peered":"2024-03-24T10:39:16.701168+0000","last_unstale":"2024-03-24T10:40:05.575135+0000","last_undegraded":"2024-03-24T10:40:05.575135+0000","last_fullsized":"2024-03-24T10:40:05.575135+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:20:34.781782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413602+0000","last_change":"2024-03-24T10:38:10.884371+0000","last_active":"2024-03-24T10:40:05.413602+0000","last_peered":"2024-03-24T10:40:05.413602+0000","last_clean":"2024-03-24T10:40:05.413602+0000","last_became_active":"2024-03-24T10:38:10.884022+0000","last_became_peered":"2024-03-24T10:38:10.884022+0000","last_unstale":"2024-03-24T10:40:05.413602+0000","last_undegraded":"2024-03-24T10:40:05.413602+0000","last_fullsized":"2024-03-24T10:40:05.413602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:47:48.778727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412949+0000","last_change":"2024-03-24T10:38:09.863231+0000","last_active":"2024-03-24T10:40:05.412949+0000","last_peered":"2024-03-24T10:40:05.412949+0000","last_clean":"2024-03-24T10:40:05.412949+0000","last_became_active":"2024-03-24T10:38:09.862848+0000","last_became_peered":"2024-03-24T10:38:09.862848+0000","last_unstale":"2024-03-24T10:40:05.412949+0000","last_undegraded":"2024-03-24T10:40:05.412949+0000","last_fullsized":"2024-03-24T10:40:05.412949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:36:57.532746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300075+0000","last_change":"2024-03-24T10:38:10.875187+0000","last_active":"2024-03-24T10:40:05.300075+0000","last_peered":"2024-03-24T10:40:05.300075+0000","last_clean":"2024-03-24T10:40:05.300075+0000","last_became_active":"2024-03-24T10:38:10.874682+0000","last_became_peered":"2024-03-24T10:38:10.874682+0000","last_unstale":"2024-03-24T10:40:05.300075+0000","last_undegraded":"2024-03-24T10:40:05.300075+0000","last_fullsized":"2024-03-24T10:40:05.300075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:53:56.915312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575456+0000","last_change":"2024-03-24T10:38:09.863430+0000","last_active":"2024-03-24T10:40:05.575456+0000","last_peered":"2024-03-24T10:40:05.575456+0000","last_clean":"2024-03-24T10:40:05.575456+0000","last_became_active":"2024-03-24T10:38:09.862877+0000","last_became_peered":"2024-03-24T10:38:09.862877+0000","last_unstale":"2024-03-24T10:40:05.575456+0000","last_undegraded":"2024-03-24T10:40:05.575456+0000","last_fullsized":"2024-03-24T10:40:05.575456+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:33:29.039896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300427+0000","last_change":"2024-03-24T10:38:10.878446+0000","last_active":"2024-03-24T10:40:05.300427+0000","last_peered":"2024-03-24T10:40:05.300427+0000","last_clean":"2024-03-24T10:40:05.300427+0000","last_became_active":"2024-03-24T10:38:10.877944+0000","last_became_peered":"2024-03-24T10:38:10.877944+0000","last_unstale":"2024-03-24T10:40:05.300427+0000","last_undegraded":"2024-03-24T10:40:05.300427+0000","last_fullsized":"2024-03-24T10:40:05.300427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:00:22.568669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.525634+0000","last_change":"2024-03-24T10:38:09.867328+0000","last_active":"2024-03-24T10:40:05.525634+0000","last_peered":"2024-03-24T10:40:05.525634+0000","last_clean":"2024-03-24T10:40:05.525634+0000","last_became_active":"2024-03-24T10:38:09.866869+0000","last_became_peered":"2024-03-24T10:38:09.866869+0000","last_unstale":"2024-03-24T10:40:05.525634+0000","last_undegraded":"2024-03-24T10:40:05.525634+0000","last_fullsized":"2024-03-24T10:40:05.525634+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:52:32.136151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300060+0000","last_change":"2024-03-24T10:39:16.705912+0000","last_active":"2024-03-24T10:40:05.300060+0000","last_peered":"2024-03-24T10:40:05.300060+0000","last_clean":"2024-03-24T10:40:05.300060+0000","last_became_active":"2024-03-24T10:39:16.699809+0000","last_became_peered":"2024-03-24T10:39:16.699809+0000","last_unstale":"2024-03-24T10:40:05.300060+0000","last_undegraded":"2024-03-24T10:40:05.300060+0000","last_fullsized":"2024-03-24T10:40:05.300060+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:31.858976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575611+0000","last_change":"2024-03-24T10:38:09.865399+0000","last_active":"2024-03-24T10:40:05.575611+0000","last_peered":"2024-03-24T10:40:05.575611+0000","last_clean":"2024-03-24T10:40:05.575611+0000","last_became_active":"2024-03-24T10:38:09.864906+0000","last_became_peered":"2024-03-24T10:38:09.864906+0000","last_unstale":"2024-03-24T10:40:05.575611+0000","last_undegraded":"2024-03-24T10:40:05.575611+0000","last_fullsized":"2024-03-24T10:40:05.575611+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:53:12.363018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300349+0000","last_change":"2024-03-24T10:38:10.876881+0000","last_active":"2024-03-24T10:40:05.300349+0000","last_peered":"2024-03-24T10:40:05.300349+0000","last_clean":"2024-03-24T10:40:05.300349+0000","last_became_active":"2024-03-24T10:38:10.876479+0000","last_became_peered":"2024-03-24T10:38:10.876479+0000","last_unstale":"2024-03-24T10:40:05.300349+0000","last_undegraded":"2024-03-24T10:40:05.300349+0000","last_fullsized":"2024-03-24T10:40:05.300349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:57:33.140451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574900+0000","last_change":"2024-03-24T10:38:10.872622+0000","last_active":"2024-03-24T10:40:05.574900+0000","last_peered":"2024-03-24T10:40:05.574900+0000","last_clean":"2024-03-24T10:40:05.574900+0000","last_became_active":"2024-03-24T10:38:10.872293+0000","last_became_peered":"2024-03-24T10:38:10.872293+0000","last_unstale":"2024-03-24T10:40:05.574900+0000","last_undegraded":"2024-03-24T10:40:05.574900+0000","last_fullsized":"2024-03-24T10:40:05.574900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:48.139589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575020+0000","last_change":"2024-03-24T10:39:16.701548+0000","last_active":"2024-03-24T10:40:05.575020+0000","last_peered":"2024-03-24T10:40:05.575020+0000","last_clean":"2024-03-24T10:40:05.575020+0000","last_became_active":"2024-03-24T10:39:16.701213+0000","last_became_peered":"2024-03-24T10:39:16.701213+0000","last_unstale":"2024-03-24T10:40:05.575020+0000","last_undegraded":"2024-03-24T10:40:05.575020+0000","last_fullsized":"2024-03-24T10:40:05.575020+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:50:28.131754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574960+0000","last_change":"2024-03-24T10:38:10.879679+0000","last_active":"2024-03-24T10:40:05.574960+0000","last_peered":"2024-03-24T10:40:05.574960+0000","last_clean":"2024-03-24T10:40:05.574960+0000","last_became_active":"2024-03-24T10:38:10.879334+0000","last_became_peered":"2024-03-24T10:38:10.879334+0000","last_unstale":"2024-03-24T10:40:05.574960+0000","last_undegraded":"2024-03-24T10:40:05.574960+0000","last_fullsized":"2024-03-24T10:40:05.574960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:27:53.517368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576067+0000","last_change":"2024-03-24T10:39:16.704444+0000","last_active":"2024-03-24T10:40:05.576067+0000","last_peered":"2024-03-24T10:40:05.576067+0000","last_clean":"2024-03-24T10:40:05.576067+0000","last_became_active":"2024-03-24T10:39:16.704085+0000","last_became_peered":"2024-03-24T10:39:16.704085+0000","last_unstale":"2024-03-24T10:40:05.576067+0000","last_undegraded":"2024-03-24T10:40:05.576067+0000","last_fullsized":"2024-03-24T10:40:05.576067+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:04:49.643195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575450+0000","last_change":"2024-03-24T10:38:10.887108+0000","last_active":"2024-03-24T10:40:05.575450+0000","last_peered":"2024-03-24T10:40:05.575450+0000","last_clean":"2024-03-24T10:40:05.575450+0000","last_became_active":"2024-03-24T10:38:10.886770+0000","last_became_peered":"2024-03-24T10:38:10.886770+0000","last_unstale":"2024-03-24T10:40:05.575450+0000","last_undegraded":"2024-03-24T10:40:05.575450+0000","last_fullsized":"2024-03-24T10:40:05.575450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:56.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575209+0000","last_change":"2024-03-24T10:38:10.873369+0000","last_active":"2024-03-24T10:40:05.575209+0000","last_peered":"2024-03-24T10:40:05.575209+0000","last_clean":"2024-03-24T10:40:05.575209+0000","last_became_active":"2024-03-24T10:38:10.873046+0000","last_became_peered":"2024-03-24T10:38:10.873046+0000","last_unstale":"2024-03-24T10:40:05.575209+0000","last_undegraded":"2024-03-24T10:40:05.575209+0000","last_fullsized":"2024-03-24T10:40:05.575209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:46.071084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575565+0000","last_change":"2024-03-24T10:38:10.878666+0000","last_active":"2024-03-24T10:40:05.575565+0000","last_peered":"2024-03-24T10:40:05.575565+0000","last_clean":"2024-03-24T10:40:05.575565+0000","last_became_active":"2024-03-24T10:38:10.877250+0000","last_became_peered":"2024-03-24T10:38:10.877250+0000","last_unstale":"2024-03-24T10:40:05.575565+0000","last_undegraded":"2024-03-24T10:40:05.575565+0000","last_fullsized":"2024-03-24T10:40:05.575565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:13:36.793584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413854+0000","last_change":"2024-03-24T10:38:10.889274+0000","last_active":"2024-03-24T10:40:05.413854+0000","last_peered":"2024-03-24T10:40:05.413854+0000","last_clean":"2024-03-24T10:40:05.413854+0000","last_became_active":"2024-03-24T10:38:10.888899+0000","last_became_peered":"2024-03-24T10:38:10.888899+0000","last_unstale":"2024-03-24T10:40:05.413854+0000","last_undegraded":"2024-03-24T10:40:05.413854+0000","last_fullsized":"2024-03-24T10:40:05.413854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:11.225562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299977+0000","last_change":"2024-03-24T10:38:10.887135+0000","last_active":"2024-03-24T10:40:05.299977+0000","last_peered":"2024-03-24T10:40:05.299977+0000","last_clean":"2024-03-24T10:40:05.299977+0000","last_became_active":"2024-03-24T10:38:10.886812+0000","last_became_peered":"2024-03-24T10:38:10.886812+0000","last_unstale":"2024-03-24T10:40:05.299977+0000","last_undegraded":"2024-03-24T10:40:05.299977+0000","last_fullsized":"2024-03-24T10:40:05.299977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:51:53.959914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413259+0000","last_change":"2024-03-24T10:38:10.885417+0000","last_active":"2024-03-24T10:40:05.413259+0000","last_peered":"2024-03-24T10:40:05.413259+0000","last_clean":"2024-03-24T10:40:05.413259+0000","last_became_active":"2024-03-24T10:38:10.883209+0000","last_became_peered":"2024-03-24T10:38:10.883209+0000","last_unstale":"2024-03-24T10:40:05.413259+0000","last_undegraded":"2024-03-24T10:40:05.413259+0000","last_fullsized":"2024-03-24T10:40:05.413259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:52:52.535611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575193+0000","last_change":"2024-03-24T10:38:10.885005+0000","last_active":"2024-03-24T10:40:05.575193+0000","last_peered":"2024-03-24T10:40:05.575193+0000","last_clean":"2024-03-24T10:40:05.575193+0000","last_became_active":"2024-03-24T10:38:10.884680+0000","last_became_peered":"2024-03-24T10:38:10.884680+0000","last_unstale":"2024-03-24T10:40:05.575193+0000","last_undegraded":"2024-03-24T10:40:05.575193+0000","last_fullsized":"2024-03-24T10:40:05.575193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:56.377777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575580+0000","last_change":"2024-03-24T10:38:10.875153+0000","last_active":"2024-03-24T10:40:05.575580+0000","last_peered":"2024-03-24T10:40:05.575580+0000","last_clean":"2024-03-24T10:40:05.575580+0000","last_became_active":"2024-03-24T10:38:10.874721+0000","last_became_peered":"2024-03-24T10:38:10.874721+0000","last_unstale":"2024-03-24T10:40:05.575580+0000","last_undegraded":"2024-03-24T10:40:05.575580+0000","last_fullsized":"2024-03-24T10:40:05.575580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:52.522797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575025+0000","last_change":"2024-03-24T10:38:10.887298+0000","last_active":"2024-03-24T10:40:05.575025+0000","last_peered":"2024-03-24T10:40:05.575025+0000","last_clean":"2024-03-24T10:40:05.575025+0000","last_became_active":"2024-03-24T10:38:10.886975+0000","last_became_peered":"2024-03-24T10:38:10.886975+0000","last_unstale":"2024-03-24T10:40:05.575025+0000","last_undegraded":"2024-03-24T10:40:05.575025+0000","last_fullsized":"2024-03-24T10:40:05.575025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:09.740303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595166,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":791,"num_read_kb":503,"num_write":285,"num_write_kb":1433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2015232,"data_stored":1785498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383716,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":465380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645353,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32936,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712216,"statfs":{"total":95995035648,"available":95961309184,"internally_reserved":0,"allocated":1802240,"data_stored":1253329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808878,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1380352,"data_stored":858949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32520,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712632,"statfs":{"total":95995035648,"available":95961735168,"internally_reserved":0,"allocated":1372160,"data_stored":858170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1576960,"data_stored":1058697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":465005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":530498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":466276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":532244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28492,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716660,"statfs":{"total":95995035648,"available":95965859840,"internally_reserved":0,"allocated":1511424,"data_stored":990015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":983040,"data_stored":466086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28700,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716452,"statfs":{"total":95995035648,"available":95965646848,"internally_reserved":0,"allocated":1724416,"data_stored":1189449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":65700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":67446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":393372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":198828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:40:10.077 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-24T10:40:10.078 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:40:10.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.217+0000 7f5300b5c640 1 -- 172.21.15.23:0/3741566021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc100f90 msgr2=0x7f52fc10d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.217+0000 7f5300b5c640 1 --2- 172.21.15.23:0/3741566021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc100f90 0x7f52fc10d450 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f52f4009ca0 tx=0x7f52f402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.217+0000 7f5300b5c640 1 -- 172.21.15.23:0/3741566021 shutdown_connections 2024-03-24T10:40:10.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.217+0000 7f5300b5c640 1 --2- 172.21.15.23:0/3741566021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc100f90 0x7f52fc10d450 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.217+0000 7f5300b5c640 1 --2- 172.21.15.23:0/3741566021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52fc1005d0 0x7f52fc100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.217+0000 7f5300b5c640 1 --2- 172.21.15.23:0/3741566021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc106690 0x7f52fc106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.217+0000 7f5300b5c640 1 -- 172.21.15.23:0/3741566021 >> 172.21.15.23:0/3741566021 conn(0x7f52fc076990 msgr2=0x7f52fc076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:10.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.217+0000 7f5300b5c640 1 -- 172.21.15.23:0/3741566021 shutdown_connections 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.217+0000 7f5300b5c640 1 -- 172.21.15.23:0/3741566021 wait complete. 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f5300b5c640 1 Processor -- start 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f5300b5c640 1 -- start start 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f5300b5c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52fc1005d0 0x7f52fc19d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f5300b5c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc100f90 0x7f52fc19ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f5300b5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc106690 0x7f52fc197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f5300b5c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc1980d0 con 0x7f52fc106690 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f5300b5c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc198240 con 0x7f52fc100f90 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f5300b5c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc198400 con 0x7f52fc1005d0 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f52fad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc106690 0x7f52fc197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f52fad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc106690 0x7f52fc197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33490/0 (socket says 172.21.15.23:33490) 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f52fa59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52fc1005d0 0x7f52fc19d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f52fad9d640 1 -- 172.21.15.23:0/1170115511 learned_addr learned my addr 172.21.15.23:0/1170115511 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f52f9d9b640 1 --2- 172.21.15.23:0/1170115511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc100f90 0x7f52fc19ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f52fad9d640 1 -- 172.21.15.23:0/1170115511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52fc1005d0 msgr2=0x7f52fc19d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f52fad9d640 1 --2- 172.21.15.23:0/1170115511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52fc1005d0 0x7f52fc19d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f52fad9d640 1 -- 172.21.15.23:0/1170115511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc100f90 msgr2=0x7f52fc19ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f52fad9d640 1 --2- 172.21.15.23:0/1170115511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc100f90 0x7f52fc19ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f52fad9d640 1 -- 172.21.15.23:0/1170115511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f4009900 con 0x7f52fc106690 2024-03-24T10:40:10.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f52fa59c640 1 --2- 172.21.15.23:0/1170115511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52fc1005d0 0x7f52fc19d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:10.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.218+0000 7f52fad9d640 1 --2- 172.21.15.23:0/1170115511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc106690 0x7f52fc197a80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f52f4005e30 tx=0x7f52f4035aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:10.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.219+0000 7f52eb7fe640 1 -- 172.21.15.23:0/1170115511 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f4035ea0 con 0x7f52fc106690 2024-03-24T10:40:10.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.219+0000 7f52eb7fe640 1 -- 172.21.15.23:0/1170115511 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52f4003e50 con 0x7f52fc106690 2024-03-24T10:40:10.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.219+0000 7f5300b5c640 1 -- 172.21.15.23:0/1170115511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52fc198680 con 0x7f52fc106690 2024-03-24T10:40:10.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.219+0000 7f52eb7fe640 1 -- 172.21.15.23:0/1170115511 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f4004830 con 0x7f52fc106690 2024-03-24T10:40:10.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.219+0000 7f5300b5c640 1 -- 172.21.15.23:0/1170115511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52fc1a4920 con 0x7f52fc106690 2024-03-24T10:40:10.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.220+0000 7f5300b5c640 1 -- 172.21.15.23:0/1170115511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52fc101cf0 con 0x7f52fc106690 2024-03-24T10:40:10.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.221+0000 7f52eb7fe640 1 -- 172.21.15.23:0/1170115511 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7f52f403b040 con 0x7f52fc106690 2024-03-24T10:40:10.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.221+0000 7f52eb7fe640 1 --2- 172.21.15.23:0/1170115511 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f52e40a2350 0x7f52e40a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:10.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.221+0000 7f52fa59c640 1 --2- 172.21.15.23:0/1170115511 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f52e40a2350 0x7f52e40a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:10.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.221+0000 7f52eb7fe640 1 -- 172.21.15.23:0/1170115511 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f52f40f0b50 con 0x7f52fc106690 2024-03-24T10:40:10.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.222+0000 7f52fa59c640 1 --2- 172.21.15.23:0/1170115511 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f52e40a2350 0x7f52e40a4810 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f52fc105730 tx=0x7f52ec005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:10.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.226+0000 7f52eb7fe640 1 -- 172.21.15.23:0/1170115511 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f52f40b96a0 con 0x7f52fc106690 2024-03-24T10:40:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi117 to 174.1M 2024-03-24T10:40:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: Unable to set osd_memory_target on smithi117 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T10:40:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-24T10:40:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: Unable to set osd_memory_target on smithi023 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-03-24T10:40:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi116 to 174.1M 2024-03-24T10:40:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: Unable to set osd_memory_target on smithi116 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-24T10:40:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-24T10:40:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T10:40:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T10:40:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:40:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: Updating smithi023:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:40:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:40:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2823891709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:40:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4158847352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:40:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:09 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi117 to 174.1M 2024-03-24T10:40:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: Unable to set osd_memory_target on smithi117 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T10:40:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-24T10:40:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: Unable to set osd_memory_target on smithi023 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-03-24T10:40:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi116 to 174.1M 2024-03-24T10:40:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: Unable to set osd_memory_target on smithi116 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-24T10:40:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-24T10:40:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T10:40:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T10:40:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:40:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: Updating smithi023:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:40:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T10:40:10.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2823891709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:40:10.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4158847352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:40:10.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:09 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:10.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.404+0000 7f5300b5c640 1 -- 172.21.15.23:0/1170115511 --> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f52fc108a50 con 0x7f52e40a2350 2024-03-24T10:40:10.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.412+0000 7f52eb7fe640 1 -- 172.21.15.23:0/1170115511 <== mgr.15288 v2:172.21.15.23:6800/581236369 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257744 (secure 0 0 0) 0x7f52fc108a50 con 0x7f52e40a2350 2024-03-24T10:40:10.415 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:40:10.416 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-24T10:40:10.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.415+0000 7f5300b5c640 1 -- 172.21.15.23:0/1170115511 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f52e40a2350 msgr2=0x7f52e40a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.415+0000 7f5300b5c640 1 --2- 172.21.15.23:0/1170115511 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f52e40a2350 0x7f52e40a4810 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f52fc105730 tx=0x7f52ec005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:10.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.415+0000 7f5300b5c640 1 -- 172.21.15.23:0/1170115511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc106690 msgr2=0x7f52fc197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.415+0000 7f5300b5c640 1 --2- 172.21.15.23:0/1170115511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc106690 0x7f52fc197a80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f52f4005e30 tx=0x7f52f4035aa0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.415+0000 7f5300b5c640 1 -- 172.21.15.23:0/1170115511 shutdown_connections 2024-03-24T10:40:10.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.415+0000 7f5300b5c640 1 --2- 172.21.15.23:0/1170115511 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f52e40a2350 0x7f52e40a4810 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.415+0000 7f5300b5c640 1 --2- 172.21.15.23:0/1170115511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc106690 0x7f52fc197a80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.415+0000 7f5300b5c640 1 --2- 172.21.15.23:0/1170115511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc100f90 0x7f52fc19ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.415+0000 7f5300b5c640 1 --2- 172.21.15.23:0/1170115511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52fc1005d0 0x7f52fc19d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.415+0000 7f5300b5c640 1 -- 172.21.15.23:0/1170115511 >> 172.21.15.23:0/1170115511 conn(0x7f52fc076990 msgr2=0x7f52fc0fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:10.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.415+0000 7f5300b5c640 1 -- 172.21.15.23:0/1170115511 shutdown_connections 2024-03-24T10:40:10.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.415+0000 7f5300b5c640 1 -- 172.21.15.23:0/1170115511 wait complete. 2024-03-24T10:40:10.426 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-03-24T10:40:09.312178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185768,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":927,"num_read_kb":622,"num_write":468,"num_write_kb":4373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1181547,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":863,"ondisk_log_size":863,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368356,"kb_used_data":15012,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124573468,"statfs":{"total":1151940427776,"available":1151563231232,"internally_reserved":0,"allocated":15372288,"data_stored":9134098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":361674043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.992192"},"pg_stats":[{"pgid":"3.3f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575582+0000","last_change":"2024-03-24T10:38:10.882385+0000","last_active":"2024-03-24T10:40:05.575582+0000","last_peered":"2024-03-24T10:40:05.575582+0000","last_clean":"2024-03-24T10:40:05.575582+0000","last_became_active":"2024-03-24T10:38:10.882062+0000","last_became_peered":"2024-03-24T10:38:10.882062+0000","last_unstale":"2024-03-24T10:40:05.575582+0000","last_undegraded":"2024-03-24T10:40:05.575582+0000","last_fullsized":"2024-03-24T10:40:05.575582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:25.151062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575927+0000","last_change":"2024-03-24T10:38:10.878001+0000","last_active":"2024-03-24T10:40:05.575927+0000","last_peered":"2024-03-24T10:40:05.575927+0000","last_clean":"2024-03-24T10:40:05.575927+0000","last_became_active":"2024-03-24T10:38:10.877661+0000","last_became_peered":"2024-03-24T10:38:10.877661+0000","last_unstale":"2024-03-24T10:40:05.575927+0000","last_undegraded":"2024-03-24T10:40:05.575927+0000","last_fullsized":"2024-03-24T10:40:05.575927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:40:19.062969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576049+0000","last_change":"2024-03-24T10:38:10.891809+0000","last_active":"2024-03-24T10:40:05.576049+0000","last_peered":"2024-03-24T10:40:05.576049+0000","last_clean":"2024-03-24T10:40:05.576049+0000","last_became_active":"2024-03-24T10:38:10.890354+0000","last_became_peered":"2024-03-24T10:38:10.890354+0000","last_unstale":"2024-03-24T10:40:05.576049+0000","last_undegraded":"2024-03-24T10:40:05.576049+0000","last_fullsized":"2024-03-24T10:40:05.576049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:59:26.037914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575921+0000","last_change":"2024-03-24T10:38:10.885097+0000","last_active":"2024-03-24T10:40:05.575921+0000","last_peered":"2024-03-24T10:40:05.575921+0000","last_clean":"2024-03-24T10:40:05.575921+0000","last_became_active":"2024-03-24T10:38:10.884842+0000","last_became_peered":"2024-03-24T10:38:10.884842+0000","last_unstale":"2024-03-24T10:40:05.575921+0000","last_undegraded":"2024-03-24T10:40:05.575921+0000","last_fullsized":"2024-03-24T10:40:05.575921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:39:44.085699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412906+0000","last_change":"2024-03-24T10:38:10.877653+0000","last_active":"2024-03-24T10:40:05.412906+0000","last_peered":"2024-03-24T10:40:05.412906+0000","last_clean":"2024-03-24T10:40:05.412906+0000","last_became_active":"2024-03-24T10:38:10.877249+0000","last_became_peered":"2024-03-24T10:38:10.877249+0000","last_unstale":"2024-03-24T10:40:05.412906+0000","last_undegraded":"2024-03-24T10:40:05.412906+0000","last_fullsized":"2024-03-24T10:40:05.412906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:17.504660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422853+0000","last_change":"2024-03-24T10:38:10.876584+0000","last_active":"2024-03-24T10:40:05.422853+0000","last_peered":"2024-03-24T10:40:05.422853+0000","last_clean":"2024-03-24T10:40:05.422853+0000","last_became_active":"2024-03-24T10:38:10.876173+0000","last_became_peered":"2024-03-24T10:38:10.876173+0000","last_unstale":"2024-03-24T10:40:05.422853+0000","last_undegraded":"2024-03-24T10:40:05.422853+0000","last_fullsized":"2024-03-24T10:40:05.422853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:33.899155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299791+0000","last_change":"2024-03-24T10:38:10.874206+0000","last_active":"2024-03-24T10:40:05.299791+0000","last_peered":"2024-03-24T10:40:05.299791+0000","last_clean":"2024-03-24T10:40:05.299791+0000","last_became_active":"2024-03-24T10:38:10.873779+0000","last_became_peered":"2024-03-24T10:38:10.873779+0000","last_unstale":"2024-03-24T10:40:05.299791+0000","last_undegraded":"2024-03-24T10:40:05.299791+0000","last_fullsized":"2024-03-24T10:40:05.299791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:08:19.928998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413254+0000","last_change":"2024-03-24T10:38:10.884219+0000","last_active":"2024-03-24T10:40:05.413254+0000","last_peered":"2024-03-24T10:40:05.413254+0000","last_clean":"2024-03-24T10:40:05.413254+0000","last_became_active":"2024-03-24T10:38:10.883863+0000","last_became_peered":"2024-03-24T10:38:10.883863+0000","last_unstale":"2024-03-24T10:40:05.413254+0000","last_undegraded":"2024-03-24T10:40:05.413254+0000","last_fullsized":"2024-03-24T10:40:05.413254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:26:34.932083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575665+0000","last_change":"2024-03-24T10:38:10.881297+0000","last_active":"2024-03-24T10:40:05.575665+0000","last_peered":"2024-03-24T10:40:05.575665+0000","last_clean":"2024-03-24T10:40:05.575665+0000","last_became_active":"2024-03-24T10:38:10.879871+0000","last_became_peered":"2024-03-24T10:38:10.879871+0000","last_unstale":"2024-03-24T10:40:05.575665+0000","last_undegraded":"2024-03-24T10:40:05.575665+0000","last_fullsized":"2024-03-24T10:40:05.575665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:05.132788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"87'16","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422954+0000","last_change":"2024-03-24T10:38:10.876648+0000","last_active":"2024-03-24T10:40:05.422954+0000","last_peered":"2024-03-24T10:40:05.422954+0000","last_clean":"2024-03-24T10:40:05.422954+0000","last_became_active":"2024-03-24T10:38:10.876113+0000","last_became_peered":"2024-03-24T10:38:10.876113+0000","last_unstale":"2024-03-24T10:40:05.422954+0000","last_undegraded":"2024-03-24T10:40:05.422954+0000","last_fullsized":"2024-03-24T10:40:05.422954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:29:18.282797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300282+0000","last_change":"2024-03-24T10:38:10.875836+0000","last_active":"2024-03-24T10:40:05.300282+0000","last_peered":"2024-03-24T10:40:05.300282+0000","last_clean":"2024-03-24T10:40:05.300282+0000","last_became_active":"2024-03-24T10:38:10.875522+0000","last_became_peered":"2024-03-24T10:38:10.875522+0000","last_unstale":"2024-03-24T10:40:05.300282+0000","last_undegraded":"2024-03-24T10:40:05.300282+0000","last_fullsized":"2024-03-24T10:40:05.300282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:41.282123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575678+0000","last_change":"2024-03-24T10:38:10.884567+0000","last_active":"2024-03-24T10:40:05.575678+0000","last_peered":"2024-03-24T10:40:05.575678+0000","last_clean":"2024-03-24T10:40:05.575678+0000","last_became_active":"2024-03-24T10:38:10.884195+0000","last_became_peered":"2024-03-24T10:38:10.884195+0000","last_unstale":"2024-03-24T10:40:05.575678+0000","last_undegraded":"2024-03-24T10:40:05.575678+0000","last_fullsized":"2024-03-24T10:40:05.575678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:29.286631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"87'10","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575744+0000","last_change":"2024-03-24T10:39:16.726561+0000","last_active":"2024-03-24T10:40:05.575744+0000","last_peered":"2024-03-24T10:40:05.575744+0000","last_clean":"2024-03-24T10:40:05.575744+0000","last_became_active":"2024-03-24T10:39:16.726088+0000","last_became_peered":"2024-03-24T10:39:16.726088+0000","last_unstale":"2024-03-24T10:40:05.575744+0000","last_undegraded":"2024-03-24T10:40:05.575744+0000","last_fullsized":"2024-03-24T10:40:05.575744+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:08:46.638610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575643+0000","last_change":"2024-03-24T10:38:10.879942+0000","last_active":"2024-03-24T10:40:05.575643+0000","last_peered":"2024-03-24T10:40:05.575643+0000","last_clean":"2024-03-24T10:40:05.575643+0000","last_became_active":"2024-03-24T10:38:10.879396+0000","last_became_peered":"2024-03-24T10:38:10.879396+0000","last_unstale":"2024-03-24T10:40:05.575643+0000","last_undegraded":"2024-03-24T10:40:05.575643+0000","last_fullsized":"2024-03-24T10:40:05.575643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:41.477988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575389+0000","last_change":"2024-03-24T10:38:10.886667+0000","last_active":"2024-03-24T10:40:05.575389+0000","last_peered":"2024-03-24T10:40:05.575389+0000","last_clean":"2024-03-24T10:40:05.575389+0000","last_became_active":"2024-03-24T10:38:10.886293+0000","last_became_peered":"2024-03-24T10:38:10.886293+0000","last_unstale":"2024-03-24T10:40:05.575389+0000","last_undegraded":"2024-03-24T10:40:05.575389+0000","last_fullsized":"2024-03-24T10:40:05.575389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:43:43.414573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574995+0000","last_change":"2024-03-24T10:38:10.883671+0000","last_active":"2024-03-24T10:40:05.574995+0000","last_peered":"2024-03-24T10:40:05.574995+0000","last_clean":"2024-03-24T10:40:05.574995+0000","last_became_active":"2024-03-24T10:38:10.883013+0000","last_became_peered":"2024-03-24T10:38:10.883013+0000","last_unstale":"2024-03-24T10:40:05.574995+0000","last_undegraded":"2024-03-24T10:40:05.574995+0000","last_fullsized":"2024-03-24T10:40:05.574995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:33.936543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575587+0000","last_change":"2024-03-24T10:38:10.887286+0000","last_active":"2024-03-24T10:40:05.575587+0000","last_peered":"2024-03-24T10:40:05.575587+0000","last_clean":"2024-03-24T10:40:05.575587+0000","last_became_active":"2024-03-24T10:38:10.886918+0000","last_became_peered":"2024-03-24T10:38:10.886918+0000","last_unstale":"2024-03-24T10:40:05.575587+0000","last_undegraded":"2024-03-24T10:40:05.575587+0000","last_fullsized":"2024-03-24T10:40:05.575587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:29:40.266589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.298339+0000","last_change":"2024-03-24T10:38:10.874476+0000","last_active":"2024-03-24T10:40:05.298339+0000","last_peered":"2024-03-24T10:40:05.298339+0000","last_clean":"2024-03-24T10:40:05.298339+0000","last_became_active":"2024-03-24T10:38:10.874123+0000","last_became_peered":"2024-03-24T10:38:10.874123+0000","last_unstale":"2024-03-24T10:40:05.298339+0000","last_undegraded":"2024-03-24T10:40:05.298339+0000","last_fullsized":"2024-03-24T10:40:05.298339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:29.759320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575455+0000","last_change":"2024-03-24T10:38:10.878758+0000","last_active":"2024-03-24T10:40:05.575455+0000","last_peered":"2024-03-24T10:40:05.575455+0000","last_clean":"2024-03-24T10:40:05.575455+0000","last_became_active":"2024-03-24T10:38:10.878424+0000","last_became_peered":"2024-03-24T10:38:10.878424+0000","last_unstale":"2024-03-24T10:40:05.575455+0000","last_undegraded":"2024-03-24T10:40:05.575455+0000","last_fullsized":"2024-03-24T10:40:05.575455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:15:19.273108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.298282+0000","last_change":"2024-03-24T10:39:16.709304+0000","last_active":"2024-03-24T10:40:05.298282+0000","last_peered":"2024-03-24T10:40:05.298282+0000","last_clean":"2024-03-24T10:40:05.298282+0000","last_became_active":"2024-03-24T10:39:16.700325+0000","last_became_peered":"2024-03-24T10:39:16.700325+0000","last_unstale":"2024-03-24T10:40:05.298282+0000","last_undegraded":"2024-03-24T10:40:05.298282+0000","last_fullsized":"2024-03-24T10:40:05.298282+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:43:57.633288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.414021+0000","last_change":"2024-03-24T10:38:10.885198+0000","last_active":"2024-03-24T10:40:05.414021+0000","last_peered":"2024-03-24T10:40:05.414021+0000","last_clean":"2024-03-24T10:40:05.414021+0000","last_became_active":"2024-03-24T10:38:10.884123+0000","last_became_peered":"2024-03-24T10:38:10.884123+0000","last_unstale":"2024-03-24T10:40:05.414021+0000","last_undegraded":"2024-03-24T10:40:05.414021+0000","last_fullsized":"2024-03-24T10:40:05.414021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:37:12.510671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.414153+0000","last_change":"2024-03-24T10:38:09.866223+0000","last_active":"2024-03-24T10:40:05.414153+0000","last_peered":"2024-03-24T10:40:05.414153+0000","last_clean":"2024-03-24T10:40:05.414153+0000","last_became_active":"2024-03-24T10:38:09.865711+0000","last_became_peered":"2024-03-24T10:38:09.865711+0000","last_unstale":"2024-03-24T10:40:05.414153+0000","last_undegraded":"2024-03-24T10:40:05.414153+0000","last_fullsized":"2024-03-24T10:40:05.414153+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:26:08.380749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576281+0000","last_change":"2024-03-24T10:38:10.890557+0000","last_active":"2024-03-24T10:40:05.576281+0000","last_peered":"2024-03-24T10:40:05.576281+0000","last_clean":"2024-03-24T10:40:05.576281+0000","last_became_active":"2024-03-24T10:38:10.889883+0000","last_became_peered":"2024-03-24T10:38:10.889883+0000","last_unstale":"2024-03-24T10:40:05.576281+0000","last_undegraded":"2024-03-24T10:40:05.576281+0000","last_fullsized":"2024-03-24T10:40:05.576281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:34.731585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413316+0000","last_change":"2024-03-24T10:38:09.870253+0000","last_active":"2024-03-24T10:40:05.413316+0000","last_peered":"2024-03-24T10:40:05.413316+0000","last_clean":"2024-03-24T10:40:05.413316+0000","last_became_active":"2024-03-24T10:38:09.869932+0000","last_became_peered":"2024-03-24T10:38:09.869932+0000","last_unstale":"2024-03-24T10:40:05.413316+0000","last_undegraded":"2024-03-24T10:40:05.413316+0000","last_fullsized":"2024-03-24T10:40:05.413316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:38:43.626390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575343+0000","last_change":"2024-03-24T10:38:10.881705+0000","last_active":"2024-03-24T10:40:05.575343+0000","last_peered":"2024-03-24T10:40:05.575343+0000","last_clean":"2024-03-24T10:40:05.575343+0000","last_became_active":"2024-03-24T10:38:10.881323+0000","last_became_peered":"2024-03-24T10:38:10.881323+0000","last_unstale":"2024-03-24T10:40:05.575343+0000","last_undegraded":"2024-03-24T10:40:05.575343+0000","last_fullsized":"2024-03-24T10:40:05.575343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:25.861184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413086+0000","last_change":"2024-03-24T10:38:09.870481+0000","last_active":"2024-03-24T10:40:05.413086+0000","last_peered":"2024-03-24T10:40:05.413086+0000","last_clean":"2024-03-24T10:40:05.413086+0000","last_became_active":"2024-03-24T10:38:09.870117+0000","last_became_peered":"2024-03-24T10:38:09.870117+0000","last_unstale":"2024-03-24T10:40:05.413086+0000","last_undegraded":"2024-03-24T10:40:05.413086+0000","last_fullsized":"2024-03-24T10:40:05.413086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:37:51.871068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"87'15","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.297975+0000","last_change":"2024-03-24T10:39:16.906560+0000","last_active":"2024-03-24T10:40:05.297975+0000","last_peered":"2024-03-24T10:40:05.297975+0000","last_clean":"2024-03-24T10:40:05.297975+0000","last_became_active":"2024-03-24T10:39:16.906009+0000","last_became_peered":"2024-03-24T10:39:16.906009+0000","last_unstale":"2024-03-24T10:40:05.297975+0000","last_undegraded":"2024-03-24T10:40:05.297975+0000","last_fullsized":"2024-03-24T10:40:05.297975+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:45:56.513953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"88'8","reported_seq":144,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:06.037385+0000","last_change":"2024-03-24T10:38:09.864160+0000","last_active":"2024-03-24T10:40:06.037385+0000","last_peered":"2024-03-24T10:40:06.037385+0000","last_clean":"2024-03-24T10:40:06.037385+0000","last_became_active":"2024-03-24T10:38:09.862066+0000","last_became_peered":"2024-03-24T10:38:09.862066+0000","last_unstale":"2024-03-24T10:40:06.037385+0000","last_undegraded":"2024-03-24T10:40:06.037385+0000","last_fullsized":"2024-03-24T10:40:06.037385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:59.003931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412874+0000","last_change":"2024-03-24T10:38:10.874433+0000","last_active":"2024-03-24T10:40:05.412874+0000","last_peered":"2024-03-24T10:40:05.412874+0000","last_clean":"2024-03-24T10:40:05.412874+0000","last_became_active":"2024-03-24T10:38:10.874090+0000","last_became_peered":"2024-03-24T10:38:10.874090+0000","last_unstale":"2024-03-24T10:40:05.412874+0000","last_undegraded":"2024-03-24T10:40:05.412874+0000","last_fullsized":"2024-03-24T10:40:05.412874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:25.979419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422847+0000","last_change":"2024-03-24T10:38:09.863028+0000","last_active":"2024-03-24T10:40:05.422847+0000","last_peered":"2024-03-24T10:40:05.422847+0000","last_clean":"2024-03-24T10:40:05.422847+0000","last_became_active":"2024-03-24T10:38:09.862682+0000","last_became_peered":"2024-03-24T10:38:09.862682+0000","last_unstale":"2024-03-24T10:40:05.422847+0000","last_undegraded":"2024-03-24T10:40:05.422847+0000","last_fullsized":"2024-03-24T10:40:05.422847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:14.962115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300267+0000","last_change":"2024-03-24T10:38:10.885950+0000","last_active":"2024-03-24T10:40:05.300267+0000","last_peered":"2024-03-24T10:40:05.300267+0000","last_clean":"2024-03-24T10:40:05.300267+0000","last_became_active":"2024-03-24T10:38:10.885685+0000","last_became_peered":"2024-03-24T10:38:10.885685+0000","last_unstale":"2024-03-24T10:40:05.300267+0000","last_undegraded":"2024-03-24T10:40:05.300267+0000","last_fullsized":"2024-03-24T10:40:05.300267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:42:09.912743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575251+0000","last_change":"2024-03-24T10:38:09.860376+0000","last_active":"2024-03-24T10:40:05.575251+0000","last_peered":"2024-03-24T10:40:05.575251+0000","last_clean":"2024-03-24T10:40:05.575251+0000","last_became_active":"2024-03-24T10:38:09.860062+0000","last_became_peered":"2024-03-24T10:38:09.860062+0000","last_unstale":"2024-03-24T10:40:05.575251+0000","last_undegraded":"2024-03-24T10:40:05.575251+0000","last_fullsized":"2024-03-24T10:40:05.575251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:33.387953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413177+0000","last_change":"2024-03-24T10:38:10.883843+0000","last_active":"2024-03-24T10:40:05.413177+0000","last_peered":"2024-03-24T10:40:05.413177+0000","last_clean":"2024-03-24T10:40:05.413177+0000","last_became_active":"2024-03-24T10:38:10.883170+0000","last_became_peered":"2024-03-24T10:38:10.883170+0000","last_unstale":"2024-03-24T10:40:05.413177+0000","last_undegraded":"2024-03-24T10:40:05.413177+0000","last_fullsized":"2024-03-24T10:40:05.413177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:28:16.798546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299912+0000","last_change":"2024-03-24T10:38:09.862299+0000","last_active":"2024-03-24T10:40:05.299912+0000","last_peered":"2024-03-24T10:40:05.299912+0000","last_clean":"2024-03-24T10:40:05.299912+0000","last_became_active":"2024-03-24T10:38:09.861895+0000","last_became_peered":"2024-03-24T10:38:09.861895+0000","last_unstale":"2024-03-24T10:40:05.299912+0000","last_undegraded":"2024-03-24T10:40:05.299912+0000","last_fullsized":"2024-03-24T10:40:05.299912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:13:19.869511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.297954+0000","last_change":"2024-03-24T10:38:10.880507+0000","last_active":"2024-03-24T10:40:05.297954+0000","last_peered":"2024-03-24T10:40:05.297954+0000","last_clean":"2024-03-24T10:40:05.297954+0000","last_became_active":"2024-03-24T10:38:10.879790+0000","last_became_peered":"2024-03-24T10:38:10.879790+0000","last_unstale":"2024-03-24T10:40:05.297954+0000","last_undegraded":"2024-03-24T10:40:05.297954+0000","last_fullsized":"2024-03-24T10:40:05.297954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:11:02.650420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'4","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.423038+0000","last_change":"2024-03-24T10:38:09.860186+0000","last_active":"2024-03-24T10:40:05.423038+0000","last_peered":"2024-03-24T10:40:05.423038+0000","last_clean":"2024-03-24T10:40:05.423038+0000","last_became_active":"2024-03-24T10:38:09.859834+0000","last_became_peered":"2024-03-24T10:38:09.859834+0000","last_unstale":"2024-03-24T10:40:05.423038+0000","last_undegraded":"2024-03-24T10:40:05.423038+0000","last_fullsized":"2024-03-24T10:40:05.423038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:33:48.029053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575833+0000","last_change":"2024-03-24T10:38:10.875634+0000","last_active":"2024-03-24T10:40:05.575833+0000","last_peered":"2024-03-24T10:40:05.575833+0000","last_clean":"2024-03-24T10:40:05.575833+0000","last_became_active":"2024-03-24T10:38:10.875101+0000","last_became_peered":"2024-03-24T10:38:10.875101+0000","last_unstale":"2024-03-24T10:40:05.575833+0000","last_undegraded":"2024-03-24T10:40:05.575833+0000","last_fullsized":"2024-03-24T10:40:05.575833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:32:58.165087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300091+0000","last_change":"2024-03-24T10:38:09.860412+0000","last_active":"2024-03-24T10:40:05.300091+0000","last_peered":"2024-03-24T10:40:05.300091+0000","last_clean":"2024-03-24T10:40:05.300091+0000","last_became_active":"2024-03-24T10:38:09.860084+0000","last_became_peered":"2024-03-24T10:38:09.860084+0000","last_unstale":"2024-03-24T10:40:05.300091+0000","last_undegraded":"2024-03-24T10:40:05.300091+0000","last_fullsized":"2024-03-24T10:40:05.300091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:44.585482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"87'9","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576331+0000","last_change":"2024-03-24T10:39:16.703566+0000","last_active":"2024-03-24T10:40:05.576331+0000","last_peered":"2024-03-24T10:40:05.576331+0000","last_clean":"2024-03-24T10:40:05.576331+0000","last_became_active":"2024-03-24T10:39:16.703213+0000","last_became_peered":"2024-03-24T10:39:16.703213+0000","last_unstale":"2024-03-24T10:40:05.576331+0000","last_undegraded":"2024-03-24T10:40:05.576331+0000","last_fullsized":"2024-03-24T10:40:05.576331+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:35:10.888996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300419+0000","last_change":"2024-03-24T10:38:09.865191+0000","last_active":"2024-03-24T10:40:05.300419+0000","last_peered":"2024-03-24T10:40:05.300419+0000","last_clean":"2024-03-24T10:40:05.300419+0000","last_became_active":"2024-03-24T10:38:09.864470+0000","last_became_peered":"2024-03-24T10:38:09.864470+0000","last_unstale":"2024-03-24T10:40:05.300419+0000","last_undegraded":"2024-03-24T10:40:05.300419+0000","last_fullsized":"2024-03-24T10:40:05.300419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:45:00.266470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575474+0000","last_change":"2024-03-24T10:38:10.889668+0000","last_active":"2024-03-24T10:40:05.575474+0000","last_peered":"2024-03-24T10:40:05.575474+0000","last_clean":"2024-03-24T10:40:05.575474+0000","last_became_active":"2024-03-24T10:38:10.889364+0000","last_became_peered":"2024-03-24T10:38:10.889364+0000","last_unstale":"2024-03-24T10:40:05.575474+0000","last_undegraded":"2024-03-24T10:40:05.575474+0000","last_fullsized":"2024-03-24T10:40:05.575474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:57.702719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'35","reported_seq":289,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:06.037599+0000","last_change":"2024-03-24T10:38:09.861809+0000","last_active":"2024-03-24T10:40:06.037599+0000","last_peered":"2024-03-24T10:40:06.037599+0000","last_clean":"2024-03-24T10:40:06.037599+0000","last_became_active":"2024-03-24T10:38:09.861444+0000","last_became_peered":"2024-03-24T10:38:09.861444+0000","last_unstale":"2024-03-24T10:40:06.037599+0000","last_undegraded":"2024-03-24T10:40:06.037599+0000","last_fullsized":"2024-03-24T10:40:06.037599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:19.652188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65118,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":233,"num_read_kb":180,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413060+0000","last_change":"2024-03-24T10:38:10.883090+0000","last_active":"2024-03-24T10:40:05.413060+0000","last_peered":"2024-03-24T10:40:05.413060+0000","last_clean":"2024-03-24T10:40:05.413060+0000","last_became_active":"2024-03-24T10:38:10.880507+0000","last_became_peered":"2024-03-24T10:38:10.880507+0000","last_unstale":"2024-03-24T10:40:05.413060+0000","last_undegraded":"2024-03-24T10:40:05.413060+0000","last_fullsized":"2024-03-24T10:40:05.413060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:01.422699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575220+0000","last_change":"2024-03-24T10:38:09.868072+0000","last_active":"2024-03-24T10:40:05.575220+0000","last_peered":"2024-03-24T10:40:05.575220+0000","last_clean":"2024-03-24T10:40:05.575220+0000","last_became_active":"2024-03-24T10:38:09.867704+0000","last_became_peered":"2024-03-24T10:38:09.867704+0000","last_unstale":"2024-03-24T10:40:05.575220+0000","last_undegraded":"2024-03-24T10:40:05.575220+0000","last_fullsized":"2024-03-24T10:40:05.575220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:09:28.726969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575265+0000","last_change":"2024-03-24T10:38:10.881882+0000","last_active":"2024-03-24T10:40:05.575265+0000","last_peered":"2024-03-24T10:40:05.575265+0000","last_clean":"2024-03-24T10:40:05.575265+0000","last_became_active":"2024-03-24T10:38:10.881495+0000","last_became_peered":"2024-03-24T10:38:10.881495+0000","last_unstale":"2024-03-24T10:40:05.575265+0000","last_undegraded":"2024-03-24T10:40:05.575265+0000","last_fullsized":"2024-03-24T10:40:05.575265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:31:43.392952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300200+0000","last_change":"2024-03-24T10:38:09.867265+0000","last_active":"2024-03-24T10:40:05.300200+0000","last_peered":"2024-03-24T10:40:05.300200+0000","last_clean":"2024-03-24T10:40:05.300200+0000","last_became_active":"2024-03-24T10:38:09.866669+0000","last_became_peered":"2024-03-24T10:38:09.866669+0000","last_unstale":"2024-03-24T10:40:05.300200+0000","last_undegraded":"2024-03-24T10:40:05.300200+0000","last_fullsized":"2024-03-24T10:40:05.300200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:34:28.230967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300253+0000","last_change":"2024-03-24T10:38:10.886634+0000","last_active":"2024-03-24T10:40:05.300253+0000","last_peered":"2024-03-24T10:40:05.300253+0000","last_clean":"2024-03-24T10:40:05.300253+0000","last_became_active":"2024-03-24T10:38:10.886207+0000","last_became_peered":"2024-03-24T10:38:10.886207+0000","last_unstale":"2024-03-24T10:40:05.300253+0000","last_undegraded":"2024-03-24T10:40:05.300253+0000","last_fullsized":"2024-03-24T10:40:05.300253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:43:41.601693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'5","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575133+0000","last_change":"2024-03-24T10:38:09.863262+0000","last_active":"2024-03-24T10:40:05.575133+0000","last_peered":"2024-03-24T10:40:05.575133+0000","last_clean":"2024-03-24T10:40:05.575133+0000","last_became_active":"2024-03-24T10:38:09.862665+0000","last_became_peered":"2024-03-24T10:38:09.862665+0000","last_unstale":"2024-03-24T10:40:05.575133+0000","last_undegraded":"2024-03-24T10:40:05.575133+0000","last_fullsized":"2024-03-24T10:40:05.575133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:12:29.117592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299650+0000","last_change":"2024-03-24T10:38:10.880793+0000","last_active":"2024-03-24T10:40:05.299650+0000","last_peered":"2024-03-24T10:40:05.299650+0000","last_clean":"2024-03-24T10:40:05.299650+0000","last_became_active":"2024-03-24T10:38:10.880449+0000","last_became_peered":"2024-03-24T10:38:10.880449+0000","last_unstale":"2024-03-24T10:40:05.299650+0000","last_undegraded":"2024-03-24T10:40:05.299650+0000","last_fullsized":"2024-03-24T10:40:05.299650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:32:57.184014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412771+0000","last_change":"2024-03-24T10:39:16.915438+0000","last_active":"2024-03-24T10:40:05.412771+0000","last_peered":"2024-03-24T10:40:05.412771+0000","last_clean":"2024-03-24T10:40:05.412771+0000","last_became_active":"2024-03-24T10:39:16.906155+0000","last_became_peered":"2024-03-24T10:39:16.906155+0000","last_unstale":"2024-03-24T10:40:05.412771+0000","last_undegraded":"2024-03-24T10:40:05.412771+0000","last_fullsized":"2024-03-24T10:40:05.412771+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:36:12.356069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299997+0000","last_change":"2024-03-24T10:38:10.881952+0000","last_active":"2024-03-24T10:40:05.299997+0000","last_peered":"2024-03-24T10:40:05.299997+0000","last_clean":"2024-03-24T10:40:05.299997+0000","last_became_active":"2024-03-24T10:38:10.881628+0000","last_became_peered":"2024-03-24T10:38:10.881628+0000","last_unstale":"2024-03-24T10:40:05.299997+0000","last_undegraded":"2024-03-24T10:40:05.299997+0000","last_fullsized":"2024-03-24T10:40:05.299997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:24.777793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413319+0000","last_change":"2024-03-24T10:38:09.868572+0000","last_active":"2024-03-24T10:40:05.413319+0000","last_peered":"2024-03-24T10:40:05.413319+0000","last_clean":"2024-03-24T10:40:05.413319+0000","last_became_active":"2024-03-24T10:38:09.867702+0000","last_became_peered":"2024-03-24T10:38:09.867702+0000","last_unstale":"2024-03-24T10:40:05.413319+0000","last_undegraded":"2024-03-24T10:40:05.413319+0000","last_fullsized":"2024-03-24T10:40:05.413319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:27.890114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575267+0000","last_change":"2024-03-24T10:38:10.875342+0000","last_active":"2024-03-24T10:40:05.575267+0000","last_peered":"2024-03-24T10:40:05.575267+0000","last_clean":"2024-03-24T10:40:05.575267+0000","last_became_active":"2024-03-24T10:38:10.874933+0000","last_became_peered":"2024-03-24T10:38:10.874933+0000","last_unstale":"2024-03-24T10:40:05.575267+0000","last_undegraded":"2024-03-24T10:40:05.575267+0000","last_fullsized":"2024-03-24T10:40:05.575267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:56:09.139478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.298000+0000","last_change":"2024-03-24T10:38:09.857867+0000","last_active":"2024-03-24T10:40:05.298000+0000","last_peered":"2024-03-24T10:40:05.298000+0000","last_clean":"2024-03-24T10:40:05.298000+0000","last_became_active":"2024-03-24T10:38:09.857525+0000","last_became_peered":"2024-03-24T10:38:09.857525+0000","last_unstale":"2024-03-24T10:40:05.298000+0000","last_undegraded":"2024-03-24T10:40:05.298000+0000","last_fullsized":"2024-03-24T10:40:05.298000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:11:37.320245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422684+0000","last_change":"2024-03-24T10:38:10.878980+0000","last_active":"2024-03-24T10:40:05.422684+0000","last_peered":"2024-03-24T10:40:05.422684+0000","last_clean":"2024-03-24T10:40:05.422684+0000","last_became_active":"2024-03-24T10:38:10.878648+0000","last_became_peered":"2024-03-24T10:38:10.878648+0000","last_unstale":"2024-03-24T10:40:05.422684+0000","last_undegraded":"2024-03-24T10:40:05.422684+0000","last_fullsized":"2024-03-24T10:40:05.422684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:07:49.988403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299957+0000","last_change":"2024-03-24T10:38:09.863457+0000","last_active":"2024-03-24T10:40:05.299957+0000","last_peered":"2024-03-24T10:40:05.299957+0000","last_clean":"2024-03-24T10:40:05.299957+0000","last_became_active":"2024-03-24T10:38:09.863130+0000","last_became_peered":"2024-03-24T10:38:09.863130+0000","last_unstale":"2024-03-24T10:40:05.299957+0000","last_undegraded":"2024-03-24T10:40:05.299957+0000","last_fullsized":"2024-03-24T10:40:05.299957+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:16.528703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.422825+0000","last_change":"2024-03-24T10:38:10.887014+0000","last_active":"2024-03-24T10:40:05.422825+0000","last_peered":"2024-03-24T10:40:05.422825+0000","last_clean":"2024-03-24T10:40:05.422825+0000","last_became_active":"2024-03-24T10:38:10.886698+0000","last_became_peered":"2024-03-24T10:38:10.886698+0000","last_unstale":"2024-03-24T10:40:05.422825+0000","last_undegraded":"2024-03-24T10:40:05.422825+0000","last_fullsized":"2024-03-24T10:40:05.422825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:47:08.571886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300143+0000","last_change":"2024-03-24T10:38:10.874811+0000","last_active":"2024-03-24T10:40:05.300143+0000","last_peered":"2024-03-24T10:40:05.300143+0000","last_clean":"2024-03-24T10:40:05.300143+0000","last_became_active":"2024-03-24T10:38:10.874483+0000","last_became_peered":"2024-03-24T10:38:10.874483+0000","last_unstale":"2024-03-24T10:40:05.300143+0000","last_undegraded":"2024-03-24T10:40:05.300143+0000","last_fullsized":"2024-03-24T10:40:05.300143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:02:57.382174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'5","reported_seq":137,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575176+0000","last_change":"2024-03-24T10:38:09.865614+0000","last_active":"2024-03-24T10:40:05.575176+0000","last_peered":"2024-03-24T10:40:05.575176+0000","last_clean":"2024-03-24T10:40:05.575176+0000","last_became_active":"2024-03-24T10:38:09.865317+0000","last_became_peered":"2024-03-24T10:38:09.865317+0000","last_unstale":"2024-03-24T10:40:05.575176+0000","last_undegraded":"2024-03-24T10:40:05.575176+0000","last_fullsized":"2024-03-24T10:40:05.575176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:21:14.968552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413502+0000","last_change":"2024-03-24T10:38:10.884847+0000","last_active":"2024-03-24T10:40:05.413502+0000","last_peered":"2024-03-24T10:40:05.413502+0000","last_clean":"2024-03-24T10:40:05.413502+0000","last_became_active":"2024-03-24T10:38:10.884526+0000","last_became_peered":"2024-03-24T10:38:10.884526+0000","last_unstale":"2024-03-24T10:40:05.413502+0000","last_undegraded":"2024-03-24T10:40:05.413502+0000","last_fullsized":"2024-03-24T10:40:05.413502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:01:49.364083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575339+0000","last_change":"2024-03-24T10:38:09.858893+0000","last_active":"2024-03-24T10:40:05.575339+0000","last_peered":"2024-03-24T10:40:05.575339+0000","last_clean":"2024-03-24T10:40:05.575339+0000","last_became_active":"2024-03-24T10:38:09.858474+0000","last_became_peered":"2024-03-24T10:38:09.858474+0000","last_unstale":"2024-03-24T10:40:05.575339+0000","last_undegraded":"2024-03-24T10:40:05.575339+0000","last_fullsized":"2024-03-24T10:40:05.575339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:03:36.709498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299715+0000","last_change":"2024-03-24T10:36:59.100728+0000","last_active":"2024-03-24T10:40:05.299715+0000","last_peered":"2024-03-24T10:40:05.299715+0000","last_clean":"2024-03-24T10:40:05.299715+0000","last_became_active":"2024-03-24T10:36:59.100159+0000","last_became_peered":"2024-03-24T10:36:59.100159+0000","last_unstale":"2024-03-24T10:40:05.299715+0000","last_undegraded":"2024-03-24T10:40:05.299715+0000","last_fullsized":"2024-03-24T10:40:05.299715+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:33:39.429553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:33:39.429553+0000","last_clean_scrub_stamp":"2024-03-24T10:33:39.429553+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:21:01.470585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574970+0000","last_change":"2024-03-24T10:38:10.885804+0000","last_active":"2024-03-24T10:40:05.574970+0000","last_peered":"2024-03-24T10:40:05.574970+0000","last_clean":"2024-03-24T10:40:05.574970+0000","last_became_active":"2024-03-24T10:38:10.885447+0000","last_became_peered":"2024-03-24T10:38:10.885447+0000","last_unstale":"2024-03-24T10:40:05.574970+0000","last_undegraded":"2024-03-24T10:40:05.574970+0000","last_fullsized":"2024-03-24T10:40:05.574970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:43:57.103205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299830+0000","last_change":"2024-03-24T10:38:09.863187+0000","last_active":"2024-03-24T10:40:05.299830+0000","last_peered":"2024-03-24T10:40:05.299830+0000","last_clean":"2024-03-24T10:40:05.299830+0000","last_became_active":"2024-03-24T10:38:09.862859+0000","last_became_peered":"2024-03-24T10:38:09.862859+0000","last_unstale":"2024-03-24T10:40:05.299830+0000","last_undegraded":"2024-03-24T10:40:05.299830+0000","last_fullsized":"2024-03-24T10:40:05.299830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:21:02.707127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413190+0000","last_change":"2024-03-24T10:38:09.867899+0000","last_active":"2024-03-24T10:40:05.413190+0000","last_peered":"2024-03-24T10:40:05.413190+0000","last_clean":"2024-03-24T10:40:05.413190+0000","last_became_active":"2024-03-24T10:38:09.867563+0000","last_became_peered":"2024-03-24T10:38:09.867563+0000","last_unstale":"2024-03-24T10:40:05.413190+0000","last_undegraded":"2024-03-24T10:40:05.413190+0000","last_fullsized":"2024-03-24T10:40:05.413190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:51.067981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575138+0000","last_change":"2024-03-24T10:38:10.870825+0000","last_active":"2024-03-24T10:40:05.575138+0000","last_peered":"2024-03-24T10:40:05.575138+0000","last_clean":"2024-03-24T10:40:05.575138+0000","last_became_active":"2024-03-24T10:38:10.870491+0000","last_became_peered":"2024-03-24T10:38:10.870491+0000","last_unstale":"2024-03-24T10:40:05.575138+0000","last_undegraded":"2024-03-24T10:40:05.575138+0000","last_fullsized":"2024-03-24T10:40:05.575138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:30:34.091299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'44","reported_seq":88,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.527925+0000","last_change":"2024-03-24T10:38:09.863724+0000","last_active":"2024-03-24T10:40:05.527925+0000","last_peered":"2024-03-24T10:40:05.527925+0000","last_clean":"2024-03-24T10:40:05.527925+0000","last_became_active":"2024-03-24T10:38:09.862900+0000","last_became_peered":"2024-03-24T10:38:09.862900+0000","last_unstale":"2024-03-24T10:40:05.527925+0000","last_undegraded":"2024-03-24T10:40:05.527925+0000","last_fullsized":"2024-03-24T10:40:05.527925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:21.646074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413886+0000","last_change":"2024-03-24T10:38:10.884661+0000","last_active":"2024-03-24T10:40:05.413886+0000","last_peered":"2024-03-24T10:40:05.413886+0000","last_clean":"2024-03-24T10:40:05.413886+0000","last_became_active":"2024-03-24T10:38:10.882901+0000","last_became_peered":"2024-03-24T10:38:10.882901+0000","last_unstale":"2024-03-24T10:40:05.413886+0000","last_undegraded":"2024-03-24T10:40:05.413886+0000","last_fullsized":"2024-03-24T10:40:05.413886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:23:46.366567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576002+0000","last_change":"2024-03-24T10:38:09.862122+0000","last_active":"2024-03-24T10:40:05.576002+0000","last_peered":"2024-03-24T10:40:05.576002+0000","last_clean":"2024-03-24T10:40:05.576002+0000","last_became_active":"2024-03-24T10:38:09.861765+0000","last_became_peered":"2024-03-24T10:38:09.861765+0000","last_unstale":"2024-03-24T10:40:05.576002+0000","last_undegraded":"2024-03-24T10:40:05.576002+0000","last_fullsized":"2024-03-24T10:40:05.576002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:40:15.148349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575995+0000","last_change":"2024-03-24T10:38:10.886797+0000","last_active":"2024-03-24T10:40:05.575995+0000","last_peered":"2024-03-24T10:40:05.575995+0000","last_clean":"2024-03-24T10:40:05.575995+0000","last_became_active":"2024-03-24T10:38:10.886406+0000","last_became_peered":"2024-03-24T10:38:10.886406+0000","last_unstale":"2024-03-24T10:40:05.575995+0000","last_undegraded":"2024-03-24T10:40:05.575995+0000","last_fullsized":"2024-03-24T10:40:05.575995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:35.763623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575166+0000","last_change":"2024-03-24T10:39:16.707034+0000","last_active":"2024-03-24T10:40:05.575166+0000","last_peered":"2024-03-24T10:40:05.575166+0000","last_clean":"2024-03-24T10:40:05.575166+0000","last_became_active":"2024-03-24T10:39:16.701995+0000","last_became_peered":"2024-03-24T10:39:16.701995+0000","last_unstale":"2024-03-24T10:40:05.575166+0000","last_undegraded":"2024-03-24T10:40:05.575166+0000","last_fullsized":"2024-03-24T10:40:05.575166+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:18:41.362740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412756+0000","last_change":"2024-03-24T10:38:10.873871+0000","last_active":"2024-03-24T10:40:05.412756+0000","last_peered":"2024-03-24T10:40:05.412756+0000","last_clean":"2024-03-24T10:40:05.412756+0000","last_became_active":"2024-03-24T10:38:10.873528+0000","last_became_peered":"2024-03-24T10:38:10.873528+0000","last_unstale":"2024-03-24T10:40:05.412756+0000","last_undegraded":"2024-03-24T10:40:05.412756+0000","last_fullsized":"2024-03-24T10:40:05.412756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:33.372195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413424+0000","last_change":"2024-03-24T10:38:09.861548+0000","last_active":"2024-03-24T10:40:05.413424+0000","last_peered":"2024-03-24T10:40:05.413424+0000","last_clean":"2024-03-24T10:40:05.413424+0000","last_became_active":"2024-03-24T10:38:09.861219+0000","last_became_peered":"2024-03-24T10:38:09.861219+0000","last_unstale":"2024-03-24T10:40:05.413424+0000","last_undegraded":"2024-03-24T10:40:05.413424+0000","last_fullsized":"2024-03-24T10:40:05.413424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:57:46.398179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574856+0000","last_change":"2024-03-24T10:38:10.880202+0000","last_active":"2024-03-24T10:40:05.574856+0000","last_peered":"2024-03-24T10:40:05.574856+0000","last_clean":"2024-03-24T10:40:05.574856+0000","last_became_active":"2024-03-24T10:38:10.879807+0000","last_became_peered":"2024-03-24T10:38:10.879807+0000","last_unstale":"2024-03-24T10:40:05.574856+0000","last_undegraded":"2024-03-24T10:40:05.574856+0000","last_fullsized":"2024-03-24T10:40:05.574856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:37.498728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575135+0000","last_change":"2024-03-24T10:39:16.704470+0000","last_active":"2024-03-24T10:40:05.575135+0000","last_peered":"2024-03-24T10:40:05.575135+0000","last_clean":"2024-03-24T10:40:05.575135+0000","last_became_active":"2024-03-24T10:39:16.701168+0000","last_became_peered":"2024-03-24T10:39:16.701168+0000","last_unstale":"2024-03-24T10:40:05.575135+0000","last_undegraded":"2024-03-24T10:40:05.575135+0000","last_fullsized":"2024-03-24T10:40:05.575135+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:20:34.781782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413602+0000","last_change":"2024-03-24T10:38:10.884371+0000","last_active":"2024-03-24T10:40:05.413602+0000","last_peered":"2024-03-24T10:40:05.413602+0000","last_clean":"2024-03-24T10:40:05.413602+0000","last_became_active":"2024-03-24T10:38:10.884022+0000","last_became_peered":"2024-03-24T10:38:10.884022+0000","last_unstale":"2024-03-24T10:40:05.413602+0000","last_undegraded":"2024-03-24T10:40:05.413602+0000","last_fullsized":"2024-03-24T10:40:05.413602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:47:48.778727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.412949+0000","last_change":"2024-03-24T10:38:09.863231+0000","last_active":"2024-03-24T10:40:05.412949+0000","last_peered":"2024-03-24T10:40:05.412949+0000","last_clean":"2024-03-24T10:40:05.412949+0000","last_became_active":"2024-03-24T10:38:09.862848+0000","last_became_peered":"2024-03-24T10:38:09.862848+0000","last_unstale":"2024-03-24T10:40:05.412949+0000","last_undegraded":"2024-03-24T10:40:05.412949+0000","last_fullsized":"2024-03-24T10:40:05.412949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:36:57.532746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"87'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300075+0000","last_change":"2024-03-24T10:38:10.875187+0000","last_active":"2024-03-24T10:40:05.300075+0000","last_peered":"2024-03-24T10:40:05.300075+0000","last_clean":"2024-03-24T10:40:05.300075+0000","last_became_active":"2024-03-24T10:38:10.874682+0000","last_became_peered":"2024-03-24T10:38:10.874682+0000","last_unstale":"2024-03-24T10:40:05.300075+0000","last_undegraded":"2024-03-24T10:40:05.300075+0000","last_fullsized":"2024-03-24T10:40:05.300075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:53:56.915312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575456+0000","last_change":"2024-03-24T10:38:09.863430+0000","last_active":"2024-03-24T10:40:05.575456+0000","last_peered":"2024-03-24T10:40:05.575456+0000","last_clean":"2024-03-24T10:40:05.575456+0000","last_became_active":"2024-03-24T10:38:09.862877+0000","last_became_peered":"2024-03-24T10:38:09.862877+0000","last_unstale":"2024-03-24T10:40:05.575456+0000","last_undegraded":"2024-03-24T10:40:05.575456+0000","last_fullsized":"2024-03-24T10:40:05.575456+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:33:29.039896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"87'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300427+0000","last_change":"2024-03-24T10:38:10.878446+0000","last_active":"2024-03-24T10:40:05.300427+0000","last_peered":"2024-03-24T10:40:05.300427+0000","last_clean":"2024-03-24T10:40:05.300427+0000","last_became_active":"2024-03-24T10:38:10.877944+0000","last_became_peered":"2024-03-24T10:38:10.877944+0000","last_unstale":"2024-03-24T10:40:05.300427+0000","last_undegraded":"2024-03-24T10:40:05.300427+0000","last_fullsized":"2024-03-24T10:40:05.300427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:00:22.568669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'39","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.525634+0000","last_change":"2024-03-24T10:38:09.867328+0000","last_active":"2024-03-24T10:40:05.525634+0000","last_peered":"2024-03-24T10:40:05.525634+0000","last_clean":"2024-03-24T10:40:05.525634+0000","last_became_active":"2024-03-24T10:38:09.866869+0000","last_became_peered":"2024-03-24T10:38:09.866869+0000","last_unstale":"2024-03-24T10:40:05.525634+0000","last_undegraded":"2024-03-24T10:40:05.525634+0000","last_fullsized":"2024-03-24T10:40:05.525634+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:52:32.136151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'7","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300060+0000","last_change":"2024-03-24T10:39:16.705912+0000","last_active":"2024-03-24T10:40:05.300060+0000","last_peered":"2024-03-24T10:40:05.300060+0000","last_clean":"2024-03-24T10:40:05.300060+0000","last_became_active":"2024-03-24T10:39:16.699809+0000","last_became_peered":"2024-03-24T10:39:16.699809+0000","last_unstale":"2024-03-24T10:40:05.300060+0000","last_undegraded":"2024-03-24T10:40:05.300060+0000","last_fullsized":"2024-03-24T10:40:05.300060+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:16:31.858976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575611+0000","last_change":"2024-03-24T10:38:09.865399+0000","last_active":"2024-03-24T10:40:05.575611+0000","last_peered":"2024-03-24T10:40:05.575611+0000","last_clean":"2024-03-24T10:40:05.575611+0000","last_became_active":"2024-03-24T10:38:09.864906+0000","last_became_peered":"2024-03-24T10:38:09.864906+0000","last_unstale":"2024-03-24T10:40:05.575611+0000","last_undegraded":"2024-03-24T10:40:05.575611+0000","last_fullsized":"2024-03-24T10:40:05.575611+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:08.835108+0000","last_clean_scrub_stamp":"2024-03-24T10:38:08.835108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:53:12.363018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.300349+0000","last_change":"2024-03-24T10:38:10.876881+0000","last_active":"2024-03-24T10:40:05.300349+0000","last_peered":"2024-03-24T10:40:05.300349+0000","last_clean":"2024-03-24T10:40:05.300349+0000","last_became_active":"2024-03-24T10:38:10.876479+0000","last_became_peered":"2024-03-24T10:38:10.876479+0000","last_unstale":"2024-03-24T10:40:05.300349+0000","last_undegraded":"2024-03-24T10:40:05.300349+0000","last_fullsized":"2024-03-24T10:40:05.300349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:57:33.140451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"87'9","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574900+0000","last_change":"2024-03-24T10:38:10.872622+0000","last_active":"2024-03-24T10:40:05.574900+0000","last_peered":"2024-03-24T10:40:05.574900+0000","last_clean":"2024-03-24T10:40:05.574900+0000","last_became_active":"2024-03-24T10:38:10.872293+0000","last_became_peered":"2024-03-24T10:38:10.872293+0000","last_unstale":"2024-03-24T10:40:05.574900+0000","last_undegraded":"2024-03-24T10:40:05.574900+0000","last_fullsized":"2024-03-24T10:40:05.574900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:48.139589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"87'12","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575020+0000","last_change":"2024-03-24T10:39:16.701548+0000","last_active":"2024-03-24T10:40:05.575020+0000","last_peered":"2024-03-24T10:40:05.575020+0000","last_clean":"2024-03-24T10:40:05.575020+0000","last_became_active":"2024-03-24T10:39:16.701213+0000","last_became_peered":"2024-03-24T10:39:16.701213+0000","last_unstale":"2024-03-24T10:40:05.575020+0000","last_undegraded":"2024-03-24T10:40:05.575020+0000","last_fullsized":"2024-03-24T10:40:05.575020+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:50:28.131754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.574960+0000","last_change":"2024-03-24T10:38:10.879679+0000","last_active":"2024-03-24T10:40:05.574960+0000","last_peered":"2024-03-24T10:40:05.574960+0000","last_clean":"2024-03-24T10:40:05.574960+0000","last_became_active":"2024-03-24T10:38:10.879334+0000","last_became_peered":"2024-03-24T10:38:10.879334+0000","last_unstale":"2024-03-24T10:40:05.574960+0000","last_undegraded":"2024-03-24T10:40:05.574960+0000","last_fullsized":"2024-03-24T10:40:05.574960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:27:53.517368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"87'7","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.576067+0000","last_change":"2024-03-24T10:39:16.704444+0000","last_active":"2024-03-24T10:40:05.576067+0000","last_peered":"2024-03-24T10:40:05.576067+0000","last_clean":"2024-03-24T10:40:05.576067+0000","last_became_active":"2024-03-24T10:39:16.704085+0000","last_became_peered":"2024-03-24T10:39:16.704085+0000","last_unstale":"2024-03-24T10:40:05.576067+0000","last_undegraded":"2024-03-24T10:40:05.576067+0000","last_fullsized":"2024-03-24T10:40:05.576067+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:04:49.643195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575450+0000","last_change":"2024-03-24T10:38:10.887108+0000","last_active":"2024-03-24T10:40:05.575450+0000","last_peered":"2024-03-24T10:40:05.575450+0000","last_clean":"2024-03-24T10:40:05.575450+0000","last_became_active":"2024-03-24T10:38:10.886770+0000","last_became_peered":"2024-03-24T10:38:10.886770+0000","last_unstale":"2024-03-24T10:40:05.575450+0000","last_undegraded":"2024-03-24T10:40:05.575450+0000","last_fullsized":"2024-03-24T10:40:05.575450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:56.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575209+0000","last_change":"2024-03-24T10:38:10.873369+0000","last_active":"2024-03-24T10:40:05.575209+0000","last_peered":"2024-03-24T10:40:05.575209+0000","last_clean":"2024-03-24T10:40:05.575209+0000","last_became_active":"2024-03-24T10:38:10.873046+0000","last_became_peered":"2024-03-24T10:38:10.873046+0000","last_unstale":"2024-03-24T10:40:05.575209+0000","last_undegraded":"2024-03-24T10:40:05.575209+0000","last_fullsized":"2024-03-24T10:40:05.575209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:46.071084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575565+0000","last_change":"2024-03-24T10:38:10.878666+0000","last_active":"2024-03-24T10:40:05.575565+0000","last_peered":"2024-03-24T10:40:05.575565+0000","last_clean":"2024-03-24T10:40:05.575565+0000","last_became_active":"2024-03-24T10:38:10.877250+0000","last_became_peered":"2024-03-24T10:38:10.877250+0000","last_unstale":"2024-03-24T10:40:05.575565+0000","last_undegraded":"2024-03-24T10:40:05.575565+0000","last_fullsized":"2024-03-24T10:40:05.575565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:13:36.793584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413854+0000","last_change":"2024-03-24T10:38:10.889274+0000","last_active":"2024-03-24T10:40:05.413854+0000","last_peered":"2024-03-24T10:40:05.413854+0000","last_clean":"2024-03-24T10:40:05.413854+0000","last_became_active":"2024-03-24T10:38:10.888899+0000","last_became_peered":"2024-03-24T10:38:10.888899+0000","last_unstale":"2024-03-24T10:40:05.413854+0000","last_undegraded":"2024-03-24T10:40:05.413854+0000","last_fullsized":"2024-03-24T10:40:05.413854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:11.225562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.299977+0000","last_change":"2024-03-24T10:38:10.887135+0000","last_active":"2024-03-24T10:40:05.299977+0000","last_peered":"2024-03-24T10:40:05.299977+0000","last_clean":"2024-03-24T10:40:05.299977+0000","last_became_active":"2024-03-24T10:38:10.886812+0000","last_became_peered":"2024-03-24T10:38:10.886812+0000","last_unstale":"2024-03-24T10:40:05.299977+0000","last_undegraded":"2024-03-24T10:40:05.299977+0000","last_fullsized":"2024-03-24T10:40:05.299977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:51:53.959914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.413259+0000","last_change":"2024-03-24T10:38:10.885417+0000","last_active":"2024-03-24T10:40:05.413259+0000","last_peered":"2024-03-24T10:40:05.413259+0000","last_clean":"2024-03-24T10:40:05.413259+0000","last_became_active":"2024-03-24T10:38:10.883209+0000","last_became_peered":"2024-03-24T10:38:10.883209+0000","last_unstale":"2024-03-24T10:40:05.413259+0000","last_undegraded":"2024-03-24T10:40:05.413259+0000","last_fullsized":"2024-03-24T10:40:05.413259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:52:52.535611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575193+0000","last_change":"2024-03-24T10:38:10.885005+0000","last_active":"2024-03-24T10:40:05.575193+0000","last_peered":"2024-03-24T10:40:05.575193+0000","last_clean":"2024-03-24T10:40:05.575193+0000","last_became_active":"2024-03-24T10:38:10.884680+0000","last_became_peered":"2024-03-24T10:38:10.884680+0000","last_unstale":"2024-03-24T10:40:05.575193+0000","last_undegraded":"2024-03-24T10:40:05.575193+0000","last_fullsized":"2024-03-24T10:40:05.575193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:32:56.377777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"87'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575580+0000","last_change":"2024-03-24T10:38:10.875153+0000","last_active":"2024-03-24T10:40:05.575580+0000","last_peered":"2024-03-24T10:40:05.575580+0000","last_clean":"2024-03-24T10:40:05.575580+0000","last_became_active":"2024-03-24T10:38:10.874721+0000","last_became_peered":"2024-03-24T10:38:10.874721+0000","last_unstale":"2024-03-24T10:40:05.575580+0000","last_undegraded":"2024-03-24T10:40:05.575580+0000","last_fullsized":"2024-03-24T10:40:05.575580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:52.522797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"87'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T10:40:05.575025+0000","last_change":"2024-03-24T10:38:10.887298+0000","last_active":"2024-03-24T10:40:05.575025+0000","last_peered":"2024-03-24T10:40:05.575025+0000","last_clean":"2024-03-24T10:40:05.575025+0000","last_became_active":"2024-03-24T10:38:10.886975+0000","last_became_peered":"2024-03-24T10:38:10.886975+0000","last_unstale":"2024-03-24T10:40:05.575025+0000","last_undegraded":"2024-03-24T10:40:05.575025+0000","last_fullsized":"2024-03-24T10:40:05.575025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:38:09.841439+0000","last_clean_scrub_stamp":"2024-03-24T10:38:09.841439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:09.740303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595166,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":791,"num_read_kb":503,"num_write":285,"num_write_kb":1433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2015232,"data_stored":1785498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383716,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":465380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645353,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32936,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712216,"statfs":{"total":95995035648,"available":95961309184,"internally_reserved":0,"allocated":1802240,"data_stored":1253329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808878,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1380352,"data_stored":858949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32520,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712632,"statfs":{"total":95995035648,"available":95961735168,"internally_reserved":0,"allocated":1372160,"data_stored":858170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1576960,"data_stored":1058697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":465005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":530498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":466276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":532244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28492,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716660,"statfs":{"total":95995035648,"available":95965859840,"internally_reserved":0,"allocated":1511424,"data_stored":990015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":983040,"data_stored":466086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28700,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716452,"statfs":{"total":95995035648,"available":95965646848,"internally_reserved":0,"allocated":1724416,"data_stored":1189449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":65700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":67446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":393372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":198828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:40:10.431 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:40:10.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.569+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1944308216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ad4107710 msgr2=0x7f7ad4107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.569+0000 7f7ad9e40640 1 --2- 172.21.15.23:0/1944308216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ad4107710 0x7f7ad4107af0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8009930 tx=0x7f7ac802bcd0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.569+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1944308216 shutdown_connections 2024-03-24T10:40:10.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.569+0000 7f7ad9e40640 1 --2- 172.21.15.23:0/1944308216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad406a760 0x7f7ad410e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.569+0000 7f7ad9e40640 1 --2- 172.21.15.23:0/1944308216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ad4069dc0 0x7f7ad406a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.569+0000 7f7ad9e40640 1 --2- 172.21.15.23:0/1944308216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ad4107710 0x7f7ad4107af0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.569+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1944308216 >> 172.21.15.23:0/1944308216 conn(0x7f7ad40776b0 msgr2=0x7f7ad4077ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:10.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.569+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1944308216 shutdown_connections 2024-03-24T10:40:10.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.569+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1944308216 wait complete. 2024-03-24T10:40:10.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.569+0000 7f7ad9e40640 1 Processor -- start 2024-03-24T10:40:10.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad9e40640 1 -- start start 2024-03-24T10:40:10.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad9e40640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ad4069dc0 0x7f7ad4197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:10.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad9e40640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ad406a760 0x7f7ad41982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:10.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad9e40640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad4107710 0x7f7ad419c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:10.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad9e40640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad419cbe0 con 0x7f7ad406a760 2024-03-24T10:40:10.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad9e40640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad419cd50 con 0x7f7ad4107710 2024-03-24T10:40:10.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad9e40640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad419cf10 con 0x7f7ad4069dc0 2024-03-24T10:40:10.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ad406a760 0x7f7ad41982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:10.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ad406a760 0x7f7ad41982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33510/0 (socket says 172.21.15.23:33510) 2024-03-24T10:40:10.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad2ffd640 1 -- 172.21.15.23:0/1520188649 learned_addr learned my addr 172.21.15.23:0/1520188649 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:10.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad37fe640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ad4069dc0 0x7f7ad4197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad3fff640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad4107710 0x7f7ad419c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad2ffd640 1 -- 172.21.15.23:0/1520188649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ad4069dc0 msgr2=0x7f7ad4197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad2ffd640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ad4069dc0 0x7f7ad4197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad2ffd640 1 -- 172.21.15.23:0/1520188649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad4107710 msgr2=0x7f7ad419c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad2ffd640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad4107710 0x7f7ad419c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad2ffd640 1 -- 172.21.15.23:0/1520188649 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ac8009560 con 0x7f7ad406a760 2024-03-24T10:40:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad37fe640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ad4069dc0 0x7f7ad4197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.570+0000 7f7ad2ffd640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ad406a760 0x7f7ad41982b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400e9f0 tx=0x7f7ac400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.571+0000 7f7ad0ff9640 1 -- 172.21.15.23:0/1520188649 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4009720 con 0x7f7ad406a760 2024-03-24T10:40:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.571+0000 7f7ad0ff9640 1 -- 172.21.15.23:0/1520188649 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ac400cd80 con 0x7f7ad406a760 2024-03-24T10:40:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.571+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1520188649 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ad419d1f0 con 0x7f7ad406a760 2024-03-24T10:40:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.571+0000 7f7ad0ff9640 1 -- 172.21.15.23:0/1520188649 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4010690 con 0x7f7ad406a760 2024-03-24T10:40:10.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.571+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1520188649 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ad41a4a70 con 0x7f7ad406a760 2024-03-24T10:40:10.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.572+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1520188649 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ad4068950 con 0x7f7ad406a760 2024-03-24T10:40:10.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.573+0000 7f7ad0ff9640 1 -- 172.21.15.23:0/1520188649 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7f7ac40046c0 con 0x7f7ad406a760 2024-03-24T10:40:10.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.573+0000 7f7ad0ff9640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7abc0a2350 0x7f7abc0a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:10.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.574+0000 7f7ad37fe640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7abc0a2350 0x7f7abc0a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:10.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.574+0000 7f7ad0ff9640 1 -- 172.21.15.23:0/1520188649 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f7ac4014070 con 0x7f7ad406a760 2024-03-24T10:40:10.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.574+0000 7f7ad37fe640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7abc0a2350 0x7f7abc0a4810 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8005ab0 tx=0x7f7ac8005fd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:10.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.577+0000 7f7ad0ff9640 1 -- 172.21.15.23:0/1520188649 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7ac40972e0 con 0x7f7ad406a760 2024-03-24T10:40:10.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.760+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1520188649 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ad4199070 con 0x7f7ad406a760 2024-03-24T10:40:10.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.761+0000 7f7ad0ff9640 1 -- 172.21.15.23:0/1520188649 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23079 (secure 0 0 0) 0x7f7ac409a9a0 con 0x7f7ad406a760 2024-03-24T10:40:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:40:10.762 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:40:05.291943+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2350021632":"2024-03-25T10:40:05.291912+0000","172.21.15.23:6800/2350021632":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3614005933":"2024-03-25T10:40:05.291912+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/256452041":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/352422381":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/725846898":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:40:10.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.763+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1520188649 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7abc0a2350 msgr2=0x7f7abc0a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.763+0000 7f7ad9e40640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7abc0a2350 0x7f7abc0a4810 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8005ab0 tx=0x7f7ac8005fd0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.763+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1520188649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ad406a760 msgr2=0x7f7ad41982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.763+0000 7f7ad9e40640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ad406a760 0x7f7ad41982b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400e9f0 tx=0x7f7ac400edf0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.763+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1520188649 shutdown_connections 2024-03-24T10:40:10.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.763+0000 7f7ad9e40640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7abc0a2350 0x7f7abc0a4810 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.763+0000 7f7ad9e40640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad4107710 0x7f7ad419c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.763+0000 7f7ad9e40640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ad406a760 0x7f7ad41982b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.763+0000 7f7ad9e40640 1 --2- 172.21.15.23:0/1520188649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ad4069dc0 0x7f7ad4197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.763+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1520188649 >> 172.21.15.23:0/1520188649 conn(0x7f7ad40776b0 msgr2=0x7f7ad40ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:10.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.763+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1520188649 shutdown_connections 2024-03-24T10:40:10.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.763+0000 7f7ad9e40640 1 -- 172.21.15.23:0/1520188649 wait complete. 2024-03-24T10:40:10.775 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:39.056452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:38:08.473472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:38:09.414667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:40:10.775 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:40:10.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.883+0000 7ff6684c3640 1 -- 172.21.15.23:0/1169130624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6601087c0 msgr2=0x7ff660108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.883+0000 7ff6684c3640 1 --2- 172.21.15.23:0/1169130624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6601087c0 0x7ff660108ba0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009930 tx=0x7ff65c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:10.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.883+0000 7ff6684c3640 1 -- 172.21.15.23:0/1169130624 shutdown_connections 2024-03-24T10:40:10.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.883+0000 7ff6684c3640 1 --2- 172.21.15.23:0/1169130624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff66010d0c0 0x7ff660114be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.883+0000 7ff6684c3640 1 --2- 172.21.15.23:0/1169130624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff660109170 0x7ff66010c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.883+0000 7ff6684c3640 1 --2- 172.21.15.23:0/1169130624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6601087c0 0x7ff660108ba0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.883+0000 7ff6684c3640 1 -- 172.21.15.23:0/1169130624 >> 172.21.15.23:0/1169130624 conn(0x7ff660077490 msgr2=0x7ff6600798b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:10.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.883+0000 7ff6684c3640 1 -- 172.21.15.23:0/1169130624 shutdown_connections 2024-03-24T10:40:10.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.883+0000 7ff6684c3640 1 -- 172.21.15.23:0/1169130624 wait complete. 2024-03-24T10:40:10.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.883+0000 7ff6684c3640 1 Processor -- start 2024-03-24T10:40:10.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.883+0000 7ff6684c3640 1 -- start start 2024-03-24T10:40:10.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff6684c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6601087c0 0x7ff66019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:10.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff6684c3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff66010d0c0 0x7ff66019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:10.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff6684c3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6601a2ca0 0x7ff6601a3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:10.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff6684c3640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff6601a3690 con 0x7ff6601a2ca0 2024-03-24T10:40:10.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff6684c3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff6601a3800 con 0x7ff6601087c0 2024-03-24T10:40:10.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff6684c3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff6601a3970 con 0x7ff66010d0c0 2024-03-24T10:40:10.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff666a60640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6601a2ca0 0x7ff6601a3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:10.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff66625f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6601087c0 0x7ff66019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:10.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff66625f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6601087c0 0x7ff66019e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:57360/0 (socket says 172.21.15.23:57360) 2024-03-24T10:40:10.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff66625f640 1 -- 172.21.15.23:0/648472025 learned_addr learned my addr 172.21.15.23:0/648472025 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:10.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff666a60640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6601a2ca0 0x7ff6601a3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33516/0 (socket says 172.21.15.23:33516) 2024-03-24T10:40:10.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff665a5e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff66010d0c0 0x7ff66019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:10.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff66625f640 1 -- 172.21.15.23:0/648472025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff66010d0c0 msgr2=0x7ff66019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff66625f640 1 --2- 172.21.15.23:0/648472025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff66010d0c0 0x7ff66019e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff66625f640 1 -- 172.21.15.23:0/648472025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6601a2ca0 msgr2=0x7ff6601a3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:10.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff66625f640 1 --2- 172.21.15.23:0/648472025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6601a2ca0 0x7ff6601a3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:10.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.884+0000 7ff66625f640 1 -- 172.21.15.23:0/648472025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff650009490 con 0x7ff6601087c0 2024-03-24T10:40:10.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.885+0000 7ff66625f640 1 --2- 172.21.15.23:0/648472025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6601087c0 0x7ff66019e430 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009230 tx=0x7ff65c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:10.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.885+0000 7ff6577fe640 1 -- 172.21.15.23:0/648472025 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65c004350 con 0x7ff6601087c0 2024-03-24T10:40:10.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.885+0000 7ff6684c3640 1 -- 172.21.15.23:0/648472025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff658009840 con 0x7ff6601087c0 2024-03-24T10:40:10.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.885+0000 7ff6577fe640 1 -- 172.21.15.23:0/648472025 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff65c0044b0 con 0x7ff6601087c0 2024-03-24T10:40:10.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.885+0000 7ff6577fe640 1 -- 172.21.15.23:0/648472025 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65c003bf0 con 0x7ff6601087c0 2024-03-24T10:40:10.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.885+0000 7ff6684c3640 1 -- 172.21.15.23:0/648472025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff65c009560 con 0x7ff6601087c0 2024-03-24T10:40:10.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.886+0000 7ff6684c3640 1 -- 172.21.15.23:0/648472025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff66019f730 con 0x7ff6601087c0 2024-03-24T10:40:10.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.887+0000 7ff6577fe640 1 -- 172.21.15.23:0/648472025 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7ff65c003d50 con 0x7ff6601087c0 2024-03-24T10:40:10.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.888+0000 7ff6577fe640 1 --2- 172.21.15.23:0/648472025 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff64c0a2350 0x7ff64c0a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:10.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.888+0000 7ff665a5e640 1 --2- 172.21.15.23:0/648472025 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff64c0a2350 0x7ff64c0a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:10.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.888+0000 7ff6577fe640 1 -- 172.21.15.23:0/648472025 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7ff65c0f09b0 con 0x7ff6601087c0 2024-03-24T10:40:10.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.888+0000 7ff665a5e640 1 --2- 172.21.15.23:0/648472025 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff64c0a2350 0x7ff64c0a4810 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff66010bc30 tx=0x7ff650005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:10.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:10.892+0000 7ff6577fe640 1 -- 172.21.15.23:0/648472025 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff65c0b9500 con 0x7ff6601087c0 2024-03-24T10:40:11.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.076+0000 7ff6684c3640 1 -- 172.21.15.23:0/648472025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff660118030 con 0x7ff6601087c0 2024-03-24T10:40:11.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.076+0000 7ff6577fe640 1 -- 172.21.15.23:0/648472025 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7ff65c0bcbc0 con 0x7ff6601087c0 2024-03-24T10:40:11.077 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-24T10:40:11.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.078+0000 7ff6684c3640 1 -- 172.21.15.23:0/648472025 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff64c0a2350 msgr2=0x7ff64c0a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.078+0000 7ff6684c3640 1 --2- 172.21.15.23:0/648472025 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff64c0a2350 0x7ff64c0a4810 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff66010bc30 tx=0x7ff650005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:11.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.078+0000 7ff6684c3640 1 -- 172.21.15.23:0/648472025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6601087c0 msgr2=0x7ff66019e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.078+0000 7ff6684c3640 1 --2- 172.21.15.23:0/648472025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6601087c0 0x7ff66019e430 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009230 tx=0x7ff65c02c650 comp rx=0 tx=0).stop 2024-03-24T10:40:11.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.078+0000 7ff6684c3640 1 -- 172.21.15.23:0/648472025 shutdown_connections 2024-03-24T10:40:11.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.078+0000 7ff6684c3640 1 --2- 172.21.15.23:0/648472025 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff64c0a2350 0x7ff64c0a4810 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.078+0000 7ff6684c3640 1 --2- 172.21.15.23:0/648472025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6601a2ca0 0x7ff6601a3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.078+0000 7ff6684c3640 1 --2- 172.21.15.23:0/648472025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff66010d0c0 0x7ff66019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.078+0000 7ff6684c3640 1 --2- 172.21.15.23:0/648472025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6601087c0 0x7ff66019e430 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.078+0000 7ff6684c3640 1 -- 172.21.15.23:0/648472025 >> 172.21.15.23:0/648472025 conn(0x7ff660077490 msgr2=0x7ff660079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:11.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.078+0000 7ff6684c3640 1 -- 172.21.15.23:0/648472025 shutdown_connections 2024-03-24T10:40:11.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.078+0000 7ff6684c3640 1 -- 172.21.15.23:0/648472025 wait complete. 2024-03-24T10:40:11.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:10 smithi023 ceph-mon[32466]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:40:11.090 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:10 smithi023 ceph-mon[32466]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:40:11.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:10 smithi023 ceph-mon[32466]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:40:11.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:10 smithi023 ceph-mon[32466]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:40:11.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:10 smithi023 ceph-mon[32466]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:40:11.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:10 smithi023 ceph-mon[32466]: Updating smithi023:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:40:11.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:10 smithi023 ceph-mon[32466]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:40:11.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:10 smithi023 ceph-mon[32466]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:40:11.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:10 smithi023 ceph-mon[32466]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-24T10:40:11.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:10 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:40:11.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:10 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:40:11.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:10 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:40:11.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:10 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:11.091 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:10 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1520188649' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:11.093 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:40:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.199+0000 7fe5b47b5640 1 -- 172.21.15.23:0/1119003354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5ac1005a0 msgr2=0x7fe5ac100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.199+0000 7fe5b47b5640 1 --2- 172.21.15.23:0/1119003354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5ac1005a0 0x7fe5ac100a20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0009900 tx=0x7fe5a002bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 -- 172.21.15.23:0/1119003354 shutdown_connections 2024-03-24T10:40:11.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 --2- 172.21.15.23:0/1119003354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5ac100f60 0x7fe5ac10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 --2- 172.21.15.23:0/1119003354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5ac1005a0 0x7fe5ac100a20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 --2- 172.21.15.23:0/1119003354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5ac106660 0x7fe5ac106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 -- 172.21.15.23:0/1119003354 >> 172.21.15.23:0/1119003354 conn(0x7fe5ac077710 msgr2=0x7fe5ac077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:11.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 -- 172.21.15.23:0/1119003354 shutdown_connections 2024-03-24T10:40:11.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 -- 172.21.15.23:0/1119003354 wait complete. 2024-03-24T10:40:11.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 Processor -- start 2024-03-24T10:40:11.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 -- start start 2024-03-24T10:40:11.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5ac1005a0 0x7fe5ac198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:11.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5ac100f60 0x7fe5ac1990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:11.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5ac106660 0x7fe5ac1a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:11.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac199940 con 0x7fe5ac106660 2024-03-24T10:40:11.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac1a4160 con 0x7fe5ac100f60 2024-03-24T10:40:11.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b47b5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac1a42a0 con 0x7fe5ac1005a0 2024-03-24T10:40:11.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b2d52640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5ac106660 0x7fe5ac1a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:11.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b2551640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5ac1005a0 0x7fe5ac198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:11.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b2d52640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5ac106660 0x7fe5ac1a3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33534/0 (socket says 172.21.15.23:33534) 2024-03-24T10:40:11.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b2d52640 1 -- 172.21.15.23:0/2658282537 learned_addr learned my addr 172.21.15.23:0/2658282537 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:11.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.200+0000 7fe5b1d50640 1 --2- 172.21.15.23:0/2658282537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5ac100f60 0x7fe5ac1990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:11.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.201+0000 7fe5b2d52640 1 -- 172.21.15.23:0/2658282537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5ac1005a0 msgr2=0x7fe5ac198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.201+0000 7fe5b2d52640 1 --2- 172.21.15.23:0/2658282537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5ac1005a0 0x7fe5ac198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.201+0000 7fe5b2d52640 1 -- 172.21.15.23:0/2658282537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5ac100f60 msgr2=0x7fe5ac1990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.201+0000 7fe5b2d52640 1 --2- 172.21.15.23:0/2658282537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5ac100f60 0x7fe5ac1990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.201+0000 7fe5b2d52640 1 -- 172.21.15.23:0/2658282537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5a0009560 con 0x7fe5ac106660 2024-03-24T10:40:11.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.201+0000 7fe5b2d52640 1 --2- 172.21.15.23:0/2658282537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5ac106660 0x7fe5ac1a3c20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00eb10 tx=0x7fe59c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:11.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.201+0000 7fe5ab7fe640 1 -- 172.21.15.23:0/2658282537 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59c004b20 con 0x7fe5ac106660 2024-03-24T10:40:11.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.201+0000 7fe5b47b5640 1 -- 172.21.15.23:0/2658282537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ac1a4500 con 0x7fe5ac106660 2024-03-24T10:40:11.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.201+0000 7fe5ab7fe640 1 -- 172.21.15.23:0/2658282537 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe59c004c80 con 0x7fe5ac106660 2024-03-24T10:40:11.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.201+0000 7fe5ab7fe640 1 -- 172.21.15.23:0/2658282537 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59c010650 con 0x7fe5ac106660 2024-03-24T10:40:11.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.201+0000 7fe5b47b5640 1 -- 172.21.15.23:0/2658282537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ac1a4990 con 0x7fe5ac106660 2024-03-24T10:40:11.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.202+0000 7fe5b47b5640 1 -- 172.21.15.23:0/2658282537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5ac068500 con 0x7fe5ac106660 2024-03-24T10:40:11.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.203+0000 7fe5ab7fe640 1 -- 172.21.15.23:0/2658282537 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7fe59c005020 con 0x7fe5ac106660 2024-03-24T10:40:11.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.203+0000 7fe5ab7fe640 1 --2- 172.21.15.23:0/2658282537 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe5940a2350 0x7fe5940a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:11.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.204+0000 7fe5b2551640 1 --2- 172.21.15.23:0/2658282537 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe5940a2350 0x7fe5940a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:11.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.204+0000 7fe5ab7fe640 1 -- 172.21.15.23:0/2658282537 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe59c014070 con 0x7fe5ac106660 2024-03-24T10:40:11.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.204+0000 7fe5b2551640 1 --2- 172.21.15.23:0/2658282537 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe5940a2350 0x7fe5940a4810 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009b50 tx=0x7fe5a4005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:11.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.208+0000 7fe5ab7fe640 1 -- 172.21.15.23:0/2658282537 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe59c097360 con 0x7fe5ac106660 2024-03-24T10:40:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:10 smithi117 ceph-mon[36368]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:40:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:10 smithi117 ceph-mon[36368]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:40:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:10 smithi117 ceph-mon[36368]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:40:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:10 smithi117 ceph-mon[36368]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:40:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:10 smithi117 ceph-mon[36368]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:40:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:10 smithi117 ceph-mon[36368]: Updating smithi023:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:40:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:10 smithi117 ceph-mon[36368]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:40:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:10 smithi117 ceph-mon[36368]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:40:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:10 smithi117 ceph-mon[36368]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-24T10:40:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:10 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:40:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:10 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:40:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:10 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:40:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:10 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:10 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1520188649' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:11.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:10 smithi116 ceph-mon[36248]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:40:11.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:10 smithi116 ceph-mon[36248]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:40:11.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:10 smithi116 ceph-mon[36248]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-24T10:40:11.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:10 smithi116 ceph-mon[36248]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T10:40:11.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:10 smithi116 ceph-mon[36248]: Updating smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:40:11.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:10 smithi116 ceph-mon[36248]: Updating smithi023:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:40:11.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:10 smithi116 ceph-mon[36248]: Updating smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T10:40:11.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:10 smithi116 ceph-mon[36248]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:40:11.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:10 smithi116 ceph-mon[36248]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-24T10:40:11.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:10 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:40:11.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:10 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:40:11.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:10 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:40:11.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:10 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:40:11.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:10 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1520188649' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:11.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.393+0000 7fe5b47b5640 1 -- 172.21.15.23:0/2658282537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe5ac108a50 con 0x7fe5ac106660 2024-03-24T10:40:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.393+0000 7fe5ab7fe640 1 -- 172.21.15.23:0/2658282537 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fe59c09aa20 con 0x7fe5ac106660 2024-03-24T10:40:11.394 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-24T10:40:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.395+0000 7fe5b47b5640 1 -- 172.21.15.23:0/2658282537 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe5940a2350 msgr2=0x7fe5940a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.395+0000 7fe5b47b5640 1 --2- 172.21.15.23:0/2658282537 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe5940a2350 0x7fe5940a4810 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009b50 tx=0x7fe5a4005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.395+0000 7fe5b47b5640 1 -- 172.21.15.23:0/2658282537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5ac106660 msgr2=0x7fe5ac1a3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.395+0000 7fe5b47b5640 1 --2- 172.21.15.23:0/2658282537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5ac106660 0x7fe5ac1a3c20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00eb10 tx=0x7fe59c00c2d0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.395+0000 7fe5b47b5640 1 -- 172.21.15.23:0/2658282537 shutdown_connections 2024-03-24T10:40:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.395+0000 7fe5b47b5640 1 --2- 172.21.15.23:0/2658282537 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe5940a2350 0x7fe5940a4810 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.395+0000 7fe5b47b5640 1 --2- 172.21.15.23:0/2658282537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5ac106660 0x7fe5ac1a3c20 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.395+0000 7fe5b47b5640 1 --2- 172.21.15.23:0/2658282537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5ac100f60 0x7fe5ac1990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.395+0000 7fe5b47b5640 1 --2- 172.21.15.23:0/2658282537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5ac1005a0 0x7fe5ac198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.395+0000 7fe5b47b5640 1 -- 172.21.15.23:0/2658282537 >> 172.21.15.23:0/2658282537 conn(0x7fe5ac077710 msgr2=0x7fe5ac0ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:11.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.395+0000 7fe5b47b5640 1 -- 172.21.15.23:0/2658282537 shutdown_connections 2024-03-24T10:40:11.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.395+0000 7fe5b47b5640 1 -- 172.21.15.23:0/2658282537 wait complete. 2024-03-24T10:40:11.406 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:40:11.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.512+0000 7f65bc24e640 1 -- 172.21.15.23:0/198055134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b4106690 msgr2=0x7f65b4106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.512+0000 7f65bc24e640 1 --2- 172.21.15.23:0/198055134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b4106690 0x7f65b4106a70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f65b000bac0 tx=0x7f65b002d990 comp rx=0 tx=0).stop 2024-03-24T10:40:11.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.512+0000 7f65bc24e640 1 -- 172.21.15.23:0/198055134 shutdown_connections 2024-03-24T10:40:11.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.512+0000 7f65bc24e640 1 --2- 172.21.15.23:0/198055134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65b4101200 0x7f65b410d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.512+0000 7f65bc24e640 1 --2- 172.21.15.23:0/198055134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65b4100860 0x7f65b4100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.512+0000 7f65bc24e640 1 --2- 172.21.15.23:0/198055134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b4106690 0x7f65b4106a70 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.512+0000 7f65bc24e640 1 -- 172.21.15.23:0/198055134 >> 172.21.15.23:0/198055134 conn(0x7f65b40773b0 msgr2=0x7f65b40777c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:11.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.512+0000 7f65bc24e640 1 -- 172.21.15.23:0/198055134 shutdown_connections 2024-03-24T10:40:11.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.512+0000 7f65bc24e640 1 -- 172.21.15.23:0/198055134 wait complete. 2024-03-24T10:40:11.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65bc24e640 1 Processor -- start 2024-03-24T10:40:11.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65bc24e640 1 -- start start 2024-03-24T10:40:11.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65bc24e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65b4100860 0x7f65b419b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:11.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65bc24e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65b4101200 0x7f65b419bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:11.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65bc24e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b4106690 0x7f65b41959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:11.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65bc24e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f65b4196000 con 0x7f65b4106690 2024-03-24T10:40:11.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65bc24e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f65b4196170 con 0x7f65b4100860 2024-03-24T10:40:11.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65bc24e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f65b4196330 con 0x7f65b4101200 2024-03-24T10:40:11.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65ba7eb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b4106690 0x7f65b41959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:11.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65ba7eb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b4106690 0x7f65b41959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33560/0 (socket says 172.21.15.23:33560) 2024-03-24T10:40:11.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65ba7eb640 1 -- 172.21.15.23:0/3923967677 learned_addr learned my addr 172.21.15.23:0/3923967677 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:11.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65b97e9640 1 --2- 172.21.15.23:0/3923967677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65b4101200 0x7f65b419bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:11.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65b9fea640 1 --2- 172.21.15.23:0/3923967677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65b4100860 0x7f65b419b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:11.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65ba7eb640 1 -- 172.21.15.23:0/3923967677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65b4101200 msgr2=0x7f65b419bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65ba7eb640 1 --2- 172.21.15.23:0/3923967677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65b4101200 0x7f65b419bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.513+0000 7f65ba7eb640 1 -- 172.21.15.23:0/3923967677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65b4100860 msgr2=0x7f65b419b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.514+0000 7f65ba7eb640 1 --2- 172.21.15.23:0/3923967677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65b4100860 0x7f65b419b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.514+0000 7f65ba7eb640 1 -- 172.21.15.23:0/3923967677 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b000b770 con 0x7f65b4106690 2024-03-24T10:40:11.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.514+0000 7f65ba7eb640 1 --2- 172.21.15.23:0/3923967677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b4106690 0x7f65b41959b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00eb10 tx=0x7f65ac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:11.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.514+0000 7f65aaffd640 1 -- 172.21.15.23:0/3923967677 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac004b20 con 0x7f65b4106690 2024-03-24T10:40:11.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.514+0000 7f65aaffd640 1 -- 172.21.15.23:0/3923967677 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65ac004c80 con 0x7f65b4106690 2024-03-24T10:40:11.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.514+0000 7f65bc24e640 1 -- 172.21.15.23:0/3923967677 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b4196610 con 0x7f65b4106690 2024-03-24T10:40:11.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.514+0000 7f65aaffd640 1 -- 172.21.15.23:0/3923967677 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac010650 con 0x7f65b4106690 2024-03-24T10:40:11.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.514+0000 7f65bc24e640 1 -- 172.21.15.23:0/3923967677 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b41a28a0 con 0x7f65b4106690 2024-03-24T10:40:11.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.515+0000 7f65bc24e640 1 -- 172.21.15.23:0/3923967677 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65b4101cb0 con 0x7f65b4106690 2024-03-24T10:40:11.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.516+0000 7f65aaffd640 1 -- 172.21.15.23:0/3923967677 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7f65ac005020 con 0x7f65b4106690 2024-03-24T10:40:11.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.516+0000 7f65aaffd640 1 --2- 172.21.15.23:0/3923967677 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f65a00a23d0 0x7f65a00a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:11.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.517+0000 7f65b9fea640 1 --2- 172.21.15.23:0/3923967677 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f65a00a23d0 0x7f65a00a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:11.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.517+0000 7f65aaffd640 1 -- 172.21.15.23:0/3923967677 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f65ac014070 con 0x7f65b4106690 2024-03-24T10:40:11.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.517+0000 7f65b9fea640 1 --2- 172.21.15.23:0/3923967677 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f65a00a23d0 0x7f65a00a4890 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f65b000ba90 tx=0x7f65b0005fd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:11.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.521+0000 7f65aaffd640 1 -- 172.21.15.23:0/3923967677 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f65ac097280 con 0x7f65b4106690 2024-03-24T10:40:11.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.704+0000 7f65bc24e640 1 -- 172.21.15.23:0/3923967677 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f65b41089f0 con 0x7f65b4106690 2024-03-24T10:40:11.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.704+0000 7f65aaffd640 1 -- 172.21.15.23:0/3923967677 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f65ac09a940 con 0x7f65b4106690 2024-03-24T10:40:11.705 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-24T10:40:11.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.706+0000 7f65bc24e640 1 -- 172.21.15.23:0/3923967677 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f65a00a23d0 msgr2=0x7f65a00a4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.706+0000 7f65bc24e640 1 --2- 172.21.15.23:0/3923967677 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f65a00a23d0 0x7f65a00a4890 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f65b000ba90 tx=0x7f65b0005fd0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.706+0000 7f65bc24e640 1 -- 172.21.15.23:0/3923967677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b4106690 msgr2=0x7f65b41959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.706+0000 7f65bc24e640 1 --2- 172.21.15.23:0/3923967677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b4106690 0x7f65b41959b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00eb10 tx=0x7f65ac00c2d0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.707+0000 7f65bc24e640 1 -- 172.21.15.23:0/3923967677 shutdown_connections 2024-03-24T10:40:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.707+0000 7f65bc24e640 1 --2- 172.21.15.23:0/3923967677 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f65a00a23d0 0x7f65a00a4890 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.707+0000 7f65bc24e640 1 --2- 172.21.15.23:0/3923967677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b4106690 0x7f65b41959b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.707+0000 7f65bc24e640 1 --2- 172.21.15.23:0/3923967677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65b4101200 0x7f65b419bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.707+0000 7f65bc24e640 1 --2- 172.21.15.23:0/3923967677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65b4100860 0x7f65b419b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.707+0000 7f65bc24e640 1 -- 172.21.15.23:0/3923967677 >> 172.21.15.23:0/3923967677 conn(0x7f65b40773b0 msgr2=0x7f65b40ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.707+0000 7f65bc24e640 1 -- 172.21.15.23:0/3923967677 shutdown_connections 2024-03-24T10:40:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.707+0000 7f65bc24e640 1 -- 172.21.15.23:0/3923967677 wait complete. 2024-03-24T10:40:11.718 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:40:11.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 -- 172.21.15.23:0/2203507826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64b4106690 msgr2=0x7f64b4106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 --2- 172.21.15.23:0/2203507826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64b4106690 0x7f64b4106a70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f64ac009ca0 tx=0x7f64ac02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 -- 172.21.15.23:0/2203507826 shutdown_connections 2024-03-24T10:40:11.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 --2- 172.21.15.23:0/2203507826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64b4100f90 0x7f64b410d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 --2- 172.21.15.23:0/2203507826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64b41005d0 0x7f64b4100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 --2- 172.21.15.23:0/2203507826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64b4106690 0x7f64b4106a70 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 -- 172.21.15.23:0/2203507826 >> 172.21.15.23:0/2203507826 conn(0x7f64b4076990 msgr2=0x7f64b4076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:11.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 -- 172.21.15.23:0/2203507826 shutdown_connections 2024-03-24T10:40:11.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 -- 172.21.15.23:0/2203507826 wait complete. 2024-03-24T10:40:11.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 Processor -- start 2024-03-24T10:40:11.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 -- start start 2024-03-24T10:40:11.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64b41005d0 0x7f64b4199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:11.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64b4100f90 0x7f64b419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:11.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64b4106690 0x7f64b419e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:11.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f64b419ed40 con 0x7f64b4106690 2024-03-24T10:40:11.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f64b419eeb0 con 0x7f64b4100f90 2024-03-24T10:40:11.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.821+0000 7f64bad43640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f64b419f070 con 0x7f64b41005d0 2024-03-24T10:40:11.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64b92e0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64b4106690 0x7f64b419e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:11.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64b92e0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64b4106690 0x7f64b419e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33578/0 (socket says 172.21.15.23:33578) 2024-03-24T10:40:11.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64b8adf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64b41005d0 0x7f64b4199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:11.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64b92e0640 1 -- 172.21.15.23:0/1148409755 learned_addr learned my addr 172.21.15.23:0/1148409755 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:11.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64b3fff640 1 --2- 172.21.15.23:0/1148409755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64b4100f90 0x7f64b419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:11.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64b92e0640 1 -- 172.21.15.23:0/1148409755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64b41005d0 msgr2=0x7f64b4199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64b92e0640 1 --2- 172.21.15.23:0/1148409755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64b41005d0 0x7f64b4199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64b92e0640 1 -- 172.21.15.23:0/1148409755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64b4100f90 msgr2=0x7f64b419a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:11.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64b92e0640 1 --2- 172.21.15.23:0/1148409755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64b4100f90 0x7f64b419a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:11.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64b92e0640 1 -- 172.21.15.23:0/1148409755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ac009900 con 0x7f64b4106690 2024-03-24T10:40:11.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64b92e0640 1 --2- 172.21.15.23:0/1148409755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64b4106690 0x7f64b419e740 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f64a400e970 tx=0x7f64a400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:11.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64b1ffb640 1 -- 172.21.15.23:0/1148409755 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4009720 con 0x7f64b4106690 2024-03-24T10:40:11.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64b1ffb640 1 -- 172.21.15.23:0/1148409755 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64a4005500 con 0x7f64b4106690 2024-03-24T10:40:11.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64bad43640 1 -- 172.21.15.23:0/1148409755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b419f350 con 0x7f64b4106690 2024-03-24T10:40:11.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64b1ffb640 1 -- 172.21.15.23:0/1148409755 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4010430 con 0x7f64b4106690 2024-03-24T10:40:11.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.822+0000 7f64bad43640 1 -- 172.21.15.23:0/1148409755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b41a6bd0 con 0x7f64b4106690 2024-03-24T10:40:11.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.824+0000 7f64bad43640 1 -- 172.21.15.23:0/1148409755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64b4101cf0 con 0x7f64b4106690 2024-03-24T10:40:11.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.824+0000 7f64b1ffb640 1 -- 172.21.15.23:0/1148409755 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7f64a4004ff0 con 0x7f64b4106690 2024-03-24T10:40:11.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.825+0000 7f64b1ffb640 1 --2- 172.21.15.23:0/1148409755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f64a00a2350 0x7f64a00a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:11.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.825+0000 7f64b8adf640 1 --2- 172.21.15.23:0/1148409755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f64a00a2350 0x7f64a00a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:11.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.825+0000 7f64b1ffb640 1 -- 172.21.15.23:0/1148409755 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f64a4014070 con 0x7f64b4106690 2024-03-24T10:40:11.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.825+0000 7f64b8adf640 1 --2- 172.21.15.23:0/1148409755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f64a00a2350 0x7f64a00a4810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f64ac004af0 tx=0x7f64ac005c50 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:11.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:11.829+0000 7f64b1ffb640 1 -- 172.21.15.23:0/1148409755 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f64a40970a0 con 0x7f64b4106690 2024-03-24T10:40:12.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.049+0000 7f64bad43640 1 -- 172.21.15.23:0/1148409755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64b4068950 con 0x7f64b4106690 2024-03-24T10:40:12.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.050+0000 7f64b1ffb640 1 -- 172.21.15.23:0/1148409755 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11866 (secure 0 0 0) 0x7f64a409a760 con 0x7f64b4106690 2024-03-24T10:40:12.051 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:40:12.051 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:40:03.697229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24539,24503,24505]},"id":1}]} 2024-03-24T10:40:12.051 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 23 2024-03-24T10:40:12.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.052+0000 7f64bad43640 1 -- 172.21.15.23:0/1148409755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f64a00a2350 msgr2=0x7f64a00a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.052+0000 7f64bad43640 1 --2- 172.21.15.23:0/1148409755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f64a00a2350 0x7f64a00a4810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f64ac004af0 tx=0x7f64ac005c50 comp rx=0 tx=0).stop 2024-03-24T10:40:12.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.052+0000 7f64bad43640 1 -- 172.21.15.23:0/1148409755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64b4106690 msgr2=0x7f64b419e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.052+0000 7f64bad43640 1 --2- 172.21.15.23:0/1148409755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64b4106690 0x7f64b419e740 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f64a400e970 tx=0x7f64a400ed70 comp rx=0 tx=0).stop 2024-03-24T10:40:12.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.052+0000 7f64bad43640 1 -- 172.21.15.23:0/1148409755 shutdown_connections 2024-03-24T10:40:12.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.052+0000 7f64bad43640 1 --2- 172.21.15.23:0/1148409755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f64a00a2350 0x7f64a00a4810 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.052+0000 7f64bad43640 1 --2- 172.21.15.23:0/1148409755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64b4106690 0x7f64b419e740 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.052+0000 7f64bad43640 1 --2- 172.21.15.23:0/1148409755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64b4100f90 0x7f64b419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.052+0000 7f64bad43640 1 --2- 172.21.15.23:0/1148409755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64b41005d0 0x7f64b4199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.052+0000 7f64bad43640 1 -- 172.21.15.23:0/1148409755 >> 172.21.15.23:0/1148409755 conn(0x7f64b4076990 msgr2=0x7f64b4069d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.052+0000 7f64bad43640 1 -- 172.21.15.23:0/1148409755 shutdown_connections 2024-03-24T10:40:12.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.052+0000 7f64bad43640 1 -- 172.21.15.23:0/1148409755 wait complete. 2024-03-24T10:40:12.062 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:11 smithi023 ceph-mon[32466]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:40:12.062 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:11 smithi023 ceph-mon[32466]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:40:12.062 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:11 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/648472025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:40:12.063 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:11 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2658282537' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:40:12.063 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:11 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3923967677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:40:12.064 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:40:12.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.167+0000 7fad17c15640 1 -- 172.21.15.23:0/49677752 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad10100590 msgr2=0x7fad1010e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.167+0000 7fad17c15640 1 --2- 172.21.15.23:0/49677752 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad10100590 0x7fad1010e590 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fad08009cd0 tx=0x7fad0802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.167+0000 7fad17c15640 1 -- 172.21.15.23:0/49677752 shutdown_connections 2024-03-24T10:40:12.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.167+0000 7fad17c15640 1 --2- 172.21.15.23:0/49677752 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad10100590 0x7fad1010e590 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.167+0000 7fad17c15640 1 --2- 172.21.15.23:0/49677752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad100ffc10 0x7fad10100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.167+0000 7fad17c15640 1 --2- 172.21.15.23:0/49677752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad101076d0 0x7fad10107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.167+0000 7fad17c15640 1 -- 172.21.15.23:0/49677752 >> 172.21.15.23:0/49677752 conn(0x7fad10076ba0 msgr2=0x7fad100fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.167+0000 7fad17c15640 1 -- 172.21.15.23:0/49677752 shutdown_connections 2024-03-24T10:40:12.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.167+0000 7fad17c15640 1 -- 172.21.15.23:0/49677752 wait complete. 2024-03-24T10:40:12.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.167+0000 7fad17c15640 1 Processor -- start 2024-03-24T10:40:12.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.167+0000 7fad17c15640 1 -- start start 2024-03-24T10:40:12.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad17c15640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad100ffc10 0x7fad10197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad17c15640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad10100590 0x7fad10198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad17c15640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad101076d0 0x7fad1019c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad17c15640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fad1019cbf0 con 0x7fad100ffc10 2024-03-24T10:40:12.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad17c15640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad1019cd60 con 0x7fad101076d0 2024-03-24T10:40:12.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad17c15640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fad1019cf20 con 0x7fad10100590 2024-03-24T10:40:12.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad159b1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad100ffc10 0x7fad10197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad159b1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad100ffc10 0x7fad10197d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33596/0 (socket says 172.21.15.23:33596) 2024-03-24T10:40:12.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad159b1640 1 -- 172.21.15.23:0/2111010293 learned_addr learned my addr 172.21.15.23:0/2111010293 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:12.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad151b0640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad10100590 0x7fad10198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad161b2640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad101076d0 0x7fad1019c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad159b1640 1 -- 172.21.15.23:0/2111010293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad10100590 msgr2=0x7fad10198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad159b1640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad10100590 0x7fad10198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad159b1640 1 -- 172.21.15.23:0/2111010293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad101076d0 msgr2=0x7fad1019c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad159b1640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad101076d0 0x7fad1019c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad159b1640 1 -- 172.21.15.23:0/2111010293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad08009900 con 0x7fad100ffc10 2024-03-24T10:40:12.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad161b2640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad101076d0 0x7fad1019c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:40:12.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad151b0640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad10100590 0x7fad10198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:40:12.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad159b1640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad100ffc10 0x7fad10197d30 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fad0000e9f0 tx=0x7fad0000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad06ffd640 1 -- 172.21.15.23:0/2111010293 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad00009720 con 0x7fad100ffc10 2024-03-24T10:40:12.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad06ffd640 1 -- 172.21.15.23:0/2111010293 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad0000cd80 con 0x7fad100ffc10 2024-03-24T10:40:12.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad06ffd640 1 -- 172.21.15.23:0/2111010293 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad00010690 con 0x7fad100ffc10 2024-03-24T10:40:12.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad17c15640 1 -- 172.21.15.23:0/2111010293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad1019d200 con 0x7fad100ffc10 2024-03-24T10:40:12.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.168+0000 7fad17c15640 1 -- 172.21.15.23:0/2111010293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad101a4a80 con 0x7fad100ffc10 2024-03-24T10:40:12.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.169+0000 7fad17c15640 1 -- 172.21.15.23:0/2111010293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad10068950 con 0x7fad100ffc10 2024-03-24T10:40:12.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.170+0000 7fad06ffd640 1 -- 172.21.15.23:0/2111010293 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7fad000046c0 con 0x7fad100ffc10 2024-03-24T10:40:12.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.171+0000 7fad06ffd640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7facfc0a2350 0x7facfc0a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.171+0000 7fad151b0640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7facfc0a2350 0x7facfc0a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.171+0000 7fad06ffd640 1 -- 172.21.15.23:0/2111010293 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fad00014070 con 0x7fad100ffc10 2024-03-24T10:40:12.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.171+0000 7fad151b0640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7facfc0a2350 0x7facfc0a4810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fad100fdb60 tx=0x7fad0c005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.175+0000 7fad06ffd640 1 -- 172.21.15.23:0/2111010293 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fad000971c0 con 0x7fad100ffc10 2024-03-24T10:40:12.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:11 smithi117 ceph-mon[36368]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:40:12.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:11 smithi117 ceph-mon[36368]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:40:12.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:11 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/648472025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:40:12.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:11 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2658282537' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:40:12.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:11 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3923967677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:40:12.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:11 smithi116 ceph-mon[36248]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:40:12.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:11 smithi116 ceph-mon[36248]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T10:40:12.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:11 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/648472025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:40:12.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:11 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2658282537' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:40:12.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:11 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3923967677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:40:12.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.358+0000 7fad17c15640 1 -- 172.21.15.23:0/2111010293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad10109be0 con 0x7fad100ffc10 2024-03-24T10:40:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.359+0000 7fad06ffd640 1 -- 172.21.15.23:0/2111010293 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+23079 (secure 0 0 0) 0x7fad0009a880 con 0x7fad100ffc10 2024-03-24T10:40:12.360 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:40:12.360 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:40:05.291943+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2350021632":"2024-03-25T10:40:05.291912+0000","172.21.15.23:6800/2350021632":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3614005933":"2024-03-25T10:40:05.291912+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/256452041":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/352422381":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/725846898":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:40:12.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.361+0000 7fad17c15640 1 -- 172.21.15.23:0/2111010293 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7facfc0a2350 msgr2=0x7facfc0a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.361+0000 7fad17c15640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7facfc0a2350 0x7facfc0a4810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fad100fdb60 tx=0x7fad0c005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:12.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.361+0000 7fad17c15640 1 -- 172.21.15.23:0/2111010293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad100ffc10 msgr2=0x7fad10197d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.361+0000 7fad17c15640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad100ffc10 0x7fad10197d30 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fad0000e9f0 tx=0x7fad0000edf0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.362+0000 7fad17c15640 1 -- 172.21.15.23:0/2111010293 shutdown_connections 2024-03-24T10:40:12.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.362+0000 7fad17c15640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7facfc0a2350 0x7facfc0a4810 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.362+0000 7fad17c15640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad101076d0 0x7fad1019c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.362+0000 7fad17c15640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad10100590 0x7fad10198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.362+0000 7fad17c15640 1 --2- 172.21.15.23:0/2111010293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad100ffc10 0x7fad10197d30 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.362+0000 7fad17c15640 1 -- 172.21.15.23:0/2111010293 >> 172.21.15.23:0/2111010293 conn(0x7fad10076ba0 msgr2=0x7fad100fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.362+0000 7fad17c15640 1 -- 172.21.15.23:0/2111010293 shutdown_connections 2024-03-24T10:40:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.362+0000 7fad17c15640 1 -- 172.21.15.23:0/2111010293 wait complete. 2024-03-24T10:40:12.373 DEBUG:tasks.fwd_scrub:Yielding 2024-03-24T10:40:12.374 INFO:teuthology.run_tasks:Running task exec... 2024-03-24T10:40:12.384 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T10:40:12.384 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T10:40:12.384 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-24T10:40:12.384 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-24T10:40:12.385 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:40:12.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:40:12.385 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:40:12.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/2233156994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c14068d90 msgr2=0x7f0c140691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7f0c1a5c9640 1 --2- 172.21.15.23:0/2233156994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c14068d90 0x7f0c140691f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0c08009930 tx=0x7f0c0802bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:12.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/2233156994 shutdown_connections 2024-03-24T10:40:12.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7f0c1a5c9640 1 --2- 172.21.15.23:0/2233156994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c14069730 0x7f0c1410e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7f0c1a5c9640 1 --2- 172.21.15.23:0/2233156994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c14068d90 0x7f0c140691f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7f0c1a5c9640 1 --2- 172.21.15.23:0/2233156994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c14107780 0x7f0c14107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/2233156994 >> 172.21.15.23:0/2233156994 conn(0x7f0c14077460 msgr2=0x7f0c14077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/2233156994 shutdown_connections 2024-03-24T10:40:12.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/2233156994 wait complete. 2024-03-24T10:40:12.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7f0c1a5c9640 1 Processor -- start 2024-03-24T10:40:12.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7feac1901640 1 -- 172.21.15.23:0/558875719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feabc1098c0 msgr2=0x7feabc109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7feac1901640 1 --2- 172.21.15.23:0/558875719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feabc1098c0 0x7feabc109ca0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7feaac009900 tx=0x7feaac02bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:12.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7f0c1a5c9640 1 -- start start 2024-03-24T10:40:12.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7feac1901640 1 -- 172.21.15.23:0/558875719 shutdown_connections 2024-03-24T10:40:12.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7feac1901640 1 --2- 172.21.15.23:0/558875719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc104270 0x7feabc110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7feac1901640 1 --2- 172.21.15.23:0/558875719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc1038b0 0x7feabc103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7feac1901640 1 --2- 172.21.15.23:0/558875719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feabc1098c0 0x7feabc109ca0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7feac1901640 1 -- 172.21.15.23:0/558875719 >> 172.21.15.23:0/558875719 conn(0x7feabc0ff380 msgr2=0x7feabc1017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7feac1901640 1 -- 172.21.15.23:0/558875719 shutdown_connections 2024-03-24T10:40:12.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.490+0000 7feac1901640 1 -- 172.21.15.23:0/558875719 wait complete. 2024-03-24T10:40:12.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c1a5c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c14068d90 0x7f0c1419d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c1a5c9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c14069730 0x7f0c1419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c1a5c9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c14107780 0x7f0c14197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c1a5c9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0c141980e0 con 0x7f0c14107780 2024-03-24T10:40:12.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c1a5c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0c14198250 con 0x7f0c14068d90 2024-03-24T10:40:12.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c1a5c9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0c14198410 con 0x7f0c14069730 2024-03-24T10:40:12.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c13fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c14068d90 0x7f0c1419d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c18b66640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c14107780 0x7f0c14197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c18b66640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c14107780 0x7f0c14197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33608/0 (socket says 172.21.15.23:33608) 2024-03-24T10:40:12.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c18b66640 1 -- 172.21.15.23:0/1496035716 learned_addr learned my addr 172.21.15.23:0/1496035716 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:12.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c13fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c14068d90 0x7f0c1419d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:57472/0 (socket says 172.21.15.23:57472) 2024-03-24T10:40:12.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c137fe640 1 --2- 172.21.15.23:0/1496035716 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c14069730 0x7f0c1419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7feac1901640 1 Processor -- start 2024-03-24T10:40:12.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c18b66640 1 -- 172.21.15.23:0/1496035716 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c14069730 msgr2=0x7f0c1419de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c18b66640 1 --2- 172.21.15.23:0/1496035716 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c14069730 0x7f0c1419de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c18b66640 1 -- 172.21.15.23:0/1496035716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c14068d90 msgr2=0x7f0c1419d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c18b66640 1 --2- 172.21.15.23:0/1496035716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c14068d90 0x7f0c1419d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c18b66640 1 -- 172.21.15.23:0/1496035716 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0c009840 con 0x7f0c14107780 2024-03-24T10:40:12.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c13fff640 1 --2- 172.21.15.23:0/1496035716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c14068d90 0x7f0c1419d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:12.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7feac1901640 1 -- start start 2024-03-24T10:40:12.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.491+0000 7f0c18b66640 1 --2- 172.21.15.23:0/1496035716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c14107780 0x7f0c14197a90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c00ec70 tx=0x7f0c0c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feac1901640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc1038b0 0x7feabc199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feac1901640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feabc104270 0x7feabc19a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7f0c117fa640 1 -- 172.21.15.23:0/1496035716 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c003b70 con 0x7f0c14107780 2024-03-24T10:40:12.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feac1901640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc1098c0 0x7feabc19e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feac1901640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7feabc19ede0 con 0x7feabc1098c0 2024-03-24T10:40:12.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/1496035716 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c08009560 con 0x7f0c14107780 2024-03-24T10:40:12.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feac1901640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feabc19ef50 con 0x7feabc104270 2024-03-24T10:40:12.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feac1901640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feabc19f110 con 0x7feabc1038b0 2024-03-24T10:40:12.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/1496035716 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c141a47a0 con 0x7f0c14107780 2024-03-24T10:40:12.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7f0c117fa640 1 -- 172.21.15.23:0/1496035716 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c0c00ee80 con 0x7f0c14107780 2024-03-24T10:40:12.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7f0c117fa640 1 -- 172.21.15.23:0/1496035716 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c010810 con 0x7f0c14107780 2024-03-24T10:40:12.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feaba7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feabc104270 0x7feabc19a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feabaffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc1038b0 0x7feabc199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feabb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc1098c0 0x7feabc19e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feabaffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc1038b0 0x7feabc199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57204/0 (socket says 172.21.15.23:57204) 2024-03-24T10:40:12.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feaba7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feabc104270 0x7feabc19a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:57484/0 (socket says 172.21.15.23:57484) 2024-03-24T10:40:12.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feabaffd640 1 -- 172.21.15.23:0/439232536 learned_addr learned my addr 172.21.15.23:0/439232536 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:12.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feabaffd640 1 -- 172.21.15.23:0/439232536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feabc104270 msgr2=0x7feabc19a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feabaffd640 1 --2- 172.21.15.23:0/439232536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feabc104270 0x7feabc19a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feabaffd640 1 -- 172.21.15.23:0/439232536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc1098c0 msgr2=0x7feabc19e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feabaffd640 1 --2- 172.21.15.23:0/439232536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc1098c0 0x7feabc19e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feabaffd640 1 -- 172.21.15.23:0/439232536 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab4009840 con 0x7feabc1038b0 2024-03-24T10:40:12.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feabaffd640 1 --2- 172.21.15.23:0/439232536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc1038b0 0x7feabc199f70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7feaac005940 tx=0x7feaac02c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.492+0000 7feac08ff640 1 -- 172.21.15.23:0/439232536 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac0042d0 con 0x7feabc1038b0 2024-03-24T10:40:12.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.493+0000 7feac1901640 1 -- 172.21.15.23:0/439232536 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaac009560 con 0x7feabc1038b0 2024-03-24T10:40:12.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.493+0000 7feac08ff640 1 -- 172.21.15.23:0/439232536 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaac004430 con 0x7feabc1038b0 2024-03-24T10:40:12.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.493+0000 7feac08ff640 1 -- 172.21.15.23:0/439232536 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac003b50 con 0x7feabc1038b0 2024-03-24T10:40:12.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.493+0000 7feac1901640 1 -- 172.21.15.23:0/439232536 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feabc1a6a90 con 0x7feabc1038b0 2024-03-24T10:40:12.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.493+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/1496035716 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c1404f470 con 0x7f0c14107780 2024-03-24T10:40:12.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.494+0000 7f0c117fa640 1 -- 172.21.15.23:0/1496035716 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7f0c0c004b40 con 0x7f0c14107780 2024-03-24T10:40:12.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.495+0000 7feac08ff640 1 -- 172.21.15.23:0/439232536 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7feaac003cb0 con 0x7feabc1038b0 2024-03-24T10:40:12.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.495+0000 7f0c117fa640 1 --2- 172.21.15.23:0/1496035716 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0bfc0a2470 0x7f0bfc0a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.495+0000 7f0c13fff640 1 --2- 172.21.15.23:0/1496035716 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0bfc0a2470 0x7f0bfc0a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.495+0000 7feac08ff640 1 --2- 172.21.15.23:0/439232536 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feaa40a2350 0x7feaa40a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.495+0000 7f0c117fa640 1 -- 172.21.15.23:0/1496035716 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0c0c014070 con 0x7f0c14107780 2024-03-24T10:40:12.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.495+0000 7feac45b6740 1 Processor -- start 2024-03-24T10:40:12.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.495+0000 7feaba7fc640 1 --2- 172.21.15.23:0/439232536 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feaa40a2350 0x7feaa40a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.495+0000 7feac45b6740 1 -- start start 2024-03-24T10:40:12.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.495+0000 7feac08ff640 1 -- 172.21.15.23:0/439232536 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7feaac0f0350 con 0x7feabc1038b0 2024-03-24T10:40:12.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feac45b6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffa8e140 0x55daffa8e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feac45b6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffa9ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feac45b6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffaa0470 0x55daffaa2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feac45b6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55daff9fc530 con 0x55daffaa0470 2024-03-24T10:40:12.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feac45b6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55daff9f9a60 con 0x55daffa8e140 2024-03-24T10:40:12.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feac45b6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55daff9f5240 con 0x55daffa8eaf0 2024-03-24T10:40:12.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feabb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffaa0470 0x55daffaa2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7f0c13fff640 1 --2- 172.21.15.23:0/1496035716 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0bfc0a2470 0x7f0bfc0a4930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0c14198bd0 tx=0x7f0c04005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feabb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffaa0470 0x55daffaa2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33632/0 (socket says 172.21.15.23:33632) 2024-03-24T10:40:12.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feabb7fe640 1 -- 172.21.15.23:0/3283692025 learned_addr learned my addr 172.21.15.23:0/3283692025 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:12.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feabaffd640 1 --2- 172.21.15.23:0/3283692025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffa8e140 0x55daffa8e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feaba7fc640 1 --2- 172.21.15.23:0/439232536 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feaa40a2350 0x7feaa40a4810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feabc108f40 tx=0x7feab0005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feaba7fc640 1 --2- 172.21.15.23:0/3283692025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffa9ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feabaffd640 1 -- 172.21.15.23:0/3283692025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 msgr2=0x55daffa9ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feabaffd640 1 --2- 172.21.15.23:0/3283692025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffa9ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feabaffd640 1 -- 172.21.15.23:0/3283692025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffaa0470 msgr2=0x55daffaa2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feabaffd640 1 --2- 172.21.15.23:0/3283692025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffaa0470 0x55daffaa2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feabaffd640 1 -- 172.21.15.23:0/3283692025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daff9f38c0 con 0x55daffa8e140 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feabb7fe640 1 --2- 172.21.15.23:0/3283692025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffaa0470 0x55daffaa2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.496+0000 7feabaffd640 1 --2- 172.21.15.23:0/3283692025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffa8e140 0x55daffa8e520 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feaac0eda30 tx=0x7feaac0bc740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=701f5f3e76d03ed2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.497+0000 7feaa1ffb640 1 -- 172.21.15.23:0/3283692025 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac0bcc00 con 0x55daffa8e140 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.497+0000 7feaa1ffb640 1 -- 172.21.15.23:0/3283692025 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaac0a7050 con 0x55daffa8e140 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.497+0000 7feaa1ffb640 1 -- 172.21.15.23:0/3283692025 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac0c4f10 con 0x55daffa8e140 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.497+0000 7feac45b6740 1 -- 172.21.15.23:0/3283692025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffa8e140 msgr2=0x55daffa8e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.497+0000 7feac45b6740 1 --2- 172.21.15.23:0/3283692025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffa8e140 0x55daffa8e520 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feaac0eda30 tx=0x7feaac0bc740 comp rx=0 tx=0).stop 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.497+0000 7feac45b6740 1 -- 172.21.15.23:0/3283692025 shutdown_connections 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.497+0000 7feac45b6740 1 --2- 172.21.15.23:0/3283692025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffaa0470 0x55daffaa2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.497+0000 7feac45b6740 1 --2- 172.21.15.23:0/3283692025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffa9ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.497+0000 7feac45b6740 1 --2- 172.21.15.23:0/3283692025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffa8e140 0x55daffa8e520 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.497+0000 7feac45b6740 1 -- 172.21.15.23:0/3283692025 >> 172.21.15.23:0/3283692025 conn(0x55daffa88e90 msgr2=0x55daffa89270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.497+0000 7feac45b6740 1 -- 172.21.15.23:0/3283692025 shutdown_connections 2024-03-24T10:40:12.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.497+0000 7feac45b6740 1 -- 172.21.15.23:0/3283692025 wait complete. 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.498+0000 7feac45b6740 1 -- start start 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.498+0000 7feac45b6740 1 Processor -- start 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.498+0000 7feac45b6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffa8e140 0x55daffa9d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.498+0000 7feac45b6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffa9da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.498+0000 7feac45b6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffaa0470 0x55daffa9dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.498+0000 7feac45b6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55daff999f80 con 0x55daffa8e140 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.498+0000 7feac45b6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55daff6c84b0 con 0x55daffaa0470 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.498+0000 7feac45b6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55daff97aad0 con 0x55daffa8eaf0 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.498+0000 7feac45b6740 10 client.0 _ll_register_callbacks cb 0x55daff9adb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.498+0000 7feabaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffa8e140 0x55daffa9d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.498+0000 7feabaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffa8e140 0x55daffa9d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33636/0 (socket says 172.21.15.23:33636) 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.498+0000 7feabaffd640 1 -- 172.21.15.23:0/585339920 learned_addr learned my addr 172.21.15.23:0/585339920 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.498+0000 7feaba7fc640 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffa9da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.498+0000 7feabb7fe640 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffaa0470 0x55daffa9dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.499+0000 7feabaffd640 1 -- 172.21.15.23:0/585339920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 msgr2=0x55daffa9da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.499+0000 7feabaffd640 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffa9da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.499+0000 7feabaffd640 1 -- 172.21.15.23:0/585339920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffaa0470 msgr2=0x55daffa9dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.499+0000 7feabaffd640 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffaa0470 0x55daffa9dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.499+0000 7feabaffd640 1 -- 172.21.15.23:0/585339920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feabc1a6a90 con 0x55daffa8e140 2024-03-24T10:40:12.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.499+0000 7feaba7fc640 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffa9da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:12.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.499+0000 7feabb7fe640 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffaa0470 0x55daffa9dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:12.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.499+0000 7feabaffd640 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffa8e140 0x55daffa9d520 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7feaac0c4e70 tx=0x7feaac000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb2a2b7d2e8fc6b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.499+0000 7fea9affd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:12.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7fea9affd640 1 -- 172.21.15.23:0/585339920 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac0c0d50 con 0x55daffa8e140 2024-03-24T10:40:12.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7fea9affd640 1 -- 172.21.15.23:0/585339920 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7feaac0c53a0 con 0x55daffa8e140 2024-03-24T10:40:12.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7feac45b6740 1 -- 172.21.15.23:0/585339920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55daffa197a0 con 0x55daffa8e140 2024-03-24T10:40:12.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7f0c117fa640 1 -- 172.21.15.23:0/1496035716 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0c0c097640 con 0x7f0c14107780 2024-03-24T10:40:12.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7fea9affd640 1 -- 172.21.15.23:0/585339920 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7feaac0c5120 con 0x55daffa8e140 2024-03-24T10:40:12.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7feac45b6740 10 client.15492 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:12.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7feac45b6740 1 -- 172.21.15.23:0/585339920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55daff9ab6d0 con 0x55daffa8e140 2024-03-24T10:40:12.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7fea9affd640 1 -- 172.21.15.23:0/585339920 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaac09f080 con 0x55daffa8e140 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7fea9affd640 1 -- 172.21.15.23:0/585339920 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7feaac03c070 con 0x55daffa8e140 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7feac45b6740 10 client.15492 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7feac45b6740 10 client.15492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7feac45b6740 20 client.15492 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7feac45b6740 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55daffaaff30 0x55daffab2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7feac45b6740 4 client.15492 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7feac45b6740 1 -- 172.21.15.23:0/585339920 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55daff9eb1f0 con 0x55daffaaff30 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.500+0000 7feaba7fc640 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55daffaaff30 0x55daffab2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.501+0000 7fea9affd640 10 client.15492 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.501+0000 7feaba7fc640 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55daffaaff30 0x55daffab2320 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74689d7caed63cc9 server_cookie=a2cd7c463600726b in_seq=0 out_seq=0 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7fea9affd640 1 -- 172.21.15.23:0/585339920 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55daff9eb1f0 con 0x55daffaaff30 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7fea9affd640 10 client.15492 handle_command_reply: tid=0 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7feac45b6740 1 client.15492 shutdown 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7feac45b6740 1 -- 172.21.15.23:0/585339920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffa8e140 msgr2=0x55daffa9d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7feac45b6740 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffa8e140 0x55daffa9d520 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7feaac0c4e70 tx=0x7feaac000bc0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7feac45b6740 1 -- 172.21.15.23:0/585339920 shutdown_connections 2024-03-24T10:40:12.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7feac45b6740 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffaa0470 0x55daffa9dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7feac45b6740 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55daffaaff30 0x55daffab2320 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7feac45b6740 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffa9da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7feac45b6740 1 --2- 172.21.15.23:0/585339920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffa8e140 0x55daffa9d520 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7fea9affd640 0 client.15492 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:12.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7feac45b6740 1 -- 172.21.15.23:0/585339920 >> 172.21.15.23:0/585339920 conn(0x55daffa88e90 msgr2=0x55daffa8abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7feac45b6740 1 -- 172.21.15.23:0/585339920 shutdown_connections 2024-03-24T10:40:12.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7feac45b6740 1 -- 172.21.15.23:0/585339920 wait complete. 2024-03-24T10:40:12.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.502+0000 7feac45b6740 20 client.15492 trim_cache size 0 max 16384 2024-03-24T10:40:12.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feac45b6740 1 Processor -- start 2024-03-24T10:40:12.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feac45b6740 1 -- start start 2024-03-24T10:40:12.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feac45b6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffa8e140 0x55daffa9fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feac45b6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffaaba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feac45b6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffaa0470 0x55daffab7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feac45b6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55daffaabfd0 con 0x55daffa8e140 2024-03-24T10:40:12.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feac45b6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55daffaacdd0 con 0x55daffaa0470 2024-03-24T10:40:12.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feac45b6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55daffab2980 con 0x55daffa8eaf0 2024-03-24T10:40:12.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feabaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffa8e140 0x55daffa9fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feabaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffa8e140 0x55daffa9fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33646/0 (socket says 172.21.15.23:33646) 2024-03-24T10:40:12.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feabaffd640 1 -- 172.21.15.23:0/3860719828 learned_addr learned my addr 172.21.15.23:0/3860719828 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:12.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feaba7fc640 1 --2- 172.21.15.23:0/3860719828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffaaba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feabb7fe640 1 --2- 172.21.15.23:0/3860719828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffaa0470 0x55daffab7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feabaffd640 1 -- 172.21.15.23:0/3860719828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 msgr2=0x55daffaaba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feabaffd640 1 --2- 172.21.15.23:0/3860719828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffaaba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feabaffd640 1 -- 172.21.15.23:0/3860719828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffaa0470 msgr2=0x55daffab7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feabaffd640 1 --2- 172.21.15.23:0/3860719828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffaa0470 0x55daffab7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feabaffd640 1 -- 172.21.15.23:0/3860719828 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daff98fc60 con 0x55daffa8e140 2024-03-24T10:40:12.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.520+0000 7feabaffd640 1 --2- 172.21.15.23:0/3860719828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffa8e140 0x55daffa9fc50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7feaac0ee0a0 tx=0x7feaac0db220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97b0869b2e1ca27 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.521+0000 7feaa17fa640 1 -- 172.21.15.23:0/3860719828 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac0881b0 con 0x55daffa8e140 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.521+0000 7feaa17fa640 1 -- 172.21.15.23:0/3860719828 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55daffa197a0 con 0x55daffa8e140 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.521+0000 7feaa17fa640 1 -- 172.21.15.23:0/3860719828 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac0bbc80 con 0x55daffa8e140 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.521+0000 7feac45b6740 1 -- 172.21.15.23:0/3860719828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffa8e140 msgr2=0x55daffa9fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.521+0000 7feac45b6740 1 --2- 172.21.15.23:0/3860719828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffa8e140 0x55daffa9fc50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7feaac0ee0a0 tx=0x7feaac0db220 comp rx=0 tx=0).stop 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.521+0000 7feac45b6740 1 -- 172.21.15.23:0/3860719828 shutdown_connections 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.521+0000 7feac45b6740 1 --2- 172.21.15.23:0/3860719828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffaa0470 0x55daffab7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.521+0000 7feac45b6740 1 --2- 172.21.15.23:0/3860719828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffaaba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.521+0000 7feac45b6740 1 --2- 172.21.15.23:0/3860719828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffa8e140 0x55daffa9fc50 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.521+0000 7feac45b6740 1 -- 172.21.15.23:0/3860719828 >> 172.21.15.23:0/3860719828 conn(0x55daffa88e90 msgr2=0x55daffa87ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.521+0000 7feac45b6740 1 -- 172.21.15.23:0/3860719828 shutdown_connections 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.521+0000 7feac45b6740 1 -- 172.21.15.23:0/3860719828 wait complete. 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feac45b6740 1 -- start start 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feac45b6740 1 Processor -- start 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feac45b6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffa8e140 0x55daffaa80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feac45b6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffaa5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feac45b6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffaa0470 0x55daffaa55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feac45b6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55daffaa87c0 con 0x55daffaa0470 2024-03-24T10:40:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feac45b6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55daffaa8930 con 0x55daffa8e140 2024-03-24T10:40:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feac45b6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55daffaa5b10 con 0x55daffa8eaf0 2024-03-24T10:40:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feac45b6740 10 client.0 _ll_register_callbacks cb 0x7feaac0b8f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feabb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffaa0470 0x55daffaa55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feabaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffa8e140 0x55daffaa80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feaba7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffaa5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feabb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffaa0470 0x55daffaa55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33658/0 (socket says 172.21.15.23:33658) 2024-03-24T10:40:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feabaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffa8e140 0x55daffaa80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:57528/0 (socket says 172.21.15.23:57528) 2024-03-24T10:40:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.522+0000 7feabb7fe640 1 -- 172.21.15.23:0/588147023 learned_addr learned my addr 172.21.15.23:0/588147023 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7feabb7fe640 1 -- 172.21.15.23:0/588147023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 msgr2=0x55daffaa5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7feabb7fe640 1 --2- 172.21.15.23:0/588147023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffaa5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7feabb7fe640 1 -- 172.21.15.23:0/588147023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffa8e140 msgr2=0x55daffaa80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7feabb7fe640 1 --2- 172.21.15.23:0/588147023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffa8e140 0x55daffaa80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7feabb7fe640 1 -- 172.21.15.23:0/588147023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daff9ab6d0 con 0x55daffaa0470 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7feaba7fc640 1 --2- 172.21.15.23:0/588147023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffaa5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7feabaffd640 1 --2- 172.21.15.23:0/588147023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffa8e140 0x55daffaa80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7feabb7fe640 1 --2- 172.21.15.23:0/588147023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffaa0470 0x55daffaa55d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7feab40049d0 tx=0x7feab400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89b333828153053 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7fea9affd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7fea9affd640 1 -- 172.21.15.23:0/588147023 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab4017050 con 0x55daffaa0470 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7fea9affd640 1 -- 172.21.15.23:0/588147023 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7feab4004b20 con 0x55daffaa0470 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7feac45b6740 1 -- 172.21.15.23:0/588147023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feaac0a7050 con 0x55daffaa0470 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7fea9affd640 1 -- 172.21.15.23:0/588147023 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7feab40122e0 con 0x55daffaa0470 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7feac45b6740 10 client.15498 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7feac45b6740 1 -- 172.21.15.23:0/588147023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55daff9b5970 con 0x55daffaa0470 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.523+0000 7fea9affd640 1 -- 172.21.15.23:0/588147023 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaac0a7050 con 0x55daffaa0470 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.524+0000 7fea9affd640 1 -- 172.21.15.23:0/588147023 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7feab4010730 con 0x55daffaa0470 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.524+0000 7feac45b6740 10 client.15498 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.524+0000 7feac45b6740 10 client.15498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:12.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.524+0000 7feac45b6740 20 client.15498 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:12.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.524+0000 7feac45b6740 1 --2- 172.21.15.23:0/588147023 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55daffb17590 0x55daffb199b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.524+0000 7feac45b6740 4 client.15498 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:40:12.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.524+0000 7feac45b6740 1 -- 172.21.15.23:0/588147023 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55daff6f4860 con 0x55daffb17590 2024-03-24T10:40:12.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.524+0000 7feabaffd640 1 --2- 172.21.15.23:0/588147023 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55daffb17590 0x55daffb199b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.524+0000 7feabaffd640 1 --2- 172.21.15.23:0/588147023 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55daffb17590 0x55daffb199b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a201634c0a394b56 server_cookie=e399cd8afea746c7 in_seq=0 out_seq=0 2024-03-24T10:40:12.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.524+0000 7fea9affd640 10 client.15498 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:12.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.525+0000 7fea9affd640 1 -- 172.21.15.23:0/588147023 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55daff6f4860 con 0x55daffb17590 2024-03-24T10:40:12.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.525+0000 7fea9affd640 10 client.15498 handle_command_reply: tid=0 2024-03-24T10:40:12.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.525+0000 7feac45b6740 1 client.15498 shutdown 2024-03-24T10:40:12.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.525+0000 7feac45b6740 1 -- 172.21.15.23:0/588147023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffaa0470 msgr2=0x55daffaa55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.525+0000 7feac45b6740 1 --2- 172.21.15.23:0/588147023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffaa0470 0x55daffaa55d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7feab40049d0 tx=0x7feab400f790 comp rx=0 tx=0).stop 2024-03-24T10:40:12.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.525+0000 7feac45b6740 1 -- 172.21.15.23:0/588147023 shutdown_connections 2024-03-24T10:40:12.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.525+0000 7feac45b6740 1 --2- 172.21.15.23:0/588147023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55daffaa0470 0x55daffaa55d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.525+0000 7feac45b6740 1 --2- 172.21.15.23:0/588147023 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55daffb17590 0x55daffb199b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.525+0000 7feac45b6740 1 --2- 172.21.15.23:0/588147023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daffa8eaf0 0x55daffaa5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.525+0000 7feac45b6740 1 --2- 172.21.15.23:0/588147023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55daffa8e140 0x55daffaa80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.525+0000 7fea9affd640 0 client.15498 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:12.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.525+0000 7feac45b6740 1 -- 172.21.15.23:0/588147023 >> 172.21.15.23:0/588147023 conn(0x55daffa88e90 msgr2=0x55daffa9d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.525+0000 7feac45b6740 1 -- 172.21.15.23:0/588147023 shutdown_connections 2024-03-24T10:40:12.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.525+0000 7feac45b6740 1 -- 172.21.15.23:0/588147023 wait complete. 2024-03-24T10:40:12.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.526+0000 7feac45b6740 20 client.15498 trim_cache size 0 max 16384 2024-03-24T10:40:12.532 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:40:12.533 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:40:12.533 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "cc624829-28fc-464f-bcd2-4462091520fb", 2024-03-24T10:40:12.533 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:40:12.533 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:40:12.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.526+0000 7feac1901640 1 -- 172.21.15.23:0/439232536 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feaa40a2350 msgr2=0x7feaa40a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.526+0000 7feac1901640 1 --2- 172.21.15.23:0/439232536 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feaa40a2350 0x7feaa40a4810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feabc108f40 tx=0x7feab0005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:12.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.526+0000 7feac1901640 1 -- 172.21.15.23:0/439232536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc1038b0 msgr2=0x7feabc199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.526+0000 7feac1901640 1 --2- 172.21.15.23:0/439232536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc1038b0 0x7feabc199f70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7feaac005940 tx=0x7feaac02c670 comp rx=0 tx=0).stop 2024-03-24T10:40:12.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.526+0000 7feac1901640 1 -- 172.21.15.23:0/439232536 shutdown_connections 2024-03-24T10:40:12.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.526+0000 7feac1901640 1 --2- 172.21.15.23:0/439232536 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feaa40a2350 0x7feaa40a4810 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.526+0000 7feac1901640 1 --2- 172.21.15.23:0/439232536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc1098c0 0x7feabc19e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.526+0000 7feac1901640 1 --2- 172.21.15.23:0/439232536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feabc104270 0x7feabc19a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.526+0000 7feac1901640 1 --2- 172.21.15.23:0/439232536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc1038b0 0x7feabc199f70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.526+0000 7feac1901640 1 -- 172.21.15.23:0/439232536 >> 172.21.15.23:0/439232536 conn(0x7feabc0ff380 msgr2=0x7feabc0ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.526+0000 7feac1901640 1 -- 172.21.15.23:0/439232536 shutdown_connections 2024-03-24T10:40:12.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.526+0000 7feac1901640 1 -- 172.21.15.23:0/439232536 wait complete. 2024-03-24T10:40:12.538 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:40:12.538 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:40:12.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.642+0000 7ff52d147640 1 -- 172.21.15.23:0/86087572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff528100f60 msgr2=0x7ff52810d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.642+0000 7ff52d147640 1 --2- 172.21.15.23:0/86087572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff528100f60 0x7ff52810d420 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff520009ca0 tx=0x7ff52002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.642+0000 7ff52d147640 1 -- 172.21.15.23:0/86087572 shutdown_connections 2024-03-24T10:40:12.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.642+0000 7ff52d147640 1 --2- 172.21.15.23:0/86087572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff528100f60 0x7ff52810d420 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.642+0000 7ff52d147640 1 --2- 172.21.15.23:0/86087572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5281005a0 0x7ff528100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.642+0000 7ff52d147640 1 --2- 172.21.15.23:0/86087572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff528106660 0x7ff528106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.642+0000 7ff52d147640 1 -- 172.21.15.23:0/86087572 >> 172.21.15.23:0/86087572 conn(0x7ff528077710 msgr2=0x7ff528077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.642+0000 7ff52d147640 1 -- 172.21.15.23:0/86087572 shutdown_connections 2024-03-24T10:40:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.642+0000 7ff52d147640 1 -- 172.21.15.23:0/86087572 wait complete. 2024-03-24T10:40:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.643+0000 7ff52d147640 1 Processor -- start 2024-03-24T10:40:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.643+0000 7ff52d147640 1 -- start start 2024-03-24T10:40:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.643+0000 7ff52d147640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5281005a0 0x7ff52819d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.643+0000 7ff52d147640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff528100f60 0x7ff52819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.643+0000 7ff52d147640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff528106660 0x7ff528197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.643+0000 7ff52d147640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff528198100 con 0x7ff5281005a0 2024-03-24T10:40:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.643+0000 7ff52d147640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff528198270 con 0x7ff528100f60 2024-03-24T10:40:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.643+0000 7ff52d147640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff528198430 con 0x7ff528106660 2024-03-24T10:40:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.643+0000 7ff526d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5281005a0 0x7ff52819d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.643+0000 7ff526d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5281005a0 0x7ff52819d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33672/0 (socket says 172.21.15.23:33672) 2024-03-24T10:40:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.643+0000 7ff526d9d640 1 -- 172.21.15.23:0/1621734414 learned_addr learned my addr 172.21.15.23:0/1621734414 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.643+0000 7ff52659c640 1 --2- 172.21.15.23:0/1621734414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff528100f60 0x7ff52819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.643+0000 7ff52759e640 1 --2- 172.21.15.23:0/1621734414 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff528106660 0x7ff528197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.644+0000 7ff526d9d640 1 -- 172.21.15.23:0/1621734414 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff528106660 msgr2=0x7ff528197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.644+0000 7ff526d9d640 1 --2- 172.21.15.23:0/1621734414 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff528106660 0x7ff528197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.644+0000 7ff526d9d640 1 -- 172.21.15.23:0/1621734414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff528100f60 msgr2=0x7ff52819de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.644+0000 7ff526d9d640 1 --2- 172.21.15.23:0/1621734414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff528100f60 0x7ff52819de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.644+0000 7ff526d9d640 1 -- 172.21.15.23:0/1621734414 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff518009490 con 0x7ff5281005a0 2024-03-24T10:40:12.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.644+0000 7ff526d9d640 1 --2- 172.21.15.23:0/1621734414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5281005a0 0x7ff52819d900 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff51800ea60 tx=0x7ff51800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.644+0000 7ff50ffff640 1 -- 172.21.15.23:0/1621734414 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51800cde0 con 0x7ff5281005a0 2024-03-24T10:40:12.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.644+0000 7ff52d147640 1 -- 172.21.15.23:0/1621734414 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff520009900 con 0x7ff5281005a0 2024-03-24T10:40:12.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.644+0000 7ff50ffff640 1 -- 172.21.15.23:0/1621734414 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5180035f0 con 0x7ff5281005a0 2024-03-24T10:40:12.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.644+0000 7ff52d147640 1 -- 172.21.15.23:0/1621734414 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff528198900 con 0x7ff5281005a0 2024-03-24T10:40:12.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.644+0000 7ff50ffff640 1 -- 172.21.15.23:0/1621734414 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff518010690 con 0x7ff5281005a0 2024-03-24T10:40:12.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff50ffff640 1 -- 172.21.15.23:0/1621734414 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7ff518003af0 con 0x7ff5281005a0 2024-03-24T10:40:12.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52fdfd740 1 Processor -- start 2024-03-24T10:40:12.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52fdfd740 1 -- start start 2024-03-24T10:40:12.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52fdfd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03c1d0 0x55b5ad03c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52fdfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03caf0 0x55b5ad045190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52fdfd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad0456d0 0x55b5ad047ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52fdfd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b5acf60f30 con 0x55b5ad0456d0 2024-03-24T10:40:12.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52fdfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5acfaa810 con 0x55b5ad03caf0 2024-03-24T10:40:12.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52fdfd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b5acfa7d40 con 0x55b5ad03c1d0 2024-03-24T10:40:12.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52759e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03c1d0 0x55b5ad03c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff526d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03caf0 0x55b5ad045190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52759e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03c1d0 0x55b5ad03c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57266/0 (socket says 172.21.15.23:57266) 2024-03-24T10:40:12.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52759e640 1 -- 172.21.15.23:0/1345052149 learned_addr learned my addr 172.21.15.23:0/1345052149 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:12.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52659c640 1 --2- 172.21.15.23:0/1345052149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad0456d0 0x55b5ad047ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff50ffff640 1 --2- 172.21.15.23:0/1621734414 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff5100a2350 0x7ff5100a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52759e640 1 -- 172.21.15.23:0/1345052149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03caf0 msgr2=0x55b5ad045190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52759e640 1 --2- 172.21.15.23:0/1345052149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03caf0 0x55b5ad045190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52759e640 1 -- 172.21.15.23:0/1345052149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad0456d0 msgr2=0x55b5ad047ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52759e640 1 --2- 172.21.15.23:0/1345052149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad0456d0 0x55b5ad047ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52759e640 1 -- 172.21.15.23:0/1345052149 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5acf64ea0 con 0x55b5ad03c1d0 2024-03-24T10:40:12.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff50ffff640 1 -- 172.21.15.23:0/1621734414 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7ff518014070 con 0x7ff5281005a0 2024-03-24T10:40:12.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.647+0000 7ff52659c640 1 --2- 172.21.15.23:0/1345052149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad0456d0 0x55b5ad047ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:12.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.648+0000 7ff52759e640 1 --2- 172.21.15.23:0/1621734414 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff5100a2350 0x7ff5100a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.648+0000 7ff52759e640 1 --2- 172.21.15.23:0/1345052149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03c1d0 0x55b5ad03c5b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff520037040 tx=0x7ff520032de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=287bb04c7ef8c870 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.648+0000 7ff50d7fa640 1 -- 172.21.15.23:0/1345052149 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5200326a0 con 0x55b5ad03c1d0 2024-03-24T10:40:12.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.648+0000 7ff50d7fa640 1 -- 172.21.15.23:0/1345052149 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5200324e0 con 0x55b5ad03c1d0 2024-03-24T10:40:12.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.648+0000 7ff50d7fa640 1 -- 172.21.15.23:0/1345052149 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5200043d0 con 0x55b5ad03c1d0 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 -- 172.21.15.23:0/1345052149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03c1d0 msgr2=0x55b5ad03c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/1345052149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03c1d0 0x55b5ad03c5b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff520037040 tx=0x7ff520032de0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52759e640 1 --2- 172.21.15.23:0/1621734414 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff5100a2350 0x7ff5100a4810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff52004d000 tx=0x7ff52004a000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 -- 172.21.15.23:0/1345052149 shutdown_connections 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/1345052149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad0456d0 0x55b5ad047ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/1345052149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03caf0 0x55b5ad045190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/1345052149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03c1d0 0x55b5ad03c5b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 -- 172.21.15.23:0/1345052149 >> 172.21.15.23:0/1345052149 conn(0x55b5ad036900 msgr2=0x55b5ad036ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 -- 172.21.15.23:0/1345052149 shutdown_connections 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 -- 172.21.15.23:0/1345052149 wait complete. 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 -- start start 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 Processor -- start 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad03c1d0 0x55b5ad0426d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03caf0 0x55b5ad042c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad0456d0 0x55b5ad043150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b5acfa3520 con 0x55b5ad03c1d0 2024-03-24T10:40:12.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5acc76430 con 0x55b5ad0456d0 2024-03-24T10:40:12.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b5acc32e00 con 0x55b5ad03caf0 2024-03-24T10:40:12.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff52fdfd740 10 client.0 _ll_register_callbacks cb 0x55b5acf41d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:12.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff526d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad03c1d0 0x55b5ad0426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.649+0000 7ff526d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad03c1d0 0x55b5ad0426d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33692/0 (socket says 172.21.15.23:33692) 2024-03-24T10:40:12.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff526d9d640 1 -- 172.21.15.23:0/3465649788 learned_addr learned my addr 172.21.15.23:0/3465649788 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:12.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff52659c640 1 --2- 172.21.15.23:0/3465649788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03caf0 0x55b5ad042c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff52759e640 1 --2- 172.21.15.23:0/3465649788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad0456d0 0x55b5ad043150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff526d9d640 1 -- 172.21.15.23:0/3465649788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03caf0 msgr2=0x55b5ad042c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff526d9d640 1 --2- 172.21.15.23:0/3465649788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03caf0 0x55b5ad042c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff526d9d640 1 -- 172.21.15.23:0/3465649788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad0456d0 msgr2=0x55b5ad043150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff526d9d640 1 --2- 172.21.15.23:0/3465649788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad0456d0 0x55b5ad043150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff526d9d640 1 -- 172.21.15.23:0/3465649788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff52810dd20 con 0x55b5ad03c1d0 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff526d9d640 1 --2- 172.21.15.23:0/3465649788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad03c1d0 0x55b5ad0426d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff51809aeb0 tx=0x7ff5180d1f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68832cbd5608fed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff50aff5640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff50aff5640 1 -- 172.21.15.23:0/3465649788 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff518003960 con 0x55b5ad03c1d0 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff50aff5640 1 -- 172.21.15.23:0/3465649788 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7ff51809b620 con 0x55b5ad03c1d0 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff52fdfd740 1 -- 172.21.15.23:0/3465649788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b5acca27e0 con 0x55b5ad03c1d0 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff50aff5640 1 -- 172.21.15.23:0/3465649788 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff51809a880 con 0x55b5ad03c1d0 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff52fdfd740 10 client.15510 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff52fdfd740 1 -- 172.21.15.23:0/3465649788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b5acfa1ba0 con 0x55b5ad03c1d0 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.650+0000 7ff50aff5640 1 -- 172.21.15.23:0/3465649788 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b5acca27e0 con 0x55b5ad03c1d0 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.651+0000 7ff50aff5640 1 -- 172.21.15.23:0/3465649788 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff5180cc030 con 0x55b5ad03c1d0 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.651+0000 7ff52fdfd740 10 client.15510 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.651+0000 7ff52fdfd740 10 client.15510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.651+0000 7ff52fdfd740 20 client.15510 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:12.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.651+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/3465649788 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5ad055a40 0x55b5ad057e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.651+0000 7ff52fdfd740 4 client.15510 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.651+0000 7ff52fdfd740 1 -- 172.21.15.23:0/3465649788 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5accdf720 con 0x55b5ad055a40 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.651+0000 7ff52659c640 1 --2- 172.21.15.23:0/3465649788 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5ad055a40 0x55b5ad057e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.651+0000 7ff50aff5640 10 client.15510 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.651+0000 7ff52659c640 1 --2- 172.21.15.23:0/3465649788 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5ad055a40 0x55b5ad057e30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10de10f102dc8b9e server_cookie=51c1c8e3cbd2185e in_seq=0 out_seq=0 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.652+0000 7ff50aff5640 1 -- 172.21.15.23:0/3465649788 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b5accdf720 con 0x55b5ad055a40 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.652+0000 7ff50aff5640 10 client.15510 handle_command_reply: tid=0 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.652+0000 7ff52fdfd740 1 client.15510 shutdown 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.652+0000 7ff52fdfd740 1 -- 172.21.15.23:0/3465649788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad03c1d0 msgr2=0x55b5ad0426d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.652+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/3465649788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad03c1d0 0x55b5ad0426d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff51809aeb0 tx=0x7ff5180d1f60 comp rx=0 tx=0).stop 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.652+0000 7ff52fdfd740 1 -- 172.21.15.23:0/3465649788 shutdown_connections 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.652+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/3465649788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad0456d0 0x55b5ad043150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.652+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/3465649788 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5ad055a40 0x55b5ad057e30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.652+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/3465649788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03caf0 0x55b5ad042c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.652+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/3465649788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad03c1d0 0x55b5ad0426d0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.652+0000 7ff50aff5640 0 client.15510 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.652+0000 7ff52fdfd740 1 -- 172.21.15.23:0/3465649788 >> 172.21.15.23:0/3465649788 conn(0x55b5ad036900 msgr2=0x55b5ad035b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.652+0000 7ff52fdfd740 1 -- 172.21.15.23:0/3465649788 shutdown_connections 2024-03-24T10:40:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.652+0000 7ff52fdfd740 1 -- 172.21.15.23:0/3465649788 wait complete. 2024-03-24T10:40:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.653+0000 7ff52fdfd740 20 client.15510 trim_cache size 0 max 16384 2024-03-24T10:40:12.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff52fdfd740 1 Processor -- start 2024-03-24T10:40:12.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff52fdfd740 1 -- start start 2024-03-24T10:40:12.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff52fdfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03c1d0 0x55b5ad044e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff52fdfd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad03caf0 0x55b5ad035b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff52fdfd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad0456d0 0x55b5ad042f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff52fdfd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b5acf28430 con 0x55b5ad03caf0 2024-03-24T10:40:12.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff52fdfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5ad0360a0 con 0x55b5ad03c1d0 2024-03-24T10:40:12.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff52fdfd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b5ad037900 con 0x55b5ad0456d0 2024-03-24T10:40:12.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff52659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad03caf0 0x55b5ad035b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff52659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad03caf0 0x55b5ad035b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33708/0 (socket says 172.21.15.23:33708) 2024-03-24T10:40:12.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff52659c640 1 -- 172.21.15.23:0/2455125004 learned_addr learned my addr 172.21.15.23:0/2455125004 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:12.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff526d9d640 1 --2- 172.21.15.23:0/2455125004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03c1d0 0x55b5ad044e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff52759e640 1 --2- 172.21.15.23:0/2455125004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad0456d0 0x55b5ad042f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff52659c640 1 -- 172.21.15.23:0/2455125004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad0456d0 msgr2=0x55b5ad042f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.670+0000 7ff52659c640 1 --2- 172.21.15.23:0/2455125004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad0456d0 0x55b5ad042f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.671+0000 7ff52659c640 1 -- 172.21.15.23:0/2455125004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03c1d0 msgr2=0x55b5ad044e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.671+0000 7ff52659c640 1 --2- 172.21.15.23:0/2455125004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03c1d0 0x55b5ad044e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.671+0000 7ff52659c640 1 -- 172.21.15.23:0/2455125004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5acf3db20 con 0x55b5ad03caf0 2024-03-24T10:40:12.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.671+0000 7ff52659c640 1 --2- 172.21.15.23:0/2455125004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad03caf0 0x55b5ad035b60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff51c005e00 tx=0x7ff51c0096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df682f13e4384d6e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.671+0000 7ff50cff9640 1 -- 172.21.15.23:0/2455125004 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c017040 con 0x55b5ad03caf0 2024-03-24T10:40:12.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.671+0000 7ff50cff9640 1 -- 172.21.15.23:0/2455125004 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff51c018040 con 0x55b5ad03caf0 2024-03-24T10:40:12.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.671+0000 7ff50cff9640 1 -- 172.21.15.23:0/2455125004 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c004d90 con 0x55b5ad03caf0 2024-03-24T10:40:12.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 -- 172.21.15.23:0/2455125004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad03caf0 msgr2=0x55b5ad035b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/2455125004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad03caf0 0x55b5ad035b60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff51c005e00 tx=0x7ff51c0096d0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 -- 172.21.15.23:0/2455125004 shutdown_connections 2024-03-24T10:40:12.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/2455125004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad0456d0 0x55b5ad042f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/2455125004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad03caf0 0x55b5ad035b60 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/2455125004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03c1d0 0x55b5ad044e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 -- 172.21.15.23:0/2455125004 >> 172.21.15.23:0/2455125004 conn(0x55b5ad036900 msgr2=0x55b5ad037b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 -- 172.21.15.23:0/2455125004 shutdown_connections 2024-03-24T10:40:12.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 -- 172.21.15.23:0/2455125004 wait complete. 2024-03-24T10:40:12.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 -- start start 2024-03-24T10:40:12.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 Processor -- start 2024-03-24T10:40:12.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03c1d0 0x55b5ad049090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03caf0 0x55b5ad0495d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad0456d0 0x55b5ad049b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b5ad0bae20 con 0x55b5ad0456d0 2024-03-24T10:40:12.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5ad0baf90 con 0x55b5ad03c1d0 2024-03-24T10:40:12.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b5ad0bb150 con 0x55b5ad03caf0 2024-03-24T10:40:12.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52fdfd740 10 client.0 _ll_register_callbacks cb 0x7ff51800ef90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:12.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52759e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad0456d0 0x55b5ad049b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52759e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad0456d0 0x55b5ad049b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33710/0 (socket says 172.21.15.23:33710) 2024-03-24T10:40:12.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52759e640 1 -- 172.21.15.23:0/2930979534 learned_addr learned my addr 172.21.15.23:0/2930979534 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:12.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff526d9d640 1 --2- 172.21.15.23:0/2930979534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03c1d0 0x55b5ad049090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.672+0000 7ff52659c640 1 --2- 172.21.15.23:0/2930979534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03caf0 0x55b5ad0495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff52759e640 1 -- 172.21.15.23:0/2930979534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03caf0 msgr2=0x55b5ad0495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff52759e640 1 --2- 172.21.15.23:0/2930979534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03caf0 0x55b5ad0495d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff52759e640 1 -- 172.21.15.23:0/2930979534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03c1d0 msgr2=0x55b5ad049090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:12.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff52759e640 1 --2- 172.21.15.23:0/2930979534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03c1d0 0x55b5ad049090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff52759e640 1 -- 172.21.15.23:0/2930979534 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff51c009b00 con 0x55b5ad0456d0 2024-03-24T10:40:12.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff526d9d640 1 --2- 172.21.15.23:0/2930979534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03c1d0 0x55b5ad049090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:12.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff52759e640 1 --2- 172.21.15.23:0/2930979534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad0456d0 0x55b5ad049b10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff52002c630 tx=0x7ff5200348b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f4c58a25d76012 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff50aff5640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff50aff5640 1 -- 172.21.15.23:0/2930979534 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5200326a0 con 0x55b5ad0456d0 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff50aff5640 1 -- 172.21.15.23:0/2930979534 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7ff520032800 con 0x55b5ad0456d0 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff52fdfd740 1 -- 172.21.15.23:0/2930979534 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5200324e0 con 0x55b5ad0456d0 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff50aff5640 1 -- 172.21.15.23:0/2930979534 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff5200329c0 con 0x55b5ad0456d0 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff52fdfd740 10 client.15516 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff52fdfd740 1 -- 172.21.15.23:0/2930979534 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b5acf63700 con 0x55b5ad0456d0 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff50aff5640 1 -- 172.21.15.23:0/2930979534 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff52006c080 con 0x55b5ad0456d0 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.673+0000 7ff50aff5640 1 -- 172.21.15.23:0/2930979534 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff520066030 con 0x55b5ad0456d0 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 10 client.15516 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 10 client.15516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 20 client.15516 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/2930979534 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5ad0bc710 0x55b5ad0beb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 4 client.15516 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 1 -- 172.21.15.23:0/2930979534 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b5acd2e890 con 0x55b5ad0bc710 2024-03-24T10:40:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff526d9d640 1 --2- 172.21.15.23:0/2930979534 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5ad0bc710 0x55b5ad0beb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff526d9d640 1 --2- 172.21.15.23:0/2930979534 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5ad0bc710 0x55b5ad0beb30 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25f6298670687e9e server_cookie=29922eb1d229d3f in_seq=0 out_seq=0 2024-03-24T10:40:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff50aff5640 10 client.15516 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff50aff5640 1 -- 172.21.15.23:0/2930979534 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b5acd2e890 con 0x55b5ad0bc710 2024-03-24T10:40:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff50aff5640 10 client.15516 handle_command_reply: tid=0 2024-03-24T10:40:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 1 client.15516 shutdown 2024-03-24T10:40:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 1 -- 172.21.15.23:0/2930979534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad0456d0 msgr2=0x55b5ad049b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/2930979534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad0456d0 0x55b5ad049b10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff52002c630 tx=0x7ff5200348b0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 1 -- 172.21.15.23:0/2930979534 shutdown_connections 2024-03-24T10:40:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/2930979534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5ad0456d0 0x55b5ad049b10 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/2930979534 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5ad0bc710 0x55b5ad0beb30 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/2930979534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5ad03caf0 0x55b5ad0495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 1 --2- 172.21.15.23:0/2930979534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5ad03c1d0 0x55b5ad049090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff50aff5640 0 client.15516 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 1 -- 172.21.15.23:0/2930979534 >> 172.21.15.23:0/2930979534 conn(0x55b5ad036900 msgr2=0x55b5ad038320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 1 -- 172.21.15.23:0/2930979534 shutdown_connections 2024-03-24T10:40:12.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.674+0000 7ff52fdfd740 1 -- 172.21.15.23:0/2930979534 wait complete. 2024-03-24T10:40:12.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.675+0000 7ff52fdfd740 20 client.15516 trim_cache size 0 max 16384 2024-03-24T10:40:12.683 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:40:12.683 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:40:12.683 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:40:12.683 INFO:teuthology.orchestra.run.smithi023.stdout: "cc624829-28fc-464f-bcd2-4462091520fb": { 2024-03-24T10:40:12.683 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:40:12.683 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "cc624829-28fc-464f-bcd2-4462091520fb", 2024-03-24T10:40:12.683 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:40:12.684 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:40:12.684 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:40:12.684 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:40:12.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.675+0000 7ff52d147640 1 -- 172.21.15.23:0/1621734414 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff5100a2350 msgr2=0x7ff5100a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.675+0000 7ff52d147640 1 --2- 172.21.15.23:0/1621734414 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff5100a2350 0x7ff5100a4810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff52004d000 tx=0x7ff52004a000 comp rx=0 tx=0).stop 2024-03-24T10:40:12.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.675+0000 7ff52d147640 1 -- 172.21.15.23:0/1621734414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5281005a0 msgr2=0x7ff52819d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.675+0000 7ff52d147640 1 --2- 172.21.15.23:0/1621734414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5281005a0 0x7ff52819d900 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff51800ea60 tx=0x7ff51800ef20 comp rx=0 tx=0).stop 2024-03-24T10:40:12.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.675+0000 7ff52d147640 1 -- 172.21.15.23:0/1621734414 shutdown_connections 2024-03-24T10:40:12.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.675+0000 7ff52d147640 1 --2- 172.21.15.23:0/1621734414 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff5100a2350 0x7ff5100a4810 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.675+0000 7ff52d147640 1 --2- 172.21.15.23:0/1621734414 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff528106660 0x7ff528197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.675+0000 7ff52d147640 1 --2- 172.21.15.23:0/1621734414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff528100f60 0x7ff52819de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.675+0000 7ff52d147640 1 --2- 172.21.15.23:0/1621734414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5281005a0 0x7ff52819d900 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.675+0000 7ff52d147640 1 -- 172.21.15.23:0/1621734414 >> 172.21.15.23:0/1621734414 conn(0x7ff528077710 msgr2=0x7ff5280ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.675+0000 7ff52d147640 1 -- 172.21.15.23:0/1621734414 shutdown_connections 2024-03-24T10:40:12.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.675+0000 7ff52d147640 1 -- 172.21.15.23:0/1621734414 wait complete. 2024-03-24T10:40:12.687 INFO:tasks.cephfs.filesystem:scrub status for tag:cc624829-28fc-464f-bcd2-4462091520fb - {'path': '/', 'tag': 'cc624829-28fc-464f-bcd2-4462091520fb', 'options': 'recursive,force'} 2024-03-24T10:40:12.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.713+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/1496035716 --> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f0c14111850 con 0x7f0bfc0a2470 2024-03-24T10:40:12.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.775+0000 7f0c117fa640 1 -- 172.21.15.23:0/1496035716 <== mgr.15288 v2:172.21.15.23:6800/581236369 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0c14111850 con 0x7f0bfc0a2470 2024-03-24T10:40:12.776 INFO:teuthology.orchestra.run.smithi023.stdout:Schedule set for path / 2024-03-24T10:40:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.777+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/1496035716 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0bfc0a2470 msgr2=0x7f0bfc0a4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.777+0000 7f0c1a5c9640 1 --2- 172.21.15.23:0/1496035716 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0bfc0a2470 0x7f0bfc0a4930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0c14198bd0 tx=0x7f0c04005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.777+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/1496035716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c14107780 msgr2=0x7f0c14197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.777+0000 7f0c1a5c9640 1 --2- 172.21.15.23:0/1496035716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c14107780 0x7f0c14197a90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c00ec70 tx=0x7f0c0c00c6a0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.777+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/1496035716 shutdown_connections 2024-03-24T10:40:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.777+0000 7f0c1a5c9640 1 --2- 172.21.15.23:0/1496035716 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0bfc0a2470 0x7f0bfc0a4930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.777+0000 7f0c1a5c9640 1 --2- 172.21.15.23:0/1496035716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c14107780 0x7f0c14197a90 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.777+0000 7f0c1a5c9640 1 --2- 172.21.15.23:0/1496035716 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c14069730 0x7f0c1419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.777+0000 7f0c1a5c9640 1 --2- 172.21.15.23:0/1496035716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c14068d90 0x7f0c1419d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.777+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/1496035716 >> 172.21.15.23:0/1496035716 conn(0x7f0c14077460 msgr2=0x7f0c140ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.777+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/1496035716 shutdown_connections 2024-03-24T10:40:12.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.777+0000 7f0c1a5c9640 1 -- 172.21.15.23:0/1496035716 wait complete. 2024-03-24T10:40:12.788 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-24T10:40:12.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.933+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1647737777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2681016c0 msgr2=0x7fb268101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.933+0000 7fb26ff1e640 1 --2- 172.21.15.23:0/1647737777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2681016c0 0x7fb268101b40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009930 tx=0x7fb25c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:12.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.933+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1647737777 shutdown_connections 2024-03-24T10:40:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.933+0000 7fb26ff1e640 1 --2- 172.21.15.23:0/1647737777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb268102080 0x7fb26810e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.933+0000 7fb26ff1e640 1 --2- 172.21.15.23:0/1647737777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2681016c0 0x7fb268101b40 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.933+0000 7fb26ff1e640 1 --2- 172.21.15.23:0/1647737777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2681076d0 0x7fb268107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.933+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1647737777 >> 172.21.15.23:0/1647737777 conn(0x7fb2680fd1d0 msgr2=0x7fb2680ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.933+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1647737777 shutdown_connections 2024-03-24T10:40:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.933+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1647737777 wait complete. 2024-03-24T10:40:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.934+0000 7fb26ff1e640 1 Processor -- start 2024-03-24T10:40:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.934+0000 7fb26ff1e640 1 -- start start 2024-03-24T10:40:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.934+0000 7fb26ff1e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2681016c0 0x7fb26819d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.934+0000 7fb26ff1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb268102080 0x7fb26819de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.934+0000 7fb26ff1e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2681076d0 0x7fb268197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.934+0000 7fb26ff1e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb268198140 con 0x7fb2681016c0 2024-03-24T10:40:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.934+0000 7fb26ff1e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb2681982b0 con 0x7fb268102080 2024-03-24T10:40:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.934+0000 7fb26ff1e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb268198470 con 0x7fb2681076d0 2024-03-24T10:40:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.934+0000 7fb26dcba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2681016c0 0x7fb26819d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.934+0000 7fb26dcba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2681016c0 0x7fb26819d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33720/0 (socket says 172.21.15.23:33720) 2024-03-24T10:40:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.934+0000 7fb26dcba640 1 -- 172.21.15.23:0/1178824593 learned_addr learned my addr 172.21.15.23:0/1178824593 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.934+0000 7fb26d4b9640 1 --2- 172.21.15.23:0/1178824593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb268102080 0x7fb26819de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.934+0000 7fb26e4bb640 1 --2- 172.21.15.23:0/1178824593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2681076d0 0x7fb268197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.935+0000 7fb26dcba640 1 -- 172.21.15.23:0/1178824593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2681076d0 msgr2=0x7fb268197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.935+0000 7fb26dcba640 1 --2- 172.21.15.23:0/1178824593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2681076d0 0x7fb268197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.935+0000 7fb26dcba640 1 -- 172.21.15.23:0/1178824593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb268102080 msgr2=0x7fb26819de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.935+0000 7fb26dcba640 1 --2- 172.21.15.23:0/1178824593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb268102080 0x7fb26819de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.935+0000 7fb26dcba640 1 -- 172.21.15.23:0/1178824593 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb25c009560 con 0x7fb2681016c0 2024-03-24T10:40:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.935+0000 7fb26dcba640 1 --2- 172.21.15.23:0/1178824593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2681016c0 0x7fb26819d940 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb26000e9f0 tx=0x7fb26000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.935+0000 7fb266ffd640 1 -- 172.21.15.23:0/1178824593 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb260009720 con 0x7fb2681016c0 2024-03-24T10:40:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.935+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1178824593 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb268198750 con 0x7fb2681016c0 2024-03-24T10:40:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.935+0000 7fb266ffd640 1 -- 172.21.15.23:0/1178824593 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb26000cd80 con 0x7fb2681016c0 2024-03-24T10:40:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.935+0000 7fb266ffd640 1 -- 172.21.15.23:0/1178824593 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb260010690 con 0x7fb2681016c0 2024-03-24T10:40:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.935+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1178824593 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2681a49e0 con 0x7fb2681016c0 2024-03-24T10:40:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.936+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1178824593 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb26804f470 con 0x7fb2681016c0 2024-03-24T10:40:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.937+0000 7fb266ffd640 1 -- 172.21.15.23:0/1178824593 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7fb2600046c0 con 0x7fb2681016c0 2024-03-24T10:40:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.937+0000 7fb266ffd640 1 --2- 172.21.15.23:0/1178824593 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb2500a23f0 0x7fb2500a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.937+0000 7fb266ffd640 1 -- 172.21.15.23:0/1178824593 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fb260014070 con 0x7fb2681016c0 2024-03-24T10:40:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.937+0000 7fb26d4b9640 1 --2- 172.21.15.23:0/1178824593 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb2500a23f0 0x7fb2500a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.938+0000 7fb26d4b9640 1 --2- 172.21.15.23:0/1178824593 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb2500a23f0 0x7fb2500a48b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009900 tx=0x7fb25c005a20 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:12.942+0000 7fb266ffd640 1 -- 172.21.15.23:0/1178824593 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb2600970b0 con 0x7fb2681016c0 2024-03-24T10:40:13.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.147+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1178824593 --> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fb2681118b0 con 0x7fb2500a23f0 2024-03-24T10:40:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:12 smithi023 ceph-mon[32466]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 410 B/s wr, 25 op/s 2024-03-24T10:40:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:12 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1148409755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:40:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:12 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2111010293' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:13.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.200+0000 7fb266ffd640 1 -- 172.21.15.23:0/1178824593 <== mgr.15288 v2:172.21.15.23:6800/581236369 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fb2681118b0 con 0x7fb2500a23f0 2024-03-24T10:40:13.200 INFO:teuthology.orchestra.run.smithi023.stdout:Retention added to path / 2024-03-24T10:40:13.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.202+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1178824593 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb2500a23f0 msgr2=0x7fb2500a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:13.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.202+0000 7fb26ff1e640 1 --2- 172.21.15.23:0/1178824593 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb2500a23f0 0x7fb2500a48b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb25c009900 tx=0x7fb25c005a20 comp rx=0 tx=0).stop 2024-03-24T10:40:13.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.202+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1178824593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2681016c0 msgr2=0x7fb26819d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:13.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.202+0000 7fb26ff1e640 1 --2- 172.21.15.23:0/1178824593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2681016c0 0x7fb26819d940 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb26000e9f0 tx=0x7fb26000edf0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.202+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1178824593 shutdown_connections 2024-03-24T10:40:13.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.202+0000 7fb26ff1e640 1 --2- 172.21.15.23:0/1178824593 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb2500a23f0 0x7fb2500a48b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.202+0000 7fb26ff1e640 1 --2- 172.21.15.23:0/1178824593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2681076d0 0x7fb268197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.202+0000 7fb26ff1e640 1 --2- 172.21.15.23:0/1178824593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb268102080 0x7fb26819de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.202+0000 7fb26ff1e640 1 --2- 172.21.15.23:0/1178824593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2681016c0 0x7fb26819d940 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.202+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1178824593 >> 172.21.15.23:0/1178824593 conn(0x7fb2680fd1d0 msgr2=0x7fb2680fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:13.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.202+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1178824593 shutdown_connections 2024-03-24T10:40:13.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.202+0000 7fb26ff1e640 1 -- 172.21.15.23:0/1178824593 wait complete. 2024-03-24T10:40:13.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-24T10:40:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:12 smithi117 ceph-mon[36368]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 410 B/s wr, 25 op/s 2024-03-24T10:40:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:12 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1148409755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:40:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:12 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2111010293' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:13.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:12 smithi116 ceph-mon[36248]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 410 B/s wr, 25 op/s 2024-03-24T10:40:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:12 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1148409755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:40:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:12 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2111010293' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:40:13.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.316+0000 7f3eaf06f640 1 -- 172.21.15.23:0/2868763812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea81076f0 msgr2=0x7f3ea8107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:13.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.316+0000 7f3eaf06f640 1 --2- 172.21.15.23:0/2868763812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea81076f0 0x7f3ea8107ad0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c009930 tx=0x7f3e9c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:13.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.316+0000 7f3eaf06f640 1 -- 172.21.15.23:0/2868763812 shutdown_connections 2024-03-24T10:40:13.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.316+0000 7f3eaf06f640 1 --2- 172.21.15.23:0/2868763812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ea81020a0 0x7f3ea810e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.316+0000 7f3eaf06f640 1 --2- 172.21.15.23:0/2868763812 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ea81016e0 0x7f3ea8101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.316+0000 7f3eaf06f640 1 --2- 172.21.15.23:0/2868763812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea81076f0 0x7f3ea8107ad0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.316+0000 7f3eaf06f640 1 -- 172.21.15.23:0/2868763812 >> 172.21.15.23:0/2868763812 conn(0x7f3ea80fd1f0 msgr2=0x7f3ea80ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:13.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.316+0000 7f3eaf06f640 1 -- 172.21.15.23:0/2868763812 shutdown_connections 2024-03-24T10:40:13.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.316+0000 7f3eaf06f640 1 -- 172.21.15.23:0/2868763812 wait complete. 2024-03-24T10:40:13.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3eaf06f640 1 Processor -- start 2024-03-24T10:40:13.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3eaf06f640 1 -- start start 2024-03-24T10:40:13.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3eaf06f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ea81016e0 0x7f3ea8197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:13.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3eaf06f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ea81020a0 0x7f3ea8198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:13.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3eaf06f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea81076f0 0x7f3ea819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:13.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3eaf06f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea819cbf0 con 0x7f3ea81076f0 2024-03-24T10:40:13.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3eaf06f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea819cd60 con 0x7f3ea81020a0 2024-03-24T10:40:13.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3eaf06f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea819cf20 con 0x7f3ea81016e0 2024-03-24T10:40:13.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3ead60c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea81076f0 0x7f3ea819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:13.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3eace0b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ea81016e0 0x7f3ea8197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:13.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3ead60c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea81076f0 0x7f3ea819c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33746/0 (socket says 172.21.15.23:33746) 2024-03-24T10:40:13.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3ead60c640 1 -- 172.21.15.23:0/4153404747 learned_addr learned my addr 172.21.15.23:0/4153404747 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:13.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3ea7fff640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ea81020a0 0x7f3ea8198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:13.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3ead60c640 1 -- 172.21.15.23:0/4153404747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ea81016e0 msgr2=0x7f3ea8197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:13.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3ead60c640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ea81016e0 0x7f3ea8197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3ead60c640 1 -- 172.21.15.23:0/4153404747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ea81020a0 msgr2=0x7f3ea8198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:13.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3ead60c640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ea81020a0 0x7f3ea8198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3ead60c640 1 -- 172.21.15.23:0/4153404747 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9c009560 con 0x7f3ea81076f0 2024-03-24T10:40:13.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3eace0b640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ea81016e0 0x7f3ea8197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:13.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3ea7fff640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ea81020a0 0x7f3ea8198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:13.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3ead60c640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea81076f0 0x7f3ea819c5f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000eb10 tx=0x7f3ea000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:13.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.317+0000 7f3ea5ffb640 1 -- 172.21.15.23:0/4153404747 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0004b20 con 0x7f3ea81076f0 2024-03-24T10:40:13.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.318+0000 7f3ea5ffb640 1 -- 172.21.15.23:0/4153404747 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ea0004c80 con 0x7f3ea81076f0 2024-03-24T10:40:13.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.318+0000 7f3eaf06f640 1 -- 172.21.15.23:0/4153404747 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ea819d200 con 0x7f3ea81076f0 2024-03-24T10:40:13.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.318+0000 7f3ea5ffb640 1 -- 172.21.15.23:0/4153404747 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0010650 con 0x7f3ea81076f0 2024-03-24T10:40:13.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.318+0000 7f3eaf06f640 1 -- 172.21.15.23:0/4153404747 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ea81a4a80 con 0x7f3ea81076f0 2024-03-24T10:40:13.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.319+0000 7f3eaf06f640 1 -- 172.21.15.23:0/4153404747 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ea804f470 con 0x7f3ea81076f0 2024-03-24T10:40:13.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.320+0000 7f3ea5ffb640 1 -- 172.21.15.23:0/4153404747 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7f3ea0005020 con 0x7f3ea81076f0 2024-03-24T10:40:13.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.320+0000 7f3ea5ffb640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3e980a23f0 0x7f3e980a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:13.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.320+0000 7f3ea5ffb640 1 -- 172.21.15.23:0/4153404747 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f3ea0014070 con 0x7f3ea81076f0 2024-03-24T10:40:13.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.320+0000 7f3eace0b640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3e980a23f0 0x7f3e980a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:13.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.321+0000 7f3eace0b640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3e980a23f0 0x7f3e980a48b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c009900 tx=0x7f3e9c005a20 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:13.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.325+0000 7f3ea5ffb640 1 -- 172.21.15.23:0/4153404747 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3ea00972c0 con 0x7f3ea81076f0 2024-03-24T10:40:13.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.530+0000 7f3eaf06f640 1 -- 172.21.15.23:0/4153404747 --> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f3ea8109ce0 con 0x7f3e980a23f0 2024-03-24T10:40:13.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.537+0000 7f3ea5ffb640 1 -- 172.21.15.23:0/4153404747 <== mgr.15288 v2:172.21.15.23:6800/581236369 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f3ea8109ce0 con 0x7f3e980a23f0 2024-03-24T10:40:13.538 INFO:teuthology.orchestra.run.smithi023.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-24T00:00:00", "created": "2024-03-24T10:40:12", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-24T10:40:13.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.539+0000 7f3eaf06f640 1 -- 172.21.15.23:0/4153404747 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3e980a23f0 msgr2=0x7f3e980a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:13.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.539+0000 7f3eaf06f640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3e980a23f0 0x7f3e980a48b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c009900 tx=0x7f3e9c005a20 comp rx=0 tx=0).stop 2024-03-24T10:40:13.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.539+0000 7f3eaf06f640 1 -- 172.21.15.23:0/4153404747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea81076f0 msgr2=0x7f3ea819c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:13.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.539+0000 7f3eaf06f640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea81076f0 0x7f3ea819c5f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000eb10 tx=0x7f3ea000c2d0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.539+0000 7f3eaf06f640 1 -- 172.21.15.23:0/4153404747 shutdown_connections 2024-03-24T10:40:13.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.539+0000 7f3eaf06f640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3e980a23f0 0x7f3e980a48b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.539+0000 7f3eaf06f640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea81076f0 0x7f3ea819c5f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.539+0000 7f3eaf06f640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ea81020a0 0x7f3ea8198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.539+0000 7f3eaf06f640 1 --2- 172.21.15.23:0/4153404747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ea81016e0 0x7f3ea8197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.539+0000 7f3eaf06f640 1 -- 172.21.15.23:0/4153404747 >> 172.21.15.23:0/4153404747 conn(0x7f3ea80fd1f0 msgr2=0x7f3ea80fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:13.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.540+0000 7f3eaf06f640 1 -- 172.21.15.23:0/4153404747 shutdown_connections 2024-03-24T10:40:13.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.540+0000 7f3eaf06f640 1 -- 172.21.15.23:0/4153404747 wait complete. 2024-03-24T10:40:13.554 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-24T10:40:13.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.658+0000 7f2a4d54b640 1 -- 172.21.15.23:0/380589077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a48105590 msgr2=0x7f2a48105970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.658+0000 7f2a4d54b640 1 --2- 172.21.15.23:0/380589077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a48105590 0x7f2a48105970 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2a38009900 tx=0x7f2a3802bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.658+0000 7f2a4d54b640 1 -- 172.21.15.23:0/380589077 shutdown_connections 2024-03-24T10:40:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.658+0000 7f2a4d54b640 1 --2- 172.21.15.23:0/380589077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a48101200 0x7f2a4810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.658+0000 7f2a4d54b640 1 --2- 172.21.15.23:0/380589077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a48100860 0x7f2a48100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.658+0000 7f2a4d54b640 1 --2- 172.21.15.23:0/380589077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a48105590 0x7f2a48105970 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.658+0000 7f2a4d54b640 1 -- 172.21.15.23:0/380589077 >> 172.21.15.23:0/380589077 conn(0x7f2a480773e0 msgr2=0x7f2a480777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.658+0000 7f2a4d54b640 1 -- 172.21.15.23:0/380589077 shutdown_connections 2024-03-24T10:40:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.658+0000 7f2a4d54b640 1 -- 172.21.15.23:0/380589077 wait complete. 2024-03-24T10:40:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a4d54b640 1 Processor -- start 2024-03-24T10:40:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a4d54b640 1 -- start start 2024-03-24T10:40:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a4d54b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a48100860 0x7f2a48197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a4d54b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a48101200 0x7f2a48198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a4d54b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a48105590 0x7f2a4819c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a4d54b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4819cbe0 con 0x7f2a48101200 2024-03-24T10:40:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a4d54b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4819cd50 con 0x7f2a48100860 2024-03-24T10:40:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a4d54b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4819cf10 con 0x7f2a48105590 2024-03-24T10:40:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a467fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a48101200 0x7f2a48198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a467fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a48101200 0x7f2a48198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33766/0 (socket says 172.21.15.23:33766) 2024-03-24T10:40:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a467fc640 1 -- 172.21.15.23:0/1228405327 learned_addr learned my addr 172.21.15.23:0/1228405327 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a46ffd640 1 --2- 172.21.15.23:0/1228405327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a48100860 0x7f2a48197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a477fe640 1 --2- 172.21.15.23:0/1228405327 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a48105590 0x7f2a4819c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a467fc640 1 -- 172.21.15.23:0/1228405327 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a48105590 msgr2=0x7f2a4819c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a467fc640 1 --2- 172.21.15.23:0/1228405327 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a48105590 0x7f2a4819c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a467fc640 1 -- 172.21.15.23:0/1228405327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a48100860 msgr2=0x7f2a48197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a467fc640 1 --2- 172.21.15.23:0/1228405327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a48100860 0x7f2a48197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.659+0000 7f2a467fc640 1 -- 172.21.15.23:0/1228405327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3c009490 con 0x7f2a48101200 2024-03-24T10:40:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.660+0000 7f2a467fc640 1 --2- 172.21.15.23:0/1228405327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a48101200 0x7f2a48198260 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00ea60 tx=0x7f2a3c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:13.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.660+0000 7f2a2ffff640 1 -- 172.21.15.23:0/1228405327 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c00cde0 con 0x7f2a48101200 2024-03-24T10:40:13.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.660+0000 7f2a2ffff640 1 -- 172.21.15.23:0/1228405327 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a3c0035f0 con 0x7f2a48101200 2024-03-24T10:40:13.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.660+0000 7f2a4d54b640 1 -- 172.21.15.23:0/1228405327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a38009560 con 0x7f2a48101200 2024-03-24T10:40:13.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.660+0000 7f2a2ffff640 1 -- 172.21.15.23:0/1228405327 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c010690 con 0x7f2a48101200 2024-03-24T10:40:13.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.660+0000 7f2a4d54b640 1 -- 172.21.15.23:0/1228405327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a481a4890 con 0x7f2a48101200 2024-03-24T10:40:13.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.661+0000 7f2a4d54b640 1 -- 172.21.15.23:0/1228405327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a48101d20 con 0x7f2a48101200 2024-03-24T10:40:13.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.662+0000 7f2a2ffff640 1 -- 172.21.15.23:0/1228405327 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7f2a3c003af0 con 0x7f2a48101200 2024-03-24T10:40:13.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.662+0000 7f2a2ffff640 1 --2- 172.21.15.23:0/1228405327 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2a300a23f0 0x7f2a300a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:13.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.663+0000 7f2a46ffd640 1 --2- 172.21.15.23:0/1228405327 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2a300a23f0 0x7f2a300a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:13.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.663+0000 7f2a2ffff640 1 -- 172.21.15.23:0/1228405327 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f2a3c014070 con 0x7f2a48101200 2024-03-24T10:40:13.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.663+0000 7f2a46ffd640 1 --2- 172.21.15.23:0/1228405327 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2a300a23f0 0x7f2a300a48b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2a38005ab0 tx=0x7f2a38005a20 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:13.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.667+0000 7f2a2ffff640 1 -- 172.21.15.23:0/1228405327 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2a3c0971e0 con 0x7f2a48101200 2024-03-24T10:40:13.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.872+0000 7f2a4d54b640 1 -- 172.21.15.23:0/1228405327 --> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f2a48110870 con 0x7f2a300a23f0 2024-03-24T10:40:13.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.878+0000 7f2a2ffff640 1 -- 172.21.15.23:0/1228405327 <== mgr.15288 v2:172.21.15.23:6800/581236369 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f2a48110870 con 0x7f2a300a23f0 2024-03-24T10:40:13.878 INFO:teuthology.orchestra.run.smithi023.stdout:/ 1m 6m3h 2024-03-24T10:40:13.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.880+0000 7f2a4d54b640 1 -- 172.21.15.23:0/1228405327 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2a300a23f0 msgr2=0x7f2a300a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:13.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.880+0000 7f2a4d54b640 1 --2- 172.21.15.23:0/1228405327 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2a300a23f0 0x7f2a300a48b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2a38005ab0 tx=0x7f2a38005a20 comp rx=0 tx=0).stop 2024-03-24T10:40:13.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.880+0000 7f2a4d54b640 1 -- 172.21.15.23:0/1228405327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a48101200 msgr2=0x7f2a48198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:13.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.880+0000 7f2a4d54b640 1 --2- 172.21.15.23:0/1228405327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a48101200 0x7f2a48198260 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00ea60 tx=0x7f2a3c00ef20 comp rx=0 tx=0).stop 2024-03-24T10:40:13.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.880+0000 7f2a4d54b640 1 -- 172.21.15.23:0/1228405327 shutdown_connections 2024-03-24T10:40:13.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.880+0000 7f2a4d54b640 1 --2- 172.21.15.23:0/1228405327 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2a300a23f0 0x7f2a300a48b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.880+0000 7f2a4d54b640 1 --2- 172.21.15.23:0/1228405327 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a48105590 0x7f2a4819c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.880+0000 7f2a4d54b640 1 --2- 172.21.15.23:0/1228405327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a48101200 0x7f2a48198260 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.880+0000 7f2a4d54b640 1 --2- 172.21.15.23:0/1228405327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a48100860 0x7f2a48197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:13.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.880+0000 7f2a4d54b640 1 -- 172.21.15.23:0/1228405327 >> 172.21.15.23:0/1228405327 conn(0x7f2a480773e0 msgr2=0x7f2a4806a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:13.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.880+0000 7f2a4d54b640 1 -- 172.21.15.23:0/1228405327 shutdown_connections 2024-03-24T10:40:13.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:13.880+0000 7f2a4d54b640 1 -- 172.21.15.23:0/1228405327 wait complete. 2024-03-24T10:40:13.891 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-24T10:40:13.918 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-24T10:40:13.929 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-24T10:40:13.938 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T10:40:13.939 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-24T10:40:13.961 INFO:teuthology.run_tasks:Running task workunit... 2024-03-24T10:40:13.973 INFO:tasks.workunit:Pulling workunits from ref d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b 2024-03-24T10:40:13.974 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-24T10:40:13.974 INFO:tasks.workunit:timeout=3h 2024-03-24T10:40:13.974 INFO:tasks.workunit:cleanup=True 2024-03-24T10:40:13.974 DEBUG:teuthology.orchestra.run.smithi023:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-24T10:40:13.988 INFO:teuthology.orchestra.run.smithi023.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-24T10:40:13.989 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-24T10:40:13.989 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-24T10:40:13.989 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:40:13.989 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-24T10:40:13.989 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-24 10:38:17.332158605 +0000 2024-03-24T10:40:13.989 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-24 10:38:17.332158605 +0000 2024-03-24T10:40:13.989 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-24 10:40:03.035689286 +0000 2024-03-24T10:40:13.989 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-24 10:38:17.332158605 +0000 2024-03-24T10:40:13.989 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-24T10:40:13.989 DEBUG:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-24T10:40:14.053 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:13 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:40:14.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:13 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:14.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:13 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:40:14.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:13 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:14.054 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:13 smithi023 ceph-mon[32466]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:14.063 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b 2024-03-24T10:40:14.118 INFO:tasks.workunit.client.0.smithi023.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-24T10:40:14.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:13 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:40:14.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:13 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:14.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:13 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:40:14.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:13 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:14.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:13 smithi117 ceph-mon[36368]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:14.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:13 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:40:14.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:13 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:14.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:13 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:40:14.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:13 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:14.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:13 smithi116 ceph-mon[36248]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:14 smithi023 ceph-mon[32466]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:14 smithi023 ceph-mon[32466]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.3 KiB/s wr, 23 op/s 2024-03-24T10:40:15.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:14 smithi023 ceph-mon[32466]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:15.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:14 smithi023 ceph-mon[32466]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:15.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:14 smithi117 ceph-mon[36368]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:15.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:14 smithi117 ceph-mon[36368]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.3 KiB/s wr, 23 op/s 2024-03-24T10:40:15.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:14 smithi117 ceph-mon[36368]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:15.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:14 smithi117 ceph-mon[36368]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:15.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:14 smithi116 ceph-mon[36248]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:15.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:14 smithi116 ceph-mon[36248]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.3 KiB/s wr, 23 op/s 2024-03-24T10:40:15.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:14 smithi116 ceph-mon[36248]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:15.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:14 smithi116 ceph-mon[36248]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:40:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:15 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:40:16.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:15 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:40:16.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:15 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:40:17.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:16 smithi023 ceph-mon[32466]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1 KiB/s wr, 18 op/s 2024-03-24T10:40:17.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:16 smithi117 ceph-mon[36368]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1 KiB/s wr, 18 op/s 2024-03-24T10:40:17.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:16 smithi116 ceph-mon[36248]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1 KiB/s wr, 18 op/s 2024-03-24T10:40:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:18 smithi023 ceph-mon[32466]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 85 KiB/s wr, 53 op/s 2024-03-24T10:40:19.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:18 smithi117 ceph-mon[36368]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 85 KiB/s wr, 53 op/s 2024-03-24T10:40:19.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:18 smithi116 ceph-mon[36248]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 85 KiB/s wr, 53 op/s 2024-03-24T10:40:21.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:20 smithi023 ceph-mon[32466]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 78 KiB/s wr, 48 op/s 2024-03-24T10:40:21.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:20 smithi117 ceph-mon[36368]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 78 KiB/s wr, 48 op/s 2024-03-24T10:40:21.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:20 smithi116 ceph-mon[36248]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 78 KiB/s wr, 48 op/s 2024-03-24T10:40:22.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:22 smithi023 ceph-mon[32466]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 78 KiB/s wr, 48 op/s 2024-03-24T10:40:22.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:22 smithi117 ceph-mon[36368]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 78 KiB/s wr, 48 op/s 2024-03-24T10:40:22.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:22 smithi116 ceph-mon[36248]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 78 KiB/s wr, 48 op/s 2024-03-24T10:40:24.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:24 smithi023 ceph-mon[32466]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-03-24T10:40:24.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:24 smithi117 ceph-mon[36368]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-03-24T10:40:24.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:24 smithi116 ceph-mon[36248]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-03-24T10:40:26.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:26 smithi023 ceph-mon[32466]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-03-24T10:40:26.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:26 smithi117 ceph-mon[36368]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-03-24T10:40:26.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:26 smithi116 ceph-mon[36248]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-03-24T10:40:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.091+0000 7f89abf77640 1 -- 172.21.15.23:0/1673663690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a4101220 msgr2=0x7f89a410d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.091+0000 7f89abf77640 1 --2- 172.21.15.23:0/1673663690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a4101220 0x7f89a410d440 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f899c009ca0 tx=0x7f899c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 -- 172.21.15.23:0/1673663690 shutdown_connections 2024-03-24T10:40:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 --2- 172.21.15.23:0/1673663690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a4101220 0x7f89a410d440 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 --2- 172.21.15.23:0/1673663690 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89a4100860 0x7f89a4100ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 --2- 172.21.15.23:0/1673663690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89a41055b0 0x7f89a4105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 -- 172.21.15.23:0/1673663690 >> 172.21.15.23:0/1673663690 conn(0x7f89a4077460 msgr2=0x7f89a4077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 -- 172.21.15.23:0/1673663690 shutdown_connections 2024-03-24T10:40:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 -- 172.21.15.23:0/1673663690 wait complete. 2024-03-24T10:40:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 Processor -- start 2024-03-24T10:40:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 -- start start 2024-03-24T10:40:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89a4100860 0x7f89a419d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a4101220 0x7f89a419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89a41055b0 0x7f89a4197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f89a4198130 con 0x7f89a4101220 2024-03-24T10:40:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f89a41982a0 con 0x7f89a41055b0 2024-03-24T10:40:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.092+0000 7f89abf77640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f89a4198460 con 0x7f89a4100860 2024-03-24T10:40:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f89a9512640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a4101220 0x7f89a419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f89a9512640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a4101220 0x7f89a419de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53874/0 (socket says 172.21.15.23:53874) 2024-03-24T10:40:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f89a9512640 1 -- 172.21.15.23:0/322345254 learned_addr learned my addr 172.21.15.23:0/322345254 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f89aa514640 1 --2- 172.21.15.23:0/322345254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89a41055b0 0x7f89a4197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f89a9d13640 1 --2- 172.21.15.23:0/322345254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89a4100860 0x7f89a419d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f89a9512640 1 -- 172.21.15.23:0/322345254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89a4100860 msgr2=0x7f89a419d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f89a9512640 1 --2- 172.21.15.23:0/322345254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89a4100860 0x7f89a419d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f89a9512640 1 -- 172.21.15.23:0/322345254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89a41055b0 msgr2=0x7f89a4197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f89a9512640 1 --2- 172.21.15.23:0/322345254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89a41055b0 0x7f89a4197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f89a9512640 1 -- 172.21.15.23:0/322345254 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f899c009900 con 0x7f89a4101220 2024-03-24T10:40:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f89a9512640 1 --2- 172.21.15.23:0/322345254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a4101220 0x7f89a419de20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f899400e9f0 tx=0x7f899400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:27.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f899affd640 1 -- 172.21.15.23:0/322345254 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8994009720 con 0x7f89a4101220 2024-03-24T10:40:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f899affd640 1 -- 172.21.15.23:0/322345254 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f899400cd80 con 0x7f89a4101220 2024-03-24T10:40:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f89abf77640 1 -- 172.21.15.23:0/322345254 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a4198740 con 0x7f89a4101220 2024-03-24T10:40:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f899affd640 1 -- 172.21.15.23:0/322345254 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8994010690 con 0x7f89a4101220 2024-03-24T10:40:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.093+0000 7f89abf77640 1 -- 172.21.15.23:0/322345254 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a41a4980 con 0x7f89a4101220 2024-03-24T10:40:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.095+0000 7f899affd640 1 -- 172.21.15.23:0/322345254 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7f89940046c0 con 0x7f89a4101220 2024-03-24T10:40:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.095+0000 7f89aec2f740 1 Processor -- start 2024-03-24T10:40:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89aec2f740 1 -- start start 2024-03-24T10:40:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f899affd640 1 --2- 172.21.15.23:0/322345254 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f89900a23f0 0x7f89900a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89aec2f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 0x560df82039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89aec2f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df8203f00 0x560df820c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89aec2f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df820ccd0 0x560df820f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89aec2f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x560df81be1b0 con 0x560df820ccd0 2024-03-24T10:40:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89aec2f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560df8170f80 con 0x560df82035e0 2024-03-24T10:40:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89aec2f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560df816e4b0 con 0x560df8203f00 2024-03-24T10:40:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89aa514640 1 --2- 172.21.15.23:0/322345254 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f89900a23f0 0x7f89900a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89a9d13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 0x560df82039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f899affd640 1 -- 172.21.15.23:0/322345254 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f8994014070 con 0x7f89a4101220 2024-03-24T10:40:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89a9d13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 0x560df82039c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:39798/0 (socket says 172.21.15.23:39798) 2024-03-24T10:40:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89a9d13640 1 -- 172.21.15.23:0/4044254166 learned_addr learned my addr 172.21.15.23:0/4044254166 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89a9512640 1 --2- 172.21.15.23:0/4044254166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df8203f00 0x560df820c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89aa514640 1 --2- 172.21.15.23:0/4044254166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df820ccd0 0x560df820f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89a9512640 1 -- 172.21.15.23:0/4044254166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 msgr2=0x560df82039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89a9512640 1 --2- 172.21.15.23:0/4044254166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 0x560df82039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89a9512640 1 -- 172.21.15.23:0/4044254166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df820ccd0 msgr2=0x560df820f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89a9512640 1 --2- 172.21.15.23:0/4044254166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df820ccd0 0x560df820f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89a9512640 1 -- 172.21.15.23:0/4044254166 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560df80ff810 con 0x560df8203f00 2024-03-24T10:40:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.096+0000 7f89a9d13640 1 --2- 172.21.15.23:0/4044254166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 0x560df82039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.097+0000 7f89aa514640 1 --2- 172.21.15.23:0/322345254 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f89900a23f0 0x7f89900a48b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f899c004af0 tx=0x7f899c032eb0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.097+0000 7f89a9512640 1 --2- 172.21.15.23:0/4044254166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df8203f00 0x560df820c790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f89940cbc90 tx=0x7f899409aa60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae9dbd53317cf34b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.097+0000 7f898ffff640 1 -- 172.21.15.23:0/4044254166 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899409b560 con 0x560df8203f00 2024-03-24T10:40:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.097+0000 7f898ffff640 1 -- 172.21.15.23:0/4044254166 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f899409ae60 con 0x560df8203f00 2024-03-24T10:40:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.097+0000 7f898ffff640 1 -- 172.21.15.23:0/4044254166 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89940758f0 con 0x560df8203f00 2024-03-24T10:40:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.097+0000 7f89aec2f740 1 -- 172.21.15.23:0/4044254166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df8203f00 msgr2=0x560df820c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 --2- 172.21.15.23:0/4044254166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df8203f00 0x560df820c790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f89940cbc90 tx=0x7f899409aa60 comp rx=0 tx=0).stop 2024-03-24T10:40:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 -- 172.21.15.23:0/4044254166 shutdown_connections 2024-03-24T10:40:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 --2- 172.21.15.23:0/4044254166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df820ccd0 0x560df820f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 --2- 172.21.15.23:0/4044254166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df8203f00 0x560df820c790 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 --2- 172.21.15.23:0/4044254166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 0x560df82039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 -- 172.21.15.23:0/4044254166 >> 172.21.15.23:0/4044254166 conn(0x560df81fdf00 msgr2=0x560df81fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 -- 172.21.15.23:0/4044254166 shutdown_connections 2024-03-24T10:40:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 -- 172.21.15.23:0/4044254166 wait complete. 2024-03-24T10:40:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 -- start start 2024-03-24T10:40:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 Processor -- start 2024-03-24T10:40:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 0x560df8209df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df820a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df820ccd0 0x560df820a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x560df8169c90 con 0x560df8203f00 2024-03-24T10:40:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560df810ec50 con 0x560df82035e0 2024-03-24T10:40:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560df7e3d250 con 0x560df820ccd0 2024-03-24T10:40:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.098+0000 7f89aec2f740 10 client.0 _ll_register_callbacks cb 0x560df815d290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f89a9512640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df820a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f89a9512640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df820a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53892/0 (socket says 172.21.15.23:53892) 2024-03-24T10:40:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f89a9512640 1 -- 172.21.15.23:0/4020560614 learned_addr learned my addr 172.21.15.23:0/4020560614 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f89a9d13640 1 --2- 172.21.15.23:0/4020560614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 0x560df8209df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f89aa514640 1 --2- 172.21.15.23:0/4020560614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df820ccd0 0x560df820a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f89a9512640 1 -- 172.21.15.23:0/4020560614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df820ccd0 msgr2=0x560df820a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f89a9512640 1 --2- 172.21.15.23:0/4020560614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df820ccd0 0x560df820a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f89a9512640 1 -- 172.21.15.23:0/4020560614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 msgr2=0x560df8209df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f89a9512640 1 --2- 172.21.15.23:0/4020560614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 0x560df8209df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f89a9512640 1 -- 172.21.15.23:0/4020560614 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a41a4980 con 0x560df8203f00 2024-03-24T10:40:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f89a9512640 1 --2- 172.21.15.23:0/4020560614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df820a330 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f89940cbda0 tx=0x7f89940a33a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98bddd38cf1885b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f898d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f898d7fa640 1 -- 172.21.15.23:0/4020560614 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899409b560 con 0x560df8203f00 2024-03-24T10:40:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f898d7fa640 1 -- 172.21.15.23:0/4020560614 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f899409d070 con 0x560df8203f00 2024-03-24T10:40:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f89aec2f740 1 -- 172.21.15.23:0/4020560614 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560df818e790 con 0x560df8203f00 2024-03-24T10:40:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f898d7fa640 1 -- 172.21.15.23:0/4020560614 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f89940974d0 con 0x560df8203f00 2024-03-24T10:40:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f89aec2f740 10 client.15546 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f89aec2f740 1 -- 172.21.15.23:0/4020560614 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x560df8168310 con 0x560df8203f00 2024-03-24T10:40:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.099+0000 7f898d7fa640 1 -- 172.21.15.23:0/4020560614 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560df818e790 con 0x560df8203f00 2024-03-24T10:40:27.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.100+0000 7f898d7fa640 1 -- 172.21.15.23:0/4020560614 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f899401f060 con 0x560df8203f00 2024-03-24T10:40:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.100+0000 7f89aec2f740 10 client.15546 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.100+0000 7f89aec2f740 10 client.15546 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.100+0000 7f89aec2f740 20 client.15546 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.100+0000 7f89aec2f740 1 --2- 172.21.15.23:0/4020560614 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x560df821c980 0x560df821ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.100+0000 7f89aec2f740 4 client.15546 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.100+0000 7f89aec2f740 1 -- 172.21.15.23:0/4020560614 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560df7e69600 con 0x560df821c980 2024-03-24T10:40:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.100+0000 7f89a9d13640 1 --2- 172.21.15.23:0/4020560614 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x560df821c980 0x560df821ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.100+0000 7f89a9d13640 1 --2- 172.21.15.23:0/4020560614 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x560df821c980 0x560df821ed70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf8d3a02a3851f8b server_cookie=5f93e83f509b097e in_seq=0 out_seq=0 2024-03-24T10:40:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.100+0000 7f898d7fa640 10 client.15546 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f898d7fa640 1 -- 172.21.15.23:0/4020560614 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560df7e69600 con 0x560df821c980 2024-03-24T10:40:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f898d7fa640 10 client.15546 handle_command_reply: tid=0 2024-03-24T10:40:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f89aec2f740 1 client.15546 shutdown 2024-03-24T10:40:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f89aec2f740 1 -- 172.21.15.23:0/4020560614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 msgr2=0x560df820a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f89aec2f740 1 --2- 172.21.15.23:0/4020560614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df820a330 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f89940cbda0 tx=0x7f89940a33a0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f89aec2f740 1 -- 172.21.15.23:0/4020560614 shutdown_connections 2024-03-24T10:40:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f89aec2f740 1 --2- 172.21.15.23:0/4020560614 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x560df821c980 0x560df821ed70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f89aec2f740 1 --2- 172.21.15.23:0/4020560614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df820ccd0 0x560df820a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f89aec2f740 1 --2- 172.21.15.23:0/4020560614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df820a330 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f898d7fa640 0 client.15546 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f89aec2f740 1 --2- 172.21.15.23:0/4020560614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 0x560df8209df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f89aec2f740 1 -- 172.21.15.23:0/4020560614 >> 172.21.15.23:0/4020560614 conn(0x560df81fdf00 msgr2=0x560df81ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f89aec2f740 1 -- 172.21.15.23:0/4020560614 shutdown_connections 2024-03-24T10:40:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f89aec2f740 1 -- 172.21.15.23:0/4020560614 wait complete. 2024-03-24T10:40:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.101+0000 7f89aec2f740 20 client.15546 trim_cache size 0 max 16384 2024-03-24T10:40:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.116+0000 7f89aec2f740 1 Processor -- start 2024-03-24T10:40:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.116+0000 7f89aec2f740 1 -- start start 2024-03-24T10:40:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.116+0000 7f89aec2f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df82035e0 0x560df820c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.116+0000 7f89aec2f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df8218340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.116+0000 7f89aec2f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df820ccd0 0x560df8224630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.116+0000 7f89aec2f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x560df80ef660 con 0x560df8203f00 2024-03-24T10:40:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.116+0000 7f89aec2f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560df7e69600 con 0x560df820ccd0 2024-03-24T10:40:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.116+0000 7f89aec2f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560df81fc3f0 con 0x560df82035e0 2024-03-24T10:40:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f89a9512640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df8218340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f89a9512640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df8218340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53906/0 (socket says 172.21.15.23:53906) 2024-03-24T10:40:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f89a9512640 1 -- 172.21.15.23:0/3939498795 learned_addr learned my addr 172.21.15.23:0/3939498795 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f89a9d13640 1 --2- 172.21.15.23:0/3939498795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df82035e0 0x560df820c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f89aa514640 1 --2- 172.21.15.23:0/3939498795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df820ccd0 0x560df8224630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f89a9512640 1 -- 172.21.15.23:0/3939498795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df82035e0 msgr2=0x560df820c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f89a9512640 1 --2- 172.21.15.23:0/3939498795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df82035e0 0x560df820c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f89a9512640 1 -- 172.21.15.23:0/3939498795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df820ccd0 msgr2=0x560df8224630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f89a9512640 1 --2- 172.21.15.23:0/3939498795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df820ccd0 0x560df8224630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f89a9512640 1 -- 172.21.15.23:0/3939498795 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560df8104810 con 0x560df8203f00 2024-03-24T10:40:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f89a9d13640 1 --2- 172.21.15.23:0/3939498795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df82035e0 0x560df820c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:40:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f89aa514640 1 --2- 172.21.15.23:0/3939498795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df820ccd0 0x560df8224630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:40:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f89a9512640 1 --2- 172.21.15.23:0/3939498795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df8218340 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f89940972c0 tx=0x7f8994004b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319712a0a4343a0f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f898f7fe640 1 -- 172.21.15.23:0/3939498795 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899409b560 con 0x560df8203f00 2024-03-24T10:40:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f898f7fe640 1 -- 172.21.15.23:0/3939498795 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89940a3410 con 0x560df8203f00 2024-03-24T10:40:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.117+0000 7f898f7fe640 1 -- 172.21.15.23:0/3939498795 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899409a2b0 con 0x560df8203f00 2024-03-24T10:40:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.118+0000 7f89aec2f740 1 -- 172.21.15.23:0/3939498795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 msgr2=0x560df8218340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.118+0000 7f89aec2f740 1 --2- 172.21.15.23:0/3939498795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df8218340 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f89940972c0 tx=0x7f8994004b30 comp rx=0 tx=0).stop 2024-03-24T10:40:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.118+0000 7f89aec2f740 1 -- 172.21.15.23:0/3939498795 shutdown_connections 2024-03-24T10:40:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.118+0000 7f89aec2f740 1 --2- 172.21.15.23:0/3939498795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df820ccd0 0x560df8224630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.118+0000 7f89aec2f740 1 --2- 172.21.15.23:0/3939498795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df8218340 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.118+0000 7f89aec2f740 1 --2- 172.21.15.23:0/3939498795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df82035e0 0x560df820c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.118+0000 7f89aec2f740 1 -- 172.21.15.23:0/3939498795 >> 172.21.15.23:0/3939498795 conn(0x560df81fdf00 msgr2=0x560df82050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:27.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.118+0000 7f89aec2f740 1 -- 172.21.15.23:0/3939498795 shutdown_connections 2024-03-24T10:40:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.118+0000 7f89aec2f740 1 -- 172.21.15.23:0/3939498795 wait complete. 2024-03-24T10:40:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.118+0000 7f89aec2f740 1 -- start start 2024-03-24T10:40:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.118+0000 7f89aec2f740 1 Processor -- start 2024-03-24T10:40:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89aec2f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 0x560df8214970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89aec2f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df8206430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89aec2f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df820ccd0 0x560df8206970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89aec2f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x560df8215110 con 0x560df8203f00 2024-03-24T10:40:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89aec2f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560df8206eb0 con 0x560df82035e0 2024-03-24T10:40:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89aec2f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560df8207070 con 0x560df820ccd0 2024-03-24T10:40:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89aec2f740 10 client.0 _ll_register_callbacks cb 0x7f899c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89a9512640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df8206430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89a9512640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df8206430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53920/0 (socket says 172.21.15.23:53920) 2024-03-24T10:40:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89a9512640 1 -- 172.21.15.23:0/2610602000 learned_addr learned my addr 172.21.15.23:0/2610602000 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89a9d13640 1 --2- 172.21.15.23:0/2610602000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 0x560df8214970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89aa514640 1 --2- 172.21.15.23:0/2610602000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df820ccd0 0x560df8206970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89a9512640 1 -- 172.21.15.23:0/2610602000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df820ccd0 msgr2=0x560df8206970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89a9512640 1 --2- 172.21.15.23:0/2610602000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df820ccd0 0x560df8206970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89a9512640 1 -- 172.21.15.23:0/2610602000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 msgr2=0x560df8214970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89a9512640 1 --2- 172.21.15.23:0/2610602000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 0x560df8214970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89a9512640 1 -- 172.21.15.23:0/2610602000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560df8168310 con 0x560df8203f00 2024-03-24T10:40:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89a9512640 1 --2- 172.21.15.23:0/2610602000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df8206430 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f89940cc410 tx=0x7f8994099370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ade6674f2be32e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f898d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f898d7fa640 1 -- 172.21.15.23:0/2610602000 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89940a3600 con 0x560df8203f00 2024-03-24T10:40:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f898d7fa640 1 -- 172.21.15.23:0/2610602000 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f899409d070 con 0x560df8203f00 2024-03-24T10:40:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89aec2f740 1 -- 172.21.15.23:0/2610602000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f899409ae60 con 0x560df8203f00 2024-03-24T10:40:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f898d7fa640 1 -- 172.21.15.23:0/2610602000 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f899409efb0 con 0x560df8203f00 2024-03-24T10:40:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89aec2f740 10 client.15552 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.119+0000 7f89aec2f740 1 -- 172.21.15.23:0/2610602000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x560df812a5a0 con 0x560df8203f00 2024-03-24T10:40:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.120+0000 7f898d7fa640 1 -- 172.21.15.23:0/2610602000 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f899409ae60 con 0x560df8203f00 2024-03-24T10:40:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.120+0000 7f898d7fa640 1 -- 172.21.15.23:0/2610602000 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f899401f070 con 0x560df8203f00 2024-03-24T10:40:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.120+0000 7f89aec2f740 10 client.15552 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.120+0000 7f89aec2f740 10 client.15552 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.120+0000 7f89aec2f740 20 client.15552 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.120+0000 7f89aec2f740 1 --2- 172.21.15.23:0/2610602000 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x560df8282af0 0x560df8284ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.120+0000 7f89aec2f740 4 client.15552 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:40:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.120+0000 7f89aec2f740 1 -- 172.21.15.23:0/2610602000 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560df7eba1b0 con 0x560df8282af0 2024-03-24T10:40:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.120+0000 7f89a9d13640 1 --2- 172.21.15.23:0/2610602000 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x560df8282af0 0x560df8284ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.120+0000 7f898d7fa640 10 client.15552 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.120+0000 7f89a9d13640 1 --2- 172.21.15.23:0/2610602000 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x560df8282af0 0x560df8284ee0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73da722cc593c717 server_cookie=376763d9ebfeb7b in_seq=0 out_seq=0 2024-03-24T10:40:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.130+0000 7f898d7fa640 1 -- 172.21.15.23:0/2610602000 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560df7eba1b0 con 0x560df8282af0 2024-03-24T10:40:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.130+0000 7f898d7fa640 10 client.15552 handle_command_reply: tid=0 2024-03-24T10:40:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.130+0000 7f89aec2f740 1 client.15552 shutdown 2024-03-24T10:40:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.130+0000 7f89aec2f740 1 -- 172.21.15.23:0/2610602000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 msgr2=0x560df8206430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.130+0000 7f89aec2f740 1 --2- 172.21.15.23:0/2610602000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df8206430 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f89940cc410 tx=0x7f8994099370 comp rx=0 tx=0).stop 2024-03-24T10:40:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.130+0000 7f89aec2f740 1 -- 172.21.15.23:0/2610602000 shutdown_connections 2024-03-24T10:40:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.130+0000 7f89aec2f740 1 --2- 172.21.15.23:0/2610602000 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x560df8282af0 0x560df8284ee0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.130+0000 7f89aec2f740 1 --2- 172.21.15.23:0/2610602000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560df820ccd0 0x560df8206970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.130+0000 7f89aec2f740 1 --2- 172.21.15.23:0/2610602000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560df8203f00 0x560df8206430 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.130+0000 7f89aec2f740 1 --2- 172.21.15.23:0/2610602000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560df82035e0 0x560df8214970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.130+0000 7f898d7fa640 0 client.15552 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.130+0000 7f89aec2f740 1 -- 172.21.15.23:0/2610602000 >> 172.21.15.23:0/2610602000 conn(0x560df81fdf00 msgr2=0x560df8209d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89aec2f740 1 -- 172.21.15.23:0/2610602000 shutdown_connections 2024-03-24T10:40:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89aec2f740 1 -- 172.21.15.23:0/2610602000 wait complete. 2024-03-24T10:40:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89aec2f740 20 client.15552 trim_cache size 0 max 16384 2024-03-24T10:40:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:40:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:40:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:40:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:40:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89abf77640 1 -- 172.21.15.23:0/322345254 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f89900a23f0 msgr2=0x7f89900a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89abf77640 1 --2- 172.21.15.23:0/322345254 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f89900a23f0 0x7f89900a48b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f899c004af0 tx=0x7f899c032eb0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89abf77640 1 -- 172.21.15.23:0/322345254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a4101220 msgr2=0x7f89a419de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89abf77640 1 --2- 172.21.15.23:0/322345254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a4101220 0x7f89a419de20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f899400e9f0 tx=0x7f899400edf0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89abf77640 1 -- 172.21.15.23:0/322345254 shutdown_connections 2024-03-24T10:40:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89abf77640 1 --2- 172.21.15.23:0/322345254 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f89900a23f0 0x7f89900a48b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89abf77640 1 --2- 172.21.15.23:0/322345254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89a41055b0 0x7f89a4197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89abf77640 1 --2- 172.21.15.23:0/322345254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a4101220 0x7f89a419de20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89abf77640 1 --2- 172.21.15.23:0/322345254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89a4100860 0x7f89a419d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89abf77640 1 -- 172.21.15.23:0/322345254 >> 172.21.15.23:0/322345254 conn(0x7f89a4077460 msgr2=0x7f89a4069620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89abf77640 1 -- 172.21.15.23:0/322345254 shutdown_connections 2024-03-24T10:40:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:27.131+0000 7f89abf77640 1 -- 172.21.15.23:0/322345254 wait complete. 2024-03-24T10:40:28.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:28 smithi023 ceph-mon[32466]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-03-24T10:40:28.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:28 smithi117 ceph-mon[36368]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-03-24T10:40:28.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:28 smithi116 ceph-mon[36248]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-03-24T10:40:30.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:30 smithi023 ceph-mon[32466]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:40:30.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:30 smithi117 ceph-mon[36368]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:40:30.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:30 smithi116 ceph-mon[36248]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:40:32.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:32 smithi023 ceph-mon[32466]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-24T10:40:32.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:32 smithi117 ceph-mon[36368]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-24T10:40:32.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:32 smithi116 ceph-mon[36248]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-24T10:40:34.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:34 smithi023 ceph-mon[32466]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-24T10:40:34.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:34 smithi117 ceph-mon[36368]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-24T10:40:34.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:34 smithi116 ceph-mon[36248]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-24T10:40:36.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:36 smithi116 ceph-mon[36248]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-24T10:40:36.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:36 smithi023 ceph-mon[32466]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-24T10:40:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:36 smithi117 ceph-mon[36368]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-24T10:40:38.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:38 smithi023 ceph-mon[32466]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-24T10:40:38.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:38 smithi117 ceph-mon[36368]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-24T10:40:38.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:38 smithi116 ceph-mon[36248]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-24T10:40:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.218+0000 7f2eb799a640 1 -- 172.21.15.23:0/3059397934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb0069730 msgr2=0x7f2eb010e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.218+0000 7f2eb799a640 1 --2- 172.21.15.23:0/3059397934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb0069730 0x7f2eb010e5a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0009cd0 tx=0x7f2ea002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 -- 172.21.15.23:0/3059397934 shutdown_connections 2024-03-24T10:40:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 --2- 172.21.15.23:0/3059397934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb0069730 0x7f2eb010e5a0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 --2- 172.21.15.23:0/3059397934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0068d90 0x7f2eb00691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 --2- 172.21.15.23:0/3059397934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2eb01077e0 0x7f2eb0107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 -- 172.21.15.23:0/3059397934 >> 172.21.15.23:0/3059397934 conn(0x7f2eb0077460 msgr2=0x7f2eb0077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 -- 172.21.15.23:0/3059397934 shutdown_connections 2024-03-24T10:40:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 -- 172.21.15.23:0/3059397934 wait complete. 2024-03-24T10:40:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 Processor -- start 2024-03-24T10:40:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 -- start start 2024-03-24T10:40:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb0068d90 0x7f2eb0100930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0069730 0x7f2eb0100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2eb01077e0 0x7f2eb01013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb01a5fe0 con 0x7f2eb0068d90 2024-03-24T10:40:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb01a6150 con 0x7f2eb0069730 2024-03-24T10:40:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.219+0000 7f2eb799a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb01a6310 con 0x7f2eb01077e0 2024-03-24T10:40:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb5736640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb0068d90 0x7f2eb0100930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb4f35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0069730 0x7f2eb0100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb5736640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb0068d90 0x7f2eb0100930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38096/0 (socket says 172.21.15.23:38096) 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb5736640 1 -- 172.21.15.23:0/2955094339 learned_addr learned my addr 172.21.15.23:0/2955094339 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb5f37640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2eb01077e0 0x7f2eb01013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb4f35640 1 -- 172.21.15.23:0/2955094339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2eb01077e0 msgr2=0x7f2eb01013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb4f35640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2eb01077e0 0x7f2eb01013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb4f35640 1 -- 172.21.15.23:0/2955094339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb0068d90 msgr2=0x7f2eb0100930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb4f35640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb0068d90 0x7f2eb0100930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb4f35640 1 -- 172.21.15.23:0/2955094339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea0009900 con 0x7f2eb0069730 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb5f37640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2eb01077e0 0x7f2eb01013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb5736640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb0068d90 0x7f2eb0100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb4f35640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0069730 0x7f2eb0100e70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400e9f0 tx=0x7f2ea400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eae7fc640 1 -- 172.21.15.23:0/2955094339 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea400ccc0 con 0x7f2eb0069730 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb799a640 1 -- 172.21.15.23:0/2955094339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb01a65f0 con 0x7f2eb0069730 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eae7fc640 1 -- 172.21.15.23:0/2955094339 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ea400ce20 con 0x7f2eb0069730 2024-03-24T10:40:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eae7fc640 1 -- 172.21.15.23:0/2955094339 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea4018920 con 0x7f2eb0069730 2024-03-24T10:40:40.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.220+0000 7f2eb799a640 1 -- 172.21.15.23:0/2955094339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb01a6b00 con 0x7f2eb0069730 2024-03-24T10:40:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eae7fc640 1 -- 172.21.15.23:0/2955094339 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7f2ea4018a80 con 0x7f2eb0069730 2024-03-24T10:40:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eba580740 1 Processor -- start 2024-03-24T10:40:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eba580740 1 -- start start 2024-03-24T10:40:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eba580740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baf5e0 0x55b4c5baf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eba580740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5baff00 0x55b4c5bb8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eba580740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bbb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eba580740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b4c5b6a1b0 con 0x55b4c5baff00 2024-03-24T10:40:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eba580740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b4c5b1cf80 con 0x55b4c5baf5e0 2024-03-24T10:40:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eba580740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b4c5b1a4b0 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eb5736640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5baff00 0x55b4c5bb8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eb5736640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5baff00 0x55b4c5bb8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38108/0 (socket says 172.21.15.23:38108) 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eb5736640 1 -- 172.21.15.23:0/2463047049 learned_addr learned my addr 172.21.15.23:0/2463047049 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eb5f37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baf5e0 0x55b4c5baf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eb4f35640 1 --2- 172.21.15.23:0/2463047049 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bbb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eae7fc640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2e980a23f0 0x7f2e980a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eb4f35640 1 -- 172.21.15.23:0/2463047049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baf5e0 msgr2=0x55b4c5baf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eb4f35640 1 --2- 172.21.15.23:0/2463047049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baf5e0 0x55b4c5baf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eb4f35640 1 -- 172.21.15.23:0/2463047049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5baff00 msgr2=0x55b4c5bb8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eb4f35640 1 --2- 172.21.15.23:0/2463047049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5baff00 0x55b4c5bb8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eb4f35640 1 -- 172.21.15.23:0/2463047049 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4c5aab810 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eb5f37640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2e980a23f0 0x7f2e980a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eb5736640 1 --2- 172.21.15.23:0/2463047049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5baff00 0x55b4c5bb8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.224+0000 7f2eae7fc640 1 -- 172.21.15.23:0/2955094339 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f2ea4014070 con 0x7f2eb0069730 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.225+0000 7f2eb4f35640 1 --2- 172.21.15.23:0/2463047049 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bbb0c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40cbbb0 tx=0x7f2ea4010f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d25c9eda3423ce2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.225+0000 7f2eb5f37640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2e980a23f0 0x7f2e980a48b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0005cc0 tx=0x7f2ea002e040 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.225+0000 7f2e977fe640 1 -- 172.21.15.23:0/2463047049 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea409a790 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.225+0000 7f2e977fe640 1 -- 172.21.15.23:0/2463047049 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ea409ae30 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.225+0000 7f2e977fe640 1 -- 172.21.15.23:0/2463047049 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea40974a0 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.225+0000 7f2eba580740 1 -- 172.21.15.23:0/2463047049 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5bb8cd0 msgr2=0x55b4c5bbb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.225+0000 7f2eba580740 1 --2- 172.21.15.23:0/2463047049 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bbb0c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40cbbb0 tx=0x7f2ea4010f00 comp rx=0 tx=0).stop 2024-03-24T10:40:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 -- 172.21.15.23:0/2463047049 shutdown_connections 2024-03-24T10:40:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 --2- 172.21.15.23:0/2463047049 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bbb0c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 --2- 172.21.15.23:0/2463047049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5baff00 0x55b4c5bb8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 --2- 172.21.15.23:0/2463047049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baf5e0 0x55b4c5baf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 -- 172.21.15.23:0/2463047049 >> 172.21.15.23:0/2463047049 conn(0x55b4c5ba9f00 msgr2=0x55b4c5baa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 -- 172.21.15.23:0/2463047049 shutdown_connections 2024-03-24T10:40:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 -- 172.21.15.23:0/2463047049 wait complete. 2024-03-24T10:40:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 -- start start 2024-03-24T10:40:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 Processor -- start 2024-03-24T10:40:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 0x55b4c5bb5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baff00 0x55b4c5bb6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bb67a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b4c5b15c90 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b4c5abac50 con 0x55b4c5baff00 2024-03-24T10:40:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b4c57e9250 con 0x55b4c5baf5e0 2024-03-24T10:40:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.226+0000 7f2eba580740 10 client.0 _ll_register_callbacks cb 0x55b4c5b09290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2eb5f37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bb67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2eb5736640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 0x55b4c5bb5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2eb5f37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bb67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38116/0 (socket says 172.21.15.23:38116) 2024-03-24T10:40:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2eb5f37640 1 -- 172.21.15.23:0/1041782052 learned_addr learned my addr 172.21.15.23:0/1041782052 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2eb4f35640 1 --2- 172.21.15.23:0/1041782052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baff00 0x55b4c5bb6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2eb5f37640 1 -- 172.21.15.23:0/1041782052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 msgr2=0x55b4c5bb5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2eb5f37640 1 --2- 172.21.15.23:0/1041782052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 0x55b4c5bb5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2eb5f37640 1 -- 172.21.15.23:0/1041782052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baff00 msgr2=0x55b4c5bb6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2eb5f37640 1 --2- 172.21.15.23:0/1041782052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baff00 0x55b4c5bb6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2eb5f37640 1 -- 172.21.15.23:0/1041782052 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb01a6b00 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2eb5f37640 1 --2- 172.21.15.23:0/1041782052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bb67a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0036000 tx=0x7f2ea0005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=144f5dd1fbf3eb30 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2e94ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2e94ff9640 1 -- 172.21.15.23:0/1041782052 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea003fbf0 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2e94ff9640 1 -- 172.21.15.23:0/1041782052 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f2ea003fd50 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2eba580740 1 -- 172.21.15.23:0/1041782052 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b4c5b3a790 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2e94ff9640 1 -- 172.21.15.23:0/1041782052 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2ea00324c0 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2eba580740 10 client.15564 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.227+0000 7f2eba580740 1 -- 172.21.15.23:0/1041782052 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b4c5b14310 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.228+0000 7f2e94ff9640 1 -- 172.21.15.23:0/1041782052 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b4c5b3a790 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.228+0000 7f2e94ff9640 1 -- 172.21.15.23:0/1041782052 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f2ea0048040 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.228+0000 7f2eba580740 10 client.15564 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.228+0000 7f2eba580740 10 client.15564 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.228+0000 7f2eba580740 20 client.15564 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.228+0000 7f2eba580740 1 --2- 172.21.15.23:0/1041782052 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b4c5bc8b60 0x55b4c5bcaf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.228+0000 7f2eba580740 4 client.15564 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.228+0000 7f2eba580740 1 -- 172.21.15.23:0/1041782052 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4c5815600 con 0x55b4c5bc8b60 2024-03-24T10:40:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.228+0000 7f2eb5736640 1 --2- 172.21.15.23:0/1041782052 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b4c5bc8b60 0x55b4c5bcaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.228+0000 7f2eb5736640 1 --2- 172.21.15.23:0/1041782052 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b4c5bc8b60 0x55b4c5bcaf50 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c400617359fd5dc7 server_cookie=4fcbb343cd9ef61b in_seq=0 out_seq=0 2024-03-24T10:40:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.228+0000 7f2e94ff9640 10 client.15564 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.229+0000 7f2e94ff9640 1 -- 172.21.15.23:0/1041782052 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b4c5815600 con 0x55b4c5bc8b60 2024-03-24T10:40:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.229+0000 7f2e94ff9640 10 client.15564 handle_command_reply: tid=0 2024-03-24T10:40:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.229+0000 7f2eba580740 1 client.15564 shutdown 2024-03-24T10:40:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.229+0000 7f2eba580740 1 -- 172.21.15.23:0/1041782052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5bb8cd0 msgr2=0x55b4c5bb67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.229+0000 7f2eba580740 1 --2- 172.21.15.23:0/1041782052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bb67a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0036000 tx=0x7f2ea0005c50 comp rx=0 tx=0).stop 2024-03-24T10:40:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.229+0000 7f2eba580740 1 -- 172.21.15.23:0/1041782052 shutdown_connections 2024-03-24T10:40:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.229+0000 7f2eba580740 1 --2- 172.21.15.23:0/1041782052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bb67a0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.229+0000 7f2eba580740 1 --2- 172.21.15.23:0/1041782052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baff00 0x55b4c5bb6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.229+0000 7f2eba580740 1 --2- 172.21.15.23:0/1041782052 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b4c5bc8b60 0x55b4c5bcaf50 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.229+0000 7f2eba580740 1 --2- 172.21.15.23:0/1041782052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 0x55b4c5bb5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.229+0000 7f2e94ff9640 0 client.15564 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.229+0000 7f2eba580740 1 -- 172.21.15.23:0/1041782052 >> 172.21.15.23:0/1041782052 conn(0x55b4c5ba9f00 msgr2=0x55b4c5babc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.229+0000 7f2eba580740 1 -- 172.21.15.23:0/1041782052 shutdown_connections 2024-03-24T10:40:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.230+0000 7f2eba580740 1 -- 172.21.15.23:0/1041782052 wait complete. 2024-03-24T10:40:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.230+0000 7f2eba580740 20 client.15564 trim_cache size 0 max 16384 2024-03-24T10:40:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.244+0000 7f2eba580740 1 Processor -- start 2024-03-24T10:40:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.244+0000 7f2eba580740 1 -- start start 2024-03-24T10:40:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eba580740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 0x55b4c5bc4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eba580740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baff00 0x55b4c5bb65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eba580740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5baba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eba580740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b4c5a9b660 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eba580740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b4c5bc5610 con 0x55b4c5baff00 2024-03-24T10:40:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eba580740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b4c5baaad0 con 0x55b4c5baf5e0 2024-03-24T10:40:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eb5f37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5baba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eb4f35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baff00 0x55b4c5bb65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eb5736640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 0x55b4c5bc4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eb5f37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5baba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38132/0 (socket says 172.21.15.23:38132) 2024-03-24T10:40:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eb5f37640 1 -- 172.21.15.23:0/1640862943 learned_addr learned my addr 172.21.15.23:0/1640862943 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eb4f35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baff00 0x55b4c5bb65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:44550/0 (socket says 172.21.15.23:44550) 2024-03-24T10:40:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eb5f37640 1 -- 172.21.15.23:0/1640862943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 msgr2=0x55b4c5bc4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eb5f37640 1 --2- 172.21.15.23:0/1640862943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 0x55b4c5bc4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eb5f37640 1 -- 172.21.15.23:0/1640862943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baff00 msgr2=0x55b4c5bb65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eb5f37640 1 --2- 172.21.15.23:0/1640862943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baff00 0x55b4c5bb65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eb5f37640 1 -- 172.21.15.23:0/1640862943 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4c5ab0810 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eb5736640 1 --2- 172.21.15.23:0/1640862943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 0x55b4c5bc4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eb4f35640 1 --2- 172.21.15.23:0/1640862943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baff00 0x55b4c5bb65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2eb5f37640 1 --2- 172.21.15.23:0/1640862943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5baba10 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0006010 tx=0x7f2ea0032bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e321a7f160ae99c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2e96ffd640 1 -- 172.21.15.23:0/1640862943 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea004b410 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2e96ffd640 1 -- 172.21.15.23:0/1640862943 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ea0003e50 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.245+0000 7f2e96ffd640 1 -- 172.21.15.23:0/1640862943 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea003fe10 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.246+0000 7f2eba580740 1 -- 172.21.15.23:0/1640862943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5bb8cd0 msgr2=0x55b4c5baba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:40.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.246+0000 7f2eba580740 1 --2- 172.21.15.23:0/1640862943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5baba10 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0006010 tx=0x7f2ea0032bf0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.246+0000 7f2eba580740 1 -- 172.21.15.23:0/1640862943 shutdown_connections 2024-03-24T10:40:40.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.246+0000 7f2eba580740 1 --2- 172.21.15.23:0/1640862943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5baba10 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.246+0000 7f2eba580740 1 --2- 172.21.15.23:0/1640862943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5baff00 0x55b4c5bb65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.246+0000 7f2eba580740 1 --2- 172.21.15.23:0/1640862943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 0x55b4c5bc4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.246+0000 7f2eba580740 1 -- 172.21.15.23:0/1640862943 >> 172.21.15.23:0/1640862943 conn(0x55b4c5ba9f00 msgr2=0x55b4c5bab300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:40.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.246+0000 7f2eba580740 1 -- 172.21.15.23:0/1640862943 shutdown_connections 2024-03-24T10:40:40.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.246+0000 7f2eba580740 1 -- 172.21.15.23:0/1640862943 wait complete. 2024-03-24T10:40:40.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.246+0000 7f2eba580740 1 -- start start 2024-03-24T10:40:40.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eba580740 1 Processor -- start 2024-03-24T10:40:40.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eba580740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 0x55b4c5bc0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.251 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eba580740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5baff00 0x55b4c5bb2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eba580740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bb28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eba580740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b4c5bc1020 con 0x55b4c5baff00 2024-03-24T10:40:40.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eba580740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b4c5bc1190 con 0x55b4c5bb8cd0 2024-03-24T10:40:40.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eba580740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b4c5bb2e10 con 0x55b4c5baf5e0 2024-03-24T10:40:40.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eba580740 10 client.0 _ll_register_callbacks cb 0x7f2ea4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:40.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eb4f35640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5baff00 0x55b4c5bb2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eb4f35640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5baff00 0x55b4c5bb2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38134/0 (socket says 172.21.15.23:38134) 2024-03-24T10:40:40.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eb4f35640 1 -- 172.21.15.23:0/544756649 learned_addr learned my addr 172.21.15.23:0/544756649 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:40.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eb5736640 1 --2- 172.21.15.23:0/544756649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 0x55b4c5bc0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eb5f37640 1 --2- 172.21.15.23:0/544756649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bb28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eb4f35640 1 -- 172.21.15.23:0/544756649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 msgr2=0x55b4c5bc0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:40.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eb4f35640 1 --2- 172.21.15.23:0/544756649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 0x55b4c5bc0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eb4f35640 1 -- 172.21.15.23:0/544756649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5bb8cd0 msgr2=0x55b4c5bb28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eb4f35640 1 --2- 172.21.15.23:0/544756649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bb28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eb4f35640 1 -- 172.21.15.23:0/544756649 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ea8012120 con 0x55b4c5baff00 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2eb4f35640 1 --2- 172.21.15.23:0/544756649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5baff00 0x55b4c5bb2360 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40cc330 tx=0x7f2ea409a790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff48b9a8022253e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.247+0000 7f2e94ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2e94ff9640 1 -- 172.21.15.23:0/544756649 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea409eed0 con 0x55b4c5baff00 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2e94ff9640 1 -- 172.21.15.23:0/544756649 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f2ea40a3520 con 0x55b4c5baff00 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2eba580740 1 -- 172.21.15.23:0/544756649 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ea409ae30 con 0x55b4c5baff00 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2e94ff9640 1 -- 172.21.15.23:0/544756649 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2ea4096b80 con 0x55b4c5baff00 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2eba580740 10 client.15570 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2eba580740 1 -- 172.21.15.23:0/544756649 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b4c5aab810 con 0x55b4c5baff00 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2e94ff9640 1 -- 172.21.15.23:0/544756649 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ea409ae30 con 0x55b4c5baff00 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2e94ff9640 1 -- 172.21.15.23:0/544756649 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f2ea40cbe90 con 0x55b4c5baff00 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2eba580740 10 client.15570 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2eba580740 10 client.15570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2eba580740 20 client.15570 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:40.254 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2eba580740 1 --2- 172.21.15.23:0/544756649 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b4c5c2ef00 0x55b4c5c31320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:40.254 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2eba580740 4 client.15570 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:40:40.254 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2eba580740 1 -- 172.21.15.23:0/544756649 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b4c58661b0 con 0x55b4c5c2ef00 2024-03-24T10:40:40.254 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.248+0000 7f2eb5736640 1 --2- 172.21.15.23:0/544756649 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b4c5c2ef00 0x55b4c5c31320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:40.254 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.249+0000 7f2eb5736640 1 --2- 172.21.15.23:0/544756649 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b4c5c2ef00 0x55b4c5c31320 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd93de44f77e5e98 server_cookie=237345c53c06a19b in_seq=0 out_seq=0 2024-03-24T10:40:40.254 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.249+0000 7f2e94ff9640 10 client.15570 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:40.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.251+0000 7f2e94ff9640 1 -- 172.21.15.23:0/544756649 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b4c58661b0 con 0x55b4c5c2ef00 2024-03-24T10:40:40.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.251+0000 7f2e94ff9640 10 client.15570 handle_command_reply: tid=0 2024-03-24T10:40:40.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.251+0000 7f2eba580740 1 client.15570 shutdown 2024-03-24T10:40:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.252+0000 7f2eba580740 1 -- 172.21.15.23:0/544756649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5baff00 msgr2=0x55b4c5bb2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.252+0000 7f2eba580740 1 --2- 172.21.15.23:0/544756649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5baff00 0x55b4c5bb2360 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40cc330 tx=0x7f2ea409a790 comp rx=0 tx=0).stop 2024-03-24T10:40:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.252+0000 7f2eba580740 1 -- 172.21.15.23:0/544756649 shutdown_connections 2024-03-24T10:40:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.252+0000 7f2eba580740 1 --2- 172.21.15.23:0/544756649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4c5bb8cd0 0x55b4c5bb28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.252+0000 7f2eba580740 1 --2- 172.21.15.23:0/544756649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4c5baff00 0x55b4c5bb2360 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.252+0000 7f2eba580740 1 --2- 172.21.15.23:0/544756649 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b4c5c2ef00 0x55b4c5c31320 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.252+0000 7f2eba580740 1 --2- 172.21.15.23:0/544756649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4c5baf5e0 0x55b4c5bc0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.252+0000 7f2e94ff9640 0 client.15570 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.252+0000 7f2eba580740 1 -- 172.21.15.23:0/544756649 >> 172.21.15.23:0/544756649 conn(0x55b4c5ba9f00 msgr2=0x55b4c5babb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.252+0000 7f2eba580740 1 -- 172.21.15.23:0/544756649 shutdown_connections 2024-03-24T10:40:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.252+0000 7f2eba580740 1 -- 172.21.15.23:0/544756649 wait complete. 2024-03-24T10:40:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.252+0000 7f2eba580740 20 client.15570 trim_cache size 0 max 16384 2024-03-24T10:40:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:40:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:40:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:40:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:40:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.253+0000 7f2eb799a640 1 -- 172.21.15.23:0/2955094339 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2e980a23f0 msgr2=0x7f2e980a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.253+0000 7f2eb799a640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2e980a23f0 0x7f2e980a48b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0005cc0 tx=0x7f2ea002e040 comp rx=0 tx=0).stop 2024-03-24T10:40:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.253+0000 7f2eb799a640 1 -- 172.21.15.23:0/2955094339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0069730 msgr2=0x7f2eb0100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.253+0000 7f2eb799a640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0069730 0x7f2eb0100e70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400e9f0 tx=0x7f2ea400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.253+0000 7f2eb799a640 1 -- 172.21.15.23:0/2955094339 shutdown_connections 2024-03-24T10:40:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.253+0000 7f2eb799a640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2e980a23f0 0x7f2e980a48b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.253+0000 7f2eb799a640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2eb01077e0 0x7f2eb01013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.253+0000 7f2eb799a640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0069730 0x7f2eb0100e70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.253+0000 7f2eb799a640 1 --2- 172.21.15.23:0/2955094339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb0068d90 0x7f2eb0100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.253+0000 7f2eb799a640 1 -- 172.21.15.23:0/2955094339 >> 172.21.15.23:0/2955094339 conn(0x7f2eb0077460 msgr2=0x7f2eb00fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.253+0000 7f2eb799a640 1 -- 172.21.15.23:0/2955094339 shutdown_connections 2024-03-24T10:40:40.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:40.253+0000 7f2eb799a640 1 -- 172.21.15.23:0/2955094339 wait complete. 2024-03-24T10:40:40.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:40 smithi023 ceph-mon[32466]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-24T10:40:40.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:40 smithi117 ceph-mon[36368]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-24T10:40:40.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:40 smithi116 ceph-mon[36248]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-03-24T10:40:42.687 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:40:42.688 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:40:42.712 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:42 smithi023 ceph-mon[32466]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-03-24T10:40:42.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:42 smithi117 ceph-mon[36368]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-03-24T10:40:42.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:42 smithi116 ceph-mon[36248]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-03-24T10:40:42.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.794+0000 7fdd57f81640 1 -- 172.21.15.23:0/1693525103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd501005a0 msgr2=0x7fdd50100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:42.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.794+0000 7fdd57f81640 1 --2- 172.21.15.23:0/1693525103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd501005a0 0x7fdd50100a20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fdd40009900 tx=0x7fdd4002bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:42.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.794+0000 7fdd57f81640 1 -- 172.21.15.23:0/1693525103 shutdown_connections 2024-03-24T10:40:42.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.794+0000 7fdd57f81640 1 --2- 172.21.15.23:0/1693525103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd50100f60 0x7fdd5010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.794+0000 7fdd57f81640 1 --2- 172.21.15.23:0/1693525103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd501005a0 0x7fdd50100a20 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.794+0000 7fdd57f81640 1 --2- 172.21.15.23:0/1693525103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd50106660 0x7fdd50106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.794+0000 7fdd57f81640 1 -- 172.21.15.23:0/1693525103 >> 172.21.15.23:0/1693525103 conn(0x7fdd50077710 msgr2=0x7fdd50077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:42.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.794+0000 7fdd57f81640 1 -- 172.21.15.23:0/1693525103 shutdown_connections 2024-03-24T10:40:42.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.794+0000 7fdd57f81640 1 -- 172.21.15.23:0/1693525103 wait complete. 2024-03-24T10:40:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd57f81640 1 Processor -- start 2024-03-24T10:40:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd57f81640 1 -- start start 2024-03-24T10:40:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd57f81640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd501005a0 0x7fdd5019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd57f81640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd50100f60 0x7fdd5019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd57f81640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd50106660 0x7fdd50196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd57f81640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fdd501970c0 con 0x7fdd501005a0 2024-03-24T10:40:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd57f81640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdd50197230 con 0x7fdd50106660 2024-03-24T10:40:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd57f81640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdd501973f0 con 0x7fdd50100f60 2024-03-24T10:40:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd55d1d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd501005a0 0x7fdd5019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd55d1d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd501005a0 0x7fdd5019c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38156/0 (socket says 172.21.15.23:38156) 2024-03-24T10:40:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd55d1d640 1 -- 172.21.15.23:0/3310643720 learned_addr learned my addr 172.21.15.23:0/3310643720 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd5651e640 1 --2- 172.21.15.23:0/3310643720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd50106660 0x7fdd50196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd5551c640 1 --2- 172.21.15.23:0/3310643720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd50100f60 0x7fdd5019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd55d1d640 1 -- 172.21.15.23:0/3310643720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd50100f60 msgr2=0x7fdd5019ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd55d1d640 1 --2- 172.21.15.23:0/3310643720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd50100f60 0x7fdd5019ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd55d1d640 1 -- 172.21.15.23:0/3310643720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd50106660 msgr2=0x7fdd50196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd55d1d640 1 --2- 172.21.15.23:0/3310643720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd50106660 0x7fdd50196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.795+0000 7fdd55d1d640 1 -- 172.21.15.23:0/3310643720 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd40009560 con 0x7fdd501005a0 2024-03-24T10:40:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.796+0000 7fdd55d1d640 1 --2- 172.21.15.23:0/3310643720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd501005a0 0x7fdd5019c8c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00e9f0 tx=0x7fdd4c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.796+0000 7fdd46ffd640 1 -- 172.21.15.23:0/3310643720 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c009720 con 0x7fdd501005a0 2024-03-24T10:40:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.796+0000 7fdd46ffd640 1 -- 172.21.15.23:0/3310643720 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd4c00cd80 con 0x7fdd501005a0 2024-03-24T10:40:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.796+0000 7fdd57f81640 1 -- 172.21.15.23:0/3310643720 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd501976d0 con 0x7fdd501005a0 2024-03-24T10:40:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.796+0000 7fdd46ffd640 1 -- 172.21.15.23:0/3310643720 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c010690 con 0x7fdd501005a0 2024-03-24T10:40:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.796+0000 7fdd57f81640 1 -- 172.21.15.23:0/3310643720 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd501a3960 con 0x7fdd501005a0 2024-03-24T10:40:42.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.798+0000 7fdd46ffd640 1 -- 172.21.15.23:0/3310643720 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7fdd4c0046c0 con 0x7fdd501005a0 2024-03-24T10:40:42.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.798+0000 7fdd5ab64740 1 Processor -- start 2024-03-24T10:40:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5ab64740 1 -- start start 2024-03-24T10:40:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd46ffd640 1 --2- 172.21.15.23:0/3310643720 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdd3c0a2350 0x7fdd3c0a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5ab64740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef6461d0 0x55f9ef6465b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5ab64740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef646af0 0x55f9ef64f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5ab64740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef64f6d0 0x55f9ef651ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5ab64740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f9ef56af30 con 0x55f9ef646af0 2024-03-24T10:40:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5ab64740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f9ef5b4810 con 0x55f9ef6461d0 2024-03-24T10:40:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5ab64740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f9ef5b1d40 con 0x55f9ef64f6d0 2024-03-24T10:40:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd46ffd640 1 -- 172.21.15.23:0/3310643720 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fdd4c014070 con 0x7fdd501005a0 2024-03-24T10:40:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd55d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef6461d0 0x55f9ef6465b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5651e640 1 --2- 172.21.15.23:0/3310643720 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdd3c0a2350 0x7fdd3c0a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5551c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef646af0 0x55f9ef64f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5651e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef64f6d0 0x55f9ef651ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5651e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef64f6d0 0x55f9ef651ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:53122/0 (socket says 172.21.15.23:53122) 2024-03-24T10:40:42.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5551c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef646af0 0x55f9ef64f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38158/0 (socket says 172.21.15.23:38158) 2024-03-24T10:40:42.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5651e640 1 -- 172.21.15.23:0/189292990 learned_addr learned my addr 172.21.15.23:0/189292990 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:42.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5651e640 1 -- 172.21.15.23:0/189292990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef6461d0 msgr2=0x55f9ef6465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:42.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5651e640 1 --2- 172.21.15.23:0/189292990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef6461d0 0x55f9ef6465b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5651e640 1 -- 172.21.15.23:0/189292990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef646af0 msgr2=0x55f9ef64f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:42.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5651e640 1 --2- 172.21.15.23:0/189292990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef646af0 0x55f9ef64f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd5651e640 1 -- 172.21.15.23:0/189292990 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9ef56eea0 con 0x55f9ef64f6d0 2024-03-24T10:40:42.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.799+0000 7fdd55d1d640 1 --2- 172.21.15.23:0/189292990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef6461d0 0x55f9ef6465b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:42.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.800+0000 7fdd5651e640 1 --2- 172.21.15.23:0/189292990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef64f6d0 0x55f9ef651ac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdd48010e50 tx=0x7fdd4800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8334826e4bee4fec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:42.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.800+0000 7fdd5651e640 1 --2- 172.21.15.23:0/3310643720 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdd3c0a2350 0x7fdd3c0a4810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800fbd0 tx=0x7fdd48018000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:42.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.800+0000 7fdd3bfff640 1 -- 172.21.15.23:0/189292990 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd48009ea0 con 0x55f9ef64f6d0 2024-03-24T10:40:42.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.800+0000 7fdd3bfff640 1 -- 172.21.15.23:0/189292990 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd48010730 con 0x55f9ef64f6d0 2024-03-24T10:40:42.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.800+0000 7fdd3bfff640 1 -- 172.21.15.23:0/189292990 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd48005780 con 0x55f9ef64f6d0 2024-03-24T10:40:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.800+0000 7fdd5ab64740 1 -- 172.21.15.23:0/189292990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef64f6d0 msgr2=0x55f9ef651ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.800+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/189292990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef64f6d0 0x55f9ef651ac0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdd48010e50 tx=0x7fdd4800f790 comp rx=0 tx=0).stop 2024-03-24T10:40:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.800+0000 7fdd5ab64740 1 -- 172.21.15.23:0/189292990 shutdown_connections 2024-03-24T10:40:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.800+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/189292990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef64f6d0 0x55f9ef651ac0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.800+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/189292990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef646af0 0x55f9ef64f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.800+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/189292990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef6461d0 0x55f9ef6465b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.800+0000 7fdd5ab64740 1 -- 172.21.15.23:0/189292990 >> 172.21.15.23:0/189292990 conn(0x55f9ef640900 msgr2=0x55f9ef640ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.801+0000 7fdd5ab64740 1 -- 172.21.15.23:0/189292990 shutdown_connections 2024-03-24T10:40:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.801+0000 7fdd5ab64740 1 -- 172.21.15.23:0/189292990 wait complete. 2024-03-24T10:40:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.801+0000 7fdd5ab64740 1 -- start start 2024-03-24T10:40:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.801+0000 7fdd5ab64740 1 Processor -- start 2024-03-24T10:40:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.801+0000 7fdd5ab64740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef6461d0 0x55f9ef64c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.801+0000 7fdd5ab64740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef646af0 0x55f9ef64cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.801+0000 7fdd5ab64740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef64f6d0 0x55f9ef64d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.801+0000 7fdd5ab64740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f9ef5ad520 con 0x55f9ef6461d0 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.801+0000 7fdd5ab64740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f9ef280430 con 0x55f9ef64f6d0 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.801+0000 7fdd5ab64740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f9ef23ce00 con 0x55f9ef646af0 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.801+0000 7fdd5ab64740 10 client.0 _ll_register_callbacks cb 0x55f9ef54bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.801+0000 7fdd55d1d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef6461d0 0x55f9ef64c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.801+0000 7fdd55d1d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef6461d0 0x55f9ef64c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38164/0 (socket says 172.21.15.23:38164) 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.801+0000 7fdd55d1d640 1 -- 172.21.15.23:0/3044735159 learned_addr learned my addr 172.21.15.23:0/3044735159 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd5651e640 1 --2- 172.21.15.23:0/3044735159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef64f6d0 0x55f9ef64d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd5551c640 1 --2- 172.21.15.23:0/3044735159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef646af0 0x55f9ef64cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd55d1d640 1 -- 172.21.15.23:0/3044735159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef646af0 msgr2=0x55f9ef64cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd55d1d640 1 --2- 172.21.15.23:0/3044735159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef646af0 0x55f9ef64cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd55d1d640 1 -- 172.21.15.23:0/3044735159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef64f6d0 msgr2=0x55f9ef64d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd55d1d640 1 --2- 172.21.15.23:0/3044735159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef64f6d0 0x55f9ef64d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd55d1d640 1 -- 172.21.15.23:0/3044735159 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd48010a30 con 0x55f9ef6461d0 2024-03-24T10:40:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd5551c640 1 --2- 172.21.15.23:0/3044735159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef646af0 0x55f9ef64cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd55d1d640 1 --2- 172.21.15.23:0/3044735159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef6461d0 0x55f9ef64c7c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00d010 tx=0x7fdd4c09b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1127c3e1cebb9894 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd397fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd397fa640 1 -- 172.21.15.23:0/3044735159 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c08a690 con 0x55f9ef6461d0 2024-03-24T10:40:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd397fa640 1 -- 172.21.15.23:0/3044735159 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fdd4c0a35b0 con 0x55f9ef6461d0 2024-03-24T10:40:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd5ab64740 1 -- 172.21.15.23:0/3044735159 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f9ef2ac7e0 con 0x55f9ef6461d0 2024-03-24T10:40:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd397fa640 1 -- 172.21.15.23:0/3044735159 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdd4c0a3120 con 0x55f9ef6461d0 2024-03-24T10:40:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd5ab64740 10 client.15582 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd5ab64740 1 -- 172.21.15.23:0/3044735159 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55f9ef5abba0 con 0x55f9ef6461d0 2024-03-24T10:40:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.802+0000 7fdd397fa640 1 -- 172.21.15.23:0/3044735159 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f9ef2ac7e0 con 0x55f9ef6461d0 2024-03-24T10:40:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.803+0000 7fdd397fa640 1 -- 172.21.15.23:0/3044735159 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fdd4c01f060 con 0x55f9ef6461d0 2024-03-24T10:40:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.803+0000 7fdd5ab64740 10 client.15582 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.803+0000 7fdd5ab64740 10 client.15582 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.803+0000 7fdd5ab64740 20 client.15582 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.803+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/3044735159 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f9ef65fb40 0x55f9ef661f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.803+0000 7fdd5ab64740 4 client.15582 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.803+0000 7fdd5ab64740 1 -- 172.21.15.23:0/3044735159 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9ef2e9720 con 0x55f9ef65fb40 2024-03-24T10:40:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.803+0000 7fdd5551c640 1 --2- 172.21.15.23:0/3044735159 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f9ef65fb40 0x55f9ef661f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.803+0000 7fdd5551c640 1 --2- 172.21.15.23:0/3044735159 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f9ef65fb40 0x55f9ef661f50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7331b4ec3bf51045 server_cookie=dc7fa45e8e059321 in_seq=0 out_seq=0 2024-03-24T10:40:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.803+0000 7fdd397fa640 10 client.15582 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd397fa640 1 -- 172.21.15.23:0/3044735159 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f9ef2e9720 con 0x55f9ef65fb40 2024-03-24T10:40:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd397fa640 10 client.15582 handle_command_reply: tid=0 2024-03-24T10:40:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd5ab64740 1 client.15582 shutdown 2024-03-24T10:40:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd5ab64740 1 -- 172.21.15.23:0/3044735159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef6461d0 msgr2=0x55f9ef64c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/3044735159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef6461d0 0x55f9ef64c7c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00d010 tx=0x7fdd4c09b650 comp rx=0 tx=0).stop 2024-03-24T10:40:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd5ab64740 1 -- 172.21.15.23:0/3044735159 shutdown_connections 2024-03-24T10:40:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/3044735159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef64f6d0 0x55f9ef64d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/3044735159 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f9ef65fb40 0x55f9ef661f50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/3044735159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef646af0 0x55f9ef64cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/3044735159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef6461d0 0x55f9ef64c7c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd5ab64740 1 -- 172.21.15.23:0/3044735159 >> 172.21.15.23:0/3044735159 conn(0x55f9ef640900 msgr2=0x55f9ef642650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd397fa640 0 client.15582 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd5ab64740 1 -- 172.21.15.23:0/3044735159 shutdown_connections 2024-03-24T10:40:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd5ab64740 1 -- 172.21.15.23:0/3044735159 wait complete. 2024-03-24T10:40:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.804+0000 7fdd5ab64740 20 client.15582 trim_cache size 0 max 16384 2024-03-24T10:40:42.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.827+0000 7fdd5ab64740 1 Processor -- start 2024-03-24T10:40:42.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5ab64740 1 -- start start 2024-03-24T10:40:42.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5ab64740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef6461d0 0x55f9ef65af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5ab64740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef646af0 0x55f9ef657720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5ab64740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef64f6d0 0x55f9ef667800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5ab64740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f9ef532430 con 0x55f9ef64f6d0 2024-03-24T10:40:42.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5ab64740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f9ef645c40 con 0x55f9ef646af0 2024-03-24T10:40:42.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5ab64740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f9ef65b4c0 con 0x55f9ef6461d0 2024-03-24T10:40:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5651e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef64f6d0 0x55f9ef667800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd55d1d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef6461d0 0x55f9ef65af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5651e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef64f6d0 0x55f9ef667800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38180/0 (socket says 172.21.15.23:38180) 2024-03-24T10:40:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5651e640 1 -- 172.21.15.23:0/2654861475 learned_addr learned my addr 172.21.15.23:0/2654861475 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5551c640 1 --2- 172.21.15.23:0/2654861475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef646af0 0x55f9ef657720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5651e640 1 -- 172.21.15.23:0/2654861475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef6461d0 msgr2=0x55f9ef65af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5651e640 1 --2- 172.21.15.23:0/2654861475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef6461d0 0x55f9ef65af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5651e640 1 -- 172.21.15.23:0/2654861475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef646af0 msgr2=0x55f9ef657720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5651e640 1 --2- 172.21.15.23:0/2654861475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef646af0 0x55f9ef657720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5651e640 1 -- 172.21.15.23:0/2654861475 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9ef547b20 con 0x55f9ef64f6d0 2024-03-24T10:40:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd55d1d640 1 --2- 172.21.15.23:0/2654861475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef6461d0 0x55f9ef65af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5551c640 1 --2- 172.21.15.23:0/2654861475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef646af0 0x55f9ef657720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd5651e640 1 --2- 172.21.15.23:0/2654861475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef64f6d0 0x55f9ef667800 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdd48005370 tx=0x7fdd480052e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68e07f49323c655 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.828+0000 7fdd3b7fe640 1 -- 172.21.15.23:0/2654861475 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4802a040 con 0x55f9ef64f6d0 2024-03-24T10:40:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.829+0000 7fdd3b7fe640 1 -- 172.21.15.23:0/2654861475 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd480059f0 con 0x55f9ef64f6d0 2024-03-24T10:40:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.829+0000 7fdd3b7fe640 1 -- 172.21.15.23:0/2654861475 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd48003bd0 con 0x55f9ef64f6d0 2024-03-24T10:40:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.829+0000 7fdd5ab64740 1 -- 172.21.15.23:0/2654861475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef64f6d0 msgr2=0x55f9ef667800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.829+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/2654861475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef64f6d0 0x55f9ef667800 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdd48005370 tx=0x7fdd480052e0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.829+0000 7fdd5ab64740 1 -- 172.21.15.23:0/2654861475 shutdown_connections 2024-03-24T10:40:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.829+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/2654861475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef64f6d0 0x55f9ef667800 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.829+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/2654861475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef646af0 0x55f9ef657720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.829+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/2654861475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef6461d0 0x55f9ef65af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.829+0000 7fdd5ab64740 1 -- 172.21.15.23:0/2654861475 >> 172.21.15.23:0/2654861475 conn(0x55f9ef640900 msgr2=0x55f9ef63fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.829+0000 7fdd5ab64740 1 -- 172.21.15.23:0/2654861475 shutdown_connections 2024-03-24T10:40:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.829+0000 7fdd5ab64740 1 -- 172.21.15.23:0/2654861475 wait complete. 2024-03-24T10:40:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.829+0000 7fdd5ab64740 1 -- start start 2024-03-24T10:40:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5ab64740 1 Processor -- start 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5ab64740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef6461d0 0x55f9ef649d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5ab64740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef646af0 0x55f9ef64a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5ab64740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef64f6d0 0x55f9ef648a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5ab64740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f9ef649040 con 0x55f9ef646af0 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5ab64740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f9ef6491b0 con 0x55f9ef6461d0 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5ab64740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f9ef649370 con 0x55f9ef64f6d0 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5ab64740 10 client.0 _ll_register_callbacks cb 0x7fdd4c09a770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5551c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef646af0 0x55f9ef64a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5551c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef646af0 0x55f9ef64a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38190/0 (socket says 172.21.15.23:38190) 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5551c640 1 -- 172.21.15.23:0/320930798 learned_addr learned my addr 172.21.15.23:0/320930798 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5651e640 1 --2- 172.21.15.23:0/320930798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef64f6d0 0x55f9ef648a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd55d1d640 1 --2- 172.21.15.23:0/320930798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef6461d0 0x55f9ef649d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5551c640 1 -- 172.21.15.23:0/320930798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef64f6d0 msgr2=0x55f9ef648a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5551c640 1 --2- 172.21.15.23:0/320930798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef64f6d0 0x55f9ef648a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5551c640 1 -- 172.21.15.23:0/320930798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef6461d0 msgr2=0x55f9ef649d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5551c640 1 --2- 172.21.15.23:0/320930798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef6461d0 0x55f9ef649d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.830+0000 7fdd5551c640 1 -- 172.21.15.23:0/320930798 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9ef5abba0 con 0x55f9ef646af0 2024-03-24T10:40:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd5551c640 1 --2- 172.21.15.23:0/320930798 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef646af0 0x55f9ef64a250 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdd40005ab0 tx=0x7fdd4004def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a8b305c4624213 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd397fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd397fa640 1 -- 172.21.15.23:0/320930798 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd40045040 con 0x55f9ef646af0 2024-03-24T10:40:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd397fa640 1 -- 172.21.15.23:0/320930798 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fdd4002cb60 con 0x55f9ef646af0 2024-03-24T10:40:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd5ab64740 1 -- 172.21.15.23:0/320930798 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f9ef338890 con 0x55f9ef646af0 2024-03-24T10:40:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd397fa640 1 -- 172.21.15.23:0/320930798 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdd4002dea0 con 0x55f9ef646af0 2024-03-24T10:40:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd5ab64740 10 client.15588 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd5ab64740 1 -- 172.21.15.23:0/320930798 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x7fdd5010dd20 con 0x55f9ef646af0 2024-03-24T10:40:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd397fa640 1 -- 172.21.15.23:0/320930798 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f9ef338890 con 0x55f9ef646af0 2024-03-24T10:40:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd397fa640 1 -- 172.21.15.23:0/320930798 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fdd4003c040 con 0x55f9ef646af0 2024-03-24T10:40:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd5ab64740 10 client.15588 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd5ab64740 10 client.15588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd5ab64740 20 client.15588 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/320930798 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f9ef6c9090 0x55f9ef6cb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd5ab64740 4 client.15588 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:40:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.831+0000 7fdd5ab64740 1 -- 172.21.15.23:0/320930798 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f9ef258700 con 0x55f9ef6c9090 2024-03-24T10:40:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd55d1d640 1 --2- 172.21.15.23:0/320930798 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f9ef6c9090 0x55f9ef6cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd397fa640 10 client.15588 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd55d1d640 1 --2- 172.21.15.23:0/320930798 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f9ef6c9090 0x55f9ef6cb480 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d05a936493a2cad server_cookie=14bdac4e0e000f79 in_seq=0 out_seq=0 2024-03-24T10:40:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd397fa640 1 -- 172.21.15.23:0/320930798 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f9ef258700 con 0x55f9ef6c9090 2024-03-24T10:40:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd397fa640 10 client.15588 handle_command_reply: tid=0 2024-03-24T10:40:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd5ab64740 1 client.15588 shutdown 2024-03-24T10:40:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd5ab64740 1 -- 172.21.15.23:0/320930798 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef646af0 msgr2=0x55f9ef64a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/320930798 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef646af0 0x55f9ef64a250 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdd40005ab0 tx=0x7fdd4004def0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd5ab64740 1 -- 172.21.15.23:0/320930798 shutdown_connections 2024-03-24T10:40:42.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/320930798 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f9ef6c9090 0x55f9ef6cb480 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/320930798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9ef64f6d0 0x55f9ef648a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/320930798 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9ef646af0 0x55f9ef64a250 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd397fa640 0 client.15588 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:42.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd5ab64740 1 --2- 172.21.15.23:0/320930798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9ef6461d0 0x55f9ef649d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.832+0000 7fdd5ab64740 1 -- 172.21.15.23:0/320930798 >> 172.21.15.23:0/320930798 conn(0x55f9ef640900 msgr2=0x55f9ef641a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:42.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd5ab64740 1 -- 172.21.15.23:0/320930798 shutdown_connections 2024-03-24T10:40:42.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd5ab64740 1 -- 172.21.15.23:0/320930798 wait complete. 2024-03-24T10:40:42.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd5ab64740 20 client.15588 trim_cache size 0 max 16384 2024-03-24T10:40:42.839 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:40:42.839 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:40:42.840 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:40:42.840 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:40:42.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd57f81640 1 -- 172.21.15.23:0/3310643720 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdd3c0a2350 msgr2=0x7fdd3c0a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd57f81640 1 --2- 172.21.15.23:0/3310643720 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdd3c0a2350 0x7fdd3c0a4810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800fbd0 tx=0x7fdd48018000 comp rx=0 tx=0).stop 2024-03-24T10:40:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd57f81640 1 -- 172.21.15.23:0/3310643720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd501005a0 msgr2=0x7fdd5019c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd57f81640 1 --2- 172.21.15.23:0/3310643720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd501005a0 0x7fdd5019c8c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00e9f0 tx=0x7fdd4c00edf0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd57f81640 1 -- 172.21.15.23:0/3310643720 shutdown_connections 2024-03-24T10:40:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd57f81640 1 --2- 172.21.15.23:0/3310643720 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdd3c0a2350 0x7fdd3c0a4810 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd57f81640 1 --2- 172.21.15.23:0/3310643720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd50106660 0x7fdd50196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd57f81640 1 --2- 172.21.15.23:0/3310643720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd50100f60 0x7fdd5019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd57f81640 1 --2- 172.21.15.23:0/3310643720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd501005a0 0x7fdd5019c8c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd57f81640 1 -- 172.21.15.23:0/3310643720 >> 172.21.15.23:0/3310643720 conn(0x7fdd50077710 msgr2=0x7fdd500ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd57f81640 1 -- 172.21.15.23:0/3310643720 shutdown_connections 2024-03-24T10:40:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.833+0000 7fdd57f81640 1 -- 172.21.15.23:0/3310643720 wait complete. 2024-03-24T10:40:42.844 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:40:42.844 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:40:42.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.988+0000 7f44d3781640 1 -- 172.21.15.23:0/574366677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44cc1032a0 msgr2=0x7f44cc10b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:42.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.988+0000 7f44d3781640 1 --2- 172.21.15.23:0/574366677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44cc1032a0 0x7f44cc10b780 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f44bc009900 tx=0x7f44bc02bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:42.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.988+0000 7f44d3781640 1 -- 172.21.15.23:0/574366677 shutdown_connections 2024-03-24T10:40:42.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.988+0000 7f44d3781640 1 --2- 172.21.15.23:0/574366677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44cc10bd50 0x7f44cc10e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.988+0000 7f44d3781640 1 --2- 172.21.15.23:0/574366677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44cc1032a0 0x7f44cc10b780 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.988+0000 7f44d3781640 1 --2- 172.21.15.23:0/574366677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44cc102980 0x7f44cc102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.988+0000 7f44d3781640 1 -- 172.21.15.23:0/574366677 >> 172.21.15.23:0/574366677 conn(0x7f44cc077710 msgr2=0x7f44cc077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:42.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.988+0000 7f44d3781640 1 -- 172.21.15.23:0/574366677 shutdown_connections 2024-03-24T10:40:42.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.988+0000 7f44d3781640 1 -- 172.21.15.23:0/574366677 wait complete. 2024-03-24T10:40:42.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.989+0000 7f44d3781640 1 Processor -- start 2024-03-24T10:40:42.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.989+0000 7f44d3781640 1 -- start start 2024-03-24T10:40:42.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.989+0000 7f44d3781640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44cc102980 0x7f44cc197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.989+0000 7f44d3781640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44cc1032a0 0x7f44cc198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.989+0000 7f44d3781640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44cc10bd50 0x7f44cc19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.989+0000 7f44d3781640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc19cc90 con 0x7f44cc1032a0 2024-03-24T10:40:42.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.989+0000 7f44d3781640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc19ce00 con 0x7f44cc102980 2024-03-24T10:40:42.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.989+0000 7f44d3781640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc19cfc0 con 0x7f44cc10bd50 2024-03-24T10:40:42.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.989+0000 7f44d0d1c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44cc1032a0 0x7f44cc198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.989+0000 7f44d0d1c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44cc1032a0 0x7f44cc198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38212/0 (socket says 172.21.15.23:38212) 2024-03-24T10:40:42.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.989+0000 7f44d0d1c640 1 -- 172.21.15.23:0/1224697665 learned_addr learned my addr 172.21.15.23:0/1224697665 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:42.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.989+0000 7f44d1d1e640 1 --2- 172.21.15.23:0/1224697665 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44cc10bd50 0x7f44cc19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.989+0000 7f44d151d640 1 --2- 172.21.15.23:0/1224697665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44cc102980 0x7f44cc197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.990+0000 7f44d0d1c640 1 -- 172.21.15.23:0/1224697665 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44cc10bd50 msgr2=0x7f44cc19c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:42.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.990+0000 7f44d0d1c640 1 --2- 172.21.15.23:0/1224697665 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44cc10bd50 0x7f44cc19c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.990+0000 7f44d0d1c640 1 -- 172.21.15.23:0/1224697665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44cc102980 msgr2=0x7f44cc197dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:42.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.990+0000 7f44d0d1c640 1 --2- 172.21.15.23:0/1224697665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44cc102980 0x7f44cc197dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:42.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.990+0000 7f44d0d1c640 1 -- 172.21.15.23:0/1224697665 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44c4009840 con 0x7f44cc1032a0 2024-03-24T10:40:42.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.990+0000 7f44d0d1c640 1 --2- 172.21.15.23:0/1224697665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44cc1032a0 0x7f44cc198310 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f44bc005940 tx=0x7f44bc02c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:42.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.990+0000 7f44c27fc640 1 -- 172.21.15.23:0/1224697665 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc004350 con 0x7f44cc1032a0 2024-03-24T10:40:42.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.990+0000 7f44d3781640 1 -- 172.21.15.23:0/1224697665 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44bc009560 con 0x7f44cc1032a0 2024-03-24T10:40:42.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.990+0000 7f44c27fc640 1 -- 172.21.15.23:0/1224697665 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44bc0044b0 con 0x7f44cc1032a0 2024-03-24T10:40:42.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.990+0000 7f44d3781640 1 -- 172.21.15.23:0/1224697665 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44cc1a4940 con 0x7f44cc1032a0 2024-03-24T10:40:42.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.990+0000 7f44c27fc640 1 -- 172.21.15.23:0/1224697665 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc003bc0 con 0x7f44cc1032a0 2024-03-24T10:40:42.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.991+0000 7f44d3781640 1 -- 172.21.15.23:0/1224697665 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44cc068500 con 0x7f44cc1032a0 2024-03-24T10:40:42.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.992+0000 7f44c27fc640 1 -- 172.21.15.23:0/1224697665 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7f44bc003d20 con 0x7f44cc1032a0 2024-03-24T10:40:42.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.993+0000 7f44c27fc640 1 --2- 172.21.15.23:0/1224697665 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f44b80a2350 0x7f44b80a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:42.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.993+0000 7f44d151d640 1 --2- 172.21.15.23:0/1224697665 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f44b80a2350 0x7f44b80a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:42.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.993+0000 7f44c27fc640 1 -- 172.21.15.23:0/1224697665 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f44bc0f0570 con 0x7f44cc1032a0 2024-03-24T10:40:42.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.993+0000 7f44d151d640 1 --2- 172.21.15.23:0/1224697665 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f44b80a2350 0x7f44b80a4810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009bd0 tx=0x7f44c8005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:42.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:42.998+0000 7f44c27fc640 1 -- 172.21.15.23:0/1224697665 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f44bc0b90c0 con 0x7f44cc1032a0 2024-03-24T10:40:43.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.274+0000 7f44d3781640 1 -- 172.21.15.23:0/1224697665 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44cc106e40 con 0x7f44cc1032a0 2024-03-24T10:40:43.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.275+0000 7f44c27fc640 1 -- 172.21.15.23:0/1224697665 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11866 (secure 0 0 0) 0x7f44bc0bc780 con 0x7f44cc1032a0 2024-03-24T10:40:43.275 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:40:43.275 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:40:03.697229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24539,24503,24505]},"id":1}]} 2024-03-24T10:40:43.275 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 23 2024-03-24T10:40:43.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.276+0000 7f44d3781640 1 -- 172.21.15.23:0/1224697665 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f44b80a2350 msgr2=0x7f44b80a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.277+0000 7f44d3781640 1 --2- 172.21.15.23:0/1224697665 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f44b80a2350 0x7f44b80a4810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009bd0 tx=0x7f44c8005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:43.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.277+0000 7f44d3781640 1 -- 172.21.15.23:0/1224697665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44cc1032a0 msgr2=0x7f44cc198310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.277+0000 7f44d3781640 1 --2- 172.21.15.23:0/1224697665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44cc1032a0 0x7f44cc198310 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f44bc005940 tx=0x7f44bc02c670 comp rx=0 tx=0).stop 2024-03-24T10:40:43.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.277+0000 7f44d3781640 1 -- 172.21.15.23:0/1224697665 shutdown_connections 2024-03-24T10:40:43.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.277+0000 7f44d3781640 1 --2- 172.21.15.23:0/1224697665 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f44b80a2350 0x7f44b80a4810 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.277+0000 7f44d3781640 1 --2- 172.21.15.23:0/1224697665 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44cc10bd50 0x7f44cc19c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.277+0000 7f44d3781640 1 --2- 172.21.15.23:0/1224697665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44cc1032a0 0x7f44cc198310 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.277+0000 7f44d3781640 1 --2- 172.21.15.23:0/1224697665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44cc102980 0x7f44cc197dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.277+0000 7f44d3781640 1 -- 172.21.15.23:0/1224697665 >> 172.21.15.23:0/1224697665 conn(0x7f44cc077710 msgr2=0x7f44cc0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.277+0000 7f44d3781640 1 -- 172.21.15.23:0/1224697665 shutdown_connections 2024-03-24T10:40:43.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.277+0000 7f44d3781640 1 -- 172.21.15.23:0/1224697665 wait complete. 2024-03-24T10:40:43.288 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:40:43.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.395+0000 7fb78759a640 1 -- 172.21.15.23:0/2071161298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb780068910 msgr2=0x7fb780068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.395+0000 7fb78759a640 1 --2- 172.21.15.23:0/2071161298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb780068910 0x7fb780068d90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb774009930 tx=0x7fb77402bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:43.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.395+0000 7fb78759a640 1 -- 172.21.15.23:0/2071161298 shutdown_connections 2024-03-24T10:40:43.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.395+0000 7fb78759a640 1 --2- 172.21.15.23:0/2071161298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7800692d0 0x7fb78010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.395+0000 7fb78759a640 1 --2- 172.21.15.23:0/2071161298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb780068910 0x7fb780068d90 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.395+0000 7fb78759a640 1 --2- 172.21.15.23:0/2071161298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb780107760 0x7fb780107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.395+0000 7fb78759a640 1 -- 172.21.15.23:0/2071161298 >> 172.21.15.23:0/2071161298 conn(0x7fb780077710 msgr2=0x7fb780077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.396+0000 7fb78759a640 1 -- 172.21.15.23:0/2071161298 shutdown_connections 2024-03-24T10:40:43.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.396+0000 7fb78759a640 1 -- 172.21.15.23:0/2071161298 wait complete. 2024-03-24T10:40:43.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.396+0000 7fb78759a640 1 Processor -- start 2024-03-24T10:40:43.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.396+0000 7fb78759a640 1 -- start start 2024-03-24T10:40:43.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.396+0000 7fb78759a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb780068910 0x7fb7801005d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.396+0000 7fb78759a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7800692d0 0x7fb780100b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.396+0000 7fb78759a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780107760 0x7fb780101050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.396+0000 7fb78759a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb7801a6100 con 0x7fb7800692d0 2024-03-24T10:40:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.396+0000 7fb78759a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb7801a6270 con 0x7fb780107760 2024-03-24T10:40:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.396+0000 7fb78759a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb7801a6430 con 0x7fb780068910 2024-03-24T10:40:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.396+0000 7fb784b35640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7800692d0 0x7fb780100b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb784b35640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7800692d0 0x7fb780100b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38236/0 (socket says 172.21.15.23:38236) 2024-03-24T10:40:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb784b35640 1 -- 172.21.15.23:0/2871443361 learned_addr learned my addr 172.21.15.23:0/2871443361 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb785336640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb780068910 0x7fb7801005d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb785b37640 1 --2- 172.21.15.23:0/2871443361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780107760 0x7fb780101050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb784b35640 1 -- 172.21.15.23:0/2871443361 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb780068910 msgr2=0x7fb7801005d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb784b35640 1 --2- 172.21.15.23:0/2871443361 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb780068910 0x7fb7801005d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb784b35640 1 -- 172.21.15.23:0/2871443361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780107760 msgr2=0x7fb780101050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb784b35640 1 --2- 172.21.15.23:0/2871443361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780107760 0x7fb780101050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb784b35640 1 -- 172.21.15.23:0/2871443361 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb770009840 con 0x7fb7800692d0 2024-03-24T10:40:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb784b35640 1 --2- 172.21.15.23:0/2871443361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7800692d0 0x7fb780100b10 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb774009230 tx=0x7fb774004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb77e7fc640 1 -- 172.21.15.23:0/2871443361 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7740042d0 con 0x7fb7800692d0 2024-03-24T10:40:43.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb77e7fc640 1 -- 172.21.15.23:0/2871443361 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb774004430 con 0x7fb7800692d0 2024-03-24T10:40:43.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb78759a640 1 -- 172.21.15.23:0/2871443361 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb778009490 con 0x7fb7800692d0 2024-03-24T10:40:43.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb77e7fc640 1 -- 172.21.15.23:0/2871443361 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774003b50 con 0x7fb7800692d0 2024-03-24T10:40:43.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.397+0000 7fb78759a640 1 -- 172.21.15.23:0/2871443361 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb774009560 con 0x7fb7800692d0 2024-03-24T10:40:43.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.400+0000 7fb77e7fc640 1 -- 172.21.15.23:0/2871443361 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7fb774005460 con 0x7fb7800692d0 2024-03-24T10:40:43.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.400+0000 7fb78a178740 1 Processor -- start 2024-03-24T10:40:43.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.400+0000 7fb78a178740 1 -- start start 2024-03-24T10:40:43.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb78a178740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603cc3f0 0x55b2603cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb78a178740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 0x55b2603d53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb78a178740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603d58f0 0x55b2603d7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb78a178740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b26033a840 con 0x55b2603d58f0 2024-03-24T10:40:43.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb78a178740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b260337d70 con 0x55b2603cc3f0 2024-03-24T10:40:43.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb78a178740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b260333550 con 0x55b2603ccd10 2024-03-24T10:40:43.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb77e7fc640 1 --2- 172.21.15.23:0/2871443361 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7680a2350 0x7fb7680a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb785336640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 0x55b2603d53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb784b35640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603d58f0 0x55b2603d7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb785336640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 0x55b2603d53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:53188/0 (socket says 172.21.15.23:53188) 2024-03-24T10:40:43.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb785336640 1 -- 172.21.15.23:0/2936136517 learned_addr learned my addr 172.21.15.23:0/2936136517 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb785b37640 1 --2- 172.21.15.23:0/2936136517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603cc3f0 0x55b2603cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb785b37640 1 --2- 172.21.15.23:0/2871443361 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7680a2350 0x7fb7680a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb77e7fc640 1 -- 172.21.15.23:0/2871443361 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fb7740f0300 con 0x7fb7800692d0 2024-03-24T10:40:43.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb785336640 1 -- 172.21.15.23:0/2936136517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603cc3f0 msgr2=0x55b2603cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb785336640 1 --2- 172.21.15.23:0/2936136517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603cc3f0 0x55b2603cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb785336640 1 -- 172.21.15.23:0/2936136517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603d58f0 msgr2=0x55b2603d7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb785336640 1 --2- 172.21.15.23:0/2936136517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603d58f0 0x55b2603d7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb785336640 1 -- 172.21.15.23:0/2936136517 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2602c8de0 con 0x55b2603ccd10 2024-03-24T10:40:43.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb785b37640 1 --2- 172.21.15.23:0/2936136517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603cc3f0 0x55b2603cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb785b37640 1 --2- 172.21.15.23:0/2871443361 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7680a2350 0x7fb7680a4810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb770009b80 tx=0x7fb770009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.401+0000 7fb785336640 1 --2- 172.21.15.23:0/2936136517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 0x55b2603d53b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb778009e00 tx=0x7fb778011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba5c73ff2e1812b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.402+0000 7fb7677fe640 1 -- 172.21.15.23:0/2936136517 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb778004230 con 0x55b2603ccd10 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.402+0000 7fb7677fe640 1 -- 172.21.15.23:0/2936136517 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb778004390 con 0x55b2603ccd10 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.402+0000 7fb7677fe640 1 -- 172.21.15.23:0/2936136517 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7780197c0 con 0x55b2603ccd10 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.402+0000 7fb78a178740 1 -- 172.21.15.23:0/2936136517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 msgr2=0x55b2603d53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.402+0000 7fb78a178740 1 --2- 172.21.15.23:0/2936136517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 0x55b2603d53b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb778009e00 tx=0x7fb778011810 comp rx=0 tx=0).stop 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.402+0000 7fb78a178740 1 -- 172.21.15.23:0/2936136517 shutdown_connections 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.402+0000 7fb78a178740 1 --2- 172.21.15.23:0/2936136517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603d58f0 0x55b2603d7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.402+0000 7fb78a178740 1 --2- 172.21.15.23:0/2936136517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 0x55b2603d53b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.402+0000 7fb78a178740 1 --2- 172.21.15.23:0/2936136517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603cc3f0 0x55b2603cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.402+0000 7fb78a178740 1 -- 172.21.15.23:0/2936136517 >> 172.21.15.23:0/2936136517 conn(0x55b2603c6b20 msgr2=0x55b2603c6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb78a178740 1 -- 172.21.15.23:0/2936136517 shutdown_connections 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb78a178740 1 -- 172.21.15.23:0/2936136517 wait complete. 2024-03-24T10:40:43.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb78a178740 1 -- start start 2024-03-24T10:40:43.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb78a178740 1 Processor -- start 2024-03-24T10:40:43.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb78a178740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603cc3f0 0x55b2603d2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb78a178740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 0x55b2603d2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb78a178740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603d58f0 0x55b2603d3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb78a178740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b260006430 con 0x55b2603cc3f0 2024-03-24T10:40:43.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb78a178740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b25ffc2e00 con 0x55b2603d58f0 2024-03-24T10:40:43.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb78a178740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b2602b8590 con 0x55b2603ccd10 2024-03-24T10:40:43.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb78a178740 10 client.0 _ll_register_callbacks cb 0x55b260327fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:43.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb785336640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603cc3f0 0x55b2603d2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb785336640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603cc3f0 0x55b2603d2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38252/0 (socket says 172.21.15.23:38252) 2024-03-24T10:40:43.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb785336640 1 -- 172.21.15.23:0/1866429137 learned_addr learned my addr 172.21.15.23:0/1866429137 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.403+0000 7fb785b37640 1 --2- 172.21.15.23:0/1866429137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603d58f0 0x55b2603d3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb784b35640 1 --2- 172.21.15.23:0/1866429137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 0x55b2603d2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb785336640 1 -- 172.21.15.23:0/1866429137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 msgr2=0x55b2603d2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb785336640 1 --2- 172.21.15.23:0/1866429137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 0x55b2603d2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb785336640 1 -- 172.21.15.23:0/1866429137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603d58f0 msgr2=0x55b2603d3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb785336640 1 --2- 172.21.15.23:0/1866429137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603d58f0 0x55b2603d3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb785336640 1 -- 172.21.15.23:0/1866429137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb778009a60 con 0x55b2603cc3f0 2024-03-24T10:40:43.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb785336640 1 --2- 172.21.15.23:0/1866429137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603cc3f0 0x55b2603d2910 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb778009460 tx=0x7fb778003dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da7450d92b021bf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb764ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:43.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb764ff9640 1 -- 172.21.15.23:0/1866429137 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb778005260 con 0x55b2603cc3f0 2024-03-24T10:40:43.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb764ff9640 1 -- 172.21.15.23:0/1866429137 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fb7780053c0 con 0x55b2603cc3f0 2024-03-24T10:40:43.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb78a178740 1 -- 172.21.15.23:0/1866429137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b2600327e0 con 0x55b2603cc3f0 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb764ff9640 1 -- 172.21.15.23:0/1866429137 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb778005580 con 0x55b2603cc3f0 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb78a178740 10 client.15606 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb78a178740 1 -- 172.21.15.23:0/1866429137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b260331bd0 con 0x55b2603cc3f0 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.404+0000 7fb764ff9640 1 -- 172.21.15.23:0/1866429137 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b2600327e0 con 0x55b2603cc3f0 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.405+0000 7fb764ff9640 1 -- 172.21.15.23:0/1866429137 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb778004c90 con 0x55b2603cc3f0 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.405+0000 7fb78a178740 10 client.15606 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.405+0000 7fb78a178740 10 client.15606 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.405+0000 7fb78a178740 20 client.15606 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.405+0000 7fb78a178740 1 --2- 172.21.15.23:0/1866429137 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55b2603e59e0 0x55b2603e7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.405+0000 7fb78a178740 4 client.15606 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.405+0000 7fb78a178740 1 -- 172.21.15.23:0/1866429137 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b26006f720 con 0x55b2603e59e0 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.405+0000 7fb784b35640 1 --2- 172.21.15.23:0/1866429137 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55b2603e59e0 0x55b2603e7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.405+0000 7fb784b35640 1 --2- 172.21.15.23:0/1866429137 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55b2603e59e0 0x55b2603e7df0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=75f1717ca4df44b6 server_cookie=612a56f440bdaa71 in_seq=0 out_seq=0 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.405+0000 7fb764ff9640 10 client.15606 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:40:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb764ff9640 1 -- 172.21.15.23:0/1866429137 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b26006f720 con 0x55b2603e59e0 2024-03-24T10:40:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb764ff9640 10 client.15606 handle_command_reply: tid=0 2024-03-24T10:40:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb78a178740 1 client.15606 shutdown 2024-03-24T10:40:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb78a178740 1 -- 172.21.15.23:0/1866429137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603cc3f0 msgr2=0x55b2603d2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb78a178740 1 --2- 172.21.15.23:0/1866429137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603cc3f0 0x55b2603d2910 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb778009460 tx=0x7fb778003dd0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb78a178740 1 -- 172.21.15.23:0/1866429137 shutdown_connections 2024-03-24T10:40:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb78a178740 1 --2- 172.21.15.23:0/1866429137 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55b2603e59e0 0x55b2603e7df0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb78a178740 1 --2- 172.21.15.23:0/1866429137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603d58f0 0x55b2603d3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb78a178740 1 --2- 172.21.15.23:0/1866429137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 0x55b2603d2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb78a178740 1 --2- 172.21.15.23:0/1866429137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603cc3f0 0x55b2603d2910 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb78a178740 1 -- 172.21.15.23:0/1866429137 >> 172.21.15.23:0/1866429137 conn(0x55b2603c6b20 msgr2=0x55b2603c8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb764ff9640 0 client.15606 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:40:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb78a178740 1 -- 172.21.15.23:0/1866429137 shutdown_connections 2024-03-24T10:40:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb78a178740 1 -- 172.21.15.23:0/1866429137 wait complete. 2024-03-24T10:40:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.406+0000 7fb78a178740 20 client.15606 trim_cache size 0 max 16384 2024-03-24T10:40:43.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb78a178740 1 Processor -- start 2024-03-24T10:40:43.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb78a178740 1 -- start start 2024-03-24T10:40:43.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb78a178740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603cc3f0 0x55b2603d4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb78a178740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603ccd10 0x55b2603d11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb78a178740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603d58f0 0x55b2603e1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb78a178740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b2603cc050 con 0x55b2603d58f0 2024-03-24T10:40:43.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb78a178740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b2603cb4e0 con 0x55b2603ccd10 2024-03-24T10:40:43.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb78a178740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b2603c7b20 con 0x55b2603cc3f0 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb785b37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603d58f0 0x55b2603e1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb785b37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603d58f0 0x55b2603e1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38262/0 (socket says 172.21.15.23:38262) 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb785b37640 1 -- 172.21.15.23:0/178247542 learned_addr learned my addr 172.21.15.23:0/178247542 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb784b35640 1 --2- 172.21.15.23:0/178247542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603ccd10 0x55b2603d11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb785336640 1 --2- 172.21.15.23:0/178247542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603cc3f0 0x55b2603d4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb785b37640 1 -- 172.21.15.23:0/178247542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603cc3f0 msgr2=0x55b2603d4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb785b37640 1 --2- 172.21.15.23:0/178247542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603cc3f0 0x55b2603d4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb785b37640 1 -- 172.21.15.23:0/178247542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603ccd10 msgr2=0x55b2603d11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb785b37640 1 --2- 172.21.15.23:0/178247542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603ccd10 0x55b2603d11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.420+0000 7fb785b37640 1 -- 172.21.15.23:0/178247542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2602cdc80 con 0x55b2603d58f0 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.421+0000 7fb785b37640 1 --2- 172.21.15.23:0/178247542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603d58f0 0x55b2603e1180 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb77000fdc0 tx=0x7fb770009af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f98dafe1a4c197 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.421+0000 7fb766ffd640 1 -- 172.21.15.23:0/178247542 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770004d20 con 0x55b2603d58f0 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.421+0000 7fb766ffd640 1 -- 172.21.15.23:0/178247542 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb770004e80 con 0x55b2603d58f0 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.421+0000 7fb766ffd640 1 -- 172.21.15.23:0/178247542 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7700055c0 con 0x55b2603d58f0 2024-03-24T10:40:43.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.421+0000 7fb78a178740 1 -- 172.21.15.23:0/178247542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603d58f0 msgr2=0x55b2603e1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.421+0000 7fb78a178740 1 --2- 172.21.15.23:0/178247542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603d58f0 0x55b2603e1180 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb77000fdc0 tx=0x7fb770009af0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.421+0000 7fb78a178740 1 -- 172.21.15.23:0/178247542 shutdown_connections 2024-03-24T10:40:43.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.421+0000 7fb78a178740 1 --2- 172.21.15.23:0/178247542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603d58f0 0x55b2603e1180 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.421+0000 7fb78a178740 1 --2- 172.21.15.23:0/178247542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603ccd10 0x55b2603d11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.421+0000 7fb78a178740 1 --2- 172.21.15.23:0/178247542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603cc3f0 0x55b2603d4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.421+0000 7fb78a178740 1 -- 172.21.15.23:0/178247542 >> 172.21.15.23:0/178247542 conn(0x55b2603c6b20 msgr2=0x55b2603c5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.421+0000 7fb78a178740 1 -- 172.21.15.23:0/178247542 shutdown_connections 2024-03-24T10:40:43.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.421+0000 7fb78a178740 1 -- 172.21.15.23:0/178247542 wait complete. 2024-03-24T10:40:43.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.422+0000 7fb78a178740 1 -- start start 2024-03-24T10:40:43.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.422+0000 7fb78a178740 1 Processor -- start 2024-03-24T10:40:43.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.422+0000 7fb78a178740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603cc3f0 0x55b2603dd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.422+0000 7fb78a178740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 0x55b2603da780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.422+0000 7fb78a178740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603d58f0 0x55b2603dacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.422+0000 7fb78a178740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b2603db200 con 0x55b2603cc3f0 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.422+0000 7fb78a178740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b2603db370 con 0x55b2603d58f0 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.422+0000 7fb78a178740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b2603db530 con 0x55b2603ccd10 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.422+0000 7fb78a178740 10 client.0 _ll_register_callbacks cb 0x7fb770011520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.422+0000 7fb785336640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603cc3f0 0x55b2603dd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.422+0000 7fb785336640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603cc3f0 0x55b2603dd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38268/0 (socket says 172.21.15.23:38268) 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.422+0000 7fb785336640 1 -- 172.21.15.23:0/1915924947 learned_addr learned my addr 172.21.15.23:0/1915924947 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.422+0000 7fb784b35640 1 --2- 172.21.15.23:0/1915924947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 0x55b2603da780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.422+0000 7fb785b37640 1 --2- 172.21.15.23:0/1915924947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603d58f0 0x55b2603dacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.423+0000 7fb785336640 1 -- 172.21.15.23:0/1915924947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 msgr2=0x55b2603da780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.423+0000 7fb785336640 1 --2- 172.21.15.23:0/1915924947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 0x55b2603da780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.423+0000 7fb785336640 1 -- 172.21.15.23:0/1915924947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603d58f0 msgr2=0x55b2603dacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.423+0000 7fb785336640 1 --2- 172.21.15.23:0/1915924947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603d58f0 0x55b2603dacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.423+0000 7fb785336640 1 -- 172.21.15.23:0/1915924947 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb770019040 con 0x55b2603cc3f0 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.423+0000 7fb785336640 1 --2- 172.21.15.23:0/1915924947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603cc3f0 0x55b2603dd9a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb7780195b0 tx=0x7fb778003e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de2b38ba568eac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.423+0000 7fb764ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.423+0000 7fb764ff9640 1 -- 172.21.15.23:0/1915924947 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb778019ed0 con 0x55b2603cc3f0 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.423+0000 7fb764ff9640 1 -- 172.21.15.23:0/1915924947 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fb778000e40 con 0x55b2603cc3f0 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.423+0000 7fb78a178740 1 -- 172.21.15.23:0/1915924947 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b2600be890 con 0x55b2603cc3f0 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.423+0000 7fb764ff9640 1 -- 172.21.15.23:0/1915924947 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb7780193e0 con 0x55b2603cc3f0 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.423+0000 7fb78a178740 10 client.15612 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.423+0000 7fb78a178740 1 -- 172.21.15.23:0/1915924947 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b2602f3920 con 0x55b2603cc3f0 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.423+0000 7fb764ff9640 1 -- 172.21.15.23:0/1915924947 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb778043020 con 0x55b2603cc3f0 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.424+0000 7fb764ff9640 1 -- 172.21.15.23:0/1915924947 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb778003950 con 0x55b2603cc3f0 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.424+0000 7fb78a178740 10 client.15612 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.424+0000 7fb78a178740 10 client.15612 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.424+0000 7fb78a178740 20 client.15612 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.424+0000 7fb78a178740 1 --2- 172.21.15.23:0/1915924947 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55b26044d040 0x55b26044f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.424+0000 7fb78a178740 4 client.15612 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:40:43.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.424+0000 7fb78a178740 1 -- 172.21.15.23:0/1915924947 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b25ffde700 con 0x55b26044d040 2024-03-24T10:40:43.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.424+0000 7fb784b35640 1 --2- 172.21.15.23:0/1915924947 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55b26044d040 0x55b26044f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.424+0000 7fb764ff9640 10 client.15612 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:40:43.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.424+0000 7fb784b35640 1 --2- 172.21.15.23:0/1915924947 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55b26044d040 0x55b26044f460 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=27b25167b9342a6f server_cookie=ee9d1a5b5dea37d8 in_seq=0 out_seq=0 2024-03-24T10:40:43.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.424+0000 7fb764ff9640 1 -- 172.21.15.23:0/1915924947 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b25ffde700 con 0x55b26044d040 2024-03-24T10:40:43.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.424+0000 7fb764ff9640 10 client.15612 handle_command_reply: tid=0 2024-03-24T10:40:43.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.424+0000 7fb78a178740 1 client.15612 shutdown 2024-03-24T10:40:43.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78a178740 1 -- 172.21.15.23:0/1915924947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603cc3f0 msgr2=0x55b2603dd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78a178740 1 --2- 172.21.15.23:0/1915924947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603cc3f0 0x55b2603dd9a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb7780195b0 tx=0x7fb778003e40 comp rx=0 tx=0).stop 2024-03-24T10:40:43.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78a178740 1 -- 172.21.15.23:0/1915924947 shutdown_connections 2024-03-24T10:40:43.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78a178740 1 --2- 172.21.15.23:0/1915924947 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55b26044d040 0x55b26044f460 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78a178740 1 --2- 172.21.15.23:0/1915924947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2603d58f0 0x55b2603dacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78a178740 1 --2- 172.21.15.23:0/1915924947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2603ccd10 0x55b2603da780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78a178740 1 --2- 172.21.15.23:0/1915924947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2603cc3f0 0x55b2603dd9a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78a178740 1 -- 172.21.15.23:0/1915924947 >> 172.21.15.23:0/1915924947 conn(0x55b2603c6b20 msgr2=0x55b2603c5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb764ff9640 0 client.15612 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:40:43.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78a178740 1 -- 172.21.15.23:0/1915924947 shutdown_connections 2024-03-24T10:40:43.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78a178740 1 -- 172.21.15.23:0/1915924947 wait complete. 2024-03-24T10:40:43.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78a178740 20 client.15612 trim_cache size 0 max 16384 2024-03-24T10:40:43.431 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:40:43.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78759a640 1 -- 172.21.15.23:0/2871443361 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7680a2350 msgr2=0x7fb7680a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78759a640 1 --2- 172.21.15.23:0/2871443361 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7680a2350 0x7fb7680a4810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb770009b80 tx=0x7fb770009500 comp rx=0 tx=0).stop 2024-03-24T10:40:43.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78759a640 1 -- 172.21.15.23:0/2871443361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7800692d0 msgr2=0x7fb780100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.425+0000 7fb78759a640 1 --2- 172.21.15.23:0/2871443361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7800692d0 0x7fb780100b10 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb774009230 tx=0x7fb774004ab0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.426+0000 7fb78759a640 1 -- 172.21.15.23:0/2871443361 shutdown_connections 2024-03-24T10:40:43.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.426+0000 7fb78759a640 1 --2- 172.21.15.23:0/2871443361 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7680a2350 0x7fb7680a4810 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.426+0000 7fb78759a640 1 --2- 172.21.15.23:0/2871443361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780107760 0x7fb780101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.426+0000 7fb78759a640 1 --2- 172.21.15.23:0/2871443361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7800692d0 0x7fb780100b10 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.426+0000 7fb78759a640 1 --2- 172.21.15.23:0/2871443361 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb780068910 0x7fb7801005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.426+0000 7fb78759a640 1 -- 172.21.15.23:0/2871443361 >> 172.21.15.23:0/2871443361 conn(0x7fb780077710 msgr2=0x7fb7800fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.426+0000 7fb78759a640 1 -- 172.21.15.23:0/2871443361 shutdown_connections 2024-03-24T10:40:43.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.426+0000 7fb78759a640 1 -- 172.21.15.23:0/2871443361 wait complete. 2024-03-24T10:40:43.437 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:40:43.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.547+0000 7fae23647640 1 -- 172.21.15.23:0/29774028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae1c100f60 msgr2=0x7fae1c10d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.547+0000 7fae23647640 1 --2- 172.21.15.23:0/29774028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae1c100f60 0x7fae1c10d420 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fae14009ca0 tx=0x7fae1402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.547+0000 7fae23647640 1 -- 172.21.15.23:0/29774028 shutdown_connections 2024-03-24T10:40:43.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.547+0000 7fae23647640 1 --2- 172.21.15.23:0/29774028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae1c100f60 0x7fae1c10d420 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.547+0000 7fae23647640 1 --2- 172.21.15.23:0/29774028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae1c1005a0 0x7fae1c100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.547+0000 7fae23647640 1 --2- 172.21.15.23:0/29774028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae1c106660 0x7fae1c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.547+0000 7fae23647640 1 -- 172.21.15.23:0/29774028 >> 172.21.15.23:0/29774028 conn(0x7fae1c077710 msgr2=0x7fae1c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.547+0000 7fae23647640 1 -- 172.21.15.23:0/29774028 shutdown_connections 2024-03-24T10:40:43.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.547+0000 7fae23647640 1 -- 172.21.15.23:0/29774028 wait complete. 2024-03-24T10:40:43.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.547+0000 7fae23647640 1 Processor -- start 2024-03-24T10:40:43.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae23647640 1 -- start start 2024-03-24T10:40:43.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae23647640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae1c1005a0 0x7fae1c19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae23647640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae1c100f60 0x7fae1c19ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae23647640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae1c106660 0x7fae1c196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae23647640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c196fb0 con 0x7fae1c100f60 2024-03-24T10:40:43.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae23647640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c197120 con 0x7fae1c1005a0 2024-03-24T10:40:43.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae23647640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c1972e0 con 0x7fae1c106660 2024-03-24T10:40:43.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae20be2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae1c100f60 0x7fae1c19ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae20be2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae1c100f60 0x7fae1c19ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38294/0 (socket says 172.21.15.23:38294) 2024-03-24T10:40:43.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae20be2640 1 -- 172.21.15.23:0/1092582347 learned_addr learned my addr 172.21.15.23:0/1092582347 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae213e3640 1 --2- 172.21.15.23:0/1092582347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae1c1005a0 0x7fae1c19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae21be4640 1 --2- 172.21.15.23:0/1092582347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae1c106660 0x7fae1c196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae20be2640 1 -- 172.21.15.23:0/1092582347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae1c106660 msgr2=0x7fae1c196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae20be2640 1 --2- 172.21.15.23:0/1092582347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae1c106660 0x7fae1c196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae20be2640 1 -- 172.21.15.23:0/1092582347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae1c1005a0 msgr2=0x7fae1c19c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae20be2640 1 --2- 172.21.15.23:0/1092582347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae1c1005a0 0x7fae1c19c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae20be2640 1 -- 172.21.15.23:0/1092582347 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae0c009490 con 0x7fae1c100f60 2024-03-24T10:40:43.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae20be2640 1 --2- 172.21.15.23:0/1092582347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae1c100f60 0x7fae1c19ccf0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fae0c00ea60 tx=0x7fae0c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.548+0000 7fae127fc640 1 -- 172.21.15.23:0/1092582347 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c00cde0 con 0x7fae1c100f60 2024-03-24T10:40:43.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.549+0000 7fae23647640 1 -- 172.21.15.23:0/1092582347 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae18009490 con 0x7fae1c100f60 2024-03-24T10:40:43.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.549+0000 7fae127fc640 1 -- 172.21.15.23:0/1092582347 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae0c0035f0 con 0x7fae1c100f60 2024-03-24T10:40:43.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.549+0000 7fae127fc640 1 -- 172.21.15.23:0/1092582347 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c010690 con 0x7fae1c100f60 2024-03-24T10:40:43.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.549+0000 7fae23647640 1 -- 172.21.15.23:0/1092582347 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae14009900 con 0x7fae1c100f60 2024-03-24T10:40:43.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.551+0000 7fae127fc640 1 -- 172.21.15.23:0/1092582347 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7fae0c003af0 con 0x7fae1c100f60 2024-03-24T10:40:43.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.551+0000 7fae26222740 1 Processor -- start 2024-03-24T10:40:43.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.551+0000 7fae26222740 1 -- start start 2024-03-24T10:40:43.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae26222740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247ac3f0 0x5639247ac7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae26222740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae26222740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247b58f0 0x5639247b7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae26222740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56392471a840 con 0x5639247acd10 2024-03-24T10:40:43.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae26222740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563924717d70 con 0x5639247ac3f0 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae26222740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563924713550 con 0x5639247b58f0 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae213e3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae127fc640 1 --2- 172.21.15.23:0/1092582347 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fae080a2350 0x7fae080a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae21be4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247ac3f0 0x5639247ac7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae213e3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38300/0 (socket says 172.21.15.23:38300) 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae213e3640 1 -- 172.21.15.23:0/2270339299 learned_addr learned my addr 172.21.15.23:0/2270339299 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae20be2640 1 --2- 172.21.15.23:0/2270339299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247b58f0 0x5639247b7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae21be4640 1 --2- 172.21.15.23:0/1092582347 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fae080a2350 0x7fae080a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae213e3640 1 -- 172.21.15.23:0/2270339299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247b58f0 msgr2=0x5639247b7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae213e3640 1 --2- 172.21.15.23:0/2270339299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247b58f0 0x5639247b7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae213e3640 1 -- 172.21.15.23:0/2270339299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247ac3f0 msgr2=0x5639247ac7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae213e3640 1 --2- 172.21.15.23:0/2270339299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247ac3f0 0x5639247ac7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae213e3640 1 -- 172.21.15.23:0/2270339299 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639246a8de0 con 0x5639247acd10 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae21be4640 1 --2- 172.21.15.23:0/2270339299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247ac3f0 0x5639247ac7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:43.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.552+0000 7fae213e3640 1 --2- 172.21.15.23:0/2270339299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b53b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fae18009da0 tx=0x7fae180119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f0f02042877a0f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.553+0000 7fae127fc640 1 -- 172.21.15.23:0/1092582347 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fae0c014070 con 0x7fae1c100f60 2024-03-24T10:40:43.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.553+0000 7fae077fe640 1 -- 172.21.15.23:0/2270339299 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae18016070 con 0x5639247acd10 2024-03-24T10:40:43.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.553+0000 7fae21be4640 1 --2- 172.21.15.23:0/1092582347 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fae080a2350 0x7fae080a4810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fae14004af0 tx=0x7fae1402c0f0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.553+0000 7fae077fe640 1 -- 172.21.15.23:0/2270339299 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae180035f0 con 0x5639247acd10 2024-03-24T10:40:43.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.553+0000 7fae077fe640 1 -- 172.21.15.23:0/2270339299 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1800e670 con 0x5639247acd10 2024-03-24T10:40:43.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.553+0000 7fae26222740 1 -- 172.21.15.23:0/2270339299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 msgr2=0x5639247b53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.553+0000 7fae26222740 1 --2- 172.21.15.23:0/2270339299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b53b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fae18009da0 tx=0x7fae180119a0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 -- 172.21.15.23:0/2270339299 shutdown_connections 2024-03-24T10:40:43.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 --2- 172.21.15.23:0/2270339299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247b58f0 0x5639247b7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 --2- 172.21.15.23:0/2270339299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b53b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 --2- 172.21.15.23:0/2270339299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247ac3f0 0x5639247ac7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 -- 172.21.15.23:0/2270339299 >> 172.21.15.23:0/2270339299 conn(0x5639247a6b20 msgr2=0x5639247a6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 -- 172.21.15.23:0/2270339299 shutdown_connections 2024-03-24T10:40:43.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 -- 172.21.15.23:0/2270339299 wait complete. 2024-03-24T10:40:43.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 -- start start 2024-03-24T10:40:43.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 Processor -- start 2024-03-24T10:40:43.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247ac3f0 0x5639247b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247b58f0 0x5639247b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5639243e6430 con 0x5639247acd10 2024-03-24T10:40:43.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5639243a2e00 con 0x5639247ac3f0 2024-03-24T10:40:43.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.554+0000 7fae26222740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563924698590 con 0x5639247b58f0 2024-03-24T10:40:43.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae26222740 10 client.0 _ll_register_callbacks cb 0x563924707fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:43.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae213e3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247ac3f0 0x5639247b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae20be2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae213e3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247ac3f0 0x5639247b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:44738/0 (socket says 172.21.15.23:44738) 2024-03-24T10:40:43.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae213e3640 1 -- 172.21.15.23:0/432250527 learned_addr learned my addr 172.21.15.23:0/432250527 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae21be4640 1 --2- 172.21.15.23:0/432250527 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247b58f0 0x5639247b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae20be2640 1 -- 172.21.15.23:0/432250527 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247b58f0 msgr2=0x5639247b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae20be2640 1 --2- 172.21.15.23:0/432250527 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247b58f0 0x5639247b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae20be2640 1 -- 172.21.15.23:0/432250527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247ac3f0 msgr2=0x5639247b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae20be2640 1 --2- 172.21.15.23:0/432250527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247ac3f0 0x5639247b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae20be2640 1 -- 172.21.15.23:0/432250527 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae18009a80 con 0x5639247acd10 2024-03-24T10:40:43.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae213e3640 1 --2- 172.21.15.23:0/432250527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247ac3f0 0x5639247b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:43.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae20be2640 1 --2- 172.21.15.23:0/432250527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b2e90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fae0c0cbd60 tx=0x7fae0c09ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398c5eeaf7266f42 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:43.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae04ff9640 1 -- 172.21.15.23:0/432250527 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c003960 con 0x5639247acd10 2024-03-24T10:40:43.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae04ff9640 1 -- 172.21.15.23:0/432250527 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fae0c09f080 con 0x5639247acd10 2024-03-24T10:40:43.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae26222740 1 -- 172.21.15.23:0/432250527 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5639244127e0 con 0x5639247acd10 2024-03-24T10:40:43.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae04ff9640 1 -- 172.21.15.23:0/432250527 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fae0c0a36d0 con 0x5639247acd10 2024-03-24T10:40:43.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae26222740 10 client.15624 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:43.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.555+0000 7fae26222740 1 -- 172.21.15.23:0/432250527 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563924711bd0 con 0x5639247acd10 2024-03-24T10:40:43.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.556+0000 7fae04ff9640 1 -- 172.21.15.23:0/432250527 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5639244127e0 con 0x5639247acd10 2024-03-24T10:40:43.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.556+0000 7fae04ff9640 1 -- 172.21.15.23:0/432250527 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fae0c0b9020 con 0x5639247acd10 2024-03-24T10:40:43.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.556+0000 7fae26222740 10 client.15624 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:43.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.556+0000 7fae26222740 10 client.15624 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:40:43.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.556+0000 7fae26222740 20 client.15624 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:43.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.556+0000 7fae26222740 1 --2- 172.21.15.23:0/432250527 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5639247c5990 0x5639247c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.556+0000 7fae26222740 4 client.15624 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:43.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.556+0000 7fae26222740 1 -- 172.21.15.23:0/432250527 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56392444f720 con 0x5639247c5990 2024-03-24T10:40:43.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.556+0000 7fae213e3640 1 --2- 172.21.15.23:0/432250527 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5639247c5990 0x5639247c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.556+0000 7fae213e3640 1 --2- 172.21.15.23:0/432250527 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5639247c5990 0x5639247c7df0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4dda260222835908 server_cookie=9446c95a1581a5a7 in_seq=0 out_seq=0 2024-03-24T10:40:43.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.556+0000 7fae04ff9640 10 client.15624 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:40:43.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.557+0000 7fae04ff9640 1 -- 172.21.15.23:0/432250527 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56392444f720 con 0x5639247c5990 2024-03-24T10:40:43.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.557+0000 7fae04ff9640 10 client.15624 handle_command_reply: tid=0 2024-03-24T10:40:43.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.557+0000 7fae26222740 1 client.15624 shutdown 2024-03-24T10:40:43.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.557+0000 7fae26222740 1 -- 172.21.15.23:0/432250527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 msgr2=0x5639247b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.558+0000 7fae26222740 1 --2- 172.21.15.23:0/432250527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b2e90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fae0c0cbd60 tx=0x7fae0c09ab70 comp rx=0 tx=0).stop 2024-03-24T10:40:43.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.558+0000 7fae26222740 1 -- 172.21.15.23:0/432250527 shutdown_connections 2024-03-24T10:40:43.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.558+0000 7fae26222740 1 --2- 172.21.15.23:0/432250527 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5639247c5990 0x5639247c7df0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.558+0000 7fae26222740 1 --2- 172.21.15.23:0/432250527 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247b58f0 0x5639247b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.558+0000 7fae26222740 1 --2- 172.21.15.23:0/432250527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b2e90 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.558+0000 7fae26222740 1 --2- 172.21.15.23:0/432250527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247ac3f0 0x5639247b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.558+0000 7fae04ff9640 0 client.15624 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:40:43.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.558+0000 7fae26222740 1 -- 172.21.15.23:0/432250527 >> 172.21.15.23:0/432250527 conn(0x5639247a6b20 msgr2=0x5639247a8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.558+0000 7fae26222740 1 -- 172.21.15.23:0/432250527 shutdown_connections 2024-03-24T10:40:43.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.558+0000 7fae26222740 1 -- 172.21.15.23:0/432250527 wait complete. 2024-03-24T10:40:43.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.558+0000 7fae26222740 20 client.15624 trim_cache size 0 max 16384 2024-03-24T10:40:43.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.572+0000 7fae26222740 1 Processor -- start 2024-03-24T10:40:43.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.572+0000 7fae26222740 1 -- start start 2024-03-24T10:40:43.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.572+0000 7fae26222740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247ac3f0 0x5639247c1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.572+0000 7fae26222740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247acd10 0x5639247b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.572+0000 7fae26222740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247b58f0 0x5639247cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.572+0000 7fae26222740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5639247c57a0 con 0x5639247ac3f0 2024-03-24T10:40:43.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.572+0000 7fae26222740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5639247ac050 con 0x5639247b58f0 2024-03-24T10:40:43.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.572+0000 7fae26222740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5639247a7b20 con 0x5639247acd10 2024-03-24T10:40:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.572+0000 7fae213e3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247ac3f0 0x5639247c1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.572+0000 7fae213e3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247ac3f0 0x5639247c1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38316/0 (socket says 172.21.15.23:38316) 2024-03-24T10:40:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.572+0000 7fae213e3640 1 -- 172.21.15.23:0/406252320 learned_addr learned my addr 172.21.15.23:0/406252320 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.572+0000 7fae20be2640 1 --2- 172.21.15.23:0/406252320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247acd10 0x5639247b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.572+0000 7fae21be4640 1 --2- 172.21.15.23:0/406252320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247b58f0 0x5639247cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.573+0000 7fae213e3640 1 -- 172.21.15.23:0/406252320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247acd10 msgr2=0x5639247b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.573+0000 7fae213e3640 1 --2- 172.21.15.23:0/406252320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247acd10 0x5639247b31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.573+0000 7fae213e3640 1 -- 172.21.15.23:0/406252320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247b58f0 msgr2=0x5639247cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.573+0000 7fae213e3640 1 --2- 172.21.15.23:0/406252320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247b58f0 0x5639247cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.573+0000 7fae213e3640 1 -- 172.21.15.23:0/406252320 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639246adc80 con 0x5639247ac3f0 2024-03-24T10:40:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.573+0000 7fae213e3640 1 --2- 172.21.15.23:0/406252320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247ac3f0 0x5639247c1180 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fae18000be0 tx=0x7fae18005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3835bbdd3eab637c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.573+0000 7fae06ffd640 1 -- 172.21.15.23:0/406252320 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae18016040 con 0x5639247ac3f0 2024-03-24T10:40:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.573+0000 7fae06ffd640 1 -- 172.21.15.23:0/406252320 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae18017040 con 0x5639247ac3f0 2024-03-24T10:40:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.573+0000 7fae06ffd640 1 -- 172.21.15.23:0/406252320 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1800e470 con 0x5639247ac3f0 2024-03-24T10:40:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.573+0000 7fae26222740 1 -- 172.21.15.23:0/406252320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247ac3f0 msgr2=0x5639247c1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.573+0000 7fae26222740 1 --2- 172.21.15.23:0/406252320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247ac3f0 0x5639247c1180 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fae18000be0 tx=0x7fae18005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 -- 172.21.15.23:0/406252320 shutdown_connections 2024-03-24T10:40:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 --2- 172.21.15.23:0/406252320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247b58f0 0x5639247cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 --2- 172.21.15.23:0/406252320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247acd10 0x5639247b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 --2- 172.21.15.23:0/406252320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247ac3f0 0x5639247c1180 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 -- 172.21.15.23:0/406252320 >> 172.21.15.23:0/406252320 conn(0x5639247a6b20 msgr2=0x5639247a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 -- 172.21.15.23:0/406252320 shutdown_connections 2024-03-24T10:40:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 -- 172.21.15.23:0/406252320 wait complete. 2024-03-24T10:40:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 -- start start 2024-03-24T10:40:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 Processor -- start 2024-03-24T10:40:43.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247ac3f0 0x5639247b9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247b58f0 0x56392482ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5639247ba180 con 0x5639247acd10 2024-03-24T10:40:43.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5639247ba2f0 con 0x5639247b58f0 2024-03-24T10:40:43.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56392482b2a0 con 0x5639247ac3f0 2024-03-24T10:40:43.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae26222740 10 client.0 _ll_register_callbacks cb 0x7fae0c00e9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:43.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae20be2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.574+0000 7fae20be2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38320/0 (socket says 172.21.15.23:38320) 2024-03-24T10:40:43.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae20be2640 1 -- 172.21.15.23:0/2937871737 learned_addr learned my addr 172.21.15.23:0/2937871737 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae21be4640 1 --2- 172.21.15.23:0/2937871737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247b58f0 0x56392482ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae213e3640 1 --2- 172.21.15.23:0/2937871737 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247ac3f0 0x5639247b9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae20be2640 1 -- 172.21.15.23:0/2937871737 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247ac3f0 msgr2=0x5639247b9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae20be2640 1 --2- 172.21.15.23:0/2937871737 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247ac3f0 0x5639247b9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae20be2640 1 -- 172.21.15.23:0/2937871737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247b58f0 msgr2=0x56392482ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae20be2640 1 --2- 172.21.15.23:0/2937871737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247b58f0 0x56392482ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae20be2640 1 -- 172.21.15.23:0/2937871737 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639246a8de0 con 0x5639247acd10 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae20be2640 1 --2- 172.21.15.23:0/2937871737 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b99b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fae0c0cc470 tx=0x7fae0c0b94f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e45d141fa45b0f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae04ff9640 1 -- 172.21.15.23:0/2937871737 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c096e80 con 0x5639247acd10 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae04ff9640 1 -- 172.21.15.23:0/2937871737 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fae0c096fe0 con 0x5639247acd10 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae26222740 1 -- 172.21.15.23:0/2937871737 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae180035f0 con 0x5639247acd10 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae04ff9640 1 -- 172.21.15.23:0/2937871737 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fae0c0cc250 con 0x5639247acd10 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae26222740 10 client.15630 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae26222740 1 -- 172.21.15.23:0/2937871737 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5639246d3920 con 0x5639247acd10 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.575+0000 7fae04ff9640 1 -- 172.21.15.23:0/2937871737 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae180035f0 con 0x5639247acd10 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae04ff9640 1 -- 172.21.15.23:0/2937871737 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fae0c0840a0 con 0x5639247acd10 2024-03-24T10:40:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 10 client.15630 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 10 client.15630 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:40:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 20 client.15630 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 1 --2- 172.21.15.23:0/2937871737 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56392482c6a0 0x56392482eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 4 client.15630 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:40:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 1 -- 172.21.15.23:0/2937871737 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56392449e890 con 0x56392482c6a0 2024-03-24T10:40:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae213e3640 1 --2- 172.21.15.23:0/2937871737 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56392482c6a0 0x56392482eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae213e3640 1 --2- 172.21.15.23:0/2937871737 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56392482c6a0 0x56392482eac0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b57c027f6b22beb4 server_cookie=b12bdeb1a9a5f6bb in_seq=0 out_seq=0 2024-03-24T10:40:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae04ff9640 10 client.15630 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:40:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae04ff9640 1 -- 172.21.15.23:0/2937871737 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56392449e890 con 0x56392482c6a0 2024-03-24T10:40:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae04ff9640 10 client.15630 handle_command_reply: tid=0 2024-03-24T10:40:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 1 client.15630 shutdown 2024-03-24T10:40:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 1 -- 172.21.15.23:0/2937871737 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 msgr2=0x5639247b99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 1 --2- 172.21.15.23:0/2937871737 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b99b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fae0c0cc470 tx=0x7fae0c0b94f0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 1 -- 172.21.15.23:0/2937871737 shutdown_connections 2024-03-24T10:40:43.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 1 --2- 172.21.15.23:0/2937871737 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56392482c6a0 0x56392482eac0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 1 --2- 172.21.15.23:0/2937871737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5639247b58f0 0x56392482ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 1 --2- 172.21.15.23:0/2937871737 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639247acd10 0x5639247b99b0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 1 --2- 172.21.15.23:0/2937871737 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639247ac3f0 0x5639247b9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.576+0000 7fae26222740 1 -- 172.21.15.23:0/2937871737 >> 172.21.15.23:0/2937871737 conn(0x5639247a6b20 msgr2=0x5639247a7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae04ff9640 0 client.15630 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:40:43.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae26222740 1 -- 172.21.15.23:0/2937871737 shutdown_connections 2024-03-24T10:40:43.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae26222740 1 -- 172.21.15.23:0/2937871737 wait complete. 2024-03-24T10:40:43.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae26222740 20 client.15630 trim_cache size 0 max 16384 2024-03-24T10:40:43.583 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:40:43.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae23647640 1 -- 172.21.15.23:0/1092582347 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fae080a2350 msgr2=0x7fae080a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae23647640 1 --2- 172.21.15.23:0/1092582347 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fae080a2350 0x7fae080a4810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fae14004af0 tx=0x7fae1402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae23647640 1 -- 172.21.15.23:0/1092582347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae1c100f60 msgr2=0x7fae1c19ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae23647640 1 --2- 172.21.15.23:0/1092582347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae1c100f60 0x7fae1c19ccf0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fae0c00ea60 tx=0x7fae0c00ef20 comp rx=0 tx=0).stop 2024-03-24T10:40:43.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae23647640 1 -- 172.21.15.23:0/1092582347 shutdown_connections 2024-03-24T10:40:43.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae23647640 1 --2- 172.21.15.23:0/1092582347 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fae080a2350 0x7fae080a4810 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae23647640 1 --2- 172.21.15.23:0/1092582347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae1c106660 0x7fae1c196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae23647640 1 --2- 172.21.15.23:0/1092582347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae1c100f60 0x7fae1c19ccf0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae23647640 1 --2- 172.21.15.23:0/1092582347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae1c1005a0 0x7fae1c19c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae23647640 1 -- 172.21.15.23:0/1092582347 >> 172.21.15.23:0/1092582347 conn(0x7fae1c077710 msgr2=0x7fae1c0ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae23647640 1 -- 172.21.15.23:0/1092582347 shutdown_connections 2024-03-24T10:40:43.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.577+0000 7fae23647640 1 -- 172.21.15.23:0/1092582347 wait complete. 2024-03-24T10:40:43.587 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:43 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1224697665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:40:43.588 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:40:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.695+0000 7fa12e833640 1 -- 172.21.15.23:0/909743529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1280688f0 msgr2=0x7fa128068d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.695+0000 7fa12e833640 1 --2- 172.21.15.23:0/909743529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1280688f0 0x7fa128068d70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009930 tx=0x7fa11c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.695+0000 7fa12e833640 1 -- 172.21.15.23:0/909743529 shutdown_connections 2024-03-24T10:40:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.695+0000 7fa12e833640 1 --2- 172.21.15.23:0/909743529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1280692b0 0x7fa12810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.695+0000 7fa12e833640 1 --2- 172.21.15.23:0/909743529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1280688f0 0x7fa128068d70 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.695+0000 7fa12e833640 1 --2- 172.21.15.23:0/909743529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa128107690 0x7fa128107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.695+0000 7fa12e833640 1 -- 172.21.15.23:0/909743529 >> 172.21.15.23:0/909743529 conn(0x7fa128077710 msgr2=0x7fa128077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.695+0000 7fa12e833640 1 -- 172.21.15.23:0/909743529 shutdown_connections 2024-03-24T10:40:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.695+0000 7fa12e833640 1 -- 172.21.15.23:0/909743529 wait complete. 2024-03-24T10:40:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa12e833640 1 Processor -- start 2024-03-24T10:40:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa12e833640 1 -- start start 2024-03-24T10:40:43.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa12e833640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1280688f0 0x7fa128197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa12e833640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1280692b0 0x7fa1281982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa12e833640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa128107690 0x7fa12819c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa12e833640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fa12819cc60 con 0x7fa128107690 2024-03-24T10:40:43.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa12e833640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa12819cdd0 con 0x7fa1280688f0 2024-03-24T10:40:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa12e833640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa12819cf90 con 0x7fa1280692b0 2024-03-24T10:40:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa12cdd0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa128107690 0x7fa12819c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa127fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1280688f0 0x7fa128197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa12cdd0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa128107690 0x7fa12819c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38334/0 (socket says 172.21.15.23:38334) 2024-03-24T10:40:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa12cdd0640 1 -- 172.21.15.23:0/1742166773 learned_addr learned my addr 172.21.15.23:0/1742166773 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa1277fe640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1280692b0 0x7fa1281982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa12cdd0640 1 -- 172.21.15.23:0/1742166773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1280692b0 msgr2=0x7fa1281982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.696+0000 7fa12cdd0640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1280692b0 0x7fa1281982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.697+0000 7fa12cdd0640 1 -- 172.21.15.23:0/1742166773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1280688f0 msgr2=0x7fa128197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.697+0000 7fa12cdd0640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1280688f0 0x7fa128197da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.697+0000 7fa12cdd0640 1 -- 172.21.15.23:0/1742166773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa120009840 con 0x7fa128107690 2024-03-24T10:40:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.697+0000 7fa127fff640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1280688f0 0x7fa128197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.697+0000 7fa1277fe640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1280692b0 0x7fa1281982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:43.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.697+0000 7fa12cdd0640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa128107690 0x7fa12819c660 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa12000efd0 tx=0x7fa12000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.697+0000 7fa1257fa640 1 -- 172.21.15.23:0/1742166773 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa120009b40 con 0x7fa128107690 2024-03-24T10:40:43.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.697+0000 7fa12e833640 1 -- 172.21.15.23:0/1742166773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa11c009560 con 0x7fa128107690 2024-03-24T10:40:43.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.697+0000 7fa1257fa640 1 -- 172.21.15.23:0/1742166773 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa120003b70 con 0x7fa128107690 2024-03-24T10:40:43.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.697+0000 7fa1257fa640 1 -- 172.21.15.23:0/1742166773 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1200108f0 con 0x7fa128107690 2024-03-24T10:40:43.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.697+0000 7fa12e833640 1 -- 172.21.15.23:0/1742166773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1281a4910 con 0x7fa128107690 2024-03-24T10:40:43.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.699+0000 7fa1257fa640 1 -- 172.21.15.23:0/1742166773 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7fa120003ce0 con 0x7fa128107690 2024-03-24T10:40:43.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa13140c740 1 Processor -- start 2024-03-24T10:40:43.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa13140c740 1 -- start start 2024-03-24T10:40:43.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa13140c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fe3f0 0x5611e48fe7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa13140c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fed10 0x5611e49073b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa13140c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e49078f0 0x5611e4909ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa13140c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5611e486c840 con 0x5611e49078f0 2024-03-24T10:40:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa13140c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5611e4869d70 con 0x5611e48fed10 2024-03-24T10:40:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa13140c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5611e4865550 con 0x5611e48fe3f0 2024-03-24T10:40:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa1257fa640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa1100a2350 0x7fa1100a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa1277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e49078f0 0x5611e4909ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa127fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fed10 0x5611e49073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa1277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e49078f0 0x5611e4909ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38348/0 (socket says 172.21.15.23:38348) 2024-03-24T10:40:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa1277fe640 1 -- 172.21.15.23:0/2267773160 learned_addr learned my addr 172.21.15.23:0/2267773160 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa1277fe640 1 --2- 172.21.15.23:0/2267773160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fe3f0 0x5611e48fe7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa12cdd0640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa1100a2350 0x7fa1100a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa1257fa640 1 -- 172.21.15.23:0/1742166773 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fa120014070 con 0x7fa128107690 2024-03-24T10:40:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa1277fe640 1 -- 172.21.15.23:0/2267773160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fe3f0 msgr2=0x5611e48fe7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa1277fe640 1 --2- 172.21.15.23:0/2267773160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fe3f0 0x5611e48fe7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.700+0000 7fa1277fe640 1 -- 172.21.15.23:0/2267773160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fed10 msgr2=0x5611e49073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.701+0000 7fa1277fe640 1 --2- 172.21.15.23:0/2267773160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fed10 0x5611e49073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.701+0000 7fa1277fe640 1 -- 172.21.15.23:0/2267773160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611e47fade0 con 0x5611e49078f0 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.701+0000 7fa127fff640 1 --2- 172.21.15.23:0/2267773160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fed10 0x5611e49073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.701+0000 7fa1277fe640 1 --2- 172.21.15.23:0/2267773160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e49078f0 0x5611e4909ce0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa11c036040 tx=0x7fa11c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe44869090c53f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.701+0000 7fa12cdd0640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa1100a2350 0x7fa1100a4810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa1200cbef0 tx=0x7fa120009f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.701+0000 7fa10e7fc640 1 -- 172.21.15.23:0/2267773160 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c004030 con 0x5611e49078f0 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.701+0000 7fa10e7fc640 1 -- 172.21.15.23:0/2267773160 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa11c0035f0 con 0x5611e49078f0 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.701+0000 7fa10e7fc640 1 -- 172.21.15.23:0/2267773160 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c005050 con 0x5611e49078f0 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 -- 172.21.15.23:0/2267773160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e49078f0 msgr2=0x5611e4909ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 --2- 172.21.15.23:0/2267773160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e49078f0 0x5611e4909ce0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa11c036040 tx=0x7fa11c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 -- 172.21.15.23:0/2267773160 shutdown_connections 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 --2- 172.21.15.23:0/2267773160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e49078f0 0x5611e4909ce0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 --2- 172.21.15.23:0/2267773160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fed10 0x5611e49073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 --2- 172.21.15.23:0/2267773160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fe3f0 0x5611e48fe7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 -- 172.21.15.23:0/2267773160 >> 172.21.15.23:0/2267773160 conn(0x5611e48f8b20 msgr2=0x5611e48f8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 -- 172.21.15.23:0/2267773160 shutdown_connections 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 -- 172.21.15.23:0/2267773160 wait complete. 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 -- start start 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 Processor -- start 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fe3f0 0x5611e49049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e48fed10 0x5611e4904ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e49078f0 0x5611e4905420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5611e4538430 con 0x5611e48fed10 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5611e44f4e00 con 0x5611e48fe3f0 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5611e47ea590 con 0x5611e49078f0 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.702+0000 7fa13140c740 10 client.0 _ll_register_callbacks cb 0x5611e4859fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa127fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e48fed10 0x5611e4904ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa127fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e48fed10 0x5611e4904ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38352/0 (socket says 172.21.15.23:38352) 2024-03-24T10:40:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa127fff640 1 -- 172.21.15.23:0/1392802335 learned_addr learned my addr 172.21.15.23:0/1392802335 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa1277fe640 1 --2- 172.21.15.23:0/1392802335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fe3f0 0x5611e49049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa1277fe640 1 --2- 172.21.15.23:0/1392802335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e49078f0 0x5611e4905420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa127fff640 1 -- 172.21.15.23:0/1392802335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e49078f0 msgr2=0x5611e4905420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa127fff640 1 --2- 172.21.15.23:0/1392802335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e49078f0 0x5611e4905420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa127fff640 1 -- 172.21.15.23:0/1392802335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fe3f0 msgr2=0x5611e49049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa127fff640 1 --2- 172.21.15.23:0/1392802335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fe3f0 0x5611e49049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa127fff640 1 -- 172.21.15.23:0/1392802335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa12810ee50 con 0x5611e48fed10 2024-03-24T10:40:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa1277fe640 1 --2- 172.21.15.23:0/1392802335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fe3f0 0x5611e49049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:40:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa127fff640 1 --2- 172.21.15.23:0/1392802335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e48fed10 0x5611e4904ee0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa118012d20 tx=0x7fa11800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b67f68bdf4efec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa1077fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa1077fe640 1 -- 172.21.15.23:0/1392802335 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa118017070 con 0x5611e48fed10 2024-03-24T10:40:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa1077fe640 1 -- 172.21.15.23:0/1392802335 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fa1180120d0 con 0x5611e48fed10 2024-03-24T10:40:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa13140c740 1 -- 172.21.15.23:0/1392802335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5611e45647e0 con 0x5611e48fed10 2024-03-24T10:40:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa1077fe640 1 -- 172.21.15.23:0/1392802335 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa118012290 con 0x5611e48fed10 2024-03-24T10:40:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa13140c740 10 client.15642 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.703+0000 7fa13140c740 1 -- 172.21.15.23:0/1392802335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5611e4863bd0 con 0x5611e48fed10 2024-03-24T10:40:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.704+0000 7fa1077fe640 1 -- 172.21.15.23:0/1392802335 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa118028080 con 0x5611e48fed10 2024-03-24T10:40:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.704+0000 7fa1077fe640 1 -- 172.21.15.23:0/1392802335 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fa118022030 con 0x5611e48fed10 2024-03-24T10:40:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.704+0000 7fa13140c740 10 client.15642 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.704+0000 7fa13140c740 10 client.15642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.704+0000 7fa13140c740 20 client.15642 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.704+0000 7fa13140c740 1 --2- 172.21.15.23:0/1392802335 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5611e4917ab0 0x5611e4919ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.704+0000 7fa13140c740 4 client.15642 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.704+0000 7fa13140c740 1 -- 172.21.15.23:0/1392802335 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611e45a1720 con 0x5611e4917ab0 2024-03-24T10:40:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.704+0000 7fa1277fe640 1 --2- 172.21.15.23:0/1392802335 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5611e4917ab0 0x5611e4919ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.704+0000 7fa1077fe640 10 client.15642 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.704+0000 7fa1277fe640 1 --2- 172.21.15.23:0/1392802335 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5611e4917ab0 0x5611e4919ed0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=676ed4d39cf478a server_cookie=fe90ef37405263cd in_seq=0 out_seq=0 2024-03-24T10:40:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.705+0000 7fa1077fe640 1 -- 172.21.15.23:0/1392802335 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5611e45a1720 con 0x5611e4917ab0 2024-03-24T10:40:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.705+0000 7fa1077fe640 10 client.15642 handle_command_reply: tid=0 2024-03-24T10:40:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.705+0000 7fa13140c740 1 client.15642 shutdown 2024-03-24T10:40:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.706+0000 7fa13140c740 1 -- 172.21.15.23:0/1392802335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e48fed10 msgr2=0x5611e4904ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.706+0000 7fa13140c740 1 --2- 172.21.15.23:0/1392802335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e48fed10 0x5611e4904ee0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa118012d20 tx=0x7fa11800f790 comp rx=0 tx=0).stop 2024-03-24T10:40:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.706+0000 7fa13140c740 1 -- 172.21.15.23:0/1392802335 shutdown_connections 2024-03-24T10:40:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.706+0000 7fa13140c740 1 --2- 172.21.15.23:0/1392802335 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5611e4917ab0 0x5611e4919ed0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.706+0000 7fa13140c740 1 --2- 172.21.15.23:0/1392802335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e49078f0 0x5611e4905420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.706+0000 7fa1077fe640 0 client.15642 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.706+0000 7fa13140c740 1 --2- 172.21.15.23:0/1392802335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e48fed10 0x5611e4904ee0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.706+0000 7fa13140c740 1 --2- 172.21.15.23:0/1392802335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fe3f0 0x5611e49049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.706+0000 7fa13140c740 1 -- 172.21.15.23:0/1392802335 >> 172.21.15.23:0/1392802335 conn(0x5611e48f8b20 msgr2=0x5611e48fa870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.706+0000 7fa13140c740 1 -- 172.21.15.23:0/1392802335 shutdown_connections 2024-03-24T10:40:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.706+0000 7fa13140c740 1 -- 172.21.15.23:0/1392802335 wait complete. 2024-03-24T10:40:43.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.706+0000 7fa13140c740 20 client.15642 trim_cache size 0 max 16384 2024-03-24T10:40:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.719+0000 7fa13140c740 1 Processor -- start 2024-03-24T10:40:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.719+0000 7fa13140c740 1 -- start start 2024-03-24T10:40:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.719+0000 7fa13140c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fe3f0 0x5611e4913120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.719+0000 7fa13140c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fed10 0x5611e49067d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.719+0000 7fa13140c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e49078f0 0x5611e491f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.719+0000 7fa13140c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5611e4905e30 con 0x5611e49078f0 2024-03-24T10:40:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.719+0000 7fa13140c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5611e4916220 con 0x5611e48fed10 2024-03-24T10:40:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.719+0000 7fa13140c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5611e4913690 con 0x5611e48fe3f0 2024-03-24T10:40:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.720+0000 7fa1277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e49078f0 0x5611e491f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.720+0000 7fa1277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e49078f0 0x5611e491f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38360/0 (socket says 172.21.15.23:38360) 2024-03-24T10:40:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.720+0000 7fa1277fe640 1 -- 172.21.15.23:0/3918843360 learned_addr learned my addr 172.21.15.23:0/3918843360 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.720+0000 7fa1277fe640 1 --2- 172.21.15.23:0/3918843360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fe3f0 0x5611e4913120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.720+0000 7fa127fff640 1 --2- 172.21.15.23:0/3918843360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fed10 0x5611e49067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.720+0000 7fa1277fe640 1 -- 172.21.15.23:0/3918843360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fe3f0 msgr2=0x5611e4913120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.720+0000 7fa1277fe640 1 --2- 172.21.15.23:0/3918843360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fe3f0 0x5611e4913120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.720+0000 7fa1277fe640 1 -- 172.21.15.23:0/3918843360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fed10 msgr2=0x5611e49067d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.720+0000 7fa1277fe640 1 --2- 172.21.15.23:0/3918843360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fed10 0x5611e49067d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.720+0000 7fa1277fe640 1 -- 172.21.15.23:0/3918843360 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611e47ffc80 con 0x5611e49078f0 2024-03-24T10:40:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.720+0000 7fa1277fe640 1 --2- 172.21.15.23:0/3918843360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e49078f0 0x5611e491f7e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa11c004ae0 tx=0x7fa11c02c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1b14cfd97ad5af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.720+0000 7fa10dffb640 1 -- 172.21.15.23:0/3918843360 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c053040 con 0x5611e49078f0 2024-03-24T10:40:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.720+0000 7fa10dffb640 1 -- 172.21.15.23:0/3918843360 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa11c05d070 con 0x5611e49078f0 2024-03-24T10:40:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.720+0000 7fa10dffb640 1 -- 172.21.15.23:0/3918843360 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c004500 con 0x5611e49078f0 2024-03-24T10:40:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.721+0000 7fa13140c740 1 -- 172.21.15.23:0/3918843360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e49078f0 msgr2=0x5611e491f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.721+0000 7fa13140c740 1 --2- 172.21.15.23:0/3918843360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e49078f0 0x5611e491f7e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa11c004ae0 tx=0x7fa11c02c820 comp rx=0 tx=0).stop 2024-03-24T10:40:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.721+0000 7fa13140c740 1 -- 172.21.15.23:0/3918843360 shutdown_connections 2024-03-24T10:40:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.721+0000 7fa13140c740 1 --2- 172.21.15.23:0/3918843360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e49078f0 0x5611e491f7e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.721+0000 7fa13140c740 1 --2- 172.21.15.23:0/3918843360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e48fed10 0x5611e49067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.721+0000 7fa13140c740 1 --2- 172.21.15.23:0/3918843360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fe3f0 0x5611e4913120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.721+0000 7fa13140c740 1 -- 172.21.15.23:0/3918843360 >> 172.21.15.23:0/3918843360 conn(0x5611e48f8b20 msgr2=0x5611e49060c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.721+0000 7fa13140c740 1 -- 172.21.15.23:0/3918843360 shutdown_connections 2024-03-24T10:40:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.721+0000 7fa13140c740 1 -- 172.21.15.23:0/3918843360 wait complete. 2024-03-24T10:40:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.721+0000 7fa13140c740 1 -- start start 2024-03-24T10:40:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.721+0000 7fa13140c740 1 Processor -- start 2024-03-24T10:40:43.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa13140c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e48fe3f0 0x5611e490f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa13140c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fed10 0x5611e4901060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa13140c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e49078f0 0x5611e49015a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa13140c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5611e4901ae0 con 0x5611e48fe3f0 2024-03-24T10:40:43.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa13140c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5611e4901c50 con 0x5611e49078f0 2024-03-24T10:40:43.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa13140c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5611e4901e10 con 0x5611e48fed10 2024-03-24T10:40:43.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa13140c740 10 client.0 _ll_register_callbacks cb 0x7fa118009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:43.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa1277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e48fe3f0 0x5611e490f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa1277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e48fe3f0 0x5611e490f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38362/0 (socket says 172.21.15.23:38362) 2024-03-24T10:40:43.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa1277fe640 1 -- 172.21.15.23:0/709929651 learned_addr learned my addr 172.21.15.23:0/709929651 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa127fff640 1 --2- 172.21.15.23:0/709929651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fed10 0x5611e4901060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa1277fe640 1 --2- 172.21.15.23:0/709929651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e49078f0 0x5611e49015a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa1277fe640 1 -- 172.21.15.23:0/709929651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fed10 msgr2=0x5611e4901060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa1277fe640 1 --2- 172.21.15.23:0/709929651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fed10 0x5611e4901060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa1277fe640 1 -- 172.21.15.23:0/709929651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e49078f0 msgr2=0x5611e49015a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa1277fe640 1 --2- 172.21.15.23:0/709929651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e49078f0 0x5611e49015a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa1277fe640 1 -- 172.21.15.23:0/709929651 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611e47fade0 con 0x5611e48fe3f0 2024-03-24T10:40:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa1277fe640 1 --2- 172.21.15.23:0/709929651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e48fe3f0 0x5611e490f750 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa11c003e60 tx=0x7fa11c005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9466dde78b0fd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa1077fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa1077fe640 1 -- 172.21.15.23:0/709929651 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c004480 con 0x5611e48fe3f0 2024-03-24T10:40:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa1077fe640 1 -- 172.21.15.23:0/709929651 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fa11c000e40 con 0x5611e48fe3f0 2024-03-24T10:40:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa13140c740 1 -- 172.21.15.23:0/709929651 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa11c0035f0 con 0x5611e48fe3f0 2024-03-24T10:40:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa1077fe640 1 -- 172.21.15.23:0/709929651 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa11c003fc0 con 0x5611e48fe3f0 2024-03-24T10:40:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa13140c740 10 client.15648 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.722+0000 7fa13140c740 1 -- 172.21.15.23:0/709929651 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5611e4825920 con 0x5611e48fe3f0 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.723+0000 7fa1077fe640 1 -- 172.21.15.23:0/709929651 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa11c0035f0 con 0x5611e48fe3f0 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.723+0000 7fa1077fe640 1 -- 172.21.15.23:0/709929651 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fa11c06a070 con 0x5611e48fe3f0 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.723+0000 7fa13140c740 10 client.15648 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.723+0000 7fa13140c740 10 client.15648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.723+0000 7fa13140c740 20 client.15648 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.723+0000 7fa13140c740 1 --2- 172.21.15.23:0/709929651 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5611e4917ab0 0x5611e497fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.723+0000 7fa13140c740 4 client.15648 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.723+0000 7fa13140c740 1 -- 172.21.15.23:0/709929651 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5611e45f0890 con 0x5611e4917ab0 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.723+0000 7fa127fff640 1 --2- 172.21.15.23:0/709929651 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5611e4917ab0 0x5611e497fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.723+0000 7fa127fff640 1 --2- 172.21.15.23:0/709929651 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5611e4917ab0 0x5611e497fdc0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad851d52467e2ee1 server_cookie=ce8cc5163d8e9eb7 in_seq=0 out_seq=0 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.723+0000 7fa1077fe640 10 client.15648 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa1077fe640 1 -- 172.21.15.23:0/709929651 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5611e45f0890 con 0x5611e4917ab0 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa1077fe640 10 client.15648 handle_command_reply: tid=0 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa13140c740 1 client.15648 shutdown 2024-03-24T10:40:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa13140c740 1 -- 172.21.15.23:0/709929651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e48fe3f0 msgr2=0x5611e490f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa13140c740 1 --2- 172.21.15.23:0/709929651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e48fe3f0 0x5611e490f750 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa11c003e60 tx=0x7fa11c005e50 comp rx=0 tx=0).stop 2024-03-24T10:40:43.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa13140c740 1 -- 172.21.15.23:0/709929651 shutdown_connections 2024-03-24T10:40:43.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa13140c740 1 --2- 172.21.15.23:0/709929651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5611e49078f0 0x5611e49015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa13140c740 1 --2- 172.21.15.23:0/709929651 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5611e4917ab0 0x5611e497fdc0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa13140c740 1 --2- 172.21.15.23:0/709929651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5611e48fed10 0x5611e4901060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa13140c740 1 --2- 172.21.15.23:0/709929651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611e48fe3f0 0x5611e490f750 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa1077fe640 0 client.15648 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:43.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa13140c740 1 -- 172.21.15.23:0/709929651 >> 172.21.15.23:0/709929651 conn(0x5611e48f8b20 msgr2=0x5611e49037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa13140c740 1 -- 172.21.15.23:0/709929651 shutdown_connections 2024-03-24T10:40:43.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa13140c740 1 -- 172.21.15.23:0/709929651 wait complete. 2024-03-24T10:40:43.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa13140c740 20 client.15648 trim_cache size 0 max 16384 2024-03-24T10:40:43.731 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:40:43.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa12e833640 1 -- 172.21.15.23:0/1742166773 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa1100a2350 msgr2=0x7fa1100a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa12e833640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa1100a2350 0x7fa1100a4810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa1200cbef0 tx=0x7fa120009f90 comp rx=0 tx=0).stop 2024-03-24T10:40:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa12e833640 1 -- 172.21.15.23:0/1742166773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa128107690 msgr2=0x7fa12819c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.724+0000 7fa12e833640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa128107690 0x7fa12819c660 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa12000efd0 tx=0x7fa12000c6a0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.725+0000 7fa12e833640 1 -- 172.21.15.23:0/1742166773 shutdown_connections 2024-03-24T10:40:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.725+0000 7fa12e833640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa1100a2350 0x7fa1100a4810 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.725+0000 7fa12e833640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa128107690 0x7fa12819c660 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.725+0000 7fa12e833640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1280692b0 0x7fa1281982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.725+0000 7fa12e833640 1 --2- 172.21.15.23:0/1742166773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1280688f0 0x7fa128197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.725+0000 7fa12e833640 1 -- 172.21.15.23:0/1742166773 >> 172.21.15.23:0/1742166773 conn(0x7fa128077710 msgr2=0x7fa1280fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.725+0000 7fa12e833640 1 -- 172.21.15.23:0/1742166773 shutdown_connections 2024-03-24T10:40:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:43.725+0000 7fa12e833640 1 -- 172.21.15.23:0/1742166773 wait complete. 2024-03-24T10:40:43.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:43 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1224697665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:40:43.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:43 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1224697665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:40:44.737 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:40:44.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:40:44.737 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:40:44.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:44 smithi117 ceph-mon[36368]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-24T10:40:44.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:44 smithi023 ceph-mon[32466]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-24T10:40:44.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:44 smithi116 ceph-mon[36248]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-24T10:40:44.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.846+0000 7f6be08c9640 1 -- 172.21.15.23:0/4043038662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bdc102060 msgr2=0x7f6bdc10e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:44.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.846+0000 7f6be08c9640 1 --2- 172.21.15.23:0/4043038662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bdc102060 0x7f6bdc10e570 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009930 tx=0x7f6bcc02bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:44.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.846+0000 7f6be08c9640 1 -- 172.21.15.23:0/4043038662 shutdown_connections 2024-03-24T10:40:44.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.846+0000 7f6be08c9640 1 --2- 172.21.15.23:0/4043038662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bdc102060 0x7f6bdc10e570 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.846+0000 7f6be08c9640 1 --2- 172.21.15.23:0/4043038662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bdc1016a0 0x7f6bdc101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.846+0000 7f6be08c9640 1 --2- 172.21.15.23:0/4043038662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc1076b0 0x7f6bdc107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.846+0000 7f6be08c9640 1 -- 172.21.15.23:0/4043038662 >> 172.21.15.23:0/4043038662 conn(0x7f6bdc078ad0 msgr2=0x7f6bdc078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:44.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.846+0000 7f6be08c9640 1 -- 172.21.15.23:0/4043038662 shutdown_connections 2024-03-24T10:40:44.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.846+0000 7f6be08c9640 1 -- 172.21.15.23:0/4043038662 wait complete. 2024-03-24T10:40:44.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.846+0000 7f6be08c9640 1 Processor -- start 2024-03-24T10:40:44.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.846+0000 7f6be08c9640 1 -- start start 2024-03-24T10:40:44.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6be08c9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bdc1016a0 0x7f6bdc073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6be08c9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bdc102060 0x7f6bdc073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6be08c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc1076b0 0x7f6bdc074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6be08c9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc1a60d0 con 0x7f6bdc102060 2024-03-24T10:40:44.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6be08c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc1a6240 con 0x7f6bdc1076b0 2024-03-24T10:40:44.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6be08c9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc1a6400 con 0x7f6bdc1016a0 2024-03-24T10:40:44.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6bd9d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bdc102060 0x7f6bdc073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6bd9d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bdc102060 0x7f6bdc073f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38380/0 (socket says 172.21.15.23:38380) 2024-03-24T10:40:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6bd9d9b640 1 -- 172.21.15.23:0/3097090371 learned_addr learned my addr 172.21.15.23:0/3097090371 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6bda59c640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bdc1016a0 0x7f6bdc073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6bdad9d640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc1076b0 0x7f6bdc074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6bd9d9b640 1 -- 172.21.15.23:0/3097090371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bdc1016a0 msgr2=0x7f6bdc073a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6bd9d9b640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bdc1016a0 0x7f6bdc073a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6bd9d9b640 1 -- 172.21.15.23:0/3097090371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc1076b0 msgr2=0x7f6bdc074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6bd9d9b640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc1076b0 0x7f6bdc074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6bd9d9b640 1 -- 172.21.15.23:0/3097090371 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bcc009560 con 0x7f6bdc102060 2024-03-24T10:40:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6bda59c640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bdc1016a0 0x7f6bdc073a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6bd9d9b640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bdc102060 0x7f6bdc073f50 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6bd000e9f0 tx=0x7f6bd000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.847+0000 7f6bcb7fe640 1 -- 172.21.15.23:0/3097090371 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0009720 con 0x7f6bdc102060 2024-03-24T10:40:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.848+0000 7f6be08c9640 1 -- 172.21.15.23:0/3097090371 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bdc1a66e0 con 0x7f6bdc102060 2024-03-24T10:40:44.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.848+0000 7f6bcb7fe640 1 -- 172.21.15.23:0/3097090371 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bd000cd80 con 0x7f6bdc102060 2024-03-24T10:40:44.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.848+0000 7f6bcb7fe640 1 -- 172.21.15.23:0/3097090371 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0010690 con 0x7f6bdc102060 2024-03-24T10:40:44.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.848+0000 7f6be08c9640 1 -- 172.21.15.23:0/3097090371 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bdc1a6ba0 con 0x7f6bdc102060 2024-03-24T10:40:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.850+0000 7f6bcb7fe640 1 -- 172.21.15.23:0/3097090371 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7f6bd00046c0 con 0x7f6bdc102060 2024-03-24T10:40:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.850+0000 7f6be3586740 1 Processor -- start 2024-03-24T10:40:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.850+0000 7f6be3586740 1 -- start start 2024-03-24T10:40:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.850+0000 7f6bcb7fe640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6bc40a2350 0x7f6bc40a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.850+0000 7f6be3586740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc01760 0x56266bc01b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.850+0000 7f6be3586740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc02080 0x56266bc0a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6be3586740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc0ac60 0x56266bc0d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6be3586740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56266bb6f530 con 0x56266bc01760 2024-03-24T10:40:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6be3586740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56266bb6ca60 con 0x56266bc0ac60 2024-03-24T10:40:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6be3586740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56266bb68240 con 0x56266bc02080 2024-03-24T10:40:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bdad9d640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6bc40a2350 0x7f6bc40a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bcb7fe640 1 -- 172.21.15.23:0/3097090371 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f6bd0014070 con 0x7f6bdc102060 2024-03-24T10:40:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bda59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc01760 0x56266bc01b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bd9d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc02080 0x56266bc0a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bda59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc01760 0x56266bc01b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38384/0 (socket says 172.21.15.23:38384) 2024-03-24T10:40:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bda59c640 1 -- 172.21.15.23:0/4066605533 learned_addr learned my addr 172.21.15.23:0/4066605533 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bdad9d640 1 --2- 172.21.15.23:0/4066605533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc0ac60 0x56266bc0d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bda59c640 1 -- 172.21.15.23:0/4066605533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc02080 msgr2=0x56266bc0a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bda59c640 1 --2- 172.21.15.23:0/4066605533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc02080 0x56266bc0a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bda59c640 1 -- 172.21.15.23:0/4066605533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc0ac60 msgr2=0x56266bc0d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bda59c640 1 --2- 172.21.15.23:0/4066605533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc0ac60 0x56266bc0d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bda59c640 1 -- 172.21.15.23:0/4066605533 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56266bb668c0 con 0x56266bc01760 2024-03-24T10:40:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bd9d9b640 1 --2- 172.21.15.23:0/4066605533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc02080 0x56266bc0a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bdad9d640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6bc40a2350 0x7f6bc40a4810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc005940 tx=0x7f6bcc005be0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bda59c640 1 --2- 172.21.15.23:0/4066605533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc01760 0x56266bc01b40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4005f40 tx=0x7f6bd4011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ab91b5ad2b0b3a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bc8ff9640 1 -- 172.21.15.23:0/4066605533 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4011220 con 0x56266bc01760 2024-03-24T10:40:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bc8ff9640 1 -- 172.21.15.23:0/4066605533 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bd400f920 con 0x56266bc01760 2024-03-24T10:40:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.851+0000 7f6bc8ff9640 1 -- 172.21.15.23:0/4066605533 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4004490 con 0x56266bc01760 2024-03-24T10:40:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.852+0000 7f6be3586740 1 -- 172.21.15.23:0/4066605533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc01760 msgr2=0x56266bc01b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.852+0000 7f6be3586740 1 --2- 172.21.15.23:0/4066605533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc01760 0x56266bc01b40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4005f40 tx=0x7f6bd4011cb0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.852+0000 7f6be3586740 1 -- 172.21.15.23:0/4066605533 shutdown_connections 2024-03-24T10:40:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.852+0000 7f6be3586740 1 --2- 172.21.15.23:0/4066605533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc0ac60 0x56266bc0d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.852+0000 7f6be3586740 1 --2- 172.21.15.23:0/4066605533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc02080 0x56266bc0a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.852+0000 7f6be3586740 1 --2- 172.21.15.23:0/4066605533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc01760 0x56266bc01b40 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.852+0000 7f6be3586740 1 -- 172.21.15.23:0/4066605533 >> 172.21.15.23:0/4066605533 conn(0x56266bbfbe90 msgr2=0x56266bbfc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.852+0000 7f6be3586740 1 -- 172.21.15.23:0/4066605533 shutdown_connections 2024-03-24T10:40:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.852+0000 7f6be3586740 1 -- 172.21.15.23:0/4066605533 wait complete. 2024-03-24T10:40:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.852+0000 7f6be3586740 1 -- start start 2024-03-24T10:40:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6be3586740 1 Processor -- start 2024-03-24T10:40:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6be3586740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc01760 0x56266bc07c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6be3586740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc02080 0x56266bc081d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6be3586740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc0ac60 0x56266bc08710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6be3586740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56266bb0cf80 con 0x56266bc0ac60 2024-03-24T10:40:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6be3586740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56266b83b4b0 con 0x56266bc01760 2024-03-24T10:40:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6be3586740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56266baedad0 con 0x56266bc02080 2024-03-24T10:40:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6be3586740 10 client.0 _ll_register_callbacks cb 0x56266bb20b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6bdad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc0ac60 0x56266bc08710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6bda59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc01760 0x56266bc07c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6bd9d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc02080 0x56266bc081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6bdad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc0ac60 0x56266bc08710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38388/0 (socket says 172.21.15.23:38388) 2024-03-24T10:40:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6bdad9d640 1 -- 172.21.15.23:0/4156037906 learned_addr learned my addr 172.21.15.23:0/4156037906 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6bda59c640 1 -- 172.21.15.23:0/4156037906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc02080 msgr2=0x56266bc081d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6bda59c640 1 --2- 172.21.15.23:0/4156037906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc02080 0x56266bc081d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6bda59c640 1 -- 172.21.15.23:0/4156037906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc0ac60 msgr2=0x56266bc08710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6bda59c640 1 --2- 172.21.15.23:0/4156037906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc0ac60 0x56266bc08710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6bda59c640 1 -- 172.21.15.23:0/4156037906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd4011480 con 0x56266bc01760 2024-03-24T10:40:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6bdad9d640 1 --2- 172.21.15.23:0/4156037906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc0ac60 0x56266bc08710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6bda59c640 1 --2- 172.21.15.23:0/4156037906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc01760 0x56266bc07c90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400f790 tx=0x7f6bd4004050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8aeda8dc461dbda4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.853+0000 7f6bc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:40:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6bc1ffb640 1 -- 172.21.15.23:0/4156037906 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4004a40 con 0x56266bc01760 2024-03-24T10:40:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6bc1ffb640 1 -- 172.21.15.23:0/4156037906 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f6bd4004ba0 con 0x56266bc01760 2024-03-24T10:40:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6be3586740 1 -- 172.21.15.23:0/4156037906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56266bb8c7a0 con 0x56266bc01760 2024-03-24T10:40:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6bc1ffb640 1 -- 172.21.15.23:0/4156037906 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6bd4004d60 con 0x56266bc01760 2024-03-24T10:40:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6be3586740 10 client.25535 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6be3586740 1 -- 172.21.15.23:0/4156037906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56266bb1e6d0 con 0x56266bc01760 2024-03-24T10:40:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6bc1ffb640 1 -- 172.21.15.23:0/4156037906 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56266bb8c7a0 con 0x56266bc01760 2024-03-24T10:40:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6bc1ffb640 1 -- 172.21.15.23:0/4156037906 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f6bd4005560 con 0x56266bc01760 2024-03-24T10:40:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6be3586740 10 client.25535 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6be3586740 10 client.25535 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6be3586740 20 client.25535 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6be3586740 1 --2- 172.21.15.23:0/4156037906 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56266bc1ad20 0x56266bc1d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6be3586740 4 client.25535 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6be3586740 1 -- 172.21.15.23:0/4156037906 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56266bb5e1f0 con 0x56266bc1ad20 2024-03-24T10:40:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.854+0000 7f6bd9d9b640 1 --2- 172.21.15.23:0/4156037906 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56266bc1ad20 0x56266bc1d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.855+0000 7f6bd9d9b640 1 --2- 172.21.15.23:0/4156037906 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56266bc1ad20 0x56266bc1d130 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5eaa839a9ee1d27 server_cookie=cc202fdef9d05555 in_seq=0 out_seq=0 2024-03-24T10:40:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.855+0000 7f6bc1ffb640 10 client.25535 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.855+0000 7f6bc1ffb640 1 -- 172.21.15.23:0/4156037906 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56266bb5e1f0 con 0x56266bc1ad20 2024-03-24T10:40:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.855+0000 7f6bc1ffb640 10 client.25535 handle_command_reply: tid=0 2024-03-24T10:40:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.856+0000 7f6be3586740 1 client.25535 shutdown 2024-03-24T10:40:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.856+0000 7f6be3586740 1 -- 172.21.15.23:0/4156037906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc01760 msgr2=0x56266bc07c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.856+0000 7f6be3586740 1 --2- 172.21.15.23:0/4156037906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc01760 0x56266bc07c90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400f790 tx=0x7f6bd4004050 comp rx=0 tx=0).stop 2024-03-24T10:40:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.856+0000 7f6be3586740 1 -- 172.21.15.23:0/4156037906 shutdown_connections 2024-03-24T10:40:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.856+0000 7f6be3586740 1 --2- 172.21.15.23:0/4156037906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc0ac60 0x56266bc08710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.856+0000 7f6be3586740 1 --2- 172.21.15.23:0/4156037906 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56266bc1ad20 0x56266bc1d130 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.856+0000 7f6be3586740 1 --2- 172.21.15.23:0/4156037906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc02080 0x56266bc081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.856+0000 7f6be3586740 1 --2- 172.21.15.23:0/4156037906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc01760 0x56266bc07c90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.856+0000 7f6bc1ffb640 0 client.25535 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.856+0000 7f6be3586740 1 -- 172.21.15.23:0/4156037906 >> 172.21.15.23:0/4156037906 conn(0x56266bbfbe90 msgr2=0x56266bbfdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.856+0000 7f6be3586740 1 -- 172.21.15.23:0/4156037906 shutdown_connections 2024-03-24T10:40:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.856+0000 7f6be3586740 1 -- 172.21.15.23:0/4156037906 wait complete. 2024-03-24T10:40:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.856+0000 7f6be3586740 20 client.25535 trim_cache size 0 max 16384 2024-03-24T10:40:44.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6be3586740 1 Processor -- start 2024-03-24T10:40:44.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6be3586740 1 -- start start 2024-03-24T10:40:44.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6be3586740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc01760 0x56266bc0a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6be3586740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc02080 0x56266bc164f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6be3586740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc0ac60 0x56266bc22a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6be3586740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56266bc177e0 con 0x56266bc01760 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6be3586740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56266bbf7620 con 0x56266bc02080 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6be3586740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56266bbfb630 con 0x56266bc0ac60 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6bda59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc01760 0x56266bc0a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6bda59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc01760 0x56266bc0a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38390/0 (socket says 172.21.15.23:38390) 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6bda59c640 1 -- 172.21.15.23:0/2137776878 learned_addr learned my addr 172.21.15.23:0/2137776878 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6bd9d9b640 1 --2- 172.21.15.23:0/2137776878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc02080 0x56266bc164f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6bdad9d640 1 --2- 172.21.15.23:0/2137776878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc0ac60 0x56266bc22a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6bda59c640 1 -- 172.21.15.23:0/2137776878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc0ac60 msgr2=0x56266bc22a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6bda59c640 1 --2- 172.21.15.23:0/2137776878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc0ac60 0x56266bc22a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6bda59c640 1 -- 172.21.15.23:0/2137776878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc02080 msgr2=0x56266bc164f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6bda59c640 1 --2- 172.21.15.23:0/2137776878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc02080 0x56266bc164f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6bda59c640 1 -- 172.21.15.23:0/2137776878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56266bb02c60 con 0x56266bc01760 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6bda59c640 1 --2- 172.21.15.23:0/2137776878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc01760 0x56266bc0a3c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4006010 tx=0x7f6bd4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f122af1f73a78716 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.874+0000 7f6bc3fff640 1 -- 172.21.15.23:0/2137776878 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4003dc0 con 0x56266bc01760 2024-03-24T10:40:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.875+0000 7f6bc3fff640 1 -- 172.21.15.23:0/2137776878 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bd4023ce0 con 0x56266bc01760 2024-03-24T10:40:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.875+0000 7f6bc3fff640 1 -- 172.21.15.23:0/2137776878 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd40054e0 con 0x56266bc01760 2024-03-24T10:40:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.875+0000 7f6be3586740 1 -- 172.21.15.23:0/2137776878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc01760 msgr2=0x56266bc0a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.875+0000 7f6be3586740 1 --2- 172.21.15.23:0/2137776878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc01760 0x56266bc0a3c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4006010 tx=0x7f6bd4009f80 comp rx=0 tx=0).stop 2024-03-24T10:40:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.875+0000 7f6be3586740 1 -- 172.21.15.23:0/2137776878 shutdown_connections 2024-03-24T10:40:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.875+0000 7f6be3586740 1 --2- 172.21.15.23:0/2137776878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc0ac60 0x56266bc22a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.875+0000 7f6be3586740 1 --2- 172.21.15.23:0/2137776878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc02080 0x56266bc164f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.875+0000 7f6be3586740 1 --2- 172.21.15.23:0/2137776878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc01760 0x56266bc0a3c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.875+0000 7f6be3586740 1 -- 172.21.15.23:0/2137776878 >> 172.21.15.23:0/2137776878 conn(0x56266bbfbe90 msgr2=0x56266bbfaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.875+0000 7f6be3586740 1 -- 172.21.15.23:0/2137776878 shutdown_connections 2024-03-24T10:40:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.875+0000 7f6be3586740 1 -- 172.21.15.23:0/2137776878 wait complete. 2024-03-24T10:40:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6be3586740 1 -- start start 2024-03-24T10:40:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6be3586740 1 Processor -- start 2024-03-24T10:40:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6be3586740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc01760 0x56266bc12d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6be3586740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc02080 0x56266bc0faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6be3586740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc0ac60 0x56266bc10030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6be3586740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56266bc10730 con 0x56266bc02080 2024-03-24T10:40:44.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6be3586740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56266bc108a0 con 0x56266bc01760 2024-03-24T10:40:44.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6be3586740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56266bc81080 con 0x56266bc0ac60 2024-03-24T10:40:44.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6be3586740 10 client.0 _ll_register_callbacks cb 0x7f6bd40116a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:44.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6bd9d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc02080 0x56266bc0faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6bd9d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc02080 0x56266bc0faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38398/0 (socket says 172.21.15.23:38398) 2024-03-24T10:40:44.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6bd9d9b640 1 -- 172.21.15.23:0/795908335 learned_addr learned my addr 172.21.15.23:0/795908335 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:44.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6bda59c640 1 --2- 172.21.15.23:0/795908335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc01760 0x56266bc12d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6bdad9d640 1 --2- 172.21.15.23:0/795908335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc0ac60 0x56266bc10030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6bd9d9b640 1 -- 172.21.15.23:0/795908335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc0ac60 msgr2=0x56266bc10030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:44.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6bd9d9b640 1 --2- 172.21.15.23:0/795908335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc0ac60 0x56266bc10030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6bd9d9b640 1 -- 172.21.15.23:0/795908335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc01760 msgr2=0x56266bc12d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:44.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6bd9d9b640 1 --2- 172.21.15.23:0/795908335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc01760 0x56266bc12d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6bd9d9b640 1 -- 172.21.15.23:0/795908335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56266bb1e6d0 con 0x56266bc02080 2024-03-24T10:40:44.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6bda59c640 1 --2- 172.21.15.23:0/795908335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc01760 0x56266bc12d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:44.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6bd9d9b640 1 --2- 172.21.15.23:0/795908335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc02080 0x56266bc0faf0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6bd009ab70 tx=0x7f6bd0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32931d2ee67633c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:44.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.876+0000 7f6bc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:44.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6bc1ffb640 1 -- 172.21.15.23:0/795908335 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd008a1c0 con 0x56266bc02080 2024-03-24T10:40:44.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6bc1ffb640 1 -- 172.21.15.23:0/795908335 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f6bd00a3610 con 0x56266bc02080 2024-03-24T10:40:44.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6be3586740 1 -- 172.21.15.23:0/795908335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56266bbf70d0 con 0x56266bc02080 2024-03-24T10:40:44.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6bc1ffb640 1 -- 172.21.15.23:0/795908335 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6bd00974e0 con 0x56266bc02080 2024-03-24T10:40:44.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6be3586740 10 client.15666 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:44.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6be3586740 1 -- 172.21.15.23:0/795908335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56266bb28970 con 0x56266bc02080 2024-03-24T10:40:44.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6bc1ffb640 1 -- 172.21.15.23:0/795908335 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bd00b7020 con 0x56266bc02080 2024-03-24T10:40:44.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6bc1ffb640 1 -- 172.21.15.23:0/795908335 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f6bd001f060 con 0x56266bc02080 2024-03-24T10:40:44.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6be3586740 10 client.15666 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:44.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6be3586740 10 client.15666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:44.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6be3586740 20 client.15666 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:44.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6be3586740 1 --2- 172.21.15.23:0/795908335 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56266bc82850 0x56266bc84c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:44.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6be3586740 4 client.15666 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:40:44.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6be3586740 1 -- 172.21.15.23:0/795908335 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f6bd400f920 con 0x56266bc82850 2024-03-24T10:40:44.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.877+0000 7f6bda59c640 1 --2- 172.21.15.23:0/795908335 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56266bc82850 0x56266bc84c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:44.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.878+0000 7f6bc1ffb640 10 client.15666 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:44.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.878+0000 7f6bda59c640 1 --2- 172.21.15.23:0/795908335 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56266bc82850 0x56266bc84c70 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=365ddc42bed8af6f server_cookie=7daed5d64bf85cab in_seq=0 out_seq=0 2024-03-24T10:40:44.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6bc1ffb640 1 -- 172.21.15.23:0/795908335 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f6bd400f920 con 0x56266bc82850 2024-03-24T10:40:44.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6bc1ffb640 10 client.15666 handle_command_reply: tid=0 2024-03-24T10:40:44.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6be3586740 1 client.15666 shutdown 2024-03-24T10:40:44.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6be3586740 1 -- 172.21.15.23:0/795908335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc02080 msgr2=0x56266bc0faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:44.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6be3586740 1 --2- 172.21.15.23:0/795908335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc02080 0x56266bc0faf0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6bd009ab70 tx=0x7f6bd0009310 comp rx=0 tx=0).stop 2024-03-24T10:40:44.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6be3586740 1 -- 172.21.15.23:0/795908335 shutdown_connections 2024-03-24T10:40:44.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6be3586740 1 --2- 172.21.15.23:0/795908335 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56266bc82850 0x56266bc84c70 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6be3586740 1 --2- 172.21.15.23:0/795908335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56266bc0ac60 0x56266bc10030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6be3586740 1 --2- 172.21.15.23:0/795908335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56266bc02080 0x56266bc0faf0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6be3586740 1 --2- 172.21.15.23:0/795908335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56266bc01760 0x56266bc12d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6be3586740 1 -- 172.21.15.23:0/795908335 >> 172.21.15.23:0/795908335 conn(0x56266bbfbe90 msgr2=0x56266bc07c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:44.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6bc1ffb640 0 client.15666 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:44.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6be3586740 1 -- 172.21.15.23:0/795908335 shutdown_connections 2024-03-24T10:40:44.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6be3586740 1 -- 172.21.15.23:0/795908335 wait complete. 2024-03-24T10:40:44.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.879+0000 7f6be3586740 20 client.15666 trim_cache size 0 max 16384 2024-03-24T10:40:44.885 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:40:44.885 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:40:44.886 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "59f1deef-1027-4ad0-86ed-f27a7cfac2cf", 2024-03-24T10:40:44.886 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:40:44.886 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:40:44.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.880+0000 7f6be08c9640 1 -- 172.21.15.23:0/3097090371 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6bc40a2350 msgr2=0x7f6bc40a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:44.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.880+0000 7f6be08c9640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6bc40a2350 0x7f6bc40a4810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc005940 tx=0x7f6bcc005be0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.880+0000 7f6be08c9640 1 -- 172.21.15.23:0/3097090371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bdc102060 msgr2=0x7f6bdc073f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:44.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.880+0000 7f6be08c9640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bdc102060 0x7f6bdc073f50 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6bd000e9f0 tx=0x7f6bd000edf0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.880+0000 7f6be08c9640 1 -- 172.21.15.23:0/3097090371 shutdown_connections 2024-03-24T10:40:44.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.880+0000 7f6be08c9640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6bc40a2350 0x7f6bc40a4810 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.880+0000 7f6be08c9640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc1076b0 0x7f6bdc074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.880+0000 7f6be08c9640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bdc102060 0x7f6bdc073f50 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.880+0000 7f6be08c9640 1 --2- 172.21.15.23:0/3097090371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bdc1016a0 0x7f6bdc073a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:44.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.880+0000 7f6be08c9640 1 -- 172.21.15.23:0/3097090371 >> 172.21.15.23:0/3097090371 conn(0x7f6bdc078ad0 msgr2=0x7f6bdc104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:44.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.880+0000 7f6be08c9640 1 -- 172.21.15.23:0/3097090371 shutdown_connections 2024-03-24T10:40:44.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:44.880+0000 7f6be08c9640 1 -- 172.21.15.23:0/3097090371 wait complete. 2024-03-24T10:40:44.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:40:44.892 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:40:45.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.038+0000 7ff399502640 1 -- 172.21.15.23:0/2761837235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff394107760 msgr2=0x7ff394107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.038+0000 7ff399502640 1 --2- 172.21.15.23:0/2761837235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff394107760 0x7ff394107b40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009930 tx=0x7ff38c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:45.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.039+0000 7ff399502640 1 -- 172.21.15.23:0/2761837235 shutdown_connections 2024-03-24T10:40:45.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.039+0000 7ff399502640 1 --2- 172.21.15.23:0/2761837235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3940692d0 0x7ff39410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.039+0000 7ff399502640 1 --2- 172.21.15.23:0/2761837235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394068910 0x7ff394068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.039+0000 7ff399502640 1 --2- 172.21.15.23:0/2761837235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff394107760 0x7ff394107b40 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.039+0000 7ff399502640 1 -- 172.21.15.23:0/2761837235 >> 172.21.15.23:0/2761837235 conn(0x7ff394077710 msgr2=0x7ff394077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.039+0000 7ff399502640 1 -- 172.21.15.23:0/2761837235 shutdown_connections 2024-03-24T10:40:45.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.039+0000 7ff399502640 1 -- 172.21.15.23:0/2761837235 wait complete. 2024-03-24T10:40:45.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.039+0000 7ff399502640 1 Processor -- start 2024-03-24T10:40:45.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.039+0000 7ff399502640 1 -- start start 2024-03-24T10:40:45.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff399502640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394068910 0x7ff394197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff399502640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3940692d0 0x7ff394198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff399502640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff394107760 0x7ff39419c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff399502640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff39419cbf0 con 0x7ff394107760 2024-03-24T10:40:45.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff399502640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff39419cd60 con 0x7ff3940692d0 2024-03-24T10:40:45.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff399502640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff39419cf20 con 0x7ff394068910 2024-03-24T10:40:45.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff3937fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff394107760 0x7ff39419c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff3937fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff394107760 0x7ff39419c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38416/0 (socket says 172.21.15.23:38416) 2024-03-24T10:40:45.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff3937fe640 1 -- 172.21.15.23:0/1734049076 learned_addr learned my addr 172.21.15.23:0/1734049076 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff392ffd640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394068910 0x7ff394197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff3927fc640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3940692d0 0x7ff394198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff3937fe640 1 -- 172.21.15.23:0/1734049076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394068910 msgr2=0x7ff394197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff3937fe640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394068910 0x7ff394197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff3937fe640 1 -- 172.21.15.23:0/1734049076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3940692d0 msgr2=0x7ff394198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff3937fe640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3940692d0 0x7ff394198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff3937fe640 1 -- 172.21.15.23:0/1734049076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37c009490 con 0x7ff394107760 2024-03-24T10:40:45.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff392ffd640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394068910 0x7ff394197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:45.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff3937fe640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff394107760 0x7ff39419c5f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff38400eb10 tx=0x7ff38400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff38bfff640 1 -- 172.21.15.23:0/1734049076 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384004b20 con 0x7ff394107760 2024-03-24T10:40:45.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff38bfff640 1 -- 172.21.15.23:0/1734049076 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff384004c80 con 0x7ff394107760 2024-03-24T10:40:45.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff399502640 1 -- 172.21.15.23:0/1734049076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff38c009560 con 0x7ff394107760 2024-03-24T10:40:45.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff38bfff640 1 -- 172.21.15.23:0/1734049076 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384010650 con 0x7ff394107760 2024-03-24T10:40:45.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.040+0000 7ff399502640 1 -- 172.21.15.23:0/1734049076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3941a48a0 con 0x7ff394107760 2024-03-24T10:40:45.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff38bfff640 1 -- 172.21.15.23:0/1734049076 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7ff384005020 con 0x7ff394107760 2024-03-24T10:40:45.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff39c1be740 1 Processor -- start 2024-03-24T10:40:45.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff39c1be740 1 -- start start 2024-03-24T10:40:45.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff39c1be740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bf21d0 0x55e346bf25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff39c1be740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346bfb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff39c1be740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bfb6d0 0x55e346bfdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff39c1be740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e346b16f30 con 0x55e346bfb6d0 2024-03-24T10:40:45.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff39c1be740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e346b60810 con 0x55e346bf21d0 2024-03-24T10:40:45.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff39c1be740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e346b5dd40 con 0x55e346bf2af0 2024-03-24T10:40:45.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff38bfff640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff3800a2350 0x7ff3800a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff3927fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bfb6d0 0x55e346bfdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff3927fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bfb6d0 0x55e346bfdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38426/0 (socket says 172.21.15.23:38426) 2024-03-24T10:40:45.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff3927fc640 1 -- 172.21.15.23:0/354208210 learned_addr learned my addr 172.21.15.23:0/354208210 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff392ffd640 1 --2- 172.21.15.23:0/354208210 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346bfb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff3927fc640 1 --2- 172.21.15.23:0/354208210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bf21d0 0x55e346bf25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.043+0000 7ff3937fe640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff3800a2350 0x7ff3800a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.044+0000 7ff38bfff640 1 -- 172.21.15.23:0/1734049076 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7ff384014070 con 0x7ff394107760 2024-03-24T10:40:45.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.044+0000 7ff3927fc640 1 -- 172.21.15.23:0/354208210 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 msgr2=0x55e346bfb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.044+0000 7ff3927fc640 1 --2- 172.21.15.23:0/354208210 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346bfb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.044+0000 7ff3927fc640 1 -- 172.21.15.23:0/354208210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bf21d0 msgr2=0x55e346bf25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.044+0000 7ff3927fc640 1 --2- 172.21.15.23:0/354208210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bf21d0 0x55e346bf25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.044+0000 7ff3927fc640 1 -- 172.21.15.23:0/354208210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e346b1aea0 con 0x55e346bfb6d0 2024-03-24T10:40:45.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.044+0000 7ff392ffd640 1 --2- 172.21.15.23:0/354208210 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346bfb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:45.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.044+0000 7ff3927fc640 1 --2- 172.21.15.23:0/354208210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bfb6d0 0x55e346bfdac0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff37c011990 tx=0x7ff37c009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8104c0e13a1533ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.044+0000 7ff3937fe640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff3800a2350 0x7ff3800a4810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff3840cbeb0 tx=0x7ff384084370 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.044+0000 7ff3897fa640 1 -- 172.21.15.23:0/354208210 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c011020 con 0x55e346bfb6d0 2024-03-24T10:40:45.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.044+0000 7ff3897fa640 1 -- 172.21.15.23:0/354208210 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff37c011180 con 0x55e346bfb6d0 2024-03-24T10:40:45.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.044+0000 7ff3897fa640 1 -- 172.21.15.23:0/354208210 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c0035f0 con 0x55e346bfb6d0 2024-03-24T10:40:45.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.045+0000 7ff39c1be740 1 -- 172.21.15.23:0/354208210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bfb6d0 msgr2=0x55e346bfdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.045+0000 7ff39c1be740 1 --2- 172.21.15.23:0/354208210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bfb6d0 0x55e346bfdac0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff37c011990 tx=0x7ff37c009370 comp rx=0 tx=0).stop 2024-03-24T10:40:45.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.045+0000 7ff39c1be740 1 -- 172.21.15.23:0/354208210 shutdown_connections 2024-03-24T10:40:45.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.045+0000 7ff39c1be740 1 --2- 172.21.15.23:0/354208210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bfb6d0 0x55e346bfdac0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.045+0000 7ff39c1be740 1 --2- 172.21.15.23:0/354208210 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346bfb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.045+0000 7ff39c1be740 1 --2- 172.21.15.23:0/354208210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bf21d0 0x55e346bf25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.045+0000 7ff39c1be740 1 -- 172.21.15.23:0/354208210 >> 172.21.15.23:0/354208210 conn(0x55e346bec900 msgr2=0x55e346becce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.045+0000 7ff39c1be740 1 -- 172.21.15.23:0/354208210 shutdown_connections 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.045+0000 7ff39c1be740 1 -- 172.21.15.23:0/354208210 wait complete. 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.045+0000 7ff39c1be740 1 -- start start 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.045+0000 7ff39c1be740 1 Processor -- start 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff39c1be740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bf21d0 0x55e346bf8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff39c1be740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346bf8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff39c1be740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bfb6d0 0x55e346bf91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff39c1be740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e346b59520 con 0x55e346bf21d0 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff39c1be740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e34682c430 con 0x55e346bfb6d0 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff39c1be740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e3467e8e00 con 0x55e346bf2af0 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff39c1be740 10 client.0 _ll_register_callbacks cb 0x55e346af7d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:45.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff3927fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bf21d0 0x55e346bf8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff3927fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bf21d0 0x55e346bf8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38434/0 (socket says 172.21.15.23:38434) 2024-03-24T10:40:45.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff3927fc640 1 -- 172.21.15.23:0/230949118 learned_addr learned my addr 172.21.15.23:0/230949118 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff392ffd640 1 --2- 172.21.15.23:0/230949118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346bf8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff3927fc640 1 --2- 172.21.15.23:0/230949118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bfb6d0 0x55e346bf91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff3927fc640 1 -- 172.21.15.23:0/230949118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 msgr2=0x55e346bf8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff3927fc640 1 --2- 172.21.15.23:0/230949118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346bf8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff3927fc640 1 -- 172.21.15.23:0/230949118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bfb6d0 msgr2=0x55e346bf91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff3927fc640 1 --2- 172.21.15.23:0/230949118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bfb6d0 0x55e346bf91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff3927fc640 1 -- 172.21.15.23:0/230949118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff37c019040 con 0x55e346bf21d0 2024-03-24T10:40:45.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff3927fc640 1 --2- 172.21.15.23:0/230949118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bf21d0 0x55e346bf8760 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009e20 tx=0x7ff37c00fd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37943a736ac2bc32 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff37a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:45.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff37a7fc640 1 -- 172.21.15.23:0/230949118 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c0041b0 con 0x55e346bf21d0 2024-03-24T10:40:45.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff37a7fc640 1 -- 172.21.15.23:0/230949118 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7ff37c000e40 con 0x55e346bf21d0 2024-03-24T10:40:45.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff39c1be740 1 -- 172.21.15.23:0/230949118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e3468587e0 con 0x55e346bf21d0 2024-03-24T10:40:45.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff37a7fc640 1 -- 172.21.15.23:0/230949118 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff37c004310 con 0x55e346bf21d0 2024-03-24T10:40:45.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.046+0000 7ff39c1be740 10 client.15678 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:45.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.047+0000 7ff39c1be740 1 -- 172.21.15.23:0/230949118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e346b57ba0 con 0x55e346bf21d0 2024-03-24T10:40:45.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.047+0000 7ff37a7fc640 1 -- 172.21.15.23:0/230949118 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e3468587e0 con 0x55e346bf21d0 2024-03-24T10:40:45.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.047+0000 7ff37a7fc640 1 -- 172.21.15.23:0/230949118 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff37c01d070 con 0x55e346bf21d0 2024-03-24T10:40:45.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.047+0000 7ff39c1be740 10 client.15678 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:45.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.047+0000 7ff39c1be740 10 client.15678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:45.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.047+0000 7ff39c1be740 20 client.15678 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:45.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.047+0000 7ff39c1be740 1 --2- 172.21.15.23:0/230949118 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e346c0bb90 0x55e346c0dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.047+0000 7ff39c1be740 4 client.15678 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:45.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.047+0000 7ff39c1be740 1 -- 172.21.15.23:0/230949118 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e346895720 con 0x55e346c0bb90 2024-03-24T10:40:45.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.047+0000 7ff392ffd640 1 --2- 172.21.15.23:0/230949118 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e346c0bb90 0x55e346c0dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff37a7fc640 10 client.15678 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:45.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff392ffd640 1 --2- 172.21.15.23:0/230949118 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e346c0bb90 0x55e346c0dfb0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12b9835e57d9a9ad server_cookie=5df78aff55b05357 in_seq=0 out_seq=0 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff37a7fc640 1 -- 172.21.15.23:0/230949118 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e346895720 con 0x55e346c0bb90 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff37a7fc640 10 client.15678 handle_command_reply: tid=0 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff39c1be740 1 client.15678 shutdown 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff39c1be740 1 -- 172.21.15.23:0/230949118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bf21d0 msgr2=0x55e346bf8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff39c1be740 1 --2- 172.21.15.23:0/230949118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bf21d0 0x55e346bf8760 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009e20 tx=0x7ff37c00fd20 comp rx=0 tx=0).stop 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff39c1be740 1 -- 172.21.15.23:0/230949118 shutdown_connections 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff39c1be740 1 --2- 172.21.15.23:0/230949118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bfb6d0 0x55e346bf91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff39c1be740 1 --2- 172.21.15.23:0/230949118 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e346c0bb90 0x55e346c0dfb0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff39c1be740 1 --2- 172.21.15.23:0/230949118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346bf8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff39c1be740 1 --2- 172.21.15.23:0/230949118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bf21d0 0x55e346bf8760 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff37a7fc640 0 client.15678 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff39c1be740 1 -- 172.21.15.23:0/230949118 >> 172.21.15.23:0/230949118 conn(0x55e346bec900 msgr2=0x55e346bee650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff39c1be740 1 -- 172.21.15.23:0/230949118 shutdown_connections 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.048+0000 7ff39c1be740 1 -- 172.21.15.23:0/230949118 wait complete. 2024-03-24T10:40:45.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.049+0000 7ff39c1be740 20 client.15678 trim_cache size 0 max 16384 2024-03-24T10:40:45.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.067+0000 7ff39c1be740 1 Processor -- start 2024-03-24T10:40:45.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.067+0000 7ff39c1be740 1 -- start start 2024-03-24T10:40:45.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.067+0000 7ff39c1be740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bf21d0 0x55e346bfadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.067+0000 7ff39c1be740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346c06f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.067+0000 7ff39c1be740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bfb6d0 0x55e346beb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.067+0000 7ff39c1be740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e346ade430 con 0x55e346bf21d0 2024-03-24T10:40:45.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.067+0000 7ff39c1be740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e346bf1e60 con 0x55e346bfb6d0 2024-03-24T10:40:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.067+0000 7ff39c1be740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e346bf3540 con 0x55e346bf2af0 2024-03-24T10:40:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.067+0000 7ff3927fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bf21d0 0x55e346bfadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.067+0000 7ff3927fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bf21d0 0x55e346bfadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38442/0 (socket says 172.21.15.23:38442) 2024-03-24T10:40:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.067+0000 7ff3927fc640 1 -- 172.21.15.23:0/2445069563 learned_addr learned my addr 172.21.15.23:0/2445069563 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.067+0000 7ff392ffd640 1 --2- 172.21.15.23:0/2445069563 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346c06f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.067+0000 7ff3927fc640 1 --2- 172.21.15.23:0/2445069563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bfb6d0 0x55e346beb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.068+0000 7ff3927fc640 1 -- 172.21.15.23:0/2445069563 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 msgr2=0x55e346c06f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.068+0000 7ff3927fc640 1 --2- 172.21.15.23:0/2445069563 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346c06f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.068+0000 7ff3927fc640 1 -- 172.21.15.23:0/2445069563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bfb6d0 msgr2=0x55e346beb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.068+0000 7ff3927fc640 1 --2- 172.21.15.23:0/2445069563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bfb6d0 0x55e346beb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.068+0000 7ff3927fc640 1 -- 172.21.15.23:0/2445069563 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e346af3b20 con 0x55e346bf21d0 2024-03-24T10:40:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.068+0000 7ff3927fc640 1 --2- 172.21.15.23:0/2445069563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bf21d0 0x55e346bfadf0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00a010 tx=0x7ff37c005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7b46b944d680c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.068+0000 7ff388ff9640 1 -- 172.21.15.23:0/2445069563 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c009c20 con 0x55e346bf21d0 2024-03-24T10:40:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.068+0000 7ff388ff9640 1 -- 172.21.15.23:0/2445069563 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff37c009980 con 0x55e346bf21d0 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.068+0000 7ff388ff9640 1 -- 172.21.15.23:0/2445069563 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c0051f0 con 0x55e346bf21d0 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 -- 172.21.15.23:0/2445069563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bf21d0 msgr2=0x55e346bfadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 --2- 172.21.15.23:0/2445069563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bf21d0 0x55e346bfadf0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00a010 tx=0x7ff37c005970 comp rx=0 tx=0).stop 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 -- 172.21.15.23:0/2445069563 shutdown_connections 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 --2- 172.21.15.23:0/2445069563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bfb6d0 0x55e346beb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 --2- 172.21.15.23:0/2445069563 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346c06f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 --2- 172.21.15.23:0/2445069563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bf21d0 0x55e346bfadf0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 -- 172.21.15.23:0/2445069563 >> 172.21.15.23:0/2445069563 conn(0x55e346bec900 msgr2=0x55e346beb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 -- 172.21.15.23:0/2445069563 shutdown_connections 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 -- 172.21.15.23:0/2445069563 wait complete. 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 -- start start 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 Processor -- start 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bf21d0 0x55e346c03980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346bfecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bfb6d0 0x55e346bff430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e346bffb00 con 0x55e346bfb6d0 2024-03-24T10:40:45.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e346c70eb0 con 0x55e346bf21d0 2024-03-24T10:40:45.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e346c70ff0 con 0x55e346bf2af0 2024-03-24T10:40:45.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff39c1be740 10 client.0 _ll_register_callbacks cb 0x7ff37c011480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:45.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff3927fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bfb6d0 0x55e346bff430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff3927fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bfb6d0 0x55e346bff430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38458/0 (socket says 172.21.15.23:38458) 2024-03-24T10:40:45.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.069+0000 7ff3927fc640 1 -- 172.21.15.23:0/1922087021 learned_addr learned my addr 172.21.15.23:0/1922087021 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff392ffd640 1 --2- 172.21.15.23:0/1922087021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346bfecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff3927fc640 1 --2- 172.21.15.23:0/1922087021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bf21d0 0x55e346c03980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff3927fc640 1 -- 172.21.15.23:0/1922087021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 msgr2=0x55e346bfecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff3927fc640 1 --2- 172.21.15.23:0/1922087021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346bfecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff3927fc640 1 -- 172.21.15.23:0/1922087021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bf21d0 msgr2=0x55e346c03980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff3927fc640 1 --2- 172.21.15.23:0/1922087021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bf21d0 0x55e346c03980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff3927fc640 1 -- 172.21.15.23:0/1922087021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e346b57ba0 con 0x55e346bfb6d0 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff3927fc640 1 --2- 172.21.15.23:0/1922087021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bfb6d0 0x55e346bff430 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009ad0 tx=0x7ff37c004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2dd83830e56d187 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff37a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff37a7fc640 1 -- 172.21.15.23:0/1922087021 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c005d30 con 0x55e346bfb6d0 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff37a7fc640 1 -- 172.21.15.23:0/1922087021 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7ff37c0041b0 con 0x55e346bfb6d0 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff39c1be740 1 -- 172.21.15.23:0/1922087021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e3468e4890 con 0x55e346bfb6d0 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff37a7fc640 1 -- 172.21.15.23:0/1922087021 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff37c004370 con 0x55e346bfb6d0 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff39c1be740 10 client.15684 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff39c1be740 1 -- 172.21.15.23:0/1922087021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e346b19700 con 0x55e346bfb6d0 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.070+0000 7ff37a7fc640 1 -- 172.21.15.23:0/1922087021 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e3468e4890 con 0x55e346bfb6d0 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff37a7fc640 1 -- 172.21.15.23:0/1922087021 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff37c048070 con 0x55e346bfb6d0 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 10 client.15684 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 10 client.15684 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 20 client.15684 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 1 --2- 172.21.15.23:0/1922087021 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e346c721f0 0x55e346c745e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 4 client.15684 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:40:45.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 1 -- 172.21.15.23:0/1922087021 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e346804700 con 0x55e346c721f0 2024-03-24T10:40:45.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff392ffd640 1 --2- 172.21.15.23:0/1922087021 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e346c721f0 0x55e346c745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff392ffd640 1 --2- 172.21.15.23:0/1922087021 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e346c721f0 0x55e346c745e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f910a375828c32b server_cookie=ad4cd54e695f9325 in_seq=0 out_seq=0 2024-03-24T10:40:45.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff37a7fc640 10 client.15684 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:45.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff37a7fc640 1 -- 172.21.15.23:0/1922087021 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e346804700 con 0x55e346c721f0 2024-03-24T10:40:45.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff37a7fc640 10 client.15684 handle_command_reply: tid=0 2024-03-24T10:40:45.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 1 client.15684 shutdown 2024-03-24T10:40:45.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 1 -- 172.21.15.23:0/1922087021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bfb6d0 msgr2=0x55e346bff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 1 --2- 172.21.15.23:0/1922087021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bfb6d0 0x55e346bff430 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009ad0 tx=0x7ff37c004cc0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 1 -- 172.21.15.23:0/1922087021 shutdown_connections 2024-03-24T10:40:45.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 1 --2- 172.21.15.23:0/1922087021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e346bfb6d0 0x55e346bff430 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 1 --2- 172.21.15.23:0/1922087021 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e346c721f0 0x55e346c745e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 1 --2- 172.21.15.23:0/1922087021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e346bf2af0 0x55e346bfecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 1 --2- 172.21.15.23:0/1922087021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e346bf21d0 0x55e346c03980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff39c1be740 1 -- 172.21.15.23:0/1922087021 >> 172.21.15.23:0/1922087021 conn(0x55e346bec900 msgr2=0x55e346bebb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.071+0000 7ff37a7fc640 0 client.15684 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:45.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff39c1be740 1 -- 172.21.15.23:0/1922087021 shutdown_connections 2024-03-24T10:40:45.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff39c1be740 1 -- 172.21.15.23:0/1922087021 wait complete. 2024-03-24T10:40:45.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff39c1be740 20 client.15684 trim_cache size 0 max 16384 2024-03-24T10:40:45.079 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:40:45.079 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:40:45.079 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:40:45.079 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:40:45.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff399502640 1 -- 172.21.15.23:0/1734049076 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff3800a2350 msgr2=0x7ff3800a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff399502640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff3800a2350 0x7ff3800a4810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff3840cbeb0 tx=0x7ff384084370 comp rx=0 tx=0).stop 2024-03-24T10:40:45.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff399502640 1 -- 172.21.15.23:0/1734049076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff394107760 msgr2=0x7ff39419c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff399502640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff394107760 0x7ff39419c5f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff38400eb10 tx=0x7ff38400c2d0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff399502640 1 -- 172.21.15.23:0/1734049076 shutdown_connections 2024-03-24T10:40:45.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff399502640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff3800a2350 0x7ff3800a4810 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff399502640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff394107760 0x7ff39419c5f0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff399502640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3940692d0 0x7ff394198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff399502640 1 --2- 172.21.15.23:0/1734049076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394068910 0x7ff394197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff399502640 1 -- 172.21.15.23:0/1734049076 >> 172.21.15.23:0/1734049076 conn(0x7ff394077710 msgr2=0x7ff3940fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff399502640 1 -- 172.21.15.23:0/1734049076 shutdown_connections 2024-03-24T10:40:45.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.072+0000 7ff399502640 1 -- 172.21.15.23:0/1734049076 wait complete. 2024-03-24T10:40:45.083 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:40:45.083 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:40:45.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.189+0000 7fe8da356640 1 -- 172.21.15.23:0/3116688984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8d410bd10 msgr2=0x7fe8d410e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.189+0000 7fe8da356640 1 --2- 172.21.15.23:0/3116688984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8d410bd10 0x7fe8d410e140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc009be0 tx=0x7fe8cc02bf80 comp rx=0 tx=0).stop 2024-03-24T10:40:45.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.189+0000 7fe8da356640 1 -- 172.21.15.23:0/3116688984 shutdown_connections 2024-03-24T10:40:45.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.189+0000 7fe8da356640 1 --2- 172.21.15.23:0/3116688984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8d410bd10 0x7fe8d410e140 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.189+0000 7fe8da356640 1 --2- 172.21.15.23:0/3116688984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8d4103260 0x7fe8d410b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.189+0000 7fe8da356640 1 --2- 172.21.15.23:0/3116688984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8d4102940 0x7fe8d4102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.189+0000 7fe8da356640 1 -- 172.21.15.23:0/3116688984 >> 172.21.15.23:0/3116688984 conn(0x7fe8d4076990 msgr2=0x7fe8d4076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.189+0000 7fe8da356640 1 -- 172.21.15.23:0/3116688984 shutdown_connections 2024-03-24T10:40:45.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.189+0000 7fe8da356640 1 -- 172.21.15.23:0/3116688984 wait complete. 2024-03-24T10:40:45.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.190+0000 7fe8da356640 1 Processor -- start 2024-03-24T10:40:45.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.190+0000 7fe8da356640 1 -- start start 2024-03-24T10:40:45.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.190+0000 7fe8da356640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8d4102940 0x7fe8d419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.190+0000 7fe8da356640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8d4103260 0x7fe8d419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.190+0000 7fe8da356640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8d410bd10 0x7fe8d4197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.190+0000 7fe8da356640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d41982b0 con 0x7fe8d410bd10 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.190+0000 7fe8da356640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d4198420 con 0x7fe8d4103260 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.190+0000 7fe8da356640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d41985e0 con 0x7fe8d4102940 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.190+0000 7fe8d88f3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8d410bd10 0x7fe8d4197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.190+0000 7fe8d37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8d4103260 0x7fe8d419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.190+0000 7fe8d3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8d4102940 0x7fe8d419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.190+0000 7fe8d88f3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8d410bd10 0x7fe8d4197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38482/0 (socket says 172.21.15.23:38482) 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.190+0000 7fe8d37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8d4103260 0x7fe8d419dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:44966/0 (socket says 172.21.15.23:44966) 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.190+0000 7fe8d88f3640 1 -- 172.21.15.23:0/1391125969 learned_addr learned my addr 172.21.15.23:0/1391125969 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.191+0000 7fe8d88f3640 1 -- 172.21.15.23:0/1391125969 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8d4102940 msgr2=0x7fe8d419da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.191+0000 7fe8d88f3640 1 --2- 172.21.15.23:0/1391125969 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8d4102940 0x7fe8d419da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.191+0000 7fe8d88f3640 1 -- 172.21.15.23:0/1391125969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8d4103260 msgr2=0x7fe8d419dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.191+0000 7fe8d88f3640 1 --2- 172.21.15.23:0/1391125969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8d4103260 0x7fe8d419dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.191+0000 7fe8d88f3640 1 -- 172.21.15.23:0/1391125969 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c8009560 con 0x7fe8d410bd10 2024-03-24T10:40:45.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.191+0000 7fe8d37fe640 1 --2- 172.21.15.23:0/1391125969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8d4103260 0x7fe8d419dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:45.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.191+0000 7fe8d3fff640 1 --2- 172.21.15.23:0/1391125969 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8d4102940 0x7fe8d419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:45.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.191+0000 7fe8d88f3640 1 --2- 172.21.15.23:0/1391125969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8d410bd10 0x7fe8d4197b20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc036040 tx=0x7fe8cc0335f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.191+0000 7fe8d17fa640 1 -- 172.21.15.23:0/1391125969 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc003970 con 0x7fe8d410bd10 2024-03-24T10:40:45.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.191+0000 7fe8da356640 1 -- 172.21.15.23:0/1391125969 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8cc009840 con 0x7fe8d410bd10 2024-03-24T10:40:45.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.191+0000 7fe8d17fa640 1 -- 172.21.15.23:0/1391125969 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8cc004040 con 0x7fe8d410bd10 2024-03-24T10:40:45.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.191+0000 7fe8d17fa640 1 -- 172.21.15.23:0/1391125969 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8cc034590 con 0x7fe8d410bd10 2024-03-24T10:40:45.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.191+0000 7fe8da356640 1 -- 172.21.15.23:0/1391125969 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8d41a49e0 con 0x7fe8d410bd10 2024-03-24T10:40:45.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.192+0000 7fe8da356640 1 -- 172.21.15.23:0/1391125969 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8d4198a50 con 0x7fe8d410bd10 2024-03-24T10:40:45.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.193+0000 7fe8d17fa640 1 -- 172.21.15.23:0/1391125969 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7fe8cc0041b0 con 0x7fe8d410bd10 2024-03-24T10:40:45.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.194+0000 7fe8d17fa640 1 --2- 172.21.15.23:0/1391125969 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe8bc0a2350 0x7fe8bc0a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.194+0000 7fe8d3fff640 1 --2- 172.21.15.23:0/1391125969 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe8bc0a2350 0x7fe8bc0a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.194+0000 7fe8d17fa640 1 -- 172.21.15.23:0/1391125969 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe8cc0f1340 con 0x7fe8d410bd10 2024-03-24T10:40:45.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.194+0000 7fe8d3fff640 1 --2- 172.21.15.23:0/1391125969 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe8bc0a2350 0x7fe8bc0a4810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4009fc0 tx=0x7fe8c4005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.198+0000 7fe8d17fa640 1 -- 172.21.15.23:0/1391125969 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe8cc0b9e90 con 0x7fe8d410bd10 2024-03-24T10:40:45.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.443+0000 7fe8da356640 1 -- 172.21.15.23:0/1391125969 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8d4198fe0 con 0x7fe8d410bd10 2024-03-24T10:40:45.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.444+0000 7fe8d17fa640 1 -- 172.21.15.23:0/1391125969 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11866 (secure 0 0 0) 0x7fe8cc043090 con 0x7fe8d410bd10 2024-03-24T10:40:45.444 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:40:45.445 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:40:03.697229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24539,24503,24505]},"id":1}]} 2024-03-24T10:40:45.445 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 23 2024-03-24T10:40:45.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.446+0000 7fe8da356640 1 -- 172.21.15.23:0/1391125969 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe8bc0a2350 msgr2=0x7fe8bc0a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.446+0000 7fe8da356640 1 --2- 172.21.15.23:0/1391125969 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe8bc0a2350 0x7fe8bc0a4810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4009fc0 tx=0x7fe8c4005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:45.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.446+0000 7fe8da356640 1 -- 172.21.15.23:0/1391125969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8d410bd10 msgr2=0x7fe8d4197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.446+0000 7fe8da356640 1 --2- 172.21.15.23:0/1391125969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8d410bd10 0x7fe8d4197b20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc036040 tx=0x7fe8cc0335f0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.446+0000 7fe8da356640 1 -- 172.21.15.23:0/1391125969 shutdown_connections 2024-03-24T10:40:45.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.446+0000 7fe8da356640 1 --2- 172.21.15.23:0/1391125969 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe8bc0a2350 0x7fe8bc0a4810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.446+0000 7fe8da356640 1 --2- 172.21.15.23:0/1391125969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8d410bd10 0x7fe8d4197b20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.446+0000 7fe8da356640 1 --2- 172.21.15.23:0/1391125969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8d4103260 0x7fe8d419dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.446+0000 7fe8da356640 1 --2- 172.21.15.23:0/1391125969 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8d4102940 0x7fe8d419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.446+0000 7fe8da356640 1 -- 172.21.15.23:0/1391125969 >> 172.21.15.23:0/1391125969 conn(0x7fe8d4076990 msgr2=0x7fe8d40fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.446+0000 7fe8da356640 1 -- 172.21.15.23:0/1391125969 shutdown_connections 2024-03-24T10:40:45.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.446+0000 7fe8da356640 1 -- 172.21.15.23:0/1391125969 wait complete. 2024-03-24T10:40:45.457 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:40:45.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.563+0000 7fb6cfe9a640 1 -- 172.21.15.23:0/495382271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c80692d0 msgr2=0x7fb6c810e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.563+0000 7fb6cfe9a640 1 --2- 172.21.15.23:0/495382271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c80692d0 0x7fb6c810e570 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0009cd0 tx=0x7fb6c002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.563+0000 7fb6cfe9a640 1 -- 172.21.15.23:0/495382271 shutdown_connections 2024-03-24T10:40:45.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.563+0000 7fb6cfe9a640 1 --2- 172.21.15.23:0/495382271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c80692d0 0x7fb6c810e570 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.563+0000 7fb6cfe9a640 1 --2- 172.21.15.23:0/495382271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8068910 0x7fb6c8068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.563+0000 7fb6cfe9a640 1 --2- 172.21.15.23:0/495382271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6c8107760 0x7fb6c8107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.563+0000 7fb6cfe9a640 1 -- 172.21.15.23:0/495382271 >> 172.21.15.23:0/495382271 conn(0x7fb6c8077710 msgr2=0x7fb6c8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.563+0000 7fb6cfe9a640 1 -- 172.21.15.23:0/495382271 shutdown_connections 2024-03-24T10:40:45.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.563+0000 7fb6cfe9a640 1 -- 172.21.15.23:0/495382271 wait complete. 2024-03-24T10:40:45.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cfe9a640 1 Processor -- start 2024-03-24T10:40:45.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cfe9a640 1 -- start start 2024-03-24T10:40:45.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cfe9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8068910 0x7fb6c819d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cfe9a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c80692d0 0x7fb6c819de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cfe9a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6c8107760 0x7fb6c8197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cfe9a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c8198160 con 0x7fb6c80692d0 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cfe9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c81982d0 con 0x7fb6c8068910 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cfe9a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c8198490 con 0x7fb6c8107760 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cd435640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c80692d0 0x7fb6c819de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cd435640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c80692d0 0x7fb6c819de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38508/0 (socket says 172.21.15.23:38508) 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cd435640 1 -- 172.21.15.23:0/2099061385 learned_addr learned my addr 172.21.15.23:0/2099061385 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cdc36640 1 --2- 172.21.15.23:0/2099061385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8068910 0x7fb6c819d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6ce437640 1 --2- 172.21.15.23:0/2099061385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6c8107760 0x7fb6c8197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cd435640 1 -- 172.21.15.23:0/2099061385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6c8107760 msgr2=0x7fb6c8197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cd435640 1 --2- 172.21.15.23:0/2099061385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6c8107760 0x7fb6c8197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cd435640 1 -- 172.21.15.23:0/2099061385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8068910 msgr2=0x7fb6c819d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.564+0000 7fb6cd435640 1 --2- 172.21.15.23:0/2099061385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8068910 0x7fb6c819d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.565+0000 7fb6cd435640 1 -- 172.21.15.23:0/2099061385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6b8009490 con 0x7fb6c80692d0 2024-03-24T10:40:45.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.565+0000 7fb6cd435640 1 --2- 172.21.15.23:0/2099061385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c80692d0 0x7fb6c819de50 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800ea60 tx=0x7fb6b800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.565+0000 7fb6beffd640 1 -- 172.21.15.23:0/2099061385 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b800cde0 con 0x7fb6c80692d0 2024-03-24T10:40:45.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.565+0000 7fb6cfe9a640 1 -- 172.21.15.23:0/2099061385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c0009900 con 0x7fb6c80692d0 2024-03-24T10:40:45.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.565+0000 7fb6beffd640 1 -- 172.21.15.23:0/2099061385 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6b80035f0 con 0x7fb6c80692d0 2024-03-24T10:40:45.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.565+0000 7fb6beffd640 1 -- 172.21.15.23:0/2099061385 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8010690 con 0x7fb6c80692d0 2024-03-24T10:40:45.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.565+0000 7fb6cfe9a640 1 -- 172.21.15.23:0/2099061385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c8198960 con 0x7fb6c80692d0 2024-03-24T10:40:45.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.567+0000 7fb6beffd640 1 -- 172.21.15.23:0/2099061385 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7fb6b8003af0 con 0x7fb6c80692d0 2024-03-24T10:40:45.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.567+0000 7fb6d2a76740 1 Processor -- start 2024-03-24T10:40:45.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6d2a76740 1 -- start start 2024-03-24T10:40:45.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6d2a76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befe3f0 0x55568befe7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6d2a76740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568befed10 0x55568bf073b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6d2a76740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568bf078f0 0x55568bf09ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6d2a76740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55568be6c840 con 0x55568bf078f0 2024-03-24T10:40:45.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6d2a76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55568be69d70 con 0x55568befe3f0 2024-03-24T10:40:45.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6d2a76740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55568be65550 con 0x55568befed10 2024-03-24T10:40:45.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6beffd640 1 --2- 172.21.15.23:0/2099061385 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb6b40a2350 0x7fb6b40a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6cd435640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568bf078f0 0x55568bf09ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6cdc36640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568befed10 0x55568bf073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6ce437640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befe3f0 0x55568befe7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6cd435640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568bf078f0 0x55568bf09ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38522/0 (socket says 172.21.15.23:38522) 2024-03-24T10:40:45.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6cd435640 1 -- 172.21.15.23:0/83807717 learned_addr learned my addr 172.21.15.23:0/83807717 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6ce437640 1 --2- 172.21.15.23:0/2099061385 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb6b40a2350 0x7fb6b40a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6cdc36640 1 -- 172.21.15.23:0/83807717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befe3f0 msgr2=0x55568befe7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6cdc36640 1 --2- 172.21.15.23:0/83807717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befe3f0 0x55568befe7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6beffd640 1 -- 172.21.15.23:0/2099061385 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fb6b8014070 con 0x7fb6c80692d0 2024-03-24T10:40:45.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.568+0000 7fb6cdc36640 1 -- 172.21.15.23:0/83807717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568bf078f0 msgr2=0x55568bf09ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.569+0000 7fb6cdc36640 1 --2- 172.21.15.23:0/83807717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568bf078f0 0x55568bf09ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.569+0000 7fb6cdc36640 1 -- 172.21.15.23:0/83807717 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55568bdfade0 con 0x55568befed10 2024-03-24T10:40:45.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.569+0000 7fb6ce437640 1 --2- 172.21.15.23:0/2099061385 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb6b40a2350 0x7fb6b40a4810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb6c002c630 tx=0x7fb6c0005d80 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.569+0000 7fb6cdc36640 1 --2- 172.21.15.23:0/83807717 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568befed10 0x55568bf073b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4009bd0 tx=0x7fb6c4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a17136081962691a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.569+0000 7fb6b3fff640 1 -- 172.21.15.23:0/83807717 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4004230 con 0x55568befed10 2024-03-24T10:40:45.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.569+0000 7fb6b3fff640 1 -- 172.21.15.23:0/83807717 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6c4009e40 con 0x55568befed10 2024-03-24T10:40:45.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.569+0000 7fb6b3fff640 1 -- 172.21.15.23:0/83807717 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4004d00 con 0x55568befed10 2024-03-24T10:40:45.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.570+0000 7fb6d2a76740 1 -- 172.21.15.23:0/83807717 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568befed10 msgr2=0x55568bf073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.570+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/83807717 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568befed10 0x55568bf073b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4009bd0 tx=0x7fb6c4011810 comp rx=0 tx=0).stop 2024-03-24T10:40:45.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.570+0000 7fb6d2a76740 1 -- 172.21.15.23:0/83807717 shutdown_connections 2024-03-24T10:40:45.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.570+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/83807717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568bf078f0 0x55568bf09ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.570+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/83807717 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568befed10 0x55568bf073b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.570+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/83807717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befe3f0 0x55568befe7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.570+0000 7fb6d2a76740 1 -- 172.21.15.23:0/83807717 >> 172.21.15.23:0/83807717 conn(0x55568bef8b20 msgr2=0x55568bef8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.570+0000 7fb6d2a76740 1 -- 172.21.15.23:0/83807717 shutdown_connections 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.570+0000 7fb6d2a76740 1 -- 172.21.15.23:0/83807717 wait complete. 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.570+0000 7fb6d2a76740 1 -- start start 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.570+0000 7fb6d2a76740 1 Processor -- start 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6d2a76740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befe3f0 0x55568bf04970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6d2a76740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568befed10 0x55568bf04eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6d2a76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568bf078f0 0x55568bf053f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6d2a76740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55568bb38430 con 0x55568befe3f0 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6d2a76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55568baf4e00 con 0x55568bf078f0 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6d2a76740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55568bdea590 con 0x55568befed10 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6d2a76740 10 client.0 _ll_register_callbacks cb 0x55568be59fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6cdc36640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befe3f0 0x55568bf04970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6cdc36640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befe3f0 0x55568bf04970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38532/0 (socket says 172.21.15.23:38532) 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6cdc36640 1 -- 172.21.15.23:0/490031186 learned_addr learned my addr 172.21.15.23:0/490031186 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6cd435640 1 --2- 172.21.15.23:0/490031186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568befed10 0x55568bf04eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6ce437640 1 --2- 172.21.15.23:0/490031186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568bf078f0 0x55568bf053f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6cdc36640 1 -- 172.21.15.23:0/490031186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568befed10 msgr2=0x55568bf04eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6cdc36640 1 --2- 172.21.15.23:0/490031186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568befed10 0x55568bf04eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6cdc36640 1 -- 172.21.15.23:0/490031186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568bf078f0 msgr2=0x55568bf053f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6cdc36640 1 --2- 172.21.15.23:0/490031186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568bf078f0 0x55568bf053f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6cdc36640 1 -- 172.21.15.23:0/490031186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c4009830 con 0x55568befe3f0 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6cdc36640 1 --2- 172.21.15.23:0/490031186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befe3f0 0x55568bf04970 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb6c40058c0 tx=0x7fb6c40050e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19da724241bcb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6b17fa640 1 -- 172.21.15.23:0/490031186 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c40054d0 con 0x55568befe3f0 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6b17fa640 1 -- 172.21.15.23:0/490031186 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fb6c4024470 con 0x55568befe3f0 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.571+0000 7fb6d2a76740 1 -- 172.21.15.23:0/490031186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55568bb647e0 con 0x55568befe3f0 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.572+0000 7fb6b17fa640 1 -- 172.21.15.23:0/490031186 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb6c4024630 con 0x55568befe3f0 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.572+0000 7fb6d2a76740 10 client.15702 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.572+0000 7fb6d2a76740 1 -- 172.21.15.23:0/490031186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55568be63bd0 con 0x55568befe3f0 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.572+0000 7fb6b17fa640 1 -- 172.21.15.23:0/490031186 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55568bb647e0 con 0x55568befe3f0 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.572+0000 7fb6b17fa640 1 -- 172.21.15.23:0/490031186 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb6c4019750 con 0x55568befe3f0 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.572+0000 7fb6d2a76740 10 client.15702 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.572+0000 7fb6d2a76740 10 client.15702 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:40:45.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.572+0000 7fb6d2a76740 20 client.15702 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:45.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.572+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/490031186 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55568bf179b0 0x55568bf19df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.572+0000 7fb6d2a76740 4 client.15702 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:45.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.572+0000 7fb6d2a76740 1 -- 172.21.15.23:0/490031186 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55568bba1720 con 0x55568bf179b0 2024-03-24T10:40:45.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.572+0000 7fb6cd435640 1 --2- 172.21.15.23:0/490031186 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55568bf179b0 0x55568bf19df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.573+0000 7fb6b17fa640 10 client.15702 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:40:45.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.573+0000 7fb6cd435640 1 --2- 172.21.15.23:0/490031186 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55568bf179b0 0x55568bf19df0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=390a36e3b70b47b7 server_cookie=4953e07f1e213386 in_seq=0 out_seq=0 2024-03-24T10:40:45.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.573+0000 7fb6b17fa640 1 -- 172.21.15.23:0/490031186 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55568bba1720 con 0x55568bf179b0 2024-03-24T10:40:45.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.574+0000 7fb6b17fa640 10 client.15702 handle_command_reply: tid=0 2024-03-24T10:40:45.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.574+0000 7fb6d2a76740 1 client.15702 shutdown 2024-03-24T10:40:45.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.574+0000 7fb6d2a76740 1 -- 172.21.15.23:0/490031186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befe3f0 msgr2=0x55568bf04970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.574+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/490031186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befe3f0 0x55568bf04970 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb6c40058c0 tx=0x7fb6c40050e0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.574+0000 7fb6d2a76740 1 -- 172.21.15.23:0/490031186 shutdown_connections 2024-03-24T10:40:45.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.574+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/490031186 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55568bf179b0 0x55568bf19df0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.574+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/490031186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568bf078f0 0x55568bf053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.574+0000 7fb6b17fa640 0 client.15702 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:40:45.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.574+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/490031186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568befed10 0x55568bf04eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.574+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/490031186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befe3f0 0x55568bf04970 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.574+0000 7fb6d2a76740 1 -- 172.21.15.23:0/490031186 >> 172.21.15.23:0/490031186 conn(0x55568bef8b20 msgr2=0x55568befa870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.574+0000 7fb6d2a76740 1 -- 172.21.15.23:0/490031186 shutdown_connections 2024-03-24T10:40:45.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.574+0000 7fb6d2a76740 1 -- 172.21.15.23:0/490031186 wait complete. 2024-03-24T10:40:45.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.574+0000 7fb6d2a76740 20 client.15702 trim_cache size 0 max 16384 2024-03-24T10:40:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.587+0000 7fb6d2a76740 1 Processor -- start 2024-03-24T10:40:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.587+0000 7fb6d2a76740 1 -- start start 2024-03-24T10:40:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.587+0000 7fb6d2a76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befe3f0 0x55568bf070a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.587+0000 7fb6d2a76740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befed10 0x55568bf13180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.587+0000 7fb6d2a76740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568bf078f0 0x55568bef7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.587+0000 7fb6d2a76740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55568bf144c0 con 0x55568befed10 2024-03-24T10:40:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.587+0000 7fb6d2a76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55568befe050 con 0x55568befe3f0 2024-03-24T10:40:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.587+0000 7fb6d2a76740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55568bee9e20 con 0x55568bf078f0 2024-03-24T10:40:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.587+0000 7fb6cd435640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befed10 0x55568bf13180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.587+0000 7fb6cd435640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befed10 0x55568bf13180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38542/0 (socket says 172.21.15.23:38542) 2024-03-24T10:40:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.587+0000 7fb6cd435640 1 -- 172.21.15.23:0/3604039389 learned_addr learned my addr 172.21.15.23:0/3604039389 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.587+0000 7fb6cdc36640 1 --2- 172.21.15.23:0/3604039389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befe3f0 0x55568bf070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.587+0000 7fb6ce437640 1 --2- 172.21.15.23:0/3604039389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568bf078f0 0x55568bef7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.588+0000 7fb6cd435640 1 -- 172.21.15.23:0/3604039389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568bf078f0 msgr2=0x55568bef7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.588+0000 7fb6cd435640 1 --2- 172.21.15.23:0/3604039389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568bf078f0 0x55568bef7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.588+0000 7fb6cd435640 1 -- 172.21.15.23:0/3604039389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befe3f0 msgr2=0x55568bf070a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.588+0000 7fb6cd435640 1 --2- 172.21.15.23:0/3604039389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befe3f0 0x55568bf070a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.588+0000 7fb6cd435640 1 -- 172.21.15.23:0/3604039389 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55568bdffc80 con 0x55568befed10 2024-03-24T10:40:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.588+0000 7fb6cd435640 1 --2- 172.21.15.23:0/3604039389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befed10 0x55568bf13180 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80973d0 tx=0x7fb6b8097210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ce871c68d8004d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.588+0000 7fb6b37fe640 1 -- 172.21.15.23:0/3604039389 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b809f0c0 con 0x55568befed10 2024-03-24T10:40:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.588+0000 7fb6b37fe640 1 -- 172.21.15.23:0/3604039389 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6b80b2070 con 0x55568befed10 2024-03-24T10:40:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.588+0000 7fb6b37fe640 1 -- 172.21.15.23:0/3604039389 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b80cc080 con 0x55568befed10 2024-03-24T10:40:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.588+0000 7fb6d2a76740 1 -- 172.21.15.23:0/3604039389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befed10 msgr2=0x55568bf13180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.588+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/3604039389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befed10 0x55568bf13180 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80973d0 tx=0x7fb6b8097210 comp rx=0 tx=0).stop 2024-03-24T10:40:45.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 -- 172.21.15.23:0/3604039389 shutdown_connections 2024-03-24T10:40:45.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/3604039389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568bf078f0 0x55568bef7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/3604039389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befed10 0x55568bf13180 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/3604039389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befe3f0 0x55568bf070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 -- 172.21.15.23:0/3604039389 >> 172.21.15.23:0/3604039389 conn(0x55568bef8b20 msgr2=0x55568bef7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 -- 172.21.15.23:0/3604039389 shutdown_connections 2024-03-24T10:40:45.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 -- 172.21.15.23:0/3604039389 wait complete. 2024-03-24T10:40:45.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 -- start start 2024-03-24T10:40:45.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 Processor -- start 2024-03-24T10:40:45.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befe3f0 0x55568bf01d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befed10 0x55568bf022b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568bf078f0 0x55568bf76d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55568bf77270 con 0x55568befe3f0 2024-03-24T10:40:45.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55568bf773e0 con 0x55568befed10 2024-03-24T10:40:45.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55568bf775a0 con 0x55568bf078f0 2024-03-24T10:40:45.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6d2a76740 10 client.0 _ll_register_callbacks cb 0x7fb6b8096b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6cdc36640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befe3f0 0x55568bf01d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6cdc36640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befe3f0 0x55568bf01d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38548/0 (socket says 172.21.15.23:38548) 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6cdc36640 1 -- 172.21.15.23:0/1420929424 learned_addr learned my addr 172.21.15.23:0/1420929424 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6cd435640 1 --2- 172.21.15.23:0/1420929424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befed10 0x55568bf022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.589+0000 7fb6ce437640 1 --2- 172.21.15.23:0/1420929424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568bf078f0 0x55568bf76d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6cdc36640 1 -- 172.21.15.23:0/1420929424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568bf078f0 msgr2=0x55568bf76d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6cdc36640 1 --2- 172.21.15.23:0/1420929424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568bf078f0 0x55568bf76d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6cdc36640 1 -- 172.21.15.23:0/1420929424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befed10 msgr2=0x55568bf022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6cdc36640 1 --2- 172.21.15.23:0/1420929424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befed10 0x55568bf022b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6cdc36640 1 -- 172.21.15.23:0/1420929424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c8198960 con 0x55568befe3f0 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6cdc36640 1 --2- 172.21.15.23:0/1420929424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befe3f0 0x55568bf01d70 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4014010 tx=0x7fb6c4009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=996e9a2772dd2ed8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6b17fa640 1 -- 172.21.15.23:0/1420929424 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4005690 con 0x55568befe3f0 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6b17fa640 1 -- 172.21.15.23:0/1420929424 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fb6c4000e40 con 0x55568befe3f0 2024-03-24T10:40:45.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6d2a76740 1 -- 172.21.15.23:0/1420929424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6c4009e40 con 0x55568befe3f0 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6b17fa640 1 -- 172.21.15.23:0/1420929424 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb6c40195f0 con 0x55568befe3f0 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6d2a76740 10 client.15708 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6d2a76740 1 -- 172.21.15.23:0/1420929424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55568be25920 con 0x55568befe3f0 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6b17fa640 1 -- 172.21.15.23:0/1420929424 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6c4044020 con 0x55568befe3f0 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6b17fa640 1 -- 172.21.15.23:0/1420929424 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb6c4004040 con 0x55568befe3f0 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6d2a76740 10 client.15708 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6d2a76740 10 client.15708 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.590+0000 7fb6d2a76740 20 client.15708 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/1420929424 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55568bf7ec10 0x55568bf81030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6d2a76740 4 client.15708 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6d2a76740 1 -- 172.21.15.23:0/1420929424 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55568bbf0890 con 0x55568bf7ec10 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6cd435640 1 --2- 172.21.15.23:0/1420929424 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55568bf7ec10 0x55568bf81030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6b17fa640 10 client.15708 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6cd435640 1 --2- 172.21.15.23:0/1420929424 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55568bf7ec10 0x55568bf81030 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b7a334c9994b01a server_cookie=568f1aa74002ca4e in_seq=0 out_seq=0 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6b17fa640 1 -- 172.21.15.23:0/1420929424 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55568bbf0890 con 0x55568bf7ec10 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6b17fa640 10 client.15708 handle_command_reply: tid=0 2024-03-24T10:40:45.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6d2a76740 1 client.15708 shutdown 2024-03-24T10:40:45.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6d2a76740 1 -- 172.21.15.23:0/1420929424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befe3f0 msgr2=0x55568bf01d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/1420929424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befe3f0 0x55568bf01d70 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4014010 tx=0x7fb6c4009b90 comp rx=0 tx=0).stop 2024-03-24T10:40:45.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6d2a76740 1 -- 172.21.15.23:0/1420929424 shutdown_connections 2024-03-24T10:40:45.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/1420929424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55568bf078f0 0x55568bf76d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/1420929424 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55568bf7ec10 0x55568bf81030 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/1420929424 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55568befed10 0x55568bf022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6d2a76740 1 --2- 172.21.15.23:0/1420929424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55568befe3f0 0x55568bf01d70 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6b17fa640 0 client.15708 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:40:45.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.591+0000 7fb6d2a76740 1 -- 172.21.15.23:0/1420929424 >> 172.21.15.23:0/1420929424 conn(0x55568bef8b20 msgr2=0x55568bf051a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6d2a76740 1 -- 172.21.15.23:0/1420929424 shutdown_connections 2024-03-24T10:40:45.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6d2a76740 1 -- 172.21.15.23:0/1420929424 wait complete. 2024-03-24T10:40:45.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6d2a76740 20 client.15708 trim_cache size 0 max 16384 2024-03-24T10:40:45.597 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:40:45.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6cfe9a640 1 -- 172.21.15.23:0/2099061385 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb6b40a2350 msgr2=0x7fb6b40a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6cfe9a640 1 --2- 172.21.15.23:0/2099061385 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb6b40a2350 0x7fb6b40a4810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb6c002c630 tx=0x7fb6c0005d80 comp rx=0 tx=0).stop 2024-03-24T10:40:45.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6cfe9a640 1 -- 172.21.15.23:0/2099061385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c80692d0 msgr2=0x7fb6c819de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6cfe9a640 1 --2- 172.21.15.23:0/2099061385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c80692d0 0x7fb6c819de50 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800ea60 tx=0x7fb6b800ef20 comp rx=0 tx=0).stop 2024-03-24T10:40:45.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6cfe9a640 1 -- 172.21.15.23:0/2099061385 shutdown_connections 2024-03-24T10:40:45.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6cfe9a640 1 --2- 172.21.15.23:0/2099061385 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb6b40a2350 0x7fb6b40a4810 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6cfe9a640 1 --2- 172.21.15.23:0/2099061385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6c8107760 0x7fb6c8197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6cfe9a640 1 --2- 172.21.15.23:0/2099061385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c80692d0 0x7fb6c819de50 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6cfe9a640 1 --2- 172.21.15.23:0/2099061385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6c8068910 0x7fb6c819d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6cfe9a640 1 -- 172.21.15.23:0/2099061385 >> 172.21.15.23:0/2099061385 conn(0x7fb6c8077710 msgr2=0x7fb6c80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6cfe9a640 1 -- 172.21.15.23:0/2099061385 shutdown_connections 2024-03-24T10:40:45.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.592+0000 7fb6cfe9a640 1 -- 172.21.15.23:0/2099061385 wait complete. 2024-03-24T10:40:45.603 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:40:45.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.707+0000 7f2e57993640 1 -- 172.21.15.23:0/590236263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e50103270 msgr2=0x7f2e5010b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.707+0000 7f2e57993640 1 --2- 172.21.15.23:0/590236263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e50103270 0x7f2e5010b750 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2e44009900 tx=0x7f2e4402bcd0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.707+0000 7f2e57993640 1 -- 172.21.15.23:0/590236263 shutdown_connections 2024-03-24T10:40:45.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.707+0000 7f2e57993640 1 --2- 172.21.15.23:0/590236263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e5010bd20 0x7f2e5010e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.707+0000 7f2e57993640 1 --2- 172.21.15.23:0/590236263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e50103270 0x7f2e5010b750 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.707+0000 7f2e57993640 1 --2- 172.21.15.23:0/590236263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e50102950 0x7f2e50102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.707+0000 7f2e57993640 1 -- 172.21.15.23:0/590236263 >> 172.21.15.23:0/590236263 conn(0x7f2e50076990 msgr2=0x7f2e50076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.707+0000 7f2e57993640 1 -- 172.21.15.23:0/590236263 shutdown_connections 2024-03-24T10:40:45.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.707+0000 7f2e57993640 1 -- 172.21.15.23:0/590236263 wait complete. 2024-03-24T10:40:45.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.707+0000 7f2e57993640 1 Processor -- start 2024-03-24T10:40:45.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.707+0000 7f2e57993640 1 -- start start 2024-03-24T10:40:45.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e57993640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e50102950 0x7f2e50197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e57993640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e50103270 0x7f2e50198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e57993640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e5010bd20 0x7f2e5019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e57993640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5019ccf0 con 0x7f2e50102950 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e57993640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5019ce60 con 0x7f2e5010bd20 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e57993640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5019d020 con 0x7f2e50103270 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e5572f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e50102950 0x7f2e50197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e5572f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e50102950 0x7f2e50197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38572/0 (socket says 172.21.15.23:38572) 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e5572f640 1 -- 172.21.15.23:0/1830597054 learned_addr learned my addr 172.21.15.23:0/1830597054 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e54f2e640 1 --2- 172.21.15.23:0/1830597054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e50103270 0x7f2e50198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e55f30640 1 --2- 172.21.15.23:0/1830597054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e5010bd20 0x7f2e5019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e5572f640 1 -- 172.21.15.23:0/1830597054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e50103270 msgr2=0x7f2e50198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e5572f640 1 --2- 172.21.15.23:0/1830597054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e50103270 0x7f2e50198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e5572f640 1 -- 172.21.15.23:0/1830597054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e5010bd20 msgr2=0x7f2e5019c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e5572f640 1 --2- 172.21.15.23:0/1830597054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e5010bd20 0x7f2e5019c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e5572f640 1 -- 172.21.15.23:0/1830597054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e44009560 con 0x7f2e50102950 2024-03-24T10:40:45.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e5572f640 1 --2- 172.21.15.23:0/1830597054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e50102950 0x7f2e50197e30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c00e9f0 tx=0x7f2e4c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e427fc640 1 -- 172.21.15.23:0/1830597054 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c009720 con 0x7f2e50102950 2024-03-24T10:40:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e427fc640 1 -- 172.21.15.23:0/1830597054 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e4c00cd80 con 0x7f2e50102950 2024-03-24T10:40:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e427fc640 1 -- 172.21.15.23:0/1830597054 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c010690 con 0x7f2e50102950 2024-03-24T10:40:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e57993640 1 -- 172.21.15.23:0/1830597054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e5019d300 con 0x7f2e50102950 2024-03-24T10:40:45.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.708+0000 7f2e57993640 1 -- 172.21.15.23:0/1830597054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e501a4b80 con 0x7f2e50102950 2024-03-24T10:40:45.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.711+0000 7f2e427fc640 1 -- 172.21.15.23:0/1830597054 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7f2e4c0046c0 con 0x7f2e50102950 2024-03-24T10:40:45.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.711+0000 7f2e5a57a740 1 Processor -- start 2024-03-24T10:40:45.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.711+0000 7f2e5a57a740 1 -- start start 2024-03-24T10:40:45.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.711+0000 7f2e5a57a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c63f0 0x5581c18c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.711+0000 7f2e5a57a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c6d10 0x5581c18cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.711+0000 7f2e5a57a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18cf8f0 0x5581c18d1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.711+0000 7f2e5a57a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5581c1834840 con 0x5581c18c6d10 2024-03-24T10:40:45.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.711+0000 7f2e5a57a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5581c1831d70 con 0x5581c18cf8f0 2024-03-24T10:40:45.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.711+0000 7f2e5a57a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5581c182d550 con 0x5581c18c63f0 2024-03-24T10:40:45.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.711+0000 7f2e427fc640 1 --2- 172.21.15.23:0/1830597054 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2e3c0a2350 0x7f2e3c0a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e5572f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c6d10 0x5581c18cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e55f30640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c63f0 0x5581c18c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e54f2e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18cf8f0 0x5581c18d1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e55f30640 1 --2- 172.21.15.23:0/1830597054 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2e3c0a2350 0x7f2e3c0a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e5572f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c6d10 0x5581c18cf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38582/0 (socket says 172.21.15.23:38582) 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e5572f640 1 -- 172.21.15.23:0/1510133291 learned_addr learned my addr 172.21.15.23:0/1510133291 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e427fc640 1 -- 172.21.15.23:0/1830597054 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f2e4c014070 con 0x7f2e50102950 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e54f2e640 1 -- 172.21.15.23:0/1510133291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c63f0 msgr2=0x5581c18c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e54f2e640 1 --2- 172.21.15.23:0/1510133291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c63f0 0x5581c18c67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e54f2e640 1 -- 172.21.15.23:0/1510133291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c6d10 msgr2=0x5581c18cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e54f2e640 1 --2- 172.21.15.23:0/1510133291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c6d10 0x5581c18cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e54f2e640 1 -- 172.21.15.23:0/1510133291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581c17c2de0 con 0x5581c18cf8f0 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e55f30640 1 --2- 172.21.15.23:0/1830597054 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2e3c0a2350 0x7f2e3c0a4810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009fd0 tx=0x7f2e48009e00 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e54f2e640 1 --2- 172.21.15.23:0/1510133291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18cf8f0 0x5581c18d1ce0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e44000c00 tx=0x7f2e4402c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c7b72944753366c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e3b7fe640 1 -- 172.21.15.23:0/1510133291 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e440047c0 con 0x5581c18cf8f0 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e3b7fe640 1 -- 172.21.15.23:0/1510133291 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e44004920 con 0x5581c18cf8f0 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.712+0000 7f2e3b7fe640 1 -- 172.21.15.23:0/1510133291 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e44004ff0 con 0x5581c18cf8f0 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.713+0000 7f2e5a57a740 1 -- 172.21.15.23:0/1510133291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18cf8f0 msgr2=0x5581c18d1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.713+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/1510133291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18cf8f0 0x5581c18d1ce0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e44000c00 tx=0x7f2e4402c740 comp rx=0 tx=0).stop 2024-03-24T10:40:45.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.713+0000 7f2e5a57a740 1 -- 172.21.15.23:0/1510133291 shutdown_connections 2024-03-24T10:40:45.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.713+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/1510133291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18cf8f0 0x5581c18d1ce0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.713+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/1510133291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c6d10 0x5581c18cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.713+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/1510133291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c63f0 0x5581c18c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.713+0000 7f2e5a57a740 1 -- 172.21.15.23:0/1510133291 >> 172.21.15.23:0/1510133291 conn(0x5581c18c0b20 msgr2=0x5581c18c0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.713+0000 7f2e5a57a740 1 -- 172.21.15.23:0/1510133291 shutdown_connections 2024-03-24T10:40:45.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.713+0000 7f2e5a57a740 1 -- 172.21.15.23:0/1510133291 wait complete. 2024-03-24T10:40:45.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.713+0000 7f2e5a57a740 1 -- start start 2024-03-24T10:40:45.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e5a57a740 1 Processor -- start 2024-03-24T10:40:45.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e5a57a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18c63f0 0x5581c18cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e5a57a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c6d10 0x5581c18ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e5a57a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18cf8f0 0x5581c18cd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e5a57a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5581c1500430 con 0x5581c18cf8f0 2024-03-24T10:40:45.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e5a57a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5581c14bce00 con 0x5581c18c63f0 2024-03-24T10:40:45.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e5a57a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5581c17b2590 con 0x5581c18c6d10 2024-03-24T10:40:45.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e5a57a740 10 client.0 _ll_register_callbacks cb 0x5581c1821fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:45.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e55f30640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18cf8f0 0x5581c18cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e55f30640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18cf8f0 0x5581c18cd320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38592/0 (socket says 172.21.15.23:38592) 2024-03-24T10:40:45.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e55f30640 1 -- 172.21.15.23:0/2465086934 learned_addr learned my addr 172.21.15.23:0/2465086934 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e5572f640 1 --2- 172.21.15.23:0/2465086934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18c63f0 0x5581c18cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e54f2e640 1 --2- 172.21.15.23:0/2465086934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c6d10 0x5581c18ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e55f30640 1 -- 172.21.15.23:0/2465086934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c6d10 msgr2=0x5581c18ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e55f30640 1 --2- 172.21.15.23:0/2465086934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c6d10 0x5581c18ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e55f30640 1 -- 172.21.15.23:0/2465086934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18c63f0 msgr2=0x5581c18cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e55f30640 1 --2- 172.21.15.23:0/2465086934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18c63f0 0x5581c18cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e55f30640 1 -- 172.21.15.23:0/2465086934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e5010edd0 con 0x5581c18cf8f0 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.714+0000 7f2e5572f640 1 --2- 172.21.15.23:0/2465086934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18c63f0 0x5581c18cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e55f30640 1 --2- 172.21.15.23:0/2465086934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18cf8f0 0x5581c18cd320 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2e4800ff20 tx=0x7f2e4801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d8c2fde472c0c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e38ff9640 1 -- 172.21.15.23:0/2465086934 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48003e00 con 0x5581c18cf8f0 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e38ff9640 1 -- 172.21.15.23:0/2465086934 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f2e480047c0 con 0x5581c18cf8f0 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2465086934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5581c152c7e0 con 0x5581c18cf8f0 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e38ff9640 1 -- 172.21.15.23:0/2465086934 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2e48020420 con 0x5581c18cf8f0 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e5a57a740 10 client.15720 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2465086934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5581c182bbd0 con 0x5581c18cf8f0 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e38ff9640 1 -- 172.21.15.23:0/2465086934 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5581c152c7e0 con 0x5581c18cf8f0 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e38ff9640 1 -- 172.21.15.23:0/2465086934 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f2e4801e580 con 0x5581c18cf8f0 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e5a57a740 10 client.15720 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:45.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e5a57a740 10 client.15720 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e5a57a740 20 client.15720 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/2465086934 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5581c18df7a0 0x5581c18e1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e5a57a740 4 client.15720 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.715+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2465086934 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581c1569720 con 0x5581c18df7a0 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.716+0000 7f2e5572f640 1 --2- 172.21.15.23:0/2465086934 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5581c18df7a0 0x5581c18e1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.716+0000 7f2e5572f640 1 --2- 172.21.15.23:0/2465086934 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5581c18df7a0 0x5581c18e1be0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2dec67bd3f6c6609 server_cookie=716f5cd6c6b497c0 in_seq=0 out_seq=0 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.716+0000 7f2e38ff9640 10 client.15720 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e38ff9640 1 -- 172.21.15.23:0/2465086934 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5581c1569720 con 0x5581c18df7a0 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e38ff9640 10 client.15720 handle_command_reply: tid=0 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e5a57a740 1 client.15720 shutdown 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2465086934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18cf8f0 msgr2=0x5581c18cd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/2465086934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18cf8f0 0x5581c18cd320 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2e4800ff20 tx=0x7f2e4801d000 comp rx=0 tx=0).stop 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2465086934 shutdown_connections 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/2465086934 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5581c18df7a0 0x5581c18e1be0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/2465086934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18cf8f0 0x5581c18cd320 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e38ff9640 0 client.15720 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:40:45.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/2465086934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c6d10 0x5581c18ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/2465086934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18c63f0 0x5581c18cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2465086934 >> 172.21.15.23:0/2465086934 conn(0x5581c18c0b20 msgr2=0x5581c18c2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2465086934 shutdown_connections 2024-03-24T10:40:45.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2465086934 wait complete. 2024-03-24T10:40:45.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.717+0000 7f2e5a57a740 20 client.15720 trim_cache size 0 max 16384 2024-03-24T10:40:45.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.742+0000 7f2e5a57a740 1 Processor -- start 2024-03-24T10:40:45.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.742+0000 7f2e5a57a740 1 -- start start 2024-03-24T10:40:45.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.742+0000 7f2e5a57a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c63f0 0x5581c18cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.742+0000 7f2e5a57a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c6d10 0x5581c18cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.742+0000 7f2e5a57a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18cf8f0 0x5581c18e74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.742+0000 7f2e5a57a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5581c18bc450 con 0x5581c18c63f0 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.742+0000 7f2e5a57a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5581c18dc270 con 0x5581c18cf8f0 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.742+0000 7f2e5a57a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5581c18c1b50 con 0x5581c18c6d10 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e5572f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c63f0 0x5581c18cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e55f30640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18cf8f0 0x5581c18e74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e54f2e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c6d10 0x5581c18cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e5572f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c63f0 0x5581c18cefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38596/0 (socket says 172.21.15.23:38596) 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e55f30640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18cf8f0 0x5581c18e74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:45096/0 (socket says 172.21.15.23:45096) 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e5572f640 1 -- 172.21.15.23:0/3060941802 learned_addr learned my addr 172.21.15.23:0/3060941802 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e54f2e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c6d10 0x5581c18cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:53566/0 (socket says 172.21.15.23:53566) 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e5572f640 1 -- 172.21.15.23:0/3060941802 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c6d10 msgr2=0x5581c18cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e5572f640 1 --2- 172.21.15.23:0/3060941802 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c6d10 0x5581c18cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e5572f640 1 -- 172.21.15.23:0/3060941802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18cf8f0 msgr2=0x5581c18e74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e5572f640 1 --2- 172.21.15.23:0/3060941802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18cf8f0 0x5581c18e74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e5572f640 1 -- 172.21.15.23:0/3060941802 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581c17c7c80 con 0x5581c18c63f0 2024-03-24T10:40:45.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e54f2e640 1 --2- 172.21.15.23:0/3060941802 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c6d10 0x5581c18cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:45.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e55f30640 1 --2- 172.21.15.23:0/3060941802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18cf8f0 0x5581c18e74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:45.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e5572f640 1 --2- 172.21.15.23:0/3060941802 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c63f0 0x5581c18cefd0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c097830 tx=0x7f2e4c0d0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0579eba2f8fc16d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e3affd640 1 -- 172.21.15.23:0/3060941802 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c09efb0 con 0x5581c18c63f0 2024-03-24T10:40:45.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e3affd640 1 -- 172.21.15.23:0/3060941802 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e4c0a3600 con 0x5581c18c63f0 2024-03-24T10:40:45.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.743+0000 7f2e3affd640 1 -- 172.21.15.23:0/3060941802 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c09b320 con 0x5581c18c63f0 2024-03-24T10:40:45.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.744+0000 7f2e5a57a740 1 -- 172.21.15.23:0/3060941802 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c63f0 msgr2=0x5581c18cefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.744+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/3060941802 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c63f0 0x5581c18cefd0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c097830 tx=0x7f2e4c0d0040 comp rx=0 tx=0).stop 2024-03-24T10:40:45.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.744+0000 7f2e5a57a740 1 -- 172.21.15.23:0/3060941802 shutdown_connections 2024-03-24T10:40:45.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.744+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/3060941802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18cf8f0 0x5581c18e74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.744+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/3060941802 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18c6d10 0x5581c18cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.744+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/3060941802 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c63f0 0x5581c18cefd0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.744+0000 7f2e5a57a740 1 -- 172.21.15.23:0/3060941802 >> 172.21.15.23:0/3060941802 conn(0x5581c18c0b20 msgr2=0x5581c18c1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.744+0000 7f2e5a57a740 1 -- 172.21.15.23:0/3060941802 shutdown_connections 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.744+0000 7f2e5a57a740 1 -- 172.21.15.23:0/3060941802 wait complete. 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.745+0000 7f2e5a57a740 1 -- start start 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.745+0000 7f2e5a57a740 1 Processor -- start 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.745+0000 7f2e5a57a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c63f0 0x5581c18c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.745+0000 7f2e5a57a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18c6d10 0x5581c18ca050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.745+0000 7f2e5a57a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18cf8f0 0x5581c1944b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.745+0000 7f2e5a57a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5581c18ca770 con 0x5581c18c63f0 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.745+0000 7f2e5a57a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5581c1945060 con 0x5581c18c6d10 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.745+0000 7f2e5a57a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5581c19451d0 con 0x5581c18cf8f0 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.745+0000 7f2e5a57a740 10 client.0 _ll_register_callbacks cb 0x7f2e48009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.745+0000 7f2e5572f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c63f0 0x5581c18c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.745+0000 7f2e54f2e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18c6d10 0x5581c18ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.745+0000 7f2e5572f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c63f0 0x5581c18c9b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38600/0 (socket says 172.21.15.23:38600) 2024-03-24T10:40:45.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.745+0000 7f2e5572f640 1 -- 172.21.15.23:0/2644021280 learned_addr learned my addr 172.21.15.23:0/2644021280 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.745+0000 7f2e55f30640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18cf8f0 0x5581c1944b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e5572f640 1 -- 172.21.15.23:0/2644021280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18cf8f0 msgr2=0x5581c1944b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e5572f640 1 --2- 172.21.15.23:0/2644021280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18cf8f0 0x5581c1944b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e5572f640 1 -- 172.21.15.23:0/2644021280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18c6d10 msgr2=0x5581c18ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e5572f640 1 --2- 172.21.15.23:0/2644021280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18c6d10 0x5581c18ca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e5572f640 1 -- 172.21.15.23:0/2644021280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581c17c2de0 con 0x5581c18c63f0 2024-03-24T10:40:45.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e55f30640 1 --2- 172.21.15.23:0/2644021280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18cf8f0 0x5581c1944b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:45.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e54f2e640 1 --2- 172.21.15.23:0/2644021280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18c6d10 0x5581c18ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:45.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e5572f640 1 --2- 172.21.15.23:0/2644021280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c63f0 0x5581c18c9b10 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0977e0 tx=0x7f2e4c097100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2963c69f057b74b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:45.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e38ff9640 1 -- 172.21.15.23:0/2644021280 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c0b9070 con 0x5581c18c63f0 2024-03-24T10:40:45.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e38ff9640 1 -- 172.21.15.23:0/2644021280 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f2e4c09d050 con 0x5581c18c63f0 2024-03-24T10:40:45.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2644021280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e44004920 con 0x5581c18c63f0 2024-03-24T10:40:45.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e38ff9640 1 -- 172.21.15.23:0/2644021280 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2e4c099160 con 0x5581c18c63f0 2024-03-24T10:40:45.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e5a57a740 10 client.15726 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:45.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2644021280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5581c182bbd0 con 0x5581c18c63f0 2024-03-24T10:40:45.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.746+0000 7f2e38ff9640 1 -- 172.21.15.23:0/2644021280 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e44004920 con 0x5581c18c63f0 2024-03-24T10:40:45.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e38ff9640 1 -- 172.21.15.23:0/2644021280 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f2e4c01f060 con 0x5581c18c63f0 2024-03-24T10:40:45.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e5a57a740 10 client.15726 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:45.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e5a57a740 10 client.15726 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:40:45.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e5a57a740 20 client.15726 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:45.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/2644021280 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5581c1946670 0x5581c1948a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e5a57a740 4 client.15726 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:40:45.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2644021280 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5581c15b8890 con 0x5581c1946670 2024-03-24T10:40:45.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e54f2e640 1 --2- 172.21.15.23:0/2644021280 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5581c1946670 0x5581c1948a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e38ff9640 10 client.15726 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e54f2e640 1 --2- 172.21.15.23:0/2644021280 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5581c1946670 0x5581c1948a60 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5ac8194ca2cb1afa server_cookie=3ef8a2646ef92271 in_seq=0 out_seq=0 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e38ff9640 1 -- 172.21.15.23:0/2644021280 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5581c15b8890 con 0x5581c1946670 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e38ff9640 10 client.15726 handle_command_reply: tid=0 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e5a57a740 1 client.15726 shutdown 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2644021280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c63f0 msgr2=0x5581c18c9b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.747+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/2644021280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c63f0 0x5581c18c9b10 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0977e0 tx=0x7f2e4c097100 comp rx=0 tx=0).stop 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.748+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2644021280 shutdown_connections 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.748+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/2644021280 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5581c1946670 0x5581c1948a60 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.748+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/2644021280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5581c18cf8f0 0x5581c1944b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.748+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/2644021280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5581c18c6d10 0x5581c18ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.748+0000 7f2e38ff9640 0 client.15726 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.748+0000 7f2e5a57a740 1 --2- 172.21.15.23:0/2644021280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5581c18c63f0 0x5581c18c9b10 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.748+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2644021280 >> 172.21.15.23:0/2644021280 conn(0x5581c18c0b20 msgr2=0x5581c18c2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.748+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2644021280 shutdown_connections 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.748+0000 7f2e5a57a740 1 -- 172.21.15.23:0/2644021280 wait complete. 2024-03-24T10:40:45.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.748+0000 7f2e5a57a740 20 client.15726 trim_cache size 0 max 16384 2024-03-24T10:40:45.754 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:40:45.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.748+0000 7f2e57993640 1 -- 172.21.15.23:0/1830597054 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2e3c0a2350 msgr2=0x7f2e3c0a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.748+0000 7f2e57993640 1 --2- 172.21.15.23:0/1830597054 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2e3c0a2350 0x7f2e3c0a4810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009fd0 tx=0x7f2e48009e00 comp rx=0 tx=0).stop 2024-03-24T10:40:45.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.748+0000 7f2e57993640 1 -- 172.21.15.23:0/1830597054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e50102950 msgr2=0x7f2e50197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.748+0000 7f2e57993640 1 --2- 172.21.15.23:0/1830597054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e50102950 0x7f2e50197e30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c00e9f0 tx=0x7f2e4c00edf0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.749+0000 7f2e57993640 1 -- 172.21.15.23:0/1830597054 shutdown_connections 2024-03-24T10:40:45.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.749+0000 7f2e57993640 1 --2- 172.21.15.23:0/1830597054 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2e3c0a2350 0x7f2e3c0a4810 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.749+0000 7f2e57993640 1 --2- 172.21.15.23:0/1830597054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e5010bd20 0x7f2e5019c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.749+0000 7f2e57993640 1 --2- 172.21.15.23:0/1830597054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e50103270 0x7f2e50198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.749+0000 7f2e57993640 1 --2- 172.21.15.23:0/1830597054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e50102950 0x7f2e50197e30 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.749+0000 7f2e57993640 1 -- 172.21.15.23:0/1830597054 >> 172.21.15.23:0/1830597054 conn(0x7f2e50076990 msgr2=0x7f2e500fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.749+0000 7f2e57993640 1 -- 172.21.15.23:0/1830597054 shutdown_connections 2024-03-24T10:40:45.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.749+0000 7f2e57993640 1 -- 172.21.15.23:0/1830597054 wait complete. 2024-03-24T10:40:45.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:45 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:40:45.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:45 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:45.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:45 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:40:45.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:45 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:45.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:45 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:40:45.760 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:45 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:40:45.760 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:45 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:45.760 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:45 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:40:45.760 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:45 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:45.760 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:45 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:40:45.761 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:40:45.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:45 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:40:45.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:45 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:45.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:45 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:40:45.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:45 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:45.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:45 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:40:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.868+0000 7ff68b60e640 1 -- 172.21.15.23:0/3233365298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff684100f60 msgr2=0x7ff68410d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.868+0000 7ff68b60e640 1 --2- 172.21.15.23:0/3233365298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff684100f60 0x7ff68410d420 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009ca0 tx=0x7ff67c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.868+0000 7ff68b60e640 1 -- 172.21.15.23:0/3233365298 shutdown_connections 2024-03-24T10:40:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.868+0000 7ff68b60e640 1 --2- 172.21.15.23:0/3233365298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff684100f60 0x7ff68410d420 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.868+0000 7ff68b60e640 1 --2- 172.21.15.23:0/3233365298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6841005a0 0x7ff684100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.868+0000 7ff68b60e640 1 --2- 172.21.15.23:0/3233365298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff684106660 0x7ff684106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.868+0000 7ff68b60e640 1 -- 172.21.15.23:0/3233365298 >> 172.21.15.23:0/3233365298 conn(0x7ff684077710 msgr2=0x7ff684077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.868+0000 7ff68b60e640 1 -- 172.21.15.23:0/3233365298 shutdown_connections 2024-03-24T10:40:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.868+0000 7ff68b60e640 1 -- 172.21.15.23:0/3233365298 wait complete. 2024-03-24T10:40:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.868+0000 7ff68b60e640 1 Processor -- start 2024-03-24T10:40:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.868+0000 7ff68b60e640 1 -- start start 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff68b60e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6841005a0 0x7ff68419c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff68b60e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff684100f60 0x7ff68419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff68b60e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff684106660 0x7ff684196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff68b60e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff684196fb0 con 0x7ff684106660 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff68b60e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff684197120 con 0x7ff684100f60 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff68b60e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff6841972e0 con 0x7ff6841005a0 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff689bab640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff684106660 0x7ff684196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff689bab640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff684106660 0x7ff684196960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38612/0 (socket says 172.21.15.23:38612) 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff689bab640 1 -- 172.21.15.23:0/554735800 learned_addr learned my addr 172.21.15.23:0/554735800 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff6893aa640 1 --2- 172.21.15.23:0/554735800 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6841005a0 0x7ff68419c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff688ba9640 1 --2- 172.21.15.23:0/554735800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff684100f60 0x7ff68419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff689bab640 1 -- 172.21.15.23:0/554735800 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6841005a0 msgr2=0x7ff68419c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff689bab640 1 --2- 172.21.15.23:0/554735800 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6841005a0 0x7ff68419c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff689bab640 1 -- 172.21.15.23:0/554735800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff684100f60 msgr2=0x7ff68419ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff689bab640 1 --2- 172.21.15.23:0/554735800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff684100f60 0x7ff68419ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff689bab640 1 -- 172.21.15.23:0/554735800 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff680009490 con 0x7ff684106660 2024-03-24T10:40:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff689bab640 1 --2- 172.21.15.23:0/554735800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff684106660 0x7ff684196960 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff67c005e30 tx=0x7ff67c035aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff67a7fc640 1 -- 172.21.15.23:0/554735800 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67c035ea0 con 0x7ff684106660 2024-03-24T10:40:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff67a7fc640 1 -- 172.21.15.23:0/554735800 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff67c003e50 con 0x7ff684106660 2024-03-24T10:40:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff67a7fc640 1 -- 172.21.15.23:0/554735800 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67c004830 con 0x7ff684106660 2024-03-24T10:40:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff68b60e640 1 -- 172.21.15.23:0/554735800 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff67c009900 con 0x7ff684106660 2024-03-24T10:40:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.869+0000 7ff68b60e640 1 -- 172.21.15.23:0/554735800 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6841a3670 con 0x7ff684106660 2024-03-24T10:40:45.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.872+0000 7ff67a7fc640 1 -- 172.21.15.23:0/554735800 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7ff67c03b040 con 0x7ff684106660 2024-03-24T10:40:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.872+0000 7ff68e1ee740 1 Processor -- start 2024-03-24T10:40:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.872+0000 7ff68e1ee740 1 -- start start 2024-03-24T10:40:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.872+0000 7ff68e1ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.872+0000 7ff68e1ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0db5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.872+0000 7ff68e1ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0dbb00 0x55c7bc0ddef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.872+0000 7ff68e1ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c7bc03e840 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.872+0000 7ff68e1ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c7bc03bd70 con 0x55c7bc0dbb00 2024-03-24T10:40:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.872+0000 7ff68e1ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c7bc037550 con 0x55c7bc0d07a0 2024-03-24T10:40:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.872+0000 7ff67a7fc640 1 --2- 172.21.15.23:0/554735800 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff6700a23d0 0x7ff6700a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff688ba9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff688ba9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0d01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38626/0 (socket says 172.21.15.23:38626) 2024-03-24T10:40:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff688ba9640 1 -- 172.21.15.23:0/2439835869 learned_addr learned my addr 172.21.15.23:0/2439835869 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff6893aa640 1 --2- 172.21.15.23:0/2439835869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0db5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff689bab640 1 --2- 172.21.15.23:0/554735800 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff6700a23d0 0x7ff6700a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff688ba9640 1 --2- 172.21.15.23:0/2439835869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0dbb00 0x55c7bc0ddef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff6893aa640 1 -- 172.21.15.23:0/2439835869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0dbb00 msgr2=0x55c7bc0ddef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff6893aa640 1 --2- 172.21.15.23:0/2439835869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0dbb00 0x55c7bc0ddef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff67a7fc640 1 -- 172.21.15.23:0/554735800 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7ff67c0f0ab0 con 0x7ff684106660 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff6893aa640 1 -- 172.21.15.23:0/2439835869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 msgr2=0x55c7bc0d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff6893aa640 1 --2- 172.21.15.23:0/2439835869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff6893aa640 1 -- 172.21.15.23:0/2439835869 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7bbfccde0 con 0x55c7bc0d07a0 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff6893aa640 1 --2- 172.21.15.23:0/2439835869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0db5c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff680009d80 tx=0x7ff680011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=291bf17daea7b43e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff689bab640 1 --2- 172.21.15.23:0/554735800 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff6700a23d0 0x7ff6700a4890 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0ee1f0 tx=0x7ff67c0e7a20 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff66f7fe640 1 -- 172.21.15.23:0/2439835869 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff680004250 con 0x55c7bc0d07a0 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff66f7fe640 1 -- 172.21.15.23:0/2439835869 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6800043b0 con 0x55c7bc0d07a0 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.873+0000 7ff66f7fe640 1 -- 172.21.15.23:0/2439835869 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800198e0 con 0x55c7bc0d07a0 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.874+0000 7ff68e1ee740 1 -- 172.21.15.23:0/2439835869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 msgr2=0x55c7bc0db5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.874+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/2439835869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0db5c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff680009d80 tx=0x7ff680011810 comp rx=0 tx=0).stop 2024-03-24T10:40:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.874+0000 7ff68e1ee740 1 -- 172.21.15.23:0/2439835869 shutdown_connections 2024-03-24T10:40:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.874+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/2439835869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0dbb00 0x55c7bc0ddef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.874+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/2439835869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0db5c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.874+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/2439835869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.874+0000 7ff68e1ee740 1 -- 172.21.15.23:0/2439835869 >> 172.21.15.23:0/2439835869 conn(0x55c7bc0cab20 msgr2=0x55c7bc0caf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.874+0000 7ff68e1ee740 1 -- 172.21.15.23:0/2439835869 shutdown_connections 2024-03-24T10:40:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.874+0000 7ff68e1ee740 1 -- 172.21.15.23:0/2439835869 wait complete. 2024-03-24T10:40:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.874+0000 7ff68e1ee740 1 -- start start 2024-03-24T10:40:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.874+0000 7ff68e1ee740 1 Processor -- start 2024-03-24T10:40:45.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff68e1ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0d8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff68e1ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0d9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff68e1ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0dbb00 0x55c7bc0d95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff68e1ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c7bbd0a430 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff68e1ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c7bbcc6e00 con 0x55c7bc0d07a0 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff68e1ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c7bbfbc590 con 0x55c7bc0dbb00 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff68e1ee740 10 client.0 _ll_register_callbacks cb 0x55c7bc02bfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff688ba9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0d8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff688ba9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0d8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38642/0 (socket says 172.21.15.23:38642) 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff688ba9640 1 -- 172.21.15.23:0/3456389173 learned_addr learned my addr 172.21.15.23:0/3456389173 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff6893aa640 1 --2- 172.21.15.23:0/3456389173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0d9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff688ba9640 1 --2- 172.21.15.23:0/3456389173 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0dbb00 0x55c7bc0d95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff688ba9640 1 -- 172.21.15.23:0/3456389173 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0dbb00 msgr2=0x55c7bc0d95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff688ba9640 1 --2- 172.21.15.23:0/3456389173 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0dbb00 0x55c7bc0d95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff688ba9640 1 -- 172.21.15.23:0/3456389173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0d07a0 msgr2=0x55c7bc0d9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff688ba9640 1 --2- 172.21.15.23:0/3456389173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0d9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff688ba9640 1 -- 172.21.15.23:0/3456389173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6800099e0 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff688ba9640 1 --2- 172.21.15.23:0/3456389173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0d8b30 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff674013d00 tx=0x7ff674005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4cbfb56c49744db server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.875+0000 7ff66cff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff66cff9640 1 -- 172.21.15.23:0/3456389173 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67400fe60 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff66cff9640 1 -- 172.21.15.23:0/3456389173 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7ff674005020 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff68e1ee740 1 -- 172.21.15.23:0/3456389173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c7bbd367e0 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff66cff9640 1 -- 172.21.15.23:0/3456389173 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff6740051e0 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff68e1ee740 10 client.15738 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff68e1ee740 1 -- 172.21.15.23:0/3456389173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c7bc035bd0 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff66cff9640 1 -- 172.21.15.23:0/3456389173 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c7bbd367e0 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff66cff9640 1 -- 172.21.15.23:0/3456389173 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff6740048c0 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff68e1ee740 10 client.15738 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff68e1ee740 10 client.15738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff68e1ee740 20 client.15738 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/3456389173 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c7bc0ebbb0 0x55c7bc0edfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff68e1ee740 4 client.15738 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff68e1ee740 1 -- 172.21.15.23:0/3456389173 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7bbd73720 con 0x55c7bc0ebbb0 2024-03-24T10:40:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.876+0000 7ff6893aa640 1 --2- 172.21.15.23:0/3456389173 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c7bc0ebbb0 0x55c7bc0edfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.877+0000 7ff66cff9640 10 client.15738 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.877+0000 7ff6893aa640 1 --2- 172.21.15.23:0/3456389173 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c7bc0ebbb0 0x55c7bc0edfd0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3171b4397a183b23 server_cookie=b6600ec5df565f7f in_seq=0 out_seq=0 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff66cff9640 1 -- 172.21.15.23:0/3456389173 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c7bbd73720 con 0x55c7bc0ebbb0 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff66cff9640 10 client.15738 handle_command_reply: tid=0 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff68e1ee740 1 client.15738 shutdown 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff68e1ee740 1 -- 172.21.15.23:0/3456389173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 msgr2=0x55c7bc0d8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/3456389173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0d8b30 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff674013d00 tx=0x7ff674005f90 comp rx=0 tx=0).stop 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff68e1ee740 1 -- 172.21.15.23:0/3456389173 shutdown_connections 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/3456389173 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c7bc0ebbb0 0x55c7bc0edfd0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/3456389173 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0dbb00 0x55c7bc0d95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/3456389173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0d9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff66cff9640 0 client.15738 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/3456389173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0d8b30 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff68e1ee740 1 -- 172.21.15.23:0/3456389173 >> 172.21.15.23:0/3456389173 conn(0x55c7bc0cab20 msgr2=0x55c7bc0cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff68e1ee740 1 -- 172.21.15.23:0/3456389173 shutdown_connections 2024-03-24T10:40:45.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff68e1ee740 1 -- 172.21.15.23:0/3456389173 wait complete. 2024-03-24T10:40:45.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.878+0000 7ff68e1ee740 20 client.15738 trim_cache size 0 max 16384 2024-03-24T10:40:45.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.891+0000 7ff68e1ee740 1 Processor -- start 2024-03-24T10:40:45.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.891+0000 7ff68e1ee740 1 -- start start 2024-03-24T10:40:45.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff68e1ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0db260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff68e1ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0c5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff68e1ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0dbb00 0x55c7bc0e73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff68e1ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c7bc0d9b00 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff68e1ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c7bc0e86c0 con 0x55c7bc0dbb00 2024-03-24T10:40:45.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff68e1ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c7bc0ddef0 con 0x55c7bc0d07a0 2024-03-24T10:40:45.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff688ba9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0db260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff688ba9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0db260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38644/0 (socket says 172.21.15.23:38644) 2024-03-24T10:40:45.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff688ba9640 1 -- 172.21.15.23:0/252576149 learned_addr learned my addr 172.21.15.23:0/252576149 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff6893aa640 1 --2- 172.21.15.23:0/252576149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0c5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff688ba9640 1 --2- 172.21.15.23:0/252576149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0dbb00 0x55c7bc0e73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff688ba9640 1 -- 172.21.15.23:0/252576149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 msgr2=0x55c7bc0c5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff688ba9640 1 --2- 172.21.15.23:0/252576149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0c5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff688ba9640 1 -- 172.21.15.23:0/252576149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0dbb00 msgr2=0x55c7bc0e73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff688ba9640 1 --2- 172.21.15.23:0/252576149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0dbb00 0x55c7bc0e73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff688ba9640 1 -- 172.21.15.23:0/252576149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7bbfd1c80 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff6893aa640 1 --2- 172.21.15.23:0/252576149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0c5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:45.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff688ba9640 1 --2- 172.21.15.23:0/252576149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0db260 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff674009ea0 tx=0x7ff6740097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12063d94679763 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.892+0000 7ff66effd640 1 -- 172.21.15.23:0/252576149 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff674013a50 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.893+0000 7ff66effd640 1 -- 172.21.15.23:0/252576149 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff674013070 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.893+0000 7ff66effd640 1 -- 172.21.15.23:0/252576149 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff674005460 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.893+0000 7ff68e1ee740 1 -- 172.21.15.23:0/252576149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 msgr2=0x55c7bc0db260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.893+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/252576149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0db260 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff674009ea0 tx=0x7ff6740097a0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.893+0000 7ff68e1ee740 1 -- 172.21.15.23:0/252576149 shutdown_connections 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.893+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/252576149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0dbb00 0x55c7bc0e73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.893+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/252576149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0c5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.893+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/252576149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0db260 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.893+0000 7ff68e1ee740 1 -- 172.21.15.23:0/252576149 >> 172.21.15.23:0/252576149 conn(0x55c7bc0cab20 msgr2=0x55c7bc0c9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.893+0000 7ff68e1ee740 1 -- 172.21.15.23:0/252576149 shutdown_connections 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.893+0000 7ff68e1ee740 1 -- 172.21.15.23:0/252576149 wait complete. 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.893+0000 7ff68e1ee740 1 -- start start 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff68e1ee740 1 Processor -- start 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff68e1ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0df6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff68e1ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0dfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff68e1ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0dbb00 0x55c7bc150f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff68e1ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c7bc0e04e0 con 0x55c7bc0dbb00 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff68e1ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c7bc151450 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff68e1ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c7bc151590 con 0x55c7bc0d07a0 2024-03-24T10:40:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff68e1ee740 10 client.0 _ll_register_callbacks cb 0x7ff674013540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff688ba9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0df6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff688ba9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0df6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:45162/0 (socket says 172.21.15.23:45162) 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff688ba9640 1 -- 172.21.15.23:0/4054175331 learned_addr learned my addr 172.21.15.23:0/4054175331 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff6893aa640 1 --2- 172.21.15.23:0/4054175331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0dfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff688ba9640 1 --2- 172.21.15.23:0/4054175331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0dbb00 0x55c7bc150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff688ba9640 1 -- 172.21.15.23:0/4054175331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 msgr2=0x55c7bc0dfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff688ba9640 1 --2- 172.21.15.23:0/4054175331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0dfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff688ba9640 1 -- 172.21.15.23:0/4054175331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0dbb00 msgr2=0x55c7bc150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff688ba9640 1 --2- 172.21.15.23:0/4054175331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0dbb00 0x55c7bc150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.894+0000 7ff688ba9640 1 -- 172.21.15.23:0/4054175331 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff68001e040 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff688ba9640 1 --2- 172.21.15.23:0/4054175331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0df6c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff674005140 tx=0x7ff674026040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d384143d3a8cbbbb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff66cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff66cff9640 1 -- 172.21.15.23:0/4054175331 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6740053e0 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff68e1ee740 1 -- 172.21.15.23:0/4054175331 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6800043b0 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff66cff9640 1 -- 172.21.15.23:0/4054175331 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7ff674034070 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff66cff9640 1 -- 172.21.15.23:0/4054175331 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff674004a80 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff68e1ee740 10 client.25619 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff68e1ee740 1 -- 172.21.15.23:0/4054175331 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c7bc035bd0 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff66cff9640 1 -- 172.21.15.23:0/4054175331 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6800043b0 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff66cff9640 1 -- 172.21.15.23:0/4054175331 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff674041070 con 0x55c7bc0cfdf0 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff68e1ee740 10 client.25619 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff68e1ee740 10 client.25619 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff68e1ee740 20 client.25619 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/4054175331 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c7bc152820 0x55c7bc154c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff68e1ee740 4 client.25619 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.895+0000 7ff68e1ee740 1 -- 172.21.15.23:0/4054175331 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c7bbdc2890 con 0x55c7bc152820 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff6893aa640 1 --2- 172.21.15.23:0/4054175331 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c7bc152820 0x55c7bc154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff6893aa640 1 --2- 172.21.15.23:0/4054175331 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c7bc152820 0x55c7bc154c10 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cf6be01c28f68f2 server_cookie=3d8fffec7b96c24f in_seq=0 out_seq=0 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff66cff9640 10 client.25619 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff66cff9640 1 -- 172.21.15.23:0/4054175331 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c7bbdc2890 con 0x55c7bc152820 2024-03-24T10:40:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff66cff9640 10 client.25619 handle_command_reply: tid=0 2024-03-24T10:40:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff68e1ee740 1 client.25619 shutdown 2024-03-24T10:40:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff68e1ee740 1 -- 172.21.15.23:0/4054175331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0cfdf0 msgr2=0x55c7bc0df6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/4054175331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0df6c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff674005140 tx=0x7ff674026040 comp rx=0 tx=0).stop 2024-03-24T10:40:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff68e1ee740 1 -- 172.21.15.23:0/4054175331 shutdown_connections 2024-03-24T10:40:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/4054175331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c7bc0dbb00 0x55c7bc150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/4054175331 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c7bc152820 0x55c7bc154c10 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/4054175331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c7bc0d07a0 0x55c7bc0dfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff66cff9640 0 client.25619 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff68e1ee740 1 --2- 172.21.15.23:0/4054175331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c7bc0cfdf0 0x55c7bc0df6c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff68e1ee740 1 -- 172.21.15.23:0/4054175331 >> 172.21.15.23:0/4054175331 conn(0x55c7bc0cab20 msgr2=0x55c7bc0c9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff68e1ee740 1 -- 172.21.15.23:0/4054175331 shutdown_connections 2024-03-24T10:40:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.896+0000 7ff68e1ee740 1 -- 172.21.15.23:0/4054175331 wait complete. 2024-03-24T10:40:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.897+0000 7ff68e1ee740 20 client.25619 trim_cache size 0 max 16384 2024-03-24T10:40:45.903 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:40:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.897+0000 7ff68b60e640 1 -- 172.21.15.23:0/554735800 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff6700a23d0 msgr2=0x7ff6700a4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.897+0000 7ff68b60e640 1 --2- 172.21.15.23:0/554735800 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff6700a23d0 0x7ff6700a4890 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0ee1f0 tx=0x7ff67c0e7a20 comp rx=0 tx=0).stop 2024-03-24T10:40:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.897+0000 7ff68b60e640 1 -- 172.21.15.23:0/554735800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff684106660 msgr2=0x7ff684196960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.897+0000 7ff68b60e640 1 --2- 172.21.15.23:0/554735800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff684106660 0x7ff684196960 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff67c005e30 tx=0x7ff67c035aa0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.897+0000 7ff68b60e640 1 -- 172.21.15.23:0/554735800 shutdown_connections 2024-03-24T10:40:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.897+0000 7ff68b60e640 1 --2- 172.21.15.23:0/554735800 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff6700a23d0 0x7ff6700a4890 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.897+0000 7ff68b60e640 1 --2- 172.21.15.23:0/554735800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff684106660 0x7ff684196960 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.897+0000 7ff68b60e640 1 --2- 172.21.15.23:0/554735800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff684100f60 0x7ff68419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.897+0000 7ff68b60e640 1 --2- 172.21.15.23:0/554735800 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6841005a0 0x7ff68419c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.897+0000 7ff68b60e640 1 -- 172.21.15.23:0/554735800 >> 172.21.15.23:0/554735800 conn(0x7ff684077710 msgr2=0x7ff6840ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.897+0000 7ff68b60e640 1 -- 172.21.15.23:0/554735800 shutdown_connections 2024-03-24T10:40:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:45.897+0000 7ff68b60e640 1 -- 172.21.15.23:0/554735800 wait complete. 2024-03-24T10:40:46.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:46 smithi117 ceph-mon[36368]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T10:40:46.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:46 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1391125969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:40:46.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:46 smithi116 ceph-mon[36248]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T10:40:46.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:46 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1391125969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:40:46.910 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:40:46.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:40:46.910 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:40:46.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:46 smithi023 ceph-mon[32466]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-24T10:40:46.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:46 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1391125969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:40:47.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.017+0000 7fcc2784e640 1 -- 172.21.15.23:0/2281983433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc201005a0 msgr2=0x7fcc20100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.017+0000 7fcc2784e640 1 --2- 172.21.15.23:0/2281983433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc201005a0 0x7fcc20100a20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009890 tx=0x7fcc1002bb70 comp rx=0 tx=0).stop 2024-03-24T10:40:47.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.017+0000 7fcc2784e640 1 -- 172.21.15.23:0/2281983433 shutdown_connections 2024-03-24T10:40:47.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.017+0000 7fcc2784e640 1 --2- 172.21.15.23:0/2281983433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc20100f60 0x7fcc2010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.017+0000 7fcc2784e640 1 --2- 172.21.15.23:0/2281983433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc201005a0 0x7fcc20100a20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.017+0000 7fcc2784e640 1 --2- 172.21.15.23:0/2281983433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc20106660 0x7fcc20106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.017+0000 7fcc2784e640 1 -- 172.21.15.23:0/2281983433 >> 172.21.15.23:0/2281983433 conn(0x7fcc20077710 msgr2=0x7fcc20077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:47.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.017+0000 7fcc2784e640 1 -- 172.21.15.23:0/2281983433 shutdown_connections 2024-03-24T10:40:47.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.017+0000 7fcc2784e640 1 -- 172.21.15.23:0/2281983433 wait complete. 2024-03-24T10:40:47.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.018+0000 7fcc2784e640 1 Processor -- start 2024-03-24T10:40:47.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.018+0000 7fcc2784e640 1 -- start start 2024-03-24T10:40:47.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.018+0000 7fcc2784e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc201005a0 0x7fcc20197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.018+0000 7fcc2784e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc20100f60 0x7fcc20198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.018+0000 7fcc2784e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc20106660 0x7fcc2019c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.018+0000 7fcc2784e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2019cbf0 con 0x7fcc20100f60 2024-03-24T10:40:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.018+0000 7fcc2784e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2019cd60 con 0x7fcc201005a0 2024-03-24T10:40:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.018+0000 7fcc2784e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2019cf20 con 0x7fcc20106660 2024-03-24T10:40:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.018+0000 7fcc24de9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc20100f60 0x7fcc20198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.018+0000 7fcc24de9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc20100f60 0x7fcc20198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38680/0 (socket says 172.21.15.23:38680) 2024-03-24T10:40:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.018+0000 7fcc24de9640 1 -- 172.21.15.23:0/3965233566 learned_addr learned my addr 172.21.15.23:0/3965233566 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.018+0000 7fcc255ea640 1 --2- 172.21.15.23:0/3965233566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc201005a0 0x7fcc20197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.018+0000 7fcc25deb640 1 --2- 172.21.15.23:0/3965233566 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc20106660 0x7fcc2019c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.019+0000 7fcc24de9640 1 -- 172.21.15.23:0/3965233566 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc20106660 msgr2=0x7fcc2019c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.019+0000 7fcc24de9640 1 --2- 172.21.15.23:0/3965233566 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc20106660 0x7fcc2019c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.019+0000 7fcc24de9640 1 -- 172.21.15.23:0/3965233566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc201005a0 msgr2=0x7fcc20197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.019+0000 7fcc24de9640 1 --2- 172.21.15.23:0/3965233566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc201005a0 0x7fcc20197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.019+0000 7fcc24de9640 1 -- 172.21.15.23:0/3965233566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1c009560 con 0x7fcc20100f60 2024-03-24T10:40:47.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.019+0000 7fcc24de9640 1 --2- 172.21.15.23:0/3965233566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc20100f60 0x7fcc20198270 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009860 tx=0x7fcc10004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:47.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.019+0000 7fcc167fc640 1 -- 172.21.15.23:0/3965233566 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10034950 con 0x7fcc20100f60 2024-03-24T10:40:47.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.019+0000 7fcc2784e640 1 -- 172.21.15.23:0/3965233566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc10009490 con 0x7fcc20100f60 2024-03-24T10:40:47.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.019+0000 7fcc167fc640 1 -- 172.21.15.23:0/3965233566 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc1002c9f0 con 0x7fcc20100f60 2024-03-24T10:40:47.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.019+0000 7fcc167fc640 1 -- 172.21.15.23:0/3965233566 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10032600 con 0x7fcc20100f60 2024-03-24T10:40:47.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.019+0000 7fcc2784e640 1 -- 172.21.15.23:0/3965233566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc2019d380 con 0x7fcc20100f60 2024-03-24T10:40:47.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.021+0000 7fcc167fc640 1 -- 172.21.15.23:0/3965233566 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7fcc1003b040 con 0x7fcc20100f60 2024-03-24T10:40:47.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc2a42f740 1 Processor -- start 2024-03-24T10:40:47.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc2a42f740 1 -- start start 2024-03-24T10:40:47.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc167fc640 1 --2- 172.21.15.23:0/3965233566 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcc0c0a2350 0x7fcc0c0a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc2a42f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7440760 0x55d3d7440b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc2a42f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7441080 0x55d3d7449720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc2a42f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d744c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc2a42f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d3d73ae530 con 0x55d3d7449c60 2024-03-24T10:40:47.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc2a42f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d3d73aba60 con 0x55d3d7440760 2024-03-24T10:40:47.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc2a42f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d3d73a7240 con 0x55d3d7441080 2024-03-24T10:40:47.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc25deb640 1 --2- 172.21.15.23:0/3965233566 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcc0c0a2350 0x7fcc0c0a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc167fc640 1 -- 172.21.15.23:0/3965233566 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fcc100efc90 con 0x7fcc20100f60 2024-03-24T10:40:47.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc25deb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d744c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc255ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7440760 0x55d3d7440b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc24de9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7441080 0x55d3d7449720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc25deb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d744c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38684/0 (socket says 172.21.15.23:38684) 2024-03-24T10:40:47.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc255ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7440760 0x55d3d7440b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:45176/0 (socket says 172.21.15.23:45176) 2024-03-24T10:40:47.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc25deb640 1 -- 172.21.15.23:0/466344719 learned_addr learned my addr 172.21.15.23:0/466344719 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:47.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc24de9640 1 --2- 172.21.15.23:0/466344719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7441080 0x55d3d7449720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:53638/0 (socket says 172.21.15.23:53638) 2024-03-24T10:40:47.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc24de9640 1 -- 172.21.15.23:0/466344719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7440760 msgr2=0x55d3d7440b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc24de9640 1 --2- 172.21.15.23:0/466344719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7440760 0x55d3d7440b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc24de9640 1 -- 172.21.15.23:0/466344719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 msgr2=0x55d3d744c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc24de9640 1 --2- 172.21.15.23:0/466344719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d744c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc24de9640 1 -- 172.21.15.23:0/466344719 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3d73a58c0 con 0x55d3d7441080 2024-03-24T10:40:47.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.022+0000 7fcc25deb640 1 --2- 172.21.15.23:0/3965233566 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcc0c0a2350 0x7fcc0c0a4810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009fd0 tx=0x7fcc18009f00 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:47.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.023+0000 7fcc24de9640 1 --2- 172.21.15.23:0/466344719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7441080 0x55d3d7449720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcc100ed370 tx=0x7fcc100bc3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3c8de7ca1ecbaae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:47.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.023+0000 7fcc0b7fe640 1 -- 172.21.15.23:0/466344719 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10099df0 con 0x55d3d7441080 2024-03-24T10:40:47.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.023+0000 7fcc0b7fe640 1 -- 172.21.15.23:0/466344719 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc10099c10 con 0x55d3d7441080 2024-03-24T10:40:47.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.023+0000 7fcc0b7fe640 1 -- 172.21.15.23:0/466344719 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc100c0690 con 0x55d3d7441080 2024-03-24T10:40:47.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.023+0000 7fcc2a42f740 1 -- 172.21.15.23:0/466344719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7441080 msgr2=0x55d3d7449720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.023+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/466344719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7441080 0x55d3d7449720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcc100ed370 tx=0x7fcc100bc3c0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.023+0000 7fcc2a42f740 1 -- 172.21.15.23:0/466344719 shutdown_connections 2024-03-24T10:40:47.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.023+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/466344719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d744c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.023+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/466344719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7441080 0x55d3d7449720 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.023+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/466344719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7440760 0x55d3d7440b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.023+0000 7fcc2a42f740 1 -- 172.21.15.23:0/466344719 >> 172.21.15.23:0/466344719 conn(0x55d3d743ae90 msgr2=0x55d3d743b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.023+0000 7fcc2a42f740 1 -- 172.21.15.23:0/466344719 shutdown_connections 2024-03-24T10:40:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc2a42f740 1 -- 172.21.15.23:0/466344719 wait complete. 2024-03-24T10:40:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc2a42f740 1 -- start start 2024-03-24T10:40:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc2a42f740 1 Processor -- start 2024-03-24T10:40:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc2a42f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7440760 0x55d3d7446b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc2a42f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 0x55d3d74470d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc2a42f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7449c60 0x55d3d7447610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc2a42f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d3d734bf80 con 0x55d3d7440760 2024-03-24T10:40:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc2a42f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d3d707a4b0 con 0x55d3d7441080 2024-03-24T10:40:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc2a42f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d3d732cad0 con 0x55d3d7449c60 2024-03-24T10:40:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc2a42f740 10 client.0 _ll_register_callbacks cb 0x55d3d735fb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc255ea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7440760 0x55d3d7446b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc255ea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7440760 0x55d3d7446b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38698/0 (socket says 172.21.15.23:38698) 2024-03-24T10:40:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc255ea640 1 -- 172.21.15.23:0/686910013 learned_addr learned my addr 172.21.15.23:0/686910013 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc24de9640 1 --2- 172.21.15.23:0/686910013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 0x55d3d74470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.024+0000 7fcc25deb640 1 --2- 172.21.15.23:0/686910013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7449c60 0x55d3d7447610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.025+0000 7fcc255ea640 1 -- 172.21.15.23:0/686910013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7449c60 msgr2=0x55d3d7447610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.025+0000 7fcc255ea640 1 --2- 172.21.15.23:0/686910013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7449c60 0x55d3d7447610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.025+0000 7fcc255ea640 1 -- 172.21.15.23:0/686910013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 msgr2=0x55d3d74470d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.025+0000 7fcc255ea640 1 --2- 172.21.15.23:0/686910013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 0x55d3d74470d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.025+0000 7fcc255ea640 1 -- 172.21.15.23:0/686910013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc100b8040 con 0x55d3d7440760 2024-03-24T10:40:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.025+0000 7fcc255ea640 1 --2- 172.21.15.23:0/686910013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7440760 0x55d3d7446b90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c012980 tx=0x7fcc1c012e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c46b1e1b9805fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.025+0000 7fcc08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.025+0000 7fcc08ff9640 1 -- 172.21.15.23:0/686910013 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c017070 con 0x55d3d7440760 2024-03-24T10:40:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.025+0000 7fcc08ff9640 1 -- 172.21.15.23:0/686910013 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fcc1c004020 con 0x55d3d7440760 2024-03-24T10:40:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.025+0000 7fcc2a42f740 1 -- 172.21.15.23:0/686910013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d3d73cb7a0 con 0x55d3d7440760 2024-03-24T10:40:47.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.025+0000 7fcc08ff9640 1 -- 172.21.15.23:0/686910013 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fcc1c0122e0 con 0x55d3d7440760 2024-03-24T10:40:47.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.025+0000 7fcc2a42f740 10 client.15756 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:47.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.025+0000 7fcc2a42f740 1 -- 172.21.15.23:0/686910013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d3d735d6d0 con 0x55d3d7440760 2024-03-24T10:40:47.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.025+0000 7fcc08ff9640 1 -- 172.21.15.23:0/686910013 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d3d73cb7a0 con 0x55d3d7440760 2024-03-24T10:40:47.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.026+0000 7fcc08ff9640 1 -- 172.21.15.23:0/686910013 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fcc1c003af0 con 0x55d3d7440760 2024-03-24T10:40:47.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.026+0000 7fcc2a42f740 10 client.15756 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:47.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.026+0000 7fcc2a42f740 10 client.15756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:47.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.026+0000 7fcc2a42f740 20 client.15756 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:47.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.026+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/686910013 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d3d7459990 0x55d3d745bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.026+0000 7fcc2a42f740 4 client.15756 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:47.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.026+0000 7fcc2a42f740 1 -- 172.21.15.23:0/686910013 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3d739d1f0 con 0x55d3d7459990 2024-03-24T10:40:47.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.026+0000 7fcc24de9640 1 --2- 172.21.15.23:0/686910013 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d3d7459990 0x55d3d745bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.026+0000 7fcc24de9640 1 --2- 172.21.15.23:0/686910013 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d3d7459990 0x55d3d745bdf0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b46f76953203748 server_cookie=a04e5940ee547fc4 in_seq=0 out_seq=0 2024-03-24T10:40:47.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.026+0000 7fcc08ff9640 10 client.15756 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.027+0000 7fcc08ff9640 1 -- 172.21.15.23:0/686910013 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d3d739d1f0 con 0x55d3d7459990 2024-03-24T10:40:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.027+0000 7fcc08ff9640 10 client.15756 handle_command_reply: tid=0 2024-03-24T10:40:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.027+0000 7fcc2a42f740 1 client.15756 shutdown 2024-03-24T10:40:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.027+0000 7fcc2a42f740 1 -- 172.21.15.23:0/686910013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7440760 msgr2=0x55d3d7446b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.027+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/686910013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7440760 0x55d3d7446b90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c012980 tx=0x7fcc1c012e40 comp rx=0 tx=0).stop 2024-03-24T10:40:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.027+0000 7fcc2a42f740 1 -- 172.21.15.23:0/686910013 shutdown_connections 2024-03-24T10:40:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.027+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/686910013 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d3d7459990 0x55d3d745bdf0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.027+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/686910013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7449c60 0x55d3d7447610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.027+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/686910013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 0x55d3d74470d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.027+0000 7fcc08ff9640 0 client.15756 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.027+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/686910013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7440760 0x55d3d7446b90 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.027+0000 7fcc2a42f740 1 -- 172.21.15.23:0/686910013 >> 172.21.15.23:0/686910013 conn(0x55d3d743ae90 msgr2=0x55d3d743cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.027+0000 7fcc2a42f740 1 -- 172.21.15.23:0/686910013 shutdown_connections 2024-03-24T10:40:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.027+0000 7fcc2a42f740 1 -- 172.21.15.23:0/686910013 wait complete. 2024-03-24T10:40:47.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.028+0000 7fcc2a42f740 20 client.15756 trim_cache size 0 max 16384 2024-03-24T10:40:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc2a42f740 1 Processor -- start 2024-03-24T10:40:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc2a42f740 1 -- start start 2024-03-24T10:40:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc2a42f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7440760 0x55d3d7455220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc2a42f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 0x55d3d7439d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc2a42f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d7461700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc2a42f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d3d743bec0 con 0x55d3d7449c60 2024-03-24T10:40:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc2a42f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d3d7436650 con 0x55d3d7441080 2024-03-24T10:40:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc2a42f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d3d7458150 con 0x55d3d7440760 2024-03-24T10:40:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc25deb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d7461700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc255ea640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7440760 0x55d3d7455220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc25deb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d7461700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38714/0 (socket says 172.21.15.23:38714) 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc25deb640 1 -- 172.21.15.23:0/2010590462 learned_addr learned my addr 172.21.15.23:0/2010590462 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc24de9640 1 --2- 172.21.15.23:0/2010590462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 0x55d3d7439d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc25deb640 1 -- 172.21.15.23:0/2010590462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7440760 msgr2=0x55d3d7455220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc25deb640 1 --2- 172.21.15.23:0/2010590462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7440760 0x55d3d7455220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc25deb640 1 -- 172.21.15.23:0/2010590462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 msgr2=0x55d3d7439d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc25deb640 1 --2- 172.21.15.23:0/2010590462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 0x55d3d7439d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc25deb640 1 -- 172.21.15.23:0/2010590462 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3d7341c60 con 0x55d3d7449c60 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc24de9640 1 --2- 172.21.15.23:0/2010590462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 0x55d3d7439d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.046+0000 7fcc25deb640 1 --2- 172.21.15.23:0/2010590462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d7461700 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800fe10 tx=0x7fcc18004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e307d976a54aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.047+0000 7fcc0affd640 1 -- 172.21.15.23:0/2010590462 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18025040 con 0x55d3d7449c60 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.047+0000 7fcc0affd640 1 -- 172.21.15.23:0/2010590462 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc18016050 con 0x55d3d7449c60 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.047+0000 7fcc0affd640 1 -- 172.21.15.23:0/2010590462 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18005950 con 0x55d3d7449c60 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.047+0000 7fcc2a42f740 1 -- 172.21.15.23:0/2010590462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 msgr2=0x55d3d7461700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.047+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/2010590462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d7461700 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800fe10 tx=0x7fcc18004ab0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.047+0000 7fcc2a42f740 1 -- 172.21.15.23:0/2010590462 shutdown_connections 2024-03-24T10:40:47.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.047+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/2010590462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d7461700 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.047+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/2010590462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 0x55d3d7439d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.047+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/2010590462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7440760 0x55d3d7455220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.047+0000 7fcc2a42f740 1 -- 172.21.15.23:0/2010590462 >> 172.21.15.23:0/2010590462 conn(0x55d3d743ae90 msgr2=0x55d3d7439660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:47.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.047+0000 7fcc2a42f740 1 -- 172.21.15.23:0/2010590462 shutdown_connections 2024-03-24T10:40:47.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.047+0000 7fcc2a42f740 1 -- 172.21.15.23:0/2010590462 wait complete. 2024-03-24T10:40:47.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc2a42f740 1 -- start start 2024-03-24T10:40:47.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc2a42f740 1 Processor -- start 2024-03-24T10:40:47.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc2a42f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7440760 0x55d3d744d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc2a42f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 0x55d3d744dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc2a42f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d74bed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc2a42f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d3d744e410 con 0x55d3d7449c60 2024-03-24T10:40:47.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc2a42f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d3d74bf270 con 0x55d3d7441080 2024-03-24T10:40:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc2a42f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d3d74bf3e0 con 0x55d3d7440760 2024-03-24T10:40:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc2a42f740 10 client.0 _ll_register_callbacks cb 0x7fcc1c012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc25deb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d74bed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc25deb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d74bed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38724/0 (socket says 172.21.15.23:38724) 2024-03-24T10:40:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc25deb640 1 -- 172.21.15.23:0/2809206635 learned_addr learned my addr 172.21.15.23:0/2809206635 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc24de9640 1 --2- 172.21.15.23:0/2809206635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 0x55d3d744dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc255ea640 1 --2- 172.21.15.23:0/2809206635 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7440760 0x55d3d744d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc25deb640 1 -- 172.21.15.23:0/2809206635 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7440760 msgr2=0x55d3d744d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc25deb640 1 --2- 172.21.15.23:0/2809206635 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7440760 0x55d3d744d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc25deb640 1 -- 172.21.15.23:0/2809206635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 msgr2=0x55d3d744dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc25deb640 1 --2- 172.21.15.23:0/2809206635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 0x55d3d744dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.048+0000 7fcc25deb640 1 -- 172.21.15.23:0/2809206635 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3d73a58c0 con 0x55d3d7449c60 2024-03-24T10:40:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc25deb640 1 --2- 172.21.15.23:0/2809206635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d74bed30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009270 tx=0x7fcc180096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f519cad10c4d6e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc08ff9640 1 -- 172.21.15.23:0/2809206635 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18025070 con 0x55d3d7449c60 2024-03-24T10:40:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc08ff9640 1 -- 172.21.15.23:0/2809206635 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fcc18004b20 con 0x55d3d7449c60 2024-03-24T10:40:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc2a42f740 1 -- 172.21.15.23:0/2809206635 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc10099c10 con 0x55d3d7449c60 2024-03-24T10:40:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc08ff9640 1 -- 172.21.15.23:0/2809206635 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fcc180052f0 con 0x55d3d7449c60 2024-03-24T10:40:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc2a42f740 10 client.15762 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc2a42f740 1 -- 172.21.15.23:0/2809206635 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x7fcc1800fb50 con 0x55d3d7449c60 2024-03-24T10:40:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc08ff9640 1 -- 172.21.15.23:0/2809206635 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc10099c10 con 0x55d3d7449c60 2024-03-24T10:40:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc08ff9640 1 -- 172.21.15.23:0/2809206635 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fcc18032040 con 0x55d3d7449c60 2024-03-24T10:40:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc2a42f740 10 client.15762 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc2a42f740 10 client.15762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc2a42f740 20 client.15762 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/2809206635 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d3d74c0c10 0x55d3d74c3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc2a42f740 4 client.15762 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:40:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc2a42f740 1 -- 172.21.15.23:0/2809206635 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d3d70a6860 con 0x55d3d74c0c10 2024-03-24T10:40:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.049+0000 7fcc255ea640 1 --2- 172.21.15.23:0/2809206635 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d3d74c0c10 0x55d3d74c3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.050+0000 7fcc08ff9640 10 client.15762 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.050+0000 7fcc255ea640 1 --2- 172.21.15.23:0/2809206635 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d3d74c0c10 0x55d3d74c3030 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cca058f4b3e2368f server_cookie=bf7bfc7bcc0d4173 in_seq=0 out_seq=0 2024-03-24T10:40:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc08ff9640 1 -- 172.21.15.23:0/2809206635 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d3d70a6860 con 0x55d3d74c0c10 2024-03-24T10:40:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc08ff9640 10 client.15762 handle_command_reply: tid=0 2024-03-24T10:40:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc2a42f740 1 client.15762 shutdown 2024-03-24T10:40:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc2a42f740 1 -- 172.21.15.23:0/2809206635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 msgr2=0x55d3d74bed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/2809206635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d74bed30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009270 tx=0x7fcc180096c0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc2a42f740 1 -- 172.21.15.23:0/2809206635 shutdown_connections 2024-03-24T10:40:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/2809206635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3d7449c60 0x55d3d74bed30 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/2809206635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d3d7441080 0x55d3d744dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/2809206635 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d3d74c0c10 0x55d3d74c3030 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc2a42f740 1 --2- 172.21.15.23:0/2809206635 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3d7440760 0x55d3d744d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc08ff9640 0 client.15762 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:47.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc2a42f740 1 -- 172.21.15.23:0/2809206635 >> 172.21.15.23:0/2809206635 conn(0x55d3d743ae90 msgr2=0x55d3d7439ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:47.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc2a42f740 1 -- 172.21.15.23:0/2809206635 shutdown_connections 2024-03-24T10:40:47.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc2a42f740 1 -- 172.21.15.23:0/2809206635 wait complete. 2024-03-24T10:40:47.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.051+0000 7fcc2a42f740 20 client.15762 trim_cache size 0 max 16384 2024-03-24T10:40:47.057 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:40:47.057 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:40:47.057 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "8ef79fc7-cad9-47cc-bef4-3a4c3e195d97", 2024-03-24T10:40:47.057 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:40:47.058 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:40:47.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.052+0000 7fcc2784e640 1 -- 172.21.15.23:0/3965233566 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcc0c0a2350 msgr2=0x7fcc0c0a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.052+0000 7fcc2784e640 1 --2- 172.21.15.23:0/3965233566 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcc0c0a2350 0x7fcc0c0a4810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009fd0 tx=0x7fcc18009f00 comp rx=0 tx=0).stop 2024-03-24T10:40:47.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.052+0000 7fcc2784e640 1 -- 172.21.15.23:0/3965233566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc20100f60 msgr2=0x7fcc20198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.052+0000 7fcc2784e640 1 --2- 172.21.15.23:0/3965233566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc20100f60 0x7fcc20198270 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009860 tx=0x7fcc10004530 comp rx=0 tx=0).stop 2024-03-24T10:40:47.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.052+0000 7fcc2784e640 1 -- 172.21.15.23:0/3965233566 shutdown_connections 2024-03-24T10:40:47.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.052+0000 7fcc2784e640 1 --2- 172.21.15.23:0/3965233566 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcc0c0a2350 0x7fcc0c0a4810 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.052+0000 7fcc2784e640 1 --2- 172.21.15.23:0/3965233566 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc20106660 0x7fcc2019c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.052+0000 7fcc2784e640 1 --2- 172.21.15.23:0/3965233566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc20100f60 0x7fcc20198270 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.052+0000 7fcc2784e640 1 --2- 172.21.15.23:0/3965233566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc201005a0 0x7fcc20197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.052+0000 7fcc2784e640 1 -- 172.21.15.23:0/3965233566 >> 172.21.15.23:0/3965233566 conn(0x7fcc20077710 msgr2=0x7fcc20068b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:47.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.052+0000 7fcc2784e640 1 -- 172.21.15.23:0/3965233566 shutdown_connections 2024-03-24T10:40:47.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.052+0000 7fcc2784e640 1 -- 172.21.15.23:0/3965233566 wait complete. 2024-03-24T10:40:47.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:40:47.063 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:40:47.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.213+0000 7fe7c42c3640 1 -- 172.21.15.23:0/287343944 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7bc100c30 msgr2=0x7fe7bc106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.213+0000 7fe7c42c3640 1 --2- 172.21.15.23:0/287343944 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7bc100c30 0x7fe7bc106360 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac009930 tx=0x7fe7ac02bc30 comp rx=0 tx=0).stop 2024-03-24T10:40:47.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.213+0000 7fe7c42c3640 1 -- 172.21.15.23:0/287343944 shutdown_connections 2024-03-24T10:40:47.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.213+0000 7fe7c42c3640 1 --2- 172.21.15.23:0/287343944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7bc106a40 0x7fe7bc10e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.213+0000 7fe7c42c3640 1 --2- 172.21.15.23:0/287343944 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7bc100c30 0x7fe7bc106360 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.213+0000 7fe7c42c3640 1 --2- 172.21.15.23:0/287343944 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7bc100280 0x7fe7bc100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.213+0000 7fe7c42c3640 1 -- 172.21.15.23:0/287343944 >> 172.21.15.23:0/287343944 conn(0x7fe7bc075870 msgr2=0x7fe7bc075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:47.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.213+0000 7fe7c42c3640 1 -- 172.21.15.23:0/287343944 shutdown_connections 2024-03-24T10:40:47.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.213+0000 7fe7c42c3640 1 -- 172.21.15.23:0/287343944 wait complete. 2024-03-24T10:40:47.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.214+0000 7fe7c42c3640 1 Processor -- start 2024-03-24T10:40:47.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.214+0000 7fe7c42c3640 1 -- start start 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.214+0000 7fe7c42c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7bc100280 0x7fe7bc197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.214+0000 7fe7c42c3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7bc106a40 0x7fe7bc198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.214+0000 7fe7c42c3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7bc19c680 0x7fe7bc19cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.214+0000 7fe7c42c3640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc19d070 con 0x7fe7bc19c680 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.214+0000 7fe7c42c3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc19d1e0 con 0x7fe7bc100280 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.214+0000 7fe7c42c3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc19d350 con 0x7fe7bc106a40 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.214+0000 7fe7c2860640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7bc19c680 0x7fe7bc19cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.214+0000 7fe7c2860640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7bc19c680 0x7fe7bc19cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38742/0 (socket says 172.21.15.23:38742) 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.214+0000 7fe7c2860640 1 -- 172.21.15.23:0/3226865619 learned_addr learned my addr 172.21.15.23:0/3226865619 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.214+0000 7fe7c185e640 1 --2- 172.21.15.23:0/3226865619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7bc106a40 0x7fe7bc198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.214+0000 7fe7c205f640 1 --2- 172.21.15.23:0/3226865619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7bc100280 0x7fe7bc197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.215+0000 7fe7c2860640 1 -- 172.21.15.23:0/3226865619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7bc106a40 msgr2=0x7fe7bc198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.215+0000 7fe7c2860640 1 --2- 172.21.15.23:0/3226865619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7bc106a40 0x7fe7bc198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.215+0000 7fe7c2860640 1 -- 172.21.15.23:0/3226865619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7bc100280 msgr2=0x7fe7bc197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.215+0000 7fe7c2860640 1 --2- 172.21.15.23:0/3226865619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7bc100280 0x7fe7bc197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.215+0000 7fe7c2860640 1 -- 172.21.15.23:0/3226865619 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7ac009560 con 0x7fe7bc19c680 2024-03-24T10:40:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.215+0000 7fe7c2860640 1 --2- 172.21.15.23:0/3226865619 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7bc19c680 0x7fe7bc19cb30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400eb10 tx=0x7fe7b400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.215+0000 7fe7b37fe640 1 -- 172.21.15.23:0/3226865619 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4004b20 con 0x7fe7bc19c680 2024-03-24T10:40:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.215+0000 7fe7b37fe640 1 -- 172.21.15.23:0/3226865619 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7b4004c80 con 0x7fe7bc19c680 2024-03-24T10:40:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.215+0000 7fe7c42c3640 1 -- 172.21.15.23:0/3226865619 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7bc1a4930 con 0x7fe7bc19c680 2024-03-24T10:40:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.215+0000 7fe7b37fe640 1 -- 172.21.15.23:0/3226865619 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4010650 con 0x7fe7bc19c680 2024-03-24T10:40:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.215+0000 7fe7c42c3640 1 -- 172.21.15.23:0/3226865619 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7bc1a4d60 con 0x7fe7bc19c680 2024-03-24T10:40:47.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.217+0000 7fe7b37fe640 1 -- 172.21.15.23:0/3226865619 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7fe7b4005020 con 0x7fe7bc19c680 2024-03-24T10:40:47.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.218+0000 7fe7c6f7e740 1 Processor -- start 2024-03-24T10:40:47.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.218+0000 7fe7c6f7e740 1 -- start start 2024-03-24T10:40:47.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.218+0000 7fe7b37fe640 1 --2- 172.21.15.23:0/3226865619 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe7a80a2350 0x7fe7a80a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.218+0000 7fe7c185e640 1 --2- 172.21.15.23:0/3226865619 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe7a80a2350 0x7fe7a80a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.218+0000 7fe7c6f7e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e148c1d0 0x5602e148c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.218+0000 7fe7b37fe640 1 -- 172.21.15.23:0/3226865619 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe7b4014070 con 0x7fe7bc19c680 2024-03-24T10:40:47.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.218+0000 7fe7c6f7e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148caf0 0x5602e1495190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.218+0000 7fe7c6f7e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e14956d0 0x5602e1497ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.218+0000 7fe7c6f7e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5602e13b0f30 con 0x5602e14956d0 2024-03-24T10:40:47.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.218+0000 7fe7c6f7e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5602e13fa810 con 0x5602e148c1d0 2024-03-24T10:40:47.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.218+0000 7fe7c6f7e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5602e13f7d40 con 0x5602e148caf0 2024-03-24T10:40:47.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7c2860640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e14956d0 0x5602e1497ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7c185e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148caf0 0x5602e1495190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7c2860640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e14956d0 0x5602e1497ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38750/0 (socket says 172.21.15.23:38750) 2024-03-24T10:40:47.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7c2860640 1 -- 172.21.15.23:0/2496468320 learned_addr learned my addr 172.21.15.23:0/2496468320 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:47.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7c205f640 1 --2- 172.21.15.23:0/2496468320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e148c1d0 0x5602e148c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7c185e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148caf0 0x5602e1495190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:53674/0 (socket says 172.21.15.23:53674) 2024-03-24T10:40:47.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7c2860640 1 -- 172.21.15.23:0/2496468320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148caf0 msgr2=0x5602e1495190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7c2860640 1 --2- 172.21.15.23:0/2496468320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148caf0 0x5602e1495190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7c2860640 1 -- 172.21.15.23:0/2496468320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e148c1d0 msgr2=0x5602e148c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7c2860640 1 --2- 172.21.15.23:0/2496468320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e148c1d0 0x5602e148c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7c2860640 1 -- 172.21.15.23:0/2496468320 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602e13b4ea0 con 0x5602e14956d0 2024-03-24T10:40:47.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7c185e640 1 --2- 172.21.15.23:0/3226865619 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe7a80a2350 0x7fe7a80a4810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac000c00 tx=0x7fe7ac004a80 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7c2860640 1 --2- 172.21.15.23:0/2496468320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e14956d0 0x5602e1497ac0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe7b40974a0 tx=0x7fe7b4009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd456bb65c32e94d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7b0bfe640 1 -- 172.21.15.23:0/2496468320 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b40b1030 con 0x5602e14956d0 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7b0bfe640 1 -- 172.21.15.23:0/2496468320 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7b409f190 con 0x5602e14956d0 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.219+0000 7fe7b0bfe640 1 -- 172.21.15.23:0/2496468320 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b40a35b0 con 0x5602e14956d0 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.220+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/2496468320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e14956d0 msgr2=0x5602e1497ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.220+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/2496468320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e14956d0 0x5602e1497ac0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe7b40974a0 tx=0x7fe7b4009a70 comp rx=0 tx=0).stop 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.220+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/2496468320 shutdown_connections 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.220+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/2496468320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e14956d0 0x5602e1497ac0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.220+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/2496468320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148caf0 0x5602e1495190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.220+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/2496468320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e148c1d0 0x5602e148c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.220+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/2496468320 >> 172.21.15.23:0/2496468320 conn(0x5602e1486900 msgr2=0x5602e1486ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.220+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/2496468320 shutdown_connections 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.220+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/2496468320 wait complete. 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.220+0000 7fe7c6f7e740 1 -- start start 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c6f7e740 1 Processor -- start 2024-03-24T10:40:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c6f7e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148c1d0 0x5602e1492700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c6f7e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e148caf0 0x5602e1492c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c6f7e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e14956d0 0x5602e1493180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c6f7e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5602e13f3520 con 0x5602e148caf0 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c6f7e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5602e10c6430 con 0x5602e14956d0 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c6f7e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5602e1082e00 con 0x5602e148c1d0 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c6f7e740 10 client.0 _ll_register_callbacks cb 0x5602e1391d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c185e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e148caf0 0x5602e1492c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c185e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e148caf0 0x5602e1492c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38762/0 (socket says 172.21.15.23:38762) 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c205f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148c1d0 0x5602e1492700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c185e640 1 -- 172.21.15.23:0/2134719142 learned_addr learned my addr 172.21.15.23:0/2134719142 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c2860640 1 --2- 172.21.15.23:0/2134719142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e14956d0 0x5602e1493180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c205f640 1 -- 172.21.15.23:0/2134719142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e14956d0 msgr2=0x5602e1493180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c205f640 1 --2- 172.21.15.23:0/2134719142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e14956d0 0x5602e1493180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c205f640 1 -- 172.21.15.23:0/2134719142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e148caf0 msgr2=0x5602e1492c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c205f640 1 --2- 172.21.15.23:0/2134719142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e148caf0 0x5602e1492c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c205f640 1 -- 172.21.15.23:0/2134719142 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7bc1a4d60 con 0x5602e148c1d0 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c185e640 1 --2- 172.21.15.23:0/2134719142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e148caf0 0x5602e1492c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7c205f640 1 --2- 172.21.15.23:0/2134719142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148c1d0 0x5602e1492700 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8012730 tx=0x7fe7b8012bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=934b53a59f9640f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.221+0000 7fe7a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7a5ffb640 1 -- 172.21.15.23:0/2134719142 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8017070 con 0x5602e148c1d0 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7a5ffb640 1 -- 172.21.15.23:0/2134719142 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe7b8012e40 con 0x5602e148c1d0 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/2134719142 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5602e10f27e0 con 0x5602e148c1d0 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7a5ffb640 1 -- 172.21.15.23:0/2134719142 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe7b800f840 con 0x5602e148c1d0 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7c6f7e740 10 client.25642 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/2134719142 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5602e13f1ba0 con 0x5602e148c1d0 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7a5ffb640 1 -- 172.21.15.23:0/2134719142 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7b8028080 con 0x5602e148c1d0 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7a5ffb640 1 -- 172.21.15.23:0/2134719142 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe7b8022030 con 0x5602e148c1d0 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7c6f7e740 10 client.25642 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7c6f7e740 10 client.25642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7c6f7e740 20 client.25642 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:47.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/2134719142 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5602e14a5c40 0x5602e14a8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7c6f7e740 4 client.25642 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:47.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/2134719142 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602e112f720 con 0x5602e14a5c40 2024-03-24T10:40:47.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.222+0000 7fe7c185e640 1 --2- 172.21.15.23:0/2134719142 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5602e14a5c40 0x5602e14a8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.223+0000 7fe7c185e640 1 --2- 172.21.15.23:0/2134719142 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5602e14a5c40 0x5602e14a8080 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=722ddc55f9037039 server_cookie=2deecbb5f945b706 in_seq=0 out_seq=0 2024-03-24T10:40:47.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.223+0000 7fe7a5ffb640 10 client.25642 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:47.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.223+0000 7fe7a5ffb640 1 -- 172.21.15.23:0/2134719142 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5602e112f720 con 0x5602e14a5c40 2024-03-24T10:40:47.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.223+0000 7fe7a5ffb640 10 client.25642 handle_command_reply: tid=0 2024-03-24T10:40:47.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.223+0000 7fe7c6f7e740 1 client.25642 shutdown 2024-03-24T10:40:47.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.223+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/2134719142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148c1d0 msgr2=0x5602e1492700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.223+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/2134719142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148c1d0 0x5602e1492700 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8012730 tx=0x7fe7b8012bf0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.223+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/2134719142 shutdown_connections 2024-03-24T10:40:47.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.223+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/2134719142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e14956d0 0x5602e1493180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.223+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/2134719142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e148caf0 0x5602e1492c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.223+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/2134719142 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5602e14a5c40 0x5602e14a8080 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.223+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/2134719142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148c1d0 0x5602e1492700 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.223+0000 7fe7a5ffb640 0 client.25642 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:47.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.224+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/2134719142 >> 172.21.15.23:0/2134719142 conn(0x5602e1486900 msgr2=0x5602e1485b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:47.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.224+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/2134719142 shutdown_connections 2024-03-24T10:40:47.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.224+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/2134719142 wait complete. 2024-03-24T10:40:47.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.224+0000 7fe7c6f7e740 20 client.25642 trim_cache size 0 max 16384 2024-03-24T10:40:47.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c6f7e740 1 Processor -- start 2024-03-24T10:40:47.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c6f7e740 1 -- start start 2024-03-24T10:40:47.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c6f7e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148c1d0 0x5602e14a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c6f7e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e148caf0 0x5602e14ad990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c6f7e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e14956d0 0x5602e1492c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c6f7e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5602e1378430 con 0x5602e14956d0 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c6f7e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5602e148bba0 con 0x5602e148caf0 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c6f7e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5602e1490c50 con 0x5602e148c1d0 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c2860640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e14956d0 0x5602e1492c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c205f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148c1d0 0x5602e14a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c205f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148c1d0 0x5602e14a0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:53678/0 (socket says 172.21.15.23:53678) 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c2860640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e14956d0 0x5602e1492c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38774/0 (socket says 172.21.15.23:38774) 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c205f640 1 -- 172.21.15.23:0/3239136308 learned_addr learned my addr 172.21.15.23:0/3239136308 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c185e640 1 --2- 172.21.15.23:0/3239136308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e148caf0 0x5602e14ad990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c205f640 1 -- 172.21.15.23:0/3239136308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e148caf0 msgr2=0x5602e14ad990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c205f640 1 --2- 172.21.15.23:0/3239136308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e148caf0 0x5602e14ad990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c205f640 1 -- 172.21.15.23:0/3239136308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e14956d0 msgr2=0x5602e1492c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c205f640 1 --2- 172.21.15.23:0/3239136308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e14956d0 0x5602e1492c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c205f640 1 -- 172.21.15.23:0/3239136308 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602e138db20 con 0x5602e148c1d0 2024-03-24T10:40:47.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c2860640 1 --2- 172.21.15.23:0/3239136308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e14956d0 0x5602e1492c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:47.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.242+0000 7fe7c205f640 1 --2- 172.21.15.23:0/3239136308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148c1d0 0x5602e14a0f60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8001eb0 tx=0x7fe7b80098b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a4da39b352bf536 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:47.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.243+0000 7fe7a7fff640 1 -- 172.21.15.23:0/3239136308 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8017070 con 0x5602e148c1d0 2024-03-24T10:40:47.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.243+0000 7fe7a7fff640 1 -- 172.21.15.23:0/3239136308 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5602e10f27e0 con 0x5602e148c1d0 2024-03-24T10:40:47.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.243+0000 7fe7a7fff640 1 -- 172.21.15.23:0/3239136308 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8010980 con 0x5602e148c1d0 2024-03-24T10:40:47.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.243+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/3239136308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148c1d0 msgr2=0x5602e14a0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.243+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/3239136308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148c1d0 0x5602e14a0f60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8001eb0 tx=0x7fe7b80098b0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.243+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/3239136308 shutdown_connections 2024-03-24T10:40:47.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.243+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/3239136308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e14956d0 0x5602e1492c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.243+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/3239136308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e148caf0 0x5602e14ad990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.243+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/3239136308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148c1d0 0x5602e14a0f60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.243+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/3239136308 >> 172.21.15.23:0/3239136308 conn(0x5602e1486900 msgr2=0x5602e1485b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:47.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.243+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/3239136308 shutdown_connections 2024-03-24T10:40:47.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.243+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/3239136308 wait complete. 2024-03-24T10:40:47.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.243+0000 7fe7c6f7e740 1 -- start start 2024-03-24T10:40:47.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.243+0000 7fe7c6f7e740 1 Processor -- start 2024-03-24T10:40:47.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c6f7e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e148c1d0 0x5602e1498f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c6f7e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148caf0 0x5602e1499490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c6f7e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e14956d0 0x5602e14999d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c6f7e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5602e150afc0 con 0x5602e148c1d0 2024-03-24T10:40:47.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c6f7e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5602e150b100 con 0x5602e14956d0 2024-03-24T10:40:47.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c6f7e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5602e150b290 con 0x5602e148caf0 2024-03-24T10:40:47.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c6f7e740 10 client.0 _ll_register_callbacks cb 0x7fe7b8009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:47.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c205f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e148c1d0 0x5602e1498f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c205f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e148c1d0 0x5602e1498f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38778/0 (socket says 172.21.15.23:38778) 2024-03-24T10:40:47.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c205f640 1 -- 172.21.15.23:0/656894957 learned_addr learned my addr 172.21.15.23:0/656894957 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:47.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c2860640 1 --2- 172.21.15.23:0/656894957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e14956d0 0x5602e14999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c185e640 1 --2- 172.21.15.23:0/656894957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148caf0 0x5602e1499490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c205f640 1 -- 172.21.15.23:0/656894957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148caf0 msgr2=0x5602e1499490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c205f640 1 --2- 172.21.15.23:0/656894957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148caf0 0x5602e1499490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c205f640 1 -- 172.21.15.23:0/656894957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e14956d0 msgr2=0x5602e14999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:47.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c205f640 1 --2- 172.21.15.23:0/656894957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e14956d0 0x5602e14999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c205f640 1 -- 172.21.15.23:0/656894957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602e13b4ea0 con 0x5602e148c1d0 2024-03-24T10:40:47.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c205f640 1 --2- 172.21.15.23:0/656894957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e148c1d0 0x5602e1498f50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8005640 tx=0x7fe7b8005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fc309f93cb7902 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:47.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:47.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7a5ffb640 1 -- 172.21.15.23:0/656894957 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8017070 con 0x5602e148c1d0 2024-03-24T10:40:47.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7a5ffb640 1 -- 172.21.15.23:0/656894957 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe7b8012e40 con 0x5602e148c1d0 2024-03-24T10:40:47.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.244+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/656894957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7b409f190 con 0x5602e148c1d0 2024-03-24T10:40:47.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.245+0000 7fe7a5ffb640 1 -- 172.21.15.23:0/656894957 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe7b8010430 con 0x5602e148c1d0 2024-03-24T10:40:47.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.245+0000 7fe7c6f7e740 10 client.15780 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:47.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.245+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/656894957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5602e13f1ba0 con 0x5602e148c1d0 2024-03-24T10:40:47.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.245+0000 7fe7a5ffb640 1 -- 172.21.15.23:0/656894957 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7b409f190 con 0x5602e148c1d0 2024-03-24T10:40:47.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.245+0000 7fe7a5ffb640 1 -- 172.21.15.23:0/656894957 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe7b8003950 con 0x5602e148c1d0 2024-03-24T10:40:47.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.245+0000 7fe7c6f7e740 10 client.15780 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:47.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.245+0000 7fe7c6f7e740 10 client.15780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:47.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.245+0000 7fe7c6f7e740 20 client.15780 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:47.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.245+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/656894957 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5602e150c770 0x5602e150eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:47.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.245+0000 7fe7c6f7e740 4 client.15780 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:40:47.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.245+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/656894957 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5602e117e890 con 0x5602e150c770 2024-03-24T10:40:47.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.245+0000 7fe7c185e640 1 --2- 172.21.15.23:0/656894957 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5602e150c770 0x5602e150eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:47.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.245+0000 7fe7a5ffb640 10 client.15780 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:47.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.245+0000 7fe7c185e640 1 --2- 172.21.15.23:0/656894957 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5602e150c770 0x5602e150eb60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e0627ec9436946e server_cookie=42cfa454111dc769 in_seq=0 out_seq=0 2024-03-24T10:40:47.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7a5ffb640 1 -- 172.21.15.23:0/656894957 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5602e117e890 con 0x5602e150c770 2024-03-24T10:40:47.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7a5ffb640 10 client.15780 handle_command_reply: tid=0 2024-03-24T10:40:47.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c6f7e740 1 client.15780 shutdown 2024-03-24T10:40:47.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/656894957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e148c1d0 msgr2=0x5602e1498f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/656894957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e148c1d0 0x5602e1498f50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8005640 tx=0x7fe7b8005670 comp rx=0 tx=0).stop 2024-03-24T10:40:47.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/656894957 shutdown_connections 2024-03-24T10:40:47.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/656894957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602e14956d0 0x5602e14999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/656894957 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5602e150c770 0x5602e150eb60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/656894957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5602e148caf0 0x5602e1499490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c6f7e740 1 --2- 172.21.15.23:0/656894957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602e148c1d0 0x5602e1498f50 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7a5ffb640 0 client.15780 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:47.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/656894957 >> 172.21.15.23:0/656894957 conn(0x5602e1486900 msgr2=0x5602e1487ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:47.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/656894957 shutdown_connections 2024-03-24T10:40:47.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c6f7e740 1 -- 172.21.15.23:0/656894957 wait complete. 2024-03-24T10:40:47.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c6f7e740 20 client.15780 trim_cache size 0 max 16384 2024-03-24T10:40:47.255 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:40:47.255 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:40:47.255 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:40:47.255 INFO:teuthology.orchestra.run.smithi023.stdout: "8ef79fc7-cad9-47cc-bef4-3a4c3e195d97": { 2024-03-24T10:40:47.255 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:40:47.255 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "8ef79fc7-cad9-47cc-bef4-3a4c3e195d97", 2024-03-24T10:40:47.255 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:40:47.255 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:40:47.255 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:40:47.256 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:40:47.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c42c3640 1 -- 172.21.15.23:0/3226865619 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe7a80a2350 msgr2=0x7fe7a80a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c42c3640 1 --2- 172.21.15.23:0/3226865619 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe7a80a2350 0x7fe7a80a4810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac000c00 tx=0x7fe7ac004a80 comp rx=0 tx=0).stop 2024-03-24T10:40:47.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c42c3640 1 -- 172.21.15.23:0/3226865619 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7bc19c680 msgr2=0x7fe7bc19cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:47.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c42c3640 1 --2- 172.21.15.23:0/3226865619 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7bc19c680 0x7fe7bc19cb30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400eb10 tx=0x7fe7b400c2d0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c42c3640 1 -- 172.21.15.23:0/3226865619 shutdown_connections 2024-03-24T10:40:47.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c42c3640 1 --2- 172.21.15.23:0/3226865619 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe7a80a2350 0x7fe7a80a4810 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c42c3640 1 --2- 172.21.15.23:0/3226865619 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7bc19c680 0x7fe7bc19cb30 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c42c3640 1 --2- 172.21.15.23:0/3226865619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7bc106a40 0x7fe7bc198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c42c3640 1 --2- 172.21.15.23:0/3226865619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7bc100280 0x7fe7bc197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:47.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.246+0000 7fe7c42c3640 1 -- 172.21.15.23:0/3226865619 >> 172.21.15.23:0/3226865619 conn(0x7fe7bc075870 msgr2=0x7fe7bc0fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:47.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.247+0000 7fe7c42c3640 1 -- 172.21.15.23:0/3226865619 shutdown_connections 2024-03-24T10:40:47.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:40:47.247+0000 7fe7c42c3640 1 -- 172.21.15.23:0/3226865619 wait complete. 2024-03-24T10:40:47.258 INFO:tasks.cephfs.filesystem:scrub status for tag:8ef79fc7-cad9-47cc-bef4-3a4c3e195d97 - {'path': '/', 'tag': '8ef79fc7-cad9-47cc-bef4-3a4c3e195d97', 'options': 'recursive,force'} 2024-03-24T10:40:48.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:48 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:40:48.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:48 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:48.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:48 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:40:48.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:48 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:48.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:48 smithi117 ceph-mon[36368]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-24T10:40:48.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:48 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:40:48.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:48 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:48.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:48 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:40:48.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:48 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:48.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:48 smithi116 ceph-mon[36248]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-24T10:40:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:48 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:40:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:48 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:48 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:40:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:48 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:40:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:48 smithi023 ceph-mon[32466]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-24T10:40:50.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:50 smithi117 ceph-mon[36368]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-24T10:40:50.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:50 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:40:50.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:50 smithi116 ceph-mon[36248]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-24T10:40:50.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:50 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:40:50.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:50 smithi023 ceph-mon[32466]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-24T10:40:50.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:50 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:40:52.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:52 smithi117 ceph-mon[36368]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-24T10:40:52.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:52 smithi116 ceph-mon[36248]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-24T10:40:52.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:52 smithi023 ceph-mon[32466]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-24T10:40:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.341+0000 7f5564d3e640 1 -- 172.21.15.23:0/1312661528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5560101220 msgr2=0x7f556010d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.341+0000 7f5564d3e640 1 --2- 172.21.15.23:0/1312661528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5560101220 0x7f556010d440 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5558009ca0 tx=0x7f555802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.341+0000 7f5564d3e640 1 -- 172.21.15.23:0/1312661528 shutdown_connections 2024-03-24T10:40:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.341+0000 7f5564d3e640 1 --2- 172.21.15.23:0/1312661528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5560101220 0x7f556010d440 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.341+0000 7f5564d3e640 1 --2- 172.21.15.23:0/1312661528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5560100860 0x7f5560100ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.341+0000 7f5564d3e640 1 --2- 172.21.15.23:0/1312661528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55601055b0 0x7f5560105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.341+0000 7f5564d3e640 1 -- 172.21.15.23:0/1312661528 >> 172.21.15.23:0/1312661528 conn(0x7f5560077460 msgr2=0x7f5560077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.341+0000 7f5564d3e640 1 -- 172.21.15.23:0/1312661528 shutdown_connections 2024-03-24T10:40:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.341+0000 7f5564d3e640 1 -- 172.21.15.23:0/1312661528 wait complete. 2024-03-24T10:40:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.342+0000 7f5564d3e640 1 Processor -- start 2024-03-24T10:40:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.342+0000 7f5564d3e640 1 -- start start 2024-03-24T10:40:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.342+0000 7f5564d3e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5560100860 0x7f5560197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.342+0000 7f5564d3e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5560101220 0x7f55601982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.342+0000 7f5564d3e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55601055b0 0x7f556019c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.342+0000 7f5564d3e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f556019cc30 con 0x7f5560100860 2024-03-24T10:40:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.342+0000 7f5564d3e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f556019cda0 con 0x7f5560101220 2024-03-24T10:40:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.342+0000 7f5564d3e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f556019cf60 con 0x7f55601055b0 2024-03-24T10:40:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.342+0000 7f555e59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5560100860 0x7f5560197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f555e59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5560100860 0x7f5560197d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45478/0 (socket says 172.21.15.23:45478) 2024-03-24T10:40:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f555e59c640 1 -- 172.21.15.23:0/2189226490 learned_addr learned my addr 172.21.15.23:0/2189226490 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f5557fff640 1 --2- 172.21.15.23:0/2189226490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5560101220 0x7f55601982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f555ed9d640 1 --2- 172.21.15.23:0/2189226490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55601055b0 0x7f556019c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f555e59c640 1 -- 172.21.15.23:0/2189226490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55601055b0 msgr2=0x7f556019c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f555e59c640 1 --2- 172.21.15.23:0/2189226490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55601055b0 0x7f556019c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f555e59c640 1 -- 172.21.15.23:0/2189226490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5560101220 msgr2=0x7f55601982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f555e59c640 1 --2- 172.21.15.23:0/2189226490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5560101220 0x7f55601982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f555e59c640 1 -- 172.21.15.23:0/2189226490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5558009900 con 0x7f5560100860 2024-03-24T10:40:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f555e59c640 1 --2- 172.21.15.23:0/2189226490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5560100860 0x7f5560197d70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f555000e9f0 tx=0x7f555000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f55577fe640 1 -- 172.21.15.23:0/2189226490 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550009700 con 0x7f5560100860 2024-03-24T10:40:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f5564d3e640 1 -- 172.21.15.23:0/2189226490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f556019d240 con 0x7f5560100860 2024-03-24T10:40:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f55577fe640 1 -- 172.21.15.23:0/2189226490 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f555000cd80 con 0x7f5560100860 2024-03-24T10:40:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f55577fe640 1 -- 172.21.15.23:0/2189226490 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550018920 con 0x7f5560100860 2024-03-24T10:40:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.343+0000 7f5564d3e640 1 -- 172.21.15.23:0/2189226490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55601a4ac0 con 0x7f5560100860 2024-03-24T10:40:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.346+0000 7f55679f2740 1 Processor -- start 2024-03-24T10:40:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.346+0000 7f55679f2740 1 -- start start 2024-03-24T10:40:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.346+0000 7f55577fe640 1 -- 172.21.15.23:0/2189226490 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) ==== 136983+0+0 (secure 0 0 0) 0x7f5550018a80 con 0x7f5560100860 2024-03-24T10:40:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.346+0000 7f55679f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820b35e0 0x5650820b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.346+0000 7f55679f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b3f00 0x5650820bc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f55679f2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820bccd0 0x5650820bf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f55679f2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56508206e1b0 con 0x5650820bccd0 2024-03-24T10:40:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f55679f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x565082020f80 con 0x5650820b35e0 2024-03-24T10:40:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f55679f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56508201e4b0 con 0x5650820b3f00 2024-03-24T10:40:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f5557fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820bccd0 0x5650820bf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f555ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820b35e0 0x5650820b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f555ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820b35e0 0x5650820b39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:45550/0 (socket says 172.21.15.23:45550) 2024-03-24T10:40:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f555e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b3f00 0x5650820bc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f555ed9d640 1 -- 172.21.15.23:0/3414109351 learned_addr learned my addr 172.21.15.23:0/3414109351 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f5557fff640 1 -- 172.21.15.23:0/3414109351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b3f00 msgr2=0x5650820bc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f55577fe640 1 --2- 172.21.15.23:0/2189226490 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f554c0a23f0 0x7f554c0a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f5557fff640 1 --2- 172.21.15.23:0/3414109351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b3f00 0x5650820bc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f5557fff640 1 -- 172.21.15.23:0/3414109351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820b35e0 msgr2=0x5650820b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f5557fff640 1 --2- 172.21.15.23:0/3414109351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820b35e0 0x5650820b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f5557fff640 1 -- 172.21.15.23:0/3414109351 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565081faf810 con 0x5650820bccd0 2024-03-24T10:40:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f555ed9d640 1 --2- 172.21.15.23:0/3414109351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820b35e0 0x5650820b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f555ed9d640 1 --2- 172.21.15.23:0/2189226490 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f554c0a23f0 0x7f554c0a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f55577fe640 1 -- 172.21.15.23:0/2189226490 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f5550014070 con 0x7f5560100860 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f5557fff640 1 --2- 172.21.15.23:0/3414109351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820bccd0 0x5650820bf0c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5548009bd0 tx=0x7f5548011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4cd43610cd387cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.347+0000 7f5554ff9640 1 -- 172.21.15.23:0/3414109351 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5548004230 con 0x5650820bccd0 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.348+0000 7f5554ff9640 1 -- 172.21.15.23:0/3414109351 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5548009e40 con 0x5650820bccd0 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.348+0000 7f5554ff9640 1 -- 172.21.15.23:0/3414109351 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5548004d90 con 0x5650820bccd0 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.348+0000 7f55679f2740 1 -- 172.21.15.23:0/3414109351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820bccd0 msgr2=0x5650820bf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.348+0000 7f555ed9d640 1 --2- 172.21.15.23:0/2189226490 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f554c0a23f0 0x7f554c0a48b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5558037040 tx=0x7f555802e040 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.348+0000 7f55679f2740 1 --2- 172.21.15.23:0/3414109351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820bccd0 0x5650820bf0c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5548009bd0 tx=0x7f5548011810 comp rx=0 tx=0).stop 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.348+0000 7f55679f2740 1 -- 172.21.15.23:0/3414109351 shutdown_connections 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.348+0000 7f55679f2740 1 --2- 172.21.15.23:0/3414109351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820bccd0 0x5650820bf0c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.348+0000 7f55679f2740 1 --2- 172.21.15.23:0/3414109351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b3f00 0x5650820bc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.348+0000 7f55679f2740 1 --2- 172.21.15.23:0/3414109351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820b35e0 0x5650820b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.348+0000 7f55679f2740 1 -- 172.21.15.23:0/3414109351 >> 172.21.15.23:0/3414109351 conn(0x5650820adf00 msgr2=0x5650820ae2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.348+0000 7f55679f2740 1 -- 172.21.15.23:0/3414109351 shutdown_connections 2024-03-24T10:40:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.348+0000 7f55679f2740 1 -- 172.21.15.23:0/3414109351 wait complete. 2024-03-24T10:40:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f55679f2740 1 -- start start 2024-03-24T10:40:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f55679f2740 1 Processor -- start 2024-03-24T10:40:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f55679f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b35e0 0x5650820b9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f55679f2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b3f00 0x5650820ba260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f55679f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 0x5650820ba7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f55679f2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x565082019c90 con 0x5650820b3f00 2024-03-24T10:40:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f55679f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x565081fbec50 con 0x5650820bccd0 2024-03-24T10:40:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f55679f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x565081ced250 con 0x5650820b35e0 2024-03-24T10:40:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f55679f2740 10 client.0 _ll_register_callbacks cb 0x56508200d290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f5557fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b3f00 0x5650820ba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f5557fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b3f00 0x5650820ba260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45498/0 (socket says 172.21.15.23:45498) 2024-03-24T10:40:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f5557fff640 1 -- 172.21.15.23:0/1273363916 learned_addr learned my addr 172.21.15.23:0/1273363916 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f555e59c640 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b35e0 0x5650820b9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f555ed9d640 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 0x5650820ba7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f5557fff640 1 -- 172.21.15.23:0/1273363916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b35e0 msgr2=0x5650820b9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f5557fff640 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b35e0 0x5650820b9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f5557fff640 1 -- 172.21.15.23:0/1273363916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 msgr2=0x5650820ba7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f5557fff640 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 0x5650820ba7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.349+0000 7f5557fff640 1 -- 172.21.15.23:0/1273363916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5548009830 con 0x5650820b3f00 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f555ed9d640 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 0x5650820ba7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f555e59c640 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b35e0 0x5650820b9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f5557fff640 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b3f00 0x5650820ba260 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5548011d10 tx=0x7f55480038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9bc264d78008d34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f5545ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f5545ffb640 1 -- 172.21.15.23:0/1273363916 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5548003cc0 con 0x5650820b3f00 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f5545ffb640 1 -- 172.21.15.23:0/1273363916 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f55480046c0 con 0x5650820b3f00 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f55679f2740 1 -- 172.21.15.23:0/1273363916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56508203e790 con 0x5650820b3f00 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f5545ffb640 1 -- 172.21.15.23:0/1273363916 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5548003e20 con 0x5650820b3f00 2024-03-24T10:40:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f55679f2740 10 client.15792 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f55679f2740 1 -- 172.21.15.23:0/1273363916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x565082018310 con 0x5650820b3f00 2024-03-24T10:40:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f5545ffb640 1 -- 172.21.15.23:0/1273363916 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56508203e790 con 0x5650820b3f00 2024-03-24T10:40:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f5545ffb640 1 -- 172.21.15.23:0/1273363916 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f55480193f0 con 0x5650820b3f00 2024-03-24T10:40:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f55679f2740 10 client.15792 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f55679f2740 10 client.15792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f55679f2740 20 client.15792 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f55679f2740 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650820d0f90 0x5650820d33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f55679f2740 4 client.15792 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:40:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.350+0000 7f55679f2740 1 -- 172.21.15.23:0/1273363916 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565081d19600 con 0x5650820d0f90 2024-03-24T10:40:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.351+0000 7f555e59c640 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650820d0f90 0x5650820d33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.351+0000 7f5545ffb640 10 client.15792 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.351+0000 7f555e59c640 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650820d0f90 0x5650820d33b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e51de8ba88d5fd1 server_cookie=492cfeed2d76492a in_seq=0 out_seq=0 2024-03-24T10:40:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f5545ffb640 1 -- 172.21.15.23:0/1273363916 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x565081d19600 con 0x5650820d0f90 2024-03-24T10:40:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f5545ffb640 10 client.15792 handle_command_reply: tid=0 2024-03-24T10:40:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f55679f2740 1 client.15792 shutdown 2024-03-24T10:40:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f55679f2740 1 -- 172.21.15.23:0/1273363916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b3f00 msgr2=0x5650820ba260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f55679f2740 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b3f00 0x5650820ba260 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5548011d10 tx=0x7f55480038a0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f55679f2740 1 -- 172.21.15.23:0/1273363916 shutdown_connections 2024-03-24T10:40:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f55679f2740 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 0x5650820ba7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f55679f2740 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b3f00 0x5650820ba260 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f55679f2740 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650820d0f90 0x5650820d33b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f55679f2740 1 --2- 172.21.15.23:0/1273363916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b35e0 0x5650820b9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f5545ffb640 0 client.15792 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f55679f2740 1 -- 172.21.15.23:0/1273363916 >> 172.21.15.23:0/1273363916 conn(0x5650820adf00 msgr2=0x5650820afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f55679f2740 1 -- 172.21.15.23:0/1273363916 shutdown_connections 2024-03-24T10:40:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f55679f2740 1 -- 172.21.15.23:0/1273363916 wait complete. 2024-03-24T10:40:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.352+0000 7f55679f2740 20 client.15792 trim_cache size 0 max 16384 2024-03-24T10:40:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f55679f2740 1 Processor -- start 2024-03-24T10:40:53.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f55679f2740 1 -- start start 2024-03-24T10:40:53.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f55679f2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b35e0 0x5650820bc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f55679f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b3f00 0x5650820cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f55679f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 0x5650820b4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f55679f2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x565081f9f660 con 0x5650820b35e0 2024-03-24T10:40:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f55679f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5650820ac3f0 con 0x5650820bccd0 2024-03-24T10:40:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f55679f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5650820b3000 con 0x5650820b3f00 2024-03-24T10:40:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f555e59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b35e0 0x5650820bc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f555e59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b35e0 0x5650820bc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45506/0 (socket says 172.21.15.23:45506) 2024-03-24T10:40:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f555e59c640 1 -- 172.21.15.23:0/3145324833 learned_addr learned my addr 172.21.15.23:0/3145324833 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f5557fff640 1 --2- 172.21.15.23:0/3145324833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b3f00 0x5650820cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f555ed9d640 1 --2- 172.21.15.23:0/3145324833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 0x5650820b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f555e59c640 1 -- 172.21.15.23:0/3145324833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b3f00 msgr2=0x5650820cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f555e59c640 1 --2- 172.21.15.23:0/3145324833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b3f00 0x5650820cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f555e59c640 1 -- 172.21.15.23:0/3145324833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 msgr2=0x5650820b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f555e59c640 1 --2- 172.21.15.23:0/3145324833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 0x5650820b4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f555e59c640 1 -- 172.21.15.23:0/3145324833 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565081fb4810 con 0x5650820b35e0 2024-03-24T10:40:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f555ed9d640 1 --2- 172.21.15.23:0/3145324833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 0x5650820b4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:40:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f555e59c640 1 --2- 172.21.15.23:0/3145324833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b35e0 0x5650820bc450 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f55500cbd10 tx=0x7f55500d0940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb75c0a4f1a1f2c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:53.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f5547fff640 1 -- 172.21.15.23:0/3145324833 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555009a9c0 con 0x5650820b35e0 2024-03-24T10:40:53.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f5547fff640 1 -- 172.21.15.23:0/3145324833 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f555009a7e0 con 0x5650820b35e0 2024-03-24T10:40:53.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.368+0000 7f5547fff640 1 -- 172.21.15.23:0/3145324833 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55500cc140 con 0x5650820b35e0 2024-03-24T10:40:53.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.369+0000 7f55679f2740 1 -- 172.21.15.23:0/3145324833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b35e0 msgr2=0x5650820bc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:53.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.369+0000 7f55679f2740 1 --2- 172.21.15.23:0/3145324833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b35e0 0x5650820bc450 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f55500cbd10 tx=0x7f55500d0940 comp rx=0 tx=0).stop 2024-03-24T10:40:53.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.369+0000 7f55679f2740 1 -- 172.21.15.23:0/3145324833 shutdown_connections 2024-03-24T10:40:53.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.369+0000 7f55679f2740 1 --2- 172.21.15.23:0/3145324833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 0x5650820b4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.369+0000 7f55679f2740 1 --2- 172.21.15.23:0/3145324833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b3f00 0x5650820cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.369+0000 7f55679f2740 1 --2- 172.21.15.23:0/3145324833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b35e0 0x5650820bc450 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.369+0000 7f55679f2740 1 -- 172.21.15.23:0/3145324833 >> 172.21.15.23:0/3145324833 conn(0x5650820adf00 msgr2=0x5650820ae2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:53.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.369+0000 7f55679f2740 1 -- 172.21.15.23:0/3145324833 shutdown_connections 2024-03-24T10:40:53.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.369+0000 7f55679f2740 1 -- 172.21.15.23:0/3145324833 wait complete. 2024-03-24T10:40:53.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.369+0000 7f55679f2740 1 -- start start 2024-03-24T10:40:53.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.369+0000 7f55679f2740 1 Processor -- start 2024-03-24T10:40:53.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f55679f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b35e0 0x5650820c8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f55679f2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b3f00 0x5650820b6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f55679f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 0x5650820b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f55679f2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5650820b6ff0 con 0x5650820b3f00 2024-03-24T10:40:53.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f55679f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5650820b7160 con 0x5650820bccd0 2024-03-24T10:40:53.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f55679f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5650820b7320 con 0x5650820b35e0 2024-03-24T10:40:53.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f55679f2740 10 client.0 _ll_register_callbacks cb 0x7f5558037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:40:53.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f555e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b35e0 0x5650820c8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f5557fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b3f00 0x5650820b6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f555e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b35e0 0x5650820c8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:45362/0 (socket says 172.21.15.23:45362) 2024-03-24T10:40:53.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f555e59c640 1 -- 172.21.15.23:0/1626330006 learned_addr learned my addr 172.21.15.23:0/1626330006 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:40:53.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f555ed9d640 1 --2- 172.21.15.23:0/1626330006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 0x5650820b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f5557fff640 1 -- 172.21.15.23:0/1626330006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b35e0 msgr2=0x5650820c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f5557fff640 1 --2- 172.21.15.23:0/1626330006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b35e0 0x5650820c8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f5557fff640 1 -- 172.21.15.23:0/1626330006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 msgr2=0x5650820b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:40:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f5557fff640 1 --2- 172.21.15.23:0/1626330006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 0x5650820b6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f5557fff640 1 -- 172.21.15.23:0/1626330006 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55601a4ac0 con 0x5650820b3f00 2024-03-24T10:40:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f555e59c640 1 --2- 172.21.15.23:0/1626330006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b35e0 0x5650820c8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:40:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f5557fff640 1 --2- 172.21.15.23:0/1626330006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b3f00 0x5650820b6570 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f55480197d0 tx=0x7f5548009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6025fec73e9299 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:40:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f5545ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:40:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f5545ffb640 1 -- 172.21.15.23:0/1626330006 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5548011dc0 con 0x5650820b3f00 2024-03-24T10:40:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f5545ffb640 1 -- 172.21.15.23:0/1626330006 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f5548003910 con 0x5650820b3f00 2024-03-24T10:40:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.370+0000 7f55679f2740 1 -- 172.21.15.23:0/1626330006 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5650820b9b50 con 0x5650820b3f00 2024-03-24T10:40:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.371+0000 7f5545ffb640 1 -- 172.21.15.23:0/1626330006 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5548019600 con 0x5650820b3f00 2024-03-24T10:40:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.371+0000 7f55679f2740 10 client.15798 fetch_fsmap learned FSMap version 23 2024-03-24T10:40:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.371+0000 7f55679f2740 1 -- 172.21.15.23:0/1626330006 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x565081fda5a0 con 0x5650820b3f00 2024-03-24T10:40:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.371+0000 7f5545ffb640 1 -- 172.21.15.23:0/1626330006 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5650820b9b50 con 0x5650820b3f00 2024-03-24T10:40:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.371+0000 7f5545ffb640 1 -- 172.21.15.23:0/1626330006 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f5548004490 con 0x5650820b3f00 2024-03-24T10:40:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.371+0000 7f55679f2740 10 client.15798 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:40:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.371+0000 7f55679f2740 10 client.15798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:40:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.371+0000 7f55679f2740 20 client.15798 populate_metadata read hostname 'smithi023' 2024-03-24T10:40:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.371+0000 7f55679f2740 1 --2- 172.21.15.23:0/1626330006 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650821375b0 0x5650821399d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:40:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.371+0000 7f55679f2740 4 client.15798 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:40:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.371+0000 7f55679f2740 1 -- 172.21.15.23:0/1626330006 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f5548009e40 con 0x5650821375b0 2024-03-24T10:40:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.371+0000 7f555e59c640 1 --2- 172.21.15.23:0/1626330006 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650821375b0 0x5650821399d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:40:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.372+0000 7f5545ffb640 10 client.15798 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.372+0000 7f555e59c640 1 --2- 172.21.15.23:0/1626330006 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650821375b0 0x5650821399d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cd4b969505f4648 server_cookie=d6906ce8ed7e22ee in_seq=0 out_seq=0 2024-03-24T10:40:53.380 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.377+0000 7f5545ffb640 1 -- 172.21.15.23:0/1626330006 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f5548009e40 con 0x5650821375b0 2024-03-24T10:40:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.377+0000 7f5545ffb640 10 client.15798 handle_command_reply: tid=0 2024-03-24T10:40:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.377+0000 7f55679f2740 1 client.15798 shutdown 2024-03-24T10:40:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.377+0000 7f55679f2740 1 -- 172.21.15.23:0/1626330006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b3f00 msgr2=0x5650820b6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.377+0000 7f55679f2740 1 --2- 172.21.15.23:0/1626330006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b3f00 0x5650820b6570 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f55480197d0 tx=0x7f5548009780 comp rx=0 tx=0).stop 2024-03-24T10:40:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.377+0000 7f55679f2740 1 -- 172.21.15.23:0/1626330006 shutdown_connections 2024-03-24T10:40:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.377+0000 7f55679f2740 1 --2- 172.21.15.23:0/1626330006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650820bccd0 0x5650820b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.377+0000 7f55679f2740 1 --2- 172.21.15.23:0/1626330006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650820b3f00 0x5650820b6570 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.377+0000 7f55679f2740 1 --2- 172.21.15.23:0/1626330006 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650821375b0 0x5650821399d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.377+0000 7f55679f2740 1 --2- 172.21.15.23:0/1626330006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650820b35e0 0x5650820c8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.377+0000 7f55679f2740 1 -- 172.21.15.23:0/1626330006 >> 172.21.15.23:0/1626330006 conn(0x5650820adf00 msgr2=0x5650820b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.377+0000 7f5545ffb640 0 client.15798 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:40:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f55679f2740 1 -- 172.21.15.23:0/1626330006 shutdown_connections 2024-03-24T10:40:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f55679f2740 1 -- 172.21.15.23:0/1626330006 wait complete. 2024-03-24T10:40:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f55679f2740 20 client.15798 trim_cache size 0 max 16384 2024-03-24T10:40:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:40:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:40:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:40:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:40:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f5564d3e640 1 -- 172.21.15.23:0/2189226490 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f554c0a23f0 msgr2=0x7f554c0a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f5564d3e640 1 --2- 172.21.15.23:0/2189226490 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f554c0a23f0 0x7f554c0a48b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5558037040 tx=0x7f555802e040 comp rx=0 tx=0).stop 2024-03-24T10:40:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f5564d3e640 1 -- 172.21.15.23:0/2189226490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5560100860 msgr2=0x7f5560197d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:40:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f5564d3e640 1 --2- 172.21.15.23:0/2189226490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5560100860 0x7f5560197d70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f555000e9f0 tx=0x7f555000eeb0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f5564d3e640 1 -- 172.21.15.23:0/2189226490 shutdown_connections 2024-03-24T10:40:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f5564d3e640 1 --2- 172.21.15.23:0/2189226490 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f554c0a23f0 0x7f554c0a48b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f5564d3e640 1 --2- 172.21.15.23:0/2189226490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55601055b0 0x7f556019c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f5564d3e640 1 --2- 172.21.15.23:0/2189226490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5560101220 0x7f55601982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f5564d3e640 1 --2- 172.21.15.23:0/2189226490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5560100860 0x7f5560197d70 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:40:53.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f5564d3e640 1 -- 172.21.15.23:0/2189226490 >> 172.21.15.23:0/2189226490 conn(0x7f5560077460 msgr2=0x7f55600fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:40:53.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f5564d3e640 1 -- 172.21.15.23:0/2189226490 shutdown_connections 2024-03-24T10:40:53.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:40:53.378+0000 7f5564d3e640 1 -- 172.21.15.23:0/2189226490 wait complete. 2024-03-24T10:40:54.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:54 smithi117 ceph-mon[36368]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-24T10:40:54.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:54 smithi116 ceph-mon[36248]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-24T10:40:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:54 smithi023 ceph-mon[32466]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-24T10:40:56.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:56 smithi117 ceph-mon[36368]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-24T10:40:56.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:56 smithi116 ceph-mon[36248]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-24T10:40:56.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:56 smithi023 ceph-mon[32466]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-24T10:40:58.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:40:58 smithi117 ceph-mon[36368]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-24T10:40:58.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:40:58 smithi116 ceph-mon[36248]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-24T10:40:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:40:58 smithi023 ceph-mon[32466]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-24T10:41:00.487 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:41:00 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:41:00.208+0000 7f5c2c52a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:41:00.487 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:41:00 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:41:00.208+0000 7f5c2c52a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:41:00.487 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:41:00 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:41:00.208+0000 7f5c2c52a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:41:00.487 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:41:00 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:41:00.208+0000 7f5c2c52a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:41:00.487 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:41:00 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:41:00.208+0000 7f5c2c52a700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T10:41:00.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:00 smithi117 ceph-mon[36368]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-24T10:41:00.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:00 smithi116 ceph-mon[36248]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-24T10:41:00.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:00 smithi023 ceph-mon[32466]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-24T10:41:02.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:02 smithi117 ceph-mon[36368]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-24T10:41:02.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:02 smithi116 ceph-mon[36248]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-24T10:41:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:02 smithi023 ceph-mon[32466]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-24T10:41:03.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:03 smithi116 ceph-mon[36248]: mgrmap e24: x(active, since 57s), standbys: y, z 2024-03-24T10:41:03.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:03 smithi023 ceph-mon[32466]: mgrmap e24: x(active, since 57s), standbys: y, z 2024-03-24T10:41:04.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:03 smithi117 ceph-mon[36368]: mgrmap e24: x(active, since 57s), standbys: y, z 2024-03-24T10:41:04.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:04 smithi116 ceph-mon[36248]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-24T10:41:04.819 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:04 smithi023 ceph-mon[32466]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-24T10:41:05.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:04 smithi117 ceph-mon[36368]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-24T10:41:06.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.468+0000 7fa921fc9640 1 -- 172.21.15.23:0/1506939825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa91c100860 msgr2=0x7fa91c100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.468+0000 7fa921fc9640 1 --2- 172.21.15.23:0/1506939825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa91c100860 0x7fa91c100ce0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa910009900 tx=0x7fa91002bc30 comp rx=0 tx=0).stop 2024-03-24T10:41:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.468+0000 7fa921fc9640 1 -- 172.21.15.23:0/1506939825 shutdown_connections 2024-03-24T10:41:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.468+0000 7fa921fc9640 1 --2- 172.21.15.23:0/1506939825 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa91c101220 0x7fa91c10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.468+0000 7fa921fc9640 1 --2- 172.21.15.23:0/1506939825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa91c100860 0x7fa91c100ce0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.468+0000 7fa921fc9640 1 --2- 172.21.15.23:0/1506939825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa91c1055b0 0x7fa91c105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.468+0000 7fa921fc9640 1 -- 172.21.15.23:0/1506939825 >> 172.21.15.23:0/1506939825 conn(0x7fa91c077460 msgr2=0x7fa91c077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.468+0000 7fa921fc9640 1 -- 172.21.15.23:0/1506939825 shutdown_connections 2024-03-24T10:41:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.468+0000 7fa921fc9640 1 -- 172.21.15.23:0/1506939825 wait complete. 2024-03-24T10:41:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.468+0000 7fa921fc9640 1 Processor -- start 2024-03-24T10:41:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa921fc9640 1 -- start start 2024-03-24T10:41:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa921fc9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa91c100860 0x7fa91c197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa921fc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa91c101220 0x7fa91c198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa921fc9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa91c1055b0 0x7fa91c19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa921fc9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c19cc80 con 0x7fa91c100860 2024-03-24T10:41:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa921fc9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c19cdf0 con 0x7fa91c101220 2024-03-24T10:41:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa921fc9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c19cfb0 con 0x7fa91c1055b0 2024-03-24T10:41:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa91b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa91c100860 0x7fa91c197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa91b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa91c100860 0x7fa91c197dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45090/0 (socket says 172.21.15.23:45090) 2024-03-24T10:41:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa91affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa91c101220 0x7fa91c198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa91b7fe640 1 -- 172.21.15.23:0/1425010360 learned_addr learned my addr 172.21.15.23:0/1425010360 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa91bfff640 1 --2- 172.21.15.23:0/1425010360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa91c1055b0 0x7fa91c19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa91affd640 1 -- 172.21.15.23:0/1425010360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa91c1055b0 msgr2=0x7fa91c19c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa91affd640 1 --2- 172.21.15.23:0/1425010360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa91c1055b0 0x7fa91c19c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa91affd640 1 -- 172.21.15.23:0/1425010360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa91c100860 msgr2=0x7fa91c197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa91affd640 1 --2- 172.21.15.23:0/1425010360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa91c100860 0x7fa91c197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa91affd640 1 -- 172.21.15.23:0/1425010360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90c009490 con 0x7fa91c101220 2024-03-24T10:41:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.469+0000 7fa91b7fe640 1 --2- 172.21.15.23:0/1425010360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa91c100860 0x7fa91c197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.470+0000 7fa91affd640 1 --2- 172.21.15.23:0/1425010360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa91c101220 0x7fa91c198300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa910036040 tx=0x7fa91002c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.470+0000 7fa918ff9640 1 -- 172.21.15.23:0/1425010360 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa910004350 con 0x7fa91c101220 2024-03-24T10:41:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.470+0000 7fa921fc9640 1 -- 172.21.15.23:0/1425010360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa910009560 con 0x7fa91c101220 2024-03-24T10:41:06.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.470+0000 7fa918ff9640 1 -- 172.21.15.23:0/1425010360 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9100044b0 con 0x7fa91c101220 2024-03-24T10:41:06.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.470+0000 7fa918ff9640 1 -- 172.21.15.23:0/1425010360 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa910003cc0 con 0x7fa91c101220 2024-03-24T10:41:06.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.470+0000 7fa921fc9640 1 -- 172.21.15.23:0/1425010360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa91c1a4930 con 0x7fa91c101220 2024-03-24T10:41:06.479 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.472+0000 7fa918ff9640 1 -- 172.21.15.23:0/1425010360 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 137040+0+0 (secure 0 0 0) 0x7fa910005560 con 0x7fa91c101220 2024-03-24T10:41:06.479 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.472+0000 7fa924bad740 1 Processor -- start 2024-03-24T10:41:06.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.472+0000 7fa924bad740 1 -- start start 2024-03-24T10:41:06.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.472+0000 7fa918ff9640 1 --2- 172.21.15.23:0/1425010360 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa9040a2480 0x7fa9040a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.472+0000 7fa924bad740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f7e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.472+0000 7fa924bad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f7ea20 0x55cd80f8bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.472+0000 7fa924bad740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f8c0e0 0x55cd80f8e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.472+0000 7fa924bad740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55cd80f391b0 con 0x55cd80f8c0e0 2024-03-24T10:41:06.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.472+0000 7fa924bad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cd80eebf80 con 0x55cd80f7ea20 2024-03-24T10:41:06.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.472+0000 7fa924bad740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cd80ee94b0 con 0x55cd80f7e100 2024-03-24T10:41:06.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.472+0000 7fa91bfff640 1 --2- 172.21.15.23:0/1425010360 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa9040a2480 0x7fa9040a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa918ff9640 1 -- 172.21.15.23:0/1425010360 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fa9100f01e0 con 0x7fa91c101220 2024-03-24T10:41:06.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa91affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f7ea20 0x55cd80f8bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa91b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f7e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa91b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f7e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:56932/0 (socket says 172.21.15.23:56932) 2024-03-24T10:41:06.480 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa91affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f7ea20 0x55cd80f8bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:59686/0 (socket says 172.21.15.23:59686) 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa91b7fe640 1 -- 172.21.15.23:0/650234964 learned_addr learned my addr 172.21.15.23:0/650234964 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa91bfff640 1 --2- 172.21.15.23:0/650234964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f8c0e0 0x55cd80f8e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa91affd640 1 -- 172.21.15.23:0/650234964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 msgr2=0x55cd80f7e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa91affd640 1 --2- 172.21.15.23:0/650234964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f7e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa91affd640 1 -- 172.21.15.23:0/650234964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f8c0e0 msgr2=0x55cd80f8e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa91affd640 1 --2- 172.21.15.23:0/650234964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f8c0e0 0x55cd80f8e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa91affd640 1 -- 172.21.15.23:0/650234964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd80e7a810 con 0x55cd80f7ea20 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa91b7fe640 1 --2- 172.21.15.23:0/650234964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f7e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa91bfff640 1 --2- 172.21.15.23:0/1425010360 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa9040a2480 0x7fa9040a4940 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa914005fd0 tx=0x7fa914013b60 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa91affd640 1 --2- 172.21.15.23:0/650234964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f7ea20 0x55cd80f8bba0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa9100ed840 tx=0x7fa91002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef6b64c7a9a0ffe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa901ffb640 1 -- 172.21.15.23:0/650234964 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9100bcae0 con 0x55cd80f7ea20 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa901ffb640 1 -- 172.21.15.23:0/650234964 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9100c0c30 con 0x55cd80f7ea20 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.473+0000 7fa901ffb640 1 -- 172.21.15.23:0/650234964 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9100c4df0 con 0x55cd80f7ea20 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.474+0000 7fa924bad740 1 -- 172.21.15.23:0/650234964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f7ea20 msgr2=0x55cd80f8bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:06.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.474+0000 7fa924bad740 1 --2- 172.21.15.23:0/650234964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f7ea20 0x55cd80f8bba0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa9100ed840 tx=0x7fa91002bc30 comp rx=0 tx=0).stop 2024-03-24T10:41:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.474+0000 7fa924bad740 1 -- 172.21.15.23:0/650234964 shutdown_connections 2024-03-24T10:41:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.474+0000 7fa924bad740 1 --2- 172.21.15.23:0/650234964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f8c0e0 0x55cd80f8e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.474+0000 7fa924bad740 1 --2- 172.21.15.23:0/650234964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f7ea20 0x55cd80f8bba0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.474+0000 7fa924bad740 1 --2- 172.21.15.23:0/650234964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f7e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.474+0000 7fa924bad740 1 -- 172.21.15.23:0/650234964 >> 172.21.15.23:0/650234964 conn(0x55cd80f78f00 msgr2=0x55cd80f792e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.474+0000 7fa924bad740 1 -- 172.21.15.23:0/650234964 shutdown_connections 2024-03-24T10:41:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.474+0000 7fa924bad740 1 -- 172.21.15.23:0/650234964 wait complete. 2024-03-24T10:41:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.474+0000 7fa924bad740 1 -- start start 2024-03-24T10:41:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.474+0000 7fa924bad740 1 Processor -- start 2024-03-24T10:41:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa924bad740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f89160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa924bad740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f7ea20 0x55cd80f896a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa924bad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f8c0e0 0x55cd80f89be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa924bad740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55cd80ee4c90 con 0x55cd80f7ea20 2024-03-24T10:41:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa924bad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cd80e89c50 con 0x55cd80f8c0e0 2024-03-24T10:41:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa924bad740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cd80bb8250 con 0x55cd80f7e100 2024-03-24T10:41:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa924bad740 10 client.0 _ll_register_callbacks cb 0x55cd80ed8290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa91affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f7ea20 0x55cd80f896a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa91b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f89160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa91affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f7ea20 0x55cd80f896a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45114/0 (socket says 172.21.15.23:45114) 2024-03-24T10:41:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa91affd640 1 -- 172.21.15.23:0/2596049744 learned_addr learned my addr 172.21.15.23:0/2596049744 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa91bfff640 1 --2- 172.21.15.23:0/2596049744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f8c0e0 0x55cd80f89be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa91affd640 1 -- 172.21.15.23:0/2596049744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 msgr2=0x55cd80f89160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa91affd640 1 --2- 172.21.15.23:0/2596049744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f89160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa91affd640 1 -- 172.21.15.23:0/2596049744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f8c0e0 msgr2=0x55cd80f89be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa91affd640 1 --2- 172.21.15.23:0/2596049744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f8c0e0 0x55cd80f89be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa91affd640 1 -- 172.21.15.23:0/2596049744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa91c1a4930 con 0x55cd80f7ea20 2024-03-24T10:41:06.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa91b7fe640 1 --2- 172.21.15.23:0/2596049744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f89160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa91affd640 1 --2- 172.21.15.23:0/2596049744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f7ea20 0x55cd80f896a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa910006010 tx=0x7fa91002c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf7a27357b7d851 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa8faffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa8faffd640 1 -- 172.21.15.23:0/2596049744 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9100b9330 con 0x55cd80f7ea20 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa8faffd640 1 -- 172.21.15.23:0/2596049744 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fa9100bcae0 con 0x55cd80f7ea20 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.475+0000 7fa924bad740 1 -- 172.21.15.23:0/2596049744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cd80f09790 con 0x55cd80f7ea20 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.476+0000 7fa8faffd640 1 -- 172.21.15.23:0/2596049744 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa9100b9150 con 0x55cd80f7ea20 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.476+0000 7fa924bad740 10 client.15816 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.476+0000 7fa924bad740 1 -- 172.21.15.23:0/2596049744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55cd80ee3310 con 0x55cd80f7ea20 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.476+0000 7fa8faffd640 1 -- 172.21.15.23:0/2596049744 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9100db030 con 0x55cd80f7ea20 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.476+0000 7fa8faffd640 1 -- 172.21.15.23:0/2596049744 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fa91003c070 con 0x55cd80f7ea20 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.476+0000 7fa924bad740 10 client.15816 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.476+0000 7fa924bad740 10 client.15816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.476+0000 7fa924bad740 20 client.15816 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.476+0000 7fa924bad740 1 --2- 172.21.15.23:0/2596049744 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55cd80f9b990 0x55cd80f9dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.476+0000 7fa924bad740 4 client.15816 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.476+0000 7fa924bad740 1 -- 172.21.15.23:0/2596049744 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd80be4600 con 0x55cd80f9b990 2024-03-24T10:41:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.476+0000 7fa91b7fe640 1 --2- 172.21.15.23:0/2596049744 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55cd80f9b990 0x55cd80f9dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.476+0000 7fa91b7fe640 1 --2- 172.21.15.23:0/2596049744 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55cd80f9b990 0x55cd80f9dda0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b5d0048800b235 server_cookie=4657379b76abf433 in_seq=0 out_seq=0 2024-03-24T10:41:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.476+0000 7fa8faffd640 10 client.15816 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.477+0000 7fa8faffd640 1 -- 172.21.15.23:0/2596049744 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cd80be4600 con 0x55cd80f9b990 2024-03-24T10:41:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.477+0000 7fa8faffd640 10 client.15816 handle_command_reply: tid=0 2024-03-24T10:41:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.477+0000 7fa924bad740 1 client.15816 shutdown 2024-03-24T10:41:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.478+0000 7fa924bad740 1 -- 172.21.15.23:0/2596049744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f7ea20 msgr2=0x55cd80f896a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.478+0000 7fa924bad740 1 --2- 172.21.15.23:0/2596049744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f7ea20 0x55cd80f896a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa910006010 tx=0x7fa91002c770 comp rx=0 tx=0).stop 2024-03-24T10:41:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.478+0000 7fa924bad740 1 -- 172.21.15.23:0/2596049744 shutdown_connections 2024-03-24T10:41:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.478+0000 7fa924bad740 1 --2- 172.21.15.23:0/2596049744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f8c0e0 0x55cd80f89be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.478+0000 7fa924bad740 1 --2- 172.21.15.23:0/2596049744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f7ea20 0x55cd80f896a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.478+0000 7fa924bad740 1 --2- 172.21.15.23:0/2596049744 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55cd80f9b990 0x55cd80f9dda0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.478+0000 7fa924bad740 1 --2- 172.21.15.23:0/2596049744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f89160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.478+0000 7fa8faffd640 0 client.15816 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.478+0000 7fa924bad740 1 -- 172.21.15.23:0/2596049744 >> 172.21.15.23:0/2596049744 conn(0x55cd80f78f00 msgr2=0x55cd80f7ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.478+0000 7fa924bad740 1 -- 172.21.15.23:0/2596049744 shutdown_connections 2024-03-24T10:41:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.478+0000 7fa924bad740 1 -- 172.21.15.23:0/2596049744 wait complete. 2024-03-24T10:41:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.478+0000 7fa924bad740 20 client.15816 trim_cache size 0 max 16384 2024-03-24T10:41:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa924bad740 1 Processor -- start 2024-03-24T10:41:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa924bad740 1 -- start start 2024-03-24T10:41:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa924bad740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f7e100 0x55cd80f8b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa924bad740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7ea20 0x55cd80f83b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa924bad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f8c0e0 0x55cd80f97540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa924bad740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55cd80e6a660 con 0x55cd80f7e100 2024-03-24T10:41:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa924bad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cd80f746c0 con 0x55cd80f8c0e0 2024-03-24T10:41:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa924bad740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cd80f9e3f0 con 0x55cd80f7ea20 2024-03-24T10:41:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa91b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f7e100 0x55cd80f8b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa91b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f7e100 0x55cd80f8b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45130/0 (socket says 172.21.15.23:45130) 2024-03-24T10:41:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa91b7fe640 1 -- 172.21.15.23:0/4257681748 learned_addr learned my addr 172.21.15.23:0/4257681748 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa91bfff640 1 --2- 172.21.15.23:0/4257681748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f8c0e0 0x55cd80f97540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa91affd640 1 --2- 172.21.15.23:0/4257681748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7ea20 0x55cd80f83b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa91b7fe640 1 -- 172.21.15.23:0/4257681748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7ea20 msgr2=0x55cd80f83b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa91b7fe640 1 --2- 172.21.15.23:0/4257681748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7ea20 0x55cd80f83b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa91b7fe640 1 -- 172.21.15.23:0/4257681748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f8c0e0 msgr2=0x55cd80f97540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa91b7fe640 1 --2- 172.21.15.23:0/4257681748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f8c0e0 0x55cd80f97540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.493+0000 7fa91b7fe640 1 -- 172.21.15.23:0/4257681748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd80e7f810 con 0x55cd80f7e100 2024-03-24T10:41:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.494+0000 7fa91b7fe640 1 --2- 172.21.15.23:0/4257681748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f7e100 0x55cd80f8b890 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa90c000c00 tx=0x7fa90c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b6ea5d7ccdb3f4b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.494+0000 7fa9017fa640 1 -- 172.21.15.23:0/4257681748 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c017040 con 0x55cd80f7e100 2024-03-24T10:41:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.494+0000 7fa9017fa640 1 -- 172.21.15.23:0/4257681748 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa90c018040 con 0x55cd80f7e100 2024-03-24T10:41:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.494+0000 7fa9017fa640 1 -- 172.21.15.23:0/4257681748 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c022170 con 0x55cd80f7e100 2024-03-24T10:41:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.494+0000 7fa924bad740 1 -- 172.21.15.23:0/4257681748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f7e100 msgr2=0x55cd80f8b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.494+0000 7fa924bad740 1 --2- 172.21.15.23:0/4257681748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f7e100 0x55cd80f8b890 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa90c000c00 tx=0x7fa90c005f90 comp rx=0 tx=0).stop 2024-03-24T10:41:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.494+0000 7fa924bad740 1 -- 172.21.15.23:0/4257681748 shutdown_connections 2024-03-24T10:41:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.494+0000 7fa924bad740 1 --2- 172.21.15.23:0/4257681748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f8c0e0 0x55cd80f97540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.494+0000 7fa924bad740 1 --2- 172.21.15.23:0/4257681748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7ea20 0x55cd80f83b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.494+0000 7fa924bad740 1 --2- 172.21.15.23:0/4257681748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f7e100 0x55cd80f8b890 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.494+0000 7fa924bad740 1 -- 172.21.15.23:0/4257681748 >> 172.21.15.23:0/4257681748 conn(0x55cd80f78f00 msgr2=0x55cd80f792e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.494+0000 7fa924bad740 1 -- 172.21.15.23:0/4257681748 shutdown_connections 2024-03-24T10:41:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.494+0000 7fa924bad740 1 -- 172.21.15.23:0/4257681748 wait complete. 2024-03-24T10:41:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.495+0000 7fa924bad740 1 -- start start 2024-03-24T10:41:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.495+0000 7fa924bad740 1 Processor -- start 2024-03-24T10:41:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.495+0000 7fa924bad740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f85bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.495+0000 7fa924bad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f7ea20 0x55cd80f86130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.495+0000 7fa924bad740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f8c0e0 0x55cd80f86670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.495+0000 7fa924bad740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55cd80f94320 con 0x55cd80f8c0e0 2024-03-24T10:41:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.495+0000 7fa924bad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cd81000cb0 con 0x55cd80f7ea20 2024-03-24T10:41:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.495+0000 7fa924bad740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cd81000df0 con 0x55cd80f7e100 2024-03-24T10:41:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.495+0000 7fa924bad740 10 client.0 _ll_register_callbacks cb 0x7fa9100ef070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.495+0000 7fa91bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f8c0e0 0x55cd80f86670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.495+0000 7fa91b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f85bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.495+0000 7fa91affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f7ea20 0x55cd80f86130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.495+0000 7fa91bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f8c0e0 0x55cd80f86670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45146/0 (socket says 172.21.15.23:45146) 2024-03-24T10:41:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.495+0000 7fa91bfff640 1 -- 172.21.15.23:0/4201787320 learned_addr learned my addr 172.21.15.23:0/4201787320 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa91affd640 1 -- 172.21.15.23:0/4201787320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 msgr2=0x55cd80f85bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa91affd640 1 --2- 172.21.15.23:0/4201787320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f85bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa91affd640 1 -- 172.21.15.23:0/4201787320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f8c0e0 msgr2=0x55cd80f86670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa91affd640 1 --2- 172.21.15.23:0/4201787320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f8c0e0 0x55cd80f86670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa91affd640 1 -- 172.21.15.23:0/4201787320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa90c012d90 con 0x55cd80f7ea20 2024-03-24T10:41:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa91bfff640 1 --2- 172.21.15.23:0/4201787320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f8c0e0 0x55cd80f86670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa91b7fe640 1 --2- 172.21.15.23:0/4201787320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f85bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa91affd640 1 --2- 172.21.15.23:0/4201787320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f7ea20 0x55cd80f86130 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa9100bc7e0 tx=0x7fa91002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=778f61c86e7da057 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa8faffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:41:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa8faffd640 1 -- 172.21.15.23:0/4201787320 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9100b8830 con 0x55cd80f7ea20 2024-03-24T10:41:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa8faffd640 1 -- 172.21.15.23:0/4201787320 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fa9100bcae0 con 0x55cd80f7ea20 2024-03-24T10:41:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa924bad740 1 -- 172.21.15.23:0/4201787320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9100c0c30 con 0x55cd80f7ea20 2024-03-24T10:41:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa8faffd640 1 -- 172.21.15.23:0/4201787320 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa9100c5280 con 0x55cd80f7ea20 2024-03-24T10:41:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa924bad740 10 client.25697 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.496+0000 7fa924bad740 1 -- 172.21.15.23:0/4201787320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55cd80ea55a0 con 0x55cd80f7ea20 2024-03-24T10:41:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.497+0000 7fa8faffd640 1 -- 172.21.15.23:0/4201787320 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9100c0c30 con 0x55cd80f7ea20 2024-03-24T10:41:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.497+0000 7fa8faffd640 1 -- 172.21.15.23:0/4201787320 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fa91003c070 con 0x55cd80f7ea20 2024-03-24T10:41:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.497+0000 7fa924bad740 10 client.25697 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.497+0000 7fa924bad740 10 client.25697 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.497+0000 7fa924bad740 20 client.25697 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.497+0000 7fa924bad740 1 --2- 172.21.15.23:0/4201787320 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55cd81002080 0x55cd81004470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.497+0000 7fa924bad740 4 client.25697 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:41:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.497+0000 7fa924bad740 1 -- 172.21.15.23:0/4201787320 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cd80c351b0 con 0x55cd81002080 2024-03-24T10:41:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.497+0000 7fa91b7fe640 1 --2- 172.21.15.23:0/4201787320 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55cd81002080 0x55cd81004470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.497+0000 7fa8faffd640 10 client.25697 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.497+0000 7fa91b7fe640 1 --2- 172.21.15.23:0/4201787320 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55cd81002080 0x55cd81004470 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bd374229f8e8dd9 server_cookie=8008cfe184eeca8f in_seq=0 out_seq=0 2024-03-24T10:41:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.502+0000 7fa8faffd640 1 -- 172.21.15.23:0/4201787320 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cd80c351b0 con 0x55cd81002080 2024-03-24T10:41:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.502+0000 7fa8faffd640 10 client.25697 handle_command_reply: tid=0 2024-03-24T10:41:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.502+0000 7fa924bad740 1 client.25697 shutdown 2024-03-24T10:41:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa924bad740 1 -- 172.21.15.23:0/4201787320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f7ea20 msgr2=0x55cd80f86130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa924bad740 1 --2- 172.21.15.23:0/4201787320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f7ea20 0x55cd80f86130 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa9100bc7e0 tx=0x7fa91002bc30 comp rx=0 tx=0).stop 2024-03-24T10:41:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa924bad740 1 -- 172.21.15.23:0/4201787320 shutdown_connections 2024-03-24T10:41:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa924bad740 1 --2- 172.21.15.23:0/4201787320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cd80f8c0e0 0x55cd80f86670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa924bad740 1 --2- 172.21.15.23:0/4201787320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cd80f7ea20 0x55cd80f86130 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa924bad740 1 --2- 172.21.15.23:0/4201787320 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55cd81002080 0x55cd81004470 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa924bad740 1 --2- 172.21.15.23:0/4201787320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd80f7e100 0x55cd80f85bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa8faffd640 0 client.25697 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa924bad740 1 -- 172.21.15.23:0/4201787320 >> 172.21.15.23:0/4201787320 conn(0x55cd80f78f00 msgr2=0x55cd80f77610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa924bad740 1 -- 172.21.15.23:0/4201787320 shutdown_connections 2024-03-24T10:41:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa924bad740 1 -- 172.21.15.23:0/4201787320 wait complete. 2024-03-24T10:41:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa924bad740 20 client.25697 trim_cache size 0 max 16384 2024-03-24T10:41:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:41:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:41:06.506 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:41:06.506 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:41:06.506 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa921fc9640 1 -- 172.21.15.23:0/1425010360 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa9040a2480 msgr2=0x7fa9040a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa921fc9640 1 --2- 172.21.15.23:0/1425010360 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa9040a2480 0x7fa9040a4940 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa914005fd0 tx=0x7fa914013b60 comp rx=0 tx=0).stop 2024-03-24T10:41:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa921fc9640 1 -- 172.21.15.23:0/1425010360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa91c101220 msgr2=0x7fa91c198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa921fc9640 1 --2- 172.21.15.23:0/1425010360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa91c101220 0x7fa91c198300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa910036040 tx=0x7fa91002c680 comp rx=0 tx=0).stop 2024-03-24T10:41:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa921fc9640 1 -- 172.21.15.23:0/1425010360 shutdown_connections 2024-03-24T10:41:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa921fc9640 1 --2- 172.21.15.23:0/1425010360 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa9040a2480 0x7fa9040a4940 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa921fc9640 1 --2- 172.21.15.23:0/1425010360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa91c1055b0 0x7fa91c19c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa921fc9640 1 --2- 172.21.15.23:0/1425010360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa91c101220 0x7fa91c198300 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa921fc9640 1 --2- 172.21.15.23:0/1425010360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa91c100860 0x7fa91c197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa921fc9640 1 -- 172.21.15.23:0/1425010360 >> 172.21.15.23:0/1425010360 conn(0x7fa91c077460 msgr2=0x7fa91c0690d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.503+0000 7fa921fc9640 1 -- 172.21.15.23:0/1425010360 shutdown_connections 2024-03-24T10:41:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:06.504+0000 7fa921fc9640 1 -- 172.21.15.23:0/1425010360 wait complete. 2024-03-24T10:41:06.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:06 smithi116 ceph-mon[36248]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-24T10:41:06.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:06 smithi023 ceph-mon[32466]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-24T10:41:07.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:06 smithi117 ceph-mon[36368]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-24T10:41:08.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:08 smithi116 ceph-mon[36248]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-24T10:41:08.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:08 smithi023 ceph-mon[32466]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-24T10:41:09.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:08 smithi117 ceph-mon[36368]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-24T10:41:10.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:10 smithi116 ceph-mon[36248]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T10:41:10.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:10 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:41:10.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:10 smithi023 ceph-mon[32466]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T10:41:10.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:10 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:41:11.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:10 smithi117 ceph-mon[36368]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-24T10:41:11.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:10 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:41:11.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:11 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:41:11.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:11 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:41:11.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:11 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:11.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:11 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:41:11.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:11 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:41:12.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:11 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:41:12.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:11 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:41:12.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:11 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:12.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:11 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:41:12.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:11 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:41:12.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:11 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:41:12.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:11 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:41:12.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:11 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:41:12.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:11 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:41:12.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:11 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:41:12.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:12 smithi023 ceph-mon[32466]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T10:41:13.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:12 smithi117 ceph-mon[36368]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T10:41:13.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:12 smithi116 ceph-mon[36248]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T10:41:14.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:14 smithi023 ceph-mon[32466]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-24T10:41:15.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:14 smithi117 ceph-mon[36368]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-24T10:41:15.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:14 smithi116 ceph-mon[36248]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-24T10:41:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:16 smithi023 ceph-mon[32466]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-24T10:41:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:16 smithi117 ceph-mon[36368]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-24T10:41:17.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:16 smithi116 ceph-mon[36248]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-24T10:41:17.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:41:17.259 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:41:17.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.375+0000 7f8ac89e2640 1 -- 172.21.15.23:0/4080457384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ac40688f0 msgr2=0x7f8ac4068d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.375+0000 7f8ac89e2640 1 --2- 172.21.15.23:0/4080457384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ac40688f0 0x7f8ac4068d70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8009930 tx=0x7f8ab802bc30 comp rx=0 tx=0).stop 2024-03-24T10:41:17.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.375+0000 7f8ac89e2640 1 -- 172.21.15.23:0/4080457384 shutdown_connections 2024-03-24T10:41:17.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.375+0000 7f8ac89e2640 1 --2- 172.21.15.23:0/4080457384 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac40692b0 0x7f8ac410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.375+0000 7f8ac89e2640 1 --2- 172.21.15.23:0/4080457384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ac40688f0 0x7f8ac4068d70 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.375+0000 7f8ac89e2640 1 --2- 172.21.15.23:0/4080457384 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ac4107690 0x7f8ac4107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.375+0000 7f8ac89e2640 1 -- 172.21.15.23:0/4080457384 >> 172.21.15.23:0/4080457384 conn(0x7f8ac4077710 msgr2=0x7f8ac4077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.375+0000 7f8ac89e2640 1 -- 172.21.15.23:0/4080457384 shutdown_connections 2024-03-24T10:41:17.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.375+0000 7f8ac89e2640 1 -- 172.21.15.23:0/4080457384 wait complete. 2024-03-24T10:41:17.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.376+0000 7f8ac89e2640 1 Processor -- start 2024-03-24T10:41:17.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.376+0000 7f8ac89e2640 1 -- start start 2024-03-24T10:41:17.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.376+0000 7f8ac89e2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac40688f0 0x7f8ac4197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.376+0000 7f8ac89e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ac40692b0 0x7f8ac41982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.376+0000 7f8ac89e2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ac4107690 0x7f8ac419c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.376+0000 7f8ac89e2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac419cc40 con 0x7f8ac4107690 2024-03-24T10:41:17.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.376+0000 7f8ac89e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac419cdb0 con 0x7f8ac40692b0 2024-03-24T10:41:17.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.376+0000 7f8ac89e2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac419cf70 con 0x7f8ac40688f0 2024-03-24T10:41:17.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.376+0000 7f8ac1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ac40692b0 0x7f8ac41982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.376+0000 7f8ac259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac40688f0 0x7f8ac4197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.376+0000 7f8ac1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ac40692b0 0x7f8ac41982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:35332/0 (socket says 172.21.15.23:35332) 2024-03-24T10:41:17.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.376+0000 7f8ac1d9b640 1 -- 172.21.15.23:0/634752888 learned_addr learned my addr 172.21.15.23:0/634752888 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:17.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.376+0000 7f8ac259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac40688f0 0x7f8ac4197d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57652/0 (socket says 172.21.15.23:57652) 2024-03-24T10:41:17.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.376+0000 7f8ac2d9d640 1 --2- 172.21.15.23:0/634752888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ac4107690 0x7f8ac419c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.377+0000 7f8ac259c640 1 -- 172.21.15.23:0/634752888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ac40692b0 msgr2=0x7f8ac41982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.377+0000 7f8ac259c640 1 --2- 172.21.15.23:0/634752888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ac40692b0 0x7f8ac41982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.377+0000 7f8ac259c640 1 -- 172.21.15.23:0/634752888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ac4107690 msgr2=0x7f8ac419c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.377+0000 7f8ac259c640 1 --2- 172.21.15.23:0/634752888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ac4107690 0x7f8ac419c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.377+0000 7f8ac259c640 1 -- 172.21.15.23:0/634752888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8abc009840 con 0x7f8ac40688f0 2024-03-24T10:41:17.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.377+0000 7f8ac259c640 1 --2- 172.21.15.23:0/634752888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac40688f0 0x7f8ac4197d80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8ab400e9f0 tx=0x7f8ab400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.377+0000 7f8ab37fe640 1 -- 172.21.15.23:0/634752888 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4009640 con 0x7f8ac40688f0 2024-03-24T10:41:17.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.377+0000 7f8ab37fe640 1 -- 172.21.15.23:0/634752888 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ab400ccc0 con 0x7f8ac40688f0 2024-03-24T10:41:17.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.377+0000 7f8ac89e2640 1 -- 172.21.15.23:0/634752888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ab8009560 con 0x7f8ac40688f0 2024-03-24T10:41:17.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.377+0000 7f8ab37fe640 1 -- 172.21.15.23:0/634752888 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4018750 con 0x7f8ac40688f0 2024-03-24T10:41:17.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.377+0000 7f8ac89e2640 1 -- 172.21.15.23:0/634752888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ac41a48f0 con 0x7f8ac40688f0 2024-03-24T10:41:17.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.380+0000 7f8ab37fe640 1 -- 172.21.15.23:0/634752888 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 24) ==== 137040+0+0 (secure 0 0 0) 0x7f8ab40109a0 con 0x7f8ac40688f0 2024-03-24T10:41:17.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.380+0000 7f8acb5c1740 1 Processor -- start 2024-03-24T10:41:17.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.380+0000 7f8acb5c1740 1 -- start start 2024-03-24T10:41:17.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8acb5c1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17b8860 0x5632d17b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8acb5c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a63b0 0x5632d17a6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8acb5c1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a6d70 0x5632d17c0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8acb5c1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5632d16d0f30 con 0x5632d17b8860 2024-03-24T10:41:17.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8acb5c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5632d171a810 con 0x5632d17a63b0 2024-03-24T10:41:17.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8acb5c1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5632d1717d40 con 0x5632d17a6d70 2024-03-24T10:41:17.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8ab37fe640 1 --2- 172.21.15.23:0/634752888 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8aac0a2460 0x7f8aac0a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8ac2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17b8860 0x5632d17b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8ac1d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a6d70 0x5632d17c0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8ac2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17b8860 0x5632d17b8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53558/0 (socket says 172.21.15.23:53558) 2024-03-24T10:41:17.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8ac2d9d640 1 -- 172.21.15.23:0/1604781523 learned_addr learned my addr 172.21.15.23:0/1604781523 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:17.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8ac259c640 1 --2- 172.21.15.23:0/1604781523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a63b0 0x5632d17a6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8ab37fe640 1 -- 172.21.15.23:0/634752888 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f8ab4014070 con 0x7f8ac40688f0 2024-03-24T10:41:17.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8ac1d9b640 1 -- 172.21.15.23:0/1604781523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a63b0 msgr2=0x5632d17a6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8ac1d9b640 1 --2- 172.21.15.23:0/1604781523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a63b0 0x5632d17a6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8ac1d9b640 1 -- 172.21.15.23:0/1604781523 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17b8860 msgr2=0x5632d17b8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8ac1d9b640 1 --2- 172.21.15.23:0/1604781523 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17b8860 0x5632d17b8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8ac1d9b640 1 -- 172.21.15.23:0/1604781523 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632d16d4ea0 con 0x5632d17a6d70 2024-03-24T10:41:17.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.381+0000 7f8ac2d9d640 1 --2- 172.21.15.23:0/1604781523 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17b8860 0x5632d17b8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:17.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.382+0000 7f8ac2d9d640 1 --2- 172.21.15.23:0/634752888 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8aac0a2460 0x7f8aac0a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.382+0000 7f8ac1d9b640 1 --2- 172.21.15.23:0/1604781523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a6d70 0x5632d17c0730 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8009900 tx=0x7f8ab802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97de6412b381d2d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.382+0000 7f8ab0ff9640 1 -- 172.21.15.23:0/1604781523 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8004880 con 0x5632d17a6d70 2024-03-24T10:41:17.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.382+0000 7f8ab0ff9640 1 -- 172.21.15.23:0/1604781523 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ab80049e0 con 0x5632d17a6d70 2024-03-24T10:41:17.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.382+0000 7f8ab0ff9640 1 -- 172.21.15.23:0/1604781523 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8005050 con 0x5632d17a6d70 2024-03-24T10:41:17.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 -- 172.21.15.23:0/1604781523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a6d70 msgr2=0x5632d17c0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/1604781523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a6d70 0x5632d17c0730 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8009900 tx=0x7f8ab802c650 comp rx=0 tx=0).stop 2024-03-24T10:41:17.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8ac2d9d640 1 --2- 172.21.15.23:0/634752888 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8aac0a2460 0x7f8aac0a4920 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x5632d17b4e90 tx=0x7f8abc004a40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 -- 172.21.15.23:0/1604781523 shutdown_connections 2024-03-24T10:41:17.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/1604781523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a6d70 0x5632d17c0730 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/1604781523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a63b0 0x5632d17a6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/1604781523 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17b8860 0x5632d17b8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 -- 172.21.15.23:0/1604781523 >> 172.21.15.23:0/1604781523 conn(0x5632d17a8e70 msgr2=0x5632d17a9250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 -- 172.21.15.23:0/1604781523 shutdown_connections 2024-03-24T10:41:17.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 -- 172.21.15.23:0/1604781523 wait complete. 2024-03-24T10:41:17.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 -- start start 2024-03-24T10:41:17.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 Processor -- start 2024-03-24T10:41:17.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a63b0 0x5632d17bdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17a6d70 0x5632d17be180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17b8860 0x5632d17be6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5632d1713520 con 0x5632d17a6d70 2024-03-24T10:41:17.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5632d13e6430 con 0x5632d17b8860 2024-03-24T10:41:17.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5632d13a2e00 con 0x5632d17a63b0 2024-03-24T10:41:17.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.383+0000 7f8acb5c1740 10 client.0 _ll_register_callbacks cb 0x5632d16b1d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:17.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8ac1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17a6d70 0x5632d17be180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8ac259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a63b0 0x5632d17bdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8ac1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17a6d70 0x5632d17be180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53564/0 (socket says 172.21.15.23:53564) 2024-03-24T10:41:17.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8ac1d9b640 1 -- 172.21.15.23:0/1967485732 learned_addr learned my addr 172.21.15.23:0/1967485732 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:17.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8ac259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a63b0 0x5632d17bdc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57656/0 (socket says 172.21.15.23:57656) 2024-03-24T10:41:17.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8ac2d9d640 1 --2- 172.21.15.23:0/1967485732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17b8860 0x5632d17be6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8ac259c640 1 -- 172.21.15.23:0/1967485732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17b8860 msgr2=0x5632d17be6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8ac259c640 1 --2- 172.21.15.23:0/1967485732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17b8860 0x5632d17be6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8ac259c640 1 -- 172.21.15.23:0/1967485732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17a6d70 msgr2=0x5632d17be180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8ac259c640 1 --2- 172.21.15.23:0/1967485732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17a6d70 0x5632d17be180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8ac259c640 1 -- 172.21.15.23:0/1967485732 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ac410ee50 con 0x5632d17a63b0 2024-03-24T10:41:17.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8ac1d9b640 1 --2- 172.21.15.23:0/1967485732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17a6d70 0x5632d17be180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:17.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8ac259c640 1 --2- 172.21.15.23:0/1967485732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a63b0 0x5632d17bdc40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40cbab0 tx=0x7f8ab4099260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e031b6d93ef39dd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8aa9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:41:17.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8aa9ffb640 1 -- 172.21.15.23:0/1967485732 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab40a3260 con 0x5632d17a63b0 2024-03-24T10:41:17.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8aa9ffb640 1 -- 172.21.15.23:0/1967485732 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f8ab409b450 con 0x5632d17a63b0 2024-03-24T10:41:17.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.384+0000 7f8acb5c1740 1 -- 172.21.15.23:0/1967485732 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5632d14127e0 con 0x5632d17a63b0 2024-03-24T10:41:17.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.385+0000 7f8aa9ffb640 1 -- 172.21.15.23:0/1967485732 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8ab40a1630 con 0x5632d17a63b0 2024-03-24T10:41:17.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.385+0000 7f8acb5c1740 10 client.25702 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:17.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.385+0000 7f8acb5c1740 1 -- 172.21.15.23:0/1967485732 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5632d1711ba0 con 0x5632d17a63b0 2024-03-24T10:41:17.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.385+0000 7f8aa9ffb640 1 -- 172.21.15.23:0/1967485732 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ab407d080 con 0x5632d17a63b0 2024-03-24T10:41:17.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.385+0000 7f8aa9ffb640 1 -- 172.21.15.23:0/1967485732 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f8ab40b9020 con 0x5632d17a63b0 2024-03-24T10:41:17.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.385+0000 7f8acb5c1740 10 client.25702 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:17.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.385+0000 7f8acb5c1740 10 client.25702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:17.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.385+0000 7f8acb5c1740 20 client.25702 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:17.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.385+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/1967485732 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5632d17ce3b0 0x5632d17d0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.385+0000 7f8acb5c1740 4 client.25702 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:17.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.385+0000 7f8acb5c1740 1 -- 172.21.15.23:0/1967485732 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632d144f720 con 0x5632d17ce3b0 2024-03-24T10:41:17.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.385+0000 7f8ac1d9b640 1 --2- 172.21.15.23:0/1967485732 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5632d17ce3b0 0x5632d17d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.386+0000 7f8ac1d9b640 1 --2- 172.21.15.23:0/1967485732 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5632d17ce3b0 0x5632d17d0810 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=720cb6a6f925fe81 server_cookie=d0976a80c74832be in_seq=0 out_seq=0 2024-03-24T10:41:17.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.386+0000 7f8aa9ffb640 10 client.25702 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:17.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.386+0000 7f8aa9ffb640 1 -- 172.21.15.23:0/1967485732 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5632d144f720 con 0x5632d17ce3b0 2024-03-24T10:41:17.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.386+0000 7f8aa9ffb640 10 client.25702 handle_command_reply: tid=0 2024-03-24T10:41:17.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.386+0000 7f8acb5c1740 1 client.25702 shutdown 2024-03-24T10:41:17.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.386+0000 7f8acb5c1740 1 -- 172.21.15.23:0/1967485732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a63b0 msgr2=0x5632d17bdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.386+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/1967485732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a63b0 0x5632d17bdc40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40cbab0 tx=0x7f8ab4099260 comp rx=0 tx=0).stop 2024-03-24T10:41:17.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.387+0000 7f8acb5c1740 1 -- 172.21.15.23:0/1967485732 shutdown_connections 2024-03-24T10:41:17.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.387+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/1967485732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17b8860 0x5632d17be6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.387+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/1967485732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17a6d70 0x5632d17be180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.387+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/1967485732 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5632d17ce3b0 0x5632d17d0810 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.387+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/1967485732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a63b0 0x5632d17bdc40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.387+0000 7f8aa9ffb640 0 client.25702 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:17.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.387+0000 7f8acb5c1740 1 -- 172.21.15.23:0/1967485732 >> 172.21.15.23:0/1967485732 conn(0x5632d17a8e70 msgr2=0x5632d17a7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.387+0000 7f8acb5c1740 1 -- 172.21.15.23:0/1967485732 shutdown_connections 2024-03-24T10:41:17.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.387+0000 7f8acb5c1740 1 -- 172.21.15.23:0/1967485732 wait complete. 2024-03-24T10:41:17.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.387+0000 7f8acb5c1740 20 client.25702 trim_cache size 0 max 16384 2024-03-24T10:41:17.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.404+0000 7f8acb5c1740 1 Processor -- start 2024-03-24T10:41:17.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.404+0000 7f8acb5c1740 1 -- start start 2024-03-24T10:41:17.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.404+0000 7f8acb5c1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a63b0 0x5632d17bb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.404+0000 7f8acb5c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a6d70 0x5632d17c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8acb5c1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17b8860 0x5632d17b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8acb5c1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5632d1698430 con 0x5632d17b8860 2024-03-24T10:41:17.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8acb5c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5632d1798500 con 0x5632d17a6d70 2024-03-24T10:41:17.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8acb5c1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5632d17a7880 con 0x5632d17a63b0 2024-03-24T10:41:17.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8ac259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a63b0 0x5632d17bb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8ac1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a6d70 0x5632d17c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8ac259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a63b0 0x5632d17bb6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57664/0 (socket says 172.21.15.23:57664) 2024-03-24T10:41:17.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8ac259c640 1 -- 172.21.15.23:0/3573335878 learned_addr learned my addr 172.21.15.23:0/3573335878 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:17.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8ac2d9d640 1 --2- 172.21.15.23:0/3573335878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17b8860 0x5632d17b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8ac1d9b640 1 -- 172.21.15.23:0/3573335878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a63b0 msgr2=0x5632d17bb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8ac1d9b640 1 --2- 172.21.15.23:0/3573335878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a63b0 0x5632d17bb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8ac1d9b640 1 -- 172.21.15.23:0/3573335878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17b8860 msgr2=0x5632d17b62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8ac1d9b640 1 --2- 172.21.15.23:0/3573335878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17b8860 0x5632d17b62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8ac1d9b640 1 -- 172.21.15.23:0/3573335878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632d16adb20 con 0x5632d17a6d70 2024-03-24T10:41:17.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8ac2d9d640 1 --2- 172.21.15.23:0/3573335878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17b8860 0x5632d17b62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:17.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8ac1d9b640 1 --2- 172.21.15.23:0/3573335878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a6d70 0x5632d17c97d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8ab802bc70 tx=0x7f8ab8041040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f155009a4200b7e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8aabfff640 1 -- 172.21.15.23:0/3573335878 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab802cc50 con 0x5632d17a6d70 2024-03-24T10:41:17.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.405+0000 7f8aabfff640 1 -- 172.21.15.23:0/3573335878 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ab8004740 con 0x5632d17a6d70 2024-03-24T10:41:17.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.406+0000 7f8aabfff640 1 -- 172.21.15.23:0/3573335878 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8005050 con 0x5632d17a6d70 2024-03-24T10:41:17.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.406+0000 7f8acb5c1740 1 -- 172.21.15.23:0/3573335878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a6d70 msgr2=0x5632d17c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.406+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/3573335878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a6d70 0x5632d17c97d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8ab802bc70 tx=0x7f8ab8041040 comp rx=0 tx=0).stop 2024-03-24T10:41:17.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.406+0000 7f8acb5c1740 1 -- 172.21.15.23:0/3573335878 shutdown_connections 2024-03-24T10:41:17.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.406+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/3573335878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17b8860 0x5632d17b62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.406+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/3573335878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a6d70 0x5632d17c97d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.406+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/3573335878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17a63b0 0x5632d17bb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.406+0000 7f8acb5c1740 1 -- 172.21.15.23:0/3573335878 >> 172.21.15.23:0/3573335878 conn(0x5632d17a8e70 msgr2=0x5632d17a52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.407+0000 7f8acb5c1740 1 -- 172.21.15.23:0/3573335878 shutdown_connections 2024-03-24T10:41:17.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.407+0000 7f8acb5c1740 1 -- 172.21.15.23:0/3573335878 wait complete. 2024-03-24T10:41:17.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.407+0000 7f8acb5c1740 1 -- start start 2024-03-24T10:41:17.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.407+0000 7f8acb5c1740 1 Processor -- start 2024-03-24T10:41:17.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.407+0000 7f8acb5c1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a63b0 0x5632d17c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.407+0000 7f8acb5c1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17a6d70 0x5632d17bd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.407+0000 7f8acb5c1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17b8860 0x5632d17bdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.407+0000 7f8acb5c1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5632d17be250 con 0x5632d17a6d70 2024-03-24T10:41:17.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.407+0000 7f8acb5c1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5632d17be3c0 con 0x5632d17a63b0 2024-03-24T10:41:17.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.407+0000 7f8acb5c1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5632d17c4f50 con 0x5632d17b8860 2024-03-24T10:41:17.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8ac1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17a6d70 0x5632d17bd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8acb5c1740 10 client.0 _ll_register_callbacks cb 0x7f8ab400ce30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:17.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8ac2d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17b8860 0x5632d17bdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8ac259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a63b0 0x5632d17c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8ac2d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17b8860 0x5632d17bdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57672/0 (socket says 172.21.15.23:57672) 2024-03-24T10:41:17.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8ac2d9d640 1 -- 172.21.15.23:0/2152452237 learned_addr learned my addr 172.21.15.23:0/2152452237 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:17.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8ac1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17a6d70 0x5632d17bd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53586/0 (socket says 172.21.15.23:53586) 2024-03-24T10:41:17.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8ac2d9d640 1 -- 172.21.15.23:0/2152452237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a63b0 msgr2=0x5632d17c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8ac2d9d640 1 --2- 172.21.15.23:0/2152452237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a63b0 0x5632d17c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8ac2d9d640 1 -- 172.21.15.23:0/2152452237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17a6d70 msgr2=0x5632d17bd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8ac2d9d640 1 --2- 172.21.15.23:0/2152452237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17a6d70 0x5632d17bd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8ac2d9d640 1 -- 172.21.15.23:0/2152452237 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632d16d4ea0 con 0x5632d17b8860 2024-03-24T10:41:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8ac259c640 1 --2- 172.21.15.23:0/2152452237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a63b0 0x5632d17c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8ac2d9d640 1 --2- 172.21.15.23:0/2152452237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17b8860 0x5632d17bdb80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8abc011d80 tx=0x7f8abc018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=640575fab4f4f8c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8aa9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:41:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8aa9ffb640 1 -- 172.21.15.23:0/2152452237 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8abc003c30 con 0x5632d17b8860 2024-03-24T10:41:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8aa9ffb640 1 -- 172.21.15.23:0/2152452237 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f8abc02b470 con 0x5632d17b8860 2024-03-24T10:41:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.408+0000 7f8acb5c1740 1 -- 172.21.15.23:0/2152452237 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ab80049e0 con 0x5632d17b8860 2024-03-24T10:41:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.409+0000 7f8aa9ffb640 1 -- 172.21.15.23:0/2152452237 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8abc02b630 con 0x5632d17b8860 2024-03-24T10:41:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.409+0000 7f8acb5c1740 10 client.25708 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.409+0000 7f8acb5c1740 1 -- 172.21.15.23:0/2152452237 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5632d16d3700 con 0x5632d17b8860 2024-03-24T10:41:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.409+0000 7f8aa9ffb640 1 -- 172.21.15.23:0/2152452237 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8abc02bc70 con 0x5632d17b8860 2024-03-24T10:41:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.409+0000 7f8aa9ffb640 1 -- 172.21.15.23:0/2152452237 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f8abc02aa40 con 0x5632d17b8860 2024-03-24T10:41:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.409+0000 7f8acb5c1740 10 client.25708 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.409+0000 7f8acb5c1740 10 client.25708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.409+0000 7f8acb5c1740 20 client.25708 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.409+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/2152452237 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5632d1835050 0x5632d1837470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.409+0000 7f8acb5c1740 4 client.25708 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:41:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.409+0000 7f8acb5c1740 1 -- 172.21.15.23:0/2152452237 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f8ab80049e0 con 0x5632d1835050 2024-03-24T10:41:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8ac259c640 1 --2- 172.21.15.23:0/2152452237 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5632d1835050 0x5632d1837470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8ac259c640 1 --2- 172.21.15.23:0/2152452237 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5632d1835050 0x5632d1837470 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c9719ffd5fc4573 server_cookie=94d157a2a45de0aa in_seq=0 out_seq=0 2024-03-24T10:41:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8aa9ffb640 10 client.25708 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8aa9ffb640 1 -- 172.21.15.23:0/2152452237 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f8ab80049e0 con 0x5632d1835050 2024-03-24T10:41:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8aa9ffb640 10 client.25708 handle_command_reply: tid=0 2024-03-24T10:41:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8acb5c1740 1 client.25708 shutdown 2024-03-24T10:41:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8acb5c1740 1 -- 172.21.15.23:0/2152452237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17b8860 msgr2=0x5632d17bdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/2152452237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17b8860 0x5632d17bdb80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8abc011d80 tx=0x7f8abc018000 comp rx=0 tx=0).stop 2024-03-24T10:41:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8acb5c1740 1 -- 172.21.15.23:0/2152452237 shutdown_connections 2024-03-24T10:41:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/2152452237 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5632d1835050 0x5632d1837470 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/2152452237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632d17b8860 0x5632d17bdb80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/2152452237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5632d17a6d70 0x5632d17bd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8aa9ffb640 0 client.25708 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8acb5c1740 1 --2- 172.21.15.23:0/2152452237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5632d17a63b0 0x5632d17c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8acb5c1740 1 -- 172.21.15.23:0/2152452237 >> 172.21.15.23:0/2152452237 conn(0x5632d17a8e70 msgr2=0x5632d17a5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8acb5c1740 1 -- 172.21.15.23:0/2152452237 shutdown_connections 2024-03-24T10:41:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8acb5c1740 1 -- 172.21.15.23:0/2152452237 wait complete. 2024-03-24T10:41:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.410+0000 7f8acb5c1740 20 client.25708 trim_cache size 0 max 16384 2024-03-24T10:41:17.419 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:41:17.419 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:41:17.419 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:41:17.419 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:41:17.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.411+0000 7f8ac89e2640 1 -- 172.21.15.23:0/634752888 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8aac0a2460 msgr2=0x7f8aac0a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.411+0000 7f8ac89e2640 1 --2- 172.21.15.23:0/634752888 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8aac0a2460 0x7f8aac0a4920 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x5632d17b4e90 tx=0x7f8abc004a40 comp rx=0 tx=0).stop 2024-03-24T10:41:17.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.411+0000 7f8ac89e2640 1 -- 172.21.15.23:0/634752888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac40688f0 msgr2=0x7f8ac4197d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.411+0000 7f8ac89e2640 1 --2- 172.21.15.23:0/634752888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac40688f0 0x7f8ac4197d80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8ab400e9f0 tx=0x7f8ab400edf0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.411+0000 7f8ac89e2640 1 -- 172.21.15.23:0/634752888 shutdown_connections 2024-03-24T10:41:17.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.411+0000 7f8ac89e2640 1 --2- 172.21.15.23:0/634752888 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8aac0a2460 0x7f8aac0a4920 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.411+0000 7f8ac89e2640 1 --2- 172.21.15.23:0/634752888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ac4107690 0x7f8ac419c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.411+0000 7f8ac89e2640 1 --2- 172.21.15.23:0/634752888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ac40692b0 0x7f8ac41982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.411+0000 7f8ac89e2640 1 --2- 172.21.15.23:0/634752888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac40688f0 0x7f8ac4197d80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.411+0000 7f8ac89e2640 1 -- 172.21.15.23:0/634752888 >> 172.21.15.23:0/634752888 conn(0x7f8ac4077710 msgr2=0x7f8ac40fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.411+0000 7f8ac89e2640 1 -- 172.21.15.23:0/634752888 shutdown_connections 2024-03-24T10:41:17.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.411+0000 7f8ac89e2640 1 -- 172.21.15.23:0/634752888 wait complete. 2024-03-24T10:41:17.422 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:41:17.423 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:41:17.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.528+0000 7fd5a1007640 1 -- 172.21.15.23:0/1553041628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c106fd0 msgr2=0x7fd59c100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.528+0000 7fd5a1007640 1 --2- 172.21.15.23:0/1553041628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c106fd0 0x7fd59c100610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd590009860 tx=0x7fd59002bc00 comp rx=0 tx=0).stop 2024-03-24T10:41:17.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.528+0000 7fd5a1007640 1 -- 172.21.15.23:0/1553041628 shutdown_connections 2024-03-24T10:41:17.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.528+0000 7fd5a1007640 1 --2- 172.21.15.23:0/1553041628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd59c100d40 0x7fd59c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.528+0000 7fd5a1007640 1 --2- 172.21.15.23:0/1553041628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c106fd0 0x7fd59c100610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.528+0000 7fd5a1007640 1 --2- 172.21.15.23:0/1553041628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59c1066b0 0x7fd59c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.528+0000 7fd5a1007640 1 -- 172.21.15.23:0/1553041628 >> 172.21.15.23:0/1553041628 conn(0x7fd59c077710 msgr2=0x7fd59c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.528+0000 7fd5a1007640 1 -- 172.21.15.23:0/1553041628 shutdown_connections 2024-03-24T10:41:17.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.528+0000 7fd5a1007640 1 -- 172.21.15.23:0/1553041628 wait complete. 2024-03-24T10:41:17.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.529+0000 7fd5a1007640 1 Processor -- start 2024-03-24T10:41:17.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd5a1007640 1 -- start start 2024-03-24T10:41:17.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd5a1007640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59c100d40 0x7fd59c197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd5a1007640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c1066b0 0x7fd59c1982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd5a1007640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd59c106fd0 0x7fd59c19c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd5a1007640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c19cc50 con 0x7fd59c106fd0 2024-03-24T10:41:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd5a1007640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c19cdc0 con 0x7fd59c100d40 2024-03-24T10:41:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd5a1007640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c19cf80 con 0x7fd59c1066b0 2024-03-24T10:41:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd59a59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c1066b0 0x7fd59c1982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd59ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59c100d40 0x7fd59c197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd59a59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c1066b0 0x7fd59c1982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57682/0 (socket says 172.21.15.23:57682) 2024-03-24T10:41:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd59ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59c100d40 0x7fd59c197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:35392/0 (socket says 172.21.15.23:35392) 2024-03-24T10:41:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd59a59c640 1 -- 172.21.15.23:0/849502430 learned_addr learned my addr 172.21.15.23:0/849502430 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd59b59e640 1 --2- 172.21.15.23:0/849502430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd59c106fd0 0x7fd59c19c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd59a59c640 1 -- 172.21.15.23:0/849502430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59c100d40 msgr2=0x7fd59c197d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd59a59c640 1 --2- 172.21.15.23:0/849502430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59c100d40 0x7fd59c197d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd59a59c640 1 -- 172.21.15.23:0/849502430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd59c106fd0 msgr2=0x7fd59c19c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd59a59c640 1 --2- 172.21.15.23:0/849502430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd59c106fd0 0x7fd59c19c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.530+0000 7fd59a59c640 1 -- 172.21.15.23:0/849502430 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd590009490 con 0x7fd59c1066b0 2024-03-24T10:41:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.531+0000 7fd59a59c640 1 --2- 172.21.15.23:0/849502430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c1066b0 0x7fd59c1982d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd59002c140 tx=0x7fd5900346f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.531+0000 7fd583fff640 1 -- 172.21.15.23:0/849502430 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59002c7a0 con 0x7fd59c1066b0 2024-03-24T10:41:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.531+0000 7fd583fff640 1 -- 172.21.15.23:0/849502430 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5900044d0 con 0x7fd59c1066b0 2024-03-24T10:41:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.531+0000 7fd583fff640 1 -- 172.21.15.23:0/849502430 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59003d650 con 0x7fd59c1066b0 2024-03-24T10:41:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.531+0000 7fd5a1007640 1 -- 172.21.15.23:0/849502430 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd59c19d200 con 0x7fd59c1066b0 2024-03-24T10:41:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.531+0000 7fd5a1007640 1 -- 172.21.15.23:0/849502430 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd59c1a4900 con 0x7fd59c1066b0 2024-03-24T10:41:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.532+0000 7fd5a1007640 1 -- 172.21.15.23:0/849502430 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd59c068500 con 0x7fd59c1066b0 2024-03-24T10:41:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.534+0000 7fd583fff640 1 -- 172.21.15.23:0/849502430 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 24) ==== 137040+0+0 (secure 0 0 0) 0x7fd59003b040 con 0x7fd59c1066b0 2024-03-24T10:41:17.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.534+0000 7fd583fff640 1 --2- 172.21.15.23:0/849502430 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd5840a2460 0x7fd5840a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.534+0000 7fd59ad9d640 1 --2- 172.21.15.23:0/849502430 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd5840a2460 0x7fd5840a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.535+0000 7fd583fff640 1 -- 172.21.15.23:0/849502430 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fd5900efb30 con 0x7fd59c1066b0 2024-03-24T10:41:17.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.535+0000 7fd59ad9d640 1 --2- 172.21.15.23:0/849502430 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd5840a2460 0x7fd5840a4920 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd59c101750 tx=0x7fd58c005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.539+0000 7fd583fff640 1 -- 172.21.15.23:0/849502430 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd5900b8740 con 0x7fd59c1066b0 2024-03-24T10:41:17.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.765+0000 7fd5a1007640 1 -- 172.21.15.23:0/849502430 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd59c1990e0 con 0x7fd59c1066b0 2024-03-24T10:41:17.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.766+0000 7fd583fff640 1 -- 172.21.15.23:0/849502430 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11866 (secure 0 0 0) 0x7fd5900bbe00 con 0x7fd59c1066b0 2024-03-24T10:41:17.766 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:41:17.767 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:40:03.697229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24539,24503,24505]},"id":1}]} 2024-03-24T10:41:17.767 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 23 2024-03-24T10:41:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.768+0000 7fd5a1007640 1 -- 172.21.15.23:0/849502430 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd5840a2460 msgr2=0x7fd5840a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.768+0000 7fd5a1007640 1 --2- 172.21.15.23:0/849502430 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd5840a2460 0x7fd5840a4920 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd59c101750 tx=0x7fd58c005f90 comp rx=0 tx=0).stop 2024-03-24T10:41:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.768+0000 7fd5a1007640 1 -- 172.21.15.23:0/849502430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c1066b0 msgr2=0x7fd59c1982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.768+0000 7fd5a1007640 1 --2- 172.21.15.23:0/849502430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c1066b0 0x7fd59c1982d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd59002c140 tx=0x7fd5900346f0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.768+0000 7fd5a1007640 1 -- 172.21.15.23:0/849502430 shutdown_connections 2024-03-24T10:41:17.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.768+0000 7fd5a1007640 1 --2- 172.21.15.23:0/849502430 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd5840a2460 0x7fd5840a4920 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.768+0000 7fd5a1007640 1 --2- 172.21.15.23:0/849502430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd59c106fd0 0x7fd59c19c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.768+0000 7fd5a1007640 1 --2- 172.21.15.23:0/849502430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c1066b0 0x7fd59c1982d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.768+0000 7fd5a1007640 1 --2- 172.21.15.23:0/849502430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd59c100d40 0x7fd59c197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.768+0000 7fd5a1007640 1 -- 172.21.15.23:0/849502430 >> 172.21.15.23:0/849502430 conn(0x7fd59c077710 msgr2=0x7fd59c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.768+0000 7fd5a1007640 1 -- 172.21.15.23:0/849502430 shutdown_connections 2024-03-24T10:41:17.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.768+0000 7fd5a1007640 1 -- 172.21.15.23:0/849502430 wait complete. 2024-03-24T10:41:17.780 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:41:17.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.886+0000 7fad971b5640 1 -- 172.21.15.23:0/3447291076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad901005a0 msgr2=0x7fad90100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.886+0000 7fad971b5640 1 --2- 172.21.15.23:0/3447291076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad901005a0 0x7fad90100a20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fad7c009930 tx=0x7fad7c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:41:17.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.886+0000 7fad971b5640 1 -- 172.21.15.23:0/3447291076 shutdown_connections 2024-03-24T10:41:17.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.886+0000 7fad971b5640 1 --2- 172.21.15.23:0/3447291076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad90100f60 0x7fad9010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.886+0000 7fad971b5640 1 --2- 172.21.15.23:0/3447291076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad901005a0 0x7fad90100a20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.886+0000 7fad971b5640 1 --2- 172.21.15.23:0/3447291076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad90106660 0x7fad90106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.886+0000 7fad971b5640 1 -- 172.21.15.23:0/3447291076 >> 172.21.15.23:0/3447291076 conn(0x7fad90077710 msgr2=0x7fad90077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.886+0000 7fad971b5640 1 -- 172.21.15.23:0/3447291076 shutdown_connections 2024-03-24T10:41:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.886+0000 7fad971b5640 1 -- 172.21.15.23:0/3447291076 wait complete. 2024-03-24T10:41:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.886+0000 7fad971b5640 1 Processor -- start 2024-03-24T10:41:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad971b5640 1 -- start start 2024-03-24T10:41:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad971b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad901005a0 0x7fad90199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad971b5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad90100f60 0x7fad9019a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad971b5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad90106660 0x7fad9019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad971b5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fad9019ee20 con 0x7fad90106660 2024-03-24T10:41:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad971b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad9019ef90 con 0x7fad901005a0 2024-03-24T10:41:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad971b5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fad9019f150 con 0x7fad90100f60 2024-03-24T10:41:17.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad94f51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad901005a0 0x7fad90199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad95752640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad90106660 0x7fad9019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad8ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad90100f60 0x7fad9019a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad8ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad90100f60 0x7fad9019a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57622/0 (socket says 172.21.15.23:57622) 2024-03-24T10:41:17.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad95752640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad90106660 0x7fad9019e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35872/0 (socket says 172.21.15.23:35872) 2024-03-24T10:41:17.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad8ffff640 1 -- 172.21.15.23:0/340637054 learned_addr learned my addr 172.21.15.23:0/340637054 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad94f51640 1 -- 172.21.15.23:0/340637054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad90100f60 msgr2=0x7fad9019a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad94f51640 1 --2- 172.21.15.23:0/340637054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad90100f60 0x7fad9019a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad94f51640 1 -- 172.21.15.23:0/340637054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad90106660 msgr2=0x7fad9019e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad94f51640 1 --2- 172.21.15.23:0/340637054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad90106660 0x7fad9019e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad94f51640 1 -- 172.21.15.23:0/340637054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad7c009560 con 0x7fad901005a0 2024-03-24T10:41:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad8ffff640 1 --2- 172.21.15.23:0/340637054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad90100f60 0x7fad9019a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.887+0000 7fad95752640 1 --2- 172.21.15.23:0/340637054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad90106660 0x7fad9019e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.888+0000 7fad94f51640 1 --2- 172.21.15.23:0/340637054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad901005a0 0x7fad90199f60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fad840097d0 tx=0x7fad84009c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.888+0000 7fad8dffb640 1 -- 172.21.15.23:0/340637054 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad84010040 con 0x7fad901005a0 2024-03-24T10:41:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.888+0000 7fad8dffb640 1 -- 172.21.15.23:0/340637054 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad8400ed80 con 0x7fad901005a0 2024-03-24T10:41:17.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.888+0000 7fad971b5640 1 -- 172.21.15.23:0/340637054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad9019f430 con 0x7fad901005a0 2024-03-24T10:41:17.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.888+0000 7fad8dffb640 1 -- 172.21.15.23:0/340637054 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad84013380 con 0x7fad901005a0 2024-03-24T10:41:17.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.888+0000 7fad971b5640 1 -- 172.21.15.23:0/340637054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad901a6cb0 con 0x7fad901005a0 2024-03-24T10:41:17.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.891+0000 7fad8dffb640 1 -- 172.21.15.23:0/340637054 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 137040+0+0 (secure 0 0 0) 0x7fad8400e8a0 con 0x7fad901005a0 2024-03-24T10:41:17.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.892+0000 7fad99d9a740 1 Processor -- start 2024-03-24T10:41:17.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.892+0000 7fad8dffb640 1 --2- 172.21.15.23:0/340637054 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fad800a24e0 0x7fad800a49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.892+0000 7fad95752640 1 --2- 172.21.15.23:0/340637054 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fad800a24e0 0x7fad800a49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.892+0000 7fad8dffb640 1 -- 172.21.15.23:0/340637054 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fad840cdfa0 con 0x7fad901005a0 2024-03-24T10:41:17.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.892+0000 7fad99d9a740 1 -- start start 2024-03-24T10:41:17.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.892+0000 7fad95752640 1 --2- 172.21.15.23:0/340637054 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fad800a24e0 0x7fad800a49a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fad88005fd0 tx=0x7fad88009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad99d9a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40dc00 0x5609fa40dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad99d9a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40e520 0x5609fa416bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad99d9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 0x5609fa4194f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad99d9a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5609fa373840 con 0x5609fa40e520 2024-03-24T10:41:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad99d9a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5609fa370d70 con 0x5609fa417100 2024-03-24T10:41:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad99d9a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5609fa36c550 con 0x5609fa40dc00 2024-03-24T10:41:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad8ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40e520 0x5609fa416bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad8ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40e520 0x5609fa416bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35886/0 (socket says 172.21.15.23:35886) 2024-03-24T10:41:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad8ffff640 1 -- 172.21.15.23:0/1359820704 learned_addr learned my addr 172.21.15.23:0/1359820704 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad95752640 1 --2- 172.21.15.23:0/1359820704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 0x5609fa4194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad94f51640 1 --2- 172.21.15.23:0/1359820704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40dc00 0x5609fa40dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad94f51640 1 -- 172.21.15.23:0/1359820704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 msgr2=0x5609fa4194f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad94f51640 1 --2- 172.21.15.23:0/1359820704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 0x5609fa4194f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad94f51640 1 -- 172.21.15.23:0/1359820704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40e520 msgr2=0x5609fa416bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad94f51640 1 --2- 172.21.15.23:0/1359820704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40e520 0x5609fa416bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad94f51640 1 -- 172.21.15.23:0/1359820704 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609fa301de0 con 0x5609fa40dc00 2024-03-24T10:41:17.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad8ffff640 1 --2- 172.21.15.23:0/1359820704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40e520 0x5609fa416bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:17.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad95752640 1 --2- 172.21.15.23:0/1359820704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 0x5609fa4194f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:17.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad94f51640 1 --2- 172.21.15.23:0/1359820704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40dc00 0x5609fa40dfe0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fad840cb4b0 tx=0x7fad8409a290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b1787117eee814 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.893+0000 7fad7affd640 1 -- 172.21.15.23:0/1359820704 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8409a750 con 0x5609fa40dc00 2024-03-24T10:41:17.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.894+0000 7fad7affd640 1 -- 172.21.15.23:0/1359820704 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad8409e8a0 con 0x5609fa40dc00 2024-03-24T10:41:17.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.894+0000 7fad99d9a740 1 -- 172.21.15.23:0/1359820704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40dc00 msgr2=0x5609fa40dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.894+0000 7fad99d9a740 1 --2- 172.21.15.23:0/1359820704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40dc00 0x5609fa40dfe0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fad840cb4b0 tx=0x7fad8409a290 comp rx=0 tx=0).stop 2024-03-24T10:41:17.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.894+0000 7fad99d9a740 1 -- 172.21.15.23:0/1359820704 shutdown_connections 2024-03-24T10:41:17.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.894+0000 7fad99d9a740 1 --2- 172.21.15.23:0/1359820704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 0x5609fa4194f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.894+0000 7fad99d9a740 1 --2- 172.21.15.23:0/1359820704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40e520 0x5609fa416bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.894+0000 7fad99d9a740 1 --2- 172.21.15.23:0/1359820704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40dc00 0x5609fa40dfe0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.894+0000 7fad99d9a740 1 -- 172.21.15.23:0/1359820704 >> 172.21.15.23:0/1359820704 conn(0x5609fa408330 msgr2=0x5609fa408710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.894+0000 7fad99d9a740 1 -- 172.21.15.23:0/1359820704 shutdown_connections 2024-03-24T10:41:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.894+0000 7fad99d9a740 1 -- 172.21.15.23:0/1359820704 wait complete. 2024-03-24T10:41:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.895+0000 7fad99d9a740 1 -- start start 2024-03-24T10:41:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.895+0000 7fad99d9a740 1 Processor -- start 2024-03-24T10:41:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.895+0000 7fad99d9a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40dc00 0x5609fa414140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.895+0000 7fad99d9a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40e520 0x5609fa414680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad99d9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 0x5609fa414bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad99d9a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5609fa03f430 con 0x5609fa40dc00 2024-03-24T10:41:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad99d9a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5609f9ffbe00 con 0x5609fa417100 2024-03-24T10:41:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad99d9a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5609fa2f1590 con 0x5609fa40e520 2024-03-24T10:41:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad94f51640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40dc00 0x5609fa414140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad94f51640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40dc00 0x5609fa414140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35892/0 (socket says 172.21.15.23:35892) 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad94f51640 1 -- 172.21.15.23:0/4046202045 learned_addr learned my addr 172.21.15.23:0/4046202045 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad8ffff640 1 --2- 172.21.15.23:0/4046202045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40e520 0x5609fa414680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad95752640 1 --2- 172.21.15.23:0/4046202045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 0x5609fa414bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad94f51640 1 -- 172.21.15.23:0/4046202045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40e520 msgr2=0x5609fa414680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad94f51640 1 --2- 172.21.15.23:0/4046202045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40e520 0x5609fa414680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad94f51640 1 -- 172.21.15.23:0/4046202045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 msgr2=0x5609fa414bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad94f51640 1 --2- 172.21.15.23:0/4046202045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 0x5609fa414bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad94f51640 1 -- 172.21.15.23:0/4046202045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad901a6cb0 con 0x5609fa40dc00 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad8ffff640 1 --2- 172.21.15.23:0/4046202045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40e520 0x5609fa414680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad99d9a740 10 client.0 _ll_register_callbacks cb 0x5609fa360fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad94f51640 1 --2- 172.21.15.23:0/4046202045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40dc00 0x5609fa414140 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fad8409a690 tx=0x7fad84098a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f876b6e991ee2cfe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad6ffff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad99d9a740 1 -- 172.21.15.23:0/4046202045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5609fa06b7e0 con 0x5609fa40dc00 2024-03-24T10:41:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad6ffff640 1 -- 172.21.15.23:0/4046202045 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad84098b50 con 0x5609fa40dc00 2024-03-24T10:41:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.896+0000 7fad6ffff640 1 -- 172.21.15.23:0/4046202045 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fad840cbe40 con 0x5609fa40dc00 2024-03-24T10:41:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.897+0000 7fad6ffff640 1 -- 172.21.15.23:0/4046202045 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fad8409a750 con 0x5609fa40dc00 2024-03-24T10:41:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.897+0000 7fad99d9a740 10 client.15858 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.897+0000 7fad99d9a740 1 -- 172.21.15.23:0/4046202045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({fsmap=23}) -- 0x5609fa36abd0 con 0x5609fa40dc00 2024-03-24T10:41:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.897+0000 7fad6ffff640 1 -- 172.21.15.23:0/4046202045 <== mon.0 v2:172.21.15.23:3300/0 4 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fad840ba0d0 con 0x5609fa40dc00 2024-03-24T10:41:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.897+0000 7fad99d9a740 10 client.15858 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.897+0000 7fad99d9a740 10 client.15858 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:41:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.897+0000 7fad99d9a740 20 client.15858 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.897+0000 7fad99d9a740 1 --2- 172.21.15.23:0/4046202045 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5609fa426bd0 0x5609fa429070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.897+0000 7fad99d9a740 4 client.15858 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.897+0000 7fad99d9a740 1 -- 172.21.15.23:0/4046202045 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609fa0a8720 con 0x5609fa426bd0 2024-03-24T10:41:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.897+0000 7fad8ffff640 1 --2- 172.21.15.23:0/4046202045 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5609fa426bd0 0x5609fa429070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.898+0000 7fad6ffff640 10 client.15858 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:41:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.898+0000 7fad8ffff640 1 --2- 172.21.15.23:0/4046202045 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5609fa426bd0 0x5609fa429070 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=646e2826e664fde3 server_cookie=f4da66ec09d05183 in_seq=0 out_seq=0 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.898+0000 7fad6ffff640 1 -- 172.21.15.23:0/4046202045 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5609fa0a8720 con 0x5609fa426bd0 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.898+0000 7fad6ffff640 10 client.15858 handle_command_reply: tid=0 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.898+0000 7fad99d9a740 1 client.15858 shutdown 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.898+0000 7fad99d9a740 1 -- 172.21.15.23:0/4046202045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40dc00 msgr2=0x5609fa414140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.899+0000 7fad99d9a740 1 --2- 172.21.15.23:0/4046202045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40dc00 0x5609fa414140 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fad8409a690 tx=0x7fad84098a50 comp rx=0 tx=0).stop 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.899+0000 7fad99d9a740 1 -- 172.21.15.23:0/4046202045 shutdown_connections 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.899+0000 7fad99d9a740 1 --2- 172.21.15.23:0/4046202045 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5609fa426bd0 0x5609fa429070 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.899+0000 7fad99d9a740 1 --2- 172.21.15.23:0/4046202045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 0x5609fa414bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.899+0000 7fad6ffff640 0 client.15858 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.899+0000 7fad99d9a740 1 --2- 172.21.15.23:0/4046202045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40e520 0x5609fa414680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.899+0000 7fad99d9a740 1 --2- 172.21.15.23:0/4046202045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40dc00 0x5609fa414140 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.899+0000 7fad99d9a740 1 -- 172.21.15.23:0/4046202045 >> 172.21.15.23:0/4046202045 conn(0x5609fa408330 msgr2=0x5609fa3fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.899+0000 7fad99d9a740 1 -- 172.21.15.23:0/4046202045 shutdown_connections 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.899+0000 7fad99d9a740 1 -- 172.21.15.23:0/4046202045 wait complete. 2024-03-24T10:41:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.899+0000 7fad99d9a740 20 client.15858 trim_cache size 0 max 16384 2024-03-24T10:41:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.925+0000 7fad99d9a740 1 Processor -- start 2024-03-24T10:41:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.925+0000 7fad99d9a740 1 -- start start 2024-03-24T10:41:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.925+0000 7fad99d9a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40dc00 0x5609fa416870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.925+0000 7fad99d9a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40e520 0x5609fa422a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.925+0000 7fad99d9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 0x5609fa42e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.925+0000 7fad99d9a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5609fa3fb250 con 0x5609fa40dc00 2024-03-24T10:41:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.925+0000 7fad99d9a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5609fa3fb760 con 0x5609fa417100 2024-03-24T10:41:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.925+0000 7fad99d9a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5609fa0a8720 con 0x5609fa40e520 2024-03-24T10:41:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.925+0000 7fad94f51640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40dc00 0x5609fa416870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.925+0000 7fad8ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40e520 0x5609fa422a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.926+0000 7fad8ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40e520 0x5609fa422a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57646/0 (socket says 172.21.15.23:57646) 2024-03-24T10:41:17.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.926+0000 7fad94f51640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40dc00 0x5609fa416870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35896/0 (socket says 172.21.15.23:35896) 2024-03-24T10:41:17.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.926+0000 7fad8ffff640 1 -- 172.21.15.23:0/989099053 learned_addr learned my addr 172.21.15.23:0/989099053 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:17.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.926+0000 7fad95752640 1 --2- 172.21.15.23:0/989099053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 0x5609fa42e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.926+0000 7fad8ffff640 1 -- 172.21.15.23:0/989099053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 msgr2=0x5609fa42e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.926+0000 7fad8ffff640 1 --2- 172.21.15.23:0/989099053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 0x5609fa42e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.926+0000 7fad8ffff640 1 -- 172.21.15.23:0/989099053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40dc00 msgr2=0x5609fa416870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.926+0000 7fad8ffff640 1 --2- 172.21.15.23:0/989099053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40dc00 0x5609fa416870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.926+0000 7fad8ffff640 1 -- 172.21.15.23:0/989099053 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609fa306c80 con 0x5609fa40e520 2024-03-24T10:41:17.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.926+0000 7fad94f51640 1 --2- 172.21.15.23:0/989099053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40dc00 0x5609fa416870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:17.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.926+0000 7fad8ffff640 1 --2- 172.21.15.23:0/989099053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40e520 0x5609fa422a30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fad7c02c210 tx=0x7fad7c005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=735c3a318a73f2a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.926+0000 7fad7a7fc640 1 -- 172.21.15.23:0/989099053 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c02c9a0 con 0x5609fa40e520 2024-03-24T10:41:17.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.926+0000 7fad7a7fc640 1 -- 172.21.15.23:0/989099053 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad7c02cb00 con 0x5609fa40e520 2024-03-24T10:41:17.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.926+0000 7fad7a7fc640 1 -- 172.21.15.23:0/989099053 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c0043a0 con 0x5609fa40e520 2024-03-24T10:41:17.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.927+0000 7fad99d9a740 1 -- 172.21.15.23:0/989099053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40e520 msgr2=0x5609fa422a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.927+0000 7fad99d9a740 1 --2- 172.21.15.23:0/989099053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40e520 0x5609fa422a30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fad7c02c210 tx=0x7fad7c005a20 comp rx=0 tx=0).stop 2024-03-24T10:41:17.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.927+0000 7fad99d9a740 1 -- 172.21.15.23:0/989099053 shutdown_connections 2024-03-24T10:41:17.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.927+0000 7fad99d9a740 1 --2- 172.21.15.23:0/989099053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa417100 0x5609fa42e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.927+0000 7fad99d9a740 1 --2- 172.21.15.23:0/989099053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa40e520 0x5609fa422a30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.927+0000 7fad99d9a740 1 --2- 172.21.15.23:0/989099053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40dc00 0x5609fa416870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.927+0000 7fad99d9a740 1 -- 172.21.15.23:0/989099053 >> 172.21.15.23:0/989099053 conn(0x5609fa408330 msgr2=0x5609fa3fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.927+0000 7fad99d9a740 1 -- 172.21.15.23:0/989099053 shutdown_connections 2024-03-24T10:41:17.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.927+0000 7fad99d9a740 1 -- 172.21.15.23:0/989099053 wait complete. 2024-03-24T10:41:17.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.928+0000 7fad99d9a740 1 -- start start 2024-03-24T10:41:17.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.928+0000 7fad99d9a740 1 Processor -- start 2024-03-24T10:41:17.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.928+0000 7fad99d9a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa40dc00 0x5609fa41ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.928+0000 7fad99d9a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40e520 0x5609fa41b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.928+0000 7fad99d9a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa417100 0x5609fa41b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.928+0000 7fad99d9a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5609fa48c120 con 0x5609fa40e520 2024-03-24T10:41:17.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.928+0000 7fad94f51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa40dc00 0x5609fa41ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.928+0000 7fad8ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40e520 0x5609fa41b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.928+0000 7fad94f51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa40dc00 0x5609fa41ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:33266/0 (socket says 172.21.15.23:33266) 2024-03-24T10:41:17.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.928+0000 7fad8ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40e520 0x5609fa41b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35904/0 (socket says 172.21.15.23:35904) 2024-03-24T10:41:17.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.928+0000 7fad94f51640 1 -- 172.21.15.23:0/672610572 learned_addr learned my addr 172.21.15.23:0/672610572 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.928+0000 7fad99d9a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5609fa48c290 con 0x5609fa40dc00 2024-03-24T10:41:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.928+0000 7fad99d9a740 1 -- 172.21.15.23:0/672610572 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5609fa48c450 con 0x5609fa417100 2024-03-24T10:41:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad95752640 1 --2- 172.21.15.23:0/672610572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa417100 0x5609fa41b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad99d9a740 10 client.0 _ll_register_callbacks cb 0x7fad84005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad8ffff640 1 -- 172.21.15.23:0/672610572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa417100 msgr2=0x5609fa41b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad8ffff640 1 --2- 172.21.15.23:0/672610572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa417100 0x5609fa41b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad8ffff640 1 -- 172.21.15.23:0/672610572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa40dc00 msgr2=0x5609fa41ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad8ffff640 1 --2- 172.21.15.23:0/672610572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa40dc00 0x5609fa41ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad8ffff640 1 -- 172.21.15.23:0/672610572 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609fa36abd0 con 0x5609fa40e520 2024-03-24T10:41:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad94f51640 1 --2- 172.21.15.23:0/672610572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa40dc00 0x5609fa41ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad99d9a740 1 -- 172.21.15.23:0/672610572 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad8409e8a0 con 0x5609fa40e520 2024-03-24T10:41:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad8ffff640 1 --2- 172.21.15.23:0/672610572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40e520 0x5609fa41b180 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x5609fa426a60 tx=0x7fad7c005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e5f61e756ad8d4a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad6ffff640 10 client.15864 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad6ffff640 1 -- 172.21.15.23:0/672610572 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c049b10 con 0x5609fa40e520 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad6ffff640 1 -- 172.21.15.23:0/672610572 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fad7c045040 con 0x5609fa40e520 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad6ffff640 1 -- 172.21.15.23:0/672610572 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fad7c049c70 con 0x5609fa40e520 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad99d9a740 10 client.15864 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.929+0000 7fad99d9a740 1 -- 172.21.15.23:0/672610572 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({fsmap=23}) -- 0x7fad7c030070 con 0x5609fa40e520 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.930+0000 7fad6ffff640 1 -- 172.21.15.23:0/672610572 <== mon.0 v2:172.21.15.23:3300/0 4 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fad7c03c040 con 0x5609fa40e520 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.930+0000 7fad99d9a740 10 client.15864 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.930+0000 7fad99d9a740 10 client.15864 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.930+0000 7fad99d9a740 20 client.15864 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.930+0000 7fad99d9a740 1 --2- 172.21.15.23:0/672610572 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5609fa426bd0 0x5609fa48f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.930+0000 7fad99d9a740 4 client.15864 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.930+0000 7fad99d9a740 1 -- 172.21.15.23:0/672610572 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5609fa0f7890 con 0x5609fa426bd0 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.930+0000 7fad94f51640 1 --2- 172.21.15.23:0/672610572 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5609fa426bd0 0x5609fa48f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.930+0000 7fad6ffff640 10 client.15864 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.930+0000 7fad94f51640 1 --2- 172.21.15.23:0/672610572 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5609fa426bd0 0x5609fa48f750 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2a4c919268a0ba89 server_cookie=29f12cce89a54d83 in_seq=0 out_seq=0 2024-03-24T10:41:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.931+0000 7fad6ffff640 1 -- 172.21.15.23:0/672610572 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5609fa0f7890 con 0x5609fa426bd0 2024-03-24T10:41:17.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.931+0000 7fad6ffff640 10 client.15864 handle_command_reply: tid=0 2024-03-24T10:41:17.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.931+0000 7fad99d9a740 1 client.15864 shutdown 2024-03-24T10:41:17.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.933+0000 7fad99d9a740 1 -- 172.21.15.23:0/672610572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40e520 msgr2=0x5609fa41b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.933+0000 7fad99d9a740 1 --2- 172.21.15.23:0/672610572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40e520 0x5609fa41b180 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x5609fa426a60 tx=0x7fad7c005c90 comp rx=0 tx=0).stop 2024-03-24T10:41:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.933+0000 7fad99d9a740 1 -- 172.21.15.23:0/672610572 shutdown_connections 2024-03-24T10:41:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.933+0000 7fad99d9a740 1 --2- 172.21.15.23:0/672610572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609fa417100 0x5609fa41b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.933+0000 7fad99d9a740 1 --2- 172.21.15.23:0/672610572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5609fa40e520 0x5609fa41b180 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.933+0000 7fad99d9a740 1 --2- 172.21.15.23:0/672610572 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5609fa426bd0 0x5609fa48f750 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.933+0000 7fad6ffff640 0 client.15864 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:41:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.933+0000 7fad99d9a740 1 --2- 172.21.15.23:0/672610572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609fa40dc00 0x5609fa41ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.933+0000 7fad99d9a740 1 -- 172.21.15.23:0/672610572 >> 172.21.15.23:0/672610572 conn(0x5609fa408330 msgr2=0x5609fa413f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.933+0000 7fad99d9a740 1 -- 172.21.15.23:0/672610572 shutdown_connections 2024-03-24T10:41:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.933+0000 7fad99d9a740 1 -- 172.21.15.23:0/672610572 wait complete. 2024-03-24T10:41:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.933+0000 7fad99d9a740 20 client.15864 trim_cache size 0 max 16384 2024-03-24T10:41:17.937 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:41:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.934+0000 7fad971b5640 1 -- 172.21.15.23:0/340637054 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fad800a24e0 msgr2=0x7fad800a49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.934+0000 7fad971b5640 1 --2- 172.21.15.23:0/340637054 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fad800a24e0 0x7fad800a49a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fad88005fd0 tx=0x7fad88009500 comp rx=0 tx=0).stop 2024-03-24T10:41:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.934+0000 7fad971b5640 1 -- 172.21.15.23:0/340637054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad901005a0 msgr2=0x7fad90199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.934+0000 7fad971b5640 1 --2- 172.21.15.23:0/340637054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad901005a0 0x7fad90199f60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fad840097d0 tx=0x7fad84009c90 comp rx=0 tx=0).stop 2024-03-24T10:41:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.934+0000 7fad971b5640 1 -- 172.21.15.23:0/340637054 shutdown_connections 2024-03-24T10:41:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.934+0000 7fad971b5640 1 --2- 172.21.15.23:0/340637054 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fad800a24e0 0x7fad800a49a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.934+0000 7fad971b5640 1 --2- 172.21.15.23:0/340637054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad90106660 0x7fad9019e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.934+0000 7fad971b5640 1 --2- 172.21.15.23:0/340637054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad90100f60 0x7fad9019a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.934+0000 7fad971b5640 1 --2- 172.21.15.23:0/340637054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad901005a0 0x7fad90199f60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:17.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.934+0000 7fad971b5640 1 -- 172.21.15.23:0/340637054 >> 172.21.15.23:0/340637054 conn(0x7fad90077710 msgr2=0x7fad9010ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:17.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.934+0000 7fad971b5640 1 -- 172.21.15.23:0/340637054 shutdown_connections 2024-03-24T10:41:17.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:17.934+0000 7fad971b5640 1 -- 172.21.15.23:0/340637054 wait complete. 2024-03-24T10:41:17.946 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:41:18.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.053+0000 7f3ec110c640 1 -- 172.21.15.23:0/1554383628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ebc102060 msgr2=0x7f3ebc10e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.053+0000 7f3ec110c640 1 --2- 172.21.15.23:0/1554383628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ebc102060 0x7f3ebc10e570 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4009cd0 tx=0x7f3eb402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.053+0000 7f3ec110c640 1 -- 172.21.15.23:0/1554383628 shutdown_connections 2024-03-24T10:41:18.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.053+0000 7f3ec110c640 1 --2- 172.21.15.23:0/1554383628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ebc102060 0x7f3ebc10e570 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.053+0000 7f3ec110c640 1 --2- 172.21.15.23:0/1554383628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ebc1016a0 0x7f3ebc101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.053+0000 7f3ec110c640 1 --2- 172.21.15.23:0/1554383628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ebc1076b0 0x7f3ebc107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.053+0000 7f3ec110c640 1 -- 172.21.15.23:0/1554383628 >> 172.21.15.23:0/1554383628 conn(0x7f3ebc0fd1b0 msgr2=0x7f3ebc0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:18.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.053+0000 7f3ec110c640 1 -- 172.21.15.23:0/1554383628 shutdown_connections 2024-03-24T10:41:18.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.053+0000 7f3ec110c640 1 -- 172.21.15.23:0/1554383628 wait complete. 2024-03-24T10:41:18.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.054+0000 7f3ec110c640 1 Processor -- start 2024-03-24T10:41:18.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.054+0000 7f3ec110c640 1 -- start start 2024-03-24T10:41:18.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.054+0000 7f3ec110c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ebc1016a0 0x7f3ebc199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.054+0000 7f3ec110c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ebc102060 0x7f3ebc19a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.054+0000 7f3ec110c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ebc1076b0 0x7f3ebc19e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.054+0000 7f3ec110c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f3ebc19edb0 con 0x7f3ebc1076b0 2024-03-24T10:41:18.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.055+0000 7f3ec110c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3ebc19ef20 con 0x7f3ebc1016a0 2024-03-24T10:41:18.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.055+0000 7f3ec110c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3ebc19f0e0 con 0x7f3ebc102060 2024-03-24T10:41:18.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.055+0000 7f3eba59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ebc102060 0x7f3ebc19a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.055+0000 7f3ebad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ebc1016a0 0x7f3ebc199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.055+0000 7f3eba59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ebc102060 0x7f3ebc19a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57666/0 (socket says 172.21.15.23:57666) 2024-03-24T10:41:18.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.055+0000 7f3eba59c640 1 -- 172.21.15.23:0/1681068903 learned_addr learned my addr 172.21.15.23:0/1681068903 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:18.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.055+0000 7f3ebb59e640 1 --2- 172.21.15.23:0/1681068903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ebc1076b0 0x7f3ebc19e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.055+0000 7f3ebad9d640 1 -- 172.21.15.23:0/1681068903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ebc102060 msgr2=0x7f3ebc19a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.055+0000 7f3ebad9d640 1 --2- 172.21.15.23:0/1681068903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ebc102060 0x7f3ebc19a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.055+0000 7f3ebad9d640 1 -- 172.21.15.23:0/1681068903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ebc1076b0 msgr2=0x7f3ebc19e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.055+0000 7f3ebad9d640 1 --2- 172.21.15.23:0/1681068903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ebc1076b0 0x7f3ebc19e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.055+0000 7f3ebad9d640 1 -- 172.21.15.23:0/1681068903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb4009900 con 0x7f3ebc1016a0 2024-03-24T10:41:18.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.055+0000 7f3eba59c640 1 --2- 172.21.15.23:0/1681068903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ebc102060 0x7f3ebc19a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:18.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.055+0000 7f3ebad9d640 1 --2- 172.21.15.23:0/1681068903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ebc1016a0 0x7f3ebc199ef0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00e9f0 tx=0x7f3eac00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:18.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.056+0000 7f3ea3fff640 1 -- 172.21.15.23:0/1681068903 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac00ccc0 con 0x7f3ebc1016a0 2024-03-24T10:41:18.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.056+0000 7f3ea3fff640 1 -- 172.21.15.23:0/1681068903 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eac00ce20 con 0x7f3ebc1016a0 2024-03-24T10:41:18.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.056+0000 7f3ec110c640 1 -- 172.21.15.23:0/1681068903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ebc19f3c0 con 0x7f3ebc1016a0 2024-03-24T10:41:18.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.056+0000 7f3ea3fff640 1 -- 172.21.15.23:0/1681068903 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac018920 con 0x7f3ebc1016a0 2024-03-24T10:41:18.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.056+0000 7f3ec110c640 1 -- 172.21.15.23:0/1681068903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ebc1a6c40 con 0x7f3ebc1016a0 2024-03-24T10:41:18.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.058+0000 7f3ec3dc8740 1 Processor -- start 2024-03-24T10:41:18.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.058+0000 7f3ec3dc8740 1 -- start start 2024-03-24T10:41:18.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.058+0000 7f3ec3dc8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813c3f0 0x55994813c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3ec3dc8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813cd10 0x5599481453b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3ec3dc8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481458f0 0x559948147ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3ec3dc8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5599480aa840 con 0x5599481458f0 2024-03-24T10:41:18.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3ec3dc8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5599480a7d70 con 0x55994813c3f0 2024-03-24T10:41:18.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3ec3dc8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5599480a3550 con 0x55994813cd10 2024-03-24T10:41:18.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3ea3fff640 1 -- 172.21.15.23:0/1681068903 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 137040+0+0 (secure 0 0 0) 0x7f3eac010430 con 0x7f3ebc1016a0 2024-03-24T10:41:18.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3ebb59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813c3f0 0x55994813c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3ebb59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813c3f0 0x55994813c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:33288/0 (socket says 172.21.15.23:33288) 2024-03-24T10:41:18.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3ebb59e640 1 -- 172.21.15.23:0/2842664694 learned_addr learned my addr 172.21.15.23:0/2842664694 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:18.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3eba59c640 1 --2- 172.21.15.23:0/2842664694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481458f0 0x559948147ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3eba59c640 1 -- 172.21.15.23:0/2842664694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813cd10 msgr2=0x5599481453b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T10:41:18.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3eba59c640 1 --2- 172.21.15.23:0/2842664694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813cd10 0x5599481453b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3eba59c640 1 -- 172.21.15.23:0/2842664694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813c3f0 msgr2=0x55994813c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3eba59c640 1 --2- 172.21.15.23:0/2842664694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813c3f0 0x55994813c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3eba59c640 1 -- 172.21.15.23:0/2842664694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559948038de0 con 0x5599481458f0 2024-03-24T10:41:18.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3ebb59e640 1 --2- 172.21.15.23:0/2842664694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813c3f0 0x55994813c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:18.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.059+0000 7f3ea3fff640 1 --2- 172.21.15.23:0/1681068903 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3ea40a2460 0x7f3ea40a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.060+0000 7f3eba59c640 1 --2- 172.21.15.23:0/2842664694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481458f0 0x559948147ce0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00114c0 tx=0x7f3eb0011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b509d7525e0a3c3d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:18.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.060+0000 7f3ea3fff640 1 -- 172.21.15.23:0/1681068903 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f3eac014070 con 0x7f3ebc1016a0 2024-03-24T10:41:18.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.060+0000 7f3ebb59e640 1 --2- 172.21.15.23:0/1681068903 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3ea40a2460 0x7f3ea40a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.060+0000 7f3ea17fa640 1 -- 172.21.15.23:0/2842664694 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0004030 con 0x5599481458f0 2024-03-24T10:41:18.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.060+0000 7f3ea17fa640 1 -- 172.21.15.23:0/2842664694 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eb00035f0 con 0x5599481458f0 2024-03-24T10:41:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.061+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/2842664694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481458f0 msgr2=0x559948147ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.061+0000 7f3ebb59e640 1 --2- 172.21.15.23:0/1681068903 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3ea40a2460 0x7f3ea40a4920 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4005e30 tx=0x7f3eb4005d80 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.061+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/2842664694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481458f0 0x559948147ce0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00114c0 tx=0x7f3eb0011980 comp rx=0 tx=0).stop 2024-03-24T10:41:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.061+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/2842664694 shutdown_connections 2024-03-24T10:41:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.061+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/2842664694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481458f0 0x559948147ce0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.061+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/2842664694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813cd10 0x5599481453b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.061+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/2842664694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813c3f0 0x55994813c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.061+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/2842664694 >> 172.21.15.23:0/2842664694 conn(0x559948136b20 msgr2=0x559948136f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.061+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/2842664694 shutdown_connections 2024-03-24T10:41:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.061+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/2842664694 wait complete. 2024-03-24T10:41:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3ec3dc8740 1 -- start start 2024-03-24T10:41:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3ec3dc8740 1 Processor -- start 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3ec3dc8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948142960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3ec3dc8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813cd10 0x559948142ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3ec3dc8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481433e0 0x55994815a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3ec3dc8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x559947d76430 con 0x5599481433e0 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3ec3dc8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559947d32e00 con 0x55994813cd10 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3ec3dc8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559948028590 con 0x55994813c3f0 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3ebad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948142960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3ebb59e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481433e0 0x55994815a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3eba59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813cd10 0x559948142ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3ebad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948142960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57688/0 (socket says 172.21.15.23:57688) 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3eba59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813cd10 0x559948142ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:33290/0 (socket says 172.21.15.23:33290) 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3ebad9d640 1 -- 172.21.15.23:0/4123127399 learned_addr learned my addr 172.21.15.23:0/4123127399 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.062+0000 7f3ec3dc8740 10 client.0 _ll_register_callbacks cb 0x559948097fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3ebad9d640 1 -- 172.21.15.23:0/4123127399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813cd10 msgr2=0x559948142ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3ebad9d640 1 --2- 172.21.15.23:0/4123127399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813cd10 0x559948142ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3ebad9d640 1 -- 172.21.15.23:0/4123127399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481433e0 msgr2=0x55994815a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3ebad9d640 1 --2- 172.21.15.23:0/4123127399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481433e0 0x55994815a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3ebad9d640 1 -- 172.21.15.23:0/4123127399 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb0009cd0 con 0x55994813c3f0 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3eba59c640 1 --2- 172.21.15.23:0/4123127399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813cd10 0x559948142ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3ebad9d640 1 --2- 172.21.15.23:0/4123127399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948142960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00c6a0 tx=0x7f3eac09aea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df0a83fa5a96cb2f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3e9eff5640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3e9eff5640 1 -- 172.21.15.23:0/4123127399 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac010bf0 con 0x55994813c3f0 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3e9eff5640 1 -- 172.21.15.23:0/4123127399 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f3eac0ce070 con 0x55994813c3f0 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/4123127399 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559947da27e0 con 0x55994813c3f0 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3e9eff5640 1 -- 172.21.15.23:0/4123127399 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3eac0ce230 con 0x55994813c3f0 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3ec3dc8740 10 client.25738 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/4123127399 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5599480a1bd0 con 0x55994813c3f0 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.063+0000 7f3e9eff5640 1 -- 172.21.15.23:0/4123127399 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559947da27e0 con 0x55994813c3f0 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.064+0000 7f3e9eff5640 1 -- 172.21.15.23:0/4123127399 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f3eac099db0 con 0x55994813c3f0 2024-03-24T10:41:18.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.064+0000 7f3ec3dc8740 10 client.25738 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:18.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.064+0000 7f3ec3dc8740 10 client.25738 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:41:18.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.064+0000 7f3ec3dc8740 20 client.25738 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:18.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.064+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/4123127399 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55994815e660 0x559948160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.064+0000 7f3ec3dc8740 4 client.25738 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:18.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.064+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/4123127399 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559947ddf720 con 0x55994815e660 2024-03-24T10:41:18.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.064+0000 7f3eba59c640 1 --2- 172.21.15.23:0/4123127399 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55994815e660 0x559948160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.064+0000 7f3e9eff5640 10 client.25738 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:41:18.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.064+0000 7f3eba59c640 1 --2- 172.21.15.23:0/4123127399 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55994815e660 0x559948160ab0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3ababd9388fe06ad server_cookie=aa3e6a512674ea3b in_seq=0 out_seq=0 2024-03-24T10:41:18.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.065+0000 7f3e9eff5640 1 -- 172.21.15.23:0/4123127399 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559947ddf720 con 0x55994815e660 2024-03-24T10:41:18.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.065+0000 7f3e9eff5640 10 client.25738 handle_command_reply: tid=0 2024-03-24T10:41:18.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.065+0000 7f3ec3dc8740 1 client.25738 shutdown 2024-03-24T10:41:18.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.065+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/4123127399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 msgr2=0x559948142960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.065+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/4123127399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948142960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00c6a0 tx=0x7f3eac09aea0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.065+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/4123127399 shutdown_connections 2024-03-24T10:41:18.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.065+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/4123127399 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55994815e660 0x559948160ab0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.065+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/4123127399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481433e0 0x55994815a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.065+0000 7f3e9eff5640 0 client.25738 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:41:18.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.065+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/4123127399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813cd10 0x559948142ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.065+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/4123127399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948142960 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.065+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/4123127399 >> 172.21.15.23:0/4123127399 conn(0x559948136b20 msgr2=0x559948138870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:18.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.065+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/4123127399 shutdown_connections 2024-03-24T10:41:18.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.065+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/4123127399 wait complete. 2024-03-24T10:41:18.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.066+0000 7f3ec3dc8740 20 client.25738 trim_cache size 0 max 16384 2024-03-24T10:41:18.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.092+0000 7f3ec3dc8740 1 Processor -- start 2024-03-24T10:41:18.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.092+0000 7f3ec3dc8740 1 -- start start 2024-03-24T10:41:18.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.092+0000 7f3ec3dc8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948138660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.092+0000 7f3ec3dc8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813cd10 0x559948166380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.092+0000 7f3ec3dc8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481433e0 0x559948159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.092+0000 7f3ec3dc8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x559948136650 con 0x5599481433e0 2024-03-24T10:41:18.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.092+0000 7f3ec3dc8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5599481485c0 con 0x55994813cd10 2024-03-24T10:41:18.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.092+0000 7f3ec3dc8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55994813b4e0 con 0x55994813c3f0 2024-03-24T10:41:18.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.092+0000 7f3eba59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813cd10 0x559948166380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.092+0000 7f3eba59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813cd10 0x559948166380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:33300/0 (socket says 172.21.15.23:33300) 2024-03-24T10:41:18.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.092+0000 7f3ebb59e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481433e0 0x559948159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.092+0000 7f3eba59c640 1 -- 172.21.15.23:0/3137271279 learned_addr learned my addr 172.21.15.23:0/3137271279 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:18.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.093+0000 7f3ebad9d640 1 --2- 172.21.15.23:0/3137271279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.093+0000 7f3eba59c640 1 -- 172.21.15.23:0/3137271279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 msgr2=0x559948138660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.093+0000 7f3eba59c640 1 --2- 172.21.15.23:0/3137271279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948138660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.093+0000 7f3eba59c640 1 -- 172.21.15.23:0/3137271279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481433e0 msgr2=0x559948159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.093+0000 7f3eba59c640 1 --2- 172.21.15.23:0/3137271279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481433e0 0x559948159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.093+0000 7f3eba59c640 1 -- 172.21.15.23:0/3137271279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55994803dc80 con 0x55994813cd10 2024-03-24T10:41:18.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.093+0000 7f3eba59c640 1 --2- 172.21.15.23:0/3137271279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813cd10 0x559948166380 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00097f0 tx=0x7f3eb0003f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3ab8f981a16059b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:18.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.093+0000 7f3ea0ff9640 1 -- 172.21.15.23:0/3137271279 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0003760 con 0x55994813cd10 2024-03-24T10:41:18.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.093+0000 7f3ea0ff9640 1 -- 172.21.15.23:0/3137271279 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eb0005590 con 0x55994813cd10 2024-03-24T10:41:18.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.093+0000 7f3ea0ff9640 1 -- 172.21.15.23:0/3137271279 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb00195d0 con 0x55994813cd10 2024-03-24T10:41:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.094+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/3137271279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813cd10 msgr2=0x559948166380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.094+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/3137271279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813cd10 0x559948166380 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00097f0 tx=0x7f3eb0003f20 comp rx=0 tx=0).stop 2024-03-24T10:41:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.094+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/3137271279 shutdown_connections 2024-03-24T10:41:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.094+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/3137271279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599481433e0 0x559948159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.094+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/3137271279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55994813cd10 0x559948166380 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.094+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/3137271279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948138660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.094+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/3137271279 >> 172.21.15.23:0/3137271279 conn(0x559948136b20 msgr2=0x559948137f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.094+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/3137271279 shutdown_connections 2024-03-24T10:41:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.094+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/3137271279 wait complete. 2024-03-24T10:41:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.094+0000 7f3ec3dc8740 1 -- start start 2024-03-24T10:41:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.094+0000 7f3ec3dc8740 1 Processor -- start 2024-03-24T10:41:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ec3dc8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948151b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ec3dc8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55994813cd10 0x559948152050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ec3dc8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599481433e0 0x559948152590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ec3dc8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5599481c39b0 con 0x55994813cd10 2024-03-24T10:41:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ec3dc8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5599481c3af0 con 0x5599481433e0 2024-03-24T10:41:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ec3dc8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5599481c3c80 con 0x55994813c3f0 2024-03-24T10:41:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ec3dc8740 10 client.0 _ll_register_callbacks cb 0x7f3eac010e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ebb59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599481433e0 0x559948152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ebad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948151b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ebb59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599481433e0 0x559948152590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:33306/0 (socket says 172.21.15.23:33306) 2024-03-24T10:41:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ebb59e640 1 -- 172.21.15.23:0/2155095339 learned_addr learned my addr 172.21.15.23:0/2155095339 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3eba59c640 1 --2- 172.21.15.23:0/2155095339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55994813cd10 0x559948152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ebad9d640 1 -- 172.21.15.23:0/2155095339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599481433e0 msgr2=0x559948152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ebad9d640 1 --2- 172.21.15.23:0/2155095339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599481433e0 0x559948152590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ebad9d640 1 -- 172.21.15.23:0/2155095339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55994813cd10 msgr2=0x559948152050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ebad9d640 1 --2- 172.21.15.23:0/2155095339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55994813cd10 0x559948152050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ebad9d640 1 -- 172.21.15.23:0/2155095339 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559948038de0 con 0x55994813c3f0 2024-03-24T10:41:18.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ebb59e640 1 --2- 172.21.15.23:0/2155095339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599481433e0 0x559948152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:18.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3ebad9d640 1 --2- 172.21.15.23:0/2155095339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948151b10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0971c0 tx=0x7f3eac0cc210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c605c55a31ab75a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:18.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.095+0000 7f3e9eff5640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:41:18.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3e9eff5640 1 -- 172.21.15.23:0/2155095339 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac099e20 con 0x55994813c3f0 2024-03-24T10:41:18.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/2155095339 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3eb00035f0 con 0x55994813c3f0 2024-03-24T10:41:18.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3e9eff5640 1 -- 172.21.15.23:0/2155095339 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f3eac010bf0 con 0x55994813c3f0 2024-03-24T10:41:18.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3e9eff5640 1 -- 172.21.15.23:0/2155095339 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3eac0cbd20 con 0x55994813c3f0 2024-03-24T10:41:18.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3ec3dc8740 10 client.25744 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/2155095339 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559948063920 con 0x55994813c3f0 2024-03-24T10:41:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3e9eff5640 1 -- 172.21.15.23:0/2155095339 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eb00035f0 con 0x55994813c3f0 2024-03-24T10:41:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3e9eff5640 1 -- 172.21.15.23:0/2155095339 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f3eac07f050 con 0x55994813c3f0 2024-03-24T10:41:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3ec3dc8740 10 client.25744 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3ec3dc8740 10 client.25744 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:41:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3ec3dc8740 20 client.25744 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/2155095339 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55994815e660 0x5599481c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3ec3dc8740 4 client.25744 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:41:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/2155095339 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559947e2e890 con 0x55994815e660 2024-03-24T10:41:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.096+0000 7f3eba59c640 1 --2- 172.21.15.23:0/2155095339 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55994815e660 0x5599481c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3e9eff5640 10 client.25744 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:41:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3eba59c640 1 --2- 172.21.15.23:0/2155095339 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55994815e660 0x5599481c72d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=27b72ef7b7fce614 server_cookie=e1bf58e72bd1a178 in_seq=0 out_seq=0 2024-03-24T10:41:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3e9eff5640 1 -- 172.21.15.23:0/2155095339 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559947e2e890 con 0x55994815e660 2024-03-24T10:41:18.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3e9eff5640 10 client.25744 handle_command_reply: tid=0 2024-03-24T10:41:18.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3ec3dc8740 1 client.25744 shutdown 2024-03-24T10:41:18.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/2155095339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 msgr2=0x559948151b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/2155095339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948151b10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0971c0 tx=0x7f3eac0cc210 comp rx=0 tx=0).stop 2024-03-24T10:41:18.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/2155095339 shutdown_connections 2024-03-24T10:41:18.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/2155095339 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55994815e660 0x5599481c72d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3e9eff5640 0 client.25744 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:41:18.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/2155095339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599481433e0 0x559948152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/2155095339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55994813cd10 0x559948152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3ec3dc8740 1 --2- 172.21.15.23:0/2155095339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55994813c3f0 0x559948151b10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/2155095339 >> 172.21.15.23:0/2155095339 conn(0x559948136b20 msgr2=0x5599481387e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:18.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/2155095339 shutdown_connections 2024-03-24T10:41:18.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3ec3dc8740 1 -- 172.21.15.23:0/2155095339 wait complete. 2024-03-24T10:41:18.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.097+0000 7f3ec3dc8740 20 client.25744 trim_cache size 0 max 16384 2024-03-24T10:41:18.104 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:41:18.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.098+0000 7f3ec110c640 1 -- 172.21.15.23:0/1681068903 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3ea40a2460 msgr2=0x7f3ea40a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.098+0000 7f3ec110c640 1 --2- 172.21.15.23:0/1681068903 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3ea40a2460 0x7f3ea40a4920 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4005e30 tx=0x7f3eb4005d80 comp rx=0 tx=0).stop 2024-03-24T10:41:18.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.098+0000 7f3ec110c640 1 -- 172.21.15.23:0/1681068903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ebc1016a0 msgr2=0x7f3ebc199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.098+0000 7f3ec110c640 1 --2- 172.21.15.23:0/1681068903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ebc1016a0 0x7f3ebc199ef0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00e9f0 tx=0x7f3eac00eeb0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.098+0000 7f3ec110c640 1 -- 172.21.15.23:0/1681068903 shutdown_connections 2024-03-24T10:41:18.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.098+0000 7f3ec110c640 1 --2- 172.21.15.23:0/1681068903 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3ea40a2460 0x7f3ea40a4920 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.098+0000 7f3ec110c640 1 --2- 172.21.15.23:0/1681068903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ebc1076b0 0x7f3ebc19e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.098+0000 7f3ec110c640 1 --2- 172.21.15.23:0/1681068903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ebc102060 0x7f3ebc19a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.098+0000 7f3ec110c640 1 --2- 172.21.15.23:0/1681068903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ebc1016a0 0x7f3ebc199ef0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.098+0000 7f3ec110c640 1 -- 172.21.15.23:0/1681068903 >> 172.21.15.23:0/1681068903 conn(0x7f3ebc0fd1b0 msgr2=0x7f3ebc10ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:18.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.098+0000 7f3ec110c640 1 -- 172.21.15.23:0/1681068903 shutdown_connections 2024-03-24T10:41:18.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.098+0000 7f3ec110c640 1 -- 172.21.15.23:0/1681068903 wait complete. 2024-03-24T10:41:18.110 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:41:18.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.218+0000 7f7dbf42a640 1 -- 172.21.15.23:0/1095065533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db8107760 msgr2=0x7f7db8107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.218+0000 7f7dbf42a640 1 --2- 172.21.15.23:0/1095065533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db8107760 0x7f7db8107b40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009960 tx=0x7f7db402bc20 comp rx=0 tx=0).stop 2024-03-24T10:41:18.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.218+0000 7f7dbf42a640 1 -- 172.21.15.23:0/1095065533 shutdown_connections 2024-03-24T10:41:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.218+0000 7f7dbf42a640 1 --2- 172.21.15.23:0/1095065533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7db80692d0 0x7f7db810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.218+0000 7f7dbf42a640 1 --2- 172.21.15.23:0/1095065533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7db8068910 0x7f7db8068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.218+0000 7f7dbf42a640 1 --2- 172.21.15.23:0/1095065533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db8107760 0x7f7db8107b40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.218+0000 7f7dbf42a640 1 -- 172.21.15.23:0/1095065533 >> 172.21.15.23:0/1095065533 conn(0x7f7db8077710 msgr2=0x7f7db8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.218+0000 7f7dbf42a640 1 -- 172.21.15.23:0/1095065533 shutdown_connections 2024-03-24T10:41:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.218+0000 7f7dbf42a640 1 -- 172.21.15.23:0/1095065533 wait complete. 2024-03-24T10:41:18.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.219+0000 7f7dbf42a640 1 Processor -- start 2024-03-24T10:41:18.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.219+0000 7f7dbf42a640 1 -- start start 2024-03-24T10:41:18.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.219+0000 7f7dbf42a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db8068910 0x7f7db8197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.219+0000 7f7dbf42a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7db80692d0 0x7f7db8198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.219+0000 7f7dbf42a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7db8107760 0x7f7db819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.219+0000 7f7dbc9c5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7db80692d0 0x7f7db8198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.219+0000 7f7dbf42a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f7db819cbf0 con 0x7f7db80692d0 2024-03-24T10:41:18.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.219+0000 7f7dbd1c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db8068910 0x7f7db8197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.219+0000 7f7dbc9c5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7db80692d0 0x7f7db8198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35978/0 (socket says 172.21.15.23:35978) 2024-03-24T10:41:18.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.219+0000 7f7dbd1c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db8068910 0x7f7db8197d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:33322/0 (socket says 172.21.15.23:33322) 2024-03-24T10:41:18.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.219+0000 7f7dbc9c5640 1 -- 172.21.15.23:0/711005146 learned_addr learned my addr 172.21.15.23:0/711005146 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:18.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.219+0000 7f7dbf42a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7db819cd60 con 0x7f7db8068910 2024-03-24T10:41:18.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.219+0000 7f7dbf42a640 1 -- 172.21.15.23:0/711005146 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7db819cf20 con 0x7f7db8107760 2024-03-24T10:41:18.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.219+0000 7f7dbd9c7640 1 --2- 172.21.15.23:0/711005146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7db8107760 0x7f7db819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.220+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/711005146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7db8107760 msgr2=0x7f7db819c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.220+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/711005146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7db8107760 0x7f7db819c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.220+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/711005146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7db80692d0 msgr2=0x7f7db8198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.220+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/711005146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7db80692d0 0x7f7db8198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.220+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/711005146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da8009490 con 0x7f7db8068910 2024-03-24T10:41:18.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.220+0000 7f7dbc9c5640 1 --2- 172.21.15.23:0/711005146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7db80692d0 0x7f7db8198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:18.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.220+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/711005146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db8068910 0x7f7db8197d30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7db4006010 tx=0x7f7db402c5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:18.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.220+0000 7f7dae7fc640 1 -- 172.21.15.23:0/711005146 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db40042d0 con 0x7f7db8068910 2024-03-24T10:41:18.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.220+0000 7f7dae7fc640 1 -- 172.21.15.23:0/711005146 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7db4004430 con 0x7f7db8068910 2024-03-24T10:41:18.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.220+0000 7f7dbf42a640 1 -- 172.21.15.23:0/711005146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7db4009560 con 0x7f7db8068910 2024-03-24T10:41:18.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.220+0000 7f7dae7fc640 1 -- 172.21.15.23:0/711005146 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db4003b50 con 0x7f7db8068910 2024-03-24T10:41:18.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.220+0000 7f7dbf42a640 1 -- 172.21.15.23:0/711005146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db81a48a0 con 0x7f7db8068910 2024-03-24T10:41:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.223+0000 7f7dc200a740 1 Processor -- start 2024-03-24T10:41:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.223+0000 7f7dc200a740 1 -- start start 2024-03-24T10:41:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.223+0000 7f7dae7fc640 1 -- 172.21.15.23:0/711005146 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 137040+0+0 (secure 0 0 0) 0x7f7db4003cb0 con 0x7f7db8068910 2024-03-24T10:41:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.223+0000 7f7dc200a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934fae37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.223+0000 7f7dc200a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934faec3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.223+0000 7f7dc200a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faeece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.223+0000 7f7dc200a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55934fa51840 con 0x55934faec8f0 2024-03-24T10:41:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.223+0000 7f7dc200a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55934fa4ed70 con 0x55934fae33f0 2024-03-24T10:41:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.223+0000 7f7dc200a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55934fa4a550 con 0x55934fae3d10 2024-03-24T10:41:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.223+0000 7f7dbd1c6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934faec3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.223+0000 7f7dbd1c6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934faec3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57744/0 (socket says 172.21.15.23:57744) 2024-03-24T10:41:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.223+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/930822140 learned_addr learned my addr 172.21.15.23:0/930822140 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7dbd9c7640 1 --2- 172.21.15.23:0/930822140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934fae37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7dbc9c5640 1 --2- 172.21.15.23:0/930822140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faeece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/930822140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 msgr2=0x55934fae37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/930822140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934fae37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/930822140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 msgr2=0x55934faeece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/930822140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faeece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/930822140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55934f9dfde0 con 0x55934fae3d10 2024-03-24T10:41:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7dbd9c7640 1 --2- 172.21.15.23:0/930822140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934fae37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7dbc9c5640 1 --2- 172.21.15.23:0/930822140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faeece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:41:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/930822140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934faec3b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7db40ed6c0 tx=0x7f7db40f8220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e752b95eb2c8b03b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7dae7fc640 1 --2- 172.21.15.23:0/711005146 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7da40a2460 0x7f7da40a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7da37fe640 1 -- 172.21.15.23:0/930822140 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db40b8c60 con 0x55934fae3d10 2024-03-24T10:41:18.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7da37fe640 1 -- 172.21.15.23:0/930822140 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7db40bc790 con 0x55934fae3d10 2024-03-24T10:41:18.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7da37fe640 1 -- 172.21.15.23:0/930822140 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db4096140 con 0x55934fae3d10 2024-03-24T10:41:18.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.224+0000 7f7dbd9c7640 1 --2- 172.21.15.23:0/711005146 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7da40a2460 0x7f7da40a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.225+0000 7f7dae7fc640 1 -- 172.21.15.23:0/711005146 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f7db40f0100 con 0x7f7db8068910 2024-03-24T10:41:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.226+0000 7f7dc200a740 1 -- 172.21.15.23:0/930822140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 msgr2=0x55934faec3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.226+0000 7f7dbd9c7640 1 --2- 172.21.15.23:0/711005146 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7da40a2460 0x7f7da40a4920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7db0005fd0 tx=0x7f7db0009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.226+0000 7f7dc200a740 1 --2- 172.21.15.23:0/930822140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934faec3b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7db40ed6c0 tx=0x7f7db40f8220 comp rx=0 tx=0).stop 2024-03-24T10:41:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.226+0000 7f7dc200a740 1 -- 172.21.15.23:0/930822140 shutdown_connections 2024-03-24T10:41:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.226+0000 7f7dc200a740 1 --2- 172.21.15.23:0/930822140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faeece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.226+0000 7f7dc200a740 1 --2- 172.21.15.23:0/930822140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934faec3b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.226+0000 7f7dc200a740 1 --2- 172.21.15.23:0/930822140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934fae37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.226+0000 7f7dc200a740 1 -- 172.21.15.23:0/930822140 >> 172.21.15.23:0/930822140 conn(0x55934faddb20 msgr2=0x55934faddf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.226+0000 7f7dc200a740 1 -- 172.21.15.23:0/930822140 shutdown_connections 2024-03-24T10:41:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.226+0000 7f7dc200a740 1 -- 172.21.15.23:0/930822140 wait complete. 2024-03-24T10:41:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.226+0000 7f7dc200a740 1 -- start start 2024-03-24T10:41:18.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.226+0000 7f7dc200a740 1 Processor -- start 2024-03-24T10:41:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.227+0000 7f7dc200a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934fae99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.227+0000 7f7dc200a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934fae9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.227+0000 7f7dc200a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faea440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.227+0000 7f7dc200a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55934f71d430 con 0x55934faec8f0 2024-03-24T10:41:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.227+0000 7f7dc200a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55934f6d9e00 con 0x55934fae33f0 2024-03-24T10:41:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.227+0000 7f7dbd1c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934fae99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.227+0000 7f7dc200a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55934f9cf590 con 0x55934fae3d10 2024-03-24T10:41:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.227+0000 7f7dbd1c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934fae99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:33340/0 (socket says 172.21.15.23:33340) 2024-03-24T10:41:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.227+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/1110498548 learned_addr learned my addr 172.21.15.23:0/1110498548 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.227+0000 7f7dbd9c7640 1 --2- 172.21.15.23:0/1110498548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faea440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.227+0000 7f7dbc9c5640 1 --2- 172.21.15.23:0/1110498548 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934fae9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.227+0000 7f7dc200a740 10 client.0 _ll_register_callbacks cb 0x55934fa3efc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.228+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/1110498548 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 msgr2=0x55934fae9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.228+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/1110498548 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934fae9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.228+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/1110498548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 msgr2=0x55934faea440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.228+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/1110498548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faea440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.228+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/1110498548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db81a48a0 con 0x55934fae33f0 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.228+0000 7f7dbd9c7640 1 --2- 172.21.15.23:0/1110498548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faea440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.228+0000 7f7dc200a740 1 -- 172.21.15.23:0/1110498548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55934f7497e0 con 0x55934fae33f0 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.228+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/1110498548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934fae99c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7db402c160 tx=0x7f7db40ed6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a57e79ec0ef5b92a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.228+0000 7f7da0ff9640 10 client.25769 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.228+0000 7f7da0ff9640 1 -- 172.21.15.23:0/1110498548 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db40dc040 con 0x55934fae33f0 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.228+0000 7f7da0ff9640 1 -- 172.21.15.23:0/1110498548 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f7db40bd0c0 con 0x55934fae33f0 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.228+0000 7f7da0ff9640 1 -- 172.21.15.23:0/1110498548 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7db40f36c0 con 0x55934fae33f0 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.228+0000 7f7dc200a740 10 client.25769 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.228+0000 7f7dc200a740 1 -- 172.21.15.23:0/1110498548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=23}) -- 0x55934fa48bd0 con 0x55934fae33f0 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.229+0000 7f7da0ff9640 1 -- 172.21.15.23:0/1110498548 <== mon.2 v2:172.21.15.116:3300/0 4 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f7db403c070 con 0x55934fae33f0 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.229+0000 7f7dc200a740 10 client.25769 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:18.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.229+0000 7f7dc200a740 10 client.25769 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.229+0000 7f7dc200a740 20 client.25769 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.229+0000 7f7dc200a740 1 --2- 172.21.15.23:0/1110498548 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55934fafc1c0 0x55934fafe5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.229+0000 7f7dc200a740 4 client.25769 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.229+0000 7f7dc200a740 1 -- 172.21.15.23:0/1110498548 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55934f786720 con 0x55934fafc1c0 2024-03-24T10:41:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.229+0000 7f7dbc9c5640 1 --2- 172.21.15.23:0/1110498548 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55934fafc1c0 0x55934fafe5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.230+0000 7f7da0ff9640 10 client.25769 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.230+0000 7f7dbc9c5640 1 --2- 172.21.15.23:0/1110498548 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55934fafc1c0 0x55934fafe5b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=348a0ce6d482a8f7 server_cookie=66053057a53e70af in_seq=0 out_seq=0 2024-03-24T10:41:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.230+0000 7f7da0ff9640 1 -- 172.21.15.23:0/1110498548 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55934f786720 con 0x55934fafc1c0 2024-03-24T10:41:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.230+0000 7f7da0ff9640 10 client.25769 handle_command_reply: tid=0 2024-03-24T10:41:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.230+0000 7f7dc200a740 1 client.25769 shutdown 2024-03-24T10:41:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.230+0000 7f7dc200a740 1 -- 172.21.15.23:0/1110498548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 msgr2=0x55934fae99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.230+0000 7f7dc200a740 1 --2- 172.21.15.23:0/1110498548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934fae99c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7db402c160 tx=0x7f7db40ed6f0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.230+0000 7f7dc200a740 1 -- 172.21.15.23:0/1110498548 shutdown_connections 2024-03-24T10:41:18.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.230+0000 7f7dc200a740 1 --2- 172.21.15.23:0/1110498548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faea440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.230+0000 7f7dc200a740 1 --2- 172.21.15.23:0/1110498548 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55934fafc1c0 0x55934fafe5b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.230+0000 7f7da0ff9640 0 client.25769 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:18.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.230+0000 7f7dc200a740 1 --2- 172.21.15.23:0/1110498548 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934fae9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.230+0000 7f7dc200a740 1 --2- 172.21.15.23:0/1110498548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934fae99c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.230+0000 7f7dc200a740 1 -- 172.21.15.23:0/1110498548 >> 172.21.15.23:0/1110498548 conn(0x55934faddb20 msgr2=0x55934fadf870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:18.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.231+0000 7f7dc200a740 1 -- 172.21.15.23:0/1110498548 shutdown_connections 2024-03-24T10:41:18.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.231+0000 7f7dc200a740 1 -- 172.21.15.23:0/1110498548 wait complete. 2024-03-24T10:41:18.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.231+0000 7f7dc200a740 20 client.25769 trim_cache size 0 max 16384 2024-03-24T10:41:18.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.244+0000 7f7dc200a740 1 Processor -- start 2024-03-24T10:41:18.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.244+0000 7f7dc200a740 1 -- start start 2024-03-24T10:41:18.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.244+0000 7f7dc200a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934faf7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.244+0000 7f7dc200a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934fae3d10 0x55934fadcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.244+0000 7f7dc200a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934faec8f0 0x55934fb03e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.244+0000 7f7dc200a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55934faef2b0 con 0x55934fae3d10 2024-03-24T10:41:18.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.244+0000 7f7dc200a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55934fadeb20 con 0x55934fae33f0 2024-03-24T10:41:18.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.244+0000 7f7dc200a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55934faf8480 con 0x55934faec8f0 2024-03-24T10:41:18.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.244+0000 7f7dbd1c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934faf7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.244+0000 7f7dbd9c7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934faec8f0 0x55934fb03e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.244+0000 7f7dbd1c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934faf7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:33346/0 (socket says 172.21.15.23:33346) 2024-03-24T10:41:18.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.244+0000 7f7dbd9c7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934faec8f0 0x55934fb03e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57762/0 (socket says 172.21.15.23:57762) 2024-03-24T10:41:18.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.244+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/3966398064 learned_addr learned my addr 172.21.15.23:0/3966398064 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:18.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.244+0000 7f7dbc9c5640 1 --2- 172.21.15.23:0/3966398064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934fae3d10 0x55934fadcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.245+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/3966398064 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934faec8f0 msgr2=0x55934fb03e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.245+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/3966398064 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934faec8f0 0x55934fb03e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.245+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/3966398064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934fae3d10 msgr2=0x55934fadcb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.245+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/3966398064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934fae3d10 0x55934fadcb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.245+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/3966398064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55934f9e4c80 con 0x55934fae33f0 2024-03-24T10:41:18.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.245+0000 7f7dbd9c7640 1 --2- 172.21.15.23:0/3966398064 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934faec8f0 0x55934fb03e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:18.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.245+0000 7f7dbc9c5640 1 --2- 172.21.15.23:0/3966398064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934fae3d10 0x55934fadcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:18.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.245+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/3966398064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934faf7f40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7db4005ab0 tx=0x7f7db40e0000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dc89416bfc2f3da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:18.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.245+0000 7f7da2ffd640 1 -- 172.21.15.23:0/3966398064 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db40dc070 con 0x55934fae33f0 2024-03-24T10:41:18.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.245+0000 7f7da2ffd640 1 -- 172.21.15.23:0/3966398064 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55934f7497e0 con 0x55934fae33f0 2024-03-24T10:41:18.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.245+0000 7f7da2ffd640 1 -- 172.21.15.23:0/3966398064 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db40b8c60 con 0x55934fae33f0 2024-03-24T10:41:18.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.246+0000 7f7dc200a740 1 -- 172.21.15.23:0/3966398064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 msgr2=0x55934faf7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.246+0000 7f7dc200a740 1 --2- 172.21.15.23:0/3966398064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934faf7f40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7db4005ab0 tx=0x7f7db40e0000 comp rx=0 tx=0).stop 2024-03-24T10:41:18.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.246+0000 7f7dc200a740 1 -- 172.21.15.23:0/3966398064 shutdown_connections 2024-03-24T10:41:18.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.246+0000 7f7dc200a740 1 --2- 172.21.15.23:0/3966398064 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934faec8f0 0x55934fb03e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.246+0000 7f7dc200a740 1 --2- 172.21.15.23:0/3966398064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934fae3d10 0x55934fadcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.246+0000 7f7dc200a740 1 --2- 172.21.15.23:0/3966398064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934faf7f40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.246+0000 7f7dc200a740 1 -- 172.21.15.23:0/3966398064 >> 172.21.15.23:0/3966398064 conn(0x55934faddb20 msgr2=0x55934fadc430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:18.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.246+0000 7f7dc200a740 1 -- 172.21.15.23:0/3966398064 shutdown_connections 2024-03-24T10:41:18.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.246+0000 7f7dc200a740 1 -- 172.21.15.23:0/3966398064 wait complete. 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.246+0000 7f7dc200a740 1 -- start start 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.246+0000 7f7dc200a740 1 Processor -- start 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.247+0000 7f7dc200a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934faf4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.247+0000 7f7dc200a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934faefba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.247+0000 7f7dc200a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faf00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.247+0000 7f7dc200a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55934faf07b0 con 0x55934faec8f0 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.247+0000 7f7dc200a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55934faf0920 con 0x55934fae33f0 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.247+0000 7f7dc200a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55934fb614b0 con 0x55934fae3d10 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.247+0000 7f7dc200a740 10 client.0 _ll_register_callbacks cb 0x7f7db4101070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.247+0000 7f7dbd1c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934faf4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.247+0000 7f7dbd1c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934faf4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:33356/0 (socket says 172.21.15.23:33356) 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.247+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/2319399895 learned_addr learned my addr 172.21.15.23:0/2319399895 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.247+0000 7f7dbd9c7640 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faf00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.247+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/2319399895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 msgr2=0x55934faefba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7dbc9c5640 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934faefba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934faefba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/2319399895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 msgr2=0x55934faf00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faf00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7dbd1c6640 1 -- 172.21.15.23:0/2319399895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da8020ba0 con 0x55934fae33f0 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7dbd9c7640 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faf00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7dbc9c5640 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934faefba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7dbd1c6640 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934faf4830 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7db40baf10 tx=0x7f7db4005670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3e22e091cf3a57d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7da0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7da0ff9640 1 -- 172.21.15.23:0/2319399895 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db40dc040 con 0x55934fae33f0 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7dc200a740 1 -- 172.21.15.23:0/2319399895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7db40bc790 con 0x55934fae33f0 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7da0ff9640 1 -- 172.21.15.23:0/2319399895 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f7db40c0b10 con 0x55934fae33f0 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7da0ff9640 1 -- 172.21.15.23:0/2319399895 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7db40f8720 con 0x55934fae33f0 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7dc200a740 10 client.25775 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7dc200a740 1 -- 172.21.15.23:0/2319399895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x7f7db0003d20 con 0x55934fae33f0 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7da0ff9640 1 -- 172.21.15.23:0/2319399895 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7db40bc790 con 0x55934fae33f0 2024-03-24T10:41:18.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.248+0000 7f7da0ff9640 1 -- 172.21.15.23:0/2319399895 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f7db403c070 con 0x55934fae33f0 2024-03-24T10:41:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.249+0000 7f7dc200a740 10 client.25775 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.249+0000 7f7dc200a740 10 client.25775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.249+0000 7f7dc200a740 20 client.25775 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.249+0000 7f7dc200a740 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55934fb6b0b0 0x55934fb6d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.249+0000 7f7dc200a740 4 client.25775 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:41:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.249+0000 7f7dc200a740 1 -- 172.21.15.23:0/2319399895 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55934f7d5890 con 0x55934fb6b0b0 2024-03-24T10:41:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.249+0000 7f7dbc9c5640 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55934fb6b0b0 0x55934fb6d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.249+0000 7f7da0ff9640 10 client.25775 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.249+0000 7f7dbc9c5640 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55934fb6b0b0 0x55934fb6d4a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53c8a855e6260766 server_cookie=544ebdbaa9daa47c in_seq=0 out_seq=0 2024-03-24T10:41:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.249+0000 7f7da0ff9640 1 -- 172.21.15.23:0/2319399895 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55934f7d5890 con 0x55934fb6b0b0 2024-03-24T10:41:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.249+0000 7f7da0ff9640 10 client.25775 handle_command_reply: tid=0 2024-03-24T10:41:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.249+0000 7f7dc200a740 1 client.25775 shutdown 2024-03-24T10:41:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.249+0000 7f7dc200a740 1 -- 172.21.15.23:0/2319399895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 msgr2=0x55934faf4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.249+0000 7f7dc200a740 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934faf4830 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7db40baf10 tx=0x7f7db4005670 comp rx=0 tx=0).stop 2024-03-24T10:41:18.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dc200a740 1 -- 172.21.15.23:0/2319399895 shutdown_connections 2024-03-24T10:41:18.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dc200a740 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55934faec8f0 0x55934faf00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dc200a740 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55934fb6b0b0 0x55934fb6d4a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dc200a740 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55934fae3d10 0x55934faefba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7da0ff9640 0 client.25775 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:18.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dc200a740 1 --2- 172.21.15.23:0/2319399895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55934fae33f0 0x55934faf4830 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dc200a740 1 -- 172.21.15.23:0/2319399895 >> 172.21.15.23:0/2319399895 conn(0x55934faddb20 msgr2=0x55934fadccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:18.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dc200a740 1 -- 172.21.15.23:0/2319399895 shutdown_connections 2024-03-24T10:41:18.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dc200a740 1 -- 172.21.15.23:0/2319399895 wait complete. 2024-03-24T10:41:18.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dc200a740 20 client.25775 trim_cache size 0 max 16384 2024-03-24T10:41:18.255 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:41:18.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dbf42a640 1 -- 172.21.15.23:0/711005146 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7da40a2460 msgr2=0x7f7da40a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dbf42a640 1 --2- 172.21.15.23:0/711005146 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7da40a2460 0x7f7da40a4920 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7db0005fd0 tx=0x7f7db0009500 comp rx=0 tx=0).stop 2024-03-24T10:41:18.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dbf42a640 1 -- 172.21.15.23:0/711005146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db8068910 msgr2=0x7f7db8197d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:18.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dbf42a640 1 --2- 172.21.15.23:0/711005146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db8068910 0x7f7db8197d30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7db4006010 tx=0x7f7db402c5c0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dbf42a640 1 -- 172.21.15.23:0/711005146 shutdown_connections 2024-03-24T10:41:18.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dbf42a640 1 --2- 172.21.15.23:0/711005146 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7da40a2460 0x7f7da40a4920 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dbf42a640 1 --2- 172.21.15.23:0/711005146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7db8107760 0x7f7db819c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dbf42a640 1 --2- 172.21.15.23:0/711005146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7db80692d0 0x7f7db8198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dbf42a640 1 --2- 172.21.15.23:0/711005146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db8068910 0x7f7db8197d30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:18.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.250+0000 7f7dbf42a640 1 -- 172.21.15.23:0/711005146 >> 172.21.15.23:0/711005146 conn(0x7f7db8077710 msgr2=0x7f7db80fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:18.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.251+0000 7f7dbf42a640 1 -- 172.21.15.23:0/711005146 shutdown_connections 2024-03-24T10:41:18.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:18.251+0000 7f7dbf42a640 1 -- 172.21.15.23:0/711005146 wait complete. 2024-03-24T10:41:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:18 smithi023 ceph-mon[32466]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-24T10:41:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:18 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/849502430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:41:19.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:18 smithi117 ceph-mon[36368]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-24T10:41:19.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:18 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/849502430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:41:19.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:18 smithi116 ceph-mon[36248]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-24T10:41:19.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:18 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/849502430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:41:19.262 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:41:19.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:19.263 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:41:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.411+0000 7f1259bc2640 1 -- 172.21.15.23:0/1493765656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1254105b40 msgr2=0x7f125410d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.411+0000 7f1259bc2640 1 --2- 172.21.15.23:0/1493765656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1254105b40 0x7f125410d450 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f124c009ca0 tx=0x7f124c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.411+0000 7f1259bc2640 1 -- 172.21.15.23:0/1493765656 shutdown_connections 2024-03-24T10:41:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.411+0000 7f1259bc2640 1 --2- 172.21.15.23:0/1493765656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1254105b40 0x7f125410d450 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.411+0000 7f1259bc2640 1 --2- 172.21.15.23:0/1493765656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12541051a0 0x7f1254105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.411+0000 7f1259bc2640 1 --2- 172.21.15.23:0/1493765656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1254100b50 0x7f1254100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.411+0000 7f1259bc2640 1 -- 172.21.15.23:0/1493765656 >> 172.21.15.23:0/1493765656 conn(0x7f1254077500 msgr2=0x7f1254077910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.411+0000 7f1259bc2640 1 -- 172.21.15.23:0/1493765656 shutdown_connections 2024-03-24T10:41:19.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.411+0000 7f1259bc2640 1 -- 172.21.15.23:0/1493765656 wait complete. 2024-03-24T10:41:19.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.412+0000 7f1259bc2640 1 Processor -- start 2024-03-24T10:41:19.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.412+0000 7f1259bc2640 1 -- start start 2024-03-24T10:41:19.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.412+0000 7f1259bc2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1254100b50 0x7f1254198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.412+0000 7f1259bc2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12541051a0 0x7f1254199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.412+0000 7f1259bc2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1254105b40 0x7f12541a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.412+0000 7f1259bc2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f1254199850 con 0x7f12541051a0 2024-03-24T10:41:19.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.412+0000 7f1259bc2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f12541999c0 con 0x7f1254105b40 2024-03-24T10:41:19.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.412+0000 7f1259bc2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f12541a4130 con 0x7f1254100b50 2024-03-24T10:41:19.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.412+0000 7f12537fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1254100b50 0x7f1254198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.412+0000 7f1252ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12541051a0 0x7f1254199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.412+0000 7f12537fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1254100b50 0x7f1254198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57790/0 (socket says 172.21.15.23:57790) 2024-03-24T10:41:19.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.412+0000 7f12537fe640 1 -- 172.21.15.23:0/589579882 learned_addr learned my addr 172.21.15.23:0/589579882 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.412+0000 7f1253fff640 1 --2- 172.21.15.23:0/589579882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1254105b40 0x7f12541a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.413+0000 7f12537fe640 1 -- 172.21.15.23:0/589579882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1254105b40 msgr2=0x7f12541a3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.413+0000 7f12537fe640 1 --2- 172.21.15.23:0/589579882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1254105b40 0x7f12541a3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.413+0000 7f12537fe640 1 -- 172.21.15.23:0/589579882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12541051a0 msgr2=0x7f1254199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.413+0000 7f12537fe640 1 --2- 172.21.15.23:0/589579882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12541051a0 0x7f1254199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.413+0000 7f12537fe640 1 -- 172.21.15.23:0/589579882 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1244009490 con 0x7f1254100b50 2024-03-24T10:41:19.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.413+0000 7f1252ffd640 1 --2- 172.21.15.23:0/589579882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12541051a0 0x7f1254199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:19.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.413+0000 7f12537fe640 1 --2- 172.21.15.23:0/589579882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1254100b50 0x7f1254198b20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f124400ea60 tx=0x7f124400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.413+0000 7f1250ff9640 1 -- 172.21.15.23:0/589579882 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124400cde0 con 0x7f1254100b50 2024-03-24T10:41:19.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.413+0000 7f1250ff9640 1 -- 172.21.15.23:0/589579882 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12440035f0 con 0x7f1254100b50 2024-03-24T10:41:19.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.413+0000 7f1250ff9640 1 -- 172.21.15.23:0/589579882 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1244010690 con 0x7f1254100b50 2024-03-24T10:41:19.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.413+0000 7f1259bc2640 1 -- 172.21.15.23:0/589579882 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f124c009900 con 0x7f1254100b50 2024-03-24T10:41:19.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.413+0000 7f1259bc2640 1 -- 172.21.15.23:0/589579882 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1254196ca0 con 0x7f1254100b50 2024-03-24T10:41:19.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.416+0000 7f125c79b740 1 Processor -- start 2024-03-24T10:41:19.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.416+0000 7f125c79b740 1 -- start start 2024-03-24T10:41:19.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.416+0000 7f125c79b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996ad9cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.416+0000 7f125c79b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ad9d080 0x55996ada5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.416+0000 7f125c79b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ada5c60 0x55996ada8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.416+0000 7f125c79b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55996ad0a530 con 0x55996ada5c60 2024-03-24T10:41:19.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.416+0000 7f125c79b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55996ad07a60 con 0x55996ad9d080 2024-03-24T10:41:19.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.416+0000 7f125c79b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55996ad03240 con 0x55996ad9c760 2024-03-24T10:41:19.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.416+0000 7f1250ff9640 1 -- 172.21.15.23:0/589579882 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 24) ==== 137040+0+0 (secure 0 0 0) 0x7f1244003af0 con 0x7f1254100b50 2024-03-24T10:41:19.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.416+0000 7f1253fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996ad9cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.416+0000 7f1253fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996ad9cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57792/0 (socket says 172.21.15.23:57792) 2024-03-24T10:41:19.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.416+0000 7f1253fff640 1 -- 172.21.15.23:0/2916550551 learned_addr learned my addr 172.21.15.23:0/2916550551 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.416+0000 7f1252ffd640 1 --2- 172.21.15.23:0/2916550551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ada5c60 0x55996ada8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.416+0000 7f12537fe640 1 --2- 172.21.15.23:0/2916550551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ad9d080 0x55996ada5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.417+0000 7f1253fff640 1 -- 172.21.15.23:0/2916550551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ad9d080 msgr2=0x55996ada5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.417+0000 7f1253fff640 1 --2- 172.21.15.23:0/2916550551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ad9d080 0x55996ada5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.417+0000 7f1253fff640 1 -- 172.21.15.23:0/2916550551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ada5c60 msgr2=0x55996ada8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.417+0000 7f1253fff640 1 --2- 172.21.15.23:0/2916550551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ada5c60 0x55996ada8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.417+0000 7f1253fff640 1 -- 172.21.15.23:0/2916550551 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55996ad018c0 con 0x55996ad9c760 2024-03-24T10:41:19.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.417+0000 7f1252ffd640 1 --2- 172.21.15.23:0/2916550551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ada5c60 0x55996ada8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:19.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.417+0000 7f1250ff9640 1 --2- 172.21.15.23:0/589579882 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f123c0a2460 0x7f123c0a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.417+0000 7f1253fff640 1 --2- 172.21.15.23:0/2916550551 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996ad9cb40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f124c02c700 tx=0x7f124c005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25b4ee777c54bf05 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.417+0000 7f1239ffb640 1 -- 172.21.15.23:0/2916550551 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c034440 con 0x55996ad9c760 2024-03-24T10:41:19.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.417+0000 7f1250ff9640 1 -- 172.21.15.23:0/589579882 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f1244014070 con 0x7f1254100b50 2024-03-24T10:41:19.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.418+0000 7f1239ffb640 1 -- 172.21.15.23:0/2916550551 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f124c0345a0 con 0x55996ad9c760 2024-03-24T10:41:19.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.418+0000 7f1239ffb640 1 -- 172.21.15.23:0/2916550551 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c032900 con 0x55996ad9c760 2024-03-24T10:41:19.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.418+0000 7f12537fe640 1 --2- 172.21.15.23:0/589579882 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f123c0a2460 0x7f123c0a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.418+0000 7f125c79b740 1 -- 172.21.15.23:0/2916550551 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 msgr2=0x55996ad9cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.418+0000 7f125c79b740 1 --2- 172.21.15.23:0/2916550551 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996ad9cb40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f124c02c700 tx=0x7f124c005f80 comp rx=0 tx=0).stop 2024-03-24T10:41:19.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.418+0000 7f12537fe640 1 --2- 172.21.15.23:0/589579882 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f123c0a2460 0x7f123c0a4920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f12440cbcd0 tx=0x7f12440d40e0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.418+0000 7f125c79b740 1 -- 172.21.15.23:0/2916550551 shutdown_connections 2024-03-24T10:41:19.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.418+0000 7f125c79b740 1 --2- 172.21.15.23:0/2916550551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ada5c60 0x55996ada8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.418+0000 7f125c79b740 1 --2- 172.21.15.23:0/2916550551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ad9d080 0x55996ada5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.418+0000 7f125c79b740 1 --2- 172.21.15.23:0/2916550551 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996ad9cb40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.418+0000 7f125c79b740 1 -- 172.21.15.23:0/2916550551 >> 172.21.15.23:0/2916550551 conn(0x55996ad96e90 msgr2=0x55996ad97270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.418+0000 7f125c79b740 1 -- 172.21.15.23:0/2916550551 shutdown_connections 2024-03-24T10:41:19.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.418+0000 7f125c79b740 1 -- 172.21.15.23:0/2916550551 wait complete. 2024-03-24T10:41:19.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.419+0000 7f125c79b740 1 -- start start 2024-03-24T10:41:19.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.419+0000 7f125c79b740 1 Processor -- start 2024-03-24T10:41:19.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.419+0000 7f125c79b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996ada2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.419+0000 7f125c79b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9d080 0x55996ada31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.419+0000 7f125c79b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ada5c60 0x55996ada3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.419+0000 7f125c79b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55996aca7f80 con 0x55996ad9d080 2024-03-24T10:41:19.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.419+0000 7f125c79b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55996a9d64b0 con 0x55996ada5c60 2024-03-24T10:41:19.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.419+0000 7f125c79b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55996ac88ad0 con 0x55996ad9c760 2024-03-24T10:41:19.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.419+0000 7f125c79b740 10 client.0 _ll_register_callbacks cb 0x55996acbbb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:19.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1253fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996ada2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1253fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996ada2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57804/0 (socket says 172.21.15.23:57804) 2024-03-24T10:41:19.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1253fff640 1 -- 172.21.15.23:0/494787685 learned_addr learned my addr 172.21.15.23:0/494787685 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1252ffd640 1 --2- 172.21.15.23:0/494787685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9d080 0x55996ada31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1253fff640 1 --2- 172.21.15.23:0/494787685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ada5c60 0x55996ada3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1253fff640 1 -- 172.21.15.23:0/494787685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ada5c60 msgr2=0x55996ada3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1253fff640 1 --2- 172.21.15.23:0/494787685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ada5c60 0x55996ada3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1253fff640 1 -- 172.21.15.23:0/494787685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9d080 msgr2=0x55996ada31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1253fff640 1 --2- 172.21.15.23:0/494787685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9d080 0x55996ada31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1253fff640 1 -- 172.21.15.23:0/494787685 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f125410dd50 con 0x55996ad9c760 2024-03-24T10:41:19.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1252ffd640 1 --2- 172.21.15.23:0/494787685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9d080 0x55996ada31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:19.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1253fff640 1 --2- 172.21.15.23:0/494787685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996ada2c80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f124c004af0 tx=0x7f124c02e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e560edf6e7e6487 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1232ffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:41:19.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1232ffd640 1 -- 172.21.15.23:0/494787685 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c044b10 con 0x55996ad9c760 2024-03-24T10:41:19.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f1232ffd640 1 -- 172.21.15.23:0/494787685 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f124c044c70 con 0x55996ad9c760 2024-03-24T10:41:19.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.420+0000 7f125c79b740 1 -- 172.21.15.23:0/494787685 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55996ad277a0 con 0x55996ad9c760 2024-03-24T10:41:19.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.421+0000 7f1232ffd640 1 -- 172.21.15.23:0/494787685 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f124c044e30 con 0x55996ad9c760 2024-03-24T10:41:19.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.421+0000 7f125c79b740 10 client.25771 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:19.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.421+0000 7f125c79b740 1 -- 172.21.15.23:0/494787685 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55996acb96d0 con 0x55996ad9c760 2024-03-24T10:41:19.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.421+0000 7f1232ffd640 1 -- 172.21.15.23:0/494787685 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55996ad277a0 con 0x55996ad9c760 2024-03-24T10:41:19.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.421+0000 7f1232ffd640 1 -- 172.21.15.23:0/494787685 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f124c004430 con 0x55996ad9c760 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.421+0000 7f125c79b740 10 client.25771 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.421+0000 7f125c79b740 10 client.25771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.421+0000 7f125c79b740 20 client.25771 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.421+0000 7f125c79b740 1 --2- 172.21.15.23:0/494787685 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55996adb5960 0x55996adb7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.421+0000 7f125c79b740 4 client.25771 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.421+0000 7f125c79b740 1 -- 172.21.15.23:0/494787685 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55996acf91f0 con 0x55996adb5960 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.421+0000 7f1252ffd640 1 --2- 172.21.15.23:0/494787685 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55996adb5960 0x55996adb7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.422+0000 7f1232ffd640 10 client.25771 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.422+0000 7f1252ffd640 1 --2- 172.21.15.23:0/494787685 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55996adb5960 0x55996adb7d80 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baf4e3ebdb6e732e server_cookie=afad037cd0ceba86 in_seq=0 out_seq=0 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.422+0000 7f1232ffd640 1 -- 172.21.15.23:0/494787685 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55996acf91f0 con 0x55996adb5960 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.422+0000 7f1232ffd640 10 client.25771 handle_command_reply: tid=0 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.422+0000 7f125c79b740 1 client.25771 shutdown 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.423+0000 7f125c79b740 1 -- 172.21.15.23:0/494787685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 msgr2=0x55996ada2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.423+0000 7f125c79b740 1 --2- 172.21.15.23:0/494787685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996ada2c80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f124c004af0 tx=0x7f124c02e040 comp rx=0 tx=0).stop 2024-03-24T10:41:19.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.423+0000 7f125c79b740 1 -- 172.21.15.23:0/494787685 shutdown_connections 2024-03-24T10:41:19.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.423+0000 7f125c79b740 1 --2- 172.21.15.23:0/494787685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ada5c60 0x55996ada3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.423+0000 7f125c79b740 1 --2- 172.21.15.23:0/494787685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9d080 0x55996ada31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.423+0000 7f125c79b740 1 --2- 172.21.15.23:0/494787685 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55996adb5960 0x55996adb7d80 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.423+0000 7f125c79b740 1 --2- 172.21.15.23:0/494787685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996ada2c80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.423+0000 7f1232ffd640 0 client.25771 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:19.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.423+0000 7f125c79b740 1 -- 172.21.15.23:0/494787685 >> 172.21.15.23:0/494787685 conn(0x55996ad96e90 msgr2=0x55996ad98be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.423+0000 7f125c79b740 1 -- 172.21.15.23:0/494787685 shutdown_connections 2024-03-24T10:41:19.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.423+0000 7f125c79b740 1 -- 172.21.15.23:0/494787685 wait complete. 2024-03-24T10:41:19.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.423+0000 7f125c79b740 20 client.25771 trim_cache size 0 max 16384 2024-03-24T10:41:19.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.440+0000 7f125c79b740 1 Processor -- start 2024-03-24T10:41:19.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f125c79b740 1 -- start start 2024-03-24T10:41:19.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f125c79b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996adb1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f125c79b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9d080 0x55996ada3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f125c79b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ada5c60 0x55996adbd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f125c79b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55996ad96cf0 con 0x55996ad9d080 2024-03-24T10:41:19.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f125c79b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55996adb83e0 con 0x55996ada5c60 2024-03-24T10:41:19.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f125c79b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55996ad9b850 con 0x55996ad9c760 2024-03-24T10:41:19.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f1253fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996adb1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f1252ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9d080 0x55996ada3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f1253fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996adb1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57812/0 (socket says 172.21.15.23:57812) 2024-03-24T10:41:19.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f1252ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9d080 0x55996ada3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36068/0 (socket says 172.21.15.23:36068) 2024-03-24T10:41:19.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f1253fff640 1 -- 172.21.15.23:0/734679303 learned_addr learned my addr 172.21.15.23:0/734679303 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f1253fff640 1 --2- 172.21.15.23:0/734679303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ada5c60 0x55996adbd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f1253fff640 1 -- 172.21.15.23:0/734679303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ada5c60 msgr2=0x55996adbd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f1253fff640 1 --2- 172.21.15.23:0/734679303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ada5c60 0x55996adbd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f1253fff640 1 -- 172.21.15.23:0/734679303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9d080 msgr2=0x55996ada3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f1253fff640 1 --2- 172.21.15.23:0/734679303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9d080 0x55996ada3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f1253fff640 1 -- 172.21.15.23:0/734679303 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55996ac9dc60 con 0x55996ad9c760 2024-03-24T10:41:19.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.441+0000 7f1252ffd640 1 --2- 172.21.15.23:0/734679303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9d080 0x55996ada3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:19.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.442+0000 7f1253fff640 1 --2- 172.21.15.23:0/734679303 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996adb1490 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f124c04f040 tx=0x7f124c02c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fab7235c300f756 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.442+0000 7f12397fa640 1 -- 172.21.15.23:0/734679303 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c032e20 con 0x55996ad9c760 2024-03-24T10:41:19.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.442+0000 7f12397fa640 1 -- 172.21.15.23:0/734679303 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f124c05a420 con 0x55996ad9c760 2024-03-24T10:41:19.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.442+0000 7f12397fa640 1 -- 172.21.15.23:0/734679303 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c003970 con 0x55996ad9c760 2024-03-24T10:41:19.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.443+0000 7f125c79b740 1 -- 172.21.15.23:0/734679303 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 msgr2=0x55996adb1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.443+0000 7f125c79b740 1 --2- 172.21.15.23:0/734679303 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996adb1490 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f124c04f040 tx=0x7f124c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.443+0000 7f125c79b740 1 -- 172.21.15.23:0/734679303 shutdown_connections 2024-03-24T10:41:19.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.443+0000 7f125c79b740 1 --2- 172.21.15.23:0/734679303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ada5c60 0x55996adbd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.443+0000 7f125c79b740 1 --2- 172.21.15.23:0/734679303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9d080 0x55996ada3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.443+0000 7f125c79b740 1 --2- 172.21.15.23:0/734679303 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ad9c760 0x55996adb1490 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.443+0000 7f125c79b740 1 -- 172.21.15.23:0/734679303 >> 172.21.15.23:0/734679303 conn(0x55996ad96e90 msgr2=0x55996ad98ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.443+0000 7f125c79b740 1 -- 172.21.15.23:0/734679303 shutdown_connections 2024-03-24T10:41:19.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.443+0000 7f125c79b740 1 -- 172.21.15.23:0/734679303 wait complete. 2024-03-24T10:41:19.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.443+0000 7f125c79b740 1 -- start start 2024-03-24T10:41:19.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.444+0000 7f125c79b740 1 Processor -- start 2024-03-24T10:41:19.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.444+0000 7f125c79b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9c760 0x55996ad9fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.444+0000 7f125c79b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ad9d080 0x55996ada03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.444+0000 7f1253fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9c760 0x55996ad9fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.444+0000 7f1253fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9c760 0x55996ad9fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36074/0 (socket says 172.21.15.23:36074) 2024-03-24T10:41:19.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.444+0000 7f1253fff640 1 -- 172.21.15.23:0/4170685553 learned_addr learned my addr 172.21.15.23:0/4170685553 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.444+0000 7f125c79b740 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ada5c60 0x55996ae1ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.444+0000 7f125c79b740 1 -- 172.21.15.23:0/4170685553 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55996ada0ac0 con 0x55996ad9c760 2024-03-24T10:41:19.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.444+0000 7f1252ffd640 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ad9d080 0x55996ada03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.444+0000 7f125c79b740 1 -- 172.21.15.23:0/4170685553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55996ae1b1c0 con 0x55996ad9d080 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.444+0000 7f125c79b740 1 -- 172.21.15.23:0/4170685553 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55996ae1b300 con 0x55996ada5c60 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.445+0000 7f125c79b740 10 client.0 _ll_register_callbacks cb 0x7f1248005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.445+0000 7f1253fff640 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ada5c60 0x55996ae1ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.445+0000 7f1252ffd640 1 -- 172.21.15.23:0/4170685553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ada5c60 msgr2=0x55996ae1ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.445+0000 7f1252ffd640 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ada5c60 0x55996ae1ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.445+0000 7f1252ffd640 1 -- 172.21.15.23:0/4170685553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9c760 msgr2=0x55996ad9fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.445+0000 7f1252ffd640 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9c760 0x55996ad9fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.445+0000 7f1252ffd640 1 -- 172.21.15.23:0/4170685553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1248004040 con 0x55996ad9d080 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.445+0000 7f1253fff640 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ada5c60 0x55996ae1ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.445+0000 7f1252ffd640 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ad9d080 0x55996ada03c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55996adb85f0 tx=0x7f1248009d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fc072e1a86d4abd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.445+0000 7f1232ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.445+0000 7f1232ffd640 1 -- 172.21.15.23:0/4170685553 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248015040 con 0x55996ad9d080 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.445+0000 7f1232ffd640 1 -- 172.21.15.23:0/4170685553 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f1248005020 con 0x55996ad9d080 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.445+0000 7f125c79b740 1 -- 172.21.15.23:0/4170685553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f124c0345a0 con 0x55996ad9d080 2024-03-24T10:41:19.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.445+0000 7f1232ffd640 1 -- 172.21.15.23:0/4170685553 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1248003e50 con 0x55996ad9d080 2024-03-24T10:41:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.446+0000 7f125c79b740 10 client.25793 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.446+0000 7f125c79b740 1 -- 172.21.15.23:0/4170685553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55996acb96d0 con 0x55996ad9d080 2024-03-24T10:41:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.446+0000 7f1232ffd640 1 -- 172.21.15.23:0/4170685553 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12480047a0 con 0x55996ad9d080 2024-03-24T10:41:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.446+0000 7f1232ffd640 1 -- 172.21.15.23:0/4170685553 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1248021150 con 0x55996ad9d080 2024-03-24T10:41:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.446+0000 7f125c79b740 10 client.25793 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.446+0000 7f125c79b740 10 client.25793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.446+0000 7f125c79b740 20 client.25793 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.446+0000 7f125c79b740 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55996ae1cbc0 0x55996ae1efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.446+0000 7f125c79b740 4 client.25793 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:41:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.446+0000 7f125c79b740 1 -- 172.21.15.23:0/4170685553 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f124c0345a0 con 0x55996ae1cbc0 2024-03-24T10:41:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.446+0000 7f1253fff640 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55996ae1cbc0 0x55996ae1efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.447+0000 7f1253fff640 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55996ae1cbc0 0x55996ae1efe0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7369acc0eff4716b server_cookie=cdf77d89aecb0a01 in_seq=0 out_seq=0 2024-03-24T10:41:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.447+0000 7f1232ffd640 10 client.25793 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f1232ffd640 1 -- 172.21.15.23:0/4170685553 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f124c0345a0 con 0x55996ae1cbc0 2024-03-24T10:41:19.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f1232ffd640 10 client.25793 handle_command_reply: tid=0 2024-03-24T10:41:19.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f125c79b740 1 client.25793 shutdown 2024-03-24T10:41:19.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f125c79b740 1 -- 172.21.15.23:0/4170685553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ad9d080 msgr2=0x55996ada03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f125c79b740 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ad9d080 0x55996ada03c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55996adb85f0 tx=0x7f1248009d90 comp rx=0 tx=0).stop 2024-03-24T10:41:19.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f125c79b740 1 -- 172.21.15.23:0/4170685553 shutdown_connections 2024-03-24T10:41:19.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f125c79b740 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55996ae1cbc0 0x55996ae1efe0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f125c79b740 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55996ada5c60 0x55996ae1ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f125c79b740 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55996ad9d080 0x55996ada03c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f125c79b740 1 --2- 172.21.15.23:0/4170685553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55996ad9c760 0x55996ad9fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f125c79b740 1 -- 172.21.15.23:0/4170685553 >> 172.21.15.23:0/4170685553 conn(0x55996ad96e90 msgr2=0x55996ad95660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f1232ffd640 0 client.25793 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:19.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f125c79b740 1 -- 172.21.15.23:0/4170685553 shutdown_connections 2024-03-24T10:41:19.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f125c79b740 1 -- 172.21.15.23:0/4170685553 wait complete. 2024-03-24T10:41:19.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.448+0000 7f125c79b740 20 client.25793 trim_cache size 0 max 16384 2024-03-24T10:41:19.454 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:41:19.454 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:41:19.454 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "6c116aab-06a9-4aa3-884e-f67e196a1e74", 2024-03-24T10:41:19.454 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:41:19.454 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:41:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.449+0000 7f1259bc2640 1 -- 172.21.15.23:0/589579882 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f123c0a2460 msgr2=0x7f123c0a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.449+0000 7f1259bc2640 1 --2- 172.21.15.23:0/589579882 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f123c0a2460 0x7f123c0a4920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f12440cbcd0 tx=0x7f12440d40e0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.449+0000 7f1259bc2640 1 -- 172.21.15.23:0/589579882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1254100b50 msgr2=0x7f1254198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.449+0000 7f1259bc2640 1 --2- 172.21.15.23:0/589579882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1254100b50 0x7f1254198b20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f124400ea60 tx=0x7f124400ef20 comp rx=0 tx=0).stop 2024-03-24T10:41:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.449+0000 7f1259bc2640 1 -- 172.21.15.23:0/589579882 shutdown_connections 2024-03-24T10:41:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.449+0000 7f1259bc2640 1 --2- 172.21.15.23:0/589579882 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f123c0a2460 0x7f123c0a4920 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.449+0000 7f1259bc2640 1 --2- 172.21.15.23:0/589579882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1254105b40 0x7f12541a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.449+0000 7f1259bc2640 1 --2- 172.21.15.23:0/589579882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12541051a0 0x7f1254199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.449+0000 7f1259bc2640 1 --2- 172.21.15.23:0/589579882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1254100b50 0x7f1254198b20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.449+0000 7f1259bc2640 1 -- 172.21.15.23:0/589579882 >> 172.21.15.23:0/589579882 conn(0x7f1254077500 msgr2=0x7f12540fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.449+0000 7f1259bc2640 1 -- 172.21.15.23:0/589579882 shutdown_connections 2024-03-24T10:41:19.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.449+0000 7f1259bc2640 1 -- 172.21.15.23:0/589579882 wait complete. 2024-03-24T10:41:19.464 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:41:19.464 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:41:19.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.572+0000 7f20e449d640 1 -- 172.21.15.23:0/1544612371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20dc074420 msgr2=0x7f20dc10d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.572+0000 7f20e449d640 1 --2- 172.21.15.23:0/1544612371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20dc074420 0x7f20dc10d9e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009900 tx=0x7f20cc02bcc0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.572+0000 7f20e449d640 1 -- 172.21.15.23:0/1544612371 shutdown_connections 2024-03-24T10:41:19.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.572+0000 7f20e449d640 1 --2- 172.21.15.23:0/1544612371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20dc10df20 0x7f20dc110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.572+0000 7f20e449d640 1 --2- 172.21.15.23:0/1544612371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20dc074420 0x7f20dc10d9e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.572+0000 7f20e449d640 1 --2- 172.21.15.23:0/1544612371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20dc073b00 0x7f20dc073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.572+0000 7f20e449d640 1 -- 172.21.15.23:0/1544612371 >> 172.21.15.23:0/1544612371 conn(0x7f20dc0fd1b0 msgr2=0x7f20dc0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.573+0000 7f20e449d640 1 -- 172.21.15.23:0/1544612371 shutdown_connections 2024-03-24T10:41:19.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.573+0000 7f20e449d640 1 -- 172.21.15.23:0/1544612371 wait complete. 2024-03-24T10:41:19.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.573+0000 7f20e449d640 1 Processor -- start 2024-03-24T10:41:19.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.573+0000 7f20e449d640 1 -- start start 2024-03-24T10:41:19.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.574+0000 7f20e449d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20dc073b00 0x7f20dc101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.574+0000 7f20e449d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20dc074420 0x7f20dc101b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.574+0000 7f20e449d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20dc10df20 0x7f20dc1020c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.574+0000 7f20e2239640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20dc073b00 0x7f20dc101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.574+0000 7f20e1a38640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20dc074420 0x7f20dc101b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.574+0000 7f20e2239640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20dc073b00 0x7f20dc101640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36084/0 (socket says 172.21.15.23:36084) 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.574+0000 7f20e1a38640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20dc074420 0x7f20dc101b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:33450/0 (socket says 172.21.15.23:33450) 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.574+0000 7f20e2239640 1 -- 172.21.15.23:0/1979570547 learned_addr learned my addr 172.21.15.23:0/1979570547 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.574+0000 7f20e2a3a640 1 --2- 172.21.15.23:0/1979570547 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20dc10df20 0x7f20dc1020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.574+0000 7f20e449d640 1 -- 172.21.15.23:0/1979570547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f20dc199b60 con 0x7f20dc073b00 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.574+0000 7f20e449d640 1 -- 172.21.15.23:0/1979570547 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f20dc199cd0 con 0x7f20dc074420 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.574+0000 7f20e449d640 1 -- 172.21.15.23:0/1979570547 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f20dc199e90 con 0x7f20dc10df20 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.575+0000 7f20e1a38640 1 -- 172.21.15.23:0/1979570547 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20dc10df20 msgr2=0x7f20dc1020c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.575+0000 7f20e1a38640 1 --2- 172.21.15.23:0/1979570547 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20dc10df20 0x7f20dc1020c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.575+0000 7f20e1a38640 1 -- 172.21.15.23:0/1979570547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20dc073b00 msgr2=0x7f20dc101640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.575+0000 7f20e1a38640 1 --2- 172.21.15.23:0/1979570547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20dc073b00 0x7f20dc101640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.575+0000 7f20e1a38640 1 -- 172.21.15.23:0/1979570547 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20cc009560 con 0x7f20dc074420 2024-03-24T10:41:19.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.575+0000 7f20e2239640 1 --2- 172.21.15.23:0/1979570547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20dc073b00 0x7f20dc101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:19.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.575+0000 7f20e1a38640 1 --2- 172.21.15.23:0/1979570547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20dc074420 0x7f20dc101b80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009850 tx=0x7f20cc02db30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.576+0000 7f20d37fe640 1 -- 172.21.15.23:0/1979570547 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20cc02c660 con 0x7f20dc074420 2024-03-24T10:41:19.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.576+0000 7f20d37fe640 1 -- 172.21.15.23:0/1979570547 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20cc0052d0 con 0x7f20dc074420 2024-03-24T10:41:19.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.576+0000 7f20d37fe640 1 -- 172.21.15.23:0/1979570547 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20cc003d00 con 0x7f20dc074420 2024-03-24T10:41:19.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.576+0000 7f20e449d640 1 -- 172.21.15.23:0/1979570547 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20dc19a110 con 0x7f20dc074420 2024-03-24T10:41:19.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.576+0000 7f20e449d640 1 -- 172.21.15.23:0/1979570547 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20dc19a5a0 con 0x7f20dc074420 2024-03-24T10:41:19.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.579+0000 7f20d37fe640 1 -- 172.21.15.23:0/1979570547 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 137040+0+0 (secure 0 0 0) 0x7f20cc03e450 con 0x7f20dc074420 2024-03-24T10:41:19.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.580+0000 7f20e7083740 1 Processor -- start 2024-03-24T10:41:19.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.580+0000 7f20e7083740 1 -- start start 2024-03-24T10:41:19.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.580+0000 7f20e7083740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9501d0 0x55fe2b9505b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.580+0000 7f20e7083740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b950af0 0x55fe2b959190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.580+0000 7f20e7083740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9596d0 0x55fe2b95bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.580+0000 7f20e7083740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fe2b874f30 con 0x55fe2b9501d0 2024-03-24T10:41:19.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.580+0000 7f20e7083740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fe2b8be810 con 0x55fe2b950af0 2024-03-24T10:41:19.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.580+0000 7f20e7083740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fe2b8bbd40 con 0x55fe2b9596d0 2024-03-24T10:41:19.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.580+0000 7f20e2239640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b950af0 0x55fe2b959190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.580+0000 7f20e2a3a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9501d0 0x55fe2b9505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.581+0000 7f20e1a38640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9596d0 0x55fe2b95bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.581+0000 7f20e2a3a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9501d0 0x55fe2b9505b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36094/0 (socket says 172.21.15.23:36094) 2024-03-24T10:41:19.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.581+0000 7f20e2a3a640 1 -- 172.21.15.23:0/784728762 learned_addr learned my addr 172.21.15.23:0/784728762 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.581+0000 7f20d37fe640 1 --2- 172.21.15.23:0/1979570547 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f20c80a2460 0x7f20c80a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.581+0000 7f20e2239640 1 -- 172.21.15.23:0/784728762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9596d0 msgr2=0x55fe2b95bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.581+0000 7f20e2239640 1 --2- 172.21.15.23:0/784728762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9596d0 0x55fe2b95bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.581+0000 7f20e2239640 1 -- 172.21.15.23:0/784728762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9501d0 msgr2=0x55fe2b9505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.581+0000 7f20e2239640 1 --2- 172.21.15.23:0/784728762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9501d0 0x55fe2b9505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.581+0000 7f20e2239640 1 -- 172.21.15.23:0/784728762 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe2b878ea0 con 0x55fe2b950af0 2024-03-24T10:41:19.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.581+0000 7f20e2a3a640 1 --2- 172.21.15.23:0/784728762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9501d0 0x55fe2b9505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:19.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.581+0000 7f20e2a3a640 1 --2- 172.21.15.23:0/1979570547 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f20c80a2460 0x7f20c80a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.581+0000 7f20d37fe640 1 -- 172.21.15.23:0/1979570547 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f20cc0f0850 con 0x7f20dc074420 2024-03-24T10:41:19.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.581+0000 7f20e2239640 1 --2- 172.21.15.23:0/784728762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b950af0 0x55fe2b959190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f20d8022500 tx=0x7f20d80229c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbe7393423203b34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.582+0000 7f20d0bfe640 1 -- 172.21.15.23:0/784728762 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d8004030 con 0x55fe2b950af0 2024-03-24T10:41:19.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.582+0000 7f20d0bfe640 1 -- 172.21.15.23:0/784728762 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20d80035f0 con 0x55fe2b950af0 2024-03-24T10:41:19.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.582+0000 7f20d0bfe640 1 -- 172.21.15.23:0/784728762 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d8020950 con 0x55fe2b950af0 2024-03-24T10:41:19.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.582+0000 7f20e2a3a640 1 --2- 172.21.15.23:0/1979570547 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f20c80a2460 0x7f20c80a4920 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f20d4003d50 tx=0x7f20d4004a40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.582+0000 7f20e7083740 1 -- 172.21.15.23:0/784728762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b950af0 msgr2=0x55fe2b959190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.582+0000 7f20e7083740 1 --2- 172.21.15.23:0/784728762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b950af0 0x55fe2b959190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f20d8022500 tx=0x7f20d80229c0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.582+0000 7f20e7083740 1 -- 172.21.15.23:0/784728762 shutdown_connections 2024-03-24T10:41:19.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.582+0000 7f20e7083740 1 --2- 172.21.15.23:0/784728762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9596d0 0x55fe2b95bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.582+0000 7f20e7083740 1 --2- 172.21.15.23:0/784728762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b950af0 0x55fe2b959190 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.582+0000 7f20e7083740 1 --2- 172.21.15.23:0/784728762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9501d0 0x55fe2b9505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.582+0000 7f20e7083740 1 -- 172.21.15.23:0/784728762 >> 172.21.15.23:0/784728762 conn(0x55fe2b94a900 msgr2=0x55fe2b94ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.582+0000 7f20e7083740 1 -- 172.21.15.23:0/784728762 shutdown_connections 2024-03-24T10:41:19.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.582+0000 7f20e7083740 1 -- 172.21.15.23:0/784728762 wait complete. 2024-03-24T10:41:19.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e7083740 1 -- start start 2024-03-24T10:41:19.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e7083740 1 Processor -- start 2024-03-24T10:41:19.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e7083740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b9501d0 0x55fe2b956720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e7083740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b950af0 0x55fe2b956c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e7083740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9596d0 0x55fe2b9571a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e7083740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fe2b8b7520 con 0x55fe2b950af0 2024-03-24T10:41:19.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e7083740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fe2b58a430 con 0x55fe2b9501d0 2024-03-24T10:41:19.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e7083740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fe2b546e00 con 0x55fe2b9596d0 2024-03-24T10:41:19.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e2239640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b9501d0 0x55fe2b956720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e2a3a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9596d0 0x55fe2b9571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e2239640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b9501d0 0x55fe2b956720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:33470/0 (socket says 172.21.15.23:33470) 2024-03-24T10:41:19.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e2239640 1 -- 172.21.15.23:0/1053697911 learned_addr learned my addr 172.21.15.23:0/1053697911 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e2a3a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9596d0 0x55fe2b9571a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57858/0 (socket says 172.21.15.23:57858) 2024-03-24T10:41:19.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e7083740 10 client.0 _ll_register_callbacks cb 0x55fe2b855d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:19.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.583+0000 7f20e1a38640 1 --2- 172.21.15.23:0/1053697911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b950af0 0x55fe2b956c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20e2239640 1 -- 172.21.15.23:0/1053697911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9596d0 msgr2=0x55fe2b9571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20e2239640 1 --2- 172.21.15.23:0/1053697911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9596d0 0x55fe2b9571a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20e2239640 1 -- 172.21.15.23:0/1053697911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b950af0 msgr2=0x55fe2b956c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20e2239640 1 --2- 172.21.15.23:0/1053697911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b950af0 0x55fe2b956c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20e2239640 1 -- 172.21.15.23:0/1053697911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d8009cd0 con 0x55fe2b9501d0 2024-03-24T10:41:19.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20e1a38640 1 --2- 172.21.15.23:0/1053697911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b950af0 0x55fe2b956c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20e2a3a640 1 --2- 172.21.15.23:0/1053697911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9596d0 0x55fe2b9571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20e2239640 1 --2- 172.21.15.23:0/1053697911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b9501d0 0x55fe2b956720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20d80224d0 tx=0x7f20d8003af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59e42b383e13caa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20c5ffb640 1 -- 172.21.15.23:0/1053697911 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d80041d0 con 0x55fe2b9501d0 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20c5ffb640 1 -- 172.21.15.23:0/1053697911 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f20d8004330 con 0x55fe2b9501d0 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20e7083740 1 -- 172.21.15.23:0/1053697911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fe2b5b67e0 con 0x55fe2b9501d0 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20c5ffb640 1 -- 172.21.15.23:0/1053697911 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f20d80044f0 con 0x55fe2b9501d0 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20e7083740 10 client.25805 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.584+0000 7f20e7083740 1 -- 172.21.15.23:0/1053697911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fe2b8b5ba0 con 0x55fe2b9501d0 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.585+0000 7f20c5ffb640 1 -- 172.21.15.23:0/1053697911 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fe2b5b67e0 con 0x55fe2b9501d0 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.585+0000 7f20c5ffb640 1 -- 172.21.15.23:0/1053697911 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f20d8004d70 con 0x55fe2b9501d0 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.585+0000 7f20e7083740 10 client.25805 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.585+0000 7f20e7083740 10 client.25805 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.585+0000 7f20e7083740 20 client.25805 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:19.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.585+0000 7f20e7083740 1 --2- 172.21.15.23:0/1053697911 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fe2b969ad0 0x55fe2b96bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.585+0000 7f20e7083740 4 client.25805 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.585+0000 7f20e7083740 1 -- 172.21.15.23:0/1053697911 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe2b5f3720 con 0x55fe2b969ad0 2024-03-24T10:41:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.586+0000 7f20e1a38640 1 --2- 172.21.15.23:0/1053697911 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fe2b969ad0 0x55fe2b96bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.586+0000 7f20c5ffb640 10 client.25805 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.586+0000 7f20e1a38640 1 --2- 172.21.15.23:0/1053697911 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fe2b969ad0 0x55fe2b96bf00 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e452769f328d634c server_cookie=eba3d7e674db1ff9 in_seq=0 out_seq=0 2024-03-24T10:41:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.586+0000 7f20c5ffb640 1 -- 172.21.15.23:0/1053697911 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fe2b5f3720 con 0x55fe2b969ad0 2024-03-24T10:41:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.587+0000 7f20c5ffb640 10 client.25805 handle_command_reply: tid=0 2024-03-24T10:41:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.587+0000 7f20e7083740 1 client.25805 shutdown 2024-03-24T10:41:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.587+0000 7f20e7083740 1 -- 172.21.15.23:0/1053697911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b9501d0 msgr2=0x55fe2b956720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.587+0000 7f20e7083740 1 --2- 172.21.15.23:0/1053697911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b9501d0 0x55fe2b956720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20d80224d0 tx=0x7f20d8003af0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.587+0000 7f20e7083740 1 -- 172.21.15.23:0/1053697911 shutdown_connections 2024-03-24T10:41:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.587+0000 7f20e7083740 1 --2- 172.21.15.23:0/1053697911 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fe2b969ad0 0x55fe2b96bf00 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.587+0000 7f20c5ffb640 0 client.25805 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:19.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.587+0000 7f20e7083740 1 --2- 172.21.15.23:0/1053697911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9596d0 0x55fe2b9571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.587+0000 7f20e7083740 1 --2- 172.21.15.23:0/1053697911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b950af0 0x55fe2b956c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.587+0000 7f20e7083740 1 --2- 172.21.15.23:0/1053697911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b9501d0 0x55fe2b956720 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.587+0000 7f20e7083740 1 -- 172.21.15.23:0/1053697911 >> 172.21.15.23:0/1053697911 conn(0x55fe2b94a900 msgr2=0x55fe2b949b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.587+0000 7f20e7083740 1 -- 172.21.15.23:0/1053697911 shutdown_connections 2024-03-24T10:41:19.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.587+0000 7f20e7083740 1 -- 172.21.15.23:0/1053697911 wait complete. 2024-03-24T10:41:19.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.587+0000 7f20e7083740 20 client.25805 trim_cache size 0 max 16384 2024-03-24T10:41:19.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.605+0000 7f20e7083740 1 Processor -- start 2024-03-24T10:41:19.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.605+0000 7f20e7083740 1 -- start start 2024-03-24T10:41:19.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.605+0000 7f20e7083740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9501d0 0x55fe2b958e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.605+0000 7f20e7083740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b950af0 0x55fe2b964fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.605+0000 7f20e7083740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9596d0 0x55fe2b971810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.605+0000 7f20e7083740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fe2b83c430 con 0x55fe2b9596d0 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.605+0000 7f20e7083740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fe2b93c4f0 con 0x55fe2b950af0 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.605+0000 7f20e7083740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fe2b94a430 con 0x55fe2b9501d0 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.605+0000 7f20e2239640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9501d0 0x55fe2b958e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.605+0000 7f20e1a38640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b950af0 0x55fe2b964fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.605+0000 7f20e2239640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9501d0 0x55fe2b958e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57868/0 (socket says 172.21.15.23:57868) 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.605+0000 7f20e2239640 1 -- 172.21.15.23:0/96441947 learned_addr learned my addr 172.21.15.23:0/96441947 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.605+0000 7f20e2a3a640 1 --2- 172.21.15.23:0/96441947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9596d0 0x55fe2b971810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.606+0000 7f20e2239640 1 -- 172.21.15.23:0/96441947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b950af0 msgr2=0x55fe2b964fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.606+0000 7f20e2239640 1 --2- 172.21.15.23:0/96441947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b950af0 0x55fe2b964fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.606+0000 7f20e2239640 1 -- 172.21.15.23:0/96441947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9596d0 msgr2=0x55fe2b971810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.606+0000 7f20e2239640 1 --2- 172.21.15.23:0/96441947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9596d0 0x55fe2b971810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.606+0000 7f20e2239640 1 -- 172.21.15.23:0/96441947 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe2b851b20 con 0x55fe2b9501d0 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.606+0000 7f20e2a3a640 1 --2- 172.21.15.23:0/96441947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9596d0 0x55fe2b971810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.606+0000 7f20e2239640 1 --2- 172.21.15.23:0/96441947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9501d0 0x55fe2b958e50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f20d8003f90 tx=0x7f20d8005db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41f507fb73ec93c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.606+0000 7f20c7fff640 1 -- 172.21.15.23:0/96441947 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d8005490 con 0x55fe2b9501d0 2024-03-24T10:41:19.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.606+0000 7f20c7fff640 1 -- 172.21.15.23:0/96441947 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20d80055f0 con 0x55fe2b9501d0 2024-03-24T10:41:19.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.606+0000 7f20c7fff640 1 -- 172.21.15.23:0/96441947 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d802a870 con 0x55fe2b9501d0 2024-03-24T10:41:19.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.607+0000 7f20e7083740 1 -- 172.21.15.23:0/96441947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9501d0 msgr2=0x55fe2b958e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.607+0000 7f20e7083740 1 --2- 172.21.15.23:0/96441947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9501d0 0x55fe2b958e50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f20d8003f90 tx=0x7f20d8005db0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.607+0000 7f20e7083740 1 -- 172.21.15.23:0/96441947 shutdown_connections 2024-03-24T10:41:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.607+0000 7f20e7083740 1 --2- 172.21.15.23:0/96441947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9596d0 0x55fe2b971810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.607+0000 7f20e7083740 1 --2- 172.21.15.23:0/96441947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b950af0 0x55fe2b964fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.607+0000 7f20e7083740 1 --2- 172.21.15.23:0/96441947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b9501d0 0x55fe2b958e50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.607+0000 7f20e7083740 1 -- 172.21.15.23:0/96441947 >> 172.21.15.23:0/96441947 conn(0x55fe2b94a900 msgr2=0x55fe2b954ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.607+0000 7f20e7083740 1 -- 172.21.15.23:0/96441947 shutdown_connections 2024-03-24T10:41:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.607+0000 7f20e7083740 1 -- 172.21.15.23:0/96441947 wait complete. 2024-03-24T10:41:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.607+0000 7f20e7083740 1 -- start start 2024-03-24T10:41:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.607+0000 7f20e7083740 1 Processor -- start 2024-03-24T10:41:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.607+0000 7f20e7083740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b9501d0 0x55fe2b953da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e7083740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b950af0 0x55fe2b9542e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e7083740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9596d0 0x55fe2b952990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e7083740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fe2b961ce0 con 0x55fe2b9596d0 2024-03-24T10:41:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e7083740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fe2b961e50 con 0x55fe2b9501d0 2024-03-24T10:41:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e7083740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fe2b952f50 con 0x55fe2b950af0 2024-03-24T10:41:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e1a38640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b950af0 0x55fe2b9542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e2239640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b9501d0 0x55fe2b953da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e2239640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b9501d0 0x55fe2b953da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:33494/0 (socket says 172.21.15.23:33494) 2024-03-24T10:41:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e2239640 1 -- 172.21.15.23:0/3560580961 learned_addr learned my addr 172.21.15.23:0/3560580961 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e2a3a640 1 --2- 172.21.15.23:0/3560580961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9596d0 0x55fe2b952990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e7083740 10 client.0 _ll_register_callbacks cb 0x7f20d802d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e2239640 1 -- 172.21.15.23:0/3560580961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b950af0 msgr2=0x55fe2b9542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e2239640 1 --2- 172.21.15.23:0/3560580961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b950af0 0x55fe2b9542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e2239640 1 -- 172.21.15.23:0/3560580961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9596d0 msgr2=0x55fe2b952990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e2239640 1 --2- 172.21.15.23:0/3560580961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9596d0 0x55fe2b952990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e2239640 1 -- 172.21.15.23:0/3560580961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe2b8b5ba0 con 0x55fe2b9501d0 2024-03-24T10:41:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20e2239640 1 --2- 172.21.15.23:0/3560580961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b9501d0 0x55fe2b953da0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f20d8003ec0 tx=0x7f20d802a9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86eaf61b3bdd4abd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:41:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20c5ffb640 1 -- 172.21.15.23:0/3560580961 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d8020a50 con 0x55fe2b9501d0 2024-03-24T10:41:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.608+0000 7f20c5ffb640 1 -- 172.21.15.23:0/3560580961 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f20d8020bb0 con 0x55fe2b9501d0 2024-03-24T10:41:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.609+0000 7f20e7083740 1 -- 172.21.15.23:0/3560580961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20d80035f0 con 0x55fe2b9501d0 2024-03-24T10:41:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.609+0000 7f20c5ffb640 1 -- 172.21.15.23:0/3560580961 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f20d8020d70 con 0x55fe2b9501d0 2024-03-24T10:41:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.609+0000 7f20e7083740 10 client.25811 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.609+0000 7f20e7083740 1 -- 172.21.15.23:0/3560580961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fe2b877700 con 0x55fe2b9501d0 2024-03-24T10:41:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.609+0000 7f20c5ffb640 1 -- 172.21.15.23:0/3560580961 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20d804f950 con 0x55fe2b9501d0 2024-03-24T10:41:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.609+0000 7f20c5ffb640 1 -- 172.21.15.23:0/3560580961 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f20d80356f0 con 0x55fe2b9501d0 2024-03-24T10:41:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.609+0000 7f20e7083740 10 client.25811 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.609+0000 7f20e7083740 10 client.25811 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.609+0000 7f20e7083740 20 client.25811 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.609+0000 7f20e7083740 1 --2- 172.21.15.23:0/3560580961 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fe2b969ad0 0x55fe2b9d1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.609+0000 7f20e7083740 4 client.25811 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:41:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.609+0000 7f20e7083740 1 -- 172.21.15.23:0/3560580961 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f20d80035f0 con 0x55fe2b969ad0 2024-03-24T10:41:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.609+0000 7f20e1a38640 1 --2- 172.21.15.23:0/3560580961 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fe2b969ad0 0x55fe2b9d1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20e1a38640 1 --2- 172.21.15.23:0/3560580961 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fe2b969ad0 0x55fe2b9d1720 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9afd74be8030c8 server_cookie=5eb59b6dcf099f8b in_seq=0 out_seq=0 2024-03-24T10:41:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20c5ffb640 10 client.25811 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20c5ffb640 1 -- 172.21.15.23:0/3560580961 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f20d80035f0 con 0x55fe2b969ad0 2024-03-24T10:41:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20c5ffb640 10 client.25811 handle_command_reply: tid=0 2024-03-24T10:41:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20e7083740 1 client.25811 shutdown 2024-03-24T10:41:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20e7083740 1 -- 172.21.15.23:0/3560580961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b9501d0 msgr2=0x55fe2b953da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20e7083740 1 --2- 172.21.15.23:0/3560580961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b9501d0 0x55fe2b953da0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f20d8003ec0 tx=0x7f20d802a9e0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20e7083740 1 -- 172.21.15.23:0/3560580961 shutdown_connections 2024-03-24T10:41:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20e7083740 1 --2- 172.21.15.23:0/3560580961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe2b9596d0 0x55fe2b952990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20e7083740 1 --2- 172.21.15.23:0/3560580961 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fe2b969ad0 0x55fe2b9d1720 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20c5ffb640 0 client.25811 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20e7083740 1 --2- 172.21.15.23:0/3560580961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fe2b950af0 0x55fe2b9542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20e7083740 1 --2- 172.21.15.23:0/3560580961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fe2b9501d0 0x55fe2b953da0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20e7083740 1 -- 172.21.15.23:0/3560580961 >> 172.21.15.23:0/3560580961 conn(0x55fe2b94a900 msgr2=0x55fe2b94bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20e7083740 1 -- 172.21.15.23:0/3560580961 shutdown_connections 2024-03-24T10:41:19.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20e7083740 1 -- 172.21.15.23:0/3560580961 wait complete. 2024-03-24T10:41:19.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.610+0000 7f20e7083740 20 client.25811 trim_cache size 0 max 16384 2024-03-24T10:41:19.616 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:41:19.616 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:41:19.616 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:41:19.616 INFO:teuthology.orchestra.run.smithi023.stdout: "6c116aab-06a9-4aa3-884e-f67e196a1e74": { 2024-03-24T10:41:19.616 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:41:19.616 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "6c116aab-06a9-4aa3-884e-f67e196a1e74", 2024-03-24T10:41:19.617 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:41:19.617 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:41:19.617 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:41:19.617 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:41:19.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.611+0000 7f20e449d640 1 -- 172.21.15.23:0/1979570547 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f20c80a2460 msgr2=0x7f20c80a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.611+0000 7f20e449d640 1 --2- 172.21.15.23:0/1979570547 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f20c80a2460 0x7f20c80a4920 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f20d4003d50 tx=0x7f20d4004a40 comp rx=0 tx=0).stop 2024-03-24T10:41:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.611+0000 7f20e449d640 1 -- 172.21.15.23:0/1979570547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20dc074420 msgr2=0x7f20dc101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.611+0000 7f20e449d640 1 --2- 172.21.15.23:0/1979570547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20dc074420 0x7f20dc101b80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009850 tx=0x7f20cc02db30 comp rx=0 tx=0).stop 2024-03-24T10:41:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.611+0000 7f20e449d640 1 -- 172.21.15.23:0/1979570547 shutdown_connections 2024-03-24T10:41:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.611+0000 7f20e449d640 1 --2- 172.21.15.23:0/1979570547 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f20c80a2460 0x7f20c80a4920 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.611+0000 7f20e449d640 1 --2- 172.21.15.23:0/1979570547 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20dc10df20 0x7f20dc1020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.611+0000 7f20e449d640 1 --2- 172.21.15.23:0/1979570547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20dc074420 0x7f20dc101b80 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009850 tx=0x7f20cc02db30 comp rx=0 tx=0).stop 2024-03-24T10:41:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.611+0000 7f20e449d640 1 --2- 172.21.15.23:0/1979570547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20dc073b00 0x7f20dc101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.611+0000 7f20e449d640 1 -- 172.21.15.23:0/1979570547 >> 172.21.15.23:0/1979570547 conn(0x7f20dc0fd1b0 msgr2=0x7f20dc10f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.611+0000 7f20e449d640 1 -- 172.21.15.23:0/1979570547 shutdown_connections 2024-03-24T10:41:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:19.611+0000 7f20e449d640 1 -- 172.21.15.23:0/1979570547 wait complete. 2024-03-24T10:41:19.623 INFO:tasks.cephfs.filesystem:scrub status for tag:6c116aab-06a9-4aa3-884e-f67e196a1e74 - {'path': '/', 'tag': '6c116aab-06a9-4aa3-884e-f67e196a1e74', 'options': 'recursive,force'} 2024-03-24T10:41:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.636+0000 7f24d3d5e640 1 -- 172.21.15.23:0/3535669974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24cc1005e0 msgr2=0x7f24cc100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.636+0000 7f24d3d5e640 1 --2- 172.21.15.23:0/3535669974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24cc1005e0 0x7f24cc100a60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009930 tx=0x7f24bc02bc30 comp rx=0 tx=0).stop 2024-03-24T10:41:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.636+0000 7f24d3d5e640 1 -- 172.21.15.23:0/3535669974 shutdown_connections 2024-03-24T10:41:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.636+0000 7f24d3d5e640 1 --2- 172.21.15.23:0/3535669974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc100fa0 0x7f24cc10d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.636+0000 7f24d3d5e640 1 --2- 172.21.15.23:0/3535669974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24cc1005e0 0x7f24cc100a60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.636+0000 7f24d3d5e640 1 --2- 172.21.15.23:0/3535669974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24cc106680 0x7f24cc106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.636+0000 7f24d3d5e640 1 -- 172.21.15.23:0/3535669974 >> 172.21.15.23:0/3535669974 conn(0x7f24cc0fc0d0 msgr2=0x7f24cc0fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.636+0000 7f24d3d5e640 1 -- 172.21.15.23:0/3535669974 shutdown_connections 2024-03-24T10:41:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.636+0000 7f24d3d5e640 1 -- 172.21.15.23:0/3535669974 wait complete. 2024-03-24T10:41:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.637+0000 7f24d3d5e640 1 Processor -- start 2024-03-24T10:41:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.637+0000 7f24d3d5e640 1 -- start start 2024-03-24T10:41:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.637+0000 7f24d3d5e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24cc1005e0 0x7f24cc19b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.637+0000 7f24d3d5e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc100fa0 0x7f24cc19bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.639 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.637+0000 7f24d3d5e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24cc106680 0x7f24cc195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d3d5e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc195eb0 con 0x7f24cc106680 2024-03-24T10:41:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d1afa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24cc1005e0 0x7f24cc19b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d3d5e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc196020 con 0x7f24cc100fa0 2024-03-24T10:41:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d1afa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24cc1005e0 0x7f24cc19b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57898/0 (socket says 172.21.15.23:57898) 2024-03-24T10:41:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d1afa640 1 -- 172.21.15.23:0/3463373128 learned_addr learned my addr 172.21.15.23:0/3463373128 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d3d5e640 1 -- 172.21.15.23:0/3463373128 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc1961e0 con 0x7f24cc1005e0 2024-03-24T10:41:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d22fb640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24cc106680 0x7f24cc195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d12f9640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc100fa0 0x7f24cc19bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d12f9640 1 -- 172.21.15.23:0/3463373128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24cc1005e0 msgr2=0x7f24cc19b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d12f9640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24cc1005e0 0x7f24cc19b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d12f9640 1 -- 172.21.15.23:0/3463373128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24cc106680 msgr2=0x7f24cc195860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d12f9640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24cc106680 0x7f24cc195860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d1afa640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24cc1005e0 0x7f24cc19b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:41:19.640 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d12f9640 1 -- 172.21.15.23:0/3463373128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24bc009560 con 0x7f24cc100fa0 2024-03-24T10:41:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d22fb640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24cc106680 0x7f24cc195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24d12f9640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc100fa0 0x7f24cc19bbf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f24bc036040 tx=0x7f24bc02c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.638+0000 7f24c2ffd640 1 -- 172.21.15.23:0/3463373128 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc0042d0 con 0x7f24cc100fa0 2024-03-24T10:41:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.639+0000 7f24c2ffd640 1 -- 172.21.15.23:0/3463373128 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24bc004430 con 0x7f24cc100fa0 2024-03-24T10:41:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.639+0000 7f24c2ffd640 1 -- 172.21.15.23:0/3463373128 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc003bf0 con 0x7f24cc100fa0 2024-03-24T10:41:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.639+0000 7f24d3d5e640 1 -- 172.21.15.23:0/3463373128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24cc196460 con 0x7f24cc100fa0 2024-03-24T10:41:19.641 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.639+0000 7f24d3d5e640 1 -- 172.21.15.23:0/3463373128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24cc1a2750 con 0x7f24cc100fa0 2024-03-24T10:41:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.641+0000 7f24c2ffd640 1 -- 172.21.15.23:0/3463373128 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 137040+0+0 (secure 0 0 0) 0x7f24bc003d50 con 0x7f24cc100fa0 2024-03-24T10:41:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.641+0000 7f24c2ffd640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24b80a6920 0x7f24b80a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.641+0000 7f24c2ffd640 1 -- 172.21.15.23:0/3463373128 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f24bc0bd7c0 con 0x7f24cc100fa0 2024-03-24T10:41:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.641+0000 7f24d1afa640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24b80a6920 0x7f24b80a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.642 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.642+0000 7f24d1afa640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24b80a6920 0x7f24b80a8de0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f24cc1969a0 tx=0x7f24c8005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.643+0000 7f24d6a1b740 1 Processor -- start 2024-03-24T10:41:19.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.643+0000 7f24d6a1b740 1 -- start start 2024-03-24T10:41:19.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.644+0000 7f24d6a1b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddc25e0 0x56420ddc29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.644+0000 7f24d6a1b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc2f00 0x56420ddcb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.644+0000 7f24d6a1b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddcbcd0 0x56420ddce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.644+0000 7f24d6a1b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56420dd7d1b0 con 0x56420ddc2f00 2024-03-24T10:41:19.645 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.644+0000 7f24d12f9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc2f00 0x56420ddcb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.645 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.644+0000 7f24d6a1b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56420dd2ff80 con 0x56420ddc25e0 2024-03-24T10:41:19.645 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.644+0000 7f24d12f9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc2f00 0x56420ddcb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36142/0 (socket says 172.21.15.23:36142) 2024-03-24T10:41:19.645 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.644+0000 7f24d12f9640 1 -- 172.21.15.23:0/537848009 learned_addr learned my addr 172.21.15.23:0/537848009 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.645 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.644+0000 7f24d22fb640 1 --2- 172.21.15.23:0/537848009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddcbcd0 0x56420ddce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.645 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.644+0000 7f24d1afa640 1 --2- 172.21.15.23:0/537848009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddc25e0 0x56420ddc29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.645 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.644+0000 7f24d6a1b740 1 -- 172.21.15.23:0/537848009 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56420dd2d4b0 con 0x56420ddcbcd0 2024-03-24T10:41:19.645 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.644+0000 7f24d12f9640 1 -- 172.21.15.23:0/537848009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddcbcd0 msgr2=0x56420ddce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.646 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.645+0000 7f24d12f9640 1 --2- 172.21.15.23:0/537848009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddcbcd0 0x56420ddce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.646 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.645+0000 7f24d12f9640 1 -- 172.21.15.23:0/537848009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddc25e0 msgr2=0x56420ddc29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.646 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.645+0000 7f24d12f9640 1 --2- 172.21.15.23:0/537848009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddc25e0 0x56420ddc29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.646 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.645+0000 7f24d12f9640 1 -- 172.21.15.23:0/537848009 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56420dcbe810 con 0x56420ddc2f00 2024-03-24T10:41:19.646 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.645+0000 7f24d12f9640 1 --2- 172.21.15.23:0/537848009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc2f00 0x56420ddcb790 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f24bc006010 tx=0x7f24bc0c20c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d82c097185933d89 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.646 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.645+0000 7f24b7fff640 1 -- 172.21.15.23:0/537848009 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc0c1020 con 0x56420ddc2f00 2024-03-24T10:41:19.646 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.645+0000 7f24b7fff640 1 -- 172.21.15.23:0/537848009 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24bc0bca20 con 0x56420ddc2f00 2024-03-24T10:41:19.646 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.645+0000 7f24b7fff640 1 -- 172.21.15.23:0/537848009 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc0bcc00 con 0x56420ddc2f00 2024-03-24T10:41:19.647 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.646+0000 7f24d6a1b740 1 -- 172.21.15.23:0/537848009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc2f00 msgr2=0x56420ddcb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.646+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/537848009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc2f00 0x56420ddcb790 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f24bc006010 tx=0x7f24bc0c20c0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.646+0000 7f24d6a1b740 1 -- 172.21.15.23:0/537848009 shutdown_connections 2024-03-24T10:41:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.646+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/537848009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddcbcd0 0x56420ddce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.646+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/537848009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc2f00 0x56420ddcb790 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.646+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/537848009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddc25e0 0x56420ddc29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.646+0000 7f24d6a1b740 1 -- 172.21.15.23:0/537848009 >> 172.21.15.23:0/537848009 conn(0x56420ddbcf00 msgr2=0x56420ddbd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.646+0000 7f24d6a1b740 1 -- 172.21.15.23:0/537848009 shutdown_connections 2024-03-24T10:41:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.646+0000 7f24d6a1b740 1 -- 172.21.15.23:0/537848009 wait complete. 2024-03-24T10:41:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.647+0000 7f24d6a1b740 1 -- start start 2024-03-24T10:41:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.647+0000 7f24d6a1b740 1 Processor -- start 2024-03-24T10:41:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.648+0000 7f24d6a1b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc25e0 0x56420ddc8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.648+0000 7f24d6a1b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc2f00 0x56420ddc9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.648+0000 7f24d6a1b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddcbcd0 0x56420ddc9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.648+0000 7f24d6a1b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56420dd28c90 con 0x56420ddc2f00 2024-03-24T10:41:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.648+0000 7f24d6a1b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56420dccdc50 con 0x56420ddcbcd0 2024-03-24T10:41:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.648+0000 7f24d12f9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc2f00 0x56420ddc9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.648+0000 7f24d1afa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc25e0 0x56420ddc8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.648+0000 7f24d12f9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc2f00 0x56420ddc9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36152/0 (socket says 172.21.15.23:36152) 2024-03-24T10:41:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.648+0000 7f24d1afa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc25e0 0x56420ddc8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57916/0 (socket says 172.21.15.23:57916) 2024-03-24T10:41:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.648+0000 7f24d12f9640 1 -- 172.21.15.23:0/1516042085 learned_addr learned my addr 172.21.15.23:0/1516042085 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.648+0000 7f24d22fb640 1 --2- 172.21.15.23:0/1516042085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddcbcd0 0x56420ddc9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.648+0000 7f24d6a1b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56420d9fc250 con 0x56420ddc25e0 2024-03-24T10:41:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.649+0000 7f24d6a1b740 10 client.0 _ll_register_callbacks cb 0x56420dd1c290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.649+0000 7f24d12f9640 1 -- 172.21.15.23:0/1516042085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc25e0 msgr2=0x56420ddc8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.649+0000 7f24d12f9640 1 --2- 172.21.15.23:0/1516042085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc25e0 0x56420ddc8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.649+0000 7f24d12f9640 1 -- 172.21.15.23:0/1516042085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddcbcd0 msgr2=0x56420ddc9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.649+0000 7f24d12f9640 1 --2- 172.21.15.23:0/1516042085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddcbcd0 0x56420ddc9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.649+0000 7f24d12f9640 1 -- 172.21.15.23:0/1516042085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24cc1a2750 con 0x56420ddc2f00 2024-03-24T10:41:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.649+0000 7f24d22fb640 1 --2- 172.21.15.23:0/1516042085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddcbcd0 0x56420ddc9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.649+0000 7f24d12f9640 1 --2- 172.21.15.23:0/1516042085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc2f00 0x56420ddc9240 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f24bc0b9fd0 tx=0x7f24bc003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c774158a103a3b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.649+0000 7f24b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.649+0000 7f24b57fa640 1 -- 172.21.15.23:0/1516042085 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc0d5070 con 0x56420ddc2f00 2024-03-24T10:41:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.649+0000 7f24b57fa640 1 -- 172.21.15.23:0/1516042085 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f24bc0bcc00 con 0x56420ddc2f00 2024-03-24T10:41:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.649+0000 7f24d6a1b740 1 -- 172.21.15.23:0/1516042085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56420dd4d790 con 0x56420ddc2f00 2024-03-24T10:41:19.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.650+0000 7f24b57fa640 1 -- 172.21.15.23:0/1516042085 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f24bc0b9390 con 0x56420ddc2f00 2024-03-24T10:41:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.650+0000 7f24d6a1b740 10 client.15945 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.650+0000 7f24d6a1b740 1 -- 172.21.15.23:0/1516042085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56420dd27310 con 0x56420ddc2f00 2024-03-24T10:41:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.650+0000 7f24b57fa640 1 -- 172.21.15.23:0/1516042085 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56420dd4d790 con 0x56420ddc2f00 2024-03-24T10:41:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.650+0000 7f24b57fa640 1 -- 172.21.15.23:0/1516042085 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f24bc03c070 con 0x56420ddc2f00 2024-03-24T10:41:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.650+0000 7f24d6a1b740 10 client.15945 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.650+0000 7f24d6a1b740 10 client.15945 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.650+0000 7f24d6a1b740 20 client.15945 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.650+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/1516042085 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56420dddb3e0 0x56420dddd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.650+0000 7f24d6a1b740 4 client.15945 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.650+0000 7f24d6a1b740 1 -- 172.21.15.23:0/1516042085 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56420da28600 con 0x56420dddb3e0 2024-03-24T10:41:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.650+0000 7f24d1afa640 1 --2- 172.21.15.23:0/1516042085 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56420dddb3e0 0x56420dddd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.651+0000 7f24b57fa640 10 client.15945 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.651+0000 7f24d1afa640 1 --2- 172.21.15.23:0/1516042085 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56420dddb3e0 0x56420dddd7d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5e8d16398762f3a server_cookie=31001e0c39698907 in_seq=0 out_seq=0 2024-03-24T10:41:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.651+0000 7f24b57fa640 1 -- 172.21.15.23:0/1516042085 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56420da28600 con 0x56420dddb3e0 2024-03-24T10:41:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.651+0000 7f24b57fa640 10 client.15945 handle_command_reply: tid=0 2024-03-24T10:41:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.651+0000 7f24d6a1b740 1 client.15945 shutdown 2024-03-24T10:41:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.652+0000 7f24d6a1b740 1 -- 172.21.15.23:0/1516042085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc2f00 msgr2=0x56420ddc9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.652+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/1516042085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc2f00 0x56420ddc9240 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f24bc0b9fd0 tx=0x7f24bc003680 comp rx=0 tx=0).stop 2024-03-24T10:41:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.652+0000 7f24d6a1b740 1 -- 172.21.15.23:0/1516042085 shutdown_connections 2024-03-24T10:41:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.652+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/1516042085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddcbcd0 0x56420ddc9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.652+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/1516042085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc2f00 0x56420ddc9240 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.652+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/1516042085 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56420dddb3e0 0x56420dddd7d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.652+0000 7f24b57fa640 0 client.15945 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.652+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/1516042085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc25e0 0x56420ddc8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.652+0000 7f24d6a1b740 1 -- 172.21.15.23:0/1516042085 >> 172.21.15.23:0/1516042085 conn(0x56420ddbcf00 msgr2=0x56420ddbec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.652+0000 7f24d6a1b740 1 -- 172.21.15.23:0/1516042085 shutdown_connections 2024-03-24T10:41:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.652+0000 7f24d6a1b740 1 -- 172.21.15.23:0/1516042085 wait complete. 2024-03-24T10:41:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.652+0000 7f24d6a1b740 20 client.15945 trim_cache size 0 max 16384 2024-03-24T10:41:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.681+0000 7f24d6a1b740 1 Processor -- start 2024-03-24T10:41:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.681+0000 7f24d6a1b740 1 -- start start 2024-03-24T10:41:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.681+0000 7f24d6a1b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc25e0 0x56420ddcb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d6a1b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddc2f00 0x56420ddb7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d6a1b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddcbcd0 0x56420ddd7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d6a1b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56420dcae660 con 0x56420ddcbcd0 2024-03-24T10:41:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d6a1b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56420ddd83d0 con 0x56420ddc2f00 2024-03-24T10:41:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d6a1b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56420ddb86c0 con 0x56420ddc25e0 2024-03-24T10:41:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d12f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddc2f00 0x56420ddb7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.683 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d1afa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc25e0 0x56420ddcb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.683 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d12f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddc2f00 0x56420ddb7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:33536/0 (socket says 172.21.15.23:33536) 2024-03-24T10:41:19.683 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d1afa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc25e0 0x56420ddcb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57922/0 (socket says 172.21.15.23:57922) 2024-03-24T10:41:19.683 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d12f9640 1 -- 172.21.15.23:0/3221356353 learned_addr learned my addr 172.21.15.23:0/3221356353 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.683 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d22fb640 1 --2- 172.21.15.23:0/3221356353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddcbcd0 0x56420ddd7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d1afa640 1 -- 172.21.15.23:0/3221356353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddc2f00 msgr2=0x56420ddb7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d1afa640 1 --2- 172.21.15.23:0/3221356353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddc2f00 0x56420ddb7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d1afa640 1 -- 172.21.15.23:0/3221356353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddcbcd0 msgr2=0x56420ddd7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d1afa640 1 --2- 172.21.15.23:0/3221356353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddcbcd0 0x56420ddd7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d1afa640 1 -- 172.21.15.23:0/3221356353 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56420dcc3810 con 0x56420ddc25e0 2024-03-24T10:41:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d12f9640 1 --2- 172.21.15.23:0/3221356353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddc2f00 0x56420ddb7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24d1afa640 1 --2- 172.21.15.23:0/3221356353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc25e0 0x56420ddcb430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x56420ddd7850 tx=0x7f24c8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b70441b394dca14e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.682+0000 7f24b77fe640 1 -- 172.21.15.23:0/3221356353 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c802e070 con 0x56420ddc25e0 2024-03-24T10:41:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.683+0000 7f24b77fe640 1 -- 172.21.15.23:0/3221356353 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24c80297a0 con 0x56420ddc25e0 2024-03-24T10:41:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.683+0000 7f24b77fe640 1 -- 172.21.15.23:0/3221356353 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c80047a0 con 0x56420ddc25e0 2024-03-24T10:41:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.683+0000 7f24d6a1b740 1 -- 172.21.15.23:0/3221356353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc25e0 msgr2=0x56420ddcb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.683+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/3221356353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc25e0 0x56420ddcb430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x56420ddd7850 tx=0x7f24c8016000 comp rx=0 tx=0).stop 2024-03-24T10:41:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.683+0000 7f24d6a1b740 1 -- 172.21.15.23:0/3221356353 shutdown_connections 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.683+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/3221356353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddcbcd0 0x56420ddd7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.683+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/3221356353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddc2f00 0x56420ddb7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.683+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/3221356353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc25e0 0x56420ddcb430 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.683+0000 7f24d6a1b740 1 -- 172.21.15.23:0/3221356353 >> 172.21.15.23:0/3221356353 conn(0x56420ddbcf00 msgr2=0x56420ddbd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.683+0000 7f24d6a1b740 1 -- 172.21.15.23:0/3221356353 shutdown_connections 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.683+0000 7f24d6a1b740 1 -- 172.21.15.23:0/3221356353 wait complete. 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.684+0000 7f24d6a1b740 1 -- start start 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.684+0000 7f24d6a1b740 1 Processor -- start 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.684+0000 7f24d6a1b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc25e0 0x56420ddd3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.684+0000 7f24d6a1b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc2f00 0x56420ddcedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.684+0000 7f24d6a1b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddcbcd0 0x56420ddcf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.684+0000 7f24d6a1b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56420ddcfa10 con 0x56420ddc25e0 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.684+0000 7f24d6a1b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56420ddcfb80 con 0x56420ddcbcd0 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.684+0000 7f24d6a1b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56420de48e80 con 0x56420ddc2f00 2024-03-24T10:41:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.684+0000 7f24d6a1b740 10 client.0 _ll_register_callbacks cb 0x7f24bc088920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.684+0000 7f24d1afa640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc25e0 0x56420ddd3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.684+0000 7f24d12f9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc2f00 0x56420ddcedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.684+0000 7f24d12f9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc2f00 0x56420ddcedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57926/0 (socket says 172.21.15.23:57926) 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.684+0000 7f24d12f9640 1 -- 172.21.15.23:0/2080352482 learned_addr learned my addr 172.21.15.23:0/2080352482 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.684+0000 7f24d22fb640 1 --2- 172.21.15.23:0/2080352482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddcbcd0 0x56420ddcf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24d12f9640 1 -- 172.21.15.23:0/2080352482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddcbcd0 msgr2=0x56420ddcf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24d12f9640 1 --2- 172.21.15.23:0/2080352482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddcbcd0 0x56420ddcf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24d12f9640 1 -- 172.21.15.23:0/2080352482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc25e0 msgr2=0x56420ddd3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24d12f9640 1 --2- 172.21.15.23:0/2080352482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc25e0 0x56420ddd3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24d12f9640 1 -- 172.21.15.23:0/2080352482 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56420dd27310 con 0x56420ddc2f00 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24d22fb640 1 --2- 172.21.15.23:0/2080352482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddcbcd0 0x56420ddcf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24d12f9640 1 --2- 172.21.15.23:0/2080352482 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc2f00 0x56420ddcedd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f24bc005ec0 tx=0x7f24bc0bcf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a889e5494856bfb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24b57fa640 1 -- 172.21.15.23:0/2080352482 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc0d5040 con 0x56420ddc2f00 2024-03-24T10:41:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24b57fa640 1 -- 172.21.15.23:0/2080352482 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f24bc0b9730 con 0x56420ddc2f00 2024-03-24T10:41:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24d6a1b740 1 -- 172.21.15.23:0/2080352482 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56420ddc3b40 con 0x56420ddc2f00 2024-03-24T10:41:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24b57fa640 1 -- 172.21.15.23:0/2080352482 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f24bc0db560 con 0x56420ddc2f00 2024-03-24T10:41:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24d6a1b740 10 client.25807 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24d6a1b740 1 -- 172.21.15.23:0/2080352482 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x7f24c8009d10 con 0x56420ddc2f00 2024-03-24T10:41:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.685+0000 7f24b57fa640 1 -- 172.21.15.23:0/2080352482 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24bc0b98f0 con 0x56420ddc2f00 2024-03-24T10:41:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.686+0000 7f24b57fa640 1 -- 172.21.15.23:0/2080352482 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f24bc03c040 con 0x56420ddc2f00 2024-03-24T10:41:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.686+0000 7f24d6a1b740 10 client.25807 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.686+0000 7f24d6a1b740 10 client.25807 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.686+0000 7f24d6a1b740 20 client.25807 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.686+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/2080352482 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56420de4a1c0 0x56420de4c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.686+0000 7f24d6a1b740 4 client.25807 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:41:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.686+0000 7f24d6a1b740 1 -- 172.21.15.23:0/2080352482 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56420ddc3b40 con 0x56420de4a1c0 2024-03-24T10:41:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.686+0000 7f24d1afa640 1 --2- 172.21.15.23:0/2080352482 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56420de4a1c0 0x56420de4c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.686+0000 7f24b57fa640 10 client.25807 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:19.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.686+0000 7f24d1afa640 1 --2- 172.21.15.23:0/2080352482 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56420de4a1c0 0x56420de4c5e0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfa955b9cdd46bad server_cookie=9d9b9be02fc15d9b in_seq=0 out_seq=0 2024-03-24T10:41:19.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.689+0000 7f24b57fa640 1 -- 172.21.15.23:0/2080352482 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56420ddc3b40 con 0x56420de4a1c0 2024-03-24T10:41:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.689+0000 7f24b57fa640 10 client.25807 handle_command_reply: tid=0 2024-03-24T10:41:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.689+0000 7f24d6a1b740 1 client.25807 shutdown 2024-03-24T10:41:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.689+0000 7f24d6a1b740 1 -- 172.21.15.23:0/2080352482 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc2f00 msgr2=0x56420ddcedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/2080352482 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc2f00 0x56420ddcedd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f24bc005ec0 tx=0x7f24bc0bcf10 comp rx=0 tx=0).stop 2024-03-24T10:41:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24d6a1b740 1 -- 172.21.15.23:0/2080352482 shutdown_connections 2024-03-24T10:41:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/2080352482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56420ddcbcd0 0x56420ddcf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/2080352482 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56420de4a1c0 0x56420de4c5e0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/2080352482 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56420ddc2f00 0x56420ddcedd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24d6a1b740 1 --2- 172.21.15.23:0/2080352482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56420ddc25e0 0x56420ddd3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24b57fa640 0 client.25807 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24d6a1b740 1 -- 172.21.15.23:0/2080352482 >> 172.21.15.23:0/2080352482 conn(0x56420ddbcf00 msgr2=0x56420ddbe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24d6a1b740 1 -- 172.21.15.23:0/2080352482 shutdown_connections 2024-03-24T10:41:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24d6a1b740 1 -- 172.21.15.23:0/2080352482 wait complete. 2024-03-24T10:41:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24d6a1b740 20 client.25807 trim_cache size 0 max 16384 2024-03-24T10:41:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:41:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:41:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:41:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:41:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24d3d5e640 1 -- 172.21.15.23:0/3463373128 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24b80a6920 msgr2=0x7f24b80a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24d3d5e640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24b80a6920 0x7f24b80a8de0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f24cc1969a0 tx=0x7f24c8005f90 comp rx=0 tx=0).stop 2024-03-24T10:41:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24d3d5e640 1 -- 172.21.15.23:0/3463373128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc100fa0 msgr2=0x7f24cc19bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.690+0000 7f24d3d5e640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc100fa0 0x7f24cc19bbf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f24bc036040 tx=0x7f24bc02c640 comp rx=0 tx=0).stop 2024-03-24T10:41:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.691+0000 7f24d3d5e640 1 -- 172.21.15.23:0/3463373128 shutdown_connections 2024-03-24T10:41:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.691+0000 7f24d3d5e640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24b80a6920 0x7f24b80a8de0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.691+0000 7f24d3d5e640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24cc106680 0x7f24cc195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.691+0000 7f24d3d5e640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24cc100fa0 0x7f24cc19bbf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.691+0000 7f24d3d5e640 1 --2- 172.21.15.23:0/3463373128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24cc1005e0 0x7f24cc19b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.691+0000 7f24d3d5e640 1 -- 172.21.15.23:0/3463373128 >> 172.21.15.23:0/3463373128 conn(0x7f24cc0fc0d0 msgr2=0x7f24cc0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:19.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.691+0000 7f24d3d5e640 1 -- 172.21.15.23:0/3463373128 shutdown_connections 2024-03-24T10:41:19.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:19.691+0000 7f24d3d5e640 1 -- 172.21.15.23:0/3463373128 wait complete. 2024-03-24T10:41:20.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:20 smithi023 ceph-mon[32466]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-24T10:41:20.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:20 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:41:20.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:20 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:20.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:20 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:41:20.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:20 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:20.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:20 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:41:21.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:20 smithi117 ceph-mon[36368]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-24T10:41:21.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:20 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:41:21.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:20 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:21.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:20 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:41:21.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:20 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:21.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:20 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:41:21.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:20 smithi116 ceph-mon[36248]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-24T10:41:21.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:20 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:41:21.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:20 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:21.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:20 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:41:21.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:20 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:21.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:20 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:41:22.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:22 smithi023 ceph-mon[32466]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-24T10:41:23.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:22 smithi117 ceph-mon[36368]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-24T10:41:23.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:22 smithi116 ceph-mon[36248]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-24T10:41:24.837 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:24 smithi116 ceph-mon[36248]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:41:24.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:24 smithi023 ceph-mon[32466]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:41:25.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:24 smithi117 ceph-mon[36368]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:41:26.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:26 smithi023 ceph-mon[32466]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:41:27.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:26 smithi117 ceph-mon[36368]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:41:27.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:26 smithi116 ceph-mon[36248]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:41:28.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:28 smithi023 ceph-mon[32466]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-24T10:41:29.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:28 smithi117 ceph-mon[36368]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-24T10:41:29.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:28 smithi116 ceph-mon[36248]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-24T10:41:30.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:30 smithi023 ceph-mon[32466]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:41:31.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:30 smithi117 ceph-mon[36368]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:41:31.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:30 smithi116 ceph-mon[36248]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-24T10:41:32.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.822+0000 7f24c880e640 1 -- 172.21.15.23:0/179679345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c41005d0 msgr2=0x7f24c410e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.822+0000 7f24c880e640 1 --2- 172.21.15.23:0/179679345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c41005d0 0x7f24c410e590 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009cd0 tx=0x7f24bc02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.822+0000 7f24c880e640 1 -- 172.21.15.23:0/179679345 shutdown_connections 2024-03-24T10:41:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.822+0000 7f24c880e640 1 --2- 172.21.15.23:0/179679345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c41005d0 0x7f24c410e590 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.822+0000 7f24c880e640 1 --2- 172.21.15.23:0/179679345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24c40ffc10 0x7f24c4100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.822+0000 7f24c880e640 1 --2- 172.21.15.23:0/179679345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24c4107710 0x7f24c4107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.822+0000 7f24c880e640 1 -- 172.21.15.23:0/179679345 >> 172.21.15.23:0/179679345 conn(0x7f24c4076b60 msgr2=0x7f24c40fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.822+0000 7f24c880e640 1 -- 172.21.15.23:0/179679345 shutdown_connections 2024-03-24T10:41:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.822+0000 7f24c880e640 1 -- 172.21.15.23:0/179679345 wait complete. 2024-03-24T10:41:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.822+0000 7f24c880e640 1 Processor -- start 2024-03-24T10:41:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c880e640 1 -- start start 2024-03-24T10:41:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c880e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24c40ffc10 0x7f24c419d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c880e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24c41005d0 0x7f24c419de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c880e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c4107710 0x7f24c4197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c880e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f24c41980f0 con 0x7f24c4107710 2024-03-24T10:41:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c880e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f24c4198260 con 0x7f24c41005d0 2024-03-24T10:41:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c880e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f24c4198420 con 0x7f24c40ffc10 2024-03-24T10:41:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c4107710 0x7f24c4197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24c41005d0 0x7f24c419de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c4107710 0x7f24c4197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40996/0 (socket says 172.21.15.23:40996) 2024-03-24T10:41:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c2d9d640 1 -- 172.21.15.23:0/3467997579 learned_addr learned my addr 172.21.15.23:0/3467997579 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24c41005d0 0x7f24c419de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:58058/0 (socket says 172.21.15.23:58058) 2024-03-24T10:41:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24c40ffc10 0x7f24c419d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c1d9b640 1 -- 172.21.15.23:0/3467997579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24c40ffc10 msgr2=0x7f24c419d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c1d9b640 1 --2- 172.21.15.23:0/3467997579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24c40ffc10 0x7f24c419d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c1d9b640 1 -- 172.21.15.23:0/3467997579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c4107710 msgr2=0x7f24c4197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c1d9b640 1 --2- 172.21.15.23:0/3467997579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c4107710 0x7f24c4197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c1d9b640 1 -- 172.21.15.23:0/3467997579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b8009490 con 0x7f24c41005d0 2024-03-24T10:41:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.823+0000 7f24c2d9d640 1 --2- 172.21.15.23:0/3467997579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c4107710 0x7f24c4197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.824+0000 7f24c1d9b640 1 --2- 172.21.15.23:0/3467997579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24c41005d0 0x7f24c419de30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f24b800e9f0 tx=0x7f24b800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.824+0000 7f24b37fe640 1 -- 172.21.15.23:0/3467997579 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b800ccc0 con 0x7f24c41005d0 2024-03-24T10:41:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.824+0000 7f24b37fe640 1 -- 172.21.15.23:0/3467997579 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24b800ce20 con 0x7f24c41005d0 2024-03-24T10:41:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.824+0000 7f24b37fe640 1 -- 172.21.15.23:0/3467997579 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b8010430 con 0x7f24c41005d0 2024-03-24T10:41:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.824+0000 7f24c880e640 1 -- 172.21.15.23:0/3467997579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24bc009900 con 0x7f24c41005d0 2024-03-24T10:41:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.824+0000 7f24c880e640 1 -- 172.21.15.23:0/3467997579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c41a47b0 con 0x7f24c41005d0 2024-03-24T10:41:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24cb3ee740 1 Processor -- start 2024-03-24T10:41:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24cb3ee740 1 -- start start 2024-03-24T10:41:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24b37fe640 1 -- 172.21.15.23:0/3467997579 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 137040+0+0 (secure 0 0 0) 0x7f24b8003f80 con 0x7f24c41005d0 2024-03-24T10:41:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24cb3ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c465e0 0x561153c469c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24cb3ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24cb3ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c4fcd0 0x561153c520c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24cb3ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x561153c011b0 con 0x561153c4fcd0 2024-03-24T10:41:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24cb3ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561153bb3f80 con 0x561153c46f00 2024-03-24T10:41:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24cb3ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561153bb14b0 con 0x561153c465e0 2024-03-24T10:41:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24c1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c4fcd0 0x561153c520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24c259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24c2d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c465e0 0x561153c469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24c259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:58072/0 (socket says 172.21.15.23:58072) 2024-03-24T10:41:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24c259c640 1 -- 172.21.15.23:0/1577028404 learned_addr learned my addr 172.21.15.23:0/1577028404 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.827+0000 7f24c1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c4fcd0 0x561153c520c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41002/0 (socket says 172.21.15.23:41002) 2024-03-24T10:41:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.828+0000 7f24c259c640 1 -- 172.21.15.23:0/1577028404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c465e0 msgr2=0x561153c469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.828+0000 7f24c259c640 1 --2- 172.21.15.23:0/1577028404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c465e0 0x561153c469c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.828+0000 7f24c259c640 1 -- 172.21.15.23:0/1577028404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c4fcd0 msgr2=0x561153c520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.828+0000 7f24c259c640 1 --2- 172.21.15.23:0/1577028404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c4fcd0 0x561153c520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.828+0000 7f24c259c640 1 -- 172.21.15.23:0/1577028404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561153b42810 con 0x561153c46f00 2024-03-24T10:41:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.828+0000 7f24b37fe640 1 --2- 172.21.15.23:0/3467997579 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24ac0a2480 0x7f24ac0a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.828+0000 7f24c259c640 1 --2- 172.21.15.23:0/1577028404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4f790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f24b400ac00 tx=0x7f24b400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a43befe9251a4a01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.828+0000 7f24b37fe640 1 -- 172.21.15.23:0/3467997579 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f24b8014070 con 0x7f24c41005d0 2024-03-24T10:41:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.828+0000 7f24c2d9d640 1 --2- 172.21.15.23:0/3467997579 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24ac0a2480 0x7f24ac0a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.828+0000 7f24b0ff9640 1 -- 172.21.15.23:0/1577028404 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4018640 con 0x561153c46f00 2024-03-24T10:41:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.828+0000 7f24b0ff9640 1 -- 172.21.15.23:0/1577028404 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24b4018c80 con 0x561153c46f00 2024-03-24T10:41:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.828+0000 7f24b0ff9640 1 -- 172.21.15.23:0/1577028404 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4010cd0 con 0x561153c46f00 2024-03-24T10:41:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.829+0000 7f24c2d9d640 1 --2- 172.21.15.23:0/3467997579 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24ac0a2480 0x7f24ac0a4940 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f24bc006010 tx=0x7f24bc005d80 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.829+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1577028404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 msgr2=0x561153c4f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.829+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/1577028404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4f790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f24b400ac00 tx=0x7f24b400f790 comp rx=0 tx=0).stop 2024-03-24T10:41:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.829+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1577028404 shutdown_connections 2024-03-24T10:41:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.829+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/1577028404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c4fcd0 0x561153c520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.829+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/1577028404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4f790 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.829+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/1577028404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c465e0 0x561153c469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.829+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1577028404 >> 172.21.15.23:0/1577028404 conn(0x561153c40f00 msgr2=0x561153c412e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.829+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1577028404 shutdown_connections 2024-03-24T10:41:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.829+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1577028404 wait complete. 2024-03-24T10:41:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.830+0000 7f24cb3ee740 1 -- start start 2024-03-24T10:41:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.830+0000 7f24cb3ee740 1 Processor -- start 2024-03-24T10:41:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.830+0000 7f24cb3ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c465e0 0x561153c4cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.830+0000 7f24cb3ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.830+0000 7f24cb3ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c4fcd0 0x561153c4d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24cb3ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x561153bacc90 con 0x561153c465e0 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24c259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c465e0 0x561153c4cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24c259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c465e0 0x561153c4cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41004/0 (socket says 172.21.15.23:41004) 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24c259c640 1 -- 172.21.15.23:0/1726423061 learned_addr learned my addr 172.21.15.23:0/1726423061 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1726423061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561153b51c50 con 0x561153c46f00 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1726423061 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561153880250 con 0x561153c4fcd0 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24c1d9b640 1 --2- 172.21.15.23:0/1726423061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24cb3ee740 10 client.0 _ll_register_callbacks cb 0x561153ba0290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24c2d9d640 1 --2- 172.21.15.23:0/1726423061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c4fcd0 0x561153c4d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24c259c640 1 -- 172.21.15.23:0/1726423061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c4fcd0 msgr2=0x561153c4d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24c259c640 1 --2- 172.21.15.23:0/1726423061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c4fcd0 0x561153c4d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24c259c640 1 -- 172.21.15.23:0/1726423061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 msgr2=0x561153c4d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24c259c640 1 --2- 172.21.15.23:0/1726423061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24c259c640 1 -- 172.21.15.23:0/1726423061 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b400a860 con 0x561153c465e0 2024-03-24T10:41:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24c2d9d640 1 --2- 172.21.15.23:0/1726423061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c4fcd0 0x561153c4d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24c259c640 1 --2- 172.21.15.23:0/1726423061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c465e0 0x561153c4cd60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f24b4010810 tx=0x7f24b4010e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b668763a2ed15bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24a9ffb640 1 -- 172.21.15.23:0/1726423061 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4010430 con 0x561153c465e0 2024-03-24T10:41:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1726423061 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561153bd1790 con 0x561153c465e0 2024-03-24T10:41:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.831+0000 7f24a9ffb640 1 -- 172.21.15.23:0/1726423061 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f24b4005250 con 0x561153c465e0 2024-03-24T10:41:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.832+0000 7f24a9ffb640 1 -- 172.21.15.23:0/1726423061 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f24b4005410 con 0x561153c465e0 2024-03-24T10:41:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.832+0000 7f24cb3ee740 10 client.15963 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.832+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1726423061 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561153bab310 con 0x561153c465e0 2024-03-24T10:41:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.832+0000 7f24a9ffb640 1 -- 172.21.15.23:0/1726423061 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561153bd1790 con 0x561153c465e0 2024-03-24T10:41:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.832+0000 7f24a9ffb640 1 -- 172.21.15.23:0/1726423061 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f24b40046c0 con 0x561153c465e0 2024-03-24T10:41:32.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.832+0000 7f24cb3ee740 10 client.15963 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.832+0000 7f24cb3ee740 10 client.15963 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.832+0000 7f24cb3ee740 20 client.15963 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.832+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/1726423061 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561153c63f10 0x561153c66300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.832+0000 7f24cb3ee740 4 client.15963 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.832+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1726423061 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611538ac600 con 0x561153c63f10 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.832+0000 7f24c1d9b640 1 --2- 172.21.15.23:0/1726423061 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561153c63f10 0x561153c66300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.833+0000 7f24a9ffb640 10 client.15963 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.833+0000 7f24c1d9b640 1 --2- 172.21.15.23:0/1726423061 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561153c63f10 0x561153c66300 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb431627f5beb36e server_cookie=cb7327c6bca9c2f in_seq=0 out_seq=0 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24a9ffb640 1 -- 172.21.15.23:0/1726423061 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5611538ac600 con 0x561153c63f10 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24a9ffb640 10 client.15963 handle_command_reply: tid=0 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24cb3ee740 1 client.15963 shutdown 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1726423061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c465e0 msgr2=0x561153c4cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/1726423061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c465e0 0x561153c4cd60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f24b4010810 tx=0x7f24b4010e80 comp rx=0 tx=0).stop 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1726423061 shutdown_connections 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/1726423061 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561153c63f10 0x561153c66300 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/1726423061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c4fcd0 0x561153c4d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/1726423061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/1726423061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c465e0 0x561153c4cd60 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1726423061 >> 172.21.15.23:0/1726423061 conn(0x561153c40f00 msgr2=0x561153c42c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24a9ffb640 0 client.15963 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1726423061 shutdown_connections 2024-03-24T10:41:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24cb3ee740 1 -- 172.21.15.23:0/1726423061 wait complete. 2024-03-24T10:41:32.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.834+0000 7f24cb3ee740 20 client.15963 trim_cache size 0 max 16384 2024-03-24T10:41:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.863+0000 7f24cb3ee740 1 Processor -- start 2024-03-24T10:41:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.863+0000 7f24cb3ee740 1 -- start start 2024-03-24T10:41:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.863+0000 7f24cb3ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.863+0000 7f24cb3ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c4fcd0 0x561153c5b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.863+0000 7f24cb3ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c6bbf0 0x561153c6c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24cb3ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x561153b32660 con 0x561153c4fcd0 2024-03-24T10:41:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24cb3ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561153c4cb80 con 0x561153c46f00 2024-03-24T10:41:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24cb3ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561153c3f3f0 con 0x561153c6bbf0 2024-03-24T10:41:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24c1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c4fcd0 0x561153c5b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24c259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24c1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c4fcd0 0x561153c5b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41020/0 (socket says 172.21.15.23:41020) 2024-03-24T10:41:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24c1d9b640 1 -- 172.21.15.23:0/3689824816 learned_addr learned my addr 172.21.15.23:0/3689824816 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24c259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:58102/0 (socket says 172.21.15.23:58102) 2024-03-24T10:41:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24c2d9d640 1 --2- 172.21.15.23:0/3689824816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c6bbf0 0x561153c6c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24c259c640 1 -- 172.21.15.23:0/3689824816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c6bbf0 msgr2=0x561153c6c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24c259c640 1 --2- 172.21.15.23:0/3689824816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c6bbf0 0x561153c6c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24c259c640 1 -- 172.21.15.23:0/3689824816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c4fcd0 msgr2=0x561153c5b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24c259c640 1 --2- 172.21.15.23:0/3689824816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c4fcd0 0x561153c5b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24c259c640 1 -- 172.21.15.23:0/3689824816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561153b47810 con 0x561153c46f00 2024-03-24T10:41:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24c1d9b640 1 --2- 172.21.15.23:0/3689824816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c4fcd0 0x561153c5b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24c2d9d640 1 --2- 172.21.15.23:0/3689824816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c6bbf0 0x561153c6c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:41:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24c259c640 1 --2- 172.21.15.23:0/3689824816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4f490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f24b400ac00 tx=0x7f24b400fc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4d31a66cb8b1d72 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24abfff640 1 -- 172.21.15.23:0/3689824816 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4039070 con 0x561153c46f00 2024-03-24T10:41:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24abfff640 1 -- 172.21.15.23:0/3689824816 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24b4004df0 con 0x561153c46f00 2024-03-24T10:41:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.864+0000 7f24abfff640 1 -- 172.21.15.23:0/3689824816 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4010430 con 0x561153c46f00 2024-03-24T10:41:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.865+0000 7f24cb3ee740 1 -- 172.21.15.23:0/3689824816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 msgr2=0x561153c4f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.865+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/3689824816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4f490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f24b400ac00 tx=0x7f24b400fc90 comp rx=0 tx=0).stop 2024-03-24T10:41:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.865+0000 7f24cb3ee740 1 -- 172.21.15.23:0/3689824816 shutdown_connections 2024-03-24T10:41:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.865+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/3689824816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c6bbf0 0x561153c6c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.866+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/3689824816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c4fcd0 0x561153c5b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.866+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/3689824816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c46f00 0x561153c4f490 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.866+0000 7f24cb3ee740 1 -- 172.21.15.23:0/3689824816 >> 172.21.15.23:0/3689824816 conn(0x561153c40f00 msgr2=0x561153c48050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.866+0000 7f24cb3ee740 1 -- 172.21.15.23:0/3689824816 shutdown_connections 2024-03-24T10:41:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.866+0000 7f24cb3ee740 1 -- 172.21.15.23:0/3689824816 wait complete. 2024-03-24T10:41:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.866+0000 7f24cb3ee740 1 -- start start 2024-03-24T10:41:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.866+0000 7f24cb3ee740 1 Processor -- start 2024-03-24T10:41:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.866+0000 7f24cb3ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c46f00 0x561153c52990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.866+0000 7f24cb3ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c4fcd0 0x561153c52ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24c259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c46f00 0x561153c52990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24c259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c46f00 0x561153c52990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41034/0 (socket says 172.21.15.23:41034) 2024-03-24T10:41:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24c1d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c4fcd0 0x561153c52ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24c1d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c4fcd0 0x561153c52ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:37336/0 (socket says 172.21.15.23:37336) 2024-03-24T10:41:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24cb3ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c6bbf0 0x561153cc8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24c259c640 1 -- 172.21.15.23:0/817881984 learned_addr learned my addr 172.21.15.23:0/817881984 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24cb3ee740 1 -- 172.21.15.23:0/817881984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x561153c53780 con 0x561153c46f00 2024-03-24T10:41:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24cb3ee740 1 -- 172.21.15.23:0/817881984 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561153cc8750 con 0x561153c6bbf0 2024-03-24T10:41:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24cb3ee740 1 -- 172.21.15.23:0/817881984 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561153cc8890 con 0x561153c4fcd0 2024-03-24T10:41:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24c2d9d640 1 --2- 172.21.15.23:0/817881984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c6bbf0 0x561153cc8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24cb3ee740 10 client.0 _ll_register_callbacks cb 0x7f24b401b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24c1d9b640 1 -- 172.21.15.23:0/817881984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c6bbf0 msgr2=0x561153cc8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24c1d9b640 1 --2- 172.21.15.23:0/817881984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c6bbf0 0x561153cc8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24c1d9b640 1 -- 172.21.15.23:0/817881984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c46f00 msgr2=0x561153c52990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24c1d9b640 1 --2- 172.21.15.23:0/817881984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c46f00 0x561153c52990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24c1d9b640 1 -- 172.21.15.23:0/817881984 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c41a47b0 con 0x561153c4fcd0 2024-03-24T10:41:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24c259c640 1 --2- 172.21.15.23:0/817881984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c46f00 0x561153c52990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24cb3ee740 1 -- 172.21.15.23:0/817881984 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24b4018c80 con 0x561153c4fcd0 2024-03-24T10:41:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24c1d9b640 1 --2- 172.21.15.23:0/817881984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c4fcd0 0x561153c52ed0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f24b800c6a0 tx=0x7f24b80d1b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d480196bfa0d07ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.867+0000 7f24a9ffb640 10 client.25819 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:41:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.868+0000 7f24a9ffb640 1 -- 172.21.15.23:0/817881984 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b809a7f0 con 0x561153c4fcd0 2024-03-24T10:41:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.868+0000 7f24a9ffb640 1 -- 172.21.15.23:0/817881984 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f24b809b590 con 0x561153c4fcd0 2024-03-24T10:41:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.868+0000 7f24a9ffb640 1 -- 172.21.15.23:0/817881984 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f24b809ae90 con 0x561153c4fcd0 2024-03-24T10:41:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.868+0000 7f24cb3ee740 10 client.25819 fetch_fsmap learned FSMap version 23 2024-03-24T10:41:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.868+0000 7f24cb3ee740 1 -- 172.21.15.23:0/817881984 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({fsmap=23}) -- 0x561153bab310 con 0x561153c4fcd0 2024-03-24T10:41:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.868+0000 7f24a9ffb640 1 -- 172.21.15.23:0/817881984 <== mon.1 v2:172.21.15.117:3300/0 4 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f24b801f060 con 0x561153c4fcd0 2024-03-24T10:41:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.868+0000 7f24cb3ee740 10 client.25819 fetch_fsmap finished waiting for FSMap version 23 2024-03-24T10:41:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.868+0000 7f24cb3ee740 10 client.25819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.868+0000 7f24cb3ee740 20 client.25819 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.868+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/817881984 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561153cc9ee0 0x561153ccc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.868+0000 7f24cb3ee740 4 client.25819 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:41:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.868+0000 7f24cb3ee740 1 -- 172.21.15.23:0/817881984 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f24b4018c80 con 0x561153cc9ee0 2024-03-24T10:41:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.868+0000 7f24c259c640 1 --2- 172.21.15.23:0/817881984 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561153cc9ee0 0x561153ccc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.869+0000 7f24a9ffb640 10 client.25819 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.869+0000 7f24c259c640 1 --2- 172.21.15.23:0/817881984 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561153cc9ee0 0x561153ccc300 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43859e54e4a64dc8 server_cookie=1e2c6ce7f49293aa in_seq=0 out_seq=0 2024-03-24T10:41:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24a9ffb640 1 -- 172.21.15.23:0/817881984 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f24b4018c80 con 0x561153cc9ee0 2024-03-24T10:41:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24a9ffb640 10 client.25819 handle_command_reply: tid=0 2024-03-24T10:41:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24cb3ee740 1 client.25819 shutdown 2024-03-24T10:41:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24cb3ee740 1 -- 172.21.15.23:0/817881984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c4fcd0 msgr2=0x561153c52ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/817881984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c4fcd0 0x561153c52ed0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f24b800c6a0 tx=0x7f24b80d1b60 comp rx=0 tx=0).stop 2024-03-24T10:41:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24cb3ee740 1 -- 172.21.15.23:0/817881984 shutdown_connections 2024-03-24T10:41:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/817881984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561153c6bbf0 0x561153cc8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/817881984 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561153cc9ee0 0x561153ccc300 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/817881984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561153c4fcd0 0x561153c52ed0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24a9ffb640 0 client.25819 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24cb3ee740 1 --2- 172.21.15.23:0/817881984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561153c46f00 0x561153c52990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24cb3ee740 1 -- 172.21.15.23:0/817881984 >> 172.21.15.23:0/817881984 conn(0x561153c40f00 msgr2=0x561153c4b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24cb3ee740 1 -- 172.21.15.23:0/817881984 shutdown_connections 2024-03-24T10:41:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24cb3ee740 1 -- 172.21.15.23:0/817881984 wait complete. 2024-03-24T10:41:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.872+0000 7f24cb3ee740 20 client.25819 trim_cache size 0 max 16384 2024-03-24T10:41:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:41:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:41:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:41:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:41:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.873+0000 7f24c880e640 1 -- 172.21.15.23:0/3467997579 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24ac0a2480 msgr2=0x7f24ac0a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.873+0000 7f24c880e640 1 --2- 172.21.15.23:0/3467997579 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24ac0a2480 0x7f24ac0a4940 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f24bc006010 tx=0x7f24bc005d80 comp rx=0 tx=0).stop 2024-03-24T10:41:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.873+0000 7f24c880e640 1 -- 172.21.15.23:0/3467997579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24c41005d0 msgr2=0x7f24c419de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.873+0000 7f24c880e640 1 --2- 172.21.15.23:0/3467997579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24c41005d0 0x7f24c419de30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f24b800e9f0 tx=0x7f24b800edf0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.873+0000 7f24c880e640 1 -- 172.21.15.23:0/3467997579 shutdown_connections 2024-03-24T10:41:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.873+0000 7f24c880e640 1 --2- 172.21.15.23:0/3467997579 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24ac0a2480 0x7f24ac0a4940 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.873+0000 7f24c880e640 1 --2- 172.21.15.23:0/3467997579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c4107710 0x7f24c4197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.873+0000 7f24c880e640 1 --2- 172.21.15.23:0/3467997579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24c41005d0 0x7f24c419de30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.873+0000 7f24c880e640 1 --2- 172.21.15.23:0/3467997579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24c40ffc10 0x7f24c419d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.873+0000 7f24c880e640 1 -- 172.21.15.23:0/3467997579 >> 172.21.15.23:0/3467997579 conn(0x7f24c4076b60 msgr2=0x7f24c40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.873+0000 7f24c880e640 1 -- 172.21.15.23:0/3467997579 shutdown_connections 2024-03-24T10:41:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:32.873+0000 7f24c880e640 1 -- 172.21.15.23:0/3467997579 wait complete. 2024-03-24T10:41:32.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:32 smithi023 ceph-mon[32466]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-24T10:41:33.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:32 smithi117 ceph-mon[36368]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-24T10:41:33.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:32 smithi116 ceph-mon[36248]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-24T10:41:34.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:34 smithi023 ceph-mon[32466]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:41:35.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:34 smithi117 ceph-mon[36368]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:41:35.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:34 smithi116 ceph-mon[36248]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:41:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:36 smithi117 ceph-mon[36368]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:41:36.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:36 smithi023 ceph-mon[32466]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:41:37.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:36 smithi116 ceph-mon[36248]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:41:37.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:37 smithi023 ceph-mon[32466]: mgrmap e25: x(active, since 91s), standbys: y, z 2024-03-24T10:41:38.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:37 smithi117 ceph-mon[36368]: mgrmap e25: x(active, since 91s), standbys: y, z 2024-03-24T10:41:38.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:37 smithi116 ceph-mon[36248]: mgrmap e25: x(active, since 91s), standbys: y, z 2024-03-24T10:41:38.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:38 smithi023 ceph-mon[32466]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-24T10:41:39.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:38 smithi117 ceph-mon[36368]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-24T10:41:39.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:38 smithi116 ceph-mon[36248]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-24T10:41:40.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:40 smithi023 ceph-mon[32466]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-24T10:41:41.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:40 smithi117 ceph-mon[36368]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-24T10:41:41.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:40 smithi116 ceph-mon[36248]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-24T10:41:41.903 INFO:tasks.workunit.client.0.smithi023.stderr:Note: switching to 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b'. 2024-03-24T10:41:41.903 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-24T10:41:41.903 INFO:tasks.workunit.client.0.smithi023.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T10:41:41.903 INFO:tasks.workunit.client.0.smithi023.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T10:41:41.903 INFO:tasks.workunit.client.0.smithi023.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T10:41:41.903 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-24T10:41:41.903 INFO:tasks.workunit.client.0.smithi023.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T10:41:41.903 INFO:tasks.workunit.client.0.smithi023.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T10:41:41.904 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-24T10:41:41.904 INFO:tasks.workunit.client.0.smithi023.stderr: git switch -c 2024-03-24T10:41:41.904 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-24T10:41:41.904 INFO:tasks.workunit.client.0.smithi023.stderr:Or undo this operation with: 2024-03-24T10:41:41.904 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-24T10:41:41.904 INFO:tasks.workunit.client.0.smithi023.stderr: git switch - 2024-03-24T10:41:41.904 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-24T10:41:41.904 INFO:tasks.workunit.client.0.smithi023.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T10:41:41.904 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-24T10:41:41.904 INFO:tasks.workunit.client.0.smithi023.stderr:HEAD is now at d39ab7bb1f1 qa/suites/fs/workload: enable snap_schedule early 2024-03-24T10:41:41.908 DEBUG:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-24T10:41:41.972 INFO:tasks.workunit.client.0.smithi023.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T10:41:41.975 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T10:41:41.975 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T10:41:42.724 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T10:41:42.799 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T10:41:42.840 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T10:41:42.842 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T10:41:42.842 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T10:41:42.881 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T10:41:42.884 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:41:42.884 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-24T10:41:42.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:42 smithi023 ceph-mon[32466]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-24T10:41:42.933 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-03-24T10:41:42.934 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-03-24T10:41:42.934 DEBUG:teuthology.orchestra.run.smithi023:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-03-24T10:41:42.994 INFO:tasks.workunit.client.0.smithi023.stderr:+ dbench 1 2024-03-24T10:41:42.999 INFO:tasks.workunit.client.0.smithi023.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-24T10:41:43.000 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-24T10:41:43.000 INFO:tasks.workunit.client.0.smithi023.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-24T10:41:43.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:42 smithi117 ceph-mon[36368]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-24T10:41:43.010 INFO:tasks.workunit.client.0.smithi023.stdout:0 of 1 processes prepared for launch 0 sec 2024-03-24T10:41:43.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:42 smithi116 ceph-mon[36248]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-24T10:41:43.111 INFO:tasks.workunit.client.0.smithi023.stdout:1 of 1 processes prepared for launch 0 sec 2024-03-24T10:41:43.111 INFO:tasks.workunit.client.0.smithi023.stdout:releasing clients 2024-03-24T10:41:44.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 3509 38.35 MB/sec warmup 1 sec latency 6.681 ms 2024-03-24T10:41:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:44 smithi023 ceph-mon[32466]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-24T10:41:45.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:44 smithi117 ceph-mon[36368]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-24T10:41:45.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:44 smithi116 ceph-mon[36248]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-24T10:41:45.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 6154 24.06 MB/sec warmup 2 sec latency 14.890 ms 2024-03-24T10:41:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.968+0000 7ff6e98bb640 1 -- 172.21.15.23:0/3449400332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e41055b0 msgr2=0x7ff6e4105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.968+0000 7ff6e98bb640 1 --2- 172.21.15.23:0/3449400332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e41055b0 0x7ff6e4105990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4009900 tx=0x7ff6d402bc30 comp rx=0 tx=0).stop 2024-03-24T10:41:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.968+0000 7ff6e98bb640 1 -- 172.21.15.23:0/3449400332 shutdown_connections 2024-03-24T10:41:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.968+0000 7ff6e98bb640 1 --2- 172.21.15.23:0/3449400332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6e4101220 0x7ff6e410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.968+0000 7ff6e98bb640 1 --2- 172.21.15.23:0/3449400332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e4100860 0x7ff6e4100ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.968+0000 7ff6e98bb640 1 --2- 172.21.15.23:0/3449400332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e41055b0 0x7ff6e4105990 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.968+0000 7ff6e98bb640 1 -- 172.21.15.23:0/3449400332 >> 172.21.15.23:0/3449400332 conn(0x7ff6e4077460 msgr2=0x7ff6e4077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.968+0000 7ff6e98bb640 1 -- 172.21.15.23:0/3449400332 shutdown_connections 2024-03-24T10:41:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.968+0000 7ff6e98bb640 1 -- 172.21.15.23:0/3449400332 wait complete. 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e98bb640 1 Processor -- start 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e98bb640 1 -- start start 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e98bb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e4100860 0x7ff6e4102280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e98bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e4101220 0x7ff6e41027c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e98bb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6e41055b0 0x7ff6e4102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e98bb640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e41a60f0 con 0x7ff6e4100860 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e98bb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e41a6260 con 0x7ff6e4101220 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e98bb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e41a6420 con 0x7ff6e41055b0 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e4100860 0x7ff6e4102280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e4100860 0x7ff6e4102280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40564/0 (socket says 172.21.15.23:40564) 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e2ffd640 1 -- 172.21.15.23:0/4094037465 learned_addr learned my addr 172.21.15.23:0/4094037465 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6dbfff640 1 --2- 172.21.15.23:0/4094037465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e4101220 0x7ff6e41027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e37fe640 1 --2- 172.21.15.23:0/4094037465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6e41055b0 0x7ff6e4102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e2ffd640 1 -- 172.21.15.23:0/4094037465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6e41055b0 msgr2=0x7ff6e4102d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e2ffd640 1 --2- 172.21.15.23:0/4094037465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6e41055b0 0x7ff6e4102d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e2ffd640 1 -- 172.21.15.23:0/4094037465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e4101220 msgr2=0x7ff6e41027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:45.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e2ffd640 1 --2- 172.21.15.23:0/4094037465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e4101220 0x7ff6e41027c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.969+0000 7ff6e2ffd640 1 -- 172.21.15.23:0/4094037465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d4009560 con 0x7ff6e4100860 2024-03-24T10:41:45.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.970+0000 7ff6e2ffd640 1 --2- 172.21.15.23:0/4094037465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e4100860 0x7ff6e4102280 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4000c00 tx=0x7ff6d402c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:45.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.970+0000 7ff6e88b9640 1 -- 172.21.15.23:0/4094037465 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d402dea0 con 0x7ff6e4100860 2024-03-24T10:41:45.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.970+0000 7ff6e88b9640 1 -- 172.21.15.23:0/4094037465 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6d4004410 con 0x7ff6e4100860 2024-03-24T10:41:45.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.970+0000 7ff6e98bb640 1 -- 172.21.15.23:0/4094037465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6e41a66a0 con 0x7ff6e4100860 2024-03-24T10:41:45.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.970+0000 7ff6e88b9640 1 -- 172.21.15.23:0/4094037465 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4003bb0 con 0x7ff6e4100860 2024-03-24T10:41:45.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.970+0000 7ff6e98bb640 1 -- 172.21.15.23:0/4094037465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6e41a6bb0 con 0x7ff6e4100860 2024-03-24T10:41:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.972+0000 7ff6e88b9640 1 -- 172.21.15.23:0/4094037465 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7ff6d4003d10 con 0x7ff6e4100860 2024-03-24T10:41:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.972+0000 7ff6ec570740 1 Processor -- start 2024-03-24T10:41:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.972+0000 7ff6ec570740 1 -- start start 2024-03-24T10:41:45.979 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.972+0000 7ff6ec570740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 0x55b5f1a909c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.972+0000 7ff6ec570740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a99790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.972+0000 7ff6ec570740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a9c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.972+0000 7ff6ec570740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b5f1a4b1b0 con 0x55b5f1a99cd0 2024-03-24T10:41:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.972+0000 7ff6ec570740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5f19fdf80 con 0x55b5f1a905e0 2024-03-24T10:41:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.972+0000 7ff6ec570740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b5f19fb4b0 con 0x55b5f1a90f00 2024-03-24T10:41:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6e88b9640 1 --2- 172.21.15.23:0/4094037465 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff6d00a22d0 0x7ff6d00a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6dbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a9c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6dbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a9c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40566/0 (socket says 172.21.15.23:40566) 2024-03-24T10:41:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6dbfff640 1 -- 172.21.15.23:0/349846748 learned_addr learned my addr 172.21.15.23:0/349846748 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6e2ffd640 1 --2- 172.21.15.23:0/349846748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a99790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6e37fe640 1 --2- 172.21.15.23:0/349846748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 0x55b5f1a909c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6e37fe640 1 --2- 172.21.15.23:0/4094037465 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff6d00a22d0 0x7ff6d00a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:45.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6e88b9640 1 -- 172.21.15.23:0/4094037465 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7ff6d40f0560 con 0x7ff6e4100860 2024-03-24T10:41:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6dbfff640 1 -- 172.21.15.23:0/349846748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 msgr2=0x55b5f1a99790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6dbfff640 1 --2- 172.21.15.23:0/349846748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a99790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6dbfff640 1 -- 172.21.15.23:0/349846748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 msgr2=0x55b5f1a909c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6dbfff640 1 --2- 172.21.15.23:0/349846748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 0x55b5f1a909c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6dbfff640 1 -- 172.21.15.23:0/349846748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5f198c810 con 0x55b5f1a99cd0 2024-03-24T10:41:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6e2ffd640 1 --2- 172.21.15.23:0/349846748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a99790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6dbfff640 1 --2- 172.21.15.23:0/349846748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a9c0c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009c90 tx=0x7ff6cc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0bfada7ed48912d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6e37fe640 1 --2- 172.21.15.23:0/4094037465 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff6d00a22d0 0x7ff6d00a4790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc005fd0 tx=0x7ff6dc009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6d9ffb640 1 -- 172.21.15.23:0/349846748 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc019630 con 0x55b5f1a99cd0 2024-03-24T10:41:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6d9ffb640 1 -- 172.21.15.23:0/349846748 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6cc019c70 con 0x55b5f1a99cd0 2024-03-24T10:41:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.973+0000 7ff6d9ffb640 1 -- 172.21.15.23:0/349846748 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc00eb90 con 0x55b5f1a99cd0 2024-03-24T10:41:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.974+0000 7ff6ec570740 1 -- 172.21.15.23:0/349846748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 msgr2=0x55b5f1a9c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:45.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.974+0000 7ff6ec570740 1 --2- 172.21.15.23:0/349846748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a9c0c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009c90 tx=0x7ff6cc011810 comp rx=0 tx=0).stop 2024-03-24T10:41:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.974+0000 7ff6ec570740 1 -- 172.21.15.23:0/349846748 shutdown_connections 2024-03-24T10:41:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.974+0000 7ff6ec570740 1 --2- 172.21.15.23:0/349846748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a9c0c0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.974+0000 7ff6ec570740 1 --2- 172.21.15.23:0/349846748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a99790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.974+0000 7ff6ec570740 1 --2- 172.21.15.23:0/349846748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 0x55b5f1a909c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.974+0000 7ff6ec570740 1 -- 172.21.15.23:0/349846748 >> 172.21.15.23:0/349846748 conn(0x55b5f1a8af00 msgr2=0x55b5f1a8b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.974+0000 7ff6ec570740 1 -- 172.21.15.23:0/349846748 shutdown_connections 2024-03-24T10:41:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.974+0000 7ff6ec570740 1 -- 172.21.15.23:0/349846748 wait complete. 2024-03-24T10:41:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.974+0000 7ff6ec570740 1 -- start start 2024-03-24T10:41:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6ec570740 1 Processor -- start 2024-03-24T10:41:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6ec570740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 0x55b5f1a96d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6ec570740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a97260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6ec570740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a977a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:45.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6ec570740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b5f19f6c90 con 0x55b5f1a99cd0 2024-03-24T10:41:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6ec570740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5f199bc50 con 0x55b5f1a905e0 2024-03-24T10:41:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6ec570740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b5f16ca250 con 0x55b5f1a90f00 2024-03-24T10:41:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6ec570740 10 client.0 _ll_register_callbacks cb 0x55b5f19ea290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6e37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6e37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40578/0 (socket says 172.21.15.23:40578) 2024-03-24T10:41:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6e37fe640 1 -- 172.21.15.23:0/2127834215 learned_addr learned my addr 172.21.15.23:0/2127834215 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6dbfff640 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a97260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6e2ffd640 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 0x55b5f1a96d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6e37fe640 1 -- 172.21.15.23:0/2127834215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 msgr2=0x55b5f1a97260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6e37fe640 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a97260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6e37fe640 1 -- 172.21.15.23:0/2127834215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 msgr2=0x55b5f1a96d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6e37fe640 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 0x55b5f1a96d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6e37fe640 1 -- 172.21.15.23:0/2127834215 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6cc0098f0 con 0x55b5f1a99cd0 2024-03-24T10:41:45.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6dbfff640 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a97260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6e2ffd640 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 0x55b5f1a96d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6e37fe640 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a977a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00ff50 tx=0x7ff6dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fccaa97d47540c37 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.975+0000 7ff6c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6c2ffd640 1 -- 172.21.15.23:0/2127834215 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc024070 con 0x55b5f1a99cd0 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6c2ffd640 1 -- 172.21.15.23:0/2127834215 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7ff6dc01fd20 con 0x55b5f1a99cd0 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6ec570740 1 -- 172.21.15.23:0/2127834215 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b5f1a1b790 con 0x55b5f1a99cd0 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6c2ffd640 1 -- 172.21.15.23:0/2127834215 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff6dc02a420 con 0x55b5f1a99cd0 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6ec570740 10 client.15981 fetch_fsmap learned FSMap version 24 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6ec570740 1 -- 172.21.15.23:0/2127834215 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b5f19f5310 con 0x55b5f1a99cd0 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6c2ffd640 1 -- 172.21.15.23:0/2127834215 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6dc037020 con 0x55b5f1a99cd0 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6c2ffd640 1 -- 172.21.15.23:0/2127834215 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) ==== 8342+0+0 (secure 0 0 0) 0x7ff6dc0324d0 con 0x55b5f1a99cd0 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6ec570740 10 client.15981 fetch_fsmap finished waiting for FSMap version 24 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6ec570740 10 client.15981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6ec570740 20 client.15981 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6ec570740 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5f1aa9c80 0x55b5f1aac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:45.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6ec570740 4 client.15981 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6ec570740 1 -- 172.21.15.23:0/2127834215 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5f16f6600 con 0x55b5f1aa9c80 2024-03-24T10:41:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.976+0000 7ff6e2ffd640 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5f1aa9c80 0x55b5f1aac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.977+0000 7ff6e2ffd640 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5f1aa9c80 0x55b5f1aac0a0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64971d04e7896039 server_cookie=adaf6a7278301bf2 in_seq=0 out_seq=0 2024-03-24T10:41:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.977+0000 7ff6c2ffd640 10 client.15981 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6c2ffd640 1 -- 172.21.15.23:0/2127834215 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b5f16f6600 con 0x55b5f1aa9c80 2024-03-24T10:41:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6c2ffd640 10 client.15981 handle_command_reply: tid=0 2024-03-24T10:41:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6ec570740 1 client.15981 shutdown 2024-03-24T10:41:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6ec570740 1 -- 172.21.15.23:0/2127834215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 msgr2=0x55b5f1a977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6ec570740 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a977a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00ff50 tx=0x7ff6dc004a40 comp rx=0 tx=0).stop 2024-03-24T10:41:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6ec570740 1 -- 172.21.15.23:0/2127834215 shutdown_connections 2024-03-24T10:41:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6ec570740 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a977a0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6ec570740 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5f1aa9c80 0x55b5f1aac0a0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6ec570740 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a97260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6ec570740 1 --2- 172.21.15.23:0/2127834215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 0x55b5f1a96d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6c2ffd640 0 client.15981 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6ec570740 1 -- 172.21.15.23:0/2127834215 >> 172.21.15.23:0/2127834215 conn(0x55b5f1a8af00 msgr2=0x55b5f1a8cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6ec570740 1 -- 172.21.15.23:0/2127834215 shutdown_connections 2024-03-24T10:41:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6ec570740 1 -- 172.21.15.23:0/2127834215 wait complete. 2024-03-24T10:41:45.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.978+0000 7ff6ec570740 20 client.15981 trim_cache size 0 max 16384 2024-03-24T10:41:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6ec570740 1 Processor -- start 2024-03-24T10:41:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6ec570740 1 -- start start 2024-03-24T10:41:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6ec570740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a905e0 0x55b5f1aa53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6ec570740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a90f00 0x55b5f1a8b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6ec570740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a975c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6ec570740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b5f197c660 con 0x55b5f1a90f00 2024-03-24T10:41:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6ec570740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5f1a8f8c0 con 0x55b5f1a99cd0 2024-03-24T10:41:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6ec570740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b5f1a8bf00 con 0x55b5f1a905e0 2024-03-24T10:41:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6dbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a90f00 0x55b5f1a8b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6dbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a90f00 0x55b5f1a8b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40582/0 (socket says 172.21.15.23:40582) 2024-03-24T10:41:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6dbfff640 1 -- 172.21.15.23:0/2014125788 learned_addr learned my addr 172.21.15.23:0/2014125788 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6e37fe640 1 --2- 172.21.15.23:0/2014125788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6e2ffd640 1 --2- 172.21.15.23:0/2014125788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a905e0 0x55b5f1aa53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6dbfff640 1 -- 172.21.15.23:0/2014125788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a905e0 msgr2=0x55b5f1aa53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6dbfff640 1 --2- 172.21.15.23:0/2014125788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a905e0 0x55b5f1aa53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6dbfff640 1 -- 172.21.15.23:0/2014125788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a99cd0 msgr2=0x55b5f1a975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6dbfff640 1 --2- 172.21.15.23:0/2014125788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.993+0000 7ff6dbfff640 1 -- 172.21.15.23:0/2014125788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5f1991810 con 0x55b5f1a90f00 2024-03-24T10:41:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.994+0000 7ff6dbfff640 1 --2- 172.21.15.23:0/2014125788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a90f00 0x55b5f1a8b2e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009820 tx=0x7ff6cc00e650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1198eb0a58de82 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.994+0000 7ff6d97fa640 1 -- 172.21.15.23:0/2014125788 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc00e420 con 0x55b5f1a90f00 2024-03-24T10:41:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.994+0000 7ff6d97fa640 1 -- 172.21.15.23:0/2014125788 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6cc005520 con 0x55b5f1a90f00 2024-03-24T10:41:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.994+0000 7ff6d97fa640 1 -- 172.21.15.23:0/2014125788 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc019860 con 0x55b5f1a90f00 2024-03-24T10:41:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.994+0000 7ff6ec570740 1 -- 172.21.15.23:0/2014125788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a90f00 msgr2=0x55b5f1a8b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.994+0000 7ff6ec570740 1 --2- 172.21.15.23:0/2014125788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a90f00 0x55b5f1a8b2e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009820 tx=0x7ff6cc00e650 comp rx=0 tx=0).stop 2024-03-24T10:41:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 -- 172.21.15.23:0/2014125788 shutdown_connections 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 --2- 172.21.15.23:0/2014125788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 --2- 172.21.15.23:0/2014125788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a90f00 0x55b5f1a8b2e0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 --2- 172.21.15.23:0/2014125788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a905e0 0x55b5f1aa53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 -- 172.21.15.23:0/2014125788 >> 172.21.15.23:0/2014125788 conn(0x55b5f1a8af00 msgr2=0x55b5f1a89ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 -- 172.21.15.23:0/2014125788 shutdown_connections 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 -- 172.21.15.23:0/2014125788 wait complete. 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 -- start start 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 Processor -- start 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 0x55b5f1a93a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a93f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a944b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b5f1b0f010 con 0x55b5f1a99cd0 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b5f1b0f180 con 0x55b5f1a905e0 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b5f1b0f340 con 0x55b5f1a90f00 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6ec570740 10 client.0 _ll_register_callbacks cb 0x7ff6dc00fdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6e37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a944b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6e37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a944b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40590/0 (socket says 172.21.15.23:40590) 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6e37fe640 1 -- 172.21.15.23:0/3016335517 learned_addr learned my addr 172.21.15.23:0/3016335517 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6dbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a93f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.995+0000 7ff6e2ffd640 1 --2- 172.21.15.23:0/3016335517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 0x55b5f1a93a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.996+0000 7ff6dbfff640 1 -- 172.21.15.23:0/3016335517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 msgr2=0x55b5f1a93a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.996+0000 7ff6dbfff640 1 --2- 172.21.15.23:0/3016335517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 0x55b5f1a93a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.996+0000 7ff6dbfff640 1 -- 172.21.15.23:0/3016335517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 msgr2=0x55b5f1a944b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.996+0000 7ff6dbfff640 1 --2- 172.21.15.23:0/3016335517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a944b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.996+0000 7ff6dbfff640 1 -- 172.21.15.23:0/3016335517 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5f198c810 con 0x55b5f1a90f00 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.996+0000 7ff6e37fe640 1 --2- 172.21.15.23:0/3016335517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a944b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.996+0000 7ff6dbfff640 1 --2- 172.21.15.23:0/3016335517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a93f70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55b5f1a85bc0 tx=0x7ff6cc00eed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=274dacf11bf3fc47 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.996+0000 7ff6c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.996+0000 7ff6c2ffd640 1 -- 172.21.15.23:0/3016335517 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc005280 con 0x55b5f1a90f00 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.996+0000 7ff6c2ffd640 1 -- 172.21.15.23:0/3016335517 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7ff6cc019430 con 0x55b5f1a90f00 2024-03-24T10:41:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.996+0000 7ff6ec570740 1 -- 172.21.15.23:0/3016335517 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6cc019c70 con 0x55b5f1a90f00 2024-03-24T10:41:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.996+0000 7ff6c2ffd640 1 -- 172.21.15.23:0/3016335517 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff6cc0195f0 con 0x55b5f1a90f00 2024-03-24T10:41:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.996+0000 7ff6ec570740 10 client.25837 fetch_fsmap learned FSMap version 24 2024-03-24T10:41:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.996+0000 7ff6ec570740 1 -- 172.21.15.23:0/3016335517 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b5f19b75a0 con 0x55b5f1a90f00 2024-03-24T10:41:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.997+0000 7ff6c2ffd640 1 -- 172.21.15.23:0/3016335517 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6cc019c70 con 0x55b5f1a90f00 2024-03-24T10:41:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.997+0000 7ff6c2ffd640 1 -- 172.21.15.23:0/3016335517 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 24) ==== 8342+0+0 (secure 0 0 0) 0x7ff6cc02a020 con 0x55b5f1a90f00 2024-03-24T10:41:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.997+0000 7ff6ec570740 10 client.25837 fetch_fsmap finished waiting for FSMap version 24 2024-03-24T10:41:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.997+0000 7ff6ec570740 10 client.25837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.997+0000 7ff6ec570740 20 client.25837 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.997+0000 7ff6ec570740 1 --2- 172.21.15.23:0/3016335517 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5f1b10650 0x55b5f1b12a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.997+0000 7ff6ec570740 4 client.25837 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:41:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.997+0000 7ff6ec570740 1 -- 172.21.15.23:0/3016335517 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b5f17471b0 con 0x55b5f1b10650 2024-03-24T10:41:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.997+0000 7ff6e2ffd640 1 --2- 172.21.15.23:0/3016335517 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5f1b10650 0x55b5f1b12a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.997+0000 7ff6e2ffd640 1 --2- 172.21.15.23:0/3016335517 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5f1b10650 0x55b5f1b12a70 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f22ff2588afab95 server_cookie=8f65ec601b6dd262 in_seq=0 out_seq=0 2024-03-24T10:41:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:45.997+0000 7ff6c2ffd640 10 client.25837 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.022+0000 7ff6c2ffd640 1 -- 172.21.15.23:0/3016335517 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b5f17471b0 con 0x55b5f1b10650 2024-03-24T10:41:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.022+0000 7ff6c2ffd640 10 client.25837 handle_command_reply: tid=0 2024-03-24T10:41:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.022+0000 7ff6ec570740 1 client.25837 shutdown 2024-03-24T10:41:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.022+0000 7ff6ec570740 1 -- 172.21.15.23:0/3016335517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 msgr2=0x55b5f1a93f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.022+0000 7ff6ec570740 1 --2- 172.21.15.23:0/3016335517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a93f70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55b5f1a85bc0 tx=0x7ff6cc00eed0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.022+0000 7ff6ec570740 1 -- 172.21.15.23:0/3016335517 shutdown_connections 2024-03-24T10:41:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.023+0000 7ff6ec570740 1 --2- 172.21.15.23:0/3016335517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b5f1a99cd0 0x55b5f1a944b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.023+0000 7ff6ec570740 1 --2- 172.21.15.23:0/3016335517 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b5f1b10650 0x55b5f1b12a70 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.023+0000 7ff6ec570740 1 --2- 172.21.15.23:0/3016335517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b5f1a90f00 0x55b5f1a93f70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.023+0000 7ff6ec570740 1 --2- 172.21.15.23:0/3016335517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b5f1a905e0 0x55b5f1a93a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.023+0000 7ff6ec570740 1 -- 172.21.15.23:0/3016335517 >> 172.21.15.23:0/3016335517 conn(0x55b5f1a8af00 msgr2=0x55b5f1a89ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.023+0000 7ff6c2ffd640 0 client.25837 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.023+0000 7ff6ec570740 1 -- 172.21.15.23:0/3016335517 shutdown_connections 2024-03-24T10:41:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.023+0000 7ff6ec570740 1 -- 172.21.15.23:0/3016335517 wait complete. 2024-03-24T10:41:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.023+0000 7ff6ec570740 20 client.25837 trim_cache size 0 max 16384 2024-03-24T10:41:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:41:46.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:41:46.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:41:46.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:41:46.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.023+0000 7ff6e98bb640 1 -- 172.21.15.23:0/4094037465 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff6d00a22d0 msgr2=0x7ff6d00a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:46.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.023+0000 7ff6e98bb640 1 --2- 172.21.15.23:0/4094037465 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff6d00a22d0 0x7ff6d00a4790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc005fd0 tx=0x7ff6dc009500 comp rx=0 tx=0).stop 2024-03-24T10:41:46.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.023+0000 7ff6e98bb640 1 -- 172.21.15.23:0/4094037465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e4100860 msgr2=0x7ff6e4102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.023+0000 7ff6e98bb640 1 --2- 172.21.15.23:0/4094037465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e4100860 0x7ff6e4102280 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4000c00 tx=0x7ff6d402c6b0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.024+0000 7ff6e98bb640 1 -- 172.21.15.23:0/4094037465 shutdown_connections 2024-03-24T10:41:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.024+0000 7ff6e98bb640 1 --2- 172.21.15.23:0/4094037465 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff6d00a22d0 0x7ff6d00a4790 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.024+0000 7ff6e98bb640 1 --2- 172.21.15.23:0/4094037465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6e41055b0 0x7ff6e4102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.024+0000 7ff6e98bb640 1 --2- 172.21.15.23:0/4094037465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e4101220 0x7ff6e41027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.024+0000 7ff6e98bb640 1 --2- 172.21.15.23:0/4094037465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6e4100860 0x7ff6e4102280 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.024+0000 7ff6e98bb640 1 -- 172.21.15.23:0/4094037465 >> 172.21.15.23:0/4094037465 conn(0x7ff6e4077460 msgr2=0x7ff6e4068f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.024+0000 7ff6e98bb640 1 -- 172.21.15.23:0/4094037465 shutdown_connections 2024-03-24T10:41:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:46.024+0000 7ff6e98bb640 1 -- 172.21.15.23:0/4094037465 wait complete. 2024-03-24T10:41:46.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 8173 20.38 MB/sec warmup 3 sec latency 48.339 ms 2024-03-24T10:41:47.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:46 smithi117 ceph-mon[36368]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-24T10:41:47.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:46 smithi117 ceph-mon[36368]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:41:47.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:46 smithi116 ceph-mon[36248]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-24T10:41:47.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:46 smithi116 ceph-mon[36248]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:41:47.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 10629 19.15 MB/sec warmup 4 sec latency 295.228 ms 2024-03-24T10:41:47.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:46 smithi023 ceph-mon[32466]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-24T10:41:47.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:46 smithi023 ceph-mon[32466]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:41:48.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 10629 15.32 MB/sec warmup 5 sec latency 1295.263 ms 2024-03-24T10:41:49.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:48 smithi117 ceph-mon[36368]: pgmap v54: 97 pgs: 97 active+clean; 6.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 491 KiB/s wr, 79 op/s 2024-03-24T10:41:49.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:48 smithi116 ceph-mon[36248]: pgmap v54: 97 pgs: 97 active+clean; 6.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 491 KiB/s wr, 79 op/s 2024-03-24T10:41:49.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 10918 13.00 MB/sec warmup 6 sec latency 2115.848 ms 2024-03-24T10:41:49.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:48 smithi023 ceph-mon[32466]: pgmap v54: 97 pgs: 97 active+clean; 6.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 491 KiB/s wr, 79 op/s 2024-03-24T10:41:49.624 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:41:49.625 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:41:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.743+0000 7fe45a593640 1 -- 172.21.15.23:0/3947592306 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454100f60 msgr2=0x7fe45410d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.743+0000 7fe45a593640 1 --2- 172.21.15.23:0/3947592306 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454100f60 0x7fe45410d420 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009ca0 tx=0x7fe44c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.743+0000 7fe45a593640 1 -- 172.21.15.23:0/3947592306 shutdown_connections 2024-03-24T10:41:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.743+0000 7fe45a593640 1 --2- 172.21.15.23:0/3947592306 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454100f60 0x7fe45410d420 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.743+0000 7fe45a593640 1 --2- 172.21.15.23:0/3947592306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4541005a0 0x7fe454100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.743+0000 7fe45a593640 1 --2- 172.21.15.23:0/3947592306 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe454106660 0x7fe454106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.743+0000 7fe45a593640 1 -- 172.21.15.23:0/3947592306 >> 172.21.15.23:0/3947592306 conn(0x7fe454077710 msgr2=0x7fe454077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.743+0000 7fe45a593640 1 -- 172.21.15.23:0/3947592306 shutdown_connections 2024-03-24T10:41:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.743+0000 7fe45a593640 1 -- 172.21.15.23:0/3947592306 wait complete. 2024-03-24T10:41:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.744+0000 7fe45a593640 1 Processor -- start 2024-03-24T10:41:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.744+0000 7fe45a593640 1 -- start start 2024-03-24T10:41:49.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.744+0000 7fe45a593640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4541005a0 0x7fe45419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.744+0000 7fe45a593640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe454100f60 0x7fe45419ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.744+0000 7fe45a593640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454106660 0x7fe454196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.744+0000 7fe45a593640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe454196f90 con 0x7fe454106660 2024-03-24T10:41:49.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.744+0000 7fe45a593640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe454197100 con 0x7fe4541005a0 2024-03-24T10:41:49.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.744+0000 7fe45a593640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe4541972c0 con 0x7fe454100f60 2024-03-24T10:41:49.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.744+0000 7fe458b30640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454106660 0x7fe454196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.744+0000 7fe458b30640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454106660 0x7fe454196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51552/0 (socket says 172.21.15.23:51552) 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.744+0000 7fe4537fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe454100f60 0x7fe45419ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.744+0000 7fe453fff640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4541005a0 0x7fe45419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.744+0000 7fe458b30640 1 -- 172.21.15.23:0/2803853454 learned_addr learned my addr 172.21.15.23:0/2803853454 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.745+0000 7fe458b30640 1 -- 172.21.15.23:0/2803853454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe454100f60 msgr2=0x7fe45419ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.745+0000 7fe458b30640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe454100f60 0x7fe45419ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.745+0000 7fe458b30640 1 -- 172.21.15.23:0/2803853454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4541005a0 msgr2=0x7fe45419c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.745+0000 7fe458b30640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4541005a0 0x7fe45419c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.745+0000 7fe458b30640 1 -- 172.21.15.23:0/2803853454 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe44c009900 con 0x7fe454106660 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.745+0000 7fe4537fe640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe454100f60 0x7fe45419ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.745+0000 7fe453fff640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4541005a0 0x7fe45419c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.745+0000 7fe458b30640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454106660 0x7fe454196940 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe44c037040 tx=0x7fe44c032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.745+0000 7fe4517fa640 1 -- 172.21.15.23:0/2803853454 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c0326a0 con 0x7fe454106660 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.745+0000 7fe4517fa640 1 -- 172.21.15.23:0/2803853454 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe44c003e50 con 0x7fe454106660 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.745+0000 7fe45a593640 1 -- 172.21.15.23:0/2803853454 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe454197540 con 0x7fe454106660 2024-03-24T10:41:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.745+0000 7fe4517fa640 1 -- 172.21.15.23:0/2803853454 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c0046a0 con 0x7fe454106660 2024-03-24T10:41:49.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.745+0000 7fe45a593640 1 -- 172.21.15.23:0/2803853454 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe454068a50 con 0x7fe454106660 2024-03-24T10:41:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.747+0000 7fe4517fa640 1 -- 172.21.15.23:0/2803853454 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fe44c03b040 con 0x7fe454106660 2024-03-24T10:41:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe45d173740 1 Processor -- start 2024-03-24T10:41:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe45d173740 1 -- start start 2024-03-24T10:41:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe4517fa640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe43c0a2230 0x7fe43c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe45d173740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe45d173740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a72af0 0x5650a2a7b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe45d173740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a7b6d0 0x5650a2a7dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe45d173740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5650a2996f30 con 0x5650a2a72af0 2024-03-24T10:41:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe45d173740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5650a29e0810 con 0x5650a2a721d0 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe45d173740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5650a29ddd40 con 0x5650a2a7b6d0 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe4537fe640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe43c0a2230 0x7fe43c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe4517fa640 1 -- 172.21.15.23:0/2803853454 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe44c0f08e0 con 0x7fe454106660 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe4537fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a72af0 0x5650a2a7b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe453fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe4537fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a72af0 0x5650a2a7b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51556/0 (socket says 172.21.15.23:51556) 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe4537fe640 1 -- 172.21.15.23:0/427290251 learned_addr learned my addr 172.21.15.23:0/427290251 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe458b30640 1 --2- 172.21.15.23:0/427290251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a7b6d0 0x5650a2a7dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe453fff640 1 -- 172.21.15.23:0/427290251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a7b6d0 msgr2=0x5650a2a7dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe453fff640 1 --2- 172.21.15.23:0/427290251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a7b6d0 0x5650a2a7dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe453fff640 1 -- 172.21.15.23:0/427290251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a72af0 msgr2=0x5650a2a7b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe453fff640 1 --2- 172.21.15.23:0/427290251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a72af0 0x5650a2a7b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.748+0000 7fe453fff640 1 -- 172.21.15.23:0/427290251 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650a299aea0 con 0x5650a2a721d0 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.749+0000 7fe4537fe640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe43c0a2230 0x7fe43c0a46f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe448009fd0 tx=0x7fe448009f20 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.749+0000 7fe453fff640 1 --2- 172.21.15.23:0/427290251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a725b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe444009f20 tx=0x7fe444005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5b50a1cea852f15 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.749+0000 7fe43a7fc640 1 -- 172.21.15.23:0/427290251 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44401c070 con 0x5650a2a721d0 2024-03-24T10:41:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.749+0000 7fe43a7fc640 1 -- 172.21.15.23:0/427290251 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe444014040 con 0x5650a2a721d0 2024-03-24T10:41:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.749+0000 7fe43a7fc640 1 -- 172.21.15.23:0/427290251 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe444013050 con 0x5650a2a721d0 2024-03-24T10:41:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.749+0000 7fe45d173740 1 -- 172.21.15.23:0/427290251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 msgr2=0x5650a2a725b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.749+0000 7fe45d173740 1 --2- 172.21.15.23:0/427290251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a725b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe444009f20 tx=0x7fe444005f90 comp rx=0 tx=0).stop 2024-03-24T10:41:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.750+0000 7fe45d173740 1 -- 172.21.15.23:0/427290251 shutdown_connections 2024-03-24T10:41:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.750+0000 7fe45d173740 1 --2- 172.21.15.23:0/427290251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a7b6d0 0x5650a2a7dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.750+0000 7fe45d173740 1 --2- 172.21.15.23:0/427290251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a72af0 0x5650a2a7b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.750+0000 7fe45d173740 1 --2- 172.21.15.23:0/427290251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a725b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.750+0000 7fe45d173740 1 -- 172.21.15.23:0/427290251 >> 172.21.15.23:0/427290251 conn(0x5650a2a6c900 msgr2=0x5650a2a6cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.750+0000 7fe45d173740 1 -- 172.21.15.23:0/427290251 shutdown_connections 2024-03-24T10:41:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.750+0000 7fe45d173740 1 -- 172.21.15.23:0/427290251 wait complete. 2024-03-24T10:41:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.750+0000 7fe45d173740 1 -- start start 2024-03-24T10:41:49.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.750+0000 7fe45d173740 1 Processor -- start 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe45d173740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a78790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe45d173740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a72af0 0x5650a2a78cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe45d173740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a7b6d0 0x5650a2a79210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe45d173740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5650a29d9520 con 0x5650a2a7b6d0 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe45d173740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5650a26ac430 con 0x5650a2a721d0 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe45d173740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5650a2668e00 con 0x5650a2a72af0 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe45d173740 10 client.0 _ll_register_callbacks cb 0x5650a2977d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe458b30640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a7b6d0 0x5650a2a79210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe453fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a78790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe4537fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a72af0 0x5650a2a78cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe453fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a78790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:49632/0 (socket says 172.21.15.23:49632) 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe458b30640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a7b6d0 0x5650a2a79210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51562/0 (socket says 172.21.15.23:51562) 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe453fff640 1 -- 172.21.15.23:0/2194827723 learned_addr learned my addr 172.21.15.23:0/2194827723 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe458b30640 1 -- 172.21.15.23:0/2194827723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a72af0 msgr2=0x5650a2a78cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:49.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe458b30640 1 --2- 172.21.15.23:0/2194827723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a72af0 0x5650a2a78cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe458b30640 1 -- 172.21.15.23:0/2194827723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 msgr2=0x5650a2a78790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe458b30640 1 --2- 172.21.15.23:0/2194827723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a78790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe458b30640 1 -- 172.21.15.23:0/2194827723 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe444009c00 con 0x5650a2a7b6d0 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe4537fe640 1 --2- 172.21.15.23:0/2194827723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a72af0 0x5650a2a78cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe458b30640 1 --2- 172.21.15.23:0/2194827723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a7b6d0 0x5650a2a79210 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe44c0ee140 tx=0x7fe44c0bcfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0e869b15153d61 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe4337fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe4337fe640 1 -- 172.21.15.23:0/2194827723 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c0bd470 con 0x5650a2a7b6d0 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe4337fe640 1 -- 172.21.15.23:0/2194827723 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe44c0c15c0 con 0x5650a2a7b6d0 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.751+0000 7fe45d173740 1 -- 172.21.15.23:0/2194827723 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5650a26d87e0 con 0x5650a2a7b6d0 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.752+0000 7fe4337fe640 1 -- 172.21.15.23:0/2194827723 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe44c0c5b80 con 0x5650a2a7b6d0 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.752+0000 7fe45d173740 10 client.15999 fetch_fsmap learned FSMap version 24 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.752+0000 7fe45d173740 1 -- 172.21.15.23:0/2194827723 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5650a29d7ba0 con 0x5650a2a7b6d0 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.752+0000 7fe4337fe640 1 -- 172.21.15.23:0/2194827723 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5650a26d87e0 con 0x5650a2a7b6d0 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.752+0000 7fe4337fe640 1 -- 172.21.15.23:0/2194827723 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) ==== 8342+0+0 (secure 0 0 0) 0x7fe44c0c56f0 con 0x5650a2a7b6d0 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.752+0000 7fe45d173740 10 client.15999 fetch_fsmap finished waiting for FSMap version 24 2024-03-24T10:41:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.752+0000 7fe45d173740 10 client.15999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.752+0000 7fe45d173740 20 client.15999 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.752+0000 7fe45d173740 1 --2- 172.21.15.23:0/2194827723 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650a2a8b970 0x5650a2a8dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.752+0000 7fe45d173740 4 client.15999 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.752+0000 7fe45d173740 1 -- 172.21.15.23:0/2194827723 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650a2715720 con 0x5650a2a8b970 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.752+0000 7fe453fff640 1 --2- 172.21.15.23:0/2194827723 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650a2a8b970 0x5650a2a8dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.753+0000 7fe453fff640 1 --2- 172.21.15.23:0/2194827723 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650a2a8b970 0x5650a2a8dd90 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83a7ea85fb8eaf53 server_cookie=80cf2606bc58a10c in_seq=0 out_seq=0 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.753+0000 7fe4337fe640 10 client.15999 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.753+0000 7fe4337fe640 1 -- 172.21.15.23:0/2194827723 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5650a2715720 con 0x5650a2a8b970 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.753+0000 7fe4337fe640 10 client.15999 handle_command_reply: tid=0 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.753+0000 7fe45d173740 1 client.15999 shutdown 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.754+0000 7fe45d173740 1 -- 172.21.15.23:0/2194827723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a7b6d0 msgr2=0x5650a2a79210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.754+0000 7fe45d173740 1 --2- 172.21.15.23:0/2194827723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a7b6d0 0x5650a2a79210 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe44c0ee140 tx=0x7fe44c0bcfb0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.754+0000 7fe45d173740 1 -- 172.21.15.23:0/2194827723 shutdown_connections 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.754+0000 7fe45d173740 1 --2- 172.21.15.23:0/2194827723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a7b6d0 0x5650a2a79210 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.754+0000 7fe45d173740 1 --2- 172.21.15.23:0/2194827723 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650a2a8b970 0x5650a2a8dd90 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.754+0000 7fe45d173740 1 --2- 172.21.15.23:0/2194827723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a72af0 0x5650a2a78cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.754+0000 7fe45d173740 1 --2- 172.21.15.23:0/2194827723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a78790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.754+0000 7fe4337fe640 0 client.15999 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:49.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.754+0000 7fe45d173740 1 -- 172.21.15.23:0/2194827723 >> 172.21.15.23:0/2194827723 conn(0x5650a2a6c900 msgr2=0x5650a2a6e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:49.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.754+0000 7fe45d173740 1 -- 172.21.15.23:0/2194827723 shutdown_connections 2024-03-24T10:41:49.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.754+0000 7fe45d173740 1 -- 172.21.15.23:0/2194827723 wait complete. 2024-03-24T10:41:49.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.754+0000 7fe45d173740 20 client.15999 trim_cache size 0 max 16384 2024-03-24T10:41:49.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.771+0000 7fe45d173740 1 Processor -- start 2024-03-24T10:41:49.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.771+0000 7fe45d173740 1 -- start start 2024-03-24T10:41:49.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe45d173740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a776d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe45d173740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a72af0 0x5650a2a86f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe45d173740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a7b6d0 0x5650a2a79030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe45d173740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5650a295e430 con 0x5650a2a7b6d0 2024-03-24T10:41:49.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe45d173740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5650a2a88670 con 0x5650a2a721d0 2024-03-24T10:41:49.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe45d173740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5650a2a874a0 con 0x5650a2a72af0 2024-03-24T10:41:49.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe458b30640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a7b6d0 0x5650a2a79030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe458b30640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a7b6d0 0x5650a2a79030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51570/0 (socket says 172.21.15.23:51570) 2024-03-24T10:41:49.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe458b30640 1 -- 172.21.15.23:0/3148052787 learned_addr learned my addr 172.21.15.23:0/3148052787 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:49.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe453fff640 1 --2- 172.21.15.23:0/3148052787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a776d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe4537fe640 1 --2- 172.21.15.23:0/3148052787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a72af0 0x5650a2a86f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe458b30640 1 -- 172.21.15.23:0/3148052787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a72af0 msgr2=0x5650a2a86f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:49.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe458b30640 1 --2- 172.21.15.23:0/3148052787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a72af0 0x5650a2a86f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe458b30640 1 -- 172.21.15.23:0/3148052787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 msgr2=0x5650a2a776d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:49.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe458b30640 1 --2- 172.21.15.23:0/3148052787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a776d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe458b30640 1 -- 172.21.15.23:0/3148052787 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650a2973b20 con 0x5650a2a7b6d0 2024-03-24T10:41:49.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe4537fe640 1 --2- 172.21.15.23:0/3148052787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a72af0 0x5650a2a86f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:49.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe453fff640 1 --2- 172.21.15.23:0/3148052787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a776d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:49.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe458b30640 1 --2- 172.21.15.23:0/3148052787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a7b6d0 0x5650a2a79030 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe44c0bb720 tx=0x7fe44c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a503e1bbafa4e927 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:49.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe439ffb640 1 -- 172.21.15.23:0/3148052787 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c0bcdd0 con 0x5650a2a7b6d0 2024-03-24T10:41:49.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe439ffb640 1 -- 172.21.15.23:0/3148052787 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe44c0c15c0 con 0x5650a2a7b6d0 2024-03-24T10:41:49.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.772+0000 7fe439ffb640 1 -- 172.21.15.23:0/3148052787 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c0b9ae0 con 0x5650a2a7b6d0 2024-03-24T10:41:49.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.773+0000 7fe45d173740 1 -- 172.21.15.23:0/3148052787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a7b6d0 msgr2=0x5650a2a79030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.773+0000 7fe45d173740 1 --2- 172.21.15.23:0/3148052787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a7b6d0 0x5650a2a79030 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe44c0bb720 tx=0x7fe44c004a40 comp rx=0 tx=0).stop 2024-03-24T10:41:49.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.773+0000 7fe45d173740 1 -- 172.21.15.23:0/3148052787 shutdown_connections 2024-03-24T10:41:49.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.773+0000 7fe45d173740 1 --2- 172.21.15.23:0/3148052787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a7b6d0 0x5650a2a79030 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.773+0000 7fe45d173740 1 --2- 172.21.15.23:0/3148052787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a72af0 0x5650a2a86f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.773+0000 7fe45d173740 1 --2- 172.21.15.23:0/3148052787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a721d0 0x5650a2a776d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.773+0000 7fe45d173740 1 -- 172.21.15.23:0/3148052787 >> 172.21.15.23:0/3148052787 conn(0x5650a2a6c900 msgr2=0x5650a2a6bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:49.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.773+0000 7fe45d173740 1 -- 172.21.15.23:0/3148052787 shutdown_connections 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.773+0000 7fe45d173740 1 -- 172.21.15.23:0/3148052787 wait complete. 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.773+0000 7fe45d173740 1 -- start start 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.773+0000 7fe45d173740 1 Processor -- start 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe45d173740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a721d0 0x5650a2a83980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe45d173740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a72af0 0x5650a2a75170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe45d173740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a7b6d0 0x5650a2a75860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe45d173740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5650a2a75f60 con 0x5650a2a721d0 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe45d173740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5650a2af0cb0 con 0x5650a2a72af0 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe45d173740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5650a2af0df0 con 0x5650a2a7b6d0 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe45d173740 10 client.0 _ll_register_callbacks cb 0x7fe44c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe453fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a721d0 0x5650a2a83980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe453fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a721d0 0x5650a2a83980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51576/0 (socket says 172.21.15.23:51576) 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe453fff640 1 -- 172.21.15.23:0/2306456451 learned_addr learned my addr 172.21.15.23:0/2306456451 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe458b30640 1 --2- 172.21.15.23:0/2306456451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a7b6d0 0x5650a2a75860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe4537fe640 1 --2- 172.21.15.23:0/2306456451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a72af0 0x5650a2a75170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe453fff640 1 -- 172.21.15.23:0/2306456451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a7b6d0 msgr2=0x5650a2a75860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe453fff640 1 --2- 172.21.15.23:0/2306456451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a7b6d0 0x5650a2a75860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe453fff640 1 -- 172.21.15.23:0/2306456451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a72af0 msgr2=0x5650a2a75170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe453fff640 1 --2- 172.21.15.23:0/2306456451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a72af0 0x5650a2a75170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe453fff640 1 -- 172.21.15.23:0/2306456451 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650a29d7ba0 con 0x5650a2a721d0 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe453fff640 1 --2- 172.21.15.23:0/2306456451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a721d0 0x5650a2a83980 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe444005b90 tx=0x7fe444005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f39702feb046626 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe4337fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe4337fe640 1 -- 172.21.15.23:0/2306456451 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44401c040 con 0x5650a2a721d0 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe4337fe640 1 -- 172.21.15.23:0/2306456451 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe44402bc40 con 0x5650a2a721d0 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.774+0000 7fe45d173740 1 -- 172.21.15.23:0/2306456451 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe444014040 con 0x5650a2a721d0 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.775+0000 7fe4337fe640 1 -- 172.21.15.23:0/2306456451 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe44402be00 con 0x5650a2a721d0 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.775+0000 7fe45d173740 10 client.16005 fetch_fsmap learned FSMap version 24 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.775+0000 7fe45d173740 1 -- 172.21.15.23:0/2306456451 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5650a2999700 con 0x5650a2a721d0 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.775+0000 7fe4337fe640 1 -- 172.21.15.23:0/2306456451 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe444014040 con 0x5650a2a721d0 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.775+0000 7fe4337fe640 1 -- 172.21.15.23:0/2306456451 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) ==== 8342+0+0 (secure 0 0 0) 0x7fe444022040 con 0x5650a2a721d0 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.775+0000 7fe45d173740 10 client.16005 fetch_fsmap finished waiting for FSMap version 24 2024-03-24T10:41:49.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.775+0000 7fe45d173740 10 client.16005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.775+0000 7fe45d173740 20 client.16005 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.775+0000 7fe45d173740 1 --2- 172.21.15.23:0/2306456451 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650a2af69a0 0x5650a2af8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.775+0000 7fe45d173740 4 client.16005 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.775+0000 7fe45d173740 1 -- 172.21.15.23:0/2306456451 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5650a2764890 con 0x5650a2af69a0 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.775+0000 7fe4537fe640 1 --2- 172.21.15.23:0/2306456451 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650a2af69a0 0x5650a2af8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.775+0000 7fe4337fe640 10 client.16005 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.775+0000 7fe4537fe640 1 --2- 172.21.15.23:0/2306456451 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650a2af69a0 0x5650a2af8dc0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0eb2393372fe847 server_cookie=efa029143954e3b6 in_seq=0 out_seq=0 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe4337fe640 1 -- 172.21.15.23:0/2306456451 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5650a2764890 con 0x5650a2af69a0 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe4337fe640 10 client.16005 handle_command_reply: tid=0 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45d173740 1 client.16005 shutdown 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45d173740 1 -- 172.21.15.23:0/2306456451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a721d0 msgr2=0x5650a2a83980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45d173740 1 --2- 172.21.15.23:0/2306456451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a721d0 0x5650a2a83980 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe444005b90 tx=0x7fe444005770 comp rx=0 tx=0).stop 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45d173740 1 -- 172.21.15.23:0/2306456451 shutdown_connections 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45d173740 1 --2- 172.21.15.23:0/2306456451 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5650a2af69a0 0x5650a2af8dc0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45d173740 1 --2- 172.21.15.23:0/2306456451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650a2a7b6d0 0x5650a2a75860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45d173740 1 --2- 172.21.15.23:0/2306456451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5650a2a72af0 0x5650a2a75170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe4337fe640 0 client.16005 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:49.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45d173740 1 --2- 172.21.15.23:0/2306456451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650a2a721d0 0x5650a2a83980 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45d173740 1 -- 172.21.15.23:0/2306456451 >> 172.21.15.23:0/2306456451 conn(0x5650a2a6c900 msgr2=0x5650a2a6d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:49.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45d173740 1 -- 172.21.15.23:0/2306456451 shutdown_connections 2024-03-24T10:41:49.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45d173740 1 -- 172.21.15.23:0/2306456451 wait complete. 2024-03-24T10:41:49.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45d173740 20 client.16005 trim_cache size 0 max 16384 2024-03-24T10:41:49.783 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:41:49.783 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:41:49.783 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:41:49.783 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:41:49.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45a593640 1 -- 172.21.15.23:0/2803853454 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe43c0a2230 msgr2=0x7fe43c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45a593640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe43c0a2230 0x7fe43c0a46f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe448009fd0 tx=0x7fe448009f20 comp rx=0 tx=0).stop 2024-03-24T10:41:49.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45a593640 1 -- 172.21.15.23:0/2803853454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454106660 msgr2=0x7fe454196940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.776+0000 7fe45a593640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454106660 0x7fe454196940 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe44c037040 tx=0x7fe44c032570 comp rx=0 tx=0).stop 2024-03-24T10:41:49.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.777+0000 7fe45a593640 1 -- 172.21.15.23:0/2803853454 shutdown_connections 2024-03-24T10:41:49.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.777+0000 7fe45a593640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe43c0a2230 0x7fe43c0a46f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.777+0000 7fe45a593640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454106660 0x7fe454196940 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.777+0000 7fe45a593640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe454100f60 0x7fe45419ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.777+0000 7fe45a593640 1 --2- 172.21.15.23:0/2803853454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4541005a0 0x7fe45419c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.777+0000 7fe45a593640 1 -- 172.21.15.23:0/2803853454 >> 172.21.15.23:0/2803853454 conn(0x7fe454077710 msgr2=0x7fe4540ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:49.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.777+0000 7fe45a593640 1 -- 172.21.15.23:0/2803853454 shutdown_connections 2024-03-24T10:41:49.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.777+0000 7fe45a593640 1 -- 172.21.15.23:0/2803853454 wait complete. 2024-03-24T10:41:49.789 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:41:49.789 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:41:49.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.891+0000 7f73ba084640 1 -- 172.21.15.23:0/1551240361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73b4100d40 msgr2=0x7f73b410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.891+0000 7f73ba084640 1 --2- 172.21.15.23:0/1551240361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73b4100d40 0x7f73b410d4e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009900 tx=0x7f73ac02bcd0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.891+0000 7f73ba084640 1 -- 172.21.15.23:0/1551240361 shutdown_connections 2024-03-24T10:41:49.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.891+0000 7f73ba084640 1 --2- 172.21.15.23:0/1551240361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73b4100d40 0x7f73b410d4e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.891+0000 7f73ba084640 1 --2- 172.21.15.23:0/1551240361 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73b4106fd0 0x7f73b4100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.891+0000 7f73ba084640 1 --2- 172.21.15.23:0/1551240361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73b41066b0 0x7f73b4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.891+0000 7f73ba084640 1 -- 172.21.15.23:0/1551240361 >> 172.21.15.23:0/1551240361 conn(0x7f73b4077710 msgr2=0x7f73b4077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:49.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.891+0000 7f73ba084640 1 -- 172.21.15.23:0/1551240361 shutdown_connections 2024-03-24T10:41:49.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.891+0000 7f73ba084640 1 -- 172.21.15.23:0/1551240361 wait complete. 2024-03-24T10:41:49.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.892+0000 7f73ba084640 1 Processor -- start 2024-03-24T10:41:49.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.892+0000 7f73ba084640 1 -- start start 2024-03-24T10:41:49.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.892+0000 7f73ba084640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73b4100d40 0x7f73b4197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.892+0000 7f73ba084640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73b41066b0 0x7f73b41982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.892+0000 7f73ba084640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73b4106fd0 0x7f73b419c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.892+0000 7f73ba084640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f73b419cbe0 con 0x7f73b41066b0 2024-03-24T10:41:49.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.892+0000 7f73ba084640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f73b419cd50 con 0x7f73b4106fd0 2024-03-24T10:41:49.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.892+0000 7f73ba084640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f73b419cf10 con 0x7f73b4100d40 2024-03-24T10:41:49.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.892+0000 7f73b2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73b41066b0 0x7f73b41982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.892+0000 7f73b2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73b41066b0 0x7f73b41982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51598/0 (socket says 172.21.15.23:51598) 2024-03-24T10:41:49.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.892+0000 7f73b2ffd640 1 -- 172.21.15.23:0/3143242000 learned_addr learned my addr 172.21.15.23:0/3143242000 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:49.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.892+0000 7f73b37fe640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73b4100d40 0x7f73b4197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.892+0000 7f73b3fff640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73b4106fd0 0x7f73b419c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.893+0000 7f73b2ffd640 1 -- 172.21.15.23:0/3143242000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73b4100d40 msgr2=0x7f73b4197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.893+0000 7f73b2ffd640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73b4100d40 0x7f73b4197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.893+0000 7f73b2ffd640 1 -- 172.21.15.23:0/3143242000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73b4106fd0 msgr2=0x7f73b419c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:49.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.893+0000 7f73b2ffd640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73b4106fd0 0x7f73b419c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:49.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.893+0000 7f73b2ffd640 1 -- 172.21.15.23:0/3143242000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ac009560 con 0x7f73b41066b0 2024-03-24T10:41:49.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.893+0000 7f73b37fe640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73b4100d40 0x7f73b4197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:49.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.893+0000 7f73b2ffd640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73b41066b0 0x7f73b41982b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f73a800e9c0 tx=0x7f73a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:49.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.893+0000 7f73b0ff9640 1 -- 172.21.15.23:0/3143242000 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a80096c0 con 0x7f73b41066b0 2024-03-24T10:41:49.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.893+0000 7f73b0ff9640 1 -- 172.21.15.23:0/3143242000 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73a8005500 con 0x7f73b41066b0 2024-03-24T10:41:49.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.893+0000 7f73ba084640 1 -- 172.21.15.23:0/3143242000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73b419d1f0 con 0x7f73b41066b0 2024-03-24T10:41:49.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.893+0000 7f73b0ff9640 1 -- 172.21.15.23:0/3143242000 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a8010430 con 0x7f73b41066b0 2024-03-24T10:41:49.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.893+0000 7f73ba084640 1 -- 172.21.15.23:0/3143242000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73b41a4a70 con 0x7f73b41066b0 2024-03-24T10:41:49.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.894+0000 7f73ba084640 1 -- 172.21.15.23:0/3143242000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73b4068500 con 0x7f73b41066b0 2024-03-24T10:41:49.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.895+0000 7f73b0ff9640 1 -- 172.21.15.23:0/3143242000 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f73a8005020 con 0x7f73b41066b0 2024-03-24T10:41:49.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.896+0000 7f73b0ff9640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f739c0a2230 0x7f739c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:49.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.896+0000 7f73b37fe640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f739c0a2230 0x7f739c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:49.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.896+0000 7f73b0ff9640 1 -- 172.21.15.23:0/3143242000 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f73a8014070 con 0x7f73b41066b0 2024-03-24T10:41:49.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.896+0000 7f73b37fe640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f739c0a2230 0x7f739c0a46f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f73b4101750 tx=0x7f73a4009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:49.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:49.901+0000 7f73b0ff9640 1 -- 172.21.15.23:0/3143242000 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f73a8097010 con 0x7f73b41066b0 2024-03-24T10:41:50.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 13259 12.34 MB/sec warmup 7 sec latency 23.752 ms 2024-03-24T10:41:50.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.123+0000 7f73ba084640 1 -- 172.21.15.23:0/3143242000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73b4199070 con 0x7f73b41066b0 2024-03-24T10:41:50.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.124+0000 7f73b0ff9640 1 -- 172.21.15.23:0/3143242000 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11867 (secure 0 0 0) 0x7f73a809a6d0 con 0x7f73b41066b0 2024-03-24T10:41:50.125 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:41:50.125 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:41:44.922822+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24505,24539,24503]},"id":1}]} 2024-03-24T10:41:50.125 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 24 2024-03-24T10:41:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.126+0000 7f73ba084640 1 -- 172.21.15.23:0/3143242000 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f739c0a2230 msgr2=0x7f739c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.126+0000 7f73ba084640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f739c0a2230 0x7f739c0a46f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f73b4101750 tx=0x7f73a4009500 comp rx=0 tx=0).stop 2024-03-24T10:41:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.126+0000 7f73ba084640 1 -- 172.21.15.23:0/3143242000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73b41066b0 msgr2=0x7f73b41982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.126+0000 7f73ba084640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73b41066b0 0x7f73b41982b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f73a800e9c0 tx=0x7f73a800ee80 comp rx=0 tx=0).stop 2024-03-24T10:41:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.126+0000 7f73ba084640 1 -- 172.21.15.23:0/3143242000 shutdown_connections 2024-03-24T10:41:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.126+0000 7f73ba084640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f739c0a2230 0x7f739c0a46f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.126+0000 7f73ba084640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73b4106fd0 0x7f73b419c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.126+0000 7f73ba084640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73b41066b0 0x7f73b41982b0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.126+0000 7f73ba084640 1 --2- 172.21.15.23:0/3143242000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73b4100d40 0x7f73b4197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.126+0000 7f73ba084640 1 -- 172.21.15.23:0/3143242000 >> 172.21.15.23:0/3143242000 conn(0x7f73b4077710 msgr2=0x7f73b40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.126+0000 7f73ba084640 1 -- 172.21.15.23:0/3143242000 shutdown_connections 2024-03-24T10:41:50.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.126+0000 7f73ba084640 1 -- 172.21.15.23:0/3143242000 wait complete. 2024-03-24T10:41:50.138 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:41:50.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.244+0000 7fa22dfa7640 1 -- 172.21.15.23:0/4255902144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2281041d0 msgr2=0x7fa228104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.244+0000 7fa22dfa7640 1 --2- 172.21.15.23:0/4255902144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2281041d0 0x7fa228104650 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa220009cd0 tx=0x7fa22002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.244+0000 7fa22dfa7640 1 -- 172.21.15.23:0/4255902144 shutdown_connections 2024-03-24T10:41:50.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.244+0000 7fa22dfa7640 1 --2- 172.21.15.23:0/4255902144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2281041d0 0x7fa228104650 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.244+0000 7fa22dfa7640 1 --2- 172.21.15.23:0/4255902144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228068e50 0x7fa2280692b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.244+0000 7fa22dfa7640 1 --2- 172.21.15.23:0/4255902144 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa228106660 0x7fa228068910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.244+0000 7fa22dfa7640 1 -- 172.21.15.23:0/4255902144 >> 172.21.15.23:0/4255902144 conn(0x7fa228077710 msgr2=0x7fa228077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.244+0000 7fa22dfa7640 1 -- 172.21.15.23:0/4255902144 shutdown_connections 2024-03-24T10:41:50.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.244+0000 7fa22dfa7640 1 -- 172.21.15.23:0/4255902144 wait complete. 2024-03-24T10:41:50.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.244+0000 7fa22dfa7640 1 Processor -- start 2024-03-24T10:41:50.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa22dfa7640 1 -- start start 2024-03-24T10:41:50.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa22dfa7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228068e50 0x7fa22819e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa22dfa7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2281041d0 0x7fa22819e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa22dfa7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa228106660 0x7fa2281a2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa22dfa7640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fa2281a3110 con 0x7fa2281041d0 2024-03-24T10:41:50.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa22dfa7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa2281a3280 con 0x7fa228068e50 2024-03-24T10:41:50.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa22dfa7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa2281a3440 con 0x7fa228106660 2024-03-24T10:41:50.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa226ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2281041d0 0x7fa22819e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa2277fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228068e50 0x7fa22819e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa2277fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228068e50 0x7fa22819e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:49694/0 (socket says 172.21.15.23:49694) 2024-03-24T10:41:50.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa2277fe640 1 -- 172.21.15.23:0/936290324 learned_addr learned my addr 172.21.15.23:0/936290324 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa226ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2281041d0 0x7fa22819e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51608/0 (socket says 172.21.15.23:51608) 2024-03-24T10:41:50.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa227fff640 1 --2- 172.21.15.23:0/936290324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa228106660 0x7fa2281a2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa226ffd640 1 -- 172.21.15.23:0/936290324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa228106660 msgr2=0x7fa2281a2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa226ffd640 1 --2- 172.21.15.23:0/936290324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa228106660 0x7fa2281a2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa226ffd640 1 -- 172.21.15.23:0/936290324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228068e50 msgr2=0x7fa22819e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa226ffd640 1 --2- 172.21.15.23:0/936290324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228068e50 0x7fa22819e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa226ffd640 1 -- 172.21.15.23:0/936290324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa220009900 con 0x7fa2281041d0 2024-03-24T10:41:50.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.245+0000 7fa226ffd640 1 --2- 172.21.15.23:0/936290324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2281041d0 0x7fa22819e7e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00e9f0 tx=0x7fa21c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.246+0000 7fa224ff9640 1 -- 172.21.15.23:0/936290324 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c00ccc0 con 0x7fa2281041d0 2024-03-24T10:41:50.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.246+0000 7fa224ff9640 1 -- 172.21.15.23:0/936290324 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa21c00ce20 con 0x7fa2281041d0 2024-03-24T10:41:50.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.246+0000 7fa22dfa7640 1 -- 172.21.15.23:0/936290324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2281a3720 con 0x7fa2281041d0 2024-03-24T10:41:50.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.246+0000 7fa224ff9640 1 -- 172.21.15.23:0/936290324 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c018920 con 0x7fa2281041d0 2024-03-24T10:41:50.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.246+0000 7fa22dfa7640 1 -- 172.21.15.23:0/936290324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2281aafa0 con 0x7fa2281041d0 2024-03-24T10:41:50.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.248+0000 7fa224ff9640 1 -- 172.21.15.23:0/936290324 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fa21c018a80 con 0x7fa2281041d0 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.248+0000 7fa230b82740 1 Processor -- start 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.248+0000 7fa230b82740 1 -- start start 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.248+0000 7fa224ff9640 1 --2- 172.21.15.23:0/936290324 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa2100a2230 0x7fa2100a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.248+0000 7fa230b82740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 0x563ff60c17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.248+0000 7fa230b82740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.248+0000 7fa230b82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 0x563ff60ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.248+0000 7fa230b82740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563ff602f840 con 0x563ff60c1d10 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.248+0000 7fa230b82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563ff602cd70 con 0x563ff60ca8f0 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.248+0000 7fa230b82740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563ff6028550 con 0x563ff60c13f0 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa226ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa227fff640 1 --2- 172.21.15.23:0/936290324 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa2100a2230 0x7fa2100a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa226ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51614/0 (socket says 172.21.15.23:51614) 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa226ffd640 1 -- 172.21.15.23:0/2283193946 learned_addr learned my addr 172.21.15.23:0/2283193946 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa2277fe640 1 --2- 172.21.15.23:0/2283193946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 0x563ff60c17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa224ff9640 1 -- 172.21.15.23:0/936290324 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fa21c014070 con 0x7fa2281041d0 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa227fff640 1 --2- 172.21.15.23:0/2283193946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 0x563ff60ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa226ffd640 1 -- 172.21.15.23:0/2283193946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 msgr2=0x563ff60c17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa226ffd640 1 --2- 172.21.15.23:0/2283193946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 0x563ff60c17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa226ffd640 1 -- 172.21.15.23:0/2283193946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 msgr2=0x563ff60ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa226ffd640 1 --2- 172.21.15.23:0/2283193946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 0x563ff60ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa226ffd640 1 -- 172.21.15.23:0/2283193946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ff5fbdde0 con 0x563ff60c1d10 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa2277fe640 1 --2- 172.21.15.23:0/2283193946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 0x563ff60c17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa226ffd640 1 --2- 172.21.15.23:0/2283193946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60ca3b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00c6a0 tx=0x7fa21c010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c063c6e6ae024a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa227fff640 1 --2- 172.21.15.23:0/936290324 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa2100a2230 0x7fa2100a46f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa220005cc0 tx=0x7fa220032f20 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa20dffb640 1 -- 172.21.15.23:0/2283193946 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c09eea0 con 0x563ff60c1d10 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa20dffb640 1 -- 172.21.15.23:0/2283193946 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa21c0a34f0 con 0x563ff60c1d10 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.249+0000 7fa20dffb640 1 -- 172.21.15.23:0/2283193946 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c097470 con 0x563ff60c1d10 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.250+0000 7fa230b82740 1 -- 172.21.15.23:0/2283193946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 msgr2=0x563ff60ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.250+0000 7fa230b82740 1 --2- 172.21.15.23:0/2283193946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60ca3b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00c6a0 tx=0x7fa21c010f00 comp rx=0 tx=0).stop 2024-03-24T10:41:50.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.250+0000 7fa230b82740 1 -- 172.21.15.23:0/2283193946 shutdown_connections 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.250+0000 7fa230b82740 1 --2- 172.21.15.23:0/2283193946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 0x563ff60ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.250+0000 7fa230b82740 1 --2- 172.21.15.23:0/2283193946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60ca3b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.250+0000 7fa230b82740 1 --2- 172.21.15.23:0/2283193946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 0x563ff60c17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.250+0000 7fa230b82740 1 -- 172.21.15.23:0/2283193946 >> 172.21.15.23:0/2283193946 conn(0x563ff60bbb20 msgr2=0x563ff60bbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.250+0000 7fa230b82740 1 -- 172.21.15.23:0/2283193946 shutdown_connections 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.250+0000 7fa230b82740 1 -- 172.21.15.23:0/2283193946 wait complete. 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.250+0000 7fa230b82740 1 -- start start 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa230b82740 1 Processor -- start 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa230b82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60c13f0 0x563ff60c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa230b82740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa230b82740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60ca8f0 0x563ff60c8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa230b82740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563ff5cfb430 con 0x563ff60c1d10 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa230b82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563ff5cb7e00 con 0x563ff60c13f0 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa230b82740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563ff5fad590 con 0x563ff60ca8f0 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa230b82740 10 client.0 _ll_register_callbacks cb 0x563ff601cfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:50.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa226ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa226ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60c7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51624/0 (socket says 172.21.15.23:51624) 2024-03-24T10:41:50.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa226ffd640 1 -- 172.21.15.23:0/1348336588 learned_addr learned my addr 172.21.15.23:0/1348336588 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa227fff640 1 --2- 172.21.15.23:0/1348336588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60ca8f0 0x563ff60c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa2277fe640 1 --2- 172.21.15.23:0/1348336588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60c13f0 0x563ff60c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa226ffd640 1 -- 172.21.15.23:0/1348336588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60ca8f0 msgr2=0x563ff60c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa226ffd640 1 --2- 172.21.15.23:0/1348336588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60ca8f0 0x563ff60c8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa226ffd640 1 -- 172.21.15.23:0/1348336588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60c13f0 msgr2=0x563ff60c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa226ffd640 1 --2- 172.21.15.23:0/1348336588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60c13f0 0x563ff60c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa226ffd640 1 -- 172.21.15.23:0/1348336588 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2281aafa0 con 0x563ff60c1d10 2024-03-24T10:41:50.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa226ffd640 1 --2- 172.21.15.23:0/1348336588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60c7ed0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa21c011fd0 tx=0x7fa21c010d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b588b459edd0f76a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa206ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:50.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa206ffd640 1 -- 172.21.15.23:0/1348336588 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c09eea0 con 0x563ff60c1d10 2024-03-24T10:41:50.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa206ffd640 1 -- 172.21.15.23:0/1348336588 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fa21c097470 con 0x563ff60c1d10 2024-03-24T10:41:50.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.251+0000 7fa230b82740 1 -- 172.21.15.23:0/1348336588 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563ff5d277e0 con 0x563ff60c1d10 2024-03-24T10:41:50.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.252+0000 7fa206ffd640 1 -- 172.21.15.23:0/1348336588 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa21c0c6900 con 0x563ff60c1d10 2024-03-24T10:41:50.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.252+0000 7fa230b82740 10 client.16023 fetch_fsmap learned FSMap version 24 2024-03-24T10:41:50.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.252+0000 7fa230b82740 1 -- 172.21.15.23:0/1348336588 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563ff6026bd0 con 0x563ff60c1d10 2024-03-24T10:41:50.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.252+0000 7fa206ffd640 1 -- 172.21.15.23:0/1348336588 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563ff5d277e0 con 0x563ff60c1d10 2024-03-24T10:41:50.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.252+0000 7fa206ffd640 1 -- 172.21.15.23:0/1348336588 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) ==== 8342+0+0 (secure 0 0 0) 0x7fa21c0603e0 con 0x563ff60c1d10 2024-03-24T10:41:50.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.252+0000 7fa230b82740 10 client.16023 fetch_fsmap finished waiting for FSMap version 24 2024-03-24T10:41:50.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.252+0000 7fa230b82740 10 client.16023 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:41:50.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.252+0000 7fa230b82740 20 client.16023 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:50.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.252+0000 7fa230b82740 1 --2- 172.21.15.23:0/1348336588 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x563ff60da620 0x563ff60dca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.252+0000 7fa230b82740 4 client.16023 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:50.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.252+0000 7fa230b82740 1 -- 172.21.15.23:0/1348336588 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ff5d64720 con 0x563ff60da620 2024-03-24T10:41:50.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.252+0000 7fa2277fe640 1 --2- 172.21.15.23:0/1348336588 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x563ff60da620 0x563ff60dca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.253+0000 7fa2277fe640 1 --2- 172.21.15.23:0/1348336588 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x563ff60da620 0x563ff60dca30 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3b33c6541b7551f4 server_cookie=b0bffe62d6aef41c in_seq=0 out_seq=0 2024-03-24T10:41:50.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.253+0000 7fa206ffd640 10 client.16023 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:41:50.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.255+0000 7fa206ffd640 1 -- 172.21.15.23:0/1348336588 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ff5d64720 con 0x563ff60da620 2024-03-24T10:41:50.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.255+0000 7fa206ffd640 10 client.16023 handle_command_reply: tid=0 2024-03-24T10:41:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.255+0000 7fa230b82740 1 client.16023 shutdown 2024-03-24T10:41:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.255+0000 7fa230b82740 1 -- 172.21.15.23:0/1348336588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 msgr2=0x563ff60c7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.255+0000 7fa230b82740 1 --2- 172.21.15.23:0/1348336588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60c7ed0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa21c011fd0 tx=0x7fa21c010d60 comp rx=0 tx=0).stop 2024-03-24T10:41:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.255+0000 7fa230b82740 1 -- 172.21.15.23:0/1348336588 shutdown_connections 2024-03-24T10:41:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.255+0000 7fa230b82740 1 --2- 172.21.15.23:0/1348336588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60ca8f0 0x563ff60c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.255+0000 7fa230b82740 1 --2- 172.21.15.23:0/1348336588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60c7ed0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.255+0000 7fa230b82740 1 --2- 172.21.15.23:0/1348336588 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x563ff60da620 0x563ff60dca30 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.255+0000 7fa230b82740 1 --2- 172.21.15.23:0/1348336588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60c13f0 0x563ff60c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.255+0000 7fa206ffd640 0 client.16023 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:41:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.255+0000 7fa230b82740 1 -- 172.21.15.23:0/1348336588 >> 172.21.15.23:0/1348336588 conn(0x563ff60bbb20 msgr2=0x563ff60bd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.256+0000 7fa230b82740 1 -- 172.21.15.23:0/1348336588 shutdown_connections 2024-03-24T10:41:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.256+0000 7fa230b82740 1 -- 172.21.15.23:0/1348336588 wait complete. 2024-03-24T10:41:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.256+0000 7fa230b82740 20 client.16023 trim_cache size 0 max 16384 2024-03-24T10:41:50.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.269+0000 7fa230b82740 1 Processor -- start 2024-03-24T10:41:50.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.269+0000 7fa230b82740 1 -- start start 2024-03-24T10:41:50.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.269+0000 7fa230b82740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 0x563ff60ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.269+0000 7fa230b82740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60d6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.269+0000 7fa230b82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 0x563ff60e2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.269+0000 7fa230b82740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563ff60dd0b0 con 0x563ff60c1d10 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.269+0000 7fa230b82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563ff60c0e30 con 0x563ff60ca8f0 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.269+0000 7fa230b82740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563ff60d6660 con 0x563ff60c13f0 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.270+0000 7fa226ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60d6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.270+0000 7fa226ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60d6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51638/0 (socket says 172.21.15.23:51638) 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.270+0000 7fa226ffd640 1 -- 172.21.15.23:0/435174408 learned_addr learned my addr 172.21.15.23:0/435174408 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.270+0000 7fa2277fe640 1 --2- 172.21.15.23:0/435174408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 0x563ff60ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.270+0000 7fa227fff640 1 --2- 172.21.15.23:0/435174408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 0x563ff60e2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.270+0000 7fa226ffd640 1 -- 172.21.15.23:0/435174408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 msgr2=0x563ff60ca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.270+0000 7fa226ffd640 1 --2- 172.21.15.23:0/435174408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 0x563ff60ca0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.270+0000 7fa226ffd640 1 -- 172.21.15.23:0/435174408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 msgr2=0x563ff60e2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.270+0000 7fa226ffd640 1 --2- 172.21.15.23:0/435174408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 0x563ff60e2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.270+0000 7fa226ffd640 1 -- 172.21.15.23:0/435174408 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ff5fc2c80 con 0x563ff60c1d10 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.270+0000 7fa226ffd640 1 --2- 172.21.15.23:0/435174408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60d6120 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa21c09ab00 tx=0x7fa21c010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265895946a26e50f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.270+0000 7fa20d7fa640 1 -- 172.21.15.23:0/435174408 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c010bf0 con 0x563ff60c1d10 2024-03-24T10:41:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.270+0000 7fa20d7fa640 1 -- 172.21.15.23:0/435174408 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa21c09a760 con 0x563ff60c1d10 2024-03-24T10:41:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.270+0000 7fa20d7fa640 1 -- 172.21.15.23:0/435174408 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c060350 con 0x563ff60c1d10 2024-03-24T10:41:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.271+0000 7fa230b82740 1 -- 172.21.15.23:0/435174408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 msgr2=0x563ff60d6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.271+0000 7fa230b82740 1 --2- 172.21.15.23:0/435174408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60d6120 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa21c09ab00 tx=0x7fa21c010f00 comp rx=0 tx=0).stop 2024-03-24T10:41:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.271+0000 7fa230b82740 1 -- 172.21.15.23:0/435174408 shutdown_connections 2024-03-24T10:41:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.271+0000 7fa230b82740 1 --2- 172.21.15.23:0/435174408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 0x563ff60e2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.271+0000 7fa230b82740 1 --2- 172.21.15.23:0/435174408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60d6120 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.271+0000 7fa230b82740 1 --2- 172.21.15.23:0/435174408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 0x563ff60ca0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.271+0000 7fa230b82740 1 -- 172.21.15.23:0/435174408 >> 172.21.15.23:0/435174408 conn(0x563ff60bbb20 msgr2=0x563ff60b6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.271+0000 7fa230b82740 1 -- 172.21.15.23:0/435174408 shutdown_connections 2024-03-24T10:41:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.271+0000 7fa230b82740 1 -- 172.21.15.23:0/435174408 wait complete. 2024-03-24T10:41:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.271+0000 7fa230b82740 1 -- start start 2024-03-24T10:41:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.271+0000 7fa230b82740 1 Processor -- start 2024-03-24T10:41:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa230b82740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 0x563ff60d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa230b82740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa230b82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 0x563ff60ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa230b82740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563ff60ce8f0 con 0x563ff60c1d10 2024-03-24T10:41:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa230b82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563ff60cea60 con 0x563ff60ca8f0 2024-03-24T10:41:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa230b82740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563ff613f930 con 0x563ff60c13f0 2024-03-24T10:41:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa230b82740 10 client.0 _ll_register_callbacks cb 0x7fa21c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa226ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa226ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60cdd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51654/0 (socket says 172.21.15.23:51654) 2024-03-24T10:41:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa226ffd640 1 -- 172.21.15.23:0/503028530 learned_addr learned my addr 172.21.15.23:0/503028530 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa227fff640 1 --2- 172.21.15.23:0/503028530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 0x563ff60ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa2277fe640 1 --2- 172.21.15.23:0/503028530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 0x563ff60d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa226ffd640 1 -- 172.21.15.23:0/503028530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 msgr2=0x563ff60d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa226ffd640 1 --2- 172.21.15.23:0/503028530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 0x563ff60d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa226ffd640 1 -- 172.21.15.23:0/503028530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 msgr2=0x563ff60ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa226ffd640 1 --2- 172.21.15.23:0/503028530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 0x563ff60ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa226ffd640 1 -- 172.21.15.23:0/503028530 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ff6026bd0 con 0x563ff60c1d10 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa2277fe640 1 --2- 172.21.15.23:0/503028530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 0x563ff60d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa226ffd640 1 --2- 172.21.15.23:0/503028530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60cdd30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa21c0cc0b0 tx=0x7fa21c09a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caea4dc90d8bb7bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa206ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa206ffd640 1 -- 172.21.15.23:0/503028530 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c09a370 con 0x563ff60c1d10 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa206ffd640 1 -- 172.21.15.23:0/503028530 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fa21c09eea0 con 0x563ff60c1d10 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa230b82740 1 -- 172.21.15.23:0/503028530 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563ff60c77c0 con 0x563ff60c1d10 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa206ffd640 1 -- 172.21.15.23:0/503028530 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa21c098f60 con 0x563ff60c1d10 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa230b82740 10 client.16029 fetch_fsmap learned FSMap version 24 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.272+0000 7fa230b82740 1 -- 172.21.15.23:0/503028530 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563ff5fe8920 con 0x563ff60c1d10 2024-03-24T10:41:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.273+0000 7fa206ffd640 1 -- 172.21.15.23:0/503028530 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563ff60c77c0 con 0x563ff60c1d10 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.273+0000 7fa206ffd640 1 -- 172.21.15.23:0/503028530 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) ==== 8342+0+0 (secure 0 0 0) 0x7fa21c09fdc0 con 0x563ff60c1d10 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.273+0000 7fa230b82740 10 client.16029 fetch_fsmap finished waiting for FSMap version 24 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.273+0000 7fa230b82740 10 client.16029 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.273+0000 7fa230b82740 20 client.16029 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.273+0000 7fa230b82740 1 --2- 172.21.15.23:0/503028530 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x563ff6140ba0 0x563ff6142f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.273+0000 7fa230b82740 4 client.16029 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.273+0000 7fa230b82740 1 -- 172.21.15.23:0/503028530 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fa21c0a34f0 con 0x563ff6140ba0 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.273+0000 7fa2277fe640 1 --2- 172.21.15.23:0/503028530 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x563ff6140ba0 0x563ff6142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.273+0000 7fa2277fe640 1 --2- 172.21.15.23:0/503028530 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x563ff6140ba0 0x563ff6142f90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bbfed92bc0cf18bf server_cookie=52da7945bf373bd2 in_seq=0 out_seq=0 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.273+0000 7fa206ffd640 10 client.16029 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.274+0000 7fa206ffd640 1 -- 172.21.15.23:0/503028530 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fa21c0a34f0 con 0x563ff6140ba0 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.274+0000 7fa206ffd640 10 client.16029 handle_command_reply: tid=0 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.274+0000 7fa230b82740 1 client.16029 shutdown 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.274+0000 7fa230b82740 1 -- 172.21.15.23:0/503028530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 msgr2=0x563ff60cdd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.274+0000 7fa230b82740 1 --2- 172.21.15.23:0/503028530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60cdd30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa21c0cc0b0 tx=0x7fa21c09a550 comp rx=0 tx=0).stop 2024-03-24T10:41:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.274+0000 7fa230b82740 1 -- 172.21.15.23:0/503028530 shutdown_connections 2024-03-24T10:41:50.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.274+0000 7fa230b82740 1 --2- 172.21.15.23:0/503028530 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x563ff6140ba0 0x563ff6142f90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.274+0000 7fa230b82740 1 --2- 172.21.15.23:0/503028530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ff60ca8f0 0x563ff60ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.274+0000 7fa230b82740 1 --2- 172.21.15.23:0/503028530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563ff60c1d10 0x563ff60cdd30 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.274+0000 7fa206ffd640 0 client.16029 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:41:50.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.274+0000 7fa230b82740 1 --2- 172.21.15.23:0/503028530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563ff60c13f0 0x563ff60d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.274+0000 7fa230b82740 1 -- 172.21.15.23:0/503028530 >> 172.21.15.23:0/503028530 conn(0x563ff60bbb20 msgr2=0x563ff60ba430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.274+0000 7fa230b82740 1 -- 172.21.15.23:0/503028530 shutdown_connections 2024-03-24T10:41:50.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.274+0000 7fa230b82740 1 -- 172.21.15.23:0/503028530 wait complete. 2024-03-24T10:41:50.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.275+0000 7fa230b82740 20 client.16029 trim_cache size 0 max 16384 2024-03-24T10:41:50.281 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:41:50.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.275+0000 7fa22dfa7640 1 -- 172.21.15.23:0/936290324 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa2100a2230 msgr2=0x7fa2100a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.275+0000 7fa22dfa7640 1 --2- 172.21.15.23:0/936290324 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa2100a2230 0x7fa2100a46f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa220005cc0 tx=0x7fa220032f20 comp rx=0 tx=0).stop 2024-03-24T10:41:50.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.275+0000 7fa22dfa7640 1 -- 172.21.15.23:0/936290324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2281041d0 msgr2=0x7fa22819e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.275+0000 7fa22dfa7640 1 --2- 172.21.15.23:0/936290324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2281041d0 0x7fa22819e7e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00e9f0 tx=0x7fa21c00eeb0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.275+0000 7fa22dfa7640 1 -- 172.21.15.23:0/936290324 shutdown_connections 2024-03-24T10:41:50.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.275+0000 7fa22dfa7640 1 --2- 172.21.15.23:0/936290324 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa2100a2230 0x7fa2100a46f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.275+0000 7fa22dfa7640 1 --2- 172.21.15.23:0/936290324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa228106660 0x7fa2281a2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.275+0000 7fa22dfa7640 1 --2- 172.21.15.23:0/936290324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2281041d0 0x7fa22819e7e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.275+0000 7fa22dfa7640 1 --2- 172.21.15.23:0/936290324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa228068e50 0x7fa22819e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.275+0000 7fa22dfa7640 1 -- 172.21.15.23:0/936290324 >> 172.21.15.23:0/936290324 conn(0x7fa228077710 msgr2=0x7fa228101580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.275+0000 7fa22dfa7640 1 -- 172.21.15.23:0/936290324 shutdown_connections 2024-03-24T10:41:50.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.275+0000 7fa22dfa7640 1 -- 172.21.15.23:0/936290324 wait complete. 2024-03-24T10:41:50.288 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:41:50.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.392+0000 7f35cbd3a640 1 -- 172.21.15.23:0/3727232877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35c4102980 msgr2=0x7f35c4102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.392+0000 7f35cbd3a640 1 --2- 172.21.15.23:0/3727232877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35c4102980 0x7f35c4102d60 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f35c0009900 tx=0x7f35c002bcd0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.392+0000 7f35cbd3a640 1 -- 172.21.15.23:0/3727232877 shutdown_connections 2024-03-24T10:41:50.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.392+0000 7f35cbd3a640 1 --2- 172.21.15.23:0/3727232877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35c410bd50 0x7f35c410e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.392+0000 7f35cbd3a640 1 --2- 172.21.15.23:0/3727232877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c41032a0 0x7f35c410b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.392+0000 7f35cbd3a640 1 --2- 172.21.15.23:0/3727232877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35c4102980 0x7f35c4102d60 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.392+0000 7f35cbd3a640 1 -- 172.21.15.23:0/3727232877 >> 172.21.15.23:0/3727232877 conn(0x7f35c4077770 msgr2=0x7f35c4077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.392+0000 7f35cbd3a640 1 -- 172.21.15.23:0/3727232877 shutdown_connections 2024-03-24T10:41:50.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.392+0000 7f35cbd3a640 1 -- 172.21.15.23:0/3727232877 wait complete. 2024-03-24T10:41:50.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.393+0000 7f35cbd3a640 1 Processor -- start 2024-03-24T10:41:50.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.393+0000 7f35cbd3a640 1 -- start start 2024-03-24T10:41:50.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.393+0000 7f35cbd3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35c4102980 0x7f35c4197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.393+0000 7f35cbd3a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35c41032a0 0x7f35c4198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.393+0000 7f35cbd3a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c410bd50 0x7f35c419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.393+0000 7f35cbd3a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f35c419cd10 con 0x7f35c41032a0 2024-03-24T10:41:50.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.393+0000 7f35cbd3a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f35c419ce80 con 0x7f35c4102980 2024-03-24T10:41:50.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.393+0000 7f35cbd3a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f35c419d040 con 0x7f35c410bd50 2024-03-24T10:41:50.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.393+0000 7f35c92d5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35c41032a0 0x7f35c4198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.393+0000 7f35c92d5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35c41032a0 0x7f35c4198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51674/0 (socket says 172.21.15.23:51674) 2024-03-24T10:41:50.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.393+0000 7f35c92d5640 1 -- 172.21.15.23:0/4166188227 learned_addr learned my addr 172.21.15.23:0/4166188227 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.393+0000 7f35c9ad6640 1 --2- 172.21.15.23:0/4166188227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35c4102980 0x7f35c4197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.393+0000 7f35ca2d7640 1 --2- 172.21.15.23:0/4166188227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c410bd50 0x7f35c419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.394+0000 7f35c92d5640 1 -- 172.21.15.23:0/4166188227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c410bd50 msgr2=0x7f35c419c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.394+0000 7f35c92d5640 1 --2- 172.21.15.23:0/4166188227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c410bd50 0x7f35c419c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.394+0000 7f35c92d5640 1 -- 172.21.15.23:0/4166188227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35c4102980 msgr2=0x7f35c4197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.394+0000 7f35c92d5640 1 --2- 172.21.15.23:0/4166188227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35c4102980 0x7f35c4197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.394+0000 7f35c92d5640 1 -- 172.21.15.23:0/4166188227 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c0009560 con 0x7f35c41032a0 2024-03-24T10:41:50.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.394+0000 7f35c92d5640 1 --2- 172.21.15.23:0/4166188227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35c41032a0 0x7f35c4198390 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f35b400e9f0 tx=0x7f35b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.394+0000 7f35baffd640 1 -- 172.21.15.23:0/4166188227 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b4009700 con 0x7f35c41032a0 2024-03-24T10:41:50.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.394+0000 7f35baffd640 1 -- 172.21.15.23:0/4166188227 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35b400cd80 con 0x7f35c41032a0 2024-03-24T10:41:50.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.394+0000 7f35baffd640 1 -- 172.21.15.23:0/4166188227 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b4018920 con 0x7f35c41032a0 2024-03-24T10:41:50.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.394+0000 7f35cbd3a640 1 -- 172.21.15.23:0/4166188227 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35c419d320 con 0x7f35c41032a0 2024-03-24T10:41:50.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.394+0000 7f35cbd3a640 1 -- 172.21.15.23:0/4166188227 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c41a4ba0 con 0x7f35c41032a0 2024-03-24T10:41:50.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.396+0000 7f35baffd640 1 -- 172.21.15.23:0/4166188227 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f35b4018a80 con 0x7f35c41032a0 2024-03-24T10:41:50.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.396+0000 7f35ce9ee740 1 Processor -- start 2024-03-24T10:41:50.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35ce9ee740 1 -- start start 2024-03-24T10:41:50.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35ce9ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd5fc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35ce9ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fcd10 0x55fcfd6053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35ce9ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd6058f0 0x55fcfd607ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35ce9ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fcfd56a840 con 0x55fcfd6058f0 2024-03-24T10:41:50.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35ce9ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fcfd567d70 con 0x55fcfd5fcd10 2024-03-24T10:41:50.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35ce9ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fcfd563550 con 0x55fcfd5fc3f0 2024-03-24T10:41:50.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35baffd640 1 --2- 172.21.15.23:0/4166188227 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f35b00a22b0 0x7f35b00a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35c92d5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd6058f0 0x55fcfd607ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35c9ad6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fcd10 0x55fcfd6053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35ca2d7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd5fc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35c9ad6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fcd10 0x55fcfd6053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:49758/0 (socket says 172.21.15.23:49758) 2024-03-24T10:41:50.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35ca2d7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd5fc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:38766/0 (socket says 172.21.15.23:38766) 2024-03-24T10:41:50.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35c9ad6640 1 -- 172.21.15.23:0/1744524461 learned_addr learned my addr 172.21.15.23:0/1744524461 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35ca2d7640 1 --2- 172.21.15.23:0/4166188227 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f35b00a22b0 0x7f35b00a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35baffd640 1 -- 172.21.15.23:0/4166188227 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f35b4014070 con 0x7f35c41032a0 2024-03-24T10:41:50.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35c92d5640 1 -- 172.21.15.23:0/1744524461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fc3f0 msgr2=0x55fcfd5fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35c92d5640 1 --2- 172.21.15.23:0/1744524461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd5fc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35c92d5640 1 -- 172.21.15.23:0/1744524461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fcd10 msgr2=0x55fcfd6053b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35c92d5640 1 --2- 172.21.15.23:0/1744524461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fcd10 0x55fcfd6053b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.397+0000 7f35c92d5640 1 -- 172.21.15.23:0/1744524461 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcfd4f8de0 con 0x55fcfd6058f0 2024-03-24T10:41:50.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.398+0000 7f35c9ad6640 1 --2- 172.21.15.23:0/1744524461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fcd10 0x55fcfd6053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:50.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.398+0000 7f35c92d5640 1 --2- 172.21.15.23:0/1744524461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd6058f0 0x55fcfd607ce0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f35b40cbad0 tx=0x7f35b406df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f79d7ad72bf071 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.398+0000 7f35ca2d7640 1 --2- 172.21.15.23:0/4166188227 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f35b00a22b0 0x7f35b00a4770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009990 tx=0x7f35bc0096a0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.398+0000 7f35affff640 1 -- 172.21.15.23:0/1744524461 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b409be60 con 0x55fcfd6058f0 2024-03-24T10:41:50.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.398+0000 7f35affff640 1 -- 172.21.15.23:0/1744524461 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35b409ae20 con 0x55fcfd6058f0 2024-03-24T10:41:50.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.398+0000 7f35affff640 1 -- 172.21.15.23:0/1744524461 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b40a3510 con 0x55fcfd6058f0 2024-03-24T10:41:50.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.399+0000 7f35ce9ee740 1 -- 172.21.15.23:0/1744524461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd6058f0 msgr2=0x55fcfd607ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.399+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/1744524461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd6058f0 0x55fcfd607ce0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f35b40cbad0 tx=0x7f35b406df50 comp rx=0 tx=0).stop 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.399+0000 7f35ce9ee740 1 -- 172.21.15.23:0/1744524461 shutdown_connections 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.399+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/1744524461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd6058f0 0x55fcfd607ce0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.399+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/1744524461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fcd10 0x55fcfd6053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.399+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/1744524461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd5fc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.399+0000 7f35ce9ee740 1 -- 172.21.15.23:0/1744524461 >> 172.21.15.23:0/1744524461 conn(0x55fcfd5f6b20 msgr2=0x55fcfd5f6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.399+0000 7f35ce9ee740 1 -- 172.21.15.23:0/1744524461 shutdown_connections 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.399+0000 7f35ce9ee740 1 -- 172.21.15.23:0/1744524461 wait complete. 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.399+0000 7f35ce9ee740 1 -- start start 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.399+0000 7f35ce9ee740 1 Processor -- start 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.399+0000 7f35ce9ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd6029b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.399+0000 7f35ce9ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fcd10 0x55fcfd602ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.399+0000 7f35ce9ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd6058f0 0x55fcfd603430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35ce9ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fcfd236430 con 0x55fcfd5fc3f0 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35ce9ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fcfd1f2e00 con 0x55fcfd5fcd10 2024-03-24T10:41:50.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35ce9ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fcfd4e8590 con 0x55fcfd6058f0 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35ce9ee740 10 client.0 _ll_register_callbacks cb 0x55fcfd557fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35c9ad6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd6029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35c9ad6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd6029b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51704/0 (socket says 172.21.15.23:51704) 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35c9ad6640 1 -- 172.21.15.23:0/2333502629 learned_addr learned my addr 172.21.15.23:0/2333502629 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35c92d5640 1 --2- 172.21.15.23:0/2333502629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fcd10 0x55fcfd602ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35ca2d7640 1 --2- 172.21.15.23:0/2333502629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd6058f0 0x55fcfd603430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35c9ad6640 1 -- 172.21.15.23:0/2333502629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd6058f0 msgr2=0x55fcfd603430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35c9ad6640 1 --2- 172.21.15.23:0/2333502629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd6058f0 0x55fcfd603430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35c9ad6640 1 -- 172.21.15.23:0/2333502629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fcd10 msgr2=0x55fcfd602ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35c9ad6640 1 --2- 172.21.15.23:0/2333502629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fcd10 0x55fcfd602ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35c9ad6640 1 -- 172.21.15.23:0/2333502629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c41a4ba0 con 0x55fcfd5fc3f0 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35c9ad6640 1 --2- 172.21.15.23:0/2333502629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd6029b0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f35c0036040 tx=0x7f35c0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1853425fa334844 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35ad7fa640 1 -- 172.21.15.23:0/2333502629 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c002c880 con 0x55fcfd5fc3f0 2024-03-24T10:41:50.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35ad7fa640 1 -- 172.21.15.23:0/2333502629 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f35c002c9e0 con 0x55fcfd5fc3f0 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35ce9ee740 1 -- 172.21.15.23:0/2333502629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fcfd2627e0 con 0x55fcfd5fc3f0 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35ad7fa640 1 -- 172.21.15.23:0/2333502629 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f35c002cba0 con 0x55fcfd5fc3f0 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35ce9ee740 10 client.16041 fetch_fsmap learned FSMap version 24 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.400+0000 7f35ce9ee740 1 -- 172.21.15.23:0/2333502629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fcfd561bd0 con 0x55fcfd5fc3f0 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.401+0000 7f35ad7fa640 1 -- 172.21.15.23:0/2333502629 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fcfd2627e0 con 0x55fcfd5fc3f0 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.401+0000 7f35ad7fa640 1 -- 172.21.15.23:0/2333502629 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) ==== 8342+0+0 (secure 0 0 0) 0x7f35c0046020 con 0x55fcfd5fc3f0 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.401+0000 7f35ce9ee740 10 client.16041 fetch_fsmap finished waiting for FSMap version 24 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.401+0000 7f35ce9ee740 10 client.16041 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.401+0000 7f35ce9ee740 20 client.16041 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.401+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/2333502629 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55fcfd615990 0x55fcfd617da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.401+0000 7f35ce9ee740 4 client.16041 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.401+0000 7f35ce9ee740 1 -- 172.21.15.23:0/2333502629 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcfd29f720 con 0x55fcfd615990 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.401+0000 7f35c92d5640 1 --2- 172.21.15.23:0/2333502629 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55fcfd615990 0x55fcfd617da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.401+0000 7f35ad7fa640 10 client.16041 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.401+0000 7f35c92d5640 1 --2- 172.21.15.23:0/2333502629 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55fcfd615990 0x55fcfd617da0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d1bf085fe284f363 server_cookie=95a891668523e02f in_seq=0 out_seq=0 2024-03-24T10:41:50.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.402+0000 7f35ad7fa640 1 -- 172.21.15.23:0/2333502629 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fcfd29f720 con 0x55fcfd615990 2024-03-24T10:41:50.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.402+0000 7f35ad7fa640 10 client.16041 handle_command_reply: tid=0 2024-03-24T10:41:50.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.402+0000 7f35ce9ee740 1 client.16041 shutdown 2024-03-24T10:41:50.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.402+0000 7f35ce9ee740 1 -- 172.21.15.23:0/2333502629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd5fc3f0 msgr2=0x55fcfd6029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.402+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/2333502629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd6029b0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f35c0036040 tx=0x7f35c0004270 comp rx=0 tx=0).stop 2024-03-24T10:41:50.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.402+0000 7f35ce9ee740 1 -- 172.21.15.23:0/2333502629 shutdown_connections 2024-03-24T10:41:50.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.402+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/2333502629 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55fcfd615990 0x55fcfd617da0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.402+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/2333502629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd6058f0 0x55fcfd603430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.402+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/2333502629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fcd10 0x55fcfd602ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.402+0000 7f35ad7fa640 0 client.16041 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:41:50.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.402+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/2333502629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd6029b0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.402+0000 7f35ce9ee740 1 -- 172.21.15.23:0/2333502629 >> 172.21.15.23:0/2333502629 conn(0x55fcfd5f6b20 msgr2=0x55fcfd5f8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.402+0000 7f35ce9ee740 1 -- 172.21.15.23:0/2333502629 shutdown_connections 2024-03-24T10:41:50.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.402+0000 7f35ce9ee740 1 -- 172.21.15.23:0/2333502629 wait complete. 2024-03-24T10:41:50.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.403+0000 7f35ce9ee740 20 client.16041 trim_cache size 0 max 16384 2024-03-24T10:41:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.416+0000 7f35ce9ee740 1 Processor -- start 2024-03-24T10:41:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.416+0000 7f35ce9ee740 1 -- start start 2024-03-24T10:41:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.416+0000 7f35ce9ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd611120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.416+0000 7f35ce9ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fcd10 0x55fcfd603250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.416+0000 7f35ce9ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd6058f0 0x55fcfd61d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.416+0000 7f35ce9ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fcfd611660 con 0x55fcfd6058f0 2024-03-24T10:41:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.416+0000 7f35ce9ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fcfd5fc050 con 0x55fcfd5fc3f0 2024-03-24T10:41:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.416+0000 7f35ce9ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fcfd5f6650 con 0x55fcfd5fcd10 2024-03-24T10:41:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.416+0000 7f35ca2d7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd6058f0 0x55fcfd61d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.416+0000 7f35ca2d7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd6058f0 0x55fcfd61d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51708/0 (socket says 172.21.15.23:51708) 2024-03-24T10:41:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.416+0000 7f35ca2d7640 1 -- 172.21.15.23:0/536930144 learned_addr learned my addr 172.21.15.23:0/536930144 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.416+0000 7f35c9ad6640 1 --2- 172.21.15.23:0/536930144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd611120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.416+0000 7f35c92d5640 1 --2- 172.21.15.23:0/536930144 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fcd10 0x55fcfd603250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.417+0000 7f35ca2d7640 1 -- 172.21.15.23:0/536930144 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fcd10 msgr2=0x55fcfd603250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.417+0000 7f35ca2d7640 1 --2- 172.21.15.23:0/536930144 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fcd10 0x55fcfd603250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.417+0000 7f35ca2d7640 1 -- 172.21.15.23:0/536930144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fc3f0 msgr2=0x55fcfd611120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.417+0000 7f35ca2d7640 1 --2- 172.21.15.23:0/536930144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd611120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.417+0000 7f35ca2d7640 1 -- 172.21.15.23:0/536930144 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcfd4fdc80 con 0x55fcfd6058f0 2024-03-24T10:41:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.417+0000 7f35ca2d7640 1 --2- 172.21.15.23:0/536930144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd6058f0 0x55fcfd61d6b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f35bc005c40 tx=0x7f35bc009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a428b6d2497bfee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.417+0000 7f35af7fe640 1 -- 172.21.15.23:0/536930144 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc025070 con 0x55fcfd6058f0 2024-03-24T10:41:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.417+0000 7f35af7fe640 1 -- 172.21.15.23:0/536930144 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35bc00ba30 con 0x55fcfd6058f0 2024-03-24T10:41:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.417+0000 7f35af7fe640 1 -- 172.21.15.23:0/536930144 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc0045b0 con 0x55fcfd6058f0 2024-03-24T10:41:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 -- 172.21.15.23:0/536930144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd6058f0 msgr2=0x55fcfd61d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/536930144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd6058f0 0x55fcfd61d6b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f35bc005c40 tx=0x7f35bc009920 comp rx=0 tx=0).stop 2024-03-24T10:41:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 -- 172.21.15.23:0/536930144 shutdown_connections 2024-03-24T10:41:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/536930144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd6058f0 0x55fcfd61d6b0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/536930144 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fcd10 0x55fcfd603250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/536930144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd611120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 -- 172.21.15.23:0/536930144 >> 172.21.15.23:0/536930144 conn(0x55fcfd5f6b20 msgr2=0x55fcfd5f5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 -- 172.21.15.23:0/536930144 shutdown_connections 2024-03-24T10:41:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 -- 172.21.15.23:0/536930144 wait complete. 2024-03-24T10:41:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 -- start start 2024-03-24T10:41:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 Processor -- start 2024-03-24T10:41:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd609450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd5fcd10 0x55fcfd609990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd6058f0 0x55fcfd67ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fcfd60a240 con 0x55fcfd5fcd10 2024-03-24T10:41:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fcfd67b220 con 0x55fcfd6058f0 2024-03-24T10:41:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fcfd67b360 con 0x55fcfd5fc3f0 2024-03-24T10:41:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.418+0000 7f35ce9ee740 10 client.0 _ll_register_callbacks cb 0x7f35c002c690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35c92d5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd5fcd10 0x55fcfd609990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35c92d5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd5fcd10 0x55fcfd609990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51720/0 (socket says 172.21.15.23:51720) 2024-03-24T10:41:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35c92d5640 1 -- 172.21.15.23:0/93322948 learned_addr learned my addr 172.21.15.23:0/93322948 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35c9ad6640 1 --2- 172.21.15.23:0/93322948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd609450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35ca2d7640 1 --2- 172.21.15.23:0/93322948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd6058f0 0x55fcfd67ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35c92d5640 1 -- 172.21.15.23:0/93322948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fc3f0 msgr2=0x55fcfd609450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35c92d5640 1 --2- 172.21.15.23:0/93322948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd609450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35c92d5640 1 -- 172.21.15.23:0/93322948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd6058f0 msgr2=0x55fcfd67ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35c92d5640 1 --2- 172.21.15.23:0/93322948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd6058f0 0x55fcfd67ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35c92d5640 1 -- 172.21.15.23:0/93322948 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35bc01a040 con 0x55fcfd5fcd10 2024-03-24T10:41:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35c92d5640 1 --2- 172.21.15.23:0/93322948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd5fcd10 0x55fcfd609990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f35b409ab20 tx=0x7f35b4010f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be3952ca5209b46 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35ad7fa640 1 -- 172.21.15.23:0/93322948 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b409eec0 con 0x55fcfd5fcd10 2024-03-24T10:41:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35ad7fa640 1 -- 172.21.15.23:0/93322948 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f35b409bc00 con 0x55fcfd5fcd10 2024-03-24T10:41:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35ce9ee740 1 -- 172.21.15.23:0/93322948 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35b409ae20 con 0x55fcfd5fcd10 2024-03-24T10:41:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35ad7fa640 1 -- 172.21.15.23:0/93322948 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f35b409abf0 con 0x55fcfd5fcd10 2024-03-24T10:41:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35ce9ee740 10 client.16047 fetch_fsmap learned FSMap version 24 2024-03-24T10:41:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35ce9ee740 1 -- 172.21.15.23:0/93322948 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fcfd523920 con 0x55fcfd5fcd10 2024-03-24T10:41:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.419+0000 7f35ad7fa640 1 -- 172.21.15.23:0/93322948 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35b409ae20 con 0x55fcfd5fcd10 2024-03-24T10:41:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35ad7fa640 1 -- 172.21.15.23:0/93322948 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) ==== 8342+0+0 (secure 0 0 0) 0x7f35b40a0070 con 0x55fcfd5fcd10 2024-03-24T10:41:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35ce9ee740 10 client.16047 fetch_fsmap finished waiting for FSMap version 24 2024-03-24T10:41:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35ce9ee740 10 client.16047 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:41:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35ce9ee740 20 client.16047 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/93322948 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55fcfd67c980 0x55fcfd67eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35ce9ee740 4 client.16047 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:41:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35ce9ee740 1 -- 172.21.15.23:0/93322948 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fcfd2ee890 con 0x55fcfd67c980 2024-03-24T10:41:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35c9ad6640 1 --2- 172.21.15.23:0/93322948 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55fcfd67c980 0x55fcfd67eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35c9ad6640 1 --2- 172.21.15.23:0/93322948 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55fcfd67c980 0x55fcfd67eda0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=42e5207595d8cdae server_cookie=4171a900145723ba in_seq=0 out_seq=0 2024-03-24T10:41:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35ad7fa640 10 client.16047 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:41:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35ad7fa640 1 -- 172.21.15.23:0/93322948 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fcfd2ee890 con 0x55fcfd67c980 2024-03-24T10:41:50.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35ad7fa640 10 client.16047 handle_command_reply: tid=0 2024-03-24T10:41:50.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35ce9ee740 1 client.16047 shutdown 2024-03-24T10:41:50.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35ce9ee740 1 -- 172.21.15.23:0/93322948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd5fcd10 msgr2=0x55fcfd609990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.420+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/93322948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd5fcd10 0x55fcfd609990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f35b409ab20 tx=0x7f35b4010f50 comp rx=0 tx=0).stop 2024-03-24T10:41:50.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35ce9ee740 1 -- 172.21.15.23:0/93322948 shutdown_connections 2024-03-24T10:41:50.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/93322948 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55fcfd67c980 0x55fcfd67eda0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/93322948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fcfd6058f0 0x55fcfd67ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35ad7fa640 0 client.16047 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:41:50.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/93322948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fcfd5fcd10 0x55fcfd609990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35ce9ee740 1 --2- 172.21.15.23:0/93322948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcfd5fc3f0 0x55fcfd609450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35ce9ee740 1 -- 172.21.15.23:0/93322948 >> 172.21.15.23:0/93322948 conn(0x55fcfd5f6b20 msgr2=0x55fcfd5f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35ce9ee740 1 -- 172.21.15.23:0/93322948 shutdown_connections 2024-03-24T10:41:50.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35ce9ee740 1 -- 172.21.15.23:0/93322948 wait complete. 2024-03-24T10:41:50.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35ce9ee740 20 client.16047 trim_cache size 0 max 16384 2024-03-24T10:41:50.429 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:41:50.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35cbd3a640 1 -- 172.21.15.23:0/4166188227 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f35b00a22b0 msgr2=0x7f35b00a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35cbd3a640 1 --2- 172.21.15.23:0/4166188227 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f35b00a22b0 0x7f35b00a4770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009990 tx=0x7f35bc0096a0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35cbd3a640 1 -- 172.21.15.23:0/4166188227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35c41032a0 msgr2=0x7f35c4198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35cbd3a640 1 --2- 172.21.15.23:0/4166188227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35c41032a0 0x7f35c4198390 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f35b400e9f0 tx=0x7f35b400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35cbd3a640 1 -- 172.21.15.23:0/4166188227 shutdown_connections 2024-03-24T10:41:50.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35cbd3a640 1 --2- 172.21.15.23:0/4166188227 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f35b00a22b0 0x7f35b00a4770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35cbd3a640 1 --2- 172.21.15.23:0/4166188227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c410bd50 0x7f35c419c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35cbd3a640 1 --2- 172.21.15.23:0/4166188227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35c41032a0 0x7f35c4198390 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35cbd3a640 1 --2- 172.21.15.23:0/4166188227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35c4102980 0x7f35c4197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35cbd3a640 1 -- 172.21.15.23:0/4166188227 >> 172.21.15.23:0/4166188227 conn(0x7f35c4077770 msgr2=0x7f35c40fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35cbd3a640 1 -- 172.21.15.23:0/4166188227 shutdown_connections 2024-03-24T10:41:50.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.421+0000 7f35cbd3a640 1 -- 172.21.15.23:0/4166188227 wait complete. 2024-03-24T10:41:50.433 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:41:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.538+0000 7f513cee6640 1 -- 172.21.15.23:0/3431484241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138100630 msgr2=0x7f5138100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.538+0000 7f513cee6640 1 --2- 172.21.15.23:0/3431484241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138100630 0x7f5138100ab0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f512c009930 tx=0x7f512c02bc20 comp rx=0 tx=0).stop 2024-03-24T10:41:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.539+0000 7f513cee6640 1 -- 172.21.15.23:0/3431484241 shutdown_connections 2024-03-24T10:41:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.539+0000 7f513cee6640 1 --2- 172.21.15.23:0/3431484241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138100ff0 0x7f513810d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.539+0000 7f513cee6640 1 --2- 172.21.15.23:0/3431484241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138100630 0x7f5138100ab0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.539+0000 7f513cee6640 1 --2- 172.21.15.23:0/3431484241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5138106640 0x7f5138106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.539+0000 7f513cee6640 1 -- 172.21.15.23:0/3431484241 >> 172.21.15.23:0/3431484241 conn(0x7f5138077770 msgr2=0x7f5138077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.539+0000 7f513cee6640 1 -- 172.21.15.23:0/3431484241 shutdown_connections 2024-03-24T10:41:50.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.539+0000 7f513cee6640 1 -- 172.21.15.23:0/3431484241 wait complete. 2024-03-24T10:41:50.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.539+0000 7f513cee6640 1 Processor -- start 2024-03-24T10:41:50.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.539+0000 7f513cee6640 1 -- start start 2024-03-24T10:41:50.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f513cee6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5138100630 0x7f5138197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f513cee6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138100ff0 0x7f5138198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f513cee6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138106640 0x7f513819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f513cee6640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f513819cc80 con 0x7f5138106640 2024-03-24T10:41:50.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f513cee6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f513819cdf0 con 0x7f5138100630 2024-03-24T10:41:50.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f513cee6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f513819cfb0 con 0x7f5138100ff0 2024-03-24T10:41:50.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f5136d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138106640 0x7f513819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f5136d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138106640 0x7f513819c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51734/0 (socket says 172.21.15.23:51734) 2024-03-24T10:41:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f5136d9d640 1 -- 172.21.15.23:0/989389057 learned_addr learned my addr 172.21.15.23:0/989389057 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f513659c640 1 --2- 172.21.15.23:0/989389057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5138100630 0x7f5138197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f5135d9b640 1 --2- 172.21.15.23:0/989389057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138100ff0 0x7f5138198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f5136d9d640 1 -- 172.21.15.23:0/989389057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138100ff0 msgr2=0x7f5138198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f5136d9d640 1 --2- 172.21.15.23:0/989389057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138100ff0 0x7f5138198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f5136d9d640 1 -- 172.21.15.23:0/989389057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5138100630 msgr2=0x7f5138197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f5136d9d640 1 --2- 172.21.15.23:0/989389057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5138100630 0x7f5138197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f5136d9d640 1 -- 172.21.15.23:0/989389057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5130009840 con 0x7f5138106640 2024-03-24T10:41:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f5136d9d640 1 --2- 172.21.15.23:0/989389057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138106640 0x7f513819c680 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f513000efd0 tx=0x7f513000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f51277fe640 1 -- 172.21.15.23:0/989389057 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5130009b40 con 0x7f5138106640 2024-03-24T10:41:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f513cee6640 1 -- 172.21.15.23:0/989389057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f512c009560 con 0x7f5138106640 2024-03-24T10:41:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f51277fe640 1 -- 172.21.15.23:0/989389057 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5130003b70 con 0x7f5138106640 2024-03-24T10:41:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f51277fe640 1 -- 172.21.15.23:0/989389057 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51300108f0 con 0x7f5138106640 2024-03-24T10:41:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.540+0000 7f513cee6640 1 -- 172.21.15.23:0/989389057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51381a4930 con 0x7f5138106640 2024-03-24T10:41:50.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.543+0000 7f51277fe640 1 -- 172.21.15.23:0/989389057 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f5130003ce0 con 0x7f5138106640 2024-03-24T10:41:50.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.543+0000 7f513fb9d740 1 Processor -- start 2024-03-24T10:41:50.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.543+0000 7f513fb9d740 1 -- start start 2024-03-24T10:41:50.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.543+0000 7f513fb9d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c73f0 0x55c42f6c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.543+0000 7f513fb9d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.543+0000 7f513fb9d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6d08f0 0x55c42f6d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.543+0000 7f513fb9d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c42f635840 con 0x55c42f6c7d10 2024-03-24T10:41:50.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.543+0000 7f513fb9d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c42f632d70 con 0x55c42f6d08f0 2024-03-24T10:41:50.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.543+0000 7f513fb9d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c42f62e550 con 0x55c42f6c73f0 2024-03-24T10:41:50.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f51277fe640 1 --2- 172.21.15.23:0/989389057 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f51200a2230 0x7f51200a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f513659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f513659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6d03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51736/0 (socket says 172.21.15.23:51736) 2024-03-24T10:41:50.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f513659c640 1 -- 172.21.15.23:0/2773100357 learned_addr learned my addr 172.21.15.23:0/2773100357 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f5135d9b640 1 --2- 172.21.15.23:0/2773100357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c73f0 0x55c42f6c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f5136d9d640 1 --2- 172.21.15.23:0/989389057 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f51200a2230 0x7f51200a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f5135d9b640 1 --2- 172.21.15.23:0/2773100357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6d08f0 0x55c42f6d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f51277fe640 1 -- 172.21.15.23:0/989389057 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f5130014070 con 0x7f5138106640 2024-03-24T10:41:50.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f5135d9b640 1 -- 172.21.15.23:0/2773100357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6d08f0 msgr2=0x55c42f6d2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f5135d9b640 1 --2- 172.21.15.23:0/2773100357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6d08f0 0x55c42f6d2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f5135d9b640 1 -- 172.21.15.23:0/2773100357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 msgr2=0x55c42f6d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f5135d9b640 1 --2- 172.21.15.23:0/2773100357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f5135d9b640 1 -- 172.21.15.23:0/2773100357 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c42f5c3de0 con 0x55c42f6c73f0 2024-03-24T10:41:50.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f513659c640 1 --2- 172.21.15.23:0/2773100357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6d03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:50.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f5135d9b640 1 --2- 172.21.15.23:0/2773100357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c73f0 0x55c42f6c77d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f512c036040 tx=0x7f512c005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ef206814b4ddcd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f5136d9d640 1 --2- 172.21.15.23:0/989389057 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f51200a2230 0x7f51200a46f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5130005c40 tx=0x7f5130009f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f5124ff9640 1 -- 172.21.15.23:0/2773100357 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c02dea0 con 0x55c42f6c73f0 2024-03-24T10:41:50.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f5124ff9640 1 -- 172.21.15.23:0/2773100357 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f512c004020 con 0x55c42f6c73f0 2024-03-24T10:41:50.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.544+0000 7f5124ff9640 1 -- 172.21.15.23:0/2773100357 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c003980 con 0x55c42f6c73f0 2024-03-24T10:41:50.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.545+0000 7f513fb9d740 1 -- 172.21.15.23:0/2773100357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c73f0 msgr2=0x55c42f6c77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.545+0000 7f513fb9d740 1 --2- 172.21.15.23:0/2773100357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c73f0 0x55c42f6c77d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f512c036040 tx=0x7f512c005e00 comp rx=0 tx=0).stop 2024-03-24T10:41:50.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.545+0000 7f513fb9d740 1 -- 172.21.15.23:0/2773100357 shutdown_connections 2024-03-24T10:41:50.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.545+0000 7f513fb9d740 1 --2- 172.21.15.23:0/2773100357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6d08f0 0x55c42f6d2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.545+0000 7f513fb9d740 1 --2- 172.21.15.23:0/2773100357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6d03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.545+0000 7f513fb9d740 1 --2- 172.21.15.23:0/2773100357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c73f0 0x55c42f6c77d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.545+0000 7f513fb9d740 1 -- 172.21.15.23:0/2773100357 >> 172.21.15.23:0/2773100357 conn(0x55c42f6c1b20 msgr2=0x55c42f6c1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.545+0000 7f513fb9d740 1 -- 172.21.15.23:0/2773100357 shutdown_connections 2024-03-24T10:41:50.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.545+0000 7f513fb9d740 1 -- 172.21.15.23:0/2773100357 wait complete. 2024-03-24T10:41:50.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.545+0000 7f513fb9d740 1 -- start start 2024-03-24T10:41:50.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513fb9d740 1 Processor -- start 2024-03-24T10:41:50.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513fb9d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c73f0 0x55c42f6cd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513fb9d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6cded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513fb9d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6d08f0 0x55c42f6ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513fb9d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c42f301430 con 0x55c42f6c7d10 2024-03-24T10:41:50.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513fb9d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c42f2bde00 con 0x55c42f6d08f0 2024-03-24T10:41:50.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513fb9d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c42f5b3590 con 0x55c42f6c73f0 2024-03-24T10:41:50.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513fb9d740 10 client.0 _ll_register_callbacks cb 0x55c42f622fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:50.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f5135d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c73f0 0x55c42f6cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6cded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51746/0 (socket says 172.21.15.23:51746) 2024-03-24T10:41:50.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513659c640 1 -- 172.21.15.23:0/3142478761 learned_addr learned my addr 172.21.15.23:0/3142478761 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f5135d9b640 1 --2- 172.21.15.23:0/3142478761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6d08f0 0x55c42f6ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513659c640 1 -- 172.21.15.23:0/3142478761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c73f0 msgr2=0x55c42f6cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513659c640 1 --2- 172.21.15.23:0/3142478761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c73f0 0x55c42f6cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513659c640 1 -- 172.21.15.23:0/3142478761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6d08f0 msgr2=0x55c42f6ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513659c640 1 --2- 172.21.15.23:0/3142478761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6d08f0 0x55c42f6ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513659c640 1 -- 172.21.15.23:0/3142478761 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5138069cc0 con 0x55c42f6c7d10 2024-03-24T10:41:50.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f5135d9b640 1 --2- 172.21.15.23:0/3142478761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c73f0 0x55c42f6cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:50.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f513659c640 1 --2- 172.21.15.23:0/3142478761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6cded0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f51280126e0 tx=0x7f5128012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c4d983b236248f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.546+0000 7f511dffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:50.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f511dffb640 1 -- 172.21.15.23:0/3142478761 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5128017070 con 0x55c42f6c7d10 2024-03-24T10:41:50.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f511dffb640 1 -- 172.21.15.23:0/3142478761 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f51280121d0 con 0x55c42f6c7d10 2024-03-24T10:41:50.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f513fb9d740 1 -- 172.21.15.23:0/3142478761 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c42f32d7e0 con 0x55c42f6c7d10 2024-03-24T10:41:50.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f511dffb640 1 -- 172.21.15.23:0/3142478761 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5128004050 con 0x55c42f6c7d10 2024-03-24T10:41:50.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f513fb9d740 10 client.16059 fetch_fsmap learned FSMap version 24 2024-03-24T10:41:50.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f513fb9d740 1 -- 172.21.15.23:0/3142478761 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c42f62cbd0 con 0x55c42f6c7d10 2024-03-24T10:41:50.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f511dffb640 1 -- 172.21.15.23:0/3142478761 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c42f32d7e0 con 0x55c42f6c7d10 2024-03-24T10:41:50.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f511dffb640 1 -- 172.21.15.23:0/3142478761 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) ==== 8342+0+0 (secure 0 0 0) 0x7f5128003850 con 0x55c42f6c7d10 2024-03-24T10:41:50.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f513fb9d740 10 client.16059 fetch_fsmap finished waiting for FSMap version 24 2024-03-24T10:41:50.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f513fb9d740 10 client.16059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:50.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f513fb9d740 20 client.16059 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:50.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f513fb9d740 1 --2- 172.21.15.23:0/3142478761 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c42f6e0a10 0x55c42f6e2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f513fb9d740 4 client.16059 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:50.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f513fb9d740 1 -- 172.21.15.23:0/3142478761 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c42f36a720 con 0x55c42f6e0a10 2024-03-24T10:41:50.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.547+0000 7f5135d9b640 1 --2- 172.21.15.23:0/3142478761 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c42f6e0a10 0x55c42f6e2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.548+0000 7f5135d9b640 1 --2- 172.21.15.23:0/3142478761 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c42f6e0a10 0x55c42f6e2e30 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6c21ac167d02d36 server_cookie=e0aa71fb299c1c24 in_seq=0 out_seq=0 2024-03-24T10:41:50.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.548+0000 7f511dffb640 10 client.16059 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:50.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f511dffb640 1 -- 172.21.15.23:0/3142478761 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c42f36a720 con 0x55c42f6e0a10 2024-03-24T10:41:50.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f511dffb640 10 client.16059 handle_command_reply: tid=0 2024-03-24T10:41:50.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f513fb9d740 1 client.16059 shutdown 2024-03-24T10:41:50.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f513fb9d740 1 -- 172.21.15.23:0/3142478761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 msgr2=0x55c42f6cded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f513fb9d740 1 --2- 172.21.15.23:0/3142478761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6cded0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f51280126e0 tx=0x7f5128012ba0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f513fb9d740 1 -- 172.21.15.23:0/3142478761 shutdown_connections 2024-03-24T10:41:50.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f513fb9d740 1 --2- 172.21.15.23:0/3142478761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6d08f0 0x55c42f6ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f513fb9d740 1 --2- 172.21.15.23:0/3142478761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6cded0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f513fb9d740 1 --2- 172.21.15.23:0/3142478761 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c42f6e0a10 0x55c42f6e2e30 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f513fb9d740 1 --2- 172.21.15.23:0/3142478761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c73f0 0x55c42f6cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f511dffb640 0 client.16059 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:50.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f513fb9d740 1 -- 172.21.15.23:0/3142478761 >> 172.21.15.23:0/3142478761 conn(0x55c42f6c1b20 msgr2=0x55c42f6c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f513fb9d740 1 -- 172.21.15.23:0/3142478761 shutdown_connections 2024-03-24T10:41:50.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f513fb9d740 1 -- 172.21.15.23:0/3142478761 wait complete. 2024-03-24T10:41:50.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.549+0000 7f513fb9d740 20 client.16059 trim_cache size 0 max 16384 2024-03-24T10:41:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513fb9d740 1 Processor -- start 2024-03-24T10:41:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513fb9d740 1 -- start start 2024-03-24T10:41:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513fb9d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6c73f0 0x55c42f6d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513fb9d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6dc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513fb9d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6d08f0 0x55c42f6ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513fb9d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c42f6df1e0 con 0x55c42f6c7d10 2024-03-24T10:41:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513fb9d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c42f6e3490 con 0x55c42f6c73f0 2024-03-24T10:41:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513fb9d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c42f6c7b00 con 0x55c42f6d08f0 2024-03-24T10:41:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6dc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6dc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51758/0 (socket says 172.21.15.23:51758) 2024-03-24T10:41:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513659c640 1 -- 172.21.15.23:0/2400842287 learned_addr learned my addr 172.21.15.23:0/2400842287 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f5135d9b640 1 --2- 172.21.15.23:0/2400842287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6c73f0 0x55c42f6d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f5135d9b640 1 --2- 172.21.15.23:0/2400842287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6d08f0 0x55c42f6ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513659c640 1 -- 172.21.15.23:0/2400842287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6d08f0 msgr2=0x55c42f6ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513659c640 1 --2- 172.21.15.23:0/2400842287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6d08f0 0x55c42f6ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513659c640 1 -- 172.21.15.23:0/2400842287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6c73f0 msgr2=0x55c42f6d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513659c640 1 --2- 172.21.15.23:0/2400842287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6c73f0 0x55c42f6d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513659c640 1 -- 172.21.15.23:0/2400842287 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c42f5c8c80 con 0x55c42f6c7d10 2024-03-24T10:41:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f5135d9b640 1 --2- 172.21.15.23:0/2400842287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6d08f0 0x55c42f6ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.563+0000 7f513659c640 1 --2- 172.21.15.23:0/2400842287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6dc120 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f512800f890 tx=0x7f5128025480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437907316fab72f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.564+0000 7f511ffff640 1 -- 172.21.15.23:0/2400842287 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512802a070 con 0x55c42f6c7d10 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.564+0000 7f511ffff640 1 -- 172.21.15.23:0/2400842287 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51280121d0 con 0x55c42f6c7d10 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.564+0000 7f511ffff640 1 -- 172.21.15.23:0/2400842287 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5128017070 con 0x55c42f6c7d10 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.564+0000 7f513fb9d740 1 -- 172.21.15.23:0/2400842287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 msgr2=0x55c42f6dc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.564+0000 7f513fb9d740 1 --2- 172.21.15.23:0/2400842287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6dc120 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f512800f890 tx=0x7f5128025480 comp rx=0 tx=0).stop 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.564+0000 7f513fb9d740 1 -- 172.21.15.23:0/2400842287 shutdown_connections 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.564+0000 7f513fb9d740 1 --2- 172.21.15.23:0/2400842287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6d08f0 0x55c42f6ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.564+0000 7f513fb9d740 1 --2- 172.21.15.23:0/2400842287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6c7d10 0x55c42f6dc120 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.564+0000 7f513fb9d740 1 --2- 172.21.15.23:0/2400842287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6c73f0 0x55c42f6d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.564+0000 7f513fb9d740 1 -- 172.21.15.23:0/2400842287 >> 172.21.15.23:0/2400842287 conn(0x55c42f6c1b20 msgr2=0x55c42f6bcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.564+0000 7f513fb9d740 1 -- 172.21.15.23:0/2400842287 shutdown_connections 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.564+0000 7f513fb9d740 1 -- 172.21.15.23:0/2400842287 wait complete. 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f513fb9d740 1 -- start start 2024-03-24T10:41:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f513fb9d740 1 Processor -- start 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f513fb9d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6c73f0 0x55c42f6d4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f513fb9d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c7d10 0x55c42f6d4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f513fb9d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6d08f0 0x55c42f6d4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f513fb9d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c42f745d50 con 0x55c42f6d08f0 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f513fb9d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c42f745e90 con 0x55c42f6c73f0 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f513fb9d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c42f746020 con 0x55c42f6c7d10 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f513fb9d740 10 client.0 _ll_register_callbacks cb 0x7f5128009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f5135d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6d08f0 0x55c42f6d4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f5135d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6d08f0 0x55c42f6d4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51764/0 (socket says 172.21.15.23:51764) 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f5135d9b640 1 -- 172.21.15.23:0/3213573762 learned_addr learned my addr 172.21.15.23:0/3213573762 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f5135d9b640 1 --2- 172.21.15.23:0/3213573762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6c73f0 0x55c42f6d4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f513659c640 1 --2- 172.21.15.23:0/3213573762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c7d10 0x55c42f6d4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f5135d9b640 1 -- 172.21.15.23:0/3213573762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c7d10 msgr2=0x55c42f6d4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f5135d9b640 1 --2- 172.21.15.23:0/3213573762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c7d10 0x55c42f6d4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f5135d9b640 1 -- 172.21.15.23:0/3213573762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6c73f0 msgr2=0x55c42f6d4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f5135d9b640 1 --2- 172.21.15.23:0/3213573762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6c73f0 0x55c42f6d4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f5135d9b640 1 -- 172.21.15.23:0/3213573762 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c42f62cbd0 con 0x55c42f6d08f0 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f5135d9b640 1 --2- 172.21.15.23:0/3213573762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6d08f0 0x55c42f6d4b90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f512c02c200 tx=0x7f512c045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a16bf0f665843be0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.565+0000 7f511dffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f511dffb640 1 -- 172.21.15.23:0/3213573762 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c02c970 con 0x55c42f6d08f0 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f511dffb640 1 -- 172.21.15.23:0/3213573762 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f512c02cad0 con 0x55c42f6d08f0 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f513fb9d740 1 -- 172.21.15.23:0/3213573762 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f512c004020 con 0x55c42f6d08f0 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f511dffb640 1 -- 172.21.15.23:0/3213573762 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f512c02cc90 con 0x55c42f6d08f0 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f513fb9d740 10 client.16065 fetch_fsmap learned FSMap version 24 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f513fb9d740 1 -- 172.21.15.23:0/3213573762 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c42f5ee920 con 0x55c42f6d08f0 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f511dffb640 1 -- 172.21.15.23:0/3213573762 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f512c004020 con 0x55c42f6d08f0 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f511dffb640 1 -- 172.21.15.23:0/3213573762 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) ==== 8342+0+0 (secure 0 0 0) 0x7f512c003820 con 0x55c42f6d08f0 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f513fb9d740 10 client.16065 fetch_fsmap finished waiting for FSMap version 24 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f513fb9d740 10 client.16065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f513fb9d740 20 client.16065 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f513fb9d740 1 --2- 172.21.15.23:0/3213573762 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c42f747550 0x55c42f749970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f513fb9d740 4 client.16065 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:41:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f513fb9d740 1 -- 172.21.15.23:0/3213573762 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c42f3b9890 con 0x55c42f747550 2024-03-24T10:41:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.566+0000 7f513659c640 1 --2- 172.21.15.23:0/3213573762 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c42f747550 0x55c42f749970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f511dffb640 10 client.16065 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f513659c640 1 --2- 172.21.15.23:0/3213573762 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c42f747550 0x55c42f749970 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54a724687ff35d23 server_cookie=4d30b0fcbaa5b443 in_seq=0 out_seq=0 2024-03-24T10:41:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f511dffb640 1 -- 172.21.15.23:0/3213573762 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c42f3b9890 con 0x55c42f747550 2024-03-24T10:41:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f511dffb640 10 client.16065 handle_command_reply: tid=0 2024-03-24T10:41:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f513fb9d740 1 client.16065 shutdown 2024-03-24T10:41:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f513fb9d740 1 -- 172.21.15.23:0/3213573762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6d08f0 msgr2=0x55c42f6d4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f513fb9d740 1 --2- 172.21.15.23:0/3213573762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6d08f0 0x55c42f6d4b90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f512c02c200 tx=0x7f512c045040 comp rx=0 tx=0).stop 2024-03-24T10:41:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f513fb9d740 1 -- 172.21.15.23:0/3213573762 shutdown_connections 2024-03-24T10:41:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f513fb9d740 1 --2- 172.21.15.23:0/3213573762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c42f6d08f0 0x55c42f6d4b90 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f513fb9d740 1 --2- 172.21.15.23:0/3213573762 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c42f747550 0x55c42f749970 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f513fb9d740 1 --2- 172.21.15.23:0/3213573762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c42f6c7d10 0x55c42f6d4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f511dffb640 0 client.16065 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:50.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f513fb9d740 1 --2- 172.21.15.23:0/3213573762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c42f6c73f0 0x55c42f6d4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f513fb9d740 1 -- 172.21.15.23:0/3213573762 >> 172.21.15.23:0/3213573762 conn(0x55c42f6c1b20 msgr2=0x55c42f6c0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f513fb9d740 1 -- 172.21.15.23:0/3213573762 shutdown_connections 2024-03-24T10:41:50.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f513fb9d740 1 -- 172.21.15.23:0/3213573762 wait complete. 2024-03-24T10:41:50.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.567+0000 7f513fb9d740 20 client.16065 trim_cache size 0 max 16384 2024-03-24T10:41:50.575 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:41:50.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.568+0000 7f513cee6640 1 -- 172.21.15.23:0/989389057 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f51200a2230 msgr2=0x7f51200a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.568+0000 7f513cee6640 1 --2- 172.21.15.23:0/989389057 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f51200a2230 0x7f51200a46f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5130005c40 tx=0x7f5130009f90 comp rx=0 tx=0).stop 2024-03-24T10:41:50.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.568+0000 7f513cee6640 1 -- 172.21.15.23:0/989389057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138106640 msgr2=0x7f513819c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:50.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.568+0000 7f513cee6640 1 --2- 172.21.15.23:0/989389057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138106640 0x7f513819c680 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f513000efd0 tx=0x7f513000c6a0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.568+0000 7f513cee6640 1 -- 172.21.15.23:0/989389057 shutdown_connections 2024-03-24T10:41:50.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.568+0000 7f513cee6640 1 --2- 172.21.15.23:0/989389057 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f51200a2230 0x7f51200a46f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.568+0000 7f513cee6640 1 --2- 172.21.15.23:0/989389057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5138106640 0x7f513819c680 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.568+0000 7f513cee6640 1 --2- 172.21.15.23:0/989389057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5138100ff0 0x7f5138198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.568+0000 7f513cee6640 1 --2- 172.21.15.23:0/989389057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5138100630 0x7f5138197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:50.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.568+0000 7f513cee6640 1 -- 172.21.15.23:0/989389057 >> 172.21.15.23:0/989389057 conn(0x7f5138077770 msgr2=0x7f5138103200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:50.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.568+0000 7f513cee6640 1 -- 172.21.15.23:0/989389057 shutdown_connections 2024-03-24T10:41:50.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:50.568+0000 7f513cee6640 1 -- 172.21.15.23:0/989389057 wait complete. 2024-03-24T10:41:50.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:50 smithi023 ceph-mon[32466]: pgmap v55: 97 pgs: 97 active+clean; 6.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 490 KiB/s wr, 74 op/s 2024-03-24T10:41:50.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:50 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3143242000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:41:51.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:50 smithi117 ceph-mon[36368]: pgmap v55: 97 pgs: 97 active+clean; 6.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 490 KiB/s wr, 74 op/s 2024-03-24T10:41:51.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:50 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3143242000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:41:51.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:50 smithi116 ceph-mon[36248]: pgmap v55: 97 pgs: 97 active+clean; 6.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 490 KiB/s wr, 74 op/s 2024-03-24T10:41:51.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:50 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3143242000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:41:51.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 15379 12.44 MB/sec warmup 8 sec latency 18.344 ms 2024-03-24T10:41:51.580 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:41:51.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:41:51.581 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:41:51.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.688+0000 7f84d3ec0640 1 -- 172.21.15.23:0/1670931422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84cc100f60 msgr2=0x7f84cc10d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.688+0000 7f84d3ec0640 1 --2- 172.21.15.23:0/1670931422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84cc100f60 0x7f84cc10d420 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009ca0 tx=0x7f84c402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.688+0000 7f84d3ec0640 1 -- 172.21.15.23:0/1670931422 shutdown_connections 2024-03-24T10:41:51.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.688+0000 7f84d3ec0640 1 --2- 172.21.15.23:0/1670931422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84cc100f60 0x7f84cc10d420 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.688+0000 7f84d3ec0640 1 --2- 172.21.15.23:0/1670931422 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84cc1005a0 0x7f84cc100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.688+0000 7f84d3ec0640 1 --2- 172.21.15.23:0/1670931422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84cc106660 0x7f84cc106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.689+0000 7f84d3ec0640 1 -- 172.21.15.23:0/1670931422 >> 172.21.15.23:0/1670931422 conn(0x7f84cc077710 msgr2=0x7f84cc077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:51.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.689+0000 7f84d3ec0640 1 -- 172.21.15.23:0/1670931422 shutdown_connections 2024-03-24T10:41:51.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.689+0000 7f84d3ec0640 1 -- 172.21.15.23:0/1670931422 wait complete. 2024-03-24T10:41:51.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.689+0000 7f84d3ec0640 1 Processor -- start 2024-03-24T10:41:51.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.689+0000 7f84d3ec0640 1 -- start start 2024-03-24T10:41:51.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.689+0000 7f84d3ec0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84cc1005a0 0x7f84cc19d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.689+0000 7f84d3ec0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84cc100f60 0x7f84cc19dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.689+0000 7f84d3ec0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84cc106660 0x7f84cc197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.689+0000 7f84d3ec0640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc1980a0 con 0x7f84cc100f60 2024-03-24T10:41:51.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.689+0000 7f84d3ec0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc198210 con 0x7f84cc1005a0 2024-03-24T10:41:51.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.689+0000 7f84d3ec0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc1983d0 con 0x7f84cc106660 2024-03-24T10:41:51.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.689+0000 7f84d145b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84cc100f60 0x7f84cc19dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.690+0000 7f84d145b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84cc100f60 0x7f84cc19dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51768/0 (socket says 172.21.15.23:51768) 2024-03-24T10:41:51.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.690+0000 7f84d145b640 1 -- 172.21.15.23:0/326493105 learned_addr learned my addr 172.21.15.23:0/326493105 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:51.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.690+0000 7f84d1c5c640 1 --2- 172.21.15.23:0/326493105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84cc1005a0 0x7f84cc19d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.690+0000 7f84d245d640 1 --2- 172.21.15.23:0/326493105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84cc106660 0x7f84cc197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.690+0000 7f84d145b640 1 -- 172.21.15.23:0/326493105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84cc106660 msgr2=0x7f84cc197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.690+0000 7f84d145b640 1 --2- 172.21.15.23:0/326493105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84cc106660 0x7f84cc197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.690+0000 7f84d145b640 1 -- 172.21.15.23:0/326493105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84cc1005a0 msgr2=0x7f84cc19d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.690+0000 7f84d145b640 1 --2- 172.21.15.23:0/326493105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84cc1005a0 0x7f84cc19d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.690+0000 7f84d145b640 1 -- 172.21.15.23:0/326493105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c4009900 con 0x7f84cc100f60 2024-03-24T10:41:51.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.690+0000 7f84d145b640 1 --2- 172.21.15.23:0/326493105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84cc100f60 0x7f84cc19dde0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00e9f0 tx=0x7f84bc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:51.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.693+0000 7f84c2ffd640 1 -- 172.21.15.23:0/326493105 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc009720 con 0x7f84cc100f60 2024-03-24T10:41:51.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.693+0000 7f84c2ffd640 1 -- 172.21.15.23:0/326493105 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84bc00cd80 con 0x7f84cc100f60 2024-03-24T10:41:51.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.693+0000 7f84d3ec0640 1 -- 172.21.15.23:0/326493105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84cc1986b0 con 0x7f84cc100f60 2024-03-24T10:41:51.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.693+0000 7f84c2ffd640 1 -- 172.21.15.23:0/326493105 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc010690 con 0x7f84cc100f60 2024-03-24T10:41:51.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.693+0000 7f84d3ec0640 1 -- 172.21.15.23:0/326493105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84cc1a4940 con 0x7f84cc100f60 2024-03-24T10:41:51.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.695+0000 7f84c2ffd640 1 -- 172.21.15.23:0/326493105 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f84bc0046c0 con 0x7f84cc100f60 2024-03-24T10:41:51.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d6b78740 1 Processor -- start 2024-03-24T10:41:51.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d6b78740 1 -- start start 2024-03-24T10:41:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84c2ffd640 1 --2- 172.21.15.23:0/326493105 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f84b80a2230 0x7f84b80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84c2ffd640 1 -- 172.21.15.23:0/326493105 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f84bc014070 con 0x7f84cc100f60 2024-03-24T10:41:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d245d640 1 --2- 172.21.15.23:0/326493105 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f84b80a2230 0x7f84b80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d6b78740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f2eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d6b78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f37720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d6b78740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f3a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d6b78740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558150e9c530 con 0x558150f2e760 2024-03-24T10:41:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d6b78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558150e99a60 con 0x558150f2f080 2024-03-24T10:41:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d6b78740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558150e95240 con 0x558150f37c60 2024-03-24T10:41:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d1c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f2eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d1c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f2eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51776/0 (socket says 172.21.15.23:51776) 2024-03-24T10:41:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d1c5c640 1 -- 172.21.15.23:0/12589123 learned_addr learned my addr 172.21.15.23:0/12589123 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d145b640 1 --2- 172.21.15.23:0/12589123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f37720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d245d640 1 --2- 172.21.15.23:0/12589123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f3a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d1c5c640 1 -- 172.21.15.23:0/12589123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 msgr2=0x558150f3a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.696+0000 7f84d1c5c640 1 --2- 172.21.15.23:0/12589123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f3a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.697+0000 7f84d1c5c640 1 -- 172.21.15.23:0/12589123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 msgr2=0x558150f37720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.697+0000 7f84d1c5c640 1 --2- 172.21.15.23:0/12589123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f37720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.697+0000 7f84d1c5c640 1 -- 172.21.15.23:0/12589123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558150e938c0 con 0x558150f2e760 2024-03-24T10:41:51.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.697+0000 7f84d245d640 1 --2- 172.21.15.23:0/12589123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f3a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:41:51.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.697+0000 7f84d245d640 1 --2- 172.21.15.23:0/326493105 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f84b80a2230 0x7f84b80a46f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f84c4004af0 tx=0x7f84c40325f0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:51.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.697+0000 7f84d1c5c640 1 --2- 172.21.15.23:0/12589123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f2eb40 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009c90 tx=0x7f84c8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e8d64552f5e1cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:51.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.697+0000 7f84b7fff640 1 -- 172.21.15.23:0/12589123 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c8009e40 con 0x558150f2e760 2024-03-24T10:41:51.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.697+0000 7f84b7fff640 1 -- 172.21.15.23:0/12589123 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84c80198f0 con 0x558150f2e760 2024-03-24T10:41:51.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.697+0000 7f84b7fff640 1 -- 172.21.15.23:0/12589123 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c80188b0 con 0x558150f2e760 2024-03-24T10:41:51.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.698+0000 7f84d6b78740 1 -- 172.21.15.23:0/12589123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 msgr2=0x558150f2eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.698+0000 7f84d6b78740 1 --2- 172.21.15.23:0/12589123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f2eb40 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009c90 tx=0x7f84c8011810 comp rx=0 tx=0).stop 2024-03-24T10:41:51.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.698+0000 7f84d6b78740 1 -- 172.21.15.23:0/12589123 shutdown_connections 2024-03-24T10:41:51.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.698+0000 7f84d6b78740 1 --2- 172.21.15.23:0/12589123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f3a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.698+0000 7f84d6b78740 1 --2- 172.21.15.23:0/12589123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f37720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.698+0000 7f84d6b78740 1 --2- 172.21.15.23:0/12589123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f2eb40 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.698+0000 7f84d6b78740 1 -- 172.21.15.23:0/12589123 >> 172.21.15.23:0/12589123 conn(0x558150f28e90 msgr2=0x558150f29270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:51.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.698+0000 7f84d6b78740 1 -- 172.21.15.23:0/12589123 shutdown_connections 2024-03-24T10:41:51.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.698+0000 7f84d6b78740 1 -- 172.21.15.23:0/12589123 wait complete. 2024-03-24T10:41:51.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.698+0000 7f84d6b78740 1 -- start start 2024-03-24T10:41:51.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.698+0000 7f84d6b78740 1 Processor -- start 2024-03-24T10:41:51.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d6b78740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f34cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d6b78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f35200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d6b78740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f35740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d6b78740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558150e39f80 con 0x558150f2e760 2024-03-24T10:41:51.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d6b78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558150b684b0 con 0x558150f2f080 2024-03-24T10:41:51.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d6b78740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558150e1aad0 con 0x558150f37c60 2024-03-24T10:41:51.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d6b78740 10 client.0 _ll_register_callbacks cb 0x558150e4db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:51.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d1c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f34cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d1c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f34cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51790/0 (socket says 172.21.15.23:51790) 2024-03-24T10:41:51.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d1c5c640 1 -- 172.21.15.23:0/1636583103 learned_addr learned my addr 172.21.15.23:0/1636583103 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:51.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d145b640 1 --2- 172.21.15.23:0/1636583103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f35200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d245d640 1 --2- 172.21.15.23:0/1636583103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f35740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d1c5c640 1 -- 172.21.15.23:0/1636583103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 msgr2=0x558150f35740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d1c5c640 1 --2- 172.21.15.23:0/1636583103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f35740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d1c5c640 1 -- 172.21.15.23:0/1636583103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 msgr2=0x558150f35200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d1c5c640 1 --2- 172.21.15.23:0/1636583103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f35200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d1c5c640 1 -- 172.21.15.23:0/1636583103 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c80098f0 con 0x558150f2e760 2024-03-24T10:41:51.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84d1c5c640 1 --2- 172.21.15.23:0/1636583103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f34cc0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f84c8005890 tx=0x7f84c8019d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce7b354cbc17698 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:51.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.699+0000 7f84b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:51.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.705+0000 7f84b57fa640 1 -- 172.21.15.23:0/1636583103 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c8009e40 con 0x558150f2e760 2024-03-24T10:41:51.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.705+0000 7f84b57fa640 1 -- 172.21.15.23:0/1636583103 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f84c8005250 con 0x558150f2e760 2024-03-24T10:41:51.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.705+0000 7f84d6b78740 1 -- 172.21.15.23:0/1636583103 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558150eb97a0 con 0x558150f2e760 2024-03-24T10:41:51.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.705+0000 7f84b57fa640 1 -- 172.21.15.23:0/1636583103 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f84c8005410 con 0x558150f2e760 2024-03-24T10:41:51.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.705+0000 7f84d6b78740 10 client.16077 fetch_fsmap learned FSMap version 25 2024-03-24T10:41:51.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.705+0000 7f84d6b78740 1 -- 172.21.15.23:0/1636583103 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558150e4b6d0 con 0x558150f2e760 2024-03-24T10:41:51.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.706+0000 7f84b57fa640 1 -- 172.21.15.23:0/1636583103 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558150eb97a0 con 0x558150f2e760 2024-03-24T10:41:51.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.706+0000 7f84b57fa640 1 -- 172.21.15.23:0/1636583103 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f84c8004b00 con 0x558150f2e760 2024-03-24T10:41:51.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.706+0000 7f84d6b78740 10 client.16077 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:41:51.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.706+0000 7f84d6b78740 10 client.16077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:51.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.706+0000 7f84d6b78740 20 client.16077 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:51.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.706+0000 7f84d6b78740 1 --2- 172.21.15.23:0/1636583103 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558150f4c1a0 0x558150f4e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.706+0000 7f84d6b78740 4 client.16077 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:51.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.706+0000 7f84d6b78740 1 -- 172.21.15.23:0/1636583103 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558150e8b1f0 con 0x558150f4c1a0 2024-03-24T10:41:51.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.706+0000 7f84d145b640 1 --2- 172.21.15.23:0/1636583103 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558150f4c1a0 0x558150f4e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.706+0000 7f84b57fa640 10 client.16077 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:51.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.706+0000 7f84d145b640 1 --2- 172.21.15.23:0/1636583103 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558150f4c1a0 0x558150f4e600 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93acc19b9704ec78 server_cookie=9571ce4e6f7a3307 in_seq=0 out_seq=0 2024-03-24T10:41:51.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.707+0000 7f84b57fa640 1 -- 172.21.15.23:0/1636583103 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558150e8b1f0 con 0x558150f4c1a0 2024-03-24T10:41:51.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.707+0000 7f84b57fa640 10 client.16077 handle_command_reply: tid=0 2024-03-24T10:41:51.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.707+0000 7f84d6b78740 1 client.16077 shutdown 2024-03-24T10:41:51.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.708+0000 7f84d6b78740 1 -- 172.21.15.23:0/1636583103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 msgr2=0x558150f34cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.708+0000 7f84d6b78740 1 --2- 172.21.15.23:0/1636583103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f34cc0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f84c8005890 tx=0x7f84c8019d10 comp rx=0 tx=0).stop 2024-03-24T10:41:51.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.708+0000 7f84d6b78740 1 -- 172.21.15.23:0/1636583103 shutdown_connections 2024-03-24T10:41:51.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.708+0000 7f84d6b78740 1 --2- 172.21.15.23:0/1636583103 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558150f4c1a0 0x558150f4e600 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.708+0000 7f84d6b78740 1 --2- 172.21.15.23:0/1636583103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f35740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.708+0000 7f84d6b78740 1 --2- 172.21.15.23:0/1636583103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f35200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.708+0000 7f84d6b78740 1 --2- 172.21.15.23:0/1636583103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f34cc0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.708+0000 7f84b57fa640 0 client.16077 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:51.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.708+0000 7f84d6b78740 1 -- 172.21.15.23:0/1636583103 >> 172.21.15.23:0/1636583103 conn(0x558150f28e90 msgr2=0x558150f2abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:51.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.708+0000 7f84d6b78740 1 -- 172.21.15.23:0/1636583103 shutdown_connections 2024-03-24T10:41:51.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.708+0000 7f84d6b78740 1 -- 172.21.15.23:0/1636583103 wait complete. 2024-03-24T10:41:51.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.708+0000 7f84d6b78740 20 client.16077 trim_cache size 0 max 16384 2024-03-24T10:41:51.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.726+0000 7f84d6b78740 1 Processor -- start 2024-03-24T10:41:51.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.726+0000 7f84d6b78740 1 -- start start 2024-03-24T10:41:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.726+0000 7f84d6b78740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.726+0000 7f84d6b78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f47900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.726+0000 7f84d6b78740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f53f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.726+0000 7f84d6b78740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558150f2db60 con 0x558150f2e760 2024-03-24T10:41:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.726+0000 7f84d6b78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558150f29ec0 con 0x558150f2f080 2024-03-24T10:41:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.726+0000 7f84d6b78740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558150f35c50 con 0x558150f37c60 2024-03-24T10:41:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.726+0000 7f84d1c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.726+0000 7f84d1c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f373f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51794/0 (socket says 172.21.15.23:51794) 2024-03-24T10:41:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.726+0000 7f84d1c5c640 1 -- 172.21.15.23:0/135797502 learned_addr learned my addr 172.21.15.23:0/135797502 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.726+0000 7f84d245d640 1 --2- 172.21.15.23:0/135797502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f53f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.726+0000 7f84d145b640 1 --2- 172.21.15.23:0/135797502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f47900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.727+0000 7f84d1c5c640 1 -- 172.21.15.23:0/135797502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 msgr2=0x558150f53f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.727+0000 7f84d1c5c640 1 --2- 172.21.15.23:0/135797502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f53f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.727+0000 7f84d1c5c640 1 -- 172.21.15.23:0/135797502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 msgr2=0x558150f47900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.727+0000 7f84d1c5c640 1 --2- 172.21.15.23:0/135797502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f47900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.727+0000 7f84d1c5c640 1 -- 172.21.15.23:0/135797502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558150e2fc60 con 0x558150f2e760 2024-03-24T10:41:51.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.727+0000 7f84d1c5c640 1 --2- 172.21.15.23:0/135797502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f373f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f84c8006010 tx=0x7f84c80097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5beb520ffe009d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:51.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.727+0000 7f84b77fe640 1 -- 172.21.15.23:0/135797502 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c8011dc0 con 0x558150f2e760 2024-03-24T10:41:51.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.727+0000 7f84b77fe640 1 -- 172.21.15.23:0/135797502 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84c8019d80 con 0x558150f2e760 2024-03-24T10:41:51.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.727+0000 7f84b77fe640 1 -- 172.21.15.23:0/135797502 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c80187c0 con 0x558150f2e760 2024-03-24T10:41:51.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 -- 172.21.15.23:0/135797502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 msgr2=0x558150f373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 --2- 172.21.15.23:0/135797502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f373f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f84c8006010 tx=0x7f84c80097a0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 -- 172.21.15.23:0/135797502 shutdown_connections 2024-03-24T10:41:51.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 --2- 172.21.15.23:0/135797502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f53f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 --2- 172.21.15.23:0/135797502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f47900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 --2- 172.21.15.23:0/135797502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f373f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 -- 172.21.15.23:0/135797502 >> 172.21.15.23:0/135797502 conn(0x558150f28e90 msgr2=0x558150f35510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:51.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 -- 172.21.15.23:0/135797502 shutdown_connections 2024-03-24T10:41:51.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 -- 172.21.15.23:0/135797502 wait complete. 2024-03-24T10:41:51.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 -- start start 2024-03-24T10:41:51.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 Processor -- start 2024-03-24T10:41:51.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f43f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f3bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f3c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558150f3c920 con 0x558150f2e760 2024-03-24T10:41:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558150f3ca90 con 0x558150f2f080 2024-03-24T10:41:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.728+0000 7f84d6b78740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558150f3cc50 con 0x558150f37c60 2024-03-24T10:41:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d6b78740 10 client.0 _ll_register_callbacks cb 0x7f84c8009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d1c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f43f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d1c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f43f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51796/0 (socket says 172.21.15.23:51796) 2024-03-24T10:41:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d1c5c640 1 -- 172.21.15.23:0/3213463077 learned_addr learned my addr 172.21.15.23:0/3213463077 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d145b640 1 --2- 172.21.15.23:0/3213463077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f3bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d245d640 1 --2- 172.21.15.23:0/3213463077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f3c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d1c5c640 1 -- 172.21.15.23:0/3213463077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 msgr2=0x558150f3c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d1c5c640 1 --2- 172.21.15.23:0/3213463077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f3c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d1c5c640 1 -- 172.21.15.23:0/3213463077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 msgr2=0x558150f3bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d1c5c640 1 --2- 172.21.15.23:0/3213463077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f3bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d1c5c640 1 -- 172.21.15.23:0/3213463077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558150e4b6d0 con 0x558150f2e760 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d1c5c640 1 --2- 172.21.15.23:0/3213463077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f43f30 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009be0 tx=0x7f84c8005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d9a72c1d75f3491 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84b57fa640 1 -- 172.21.15.23:0/3213463077 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c8005450 con 0x558150f2e760 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d6b78740 1 -- 172.21.15.23:0/3213463077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84c80198f0 con 0x558150f2e760 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84b57fa640 1 -- 172.21.15.23:0/3213463077 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f84c8000e40 con 0x558150f2e760 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84b57fa640 1 -- 172.21.15.23:0/3213463077 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f84c8009240 con 0x558150f2e760 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d6b78740 10 client.16083 fetch_fsmap learned FSMap version 25 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.729+0000 7f84d6b78740 1 -- 172.21.15.23:0/3213463077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558150e55970 con 0x558150f2e760 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.730+0000 7f84b57fa640 1 -- 172.21.15.23:0/3213463077 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84c804b020 con 0x558150f2e760 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.730+0000 7f84b57fa640 1 -- 172.21.15.23:0/3213463077 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f84c803c070 con 0x558150f2e760 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.730+0000 7f84d6b78740 10 client.16083 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.730+0000 7f84d6b78740 10 client.16083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.730+0000 7f84d6b78740 20 client.16083 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.730+0000 7f84d6b78740 1 --2- 172.21.15.23:0/3213463077 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558150fb7e20 0x558150fba240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.730+0000 7f84d6b78740 4 client.16083 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:41:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.730+0000 7f84d6b78740 1 -- 172.21.15.23:0/3213463077 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558150b94860 con 0x558150fb7e20 2024-03-24T10:41:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.730+0000 7f84d145b640 1 --2- 172.21.15.23:0/3213463077 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558150fb7e20 0x558150fba240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.730+0000 7f84d145b640 1 --2- 172.21.15.23:0/3213463077 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558150fb7e20 0x558150fba240 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b600802d5dbf1d88 server_cookie=91a0c8bedf362522 in_seq=0 out_seq=0 2024-03-24T10:41:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.730+0000 7f84b57fa640 10 client.16083 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.731+0000 7f84b57fa640 1 -- 172.21.15.23:0/3213463077 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558150b94860 con 0x558150fb7e20 2024-03-24T10:41:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.731+0000 7f84b57fa640 10 client.16083 handle_command_reply: tid=0 2024-03-24T10:41:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.731+0000 7f84d6b78740 1 client.16083 shutdown 2024-03-24T10:41:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.731+0000 7f84d6b78740 1 -- 172.21.15.23:0/3213463077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 msgr2=0x558150f43f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.731+0000 7f84d6b78740 1 --2- 172.21.15.23:0/3213463077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f43f30 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009be0 tx=0x7f84c8005970 comp rx=0 tx=0).stop 2024-03-24T10:41:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.731+0000 7f84d6b78740 1 -- 172.21.15.23:0/3213463077 shutdown_connections 2024-03-24T10:41:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.731+0000 7f84d6b78740 1 --2- 172.21.15.23:0/3213463077 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558150fb7e20 0x558150fba240 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.731+0000 7f84d6b78740 1 --2- 172.21.15.23:0/3213463077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558150f37c60 0x558150f3c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.731+0000 7f84d6b78740 1 --2- 172.21.15.23:0/3213463077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558150f2f080 0x558150f3bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.731+0000 7f84b57fa640 0 client.16083 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:51.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.731+0000 7f84d6b78740 1 --2- 172.21.15.23:0/3213463077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558150f2e760 0x558150f43f30 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.731+0000 7f84d6b78740 1 -- 172.21.15.23:0/3213463077 >> 172.21.15.23:0/3213463077 conn(0x558150f28e90 msgr2=0x558150f27660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:51.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.731+0000 7f84d6b78740 1 -- 172.21.15.23:0/3213463077 shutdown_connections 2024-03-24T10:41:51.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.732+0000 7f84d6b78740 1 -- 172.21.15.23:0/3213463077 wait complete. 2024-03-24T10:41:51.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.732+0000 7f84d6b78740 20 client.16083 trim_cache size 0 max 16384 2024-03-24T10:41:51.739 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:41:51.739 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:41:51.739 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "148c2f74-7077-47e6-a855-fe7d979980de", 2024-03-24T10:41:51.739 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:41:51.739 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:41:51.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.732+0000 7f84d3ec0640 1 -- 172.21.15.23:0/326493105 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f84b80a2230 msgr2=0x7f84b80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.732+0000 7f84d3ec0640 1 --2- 172.21.15.23:0/326493105 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f84b80a2230 0x7f84b80a46f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f84c4004af0 tx=0x7f84c40325f0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.732+0000 7f84d3ec0640 1 -- 172.21.15.23:0/326493105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84cc100f60 msgr2=0x7f84cc19dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.732+0000 7f84d3ec0640 1 --2- 172.21.15.23:0/326493105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84cc100f60 0x7f84cc19dde0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00e9f0 tx=0x7f84bc00edf0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.732+0000 7f84d3ec0640 1 -- 172.21.15.23:0/326493105 shutdown_connections 2024-03-24T10:41:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.732+0000 7f84d3ec0640 1 --2- 172.21.15.23:0/326493105 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f84b80a2230 0x7f84b80a46f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.732+0000 7f84d3ec0640 1 --2- 172.21.15.23:0/326493105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84cc106660 0x7f84cc197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.732+0000 7f84d3ec0640 1 --2- 172.21.15.23:0/326493105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84cc100f60 0x7f84cc19dde0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.732+0000 7f84d3ec0640 1 --2- 172.21.15.23:0/326493105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84cc1005a0 0x7f84cc19d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.732+0000 7f84d3ec0640 1 -- 172.21.15.23:0/326493105 >> 172.21.15.23:0/326493105 conn(0x7f84cc077710 msgr2=0x7f84cc0ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.732+0000 7f84d3ec0640 1 -- 172.21.15.23:0/326493105 shutdown_connections 2024-03-24T10:41:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.732+0000 7f84d3ec0640 1 -- 172.21.15.23:0/326493105 wait complete. 2024-03-24T10:41:51.744 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:41:51.744 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:41:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.850+0000 7f34ff93e640 1 -- 172.21.15.23:0/3448834391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34f8106640 msgr2=0x7f34f8106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.850+0000 7f34ff93e640 1 --2- 172.21.15.23:0/3448834391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34f8106640 0x7f34f8106a20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f34f4009930 tx=0x7f34f402bc30 comp rx=0 tx=0).stop 2024-03-24T10:41:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.850+0000 7f34ff93e640 1 -- 172.21.15.23:0/3448834391 shutdown_connections 2024-03-24T10:41:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.850+0000 7f34ff93e640 1 --2- 172.21.15.23:0/3448834391 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34f8100ff0 0x7f34f810d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.850+0000 7f34ff93e640 1 --2- 172.21.15.23:0/3448834391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34f8100630 0x7f34f8100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.850+0000 7f34ff93e640 1 --2- 172.21.15.23:0/3448834391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34f8106640 0x7f34f8106a20 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.850+0000 7f34ff93e640 1 -- 172.21.15.23:0/3448834391 >> 172.21.15.23:0/3448834391 conn(0x7f34f8077770 msgr2=0x7f34f8077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:51.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.850+0000 7f34ff93e640 1 -- 172.21.15.23:0/3448834391 shutdown_connections 2024-03-24T10:41:51.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.850+0000 7f34ff93e640 1 -- 172.21.15.23:0/3448834391 wait complete. 2024-03-24T10:41:51.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.850+0000 7f34ff93e640 1 Processor -- start 2024-03-24T10:41:51.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.850+0000 7f34ff93e640 1 -- start start 2024-03-24T10:41:51.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34ff93e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34f8100630 0x7f34f8197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34ff93e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34f8100ff0 0x7f34f81982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34ff93e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34f8106640 0x7f34f819c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34ff93e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f34f819cc40 con 0x7f34f8106640 2024-03-24T10:41:51.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34ff93e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f34f819cdb0 con 0x7f34f8100630 2024-03-24T10:41:51.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34ff93e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f34f819cf70 con 0x7f34f8100ff0 2024-03-24T10:41:51.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34fdedb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34f8106640 0x7f34f819c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34fdedb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34f8106640 0x7f34f819c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51826/0 (socket says 172.21.15.23:51826) 2024-03-24T10:41:51.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34fdedb640 1 -- 172.21.15.23:0/4170521128 learned_addr learned my addr 172.21.15.23:0/4170521128 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34fd6da640 1 --2- 172.21.15.23:0/4170521128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34f8100630 0x7f34f8197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34fced9640 1 --2- 172.21.15.23:0/4170521128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34f8100ff0 0x7f34f81982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34fdedb640 1 -- 172.21.15.23:0/4170521128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34f8100ff0 msgr2=0x7f34f81982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34fdedb640 1 --2- 172.21.15.23:0/4170521128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34f8100ff0 0x7f34f81982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34fdedb640 1 -- 172.21.15.23:0/4170521128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34f8100630 msgr2=0x7f34f8197d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34fdedb640 1 --2- 172.21.15.23:0/4170521128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34f8100630 0x7f34f8197d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34fdedb640 1 -- 172.21.15.23:0/4170521128 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f4009560 con 0x7f34f8106640 2024-03-24T10:41:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34fdedb640 1 --2- 172.21.15.23:0/4170521128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34f8106640 0x7f34f819c640 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f34f000eb10 tx=0x7f34f000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34ee7fc640 1 -- 172.21.15.23:0/4170521128 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0004b20 con 0x7f34f8106640 2024-03-24T10:41:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.851+0000 7f34ee7fc640 1 -- 172.21.15.23:0/4170521128 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f0004c80 con 0x7f34f8106640 2024-03-24T10:41:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.852+0000 7f34ff93e640 1 -- 172.21.15.23:0/4170521128 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34f819d250 con 0x7f34f8106640 2024-03-24T10:41:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.852+0000 7f34ee7fc640 1 -- 172.21.15.23:0/4170521128 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0010650 con 0x7f34f8106640 2024-03-24T10:41:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.852+0000 7f34ff93e640 1 -- 172.21.15.23:0/4170521128 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f81a4ad0 con 0x7f34f8106640 2024-03-24T10:41:51.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.854+0000 7f34ee7fc640 1 -- 172.21.15.23:0/4170521128 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f34f0005020 con 0x7f34f8106640 2024-03-24T10:41:51.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.854+0000 7f35025f7740 1 Processor -- start 2024-03-24T10:41:51.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.854+0000 7f35025f7740 1 -- start start 2024-03-24T10:41:51.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.854+0000 7f35025f7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6f1d0 0x564c7ee6f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.854+0000 7f35025f7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6faf0 0x564c7ee78190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.854+0000 7f35025f7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee786d0 0x564c7ee7aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.854+0000 7f35025f7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x564c7ed93f30 con 0x564c7ee786d0 2024-03-24T10:41:51.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.854+0000 7f35025f7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564c7eddd810 con 0x564c7ee6f1d0 2024-03-24T10:41:51.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.854+0000 7f35025f7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564c7eddad40 con 0x564c7ee6faf0 2024-03-24T10:41:51.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34fced9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee786d0 0x564c7ee7aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34fd6da640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6faf0 0x564c7ee78190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34fced9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee786d0 0x564c7ee7aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51838/0 (socket says 172.21.15.23:51838) 2024-03-24T10:41:51.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34fced9640 1 -- 172.21.15.23:0/1911336296 learned_addr learned my addr 172.21.15.23:0/1911336296 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:51.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34ee7fc640 1 --2- 172.21.15.23:0/4170521128 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34e40a2230 0x7f34e40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34fced9640 1 --2- 172.21.15.23:0/1911336296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6f1d0 0x564c7ee6f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34fdedb640 1 --2- 172.21.15.23:0/4170521128 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34e40a2230 0x7f34e40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34fd6da640 1 -- 172.21.15.23:0/1911336296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6f1d0 msgr2=0x564c7ee6f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34fd6da640 1 --2- 172.21.15.23:0/1911336296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6f1d0 0x564c7ee6f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34fd6da640 1 -- 172.21.15.23:0/1911336296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee786d0 msgr2=0x564c7ee7aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34fd6da640 1 --2- 172.21.15.23:0/1911336296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee786d0 0x564c7ee7aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34fd6da640 1 -- 172.21.15.23:0/1911336296 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c7ed97ea0 con 0x564c7ee6faf0 2024-03-24T10:41:51.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34ee7fc640 1 -- 172.21.15.23:0/4170521128 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f34f0014070 con 0x7f34f8106640 2024-03-24T10:41:51.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34fd6da640 1 --2- 172.21.15.23:0/1911336296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6faf0 0x564c7ee78190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f34f4009230 tx=0x7f34f402c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bdbe0b3118c3336 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:51.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34fdedb640 1 --2- 172.21.15.23:0/4170521128 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34e40a2230 0x7f34e40a46f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f34f009ac80 tx=0x7f34f000efd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:51.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34e37fe640 1 -- 172.21.15.23:0/1911336296 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4004800 con 0x564c7ee6faf0 2024-03-24T10:41:51.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34e37fe640 1 -- 172.21.15.23:0/1911336296 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f4004960 con 0x564c7ee6faf0 2024-03-24T10:41:51.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.855+0000 7f34e37fe640 1 -- 172.21.15.23:0/1911336296 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4005050 con 0x564c7ee6faf0 2024-03-24T10:41:51.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.856+0000 7f35025f7740 1 -- 172.21.15.23:0/1911336296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6faf0 msgr2=0x564c7ee78190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.856+0000 7f35025f7740 1 --2- 172.21.15.23:0/1911336296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6faf0 0x564c7ee78190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f34f4009230 tx=0x7f34f402c5d0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.856+0000 7f35025f7740 1 -- 172.21.15.23:0/1911336296 shutdown_connections 2024-03-24T10:41:51.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.856+0000 7f35025f7740 1 --2- 172.21.15.23:0/1911336296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee786d0 0x564c7ee7aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.856+0000 7f35025f7740 1 --2- 172.21.15.23:0/1911336296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6faf0 0x564c7ee78190 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.856+0000 7f35025f7740 1 --2- 172.21.15.23:0/1911336296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6f1d0 0x564c7ee6f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.856+0000 7f35025f7740 1 -- 172.21.15.23:0/1911336296 >> 172.21.15.23:0/1911336296 conn(0x564c7ee69900 msgr2=0x564c7ee69ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:51.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.856+0000 7f35025f7740 1 -- 172.21.15.23:0/1911336296 shutdown_connections 2024-03-24T10:41:51.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.856+0000 7f35025f7740 1 -- 172.21.15.23:0/1911336296 wait complete. 2024-03-24T10:41:51.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.856+0000 7f35025f7740 1 -- start start 2024-03-24T10:41:51.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.856+0000 7f35025f7740 1 Processor -- start 2024-03-24T10:41:51.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f35025f7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6f1d0 0x564c7ee75700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f35025f7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee6faf0 0x564c7ee75c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f35025f7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee786d0 0x564c7ee76180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f35025f7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x564c7edd6520 con 0x564c7ee6faf0 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f35025f7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564c7eaa9430 con 0x564c7ee786d0 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f35025f7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564c7ea65e00 con 0x564c7ee6f1d0 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f35025f7740 10 client.0 _ll_register_callbacks cb 0x564c7ed74d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f34fd6da640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee6faf0 0x564c7ee75c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f34fd6da640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee6faf0 0x564c7ee75c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51854/0 (socket says 172.21.15.23:51854) 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f34fced9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6f1d0 0x564c7ee75700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f34fd6da640 1 -- 172.21.15.23:0/257684451 learned_addr learned my addr 172.21.15.23:0/257684451 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f34fced9640 1 --2- 172.21.15.23:0/257684451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee786d0 0x564c7ee76180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f34fd6da640 1 -- 172.21.15.23:0/257684451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6f1d0 msgr2=0x564c7ee75700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f34fd6da640 1 --2- 172.21.15.23:0/257684451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6f1d0 0x564c7ee75700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f34fd6da640 1 -- 172.21.15.23:0/257684451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee786d0 msgr2=0x564c7ee76180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f34fd6da640 1 --2- 172.21.15.23:0/257684451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee786d0 0x564c7ee76180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f34fd6da640 1 -- 172.21.15.23:0/257684451 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f810de00 con 0x564c7ee6faf0 2024-03-24T10:41:51.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f34fced9640 1 --2- 172.21.15.23:0/257684451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6f1d0 0x564c7ee75700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:51.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f34fd6da640 1 --2- 172.21.15.23:0/257684451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee6faf0 0x564c7ee75c40 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f34f402c170 tx=0x7f34f4004330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c6ef1675dc0842 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:51.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.857+0000 7f34e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:51.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f34e0ff9640 1 -- 172.21.15.23:0/257684451 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4004800 con 0x564c7ee6faf0 2024-03-24T10:41:51.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f34e0ff9640 1 -- 172.21.15.23:0/257684451 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f34f40044b0 con 0x564c7ee6faf0 2024-03-24T10:41:51.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f35025f7740 1 -- 172.21.15.23:0/257684451 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564c7ead57e0 con 0x564c7ee6faf0 2024-03-24T10:41:51.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f34e0ff9640 1 -- 172.21.15.23:0/257684451 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f34f4004ad0 con 0x564c7ee6faf0 2024-03-24T10:41:51.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f35025f7740 10 client.16095 fetch_fsmap learned FSMap version 25 2024-03-24T10:41:51.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f35025f7740 1 -- 172.21.15.23:0/257684451 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564c7edd4ba0 con 0x564c7ee6faf0 2024-03-24T10:41:51.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f34e0ff9640 1 -- 172.21.15.23:0/257684451 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564c7ead57e0 con 0x564c7ee6faf0 2024-03-24T10:41:51.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f34e0ff9640 1 -- 172.21.15.23:0/257684451 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f34f403c050 con 0x564c7ee6faf0 2024-03-24T10:41:51.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f35025f7740 10 client.16095 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:41:51.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f35025f7740 10 client.16095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:51.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f35025f7740 20 client.16095 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:51.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f35025f7740 1 --2- 172.21.15.23:0/257684451 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564c7ee887c0 0x564c7ee8abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f35025f7740 4 client.16095 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f35025f7740 1 -- 172.21.15.23:0/257684451 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c7eb12720 con 0x564c7ee887c0 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.858+0000 7f34fced9640 1 --2- 172.21.15.23:0/257684451 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564c7ee887c0 0x564c7ee8abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.859+0000 7f34fced9640 1 --2- 172.21.15.23:0/257684451 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564c7ee887c0 0x564c7ee8abe0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54a64ac51055c95f server_cookie=593fe9f481332052 in_seq=0 out_seq=0 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.859+0000 7f34e0ff9640 10 client.16095 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.859+0000 7f34e0ff9640 1 -- 172.21.15.23:0/257684451 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564c7eb12720 con 0x564c7ee887c0 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.859+0000 7f34e0ff9640 10 client.16095 handle_command_reply: tid=0 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.860+0000 7f35025f7740 1 client.16095 shutdown 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.860+0000 7f35025f7740 1 -- 172.21.15.23:0/257684451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee6faf0 msgr2=0x564c7ee75c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.860+0000 7f35025f7740 1 --2- 172.21.15.23:0/257684451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee6faf0 0x564c7ee75c40 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f34f402c170 tx=0x7f34f4004330 comp rx=0 tx=0).stop 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.860+0000 7f35025f7740 1 -- 172.21.15.23:0/257684451 shutdown_connections 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.860+0000 7f35025f7740 1 --2- 172.21.15.23:0/257684451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee786d0 0x564c7ee76180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.860+0000 7f35025f7740 1 --2- 172.21.15.23:0/257684451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee6faf0 0x564c7ee75c40 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.860+0000 7f35025f7740 1 --2- 172.21.15.23:0/257684451 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564c7ee887c0 0x564c7ee8abe0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.860+0000 7f35025f7740 1 --2- 172.21.15.23:0/257684451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6f1d0 0x564c7ee75700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.860+0000 7f35025f7740 1 -- 172.21.15.23:0/257684451 >> 172.21.15.23:0/257684451 conn(0x564c7ee69900 msgr2=0x564c7ee68b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:51.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.860+0000 7f34e0ff9640 0 client.16095 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:51.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.860+0000 7f35025f7740 1 -- 172.21.15.23:0/257684451 shutdown_connections 2024-03-24T10:41:51.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.860+0000 7f35025f7740 1 -- 172.21.15.23:0/257684451 wait complete. 2024-03-24T10:41:51.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.860+0000 7f35025f7740 20 client.16095 trim_cache size 0 max 16384 2024-03-24T10:41:51.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f35025f7740 1 Processor -- start 2024-03-24T10:41:51.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f35025f7740 1 -- start start 2024-03-24T10:41:51.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f35025f7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee6f1d0 0x564c7ee77e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f35025f7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6faf0 0x564c7ee83f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f35025f7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee786d0 0x564c7ee689e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f35025f7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x564c7ed5b430 con 0x564c7ee6f1d0 2024-03-24T10:41:51.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f35025f7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564c7ee766d0 con 0x564c7ee6faf0 2024-03-24T10:41:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f35025f7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564c7ee6e370 con 0x564c7ee786d0 2024-03-24T10:41:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f34fced9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee6f1d0 0x564c7ee77e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f34fced9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee6f1d0 0x564c7ee77e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51862/0 (socket says 172.21.15.23:51862) 2024-03-24T10:41:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f34fced9640 1 -- 172.21.15.23:0/142704663 learned_addr learned my addr 172.21.15.23:0/142704663 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f34fced9640 1 --2- 172.21.15.23:0/142704663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee786d0 0x564c7ee689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f34fd6da640 1 --2- 172.21.15.23:0/142704663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6faf0 0x564c7ee83f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f34fced9640 1 -- 172.21.15.23:0/142704663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee786d0 msgr2=0x564c7ee689e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f34fced9640 1 --2- 172.21.15.23:0/142704663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee786d0 0x564c7ee689e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f34fced9640 1 -- 172.21.15.23:0/142704663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6faf0 msgr2=0x564c7ee83f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f34fced9640 1 --2- 172.21.15.23:0/142704663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6faf0 0x564c7ee83f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.878+0000 7f34fced9640 1 -- 172.21.15.23:0/142704663 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c7ed70b20 con 0x564c7ee6f1d0 2024-03-24T10:41:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.879+0000 7f34fced9640 1 --2- 172.21.15.23:0/142704663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee6f1d0 0x564c7ee77e30 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009fd0 tx=0x7f34e8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b32845ff661e27 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.879+0000 7f34e2ffd640 1 -- 172.21.15.23:0/142704663 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e8016040 con 0x564c7ee6f1d0 2024-03-24T10:41:51.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.879+0000 7f34e2ffd640 1 -- 172.21.15.23:0/142704663 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34e80044d0 con 0x564c7ee6f1d0 2024-03-24T10:41:51.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.879+0000 7f34e2ffd640 1 -- 172.21.15.23:0/142704663 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e8004820 con 0x564c7ee6f1d0 2024-03-24T10:41:51.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.879+0000 7f35025f7740 1 -- 172.21.15.23:0/142704663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee6f1d0 msgr2=0x564c7ee77e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.879+0000 7f35025f7740 1 --2- 172.21.15.23:0/142704663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee6f1d0 0x564c7ee77e30 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009fd0 tx=0x7f34e8005f90 comp rx=0 tx=0).stop 2024-03-24T10:41:51.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 -- 172.21.15.23:0/142704663 shutdown_connections 2024-03-24T10:41:51.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 --2- 172.21.15.23:0/142704663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee786d0 0x564c7ee689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 --2- 172.21.15.23:0/142704663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6faf0 0x564c7ee83f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 --2- 172.21.15.23:0/142704663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee6f1d0 0x564c7ee77e30 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 -- 172.21.15.23:0/142704663 >> 172.21.15.23:0/142704663 conn(0x564c7ee69900 msgr2=0x564c7ee682d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:51.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 -- 172.21.15.23:0/142704663 shutdown_connections 2024-03-24T10:41:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 -- 172.21.15.23:0/142704663 wait complete. 2024-03-24T10:41:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 -- start start 2024-03-24T10:41:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 Processor -- start 2024-03-24T10:41:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6f1d0 0x564c7ee80730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6faf0 0x564c7ee7baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee786d0 0x564c7ee7c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x564c7ee7c700 con 0x564c7ee786d0 2024-03-24T10:41:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564c7ee7c870 con 0x564c7ee6f1d0 2024-03-24T10:41:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564c7eeedae0 con 0x564c7ee6faf0 2024-03-24T10:41:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f35025f7740 10 client.0 _ll_register_callbacks cb 0x7f34e8009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.880+0000 7f34fced9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee786d0 0x564c7ee7c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34fced9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee786d0 0x564c7ee7c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51874/0 (socket says 172.21.15.23:51874) 2024-03-24T10:41:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34fced9640 1 -- 172.21.15.23:0/2094782518 learned_addr learned my addr 172.21.15.23:0/2094782518 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34fced9640 1 --2- 172.21.15.23:0/2094782518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6f1d0 0x564c7ee80730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34fd6da640 1 --2- 172.21.15.23:0/2094782518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6faf0 0x564c7ee7baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34fced9640 1 -- 172.21.15.23:0/2094782518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6faf0 msgr2=0x564c7ee7baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34fced9640 1 --2- 172.21.15.23:0/2094782518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6faf0 0x564c7ee7baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34fced9640 1 -- 172.21.15.23:0/2094782518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6f1d0 msgr2=0x564c7ee80730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34fced9640 1 --2- 172.21.15.23:0/2094782518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6f1d0 0x564c7ee80730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34fced9640 1 -- 172.21.15.23:0/2094782518 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e8019040 con 0x564c7ee786d0 2024-03-24T10:41:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34fced9640 1 --2- 172.21.15.23:0/2094782518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee786d0 0x564c7ee7c030 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009800 tx=0x7f34e8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9edc5eacb05f75f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34e0ff9640 1 -- 172.21.15.23:0/2094782518 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e8016070 con 0x564c7ee786d0 2024-03-24T10:41:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34e0ff9640 1 -- 172.21.15.23:0/2094782518 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f34e8005240 con 0x564c7ee786d0 2024-03-24T10:41:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f35025f7740 1 -- 172.21.15.23:0/2094782518 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34f4004960 con 0x564c7ee786d0 2024-03-24T10:41:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34e0ff9640 1 -- 172.21.15.23:0/2094782518 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f34e8005400 con 0x564c7ee786d0 2024-03-24T10:41:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f35025f7740 10 client.16101 fetch_fsmap learned FSMap version 25 2024-03-24T10:41:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f35025f7740 1 -- 172.21.15.23:0/2094782518 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564c7ed96700 con 0x564c7ee786d0 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.881+0000 7f34e0ff9640 1 -- 172.21.15.23:0/2094782518 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34e8041080 con 0x564c7ee786d0 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f34e0ff9640 1 -- 172.21.15.23:0/2094782518 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f34e80049c0 con 0x564c7ee786d0 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f35025f7740 10 client.16101 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f35025f7740 10 client.16101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f35025f7740 20 client.16101 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f35025f7740 1 --2- 172.21.15.23:0/2094782518 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564c7eeef100 0x564c7eef1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f35025f7740 4 client.16101 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f35025f7740 1 -- 172.21.15.23:0/2094782518 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564c7eb61890 con 0x564c7eeef100 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f34fd6da640 1 --2- 172.21.15.23:0/2094782518 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564c7eeef100 0x564c7eef1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f34e0ff9640 10 client.16101 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f34fd6da640 1 --2- 172.21.15.23:0/2094782518 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564c7eeef100 0x564c7eef1520 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94e887468872922a server_cookie=b047150f545815e in_seq=0 out_seq=0 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f34e0ff9640 1 -- 172.21.15.23:0/2094782518 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564c7eb61890 con 0x564c7eeef100 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f34e0ff9640 10 client.16101 handle_command_reply: tid=0 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f35025f7740 1 client.16101 shutdown 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f35025f7740 1 -- 172.21.15.23:0/2094782518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee786d0 msgr2=0x564c7ee7c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.882+0000 7f35025f7740 1 --2- 172.21.15.23:0/2094782518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee786d0 0x564c7ee7c030 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f34e8009800 tx=0x7f34e8005850 comp rx=0 tx=0).stop 2024-03-24T10:41:51.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f35025f7740 1 -- 172.21.15.23:0/2094782518 shutdown_connections 2024-03-24T10:41:51.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f35025f7740 1 --2- 172.21.15.23:0/2094782518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c7ee786d0 0x564c7ee7c030 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f35025f7740 1 --2- 172.21.15.23:0/2094782518 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564c7eeef100 0x564c7eef1520 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f35025f7740 1 --2- 172.21.15.23:0/2094782518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564c7ee6faf0 0x564c7ee7baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f35025f7740 1 --2- 172.21.15.23:0/2094782518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564c7ee6f1d0 0x564c7ee80730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f34e0ff9640 0 client.16101 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:51.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f35025f7740 1 -- 172.21.15.23:0/2094782518 >> 172.21.15.23:0/2094782518 conn(0x564c7ee69900 msgr2=0x564c7ee68b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:51.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f35025f7740 1 -- 172.21.15.23:0/2094782518 shutdown_connections 2024-03-24T10:41:51.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f35025f7740 1 -- 172.21.15.23:0/2094782518 wait complete. 2024-03-24T10:41:51.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f35025f7740 20 client.16101 trim_cache size 0 max 16384 2024-03-24T10:41:51.891 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:41:51.891 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:41:51.891 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:41:51.891 INFO:teuthology.orchestra.run.smithi023.stdout: "148c2f74-7077-47e6-a855-fe7d979980de": { 2024-03-24T10:41:51.891 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:41:51.891 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "148c2f74-7077-47e6-a855-fe7d979980de", 2024-03-24T10:41:51.891 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:41:51.891 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:41:51.891 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:41:51.891 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:41:51.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f34ff93e640 1 -- 172.21.15.23:0/4170521128 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34e40a2230 msgr2=0x7f34e40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f34ff93e640 1 --2- 172.21.15.23:0/4170521128 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34e40a2230 0x7f34e40a46f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f34f009ac80 tx=0x7f34f000efd0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f34ff93e640 1 -- 172.21.15.23:0/4170521128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34f8106640 msgr2=0x7f34f819c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:51.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f34ff93e640 1 --2- 172.21.15.23:0/4170521128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34f8106640 0x7f34f819c640 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f34f000eb10 tx=0x7f34f000c2d0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f34ff93e640 1 -- 172.21.15.23:0/4170521128 shutdown_connections 2024-03-24T10:41:51.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f34ff93e640 1 --2- 172.21.15.23:0/4170521128 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34e40a2230 0x7f34e40a46f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f34ff93e640 1 --2- 172.21.15.23:0/4170521128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34f8106640 0x7f34f819c640 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f34ff93e640 1 --2- 172.21.15.23:0/4170521128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34f8100ff0 0x7f34f81982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f34ff93e640 1 --2- 172.21.15.23:0/4170521128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34f8100630 0x7f34f8197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:51.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f34ff93e640 1 -- 172.21.15.23:0/4170521128 >> 172.21.15.23:0/4170521128 conn(0x7f34f8077770 msgr2=0x7f34f806a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:51.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f34ff93e640 1 -- 172.21.15.23:0/4170521128 shutdown_connections 2024-03-24T10:41:51.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:41:51.883+0000 7f34ff93e640 1 -- 172.21.15.23:0/4170521128 wait complete. 2024-03-24T10:41:51.894 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:51 smithi023 ceph-mon[32466]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:41:51.895 INFO:tasks.cephfs.filesystem:scrub status for tag:148c2f74-7077-47e6-a855-fe7d979980de - {'path': '/', 'tag': '148c2f74-7077-47e6-a855-fe7d979980de', 'options': 'recursive,force'} 2024-03-24T10:41:52.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:51 smithi117 ceph-mon[36368]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:41:52.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:51 smithi116 ceph-mon[36248]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:41:52.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 18174 12.95 MB/sec warmup 9 sec latency 11.385 ms 2024-03-24T10:41:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:52 smithi117 ceph-mon[36368]: pgmap v56: 97 pgs: 97 active+clean; 9.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 833 KiB/s wr, 118 op/s 2024-03-24T10:41:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:52 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:41:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:52 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:52 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:41:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:52 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:41:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:52 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:53.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:52 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:53.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:52 smithi116 ceph-mon[36248]: pgmap v56: 97 pgs: 97 active+clean; 9.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 833 KiB/s wr, 118 op/s 2024-03-24T10:41:53.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:52 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:41:53.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:52 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:53.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:52 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:41:53.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:52 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:41:53.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:52 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:53.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:52 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:53.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 20764 12.92 MB/sec warmup 10 sec latency 23.309 ms 2024-03-24T10:41:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:52 smithi023 ceph-mon[32466]: pgmap v56: 97 pgs: 97 active+clean; 9.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 833 KiB/s wr, 118 op/s 2024-03-24T10:41:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:52 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:41:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:52 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:52 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:41:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:52 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:41:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:52 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:52 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:41:54.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 22830 12.80 MB/sec warmup 11 sec latency 14.718 ms 2024-03-24T10:41:55.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:54 smithi117 ceph-mon[36368]: pgmap v57: 97 pgs: 97 active+clean; 9.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 833 KiB/s wr, 116 op/s 2024-03-24T10:41:55.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:54 smithi116 ceph-mon[36248]: pgmap v57: 97 pgs: 97 active+clean; 9.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 833 KiB/s wr, 116 op/s 2024-03-24T10:41:55.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 25483 12.91 MB/sec warmup 12 sec latency 8.715 ms 2024-03-24T10:41:55.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:54 smithi023 ceph-mon[32466]: pgmap v57: 97 pgs: 97 active+clean; 9.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 833 KiB/s wr, 116 op/s 2024-03-24T10:41:56.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 28506 13.27 MB/sec warmup 13 sec latency 23.489 ms 2024-03-24T10:41:57.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:56 smithi117 ceph-mon[36368]: pgmap v58: 97 pgs: 97 active+clean; 9.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 833 KiB/s wr, 116 op/s 2024-03-24T10:41:57.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:56 smithi116 ceph-mon[36248]: pgmap v58: 97 pgs: 97 active+clean; 9.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 833 KiB/s wr, 116 op/s 2024-03-24T10:41:57.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 30825 12.93 MB/sec warmup 14 sec latency 25.142 ms 2024-03-24T10:41:57.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:56 smithi023 ceph-mon[32466]: pgmap v58: 97 pgs: 97 active+clean; 9.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 833 KiB/s wr, 116 op/s 2024-03-24T10:41:58.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 33023 13.00 MB/sec warmup 15 sec latency 17.046 ms 2024-03-24T10:41:59.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:41:58 smithi117 ceph-mon[36368]: pgmap v59: 97 pgs: 97 active+clean; 15 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 1.4 MiB/s wr, 195 op/s 2024-03-24T10:41:59.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:41:58 smithi116 ceph-mon[36248]: pgmap v59: 97 pgs: 97 active+clean; 15 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 1.4 MiB/s wr, 195 op/s 2024-03-24T10:41:59.111 INFO:tasks.workunit.client.0.smithi023.stdout: 1 35848 13.26 MB/sec warmup 16 sec latency 21.526 ms 2024-03-24T10:41:59.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.126+0000 7feb1e49d640 1 -- 172.21.15.23:0/381912120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb1810ded0 msgr2=0x7feb18110340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:59.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.126+0000 7feb1e49d640 1 --2- 172.21.15.23:0/381912120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb1810ded0 0x7feb18110340 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7feb10009ca0 tx=0x7feb1002c180 comp rx=0 tx=0).stop 2024-03-24T10:41:59.126 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.126+0000 7feb1e49d640 1 -- 172.21.15.23:0/381912120 shutdown_connections 2024-03-24T10:41:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.126+0000 7feb1e49d640 1 --2- 172.21.15.23:0/381912120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb1810ded0 0x7feb18110340 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.126+0000 7feb1e49d640 1 --2- 172.21.15.23:0/381912120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb180692d0 0x7feb18069710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.126+0000 7feb1e49d640 1 --2- 172.21.15.23:0/381912120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb18102690 0x7feb18068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.126+0000 7feb1e49d640 1 -- 172.21.15.23:0/381912120 >> 172.21.15.23:0/381912120 conn(0x7feb18077460 msgr2=0x7feb18077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.126+0000 7feb1e49d640 1 -- 172.21.15.23:0/381912120 shutdown_connections 2024-03-24T10:41:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.126+0000 7feb1e49d640 1 -- 172.21.15.23:0/381912120 wait complete. 2024-03-24T10:41:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.126+0000 7feb1e49d640 1 Processor -- start 2024-03-24T10:41:59.127 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.126+0000 7feb1e49d640 1 -- start start 2024-03-24T10:41:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb1e49d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb180692d0 0x7feb18199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb1e49d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb18102690 0x7feb1819a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb1e49d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb1810ded0 0x7feb1819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb1e49d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7feb1819ec80 con 0x7feb18102690 2024-03-24T10:41:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb1e49d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feb1819edf0 con 0x7feb180692d0 2024-03-24T10:41:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb1e49d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feb1819efb0 con 0x7feb1810ded0 2024-03-24T10:41:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb177fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb18102690 0x7feb1819a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb180692d0 0x7feb18199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb177fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb18102690 0x7feb1819a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58378/0 (socket says 172.21.15.23:58378) 2024-03-24T10:41:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb177fe640 1 -- 172.21.15.23:0/1954256163 learned_addr learned my addr 172.21.15.23:0/1954256163 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb1ca3a640 1 --2- 172.21.15.23:0/1954256163 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb1810ded0 0x7feb1819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb177fe640 1 -- 172.21.15.23:0/1954256163 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb1810ded0 msgr2=0x7feb1819e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb177fe640 1 --2- 172.21.15.23:0/1954256163 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb1810ded0 0x7feb1819e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb177fe640 1 -- 172.21.15.23:0/1954256163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb180692d0 msgr2=0x7feb18199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb177fe640 1 --2- 172.21.15.23:0/1954256163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb180692d0 0x7feb18199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb177fe640 1 -- 172.21.15.23:0/1954256163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb10009900 con 0x7feb18102690 2024-03-24T10:41:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb177fe640 1 --2- 172.21.15.23:0/1954256163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb18102690 0x7feb1819a300 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00e9f0 tx=0x7feb0c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb157fa640 1 -- 172.21.15.23:0/1954256163 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c009700 con 0x7feb18102690 2024-03-24T10:41:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb157fa640 1 -- 172.21.15.23:0/1954256163 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0c00cd80 con 0x7feb18102690 2024-03-24T10:41:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb1e49d640 1 -- 172.21.15.23:0/1954256163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1819f290 con 0x7feb18102690 2024-03-24T10:41:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb157fa640 1 -- 172.21.15.23:0/1954256163 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c018920 con 0x7feb18102690 2024-03-24T10:41:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.127+0000 7feb1e49d640 1 -- 172.21.15.23:0/1954256163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb181a6b10 con 0x7feb18102690 2024-03-24T10:41:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb157fa640 1 -- 172.21.15.23:0/1954256163 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7feb0c018a80 con 0x7feb18102690 2024-03-24T10:41:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb21080740 1 Processor -- start 2024-03-24T10:41:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb21080740 1 -- start start 2024-03-24T10:41:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb21080740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0d5e0 0x56381be0d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb21080740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be0df00 0x56381be16790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb21080740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be16cd0 0x56381be190c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb157fa640 1 --2- 172.21.15.23:0/1954256163 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feb000a22d0 0x7feb000a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb21080740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56381bdc81b0 con 0x56381be16cd0 2024-03-24T10:41:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb21080740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56381bd7af80 con 0x56381be0d5e0 2024-03-24T10:41:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb21080740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56381bd784b0 con 0x56381be0df00 2024-03-24T10:41:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb1ca3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0d5e0 0x56381be0d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb177fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be16cd0 0x56381be190c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb1ca3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0d5e0 0x56381be0d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:52326/0 (socket says 172.21.15.23:52326) 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb1ca3a640 1 -- 172.21.15.23:0/697024096 learned_addr learned my addr 172.21.15.23:0/697024096 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.130+0000 7feb17fff640 1 --2- 172.21.15.23:0/697024096 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be0df00 0x56381be16790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.131+0000 7feb1ca3a640 1 --2- 172.21.15.23:0/1954256163 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feb000a22d0 0x7feb000a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.131+0000 7feb157fa640 1 -- 172.21.15.23:0/1954256163 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7feb0c014070 con 0x7feb18102690 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.131+0000 7feb17fff640 1 -- 172.21.15.23:0/697024096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0d5e0 msgr2=0x56381be0d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.131+0000 7feb17fff640 1 --2- 172.21.15.23:0/697024096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0d5e0 0x56381be0d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.131+0000 7feb17fff640 1 -- 172.21.15.23:0/697024096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be16cd0 msgr2=0x56381be190c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.131+0000 7feb17fff640 1 --2- 172.21.15.23:0/697024096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be16cd0 0x56381be190c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.131+0000 7feb17fff640 1 -- 172.21.15.23:0/697024096 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56381bd09810 con 0x56381be0df00 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.131+0000 7feb1ca3a640 1 --2- 172.21.15.23:0/697024096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0d5e0 0x56381be0d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.131+0000 7feb17fff640 1 --2- 172.21.15.23:0/697024096 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be0df00 0x56381be16790 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7feb08009c00 tx=0x7feb08011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f6d44939faf07ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.131+0000 7feb1ca3a640 1 --2- 172.21.15.23:0/1954256163 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feb000a22d0 0x7feb000a4790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feb10032730 tx=0x7feb10005c50 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.131+0000 7feafe7fc640 1 -- 172.21.15.23:0/697024096 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08004230 con 0x56381be0df00 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.131+0000 7feafe7fc640 1 -- 172.21.15.23:0/697024096 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb08009e70 con 0x56381be0df00 2024-03-24T10:41:59.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.131+0000 7feafe7fc640 1 -- 172.21.15.23:0/697024096 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08004d40 con 0x56381be0df00 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.132+0000 7feb21080740 1 -- 172.21.15.23:0/697024096 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be0df00 msgr2=0x56381be16790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.132+0000 7feb21080740 1 --2- 172.21.15.23:0/697024096 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be0df00 0x56381be16790 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7feb08009c00 tx=0x7feb08011810 comp rx=0 tx=0).stop 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.132+0000 7feb21080740 1 -- 172.21.15.23:0/697024096 shutdown_connections 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.132+0000 7feb21080740 1 --2- 172.21.15.23:0/697024096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be16cd0 0x56381be190c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.132+0000 7feb21080740 1 --2- 172.21.15.23:0/697024096 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be0df00 0x56381be16790 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.132+0000 7feb21080740 1 --2- 172.21.15.23:0/697024096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0d5e0 0x56381be0d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.132+0000 7feb21080740 1 -- 172.21.15.23:0/697024096 >> 172.21.15.23:0/697024096 conn(0x56381be07f00 msgr2=0x56381be082e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.132+0000 7feb21080740 1 -- 172.21.15.23:0/697024096 shutdown_connections 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.132+0000 7feb21080740 1 -- 172.21.15.23:0/697024096 wait complete. 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.132+0000 7feb21080740 1 -- start start 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb21080740 1 Processor -- start 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb21080740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0d5e0 0x56381be13cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb21080740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be0df00 0x56381be14230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb21080740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be16cd0 0x56381be14770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb21080740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56381bd73c90 con 0x56381be0d5e0 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb21080740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56381bd18c50 con 0x56381be16cd0 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb21080740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56381ba47250 con 0x56381be0df00 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb21080740 10 client.0 _ll_register_callbacks cb 0x56381bd67290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb17fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0d5e0 0x56381be13cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb17fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0d5e0 0x56381be13cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58398/0 (socket says 172.21.15.23:58398) 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb17fff640 1 -- 172.21.15.23:0/1793865834 learned_addr learned my addr 172.21.15.23:0/1793865834 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb177fe640 1 --2- 172.21.15.23:0/1793865834 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be0df00 0x56381be14230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb1ca3a640 1 --2- 172.21.15.23:0/1793865834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be16cd0 0x56381be14770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb17fff640 1 -- 172.21.15.23:0/1793865834 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be0df00 msgr2=0x56381be14230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb17fff640 1 --2- 172.21.15.23:0/1793865834 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be0df00 0x56381be14230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb17fff640 1 -- 172.21.15.23:0/1793865834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be16cd0 msgr2=0x56381be14770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb17fff640 1 --2- 172.21.15.23:0/1793865834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be16cd0 0x56381be14770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb17fff640 1 -- 172.21.15.23:0/1793865834 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb08009830 con 0x56381be0d5e0 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb17fff640 1 --2- 172.21.15.23:0/1793865834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0d5e0 0x56381be13cf0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7feb08005dd0 tx=0x7feb080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5af2fbeaa1551c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feaf77fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:59.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feaf77fe640 1 -- 172.21.15.23:0/1793865834 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb080039b0 con 0x56381be0d5e0 2024-03-24T10:41:59.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feaf77fe640 1 -- 172.21.15.23:0/1793865834 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7feb08003b10 con 0x56381be0d5e0 2024-03-24T10:41:59.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.133+0000 7feb21080740 1 -- 172.21.15.23:0/1793865834 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56381bd98790 con 0x56381be0d5e0 2024-03-24T10:41:59.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.134+0000 7feaf77fe640 1 -- 172.21.15.23:0/1793865834 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7feb08003cd0 con 0x56381be0d5e0 2024-03-24T10:41:59.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.134+0000 7feb21080740 10 client.16113 fetch_fsmap learned FSMap version 25 2024-03-24T10:41:59.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.134+0000 7feb21080740 1 -- 172.21.15.23:0/1793865834 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56381bd72310 con 0x56381be0d5e0 2024-03-24T10:41:59.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.134+0000 7feaf77fe640 1 -- 172.21.15.23:0/1793865834 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56381bd98790 con 0x56381be0d5e0 2024-03-24T10:41:59.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.134+0000 7feaf77fe640 1 -- 172.21.15.23:0/1793865834 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7feb08019460 con 0x56381be0d5e0 2024-03-24T10:41:59.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.134+0000 7feb21080740 10 client.16113 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:41:59.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.134+0000 7feb21080740 10 client.16113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:59.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.134+0000 7feb21080740 20 client.16113 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:59.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.134+0000 7feb21080740 1 --2- 172.21.15.23:0/1793865834 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56381be26b10 0x56381be28f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.134+0000 7feb21080740 4 client.16113 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:41:59.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.134+0000 7feb21080740 1 -- 172.21.15.23:0/1793865834 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56381ba73600 con 0x56381be26b10 2024-03-24T10:41:59.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.134+0000 7feb177fe640 1 --2- 172.21.15.23:0/1793865834 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56381be26b10 0x56381be28f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.135+0000 7feb177fe640 1 --2- 172.21.15.23:0/1793865834 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56381be26b10 0x56381be28f50 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3427952954fb289 server_cookie=a1baf19069a936b5 in_seq=0 out_seq=0 2024-03-24T10:41:59.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.135+0000 7feaf77fe640 10 client.16113 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:59.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feaf77fe640 1 -- 172.21.15.23:0/1793865834 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56381ba73600 con 0x56381be26b10 2024-03-24T10:41:59.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feaf77fe640 10 client.16113 handle_command_reply: tid=0 2024-03-24T10:41:59.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feb21080740 1 client.16113 shutdown 2024-03-24T10:41:59.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feb21080740 1 -- 172.21.15.23:0/1793865834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0d5e0 msgr2=0x56381be13cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:59.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feb21080740 1 --2- 172.21.15.23:0/1793865834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0d5e0 0x56381be13cf0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7feb08005dd0 tx=0x7feb080046c0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feb21080740 1 -- 172.21.15.23:0/1793865834 shutdown_connections 2024-03-24T10:41:59.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feb21080740 1 --2- 172.21.15.23:0/1793865834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be16cd0 0x56381be14770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feb21080740 1 --2- 172.21.15.23:0/1793865834 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56381be26b10 0x56381be28f50 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feb21080740 1 --2- 172.21.15.23:0/1793865834 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be0df00 0x56381be14230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feb21080740 1 --2- 172.21.15.23:0/1793865834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0d5e0 0x56381be13cf0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feb21080740 1 -- 172.21.15.23:0/1793865834 >> 172.21.15.23:0/1793865834 conn(0x56381be07f00 msgr2=0x56381be09c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:59.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feaf77fe640 0 client.16113 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:59.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feb21080740 1 -- 172.21.15.23:0/1793865834 shutdown_connections 2024-03-24T10:41:59.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feb21080740 1 -- 172.21.15.23:0/1793865834 wait complete. 2024-03-24T10:41:59.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.136+0000 7feb21080740 20 client.16113 trim_cache size 0 max 16384 2024-03-24T10:41:59.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.151+0000 7feb21080740 1 Processor -- start 2024-03-24T10:41:59.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.151+0000 7feb21080740 1 -- start start 2024-03-24T10:41:59.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.151+0000 7feb21080740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0d5e0 0x56381be16420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.151+0000 7feb21080740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0df00 0x56381be223a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.151+0000 7feb21080740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be16cd0 0x56381be2e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.151+0000 7feb21080740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56381bcf9660 con 0x56381be0df00 2024-03-24T10:41:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.151+0000 7feb21080740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56381be23630 con 0x56381be0d5e0 2024-03-24T10:41:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.151+0000 7feb21080740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56381be0d030 con 0x56381be16cd0 2024-03-24T10:41:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.151+0000 7feb177fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0df00 0x56381be223a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.151+0000 7feb177fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0df00 0x56381be223a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58400/0 (socket says 172.21.15.23:58400) 2024-03-24T10:41:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.151+0000 7feb177fe640 1 -- 172.21.15.23:0/1012868214 learned_addr learned my addr 172.21.15.23:0/1012868214 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.151+0000 7feb17fff640 1 --2- 172.21.15.23:0/1012868214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0d5e0 0x56381be16420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.151+0000 7feb1ca3a640 1 --2- 172.21.15.23:0/1012868214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be16cd0 0x56381be2e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.152+0000 7feb177fe640 1 -- 172.21.15.23:0/1012868214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be16cd0 msgr2=0x56381be2e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.152+0000 7feb177fe640 1 --2- 172.21.15.23:0/1012868214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be16cd0 0x56381be2e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.152+0000 7feb177fe640 1 -- 172.21.15.23:0/1012868214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0d5e0 msgr2=0x56381be16420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.152+0000 7feb177fe640 1 --2- 172.21.15.23:0/1012868214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0d5e0 0x56381be16420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.157 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.152+0000 7feb177fe640 1 -- 172.21.15.23:0/1012868214 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56381bd0e810 con 0x56381be0df00 2024-03-24T10:41:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.152+0000 7feb177fe640 1 --2- 172.21.15.23:0/1012868214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0df00 0x56381be223a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0cc190 tx=0x7feb0c06cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e8a7407ad8173b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.152+0000 7feafdffb640 1 -- 172.21.15.23:0/1012868214 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c09ef80 con 0x56381be0df00 2024-03-24T10:41:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.152+0000 7feafdffb640 1 -- 172.21.15.23:0/1012868214 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0c0a35d0 con 0x56381be0df00 2024-03-24T10:41:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.152+0000 7feafdffb640 1 -- 172.21.15.23:0/1012868214 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c0b3b60 con 0x56381be0df00 2024-03-24T10:41:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.153+0000 7feb21080740 1 -- 172.21.15.23:0/1012868214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0df00 msgr2=0x56381be223a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.153+0000 7feb21080740 1 --2- 172.21.15.23:0/1012868214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0df00 0x56381be223a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0cc190 tx=0x7feb0c06cf60 comp rx=0 tx=0).stop 2024-03-24T10:41:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.153+0000 7feb21080740 1 -- 172.21.15.23:0/1012868214 shutdown_connections 2024-03-24T10:41:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.153+0000 7feb21080740 1 --2- 172.21.15.23:0/1012868214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be16cd0 0x56381be2e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.153+0000 7feb21080740 1 --2- 172.21.15.23:0/1012868214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0df00 0x56381be223a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.153+0000 7feb21080740 1 --2- 172.21.15.23:0/1012868214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0d5e0 0x56381be16420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.153+0000 7feb21080740 1 -- 172.21.15.23:0/1012868214 >> 172.21.15.23:0/1012868214 conn(0x56381be07f00 msgr2=0x56381be0e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.153+0000 7feb21080740 1 -- 172.21.15.23:0/1012868214 shutdown_connections 2024-03-24T10:41:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.153+0000 7feb21080740 1 -- 172.21.15.23:0/1012868214 wait complete. 2024-03-24T10:41:59.158 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.153+0000 7feb21080740 1 -- start start 2024-03-24T10:41:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.153+0000 7feb21080740 1 Processor -- start 2024-03-24T10:41:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb21080740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0d5e0 0x56381be1edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb21080740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0df00 0x56381be1a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb21080740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be16cd0 0x56381be1a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb21080740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56381be1af30 con 0x56381be0d5e0 2024-03-24T10:41:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb21080740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56381be8be90 con 0x56381be0df00 2024-03-24T10:41:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb21080740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56381be8bfd0 con 0x56381be16cd0 2024-03-24T10:41:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb21080740 10 client.0 _ll_register_callbacks cb 0x7feb0c00cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:41:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb17fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0d5e0 0x56381be1edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb17fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0d5e0 0x56381be1edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58414/0 (socket says 172.21.15.23:58414) 2024-03-24T10:41:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb17fff640 1 -- 172.21.15.23:0/3060105235 learned_addr learned my addr 172.21.15.23:0/3060105235 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:41:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb177fe640 1 --2- 172.21.15.23:0/3060105235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0df00 0x56381be1a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb1ca3a640 1 --2- 172.21.15.23:0/3060105235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be16cd0 0x56381be1a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb17fff640 1 -- 172.21.15.23:0/3060105235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be16cd0 msgr2=0x56381be1a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb17fff640 1 --2- 172.21.15.23:0/3060105235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be16cd0 0x56381be1a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb17fff640 1 -- 172.21.15.23:0/3060105235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0df00 msgr2=0x56381be1a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:41:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb17fff640 1 --2- 172.21.15.23:0/3060105235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0df00 0x56381be1a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb17fff640 1 -- 172.21.15.23:0/3060105235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb0c06ae70 con 0x56381be0d5e0 2024-03-24T10:41:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb17fff640 1 --2- 172.21.15.23:0/3060105235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0d5e0 0x56381be1edb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7feb08011850 tx=0x7feb080043e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a222ec66bd3721d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:41:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feaf77fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:41:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feaf77fe640 1 -- 172.21.15.23:0/3060105235 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb080195f0 con 0x56381be0d5e0 2024-03-24T10:41:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feaf77fe640 1 -- 172.21.15.23:0/3060105235 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7feb08019cc0 con 0x56381be0d5e0 2024-03-24T10:41:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb21080740 1 -- 172.21.15.23:0/3060105235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb08009e70 con 0x56381be0d5e0 2024-03-24T10:41:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feaf77fe640 1 -- 172.21.15.23:0/3060105235 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7feb080037b0 con 0x56381be0d5e0 2024-03-24T10:41:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.154+0000 7feb21080740 10 client.16119 fetch_fsmap learned FSMap version 25 2024-03-24T10:41:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.155+0000 7feb21080740 1 -- 172.21.15.23:0/3060105235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56381bd345a0 con 0x56381be0d5e0 2024-03-24T10:41:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.155+0000 7feaf77fe640 1 -- 172.21.15.23:0/3060105235 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb08043080 con 0x56381be0d5e0 2024-03-24T10:41:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.155+0000 7feaf77fe640 1 -- 172.21.15.23:0/3060105235 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7feb0803d030 con 0x56381be0d5e0 2024-03-24T10:41:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.155+0000 7feb21080740 10 client.16119 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:41:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.155+0000 7feb21080740 10 client.16119 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:41:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.155+0000 7feb21080740 20 client.16119 populate_metadata read hostname 'smithi023' 2024-03-24T10:41:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.155+0000 7feb21080740 1 --2- 172.21.15.23:0/3060105235 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56381be8d550 0x56381be8f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:41:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.155+0000 7feb21080740 4 client.16119 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:41:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.155+0000 7feb21080740 1 -- 172.21.15.23:0/3060105235 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56381bac41b0 con 0x56381be8d550 2024-03-24T10:41:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.155+0000 7feb177fe640 1 --2- 172.21.15.23:0/3060105235 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56381be8d550 0x56381be8f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:41:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.155+0000 7feb177fe640 1 --2- 172.21.15.23:0/3060105235 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56381be8d550 0x56381be8f970 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20fa72fa313bcf64 server_cookie=9cc67f49e48615bf in_seq=0 out_seq=0 2024-03-24T10:41:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.155+0000 7feaf77fe640 10 client.16119 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feaf77fe640 1 -- 172.21.15.23:0/3060105235 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56381bac41b0 con 0x56381be8d550 2024-03-24T10:41:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feaf77fe640 10 client.16119 handle_command_reply: tid=0 2024-03-24T10:41:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feb21080740 1 client.16119 shutdown 2024-03-24T10:41:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feb21080740 1 -- 172.21.15.23:0/3060105235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0d5e0 msgr2=0x56381be1edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feb21080740 1 --2- 172.21.15.23:0/3060105235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0d5e0 0x56381be1edb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7feb08011850 tx=0x7feb080043e0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feb21080740 1 -- 172.21.15.23:0/3060105235 shutdown_connections 2024-03-24T10:41:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feb21080740 1 --2- 172.21.15.23:0/3060105235 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56381be8d550 0x56381be8f970 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feb21080740 1 --2- 172.21.15.23:0/3060105235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56381be16cd0 0x56381be1a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feaf77fe640 0 client.16119 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:41:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feb21080740 1 --2- 172.21.15.23:0/3060105235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56381be0df00 0x56381be1a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feb21080740 1 --2- 172.21.15.23:0/3060105235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56381be0d5e0 0x56381be1edb0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feb21080740 1 -- 172.21.15.23:0/3060105235 >> 172.21.15.23:0/3060105235 conn(0x56381be07f00 msgr2=0x56381be06610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feb21080740 1 -- 172.21.15.23:0/3060105235 shutdown_connections 2024-03-24T10:41:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feb21080740 1 -- 172.21.15.23:0/3060105235 wait complete. 2024-03-24T10:41:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.164+0000 7feb21080740 20 client.16119 trim_cache size 0 max 16384 2024-03-24T10:41:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:41:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:41:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:41:59.166 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:41:59.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.165+0000 7feb1e49d640 1 -- 172.21.15.23:0/1954256163 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feb000a22d0 msgr2=0x7feb000a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:59.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.165+0000 7feb1e49d640 1 --2- 172.21.15.23:0/1954256163 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feb000a22d0 0x7feb000a4790 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feb10032730 tx=0x7feb10005c50 comp rx=0 tx=0).stop 2024-03-24T10:41:59.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.165+0000 7feb1e49d640 1 -- 172.21.15.23:0/1954256163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb18102690 msgr2=0x7feb1819a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:41:59.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.165+0000 7feb1e49d640 1 --2- 172.21.15.23:0/1954256163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb18102690 0x7feb1819a300 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00e9f0 tx=0x7feb0c00eeb0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.165+0000 7feb1e49d640 1 -- 172.21.15.23:0/1954256163 shutdown_connections 2024-03-24T10:41:59.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.165+0000 7feb1e49d640 1 --2- 172.21.15.23:0/1954256163 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feb000a22d0 0x7feb000a4790 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.165+0000 7feb1e49d640 1 --2- 172.21.15.23:0/1954256163 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb1810ded0 0x7feb1819e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.165+0000 7feb1e49d640 1 --2- 172.21.15.23:0/1954256163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb18102690 0x7feb1819a300 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.165+0000 7feb1e49d640 1 --2- 172.21.15.23:0/1954256163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb180692d0 0x7feb18199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:41:59.167 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.165+0000 7feb1e49d640 1 -- 172.21.15.23:0/1954256163 >> 172.21.15.23:0/1954256163 conn(0x7feb18077460 msgr2=0x7feb180ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:41:59.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.165+0000 7feb1e49d640 1 -- 172.21.15.23:0/1954256163 shutdown_connections 2024-03-24T10:41:59.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:41:59.165+0000 7feb1e49d640 1 -- 172.21.15.23:0/1954256163 wait complete. 2024-03-24T10:41:59.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:41:58 smithi023 ceph-mon[32466]: pgmap v59: 97 pgs: 97 active+clean; 15 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 1.4 MiB/s wr, 195 op/s 2024-03-24T10:42:00.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 38486 13.22 MB/sec warmup 17 sec latency 23.518 ms 2024-03-24T10:42:01.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:00 smithi117 ceph-mon[36368]: pgmap v60: 97 pgs: 97 active+clean; 15 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 952 KiB/s wr, 122 op/s 2024-03-24T10:42:01.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:00 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:42:01.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:00 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:42:01.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:00 smithi116 ceph-mon[36248]: pgmap v60: 97 pgs: 97 active+clean; 15 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 952 KiB/s wr, 122 op/s 2024-03-24T10:42:01.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:00 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:42:01.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:00 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:42:01.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 39790 12.87 MB/sec warmup 18 sec latency 25.434 ms 2024-03-24T10:42:01.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:00 smithi023 ceph-mon[32466]: pgmap v60: 97 pgs: 97 active+clean; 15 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 952 KiB/s wr, 122 op/s 2024-03-24T10:42:01.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:00 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:42:01.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:00 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:42:02.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 42739 13.11 MB/sec warmup 19 sec latency 18.850 ms 2024-03-24T10:42:03.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:02 smithi117 ceph-mon[36368]: pgmap v61: 97 pgs: 97 active+clean; 22 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 1.6 MiB/s wr, 230 op/s 2024-03-24T10:42:03.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:02 smithi116 ceph-mon[36248]: pgmap v61: 97 pgs: 97 active+clean; 22 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 1.6 MiB/s wr, 230 op/s 2024-03-24T10:42:03.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 44998 12.87 MB/sec warmup 20 sec latency 16.600 ms 2024-03-24T10:42:03.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:02 smithi023 ceph-mon[32466]: pgmap v61: 97 pgs: 97 active+clean; 22 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 1.6 MiB/s wr, 230 op/s 2024-03-24T10:42:04.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 47060 12.87 MB/sec warmup 21 sec latency 15.856 ms 2024-03-24T10:42:04.969 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:04 smithi023 ceph-mon[32466]: pgmap v62: 97 pgs: 97 active+clean; 22 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-03-24T10:42:05.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:04 smithi117 ceph-mon[36368]: pgmap v62: 97 pgs: 97 active+clean; 22 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-03-24T10:42:05.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:04 smithi116 ceph-mon[36248]: pgmap v62: 97 pgs: 97 active+clean; 22 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-03-24T10:42:05.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 50051 13.12 MB/sec warmup 22 sec latency 8.060 ms 2024-03-24T10:42:06.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 52609 13.10 MB/sec warmup 23 sec latency 23.589 ms 2024-03-24T10:42:07.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:06 smithi117 ceph-mon[36368]: pgmap v63: 97 pgs: 97 active+clean; 22 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-03-24T10:42:07.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:06 smithi116 ceph-mon[36248]: pgmap v63: 97 pgs: 97 active+clean; 22 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-03-24T10:42:07.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 54845 13.04 MB/sec warmup 24 sec latency 18.024 ms 2024-03-24T10:42:07.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:06 smithi023 ceph-mon[32466]: pgmap v63: 97 pgs: 97 active+clean; 22 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-03-24T10:42:08.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 57392 13.08 MB/sec warmup 25 sec latency 12.662 ms 2024-03-24T10:42:09.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:08 smithi117 ceph-mon[36368]: pgmap v64: 97 pgs: 97 active+clean; 28 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.9 MiB/s wr, 263 op/s 2024-03-24T10:42:09.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:08 smithi116 ceph-mon[36248]: pgmap v64: 97 pgs: 97 active+clean; 28 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.9 MiB/s wr, 263 op/s 2024-03-24T10:42:09.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 60412 13.25 MB/sec warmup 26 sec latency 25.269 ms 2024-03-24T10:42:09.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:08 smithi023 ceph-mon[32466]: pgmap v64: 97 pgs: 97 active+clean; 28 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.9 MiB/s wr, 263 op/s 2024-03-24T10:42:10.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 62778 13.08 MB/sec warmup 27 sec latency 16.124 ms 2024-03-24T10:42:11.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:10 smithi117 ceph-mon[36368]: pgmap v65: 97 pgs: 97 active+clean; 28 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 1.3 MiB/s wr, 184 op/s 2024-03-24T10:42:11.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:10 smithi116 ceph-mon[36248]: pgmap v65: 97 pgs: 97 active+clean; 28 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 1.3 MiB/s wr, 184 op/s 2024-03-24T10:42:11.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 64938 13.11 MB/sec warmup 28 sec latency 22.790 ms 2024-03-24T10:42:11.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:10 smithi023 ceph-mon[32466]: pgmap v65: 97 pgs: 97 active+clean; 28 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 1.3 MiB/s wr, 184 op/s 2024-03-24T10:42:12.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:11 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:42:12.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:11 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:42:12.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 67721 13.24 MB/sec warmup 29 sec latency 12.898 ms 2024-03-24T10:42:12.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:11 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:42:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.256+0000 7f06b3688640 1 -- 172.21.15.23:0/4239116767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ac100860 msgr2=0x7f06ac100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.256+0000 7f06b3688640 1 --2- 172.21.15.23:0/4239116767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ac100860 0x7f06ac100ce0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f069c009900 tx=0x7f069c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:42:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.256+0000 7f06b3688640 1 -- 172.21.15.23:0/4239116767 shutdown_connections 2024-03-24T10:42:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.256+0000 7f06b3688640 1 --2- 172.21.15.23:0/4239116767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06ac101220 0x7f06ac10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.256+0000 7f06b3688640 1 --2- 172.21.15.23:0/4239116767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ac100860 0x7f06ac100ce0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.256+0000 7f06b3688640 1 --2- 172.21.15.23:0/4239116767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06ac1055b0 0x7f06ac105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.256+0000 7f06b3688640 1 -- 172.21.15.23:0/4239116767 >> 172.21.15.23:0/4239116767 conn(0x7f06ac077460 msgr2=0x7f06ac077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.256+0000 7f06b3688640 1 -- 172.21.15.23:0/4239116767 shutdown_connections 2024-03-24T10:42:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.256+0000 7f06b3688640 1 -- 172.21.15.23:0/4239116767 wait complete. 2024-03-24T10:42:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b3688640 1 Processor -- start 2024-03-24T10:42:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b3688640 1 -- start start 2024-03-24T10:42:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b3688640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06ac100860 0x7f06ac19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b3688640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ac101220 0x7f06ac19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b3688640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06ac1055b0 0x7f06ac196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b3688640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f06ac197080 con 0x7f06ac101220 2024-03-24T10:42:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b3688640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f06ac1971f0 con 0x7f06ac1055b0 2024-03-24T10:42:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b3688640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f06ac1973b0 con 0x7f06ac100860 2024-03-24T10:42:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b0c23640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ac101220 0x7f06ac19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b0c23640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ac101220 0x7f06ac19cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48606/0 (socket says 172.21.15.23:48606) 2024-03-24T10:42:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b0c23640 1 -- 172.21.15.23:0/3235313009 learned_addr learned my addr 172.21.15.23:0/3235313009 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b1424640 1 --2- 172.21.15.23:0/3235313009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06ac100860 0x7f06ac19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b1c25640 1 --2- 172.21.15.23:0/3235313009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06ac1055b0 0x7f06ac196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b0c23640 1 -- 172.21.15.23:0/3235313009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06ac100860 msgr2=0x7f06ac19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b0c23640 1 --2- 172.21.15.23:0/3235313009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06ac100860 0x7f06ac19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b0c23640 1 -- 172.21.15.23:0/3235313009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06ac1055b0 msgr2=0x7f06ac196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.257+0000 7f06b0c23640 1 --2- 172.21.15.23:0/3235313009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06ac1055b0 0x7f06ac196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.258+0000 7f06b0c23640 1 -- 172.21.15.23:0/3235313009 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f069c009560 con 0x7f06ac101220 2024-03-24T10:42:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.258+0000 7f06b0c23640 1 --2- 172.21.15.23:0/3235313009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ac101220 0x7f06ac19cd70 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f069c005940 tx=0x7f069c02c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.258+0000 7f06a27fc640 1 -- 172.21.15.23:0/3235313009 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c0043d0 con 0x7f06ac101220 2024-03-24T10:42:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.258+0000 7f06a27fc640 1 -- 172.21.15.23:0/3235313009 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f069c0052d0 con 0x7f06ac101220 2024-03-24T10:42:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.258+0000 7f06b3688640 1 -- 172.21.15.23:0/3235313009 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06ac197630 con 0x7f06ac101220 2024-03-24T10:42:12.261 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.258+0000 7f06a27fc640 1 -- 172.21.15.23:0/3235313009 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c003cc0 con 0x7f06ac101220 2024-03-24T10:42:12.261 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.258+0000 7f06b3688640 1 -- 172.21.15.23:0/3235313009 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06ac1a38d0 con 0x7f06ac101220 2024-03-24T10:42:12.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.260+0000 7f06a27fc640 1 -- 172.21.15.23:0/3235313009 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f069c005440 con 0x7f06ac101220 2024-03-24T10:42:12.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b626e740 1 Processor -- start 2024-03-24T10:42:12.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b626e740 1 -- start start 2024-03-24T10:42:12.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06a27fc640 1 --2- 172.21.15.23:0/3235313009 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06980a2350 0x7f06980a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b626e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11f5e0 0x557bec11f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b626e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11ff00 0x557bec128790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b626e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec128cd0 0x557bec12b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b626e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557bec0da1b0 con 0x557bec11f5e0 2024-03-24T10:42:12.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b626e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557bec08cf80 con 0x557bec11ff00 2024-03-24T10:42:12.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b626e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557bec08a4b0 con 0x557bec128cd0 2024-03-24T10:42:12.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b1424640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11f5e0 0x557bec11f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b1c25640 1 --2- 172.21.15.23:0/3235313009 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06980a2350 0x7f06980a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b1424640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11f5e0 0x557bec11f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48618/0 (socket says 172.21.15.23:48618) 2024-03-24T10:42:12.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b1424640 1 -- 172.21.15.23:0/2293426417 learned_addr learned my addr 172.21.15.23:0/2293426417 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:12.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06a27fc640 1 -- 172.21.15.23:0/3235313009 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f069c0f0020 con 0x7f06ac101220 2024-03-24T10:42:12.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b0c23640 1 --2- 172.21.15.23:0/2293426417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11ff00 0x557bec128790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b1c25640 1 --2- 172.21.15.23:0/2293426417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec128cd0 0x557bec12b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b1424640 1 -- 172.21.15.23:0/2293426417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec128cd0 msgr2=0x557bec12b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:12.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b1424640 1 --2- 172.21.15.23:0/2293426417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec128cd0 0x557bec12b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b1424640 1 -- 172.21.15.23:0/2293426417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11ff00 msgr2=0x557bec128790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:12.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b1424640 1 --2- 172.21.15.23:0/2293426417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11ff00 0x557bec128790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.261+0000 7f06b1424640 1 -- 172.21.15.23:0/2293426417 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bec01b810 con 0x557bec11f5e0 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.262+0000 7f06b1c25640 1 --2- 172.21.15.23:0/3235313009 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06980a2350 0x7f06980a4810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f06a4005fd0 tx=0x7f06a4009df0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.262+0000 7f06b1424640 1 --2- 172.21.15.23:0/2293426417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11f5e0 0x557bec11f9c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f06a800bc90 tx=0x7f06a80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839730e467bb206d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.262+0000 7f06977fe640 1 -- 172.21.15.23:0/2293426417 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8005020 con 0x557bec11f5e0 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.262+0000 7f06977fe640 1 -- 172.21.15.23:0/2293426417 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06a8015070 con 0x557bec11f5e0 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.262+0000 7f06977fe640 1 -- 172.21.15.23:0/2293426417 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8011440 con 0x557bec11f5e0 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.263+0000 7f06b626e740 1 -- 172.21.15.23:0/2293426417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11f5e0 msgr2=0x557bec11f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.263+0000 7f06b626e740 1 --2- 172.21.15.23:0/2293426417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11f5e0 0x557bec11f9c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f06a800bc90 tx=0x7f06a80107a0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.263+0000 7f06b626e740 1 -- 172.21.15.23:0/2293426417 shutdown_connections 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.263+0000 7f06b626e740 1 --2- 172.21.15.23:0/2293426417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec128cd0 0x557bec12b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.263+0000 7f06b626e740 1 --2- 172.21.15.23:0/2293426417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11ff00 0x557bec128790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.263+0000 7f06b626e740 1 --2- 172.21.15.23:0/2293426417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11f5e0 0x557bec11f9c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.263+0000 7f06b626e740 1 -- 172.21.15.23:0/2293426417 >> 172.21.15.23:0/2293426417 conn(0x557bec119f00 msgr2=0x557bec11a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.263+0000 7f06b626e740 1 -- 172.21.15.23:0/2293426417 shutdown_connections 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.263+0000 7f06b626e740 1 -- 172.21.15.23:0/2293426417 wait complete. 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.263+0000 7f06b626e740 1 -- start start 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.263+0000 7f06b626e740 1 Processor -- start 2024-03-24T10:42:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.263+0000 7f06b626e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11f5e0 0x557bec125d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.263+0000 7f06b626e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11ff00 0x557bec126240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b626e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec128cd0 0x557bec126780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b626e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557bec085c90 con 0x557bec11ff00 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b626e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557bec02ac50 con 0x557bec11f5e0 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b626e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557bebd59250 con 0x557bec128cd0 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b626e740 10 client.0 _ll_register_callbacks cb 0x557bec079290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b0c23640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11ff00 0x557bec126240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b0c23640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11ff00 0x557bec126240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48628/0 (socket says 172.21.15.23:48628) 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b0c23640 1 -- 172.21.15.23:0/2511291260 learned_addr learned my addr 172.21.15.23:0/2511291260 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b1c25640 1 --2- 172.21.15.23:0/2511291260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec128cd0 0x557bec126780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b1424640 1 --2- 172.21.15.23:0/2511291260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11f5e0 0x557bec125d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b0c23640 1 -- 172.21.15.23:0/2511291260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec128cd0 msgr2=0x557bec126780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b0c23640 1 --2- 172.21.15.23:0/2511291260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec128cd0 0x557bec126780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b0c23640 1 -- 172.21.15.23:0/2511291260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11f5e0 msgr2=0x557bec125d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b0c23640 1 --2- 172.21.15.23:0/2511291260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11f5e0 0x557bec125d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b0c23640 1 -- 172.21.15.23:0/2511291260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a800b8c0 con 0x557bec11ff00 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b0c23640 1 --2- 172.21.15.23:0/2511291260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11ff00 0x557bec126240 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f069c0b8f80 tx=0x7f069c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342ab8e370f69512 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f0694ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f0694ff9640 1 -- 172.21.15.23:0/2511291260 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c0d5850 con 0x557bec11ff00 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f0694ff9640 1 -- 172.21.15.23:0/2511291260 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f069c0bd220 con 0x557bec11ff00 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b626e740 1 -- 172.21.15.23:0/2511291260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557bec0aa790 con 0x557bec11ff00 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f0694ff9640 1 -- 172.21.15.23:0/2511291260 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f069c003e20 con 0x557bec11ff00 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b626e740 10 client.16131 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.264+0000 7f06b626e740 1 -- 172.21.15.23:0/2511291260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557bec084310 con 0x557bec11ff00 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.265+0000 7f0694ff9640 1 -- 172.21.15.23:0/2511291260 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557bec0aa790 con 0x557bec11ff00 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.265+0000 7f0694ff9640 1 -- 172.21.15.23:0/2511291260 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f069c03c070 con 0x557bec11ff00 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.265+0000 7f06b626e740 10 client.16131 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.265+0000 7f06b626e740 10 client.16131 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.265+0000 7f06b626e740 20 client.16131 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.265+0000 7f06b626e740 1 --2- 172.21.15.23:0/2511291260 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557bec138990 0x557bec13add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.265+0000 7f06b626e740 4 client.16131 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:12.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.265+0000 7f06b626e740 1 -- 172.21.15.23:0/2511291260 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bebd85600 con 0x557bec138990 2024-03-24T10:42:12.271 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.265+0000 7f06b1424640 1 --2- 172.21.15.23:0/2511291260 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557bec138990 0x557bec13add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.271 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.265+0000 7f06b1424640 1 --2- 172.21.15.23:0/2511291260 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557bec138990 0x557bec13add0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2200768f10ce1883 server_cookie=c8a651a49bde255b in_seq=0 out_seq=0 2024-03-24T10:42:12.271 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.265+0000 7f0694ff9640 10 client.16131 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.266+0000 7f0694ff9640 1 -- 172.21.15.23:0/2511291260 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557bebd85600 con 0x557bec138990 2024-03-24T10:42:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.266+0000 7f0694ff9640 10 client.16131 handle_command_reply: tid=0 2024-03-24T10:42:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.266+0000 7f06b626e740 1 client.16131 shutdown 2024-03-24T10:42:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.267+0000 7f06b626e740 1 -- 172.21.15.23:0/2511291260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11ff00 msgr2=0x557bec126240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.267+0000 7f06b626e740 1 --2- 172.21.15.23:0/2511291260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11ff00 0x557bec126240 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f069c0b8f80 tx=0x7f069c005740 comp rx=0 tx=0).stop 2024-03-24T10:42:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.267+0000 7f06b626e740 1 -- 172.21.15.23:0/2511291260 shutdown_connections 2024-03-24T10:42:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.267+0000 7f06b626e740 1 --2- 172.21.15.23:0/2511291260 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557bec138990 0x557bec13add0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.267+0000 7f06b626e740 1 --2- 172.21.15.23:0/2511291260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec128cd0 0x557bec126780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.267+0000 7f06b626e740 1 --2- 172.21.15.23:0/2511291260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11ff00 0x557bec126240 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.267+0000 7f06b626e740 1 --2- 172.21.15.23:0/2511291260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11f5e0 0x557bec125d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.267+0000 7f0694ff9640 0 client.16131 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.267+0000 7f06b626e740 1 -- 172.21.15.23:0/2511291260 >> 172.21.15.23:0/2511291260 conn(0x557bec119f00 msgr2=0x557bec11bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:12.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.267+0000 7f06b626e740 1 -- 172.21.15.23:0/2511291260 shutdown_connections 2024-03-24T10:42:12.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.267+0000 7f06b626e740 1 -- 172.21.15.23:0/2511291260 wait complete. 2024-03-24T10:42:12.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.267+0000 7f06b626e740 20 client.16131 trim_cache size 0 max 16384 2024-03-24T10:42:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b626e740 1 Processor -- start 2024-03-24T10:42:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b626e740 1 -- start start 2024-03-24T10:42:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b626e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec119f00 0x557bec128430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b626e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11f5e0 0x557bec1343a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b626e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec11ff00 0x557bec1265a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b626e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557bec00b660 con 0x557bec119f00 2024-03-24T10:42:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b626e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557bec11b0c0 con 0x557bec11f5e0 2024-03-24T10:42:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b626e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557bec1348e0 con 0x557bec11ff00 2024-03-24T10:42:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b1424640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec119f00 0x557bec128430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b1424640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec119f00 0x557bec128430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48638/0 (socket says 172.21.15.23:48638) 2024-03-24T10:42:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b1424640 1 -- 172.21.15.23:0/2371335264 learned_addr learned my addr 172.21.15.23:0/2371335264 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b0c23640 1 --2- 172.21.15.23:0/2371335264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11f5e0 0x557bec1343a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b1c25640 1 --2- 172.21.15.23:0/2371335264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec11ff00 0x557bec1265a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b1424640 1 -- 172.21.15.23:0/2371335264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec11ff00 msgr2=0x557bec1265a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b1424640 1 --2- 172.21.15.23:0/2371335264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec11ff00 0x557bec1265a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b1424640 1 -- 172.21.15.23:0/2371335264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11f5e0 msgr2=0x557bec1343a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b1424640 1 --2- 172.21.15.23:0/2371335264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11f5e0 0x557bec1343a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.283+0000 7f06b1424640 1 -- 172.21.15.23:0/2371335264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bec020810 con 0x557bec119f00 2024-03-24T10:42:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.284+0000 7f06b1424640 1 --2- 172.21.15.23:0/2371335264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec119f00 0x557bec128430 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f06a800bc60 tx=0x7f06a8004b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8685da0061415987 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.284+0000 7f0696ffd640 1 -- 172.21.15.23:0/2371335264 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8021040 con 0x557bec119f00 2024-03-24T10:42:12.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.284+0000 7f0696ffd640 1 -- 172.21.15.23:0/2371335264 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06a8022040 con 0x557bec119f00 2024-03-24T10:42:12.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.284+0000 7f0696ffd640 1 -- 172.21.15.23:0/2371335264 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8027980 con 0x557bec119f00 2024-03-24T10:42:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.284+0000 7f06b626e740 1 -- 172.21.15.23:0/2371335264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec119f00 msgr2=0x557bec128430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.284+0000 7f06b626e740 1 --2- 172.21.15.23:0/2371335264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec119f00 0x557bec128430 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f06a800bc60 tx=0x7f06a8004b80 comp rx=0 tx=0).stop 2024-03-24T10:42:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 -- 172.21.15.23:0/2371335264 shutdown_connections 2024-03-24T10:42:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 --2- 172.21.15.23:0/2371335264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec11ff00 0x557bec1265a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 --2- 172.21.15.23:0/2371335264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11f5e0 0x557bec1343a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 --2- 172.21.15.23:0/2371335264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec119f00 0x557bec128430 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 -- 172.21.15.23:0/2371335264 >> 172.21.15.23:0/2371335264 conn(0x557bec114ec0 msgr2=0x557bec121050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 -- 172.21.15.23:0/2371335264 shutdown_connections 2024-03-24T10:42:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 -- 172.21.15.23:0/2371335264 wait complete. 2024-03-24T10:42:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 -- start start 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 Processor -- start 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec119f00 0x557bec12c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11f5e0 0x557bec12ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11ff00 0x557bec12cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557bec1a2240 con 0x557bec11f5e0 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557bec1a23b0 con 0x557bec11ff00 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557bec1a2570 con 0x557bec119f00 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b626e740 10 client.0 _ll_register_callbacks cb 0x7f069c009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b0c23640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11f5e0 0x557bec12ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b0c23640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11f5e0 0x557bec12ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48644/0 (socket says 172.21.15.23:48644) 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.285+0000 7f06b0c23640 1 -- 172.21.15.23:0/3454634764 learned_addr learned my addr 172.21.15.23:0/3454634764 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f06b1424640 1 --2- 172.21.15.23:0/3454634764 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec119f00 0x557bec12c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f06b1c25640 1 --2- 172.21.15.23:0/3454634764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11ff00 0x557bec12cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f06b0c23640 1 -- 172.21.15.23:0/3454634764 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec119f00 msgr2=0x557bec12c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f06b0c23640 1 --2- 172.21.15.23:0/3454634764 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec119f00 0x557bec12c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f06b0c23640 1 -- 172.21.15.23:0/3454634764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11ff00 msgr2=0x557bec12cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f06b0c23640 1 --2- 172.21.15.23:0/3454634764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11ff00 0x557bec12cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f06b0c23640 1 -- 172.21.15.23:0/3454634764 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bec01b810 con 0x557bec11f5e0 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f06b1c25640 1 --2- 172.21.15.23:0/3454634764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11ff00 0x557bec12cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f06b0c23640 1 --2- 172.21.15.23:0/3454634764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11f5e0 0x557bec12ca80 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f069c0bcab0 tx=0x7f069c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb8fda56d48d6ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f0694ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f0694ff9640 1 -- 172.21.15.23:0/3454634764 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c0db0b0 con 0x557bec11f5e0 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f0694ff9640 1 -- 172.21.15.23:0/3454634764 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f069c0bd220 con 0x557bec11f5e0 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f06b626e740 1 -- 172.21.15.23:0/3454634764 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06a8015070 con 0x557bec11f5e0 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f0694ff9640 1 -- 172.21.15.23:0/3454634764 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f069c0bcb20 con 0x557bec11f5e0 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f06b626e740 10 client.16137 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f06b626e740 1 -- 172.21.15.23:0/3454634764 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557bec0465a0 con 0x557bec11f5e0 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.286+0000 7f0694ff9640 1 -- 172.21.15.23:0/3454634764 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f069c0a1080 con 0x557bec11f5e0 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.287+0000 7f0694ff9640 1 -- 172.21.15.23:0/3454634764 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f069c03c050 con 0x557bec11f5e0 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.287+0000 7f06b626e740 10 client.16137 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.287+0000 7f06b626e740 10 client.16137 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.287+0000 7f06b626e740 20 client.16137 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.287+0000 7f06b626e740 1 --2- 172.21.15.23:0/3454634764 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557bec128cd0 0x557bec1a5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.287+0000 7f06b626e740 4 client.16137 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:42:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.287+0000 7f06b626e740 1 -- 172.21.15.23:0/3454634764 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557bebdd61b0 con 0x557bec128cd0 2024-03-24T10:42:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.287+0000 7f06b1424640 1 --2- 172.21.15.23:0/3454634764 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557bec128cd0 0x557bec1a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.287+0000 7f06b1424640 1 --2- 172.21.15.23:0/3454634764 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557bec128cd0 0x557bec1a5840 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efc2c999fb7bdc96 server_cookie=1ba044a97f76f37c in_seq=0 out_seq=0 2024-03-24T10:42:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.287+0000 7f0694ff9640 10 client.16137 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.297+0000 7f0694ff9640 1 -- 172.21.15.23:0/3454634764 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557bebdd61b0 con 0x557bec128cd0 2024-03-24T10:42:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.297+0000 7f0694ff9640 10 client.16137 handle_command_reply: tid=0 2024-03-24T10:42:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.297+0000 7f06b626e740 1 client.16137 shutdown 2024-03-24T10:42:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.297+0000 7f06b626e740 1 -- 172.21.15.23:0/3454634764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11f5e0 msgr2=0x557bec12ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.297+0000 7f06b626e740 1 --2- 172.21.15.23:0/3454634764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11f5e0 0x557bec12ca80 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f069c0bcab0 tx=0x7f069c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:42:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.297+0000 7f06b626e740 1 -- 172.21.15.23:0/3454634764 shutdown_connections 2024-03-24T10:42:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.297+0000 7f06b626e740 1 --2- 172.21.15.23:0/3454634764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557bec11ff00 0x557bec12cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.297+0000 7f06b626e740 1 --2- 172.21.15.23:0/3454634764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bec11f5e0 0x557bec12ca80 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.297+0000 7f06b626e740 1 --2- 172.21.15.23:0/3454634764 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557bec128cd0 0x557bec1a5840 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.297+0000 7f06b626e740 1 --2- 172.21.15.23:0/3454634764 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557bec119f00 0x557bec12c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.297+0000 7f0694ff9640 0 client.16137 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.297+0000 7f06b626e740 1 -- 172.21.15.23:0/3454634764 >> 172.21.15.23:0/3454634764 conn(0x557bec114ec0 msgr2=0x557bec118610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.297+0000 7f06b626e740 1 -- 172.21.15.23:0/3454634764 shutdown_connections 2024-03-24T10:42:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.298+0000 7f06b626e740 1 -- 172.21.15.23:0/3454634764 wait complete. 2024-03-24T10:42:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.298+0000 7f06b626e740 20 client.16137 trim_cache size 0 max 16384 2024-03-24T10:42:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:42:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:42:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:42:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:42:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.298+0000 7f06b3688640 1 -- 172.21.15.23:0/3235313009 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06980a2350 msgr2=0x7f06980a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.298+0000 7f06b3688640 1 --2- 172.21.15.23:0/3235313009 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06980a2350 0x7f06980a4810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f06a4005fd0 tx=0x7f06a4009df0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.298+0000 7f06b3688640 1 -- 172.21.15.23:0/3235313009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ac101220 msgr2=0x7f06ac19cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.298+0000 7f06b3688640 1 --2- 172.21.15.23:0/3235313009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ac101220 0x7f06ac19cd70 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f069c005940 tx=0x7f069c02c700 comp rx=0 tx=0).stop 2024-03-24T10:42:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.298+0000 7f06b3688640 1 -- 172.21.15.23:0/3235313009 shutdown_connections 2024-03-24T10:42:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.298+0000 7f06b3688640 1 --2- 172.21.15.23:0/3235313009 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06980a2350 0x7f06980a4810 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.298+0000 7f06b3688640 1 --2- 172.21.15.23:0/3235313009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06ac1055b0 0x7f06ac196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.298+0000 7f06b3688640 1 --2- 172.21.15.23:0/3235313009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ac101220 0x7f06ac19cd70 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.298+0000 7f06b3688640 1 --2- 172.21.15.23:0/3235313009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06ac100860 0x7f06ac19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.298+0000 7f06b3688640 1 -- 172.21.15.23:0/3235313009 >> 172.21.15.23:0/3235313009 conn(0x7f06ac077460 msgr2=0x7f06ac0fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.298+0000 7f06b3688640 1 -- 172.21.15.23:0/3235313009 shutdown_connections 2024-03-24T10:42:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:12.298+0000 7f06b3688640 1 -- 172.21.15.23:0/3235313009 wait complete. 2024-03-24T10:42:13.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 69857 13.05 MB/sec warmup 30 sec latency 19.506 ms 2024-03-24T10:42:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:12 smithi023 ceph-mon[32466]: pgmap v66: 97 pgs: 97 active+clean; 35 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 263 op/s 2024-03-24T10:42:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:12 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:12 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:12 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:12 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:12 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:42:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:12 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:42:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:12 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:13.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:12 smithi117 ceph-mon[36368]: pgmap v66: 97 pgs: 97 active+clean; 35 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 263 op/s 2024-03-24T10:42:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:12 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:12 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:12 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:12 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:12 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:42:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:12 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:42:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:12 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:13.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:12 smithi116 ceph-mon[36248]: pgmap v66: 97 pgs: 97 active+clean; 35 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 263 op/s 2024-03-24T10:42:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:12 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:12 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:12 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:12 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:12 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:42:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:12 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:42:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:12 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:42:14.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 71752 13.03 MB/sec warmup 31 sec latency 45.302 ms 2024-03-24T10:42:15.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 74680 13.16 MB/sec warmup 32 sec latency 43.806 ms 2024-03-24T10:42:15.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:14 smithi023 ceph-mon[32466]: pgmap v67: 97 pgs: 97 active+clean; 35 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-03-24T10:42:15.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:14 smithi117 ceph-mon[36368]: pgmap v67: 97 pgs: 97 active+clean; 35 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-03-24T10:42:15.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:14 smithi116 ceph-mon[36248]: pgmap v67: 97 pgs: 97 active+clean; 35 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-03-24T10:42:16.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 76684 12.98 MB/sec warmup 33 sec latency 13.308 ms 2024-03-24T10:42:17.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 78887 13.01 MB/sec warmup 34 sec latency 15.300 ms 2024-03-24T10:42:17.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:16 smithi023 ceph-mon[32466]: pgmap v68: 97 pgs: 97 active+clean; 35 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-03-24T10:42:17.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:16 smithi117 ceph-mon[36368]: pgmap v68: 97 pgs: 97 active+clean; 35 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-03-24T10:42:17.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:16 smithi116 ceph-mon[36248]: pgmap v68: 97 pgs: 97 active+clean; 35 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-03-24T10:42:18.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 81866 13.16 MB/sec warmup 35 sec latency 9.443 ms 2024-03-24T10:42:19.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 84013 13.00 MB/sec warmup 36 sec latency 20.952 ms 2024-03-24T10:42:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:18 smithi023 ceph-mon[32466]: pgmap v69: 97 pgs: 97 active+clean; 62 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 244 op/s 2024-03-24T10:42:19.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:18 smithi117 ceph-mon[36368]: pgmap v69: 97 pgs: 97 active+clean; 62 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 244 op/s 2024-03-24T10:42:19.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:18 smithi116 ceph-mon[36248]: pgmap v69: 97 pgs: 97 active+clean; 62 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 244 op/s 2024-03-24T10:42:20.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 85796 12.97 MB/sec warmup 37 sec latency 37.975 ms 2024-03-24T10:42:21.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 88565 13.01 MB/sec warmup 38 sec latency 33.174 ms 2024-03-24T10:42:21.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:20 smithi023 ceph-mon[32466]: pgmap v70: 97 pgs: 97 active+clean; 62 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 3.3 MiB/s wr, 168 op/s 2024-03-24T10:42:21.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:20 smithi117 ceph-mon[36368]: pgmap v70: 97 pgs: 97 active+clean; 62 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 3.3 MiB/s wr, 168 op/s 2024-03-24T10:42:21.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:20 smithi116 ceph-mon[36248]: pgmap v70: 97 pgs: 97 active+clean; 62 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 3.3 MiB/s wr, 168 op/s 2024-03-24T10:42:21.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:42:21.896 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:42:22.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.009+0000 7fef9158a640 1 -- 172.21.15.23:0/1546626468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef8c100f80 msgr2=0x7fef8c10d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.009+0000 7fef9158a640 1 --2- 172.21.15.23:0/1546626468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef8c100f80 0x7fef8c10d490 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fef84009cd0 tx=0x7fef8402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.009+0000 7fef9158a640 1 -- 172.21.15.23:0/1546626468 shutdown_connections 2024-03-24T10:42:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.009+0000 7fef9158a640 1 --2- 172.21.15.23:0/1546626468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef8c100f80 0x7fef8c10d490 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.009+0000 7fef9158a640 1 --2- 172.21.15.23:0/1546626468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef8c1005c0 0x7fef8c100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.009+0000 7fef9158a640 1 --2- 172.21.15.23:0/1546626468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef8c106660 0x7fef8c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.009+0000 7fef9158a640 1 -- 172.21.15.23:0/1546626468 >> 172.21.15.23:0/1546626468 conn(0x7fef8c076990 msgr2=0x7fef8c076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.010+0000 7fef9158a640 1 -- 172.21.15.23:0/1546626468 shutdown_connections 2024-03-24T10:42:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.010+0000 7fef9158a640 1 -- 172.21.15.23:0/1546626468 wait complete. 2024-03-24T10:42:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.010+0000 7fef9158a640 1 Processor -- start 2024-03-24T10:42:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.010+0000 7fef9158a640 1 -- start start 2024-03-24T10:42:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.010+0000 7fef9158a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef8c1005c0 0x7fef8c198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.010+0000 7fef9158a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef8c100f80 0x7fef8c198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.010+0000 7fef9158a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef8c106660 0x7fef8c1a3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.010+0000 7fef9158a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c199850 con 0x7fef8c1005c0 2024-03-24T10:42:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.010+0000 7fef9158a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c1a4070 con 0x7fef8c106660 2024-03-24T10:42:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.010+0000 7fef9158a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c1a41b0 con 0x7fef8c100f80 2024-03-24T10:42:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.010+0000 7fef8affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef8c1005c0 0x7fef8c198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.010+0000 7fef8affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef8c1005c0 0x7fef8c198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55096/0 (socket says 172.21.15.23:55096) 2024-03-24T10:42:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.010+0000 7fef8affd640 1 -- 172.21.15.23:0/4011624760 learned_addr learned my addr 172.21.15.23:0/4011624760 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.011+0000 7fef83fff640 1 --2- 172.21.15.23:0/4011624760 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef8c100f80 0x7fef8c198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.011+0000 7fef8b7fe640 1 --2- 172.21.15.23:0/4011624760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef8c106660 0x7fef8c1a3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.011+0000 7fef8affd640 1 -- 172.21.15.23:0/4011624760 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef8c100f80 msgr2=0x7fef8c198ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.011+0000 7fef8affd640 1 --2- 172.21.15.23:0/4011624760 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef8c100f80 0x7fef8c198ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.011+0000 7fef8affd640 1 -- 172.21.15.23:0/4011624760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef8c106660 msgr2=0x7fef8c1a3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.011+0000 7fef8affd640 1 --2- 172.21.15.23:0/4011624760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef8c106660 0x7fef8c1a3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.011+0000 7fef8affd640 1 -- 172.21.15.23:0/4011624760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef84009900 con 0x7fef8c1005c0 2024-03-24T10:42:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.011+0000 7fef8affd640 1 --2- 172.21.15.23:0/4011624760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef8c1005c0 0x7fef8c198ab0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00e9c0 tx=0x7fef7c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.011+0000 7fef88ff9640 1 -- 172.21.15.23:0/4011624760 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c009700 con 0x7fef8c1005c0 2024-03-24T10:42:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.011+0000 7fef88ff9640 1 -- 172.21.15.23:0/4011624760 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef7c00cd80 con 0x7fef8c1005c0 2024-03-24T10:42:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.011+0000 7fef9158a640 1 -- 172.21.15.23:0/4011624760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef8c1a4410 con 0x7fef8c1005c0 2024-03-24T10:42:22.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.011+0000 7fef88ff9640 1 -- 172.21.15.23:0/4011624760 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c010430 con 0x7fef8c1005c0 2024-03-24T10:42:22.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.011+0000 7fef9158a640 1 -- 172.21.15.23:0/4011624760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef8c1a4970 con 0x7fef8c1005c0 2024-03-24T10:42:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.014+0000 7fef88ff9640 1 -- 172.21.15.23:0/4011624760 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fef7c003680 con 0x7fef8c1005c0 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.014+0000 7fef94168740 1 Processor -- start 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.014+0000 7fef94168740 1 -- start start 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.014+0000 7fef94168740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c31d0 0x5616eb8c35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.014+0000 7fef94168740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8c3af0 0x5616eb8cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.014+0000 7fef94168740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8cc6d0 0x5616eb8ceac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.014+0000 7fef94168740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5616eb7e7f30 con 0x5616eb8c31d0 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.014+0000 7fef94168740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5616eb831810 con 0x5616eb8cc6d0 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.014+0000 7fef94168740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5616eb82ed40 con 0x5616eb8c3af0 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.014+0000 7fef88ff9640 1 --2- 172.21.15.23:0/4011624760 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fef780a22b0 0x7fef780a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.014+0000 7fef8affd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8c3af0 0x5616eb8cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.014+0000 7fef83fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8cc6d0 0x5616eb8ceac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.014+0000 7fef8b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c31d0 0x5616eb8c35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.014+0000 7fef83fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8cc6d0 0x5616eb8ceac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:45414/0 (socket says 172.21.15.23:45414) 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.015+0000 7fef83fff640 1 -- 172.21.15.23:0/2696221799 learned_addr learned my addr 172.21.15.23:0/2696221799 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.015+0000 7fef8b7fe640 1 --2- 172.21.15.23:0/4011624760 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fef780a22b0 0x7fef780a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.015+0000 7fef88ff9640 1 -- 172.21.15.23:0/4011624760 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fef7c014070 con 0x7fef8c1005c0 2024-03-24T10:42:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.015+0000 7fef83fff640 1 -- 172.21.15.23:0/2696221799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8c3af0 msgr2=0x5616eb8cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.015+0000 7fef83fff640 1 --2- 172.21.15.23:0/2696221799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8c3af0 0x5616eb8cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.015+0000 7fef83fff640 1 -- 172.21.15.23:0/2696221799 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c31d0 msgr2=0x5616eb8c35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.015+0000 7fef83fff640 1 --2- 172.21.15.23:0/2696221799 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c31d0 0x5616eb8c35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.015+0000 7fef83fff640 1 -- 172.21.15.23:0/2696221799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616eb7ebea0 con 0x5616eb8cc6d0 2024-03-24T10:42:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.015+0000 7fef8b7fe640 1 --2- 172.21.15.23:0/4011624760 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fef780a22b0 0x7fef780a4770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fef84005cc0 tx=0x7fef84005ef0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.015+0000 7fef83fff640 1 --2- 172.21.15.23:0/2696221799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8cc6d0 0x5616eb8ceac0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fef74009c90 tx=0x7fef74011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad0c59a8d776248c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.015+0000 7fef817fa640 1 -- 172.21.15.23:0/2696221799 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef74019630 con 0x5616eb8cc6d0 2024-03-24T10:42:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.015+0000 7fef817fa640 1 -- 172.21.15.23:0/2696221799 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef74019c70 con 0x5616eb8cc6d0 2024-03-24T10:42:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.015+0000 7fef817fa640 1 -- 172.21.15.23:0/2696221799 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7400eac0 con 0x5616eb8cc6d0 2024-03-24T10:42:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.016+0000 7fef94168740 1 -- 172.21.15.23:0/2696221799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8cc6d0 msgr2=0x5616eb8ceac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.016+0000 7fef94168740 1 --2- 172.21.15.23:0/2696221799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8cc6d0 0x5616eb8ceac0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fef74009c90 tx=0x7fef74011810 comp rx=0 tx=0).stop 2024-03-24T10:42:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.016+0000 7fef94168740 1 -- 172.21.15.23:0/2696221799 shutdown_connections 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.016+0000 7fef94168740 1 --2- 172.21.15.23:0/2696221799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8cc6d0 0x5616eb8ceac0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.016+0000 7fef94168740 1 --2- 172.21.15.23:0/2696221799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8c3af0 0x5616eb8cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.016+0000 7fef94168740 1 --2- 172.21.15.23:0/2696221799 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c31d0 0x5616eb8c35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.016+0000 7fef94168740 1 -- 172.21.15.23:0/2696221799 >> 172.21.15.23:0/2696221799 conn(0x5616eb8bd900 msgr2=0x5616eb8bdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.016+0000 7fef94168740 1 -- 172.21.15.23:0/2696221799 shutdown_connections 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.016+0000 7fef94168740 1 -- 172.21.15.23:0/2696221799 wait complete. 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.016+0000 7fef94168740 1 -- start start 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.017+0000 7fef94168740 1 Processor -- start 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.017+0000 7fef94168740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c31d0 0x5616eb8c9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.017+0000 7fef94168740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c3af0 0x5616eb8c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.017+0000 7fef94168740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8cc6d0 0x5616eb8ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.017+0000 7fef94168740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5616eb82a520 con 0x5616eb8c3af0 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.017+0000 7fef94168740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5616eb4fd430 con 0x5616eb8c31d0 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.017+0000 7fef94168740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5616eb4b9e00 con 0x5616eb8cc6d0 2024-03-24T10:42:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.017+0000 7fef94168740 10 client.0 _ll_register_callbacks cb 0x5616eb7c8d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.017+0000 7fef83fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c3af0 0x5616eb8c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.017+0000 7fef8b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8cc6d0 0x5616eb8ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.017+0000 7fef8b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8cc6d0 0x5616eb8ca1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:45278/0 (socket says 172.21.15.23:45278) 2024-03-24T10:42:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.017+0000 7fef8b7fe640 1 -- 172.21.15.23:0/3962016443 learned_addr learned my addr 172.21.15.23:0/3962016443 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.017+0000 7fef83fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c3af0 0x5616eb8c9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55108/0 (socket says 172.21.15.23:55108) 2024-03-24T10:42:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.017+0000 7fef8affd640 1 --2- 172.21.15.23:0/3962016443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c31d0 0x5616eb8c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.018+0000 7fef8b7fe640 1 -- 172.21.15.23:0/3962016443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c31d0 msgr2=0x5616eb8c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.018+0000 7fef8b7fe640 1 --2- 172.21.15.23:0/3962016443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c31d0 0x5616eb8c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.018+0000 7fef8b7fe640 1 -- 172.21.15.23:0/3962016443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c3af0 msgr2=0x5616eb8c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.018+0000 7fef8b7fe640 1 --2- 172.21.15.23:0/3962016443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c3af0 0x5616eb8c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.018+0000 7fef8b7fe640 1 -- 172.21.15.23:0/3962016443 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef740098f0 con 0x5616eb8cc6d0 2024-03-24T10:42:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.018+0000 7fef8b7fe640 1 --2- 172.21.15.23:0/3962016443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8cc6d0 0x5616eb8ca1c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fef84036000 tx=0x7fef84009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f7866e65576661f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.018+0000 7fef727fc640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:42:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.018+0000 7fef727fc640 1 -- 172.21.15.23:0/3962016443 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef84048070 con 0x5616eb8cc6d0 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.018+0000 7fef94168740 1 -- 172.21.15.23:0/3962016443 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5616eb5297e0 con 0x5616eb8cc6d0 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.018+0000 7fef727fc640 1 -- 172.21.15.23:0/3962016443 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fef84032a40 con 0x5616eb8cc6d0 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.018+0000 7fef727fc640 1 -- 172.21.15.23:0/3962016443 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fef84003970 con 0x5616eb8cc6d0 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.018+0000 7fef94168740 10 client.25999 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.018+0000 7fef94168740 1 -- 172.21.15.23:0/3962016443 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5616eb828ba0 con 0x5616eb8cc6d0 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.018+0000 7fef727fc640 1 -- 172.21.15.23:0/3962016443 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5616eb5297e0 con 0x5616eb8cc6d0 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.019+0000 7fef727fc640 1 -- 172.21.15.23:0/3962016443 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fef8404b070 con 0x5616eb8cc6d0 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.019+0000 7fef94168740 10 client.25999 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.019+0000 7fef94168740 10 client.25999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.019+0000 7fef94168740 20 client.25999 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.019+0000 7fef94168740 1 --2- 172.21.15.23:0/3962016443 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616eb8e1020 0x5616eb8e3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.019+0000 7fef94168740 4 client.25999 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.019+0000 7fef94168740 1 -- 172.21.15.23:0/3962016443 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616eb566720 con 0x5616eb8e1020 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.019+0000 7fef8affd640 1 --2- 172.21.15.23:0/3962016443 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616eb8e1020 0x5616eb8e3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.019+0000 7fef8affd640 1 --2- 172.21.15.23:0/3962016443 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616eb8e1020 0x5616eb8e3460 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=152cfee7a5727cca server_cookie=83624405c338cc66 in_seq=0 out_seq=0 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.019+0000 7fef727fc640 10 client.25999 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.020+0000 7fef727fc640 1 -- 172.21.15.23:0/3962016443 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5616eb566720 con 0x5616eb8e1020 2024-03-24T10:42:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.020+0000 7fef727fc640 10 client.25999 handle_command_reply: tid=0 2024-03-24T10:42:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.020+0000 7fef94168740 1 client.25999 shutdown 2024-03-24T10:42:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.020+0000 7fef94168740 1 -- 172.21.15.23:0/3962016443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8cc6d0 msgr2=0x5616eb8ca1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.020+0000 7fef94168740 1 --2- 172.21.15.23:0/3962016443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8cc6d0 0x5616eb8ca1c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fef84036000 tx=0x7fef84009360 comp rx=0 tx=0).stop 2024-03-24T10:42:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.020+0000 7fef94168740 1 -- 172.21.15.23:0/3962016443 shutdown_connections 2024-03-24T10:42:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.020+0000 7fef94168740 1 --2- 172.21.15.23:0/3962016443 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616eb8e1020 0x5616eb8e3460 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.020+0000 7fef94168740 1 --2- 172.21.15.23:0/3962016443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8cc6d0 0x5616eb8ca1c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.020+0000 7fef727fc640 0 client.25999 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.020+0000 7fef94168740 1 --2- 172.21.15.23:0/3962016443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c3af0 0x5616eb8c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.020+0000 7fef94168740 1 --2- 172.21.15.23:0/3962016443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c31d0 0x5616eb8c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.020+0000 7fef94168740 1 -- 172.21.15.23:0/3962016443 >> 172.21.15.23:0/3962016443 conn(0x5616eb8bd900 msgr2=0x5616eb8bf650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.020+0000 7fef94168740 1 -- 172.21.15.23:0/3962016443 shutdown_connections 2024-03-24T10:42:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.020+0000 7fef94168740 1 -- 172.21.15.23:0/3962016443 wait complete. 2024-03-24T10:42:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.021+0000 7fef94168740 20 client.25999 trim_cache size 0 max 16384 2024-03-24T10:42:22.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.039+0000 7fef94168740 1 Processor -- start 2024-03-24T10:42:22.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.039+0000 7fef94168740 1 -- start start 2024-03-24T10:42:22.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.039+0000 7fef94168740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8c31d0 0x5616eb8c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.039+0000 7fef94168740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c3af0 0x5616eb8c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.039+0000 7fef94168740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8cc6d0 0x5616eb8d7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.039+0000 7fef94168740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5616eb7af430 con 0x5616eb8cc6d0 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.039+0000 7fef94168740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5616eb566720 con 0x5616eb8c3af0 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.039+0000 7fef94168740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5616eb5297e0 con 0x5616eb8c31d0 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.039+0000 7fef8b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8cc6d0 0x5616eb8d7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.039+0000 7fef8b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8cc6d0 0x5616eb8d7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55122/0 (socket says 172.21.15.23:55122) 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.039+0000 7fef8b7fe640 1 -- 172.21.15.23:0/1477938045 learned_addr learned my addr 172.21.15.23:0/1477938045 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.039+0000 7fef8affd640 1 --2- 172.21.15.23:0/1477938045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8c31d0 0x5616eb8c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.039+0000 7fef83fff640 1 --2- 172.21.15.23:0/1477938045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c3af0 0x5616eb8c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.040+0000 7fef8b7fe640 1 -- 172.21.15.23:0/1477938045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8c31d0 msgr2=0x5616eb8c86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.040+0000 7fef8b7fe640 1 --2- 172.21.15.23:0/1477938045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8c31d0 0x5616eb8c86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.040+0000 7fef8b7fe640 1 -- 172.21.15.23:0/1477938045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c3af0 msgr2=0x5616eb8c4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.040+0000 7fef8b7fe640 1 --2- 172.21.15.23:0/1477938045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c3af0 0x5616eb8c4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.040+0000 7fef8b7fe640 1 -- 172.21.15.23:0/1477938045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616eb7c4b20 con 0x5616eb8cc6d0 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.040+0000 7fef8affd640 1 --2- 172.21.15.23:0/1477938045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8c31d0 0x5616eb8c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:22.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.040+0000 7fef8b7fe640 1 --2- 172.21.15.23:0/1477938045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8cc6d0 0x5616eb8d7f60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fef84037040 tx=0x7fef84009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dff55245462b7a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.040+0000 7fef80ff9640 1 -- 172.21.15.23:0/1477938045 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef84048070 con 0x5616eb8cc6d0 2024-03-24T10:42:22.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.040+0000 7fef80ff9640 1 -- 172.21.15.23:0/1477938045 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef84003e50 con 0x5616eb8cc6d0 2024-03-24T10:42:22.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.040+0000 7fef80ff9640 1 -- 172.21.15.23:0/1477938045 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef840042a0 con 0x5616eb8cc6d0 2024-03-24T10:42:22.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.041+0000 7fef94168740 1 -- 172.21.15.23:0/1477938045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8cc6d0 msgr2=0x5616eb8d7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.041+0000 7fef94168740 1 --2- 172.21.15.23:0/1477938045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8cc6d0 0x5616eb8d7f60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fef84037040 tx=0x7fef84009b20 comp rx=0 tx=0).stop 2024-03-24T10:42:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.041+0000 7fef94168740 1 -- 172.21.15.23:0/1477938045 shutdown_connections 2024-03-24T10:42:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.041+0000 7fef94168740 1 --2- 172.21.15.23:0/1477938045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8cc6d0 0x5616eb8d7f60 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.041+0000 7fef94168740 1 --2- 172.21.15.23:0/1477938045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c3af0 0x5616eb8c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.041+0000 7fef94168740 1 --2- 172.21.15.23:0/1477938045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8c31d0 0x5616eb8c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.041+0000 7fef94168740 1 -- 172.21.15.23:0/1477938045 >> 172.21.15.23:0/1477938045 conn(0x5616eb8bd900 msgr2=0x5616eb8bcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.041+0000 7fef94168740 1 -- 172.21.15.23:0/1477938045 shutdown_connections 2024-03-24T10:42:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.041+0000 7fef94168740 1 -- 172.21.15.23:0/1477938045 wait complete. 2024-03-24T10:42:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.041+0000 7fef94168740 1 -- start start 2024-03-24T10:42:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.041+0000 7fef94168740 1 Processor -- start 2024-03-24T10:42:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef94168740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c31d0 0x5616eb8c6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef94168740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c3af0 0x5616eb8c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef94168740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8cc6d0 0x5616eb9463a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef94168740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5616eb8c7570 con 0x5616eb8c31d0 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef94168740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5616eb9468e0 con 0x5616eb8c3af0 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef94168740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5616eb946a20 con 0x5616eb8cc6d0 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef94168740 10 client.0 _ll_register_callbacks cb 0x7fef7c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef8affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c31d0 0x5616eb8c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef8affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c31d0 0x5616eb8c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55128/0 (socket says 172.21.15.23:55128) 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef8affd640 1 -- 172.21.15.23:0/1649750625 learned_addr learned my addr 172.21.15.23:0/1649750625 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef83fff640 1 --2- 172.21.15.23:0/1649750625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c3af0 0x5616eb8c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef8b7fe640 1 --2- 172.21.15.23:0/1649750625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8cc6d0 0x5616eb9463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef8affd640 1 -- 172.21.15.23:0/1649750625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8cc6d0 msgr2=0x5616eb9463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef8affd640 1 --2- 172.21.15.23:0/1649750625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8cc6d0 0x5616eb9463a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef8affd640 1 -- 172.21.15.23:0/1649750625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c3af0 msgr2=0x5616eb8c6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef8affd640 1 --2- 172.21.15.23:0/1649750625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c3af0 0x5616eb8c6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef8affd640 1 -- 172.21.15.23:0/1649750625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef8c1a4970 con 0x5616eb8c31d0 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef83fff640 1 --2- 172.21.15.23:0/1649750625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c3af0 0x5616eb8c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:42:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef8affd640 1 --2- 172.21.15.23:0/1649750625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c31d0 0x5616eb8c6980 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0971f0 tx=0x7fef7c0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa685fb34b285761 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef727fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef727fc640 1 -- 172.21.15.23:0/1649750625 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c09e040 con 0x5616eb8c31d0 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef727fc640 1 -- 172.21.15.23:0/1649750625 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fef7c0a3530 con 0x5616eb8c31d0 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef94168740 1 -- 172.21.15.23:0/1649750625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5616eb8c5150 con 0x5616eb8c31d0 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef727fc640 1 -- 172.21.15.23:0/1649750625 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fef7c097400 con 0x5616eb8c31d0 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef94168740 10 client.16155 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.042+0000 7fef94168740 1 -- 172.21.15.23:0/1649750625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5616eb828ba0 con 0x5616eb8c31d0 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.043+0000 7fef727fc640 1 -- 172.21.15.23:0/1649750625 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5616eb8c5150 con 0x5616eb8c31d0 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.043+0000 7fef727fc640 1 -- 172.21.15.23:0/1649750625 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fef7c0ce040 con 0x5616eb8c31d0 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.043+0000 7fef94168740 10 client.16155 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.043+0000 7fef94168740 10 client.16155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.043+0000 7fef94168740 20 client.16155 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.043+0000 7fef94168740 1 --2- 172.21.15.23:0/1649750625 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616eb948180 0x5616eb94a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.043+0000 7fef94168740 4 client.16155 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.043+0000 7fef94168740 1 -- 172.21.15.23:0/1649750625 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fef74019c70 con 0x5616eb948180 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.043+0000 7fef83fff640 1 --2- 172.21.15.23:0/1649750625 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616eb948180 0x5616eb94a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.043+0000 7fef83fff640 1 --2- 172.21.15.23:0/1649750625 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616eb948180 0x5616eb94a5e0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1764cd9758898a8 server_cookie=264137afee59d36a in_seq=0 out_seq=0 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.043+0000 7fef727fc640 10 client.16155 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.043+0000 7fef727fc640 1 -- 172.21.15.23:0/1649750625 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fef74019c70 con 0x5616eb948180 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.043+0000 7fef727fc640 10 client.16155 handle_command_reply: tid=0 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef94168740 1 client.16155 shutdown 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef94168740 1 -- 172.21.15.23:0/1649750625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c31d0 msgr2=0x5616eb8c6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef94168740 1 --2- 172.21.15.23:0/1649750625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c31d0 0x5616eb8c6980 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0971f0 tx=0x7fef7c0035f0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef94168740 1 -- 172.21.15.23:0/1649750625 shutdown_connections 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef94168740 1 --2- 172.21.15.23:0/1649750625 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5616eb948180 0x5616eb94a5e0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef94168740 1 --2- 172.21.15.23:0/1649750625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5616eb8cc6d0 0x5616eb9463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef94168740 1 --2- 172.21.15.23:0/1649750625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616eb8c3af0 0x5616eb8c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef727fc640 0 client.16155 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef94168740 1 --2- 172.21.15.23:0/1649750625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616eb8c31d0 0x5616eb8c6980 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef94168740 1 -- 172.21.15.23:0/1649750625 >> 172.21.15.23:0/1649750625 conn(0x5616eb8bd900 msgr2=0x5616eb8cadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef94168740 1 -- 172.21.15.23:0/1649750625 shutdown_connections 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef94168740 1 -- 172.21.15.23:0/1649750625 wait complete. 2024-03-24T10:42:22.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef94168740 20 client.16155 trim_cache size 0 max 16384 2024-03-24T10:42:22.050 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:42:22.050 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:42:22.050 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:42:22.050 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:42:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef9158a640 1 -- 172.21.15.23:0/4011624760 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fef780a22b0 msgr2=0x7fef780a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef9158a640 1 --2- 172.21.15.23:0/4011624760 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fef780a22b0 0x7fef780a4770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fef84005cc0 tx=0x7fef84005ef0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.044+0000 7fef9158a640 1 -- 172.21.15.23:0/4011624760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef8c1005c0 msgr2=0x7fef8c198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.045+0000 7fef9158a640 1 --2- 172.21.15.23:0/4011624760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef8c1005c0 0x7fef8c198ab0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00e9c0 tx=0x7fef7c00ee80 comp rx=0 tx=0).stop 2024-03-24T10:42:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.045+0000 7fef9158a640 1 -- 172.21.15.23:0/4011624760 shutdown_connections 2024-03-24T10:42:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.045+0000 7fef9158a640 1 --2- 172.21.15.23:0/4011624760 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fef780a22b0 0x7fef780a4770 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.045+0000 7fef9158a640 1 --2- 172.21.15.23:0/4011624760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef8c106660 0x7fef8c1a3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.045+0000 7fef9158a640 1 --2- 172.21.15.23:0/4011624760 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef8c100f80 0x7fef8c198ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.045+0000 7fef9158a640 1 --2- 172.21.15.23:0/4011624760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef8c1005c0 0x7fef8c198ab0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.045+0000 7fef9158a640 1 -- 172.21.15.23:0/4011624760 >> 172.21.15.23:0/4011624760 conn(0x7fef8c076990 msgr2=0x7fef8c0fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.045+0000 7fef9158a640 1 -- 172.21.15.23:0/4011624760 shutdown_connections 2024-03-24T10:42:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.045+0000 7fef9158a640 1 -- 172.21.15.23:0/4011624760 wait complete. 2024-03-24T10:42:22.057 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:42:22.057 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:42:22.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 90418 12.93 MB/sec warmup 39 sec latency 16.165 ms 2024-03-24T10:42:22.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.173+0000 7fe3aeed4640 1 -- 172.21.15.23:0/2688567964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a81016a0 msgr2=0x7fe3a8101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.173+0000 7fe3aeed4640 1 --2- 172.21.15.23:0/2688567964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a81016a0 0x7fe3a8101b20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe398009930 tx=0x7fe39802bc30 comp rx=0 tx=0).stop 2024-03-24T10:42:22.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.173+0000 7fe3aeed4640 1 -- 172.21.15.23:0/2688567964 shutdown_connections 2024-03-24T10:42:22.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.173+0000 7fe3aeed4640 1 --2- 172.21.15.23:0/2688567964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3a8102060 0x7fe3a810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.173+0000 7fe3aeed4640 1 --2- 172.21.15.23:0/2688567964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a81016a0 0x7fe3a8101b20 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.173+0000 7fe3aeed4640 1 --2- 172.21.15.23:0/2688567964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3a81076b0 0x7fe3a8107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.173+0000 7fe3aeed4640 1 -- 172.21.15.23:0/2688567964 >> 172.21.15.23:0/2688567964 conn(0x7fe3a80fd1b0 msgr2=0x7fe3a80ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.173+0000 7fe3aeed4640 1 -- 172.21.15.23:0/2688567964 shutdown_connections 2024-03-24T10:42:22.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.173+0000 7fe3aeed4640 1 -- 172.21.15.23:0/2688567964 wait complete. 2024-03-24T10:42:22.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3aeed4640 1 Processor -- start 2024-03-24T10:42:22.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3aeed4640 1 -- start start 2024-03-24T10:42:22.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3aeed4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a81016a0 0x7fe3a8197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3aeed4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3a8102060 0x7fe3a81982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3aeed4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3a81076b0 0x7fe3a819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3aeed4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a819cc30 con 0x7fe3a81016a0 2024-03-24T10:42:22.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3aeed4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a819cda0 con 0x7fe3a8102060 2024-03-24T10:42:22.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3aeed4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a819cf60 con 0x7fe3a81076b0 2024-03-24T10:42:22.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3acc70640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a81016a0 0x7fe3a8197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3acc70640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a81016a0 0x7fe3a8197d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55148/0 (socket says 172.21.15.23:55148) 2024-03-24T10:42:22.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3acc70640 1 -- 172.21.15.23:0/746002886 learned_addr learned my addr 172.21.15.23:0/746002886 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3ad471640 1 --2- 172.21.15.23:0/746002886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3a81076b0 0x7fe3a819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe39ffff640 1 --2- 172.21.15.23:0/746002886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3a8102060 0x7fe3a81982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3acc70640 1 -- 172.21.15.23:0/746002886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3a81076b0 msgr2=0x7fe3a819c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3acc70640 1 --2- 172.21.15.23:0/746002886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3a81076b0 0x7fe3a819c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3acc70640 1 -- 172.21.15.23:0/746002886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3a8102060 msgr2=0x7fe3a81982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3acc70640 1 --2- 172.21.15.23:0/746002886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3a8102060 0x7fe3a81982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3acc70640 1 -- 172.21.15.23:0/746002886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe398009560 con 0x7fe3a81016a0 2024-03-24T10:42:22.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.174+0000 7fe3acc70640 1 --2- 172.21.15.23:0/746002886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a81016a0 0x7fe3a8197d70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe3a400e9f0 tx=0x7fe3a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.175+0000 7fe39dffb640 1 -- 172.21.15.23:0/746002886 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4009700 con 0x7fe3a81016a0 2024-03-24T10:42:22.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.175+0000 7fe39dffb640 1 -- 172.21.15.23:0/746002886 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3a400cd80 con 0x7fe3a81016a0 2024-03-24T10:42:22.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.175+0000 7fe3aeed4640 1 -- 172.21.15.23:0/746002886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3a819d240 con 0x7fe3a81016a0 2024-03-24T10:42:22.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.175+0000 7fe39dffb640 1 -- 172.21.15.23:0/746002886 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4018920 con 0x7fe3a81016a0 2024-03-24T10:42:22.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.175+0000 7fe3aeed4640 1 -- 172.21.15.23:0/746002886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3a81a4ac0 con 0x7fe3a81016a0 2024-03-24T10:42:22.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.176+0000 7fe3aeed4640 1 -- 172.21.15.23:0/746002886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3a8068500 con 0x7fe3a81016a0 2024-03-24T10:42:22.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.178+0000 7fe39dffb640 1 -- 172.21.15.23:0/746002886 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fe3a4018a80 con 0x7fe3a81016a0 2024-03-24T10:42:22.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.178+0000 7fe39dffb640 1 --2- 172.21.15.23:0/746002886 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe3900a2230 0x7fe3900a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.178+0000 7fe39ffff640 1 --2- 172.21.15.23:0/746002886 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe3900a2230 0x7fe3900a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.179+0000 7fe39dffb640 1 -- 172.21.15.23:0/746002886 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe3a4014070 con 0x7fe3a81016a0 2024-03-24T10:42:22.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.179+0000 7fe39ffff640 1 --2- 172.21.15.23:0/746002886 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe3900a2230 0x7fe3900a46f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe398009900 tx=0x7fe398005a20 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.182+0000 7fe39dffb640 1 -- 172.21.15.23:0/746002886 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe3a4097500 con 0x7fe3a81016a0 2024-03-24T10:42:22.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:22 smithi023 ceph-mon[32466]: pgmap v71: 97 pgs: 97 active+clean; 67 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 267 op/s 2024-03-24T10:42:22.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.423+0000 7fe3aeed4640 1 -- 172.21.15.23:0/746002886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3a81990c0 con 0x7fe3a81016a0 2024-03-24T10:42:22.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.423+0000 7fe39dffb640 1 -- 172.21.15.23:0/746002886 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11869 (secure 0 0 0) 0x7fe3a409abc0 con 0x7fe3a81016a0 2024-03-24T10:42:22.424 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:42:22.424 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:41:49.922577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:42:22.424 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 25 2024-03-24T10:42:22.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.426+0000 7fe3aeed4640 1 -- 172.21.15.23:0/746002886 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe3900a2230 msgr2=0x7fe3900a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.426+0000 7fe3aeed4640 1 --2- 172.21.15.23:0/746002886 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe3900a2230 0x7fe3900a46f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe398009900 tx=0x7fe398005a20 comp rx=0 tx=0).stop 2024-03-24T10:42:22.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.426+0000 7fe3aeed4640 1 -- 172.21.15.23:0/746002886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a81016a0 msgr2=0x7fe3a8197d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.426+0000 7fe3aeed4640 1 --2- 172.21.15.23:0/746002886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a81016a0 0x7fe3a8197d70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe3a400e9f0 tx=0x7fe3a400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.426+0000 7fe3aeed4640 1 -- 172.21.15.23:0/746002886 shutdown_connections 2024-03-24T10:42:22.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.426+0000 7fe3aeed4640 1 --2- 172.21.15.23:0/746002886 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe3900a2230 0x7fe3900a46f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.426+0000 7fe3aeed4640 1 --2- 172.21.15.23:0/746002886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3a81076b0 0x7fe3a819c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.426+0000 7fe3aeed4640 1 --2- 172.21.15.23:0/746002886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3a8102060 0x7fe3a81982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.426+0000 7fe3aeed4640 1 --2- 172.21.15.23:0/746002886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a81016a0 0x7fe3a8197d70 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.426+0000 7fe3aeed4640 1 -- 172.21.15.23:0/746002886 >> 172.21.15.23:0/746002886 conn(0x7fe3a80fd1b0 msgr2=0x7fe3a80fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.426+0000 7fe3aeed4640 1 -- 172.21.15.23:0/746002886 shutdown_connections 2024-03-24T10:42:22.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.426+0000 7fe3aeed4640 1 -- 172.21.15.23:0/746002886 wait complete. 2024-03-24T10:42:22.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:42:22.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:22 smithi117 ceph-mon[36368]: pgmap v71: 97 pgs: 97 active+clean; 67 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 267 op/s 2024-03-24T10:42:22.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:22 smithi116 ceph-mon[36248]: pgmap v71: 97 pgs: 97 active+clean; 67 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 267 op/s 2024-03-24T10:42:22.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.550+0000 7fa72e0eb640 1 -- 172.21.15.23:0/765343396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7281005a0 msgr2=0x7fa728100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.550+0000 7fa72e0eb640 1 --2- 172.21.15.23:0/765343396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7281005a0 0x7fa728100a20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009900 tx=0x7fa71c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:42:22.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.550+0000 7fa72e0eb640 1 -- 172.21.15.23:0/765343396 shutdown_connections 2024-03-24T10:42:22.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.550+0000 7fa72e0eb640 1 --2- 172.21.15.23:0/765343396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa728100f60 0x7fa72810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.550+0000 7fa72e0eb640 1 --2- 172.21.15.23:0/765343396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7281005a0 0x7fa728100a20 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.550+0000 7fa72e0eb640 1 --2- 172.21.15.23:0/765343396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa728106660 0x7fa728106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.550+0000 7fa72e0eb640 1 -- 172.21.15.23:0/765343396 >> 172.21.15.23:0/765343396 conn(0x7fa728077710 msgr2=0x7fa728077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.550+0000 7fa72e0eb640 1 -- 172.21.15.23:0/765343396 shutdown_connections 2024-03-24T10:42:22.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.550+0000 7fa72e0eb640 1 -- 172.21.15.23:0/765343396 wait complete. 2024-03-24T10:42:22.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.550+0000 7fa72e0eb640 1 Processor -- start 2024-03-24T10:42:22.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa72e0eb640 1 -- start start 2024-03-24T10:42:22.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa72e0eb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7281005a0 0x7fa728198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa72e0eb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa728100f60 0x7fa728199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa72e0eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa728106660 0x7fa7281a3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa72e0eb640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fa7281998a0 con 0x7fa7281005a0 2024-03-24T10:42:22.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa72e0eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa7281a4080 con 0x7fa728106660 2024-03-24T10:42:22.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa72e0eb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa7281a41c0 con 0x7fa728100f60 2024-03-24T10:42:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa7277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7281005a0 0x7fa728198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa7277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7281005a0 0x7fa728198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55160/0 (socket says 172.21.15.23:55160) 2024-03-24T10:42:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa7277fe640 1 -- 172.21.15.23:0/1746355749 learned_addr learned my addr 172.21.15.23:0/1746355749 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa727fff640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa728106660 0x7fa7281a3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa726ffd640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa728100f60 0x7fa728199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa7277fe640 1 -- 172.21.15.23:0/1746355749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa728100f60 msgr2=0x7fa728199000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa7277fe640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa728100f60 0x7fa728199000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa7277fe640 1 -- 172.21.15.23:0/1746355749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa728106660 msgr2=0x7fa7281a3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa7277fe640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa728106660 0x7fa7281a3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa7277fe640 1 -- 172.21.15.23:0/1746355749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa71c009560 con 0x7fa7281005a0 2024-03-24T10:42:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.551+0000 7fa727fff640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa728106660 0x7fa7281a3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.552+0000 7fa7277fe640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7281005a0 0x7fa728198ac0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa71800ea20 tx=0x7fa71800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.552+0000 7fa724ff9640 1 -- 172.21.15.23:0/1746355749 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa718009700 con 0x7fa7281005a0 2024-03-24T10:42:22.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.552+0000 7fa724ff9640 1 -- 172.21.15.23:0/1746355749 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa71800cd80 con 0x7fa7281005a0 2024-03-24T10:42:22.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.552+0000 7fa72e0eb640 1 -- 172.21.15.23:0/1746355749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7281a4470 con 0x7fa7281005a0 2024-03-24T10:42:22.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.552+0000 7fa724ff9640 1 -- 172.21.15.23:0/1746355749 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa718018840 con 0x7fa7281005a0 2024-03-24T10:42:22.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.552+0000 7fa72e0eb640 1 -- 172.21.15.23:0/1746355749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7281a4950 con 0x7fa7281005a0 2024-03-24T10:42:22.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa724ff9640 1 -- 172.21.15.23:0/1746355749 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fa718010a80 con 0x7fa7281005a0 2024-03-24T10:42:22.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa730da3740 1 Processor -- start 2024-03-24T10:42:22.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa730da3740 1 -- start start 2024-03-24T10:42:22.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa730da3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086473f0 0x55ee086477d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa730da3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee08647d10 0x55ee086503b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa730da3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee086508f0 0x55ee08652ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa730da3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ee085b5840 con 0x55ee08647d10 2024-03-24T10:42:22.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa730da3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ee085b2d70 con 0x55ee086473f0 2024-03-24T10:42:22.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa730da3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ee085ae550 con 0x55ee086508f0 2024-03-24T10:42:22.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa724ff9640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa7100a2230 0x7fa7100a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa7277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee08647d10 0x55ee086503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa7277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee08647d10 0x55ee086503b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55164/0 (socket says 172.21.15.23:55164) 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa7277fe640 1 -- 172.21.15.23:0/1152659071 learned_addr learned my addr 172.21.15.23:0/1152659071 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa726ffd640 1 --2- 172.21.15.23:0/1152659071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee086508f0 0x55ee08652ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.555+0000 7fa727fff640 1 --2- 172.21.15.23:0/1152659071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086473f0 0x55ee086477d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.556+0000 7fa727fff640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa7100a2230 0x7fa7100a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.556+0000 7fa724ff9640 1 -- 172.21.15.23:0/1746355749 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fa718014070 con 0x7fa7281005a0 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.556+0000 7fa726ffd640 1 -- 172.21.15.23:0/1152659071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086473f0 msgr2=0x55ee086477d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.556+0000 7fa726ffd640 1 --2- 172.21.15.23:0/1152659071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086473f0 0x55ee086477d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.556+0000 7fa726ffd640 1 -- 172.21.15.23:0/1152659071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee08647d10 msgr2=0x55ee086503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.556+0000 7fa726ffd640 1 --2- 172.21.15.23:0/1152659071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee08647d10 0x55ee086503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.556+0000 7fa726ffd640 1 -- 172.21.15.23:0/1152659071 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee08543de0 con 0x55ee086508f0 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.556+0000 7fa7277fe640 1 --2- 172.21.15.23:0/1152659071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee08647d10 0x55ee086503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.556+0000 7fa727fff640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa7100a2230 0x7fa7100a46f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa7200099e0 tx=0x7fa720009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.556+0000 7fa726ffd640 1 --2- 172.21.15.23:0/1152659071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee086508f0 0x55ee08652ce0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa71c000c00 tx=0x7fa71c02c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df8eeae641874fef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.556+0000 7fa70dffb640 1 -- 172.21.15.23:0/1152659071 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c02c910 con 0x55ee086508f0 2024-03-24T10:42:22.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.556+0000 7fa70dffb640 1 -- 172.21.15.23:0/1152659071 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa71c02ca70 con 0x55ee086508f0 2024-03-24T10:42:22.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.556+0000 7fa70dffb640 1 -- 172.21.15.23:0/1152659071 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c005170 con 0x55ee086508f0 2024-03-24T10:42:22.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.557+0000 7fa730da3740 1 -- 172.21.15.23:0/1152659071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee086508f0 msgr2=0x55ee08652ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.557+0000 7fa730da3740 1 --2- 172.21.15.23:0/1152659071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee086508f0 0x55ee08652ce0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa71c000c00 tx=0x7fa71c02c6a0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.557+0000 7fa730da3740 1 -- 172.21.15.23:0/1152659071 shutdown_connections 2024-03-24T10:42:22.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.557+0000 7fa730da3740 1 --2- 172.21.15.23:0/1152659071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee086508f0 0x55ee08652ce0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.557+0000 7fa730da3740 1 --2- 172.21.15.23:0/1152659071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee08647d10 0x55ee086503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.557+0000 7fa730da3740 1 --2- 172.21.15.23:0/1152659071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086473f0 0x55ee086477d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.557+0000 7fa730da3740 1 -- 172.21.15.23:0/1152659071 >> 172.21.15.23:0/1152659071 conn(0x55ee08641b20 msgr2=0x55ee08641f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.557+0000 7fa730da3740 1 -- 172.21.15.23:0/1152659071 shutdown_connections 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.557+0000 7fa730da3740 1 -- 172.21.15.23:0/1152659071 wait complete. 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.558+0000 7fa730da3740 1 -- start start 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.558+0000 7fa730da3740 1 Processor -- start 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.558+0000 7fa730da3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086473f0 0x55ee0864d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.558+0000 7fa730da3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee08647d10 0x55ee0864de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.558+0000 7fa730da3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086508f0 0x55ee0864e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.558+0000 7fa730da3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ee08281430 con 0x55ee086473f0 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.558+0000 7fa730da3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ee0823de00 con 0x55ee086508f0 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.558+0000 7fa730da3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ee08533590 con 0x55ee08647d10 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.558+0000 7fa730da3740 10 client.0 _ll_register_callbacks cb 0x55ee085a2fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.558+0000 7fa726ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee08647d10 0x55ee0864de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.558+0000 7fa7277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086473f0 0x55ee0864d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.558+0000 7fa726ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee08647d10 0x55ee0864de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:45350/0 (socket says 172.21.15.23:45350) 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.559+0000 7fa7277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086473f0 0x55ee0864d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55170/0 (socket says 172.21.15.23:55170) 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.559+0000 7fa726ffd640 1 -- 172.21.15.23:0/2556258778 learned_addr learned my addr 172.21.15.23:0/2556258778 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.559+0000 7fa727fff640 1 --2- 172.21.15.23:0/2556258778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086508f0 0x55ee0864e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.559+0000 7fa726ffd640 1 -- 172.21.15.23:0/2556258778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086508f0 msgr2=0x55ee0864e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.559+0000 7fa726ffd640 1 --2- 172.21.15.23:0/2556258778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086508f0 0x55ee0864e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.559+0000 7fa726ffd640 1 -- 172.21.15.23:0/2556258778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086473f0 msgr2=0x55ee0864d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.559+0000 7fa726ffd640 1 --2- 172.21.15.23:0/2556258778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086473f0 0x55ee0864d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.559+0000 7fa726ffd640 1 -- 172.21.15.23:0/2556258778 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa72810dd20 con 0x55ee08647d10 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.559+0000 7fa727fff640 1 --2- 172.21.15.23:0/2556258778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086508f0 0x55ee0864e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.559+0000 7fa726ffd640 1 --2- 172.21.15.23:0/2556258778 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee08647d10 0x55ee0864de30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa71c02c170 tx=0x7fa71c004500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d02c4f28d95442e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.559+0000 7fa706ffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.559+0000 7fa706ffd640 1 -- 172.21.15.23:0/2556258778 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71c005300 con 0x55ee08647d10 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.559+0000 7fa706ffd640 1 -- 172.21.15.23:0/2556258778 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fa71c005460 con 0x55ee08647d10 2024-03-24T10:42:22.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.559+0000 7fa730da3740 1 -- 172.21.15.23:0/2556258778 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ee082ad7e0 con 0x55ee08647d10 2024-03-24T10:42:22.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.560+0000 7fa706ffd640 1 -- 172.21.15.23:0/2556258778 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa71c005620 con 0x55ee08647d10 2024-03-24T10:42:22.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.560+0000 7fa730da3740 10 client.26023 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:22.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.560+0000 7fa730da3740 1 -- 172.21.15.23:0/2556258778 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ee085acbd0 con 0x55ee08647d10 2024-03-24T10:42:22.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.560+0000 7fa706ffd640 1 -- 172.21.15.23:0/2556258778 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ee082ad7e0 con 0x55ee08647d10 2024-03-24T10:42:22.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.560+0000 7fa706ffd640 1 -- 172.21.15.23:0/2556258778 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa71c03c050 con 0x55ee08647d10 2024-03-24T10:42:22.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.560+0000 7fa730da3740 10 client.26023 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:22.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.560+0000 7fa730da3740 10 client.26023 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:42:22.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.560+0000 7fa730da3740 20 client.26023 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:22.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.560+0000 7fa730da3740 1 --2- 172.21.15.23:0/2556258778 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55ee086605e0 0x55ee08662a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.560+0000 7fa730da3740 4 client.26023 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:22.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.560+0000 7fa730da3740 1 -- 172.21.15.23:0/2556258778 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee082ea720 con 0x55ee086605e0 2024-03-24T10:42:22.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.560+0000 7fa7277fe640 1 --2- 172.21.15.23:0/2556258778 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55ee086605e0 0x55ee08662a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.561+0000 7fa706ffd640 10 client.26023 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:42:22.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.561+0000 7fa7277fe640 1 --2- 172.21.15.23:0/2556258778 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55ee086605e0 0x55ee08662a00 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=14d664013b2603f6 server_cookie=f5bfed323fdf3650 in_seq=0 out_seq=0 2024-03-24T10:42:22.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa706ffd640 1 -- 172.21.15.23:0/2556258778 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ee082ea720 con 0x55ee086605e0 2024-03-24T10:42:22.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa706ffd640 10 client.26023 handle_command_reply: tid=0 2024-03-24T10:42:22.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa730da3740 1 client.26023 shutdown 2024-03-24T10:42:22.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa730da3740 1 -- 172.21.15.23:0/2556258778 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee08647d10 msgr2=0x55ee0864de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa730da3740 1 --2- 172.21.15.23:0/2556258778 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee08647d10 0x55ee0864de30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa71c02c170 tx=0x7fa71c004500 comp rx=0 tx=0).stop 2024-03-24T10:42:22.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa730da3740 1 -- 172.21.15.23:0/2556258778 shutdown_connections 2024-03-24T10:42:22.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa730da3740 1 --2- 172.21.15.23:0/2556258778 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55ee086605e0 0x55ee08662a00 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa730da3740 1 --2- 172.21.15.23:0/2556258778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086508f0 0x55ee0864e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa730da3740 1 --2- 172.21.15.23:0/2556258778 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee08647d10 0x55ee0864de30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa730da3740 1 --2- 172.21.15.23:0/2556258778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086473f0 0x55ee0864d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa706ffd640 0 client.26023 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:42:22.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa730da3740 1 -- 172.21.15.23:0/2556258778 >> 172.21.15.23:0/2556258778 conn(0x55ee08641b20 msgr2=0x55ee08643870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa730da3740 1 -- 172.21.15.23:0/2556258778 shutdown_connections 2024-03-24T10:42:22.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa730da3740 1 -- 172.21.15.23:0/2556258778 wait complete. 2024-03-24T10:42:22.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.563+0000 7fa730da3740 20 client.26023 trim_cache size 0 max 16384 2024-03-24T10:42:22.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.578+0000 7fa730da3740 1 Processor -- start 2024-03-24T10:42:22.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.578+0000 7fa730da3740 1 -- start start 2024-03-24T10:42:22.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.578+0000 7fa730da3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086473f0 0x55ee08650020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.578+0000 7fa730da3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee08647d10 0x55ee0865c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.578+0000 7fa730da3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086508f0 0x55ee08668310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.578+0000 7fa730da3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ee0864d6e0 con 0x55ee086508f0 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.578+0000 7fa730da3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ee08642b20 con 0x55ee086473f0 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.578+0000 7fa730da3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ee08641950 con 0x55ee08647d10 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.578+0000 7fa727fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086508f0 0x55ee08668310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.578+0000 7fa727fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086508f0 0x55ee08668310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55180/0 (socket says 172.21.15.23:55180) 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.578+0000 7fa727fff640 1 -- 172.21.15.23:0/4265111709 learned_addr learned my addr 172.21.15.23:0/4265111709 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.578+0000 7fa7277fe640 1 --2- 172.21.15.23:0/4265111709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086473f0 0x55ee08650020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.578+0000 7fa726ffd640 1 --2- 172.21.15.23:0/4265111709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee08647d10 0x55ee0865c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.579+0000 7fa727fff640 1 -- 172.21.15.23:0/4265111709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee08647d10 msgr2=0x55ee0865c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.579+0000 7fa727fff640 1 --2- 172.21.15.23:0/4265111709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee08647d10 0x55ee0865c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.579+0000 7fa727fff640 1 -- 172.21.15.23:0/4265111709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086473f0 msgr2=0x55ee08650020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.579+0000 7fa727fff640 1 --2- 172.21.15.23:0/4265111709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086473f0 0x55ee08650020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.579+0000 7fa727fff640 1 -- 172.21.15.23:0/4265111709 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee08548c80 con 0x55ee086508f0 2024-03-24T10:42:22.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.579+0000 7fa726ffd640 1 --2- 172.21.15.23:0/4265111709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee08647d10 0x55ee0865c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.579+0000 7fa7277fe640 1 --2- 172.21.15.23:0/4265111709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086473f0 0x55ee08650020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.579+0000 7fa727fff640 1 --2- 172.21.15.23:0/4265111709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086508f0 0x55ee08668310 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa72000fe70 tx=0x7fa720008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6656071f9a84ded2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.579+0000 7fa70d7fa640 1 -- 172.21.15.23:0/4265111709 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa720026070 con 0x55ee086508f0 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.579+0000 7fa70d7fa640 1 -- 172.21.15.23:0/4265111709 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa720003e00 con 0x55ee086508f0 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.579+0000 7fa70d7fa640 1 -- 172.21.15.23:0/4265111709 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72001fdb0 con 0x55ee086508f0 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.580+0000 7fa730da3740 1 -- 172.21.15.23:0/4265111709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086508f0 msgr2=0x55ee08668310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.580+0000 7fa730da3740 1 --2- 172.21.15.23:0/4265111709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086508f0 0x55ee08668310 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa72000fe70 tx=0x7fa720008000 comp rx=0 tx=0).stop 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.580+0000 7fa730da3740 1 -- 172.21.15.23:0/4265111709 shutdown_connections 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.580+0000 7fa730da3740 1 --2- 172.21.15.23:0/4265111709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086508f0 0x55ee08668310 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.580+0000 7fa730da3740 1 --2- 172.21.15.23:0/4265111709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee08647d10 0x55ee0865c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.580+0000 7fa730da3740 1 --2- 172.21.15.23:0/4265111709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee086473f0 0x55ee08650020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.580+0000 7fa730da3740 1 -- 172.21.15.23:0/4265111709 >> 172.21.15.23:0/4265111709 conn(0x55ee08641b20 msgr2=0x55ee08640cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.580+0000 7fa730da3740 1 -- 172.21.15.23:0/4265111709 shutdown_connections 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.580+0000 7fa730da3740 1 -- 172.21.15.23:0/4265111709 wait complete. 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.580+0000 7fa730da3740 1 -- start start 2024-03-24T10:42:22.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.580+0000 7fa730da3740 1 Processor -- start 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa730da3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086473f0 0x55ee086589b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa730da3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee08647d10 0x55ee08654d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa730da3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee086508f0 0x55ee08655270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa730da3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ee08655940 con 0x55ee086473f0 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa730da3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ee08655ab0 con 0x55ee08647d10 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa730da3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ee086c6950 con 0x55ee086508f0 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa730da3740 10 client.0 _ll_register_callbacks cb 0x7fa718010d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa7277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086473f0 0x55ee086589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa7277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086473f0 0x55ee086589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55182/0 (socket says 172.21.15.23:55182) 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa7277fe640 1 -- 172.21.15.23:0/2287110796 learned_addr learned my addr 172.21.15.23:0/2287110796 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa726ffd640 1 --2- 172.21.15.23:0/2287110796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee08647d10 0x55ee08654d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa727fff640 1 --2- 172.21.15.23:0/2287110796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee086508f0 0x55ee08655270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa7277fe640 1 -- 172.21.15.23:0/2287110796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee086508f0 msgr2=0x55ee08655270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa7277fe640 1 --2- 172.21.15.23:0/2287110796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee086508f0 0x55ee08655270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa7277fe640 1 -- 172.21.15.23:0/2287110796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee08647d10 msgr2=0x55ee08654d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa7277fe640 1 --2- 172.21.15.23:0/2287110796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee08647d10 0x55ee08654d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa7277fe640 1 -- 172.21.15.23:0/2287110796 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7281a4470 con 0x55ee086473f0 2024-03-24T10:42:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa727fff640 1 --2- 172.21.15.23:0/2287110796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee086508f0 0x55ee08655270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:42:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa7277fe640 1 --2- 172.21.15.23:0/2287110796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086473f0 0x55ee086589b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa71800e9f0 tx=0x7fa7180ae040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8862c657f83904e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa706ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.581+0000 7fa706ffd640 1 -- 172.21.15.23:0/2287110796 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7180cbd10 con 0x55ee086473f0 2024-03-24T10:42:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.582+0000 7fa706ffd640 1 -- 172.21.15.23:0/2287110796 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fa7180cbe70 con 0x55ee086473f0 2024-03-24T10:42:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.582+0000 7fa730da3740 1 -- 172.21.15.23:0/2287110796 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa71c02ca70 con 0x55ee086473f0 2024-03-24T10:42:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.582+0000 7fa706ffd640 1 -- 172.21.15.23:0/2287110796 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa718097450 con 0x55ee086473f0 2024-03-24T10:42:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.582+0000 7fa730da3740 10 client.16179 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.582+0000 7fa730da3740 1 -- 172.21.15.23:0/2287110796 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ee085acbd0 con 0x55ee086473f0 2024-03-24T10:42:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.582+0000 7fa706ffd640 1 -- 172.21.15.23:0/2287110796 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa71807d020 con 0x55ee086473f0 2024-03-24T10:42:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.582+0000 7fa706ffd640 1 -- 172.21.15.23:0/2287110796 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa71809a230 con 0x55ee086473f0 2024-03-24T10:42:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.582+0000 7fa730da3740 10 client.16179 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:22.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.582+0000 7fa730da3740 10 client.16179 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:42:22.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.582+0000 7fa730da3740 20 client.16179 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:22.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.582+0000 7fa730da3740 1 --2- 172.21.15.23:0/2287110796 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55ee086c8060 0x55ee086ca480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.582+0000 7fa730da3740 4 client.16179 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:42:22.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.582+0000 7fa730da3740 1 -- 172.21.15.23:0/2287110796 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fa71c02ca70 con 0x55ee086c8060 2024-03-24T10:42:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.583+0000 7fa726ffd640 1 --2- 172.21.15.23:0/2287110796 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55ee086c8060 0x55ee086ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.583+0000 7fa726ffd640 1 --2- 172.21.15.23:0/2287110796 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55ee086c8060 0x55ee086ca480 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c835713338632f98 server_cookie=fca4aa294ab0c52 in_seq=0 out_seq=0 2024-03-24T10:42:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.583+0000 7fa706ffd640 10 client.16179 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:42:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa706ffd640 1 -- 172.21.15.23:0/2287110796 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fa71c02ca70 con 0x55ee086c8060 2024-03-24T10:42:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa706ffd640 10 client.16179 handle_command_reply: tid=0 2024-03-24T10:42:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa730da3740 1 client.16179 shutdown 2024-03-24T10:42:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa730da3740 1 -- 172.21.15.23:0/2287110796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086473f0 msgr2=0x55ee086589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa730da3740 1 --2- 172.21.15.23:0/2287110796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086473f0 0x55ee086589b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa71800e9f0 tx=0x7fa7180ae040 comp rx=0 tx=0).stop 2024-03-24T10:42:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa730da3740 1 -- 172.21.15.23:0/2287110796 shutdown_connections 2024-03-24T10:42:22.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa730da3740 1 --2- 172.21.15.23:0/2287110796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee086508f0 0x55ee08655270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa730da3740 1 --2- 172.21.15.23:0/2287110796 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55ee086c8060 0x55ee086ca480 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa730da3740 1 --2- 172.21.15.23:0/2287110796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee08647d10 0x55ee08654d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa730da3740 1 --2- 172.21.15.23:0/2287110796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee086473f0 0x55ee086589b0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa706ffd640 0 client.16179 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:42:22.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa730da3740 1 -- 172.21.15.23:0/2287110796 >> 172.21.15.23:0/2287110796 conn(0x55ee08641b20 msgr2=0x55ee0864d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa730da3740 1 -- 172.21.15.23:0/2287110796 shutdown_connections 2024-03-24T10:42:22.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa730da3740 1 -- 172.21.15.23:0/2287110796 wait complete. 2024-03-24T10:42:22.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.586+0000 7fa730da3740 20 client.16179 trim_cache size 0 max 16384 2024-03-24T10:42:22.589 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:42:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.587+0000 7fa72e0eb640 1 -- 172.21.15.23:0/1746355749 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa7100a2230 msgr2=0x7fa7100a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.587+0000 7fa72e0eb640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa7100a2230 0x7fa7100a46f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa7200099e0 tx=0x7fa720009500 comp rx=0 tx=0).stop 2024-03-24T10:42:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.587+0000 7fa72e0eb640 1 -- 172.21.15.23:0/1746355749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7281005a0 msgr2=0x7fa728198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.587+0000 7fa72e0eb640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7281005a0 0x7fa728198ac0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa71800ea20 tx=0x7fa71800eee0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.587+0000 7fa72e0eb640 1 -- 172.21.15.23:0/1746355749 shutdown_connections 2024-03-24T10:42:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.587+0000 7fa72e0eb640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa7100a2230 0x7fa7100a46f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.587+0000 7fa72e0eb640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa728106660 0x7fa7281a3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.587+0000 7fa72e0eb640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa728100f60 0x7fa728199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.587+0000 7fa72e0eb640 1 --2- 172.21.15.23:0/1746355749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7281005a0 0x7fa728198ac0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.587+0000 7fa72e0eb640 1 -- 172.21.15.23:0/1746355749 >> 172.21.15.23:0/1746355749 conn(0x7fa728077710 msgr2=0x7fa7280ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.587+0000 7fa72e0eb640 1 -- 172.21.15.23:0/1746355749 shutdown_connections 2024-03-24T10:42:22.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.587+0000 7fa72e0eb640 1 -- 172.21.15.23:0/1746355749 wait complete. 2024-03-24T10:42:22.599 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:42:22.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.713+0000 7f14c2081640 1 -- 172.21.15.23:0/3049312853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc100f60 msgr2=0x7f14bc10d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.713+0000 7f14c2081640 1 --2- 172.21.15.23:0/3049312853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc100f60 0x7f14bc10d420 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009ca0 tx=0x7f14b402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.713+0000 7f14c2081640 1 -- 172.21.15.23:0/3049312853 shutdown_connections 2024-03-24T10:42:22.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.713+0000 7f14c2081640 1 --2- 172.21.15.23:0/3049312853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc100f60 0x7f14bc10d420 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.713+0000 7f14c2081640 1 --2- 172.21.15.23:0/3049312853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14bc1005a0 0x7f14bc100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.713+0000 7f14c2081640 1 --2- 172.21.15.23:0/3049312853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14bc106660 0x7f14bc106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.713+0000 7f14c2081640 1 -- 172.21.15.23:0/3049312853 >> 172.21.15.23:0/3049312853 conn(0x7f14bc077710 msgr2=0x7f14bc077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.713+0000 7f14c2081640 1 -- 172.21.15.23:0/3049312853 shutdown_connections 2024-03-24T10:42:22.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.713+0000 7f14c2081640 1 -- 172.21.15.23:0/3049312853 wait complete. 2024-03-24T10:42:22.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.713+0000 7f14c2081640 1 Processor -- start 2024-03-24T10:42:22.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.713+0000 7f14c2081640 1 -- start start 2024-03-24T10:42:22.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14c2081640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14bc1005a0 0x7f14bc102580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14c2081640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc100f60 0x7f14bc102ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14c2081640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14bc106660 0x7f14bc103000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14c2081640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc1a5fa0 con 0x7f14bc100f60 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14c2081640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc1a6110 con 0x7f14bc106660 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14c2081640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc1a62d0 con 0x7f14bc1005a0 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14baffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc100f60 0x7f14bc102ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14baffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc100f60 0x7f14bc102ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55200/0 (socket says 172.21.15.23:55200) 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14bb7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14bc1005a0 0x7f14bc102580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14baffd640 1 -- 172.21.15.23:0/1020664067 learned_addr learned my addr 172.21.15.23:0/1020664067 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14bbfff640 1 --2- 172.21.15.23:0/1020664067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14bc106660 0x7f14bc103000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14bb7fe640 1 -- 172.21.15.23:0/1020664067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14bc106660 msgr2=0x7f14bc103000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14bb7fe640 1 --2- 172.21.15.23:0/1020664067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14bc106660 0x7f14bc103000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14bb7fe640 1 -- 172.21.15.23:0/1020664067 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc100f60 msgr2=0x7f14bc102ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14bb7fe640 1 --2- 172.21.15.23:0/1020664067 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc100f60 0x7f14bc102ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14bb7fe640 1 -- 172.21.15.23:0/1020664067 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14a4009490 con 0x7f14bc1005a0 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14baffd640 1 --2- 172.21.15.23:0/1020664067 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc100f60 0x7f14bc102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14bb7fe640 1 --2- 172.21.15.23:0/1020664067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14bc1005a0 0x7f14bc102580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f14a400e9f0 tx=0x7f14a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14b8ff9640 1 -- 172.21.15.23:0/1020664067 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a400cd80 con 0x7f14bc1005a0 2024-03-24T10:42:22.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14b8ff9640 1 -- 172.21.15.23:0/1020664067 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14a40035f0 con 0x7f14bc1005a0 2024-03-24T10:42:22.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14c2081640 1 -- 172.21.15.23:0/1020664067 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b4009900 con 0x7f14bc1005a0 2024-03-24T10:42:22.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14b8ff9640 1 -- 172.21.15.23:0/1020664067 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a4010690 con 0x7f14bc1005a0 2024-03-24T10:42:22.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.714+0000 7f14c2081640 1 -- 172.21.15.23:0/1020664067 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14bc1a6930 con 0x7f14bc1005a0 2024-03-24T10:42:22.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14b8ff9640 1 -- 172.21.15.23:0/1020664067 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f14a4003af0 con 0x7f14bc1005a0 2024-03-24T10:42:22.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14c4c5e740 1 Processor -- start 2024-03-24T10:42:22.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14c4c5e740 1 -- start start 2024-03-24T10:42:22.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14c4c5e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dba67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14c4c5e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba6d10 0x55d05dbaf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14c4c5e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbb1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14c4c5e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d05db14840 con 0x55d05dbaf8f0 2024-03-24T10:42:22.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14c4c5e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d05db11d70 con 0x55d05dba63f0 2024-03-24T10:42:22.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14c4c5e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d05db0d550 con 0x55d05dba6d10 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14baffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbb1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dba67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dba67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:45536/0 (socket says 172.21.15.23:45536) 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14bbfff640 1 -- 172.21.15.23:0/4212347941 learned_addr learned my addr 172.21.15.23:0/4212347941 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14bb7fe640 1 --2- 172.21.15.23:0/4212347941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba6d10 0x55d05dbaf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14b8ff9640 1 --2- 172.21.15.23:0/1020664067 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f14a80a2260 0x7f14a80a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14bbfff640 1 -- 172.21.15.23:0/4212347941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba6d10 msgr2=0x55d05dbaf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14bbfff640 1 --2- 172.21.15.23:0/4212347941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba6d10 0x55d05dbaf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14bbfff640 1 -- 172.21.15.23:0/4212347941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 msgr2=0x55d05dbb1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14bbfff640 1 --2- 172.21.15.23:0/4212347941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbb1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14bbfff640 1 -- 172.21.15.23:0/4212347941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d05daa2de0 con 0x55d05dba63f0 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14baffd640 1 --2- 172.21.15.23:0/4212347941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbb1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14bbfff640 1 --2- 172.21.15.23:0/1020664067 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f14a80a2260 0x7f14a80a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.717+0000 7f14b8ff9640 1 -- 172.21.15.23:0/1020664067 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f14a4014070 con 0x7f14bc1005a0 2024-03-24T10:42:22.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.718+0000 7f14bbfff640 1 --2- 172.21.15.23:0/4212347941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dba67d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f14b4004af0 tx=0x7f14b4032d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=800e987d7c425a07 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.718+0000 7f14b1ffb640 1 -- 172.21.15.23:0/4212347941 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b40326a0 con 0x55d05dba63f0 2024-03-24T10:42:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.718+0000 7f14bbfff640 1 --2- 172.21.15.23:0/1020664067 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f14a80a2260 0x7f14a80a4720 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f14b4006010 tx=0x7f14b404a000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.718+0000 7f14b1ffb640 1 -- 172.21.15.23:0/4212347941 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14b4003e50 con 0x55d05dba63f0 2024-03-24T10:42:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.718+0000 7f14b1ffb640 1 -- 172.21.15.23:0/4212347941 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b40044a0 con 0x55d05dba63f0 2024-03-24T10:42:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.718+0000 7f14c4c5e740 1 -- 172.21.15.23:0/4212347941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 msgr2=0x55d05dba67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.718+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/4212347941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dba67d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f14b4004af0 tx=0x7f14b4032d60 comp rx=0 tx=0).stop 2024-03-24T10:42:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 -- 172.21.15.23:0/4212347941 shutdown_connections 2024-03-24T10:42:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/4212347941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbb1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/4212347941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba6d10 0x55d05dbaf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/4212347941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dba67d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 -- 172.21.15.23:0/4212347941 >> 172.21.15.23:0/4212347941 conn(0x55d05dba0b20 msgr2=0x55d05dba0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 -- 172.21.15.23:0/4212347941 shutdown_connections 2024-03-24T10:42:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 -- 172.21.15.23:0/4212347941 wait complete. 2024-03-24T10:42:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 -- start start 2024-03-24T10:42:22.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 Processor -- start 2024-03-24T10:42:22.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dbac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dba6d10 0x55d05dbacf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dbaf8f0 0x55d05dbad470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d05d7e0430 con 0x55d05dba6d10 2024-03-24T10:42:22.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d05d79ce00 con 0x55d05dba63f0 2024-03-24T10:42:22.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.719+0000 7f14c4c5e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d05da92590 con 0x55d05dbaf8f0 2024-03-24T10:42:22.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.720+0000 7f14c4c5e740 10 client.0 _ll_register_callbacks cb 0x55d05db01fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:22.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.720+0000 7f14baffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dba6d10 0x55d05dbacf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.720+0000 7f14bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dbac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.720+0000 7f14bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dbac9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:45542/0 (socket says 172.21.15.23:45542) 2024-03-24T10:42:22.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.720+0000 7f14bb7fe640 1 -- 172.21.15.23:0/1815120335 learned_addr learned my addr 172.21.15.23:0/1815120335 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.720+0000 7f14bb7fe640 1 -- 172.21.15.23:0/1815120335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dbaf8f0 msgr2=0x55d05dbad470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T10:42:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.720+0000 7f14bb7fe640 1 --2- 172.21.15.23:0/1815120335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dbaf8f0 0x55d05dbad470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.720+0000 7f14bb7fe640 1 -- 172.21.15.23:0/1815120335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dba6d10 msgr2=0x55d05dbacf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.720+0000 7f14bb7fe640 1 --2- 172.21.15.23:0/1815120335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dba6d10 0x55d05dbacf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.720+0000 7f14bb7fe640 1 -- 172.21.15.23:0/1815120335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14bc10dd20 con 0x55d05dba63f0 2024-03-24T10:42:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.720+0000 7f14bb7fe640 1 --2- 172.21.15.23:0/1815120335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dbac9f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f14a400f010 tx=0x7f14a409ac80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8246273b0e5cd93a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.720+0000 7f149affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:42:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.720+0000 7f149affd640 1 -- 172.21.15.23:0/1815120335 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a409adf0 con 0x55d05dba63f0 2024-03-24T10:42:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.721+0000 7f149affd640 1 -- 172.21.15.23:0/1815120335 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f14a409ef40 con 0x55d05dba63f0 2024-03-24T10:42:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.721+0000 7f14c4c5e740 1 -- 172.21.15.23:0/1815120335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d05d80c7e0 con 0x55d05dba63f0 2024-03-24T10:42:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.721+0000 7f149affd640 1 -- 172.21.15.23:0/1815120335 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f14a40a3590 con 0x55d05dba63f0 2024-03-24T10:42:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.721+0000 7f14c4c5e740 10 client.26069 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.721+0000 7f14c4c5e740 1 -- 172.21.15.23:0/1815120335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d05db0bbd0 con 0x55d05dba63f0 2024-03-24T10:42:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.721+0000 7f149affd640 1 -- 172.21.15.23:0/1815120335 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d05d80c7e0 con 0x55d05dba63f0 2024-03-24T10:42:22.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.721+0000 7f149affd640 1 -- 172.21.15.23:0/1815120335 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f14a4003760 con 0x55d05dba63f0 2024-03-24T10:42:22.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.721+0000 7f14c4c5e740 10 client.26069 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:22.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.721+0000 7f14c4c5e740 10 client.26069 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:42:22.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.721+0000 7f14c4c5e740 20 client.26069 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:22.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.721+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/1815120335 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d05dbbf910 0x55d05dbc1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.721+0000 7f14c4c5e740 4 client.26069 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:22.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.721+0000 7f14c4c5e740 1 -- 172.21.15.23:0/1815120335 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d05d849720 con 0x55d05dbbf910 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.722+0000 7f14baffd640 1 --2- 172.21.15.23:0/1815120335 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d05dbbf910 0x55d05dbc1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.722+0000 7f14baffd640 1 --2- 172.21.15.23:0/1815120335 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d05dbbf910 0x55d05dbc1d50 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=15c6f6fa9be5ebe6 server_cookie=18c6c2525ee4be6a in_seq=0 out_seq=0 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.722+0000 7f149affd640 10 client.26069 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.722+0000 7f149affd640 1 -- 172.21.15.23:0/1815120335 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d05d849720 con 0x55d05dbbf910 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.722+0000 7f149affd640 10 client.26069 handle_command_reply: tid=0 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.723+0000 7f14c4c5e740 1 client.26069 shutdown 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.723+0000 7f14c4c5e740 1 -- 172.21.15.23:0/1815120335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 msgr2=0x55d05dbac9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.723+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/1815120335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dbac9f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f14a400f010 tx=0x7f14a409ac80 comp rx=0 tx=0).stop 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.723+0000 7f14c4c5e740 1 -- 172.21.15.23:0/1815120335 shutdown_connections 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.723+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/1815120335 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d05dbbf910 0x55d05dbc1d50 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.723+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/1815120335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dbaf8f0 0x55d05dbad470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.723+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/1815120335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dba6d10 0x55d05dbacf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.723+0000 7f149affd640 0 client.26069 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.723+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/1815120335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dbac9f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.723+0000 7f14c4c5e740 1 -- 172.21.15.23:0/1815120335 >> 172.21.15.23:0/1815120335 conn(0x55d05dba0b20 msgr2=0x55d05dba2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.723+0000 7f14c4c5e740 1 -- 172.21.15.23:0/1815120335 shutdown_connections 2024-03-24T10:42:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.723+0000 7f14c4c5e740 1 -- 172.21.15.23:0/1815120335 wait complete. 2024-03-24T10:42:22.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.723+0000 7f14c4c5e740 20 client.26069 trim_cache size 0 max 16384 2024-03-24T10:42:22.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.737+0000 7f14c4c5e740 1 Processor -- start 2024-03-24T10:42:22.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.737+0000 7f14c4c5e740 1 -- start start 2024-03-24T10:42:22.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.737+0000 7f14c4c5e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba63f0 0x55d05dbbb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.737+0000 7f14c4c5e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba6d10 0x55d05db9fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.737+0000 7f14c4c5e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbc7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.737+0000 7f14c4c5e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d05dbad9f0 con 0x55d05dbaf8f0 2024-03-24T10:42:22.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.737+0000 7f14c4c5e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d05dba5710 con 0x55d05dba6d10 2024-03-24T10:42:22.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.737+0000 7f14c4c5e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d05dba5b90 con 0x55d05dba63f0 2024-03-24T10:42:22.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.737+0000 7f14bbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbc7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.737+0000 7f14bbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbc7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55222/0 (socket says 172.21.15.23:55222) 2024-03-24T10:42:22.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.737+0000 7f14bbfff640 1 -- 172.21.15.23:0/609060045 learned_addr learned my addr 172.21.15.23:0/609060045 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.737+0000 7f14bb7fe640 1 --2- 172.21.15.23:0/609060045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba63f0 0x55d05dbbb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.737+0000 7f14baffd640 1 --2- 172.21.15.23:0/609060045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba6d10 0x55d05db9fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.738+0000 7f14bbfff640 1 -- 172.21.15.23:0/609060045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba63f0 msgr2=0x55d05dbbb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.738+0000 7f14bbfff640 1 --2- 172.21.15.23:0/609060045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba63f0 0x55d05dbbb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.738+0000 7f14bbfff640 1 -- 172.21.15.23:0/609060045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba6d10 msgr2=0x55d05db9fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.738+0000 7f14bbfff640 1 --2- 172.21.15.23:0/609060045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba6d10 0x55d05db9fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.738+0000 7f14bbfff640 1 -- 172.21.15.23:0/609060045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d05daa7c80 con 0x55d05dbaf8f0 2024-03-24T10:42:22.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.738+0000 7f14bb7fe640 1 --2- 172.21.15.23:0/609060045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba63f0 0x55d05dbbb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:22.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.738+0000 7f14bbfff640 1 --2- 172.21.15.23:0/609060045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbc7660 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f14b4032e70 tx=0x7f14b4005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76519e4cf79aeade server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.738+0000 7f14b17fa640 1 -- 172.21.15.23:0/609060045 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4032460 con 0x55d05dbaf8f0 2024-03-24T10:42:22.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.738+0000 7f14b17fa640 1 -- 172.21.15.23:0/609060045 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14b4003b70 con 0x55d05dbaf8f0 2024-03-24T10:42:22.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.738+0000 7f14b17fa640 1 -- 172.21.15.23:0/609060045 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4004830 con 0x55d05dbaf8f0 2024-03-24T10:42:22.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 -- 172.21.15.23:0/609060045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 msgr2=0x55d05dbc7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/609060045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbc7660 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f14b4032e70 tx=0x7f14b4005bd0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 -- 172.21.15.23:0/609060045 shutdown_connections 2024-03-24T10:42:22.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/609060045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbc7660 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/609060045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba6d10 0x55d05db9fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/609060045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba63f0 0x55d05dbbb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 -- 172.21.15.23:0/609060045 >> 172.21.15.23:0/609060045 conn(0x55d05dba0b20 msgr2=0x55d05db9f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 -- 172.21.15.23:0/609060045 shutdown_connections 2024-03-24T10:42:22.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 -- 172.21.15.23:0/609060045 wait complete. 2024-03-24T10:42:22.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 -- start start 2024-03-24T10:42:22.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 Processor -- start 2024-03-24T10:42:22.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dbb77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba6d10 0x55d05dbb4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbb4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d05dbb7dc0 con 0x55d05dbaf8f0 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d05dbb7f30 con 0x55d05dba63f0 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d05dbb5200 con 0x55d05dba6d10 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.739+0000 7f14c4c5e740 10 client.0 _ll_register_callbacks cb 0x7f14a40d1a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f14bbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbb4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f14bbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbb4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55232/0 (socket says 172.21.15.23:55232) 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f14bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dbb77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f14bbfff640 1 -- 172.21.15.23:0/4256942268 learned_addr learned my addr 172.21.15.23:0/4256942268 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f14baffd640 1 --2- 172.21.15.23:0/4256942268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba6d10 0x55d05dbb4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f14bbfff640 1 -- 172.21.15.23:0/4256942268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba6d10 msgr2=0x55d05dbb4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f14bbfff640 1 --2- 172.21.15.23:0/4256942268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba6d10 0x55d05dbb4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f14bbfff640 1 -- 172.21.15.23:0/4256942268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 msgr2=0x55d05dbb77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f14bbfff640 1 --2- 172.21.15.23:0/4256942268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dbb77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f14bbfff640 1 -- 172.21.15.23:0/4256942268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d05db0bbd0 con 0x55d05dbaf8f0 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f14bbfff640 1 --2- 172.21.15.23:0/4256942268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbb4cc0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f14b402c630 tx=0x7f14b4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65fbebfde496a0a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f149affd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f149affd640 1 -- 172.21.15.23:0/4256942268 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4032460 con 0x55d05dbaf8f0 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f14c4c5e740 1 -- 172.21.15.23:0/4256942268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14b4003e50 con 0x55d05dbaf8f0 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f149affd640 1 -- 172.21.15.23:0/4256942268 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f14b405e470 con 0x55d05dbaf8f0 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f149affd640 1 -- 172.21.15.23:0/4256942268 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f14b4003ce0 con 0x55d05dbaf8f0 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f14c4c5e740 10 client.16197 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f14c4c5e740 1 -- 172.21.15.23:0/4256942268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d05daa2de0 con 0x55d05dbaf8f0 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.740+0000 7f149affd640 1 -- 172.21.15.23:0/4256942268 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14b4003e50 con 0x55d05dbaf8f0 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.741+0000 7f149affd640 1 -- 172.21.15.23:0/4256942268 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f14b406d070 con 0x55d05dbaf8f0 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.741+0000 7f14c4c5e740 10 client.16197 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.741+0000 7f14c4c5e740 10 client.16197 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.741+0000 7f14c4c5e740 20 client.16197 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.741+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/4256942268 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d05dbbf910 0x55d05dc28ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.741+0000 7f14c4c5e740 4 client.16197 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.741+0000 7f14c4c5e740 1 -- 172.21.15.23:0/4256942268 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d05d898890 con 0x55d05dbbf910 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.741+0000 7f14bb7fe640 1 --2- 172.21.15.23:0/4256942268 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d05dbbf910 0x55d05dc28ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.741+0000 7f149affd640 10 client.16197 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.741+0000 7f14bb7fe640 1 --2- 172.21.15.23:0/4256942268 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d05dbbf910 0x55d05dc28ae0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d162e27c207e531c server_cookie=b4ee39f83489b21d in_seq=0 out_seq=0 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.741+0000 7f149affd640 1 -- 172.21.15.23:0/4256942268 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d05d898890 con 0x55d05dbbf910 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.741+0000 7f149affd640 10 client.16197 handle_command_reply: tid=0 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.741+0000 7f14c4c5e740 1 client.16197 shutdown 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c4c5e740 1 -- 172.21.15.23:0/4256942268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 msgr2=0x55d05dbb4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/4256942268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbb4cc0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f14b402c630 tx=0x7f14b4004040 comp rx=0 tx=0).stop 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c4c5e740 1 -- 172.21.15.23:0/4256942268 shutdown_connections 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/4256942268 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d05dbbf910 0x55d05dc28ae0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/4256942268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05dbaf8f0 0x55d05dbb4cc0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/4256942268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d05dba6d10 0x55d05dbb4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f149affd640 0 client.16197 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:42:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c4c5e740 1 --2- 172.21.15.23:0/4256942268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d05dba63f0 0x55d05dbb77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c4c5e740 1 -- 172.21.15.23:0/4256942268 >> 172.21.15.23:0/4256942268 conn(0x55d05dba0b20 msgr2=0x55d05db9fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c4c5e740 1 -- 172.21.15.23:0/4256942268 shutdown_connections 2024-03-24T10:42:22.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c4c5e740 1 -- 172.21.15.23:0/4256942268 wait complete. 2024-03-24T10:42:22.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c4c5e740 20 client.16197 trim_cache size 0 max 16384 2024-03-24T10:42:22.748 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:42:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c2081640 1 -- 172.21.15.23:0/1020664067 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f14a80a2260 msgr2=0x7f14a80a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c2081640 1 --2- 172.21.15.23:0/1020664067 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f14a80a2260 0x7f14a80a4720 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f14b4006010 tx=0x7f14b404a000 comp rx=0 tx=0).stop 2024-03-24T10:42:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c2081640 1 -- 172.21.15.23:0/1020664067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14bc1005a0 msgr2=0x7f14bc102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c2081640 1 --2- 172.21.15.23:0/1020664067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14bc1005a0 0x7f14bc102580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f14a400e9f0 tx=0x7f14a400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c2081640 1 -- 172.21.15.23:0/1020664067 shutdown_connections 2024-03-24T10:42:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c2081640 1 --2- 172.21.15.23:0/1020664067 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f14a80a2260 0x7f14a80a4720 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c2081640 1 --2- 172.21.15.23:0/1020664067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14bc106660 0x7f14bc103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c2081640 1 --2- 172.21.15.23:0/1020664067 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14bc100f60 0x7f14bc102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c2081640 1 --2- 172.21.15.23:0/1020664067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14bc1005a0 0x7f14bc102580 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c2081640 1 -- 172.21.15.23:0/1020664067 >> 172.21.15.23:0/1020664067 conn(0x7f14bc077710 msgr2=0x7f14bc1048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.742+0000 7f14c2081640 1 -- 172.21.15.23:0/1020664067 shutdown_connections 2024-03-24T10:42:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.743+0000 7f14c2081640 1 -- 172.21.15.23:0/1020664067 wait complete. 2024-03-24T10:42:22.754 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:42:22.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.865+0000 7f463b42a640 1 -- 172.21.15.23:0/2137696081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46341032a0 msgr2=0x7f463410b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.865+0000 7f463b42a640 1 --2- 172.21.15.23:0/2137696081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46341032a0 0x7f463410b780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4624009860 tx=0x7f462402bc00 comp rx=0 tx=0).stop 2024-03-24T10:42:22.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.865+0000 7f463b42a640 1 -- 172.21.15.23:0/2137696081 shutdown_connections 2024-03-24T10:42:22.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.865+0000 7f463b42a640 1 --2- 172.21.15.23:0/2137696081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f463410bd50 0x7f463410e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.865+0000 7f463b42a640 1 --2- 172.21.15.23:0/2137696081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46341032a0 0x7f463410b780 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.865+0000 7f463b42a640 1 --2- 172.21.15.23:0/2137696081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4634102980 0x7f4634102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.865+0000 7f463b42a640 1 -- 172.21.15.23:0/2137696081 >> 172.21.15.23:0/2137696081 conn(0x7f4634077770 msgr2=0x7f4634077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.865+0000 7f463b42a640 1 -- 172.21.15.23:0/2137696081 shutdown_connections 2024-03-24T10:42:22.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.865+0000 7f463b42a640 1 -- 172.21.15.23:0/2137696081 wait complete. 2024-03-24T10:42:22.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f463b42a640 1 Processor -- start 2024-03-24T10:42:22.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f463b42a640 1 -- start start 2024-03-24T10:42:22.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f463b42a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4634102980 0x7f4634197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f463b42a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46341032a0 0x7f4634198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f463b42a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f463410bd50 0x7f463419c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f463b42a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f463419ccc0 con 0x7f4634102980 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f463b42a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f463419ce30 con 0x7f463410bd50 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f463b42a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f463419cff0 con 0x7f46341032a0 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f46391c6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4634102980 0x7f4634197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f46391c6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4634102980 0x7f4634197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55242/0 (socket says 172.21.15.23:55242) 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f46391c6640 1 -- 172.21.15.23:0/1037259573 learned_addr learned my addr 172.21.15.23:0/1037259573 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f46389c5640 1 --2- 172.21.15.23:0/1037259573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46341032a0 0x7f4634198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f46399c7640 1 --2- 172.21.15.23:0/1037259573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f463410bd50 0x7f463419c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f46391c6640 1 -- 172.21.15.23:0/1037259573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46341032a0 msgr2=0x7f4634198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f46391c6640 1 --2- 172.21.15.23:0/1037259573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46341032a0 0x7f4634198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f46391c6640 1 -- 172.21.15.23:0/1037259573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f463410bd50 msgr2=0x7f463419c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f46391c6640 1 --2- 172.21.15.23:0/1037259573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f463410bd50 0x7f463419c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f46391c6640 1 -- 172.21.15.23:0/1037259573 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f462c009900 con 0x7f4634102980 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.866+0000 7f46391c6640 1 --2- 172.21.15.23:0/1037259573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4634102980 0x7f4634197e00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f463000e9f0 tx=0x7f463000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.867+0000 7f462a7fc640 1 -- 172.21.15.23:0/1037259573 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630009700 con 0x7f4634102980 2024-03-24T10:42:22.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.867+0000 7f463b42a640 1 -- 172.21.15.23:0/1037259573 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4624009490 con 0x7f4634102980 2024-03-24T10:42:22.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.867+0000 7f462a7fc640 1 -- 172.21.15.23:0/1037259573 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f463000cd80 con 0x7f4634102980 2024-03-24T10:42:22.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.867+0000 7f462a7fc640 1 -- 172.21.15.23:0/1037259573 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630018920 con 0x7f4634102980 2024-03-24T10:42:22.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.867+0000 7f463b42a640 1 -- 172.21.15.23:0/1037259573 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46341a4970 con 0x7f4634102980 2024-03-24T10:42:22.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.869+0000 7f462a7fc640 1 -- 172.21.15.23:0/1037259573 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f4630018a80 con 0x7f4634102980 2024-03-24T10:42:22.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.869+0000 7f462a7fc640 1 --2- 172.21.15.23:0/1037259573 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f46200a2230 0x7f46200a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f46389c5640 1 --2- 172.21.15.23:0/1037259573 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f46200a2230 0x7f46200a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f462a7fc640 1 -- 172.21.15.23:0/1037259573 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f4630014070 con 0x7f4634102980 2024-03-24T10:42:22.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f463e00a740 1 Processor -- start 2024-03-24T10:42:22.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f463e00a740 1 -- start start 2024-03-24T10:42:22.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f46389c5640 1 --2- 172.21.15.23:0/1037259573 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f46200a2230 0x7f46200a46f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f462402c210 tx=0x7f46240056d0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f463e00a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32b3f0 0x55938c32b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f463e00a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32bd10 0x55938c3343b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f463e00a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 0x55938c336ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f463e00a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55938c299840 con 0x55938c32bd10 2024-03-24T10:42:22.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f463e00a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55938c296d70 con 0x55938c3348f0 2024-03-24T10:42:22.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f463e00a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55938c292550 con 0x55938c32b3f0 2024-03-24T10:42:22.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f46389c5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32bd10 0x55938c3343b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f46389c5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32bd10 0x55938c3343b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55250/0 (socket says 172.21.15.23:55250) 2024-03-24T10:42:22.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f46389c5640 1 -- 172.21.15.23:0/767654817 learned_addr learned my addr 172.21.15.23:0/767654817 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.870+0000 7f46399c7640 1 --2- 172.21.15.23:0/767654817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 0x55938c336ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.871+0000 7f46391c6640 1 --2- 172.21.15.23:0/767654817 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32b3f0 0x55938c32b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.871+0000 7f46399c7640 1 -- 172.21.15.23:0/767654817 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32b3f0 msgr2=0x55938c32b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.871+0000 7f46399c7640 1 --2- 172.21.15.23:0/767654817 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32b3f0 0x55938c32b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.871+0000 7f46399c7640 1 -- 172.21.15.23:0/767654817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32bd10 msgr2=0x55938c3343b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.871+0000 7f46399c7640 1 --2- 172.21.15.23:0/767654817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32bd10 0x55938c3343b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.871+0000 7f46399c7640 1 -- 172.21.15.23:0/767654817 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55938c227de0 con 0x55938c3348f0 2024-03-24T10:42:22.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.871+0000 7f46399c7640 1 --2- 172.21.15.23:0/767654817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 0x55938c336ce0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f462c00e550 tx=0x7f462c0095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f79b0e59eb41c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.871+0000 7f461f7fe640 1 -- 172.21.15.23:0/767654817 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c013040 con 0x55938c3348f0 2024-03-24T10:42:22.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.871+0000 7f461f7fe640 1 -- 172.21.15.23:0/767654817 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f462c003e40 con 0x55938c3348f0 2024-03-24T10:42:22.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.871+0000 7f461f7fe640 1 -- 172.21.15.23:0/767654817 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c00dbb0 con 0x55938c3348f0 2024-03-24T10:42:22.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.872+0000 7f463e00a740 1 -- 172.21.15.23:0/767654817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 msgr2=0x55938c336ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.872+0000 7f463e00a740 1 --2- 172.21.15.23:0/767654817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 0x55938c336ce0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f462c00e550 tx=0x7f462c0095c0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.872+0000 7f463e00a740 1 -- 172.21.15.23:0/767654817 shutdown_connections 2024-03-24T10:42:22.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.872+0000 7f463e00a740 1 --2- 172.21.15.23:0/767654817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 0x55938c336ce0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.872+0000 7f463e00a740 1 --2- 172.21.15.23:0/767654817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32bd10 0x55938c3343b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.872+0000 7f463e00a740 1 --2- 172.21.15.23:0/767654817 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32b3f0 0x55938c32b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.872+0000 7f463e00a740 1 -- 172.21.15.23:0/767654817 >> 172.21.15.23:0/767654817 conn(0x55938c325b20 msgr2=0x55938c325f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.872+0000 7f463e00a740 1 -- 172.21.15.23:0/767654817 shutdown_connections 2024-03-24T10:42:22.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.872+0000 7f463e00a740 1 -- 172.21.15.23:0/767654817 wait complete. 2024-03-24T10:42:22.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.872+0000 7f463e00a740 1 -- start start 2024-03-24T10:42:22.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.873+0000 7f463e00a740 1 Processor -- start 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.873+0000 7f463e00a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c32b3f0 0x55938c3319c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.873+0000 7f463e00a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32bd10 0x55938c331f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.873+0000 7f463e00a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c3348f0 0x55938c332440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.873+0000 7f463e00a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55938bf65430 con 0x55938c3348f0 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.873+0000 7f463e00a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55938bf21e00 con 0x55938c32b3f0 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.873+0000 7f463e00a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55938c217590 con 0x55938c32bd10 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.873+0000 7f463e00a740 10 client.0 _ll_register_callbacks cb 0x55938c286fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.873+0000 7f46399c7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c3348f0 0x55938c332440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.873+0000 7f46391c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c32b3f0 0x55938c3319c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.873+0000 7f46389c5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32bd10 0x55938c331f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.873+0000 7f46399c7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c3348f0 0x55938c332440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55266/0 (socket says 172.21.15.23:55266) 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.873+0000 7f46399c7640 1 -- 172.21.15.23:0/95814316 learned_addr learned my addr 172.21.15.23:0/95814316 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.873+0000 7f46391c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c32b3f0 0x55938c3319c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:45590/0 (socket says 172.21.15.23:45590) 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f46399c7640 1 -- 172.21.15.23:0/95814316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32bd10 msgr2=0x55938c331f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f46399c7640 1 --2- 172.21.15.23:0/95814316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32bd10 0x55938c331f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f46399c7640 1 -- 172.21.15.23:0/95814316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c32b3f0 msgr2=0x55938c3319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f46399c7640 1 --2- 172.21.15.23:0/95814316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c32b3f0 0x55938c3319c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f46399c7640 1 -- 172.21.15.23:0/95814316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f462c009d70 con 0x55938c3348f0 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f46389c5640 1 --2- 172.21.15.23:0/95814316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32bd10 0x55938c331f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f46399c7640 1 --2- 172.21.15.23:0/95814316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c3348f0 0x55938c332440 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f462c00eea0 tx=0x7f462c004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cb0dc9eec06dab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f461cff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f461cff9640 1 -- 172.21.15.23:0/95814316 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c00db40 con 0x55938c3348f0 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f461cff9640 1 -- 172.21.15.23:0/95814316 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f462c00dca0 con 0x55938c3348f0 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f463e00a740 1 -- 172.21.15.23:0/95814316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55938bf917e0 con 0x55938c3348f0 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f461cff9640 1 -- 172.21.15.23:0/95814316 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f462c00de60 con 0x55938c3348f0 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f463e00a740 10 client.16209 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f463e00a740 1 -- 172.21.15.23:0/95814316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55938c290bd0 con 0x55938c3348f0 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.874+0000 7f461cff9640 1 -- 172.21.15.23:0/95814316 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f462c004530 con 0x55938c3348f0 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.875+0000 7f461cff9640 1 -- 172.21.15.23:0/95814316 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f462c005450 con 0x55938c3348f0 2024-03-24T10:42:22.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.875+0000 7f463e00a740 10 client.16209 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.875+0000 7f463e00a740 10 client.16209 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.875+0000 7f463e00a740 20 client.16209 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.875+0000 7f463e00a740 1 --2- 172.21.15.23:0/95814316 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55938c344920 0x55938c346da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.875+0000 7f463e00a740 4 client.16209 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.875+0000 7f463e00a740 1 -- 172.21.15.23:0/95814316 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55938bf917e0 con 0x55938c344920 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.875+0000 7f46391c6640 1 --2- 172.21.15.23:0/95814316 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55938c344920 0x55938c346da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.875+0000 7f46391c6640 1 --2- 172.21.15.23:0/95814316 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55938c344920 0x55938c346da0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8ea9b3a014257a2 server_cookie=cb21efad95787181 in_seq=0 out_seq=0 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.875+0000 7f461cff9640 10 client.16209 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.876+0000 7f461cff9640 1 -- 172.21.15.23:0/95814316 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55938bf917e0 con 0x55938c344920 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.876+0000 7f461cff9640 10 client.16209 handle_command_reply: tid=0 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.876+0000 7f463e00a740 1 client.16209 shutdown 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.876+0000 7f463e00a740 1 -- 172.21.15.23:0/95814316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c3348f0 msgr2=0x55938c332440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.876+0000 7f463e00a740 1 --2- 172.21.15.23:0/95814316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c3348f0 0x55938c332440 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f462c00eea0 tx=0x7f462c004c70 comp rx=0 tx=0).stop 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.876+0000 7f463e00a740 1 -- 172.21.15.23:0/95814316 shutdown_connections 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.876+0000 7f463e00a740 1 --2- 172.21.15.23:0/95814316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c3348f0 0x55938c332440 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.876+0000 7f463e00a740 1 --2- 172.21.15.23:0/95814316 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55938c344920 0x55938c346da0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.876+0000 7f463e00a740 1 --2- 172.21.15.23:0/95814316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32bd10 0x55938c331f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.876+0000 7f463e00a740 1 --2- 172.21.15.23:0/95814316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c32b3f0 0x55938c3319c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.877+0000 7f463e00a740 1 -- 172.21.15.23:0/95814316 >> 172.21.15.23:0/95814316 conn(0x55938c325b20 msgr2=0x55938c327870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.876+0000 7f461cff9640 0 client.16209 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:22.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.877+0000 7f463e00a740 1 -- 172.21.15.23:0/95814316 shutdown_connections 2024-03-24T10:42:22.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.877+0000 7f463e00a740 1 -- 172.21.15.23:0/95814316 wait complete. 2024-03-24T10:42:22.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.877+0000 7f463e00a740 20 client.16209 trim_cache size 0 max 16384 2024-03-24T10:42:22.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f463e00a740 1 Processor -- start 2024-03-24T10:42:22.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f463e00a740 1 -- start start 2024-03-24T10:42:22.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f463e00a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32b3f0 0x55938c340180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f463e00a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32bd10 0x55938c332260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f463e00a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 0x55938c34c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f463e00a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55938c3256a0 con 0x55938c32b3f0 2024-03-24T10:42:22.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f463e00a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55938c3446d0 con 0x55938c3348f0 2024-03-24T10:42:22.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f463e00a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55938c340740 con 0x55938c32bd10 2024-03-24T10:42:22.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f46391c6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32b3f0 0x55938c340180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f46391c6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32b3f0 0x55938c340180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55274/0 (socket says 172.21.15.23:55274) 2024-03-24T10:42:22.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f46391c6640 1 -- 172.21.15.23:0/2014892472 learned_addr learned my addr 172.21.15.23:0/2014892472 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f46389c5640 1 --2- 172.21.15.23:0/2014892472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32bd10 0x55938c332260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f46399c7640 1 --2- 172.21.15.23:0/2014892472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 0x55938c34c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f46391c6640 1 -- 172.21.15.23:0/2014892472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32bd10 msgr2=0x55938c332260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f46391c6640 1 --2- 172.21.15.23:0/2014892472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32bd10 0x55938c332260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f46391c6640 1 -- 172.21.15.23:0/2014892472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 msgr2=0x55938c34c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f46391c6640 1 --2- 172.21.15.23:0/2014892472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 0x55938c34c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.891+0000 7f46391c6640 1 -- 172.21.15.23:0/2014892472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55938c22cc80 con 0x55938c32b3f0 2024-03-24T10:42:22.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.892+0000 7f46391c6640 1 --2- 172.21.15.23:0/2014892472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32b3f0 0x55938c340180 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f46300978b0 tx=0x7f463001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8cb8c2b93bfef0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.892+0000 7f461effd640 1 -- 172.21.15.23:0/2014892472 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630098f80 con 0x55938c32b3f0 2024-03-24T10:42:22.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.892+0000 7f461effd640 1 -- 172.21.15.23:0/2014892472 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46300991e0 con 0x55938c32b3f0 2024-03-24T10:42:22.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.892+0000 7f461effd640 1 -- 172.21.15.23:0/2014892472 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630097640 con 0x55938c32b3f0 2024-03-24T10:42:22.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.892+0000 7f463e00a740 1 -- 172.21.15.23:0/2014892472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32b3f0 msgr2=0x55938c340180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.892+0000 7f463e00a740 1 --2- 172.21.15.23:0/2014892472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32b3f0 0x55938c340180 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f46300978b0 tx=0x7f463001a040 comp rx=0 tx=0).stop 2024-03-24T10:42:22.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.892+0000 7f463e00a740 1 -- 172.21.15.23:0/2014892472 shutdown_connections 2024-03-24T10:42:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.892+0000 7f463e00a740 1 --2- 172.21.15.23:0/2014892472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 0x55938c34c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.892+0000 7f463e00a740 1 --2- 172.21.15.23:0/2014892472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32bd10 0x55938c332260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.892+0000 7f463e00a740 1 --2- 172.21.15.23:0/2014892472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32b3f0 0x55938c340180 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.892+0000 7f463e00a740 1 -- 172.21.15.23:0/2014892472 >> 172.21.15.23:0/2014892472 conn(0x55938c325b20 msgr2=0x55938c324cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f463e00a740 1 -- 172.21.15.23:0/2014892472 shutdown_connections 2024-03-24T10:42:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f463e00a740 1 -- 172.21.15.23:0/2014892472 wait complete. 2024-03-24T10:42:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f463e00a740 1 -- start start 2024-03-24T10:42:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f463e00a740 1 Processor -- start 2024-03-24T10:42:22.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f463e00a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32b3f0 0x55938c33cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f463e00a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32bd10 0x55938c337e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f463e00a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 0x55938c338390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f463e00a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55938c338a90 con 0x55938c32bd10 2024-03-24T10:42:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f463e00a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55938c338c00 con 0x55938c3348f0 2024-03-24T10:42:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f463e00a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55938c3a9ce0 con 0x55938c32b3f0 2024-03-24T10:42:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f463e00a740 10 client.0 _ll_register_callbacks cb 0x7f462c01b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f46389c5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32bd10 0x55938c337e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f46389c5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32bd10 0x55938c337e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55288/0 (socket says 172.21.15.23:55288) 2024-03-24T10:42:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f46389c5640 1 -- 172.21.15.23:0/1140904817 learned_addr learned my addr 172.21.15.23:0/1140904817 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f46391c6640 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32b3f0 0x55938c33cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.893+0000 7f46399c7640 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 0x55938c338390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f46389c5640 1 -- 172.21.15.23:0/1140904817 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32b3f0 msgr2=0x55938c33cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f46389c5640 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32b3f0 0x55938c33cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f46389c5640 1 -- 172.21.15.23:0/1140904817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 msgr2=0x55938c338390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f46389c5640 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 0x55938c338390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f46389c5640 1 -- 172.21.15.23:0/1140904817 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f463009d040 con 0x55938c32bd10 2024-03-24T10:42:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f46391c6640 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32b3f0 0x55938c33cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f46399c7640 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 0x55938c338390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f46389c5640 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32bd10 0x55938c337e30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x55938c32aec0 tx=0x7f4624005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb282cf048b121b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f461cff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f461cff9640 1 -- 172.21.15.23:0/1140904817 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4624051070 con 0x55938c32bd10 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f461cff9640 1 -- 172.21.15.23:0/1140904817 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f4624045040 con 0x55938c32bd10 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f463e00a740 1 -- 172.21.15.23:0/1140904817 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f462c003e40 con 0x55938c32bd10 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f461cff9640 1 -- 172.21.15.23:0/1140904817 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4624003a80 con 0x55938c32bd10 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f463e00a740 10 client.16215 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f463e00a740 1 -- 172.21.15.23:0/1140904817 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55938c252920 con 0x55938c32bd10 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.894+0000 7f461cff9640 1 -- 172.21.15.23:0/1140904817 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4624062020 con 0x55938c32bd10 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f461cff9640 1 -- 172.21.15.23:0/1140904817 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4624055410 con 0x55938c32bd10 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f463e00a740 10 client.16215 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f463e00a740 10 client.16215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f463e00a740 20 client.16215 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f463e00a740 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55938c3ab560 0x55938c3ad980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f463e00a740 4 client.16215 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f463e00a740 1 -- 172.21.15.23:0/1140904817 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55938bfce720 con 0x55938c3ab560 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f46391c6640 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55938c3ab560 0x55938c3ad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f461cff9640 10 client.16215 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f46391c6640 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55938c3ab560 0x55938c3ad980 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab7378fa264ba352 server_cookie=5ee68ce568bf92c8 in_seq=0 out_seq=0 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f461cff9640 1 -- 172.21.15.23:0/1140904817 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55938bfce720 con 0x55938c3ab560 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f461cff9640 10 client.16215 handle_command_reply: tid=0 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f463e00a740 1 client.16215 shutdown 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f463e00a740 1 -- 172.21.15.23:0/1140904817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32bd10 msgr2=0x55938c337e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.895+0000 7f463e00a740 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32bd10 0x55938c337e30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x55938c32aec0 tx=0x7f4624005f90 comp rx=0 tx=0).stop 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463e00a740 1 -- 172.21.15.23:0/1140904817 shutdown_connections 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463e00a740 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55938c3348f0 0x55938c338390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463e00a740 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55938c32bd10 0x55938c337e30 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463e00a740 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55938c3ab560 0x55938c3ad980 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463e00a740 1 --2- 172.21.15.23:0/1140904817 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55938c32b3f0 0x55938c33cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463e00a740 1 -- 172.21.15.23:0/1140904817 >> 172.21.15.23:0/1140904817 conn(0x55938c325b20 msgr2=0x55938c326b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f461cff9640 0 client.16215 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:22.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463e00a740 1 -- 172.21.15.23:0/1140904817 shutdown_connections 2024-03-24T10:42:22.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463e00a740 1 -- 172.21.15.23:0/1140904817 wait complete. 2024-03-24T10:42:22.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463e00a740 20 client.16215 trim_cache size 0 max 16384 2024-03-24T10:42:22.902 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:42:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463b42a640 1 -- 172.21.15.23:0/1037259573 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f46200a2230 msgr2=0x7f46200a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463b42a640 1 --2- 172.21.15.23:0/1037259573 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f46200a2230 0x7f46200a46f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f462402c210 tx=0x7f46240056d0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463b42a640 1 -- 172.21.15.23:0/1037259573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4634102980 msgr2=0x7f4634197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463b42a640 1 --2- 172.21.15.23:0/1037259573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4634102980 0x7f4634197e00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f463000e9f0 tx=0x7f463000eeb0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463b42a640 1 -- 172.21.15.23:0/1037259573 shutdown_connections 2024-03-24T10:42:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463b42a640 1 --2- 172.21.15.23:0/1037259573 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f46200a2230 0x7f46200a46f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463b42a640 1 --2- 172.21.15.23:0/1037259573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f463410bd50 0x7f463419c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463b42a640 1 --2- 172.21.15.23:0/1037259573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46341032a0 0x7f4634198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463b42a640 1 --2- 172.21.15.23:0/1037259573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4634102980 0x7f4634197e00 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.896+0000 7f463b42a640 1 -- 172.21.15.23:0/1037259573 >> 172.21.15.23:0/1037259573 conn(0x7f4634077770 msgr2=0x7f46340fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.897+0000 7f463b42a640 1 -- 172.21.15.23:0/1037259573 shutdown_connections 2024-03-24T10:42:22.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:22.897+0000 7f463b42a640 1 -- 172.21.15.23:0/1037259573 wait complete. 2024-03-24T10:42:23.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 92930 12.96 MB/sec warmup 40 sec latency 21.638 ms 2024-03-24T10:42:23.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:23 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/746002886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:42:23.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:23 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/746002886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:42:23.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:23 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/746002886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:42:23.908 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:42:23.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:42:23.909 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:42:24.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.025+0000 7fecdaa64640 1 -- 172.21.15.23:0/1213733414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd4101350 msgr2=0x7fecd4101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.025+0000 7fecdaa64640 1 --2- 172.21.15.23:0/1213733414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd4101350 0x7fecd4101730 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fecc4009930 tx=0x7fecc402bc30 comp rx=0 tx=0).stop 2024-03-24T10:42:24.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.025+0000 7fecdaa64640 1 -- 172.21.15.23:0/1213733414 shutdown_connections 2024-03-24T10:42:24.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.025+0000 7fecdaa64640 1 --2- 172.21.15.23:0/1213733414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecd41059d0 0x7fecd410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.025+0000 7fecdaa64640 1 --2- 172.21.15.23:0/1213733414 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecd4101d00 0x7fecd41052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.025+0000 7fecdaa64640 1 --2- 172.21.15.23:0/1213733414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd4101350 0x7fecd4101730 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.025+0000 7fecdaa64640 1 -- 172.21.15.23:0/1213733414 >> 172.21.15.23:0/1213733414 conn(0x7fecd4075950 msgr2=0x7fecd4075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.025+0000 7fecdaa64640 1 -- 172.21.15.23:0/1213733414 shutdown_connections 2024-03-24T10:42:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.025+0000 7fecdaa64640 1 -- 172.21.15.23:0/1213733414 wait complete. 2024-03-24T10:42:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecdaa64640 1 Processor -- start 2024-03-24T10:42:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecdaa64640 1 -- start start 2024-03-24T10:42:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecdaa64640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd4101350 0x7fecd4198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecdaa64640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecd41059d0 0x7fecd4199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecdaa64640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecd4199690 0x7fecd41a3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecdaa64640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fecd41a42e0 con 0x7fecd4101350 2024-03-24T10:42:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecdaa64640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fecd41a4450 con 0x7fecd4199690 2024-03-24T10:42:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecdaa64640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fecd41a4610 con 0x7fecd41059d0 2024-03-24T10:42:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecd8800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd4101350 0x7fecd4198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecd8800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd4101350 0x7fecd4198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55306/0 (socket says 172.21.15.23:55306) 2024-03-24T10:42:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecd8800640 1 -- 172.21.15.23:0/3936631755 learned_addr learned my addr 172.21.15.23:0/3936631755 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecd3fff640 1 --2- 172.21.15.23:0/3936631755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecd41059d0 0x7fecd4199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecd9001640 1 --2- 172.21.15.23:0/3936631755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecd4199690 0x7fecd41a3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecd8800640 1 -- 172.21.15.23:0/3936631755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecd41059d0 msgr2=0x7fecd4199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecd8800640 1 --2- 172.21.15.23:0/3936631755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecd41059d0 0x7fecd4199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecd8800640 1 -- 172.21.15.23:0/3936631755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecd4199690 msgr2=0x7fecd41a3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecd8800640 1 --2- 172.21.15.23:0/3936631755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecd4199690 0x7fecd41a3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.026+0000 7fecd8800640 1 -- 172.21.15.23:0/3936631755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc4009560 con 0x7fecd4101350 2024-03-24T10:42:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.027+0000 7fecd8800640 1 --2- 172.21.15.23:0/3936631755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd4101350 0x7fecd4198c10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fecc4000c00 tx=0x7fecc402cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.027+0000 7fecd1ffb640 1 -- 172.21.15.23:0/3936631755 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4004350 con 0x7fecd4101350 2024-03-24T10:42:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.027+0000 7fecd1ffb640 1 -- 172.21.15.23:0/3936631755 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecc40044b0 con 0x7fecd4101350 2024-03-24T10:42:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.027+0000 7fecdaa64640 1 -- 172.21.15.23:0/3936631755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecd41a4890 con 0x7fecd4101350 2024-03-24T10:42:24.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.027+0000 7fecd1ffb640 1 -- 172.21.15.23:0/3936631755 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4003d00 con 0x7fecd4101350 2024-03-24T10:42:24.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.027+0000 7fecdaa64640 1 -- 172.21.15.23:0/3936631755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecd41a4da0 con 0x7fecd4101350 2024-03-24T10:42:24.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.029+0000 7fecd1ffb640 1 -- 172.21.15.23:0/3936631755 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fecc4004df0 con 0x7fecd4101350 2024-03-24T10:42:24.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.029+0000 7fecdd640740 1 Processor -- start 2024-03-24T10:42:24.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.029+0000 7fecdd640740 1 -- start start 2024-03-24T10:42:24.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd1ffb640 1 --2- 172.21.15.23:0/3936631755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fecc00a2230 0x7fecc00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecdd640740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d6d760 0x55aa66d6db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecdd640740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d76720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecdd640740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d76c60 0x55aa66d79050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecdd640740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55aa66cdb530 con 0x55aa66d6e080 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecdd640740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55aa66cd8a60 con 0x55aa66d6d760 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecdd640740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55aa66cd4240 con 0x55aa66d76c60 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd8800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d6d760 0x55aa66d6db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd9001640 1 --2- 172.21.15.23:0/3936631755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fecc00a2230 0x7fecc00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd8800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d6d760 0x55aa66d6db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:45628/0 (socket says 172.21.15.23:45628) 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd8800640 1 -- 172.21.15.23:0/1406682617 learned_addr learned my addr 172.21.15.23:0/1406682617 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd1ffb640 1 -- 172.21.15.23:0/3936631755 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fecc40f0640 con 0x7fecd4101350 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd3fff640 1 --2- 172.21.15.23:0/1406682617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d76720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd9001640 1 --2- 172.21.15.23:0/1406682617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d76c60 0x55aa66d79050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd8800640 1 -- 172.21.15.23:0/1406682617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d76c60 msgr2=0x55aa66d79050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd8800640 1 --2- 172.21.15.23:0/1406682617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d76c60 0x55aa66d79050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd8800640 1 -- 172.21.15.23:0/1406682617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 msgr2=0x55aa66d76720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd8800640 1 --2- 172.21.15.23:0/1406682617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d76720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd8800640 1 -- 172.21.15.23:0/1406682617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa66cd28c0 con 0x55aa66d6d760 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd9001640 1 --2- 172.21.15.23:0/3936631755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fecc00a2230 0x7fecc00a46f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x55aa66d69020 tx=0x7feccc009b40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7fecd8800640 1 --2- 172.21.15.23:0/1406682617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d6d760 0x55aa66d6db40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fecc404dfb0 tx=0x7fecc4045f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c08a278c5a3a2db server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.030+0000 7feccaffd640 1 -- 172.21.15.23:0/1406682617 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4003e60 con 0x55aa66d6d760 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.031+0000 7feccaffd640 1 -- 172.21.15.23:0/1406682617 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecc40035f0 con 0x55aa66d6d760 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.031+0000 7feccaffd640 1 -- 172.21.15.23:0/1406682617 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc40be850 con 0x55aa66d6d760 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.031+0000 7fecdd640740 1 -- 172.21.15.23:0/1406682617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d6d760 msgr2=0x55aa66d6db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.031+0000 7fecdd640740 1 --2- 172.21.15.23:0/1406682617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d6d760 0x55aa66d6db40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fecc404dfb0 tx=0x7fecc4045f90 comp rx=0 tx=0).stop 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.031+0000 7fecdd640740 1 -- 172.21.15.23:0/1406682617 shutdown_connections 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.031+0000 7fecdd640740 1 --2- 172.21.15.23:0/1406682617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d76c60 0x55aa66d79050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.031+0000 7fecdd640740 1 --2- 172.21.15.23:0/1406682617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d76720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.031+0000 7fecdd640740 1 --2- 172.21.15.23:0/1406682617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d6d760 0x55aa66d6db40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.031+0000 7fecdd640740 1 -- 172.21.15.23:0/1406682617 >> 172.21.15.23:0/1406682617 conn(0x55aa66d67e90 msgr2=0x55aa66d68270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.031+0000 7fecdd640740 1 -- 172.21.15.23:0/1406682617 shutdown_connections 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.031+0000 7fecdd640740 1 -- 172.21.15.23:0/1406682617 wait complete. 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.032+0000 7fecdd640740 1 -- start start 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.032+0000 7fecdd640740 1 Processor -- start 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.032+0000 7fecdd640740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6d760 0x55aa66d73b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.032+0000 7fecdd640740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d74080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.032+0000 7fecdd640740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d76c60 0x55aa66d745c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.032+0000 7fecdd640740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55aa66c78f80 con 0x55aa66d6e080 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.032+0000 7fecdd640740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55aa669a74b0 con 0x55aa66d76c60 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.032+0000 7fecdd640740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55aa66c59ad0 con 0x55aa66d6d760 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.032+0000 7fecd3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d74080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.032+0000 7fecdd640740 10 client.0 _ll_register_callbacks cb 0x55aa66c8cb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.032+0000 7fecd3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d74080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55332/0 (socket says 172.21.15.23:55332) 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.032+0000 7fecd3fff640 1 -- 172.21.15.23:0/1048797316 learned_addr learned my addr 172.21.15.23:0/1048797316 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.032+0000 7fecd8800640 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6d760 0x55aa66d73b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.032+0000 7fecd9001640 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d76c60 0x55aa66d745c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecd3fff640 1 -- 172.21.15.23:0/1048797316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6d760 msgr2=0x55aa66d73b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecd3fff640 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6d760 0x55aa66d73b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecd3fff640 1 -- 172.21.15.23:0/1048797316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d76c60 msgr2=0x55aa66d745c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecd3fff640 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d76c60 0x55aa66d745c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecd3fff640 1 -- 172.21.15.23:0/1048797316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecd41a4da0 con 0x55aa66d6e080 2024-03-24T10:42:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecd8800640 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6d760 0x55aa66d73b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecd9001640 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d76c60 0x55aa66d745c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecd3fff640 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d74080 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fecbc013f60 tx=0x7fecbc013f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f589c61a7a52533 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecb3fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecb3fff640 1 -- 172.21.15.23:0/1048797316 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc011cf0 con 0x55aa66d6e080 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecb3fff640 1 -- 172.21.15.23:0/1048797316 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fecbc011e50 con 0x55aa66d6e080 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecdd640740 1 -- 172.21.15.23:0/1048797316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55aa66cf87a0 con 0x55aa66d6e080 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecb3fff640 1 -- 172.21.15.23:0/1048797316 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fecbc009e70 con 0x55aa66d6e080 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecdd640740 10 client.16227 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.033+0000 7fecdd640740 1 -- 172.21.15.23:0/1048797316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55aa66c8a6d0 con 0x55aa66d6e080 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.034+0000 7fecb3fff640 1 -- 172.21.15.23:0/1048797316 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55aa66cf87a0 con 0x55aa66d6e080 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.034+0000 7fecb3fff640 1 -- 172.21.15.23:0/1048797316 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fecbc00f5c0 con 0x55aa66d6e080 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.034+0000 7fecdd640740 10 client.16227 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.034+0000 7fecdd640740 10 client.16227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.034+0000 7fecdd640740 20 client.16227 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.034+0000 7fecdd640740 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55aa66d8f2e0 0x55aa66d91700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.034+0000 7fecdd640740 4 client.16227 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:24.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.034+0000 7fecdd640740 1 -- 172.21.15.23:0/1048797316 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa66cca1f0 con 0x55aa66d8f2e0 2024-03-24T10:42:24.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.034+0000 7fecd8800640 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55aa66d8f2e0 0x55aa66d91700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.034+0000 7fecd8800640 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55aa66d8f2e0 0x55aa66d91700 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c30e50a9356d246b server_cookie=a94a91a8981a0f0c in_seq=0 out_seq=0 2024-03-24T10:42:24.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.034+0000 7fecb3fff640 10 client.16227 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:24.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.035+0000 7fecb3fff640 1 -- 172.21.15.23:0/1048797316 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55aa66cca1f0 con 0x55aa66d8f2e0 2024-03-24T10:42:24.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.035+0000 7fecb3fff640 10 client.16227 handle_command_reply: tid=0 2024-03-24T10:42:24.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.035+0000 7fecdd640740 1 client.16227 shutdown 2024-03-24T10:42:24.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.035+0000 7fecdd640740 1 -- 172.21.15.23:0/1048797316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 msgr2=0x55aa66d74080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.035+0000 7fecdd640740 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d74080 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fecbc013f60 tx=0x7fecbc013f90 comp rx=0 tx=0).stop 2024-03-24T10:42:24.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.036+0000 7fecdd640740 1 -- 172.21.15.23:0/1048797316 shutdown_connections 2024-03-24T10:42:24.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.036+0000 7fecdd640740 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d76c60 0x55aa66d745c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.036+0000 7fecdd640740 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d74080 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.036+0000 7fecdd640740 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55aa66d8f2e0 0x55aa66d91700 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.036+0000 7fecdd640740 1 --2- 172.21.15.23:0/1048797316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6d760 0x55aa66d73b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.036+0000 7fecb3fff640 0 client.16227 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:24.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.036+0000 7fecdd640740 1 -- 172.21.15.23:0/1048797316 >> 172.21.15.23:0/1048797316 conn(0x55aa66d67e90 msgr2=0x55aa66d69be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:24.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.036+0000 7fecdd640740 1 -- 172.21.15.23:0/1048797316 shutdown_connections 2024-03-24T10:42:24.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.036+0000 7fecdd640740 1 -- 172.21.15.23:0/1048797316 wait complete. 2024-03-24T10:42:24.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.036+0000 7fecdd640740 20 client.16227 trim_cache size 0 max 16384 2024-03-24T10:42:24.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.054+0000 7fecdd640740 1 Processor -- start 2024-03-24T10:42:24.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.054+0000 7fecdd640740 1 -- start start 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.054+0000 7fecdd640740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6d760 0x55aa66d76270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.054+0000 7fecdd640740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d743e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.054+0000 7fecdd640740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d76c60 0x55aa66d97010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.054+0000 7fecdd640740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55aa66d73040 con 0x55aa66d6e080 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.054+0000 7fecdd640740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55aa66d66350 con 0x55aa66d76c60 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.054+0000 7fecdd640740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55aa66d63650 con 0x55aa66d6d760 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.054+0000 7fecd3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d743e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.054+0000 7fecd3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d743e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55338/0 (socket says 172.21.15.23:55338) 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.054+0000 7fecd3fff640 1 -- 172.21.15.23:0/108655996 learned_addr learned my addr 172.21.15.23:0/108655996 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.054+0000 7fecd8800640 1 --2- 172.21.15.23:0/108655996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6d760 0x55aa66d76270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.054+0000 7fecd9001640 1 --2- 172.21.15.23:0/108655996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d76c60 0x55aa66d97010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.055+0000 7fecd8800640 1 -- 172.21.15.23:0/108655996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d76c60 msgr2=0x55aa66d97010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.055+0000 7fecd8800640 1 --2- 172.21.15.23:0/108655996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d76c60 0x55aa66d97010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.055+0000 7fecd8800640 1 -- 172.21.15.23:0/108655996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 msgr2=0x55aa66d743e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.055+0000 7fecd8800640 1 --2- 172.21.15.23:0/108655996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d743e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.055+0000 7fecd8800640 1 -- 172.21.15.23:0/108655996 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa66c6ec60 con 0x55aa66d6d760 2024-03-24T10:42:24.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.055+0000 7fecd3fff640 1 --2- 172.21.15.23:0/108655996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:24.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.055+0000 7fecd9001640 1 --2- 172.21.15.23:0/108655996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d76c60 0x55aa66d97010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:24.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.055+0000 7fecd8800640 1 --2- 172.21.15.23:0/108655996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6d760 0x55aa66d76270 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fecc40bcff0 tx=0x7fecc402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aaacc7a2401c5716 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:24.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.055+0000 7fecca7fc640 1 -- 172.21.15.23:0/108655996 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc40c14c0 con 0x55aa66d6d760 2024-03-24T10:42:24.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.055+0000 7fecca7fc640 1 -- 172.21.15.23:0/108655996 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecc40d1040 con 0x55aa66d6d760 2024-03-24T10:42:24.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.055+0000 7fecca7fc640 1 -- 172.21.15.23:0/108655996 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc40edf80 con 0x55aa66d6d760 2024-03-24T10:42:24.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.056+0000 7fecdd640740 1 -- 172.21.15.23:0/108655996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6d760 msgr2=0x55aa66d76270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.056+0000 7fecdd640740 1 --2- 172.21.15.23:0/108655996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6d760 0x55aa66d76270 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fecc40bcff0 tx=0x7fecc402bc30 comp rx=0 tx=0).stop 2024-03-24T10:42:24.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.056+0000 7fecdd640740 1 -- 172.21.15.23:0/108655996 shutdown_connections 2024-03-24T10:42:24.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.056+0000 7fecdd640740 1 --2- 172.21.15.23:0/108655996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d76c60 0x55aa66d97010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.056+0000 7fecdd640740 1 --2- 172.21.15.23:0/108655996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d6e080 0x55aa66d743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.056+0000 7fecdd640740 1 --2- 172.21.15.23:0/108655996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6d760 0x55aa66d76270 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.056+0000 7fecdd640740 1 -- 172.21.15.23:0/108655996 >> 172.21.15.23:0/108655996 conn(0x55aa66d67e90 msgr2=0x55aa66d66660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:24.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.056+0000 7fecdd640740 1 -- 172.21.15.23:0/108655996 shutdown_connections 2024-03-24T10:42:24.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.056+0000 7fecdd640740 1 -- 172.21.15.23:0/108655996 wait complete. 2024-03-24T10:42:24.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.056+0000 7fecdd640740 1 -- start start 2024-03-24T10:42:24.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.056+0000 7fecdd640740 1 Processor -- start 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecdd640740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d6d760 0x55aa66d87130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecdd640740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6e080 0x55aa66d84100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecdd640740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d76c60 0x55aa66d84640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecdd640740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55aa66d87800 con 0x55aa66d76c60 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecdd640740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55aa66d87970 con 0x55aa66d6d760 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecdd640740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55aa66d84b80 con 0x55aa66d6e080 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecdd640740 10 client.0 _ll_register_callbacks cb 0x7fecbc013aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecd9001640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d76c60 0x55aa66d84640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecd8800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d6d760 0x55aa66d87130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecd9001640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d76c60 0x55aa66d84640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55340/0 (socket says 172.21.15.23:55340) 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecd9001640 1 -- 172.21.15.23:0/2336409881 learned_addr learned my addr 172.21.15.23:0/2336409881 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecd3fff640 1 --2- 172.21.15.23:0/2336409881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6e080 0x55aa66d84100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecd9001640 1 -- 172.21.15.23:0/2336409881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6e080 msgr2=0x55aa66d84100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecd9001640 1 --2- 172.21.15.23:0/2336409881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6e080 0x55aa66d84100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecd9001640 1 -- 172.21.15.23:0/2336409881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d6d760 msgr2=0x55aa66d87130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecd9001640 1 --2- 172.21.15.23:0/2336409881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d6d760 0x55aa66d87130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecd9001640 1 -- 172.21.15.23:0/2336409881 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa66c8a6d0 con 0x55aa66d76c60 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecd8800640 1 --2- 172.21.15.23:0/2336409881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d6d760 0x55aa66d87130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecd3fff640 1 --2- 172.21.15.23:0/2336409881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6e080 0x55aa66d84100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecd9001640 1 --2- 172.21.15.23:0/2336409881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d76c60 0x55aa66d84640 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7feccc009b10 tx=0x7feccc009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e207726d25dd404 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecb3fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecb3fff640 1 -- 172.21.15.23:0/2336409881 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feccc023040 con 0x55aa66d76c60 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecb3fff640 1 -- 172.21.15.23:0/2336409881 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7feccc004b20 con 0x55aa66d76c60 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.057+0000 7fecdd640740 1 -- 172.21.15.23:0/2336409881 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecc40035f0 con 0x55aa66d76c60 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.058+0000 7fecb3fff640 1 -- 172.21.15.23:0/2336409881 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7feccc004ce0 con 0x55aa66d76c60 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.058+0000 7fecdd640740 10 client.16233 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.058+0000 7fecdd640740 1 -- 172.21.15.23:0/2336409881 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x7feccc01a040 con 0x55aa66d76c60 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.058+0000 7fecb3fff640 1 -- 172.21.15.23:0/2336409881 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecc40035f0 con 0x55aa66d76c60 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.058+0000 7fecb3fff640 1 -- 172.21.15.23:0/2336409881 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7feccc026030 con 0x55aa66d76c60 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.058+0000 7fecdd640740 10 client.16233 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.058+0000 7fecdd640740 10 client.16233 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:24.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.058+0000 7fecdd640740 20 client.16233 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.058+0000 7fecdd640740 1 --2- 172.21.15.23:0/2336409881 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55aa66df69d0 0x55aa66df8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.058+0000 7fecdd640740 4 client.16233 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.058+0000 7fecdd640740 1 -- 172.21.15.23:0/2336409881 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55aa669d3860 con 0x55aa66df69d0 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.058+0000 7fecd8800640 1 --2- 172.21.15.23:0/2336409881 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55aa66df69d0 0x55aa66df8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.059+0000 7fecd8800640 1 --2- 172.21.15.23:0/2336409881 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55aa66df69d0 0x55aa66df8df0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7442452b848f0092 server_cookie=ec4fbbab1c1884f6 in_seq=0 out_seq=0 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.059+0000 7fecb3fff640 10 client.16233 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecb3fff640 1 -- 172.21.15.23:0/2336409881 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55aa669d3860 con 0x55aa66df69d0 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecb3fff640 10 client.16233 handle_command_reply: tid=0 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecdd640740 1 client.16233 shutdown 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecdd640740 1 -- 172.21.15.23:0/2336409881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d76c60 msgr2=0x55aa66d84640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecdd640740 1 --2- 172.21.15.23:0/2336409881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d76c60 0x55aa66d84640 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7feccc009b10 tx=0x7feccc009600 comp rx=0 tx=0).stop 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecdd640740 1 -- 172.21.15.23:0/2336409881 shutdown_connections 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecdd640740 1 --2- 172.21.15.23:0/2336409881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa66d76c60 0x55aa66d84640 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecdd640740 1 --2- 172.21.15.23:0/2336409881 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55aa66df69d0 0x55aa66df8df0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecdd640740 1 --2- 172.21.15.23:0/2336409881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aa66d6e080 0x55aa66d84100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecdd640740 1 --2- 172.21.15.23:0/2336409881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aa66d6d760 0x55aa66d87130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecdd640740 1 -- 172.21.15.23:0/2336409881 >> 172.21.15.23:0/2336409881 conn(0x55aa66d67e90 msgr2=0x55aa66d66ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:24.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecb3fff640 0 client.16233 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:24.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecdd640740 1 -- 172.21.15.23:0/2336409881 shutdown_connections 2024-03-24T10:42:24.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecdd640740 1 -- 172.21.15.23:0/2336409881 wait complete. 2024-03-24T10:42:24.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.060+0000 7fecdd640740 20 client.16233 trim_cache size 0 max 16384 2024-03-24T10:42:24.065 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:42:24.065 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:42:24.065 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "cd982041-987e-4b95-b540-cbcee4bebb0a", 2024-03-24T10:42:24.065 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:42:24.065 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:42:24.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.061+0000 7fecdaa64640 1 -- 172.21.15.23:0/3936631755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fecc00a2230 msgr2=0x7fecc00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.061+0000 7fecdaa64640 1 --2- 172.21.15.23:0/3936631755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fecc00a2230 0x7fecc00a46f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x55aa66d69020 tx=0x7feccc009b40 comp rx=0 tx=0).stop 2024-03-24T10:42:24.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.061+0000 7fecdaa64640 1 -- 172.21.15.23:0/3936631755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd4101350 msgr2=0x7fecd4198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.061+0000 7fecdaa64640 1 --2- 172.21.15.23:0/3936631755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd4101350 0x7fecd4198c10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fecc4000c00 tx=0x7fecc402cd80 comp rx=0 tx=0).stop 2024-03-24T10:42:24.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.061+0000 7fecdaa64640 1 -- 172.21.15.23:0/3936631755 shutdown_connections 2024-03-24T10:42:24.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.061+0000 7fecdaa64640 1 --2- 172.21.15.23:0/3936631755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fecc00a2230 0x7fecc00a46f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.061+0000 7fecdaa64640 1 --2- 172.21.15.23:0/3936631755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fecd4199690 0x7fecd41a3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.061+0000 7fecdaa64640 1 --2- 172.21.15.23:0/3936631755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecd41059d0 0x7fecd4199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.061+0000 7fecdaa64640 1 --2- 172.21.15.23:0/3936631755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd4101350 0x7fecd4198c10 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.061+0000 7fecdaa64640 1 -- 172.21.15.23:0/3936631755 >> 172.21.15.23:0/3936631755 conn(0x7fecd4075950 msgr2=0x7fecd40fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:24.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.061+0000 7fecdaa64640 1 -- 172.21.15.23:0/3936631755 shutdown_connections 2024-03-24T10:42:24.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.061+0000 7fecdaa64640 1 -- 172.21.15.23:0/3936631755 wait complete. 2024-03-24T10:42:24.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:42:24.072 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:42:24.112 INFO:tasks.workunit.client.0.smithi023.stdout: 1 95898 13.07 MB/sec warmup 41 sec latency 21.374 ms 2024-03-24T10:42:24.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.186+0000 7f076a055640 1 -- 172.21.15.23:0/3371754286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07641005b0 msgr2=0x7f0764100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.186+0000 7f076a055640 1 --2- 172.21.15.23:0/3371754286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07641005b0 0x7f0764100a30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f074c009930 tx=0x7f074c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:42:24.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.186+0000 7f076a055640 1 -- 172.21.15.23:0/3371754286 shutdown_connections 2024-03-24T10:42:24.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.186+0000 7f076a055640 1 --2- 172.21.15.23:0/3371754286 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0764100f70 0x7f076410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.186+0000 7f076a055640 1 --2- 172.21.15.23:0/3371754286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07641005b0 0x7f0764100a30 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.186+0000 7f076a055640 1 --2- 172.21.15.23:0/3371754286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641055d0 0x7f07641059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.186+0000 7f076a055640 1 -- 172.21.15.23:0/3371754286 >> 172.21.15.23:0/3371754286 conn(0x7f07640776a0 msgr2=0x7f0764077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:24.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.186+0000 7f076a055640 1 -- 172.21.15.23:0/3371754286 shutdown_connections 2024-03-24T10:42:24.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.186+0000 7f076a055640 1 -- 172.21.15.23:0/3371754286 wait complete. 2024-03-24T10:42:24.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f076a055640 1 Processor -- start 2024-03-24T10:42:24.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f076a055640 1 -- start start 2024-03-24T10:42:24.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f076a055640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07641005b0 0x7f076419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f076a055640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0764100f70 0x7f076419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f076a055640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641055d0 0x7f07641969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f076a055640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0764197030 con 0x7f07641005b0 2024-03-24T10:42:24.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f076a055640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f07641971a0 con 0x7f07641055d0 2024-03-24T10:42:24.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f076a055640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0764197360 con 0x7f0764100f70 2024-03-24T10:42:24.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f07637fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07641005b0 0x7f076419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f07637fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07641005b0 0x7f076419c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55370/0 (socket says 172.21.15.23:55370) 2024-03-24T10:42:24.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f07637fe640 1 -- 172.21.15.23:0/2100371651 learned_addr learned my addr 172.21.15.23:0/2100371651 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:24.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f075bfff640 1 --2- 172.21.15.23:0/2100371651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0764100f70 0x7f076419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f0763fff640 1 --2- 172.21.15.23:0/2100371651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641055d0 0x7f07641969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f07637fe640 1 -- 172.21.15.23:0/2100371651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0764100f70 msgr2=0x7f076419cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f07637fe640 1 --2- 172.21.15.23:0/2100371651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0764100f70 0x7f076419cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f07637fe640 1 -- 172.21.15.23:0/2100371651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641055d0 msgr2=0x7f07641969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f07637fe640 1 --2- 172.21.15.23:0/2100371651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641055d0 0x7f07641969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f07637fe640 1 -- 172.21.15.23:0/2100371651 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0754009490 con 0x7f07641005b0 2024-03-24T10:42:24.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.187+0000 7f07637fe640 1 --2- 172.21.15.23:0/2100371651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07641005b0 0x7f076419c830 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f075400e9c0 tx=0x7f075400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:24.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.188+0000 7f07617fa640 1 -- 172.21.15.23:0/2100371651 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075400cde0 con 0x7f07641005b0 2024-03-24T10:42:24.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.188+0000 7f07617fa640 1 -- 172.21.15.23:0/2100371651 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0754010910 con 0x7f07641005b0 2024-03-24T10:42:24.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.188+0000 7f07617fa640 1 -- 172.21.15.23:0/2100371651 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0754018a10 con 0x7f07641005b0 2024-03-24T10:42:24.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.188+0000 7f076a055640 1 -- 172.21.15.23:0/2100371651 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f074c009560 con 0x7f07641005b0 2024-03-24T10:42:24.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.188+0000 7f076a055640 1 -- 172.21.15.23:0/2100371651 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07641a36f0 con 0x7f07641005b0 2024-03-24T10:42:24.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.190+0000 7f07617fa640 1 -- 172.21.15.23:0/2100371651 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f0754010a80 con 0x7f07641005b0 2024-03-24T10:42:24.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.190+0000 7f076cc3b740 1 Processor -- start 2024-03-24T10:42:24.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.190+0000 7f076cc3b740 1 -- start start 2024-03-24T10:42:24.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.190+0000 7f07617fa640 1 --2- 172.21.15.23:0/2100371651 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f07500a2230 0x7f07500a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.190+0000 7f076cc3b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2816113e0 0x55c2816117c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.190+0000 7f076cc3b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c281611d00 0x55c28161a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.190+0000 7f0763fff640 1 --2- 172.21.15.23:0/2100371651 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f07500a2230 0x7f07500a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.190+0000 7f076cc3b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 0x55c28161ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.190+0000 7f076cc3b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c281533f30 con 0x55c2816113e0 2024-03-24T10:42:24.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.190+0000 7f076cc3b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c28157d810 con 0x55c281611d00 2024-03-24T10:42:24.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.190+0000 7f076cc3b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c28157ad40 con 0x55c28161a8e0 2024-03-24T10:42:24.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f07617fa640 1 -- 172.21.15.23:0/2100371651 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0754014070 con 0x7f07641005b0 2024-03-24T10:42:24.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f07637fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2816113e0 0x55c2816117c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f07637fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2816113e0 0x55c2816117c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55376/0 (socket says 172.21.15.23:55376) 2024-03-24T10:42:24.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f07637fe640 1 -- 172.21.15.23:0/2922530517 learned_addr learned my addr 172.21.15.23:0/2922530517 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:24.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f075bfff640 1 --2- 172.21.15.23:0/2922530517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c281611d00 0x55c28161a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f0763fff640 1 --2- 172.21.15.23:0/2922530517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 0x55c28161ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f07637fe640 1 -- 172.21.15.23:0/2922530517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 msgr2=0x55c28161ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f07637fe640 1 --2- 172.21.15.23:0/2922530517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 0x55c28161ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f07637fe640 1 -- 172.21.15.23:0/2922530517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c281611d00 msgr2=0x55c28161a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f07637fe640 1 --2- 172.21.15.23:0/2922530517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c281611d00 0x55c28161a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f07637fe640 1 -- 172.21.15.23:0/2922530517 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c281537ea0 con 0x55c2816113e0 2024-03-24T10:42:24.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f075bfff640 1 --2- 172.21.15.23:0/2922530517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c281611d00 0x55c28161a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:24.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f0763fff640 1 --2- 172.21.15.23:0/2100371651 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f07500a2230 0x7f07500a46f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f075c005fd0 tx=0x7f075c009e00 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:24.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f07637fe640 1 --2- 172.21.15.23:0/2922530517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2816113e0 0x55c2816117c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f07540cbf80 tx=0x7f075409ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56c7e99b628d1ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:24.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f0759ffb640 1 -- 172.21.15.23:0/2922530517 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0754019d70 con 0x55c2816113e0 2024-03-24T10:42:24.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f0759ffb640 1 -- 172.21.15.23:0/2922530517 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f075409b170 con 0x55c2816113e0 2024-03-24T10:42:24.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.191+0000 7f0759ffb640 1 -- 172.21.15.23:0/2922530517 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07540977e0 con 0x55c2816113e0 2024-03-24T10:42:24.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.192+0000 7f076cc3b740 1 -- 172.21.15.23:0/2922530517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2816113e0 msgr2=0x55c2816117c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.192+0000 7f076cc3b740 1 --2- 172.21.15.23:0/2922530517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2816113e0 0x55c2816117c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f07540cbf80 tx=0x7f075409ab90 comp rx=0 tx=0).stop 2024-03-24T10:42:24.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.192+0000 7f076cc3b740 1 -- 172.21.15.23:0/2922530517 shutdown_connections 2024-03-24T10:42:24.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.192+0000 7f076cc3b740 1 --2- 172.21.15.23:0/2922530517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 0x55c28161ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.192+0000 7f076cc3b740 1 --2- 172.21.15.23:0/2922530517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c281611d00 0x55c28161a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.192+0000 7f076cc3b740 1 --2- 172.21.15.23:0/2922530517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2816113e0 0x55c2816117c0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.192+0000 7f076cc3b740 1 -- 172.21.15.23:0/2922530517 >> 172.21.15.23:0/2922530517 conn(0x55c28160bb10 msgr2=0x55c28160bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:24.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.192+0000 7f076cc3b740 1 -- 172.21.15.23:0/2922530517 shutdown_connections 2024-03-24T10:42:24.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.192+0000 7f076cc3b740 1 -- 172.21.15.23:0/2922530517 wait complete. 2024-03-24T10:42:24.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.193+0000 7f076cc3b740 1 -- start start 2024-03-24T10:42:24.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.193+0000 7f076cc3b740 1 Processor -- start 2024-03-24T10:42:24.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.193+0000 7f076cc3b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2816113e0 0x55c28160adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.193+0000 7f076cc3b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c281611d00 0x55c2816092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.193+0000 7f076cc3b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 0x55c2816098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.193+0000 7f076cc3b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c281576520 con 0x55c281611d00 2024-03-24T10:42:24.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.193+0000 7f076cc3b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c281249430 con 0x55c2816113e0 2024-03-24T10:42:24.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.193+0000 7f076cc3b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c281205e00 con 0x55c28161a8e0 2024-03-24T10:42:24.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.193+0000 7f076cc3b740 10 client.0 _ll_register_callbacks cb 0x55c281514d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:24.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.193+0000 7f075bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c281611d00 0x55c2816092b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.193+0000 7f07637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2816113e0 0x55c28160adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.193+0000 7f07637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2816113e0 0x55c28160adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:45704/0 (socket says 172.21.15.23:45704) 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.193+0000 7f07637fe640 1 -- 172.21.15.23:0/1360140515 learned_addr learned my addr 172.21.15.23:0/1360140515 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.193+0000 7f0763fff640 1 --2- 172.21.15.23:0/1360140515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 0x55c2816098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f075bfff640 1 -- 172.21.15.23:0/1360140515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 msgr2=0x55c2816098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f075bfff640 1 --2- 172.21.15.23:0/1360140515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 0x55c2816098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f075bfff640 1 -- 172.21.15.23:0/1360140515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2816113e0 msgr2=0x55c28160adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f075bfff640 1 --2- 172.21.15.23:0/1360140515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2816113e0 0x55c28160adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f075bfff640 1 -- 172.21.15.23:0/1360140515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07641a36f0 con 0x55c281611d00 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f07637fe640 1 --2- 172.21.15.23:0/1360140515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2816113e0 0x55c28160adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f0763fff640 1 --2- 172.21.15.23:0/1360140515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 0x55c2816098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f075bfff640 1 --2- 172.21.15.23:0/1360140515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c281611d00 0x55c2816092b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f074c009230 tx=0x7f074c02ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2330685cf16627a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f0742ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f0742ffd640 1 -- 172.21.15.23:0/1360140515 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c004040 con 0x55c281611d00 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f0742ffd640 1 -- 172.21.15.23:0/1360140515 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f074c0041a0 con 0x55c281611d00 2024-03-24T10:42:24.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f076cc3b740 1 -- 172.21.15.23:0/1360140515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c2812757e0 con 0x55c281611d00 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f0742ffd640 1 -- 172.21.15.23:0/1360140515 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f074c004360 con 0x55c281611d00 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f076cc3b740 10 client.16245 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f076cc3b740 1 -- 172.21.15.23:0/1360140515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c281574ba0 con 0x55c281611d00 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f0742ffd640 1 -- 172.21.15.23:0/1360140515 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c2812757e0 con 0x55c281611d00 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.194+0000 7f0742ffd640 1 -- 172.21.15.23:0/1360140515 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f074c03c040 con 0x55c281611d00 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.195+0000 7f076cc3b740 10 client.16245 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.195+0000 7f076cc3b740 10 client.16245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.195+0000 7f076cc3b740 20 client.16245 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.195+0000 7f076cc3b740 1 --2- 172.21.15.23:0/1360140515 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c28162cee0 0x55c28162f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.195+0000 7f076cc3b740 4 client.16245 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.195+0000 7f076cc3b740 1 -- 172.21.15.23:0/1360140515 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2812b2720 con 0x55c28162cee0 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.195+0000 7f07637fe640 1 --2- 172.21.15.23:0/1360140515 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c28162cee0 0x55c28162f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.195+0000 7f0742ffd640 10 client.16245 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.195+0000 7f07637fe640 1 --2- 172.21.15.23:0/1360140515 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c28162cee0 0x55c28162f300 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab0b7bc651aad5df server_cookie=6f6b1479006f9082 in_seq=0 out_seq=0 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.196+0000 7f0742ffd640 1 -- 172.21.15.23:0/1360140515 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c2812b2720 con 0x55c28162cee0 2024-03-24T10:42:24.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.196+0000 7f0742ffd640 10 client.16245 handle_command_reply: tid=0 2024-03-24T10:42:24.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.196+0000 7f076cc3b740 1 client.16245 shutdown 2024-03-24T10:42:24.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.196+0000 7f076cc3b740 1 -- 172.21.15.23:0/1360140515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c281611d00 msgr2=0x55c2816092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.196+0000 7f076cc3b740 1 --2- 172.21.15.23:0/1360140515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c281611d00 0x55c2816092b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f074c009230 tx=0x7f074c02ca10 comp rx=0 tx=0).stop 2024-03-24T10:42:24.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.196+0000 7f076cc3b740 1 -- 172.21.15.23:0/1360140515 shutdown_connections 2024-03-24T10:42:24.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.197+0000 7f076cc3b740 1 --2- 172.21.15.23:0/1360140515 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c28162cee0 0x55c28162f300 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.197+0000 7f076cc3b740 1 --2- 172.21.15.23:0/1360140515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 0x55c2816098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.197+0000 7f0742ffd640 0 client.16245 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:24.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.197+0000 7f076cc3b740 1 --2- 172.21.15.23:0/1360140515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c281611d00 0x55c2816092b0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.197+0000 7f076cc3b740 1 --2- 172.21.15.23:0/1360140515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c2816113e0 0x55c28160adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.197+0000 7f076cc3b740 1 -- 172.21.15.23:0/1360140515 >> 172.21.15.23:0/1360140515 conn(0x55c28160bb10 msgr2=0x55c28160cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:24.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.197+0000 7f076cc3b740 1 -- 172.21.15.23:0/1360140515 shutdown_connections 2024-03-24T10:42:24.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.197+0000 7f076cc3b740 1 -- 172.21.15.23:0/1360140515 wait complete. 2024-03-24T10:42:24.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.197+0000 7f076cc3b740 20 client.16245 trim_cache size 0 max 16384 2024-03-24T10:42:24.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f076cc3b740 1 Processor -- start 2024-03-24T10:42:24.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f076cc3b740 1 -- start start 2024-03-24T10:42:24.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f076cc3b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2816113e0 0x55c28161a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f076cc3b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c281611d00 0x55c281634bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f076cc3b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c28161a8e0 0x55c281614ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f076cc3b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c2814fb430 con 0x55c281611d00 2024-03-24T10:42:24.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f076cc3b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c2812b2750 con 0x55c28161a8e0 2024-03-24T10:42:24.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f076cc3b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c281605240 con 0x55c2816113e0 2024-03-24T10:42:24.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f075bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c281611d00 0x55c281634bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f075bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c281611d00 0x55c281634bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55388/0 (socket says 172.21.15.23:55388) 2024-03-24T10:42:24.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f075bfff640 1 -- 172.21.15.23:0/3196784788 learned_addr learned my addr 172.21.15.23:0/3196784788 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:24.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f0763fff640 1 --2- 172.21.15.23:0/3196784788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c28161a8e0 0x55c281614ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f07637fe640 1 --2- 172.21.15.23:0/3196784788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2816113e0 0x55c28161a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f075bfff640 1 -- 172.21.15.23:0/3196784788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2816113e0 msgr2=0x55c28161a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f075bfff640 1 --2- 172.21.15.23:0/3196784788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2816113e0 0x55c28161a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f075bfff640 1 -- 172.21.15.23:0/3196784788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c28161a8e0 msgr2=0x55c281614ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f075bfff640 1 --2- 172.21.15.23:0/3196784788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c28161a8e0 0x55c281614ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f075bfff640 1 -- 172.21.15.23:0/3196784788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c281510b20 con 0x55c281611d00 2024-03-24T10:42:24.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.217+0000 7f075bfff640 1 --2- 172.21.15.23:0/3196784788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c281611d00 0x55c281634bf0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f074c02c170 tx=0x7f074c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa96a1456413eac3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:24.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.218+0000 7f07597fa640 1 -- 172.21.15.23:0/3196784788 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c0035f0 con 0x55c281611d00 2024-03-24T10:42:24.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.218+0000 7f07597fa640 1 -- 172.21.15.23:0/3196784788 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f074c004380 con 0x55c281611d00 2024-03-24T10:42:24.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.218+0000 7f07597fa640 1 -- 172.21.15.23:0/3196784788 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c005500 con 0x55c281611d00 2024-03-24T10:42:24.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.218+0000 7f076cc3b740 1 -- 172.21.15.23:0/3196784788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c281611d00 msgr2=0x55c281634bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.218+0000 7f076cc3b740 1 --2- 172.21.15.23:0/3196784788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c281611d00 0x55c281634bf0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f074c02c170 tx=0x7f074c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:42:24.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.218+0000 7f076cc3b740 1 -- 172.21.15.23:0/3196784788 shutdown_connections 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.218+0000 7f076cc3b740 1 --2- 172.21.15.23:0/3196784788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c28161a8e0 0x55c281614ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.218+0000 7f076cc3b740 1 --2- 172.21.15.23:0/3196784788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c281611d00 0x55c281634bf0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.218+0000 7f076cc3b740 1 --2- 172.21.15.23:0/3196784788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2816113e0 0x55c28161a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.218+0000 7f076cc3b740 1 -- 172.21.15.23:0/3196784788 >> 172.21.15.23:0/3196784788 conn(0x55c28160bb10 msgr2=0x55c281616150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.218+0000 7f076cc3b740 1 -- 172.21.15.23:0/3196784788 shutdown_connections 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.218+0000 7f076cc3b740 1 -- 172.21.15.23:0/3196784788 wait complete. 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.219+0000 7f076cc3b740 1 -- start start 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.219+0000 7f076cc3b740 1 Processor -- start 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.219+0000 7f076cc3b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2816113e0 0x55c2816282d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.219+0000 7f076cc3b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c281611d00 0x55c281623640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.219+0000 7f076cc3b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 0x55c281623b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.219+0000 7f076cc3b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c2816240c0 con 0x55c2816113e0 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.219+0000 7f076cc3b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c281624230 con 0x55c281611d00 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.219+0000 7f076cc3b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c2816243f0 con 0x55c28161a8e0 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.219+0000 7f076cc3b740 10 client.0 _ll_register_callbacks cb 0x7f074c004a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.219+0000 7f07637fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2816113e0 0x55c2816282d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.219+0000 7f07637fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2816113e0 0x55c2816282d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55394/0 (socket says 172.21.15.23:55394) 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.219+0000 7f07637fe640 1 -- 172.21.15.23:0/839325003 learned_addr learned my addr 172.21.15.23:0/839325003 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.219+0000 7f075bfff640 1 --2- 172.21.15.23:0/839325003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c281611d00 0x55c281623640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.219+0000 7f0763fff640 1 --2- 172.21.15.23:0/839325003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 0x55c281623b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f07637fe640 1 -- 172.21.15.23:0/839325003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 msgr2=0x55c281623b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f07637fe640 1 --2- 172.21.15.23:0/839325003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 0x55c281623b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f07637fe640 1 -- 172.21.15.23:0/839325003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c281611d00 msgr2=0x55c281623640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f07637fe640 1 --2- 172.21.15.23:0/839325003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c281611d00 0x55c281623640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f07637fe640 1 -- 172.21.15.23:0/839325003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c281574ba0 con 0x55c2816113e0 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f07637fe640 1 --2- 172.21.15.23:0/839325003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2816113e0 0x55c2816282d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f075409ae70 tx=0x7f0754048710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0aa29f824add2af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f0742ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f0742ffd640 1 -- 172.21.15.23:0/839325003 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07540b3ea0 con 0x55c2816113e0 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f0742ffd640 1 -- 172.21.15.23:0/839325003 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f07540a3910 con 0x55c2816113e0 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f076cc3b740 1 -- 172.21.15.23:0/839325003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f075409b170 con 0x55c2816113e0 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f0742ffd640 1 -- 172.21.15.23:0/839325003 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0754097410 con 0x55c2816113e0 2024-03-24T10:42:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f076cc3b740 10 client.16251 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f076cc3b740 1 -- 172.21.15.23:0/839325003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c281536700 con 0x55c2816113e0 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f0742ffd640 1 -- 172.21.15.23:0/839325003 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f075409b170 con 0x55c2816113e0 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f0742ffd640 1 -- 172.21.15.23:0/839325003 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f07540a3b60 con 0x55c2816113e0 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f076cc3b740 10 client.16251 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f076cc3b740 10 client.16251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f076cc3b740 20 client.16251 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f076cc3b740 1 --2- 172.21.15.23:0/839325003 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c281693540 0x55c281695960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f076cc3b740 4 client.16251 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.220+0000 7f076cc3b740 1 -- 172.21.15.23:0/839325003 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c281301890 con 0x55c281693540 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f075bfff640 1 --2- 172.21.15.23:0/839325003 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c281693540 0x55c281695960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f0742ffd640 10 client.16251 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f075bfff640 1 --2- 172.21.15.23:0/839325003 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c281693540 0x55c281695960 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8ad8f9332a6c5c3 server_cookie=2c000797bbe568b9 in_seq=0 out_seq=0 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f0742ffd640 1 -- 172.21.15.23:0/839325003 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c281301890 con 0x55c281693540 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f0742ffd640 10 client.16251 handle_command_reply: tid=0 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f076cc3b740 1 client.16251 shutdown 2024-03-24T10:42:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f076cc3b740 1 -- 172.21.15.23:0/839325003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2816113e0 msgr2=0x55c2816282d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f076cc3b740 1 --2- 172.21.15.23:0/839325003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2816113e0 0x55c2816282d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f075409ae70 tx=0x7f0754048710 comp rx=0 tx=0).stop 2024-03-24T10:42:24.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f076cc3b740 1 -- 172.21.15.23:0/839325003 shutdown_connections 2024-03-24T10:42:24.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f076cc3b740 1 --2- 172.21.15.23:0/839325003 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c281693540 0x55c281695960 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f076cc3b740 1 --2- 172.21.15.23:0/839325003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c28161a8e0 0x55c281623b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f076cc3b740 1 --2- 172.21.15.23:0/839325003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c281611d00 0x55c281623640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f0742ffd640 0 client.16251 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:24.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f076cc3b740 1 --2- 172.21.15.23:0/839325003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2816113e0 0x55c2816282d0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f076cc3b740 1 -- 172.21.15.23:0/839325003 >> 172.21.15.23:0/839325003 conn(0x55c28160bb10 msgr2=0x55c28160ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:24.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f076cc3b740 1 -- 172.21.15.23:0/839325003 shutdown_connections 2024-03-24T10:42:24.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.221+0000 7f076cc3b740 1 -- 172.21.15.23:0/839325003 wait complete. 2024-03-24T10:42:24.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.222+0000 7f076cc3b740 20 client.16251 trim_cache size 0 max 16384 2024-03-24T10:42:24.227 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:42:24.227 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:42:24.228 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:42:24.228 INFO:teuthology.orchestra.run.smithi023.stdout: "cd982041-987e-4b95-b540-cbcee4bebb0a": { 2024-03-24T10:42:24.228 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:42:24.228 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "cd982041-987e-4b95-b540-cbcee4bebb0a", 2024-03-24T10:42:24.228 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:42:24.228 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:42:24.228 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:42:24.228 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:42:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.222+0000 7f076a055640 1 -- 172.21.15.23:0/2100371651 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f07500a2230 msgr2=0x7f07500a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.222+0000 7f076a055640 1 --2- 172.21.15.23:0/2100371651 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f07500a2230 0x7f07500a46f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f075c005fd0 tx=0x7f075c009e00 comp rx=0 tx=0).stop 2024-03-24T10:42:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.222+0000 7f076a055640 1 -- 172.21.15.23:0/2100371651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07641005b0 msgr2=0x7f076419c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.222+0000 7f076a055640 1 --2- 172.21.15.23:0/2100371651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07641005b0 0x7f076419c830 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f075400e9c0 tx=0x7f075400ee80 comp rx=0 tx=0).stop 2024-03-24T10:42:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.222+0000 7f076a055640 1 -- 172.21.15.23:0/2100371651 shutdown_connections 2024-03-24T10:42:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.222+0000 7f076a055640 1 --2- 172.21.15.23:0/2100371651 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f07500a2230 0x7f07500a46f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.222+0000 7f076a055640 1 --2- 172.21.15.23:0/2100371651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07641055d0 0x7f07641969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.222+0000 7f076a055640 1 --2- 172.21.15.23:0/2100371651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0764100f70 0x7f076419cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.222+0000 7f076a055640 1 --2- 172.21.15.23:0/2100371651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07641005b0 0x7f076419c830 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.222+0000 7f076a055640 1 -- 172.21.15.23:0/2100371651 >> 172.21.15.23:0/2100371651 conn(0x7f07640776a0 msgr2=0x7f07640fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.223+0000 7f076a055640 1 -- 172.21.15.23:0/2100371651 shutdown_connections 2024-03-24T10:42:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:24.223+0000 7f076a055640 1 -- 172.21.15.23:0/2100371651 wait complete. 2024-03-24T10:42:24.233 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:24 smithi023 ceph-mon[32466]: pgmap v72: 97 pgs: 97 active+clean; 67 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.8 MiB/s wr, 188 op/s 2024-03-24T10:42:24.234 INFO:tasks.cephfs.filesystem:scrub status for tag:cd982041-987e-4b95-b540-cbcee4bebb0a - {'path': '/', 'tag': 'cd982041-987e-4b95-b540-cbcee4bebb0a', 'options': 'recursive,force'} 2024-03-24T10:42:24.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:24 smithi117 ceph-mon[36368]: pgmap v72: 97 pgs: 97 active+clean; 67 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.8 MiB/s wr, 188 op/s 2024-03-24T10:42:24.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:24 smithi116 ceph-mon[36248]: pgmap v72: 97 pgs: 97 active+clean; 67 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.8 MiB/s wr, 188 op/s 2024-03-24T10:42:25.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 97288 12.91 MB/sec warmup 42 sec latency 90.147 ms 2024-03-24T10:42:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.391+0000 7f36268cd640 1 -- 172.21.15.23:0/3549491593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3620068d90 msgr2=0x7f36200691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.391+0000 7f36268cd640 1 --2- 172.21.15.23:0/3549491593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3620068d90 0x7f36200691f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3614009930 tx=0x7f361402bc30 comp rx=0 tx=0).stop 2024-03-24T10:42:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.391+0000 7f36268cd640 1 -- 172.21.15.23:0/3549491593 shutdown_connections 2024-03-24T10:42:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.391+0000 7f36268cd640 1 --2- 172.21.15.23:0/3549491593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3620069730 0x7f362010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.391+0000 7f36268cd640 1 --2- 172.21.15.23:0/3549491593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3620068d90 0x7f36200691f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.391+0000 7f36268cd640 1 --2- 172.21.15.23:0/3549491593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36201077e0 0x7f3620107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.391+0000 7f36268cd640 1 -- 172.21.15.23:0/3549491593 >> 172.21.15.23:0/3549491593 conn(0x7f3620077460 msgr2=0x7f3620077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.391+0000 7f36268cd640 1 -- 172.21.15.23:0/3549491593 shutdown_connections 2024-03-24T10:42:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.391+0000 7f36268cd640 1 -- 172.21.15.23:0/3549491593 wait complete. 2024-03-24T10:42:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.392+0000 7f36268cd640 1 Processor -- start 2024-03-24T10:42:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.392+0000 7f36268cd640 1 -- start start 2024-03-24T10:42:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.392+0000 7f36268cd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3620068d90 0x7f3620100930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.392+0000 7f36268cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3620069730 0x7f3620100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.392+0000 7f36268cd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36201077e0 0x7f36201013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.392+0000 7f36268cd640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f36201a6060 con 0x7f36201077e0 2024-03-24T10:42:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.392+0000 7f36268cd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36201a61d0 con 0x7f3620069730 2024-03-24T10:42:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.392+0000 7f36268cd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f36201a6390 con 0x7f3620068d90 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.392+0000 7f3624e6a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36201077e0 0x7f36201013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.392+0000 7f3624e6a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36201077e0 0x7f36201013b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55414/0 (socket says 172.21.15.23:55414) 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.392+0000 7f361f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3620069730 0x7f3620100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.392+0000 7f3624e6a640 1 -- 172.21.15.23:0/3581233048 learned_addr learned my addr 172.21.15.23:0/3581233048 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.392+0000 7f361ffff640 1 --2- 172.21.15.23:0/3581233048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3620068d90 0x7f3620100930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.393+0000 7f3624e6a640 1 -- 172.21.15.23:0/3581233048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3620068d90 msgr2=0x7f3620100930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.393+0000 7f3624e6a640 1 --2- 172.21.15.23:0/3581233048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3620068d90 0x7f3620100930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.393+0000 7f3624e6a640 1 -- 172.21.15.23:0/3581233048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3620069730 msgr2=0x7f3620100e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.393+0000 7f3624e6a640 1 --2- 172.21.15.23:0/3581233048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3620069730 0x7f3620100e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.393+0000 7f3624e6a640 1 -- 172.21.15.23:0/3581233048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3618009840 con 0x7f36201077e0 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.393+0000 7f361ffff640 1 --2- 172.21.15.23:0/3581233048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3620068d90 0x7f3620100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.393+0000 7f3624e6a640 1 --2- 172.21.15.23:0/3581233048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36201077e0 0x7f36201013b0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f361800ec70 tx=0x7f361800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.393+0000 7f361d7fa640 1 -- 172.21.15.23:0/3581233048 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3618003b70 con 0x7f36201077e0 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.393+0000 7f361d7fa640 1 -- 172.21.15.23:0/3581233048 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f361800ee80 con 0x7f36201077e0 2024-03-24T10:42:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.393+0000 7f36268cd640 1 -- 172.21.15.23:0/3581233048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3614009560 con 0x7f36201077e0 2024-03-24T10:42:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.393+0000 7f361d7fa640 1 -- 172.21.15.23:0/3581233048 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3618010810 con 0x7f36201077e0 2024-03-24T10:42:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.393+0000 7f36268cd640 1 -- 172.21.15.23:0/3581233048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36201a6970 con 0x7f36201077e0 2024-03-24T10:42:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.396+0000 7f361d7fa640 1 -- 172.21.15.23:0/3581233048 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f3618004020 con 0x7f36201077e0 2024-03-24T10:42:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.396+0000 7f3629588740 1 Processor -- start 2024-03-24T10:42:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.396+0000 7f3629588740 1 -- start start 2024-03-24T10:42:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.396+0000 7f361d7fa640 1 --2- 172.21.15.23:0/3581233048 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f36080a22d0 0x7f36080a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.396+0000 7f3629588740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5df5e0 0x55e65f5df9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.396+0000 7f361d7fa640 1 -- 172.21.15.23:0/3581233048 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f3618014070 con 0x7f36201077e0 2024-03-24T10:42:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.396+0000 7f3629588740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5dff00 0x55e65f5e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.396+0000 7f361f7fe640 1 --2- 172.21.15.23:0/3581233048 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f36080a22d0 0x7f36080a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.396+0000 7f3629588740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5e8cd0 0x55e65f5eb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.396+0000 7f3629588740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e65f59a1b0 con 0x55e65f5df5e0 2024-03-24T10:42:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.396+0000 7f3629588740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e65f54cf80 con 0x55e65f5dff00 2024-03-24T10:42:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.396+0000 7f3629588740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e65f54a4b0 con 0x55e65f5e8cd0 2024-03-24T10:42:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.396+0000 7f361ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5df5e0 0x55e65f5df9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.397+0000 7f361ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5df5e0 0x55e65f5df9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55420/0 (socket says 172.21.15.23:55420) 2024-03-24T10:42:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.397+0000 7f361ffff640 1 -- 172.21.15.23:0/1265261837 learned_addr learned my addr 172.21.15.23:0/1265261837 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.397+0000 7f361f7fe640 1 --2- 172.21.15.23:0/1265261837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5dff00 0x55e65f5e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.397+0000 7f3624e6a640 1 --2- 172.21.15.23:0/1265261837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5e8cd0 0x55e65f5eb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.397+0000 7f361ffff640 1 -- 172.21.15.23:0/1265261837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5e8cd0 msgr2=0x55e65f5eb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.397+0000 7f361ffff640 1 --2- 172.21.15.23:0/1265261837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5e8cd0 0x55e65f5eb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.397+0000 7f361ffff640 1 -- 172.21.15.23:0/1265261837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5dff00 msgr2=0x55e65f5e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.397+0000 7f361ffff640 1 --2- 172.21.15.23:0/1265261837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5dff00 0x55e65f5e8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.397+0000 7f361ffff640 1 -- 172.21.15.23:0/1265261837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e65f4db810 con 0x55e65f5df5e0 2024-03-24T10:42:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.397+0000 7f361f7fe640 1 --2- 172.21.15.23:0/3581233048 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f36080a22d0 0x7f36080a4790 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3614005ab0 tx=0x7f361402cba0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.397+0000 7f361ffff640 1 --2- 172.21.15.23:0/1265261837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5df5e0 0x55e65f5df9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f36100114f0 tx=0x7f36100119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=552d8265e8e34174 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.397+0000 7f36067fc640 1 -- 172.21.15.23:0/1265261837 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361000f540 con 0x55e65f5df5e0 2024-03-24T10:42:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.397+0000 7f36067fc640 1 -- 172.21.15.23:0/1265261837 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f361000fb80 con 0x55e65f5df5e0 2024-03-24T10:42:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.397+0000 7f36067fc640 1 -- 172.21.15.23:0/1265261837 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3610019bf0 con 0x55e65f5df5e0 2024-03-24T10:42:25.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.398+0000 7f3629588740 1 -- 172.21.15.23:0/1265261837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5df5e0 msgr2=0x55e65f5df9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.398+0000 7f3629588740 1 --2- 172.21.15.23:0/1265261837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5df5e0 0x55e65f5df9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f36100114f0 tx=0x7f36100119b0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.398+0000 7f3629588740 1 -- 172.21.15.23:0/1265261837 shutdown_connections 2024-03-24T10:42:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.398+0000 7f3629588740 1 --2- 172.21.15.23:0/1265261837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5e8cd0 0x55e65f5eb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.398+0000 7f3629588740 1 --2- 172.21.15.23:0/1265261837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5dff00 0x55e65f5e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.398+0000 7f3629588740 1 --2- 172.21.15.23:0/1265261837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5df5e0 0x55e65f5df9c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.398+0000 7f3629588740 1 -- 172.21.15.23:0/1265261837 >> 172.21.15.23:0/1265261837 conn(0x55e65f5d9f00 msgr2=0x55e65f5da2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.398+0000 7f3629588740 1 -- 172.21.15.23:0/1265261837 shutdown_connections 2024-03-24T10:42:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.398+0000 7f3629588740 1 -- 172.21.15.23:0/1265261837 wait complete. 2024-03-24T10:42:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.398+0000 7f3629588740 1 -- start start 2024-03-24T10:42:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f3629588740 1 Processor -- start 2024-03-24T10:42:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f3629588740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5df5e0 0x55e65f5e5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f3629588740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5dff00 0x55e65f5e6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f3629588740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5e8cd0 0x55e65f5e6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f3629588740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e65f545c90 con 0x55e65f5dff00 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f3629588740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e65f4eac50 con 0x55e65f5e8cd0 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f3629588740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e65f219250 con 0x55e65f5df5e0 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f3629588740 10 client.0 _ll_register_callbacks cb 0x55e65f539290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f361f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5dff00 0x55e65f5e6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f361ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5df5e0 0x55e65f5e5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f361f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5dff00 0x55e65f5e6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55436/0 (socket says 172.21.15.23:55436) 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f361f7fe640 1 -- 172.21.15.23:0/1590530864 learned_addr learned my addr 172.21.15.23:0/1590530864 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f3624e6a640 1 --2- 172.21.15.23:0/1590530864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5e8cd0 0x55e65f5e6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f361f7fe640 1 -- 172.21.15.23:0/1590530864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5df5e0 msgr2=0x55e65f5e5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f361f7fe640 1 --2- 172.21.15.23:0/1590530864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5df5e0 0x55e65f5e5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f361f7fe640 1 -- 172.21.15.23:0/1590530864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5e8cd0 msgr2=0x55e65f5e6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f361f7fe640 1 --2- 172.21.15.23:0/1590530864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5e8cd0 0x55e65f5e6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f361f7fe640 1 -- 172.21.15.23:0/1590530864 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3610009cc0 con 0x55e65f5dff00 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f361ffff640 1 --2- 172.21.15.23:0/1590530864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5df5e0 0x55e65f5e5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f361f7fe640 1 --2- 172.21.15.23:0/1590530864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5dff00 0x55e65f5e6230 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3614006010 tx=0x7f361402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eccc0196d7e5e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.399+0000 7f35ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f35ff7fe640 1 -- 172.21.15.23:0/1590530864 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3614004450 con 0x55e65f5dff00 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f35ff7fe640 1 -- 172.21.15.23:0/1590530864 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f36140035f0 con 0x55e65f5dff00 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f3629588740 1 -- 172.21.15.23:0/1590530864 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e65f56a790 con 0x55e65f5dff00 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f35ff7fe640 1 -- 172.21.15.23:0/1590530864 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f36140037b0 con 0x55e65f5dff00 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f3629588740 10 client.16263 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f3629588740 1 -- 172.21.15.23:0/1590530864 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e65f544310 con 0x55e65f5dff00 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f35ff7fe640 1 -- 172.21.15.23:0/1590530864 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e65f56a790 con 0x55e65f5dff00 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f35ff7fe640 1 -- 172.21.15.23:0/1590530864 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f361403c040 con 0x55e65f5dff00 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f3629588740 10 client.16263 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f3629588740 10 client.16263 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f3629588740 20 client.16263 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f3629588740 1 --2- 172.21.15.23:0/1590530864 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e65f5f8bc0 0x55e65f5fb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f3629588740 4 client.16263 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f3629588740 1 -- 172.21.15.23:0/1590530864 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e65f245600 con 0x55e65f5f8bc0 2024-03-24T10:42:25.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.400+0000 7f361ffff640 1 --2- 172.21.15.23:0/1590530864 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e65f5f8bc0 0x55e65f5fb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.401+0000 7f35ff7fe640 10 client.16263 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.401+0000 7f361ffff640 1 --2- 172.21.15.23:0/1590530864 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e65f5f8bc0 0x55e65f5fb020 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db0f57f02c74368 server_cookie=50638f6a2f2c0bb2 in_seq=0 out_seq=0 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.401+0000 7f35ff7fe640 1 -- 172.21.15.23:0/1590530864 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e65f245600 con 0x55e65f5f8bc0 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.401+0000 7f35ff7fe640 10 client.16263 handle_command_reply: tid=0 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.401+0000 7f3629588740 1 client.16263 shutdown 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.402+0000 7f3629588740 1 -- 172.21.15.23:0/1590530864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5dff00 msgr2=0x55e65f5e6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.402+0000 7f3629588740 1 --2- 172.21.15.23:0/1590530864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5dff00 0x55e65f5e6230 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3614006010 tx=0x7f361402bc30 comp rx=0 tx=0).stop 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.402+0000 7f3629588740 1 -- 172.21.15.23:0/1590530864 shutdown_connections 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.402+0000 7f3629588740 1 --2- 172.21.15.23:0/1590530864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5e8cd0 0x55e65f5e6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.402+0000 7f3629588740 1 --2- 172.21.15.23:0/1590530864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5dff00 0x55e65f5e6230 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.402+0000 7f3629588740 1 --2- 172.21.15.23:0/1590530864 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e65f5f8bc0 0x55e65f5fb020 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.402+0000 7f3629588740 1 --2- 172.21.15.23:0/1590530864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5df5e0 0x55e65f5e5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.402+0000 7f35ff7fe640 0 client.16263 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.402+0000 7f3629588740 1 -- 172.21.15.23:0/1590530864 >> 172.21.15.23:0/1590530864 conn(0x55e65f5d9f00 msgr2=0x55e65f5dbc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.402+0000 7f3629588740 1 -- 172.21.15.23:0/1590530864 shutdown_connections 2024-03-24T10:42:25.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.402+0000 7f3629588740 1 -- 172.21.15.23:0/1590530864 wait complete. 2024-03-24T10:42:25.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.402+0000 7f3629588740 20 client.16263 trim_cache size 0 max 16384 2024-03-24T10:42:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.418+0000 7f3629588740 1 Processor -- start 2024-03-24T10:42:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.418+0000 7f3629588740 1 -- start start 2024-03-24T10:42:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.418+0000 7f3629588740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5df5e0 0x55e65f5e8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.418+0000 7f3629588740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5dff00 0x55e65f5d4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.418+0000 7f3629588740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5e8cd0 0x55e65f5f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.418+0000 7f3629588740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e65f4cb660 con 0x55e65f5dff00 2024-03-24T10:42:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.418+0000 7f3629588740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e65f5dfcf0 con 0x55e65f5e8cd0 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.418+0000 7f3629588740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e65f5e6590 con 0x55e65f5df5e0 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.418+0000 7f361f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5dff00 0x55e65f5d4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.419+0000 7f361f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5dff00 0x55e65f5d4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55438/0 (socket says 172.21.15.23:55438) 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.419+0000 7f361f7fe640 1 -- 172.21.15.23:0/3574759963 learned_addr learned my addr 172.21.15.23:0/3574759963 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.419+0000 7f3624e6a640 1 --2- 172.21.15.23:0/3574759963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5e8cd0 0x55e65f5f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.419+0000 7f361ffff640 1 --2- 172.21.15.23:0/3574759963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5df5e0 0x55e65f5e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.419+0000 7f361f7fe640 1 -- 172.21.15.23:0/3574759963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5df5e0 msgr2=0x55e65f5e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.419+0000 7f361f7fe640 1 --2- 172.21.15.23:0/3574759963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5df5e0 0x55e65f5e8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.419+0000 7f361f7fe640 1 -- 172.21.15.23:0/3574759963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5e8cd0 msgr2=0x55e65f5f43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.419+0000 7f361f7fe640 1 --2- 172.21.15.23:0/3574759963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5e8cd0 0x55e65f5f43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.419+0000 7f361f7fe640 1 -- 172.21.15.23:0/3574759963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e65f4e0810 con 0x55e65f5dff00 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.419+0000 7f361f7fe640 1 --2- 172.21.15.23:0/3574759963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5dff00 0x55e65f5d4d40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f361402cc80 tx=0x7f3614004bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30b0d9ac92ab297 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.419+0000 7f3605ffb640 1 -- 172.21.15.23:0/3574759963 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3614004450 con 0x55e65f5dff00 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.419+0000 7f3605ffb640 1 -- 172.21.15.23:0/3574759963 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f361403d0b0 con 0x55e65f5dff00 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.419+0000 7f3605ffb640 1 -- 172.21.15.23:0/3574759963 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361405f5e0 con 0x55e65f5dff00 2024-03-24T10:42:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.420+0000 7f3629588740 1 -- 172.21.15.23:0/3574759963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5dff00 msgr2=0x55e65f5d4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.420+0000 7f3629588740 1 --2- 172.21.15.23:0/3574759963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5dff00 0x55e65f5d4d40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f361402cc80 tx=0x7f3614004bd0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.420+0000 7f3629588740 1 -- 172.21.15.23:0/3574759963 shutdown_connections 2024-03-24T10:42:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.420+0000 7f3629588740 1 --2- 172.21.15.23:0/3574759963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5e8cd0 0x55e65f5f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.420+0000 7f3629588740 1 --2- 172.21.15.23:0/3574759963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5dff00 0x55e65f5d4d40 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.420+0000 7f3629588740 1 --2- 172.21.15.23:0/3574759963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5df5e0 0x55e65f5e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.420+0000 7f3629588740 1 -- 172.21.15.23:0/3574759963 >> 172.21.15.23:0/3574759963 conn(0x55e65f5d9f00 msgr2=0x55e65f5da2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.420+0000 7f3629588740 1 -- 172.21.15.23:0/3574759963 shutdown_connections 2024-03-24T10:42:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.420+0000 7f3629588740 1 -- 172.21.15.23:0/3574759963 wait complete. 2024-03-24T10:42:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.420+0000 7f3629588740 1 -- start start 2024-03-24T10:42:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.421+0000 7f3629588740 1 Processor -- start 2024-03-24T10:42:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.421+0000 7f3629588740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5df5e0 0x55e65f5f09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.421+0000 7f3629588740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5dff00 0x55e65f5e2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.421+0000 7f3629588740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5e8cd0 0x55e65f5e28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.421+0000 7f3629588740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e65f5f1110 con 0x55e65f5e8cd0 2024-03-24T10:42:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.421+0000 7f3629588740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e65f5e2df0 con 0x55e65f5df5e0 2024-03-24T10:42:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.421+0000 7f3629588740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e65f5e2fb0 con 0x55e65f5dff00 2024-03-24T10:42:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.421+0000 7f3629588740 10 client.0 _ll_register_callbacks cb 0x7f3614009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.421+0000 7f3624e6a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5e8cd0 0x55e65f5e28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.421+0000 7f3624e6a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5e8cd0 0x55e65f5e28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55446/0 (socket says 172.21.15.23:55446) 2024-03-24T10:42:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.421+0000 7f361ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5df5e0 0x55e65f5f09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.421+0000 7f3624e6a640 1 -- 172.21.15.23:0/597945841 learned_addr learned my addr 172.21.15.23:0/597945841 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f361f7fe640 1 --2- 172.21.15.23:0/597945841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5dff00 0x55e65f5e2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f3624e6a640 1 -- 172.21.15.23:0/597945841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5dff00 msgr2=0x55e65f5e2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f3624e6a640 1 --2- 172.21.15.23:0/597945841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5dff00 0x55e65f5e2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f3624e6a640 1 -- 172.21.15.23:0/597945841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5df5e0 msgr2=0x55e65f5f09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f3624e6a640 1 --2- 172.21.15.23:0/597945841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5df5e0 0x55e65f5f09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f3624e6a640 1 -- 172.21.15.23:0/597945841 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e65f544310 con 0x55e65f5e8cd0 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f361f7fe640 1 --2- 172.21.15.23:0/597945841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5dff00 0x55e65f5e2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f3624e6a640 1 --2- 172.21.15.23:0/597945841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5e8cd0 0x55e65f5e28b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f361809afa0 tx=0x7f361809b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead4b8303bc190e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f35ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f35ff7fe640 1 -- 172.21.15.23:0/597945841 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3618097680 con 0x55e65f5e8cd0 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f35ff7fe640 1 -- 172.21.15.23:0/597945841 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f36180977e0 con 0x55e65f5e8cd0 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f3629588740 1 -- 172.21.15.23:0/597945841 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e65f5e0b40 con 0x55e65f5e8cd0 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f35ff7fe640 1 -- 172.21.15.23:0/597945841 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f36180b9310 con 0x55e65f5e8cd0 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f3629588740 10 client.16269 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f3629588740 1 -- 172.21.15.23:0/597945841 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e65f5065a0 con 0x55e65f5e8cd0 2024-03-24T10:42:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.422+0000 7f35ff7fe640 1 -- 172.21.15.23:0/597945841 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e65f5e0b40 con 0x55e65f5e8cd0 2024-03-24T10:42:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.423+0000 7f35ff7fe640 1 -- 172.21.15.23:0/597945841 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f361809a4f0 con 0x55e65f5e8cd0 2024-03-24T10:42:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.423+0000 7f3629588740 10 client.16269 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.423+0000 7f3629588740 10 client.16269 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.423+0000 7f3629588740 20 client.16269 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.423+0000 7f3629588740 1 --2- 172.21.15.23:0/597945841 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e65f65f2c0 0x55e65f6616e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.423+0000 7f3629588740 4 client.16269 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:42:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.423+0000 7f3629588740 1 -- 172.21.15.23:0/597945841 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f361000fb80 con 0x55e65f65f2c0 2024-03-24T10:42:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.423+0000 7f361ffff640 1 --2- 172.21.15.23:0/597945841 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e65f65f2c0 0x55e65f6616e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.423+0000 7f35ff7fe640 10 client.16269 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.423+0000 7f361ffff640 1 --2- 172.21.15.23:0/597945841 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e65f65f2c0 0x55e65f6616e0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38b610cf0f3b864e server_cookie=cbb328fc0fc62e47 in_seq=0 out_seq=0 2024-03-24T10:42:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f35ff7fe640 1 -- 172.21.15.23:0/597945841 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f361000fb80 con 0x55e65f65f2c0 2024-03-24T10:42:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f35ff7fe640 10 client.16269 handle_command_reply: tid=0 2024-03-24T10:42:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f3629588740 1 client.16269 shutdown 2024-03-24T10:42:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f3629588740 1 -- 172.21.15.23:0/597945841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5e8cd0 msgr2=0x55e65f5e28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f3629588740 1 --2- 172.21.15.23:0/597945841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5e8cd0 0x55e65f5e28b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f361809afa0 tx=0x7f361809b090 comp rx=0 tx=0).stop 2024-03-24T10:42:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f3629588740 1 -- 172.21.15.23:0/597945841 shutdown_connections 2024-03-24T10:42:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f3629588740 1 --2- 172.21.15.23:0/597945841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e65f5e8cd0 0x55e65f5e28b0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f3629588740 1 --2- 172.21.15.23:0/597945841 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e65f65f2c0 0x55e65f6616e0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f35ff7fe640 0 client.16269 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f3629588740 1 --2- 172.21.15.23:0/597945841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e65f5dff00 0x55e65f5e2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f3629588740 1 --2- 172.21.15.23:0/597945841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e65f5df5e0 0x55e65f5f09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f3629588740 1 -- 172.21.15.23:0/597945841 >> 172.21.15.23:0/597945841 conn(0x55e65f5d9f00 msgr2=0x55e65f5e6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f3629588740 1 -- 172.21.15.23:0/597945841 shutdown_connections 2024-03-24T10:42:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f3629588740 1 -- 172.21.15.23:0/597945841 wait complete. 2024-03-24T10:42:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.432+0000 7f3629588740 20 client.16269 trim_cache size 0 max 16384 2024-03-24T10:42:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:42:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:42:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:42:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:42:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.433+0000 7f36268cd640 1 -- 172.21.15.23:0/3581233048 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f36080a22d0 msgr2=0x7f36080a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.433+0000 7f36268cd640 1 --2- 172.21.15.23:0/3581233048 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f36080a22d0 0x7f36080a4790 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3614005ab0 tx=0x7f361402cba0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.433+0000 7f36268cd640 1 -- 172.21.15.23:0/3581233048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36201077e0 msgr2=0x7f36201013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.433+0000 7f36268cd640 1 --2- 172.21.15.23:0/3581233048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36201077e0 0x7f36201013b0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f361800ec70 tx=0x7f361800c6a0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.433+0000 7f36268cd640 1 -- 172.21.15.23:0/3581233048 shutdown_connections 2024-03-24T10:42:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.433+0000 7f36268cd640 1 --2- 172.21.15.23:0/3581233048 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f36080a22d0 0x7f36080a4790 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.433+0000 7f36268cd640 1 --2- 172.21.15.23:0/3581233048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36201077e0 0x7f36201013b0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.433+0000 7f36268cd640 1 --2- 172.21.15.23:0/3581233048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3620069730 0x7f3620100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.433+0000 7f36268cd640 1 --2- 172.21.15.23:0/3581233048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3620068d90 0x7f3620100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.433+0000 7f36268cd640 1 -- 172.21.15.23:0/3581233048 >> 172.21.15.23:0/3581233048 conn(0x7f3620077460 msgr2=0x7f36200fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.433+0000 7f36268cd640 1 -- 172.21.15.23:0/3581233048 shutdown_connections 2024-03-24T10:42:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:25.433+0000 7f36268cd640 1 -- 172.21.15.23:0/3581233048 wait complete. 2024-03-24T10:42:25.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:25 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:42:25.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:25 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:25.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:25 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:42:25.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:25 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:42:25.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:25 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:25.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:25 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:25.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:25 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:42:25.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:25 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:25.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:25 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:42:25.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:25 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:42:25.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:25 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:25.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:25 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:25.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:25 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:42:25.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:25 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:25.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:25 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:42:25.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:25 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:42:25.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:25 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:25.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:25 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:26.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 99797 12.94 MB/sec warmup 43 sec latency 25.621 ms 2024-03-24T10:42:26.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:26 smithi117 ceph-mon[36368]: pgmap v73: 97 pgs: 97 active+clean; 67 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.8 MiB/s wr, 188 op/s 2024-03-24T10:42:26.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:26 smithi116 ceph-mon[36248]: pgmap v73: 97 pgs: 97 active+clean; 67 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.8 MiB/s wr, 188 op/s 2024-03-24T10:42:26.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:26 smithi023 ceph-mon[32466]: pgmap v73: 97 pgs: 97 active+clean; 67 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.8 MiB/s wr, 188 op/s 2024-03-24T10:42:27.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 102526 12.95 MB/sec warmup 44 sec latency 23.211 ms 2024-03-24T10:42:28.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 104335 12.90 MB/sec warmup 45 sec latency 26.445 ms 2024-03-24T10:42:28.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:28 smithi023 ceph-mon[32466]: pgmap v74: 97 pgs: 97 active+clean; 72 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.2 MiB/s wr, 297 op/s 2024-03-24T10:42:28.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:28 smithi117 ceph-mon[36368]: pgmap v74: 97 pgs: 97 active+clean; 72 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.2 MiB/s wr, 297 op/s 2024-03-24T10:42:28.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:28 smithi116 ceph-mon[36248]: pgmap v74: 97 pgs: 97 active+clean; 72 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.2 MiB/s wr, 297 op/s 2024-03-24T10:42:29.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 106918 12.93 MB/sec warmup 46 sec latency 13.481 ms 2024-03-24T10:42:30.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 109059 12.82 MB/sec warmup 47 sec latency 100.072 ms 2024-03-24T10:42:30.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:30 smithi023 ceph-mon[32466]: pgmap v75: 97 pgs: 97 active+clean; 72 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 208 op/s 2024-03-24T10:42:30.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:30 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:42:30.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:30 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:42:30.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:30 smithi117 ceph-mon[36368]: pgmap v75: 97 pgs: 97 active+clean; 72 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 208 op/s 2024-03-24T10:42:30.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:30 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:42:30.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:30 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:42:30.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:30 smithi116 ceph-mon[36248]: pgmap v75: 97 pgs: 97 active+clean; 72 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 208 op/s 2024-03-24T10:42:30.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:30 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:42:30.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:30 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:42:31.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 111064 12.83 MB/sec warmup 48 sec latency 22.273 ms 2024-03-24T10:42:32.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 113916 12.92 MB/sec warmup 49 sec latency 17.524 ms 2024-03-24T10:42:32.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:32 smithi023 ceph-mon[32466]: pgmap v76: 97 pgs: 97 active+clean; 76 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 315 op/s 2024-03-24T10:42:32.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:32 smithi117 ceph-mon[36368]: pgmap v76: 97 pgs: 97 active+clean; 76 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 315 op/s 2024-03-24T10:42:32.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:32 smithi116 ceph-mon[36248]: pgmap v76: 97 pgs: 97 active+clean; 76 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 315 op/s 2024-03-24T10:42:33.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 116441 12.90 MB/sec warmup 50 sec latency 24.071 ms 2024-03-24T10:42:34.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 118277 12.83 MB/sec warmup 51 sec latency 14.975 ms 2024-03-24T10:42:34.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:34 smithi023 ceph-mon[32466]: pgmap v77: 97 pgs: 97 active+clean; 76 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 216 op/s 2024-03-24T10:42:34.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:34 smithi117 ceph-mon[36368]: pgmap v77: 97 pgs: 97 active+clean; 76 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 216 op/s 2024-03-24T10:42:34.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:34 smithi116 ceph-mon[36248]: pgmap v77: 97 pgs: 97 active+clean; 76 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 216 op/s 2024-03-24T10:42:35.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 120904 12.90 MB/sec warmup 52 sec latency 48.201 ms 2024-03-24T10:42:36.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 123074 12.80 MB/sec warmup 53 sec latency 18.819 ms 2024-03-24T10:42:36.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:36 smithi023 ceph-mon[32466]: pgmap v78: 97 pgs: 97 active+clean; 76 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 216 op/s 2024-03-24T10:42:36.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:36 smithi117 ceph-mon[36368]: pgmap v78: 97 pgs: 97 active+clean; 76 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 216 op/s 2024-03-24T10:42:36.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:36 smithi116 ceph-mon[36248]: pgmap v78: 97 pgs: 97 active+clean; 76 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 216 op/s 2024-03-24T10:42:37.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 125116 12.80 MB/sec warmup 54 sec latency 25.933 ms 2024-03-24T10:42:38.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 127951 12.89 MB/sec warmup 55 sec latency 75.431 ms 2024-03-24T10:42:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.524+0000 7fd9b9642640 1 -- 172.21.15.23:0/474170718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b4107730 msgr2=0x7fd9b4107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.524+0000 7fd9b9642640 1 --2- 172.21.15.23:0/474170718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b4107730 0x7fd9b4107b10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009930 tx=0x7fd9a402bc30 comp rx=0 tx=0).stop 2024-03-24T10:42:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.524+0000 7fd9b9642640 1 -- 172.21.15.23:0/474170718 shutdown_connections 2024-03-24T10:42:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.524+0000 7fd9b9642640 1 --2- 172.21.15.23:0/474170718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b41005d0 0x7fd9b410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.524+0000 7fd9b9642640 1 --2- 172.21.15.23:0/474170718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b40ffc10 0x7fd9b4100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.524+0000 7fd9b9642640 1 --2- 172.21.15.23:0/474170718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b4107730 0x7fd9b4107b10 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.524+0000 7fd9b9642640 1 -- 172.21.15.23:0/474170718 >> 172.21.15.23:0/474170718 conn(0x7fd9b4076ba0 msgr2=0x7fd9b40fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.524+0000 7fd9b9642640 1 -- 172.21.15.23:0/474170718 shutdown_connections 2024-03-24T10:42:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.524+0000 7fd9b9642640 1 -- 172.21.15.23:0/474170718 wait complete. 2024-03-24T10:42:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.525+0000 7fd9b9642640 1 Processor -- start 2024-03-24T10:42:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.525+0000 7fd9b9642640 1 -- start start 2024-03-24T10:42:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.525+0000 7fd9b9642640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b40ffc10 0x7fd9b419da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.525+0000 7fd9b9642640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b41005d0 0x7fd9b419dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.525+0000 7fd9b9642640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b4107730 0x7fd9b4197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.525+0000 7fd9b9642640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b41982c0 con 0x7fd9b40ffc10 2024-03-24T10:42:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.525+0000 7fd9b9642640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b4198430 con 0x7fd9b41005d0 2024-03-24T10:42:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.525+0000 7fd9b9642640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b41985f0 con 0x7fd9b4107730 2024-03-24T10:42:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.525+0000 7fd9b2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b40ffc10 0x7fd9b419da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.525+0000 7fd9b2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b40ffc10 0x7fd9b419da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60394/0 (socket says 172.21.15.23:60394) 2024-03-24T10:42:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.525+0000 7fd9abfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b41005d0 0x7fd9b419dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.525+0000 7fd9b2ffd640 1 -- 172.21.15.23:0/798463704 learned_addr learned my addr 172.21.15.23:0/798463704 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.525+0000 7fd9b37fe640 1 --2- 172.21.15.23:0/798463704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b4107730 0x7fd9b4197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.526+0000 7fd9b2ffd640 1 -- 172.21.15.23:0/798463704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b4107730 msgr2=0x7fd9b4197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.526+0000 7fd9b2ffd640 1 --2- 172.21.15.23:0/798463704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b4107730 0x7fd9b4197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.526+0000 7fd9b2ffd640 1 -- 172.21.15.23:0/798463704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b41005d0 msgr2=0x7fd9b419dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.526+0000 7fd9b2ffd640 1 --2- 172.21.15.23:0/798463704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b41005d0 0x7fd9b419dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.526+0000 7fd9b2ffd640 1 -- 172.21.15.23:0/798463704 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a4009560 con 0x7fd9b40ffc10 2024-03-24T10:42:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.526+0000 7fd9abfff640 1 --2- 172.21.15.23:0/798463704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b41005d0 0x7fd9b419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.526+0000 7fd9b2ffd640 1 --2- 172.21.15.23:0/798463704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b40ffc10 0x7fd9b419da70 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4000c00 tx=0x7fd9a402c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.526+0000 7fd9b0ff9640 1 -- 172.21.15.23:0/798463704 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a40042d0 con 0x7fd9b40ffc10 2024-03-24T10:42:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.526+0000 7fd9b0ff9640 1 -- 172.21.15.23:0/798463704 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9a4004430 con 0x7fd9b40ffc10 2024-03-24T10:42:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.526+0000 7fd9b9642640 1 -- 172.21.15.23:0/798463704 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b4198870 con 0x7fd9b40ffc10 2024-03-24T10:42:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.526+0000 7fd9b0ff9640 1 -- 172.21.15.23:0/798463704 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4003c50 con 0x7fd9b40ffc10 2024-03-24T10:42:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.526+0000 7fd9b9642640 1 -- 172.21.15.23:0/798463704 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b41a49f0 con 0x7fd9b40ffc10 2024-03-24T10:42:38.535 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9b0ff9640 1 -- 172.21.15.23:0/798463704 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fd9a4004df0 con 0x7fd9b40ffc10 2024-03-24T10:42:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9bc21c740 1 Processor -- start 2024-03-24T10:42:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9bc21c740 1 -- start start 2024-03-24T10:42:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9bc21c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9ed949c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9bc21c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed94f00 0x559f9ed9d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9bc21c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed9dcd0 0x559f9eda00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9bc21c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x559f9ed4f1b0 con 0x559f9ed94f00 2024-03-24T10:42:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9bc21c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559f9ed01f80 con 0x559f9ed945e0 2024-03-24T10:42:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9bc21c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559f9ecff4b0 con 0x559f9ed9dcd0 2024-03-24T10:42:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9b0ff9640 1 --2- 172.21.15.23:0/798463704 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9a00a2350 0x7fd9a00a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9b2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed94f00 0x559f9ed9d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9abfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed9dcd0 0x559f9eda00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9b37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9ed949c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9b2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed94f00 0x559f9ed9d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60410/0 (socket says 172.21.15.23:60410) 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.529+0000 7fd9b2ffd640 1 -- 172.21.15.23:0/3470004313 learned_addr learned my addr 172.21.15.23:0/3470004313 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.530+0000 7fd9b37fe640 1 --2- 172.21.15.23:0/798463704 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9a00a2350 0x7fd9a00a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.530+0000 7fd9b0ff9640 1 -- 172.21.15.23:0/798463704 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fd9a40f0580 con 0x7fd9b40ffc10 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.530+0000 7fd9abfff640 1 -- 172.21.15.23:0/3470004313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 msgr2=0x559f9ed949c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.530+0000 7fd9abfff640 1 --2- 172.21.15.23:0/3470004313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9ed949c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.530+0000 7fd9abfff640 1 -- 172.21.15.23:0/3470004313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed94f00 msgr2=0x559f9ed9d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.530+0000 7fd9abfff640 1 --2- 172.21.15.23:0/3470004313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed94f00 0x559f9ed9d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.530+0000 7fd9abfff640 1 -- 172.21.15.23:0/3470004313 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f9ec90810 con 0x559f9ed9dcd0 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.530+0000 7fd9abfff640 1 --2- 172.21.15.23:0/3470004313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed9dcd0 0x559f9eda00c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009fa0 tx=0x7fd99c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e17c00d4a248b45a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.530+0000 7fd9b37fe640 1 --2- 172.21.15.23:0/798463704 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9a00a2350 0x7fd9a00a4810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac005fd0 tx=0x7fd9ac009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.530+0000 7fd9a97fa640 1 -- 172.21.15.23:0/3470004313 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c004310 con 0x559f9ed9dcd0 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.530+0000 7fd9a97fa640 1 -- 172.21.15.23:0/3470004313 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd99c004470 con 0x559f9ed9dcd0 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.530+0000 7fd9a97fa640 1 -- 172.21.15.23:0/3470004313 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c0035f0 con 0x559f9ed9dcd0 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.531+0000 7fd9bc21c740 1 -- 172.21.15.23:0/3470004313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed9dcd0 msgr2=0x559f9eda00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.531+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/3470004313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed9dcd0 0x559f9eda00c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009fa0 tx=0x7fd99c0118d0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.531+0000 7fd9bc21c740 1 -- 172.21.15.23:0/3470004313 shutdown_connections 2024-03-24T10:42:38.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.531+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/3470004313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed9dcd0 0x559f9eda00c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.531+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/3470004313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed94f00 0x559f9ed9d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.531+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/3470004313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9ed949c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.531+0000 7fd9bc21c740 1 -- 172.21.15.23:0/3470004313 >> 172.21.15.23:0/3470004313 conn(0x559f9ed8ef00 msgr2=0x559f9ed8f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:38.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.531+0000 7fd9bc21c740 1 -- 172.21.15.23:0/3470004313 shutdown_connections 2024-03-24T10:42:38.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.531+0000 7fd9bc21c740 1 -- 172.21.15.23:0/3470004313 wait complete. 2024-03-24T10:42:38.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.531+0000 7fd9bc21c740 1 -- start start 2024-03-24T10:42:38.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.531+0000 7fd9bc21c740 1 Processor -- start 2024-03-24T10:42:38.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9bc21c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9ed9ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9bc21c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9ed9b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9bc21c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9ed9b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9bc21c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x559f9ecfac90 con 0x559f9ed9dcd0 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9bc21c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559f9ec9fc50 con 0x559f9ed945e0 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9bc21c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559f9e9ce250 con 0x559f9ed94f00 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9bc21c740 10 client.0 _ll_register_callbacks cb 0x559f9ecee290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9b37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9ed9b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9b37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9ed9b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60418/0 (socket says 172.21.15.23:60418) 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9b37fe640 1 -- 172.21.15.23:0/1402913858 learned_addr learned my addr 172.21.15.23:0/1402913858 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9b2ffd640 1 --2- 172.21.15.23:0/1402913858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9ed9ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9abfff640 1 --2- 172.21.15.23:0/1402913858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9ed9b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9b2ffd640 1 -- 172.21.15.23:0/1402913858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 msgr2=0x559f9ed9b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9b2ffd640 1 --2- 172.21.15.23:0/1402913858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9ed9b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9b2ffd640 1 -- 172.21.15.23:0/1402913858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 msgr2=0x559f9ed9b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9b2ffd640 1 --2- 172.21.15.23:0/1402913858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9ed9b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9b2ffd640 1 -- 172.21.15.23:0/1402913858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd99c009c00 con 0x559f9ed945e0 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9b37fe640 1 --2- 172.21.15.23:0/1402913858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9ed9b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd9b2ffd640 1 --2- 172.21.15.23:0/1402913858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9ed9ad30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4006010 tx=0x7fd9a40036f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fd9a62b54f41676 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.532+0000 7fd99a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd99a7fc640 1 -- 172.21.15.23:0/1402913858 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a40bd780 con 0x559f9ed945e0 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd99a7fc640 1 -- 172.21.15.23:0/1402913858 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fd9a40bd080 con 0x559f9ed945e0 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd9bc21c740 1 -- 172.21.15.23:0/1402913858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559f9ed1f790 con 0x559f9ed945e0 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd99a7fc640 1 -- 172.21.15.23:0/1402913858 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd9a40c11d0 con 0x559f9ed945e0 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd9bc21c740 10 client.26159 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd9bc21c740 1 -- 172.21.15.23:0/1402913858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559f9ecf9310 con 0x559f9ed945e0 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd99a7fc640 1 -- 172.21.15.23:0/1402913858 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559f9ed1f790 con 0x559f9ed945e0 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd99a7fc640 1 -- 172.21.15.23:0/1402913858 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd9a403c070 con 0x559f9ed945e0 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd9bc21c740 10 client.26159 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd9bc21c740 10 client.26159 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd9bc21c740 20 client.26159 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/1402913858 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559f9edad970 0x559f9edafd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd9bc21c740 4 client.26159 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd9bc21c740 1 -- 172.21.15.23:0/1402913858 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f9e9fa600 con 0x559f9edad970 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.533+0000 7fd9abfff640 1 --2- 172.21.15.23:0/1402913858 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559f9edad970 0x559f9edafd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.534+0000 7fd99a7fc640 10 client.26159 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.534+0000 7fd9abfff640 1 --2- 172.21.15.23:0/1402913858 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559f9edad970 0x559f9edafd60 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d01581b127987618 server_cookie=140c19d0fcb5bfa1 in_seq=0 out_seq=0 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd99a7fc640 1 -- 172.21.15.23:0/1402913858 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559f9e9fa600 con 0x559f9edad970 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd99a7fc640 10 client.26159 handle_command_reply: tid=0 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd9bc21c740 1 client.26159 shutdown 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd9bc21c740 1 -- 172.21.15.23:0/1402913858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 msgr2=0x559f9ed9ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/1402913858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9ed9ad30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4006010 tx=0x7fd9a40036f0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd9bc21c740 1 -- 172.21.15.23:0/1402913858 shutdown_connections 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/1402913858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9ed9b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/1402913858 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559f9edad970 0x559f9edafd60 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/1402913858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9ed9b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/1402913858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9ed9ad30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd99a7fc640 0 client.26159 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd9bc21c740 1 -- 172.21.15.23:0/1402913858 >> 172.21.15.23:0/1402913858 conn(0x559f9ed8ef00 msgr2=0x559f9ed90c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd9bc21c740 1 -- 172.21.15.23:0/1402913858 shutdown_connections 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd9bc21c740 1 -- 172.21.15.23:0/1402913858 wait complete. 2024-03-24T10:42:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.535+0000 7fd9bc21c740 20 client.26159 trim_cache size 0 max 16384 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.553+0000 7fd9bc21c740 1 Processor -- start 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.553+0000 7fd9bc21c740 1 -- start start 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9bc21c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9ed9d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9bc21c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9ed90a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9bc21c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9edb5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9bc21c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x559f9ec80660 con 0x559f9ed9dcd0 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9bc21c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559f9ed8d3f0 con 0x559f9ed945e0 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9bc21c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559f9ed8ff00 con 0x559f9ed94f00 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9b37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9edb5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9abfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9ed90a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9b37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9edb5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60426/0 (socket says 172.21.15.23:60426) 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9b2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9ed9d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9b37fe640 1 -- 172.21.15.23:0/851970845 learned_addr learned my addr 172.21.15.23:0/851970845 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9abfff640 1 -- 172.21.15.23:0/851970845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 msgr2=0x559f9ed9d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9abfff640 1 --2- 172.21.15.23:0/851970845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9ed9d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9abfff640 1 -- 172.21.15.23:0/851970845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 msgr2=0x559f9edb5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9abfff640 1 --2- 172.21.15.23:0/851970845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9edb5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9abfff640 1 -- 172.21.15.23:0/851970845 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f9ec95810 con 0x559f9ed94f00 2024-03-24T10:42:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9b37fe640 1 --2- 172.21.15.23:0/851970845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9edb5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9abfff640 1 --2- 172.21.15.23:0/851970845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9ed90a60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd99c01b040 tx=0x7fd99c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81162b37a888347f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9a8ff9640 1 -- 172.21.15.23:0/851970845 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c029040 con 0x559f9ed94f00 2024-03-24T10:42:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.554+0000 7fd9a8ff9640 1 -- 172.21.15.23:0/851970845 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd99c02b7a0 con 0x559f9ed94f00 2024-03-24T10:42:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.555+0000 7fd9a8ff9640 1 -- 172.21.15.23:0/851970845 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c00f720 con 0x559f9ed94f00 2024-03-24T10:42:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 -- 172.21.15.23:0/851970845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 msgr2=0x559f9ed90a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/851970845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9ed90a60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd99c01b040 tx=0x7fd99c0118d0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 -- 172.21.15.23:0/851970845 shutdown_connections 2024-03-24T10:42:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/851970845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9edb5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/851970845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9ed90a60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/851970845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9ed9d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 -- 172.21.15.23:0/851970845 >> 172.21.15.23:0/851970845 conn(0x559f9ed8ef00 msgr2=0x559f9ed90350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 -- 172.21.15.23:0/851970845 shutdown_connections 2024-03-24T10:42:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 -- 172.21.15.23:0/851970845 wait complete. 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 -- start start 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 Processor -- start 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9eda1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9eda1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9eda1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x559f9ee12c30 con 0x559f9ed9dcd0 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559f9ee12d70 con 0x559f9ed945e0 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559f9ee12eb0 con 0x559f9ed94f00 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.556+0000 7fd9bc21c740 10 client.0 _ll_register_callbacks cb 0x7fd9a4009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9b37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9eda1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9b2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9eda1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9abfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9eda1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9b37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9eda1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60440/0 (socket says 172.21.15.23:60440) 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9b37fe640 1 -- 172.21.15.23:0/2798961005 learned_addr learned my addr 172.21.15.23:0/2798961005 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9abfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9eda1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:57092/0 (socket says 172.21.15.23:57092) 2024-03-24T10:42:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9b2ffd640 1 -- 172.21.15.23:0/2798961005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 msgr2=0x559f9eda1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9b2ffd640 1 --2- 172.21.15.23:0/2798961005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9eda1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9b2ffd640 1 -- 172.21.15.23:0/2798961005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 msgr2=0x559f9eda1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9b2ffd640 1 --2- 172.21.15.23:0/2798961005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9eda1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9b2ffd640 1 -- 172.21.15.23:0/2798961005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9ac019040 con 0x559f9ed945e0 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9b37fe640 1 --2- 172.21.15.23:0/2798961005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9eda1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9abfff640 1 --2- 172.21.15.23:0/2798961005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9eda1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9b2ffd640 1 --2- 172.21.15.23:0/2798961005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9eda1350 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40ee2d0 tx=0x7fd9a40c56e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3fc1210841f76ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd99a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd99a7fc640 1 -- 172.21.15.23:0/2798961005 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a409bea0 con 0x559f9ed945e0 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd99a7fc640 1 -- 172.21.15.23:0/2798961005 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fd9a40bd780 con 0x559f9ed945e0 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9bc21c740 1 -- 172.21.15.23:0/2798961005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd99c004470 con 0x559f9ed945e0 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd99a7fc640 1 -- 172.21.15.23:0/2798961005 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd9a40c5560 con 0x559f9ed945e0 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9bc21c740 10 client.26165 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.557+0000 7fd9bc21c740 1 -- 172.21.15.23:0/2798961005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559f9ec90810 con 0x559f9ed945e0 2024-03-24T10:42:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.558+0000 7fd99a7fc640 1 -- 172.21.15.23:0/2798961005 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd99c004470 con 0x559f9ed945e0 2024-03-24T10:42:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.558+0000 7fd99a7fc640 1 -- 172.21.15.23:0/2798961005 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd9a40a1050 con 0x559f9ed945e0 2024-03-24T10:42:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.558+0000 7fd9bc21c740 10 client.26165 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.558+0000 7fd9bc21c740 10 client.26165 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.558+0000 7fd9bc21c740 20 client.26165 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.558+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/2798961005 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559f9ee14200 0x559f9ee16620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.558+0000 7fd9bc21c740 4 client.26165 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:42:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.558+0000 7fd9bc21c740 1 -- 172.21.15.23:0/2798961005 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559f9ea4b1b0 con 0x559f9ee14200 2024-03-24T10:42:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.558+0000 7fd9abfff640 1 --2- 172.21.15.23:0/2798961005 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559f9ee14200 0x559f9ee16620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.558+0000 7fd9abfff640 1 --2- 172.21.15.23:0/2798961005 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559f9ee14200 0x559f9ee16620 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dc7c73636b29495 server_cookie=7382fc4e6afad8ec in_seq=0 out_seq=0 2024-03-24T10:42:38.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.558+0000 7fd99a7fc640 10 client.26165 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:38.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.566+0000 7fd99a7fc640 1 -- 172.21.15.23:0/2798961005 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559f9ea4b1b0 con 0x559f9ee14200 2024-03-24T10:42:38.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.566+0000 7fd99a7fc640 10 client.26165 handle_command_reply: tid=0 2024-03-24T10:42:38.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.566+0000 7fd9bc21c740 1 client.26165 shutdown 2024-03-24T10:42:38.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9bc21c740 1 -- 172.21.15.23:0/2798961005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 msgr2=0x559f9eda1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:38.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/2798961005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9eda1350 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40ee2d0 tx=0x7fd9a40c56e0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9bc21c740 1 -- 172.21.15.23:0/2798961005 shutdown_connections 2024-03-24T10:42:38.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/2798961005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f9ed9dcd0 0x559f9eda1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/2798961005 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559f9ee14200 0x559f9ee16620 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/2798961005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559f9ed94f00 0x559f9eda1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9bc21c740 1 --2- 172.21.15.23:0/2798961005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559f9ed945e0 0x559f9eda1350 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd99a7fc640 0 client.26165 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:38.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9bc21c740 1 -- 172.21.15.23:0/2798961005 >> 172.21.15.23:0/2798961005 conn(0x559f9ed8ef00 msgr2=0x559f9ed90b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:38.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9bc21c740 1 -- 172.21.15.23:0/2798961005 shutdown_connections 2024-03-24T10:42:38.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9bc21c740 1 -- 172.21.15.23:0/2798961005 wait complete. 2024-03-24T10:42:38.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9bc21c740 20 client.26165 trim_cache size 0 max 16384 2024-03-24T10:42:38.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:42:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:42:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:42:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:42:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9b9642640 1 -- 172.21.15.23:0/798463704 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9a00a2350 msgr2=0x7fd9a00a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9b9642640 1 --2- 172.21.15.23:0/798463704 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9a00a2350 0x7fd9a00a4810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac005fd0 tx=0x7fd9ac009500 comp rx=0 tx=0).stop 2024-03-24T10:42:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9b9642640 1 -- 172.21.15.23:0/798463704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b40ffc10 msgr2=0x7fd9b419da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9b9642640 1 --2- 172.21.15.23:0/798463704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b40ffc10 0x7fd9b419da70 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4000c00 tx=0x7fd9a402c5f0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9b9642640 1 -- 172.21.15.23:0/798463704 shutdown_connections 2024-03-24T10:42:38.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9b9642640 1 --2- 172.21.15.23:0/798463704 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9a00a2350 0x7fd9a00a4810 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9b9642640 1 --2- 172.21.15.23:0/798463704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b4107730 0x7fd9b4197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9b9642640 1 --2- 172.21.15.23:0/798463704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b41005d0 0x7fd9b419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9b9642640 1 --2- 172.21.15.23:0/798463704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b40ffc10 0x7fd9b419da70 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:38.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.567+0000 7fd9b9642640 1 -- 172.21.15.23:0/798463704 >> 172.21.15.23:0/798463704 conn(0x7fd9b4076ba0 msgr2=0x7fd9b4104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:38.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.568+0000 7fd9b9642640 1 -- 172.21.15.23:0/798463704 shutdown_connections 2024-03-24T10:42:38.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:38.568+0000 7fd9b9642640 1 -- 172.21.15.23:0/798463704 wait complete. 2024-03-24T10:42:38.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:38 smithi023 ceph-mon[32466]: pgmap v79: 97 pgs: 97 active+clean; 81 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.2 MiB/s wr, 321 op/s 2024-03-24T10:42:38.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:38 smithi117 ceph-mon[36368]: pgmap v79: 97 pgs: 97 active+clean; 81 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.2 MiB/s wr, 321 op/s 2024-03-24T10:42:38.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:38 smithi116 ceph-mon[36248]: pgmap v79: 97 pgs: 97 active+clean; 81 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.2 MiB/s wr, 321 op/s 2024-03-24T10:42:39.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 130117 12.79 MB/sec warmup 56 sec latency 16.990 ms 2024-03-24T10:42:40.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 131883 12.78 MB/sec warmup 57 sec latency 81.030 ms 2024-03-24T10:42:40.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:40 smithi023 ceph-mon[32466]: pgmap v80: 97 pgs: 97 active+clean; 81 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 212 op/s 2024-03-24T10:42:40.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:40 smithi117 ceph-mon[36368]: pgmap v80: 97 pgs: 97 active+clean; 81 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 212 op/s 2024-03-24T10:42:40.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:40 smithi116 ceph-mon[36248]: pgmap v80: 97 pgs: 97 active+clean; 81 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 212 op/s 2024-03-24T10:42:41.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 134787 12.86 MB/sec warmup 58 sec latency 24.689 ms 2024-03-24T10:42:42.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 136506 12.76 MB/sec warmup 59 sec latency 17.893 ms 2024-03-24T10:42:42.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:42 smithi023 ceph-mon[32466]: pgmap v81: 97 pgs: 97 active+clean; 85 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 319 op/s 2024-03-24T10:42:42.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:42 smithi117 ceph-mon[36368]: pgmap v81: 97 pgs: 97 active+clean; 85 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 319 op/s 2024-03-24T10:42:42.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:42 smithi116 ceph-mon[36248]: pgmap v81: 97 pgs: 97 active+clean; 85 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 319 op/s 2024-03-24T10:42:43.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 138935 12.78 MB/sec warmup 60 sec latency 12.814 ms 2024-03-24T10:42:44.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 141742 12.81 MB/sec warmup 61 sec latency 18.350 ms 2024-03-24T10:42:44.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:44 smithi023 ceph-mon[32466]: pgmap v82: 97 pgs: 97 active+clean; 85 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 212 op/s 2024-03-24T10:42:44.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:44 smithi117 ceph-mon[36368]: pgmap v82: 97 pgs: 97 active+clean; 85 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 212 op/s 2024-03-24T10:42:44.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:44 smithi116 ceph-mon[36248]: pgmap v82: 97 pgs: 97 active+clean; 85 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 212 op/s 2024-03-24T10:42:45.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 143098 12.71 MB/sec warmup 62 sec latency 105.459 ms 2024-03-24T10:42:46.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 145754 12.77 MB/sec warmup 63 sec latency 23.900 ms 2024-03-24T10:42:46.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:46 smithi023 ceph-mon[32466]: pgmap v83: 97 pgs: 97 active+clean; 85 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 212 op/s 2024-03-24T10:42:46.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:46 smithi117 ceph-mon[36368]: pgmap v83: 97 pgs: 97 active+clean; 85 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 212 op/s 2024-03-24T10:42:46.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:46 smithi116 ceph-mon[36248]: pgmap v83: 97 pgs: 97 active+clean; 85 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.8 MiB/s wr, 212 op/s 2024-03-24T10:42:47.113 INFO:tasks.workunit.client.0.smithi023.stdout: 1 148019 12.70 MB/sec warmup 64 sec latency 16.152 ms 2024-03-24T10:42:48.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 149924 12.69 MB/sec warmup 65 sec latency 19.559 ms 2024-03-24T10:42:48.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:48 smithi023 ceph-mon[32466]: pgmap v84: 97 pgs: 97 active+clean; 91 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 321 op/s 2024-03-24T10:42:48.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:48 smithi117 ceph-mon[36368]: pgmap v84: 97 pgs: 97 active+clean; 91 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 321 op/s 2024-03-24T10:42:48.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:48 smithi116 ceph-mon[36248]: pgmap v84: 97 pgs: 97 active+clean; 91 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 321 op/s 2024-03-24T10:42:49.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 152779 12.77 MB/sec warmup 66 sec latency 14.829 ms 2024-03-24T10:42:50.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 154325 12.67 MB/sec warmup 67 sec latency 71.379 ms 2024-03-24T10:42:50.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:50 smithi023 ceph-mon[32466]: pgmap v85: 97 pgs: 97 active+clean; 91 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 216 op/s 2024-03-24T10:42:50.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:50 smithi117 ceph-mon[36368]: pgmap v85: 97 pgs: 97 active+clean; 91 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 216 op/s 2024-03-24T10:42:50.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:50 smithi116 ceph-mon[36248]: pgmap v85: 97 pgs: 97 active+clean; 91 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 216 op/s 2024-03-24T10:42:51.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 156667 12.68 MB/sec warmup 68 sec latency 22.344 ms 2024-03-24T10:42:51.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.658+0000 7f0156302640 1 -- 172.21.15.23:0/3502605497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f015010bd80 msgr2=0x7f015010e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:51.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.658+0000 7f0156302640 1 --2- 172.21.15.23:0/3502605497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f015010bd80 0x7f015010e1d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0148009ca0 tx=0x7f014802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.658+0000 7f0156302640 1 -- 172.21.15.23:0/3502605497 shutdown_connections 2024-03-24T10:42:51.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.658+0000 7f0156302640 1 --2- 172.21.15.23:0/3502605497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f015010bd80 0x7f015010e1d0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.658+0000 7f0156302640 1 --2- 172.21.15.23:0/3502605497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0150103230 0x7f015010b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.658+0000 7f0156302640 1 --2- 172.21.15.23:0/3502605497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0150102910 0x7f0150102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.658+0000 7f0156302640 1 -- 172.21.15.23:0/3502605497 >> 172.21.15.23:0/3502605497 conn(0x7f0150077740 msgr2=0x7f0150077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:51.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.658+0000 7f0156302640 1 -- 172.21.15.23:0/3502605497 shutdown_connections 2024-03-24T10:42:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.658+0000 7f0156302640 1 -- 172.21.15.23:0/3502605497 wait complete. 2024-03-24T10:42:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.659+0000 7f0156302640 1 Processor -- start 2024-03-24T10:42:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.659+0000 7f0156302640 1 -- start start 2024-03-24T10:42:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.659+0000 7f0156302640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0150102910 0x7f015019fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.659+0000 7f0156302640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0150103230 0x7f01501a0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.659+0000 7f0156302640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f015010bd80 0x7f015019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.659+0000 7f0156302640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f015019a780 con 0x7f0150102910 2024-03-24T10:42:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.659+0000 7f0156302640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f015019a8f0 con 0x7f0150103230 2024-03-24T10:42:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.659+0000 7f0156302640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f015019aab0 con 0x7f015010bd80 2024-03-24T10:42:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.659+0000 7f014ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0150102910 0x7f015019fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.659+0000 7f015489f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f015010bd80 0x7f015019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.659+0000 7f014ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0150102910 0x7f015019fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58338/0 (socket says 172.21.15.23:58338) 2024-03-24T10:42:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.659+0000 7f014ffff640 1 -- 172.21.15.23:0/2132653850 learned_addr learned my addr 172.21.15.23:0/2132653850 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.659+0000 7f014f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0150103230 0x7f01501a0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.660+0000 7f015489f640 1 -- 172.21.15.23:0/2132653850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0150103230 msgr2=0x7f01501a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.660+0000 7f015489f640 1 --2- 172.21.15.23:0/2132653850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0150103230 0x7f01501a0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.660+0000 7f015489f640 1 -- 172.21.15.23:0/2132653850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0150102910 msgr2=0x7f015019fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.660+0000 7f015489f640 1 --2- 172.21.15.23:0/2132653850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0150102910 0x7f015019fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.660+0000 7f015489f640 1 -- 172.21.15.23:0/2132653850 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0148009900 con 0x7f015010bd80 2024-03-24T10:42:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.660+0000 7f014f7fe640 1 --2- 172.21.15.23:0/2132653850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0150103230 0x7f01501a0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.660+0000 7f014ffff640 1 --2- 172.21.15.23:0/2132653850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0150102910 0x7f015019fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.660+0000 7f015489f640 1 --2- 172.21.15.23:0/2132653850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f015010bd80 0x7f015019a130 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f014802c700 tx=0x7f0148034a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.660+0000 7f014d7fa640 1 -- 172.21.15.23:0/2132653850 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01480328b0 con 0x7f015010bd80 2024-03-24T10:42:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.660+0000 7f0156302640 1 -- 172.21.15.23:0/2132653850 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f015019ad30 con 0x7f015010bd80 2024-03-24T10:42:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.660+0000 7f014d7fa640 1 -- 172.21.15.23:0/2132653850 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0148003e50 con 0x7f015010bd80 2024-03-24T10:42:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.660+0000 7f014d7fa640 1 -- 172.21.15.23:0/2132653850 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0148004860 con 0x7f015010bd80 2024-03-24T10:42:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.660+0000 7f0156302640 1 -- 172.21.15.23:0/2132653850 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01501a6b00 con 0x7f015010bd80 2024-03-24T10:42:51.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.662+0000 7f014d7fa640 1 -- 172.21.15.23:0/2132653850 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f014800a040 con 0x7f015010bd80 2024-03-24T10:42:51.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f014d7fa640 1 --2- 172.21.15.23:0/2132653850 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f013c0a22d0 0x7f013c0a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f0158fbe740 1 Processor -- start 2024-03-24T10:42:51.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f0158fbe740 1 -- start start 2024-03-24T10:42:51.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f014ffff640 1 --2- 172.21.15.23:0/2132653850 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f013c0a22d0 0x7f013c0a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f014d7fa640 1 -- 172.21.15.23:0/2132653850 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f01480f0c40 con 0x7f015010bd80 2024-03-24T10:42:51.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f0158fbe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e5785e0 0x55bd6e5789c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f0158fbe740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e578f00 0x55bd6e581790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f0158fbe740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e5840c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f0158fbe740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bd6e5331b0 con 0x55bd6e578f00 2024-03-24T10:42:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f0158fbe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bd6e4e5f80 con 0x55bd6e5785e0 2024-03-24T10:42:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f0158fbe740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bd6e4e34b0 con 0x55bd6e581cd0 2024-03-24T10:42:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f014f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e578f00 0x55bd6e581790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f014f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e578f00 0x55bd6e581790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58342/0 (socket says 172.21.15.23:58342) 2024-03-24T10:42:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f014f7fe640 1 -- 172.21.15.23:0/1788496385 learned_addr learned my addr 172.21.15.23:0/1788496385 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f014ffff640 1 --2- 172.21.15.23:0/1788496385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e5785e0 0x55bd6e5789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f015489f640 1 --2- 172.21.15.23:0/1788496385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e5840c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f014ffff640 1 --2- 172.21.15.23:0/2132653850 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f013c0a22d0 0x7f013c0a4790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0140011570 tx=0x7f0140011970 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f014ffff640 1 -- 172.21.15.23:0/1788496385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 msgr2=0x55bd6e5840c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f014ffff640 1 --2- 172.21.15.23:0/1788496385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e5840c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f014ffff640 1 -- 172.21.15.23:0/1788496385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e578f00 msgr2=0x55bd6e581790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.663+0000 7f014ffff640 1 --2- 172.21.15.23:0/1788496385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e578f00 0x55bd6e581790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.664+0000 7f014ffff640 1 -- 172.21.15.23:0/1788496385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd6e474810 con 0x55bd6e5785e0 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.664+0000 7f014f7fe640 1 --2- 172.21.15.23:0/1788496385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e578f00 0x55bd6e581790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.664+0000 7f014ffff640 1 --2- 172.21.15.23:0/1788496385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e5785e0 0x55bd6e5789c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f014000f880 tx=0x7f014001c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5397521298be8d30 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.664+0000 7f01467fc640 1 -- 172.21.15.23:0/1788496385 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0140005220 con 0x55bd6e5785e0 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.664+0000 7f01467fc640 1 -- 172.21.15.23:0/1788496385 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0140004c80 con 0x55bd6e5785e0 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.664+0000 7f01467fc640 1 -- 172.21.15.23:0/1788496385 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014001d8e0 con 0x55bd6e5785e0 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 -- 172.21.15.23:0/1788496385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e5785e0 msgr2=0x55bd6e5789c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 --2- 172.21.15.23:0/1788496385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e5785e0 0x55bd6e5789c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f014000f880 tx=0x7f014001c000 comp rx=0 tx=0).stop 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 -- 172.21.15.23:0/1788496385 shutdown_connections 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 --2- 172.21.15.23:0/1788496385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e5840c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 --2- 172.21.15.23:0/1788496385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e578f00 0x55bd6e581790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 --2- 172.21.15.23:0/1788496385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e5785e0 0x55bd6e5789c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 -- 172.21.15.23:0/1788496385 >> 172.21.15.23:0/1788496385 conn(0x55bd6e572f00 msgr2=0x55bd6e5732e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 -- 172.21.15.23:0/1788496385 shutdown_connections 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 -- 172.21.15.23:0/1788496385 wait complete. 2024-03-24T10:42:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 -- start start 2024-03-24T10:42:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 Processor -- start 2024-03-24T10:42:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e5785e0 0x55bd6e57edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e578f00 0x55bd6e57f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e57f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bd6e4dec90 con 0x55bd6e5785e0 2024-03-24T10:42:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bd6e483c50 con 0x55bd6e578f00 2024-03-24T10:42:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bd6e1b2250 con 0x55bd6e581cd0 2024-03-24T10:42:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.665+0000 7f0158fbe740 10 client.0 _ll_register_callbacks cb 0x55bd6e4d2290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f014ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e5785e0 0x55bd6e57edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f014f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e578f00 0x55bd6e57f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f014ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e5785e0 0x55bd6e57edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58354/0 (socket says 172.21.15.23:58354) 2024-03-24T10:42:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f014ffff640 1 -- 172.21.15.23:0/3600307824 learned_addr learned my addr 172.21.15.23:0/3600307824 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:51.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f015489f640 1 --2- 172.21.15.23:0/3600307824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e57f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f014ffff640 1 -- 172.21.15.23:0/3600307824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 msgr2=0x55bd6e57f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f014ffff640 1 --2- 172.21.15.23:0/3600307824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e57f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f014ffff640 1 -- 172.21.15.23:0/3600307824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e578f00 msgr2=0x55bd6e57f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f014ffff640 1 --2- 172.21.15.23:0/3600307824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e578f00 0x55bd6e57f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f014ffff640 1 -- 172.21.15.23:0/3600307824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f014000f640 con 0x55bd6e5785e0 2024-03-24T10:42:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f014ffff640 1 --2- 172.21.15.23:0/3600307824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e5785e0 0x55bd6e57edd0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f014001da00 tx=0x7f0140005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6e628e9857b0efa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f012f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f012f7fe640 1 -- 172.21.15.23:0/3600307824 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014002b070 con 0x55bd6e5785e0 2024-03-24T10:42:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f012f7fe640 1 -- 172.21.15.23:0/3600307824 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f01400046c0 con 0x55bd6e5785e0 2024-03-24T10:42:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f0158fbe740 1 -- 172.21.15.23:0/3600307824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bd6e503790 con 0x55bd6e5785e0 2024-03-24T10:42:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f012f7fe640 1 -- 172.21.15.23:0/3600307824 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0140004880 con 0x55bd6e5785e0 2024-03-24T10:42:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f0158fbe740 10 client.16299 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:51.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.666+0000 7f0158fbe740 1 -- 172.21.15.23:0/3600307824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bd6e4dd310 con 0x55bd6e5785e0 2024-03-24T10:42:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.667+0000 7f012f7fe640 1 -- 172.21.15.23:0/3600307824 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bd6e503790 con 0x55bd6e5785e0 2024-03-24T10:42:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.667+0000 7f012f7fe640 1 -- 172.21.15.23:0/3600307824 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f01400255b0 con 0x55bd6e5785e0 2024-03-24T10:42:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.667+0000 7f0158fbe740 10 client.16299 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.667+0000 7f0158fbe740 10 client.16299 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.667+0000 7f0158fbe740 20 client.16299 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.667+0000 7f0158fbe740 1 --2- 172.21.15.23:0/3600307824 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd6e59a3f0 0x55bd6e59c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.667+0000 7f0158fbe740 4 client.16299 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.667+0000 7f0158fbe740 1 -- 172.21.15.23:0/3600307824 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd6e1de600 con 0x55bd6e59a3f0 2024-03-24T10:42:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.667+0000 7f014f7fe640 1 --2- 172.21.15.23:0/3600307824 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd6e59a3f0 0x55bd6e59c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.667+0000 7f012f7fe640 10 client.16299 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.667+0000 7f014f7fe640 1 --2- 172.21.15.23:0/3600307824 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd6e59a3f0 0x55bd6e59c830 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6e27fb204d46c5c server_cookie=17ed7bebf10ff06a in_seq=0 out_seq=0 2024-03-24T10:42:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f012f7fe640 1 -- 172.21.15.23:0/3600307824 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bd6e1de600 con 0x55bd6e59a3f0 2024-03-24T10:42:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f012f7fe640 10 client.16299 handle_command_reply: tid=0 2024-03-24T10:42:51.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f0158fbe740 1 client.16299 shutdown 2024-03-24T10:42:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f0158fbe740 1 -- 172.21.15.23:0/3600307824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e5785e0 msgr2=0x55bd6e57edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f0158fbe740 1 --2- 172.21.15.23:0/3600307824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e5785e0 0x55bd6e57edd0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f014001da00 tx=0x7f0140005d30 comp rx=0 tx=0).stop 2024-03-24T10:42:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f0158fbe740 1 -- 172.21.15.23:0/3600307824 shutdown_connections 2024-03-24T10:42:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f0158fbe740 1 --2- 172.21.15.23:0/3600307824 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd6e59a3f0 0x55bd6e59c830 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f0158fbe740 1 --2- 172.21.15.23:0/3600307824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e57f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f012f7fe640 0 client.16299 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f0158fbe740 1 --2- 172.21.15.23:0/3600307824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e578f00 0x55bd6e57f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f0158fbe740 1 --2- 172.21.15.23:0/3600307824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e5785e0 0x55bd6e57edd0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f0158fbe740 1 -- 172.21.15.23:0/3600307824 >> 172.21.15.23:0/3600307824 conn(0x55bd6e572f00 msgr2=0x55bd6e574c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f0158fbe740 1 -- 172.21.15.23:0/3600307824 shutdown_connections 2024-03-24T10:42:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f0158fbe740 1 -- 172.21.15.23:0/3600307824 wait complete. 2024-03-24T10:42:51.676 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.668+0000 7f0158fbe740 20 client.16299 trim_cache size 0 max 16384 2024-03-24T10:42:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.683+0000 7f0158fbe740 1 Processor -- start 2024-03-24T10:42:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.683+0000 7f0158fbe740 1 -- start start 2024-03-24T10:42:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f0158fbe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e5785e0 0x55bd6e581500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f0158fbe740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e578f00 0x55bd6e58d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f0158fbe740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e5a2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f0158fbe740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bd6e464660 con 0x55bd6e578f00 2024-03-24T10:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f0158fbe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bd6e58e730 con 0x55bd6e5785e0 2024-03-24T10:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f0158fbe740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bd6e573ad0 con 0x55bd6e581cd0 2024-03-24T10:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f014f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e578f00 0x55bd6e58d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f014f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e578f00 0x55bd6e58d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58370/0 (socket says 172.21.15.23:58370) 2024-03-24T10:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f014f7fe640 1 -- 172.21.15.23:0/2192020644 learned_addr learned my addr 172.21.15.23:0/2192020644 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f014ffff640 1 --2- 172.21.15.23:0/2192020644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e5785e0 0x55bd6e581500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f015489f640 1 --2- 172.21.15.23:0/2192020644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e5a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f014f7fe640 1 -- 172.21.15.23:0/2192020644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 msgr2=0x55bd6e5a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f014f7fe640 1 --2- 172.21.15.23:0/2192020644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e5a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f014f7fe640 1 -- 172.21.15.23:0/2192020644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e5785e0 msgr2=0x55bd6e581500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f014f7fe640 1 --2- 172.21.15.23:0/2192020644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e5785e0 0x55bd6e581500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f014f7fe640 1 -- 172.21.15.23:0/2192020644 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd6e479810 con 0x55bd6e578f00 2024-03-24T10:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f014f7fe640 1 --2- 172.21.15.23:0/2192020644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e578f00 0x55bd6e58d3a0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x55bd6e5803a0 tx=0x7f0138005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330caace3d11c2db server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:51.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f0145ffb640 1 -- 172.21.15.23:0/2192020644 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138017040 con 0x55bd6e578f00 2024-03-24T10:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f0145ffb640 1 -- 172.21.15.23:0/2192020644 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0138024070 con 0x55bd6e578f00 2024-03-24T10:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.684+0000 7f0145ffb640 1 -- 172.21.15.23:0/2192020644 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138005440 con 0x55bd6e578f00 2024-03-24T10:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.685+0000 7f0158fbe740 1 -- 172.21.15.23:0/2192020644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e578f00 msgr2=0x55bd6e58d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.685+0000 7f0158fbe740 1 --2- 172.21.15.23:0/2192020644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e578f00 0x55bd6e58d3a0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x55bd6e5803a0 tx=0x7f0138005f90 comp rx=0 tx=0).stop 2024-03-24T10:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.685+0000 7f0158fbe740 1 -- 172.21.15.23:0/2192020644 shutdown_connections 2024-03-24T10:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.685+0000 7f0158fbe740 1 --2- 172.21.15.23:0/2192020644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e5a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.685+0000 7f0158fbe740 1 --2- 172.21.15.23:0/2192020644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e578f00 0x55bd6e58d3a0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.685+0000 7f0158fbe740 1 --2- 172.21.15.23:0/2192020644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e5785e0 0x55bd6e581500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.685+0000 7f0158fbe740 1 -- 172.21.15.23:0/2192020644 >> 172.21.15.23:0/2192020644 conn(0x55bd6e572f00 msgr2=0x55bd6e57a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.685+0000 7f0158fbe740 1 -- 172.21.15.23:0/2192020644 shutdown_connections 2024-03-24T10:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.685+0000 7f0158fbe740 1 -- 172.21.15.23:0/2192020644 wait complete. 2024-03-24T10:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.685+0000 7f0158fbe740 1 -- start start 2024-03-24T10:42:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f0158fbe740 1 Processor -- start 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f0158fbe740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e5785e0 0x55bd6e589b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f0158fbe740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e578f00 0x55bd6e584f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f0158fbe740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e5854d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f0158fbe740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bd6e585a60 con 0x55bd6e5785e0 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f0158fbe740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bd6e585bd0 con 0x55bd6e578f00 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f0158fbe740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bd6e585d90 con 0x55bd6e581cd0 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f0158fbe740 10 client.0 _ll_register_callbacks cb 0x7f0138009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f014ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e5785e0 0x55bd6e589b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f014ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e5785e0 0x55bd6e589b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58374/0 (socket says 172.21.15.23:58374) 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f014ffff640 1 -- 172.21.15.23:0/2996991132 learned_addr learned my addr 172.21.15.23:0/2996991132 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f014f7fe640 1 --2- 172.21.15.23:0/2996991132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e578f00 0x55bd6e584f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f015489f640 1 --2- 172.21.15.23:0/2996991132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e5854d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f014ffff640 1 -- 172.21.15.23:0/2996991132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 msgr2=0x55bd6e5854d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f014ffff640 1 --2- 172.21.15.23:0/2996991132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e5854d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f014ffff640 1 -- 172.21.15.23:0/2996991132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e578f00 msgr2=0x55bd6e584f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f014ffff640 1 --2- 172.21.15.23:0/2996991132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e578f00 0x55bd6e584f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.686+0000 7f014ffff640 1 -- 172.21.15.23:0/2996991132 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01380120f0 con 0x55bd6e5785e0 2024-03-24T10:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f012f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f014ffff640 1 --2- 172.21.15.23:0/2996991132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e5785e0 0x55bd6e589b20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f0140005e00 tx=0x7f014000fef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970af4964459315 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f012f7fe640 1 -- 172.21.15.23:0/2996991132 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014002b040 con 0x55bd6e5785e0 2024-03-24T10:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f012f7fe640 1 -- 172.21.15.23:0/2996991132 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0140025600 con 0x55bd6e5785e0 2024-03-24T10:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f0158fbe740 1 -- 172.21.15.23:0/2996991132 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0140004c80 con 0x55bd6e5785e0 2024-03-24T10:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f012f7fe640 1 -- 172.21.15.23:0/2996991132 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f014001da30 con 0x55bd6e5785e0 2024-03-24T10:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f0158fbe740 10 client.16305 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f0158fbe740 1 -- 172.21.15.23:0/2996991132 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bd6e49f5a0 con 0x55bd6e5785e0 2024-03-24T10:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f012f7fe640 1 -- 172.21.15.23:0/2996991132 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0140045020 con 0x55bd6e5785e0 2024-03-24T10:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f012f7fe640 1 -- 172.21.15.23:0/2996991132 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f0140005470 con 0x55bd6e5785e0 2024-03-24T10:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f0158fbe740 10 client.16305 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f0158fbe740 10 client.16305 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f0158fbe740 20 client.16305 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f0158fbe740 1 --2- 172.21.15.23:0/2996991132 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd6e600b90 0x55bd6e602fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f0158fbe740 4 client.16305 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:42:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f0158fbe740 1 -- 172.21.15.23:0/2996991132 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bd6e22f1b0 con 0x55bd6e600b90 2024-03-24T10:42:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.687+0000 7f014f7fe640 1 --2- 172.21.15.23:0/2996991132 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd6e600b90 0x55bd6e602fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.688+0000 7f012f7fe640 10 client.16305 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.688+0000 7f014f7fe640 1 --2- 172.21.15.23:0/2996991132 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd6e600b90 0x55bd6e602fb0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bd820c6e54115cd server_cookie=4132f2a3eae5173c in_seq=0 out_seq=0 2024-03-24T10:42:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.698+0000 7f012f7fe640 1 -- 172.21.15.23:0/2996991132 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bd6e22f1b0 con 0x55bd6e600b90 2024-03-24T10:42:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.698+0000 7f012f7fe640 10 client.16305 handle_command_reply: tid=0 2024-03-24T10:42:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.698+0000 7f0158fbe740 1 client.16305 shutdown 2024-03-24T10:42:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0158fbe740 1 -- 172.21.15.23:0/2996991132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e5785e0 msgr2=0x55bd6e589b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0158fbe740 1 --2- 172.21.15.23:0/2996991132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e5785e0 0x55bd6e589b20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f0140005e00 tx=0x7f014000fef0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0158fbe740 1 -- 172.21.15.23:0/2996991132 shutdown_connections 2024-03-24T10:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0158fbe740 1 --2- 172.21.15.23:0/2996991132 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd6e600b90 0x55bd6e602fb0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0158fbe740 1 --2- 172.21.15.23:0/2996991132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd6e581cd0 0x55bd6e5854d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0158fbe740 1 --2- 172.21.15.23:0/2996991132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd6e578f00 0x55bd6e584f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f012f7fe640 0 client.16305 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0158fbe740 1 --2- 172.21.15.23:0/2996991132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd6e5785e0 0x55bd6e589b20 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0158fbe740 1 -- 172.21.15.23:0/2996991132 >> 172.21.15.23:0/2996991132 conn(0x55bd6e572f00 msgr2=0x55bd6e571610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0158fbe740 1 -- 172.21.15.23:0/2996991132 shutdown_connections 2024-03-24T10:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0158fbe740 1 -- 172.21.15.23:0/2996991132 wait complete. 2024-03-24T10:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0158fbe740 20 client.16305 trim_cache size 0 max 16384 2024-03-24T10:42:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:42:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:42:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:42:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:42:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0156302640 1 -- 172.21.15.23:0/2132653850 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f013c0a22d0 msgr2=0x7f013c0a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0156302640 1 --2- 172.21.15.23:0/2132653850 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f013c0a22d0 0x7f013c0a4790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0140011570 tx=0x7f0140011970 comp rx=0 tx=0).stop 2024-03-24T10:42:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0156302640 1 -- 172.21.15.23:0/2132653850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f015010bd80 msgr2=0x7f015019a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.699+0000 7f0156302640 1 --2- 172.21.15.23:0/2132653850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f015010bd80 0x7f015019a130 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f014802c700 tx=0x7f0148034a90 comp rx=0 tx=0).stop 2024-03-24T10:42:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.700+0000 7f0156302640 1 -- 172.21.15.23:0/2132653850 shutdown_connections 2024-03-24T10:42:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.700+0000 7f0156302640 1 --2- 172.21.15.23:0/2132653850 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f013c0a22d0 0x7f013c0a4790 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.700+0000 7f0156302640 1 --2- 172.21.15.23:0/2132653850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f015010bd80 0x7f015019a130 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.700+0000 7f0156302640 1 --2- 172.21.15.23:0/2132653850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0150103230 0x7f01501a0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.700+0000 7f0156302640 1 --2- 172.21.15.23:0/2132653850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0150102910 0x7f015019fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.700+0000 7f0156302640 1 -- 172.21.15.23:0/2132653850 >> 172.21.15.23:0/2132653850 conn(0x7f0150077740 msgr2=0x7f015010cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.700+0000 7f0156302640 1 -- 172.21.15.23:0/2132653850 shutdown_connections 2024-03-24T10:42:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:42:51.700+0000 7f0156302640 1 -- 172.21.15.23:0/2132653850 wait complete. 2024-03-24T10:42:52.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 159443 12.71 MB/sec warmup 69 sec latency 24.055 ms 2024-03-24T10:42:52.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:52 smithi117 ceph-mon[36368]: pgmap v86: 97 pgs: 97 active+clean; 96 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.6 MiB/s wr, 318 op/s 2024-03-24T10:42:52.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:52 smithi116 ceph-mon[36248]: pgmap v86: 97 pgs: 97 active+clean; 96 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.6 MiB/s wr, 318 op/s 2024-03-24T10:42:52.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:52 smithi023 ceph-mon[32466]: pgmap v86: 97 pgs: 97 active+clean; 96 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.6 MiB/s wr, 318 op/s 2024-03-24T10:42:53.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 161248 12.67 MB/sec warmup 70 sec latency 19.872 ms 2024-03-24T10:42:54.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 163783 12.69 MB/sec warmup 71 sec latency 13.091 ms 2024-03-24T10:42:54.234 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:42:54.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:42:54.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.344+0000 7fe432e15640 1 -- 172.21.15.23:0/4196239313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe42c100f60 msgr2=0x7fe42c10d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.344+0000 7fe432e15640 1 --2- 172.21.15.23:0/4196239313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe42c100f60 0x7fe42c10d420 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe424009ca0 tx=0x7fe42402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.344+0000 7fe432e15640 1 -- 172.21.15.23:0/4196239313 shutdown_connections 2024-03-24T10:42:54.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.344+0000 7fe432e15640 1 --2- 172.21.15.23:0/4196239313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe42c100f60 0x7fe42c10d420 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.344+0000 7fe432e15640 1 --2- 172.21.15.23:0/4196239313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe42c1005a0 0x7fe42c100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.344+0000 7fe432e15640 1 --2- 172.21.15.23:0/4196239313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe42c106660 0x7fe42c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.344+0000 7fe432e15640 1 -- 172.21.15.23:0/4196239313 >> 172.21.15.23:0/4196239313 conn(0x7fe42c077710 msgr2=0x7fe42c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:54.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.344+0000 7fe432e15640 1 -- 172.21.15.23:0/4196239313 shutdown_connections 2024-03-24T10:42:54.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.344+0000 7fe432e15640 1 -- 172.21.15.23:0/4196239313 wait complete. 2024-03-24T10:42:54.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe432e15640 1 Processor -- start 2024-03-24T10:42:54.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe432e15640 1 -- start start 2024-03-24T10:42:54.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe432e15640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe42c1005a0 0x7fe42c198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe432e15640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe42c100f60 0x7fe42c198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe432e15640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe42c106660 0x7fe42c1a3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe432e15640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c199860 con 0x7fe42c106660 2024-03-24T10:42:54.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe432e15640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c1a4060 con 0x7fe42c1005a0 2024-03-24T10:42:54.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe432e15640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c1a41a0 con 0x7fe42c100f60 2024-03-24T10:42:54.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe4313b2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe42c106660 0x7fe42c1a3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe4313b2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe42c106660 0x7fe42c1a3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58392/0 (socket says 172.21.15.23:58392) 2024-03-24T10:42:54.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe4313b2640 1 -- 172.21.15.23:0/1755766805 learned_addr learned my addr 172.21.15.23:0/1755766805 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:54.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe430bb1640 1 --2- 172.21.15.23:0/1755766805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe42c1005a0 0x7fe42c198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe42bfff640 1 --2- 172.21.15.23:0/1755766805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe42c100f60 0x7fe42c198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe4313b2640 1 -- 172.21.15.23:0/1755766805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe42c100f60 msgr2=0x7fe42c198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe4313b2640 1 --2- 172.21.15.23:0/1755766805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe42c100f60 0x7fe42c198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe4313b2640 1 -- 172.21.15.23:0/1755766805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe42c1005a0 msgr2=0x7fe42c198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe4313b2640 1 --2- 172.21.15.23:0/1755766805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe42c1005a0 0x7fe42c198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.345+0000 7fe4313b2640 1 -- 172.21.15.23:0/1755766805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe424009900 con 0x7fe42c106660 2024-03-24T10:42:54.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.346+0000 7fe4313b2640 1 --2- 172.21.15.23:0/1755766805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe42c106660 0x7fe42c1a3b20 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fe42402c700 tx=0x7fe424035aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:54.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.346+0000 7fe429ffb640 1 -- 172.21.15.23:0/1755766805 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe424035ea0 con 0x7fe42c106660 2024-03-24T10:42:54.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.346+0000 7fe429ffb640 1 -- 172.21.15.23:0/1755766805 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe424003e50 con 0x7fe42c106660 2024-03-24T10:42:54.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.346+0000 7fe429ffb640 1 -- 172.21.15.23:0/1755766805 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe424004860 con 0x7fe42c106660 2024-03-24T10:42:54.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.346+0000 7fe432e15640 1 -- 172.21.15.23:0/1755766805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe42c1a43a0 con 0x7fe42c106660 2024-03-24T10:42:54.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.346+0000 7fe432e15640 1 -- 172.21.15.23:0/1755766805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe42c1a4840 con 0x7fe42c106660 2024-03-24T10:42:54.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.348+0000 7fe429ffb640 1 -- 172.21.15.23:0/1755766805 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fe42403b040 con 0x7fe42c106660 2024-03-24T10:42:54.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.349+0000 7fe429ffb640 1 --2- 172.21.15.23:0/1755766805 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe41c0a22b0 0x7fe41c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.349+0000 7fe4359f2740 1 Processor -- start 2024-03-24T10:42:54.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.349+0000 7fe4359f2740 1 -- start start 2024-03-24T10:42:54.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.349+0000 7fe430bb1640 1 --2- 172.21.15.23:0/1755766805 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe41c0a22b0 0x7fe41c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.349+0000 7fe429ffb640 1 -- 172.21.15.23:0/1755766805 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe4240f0810 con 0x7fe42c106660 2024-03-24T10:42:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.349+0000 7fe4359f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c11d0 0x55baba1c15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.349+0000 7fe4359f2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c1af0 0x55baba1ca190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.349+0000 7fe4359f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.349+0000 7fe4359f2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55baba0e5f30 con 0x55baba1c1af0 2024-03-24T10:42:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.349+0000 7fe4359f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55baba12f810 con 0x55baba1ca6d0 2024-03-24T10:42:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.349+0000 7fe4359f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55baba12cd40 con 0x55baba1c11d0 2024-03-24T10:42:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe42bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c1af0 0x55baba1ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe430bb1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c11d0 0x55baba1c15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe42bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c1af0 0x55baba1ca190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58408/0 (socket says 172.21.15.23:58408) 2024-03-24T10:42:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe42bfff640 1 -- 172.21.15.23:0/1537712488 learned_addr learned my addr 172.21.15.23:0/1537712488 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe4313b2640 1 --2- 172.21.15.23:0/1537712488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe430bb1640 1 --2- 172.21.15.23:0/1755766805 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe41c0a22b0 0x7fe41c0a4770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe420009c20 tx=0x7fe420005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe42bfff640 1 -- 172.21.15.23:0/1537712488 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c11d0 msgr2=0x55baba1c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe42bfff640 1 --2- 172.21.15.23:0/1537712488 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c11d0 0x55baba1c15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe42bfff640 1 -- 172.21.15.23:0/1537712488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 msgr2=0x55baba1ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe42bfff640 1 --2- 172.21.15.23:0/1537712488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe42bfff640 1 -- 172.21.15.23:0/1537712488 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baba0e9ea0 con 0x55baba1c1af0 2024-03-24T10:42:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe4313b2640 1 --2- 172.21.15.23:0/1537712488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe42bfff640 1 --2- 172.21.15.23:0/1537712488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c1af0 0x55baba1ca190 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fe418009c90 tx=0x7fe418011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73cbbec6bb86b98c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe416ffd640 1 -- 172.21.15.23:0/1537712488 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418019630 con 0x55baba1c1af0 2024-03-24T10:42:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe416ffd640 1 -- 172.21.15.23:0/1537712488 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe418019c70 con 0x55baba1c1af0 2024-03-24T10:42:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.350+0000 7fe416ffd640 1 -- 172.21.15.23:0/1537712488 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41800ebf0 con 0x55baba1c1af0 2024-03-24T10:42:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.351+0000 7fe4359f2740 1 -- 172.21.15.23:0/1537712488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c1af0 msgr2=0x55baba1ca190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.351+0000 7fe4359f2740 1 --2- 172.21.15.23:0/1537712488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c1af0 0x55baba1ca190 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fe418009c90 tx=0x7fe418011810 comp rx=0 tx=0).stop 2024-03-24T10:42:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.351+0000 7fe4359f2740 1 -- 172.21.15.23:0/1537712488 shutdown_connections 2024-03-24T10:42:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.351+0000 7fe4359f2740 1 --2- 172.21.15.23:0/1537712488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.351+0000 7fe4359f2740 1 --2- 172.21.15.23:0/1537712488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c1af0 0x55baba1ca190 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.351+0000 7fe4359f2740 1 --2- 172.21.15.23:0/1537712488 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c11d0 0x55baba1c15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.351+0000 7fe4359f2740 1 -- 172.21.15.23:0/1537712488 >> 172.21.15.23:0/1537712488 conn(0x55baba1bb900 msgr2=0x55baba1bbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.351+0000 7fe4359f2740 1 -- 172.21.15.23:0/1537712488 shutdown_connections 2024-03-24T10:42:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.351+0000 7fe4359f2740 1 -- 172.21.15.23:0/1537712488 wait complete. 2024-03-24T10:42:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.351+0000 7fe4359f2740 1 -- start start 2024-03-24T10:42:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.352+0000 7fe4359f2740 1 Processor -- start 2024-03-24T10:42:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.352+0000 7fe4359f2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1c7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.352+0000 7fe4359f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.352+0000 7fe4359f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1c8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.352+0000 7fe4359f2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55baba128520 con 0x55baba1c11d0 2024-03-24T10:42:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.352+0000 7fe4359f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bab9dfb430 con 0x55baba1ca6d0 2024-03-24T10:42:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.352+0000 7fe4359f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bab9db7e00 con 0x55baba1c1af0 2024-03-24T10:42:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.352+0000 7fe4359f2740 10 client.0 _ll_register_callbacks cb 0x55baba0c6d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.352+0000 7fe42bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.352+0000 7fe42bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1c7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39424/0 (socket says 172.21.15.23:39424) 2024-03-24T10:42:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.352+0000 7fe42bfff640 1 -- 172.21.15.23:0/3666210048 learned_addr learned my addr 172.21.15.23:0/3666210048 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.352+0000 7fe430bb1640 1 --2- 172.21.15.23:0/3666210048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.352+0000 7fe4313b2640 1 --2- 172.21.15.23:0/3666210048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.353+0000 7fe42bfff640 1 -- 172.21.15.23:0/3666210048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 msgr2=0x55baba1c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.353+0000 7fe42bfff640 1 --2- 172.21.15.23:0/3666210048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.353+0000 7fe42bfff640 1 -- 172.21.15.23:0/3666210048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 msgr2=0x55baba1c7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.353+0000 7fe42bfff640 1 --2- 172.21.15.23:0/3666210048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1c7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.353+0000 7fe42bfff640 1 -- 172.21.15.23:0/3666210048 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4180098f0 con 0x55baba1c1af0 2024-03-24T10:42:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.353+0000 7fe42bfff640 1 --2- 172.21.15.23:0/3666210048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1c7c50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe41800ef00 tx=0x7fe41800ef30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7b7f922bfaebd5a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.353+0000 7fe40bfff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:42:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.353+0000 7fe40bfff640 1 -- 172.21.15.23:0/3666210048 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418005450 con 0x55baba1c1af0 2024-03-24T10:42:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.353+0000 7fe4359f2740 1 -- 172.21.15.23:0/3666210048 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bab9e277e0 con 0x55baba1c1af0 2024-03-24T10:42:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.353+0000 7fe40bfff640 1 -- 172.21.15.23:0/3666210048 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe418019de0 con 0x55baba1c1af0 2024-03-24T10:42:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.353+0000 7fe40bfff640 1 -- 172.21.15.23:0/3666210048 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe4180055b0 con 0x55baba1c1af0 2024-03-24T10:42:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.353+0000 7fe4359f2740 10 client.26164 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.353+0000 7fe4359f2740 1 -- 172.21.15.23:0/3666210048 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55baba126ba0 con 0x55baba1c1af0 2024-03-24T10:42:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.353+0000 7fe40bfff640 1 -- 172.21.15.23:0/3666210048 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bab9e277e0 con 0x55baba1c1af0 2024-03-24T10:42:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.354+0000 7fe40bfff640 1 -- 172.21.15.23:0/3666210048 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe418019830 con 0x55baba1c1af0 2024-03-24T10:42:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.354+0000 7fe4359f2740 10 client.26164 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.354+0000 7fe4359f2740 10 client.26164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.354+0000 7fe4359f2740 20 client.26164 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.354+0000 7fe4359f2740 1 --2- 172.21.15.23:0/3666210048 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55baba1deef0 0x55baba1e1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.354+0000 7fe4359f2740 4 client.26164 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.354+0000 7fe4359f2740 1 -- 172.21.15.23:0/3666210048 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bab9e64720 con 0x55baba1deef0 2024-03-24T10:42:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.354+0000 7fe430bb1640 1 --2- 172.21.15.23:0/3666210048 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55baba1deef0 0x55baba1e1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.354+0000 7fe40bfff640 10 client.26164 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:54.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.354+0000 7fe430bb1640 1 --2- 172.21.15.23:0/3666210048 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55baba1deef0 0x55baba1e1310 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b33209e2678f748 server_cookie=b4c9252230282a1b in_seq=0 out_seq=0 2024-03-24T10:42:54.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe40bfff640 1 -- 172.21.15.23:0/3666210048 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bab9e64720 con 0x55baba1deef0 2024-03-24T10:42:54.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe40bfff640 10 client.26164 handle_command_reply: tid=0 2024-03-24T10:42:54.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe4359f2740 1 client.26164 shutdown 2024-03-24T10:42:54.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe4359f2740 1 -- 172.21.15.23:0/3666210048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 msgr2=0x55baba1c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe4359f2740 1 --2- 172.21.15.23:0/3666210048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1c7c50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe41800ef00 tx=0x7fe41800ef30 comp rx=0 tx=0).stop 2024-03-24T10:42:54.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe4359f2740 1 -- 172.21.15.23:0/3666210048 shutdown_connections 2024-03-24T10:42:54.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe4359f2740 1 --2- 172.21.15.23:0/3666210048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe4359f2740 1 --2- 172.21.15.23:0/3666210048 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55baba1deef0 0x55baba1e1310 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe4359f2740 1 --2- 172.21.15.23:0/3666210048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1c7c50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe4359f2740 1 --2- 172.21.15.23:0/3666210048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1c7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe40bfff640 0 client.26164 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:54.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe4359f2740 1 -- 172.21.15.23:0/3666210048 >> 172.21.15.23:0/3666210048 conn(0x55baba1bb900 msgr2=0x55baba1bd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:54.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe4359f2740 1 -- 172.21.15.23:0/3666210048 shutdown_connections 2024-03-24T10:42:54.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe4359f2740 1 -- 172.21.15.23:0/3666210048 wait complete. 2024-03-24T10:42:54.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.355+0000 7fe4359f2740 20 client.26164 trim_cache size 0 max 16384 2024-03-24T10:42:54.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.375+0000 7fe4359f2740 1 Processor -- start 2024-03-24T10:42:54.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.375+0000 7fe4359f2740 1 -- start start 2024-03-24T10:42:54.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe4359f2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe4359f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1d5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe4359f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1ba9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe4359f2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55baba0ad430 con 0x55baba1c11d0 2024-03-24T10:42:54.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe4359f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55baba1d7600 con 0x55baba1ca6d0 2024-03-24T10:42:54.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe4359f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55baba1d64a0 con 0x55baba1c1af0 2024-03-24T10:42:54.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe430bb1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe42bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1d5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe42bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1d5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39436/0 (socket says 172.21.15.23:39436) 2024-03-24T10:42:54.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe430bb1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1c9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58422/0 (socket says 172.21.15.23:58422) 2024-03-24T10:42:54.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe42bfff640 1 -- 172.21.15.23:0/3282898874 learned_addr learned my addr 172.21.15.23:0/3282898874 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:54.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe4313b2640 1 --2- 172.21.15.23:0/3282898874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1ba9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe42bfff640 1 -- 172.21.15.23:0/3282898874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 msgr2=0x55baba1ba9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe42bfff640 1 --2- 172.21.15.23:0/3282898874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1ba9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe42bfff640 1 -- 172.21.15.23:0/3282898874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 msgr2=0x55baba1c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe42bfff640 1 --2- 172.21.15.23:0/3282898874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe42bfff640 1 -- 172.21.15.23:0/3282898874 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baba0c2b20 con 0x55baba1c1af0 2024-03-24T10:42:54.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe42bfff640 1 --2- 172.21.15.23:0/3282898874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1d5f60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe418000c00 tx=0x7fe418012f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d17a72c23628845 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:54.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe4167fc640 1 -- 172.21.15.23:0/3282898874 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41800e3b0 con 0x55baba1c1af0 2024-03-24T10:42:54.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe4167fc640 1 -- 172.21.15.23:0/3282898874 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe418004800 con 0x55baba1c1af0 2024-03-24T10:42:54.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.376+0000 7fe4167fc640 1 -- 172.21.15.23:0/3282898874 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418019690 con 0x55baba1c1af0 2024-03-24T10:42:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.377+0000 7fe4359f2740 1 -- 172.21.15.23:0/3282898874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 msgr2=0x55baba1d5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.377+0000 7fe4359f2740 1 --2- 172.21.15.23:0/3282898874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1d5f60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe418000c00 tx=0x7fe418012f90 comp rx=0 tx=0).stop 2024-03-24T10:42:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.377+0000 7fe4359f2740 1 -- 172.21.15.23:0/3282898874 shutdown_connections 2024-03-24T10:42:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.377+0000 7fe4359f2740 1 --2- 172.21.15.23:0/3282898874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1ba9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.377+0000 7fe4359f2740 1 --2- 172.21.15.23:0/3282898874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1d5f60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.377+0000 7fe4359f2740 1 --2- 172.21.15.23:0/3282898874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.377+0000 7fe4359f2740 1 -- 172.21.15.23:0/3282898874 >> 172.21.15.23:0/3282898874 conn(0x55baba1bb900 msgr2=0x55baba1ba2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.377+0000 7fe4359f2740 1 -- 172.21.15.23:0/3282898874 shutdown_connections 2024-03-24T10:42:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.377+0000 7fe4359f2740 1 -- 172.21.15.23:0/3282898874 wait complete. 2024-03-24T10:42:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.377+0000 7fe4359f2740 1 -- start start 2024-03-24T10:42:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.377+0000 7fe4359f2740 1 Processor -- start 2024-03-24T10:42:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe4359f2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1d2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe4359f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1d29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe4359f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe4359f2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55baba1ce1e0 con 0x55baba1c11d0 2024-03-24T10:42:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe4359f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55baba1ce350 con 0x55baba1ca6d0 2024-03-24T10:42:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe4359f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55baba1ce510 con 0x55baba1c1af0 2024-03-24T10:42:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe4359f2740 10 client.0 _ll_register_callbacks cb 0x7fe41801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe430bb1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1d2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe430bb1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1d2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58432/0 (socket says 172.21.15.23:58432) 2024-03-24T10:42:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe42bfff640 1 --2- 172.21.15.23:0/189841770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1d29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe430bb1640 1 -- 172.21.15.23:0/189841770 learned_addr learned my addr 172.21.15.23:0/189841770 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe4313b2640 1 --2- 172.21.15.23:0/189841770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe430bb1640 1 -- 172.21.15.23:0/189841770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 msgr2=0x55baba1d29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe430bb1640 1 --2- 172.21.15.23:0/189841770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1d29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe430bb1640 1 -- 172.21.15.23:0/189841770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 msgr2=0x55baba1cdb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe430bb1640 1 --2- 172.21.15.23:0/189841770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1cdb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe430bb1640 1 -- 172.21.15.23:0/189841770 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baba126ba0 con 0x55baba1c11d0 2024-03-24T10:42:54.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe42bfff640 1 --2- 172.21.15.23:0/189841770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1d29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:54.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe430bb1640 1 --2- 172.21.15.23:0/189841770 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1d2470 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x55baba1e1ba0 tx=0x7fe420013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4f92e2e52e24a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:54.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.378+0000 7fe40bfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:54.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe40bfff640 1 -- 172.21.15.23:0/189841770 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420022030 con 0x55baba1c11d0 2024-03-24T10:42:54.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe40bfff640 1 -- 172.21.15.23:0/189841770 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe4200054d0 con 0x55baba1c11d0 2024-03-24T10:42:54.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe4359f2740 1 -- 172.21.15.23:0/189841770 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bab9eb3890 con 0x55baba1c11d0 2024-03-24T10:42:54.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe40bfff640 1 -- 172.21.15.23:0/189841770 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe420005690 con 0x55baba1c11d0 2024-03-24T10:42:54.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe4359f2740 10 client.16323 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:54.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe4359f2740 1 -- 172.21.15.23:0/189841770 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55baba0e8700 con 0x55baba1c11d0 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe40bfff640 1 -- 172.21.15.23:0/189841770 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bab9eb3890 con 0x55baba1c11d0 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe40bfff640 1 -- 172.21.15.23:0/189841770 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe420003850 con 0x55baba1c11d0 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe4359f2740 10 client.16323 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe4359f2740 10 client.16323 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe4359f2740 20 client.16323 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe4359f2740 1 --2- 172.21.15.23:0/189841770 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55baba2456e0 0x55baba247b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe4359f2740 4 client.16323 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe4359f2740 1 -- 172.21.15.23:0/189841770 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bab9dd3700 con 0x55baba2456e0 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.379+0000 7fe42bfff640 1 --2- 172.21.15.23:0/189841770 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55baba2456e0 0x55baba247b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe42bfff640 1 --2- 172.21.15.23:0/189841770 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55baba2456e0 0x55baba247b00 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfc483e2fe4e1a83 server_cookie=6a9623312369f4bd in_seq=0 out_seq=0 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe40bfff640 10 client.16323 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe40bfff640 1 -- 172.21.15.23:0/189841770 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bab9dd3700 con 0x55baba2456e0 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe40bfff640 10 client.16323 handle_command_reply: tid=0 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe4359f2740 1 client.16323 shutdown 2024-03-24T10:42:54.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe4359f2740 1 -- 172.21.15.23:0/189841770 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 msgr2=0x55baba1d2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe4359f2740 1 --2- 172.21.15.23:0/189841770 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1d2470 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x55baba1e1ba0 tx=0x7fe420013040 comp rx=0 tx=0).stop 2024-03-24T10:42:54.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe4359f2740 1 -- 172.21.15.23:0/189841770 shutdown_connections 2024-03-24T10:42:54.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe4359f2740 1 --2- 172.21.15.23:0/189841770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55baba1ca6d0 0x55baba1cdb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe4359f2740 1 --2- 172.21.15.23:0/189841770 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55baba2456e0 0x55baba247b00 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe4359f2740 1 --2- 172.21.15.23:0/189841770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55baba1c1af0 0x55baba1d29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe4359f2740 1 --2- 172.21.15.23:0/189841770 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55baba1c11d0 0x55baba1d2470 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe40bfff640 0 client.16323 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:54.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe4359f2740 1 -- 172.21.15.23:0/189841770 >> 172.21.15.23:0/189841770 conn(0x55baba1bb900 msgr2=0x55baba1bab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:54.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe4359f2740 1 -- 172.21.15.23:0/189841770 shutdown_connections 2024-03-24T10:42:54.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe4359f2740 1 -- 172.21.15.23:0/189841770 wait complete. 2024-03-24T10:42:54.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.380+0000 7fe4359f2740 20 client.16323 trim_cache size 0 max 16384 2024-03-24T10:42:54.388 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:42:54.388 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:42:54.388 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:42:54.388 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:42:54.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.381+0000 7fe432e15640 1 -- 172.21.15.23:0/1755766805 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe41c0a22b0 msgr2=0x7fe41c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.381+0000 7fe432e15640 1 --2- 172.21.15.23:0/1755766805 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe41c0a22b0 0x7fe41c0a4770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe420009c20 tx=0x7fe420005f90 comp rx=0 tx=0).stop 2024-03-24T10:42:54.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.381+0000 7fe432e15640 1 -- 172.21.15.23:0/1755766805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe42c106660 msgr2=0x7fe42c1a3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.381+0000 7fe432e15640 1 --2- 172.21.15.23:0/1755766805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe42c106660 0x7fe42c1a3b20 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fe42402c700 tx=0x7fe424035aa0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.381+0000 7fe432e15640 1 -- 172.21.15.23:0/1755766805 shutdown_connections 2024-03-24T10:42:54.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.381+0000 7fe432e15640 1 --2- 172.21.15.23:0/1755766805 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe41c0a22b0 0x7fe41c0a4770 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.381+0000 7fe432e15640 1 --2- 172.21.15.23:0/1755766805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe42c106660 0x7fe42c1a3b20 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.381+0000 7fe432e15640 1 --2- 172.21.15.23:0/1755766805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe42c100f60 0x7fe42c198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.381+0000 7fe432e15640 1 --2- 172.21.15.23:0/1755766805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe42c1005a0 0x7fe42c198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.381+0000 7fe432e15640 1 -- 172.21.15.23:0/1755766805 >> 172.21.15.23:0/1755766805 conn(0x7fe42c077710 msgr2=0x7fe42c0ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:54.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.381+0000 7fe432e15640 1 -- 172.21.15.23:0/1755766805 shutdown_connections 2024-03-24T10:42:54.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.381+0000 7fe432e15640 1 -- 172.21.15.23:0/1755766805 wait complete. 2024-03-24T10:42:54.392 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:42:54.392 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:42:54.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.498+0000 7f6e8933a640 1 -- 172.21.15.23:0/2593653235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e841076d0 msgr2=0x7f6e84107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.498+0000 7f6e8933a640 1 --2- 172.21.15.23:0/2593653235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e841076d0 0x7f6e84107ab0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6e74009930 tx=0x7f6e7402bc30 comp rx=0 tx=0).stop 2024-03-24T10:42:54.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.498+0000 7f6e8933a640 1 -- 172.21.15.23:0/2593653235 shutdown_connections 2024-03-24T10:42:54.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.498+0000 7f6e8933a640 1 --2- 172.21.15.23:0/2593653235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e841005b0 0x7f6e8410e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.498+0000 7f6e8933a640 1 --2- 172.21.15.23:0/2593653235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e840ffc10 0x7f6e84100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.498+0000 7f6e8933a640 1 --2- 172.21.15.23:0/2593653235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e841076d0 0x7f6e84107ab0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.498+0000 7f6e8933a640 1 -- 172.21.15.23:0/2593653235 >> 172.21.15.23:0/2593653235 conn(0x7f6e84076ba0 msgr2=0x7f6e840fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:54.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.498+0000 7f6e8933a640 1 -- 172.21.15.23:0/2593653235 shutdown_connections 2024-03-24T10:42:54.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.498+0000 7f6e8933a640 1 -- 172.21.15.23:0/2593653235 wait complete. 2024-03-24T10:42:54.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e8933a640 1 Processor -- start 2024-03-24T10:42:54.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e8933a640 1 -- start start 2024-03-24T10:42:54.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e8933a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e840ffc10 0x7f6e84197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e8933a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e841005b0 0x7f6e84198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e8933a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e841076d0 0x7f6e8419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e8933a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8419ccb0 con 0x7f6e840ffc10 2024-03-24T10:42:54.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e8933a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8419ce20 con 0x7f6e841076d0 2024-03-24T10:42:54.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e8933a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8419cfe0 con 0x7f6e841005b0 2024-03-24T10:42:54.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e82ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e840ffc10 0x7f6e84197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e82ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e840ffc10 0x7f6e84197df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58450/0 (socket says 172.21.15.23:58450) 2024-03-24T10:42:54.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e82ffd640 1 -- 172.21.15.23:0/2914874925 learned_addr learned my addr 172.21.15.23:0/2914874925 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:54.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e837fe640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e841076d0 0x7f6e8419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e827fc640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e841005b0 0x7f6e84198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e82ffd640 1 -- 172.21.15.23:0/2914874925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e841005b0 msgr2=0x7f6e84198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e82ffd640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e841005b0 0x7f6e84198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e82ffd640 1 -- 172.21.15.23:0/2914874925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e841076d0 msgr2=0x7f6e8419c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e82ffd640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e841076d0 0x7f6e8419c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e82ffd640 1 -- 172.21.15.23:0/2914874925 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e74009560 con 0x7f6e840ffc10 2024-03-24T10:42:54.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.499+0000 7f6e827fc640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e841005b0 0x7f6e84198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:42:54.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.500+0000 7f6e82ffd640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e840ffc10 0x7f6e84197df0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6e74009230 tx=0x7f6e7402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:54.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.500+0000 7f6e6bfff640 1 -- 172.21.15.23:0/2914874925 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74004350 con 0x7f6e840ffc10 2024-03-24T10:42:54.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.500+0000 7f6e8933a640 1 -- 172.21.15.23:0/2914874925 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e8419d260 con 0x7f6e840ffc10 2024-03-24T10:42:54.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.500+0000 7f6e6bfff640 1 -- 172.21.15.23:0/2914874925 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e740044b0 con 0x7f6e840ffc10 2024-03-24T10:42:54.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.500+0000 7f6e6bfff640 1 -- 172.21.15.23:0/2914874925 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74003d00 con 0x7f6e840ffc10 2024-03-24T10:42:54.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.500+0000 7f6e8933a640 1 -- 172.21.15.23:0/2914874925 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e841a4b40 con 0x7f6e840ffc10 2024-03-24T10:42:54.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.501+0000 7f6e8933a640 1 -- 172.21.15.23:0/2914874925 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e84102f30 con 0x7f6e840ffc10 2024-03-24T10:42:54.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.502+0000 7f6e6bfff640 1 -- 172.21.15.23:0/2914874925 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f6e74004df0 con 0x7f6e840ffc10 2024-03-24T10:42:54.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.502+0000 7f6e6bfff640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e6c0a2230 0x7f6e6c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.502+0000 7f6e827fc640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e6c0a2230 0x7f6e6c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.502+0000 7f6e6bfff640 1 -- 172.21.15.23:0/2914874925 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f6e740f0000 con 0x7f6e840ffc10 2024-03-24T10:42:54.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.503+0000 7f6e827fc640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e6c0a2230 0x7f6e6c0a46f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6e840fda10 tx=0x7f6e78005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:54.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.507+0000 7f6e6bfff640 1 -- 172.21.15.23:0/2914874925 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6e740b8da0 con 0x7f6e840ffc10 2024-03-24T10:42:54.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:54 smithi023 ceph-mon[32466]: pgmap v87: 97 pgs: 97 active+clean; 96 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 212 op/s 2024-03-24T10:42:54.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.738+0000 7f6e8933a640 1 -- 172.21.15.23:0/2914874925 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e84109c50 con 0x7f6e840ffc10 2024-03-24T10:42:54.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.739+0000 7f6e6bfff640 1 -- 172.21.15.23:0/2914874925 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11869 (secure 0 0 0) 0x7f6e74003e60 con 0x7f6e840ffc10 2024-03-24T10:42:54.740 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:42:54.740 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:41:49.922577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:42:54.740 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 25 2024-03-24T10:42:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.741+0000 7f6e8933a640 1 -- 172.21.15.23:0/2914874925 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e6c0a2230 msgr2=0x7f6e6c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.741+0000 7f6e8933a640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e6c0a2230 0x7f6e6c0a46f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6e840fda10 tx=0x7f6e78005f90 comp rx=0 tx=0).stop 2024-03-24T10:42:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.741+0000 7f6e8933a640 1 -- 172.21.15.23:0/2914874925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e840ffc10 msgr2=0x7f6e84197df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.741+0000 7f6e8933a640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e840ffc10 0x7f6e84197df0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6e74009230 tx=0x7f6e7402c650 comp rx=0 tx=0).stop 2024-03-24T10:42:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.741+0000 7f6e8933a640 1 -- 172.21.15.23:0/2914874925 shutdown_connections 2024-03-24T10:42:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.741+0000 7f6e8933a640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e6c0a2230 0x7f6e6c0a46f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.741+0000 7f6e8933a640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e841076d0 0x7f6e8419c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.741+0000 7f6e8933a640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e841005b0 0x7f6e84198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.741+0000 7f6e8933a640 1 --2- 172.21.15.23:0/2914874925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e840ffc10 0x7f6e84197df0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.741+0000 7f6e8933a640 1 -- 172.21.15.23:0/2914874925 >> 172.21.15.23:0/2914874925 conn(0x7f6e84076ba0 msgr2=0x7f6e840fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:54.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.741+0000 7f6e8933a640 1 -- 172.21.15.23:0/2914874925 shutdown_connections 2024-03-24T10:42:54.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.741+0000 7f6e8933a640 1 -- 172.21.15.23:0/2914874925 wait complete. 2024-03-24T10:42:54.753 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:42:54.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:54 smithi117 ceph-mon[36368]: pgmap v87: 97 pgs: 97 active+clean; 96 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 212 op/s 2024-03-24T10:42:54.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:54 smithi116 ceph-mon[36248]: pgmap v87: 97 pgs: 97 active+clean; 96 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 212 op/s 2024-03-24T10:42:54.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.861+0000 7fb723455640 1 -- 172.21.15.23:0/2161785712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb71c100f80 msgr2=0x7fb71c10d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.861+0000 7fb723455640 1 --2- 172.21.15.23:0/2161785712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb71c100f80 0x7fb71c10d490 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fb714009cd0 tx=0x7fb71402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.861+0000 7fb723455640 1 -- 172.21.15.23:0/2161785712 shutdown_connections 2024-03-24T10:42:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.861+0000 7fb723455640 1 --2- 172.21.15.23:0/2161785712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb71c100f80 0x7fb71c10d490 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.861+0000 7fb723455640 1 --2- 172.21.15.23:0/2161785712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb71c1005c0 0x7fb71c100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.861+0000 7fb723455640 1 --2- 172.21.15.23:0/2161785712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb71c106660 0x7fb71c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.861+0000 7fb723455640 1 -- 172.21.15.23:0/2161785712 >> 172.21.15.23:0/2161785712 conn(0x7fb71c076990 msgr2=0x7fb71c076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.861+0000 7fb723455640 1 -- 172.21.15.23:0/2161785712 shutdown_connections 2024-03-24T10:42:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.861+0000 7fb723455640 1 -- 172.21.15.23:0/2161785712 wait complete. 2024-03-24T10:42:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb723455640 1 Processor -- start 2024-03-24T10:42:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb723455640 1 -- start start 2024-03-24T10:42:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb723455640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb71c1005c0 0x7fb71c19d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb723455640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb71c100f80 0x7fb71c19de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb723455640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb71c106660 0x7fb71c197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb723455640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb71c198160 con 0x7fb71c106660 2024-03-24T10:42:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb723455640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb71c1982d0 con 0x7fb71c1005c0 2024-03-24T10:42:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb723455640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb71c198490 con 0x7fb71c100f80 2024-03-24T10:42:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb7219f2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb71c106660 0x7fb71c197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb7209f0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb71c100f80 0x7fb71c19de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb7211f1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb71c1005c0 0x7fb71c19d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb7209f0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb71c100f80 0x7fb71c19de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39474/0 (socket says 172.21.15.23:39474) 2024-03-24T10:42:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb7219f2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb71c106660 0x7fb71c197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58462/0 (socket says 172.21.15.23:58462) 2024-03-24T10:42:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb7209f0640 1 -- 172.21.15.23:0/1930045305 learned_addr learned my addr 172.21.15.23:0/1930045305 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.862+0000 7fb7219f2640 1 -- 172.21.15.23:0/1930045305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb71c100f80 msgr2=0x7fb71c19de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.863+0000 7fb7219f2640 1 --2- 172.21.15.23:0/1930045305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb71c100f80 0x7fb71c19de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.863+0000 7fb7219f2640 1 -- 172.21.15.23:0/1930045305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb71c1005c0 msgr2=0x7fb71c19d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.863+0000 7fb7219f2640 1 --2- 172.21.15.23:0/1930045305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb71c1005c0 0x7fb71c19d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.863+0000 7fb7219f2640 1 -- 172.21.15.23:0/1930045305 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb714009900 con 0x7fb71c106660 2024-03-24T10:42:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.863+0000 7fb7219f2640 1 --2- 172.21.15.23:0/1930045305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb71c106660 0x7fb71c197b10 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb71402c630 tx=0x7fb714034b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:54.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.863+0000 7fb7127fc640 1 -- 172.21.15.23:0/1930045305 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7140328b0 con 0x7fb71c106660 2024-03-24T10:42:54.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.863+0000 7fb723455640 1 -- 172.21.15.23:0/1930045305 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb71c198710 con 0x7fb71c106660 2024-03-24T10:42:54.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.863+0000 7fb7127fc640 1 -- 172.21.15.23:0/1930045305 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb714003e50 con 0x7fb71c106660 2024-03-24T10:42:54.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.863+0000 7fb7127fc640 1 -- 172.21.15.23:0/1930045305 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb714040450 con 0x7fb71c106660 2024-03-24T10:42:54.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.863+0000 7fb723455640 1 -- 172.21.15.23:0/1930045305 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb71c1a49b0 con 0x7fb71c106660 2024-03-24T10:42:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.865+0000 7fb7127fc640 1 -- 172.21.15.23:0/1930045305 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fb71400a040 con 0x7fb71c106660 2024-03-24T10:42:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb7127fc640 1 --2- 172.21.15.23:0/1930045305 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7080a2230 0x7fb7080a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb726036740 1 Processor -- start 2024-03-24T10:42:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb7211f1640 1 --2- 172.21.15.23:0/1930045305 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7080a2230 0x7fb7080a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb7127fc640 1 -- 172.21.15.23:0/1930045305 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fb7140f0c20 con 0x7fb71c106660 2024-03-24T10:42:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb726036740 1 -- start start 2024-03-24T10:42:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb726036740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f183f0 0x5629c7f187d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb726036740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 0x5629c7f213b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb726036740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f218f0 0x5629c7f23ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb726036740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5629c7e86840 con 0x5629c7f183f0 2024-03-24T10:42:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb726036740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5629c7e83d70 con 0x5629c7f18d10 2024-03-24T10:42:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb726036740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5629c7e7f550 con 0x5629c7f218f0 2024-03-24T10:42:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb7211f1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f183f0 0x5629c7f187d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb7211f1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f183f0 0x5629c7f187d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58472/0 (socket says 172.21.15.23:58472) 2024-03-24T10:42:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb7211f1640 1 -- 172.21.15.23:0/1444761253 learned_addr learned my addr 172.21.15.23:0/1444761253 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.866+0000 7fb7219f2640 1 --2- 172.21.15.23:0/1444761253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f218f0 0x5629c7f23ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.867+0000 7fb7211f1640 1 --2- 172.21.15.23:0/1930045305 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7080a2230 0x7fb7080a46f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb71c1056e0 tx=0x7fb718005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.867+0000 7fb7209f0640 1 --2- 172.21.15.23:0/1444761253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 0x5629c7f213b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.867+0000 7fb7211f1640 1 -- 172.21.15.23:0/1444761253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f218f0 msgr2=0x5629c7f23ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.867+0000 7fb7211f1640 1 --2- 172.21.15.23:0/1444761253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f218f0 0x5629c7f23ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.867+0000 7fb7211f1640 1 -- 172.21.15.23:0/1444761253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 msgr2=0x5629c7f213b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.867+0000 7fb7211f1640 1 --2- 172.21.15.23:0/1444761253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 0x5629c7f213b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.867+0000 7fb7211f1640 1 -- 172.21.15.23:0/1444761253 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629c7e14de0 con 0x5629c7f183f0 2024-03-24T10:42:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.867+0000 7fb7219f2640 1 --2- 172.21.15.23:0/1444761253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f218f0 0x5629c7f23ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.867+0000 7fb7211f1640 1 --2- 172.21.15.23:0/1444761253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f183f0 0x5629c7f187d0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb718011e00 tx=0x7fb718014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88eaef484c5d5acc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.867+0000 7fb7077fe640 1 -- 172.21.15.23:0/1444761253 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718004230 con 0x5629c7f183f0 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.867+0000 7fb7077fe640 1 -- 172.21.15.23:0/1444761253 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb718021070 con 0x5629c7f183f0 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.867+0000 7fb7077fe640 1 -- 172.21.15.23:0/1444761253 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718003e20 con 0x5629c7f183f0 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.868+0000 7fb726036740 1 -- 172.21.15.23:0/1444761253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f183f0 msgr2=0x5629c7f187d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.868+0000 7fb726036740 1 --2- 172.21.15.23:0/1444761253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f183f0 0x5629c7f187d0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb718011e00 tx=0x7fb718014000 comp rx=0 tx=0).stop 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.868+0000 7fb726036740 1 -- 172.21.15.23:0/1444761253 shutdown_connections 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.868+0000 7fb726036740 1 --2- 172.21.15.23:0/1444761253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f218f0 0x5629c7f23ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.868+0000 7fb726036740 1 --2- 172.21.15.23:0/1444761253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 0x5629c7f213b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.868+0000 7fb726036740 1 --2- 172.21.15.23:0/1444761253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f183f0 0x5629c7f187d0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.868+0000 7fb726036740 1 -- 172.21.15.23:0/1444761253 >> 172.21.15.23:0/1444761253 conn(0x5629c7f12b20 msgr2=0x5629c7f12f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.868+0000 7fb726036740 1 -- 172.21.15.23:0/1444761253 shutdown_connections 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.868+0000 7fb726036740 1 -- 172.21.15.23:0/1444761253 wait complete. 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.868+0000 7fb726036740 1 -- start start 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb726036740 1 Processor -- start 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb726036740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f183f0 0x5629c7f1e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb726036740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 0x5629c7f1ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb726036740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f218f0 0x5629c7f1f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb726036740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5629c7b52430 con 0x5629c7f218f0 2024-03-24T10:42:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb726036740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5629c7b0ee00 con 0x5629c7f18d10 2024-03-24T10:42:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb726036740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5629c7e04590 con 0x5629c7f183f0 2024-03-24T10:42:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb726036740 10 client.0 _ll_register_callbacks cb 0x5629c7e73fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb7219f2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f218f0 0x5629c7f1f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb7219f2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f218f0 0x5629c7f1f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58484/0 (socket says 172.21.15.23:58484) 2024-03-24T10:42:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb7211f1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f183f0 0x5629c7f1e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb7219f2640 1 -- 172.21.15.23:0/1147471348 learned_addr learned my addr 172.21.15.23:0/1147471348 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb7209f0640 1 --2- 172.21.15.23:0/1147471348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 0x5629c7f1ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb7219f2640 1 -- 172.21.15.23:0/1147471348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f183f0 msgr2=0x5629c7f1e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb7219f2640 1 --2- 172.21.15.23:0/1147471348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f183f0 0x5629c7f1e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb7219f2640 1 -- 172.21.15.23:0/1147471348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 msgr2=0x5629c7f1ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb7219f2640 1 --2- 172.21.15.23:0/1147471348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 0x5629c7f1ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.869+0000 7fb7219f2640 1 -- 172.21.15.23:0/1147471348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb718017070 con 0x5629c7f218f0 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb7211f1640 1 --2- 172.21.15.23:0/1147471348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f183f0 0x5629c7f1e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb7209f0640 1 --2- 172.21.15.23:0/1147471348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 0x5629c7f1ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb7219f2640 1 --2- 172.21.15.23:0/1147471348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f218f0 0x5629c7f1f460 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb714005fa0 tx=0x7fb7140bd140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c422c926ff0073f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb704ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb704ff9640 1 -- 172.21.15.23:0/1147471348 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7140c1870 con 0x5629c7f218f0 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb704ff9640 1 -- 172.21.15.23:0/1147471348 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fb7140c5ec0 con 0x5629c7f218f0 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb726036740 1 -- 172.21.15.23:0/1147471348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5629c7b7e7e0 con 0x5629c7f218f0 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb704ff9640 1 -- 172.21.15.23:0/1147471348 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb7140b9d90 con 0x5629c7f218f0 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb726036740 10 client.16341 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb726036740 1 -- 172.21.15.23:0/1147471348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5629c7e7dbd0 con 0x5629c7f218f0 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb704ff9640 1 -- 172.21.15.23:0/1147471348 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7140da070 con 0x5629c7f218f0 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb704ff9640 1 -- 172.21.15.23:0/1147471348 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb7140c5a30 con 0x5629c7f218f0 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb726036740 10 client.16341 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb726036740 10 client.16341 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb726036740 20 client.16341 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb726036740 1 --2- 172.21.15.23:0/1147471348 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5629c7f318c0 0x5629c7f33ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb726036740 4 client.16341 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.870+0000 7fb726036740 1 -- 172.21.15.23:0/1147471348 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629c7bbb720 con 0x5629c7f318c0 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.871+0000 7fb7211f1640 1 --2- 172.21.15.23:0/1147471348 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5629c7f318c0 0x5629c7f33ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.871+0000 7fb7211f1640 1 --2- 172.21.15.23:0/1147471348 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5629c7f318c0 0x5629c7f33ce0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8310f759917c9e5b server_cookie=131228adce112183 in_seq=0 out_seq=0 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.871+0000 7fb704ff9640 10 client.16341 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb704ff9640 1 -- 172.21.15.23:0/1147471348 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5629c7bbb720 con 0x5629c7f318c0 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb704ff9640 10 client.16341 handle_command_reply: tid=0 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb726036740 1 client.16341 shutdown 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb726036740 1 -- 172.21.15.23:0/1147471348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f218f0 msgr2=0x5629c7f1f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb726036740 1 --2- 172.21.15.23:0/1147471348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f218f0 0x5629c7f1f460 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb714005fa0 tx=0x7fb7140bd140 comp rx=0 tx=0).stop 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb726036740 1 -- 172.21.15.23:0/1147471348 shutdown_connections 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb726036740 1 --2- 172.21.15.23:0/1147471348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f218f0 0x5629c7f1f460 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb726036740 1 --2- 172.21.15.23:0/1147471348 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5629c7f318c0 0x5629c7f33ce0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb726036740 1 --2- 172.21.15.23:0/1147471348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 0x5629c7f1ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb726036740 1 --2- 172.21.15.23:0/1147471348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f183f0 0x5629c7f1e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb704ff9640 0 client.16341 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:42:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb726036740 1 -- 172.21.15.23:0/1147471348 >> 172.21.15.23:0/1147471348 conn(0x5629c7f12b20 msgr2=0x5629c7f14870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb726036740 1 -- 172.21.15.23:0/1147471348 shutdown_connections 2024-03-24T10:42:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb726036740 1 -- 172.21.15.23:0/1147471348 wait complete. 2024-03-24T10:42:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.873+0000 7fb726036740 20 client.16341 trim_cache size 0 max 16384 2024-03-24T10:42:54.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb726036740 1 Processor -- start 2024-03-24T10:42:54.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb726036740 1 -- start start 2024-03-24T10:42:54.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb726036740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f183f0 0x5629c7f1db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb726036740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 0x5629c7f2d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb726036740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f218f0 0x5629c7f1f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb726036740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5629c7f13b50 con 0x5629c7f183f0 2024-03-24T10:42:54.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb726036740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5629c7f1e080 con 0x5629c7f18d10 2024-03-24T10:42:54.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb726036740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5629c7f24050 con 0x5629c7f218f0 2024-03-24T10:42:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb7211f1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f183f0 0x5629c7f1db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb7211f1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f183f0 0x5629c7f1db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58496/0 (socket says 172.21.15.23:58496) 2024-03-24T10:42:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb7209f0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 0x5629c7f2d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb7211f1640 1 -- 172.21.15.23:0/2322454537 learned_addr learned my addr 172.21.15.23:0/2322454537 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb7219f2640 1 --2- 172.21.15.23:0/2322454537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f218f0 0x5629c7f1f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb7211f1640 1 -- 172.21.15.23:0/2322454537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f218f0 msgr2=0x5629c7f1f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb7211f1640 1 --2- 172.21.15.23:0/2322454537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f218f0 0x5629c7f1f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb7211f1640 1 -- 172.21.15.23:0/2322454537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 msgr2=0x5629c7f2d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.887+0000 7fb7211f1640 1 --2- 172.21.15.23:0/2322454537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 0x5629c7f2d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.888+0000 7fb7211f1640 1 -- 172.21.15.23:0/2322454537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629c7e19c80 con 0x5629c7f183f0 2024-03-24T10:42:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.888+0000 7fb7211f1640 1 --2- 172.21.15.23:0/2322454537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f183f0 0x5629c7f1db40 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb718011e00 tx=0x7fb718029040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc72d945de6dba8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.888+0000 7fb706ffd640 1 -- 172.21.15.23:0/2322454537 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb71802c040 con 0x5629c7f183f0 2024-03-24T10:42:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.888+0000 7fb706ffd640 1 -- 172.21.15.23:0/2322454537 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb718003770 con 0x5629c7f183f0 2024-03-24T10:42:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.888+0000 7fb706ffd640 1 -- 172.21.15.23:0/2322454537 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718004680 con 0x5629c7f183f0 2024-03-24T10:42:54.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.889+0000 7fb726036740 1 -- 172.21.15.23:0/2322454537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f183f0 msgr2=0x5629c7f1db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.889+0000 7fb726036740 1 --2- 172.21.15.23:0/2322454537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f183f0 0x5629c7f1db40 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb718011e00 tx=0x7fb718029040 comp rx=0 tx=0).stop 2024-03-24T10:42:54.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.889+0000 7fb726036740 1 -- 172.21.15.23:0/2322454537 shutdown_connections 2024-03-24T10:42:54.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.889+0000 7fb726036740 1 --2- 172.21.15.23:0/2322454537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f218f0 0x5629c7f1f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.889+0000 7fb726036740 1 --2- 172.21.15.23:0/2322454537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f18d10 0x5629c7f2d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.889+0000 7fb726036740 1 --2- 172.21.15.23:0/2322454537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f183f0 0x5629c7f1db40 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.889+0000 7fb726036740 1 -- 172.21.15.23:0/2322454537 >> 172.21.15.23:0/2322454537 conn(0x5629c7f12b20 msgr2=0x5629c7f13f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:54.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.889+0000 7fb726036740 1 -- 172.21.15.23:0/2322454537 shutdown_connections 2024-03-24T10:42:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.889+0000 7fb726036740 1 -- 172.21.15.23:0/2322454537 wait complete. 2024-03-24T10:42:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.889+0000 7fb726036740 1 -- start start 2024-03-24T10:42:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.889+0000 7fb726036740 1 Processor -- start 2024-03-24T10:42:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb726036740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f183f0 0x5629c7f1b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb726036740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f18d10 0x5629c7f1bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb726036740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f218f0 0x5629c7f1c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb726036740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5629c7f96be0 con 0x5629c7f218f0 2024-03-24T10:42:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb726036740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5629c7f96d20 con 0x5629c7f183f0 2024-03-24T10:42:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb726036740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5629c7f96e60 con 0x5629c7f18d10 2024-03-24T10:42:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb726036740 10 client.0 _ll_register_callbacks cb 0x7fb7140b9350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb7219f2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f218f0 0x5629c7f1c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb7219f2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f218f0 0x5629c7f1c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58510/0 (socket says 172.21.15.23:58510) 2024-03-24T10:42:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb7219f2640 1 -- 172.21.15.23:0/2961641705 learned_addr learned my addr 172.21.15.23:0/2961641705 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb7211f1640 1 --2- 172.21.15.23:0/2961641705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f183f0 0x5629c7f1b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb7209f0640 1 --2- 172.21.15.23:0/2961641705 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f18d10 0x5629c7f1bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb7219f2640 1 -- 172.21.15.23:0/2961641705 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f18d10 msgr2=0x5629c7f1bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb7219f2640 1 --2- 172.21.15.23:0/2961641705 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f18d10 0x5629c7f1bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb7219f2640 1 -- 172.21.15.23:0/2961641705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f183f0 msgr2=0x5629c7f1b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb7219f2640 1 --2- 172.21.15.23:0/2961641705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f183f0 0x5629c7f1b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb7219f2640 1 -- 172.21.15.23:0/2961641705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629c7e14de0 con 0x5629c7f218f0 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb7211f1640 1 --2- 172.21.15.23:0/2961641705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f183f0 0x5629c7f1b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb7219f2640 1 --2- 172.21.15.23:0/2961641705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f218f0 0x5629c7f1c080 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb7140ba0f0 tx=0x7fb7140b9af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a09bc204216c852 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb704ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb704ff9640 1 -- 172.21.15.23:0/2961641705 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7140bd980 con 0x5629c7f218f0 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb704ff9640 1 -- 172.21.15.23:0/2961641705 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fb7140dd330 con 0x5629c7f218f0 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.890+0000 7fb726036740 1 -- 172.21.15.23:0/2961641705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb718021070 con 0x5629c7f218f0 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.891+0000 7fb704ff9640 1 -- 172.21.15.23:0/2961641705 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb7140bd260 con 0x5629c7f218f0 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.891+0000 7fb726036740 10 client.16347 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.891+0000 7fb726036740 1 -- 172.21.15.23:0/2961641705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5629c7e3f920 con 0x5629c7f218f0 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.891+0000 7fb704ff9640 1 -- 172.21.15.23:0/2961641705 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb718021070 con 0x5629c7f218f0 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.891+0000 7fb704ff9640 1 -- 172.21.15.23:0/2961641705 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb7140b9d90 con 0x5629c7f218f0 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.891+0000 7fb726036740 10 client.16347 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.891+0000 7fb726036740 10 client.16347 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.891+0000 7fb726036740 20 client.16347 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.891+0000 7fb726036740 1 --2- 172.21.15.23:0/2961641705 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5629c7f98150 0x5629c7f9a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.891+0000 7fb726036740 4 client.16347 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.891+0000 7fb726036740 1 -- 172.21.15.23:0/2961641705 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5629c7c0a890 con 0x5629c7f98150 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.891+0000 7fb7211f1640 1 --2- 172.21.15.23:0/2961641705 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5629c7f98150 0x5629c7f9a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.891+0000 7fb704ff9640 10 client.16347 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.891+0000 7fb7211f1640 1 --2- 172.21.15.23:0/2961641705 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5629c7f98150 0x5629c7f9a570 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8dbbe39f9dccc788 server_cookie=57dcf401f6bcef4d in_seq=0 out_seq=0 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb704ff9640 1 -- 172.21.15.23:0/2961641705 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5629c7c0a890 con 0x5629c7f98150 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb704ff9640 10 client.16347 handle_command_reply: tid=0 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb726036740 1 client.16347 shutdown 2024-03-24T10:42:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb726036740 1 -- 172.21.15.23:0/2961641705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f218f0 msgr2=0x5629c7f1c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb726036740 1 --2- 172.21.15.23:0/2961641705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f218f0 0x5629c7f1c080 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb7140ba0f0 tx=0x7fb7140b9af0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb726036740 1 -- 172.21.15.23:0/2961641705 shutdown_connections 2024-03-24T10:42:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb726036740 1 --2- 172.21.15.23:0/2961641705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629c7f218f0 0x5629c7f1c080 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb726036740 1 --2- 172.21.15.23:0/2961641705 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629c7f18d10 0x5629c7f1bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb726036740 1 --2- 172.21.15.23:0/2961641705 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5629c7f98150 0x5629c7f9a570 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb726036740 1 --2- 172.21.15.23:0/2961641705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629c7f183f0 0x5629c7f1b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb704ff9640 0 client.16347 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:42:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb726036740 1 -- 172.21.15.23:0/2961641705 >> 172.21.15.23:0/2961641705 conn(0x5629c7f12b20 msgr2=0x5629c7f147e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb726036740 1 -- 172.21.15.23:0/2961641705 shutdown_connections 2024-03-24T10:42:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb726036740 1 -- 172.21.15.23:0/2961641705 wait complete. 2024-03-24T10:42:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.892+0000 7fb726036740 20 client.16347 trim_cache size 0 max 16384 2024-03-24T10:42:54.899 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:42:54.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.893+0000 7fb723455640 1 -- 172.21.15.23:0/1930045305 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7080a2230 msgr2=0x7fb7080a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.893+0000 7fb723455640 1 --2- 172.21.15.23:0/1930045305 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7080a2230 0x7fb7080a46f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb71c1056e0 tx=0x7fb718005f90 comp rx=0 tx=0).stop 2024-03-24T10:42:54.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.893+0000 7fb723455640 1 -- 172.21.15.23:0/1930045305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb71c106660 msgr2=0x7fb71c197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:54.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.893+0000 7fb723455640 1 --2- 172.21.15.23:0/1930045305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb71c106660 0x7fb71c197b10 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb71402c630 tx=0x7fb714034b10 comp rx=0 tx=0).stop 2024-03-24T10:42:54.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.893+0000 7fb723455640 1 -- 172.21.15.23:0/1930045305 shutdown_connections 2024-03-24T10:42:54.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.893+0000 7fb723455640 1 --2- 172.21.15.23:0/1930045305 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7080a2230 0x7fb7080a46f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.893+0000 7fb723455640 1 --2- 172.21.15.23:0/1930045305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb71c106660 0x7fb71c197b10 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.893+0000 7fb723455640 1 --2- 172.21.15.23:0/1930045305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb71c100f80 0x7fb71c19de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.893+0000 7fb723455640 1 --2- 172.21.15.23:0/1930045305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb71c1005c0 0x7fb71c19d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:54.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.893+0000 7fb723455640 1 -- 172.21.15.23:0/1930045305 >> 172.21.15.23:0/1930045305 conn(0x7fb71c076990 msgr2=0x7fb71c0feee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:54.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.893+0000 7fb723455640 1 -- 172.21.15.23:0/1930045305 shutdown_connections 2024-03-24T10:42:54.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:54.893+0000 7fb723455640 1 -- 172.21.15.23:0/1930045305 wait complete. 2024-03-24T10:42:54.904 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:42:55.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.011+0000 7f518d242640 1 -- 172.21.15.23:0/1022877221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51881005a0 msgr2=0x7f5188100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.011+0000 7f518d242640 1 --2- 172.21.15.23:0/1022877221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51881005a0 0x7f5188100a20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f5170009900 tx=0x7f517002bc30 comp rx=0 tx=0).stop 2024-03-24T10:42:55.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.012+0000 7f518d242640 1 -- 172.21.15.23:0/1022877221 shutdown_connections 2024-03-24T10:42:55.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.012+0000 7f518d242640 1 --2- 172.21.15.23:0/1022877221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5188100f60 0x7f518810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.012+0000 7f518d242640 1 --2- 172.21.15.23:0/1022877221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51881005a0 0x7f5188100a20 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.012+0000 7f518d242640 1 --2- 172.21.15.23:0/1022877221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5188106660 0x7f5188106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.012+0000 7f518d242640 1 -- 172.21.15.23:0/1022877221 >> 172.21.15.23:0/1022877221 conn(0x7f5188077710 msgr2=0x7f5188077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.012+0000 7f518d242640 1 -- 172.21.15.23:0/1022877221 shutdown_connections 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.012+0000 7f518d242640 1 -- 172.21.15.23:0/1022877221 wait complete. 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.012+0000 7f518d242640 1 Processor -- start 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.012+0000 7f518d242640 1 -- start start 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.012+0000 7f518d242640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51881005a0 0x7f518819d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.012+0000 7f518d242640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5188100f60 0x7f518819de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f518d242640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5188106660 0x7f5188197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f518d242640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f5188198160 con 0x7f51881005a0 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f518d242640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f51881982d0 con 0x7f5188100f60 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f518d242640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5188198490 con 0x7f5188106660 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f5186d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51881005a0 0x7f518819d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f5186d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51881005a0 0x7f518819d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58524/0 (socket says 172.21.15.23:58524) 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f5186d9d640 1 -- 172.21.15.23:0/1729981428 learned_addr learned my addr 172.21.15.23:0/1729981428 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f518759e640 1 --2- 172.21.15.23:0/1729981428 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5188106660 0x7f5188197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f517ffff640 1 --2- 172.21.15.23:0/1729981428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5188100f60 0x7f518819de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f5186d9d640 1 -- 172.21.15.23:0/1729981428 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5188106660 msgr2=0x7f5188197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f5186d9d640 1 --2- 172.21.15.23:0/1729981428 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5188106660 0x7f5188197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f5186d9d640 1 -- 172.21.15.23:0/1729981428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5188100f60 msgr2=0x7f518819de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f5186d9d640 1 --2- 172.21.15.23:0/1729981428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5188100f60 0x7f518819de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f5186d9d640 1 -- 172.21.15.23:0/1729981428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5170009560 con 0x7f51881005a0 2024-03-24T10:42:55.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f5186d9d640 1 --2- 172.21.15.23:0/1729981428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51881005a0 0x7f518819d910 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f517800e9f0 tx=0x7f517800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:55.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f5184d99640 1 -- 172.21.15.23:0/1729981428 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5178009720 con 0x7f51881005a0 2024-03-24T10:42:55.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f5184d99640 1 -- 172.21.15.23:0/1729981428 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f517800cd80 con 0x7f51881005a0 2024-03-24T10:42:55.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f518d242640 1 -- 172.21.15.23:0/1729981428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5188198770 con 0x7f51881005a0 2024-03-24T10:42:55.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f5184d99640 1 -- 172.21.15.23:0/1729981428 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5178010690 con 0x7f51881005a0 2024-03-24T10:42:55.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.013+0000 7f518d242640 1 -- 172.21.15.23:0/1729981428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51881a49b0 con 0x7f51881005a0 2024-03-24T10:42:55.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.016+0000 7f5184d99640 1 -- 172.21.15.23:0/1729981428 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f51780046c0 con 0x7f51881005a0 2024-03-24T10:42:55.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.016+0000 7f518fe1a740 1 Processor -- start 2024-03-24T10:42:55.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.016+0000 7f518fe1a740 1 -- start start 2024-03-24T10:42:55.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f518fe1a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543b3f0 0x55a74543b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f518fe1a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543bd10 0x55a7454443b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f518fe1a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 0x55a745446ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f518fe1a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a7453a9840 con 0x55a74543bd10 2024-03-24T10:42:55.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f518fe1a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a7453a6d70 con 0x55a7454448f0 2024-03-24T10:42:55.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f518fe1a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a7453a2550 con 0x55a74543b3f0 2024-03-24T10:42:55.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f5184d99640 1 --2- 172.21.15.23:0/1729981428 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f51740a2230 0x7f51740a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f5186d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543bd10 0x55a7454443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f518759e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543b3f0 0x55a74543b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f517ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 0x55a745446ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f5186d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543bd10 0x55a7454443b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58534/0 (socket says 172.21.15.23:58534) 2024-03-24T10:42:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f518759e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543b3f0 0x55a74543b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39542/0 (socket says 172.21.15.23:39542) 2024-03-24T10:42:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f5186d9d640 1 -- 172.21.15.23:0/958309502 learned_addr learned my addr 172.21.15.23:0/958309502 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f518759e640 1 --2- 172.21.15.23:0/1729981428 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f51740a2230 0x7f51740a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f5184d99640 1 -- 172.21.15.23:0/1729981428 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f5178014070 con 0x7f51881005a0 2024-03-24T10:42:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f517ffff640 1 -- 172.21.15.23:0/958309502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543b3f0 msgr2=0x55a74543b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f517ffff640 1 --2- 172.21.15.23:0/958309502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543b3f0 0x55a74543b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f517ffff640 1 -- 172.21.15.23:0/958309502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543bd10 msgr2=0x55a7454443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f517ffff640 1 --2- 172.21.15.23:0/958309502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543bd10 0x55a7454443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f517ffff640 1 -- 172.21.15.23:0/958309502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a745337de0 con 0x55a7454448f0 2024-03-24T10:42:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f518759e640 1 --2- 172.21.15.23:0/1729981428 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f51740a2230 0x7f51740a46f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5180005c40 tx=0x7f5180009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.017+0000 7f517ffff640 1 --2- 172.21.15.23:0/958309502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 0x55a745446ce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5170000c00 tx=0x7f517002c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85b1c0409ac4e8d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.018+0000 7f517d7fa640 1 -- 172.21.15.23:0/958309502 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f517002c910 con 0x55a7454448f0 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.018+0000 7f517d7fa640 1 -- 172.21.15.23:0/958309502 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f517002ca70 con 0x55a7454448f0 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.018+0000 7f517d7fa640 1 -- 172.21.15.23:0/958309502 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5170005050 con 0x55a7454448f0 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.018+0000 7f518fe1a740 1 -- 172.21.15.23:0/958309502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 msgr2=0x55a745446ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.018+0000 7f518fe1a740 1 --2- 172.21.15.23:0/958309502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 0x55a745446ce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5170000c00 tx=0x7f517002c6a0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 -- 172.21.15.23:0/958309502 shutdown_connections 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 --2- 172.21.15.23:0/958309502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 0x55a745446ce0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 --2- 172.21.15.23:0/958309502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543bd10 0x55a7454443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 --2- 172.21.15.23:0/958309502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543b3f0 0x55a74543b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 -- 172.21.15.23:0/958309502 >> 172.21.15.23:0/958309502 conn(0x55a745435b20 msgr2=0x55a745435f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 -- 172.21.15.23:0/958309502 shutdown_connections 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 -- 172.21.15.23:0/958309502 wait complete. 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 -- start start 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 Processor -- start 2024-03-24T10:42:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543b3f0 0x55a745441960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543bd10 0x55a745441ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 0x55a7454423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a745075430 con 0x55a74543b3f0 2024-03-24T10:42:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a745031e00 con 0x55a7454448f0 2024-03-24T10:42:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a745327590 con 0x55a74543bd10 2024-03-24T10:42:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518fe1a740 10 client.0 _ll_register_callbacks cb 0x55a745396fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f5186d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543b3f0 0x55a745441960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f5186d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543b3f0 0x55a745441960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58538/0 (socket says 172.21.15.23:58538) 2024-03-24T10:42:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f5186d9d640 1 -- 172.21.15.23:0/166775088 learned_addr learned my addr 172.21.15.23:0/166775088 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f518759e640 1 --2- 172.21.15.23:0/166775088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 0x55a7454423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.019+0000 7f517ffff640 1 --2- 172.21.15.23:0/166775088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543bd10 0x55a745441ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.020+0000 7f5186d9d640 1 -- 172.21.15.23:0/166775088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543bd10 msgr2=0x55a745441ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.020+0000 7f5186d9d640 1 --2- 172.21.15.23:0/166775088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543bd10 0x55a745441ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.020+0000 7f5186d9d640 1 -- 172.21.15.23:0/166775088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 msgr2=0x55a7454423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.020+0000 7f5186d9d640 1 --2- 172.21.15.23:0/166775088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 0x55a7454423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.020+0000 7f5186d9d640 1 -- 172.21.15.23:0/166775088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5188068870 con 0x55a74543b3f0 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.020+0000 7f5186d9d640 1 --2- 172.21.15.23:0/166775088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543b3f0 0x55a745441960 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x55a74543aec0 tx=0x7f517809aa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f42344049e487e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.020+0000 7f516e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.020+0000 7f516e7fc640 1 -- 172.21.15.23:0/166775088 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51780ad030 con 0x55a74543b3f0 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.020+0000 7f516e7fc640 1 -- 172.21.15.23:0/166775088 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f517809e030 con 0x55a74543b3f0 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.020+0000 7f518fe1a740 1 -- 172.21.15.23:0/166775088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a7450a17e0 con 0x55a74543b3f0 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.020+0000 7f516e7fc640 1 -- 172.21.15.23:0/166775088 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5178084120 con 0x55a74543b3f0 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.020+0000 7f518fe1a740 10 client.16359 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.020+0000 7f518fe1a740 1 -- 172.21.15.23:0/166775088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a7453a0bd0 con 0x55a74543b3f0 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.020+0000 7f516e7fc640 1 -- 172.21.15.23:0/166775088 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a7450a17e0 con 0x55a74543b3f0 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.021+0000 7f516e7fc640 1 -- 172.21.15.23:0/166775088 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f517801f060 con 0x55a74543b3f0 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.021+0000 7f518fe1a740 10 client.16359 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.021+0000 7f518fe1a740 10 client.16359 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:42:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.021+0000 7f518fe1a740 20 client.16359 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.021+0000 7f518fe1a740 1 --2- 172.21.15.23:0/166775088 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55a7454549a0 0x55a745456dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.021+0000 7f518fe1a740 4 client.16359 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.021+0000 7f518fe1a740 1 -- 172.21.15.23:0/166775088 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7450de720 con 0x55a7454549a0 2024-03-24T10:42:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.021+0000 7f517ffff640 1 --2- 172.21.15.23:0/166775088 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55a7454549a0 0x55a745456dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.021+0000 7f516e7fc640 10 client.16359 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:42:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.021+0000 7f517ffff640 1 --2- 172.21.15.23:0/166775088 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55a7454549a0 0x55a745456dc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f7628048ea388f7 server_cookie=991a996bde3505cf in_seq=0 out_seq=0 2024-03-24T10:42:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f516e7fc640 1 -- 172.21.15.23:0/166775088 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a7450de720 con 0x55a7454549a0 2024-03-24T10:42:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f516e7fc640 10 client.16359 handle_command_reply: tid=0 2024-03-24T10:42:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f518fe1a740 1 client.16359 shutdown 2024-03-24T10:42:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f518fe1a740 1 -- 172.21.15.23:0/166775088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543b3f0 msgr2=0x55a745441960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f518fe1a740 1 --2- 172.21.15.23:0/166775088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543b3f0 0x55a745441960 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x55a74543aec0 tx=0x7f517809aa10 comp rx=0 tx=0).stop 2024-03-24T10:42:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f518fe1a740 1 -- 172.21.15.23:0/166775088 shutdown_connections 2024-03-24T10:42:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f518fe1a740 1 --2- 172.21.15.23:0/166775088 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55a7454549a0 0x55a745456dc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f518fe1a740 1 --2- 172.21.15.23:0/166775088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 0x55a7454423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f518fe1a740 1 --2- 172.21.15.23:0/166775088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543bd10 0x55a745441ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f516e7fc640 0 client.16359 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:42:55.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f518fe1a740 1 --2- 172.21.15.23:0/166775088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543b3f0 0x55a745441960 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f518fe1a740 1 -- 172.21.15.23:0/166775088 >> 172.21.15.23:0/166775088 conn(0x55a745435b20 msgr2=0x55a745437870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:55.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f518fe1a740 1 -- 172.21.15.23:0/166775088 shutdown_connections 2024-03-24T10:42:55.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f518fe1a740 1 -- 172.21.15.23:0/166775088 wait complete. 2024-03-24T10:42:55.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.022+0000 7f518fe1a740 20 client.16359 trim_cache size 0 max 16384 2024-03-24T10:42:55.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.039+0000 7f518fe1a740 1 Processor -- start 2024-03-24T10:42:55.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.039+0000 7f518fe1a740 1 -- start start 2024-03-24T10:42:55.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.040+0000 7f518fe1a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74543b3f0 0x55a745434cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.040+0000 7f518fe1a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543bd10 0x55a74545c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.040+0000 7f518fe1a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7454448f0 0x55a74544c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.040+0000 7f518fe1a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a745435230 con 0x55a74543bd10 2024-03-24T10:42:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.040+0000 7f518fe1a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a7454313e0 con 0x55a74543b3f0 2024-03-24T10:42:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.040+0000 7f518fe1a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a74543fed0 con 0x55a7454448f0 2024-03-24T10:42:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.040+0000 7f5186d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74543b3f0 0x55a745434cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.040+0000 7f5186d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74543b3f0 0x55a745434cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:44048/0 (socket says 172.21.15.23:44048) 2024-03-24T10:42:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.040+0000 7f5186d9d640 1 -- 172.21.15.23:0/3320636353 learned_addr learned my addr 172.21.15.23:0/3320636353 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.041+0000 7f517ffff640 1 --2- 172.21.15.23:0/3320636353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543bd10 0x55a74545c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.041+0000 7f5186d9d640 1 -- 172.21.15.23:0/3320636353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7454448f0 msgr2=0x55a74544c740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T10:42:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.041+0000 7f5186d9d640 1 --2- 172.21.15.23:0/3320636353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7454448f0 0x55a74544c740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.041+0000 7f5186d9d640 1 -- 172.21.15.23:0/3320636353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543bd10 msgr2=0x55a74545c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.041+0000 7f5186d9d640 1 --2- 172.21.15.23:0/3320636353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543bd10 0x55a74545c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.041+0000 7f5186d9d640 1 -- 172.21.15.23:0/3320636353 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a74533cc80 con 0x55a74543b3f0 2024-03-24T10:42:55.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.042+0000 7f5186d9d640 1 --2- 172.21.15.23:0/3320636353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74543b3f0 0x55a745434cc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f51780977e0 tx=0x7f517809aa80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3854c4f8a94d180f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:55.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.048+0000 7f517cff9640 1 -- 172.21.15.23:0/3320636353 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5178084120 con 0x55a74543b3f0 2024-03-24T10:42:55.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.048+0000 7f517cff9640 1 -- 172.21.15.23:0/3320636353 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f517809d020 con 0x55a74543b3f0 2024-03-24T10:42:55.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.048+0000 7f517cff9640 1 -- 172.21.15.23:0/3320636353 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51780a3120 con 0x55a74543b3f0 2024-03-24T10:42:55.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.049+0000 7f518fe1a740 1 -- 172.21.15.23:0/3320636353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74543b3f0 msgr2=0x55a745434cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.049+0000 7f518fe1a740 1 --2- 172.21.15.23:0/3320636353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74543b3f0 0x55a745434cc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f51780977e0 tx=0x7f517809aa80 comp rx=0 tx=0).stop 2024-03-24T10:42:55.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.053+0000 7f518fe1a740 1 -- 172.21.15.23:0/3320636353 shutdown_connections 2024-03-24T10:42:55.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.053+0000 7f518fe1a740 1 --2- 172.21.15.23:0/3320636353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7454448f0 0x55a74544c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.053+0000 7f518fe1a740 1 --2- 172.21.15.23:0/3320636353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543bd10 0x55a74545c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.054+0000 7f518fe1a740 1 --2- 172.21.15.23:0/3320636353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74543b3f0 0x55a745434cc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.054+0000 7f518fe1a740 1 -- 172.21.15.23:0/3320636353 >> 172.21.15.23:0/3320636353 conn(0x55a745435b20 msgr2=0x55a745436d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:55.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.055+0000 7f518fe1a740 1 -- 172.21.15.23:0/3320636353 shutdown_connections 2024-03-24T10:42:55.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.055+0000 7f518fe1a740 1 -- 172.21.15.23:0/3320636353 wait complete. 2024-03-24T10:42:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.056+0000 7f518fe1a740 1 -- start start 2024-03-24T10:42:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.056+0000 7f518fe1a740 1 Processor -- start 2024-03-24T10:42:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.056+0000 7f518fe1a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543b3f0 0x55a74544c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.057+0000 7f518fe1a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543bd10 0x55a745449720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.057+0000 7f518fe1a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 0x55a745449c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.057+0000 7f517ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543bd10 0x55a745449720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.057+0000 7f517ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543bd10 0x55a745449720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39572/0 (socket says 172.21.15.23:39572) 2024-03-24T10:42:55.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.057+0000 7f517ffff640 1 -- 172.21.15.23:0/2960700924 learned_addr learned my addr 172.21.15.23:0/2960700924 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:55.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.057+0000 7f518fe1a740 1 -- 172.21.15.23:0/2960700924 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a74544a1a0 con 0x55a74543b3f0 2024-03-24T10:42:55.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.057+0000 7f518759e640 1 --2- 172.21.15.23:0/2960700924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 0x55a745449c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.057+0000 7f518fe1a740 1 -- 172.21.15.23:0/2960700924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a74544a310 con 0x55a7454448f0 2024-03-24T10:42:55.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.057+0000 7f518fe1a740 1 -- 172.21.15.23:0/2960700924 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a74544a4d0 con 0x55a74543bd10 2024-03-24T10:42:55.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.058+0000 7f518fe1a740 10 client.0 _ll_register_callbacks cb 0x7f517800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:55.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.058+0000 7f517ffff640 1 -- 172.21.15.23:0/2960700924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 msgr2=0x55a745449c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.058+0000 7f517ffff640 1 --2- 172.21.15.23:0/2960700924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 0x55a745449c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.058+0000 7f517ffff640 1 -- 172.21.15.23:0/2960700924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543b3f0 msgr2=0x55a74544c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.058+0000 7f517ffff640 1 --2- 172.21.15.23:0/2960700924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543b3f0 0x55a74544c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.058+0000 7f517ffff640 1 -- 172.21.15.23:0/2960700924 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7453a0bd0 con 0x55a74543bd10 2024-03-24T10:42:55.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.058+0000 7f518fe1a740 1 -- 172.21.15.23:0/2960700924 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f517002ca70 con 0x55a74543bd10 2024-03-24T10:42:55.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.058+0000 7f517ffff640 1 --2- 172.21.15.23:0/2960700924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543bd10 0x55a745449720 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55a745436cb0 tx=0x7f5170004370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb7630e62b33ea34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:55.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.058+0000 7f516e7fc640 10 client.26209 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:42:55.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.059+0000 7f516e7fc640 1 -- 172.21.15.23:0/2960700924 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51700046c0 con 0x55a74543bd10 2024-03-24T10:42:55.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.059+0000 7f516e7fc640 1 -- 172.21.15.23:0/2960700924 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f517004e070 con 0x55a74543bd10 2024-03-24T10:42:55.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.059+0000 7f516e7fc640 1 -- 172.21.15.23:0/2960700924 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5170004820 con 0x55a74543bd10 2024-03-24T10:42:55.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.059+0000 7f518fe1a740 10 client.26209 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:55.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.059+0000 7f518fe1a740 1 -- 172.21.15.23:0/2960700924 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({fsmap=25}) -- 0x55a745362920 con 0x55a74543bd10 2024-03-24T10:42:55.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.060+0000 7f516e7fc640 1 -- 172.21.15.23:0/2960700924 <== mon.1 v2:172.21.15.117:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f517003c050 con 0x55a74543bd10 2024-03-24T10:42:55.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.061+0000 7f518fe1a740 10 client.26209 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:55.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.061+0000 7f518fe1a740 10 client.26209 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:42:55.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.061+0000 7f518fe1a740 20 client.26209 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:55.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.061+0000 7f518fe1a740 1 --2- 172.21.15.23:0/2960700924 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55a7454bbc50 0x55a7454be090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.061+0000 7f518fe1a740 4 client.26209 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:42:55.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.061+0000 7f518fe1a740 1 -- 172.21.15.23:0/2960700924 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f517002ca70 con 0x55a7454bbc50 2024-03-24T10:42:55.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.061+0000 7f5186d9d640 1 --2- 172.21.15.23:0/2960700924 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55a7454bbc50 0x55a7454be090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.062+0000 7f5186d9d640 1 --2- 172.21.15.23:0/2960700924 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55a7454bbc50 0x55a7454be090 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6d7958e5bd92db60 server_cookie=9119b84c9ac0cbfe in_seq=0 out_seq=0 2024-03-24T10:42:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.062+0000 7f516e7fc640 10 client.26209 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:42:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.062+0000 7f516e7fc640 1 -- 172.21.15.23:0/2960700924 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f517002ca70 con 0x55a7454bbc50 2024-03-24T10:42:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.062+0000 7f516e7fc640 10 client.26209 handle_command_reply: tid=0 2024-03-24T10:42:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.062+0000 7f518fe1a740 1 client.26209 shutdown 2024-03-24T10:42:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.063+0000 7f518fe1a740 1 -- 172.21.15.23:0/2960700924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543bd10 msgr2=0x55a745449720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.063+0000 7f518fe1a740 1 --2- 172.21.15.23:0/2960700924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543bd10 0x55a745449720 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55a745436cb0 tx=0x7f5170004370 comp rx=0 tx=0).stop 2024-03-24T10:42:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.063+0000 7f518fe1a740 1 -- 172.21.15.23:0/2960700924 shutdown_connections 2024-03-24T10:42:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.063+0000 7f518fe1a740 1 --2- 172.21.15.23:0/2960700924 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55a7454bbc50 0x55a7454be090 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.063+0000 7f518fe1a740 1 --2- 172.21.15.23:0/2960700924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7454448f0 0x55a745449c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.063+0000 7f518fe1a740 1 --2- 172.21.15.23:0/2960700924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74543bd10 0x55a745449720 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.063+0000 7f518fe1a740 1 --2- 172.21.15.23:0/2960700924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a74543b3f0 0x55a74544c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.063+0000 7f518fe1a740 1 -- 172.21.15.23:0/2960700924 >> 172.21.15.23:0/2960700924 conn(0x55a745435b20 msgr2=0x55a745437570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.063+0000 7f516e7fc640 0 client.26209 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:42:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.063+0000 7f518fe1a740 1 -- 172.21.15.23:0/2960700924 shutdown_connections 2024-03-24T10:42:55.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.063+0000 7f518fe1a740 1 -- 172.21.15.23:0/2960700924 wait complete. 2024-03-24T10:42:55.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.064+0000 7f518fe1a740 20 client.26209 trim_cache size 0 max 16384 2024-03-24T10:42:55.065 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:42:55.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.064+0000 7f518d242640 1 -- 172.21.15.23:0/1729981428 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f51740a2230 msgr2=0x7f51740a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.064+0000 7f518d242640 1 --2- 172.21.15.23:0/1729981428 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f51740a2230 0x7f51740a46f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5180005c40 tx=0x7f5180009500 comp rx=0 tx=0).stop 2024-03-24T10:42:55.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.064+0000 7f518d242640 1 -- 172.21.15.23:0/1729981428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51881005a0 msgr2=0x7f518819d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.064+0000 7f518d242640 1 --2- 172.21.15.23:0/1729981428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51881005a0 0x7f518819d910 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f517800e9f0 tx=0x7f517800edf0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.065+0000 7f518d242640 1 -- 172.21.15.23:0/1729981428 shutdown_connections 2024-03-24T10:42:55.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.065+0000 7f518d242640 1 --2- 172.21.15.23:0/1729981428 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f51740a2230 0x7f51740a46f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.065+0000 7f518d242640 1 --2- 172.21.15.23:0/1729981428 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5188106660 0x7f5188197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.065+0000 7f518d242640 1 --2- 172.21.15.23:0/1729981428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5188100f60 0x7f518819de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.065+0000 7f518d242640 1 --2- 172.21.15.23:0/1729981428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51881005a0 0x7f518819d910 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.065+0000 7f518d242640 1 -- 172.21.15.23:0/1729981428 >> 172.21.15.23:0/1729981428 conn(0x7f5188077710 msgr2=0x7f51880ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:55.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.065+0000 7f518d242640 1 -- 172.21.15.23:0/1729981428 shutdown_connections 2024-03-24T10:42:55.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.065+0000 7f518d242640 1 -- 172.21.15.23:0/1729981428 wait complete. 2024-03-24T10:42:55.076 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:42:55.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 166181 12.68 MB/sec warmup 72 sec latency 57.797 ms 2024-03-24T10:42:55.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.181+0000 7f03f67c2640 1 -- 172.21.15.23:0/2350274977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03f01055b0 msgr2=0x7f03f0105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.181+0000 7f03f67c2640 1 --2- 172.21.15.23:0/2350274977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03f01055b0 0x7f03f0105990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f03e0009930 tx=0x7f03e002bc30 comp rx=0 tx=0).stop 2024-03-24T10:42:55.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.181+0000 7f03f67c2640 1 -- 172.21.15.23:0/2350274977 shutdown_connections 2024-03-24T10:42:55.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.181+0000 7f03f67c2640 1 --2- 172.21.15.23:0/2350274977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03f0101ae0 0x7f03f010d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.181+0000 7f03f67c2640 1 --2- 172.21.15.23:0/2350274977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03f0101120 0x7f03f01015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.181+0000 7f03f67c2640 1 --2- 172.21.15.23:0/2350274977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03f01055b0 0x7f03f0105990 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.181+0000 7f03f67c2640 1 -- 172.21.15.23:0/2350274977 >> 172.21.15.23:0/2350274977 conn(0x7f03f0075870 msgr2=0x7f03f0075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:55.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.181+0000 7f03f67c2640 1 -- 172.21.15.23:0/2350274977 shutdown_connections 2024-03-24T10:42:55.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.181+0000 7f03f67c2640 1 -- 172.21.15.23:0/2350274977 wait complete. 2024-03-24T10:42:55.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.181+0000 7f03f67c2640 1 Processor -- start 2024-03-24T10:42:55.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.181+0000 7f03f67c2640 1 -- start start 2024-03-24T10:42:55.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f67c2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03f0101120 0x7f03f0198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f67c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03f0101ae0 0x7f03f0199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f67c2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03f01055b0 0x7f03f01a3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f67c2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f03f01998e0 con 0x7f03f01055b0 2024-03-24T10:42:55.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f67c2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f03f01a4100 con 0x7f03f0101ae0 2024-03-24T10:42:55.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f67c2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f03f01a4240 con 0x7f03f0101120 2024-03-24T10:42:55.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f4d5f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03f01055b0 0x7f03f01a3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f4d5f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03f01055b0 0x7f03f01a3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58590/0 (socket says 172.21.15.23:58590) 2024-03-24T10:42:55.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f4d5f640 1 -- 172.21.15.23:0/2096359239 learned_addr learned my addr 172.21.15.23:0/2096359239 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:55.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03effff640 1 --2- 172.21.15.23:0/2096359239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03f0101120 0x7f03f0198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03ef7fe640 1 --2- 172.21.15.23:0/2096359239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03f0101ae0 0x7f03f0199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f4d5f640 1 -- 172.21.15.23:0/2096359239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03f0101120 msgr2=0x7f03f0198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f4d5f640 1 --2- 172.21.15.23:0/2096359239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03f0101120 0x7f03f0198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f4d5f640 1 -- 172.21.15.23:0/2096359239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03f0101ae0 msgr2=0x7f03f0199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f4d5f640 1 --2- 172.21.15.23:0/2096359239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03f0101ae0 0x7f03f0199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f4d5f640 1 -- 172.21.15.23:0/2096359239 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e0009560 con 0x7f03f01055b0 2024-03-24T10:42:55.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f4d5f640 1 --2- 172.21.15.23:0/2096359239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03f01055b0 0x7f03f01a3bc0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009fd0 tx=0x7f03e800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:55.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03ed7fa640 1 -- 172.21.15.23:0/2096359239 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e800bea0 con 0x7f03f01055b0 2024-03-24T10:42:55.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03ed7fa640 1 -- 172.21.15.23:0/2096359239 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03e8009940 con 0x7f03f01055b0 2024-03-24T10:42:55.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f67c2640 1 -- 172.21.15.23:0/2096359239 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f01a44a0 con 0x7f03f01055b0 2024-03-24T10:42:55.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03ed7fa640 1 -- 172.21.15.23:0/2096359239 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e80106c0 con 0x7f03f01055b0 2024-03-24T10:42:55.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.182+0000 7f03f67c2640 1 -- 172.21.15.23:0/2096359239 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f01a4960 con 0x7f03f01055b0 2024-03-24T10:42:55.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.185+0000 7f03ed7fa640 1 -- 172.21.15.23:0/2096359239 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f03e8004240 con 0x7f03f01055b0 2024-03-24T10:42:55.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.185+0000 7f03f939b740 1 Processor -- start 2024-03-24T10:42:55.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.185+0000 7f03f939b740 1 -- start start 2024-03-24T10:42:55.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03f939b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d757d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03f939b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d75d10 0x5571b0d7e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03f939b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d7e8f0 0x5571b0d80ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03f939b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5571b0ce3840 con 0x5571b0d753f0 2024-03-24T10:42:55.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03f939b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5571b0ce0d70 con 0x5571b0d75d10 2024-03-24T10:42:55.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03f939b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5571b0cdc550 con 0x5571b0d7e8f0 2024-03-24T10:42:55.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03ef7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d757d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d75d10 0x5571b0d7e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03ef7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d757d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58600/0 (socket says 172.21.15.23:58600) 2024-03-24T10:42:55.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03ef7fe640 1 -- 172.21.15.23:0/3947327329 learned_addr learned my addr 172.21.15.23:0/3947327329 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:55.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03ef7fe640 1 --2- 172.21.15.23:0/3947327329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d7e8f0 0x5571b0d80ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03ed7fa640 1 --2- 172.21.15.23:0/2096359239 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f03d80a2230 0x7f03d80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03effff640 1 -- 172.21.15.23:0/3947327329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d7e8f0 msgr2=0x5571b0d80ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03effff640 1 --2- 172.21.15.23:0/3947327329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d7e8f0 0x5571b0d80ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03effff640 1 -- 172.21.15.23:0/3947327329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 msgr2=0x5571b0d757d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03effff640 1 --2- 172.21.15.23:0/3947327329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d757d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03effff640 1 -- 172.21.15.23:0/3947327329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571b0c71de0 con 0x5571b0d75d10 2024-03-24T10:42:55.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03f4d5f640 1 --2- 172.21.15.23:0/2096359239 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f03d80a2230 0x7f03d80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03ed7fa640 1 -- 172.21.15.23:0/2096359239 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f03e80cdcc0 con 0x7f03f01055b0 2024-03-24T10:42:55.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.186+0000 7f03effff640 1 --2- 172.21.15.23:0/3947327329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d75d10 0x5571b0d7e3b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f03e002c170 tx=0x7f03e0005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc83242a196d0258 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:55.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.187+0000 7f03f4d5f640 1 --2- 172.21.15.23:0/2096359239 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f03d80a2230 0x7f03d80a46f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f03e80cba10 tx=0x7f03e8085040 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:55.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.187+0000 7f03d67fc640 1 -- 172.21.15.23:0/3947327329 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e002c8d0 con 0x5571b0d75d10 2024-03-24T10:42:55.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.187+0000 7f03d67fc640 1 -- 172.21.15.23:0/3947327329 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03e002ca30 con 0x5571b0d75d10 2024-03-24T10:42:55.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.187+0000 7f03d67fc640 1 -- 172.21.15.23:0/3947327329 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e0005050 con 0x5571b0d75d10 2024-03-24T10:42:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.187+0000 7f03f939b740 1 -- 172.21.15.23:0/3947327329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d75d10 msgr2=0x5571b0d7e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.187+0000 7f03f939b740 1 --2- 172.21.15.23:0/3947327329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d75d10 0x5571b0d7e3b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f03e002c170 tx=0x7f03e0005c00 comp rx=0 tx=0).stop 2024-03-24T10:42:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.187+0000 7f03f939b740 1 -- 172.21.15.23:0/3947327329 shutdown_connections 2024-03-24T10:42:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.187+0000 7f03f939b740 1 --2- 172.21.15.23:0/3947327329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d7e8f0 0x5571b0d80ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.187+0000 7f03f939b740 1 --2- 172.21.15.23:0/3947327329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d75d10 0x5571b0d7e3b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.187+0000 7f03f939b740 1 --2- 172.21.15.23:0/3947327329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d757d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.187+0000 7f03f939b740 1 -- 172.21.15.23:0/3947327329 >> 172.21.15.23:0/3947327329 conn(0x5571b0d6fb20 msgr2=0x5571b0d6ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.187+0000 7f03f939b740 1 -- 172.21.15.23:0/3947327329 shutdown_connections 2024-03-24T10:42:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.187+0000 7f03f939b740 1 -- 172.21.15.23:0/3947327329 wait complete. 2024-03-24T10:42:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03f939b740 1 -- start start 2024-03-24T10:42:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03f939b740 1 Processor -- start 2024-03-24T10:42:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03f939b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d753f0 0x5571b0d7b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03f939b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d75d10 0x5571b0d7be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03f939b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d7e8f0 0x5571b0d7c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03f939b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5571b09af430 con 0x5571b0d75d10 2024-03-24T10:42:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03f939b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5571b096be00 con 0x5571b0d753f0 2024-03-24T10:42:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03f939b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5571b0c61590 con 0x5571b0d7e8f0 2024-03-24T10:42:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03f939b740 10 client.0 _ll_register_callbacks cb 0x5571b0cd0fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03effff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d75d10 0x5571b0d7be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03effff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d75d10 0x5571b0d7be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58614/0 (socket says 172.21.15.23:58614) 2024-03-24T10:42:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03effff640 1 -- 172.21.15.23:0/415457235 learned_addr learned my addr 172.21.15.23:0/415457235 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03ef7fe640 1 --2- 172.21.15.23:0/415457235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d753f0 0x5571b0d7b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03ef7fe640 1 --2- 172.21.15.23:0/415457235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d7e8f0 0x5571b0d7c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03effff640 1 -- 172.21.15.23:0/415457235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d7e8f0 msgr2=0x5571b0d7c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.188+0000 7f03effff640 1 --2- 172.21.15.23:0/415457235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d7e8f0 0x5571b0d7c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03effff640 1 -- 172.21.15.23:0/415457235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d753f0 msgr2=0x5571b0d7b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03effff640 1 --2- 172.21.15.23:0/415457235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d753f0 0x5571b0d7b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03effff640 1 -- 172.21.15.23:0/415457235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f010dd90 con 0x5571b0d75d10 2024-03-24T10:42:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03effff640 1 --2- 172.21.15.23:0/415457235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d75d10 0x5571b0d7be90 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f03e0036040 tx=0x7f03e0004b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba1a5959477b87c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03cf7fe640 1 -- 172.21.15.23:0/415457235 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e002c8d0 con 0x5571b0d75d10 2024-03-24T10:42:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03cf7fe640 1 -- 172.21.15.23:0/415457235 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f03e0005200 con 0x5571b0d75d10 2024-03-24T10:42:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03f939b740 1 -- 172.21.15.23:0/415457235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5571b09db7e0 con 0x5571b0d75d10 2024-03-24T10:42:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03cf7fe640 1 -- 172.21.15.23:0/415457235 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f03e00053c0 con 0x5571b0d75d10 2024-03-24T10:42:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03f939b740 10 client.16374 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03f939b740 1 -- 172.21.15.23:0/415457235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5571b0cdabd0 con 0x5571b0d75d10 2024-03-24T10:42:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03cf7fe640 1 -- 172.21.15.23:0/415457235 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5571b09db7e0 con 0x5571b0d75d10 2024-03-24T10:42:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03cf7fe640 1 -- 172.21.15.23:0/415457235 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f03e003c050 con 0x5571b0d75d10 2024-03-24T10:42:55.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03f939b740 10 client.16374 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:55.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03f939b740 10 client.16374 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:55.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03f939b740 20 client.16374 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:55.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03f939b740 1 --2- 172.21.15.23:0/415457235 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5571b0d8e600 0x5571b0d90a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03f939b740 4 client.16374 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:55.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.189+0000 7f03f939b740 1 -- 172.21.15.23:0/415457235 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571b0a18720 con 0x5571b0d8e600 2024-03-24T10:42:55.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.190+0000 7f03ef7fe640 1 --2- 172.21.15.23:0/415457235 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5571b0d8e600 0x5571b0d90a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.190+0000 7f03ef7fe640 1 --2- 172.21.15.23:0/415457235 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5571b0d8e600 0x5571b0d90a20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3a26d56eaea9a22 server_cookie=43447da0fe16ff23 in_seq=0 out_seq=0 2024-03-24T10:42:55.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.190+0000 7f03cf7fe640 10 client.16374 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:55.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03cf7fe640 1 -- 172.21.15.23:0/415457235 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5571b0a18720 con 0x5571b0d8e600 2024-03-24T10:42:55.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03cf7fe640 10 client.16374 handle_command_reply: tid=0 2024-03-24T10:42:55.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03f939b740 1 client.16374 shutdown 2024-03-24T10:42:55.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03f939b740 1 -- 172.21.15.23:0/415457235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d75d10 msgr2=0x5571b0d7be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03f939b740 1 --2- 172.21.15.23:0/415457235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d75d10 0x5571b0d7be90 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f03e0036040 tx=0x7f03e0004b30 comp rx=0 tx=0).stop 2024-03-24T10:42:55.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03f939b740 1 -- 172.21.15.23:0/415457235 shutdown_connections 2024-03-24T10:42:55.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03f939b740 1 --2- 172.21.15.23:0/415457235 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5571b0d8e600 0x5571b0d90a20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03f939b740 1 --2- 172.21.15.23:0/415457235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d7e8f0 0x5571b0d7c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03f939b740 1 --2- 172.21.15.23:0/415457235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d75d10 0x5571b0d7be90 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03cf7fe640 0 client.16374 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:55.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03f939b740 1 --2- 172.21.15.23:0/415457235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d753f0 0x5571b0d7b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03f939b740 1 -- 172.21.15.23:0/415457235 >> 172.21.15.23:0/415457235 conn(0x5571b0d6fb20 msgr2=0x5571b0d71870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:55.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03f939b740 1 -- 172.21.15.23:0/415457235 shutdown_connections 2024-03-24T10:42:55.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03f939b740 1 -- 172.21.15.23:0/415457235 wait complete. 2024-03-24T10:42:55.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.191+0000 7f03f939b740 20 client.16374 trim_cache size 0 max 16384 2024-03-24T10:42:55.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.204+0000 7f03f939b740 1 Processor -- start 2024-03-24T10:42:55.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03f939b740 1 -- start start 2024-03-24T10:42:55.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03f939b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d7e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03f939b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d75d10 0x5571b0d8a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03f939b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d7e8f0 0x5571b0d6ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03f939b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5571b0d70b20 con 0x5571b0d753f0 2024-03-24T10:42:55.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03f939b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5571b0d8b440 con 0x5571b0d7e8f0 2024-03-24T10:42:55.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03f939b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5571b0d60e20 con 0x5571b0d75d10 2024-03-24T10:42:55.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03ef7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d7e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03ef7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d7e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58628/0 (socket says 172.21.15.23:58628) 2024-03-24T10:42:55.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03effff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d75d10 0x5571b0d8a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03ef7fe640 1 -- 172.21.15.23:0/337680971 learned_addr learned my addr 172.21.15.23:0/337680971 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:55.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03ef7fe640 1 --2- 172.21.15.23:0/337680971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d7e8f0 0x5571b0d6ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03ef7fe640 1 -- 172.21.15.23:0/337680971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d75d10 msgr2=0x5571b0d8a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03ef7fe640 1 --2- 172.21.15.23:0/337680971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d75d10 0x5571b0d8a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03ef7fe640 1 -- 172.21.15.23:0/337680971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d7e8f0 msgr2=0x5571b0d6ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03ef7fe640 1 --2- 172.21.15.23:0/337680971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d7e8f0 0x5571b0d6ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03ef7fe640 1 -- 172.21.15.23:0/337680971 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571b0c76c80 con 0x5571b0d753f0 2024-03-24T10:42:55.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03effff640 1 --2- 172.21.15.23:0/337680971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d75d10 0x5571b0d8a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:55.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03ef7fe640 1 --2- 172.21.15.23:0/337680971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d7e080 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f03e4000c00 tx=0x7f03e4009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a05807c9657984d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:55.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.205+0000 7f03d5ffb640 1 -- 172.21.15.23:0/337680971 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e4013bd0 con 0x5571b0d753f0 2024-03-24T10:42:55.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.206+0000 7f03d5ffb640 1 -- 172.21.15.23:0/337680971 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03e4013d30 con 0x5571b0d753f0 2024-03-24T10:42:55.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.206+0000 7f03d5ffb640 1 -- 172.21.15.23:0/337680971 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e4003b70 con 0x5571b0d753f0 2024-03-24T10:42:55.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.206+0000 7f03f939b740 1 -- 172.21.15.23:0/337680971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 msgr2=0x5571b0d7e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.206+0000 7f03f939b740 1 --2- 172.21.15.23:0/337680971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d7e080 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f03e4000c00 tx=0x7f03e4009c30 comp rx=0 tx=0).stop 2024-03-24T10:42:55.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.206+0000 7f03f939b740 1 -- 172.21.15.23:0/337680971 shutdown_connections 2024-03-24T10:42:55.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.206+0000 7f03f939b740 1 --2- 172.21.15.23:0/337680971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d7e8f0 0x5571b0d6ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.206+0000 7f03f939b740 1 --2- 172.21.15.23:0/337680971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d75d10 0x5571b0d8a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.206+0000 7f03f939b740 1 --2- 172.21.15.23:0/337680971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d7e080 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.206+0000 7f03f939b740 1 -- 172.21.15.23:0/337680971 >> 172.21.15.23:0/337680971 conn(0x5571b0d6fb20 msgr2=0x5571b0d6ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:55.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.206+0000 7f03f939b740 1 -- 172.21.15.23:0/337680971 shutdown_connections 2024-03-24T10:42:55.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.206+0000 7f03f939b740 1 -- 172.21.15.23:0/337680971 wait complete. 2024-03-24T10:42:55.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03f939b740 1 -- start start 2024-03-24T10:42:55.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03f939b740 1 Processor -- start 2024-03-24T10:42:55.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03f939b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d82c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03f939b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d75d10 0x5571b0d83140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03f939b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d7e8f0 0x5571b0dec920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03f939b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5571b0decfd0 con 0x5571b0d753f0 2024-03-24T10:42:55.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03f939b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5571b0ded140 con 0x5571b0d7e8f0 2024-03-24T10:42:55.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03f939b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5571b0ded300 con 0x5571b0d75d10 2024-03-24T10:42:55.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03f939b740 10 client.0 _ll_register_callbacks cb 0x7f03e4009b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:55.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03ef7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d82c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03ef7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d82c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58640/0 (socket says 172.21.15.23:58640) 2024-03-24T10:42:55.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03ef7fe640 1 -- 172.21.15.23:0/2439501373 learned_addr learned my addr 172.21.15.23:0/2439501373 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:55.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03effff640 1 --2- 172.21.15.23:0/2439501373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d75d10 0x5571b0d83140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03ef7fe640 1 --2- 172.21.15.23:0/2439501373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d7e8f0 0x5571b0dec920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03ef7fe640 1 -- 172.21.15.23:0/2439501373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d75d10 msgr2=0x5571b0d83140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.207+0000 7f03ef7fe640 1 --2- 172.21.15.23:0/2439501373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d75d10 0x5571b0d83140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03ef7fe640 1 -- 172.21.15.23:0/2439501373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d7e8f0 msgr2=0x5571b0dec920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:55.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03ef7fe640 1 --2- 172.21.15.23:0/2439501373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d7e8f0 0x5571b0dec920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03ef7fe640 1 -- 172.21.15.23:0/2439501373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571b0cdabd0 con 0x5571b0d753f0 2024-03-24T10:42:55.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03ef7fe640 1 --2- 172.21.15.23:0/2439501373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d82c00 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f03e4013f50 tx=0x7f03e4003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5c908f3abe5979 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:55.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:55.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03cf7fe640 1 -- 172.21.15.23:0/2439501373 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e4005110 con 0x5571b0d753f0 2024-03-24T10:42:55.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03cf7fe640 1 -- 172.21.15.23:0/2439501373 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f03e4005270 con 0x5571b0d753f0 2024-03-24T10:42:55.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03f939b740 1 -- 172.21.15.23:0/2439501373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03e002ca30 con 0x5571b0d753f0 2024-03-24T10:42:55.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03cf7fe640 1 -- 172.21.15.23:0/2439501373 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f03e4005430 con 0x5571b0d753f0 2024-03-24T10:42:55.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03f939b740 10 client.16380 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:55.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03f939b740 1 -- 172.21.15.23:0/2439501373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x7f03e401b040 con 0x5571b0d753f0 2024-03-24T10:42:55.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03cf7fe640 1 -- 172.21.15.23:0/2439501373 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03e002ca30 con 0x5571b0d753f0 2024-03-24T10:42:55.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03cf7fe640 1 -- 172.21.15.23:0/2439501373 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f03e40275c0 con 0x5571b0d753f0 2024-03-24T10:42:55.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03f939b740 10 client.16380 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:55.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03f939b740 10 client.16380 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:55.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03f939b740 20 client.16380 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:55.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03f939b740 1 --2- 172.21.15.23:0/2439501373 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5571b0df59d0 0x5571b0df7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:55.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03f939b740 4 client.16380 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:42:55.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.208+0000 7f03f939b740 1 -- 172.21.15.23:0/2439501373 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5571b0a67890 con 0x5571b0df59d0 2024-03-24T10:42:55.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03effff640 1 --2- 172.21.15.23:0/2439501373 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5571b0df59d0 0x5571b0df7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:55.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03effff640 1 --2- 172.21.15.23:0/2439501373 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5571b0df59d0 0x5571b0df7df0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adca51b8444f86a4 server_cookie=6bd0e824112f3d3e in_seq=0 out_seq=0 2024-03-24T10:42:55.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03cf7fe640 10 client.16380 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:55.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03cf7fe640 1 -- 172.21.15.23:0/2439501373 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5571b0a67890 con 0x5571b0df59d0 2024-03-24T10:42:55.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03cf7fe640 10 client.16380 handle_command_reply: tid=0 2024-03-24T10:42:55.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03f939b740 1 client.16380 shutdown 2024-03-24T10:42:55.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03f939b740 1 -- 172.21.15.23:0/2439501373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 msgr2=0x5571b0d82c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03f939b740 1 --2- 172.21.15.23:0/2439501373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d82c00 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f03e4013f50 tx=0x7f03e4003730 comp rx=0 tx=0).stop 2024-03-24T10:42:55.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03f939b740 1 -- 172.21.15.23:0/2439501373 shutdown_connections 2024-03-24T10:42:55.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03f939b740 1 --2- 172.21.15.23:0/2439501373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571b0d7e8f0 0x5571b0dec920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03f939b740 1 --2- 172.21.15.23:0/2439501373 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5571b0df59d0 0x5571b0df7df0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03f939b740 1 --2- 172.21.15.23:0/2439501373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571b0d75d10 0x5571b0d83140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03f939b740 1 --2- 172.21.15.23:0/2439501373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571b0d753f0 0x5571b0d82c00 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03cf7fe640 0 client.16380 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:55.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03f939b740 1 -- 172.21.15.23:0/2439501373 >> 172.21.15.23:0/2439501373 conn(0x5571b0d6fb20 msgr2=0x5571b0d6ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:55.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03f939b740 1 -- 172.21.15.23:0/2439501373 shutdown_connections 2024-03-24T10:42:55.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03f939b740 1 -- 172.21.15.23:0/2439501373 wait complete. 2024-03-24T10:42:55.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.209+0000 7f03f939b740 20 client.16380 trim_cache size 0 max 16384 2024-03-24T10:42:55.219 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:42:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.210+0000 7f03f67c2640 1 -- 172.21.15.23:0/2096359239 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f03d80a2230 msgr2=0x7f03d80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.210+0000 7f03f67c2640 1 --2- 172.21.15.23:0/2096359239 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f03d80a2230 0x7f03d80a46f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f03e80cba10 tx=0x7f03e8085040 comp rx=0 tx=0).stop 2024-03-24T10:42:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.210+0000 7f03f67c2640 1 -- 172.21.15.23:0/2096359239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03f01055b0 msgr2=0x7f03f01a3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.210+0000 7f03f67c2640 1 --2- 172.21.15.23:0/2096359239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03f01055b0 0x7f03f01a3bc0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009fd0 tx=0x7f03e800b260 comp rx=0 tx=0).stop 2024-03-24T10:42:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.210+0000 7f03f67c2640 1 -- 172.21.15.23:0/2096359239 shutdown_connections 2024-03-24T10:42:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.210+0000 7f03f67c2640 1 --2- 172.21.15.23:0/2096359239 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f03d80a2230 0x7f03d80a46f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.210+0000 7f03f67c2640 1 --2- 172.21.15.23:0/2096359239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03f01055b0 0x7f03f01a3bc0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.210+0000 7f03f67c2640 1 --2- 172.21.15.23:0/2096359239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03f0101ae0 0x7f03f0199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.210+0000 7f03f67c2640 1 --2- 172.21.15.23:0/2096359239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03f0101120 0x7f03f0198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.210+0000 7f03f67c2640 1 -- 172.21.15.23:0/2096359239 >> 172.21.15.23:0/2096359239 conn(0x7f03f0075870 msgr2=0x7f03f00fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.210+0000 7f03f67c2640 1 -- 172.21.15.23:0/2096359239 shutdown_connections 2024-03-24T10:42:55.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:55.210+0000 7f03f67c2640 1 -- 172.21.15.23:0/2096359239 wait complete. 2024-03-24T10:42:55.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:55 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2914874925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:42:55.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:55 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2914874925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:42:55.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:55 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2914874925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:42:56.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 167868 12.63 MB/sec warmup 73 sec latency 58.789 ms 2024-03-24T10:42:56.223 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:42:56.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:42:56.223 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:42:56.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.339+0000 7f296233e640 1 -- 172.21.15.23:0/940386427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f295c100f60 msgr2=0x7f295c10d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.339+0000 7f296233e640 1 --2- 172.21.15.23:0/940386427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f295c100f60 0x7f295c10d420 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2954009ca0 tx=0x7f295402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.339+0000 7f296233e640 1 -- 172.21.15.23:0/940386427 shutdown_connections 2024-03-24T10:42:56.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.339+0000 7f296233e640 1 --2- 172.21.15.23:0/940386427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f295c100f60 0x7f295c10d420 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.339+0000 7f296233e640 1 --2- 172.21.15.23:0/940386427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f295c1005a0 0x7f295c100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.339+0000 7f296233e640 1 --2- 172.21.15.23:0/940386427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f295c106660 0x7f295c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.339+0000 7f296233e640 1 -- 172.21.15.23:0/940386427 >> 172.21.15.23:0/940386427 conn(0x7f295c077710 msgr2=0x7f295c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:56.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.339+0000 7f296233e640 1 -- 172.21.15.23:0/940386427 shutdown_connections 2024-03-24T10:42:56.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.339+0000 7f296233e640 1 -- 172.21.15.23:0/940386427 wait complete. 2024-03-24T10:42:56.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.339+0000 7f296233e640 1 Processor -- start 2024-03-24T10:42:56.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.339+0000 7f296233e640 1 -- start start 2024-03-24T10:42:56.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f296233e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f295c1005a0 0x7f295c199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f296233e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f295c100f60 0x7f295c19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f296233e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f295c106660 0x7f295c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f296233e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f295c19ecd0 con 0x7f295c100f60 2024-03-24T10:42:56.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f296233e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f295c19ee40 con 0x7f295c1005a0 2024-03-24T10:42:56.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f296233e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f295c19f000 con 0x7f295c106660 2024-03-24T10:42:56.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f295b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f295c100f60 0x7f295c19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f295b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f295c100f60 0x7f295c19a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58650/0 (socket says 172.21.15.23:58650) 2024-03-24T10:42:56.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f295b7fe640 1 -- 172.21.15.23:0/1251331389 learned_addr learned my addr 172.21.15.23:0/1251331389 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:56.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f29608db640 1 --2- 172.21.15.23:0/1251331389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f295c106660 0x7f295c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f295bfff640 1 --2- 172.21.15.23:0/1251331389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f295c1005a0 0x7f295c199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f295b7fe640 1 -- 172.21.15.23:0/1251331389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f295c106660 msgr2=0x7f295c19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f295b7fe640 1 --2- 172.21.15.23:0/1251331389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f295c106660 0x7f295c19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f295b7fe640 1 -- 172.21.15.23:0/1251331389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f295c1005a0 msgr2=0x7f295c199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f295b7fe640 1 --2- 172.21.15.23:0/1251331389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f295c1005a0 0x7f295c199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f295b7fe640 1 -- 172.21.15.23:0/1251331389 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2954009900 con 0x7f295c100f60 2024-03-24T10:42:56.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f295b7fe640 1 --2- 172.21.15.23:0/1251331389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f295c100f60 0x7f295c19a350 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f295000e9f0 tx=0x7f295000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:56.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.340+0000 7f29597fa640 1 -- 172.21.15.23:0/1251331389 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2950009720 con 0x7f295c100f60 2024-03-24T10:42:56.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.341+0000 7f296233e640 1 -- 172.21.15.23:0/1251331389 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f295c19f2e0 con 0x7f295c100f60 2024-03-24T10:42:56.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.341+0000 7f29597fa640 1 -- 172.21.15.23:0/1251331389 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f295000cd80 con 0x7f295c100f60 2024-03-24T10:42:56.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.341+0000 7f29597fa640 1 -- 172.21.15.23:0/1251331389 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2950010690 con 0x7f295c100f60 2024-03-24T10:42:56.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.341+0000 7f296233e640 1 -- 172.21.15.23:0/1251331389 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f295c1a6b60 con 0x7f295c100f60 2024-03-24T10:42:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.343+0000 7f29597fa640 1 -- 172.21.15.23:0/1251331389 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f29500046c0 con 0x7f295c100f60 2024-03-24T10:42:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.343+0000 7f2964fff740 1 Processor -- start 2024-03-24T10:42:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.343+0000 7f29597fa640 1 --2- 172.21.15.23:0/1251331389 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f29440a2230 0x7f29440a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.343+0000 7f2964fff740 1 -- start start 2024-03-24T10:42:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.343+0000 7f2964fff740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6490 0x5590afce6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f2964fff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6db0 0x5590afcf7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f29608db640 1 --2- 172.21.15.23:0/1251331389 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f29440a2230 0x7f29440a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f2964fff740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afcf8470 0x5590afcfa860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f2964fff740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5590afc54530 con 0x5590afcf8470 2024-03-24T10:42:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f2964fff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5590afc51a60 con 0x5590afce6db0 2024-03-24T10:42:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f2964fff740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5590afc4d240 con 0x5590afce6490 2024-03-24T10:42:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f29597fa640 1 -- 172.21.15.23:0/1251331389 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f2950014070 con 0x7f295c100f60 2024-03-24T10:42:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f29608db640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afcf8470 0x5590afcfa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f295bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6490 0x5590afce6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f295bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6490 0x5590afce6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39666/0 (socket says 172.21.15.23:39666) 2024-03-24T10:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f295bfff640 1 -- 172.21.15.23:0/1067315503 learned_addr learned my addr 172.21.15.23:0/1067315503 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f29608db640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afcf8470 0x5590afcfa860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58662/0 (socket says 172.21.15.23:58662) 2024-03-24T10:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f295bfff640 1 -- 172.21.15.23:0/1067315503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6db0 msgr2=0x5590afcf7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f295b7fe640 1 --2- 172.21.15.23:0/1067315503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6db0 0x5590afcf7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f295bfff640 1 --2- 172.21.15.23:0/1067315503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6db0 0x5590afcf7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f295bfff640 1 -- 172.21.15.23:0/1067315503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afcf8470 msgr2=0x5590afcfa860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f295bfff640 1 --2- 172.21.15.23:0/1067315503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afcf8470 0x5590afcfa860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f295bfff640 1 -- 172.21.15.23:0/1067315503 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590afc4b8c0 con 0x5590afce6490 2024-03-24T10:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f29608db640 1 --2- 172.21.15.23:0/1067315503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afcf8470 0x5590afcfa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f295b7fe640 1 --2- 172.21.15.23:0/1067315503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6db0 0x5590afcf7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.344+0000 7f29608db640 1 --2- 172.21.15.23:0/1251331389 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f29440a2230 0x7f29440a46f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2954004af0 tx=0x7f2954032f10 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.345+0000 7f295bfff640 1 --2- 172.21.15.23:0/1067315503 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6490 0x5590afce6870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f294c00bcd0 tx=0x7f294c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7aac4d05a13a1bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.345+0000 7f29427fc640 1 -- 172.21.15.23:0/1067315503 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c005220 con 0x5590afce6490 2024-03-24T10:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.345+0000 7f29427fc640 1 -- 172.21.15.23:0/1067315503 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f294c015070 con 0x5590afce6490 2024-03-24T10:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.345+0000 7f29427fc640 1 -- 172.21.15.23:0/1067315503 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c005690 con 0x5590afce6490 2024-03-24T10:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.345+0000 7f2964fff740 1 -- 172.21.15.23:0/1067315503 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6490 msgr2=0x5590afce6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.345+0000 7f2964fff740 1 --2- 172.21.15.23:0/1067315503 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6490 0x5590afce6870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f294c00bcd0 tx=0x7f294c0107a0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.345+0000 7f2964fff740 1 -- 172.21.15.23:0/1067315503 shutdown_connections 2024-03-24T10:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.345+0000 7f2964fff740 1 --2- 172.21.15.23:0/1067315503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afcf8470 0x5590afcfa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.345+0000 7f2964fff740 1 --2- 172.21.15.23:0/1067315503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6db0 0x5590afcf7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.345+0000 7f2964fff740 1 --2- 172.21.15.23:0/1067315503 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6490 0x5590afce6870 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.345+0000 7f2964fff740 1 -- 172.21.15.23:0/1067315503 >> 172.21.15.23:0/1067315503 conn(0x5590afce0e90 msgr2=0x5590afce1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f2964fff740 1 -- 172.21.15.23:0/1067315503 shutdown_connections 2024-03-24T10:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f2964fff740 1 -- 172.21.15.23:0/1067315503 wait complete. 2024-03-24T10:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f2964fff740 1 -- start start 2024-03-24T10:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f2964fff740 1 Processor -- start 2024-03-24T10:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f2964fff740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6490 0x5590afcf1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f2964fff740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afce6db0 0x5590afcf2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f2964fff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afcf8470 0x5590afcf7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f2964fff740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5590afbf1f80 con 0x5590afce6db0 2024-03-24T10:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f2964fff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5590af9204b0 con 0x5590afcf8470 2024-03-24T10:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f2964fff740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5590afbd2ad0 con 0x5590afce6490 2024-03-24T10:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f2964fff740 10 client.0 _ll_register_callbacks cb 0x5590afc05b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f295b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afce6db0 0x5590afcf2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f295bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6490 0x5590afcf1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f295b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afce6db0 0x5590afcf2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58674/0 (socket says 172.21.15.23:58674) 2024-03-24T10:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f295b7fe640 1 -- 172.21.15.23:0/2039238539 learned_addr learned my addr 172.21.15.23:0/2039238539 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.346+0000 7f29608db640 1 --2- 172.21.15.23:0/2039238539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afcf8470 0x5590afcf7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f295b7fe640 1 -- 172.21.15.23:0/2039238539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6490 msgr2=0x5590afcf1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f295b7fe640 1 --2- 172.21.15.23:0/2039238539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6490 0x5590afcf1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f295b7fe640 1 -- 172.21.15.23:0/2039238539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afcf8470 msgr2=0x5590afcf7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f295b7fe640 1 --2- 172.21.15.23:0/2039238539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afcf8470 0x5590afcf7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f295b7fe640 1 -- 172.21.15.23:0/2039238539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294c00b900 con 0x5590afce6db0 2024-03-24T10:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f295bfff640 1 --2- 172.21.15.23:0/2039238539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6490 0x5590afcf1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f295b7fe640 1 --2- 172.21.15.23:0/2039238539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afce6db0 0x5590afcf2400 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2950097260 tx=0x7f295009b640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ca44ab4f9e712 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f293b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f293b7fe640 1 -- 172.21.15.23:0/2039238539 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f295009d070 con 0x5590afce6db0 2024-03-24T10:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f293b7fe640 1 -- 172.21.15.23:0/2039238539 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f29500b9310 con 0x5590afce6db0 2024-03-24T10:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f2964fff740 1 -- 172.21.15.23:0/2039238539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5590afc717a0 con 0x5590afce6db0 2024-03-24T10:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f293b7fe640 1 -- 172.21.15.23:0/2039238539 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f29500a3110 con 0x5590afce6db0 2024-03-24T10:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f2964fff740 10 client.16392 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f2964fff740 1 -- 172.21.15.23:0/2039238539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5590afc036d0 con 0x5590afce6db0 2024-03-24T10:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f293b7fe640 1 -- 172.21.15.23:0/2039238539 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5590afc717a0 con 0x5590afce6db0 2024-03-24T10:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.347+0000 7f293b7fe640 1 -- 172.21.15.23:0/2039238539 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f29500cd040 con 0x5590afce6db0 2024-03-24T10:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.348+0000 7f2964fff740 10 client.16392 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.348+0000 7f2964fff740 10 client.16392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.348+0000 7f2964fff740 20 client.16392 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.348+0000 7f2964fff740 1 --2- 172.21.15.23:0/2039238539 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5590afd10ce0 0x5590afd13140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.348+0000 7f2964fff740 4 client.16392 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.348+0000 7f2964fff740 1 -- 172.21.15.23:0/2039238539 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590afc431f0 con 0x5590afd10ce0 2024-03-24T10:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.348+0000 7f295bfff640 1 --2- 172.21.15.23:0/2039238539 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5590afd10ce0 0x5590afd13140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.348+0000 7f293b7fe640 10 client.16392 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.348+0000 7f295bfff640 1 --2- 172.21.15.23:0/2039238539 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5590afd10ce0 0x5590afd13140 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c8b8083a31817af server_cookie=b50b8afdbd78ee52 in_seq=0 out_seq=0 2024-03-24T10:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f293b7fe640 1 -- 172.21.15.23:0/2039238539 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5590afc431f0 con 0x5590afd10ce0 2024-03-24T10:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f293b7fe640 10 client.16392 handle_command_reply: tid=0 2024-03-24T10:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f2964fff740 1 client.16392 shutdown 2024-03-24T10:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f2964fff740 1 -- 172.21.15.23:0/2039238539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afce6db0 msgr2=0x5590afcf2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f2964fff740 1 --2- 172.21.15.23:0/2039238539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afce6db0 0x5590afcf2400 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2950097260 tx=0x7f295009b640 comp rx=0 tx=0).stop 2024-03-24T10:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f2964fff740 1 -- 172.21.15.23:0/2039238539 shutdown_connections 2024-03-24T10:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f2964fff740 1 --2- 172.21.15.23:0/2039238539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afcf8470 0x5590afcf7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f2964fff740 1 --2- 172.21.15.23:0/2039238539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afce6db0 0x5590afcf2400 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f2964fff740 1 --2- 172.21.15.23:0/2039238539 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5590afd10ce0 0x5590afd13140 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f2964fff740 1 --2- 172.21.15.23:0/2039238539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6490 0x5590afcf1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f293b7fe640 0 client.16392 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f2964fff740 1 -- 172.21.15.23:0/2039238539 >> 172.21.15.23:0/2039238539 conn(0x5590afce0e90 msgr2=0x5590afce2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f2964fff740 1 -- 172.21.15.23:0/2039238539 shutdown_connections 2024-03-24T10:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f2964fff740 1 -- 172.21.15.23:0/2039238539 wait complete. 2024-03-24T10:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.349+0000 7f2964fff740 20 client.16392 trim_cache size 0 max 16384 2024-03-24T10:42:56.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f2964fff740 1 Processor -- start 2024-03-24T10:42:56.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f2964fff740 1 -- start start 2024-03-24T10:42:56.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f2964fff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6490 0x5590afcdfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f2964fff740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6db0 0x5590afd18a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f2964fff740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afcf8470 0x5590afcea580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f2964fff740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5590afc431f0 con 0x5590afcf8470 2024-03-24T10:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f2964fff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5590afc717a0 con 0x5590afce6490 2024-03-24T10:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f2964fff740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5590afce1c90 con 0x5590afce6db0 2024-03-24T10:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f29608db640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afcf8470 0x5590afcea580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f295b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6db0 0x5590afd18a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f29608db640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afcf8470 0x5590afcea580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58686/0 (socket says 172.21.15.23:58686) 2024-03-24T10:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f295bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6490 0x5590afcdfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f29608db640 1 -- 172.21.15.23:0/57569460 learned_addr learned my addr 172.21.15.23:0/57569460 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f29608db640 1 -- 172.21.15.23:0/57569460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6db0 msgr2=0x5590afd18a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f29608db640 1 --2- 172.21.15.23:0/57569460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6db0 0x5590afd18a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f29608db640 1 -- 172.21.15.23:0/57569460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6490 msgr2=0x5590afcdfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f29608db640 1 --2- 172.21.15.23:0/57569460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6490 0x5590afcdfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f29608db640 1 -- 172.21.15.23:0/57569460 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590afbe7c60 con 0x5590afcf8470 2024-03-24T10:42:56.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f295bfff640 1 --2- 172.21.15.23:0/57569460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6490 0x5590afcdfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:56.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.369+0000 7f295b7fe640 1 --2- 172.21.15.23:0/57569460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6db0 0x5590afd18a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:56.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.370+0000 7f29608db640 1 --2- 172.21.15.23:0/57569460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afcf8470 0x5590afcea580 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2954009ca0 tx=0x7f29540348b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c309bc2e5c42ab6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:56.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.370+0000 7f2941ffb640 1 -- 172.21.15.23:0/57569460 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2954048070 con 0x5590afcf8470 2024-03-24T10:42:56.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.370+0000 7f2941ffb640 1 -- 172.21.15.23:0/57569460 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2954003970 con 0x5590afcf8470 2024-03-24T10:42:56.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.370+0000 7f2941ffb640 1 -- 172.21.15.23:0/57569460 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2954004540 con 0x5590afcf8470 2024-03-24T10:42:56.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.370+0000 7f2964fff740 1 -- 172.21.15.23:0/57569460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afcf8470 msgr2=0x5590afcea580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.370+0000 7f2964fff740 1 --2- 172.21.15.23:0/57569460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afcf8470 0x5590afcea580 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2954009ca0 tx=0x7f29540348b0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.370+0000 7f2964fff740 1 -- 172.21.15.23:0/57569460 shutdown_connections 2024-03-24T10:42:56.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.370+0000 7f2964fff740 1 --2- 172.21.15.23:0/57569460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afcf8470 0x5590afcea580 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f2964fff740 1 --2- 172.21.15.23:0/57569460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afce6db0 0x5590afd18a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f2964fff740 1 --2- 172.21.15.23:0/57569460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6490 0x5590afcdfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f2964fff740 1 -- 172.21.15.23:0/57569460 >> 172.21.15.23:0/57569460 conn(0x5590afce0e90 msgr2=0x5590afcdf660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f2964fff740 1 -- 172.21.15.23:0/57569460 shutdown_connections 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f2964fff740 1 -- 172.21.15.23:0/57569460 wait complete. 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f2964fff740 1 -- start start 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f2964fff740 1 Processor -- start 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f2964fff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6490 0x5590afd22970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f2964fff740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afce6db0 0x5590afd22eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f2964fff740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afcf8470 0x5590afd233f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f2964fff740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5590afd76b50 con 0x5590afce6db0 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f2964fff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5590afd76cc0 con 0x5590afce6490 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f2964fff740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5590afd76e80 con 0x5590afcf8470 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f2964fff740 10 client.0 _ll_register_callbacks cb 0x7f295009a760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f295b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afce6db0 0x5590afd22eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f295bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6490 0x5590afd22970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f295b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afce6db0 0x5590afd22eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58692/0 (socket says 172.21.15.23:58692) 2024-03-24T10:42:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.371+0000 7f295b7fe640 1 -- 172.21.15.23:0/3374654975 learned_addr learned my addr 172.21.15.23:0/3374654975 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f29608db640 1 --2- 172.21.15.23:0/3374654975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afcf8470 0x5590afd233f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f295b7fe640 1 -- 172.21.15.23:0/3374654975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afcf8470 msgr2=0x5590afd233f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f295b7fe640 1 --2- 172.21.15.23:0/3374654975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afcf8470 0x5590afd233f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f295b7fe640 1 -- 172.21.15.23:0/3374654975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6490 msgr2=0x5590afd22970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f295b7fe640 1 --2- 172.21.15.23:0/3374654975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6490 0x5590afd22970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f295b7fe640 1 -- 172.21.15.23:0/3374654975 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590afc4b8c0 con 0x5590afce6db0 2024-03-24T10:42:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f295b7fe640 1 --2- 172.21.15.23:0/3374654975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afce6db0 0x5590afd22eb0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f295009b6b0 tx=0x7f295009a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80974ac396fff675 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f293b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f293b7fe640 1 -- 172.21.15.23:0/3374654975 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f295009d070 con 0x5590afce6db0 2024-03-24T10:42:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f293b7fe640 1 -- 172.21.15.23:0/3374654975 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f29500b92b0 con 0x5590afce6db0 2024-03-24T10:42:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f2964fff740 1 -- 172.21.15.23:0/3374654975 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f294c015070 con 0x5590afce6db0 2024-03-24T10:42:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f293b7fe640 1 -- 172.21.15.23:0/3374654975 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f29500970a0 con 0x5590afce6db0 2024-03-24T10:42:56.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f2964fff740 10 client.16398 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:56.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f2964fff740 1 -- 172.21.15.23:0/3374654975 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5590afc036d0 con 0x5590afce6db0 2024-03-24T10:42:56.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.372+0000 7f293b7fe640 1 -- 172.21.15.23:0/3374654975 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2950089020 con 0x5590afce6db0 2024-03-24T10:42:56.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.373+0000 7f293b7fe640 1 -- 172.21.15.23:0/3374654975 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f29500cd040 con 0x5590afce6db0 2024-03-24T10:42:56.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.373+0000 7f2964fff740 10 client.16398 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:56.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.373+0000 7f2964fff740 10 client.16398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:56.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.373+0000 7f2964fff740 20 client.16398 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:56.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.373+0000 7f2964fff740 1 --2- 172.21.15.23:0/3374654975 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5590afd78310 0x5590afd7a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.373+0000 7f2964fff740 4 client.16398 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:42:56.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.373+0000 7f2964fff740 1 -- 172.21.15.23:0/3374654975 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5590af94c860 con 0x5590afd78310 2024-03-24T10:42:56.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.373+0000 7f295bfff640 1 --2- 172.21.15.23:0/3374654975 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5590afd78310 0x5590afd7a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.374+0000 7f295bfff640 1 --2- 172.21.15.23:0/3374654975 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5590afd78310 0x5590afd7a730 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4e88674a808381d server_cookie=741178d3fde32895 in_seq=0 out_seq=0 2024-03-24T10:42:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.374+0000 7f293b7fe640 10 client.16398 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.375+0000 7f293b7fe640 1 -- 172.21.15.23:0/3374654975 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5590af94c860 con 0x5590afd78310 2024-03-24T10:42:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.375+0000 7f293b7fe640 10 client.16398 handle_command_reply: tid=0 2024-03-24T10:42:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.375+0000 7f2964fff740 1 client.16398 shutdown 2024-03-24T10:42:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.375+0000 7f2964fff740 1 -- 172.21.15.23:0/3374654975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afce6db0 msgr2=0x5590afd22eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.375+0000 7f2964fff740 1 --2- 172.21.15.23:0/3374654975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afce6db0 0x5590afd22eb0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f295009b6b0 tx=0x7f295009a810 comp rx=0 tx=0).stop 2024-03-24T10:42:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.375+0000 7f2964fff740 1 -- 172.21.15.23:0/3374654975 shutdown_connections 2024-03-24T10:42:56.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.375+0000 7f2964fff740 1 --2- 172.21.15.23:0/3374654975 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5590afd78310 0x5590afd7a730 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.375+0000 7f2964fff740 1 --2- 172.21.15.23:0/3374654975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5590afcf8470 0x5590afd233f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.375+0000 7f293b7fe640 0 client.16398 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:56.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.376+0000 7f2964fff740 1 --2- 172.21.15.23:0/3374654975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590afce6db0 0x5590afd22eb0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.376+0000 7f2964fff740 1 --2- 172.21.15.23:0/3374654975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5590afce6490 0x5590afd22970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.376+0000 7f2964fff740 1 -- 172.21.15.23:0/3374654975 >> 172.21.15.23:0/3374654975 conn(0x5590afce0e90 msgr2=0x5590afcdfef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:56.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.376+0000 7f2964fff740 1 -- 172.21.15.23:0/3374654975 shutdown_connections 2024-03-24T10:42:56.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.376+0000 7f2964fff740 1 -- 172.21.15.23:0/3374654975 wait complete. 2024-03-24T10:42:56.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.376+0000 7f2964fff740 20 client.16398 trim_cache size 0 max 16384 2024-03-24T10:42:56.381 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:42:56.382 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:42:56.382 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "8277813c-3dba-4650-82a1-91d44675e599", 2024-03-24T10:42:56.382 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:42:56.382 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:42:56.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.376+0000 7f296233e640 1 -- 172.21.15.23:0/1251331389 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f29440a2230 msgr2=0x7f29440a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.376+0000 7f296233e640 1 --2- 172.21.15.23:0/1251331389 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f29440a2230 0x7f29440a46f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2954004af0 tx=0x7f2954032f10 comp rx=0 tx=0).stop 2024-03-24T10:42:56.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.376+0000 7f296233e640 1 -- 172.21.15.23:0/1251331389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f295c100f60 msgr2=0x7f295c19a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.376+0000 7f296233e640 1 --2- 172.21.15.23:0/1251331389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f295c100f60 0x7f295c19a350 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f295000e9f0 tx=0x7f295000edf0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.377+0000 7f296233e640 1 -- 172.21.15.23:0/1251331389 shutdown_connections 2024-03-24T10:42:56.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.377+0000 7f296233e640 1 --2- 172.21.15.23:0/1251331389 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f29440a2230 0x7f29440a46f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.377+0000 7f296233e640 1 --2- 172.21.15.23:0/1251331389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f295c106660 0x7f295c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.377+0000 7f296233e640 1 --2- 172.21.15.23:0/1251331389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f295c100f60 0x7f295c19a350 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.377+0000 7f296233e640 1 --2- 172.21.15.23:0/1251331389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f295c1005a0 0x7f295c199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.377+0000 7f296233e640 1 -- 172.21.15.23:0/1251331389 >> 172.21.15.23:0/1251331389 conn(0x7f295c077710 msgr2=0x7f295c0688a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:56.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.377+0000 7f296233e640 1 -- 172.21.15.23:0/1251331389 shutdown_connections 2024-03-24T10:42:56.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.377+0000 7f296233e640 1 -- 172.21.15.23:0/1251331389 wait complete. 2024-03-24T10:42:56.388 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:42:56.388 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:42:56.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.499+0000 7fcef5cdf640 1 -- 172.21.15.23:0/466834097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcef0102980 msgr2=0x7fcef0102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.499+0000 7fcef5cdf640 1 --2- 172.21.15.23:0/466834097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcef0102980 0x7fcef0102d60 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fcee0009900 tx=0x7fcee002bcd0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.500+0000 7fcef5cdf640 1 -- 172.21.15.23:0/466834097 shutdown_connections 2024-03-24T10:42:56.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.500+0000 7fcef5cdf640 1 --2- 172.21.15.23:0/466834097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef010bd50 0x7fcef010e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.500+0000 7fcef5cdf640 1 --2- 172.21.15.23:0/466834097 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcef01032a0 0x7fcef010b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.500+0000 7fcef5cdf640 1 --2- 172.21.15.23:0/466834097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcef0102980 0x7fcef0102d60 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.500+0000 7fcef5cdf640 1 -- 172.21.15.23:0/466834097 >> 172.21.15.23:0/466834097 conn(0x7fcef0077770 msgr2=0x7fcef0077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:56.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.500+0000 7fcef5cdf640 1 -- 172.21.15.23:0/466834097 shutdown_connections 2024-03-24T10:42:56.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.500+0000 7fcef5cdf640 1 -- 172.21.15.23:0/466834097 wait complete. 2024-03-24T10:42:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.500+0000 7fcef5cdf640 1 Processor -- start 2024-03-24T10:42:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.500+0000 7fcef5cdf640 1 -- start start 2024-03-24T10:42:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fcef5cdf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcef0102980 0x7fcef0197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fcef5cdf640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcef01032a0 0x7fcef0198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fcef5cdf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef010bd50 0x7fcef019c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fcef5cdf640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fcef019cd10 con 0x7fcef01032a0 2024-03-24T10:42:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fcef5cdf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcef019ce80 con 0x7fcef010bd50 2024-03-24T10:42:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fcef5cdf640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcef019d040 con 0x7fcef0102980 2024-03-24T10:42:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fceeeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcef01032a0 0x7fcef0198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fceeeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcef01032a0 0x7fcef0198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58710/0 (socket says 172.21.15.23:58710) 2024-03-24T10:42:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fceeeffd640 1 -- 172.21.15.23:0/1793389276 learned_addr learned my addr 172.21.15.23:0/1793389276 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fceef7fe640 1 --2- 172.21.15.23:0/1793389276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcef0102980 0x7fcef0197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fceeffff640 1 --2- 172.21.15.23:0/1793389276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef010bd50 0x7fcef019c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fceeeffd640 1 -- 172.21.15.23:0/1793389276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcef0102980 msgr2=0x7fcef0197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fceeeffd640 1 --2- 172.21.15.23:0/1793389276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcef0102980 0x7fcef0197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fceeeffd640 1 -- 172.21.15.23:0/1793389276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef010bd50 msgr2=0x7fcef019c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fceeeffd640 1 --2- 172.21.15.23:0/1793389276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef010bd50 0x7fcef019c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fceeeffd640 1 -- 172.21.15.23:0/1793389276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee0009560 con 0x7fcef01032a0 2024-03-24T10:42:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fceeeffd640 1 --2- 172.21.15.23:0/1793389276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcef01032a0 0x7fcef0198390 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcee4009f40 tx=0x7fcee400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fceecff9640 1 -- 172.21.15.23:0/1793389276 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee4009720 con 0x7fcef01032a0 2024-03-24T10:42:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fcef5cdf640 1 -- 172.21.15.23:0/1793389276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef019d320 con 0x7fcef01032a0 2024-03-24T10:42:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fceecff9640 1 -- 172.21.15.23:0/1793389276 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcee400cd80 con 0x7fcef01032a0 2024-03-24T10:42:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.501+0000 7fceecff9640 1 -- 172.21.15.23:0/1793389276 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee40047a0 con 0x7fcef01032a0 2024-03-24T10:42:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.502+0000 7fcef5cdf640 1 -- 172.21.15.23:0/1793389276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef01006e0 con 0x7fcef01032a0 2024-03-24T10:42:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fceecff9640 1 -- 172.21.15.23:0/1793389276 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fcee401b440 con 0x7fcef01032a0 2024-03-24T10:42:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fcef8999740 1 Processor -- start 2024-03-24T10:42:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fcef8999740 1 -- start start 2024-03-24T10:42:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fcef8999740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16881d0 0x5651c16885b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fceecff9640 1 --2- 172.21.15.23:0/1793389276 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fced80a2230 0x7fced80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fcef8999740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c1688af0 0x5651c1691190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fcef8999740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c16916d0 0x5651c1693ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fcef8999740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5651c15acf30 con 0x5651c1688af0 2024-03-24T10:42:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fcef8999740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5651c15f6810 con 0x5651c16881d0 2024-03-24T10:42:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fcef8999740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5651c15f3d40 con 0x5651c16916d0 2024-03-24T10:42:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fceeeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c1688af0 0x5651c1691190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fceef7fe640 1 --2- 172.21.15.23:0/1793389276 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fced80a2230 0x7fced80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fceeeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c1688af0 0x5651c1691190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58720/0 (socket says 172.21.15.23:58720) 2024-03-24T10:42:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fceeeffd640 1 -- 172.21.15.23:0/1695617964 learned_addr learned my addr 172.21.15.23:0/1695617964 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fceeffff640 1 --2- 172.21.15.23:0/1695617964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16881d0 0x5651c16885b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fceeffff640 1 --2- 172.21.15.23:0/1695617964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c16916d0 0x5651c1693ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.504+0000 7fceecff9640 1 -- 172.21.15.23:0/1793389276 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fcee40d22d0 con 0x7fcef01032a0 2024-03-24T10:42:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.505+0000 7fceeeffd640 1 -- 172.21.15.23:0/1695617964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c16916d0 msgr2=0x5651c1693ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.505+0000 7fceeeffd640 1 --2- 172.21.15.23:0/1695617964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c16916d0 0x5651c1693ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.505+0000 7fceeeffd640 1 -- 172.21.15.23:0/1695617964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16881d0 msgr2=0x5651c16885b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.505+0000 7fceeeffd640 1 --2- 172.21.15.23:0/1695617964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16881d0 0x5651c16885b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.505+0000 7fceeeffd640 1 -- 172.21.15.23:0/1695617964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651c15b0ea0 con 0x5651c1688af0 2024-03-24T10:42:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.505+0000 7fceef7fe640 1 --2- 172.21.15.23:0/1793389276 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fced80a2230 0x7fced80a46f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcee0005ab0 tx=0x7fcee0005a20 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.505+0000 7fceeeffd640 1 --2- 172.21.15.23:0/1695617964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c1688af0 0x5651c1691190 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fcee40cfaa0 tx=0x7fcee40139e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee017d158faeeb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.505+0000 7fced5ffb640 1 -- 172.21.15.23:0/1695617964 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee40a4440 con 0x5651c1688af0 2024-03-24T10:42:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.505+0000 7fced5ffb640 1 -- 172.21.15.23:0/1695617964 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcee40a3e40 con 0x5651c1688af0 2024-03-24T10:42:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.505+0000 7fced5ffb640 1 -- 172.21.15.23:0/1695617964 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee40a4620 con 0x5651c1688af0 2024-03-24T10:42:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.506+0000 7fcef8999740 1 -- 172.21.15.23:0/1695617964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c1688af0 msgr2=0x5651c1691190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.506+0000 7fcef8999740 1 --2- 172.21.15.23:0/1695617964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c1688af0 0x5651c1691190 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fcee40cfaa0 tx=0x7fcee40139e0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.506+0000 7fcef8999740 1 -- 172.21.15.23:0/1695617964 shutdown_connections 2024-03-24T10:42:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.506+0000 7fcef8999740 1 --2- 172.21.15.23:0/1695617964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c16916d0 0x5651c1693ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.506+0000 7fcef8999740 1 --2- 172.21.15.23:0/1695617964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c1688af0 0x5651c1691190 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.506+0000 7fcef8999740 1 --2- 172.21.15.23:0/1695617964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16881d0 0x5651c16885b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.506+0000 7fcef8999740 1 -- 172.21.15.23:0/1695617964 >> 172.21.15.23:0/1695617964 conn(0x5651c1682900 msgr2=0x5651c1682ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.506+0000 7fcef8999740 1 -- 172.21.15.23:0/1695617964 shutdown_connections 2024-03-24T10:42:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.506+0000 7fcef8999740 1 -- 172.21.15.23:0/1695617964 wait complete. 2024-03-24T10:42:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.506+0000 7fcef8999740 1 -- start start 2024-03-24T10:42:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.506+0000 7fcef8999740 1 Processor -- start 2024-03-24T10:42:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fcef8999740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c16881d0 0x5651c168e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fcef8999740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c1688af0 0x5651c168ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fcef8999740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16916d0 0x5651c168f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fcef8999740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5651c15ef520 con 0x5651c16881d0 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fcef8999740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5651c12c2430 con 0x5651c16916d0 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fcef8999740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5651c127ee00 con 0x5651c1688af0 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fcef8999740 10 client.0 _ll_register_callbacks cb 0x5651c158dd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fceeeffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c1688af0 0x5651c168ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fceeffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16916d0 0x5651c168f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fceef7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c16881d0 0x5651c168e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fceeeffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c1688af0 0x5651c168ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39720/0 (socket says 172.21.15.23:39720) 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fceeeffd640 1 -- 172.21.15.23:0/1591479278 learned_addr learned my addr 172.21.15.23:0/1591479278 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fceeffff640 1 -- 172.21.15.23:0/1591479278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c1688af0 msgr2=0x5651c168ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fceeffff640 1 --2- 172.21.15.23:0/1591479278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c1688af0 0x5651c168ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fceeffff640 1 -- 172.21.15.23:0/1591479278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c16881d0 msgr2=0x5651c168e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.507+0000 7fceeffff640 1 --2- 172.21.15.23:0/1591479278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c16881d0 0x5651c168e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.508+0000 7fceeffff640 1 -- 172.21.15.23:0/1591479278 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef01006e0 con 0x5651c16916d0 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.508+0000 7fceeeffd640 1 --2- 172.21.15.23:0/1591479278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c1688af0 0x5651c168ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.508+0000 7fceef7fe640 1 --2- 172.21.15.23:0/1591479278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c16881d0 0x5651c168e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:42:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.508+0000 7fceeffff640 1 --2- 172.21.15.23:0/1591479278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16916d0 0x5651c168f190 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x5651c1683a90 tx=0x7fcee8003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd7093147dd164a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.508+0000 7fceceffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:42:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.508+0000 7fceceffd640 1 -- 172.21.15.23:0/1591479278 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee8004b20 con 0x5651c16916d0 2024-03-24T10:42:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.508+0000 7fceceffd640 1 -- 172.21.15.23:0/1591479278 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fcee8004c80 con 0x5651c16916d0 2024-03-24T10:42:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.508+0000 7fcef8999740 1 -- 172.21.15.23:0/1591479278 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5651c12ee7e0 con 0x5651c16916d0 2024-03-24T10:42:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.508+0000 7fceceffd640 1 -- 172.21.15.23:0/1591479278 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcee8004e40 con 0x5651c16916d0 2024-03-24T10:42:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.508+0000 7fcef8999740 10 client.26288 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.508+0000 7fcef8999740 1 -- 172.21.15.23:0/1591479278 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5651c15edba0 con 0x5651c16916d0 2024-03-24T10:42:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.508+0000 7fceceffd640 1 -- 172.21.15.23:0/1591479278 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5651c12ee7e0 con 0x5651c16916d0 2024-03-24T10:42:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.509+0000 7fceceffd640 1 -- 172.21.15.23:0/1591479278 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fcee8005630 con 0x5651c16916d0 2024-03-24T10:42:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.509+0000 7fcef8999740 10 client.26288 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.509+0000 7fcef8999740 10 client.26288 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.509+0000 7fcef8999740 20 client.26288 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.509+0000 7fcef8999740 1 --2- 172.21.15.23:0/1591479278 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5651c16a1cc0 0x5651c16a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.509+0000 7fcef8999740 4 client.26288 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:42:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.509+0000 7fcef8999740 1 -- 172.21.15.23:0/1591479278 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651c132b720 con 0x5651c16a1cc0 2024-03-24T10:42:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.509+0000 7fceef7fe640 1 --2- 172.21.15.23:0/1591479278 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5651c16a1cc0 0x5651c16a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.509+0000 7fceef7fe640 1 --2- 172.21.15.23:0/1591479278 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5651c16a1cc0 0x5651c16a40e0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d55ea2e31ab8a9f8 server_cookie=87258fb24965f59f in_seq=0 out_seq=0 2024-03-24T10:42:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.509+0000 7fceceffd640 10 client.26288 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.510+0000 7fceceffd640 1 -- 172.21.15.23:0/1591479278 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5651c132b720 con 0x5651c16a1cc0 2024-03-24T10:42:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.510+0000 7fceceffd640 10 client.26288 handle_command_reply: tid=0 2024-03-24T10:42:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.510+0000 7fcef8999740 1 client.26288 shutdown 2024-03-24T10:42:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.511+0000 7fcef8999740 1 -- 172.21.15.23:0/1591479278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16916d0 msgr2=0x5651c168f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.511+0000 7fcef8999740 1 --2- 172.21.15.23:0/1591479278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16916d0 0x5651c168f190 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x5651c1683a90 tx=0x7fcee8003f90 comp rx=0 tx=0).stop 2024-03-24T10:42:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.511+0000 7fcef8999740 1 -- 172.21.15.23:0/1591479278 shutdown_connections 2024-03-24T10:42:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.511+0000 7fcef8999740 1 --2- 172.21.15.23:0/1591479278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16916d0 0x5651c168f190 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.511+0000 7fcef8999740 1 --2- 172.21.15.23:0/1591479278 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5651c16a1cc0 0x5651c16a40e0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.511+0000 7fcef8999740 1 --2- 172.21.15.23:0/1591479278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c1688af0 0x5651c168ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.511+0000 7fcef8999740 1 --2- 172.21.15.23:0/1591479278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c16881d0 0x5651c168e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.511+0000 7fceceffd640 0 client.26288 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.511+0000 7fcef8999740 1 -- 172.21.15.23:0/1591479278 >> 172.21.15.23:0/1591479278 conn(0x5651c1682900 msgr2=0x5651c1681b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.511+0000 7fcef8999740 1 -- 172.21.15.23:0/1591479278 shutdown_connections 2024-03-24T10:42:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.511+0000 7fcef8999740 1 -- 172.21.15.23:0/1591479278 wait complete. 2024-03-24T10:42:56.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.511+0000 7fcef8999740 20 client.26288 trim_cache size 0 max 16384 2024-03-24T10:42:56.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.530+0000 7fcef8999740 1 Processor -- start 2024-03-24T10:42:56.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.530+0000 7fcef8999740 1 -- start start 2024-03-24T10:42:56.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fcef8999740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c16881d0 0x5651c1690e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fcef8999740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c1688af0 0x5651c169cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fcef8999740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16916d0 0x5651c16a9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fcef8999740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5651c1574430 con 0x5651c16881d0 2024-03-24T10:42:56.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fcef8999740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5651c169e600 con 0x5651c16916d0 2024-03-24T10:42:56.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fcef8999740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5651c1689570 con 0x5651c1688af0 2024-03-24T10:42:56.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fceef7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c16881d0 0x5651c1690e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fceef7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c16881d0 0x5651c1690e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58732/0 (socket says 172.21.15.23:58732) 2024-03-24T10:42:56.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fceef7fe640 1 -- 172.21.15.23:0/2284612590 learned_addr learned my addr 172.21.15.23:0/2284612590 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:56.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fceeffff640 1 --2- 172.21.15.23:0/2284612590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16916d0 0x5651c16a9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fceeeffd640 1 --2- 172.21.15.23:0/2284612590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c1688af0 0x5651c169cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fceef7fe640 1 -- 172.21.15.23:0/2284612590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c1688af0 msgr2=0x5651c169cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fceef7fe640 1 --2- 172.21.15.23:0/2284612590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c1688af0 0x5651c169cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fceef7fe640 1 -- 172.21.15.23:0/2284612590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16916d0 msgr2=0x5651c16a9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fceef7fe640 1 --2- 172.21.15.23:0/2284612590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16916d0 0x5651c16a9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fceef7fe640 1 -- 172.21.15.23:0/2284612590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651c1589b20 con 0x5651c16881d0 2024-03-24T10:42:56.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fceeffff640 1 --2- 172.21.15.23:0/2284612590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16916d0 0x5651c16a9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:42:56.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fceef7fe640 1 --2- 172.21.15.23:0/2284612590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c16881d0 0x5651c1690e40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fcee0000c00 tx=0x7fcee003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d0f6043a4bd652 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:56.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fced57fa640 1 -- 172.21.15.23:0/2284612590 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee002da10 con 0x5651c16881d0 2024-03-24T10:42:56.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fced57fa640 1 -- 172.21.15.23:0/2284612590 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcee0051070 con 0x5651c16881d0 2024-03-24T10:42:56.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.531+0000 7fced57fa640 1 -- 172.21.15.23:0/2284612590 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee0003850 con 0x5651c16881d0 2024-03-24T10:42:56.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.532+0000 7fcef8999740 1 -- 172.21.15.23:0/2284612590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c16881d0 msgr2=0x5651c1690e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.532+0000 7fcef8999740 1 --2- 172.21.15.23:0/2284612590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c16881d0 0x5651c1690e40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fcee0000c00 tx=0x7fcee003f000 comp rx=0 tx=0).stop 2024-03-24T10:42:56.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.532+0000 7fcef8999740 1 -- 172.21.15.23:0/2284612590 shutdown_connections 2024-03-24T10:42:56.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.532+0000 7fcef8999740 1 --2- 172.21.15.23:0/2284612590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16916d0 0x5651c16a9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.532+0000 7fcef8999740 1 --2- 172.21.15.23:0/2284612590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c1688af0 0x5651c169cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.532+0000 7fcef8999740 1 --2- 172.21.15.23:0/2284612590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c16881d0 0x5651c1690e40 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.532+0000 7fcef8999740 1 -- 172.21.15.23:0/2284612590 >> 172.21.15.23:0/2284612590 conn(0x5651c1682900 msgr2=0x5651c1682ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:56.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.532+0000 7fcef8999740 1 -- 172.21.15.23:0/2284612590 shutdown_connections 2024-03-24T10:42:56.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.532+0000 7fcef8999740 1 -- 172.21.15.23:0/2284612590 wait complete. 2024-03-24T10:42:56.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.532+0000 7fcef8999740 1 -- start start 2024-03-24T10:42:56.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fcef8999740 1 Processor -- start 2024-03-24T10:42:56.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fcef8999740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16881d0 0x5651c1694e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fcef8999740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c1688af0 0x5651c16953c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fcef8999740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c16916d0 0x5651c1695900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fcef8999740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5651c1706fc0 con 0x5651c1688af0 2024-03-24T10:42:56.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fcef8999740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5651c1707100 con 0x5651c16881d0 2024-03-24T10:42:56.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fcef8999740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5651c1707240 con 0x5651c16916d0 2024-03-24T10:42:56.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fcef8999740 10 client.0 _ll_register_callbacks cb 0x7fcee8003a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:42:56.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fceeeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c1688af0 0x5651c16953c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fceef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16881d0 0x5651c1694e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fceeeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c1688af0 0x5651c16953c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58736/0 (socket says 172.21.15.23:58736) 2024-03-24T10:42:56.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fceef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16881d0 0x5651c1694e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:44232/0 (socket says 172.21.15.23:44232) 2024-03-24T10:42:56.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fceeeffd640 1 -- 172.21.15.23:0/1796817871 learned_addr learned my addr 172.21.15.23:0/1796817871 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:42:56.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fceeffff640 1 --2- 172.21.15.23:0/1796817871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c16916d0 0x5651c1695900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fceeeffd640 1 -- 172.21.15.23:0/1796817871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c16916d0 msgr2=0x5651c1695900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fceeeffd640 1 --2- 172.21.15.23:0/1796817871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c16916d0 0x5651c1695900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fceeeffd640 1 -- 172.21.15.23:0/1796817871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16881d0 msgr2=0x5651c1694e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:42:56.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fceeeffd640 1 --2- 172.21.15.23:0/1796817871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16881d0 0x5651c1694e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.533+0000 7fceeeffd640 1 -- 172.21.15.23:0/1796817871 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef010ee40 con 0x5651c1688af0 2024-03-24T10:42:56.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fceeeffd640 1 --2- 172.21.15.23:0/1796817871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c1688af0 0x5651c16953c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fcee4013880 tx=0x7fcee409e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7200f07b6fb3cc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:42:56.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fceceffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:42:56.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fceceffd640 1 -- 172.21.15.23:0/1796817871 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee40a7570 con 0x5651c1688af0 2024-03-24T10:42:56.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fceceffd640 1 -- 172.21.15.23:0/1796817871 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fcee40a3be0 con 0x5651c1688af0 2024-03-24T10:42:56.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fcef8999740 1 -- 172.21.15.23:0/1796817871 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcee40a3e40 con 0x5651c1688af0 2024-03-24T10:42:56.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fceceffd640 1 -- 172.21.15.23:0/1796817871 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcee40bd300 con 0x5651c1688af0 2024-03-24T10:42:56.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fcef8999740 10 client.16416 fetch_fsmap learned FSMap version 25 2024-03-24T10:42:56.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fcef8999740 1 -- 172.21.15.23:0/1796817871 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5651c15af700 con 0x5651c1688af0 2024-03-24T10:42:56.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fceceffd640 1 -- 172.21.15.23:0/1796817871 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcee40a3e40 con 0x5651c1688af0 2024-03-24T10:42:56.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fceceffd640 1 -- 172.21.15.23:0/1796817871 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fcee40bd020 con 0x5651c1688af0 2024-03-24T10:42:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fcef8999740 10 client.16416 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:42:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fcef8999740 10 client.16416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:42:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fcef8999740 20 client.16416 populate_metadata read hostname 'smithi023' 2024-03-24T10:42:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fcef8999740 1 --2- 172.21.15.23:0/1796817871 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5651c17087a0 0x5651c170abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:42:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fcef8999740 4 client.16416 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:42:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.534+0000 7fcef8999740 1 -- 172.21.15.23:0/1796817871 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5651c137a890 con 0x5651c17087a0 2024-03-24T10:42:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.535+0000 7fceef7fe640 1 --2- 172.21.15.23:0/1796817871 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5651c17087a0 0x5651c170abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:42:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.535+0000 7fceef7fe640 1 --2- 172.21.15.23:0/1796817871 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5651c17087a0 0x5651c170abc0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adce5b42d65c69d7 server_cookie=3c415cd3dd661263 in_seq=0 out_seq=0 2024-03-24T10:42:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.535+0000 7fceceffd640 10 client.16416 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.535+0000 7fceceffd640 1 -- 172.21.15.23:0/1796817871 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5651c137a890 con 0x5651c17087a0 2024-03-24T10:42:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.535+0000 7fceceffd640 10 client.16416 handle_command_reply: tid=0 2024-03-24T10:42:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.535+0000 7fcef8999740 1 client.16416 shutdown 2024-03-24T10:42:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.535+0000 7fcef8999740 1 -- 172.21.15.23:0/1796817871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c1688af0 msgr2=0x5651c16953c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.535+0000 7fcef8999740 1 --2- 172.21.15.23:0/1796817871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c1688af0 0x5651c16953c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fcee4013880 tx=0x7fcee409e7e0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.535+0000 7fcef8999740 1 -- 172.21.15.23:0/1796817871 shutdown_connections 2024-03-24T10:42:56.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.536+0000 7fcef8999740 1 --2- 172.21.15.23:0/1796817871 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5651c17087a0 0x5651c170abc0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.536+0000 7fcef8999740 1 --2- 172.21.15.23:0/1796817871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651c16916d0 0x5651c1695900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.536+0000 7fcef8999740 1 --2- 172.21.15.23:0/1796817871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5651c1688af0 0x5651c16953c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.536+0000 7fceceffd640 0 client.16416 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:42:56.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.536+0000 7fcef8999740 1 --2- 172.21.15.23:0/1796817871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651c16881d0 0x5651c1694e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.536+0000 7fcef8999740 1 -- 172.21.15.23:0/1796817871 >> 172.21.15.23:0/1796817871 conn(0x5651c1682900 msgr2=0x5651c1683d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:56.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.536+0000 7fcef8999740 1 -- 172.21.15.23:0/1796817871 shutdown_connections 2024-03-24T10:42:56.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.536+0000 7fcef8999740 1 -- 172.21.15.23:0/1796817871 wait complete. 2024-03-24T10:42:56.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.536+0000 7fcef8999740 20 client.16416 trim_cache size 0 max 16384 2024-03-24T10:42:56.542 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:42:56.543 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:42:56.543 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:42:56.543 INFO:teuthology.orchestra.run.smithi023.stdout: "8277813c-3dba-4650-82a1-91d44675e599": { 2024-03-24T10:42:56.543 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:42:56.543 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "8277813c-3dba-4650-82a1-91d44675e599", 2024-03-24T10:42:56.543 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:42:56.543 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:42:56.543 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:42:56.543 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:42:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.536+0000 7fcef5cdf640 1 -- 172.21.15.23:0/1793389276 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fced80a2230 msgr2=0x7fced80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.536+0000 7fcef5cdf640 1 --2- 172.21.15.23:0/1793389276 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fced80a2230 0x7fced80a46f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcee0005ab0 tx=0x7fcee0005a20 comp rx=0 tx=0).stop 2024-03-24T10:42:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.536+0000 7fcef5cdf640 1 -- 172.21.15.23:0/1793389276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcef01032a0 msgr2=0x7fcef0198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:42:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.536+0000 7fcef5cdf640 1 --2- 172.21.15.23:0/1793389276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcef01032a0 0x7fcef0198390 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcee4009f40 tx=0x7fcee400ee80 comp rx=0 tx=0).stop 2024-03-24T10:42:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.537+0000 7fcef5cdf640 1 -- 172.21.15.23:0/1793389276 shutdown_connections 2024-03-24T10:42:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.537+0000 7fcef5cdf640 1 --2- 172.21.15.23:0/1793389276 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fced80a2230 0x7fced80a46f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.537+0000 7fcef5cdf640 1 --2- 172.21.15.23:0/1793389276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcef010bd50 0x7fcef019c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.537+0000 7fcef5cdf640 1 --2- 172.21.15.23:0/1793389276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcef01032a0 0x7fcef0198390 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.537+0000 7fcef5cdf640 1 --2- 172.21.15.23:0/1793389276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcef0102980 0x7fcef0197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:42:56.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.537+0000 7fcef5cdf640 1 -- 172.21.15.23:0/1793389276 >> 172.21.15.23:0/1793389276 conn(0x7fcef0077770 msgr2=0x7fcef00fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:42:56.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.537+0000 7fcef5cdf640 1 -- 172.21.15.23:0/1793389276 shutdown_connections 2024-03-24T10:42:56.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:42:56.537+0000 7fcef5cdf640 1 -- 172.21.15.23:0/1793389276 wait complete. 2024-03-24T10:42:56.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:56 smithi023 ceph-mon[32466]: pgmap v88: 97 pgs: 97 active+clean; 96 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 212 op/s 2024-03-24T10:42:56.549 INFO:tasks.cephfs.filesystem:scrub status for tag:8277813c-3dba-4650-82a1-91d44675e599 - {'path': '/', 'tag': '8277813c-3dba-4650-82a1-91d44675e599', 'options': 'recursive,force'} 2024-03-24T10:42:56.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:56 smithi117 ceph-mon[36368]: pgmap v88: 97 pgs: 97 active+clean; 96 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 212 op/s 2024-03-24T10:42:56.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:56 smithi116 ceph-mon[36248]: pgmap v88: 97 pgs: 97 active+clean; 96 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 212 op/s 2024-03-24T10:42:57.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 170595 12.68 MB/sec warmup 74 sec latency 28.184 ms 2024-03-24T10:42:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:57 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:42:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:57 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:57 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:42:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:57 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:42:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:57 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:57 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:57.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:57 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:42:57.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:57 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:57.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:57 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:42:57.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:57 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:42:57.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:57 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:57.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:57 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:57.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:57 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:42:57.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:57 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:57.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:57 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:42:57.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:57 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:42:57.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:57 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:57.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:57 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:42:58.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 172993 12.66 MB/sec warmup 75 sec latency 15.763 ms 2024-03-24T10:42:58.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:42:58 smithi117 ceph-mon[36368]: pgmap v89: 97 pgs: 97 active+clean; 100 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 320 op/s 2024-03-24T10:42:58.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:42:58 smithi116 ceph-mon[36248]: pgmap v89: 97 pgs: 97 active+clean; 100 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 320 op/s 2024-03-24T10:42:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:42:58 smithi023 ceph-mon[32466]: pgmap v89: 97 pgs: 97 active+clean; 100 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 320 op/s 2024-03-24T10:42:59.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 175043 12.63 MB/sec warmup 76 sec latency 21.330 ms 2024-03-24T10:43:00.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 177782 12.69 MB/sec warmup 77 sec latency 15.723 ms 2024-03-24T10:43:00.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:00 smithi117 ceph-mon[36368]: pgmap v90: 97 pgs: 97 active+clean; 100 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-03-24T10:43:00.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:00 smithi116 ceph-mon[36248]: pgmap v90: 97 pgs: 97 active+clean; 100 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-03-24T10:43:00.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:00 smithi023 ceph-mon[32466]: pgmap v90: 97 pgs: 97 active+clean; 100 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-03-24T10:43:01.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 178587 12.55 MB/sec warmup 78 sec latency 68.661 ms 2024-03-24T10:43:02.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 181286 12.61 MB/sec warmup 79 sec latency 19.852 ms 2024-03-24T10:43:02.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:02 smithi117 ceph-mon[36368]: pgmap v91: 97 pgs: 97 active+clean; 105 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 345 op/s 2024-03-24T10:43:02.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:02 smithi116 ceph-mon[36248]: pgmap v91: 97 pgs: 97 active+clean; 105 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 345 op/s 2024-03-24T10:43:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:02 smithi023 ceph-mon[32466]: pgmap v91: 97 pgs: 97 active+clean; 105 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 345 op/s 2024-03-24T10:43:03.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 183676 12.59 MB/sec warmup 80 sec latency 20.056 ms 2024-03-24T10:43:04.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 185514 12.56 MB/sec warmup 81 sec latency 21.621 ms 2024-03-24T10:43:04.720 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:04 smithi023 ceph-mon[32466]: pgmap v92: 97 pgs: 97 active+clean; 105 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 243 op/s 2024-03-24T10:43:04.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:04 smithi117 ceph-mon[36368]: pgmap v92: 97 pgs: 97 active+clean; 105 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 243 op/s 2024-03-24T10:43:04.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:04 smithi116 ceph-mon[36248]: pgmap v92: 97 pgs: 97 active+clean; 105 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 243 op/s 2024-03-24T10:43:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.791+0000 7f044b61e640 1 -- 172.21.15.23:0/2442817165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f044406a8d0 msgr2=0x7f044410e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.791+0000 7f044b61e640 1 --2- 172.21.15.23:0/2442817165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f044406a8d0 0x7f044410e5a0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f043c009cd0 tx=0x7f043c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.791+0000 7f044b61e640 1 -- 172.21.15.23:0/2442817165 shutdown_connections 2024-03-24T10:43:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.791+0000 7f044b61e640 1 --2- 172.21.15.23:0/2442817165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f044406a8d0 0x7f044410e5a0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.791+0000 7f044b61e640 1 --2- 172.21.15.23:0/2442817165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0444069f10 0x7f044406a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.791+0000 7f044b61e640 1 --2- 172.21.15.23:0/2442817165 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0444107790 0x7f0444107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.791+0000 7f044b61e640 1 -- 172.21.15.23:0/2442817165 >> 172.21.15.23:0/2442817165 conn(0x7f04440773e0 msgr2=0x7f04440777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.791+0000 7f044b61e640 1 -- 172.21.15.23:0/2442817165 shutdown_connections 2024-03-24T10:43:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.791+0000 7f044b61e640 1 -- 172.21.15.23:0/2442817165 wait complete. 2024-03-24T10:43:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f044b61e640 1 Processor -- start 2024-03-24T10:43:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f044b61e640 1 -- start start 2024-03-24T10:43:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f044b61e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0444069f10 0x7f0444199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f044b61e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f044406a8d0 0x7f044419a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f044b61e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0444107790 0x7f044419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f044b61e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f044419ecb0 con 0x7f044406a8d0 2024-03-24T10:43:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f044b61e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f044419ee20 con 0x7f0444069f10 2024-03-24T10:43:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f044b61e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f044419efe0 con 0x7f0444107790 2024-03-24T10:43:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f0448bb9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f044406a8d0 0x7f044419a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f0448bb9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f044406a8d0 0x7f044419a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37952/0 (socket says 172.21.15.23:37952) 2024-03-24T10:43:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f0448bb9640 1 -- 172.21.15.23:0/3247364646 learned_addr learned my addr 172.21.15.23:0/3247364646 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f04493ba640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0444069f10 0x7f0444199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f0449bbb640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0444107790 0x7f044419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f0448bb9640 1 -- 172.21.15.23:0/3247364646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0444107790 msgr2=0x7f044419e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f0448bb9640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0444107790 0x7f044419e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f0448bb9640 1 -- 172.21.15.23:0/3247364646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0444069f10 msgr2=0x7f0444199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f0448bb9640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0444069f10 0x7f0444199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f0448bb9640 1 -- 172.21.15.23:0/3247364646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f043c009900 con 0x7f044406a8d0 2024-03-24T10:43:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f04493ba640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0444069f10 0x7f0444199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.792+0000 7f0448bb9640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f044406a8d0 0x7f044419a330 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f043400e9f0 tx=0x7f043400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.793+0000 7f043a7fc640 1 -- 172.21.15.23:0/3247364646 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0434009700 con 0x7f044406a8d0 2024-03-24T10:43:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.793+0000 7f043a7fc640 1 -- 172.21.15.23:0/3247364646 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f043400cd80 con 0x7f044406a8d0 2024-03-24T10:43:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.793+0000 7f043a7fc640 1 -- 172.21.15.23:0/3247364646 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0434018920 con 0x7f044406a8d0 2024-03-24T10:43:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.793+0000 7f044b61e640 1 -- 172.21.15.23:0/3247364646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f044419f2c0 con 0x7f044406a8d0 2024-03-24T10:43:04.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.793+0000 7f044b61e640 1 -- 172.21.15.23:0/3247364646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04441a6b40 con 0x7f044406a8d0 2024-03-24T10:43:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.795+0000 7f043a7fc640 1 -- 172.21.15.23:0/3247364646 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f0434018a80 con 0x7f044406a8d0 2024-03-24T10:43:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.795+0000 7f044e1fd740 1 Processor -- start 2024-03-24T10:43:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f044e1fd740 1 -- start start 2024-03-24T10:43:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f044e1fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d3a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f043a7fc640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f04300a2350 0x7f04300a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f044e1fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3af00 0x55bc71d43790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f044e1fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d43cd0 0x55bc71d460c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f044e1fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bc71cf51b0 con 0x55bc71d43cd0 2024-03-24T10:43:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f044e1fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bc71ca7f80 con 0x55bc71d3af00 2024-03-24T10:43:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f044e1fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bc71ca54b0 con 0x55bc71d3a5e0 2024-03-24T10:43:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f0449bbb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d43cd0 0x55bc71d460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f0448bb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3af00 0x55bc71d43790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f0449bbb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d3a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f04493ba640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f04300a2350 0x7f04300a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f0448bb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3af00 0x55bc71d43790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:34156/0 (socket says 172.21.15.23:34156) 2024-03-24T10:43:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f0448bb9640 1 -- 172.21.15.23:0/2863572196 learned_addr learned my addr 172.21.15.23:0/2863572196 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f043a7fc640 1 -- 172.21.15.23:0/3247364646 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0434014070 con 0x7f044406a8d0 2024-03-24T10:43:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f0449bbb640 1 -- 172.21.15.23:0/2863572196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3af00 msgr2=0x55bc71d43790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:04.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f0449bbb640 1 --2- 172.21.15.23:0/2863572196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3af00 0x55bc71d43790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f0449bbb640 1 -- 172.21.15.23:0/2863572196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d43cd0 msgr2=0x55bc71d460c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f0449bbb640 1 --2- 172.21.15.23:0/2863572196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d43cd0 0x55bc71d460c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f0449bbb640 1 -- 172.21.15.23:0/2863572196 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc71c36810 con 0x55bc71d3a5e0 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.796+0000 7f0448bb9640 1 --2- 172.21.15.23:0/2863572196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3af00 0x55bc71d43790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.797+0000 7f04493ba640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f04300a2350 0x7f04300a4810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0440009fc0 tx=0x7f0440005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.797+0000 7f0449bbb640 1 --2- 172.21.15.23:0/2863572196 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d3a9c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f043c037040 tx=0x7f043c005d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a41ee94d4279fd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.797+0000 7f042f7fe640 1 -- 172.21.15.23:0/2863572196 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c034640 con 0x55bc71d3a5e0 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.797+0000 7f042f7fe640 1 -- 172.21.15.23:0/2863572196 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f043c0347a0 con 0x55bc71d3a5e0 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.797+0000 7f042f7fe640 1 -- 172.21.15.23:0/2863572196 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c003b70 con 0x55bc71d3a5e0 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.797+0000 7f044e1fd740 1 -- 172.21.15.23:0/2863572196 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 msgr2=0x55bc71d3a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.797+0000 7f044e1fd740 1 --2- 172.21.15.23:0/2863572196 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d3a9c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f043c037040 tx=0x7f043c005d80 comp rx=0 tx=0).stop 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.798+0000 7f044e1fd740 1 -- 172.21.15.23:0/2863572196 shutdown_connections 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.798+0000 7f044e1fd740 1 --2- 172.21.15.23:0/2863572196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d43cd0 0x55bc71d460c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.798+0000 7f044e1fd740 1 --2- 172.21.15.23:0/2863572196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3af00 0x55bc71d43790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.798+0000 7f044e1fd740 1 --2- 172.21.15.23:0/2863572196 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d3a9c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.798+0000 7f044e1fd740 1 -- 172.21.15.23:0/2863572196 >> 172.21.15.23:0/2863572196 conn(0x55bc71d34f00 msgr2=0x55bc71d352e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.798+0000 7f044e1fd740 1 -- 172.21.15.23:0/2863572196 shutdown_connections 2024-03-24T10:43:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.798+0000 7f044e1fd740 1 -- 172.21.15.23:0/2863572196 wait complete. 2024-03-24T10:43:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.798+0000 7f044e1fd740 1 -- start start 2024-03-24T10:43:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.798+0000 7f044e1fd740 1 Processor -- start 2024-03-24T10:43:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.798+0000 7f044e1fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3a5e0 0x55bc71d40d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f044e1fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d3af00 0x55bc71d412b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f044e1fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d43cd0 0x55bc71d417f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f044e1fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bc71ca0c90 con 0x55bc71d3af00 2024-03-24T10:43:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f044e1fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bc71c45c50 con 0x55bc71d3a5e0 2024-03-24T10:43:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f044e1fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bc71974250 con 0x55bc71d43cd0 2024-03-24T10:43:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f044e1fd740 10 client.0 _ll_register_callbacks cb 0x55bc71c94290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f04493ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3a5e0 0x55bc71d40d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f04493ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3a5e0 0x55bc71d40d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:34166/0 (socket says 172.21.15.23:34166) 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f04493ba640 1 -- 172.21.15.23:0/3120807002 learned_addr learned my addr 172.21.15.23:0/3120807002 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f0448bb9640 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d3af00 0x55bc71d412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f0449bbb640 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d43cd0 0x55bc71d417f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f0449bbb640 1 -- 172.21.15.23:0/3120807002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3a5e0 msgr2=0x55bc71d40d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f0449bbb640 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3a5e0 0x55bc71d40d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f0449bbb640 1 -- 172.21.15.23:0/3120807002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d3af00 msgr2=0x55bc71d412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f0449bbb640 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d3af00 0x55bc71d412b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f0449bbb640 1 -- 172.21.15.23:0/3120807002 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f044410eea0 con 0x55bc71d43cd0 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f0448bb9640 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d3af00 0x55bc71d412b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f04493ba640 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3a5e0 0x55bc71d40d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f0449bbb640 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d43cd0 0x55bc71d417f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f043c02c630 tx=0x7f043c032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7af01b566766f0da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f042cff9640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f042cff9640 1 -- 172.21.15.23:0/3120807002 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c034ea0 con 0x55bc71d43cd0 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f044e1fd740 1 -- 172.21.15.23:0/3120807002 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bc719a0600 con 0x55bc71d43cd0 2024-03-24T10:43:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.799+0000 7f042cff9640 1 -- 172.21.15.23:0/3120807002 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f043c003970 con 0x55bc71d43cd0 2024-03-24T10:43:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.800+0000 7f042cff9640 1 -- 172.21.15.23:0/3120807002 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f043c003b30 con 0x55bc71d43cd0 2024-03-24T10:43:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.800+0000 7f044e1fd740 10 client.26275 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.800+0000 7f044e1fd740 1 -- 172.21.15.23:0/3120807002 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bc71c9f310 con 0x55bc71d43cd0 2024-03-24T10:43:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.800+0000 7f042cff9640 1 -- 172.21.15.23:0/3120807002 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bc719a0600 con 0x55bc71d43cd0 2024-03-24T10:43:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.800+0000 7f042cff9640 1 -- 172.21.15.23:0/3120807002 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f043c04d450 con 0x55bc71d43cd0 2024-03-24T10:43:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.800+0000 7f044e1fd740 10 client.26275 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.800+0000 7f044e1fd740 10 client.26275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.800+0000 7f044e1fd740 20 client.26275 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.800+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc71d53890 0x55bc71d55c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.800+0000 7f044e1fd740 4 client.26275 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.800+0000 7f044e1fd740 1 -- 172.21.15.23:0/3120807002 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc719f11b0 con 0x55bc71d53890 2024-03-24T10:43:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.800+0000 7f04493ba640 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc71d53890 0x55bc71d55c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.801+0000 7f04493ba640 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc71d53890 0x55bc71d55c80 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8be77492619104e6 server_cookie=29a327dc8690093f in_seq=0 out_seq=0 2024-03-24T10:43:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.801+0000 7f042cff9640 10 client.26275 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:04.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f042cff9640 1 -- 172.21.15.23:0/3120807002 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bc719f11b0 con 0x55bc71d53890 2024-03-24T10:43:04.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f042cff9640 10 client.26275 handle_command_reply: tid=0 2024-03-24T10:43:04.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f044e1fd740 1 client.26275 shutdown 2024-03-24T10:43:04.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f044e1fd740 1 -- 172.21.15.23:0/3120807002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d43cd0 msgr2=0x55bc71d417f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:04.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d43cd0 0x55bc71d417f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f043c02c630 tx=0x7f043c032f20 comp rx=0 tx=0).stop 2024-03-24T10:43:04.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f044e1fd740 1 -- 172.21.15.23:0/3120807002 shutdown_connections 2024-03-24T10:43:04.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc71d53890 0x55bc71d55c80 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d43cd0 0x55bc71d417f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f042cff9640 0 client.26275 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:04.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d3af00 0x55bc71d412b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3120807002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3a5e0 0x55bc71d40d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f044e1fd740 1 -- 172.21.15.23:0/3120807002 >> 172.21.15.23:0/3120807002 conn(0x55bc71d34f00 msgr2=0x55bc71d36c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f044e1fd740 1 -- 172.21.15.23:0/3120807002 shutdown_connections 2024-03-24T10:43:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f044e1fd740 1 -- 172.21.15.23:0/3120807002 wait complete. 2024-03-24T10:43:04.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.802+0000 7f044e1fd740 20 client.26275 trim_cache size 0 max 16384 2024-03-24T10:43:04.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.818+0000 7f044e1fd740 1 Processor -- start 2024-03-24T10:43:04.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.818+0000 7f044e1fd740 1 -- start start 2024-03-24T10:43:04.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f044e1fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d4f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f044e1fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3af00 0x55bc71d41610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f044e1fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d43cd0 0x55bc71d5b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f044e1fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bc71d30160 con 0x55bc71d43cd0 2024-03-24T10:43:04.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f044e1fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bc71c26660 con 0x55bc71d3af00 2024-03-24T10:43:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f044e1fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bc71d536e0 con 0x55bc71d3a5e0 2024-03-24T10:43:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f0449bbb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d43cd0 0x55bc71d5b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f0449bbb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d43cd0 0x55bc71d5b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37986/0 (socket says 172.21.15.23:37986) 2024-03-24T10:43:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f0449bbb640 1 -- 172.21.15.23:0/3744170093 learned_addr learned my addr 172.21.15.23:0/3744170093 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f0448bb9640 1 --2- 172.21.15.23:0/3744170093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3af00 0x55bc71d41610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f04493ba640 1 --2- 172.21.15.23:0/3744170093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d4f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f0449bbb640 1 -- 172.21.15.23:0/3744170093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 msgr2=0x55bc71d4f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:04.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f0449bbb640 1 --2- 172.21.15.23:0/3744170093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d4f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f0449bbb640 1 -- 172.21.15.23:0/3744170093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3af00 msgr2=0x55bc71d41610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f0449bbb640 1 --2- 172.21.15.23:0/3744170093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3af00 0x55bc71d41610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f0449bbb640 1 -- 172.21.15.23:0/3744170093 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc71c3b810 con 0x55bc71d43cd0 2024-03-24T10:43:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f0449bbb640 1 --2- 172.21.15.23:0/3744170093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d43cd0 0x55bc71d5b550 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f043c0325c0 tx=0x7f043c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde042de250f065e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f042effd640 1 -- 172.21.15.23:0/3744170093 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c032950 con 0x55bc71d43cd0 2024-03-24T10:43:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f042effd640 1 -- 172.21.15.23:0/3744170093 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f043c032b30 con 0x55bc71d43cd0 2024-03-24T10:43:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.819+0000 7f042effd640 1 -- 172.21.15.23:0/3744170093 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c056690 con 0x55bc71d43cd0 2024-03-24T10:43:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.820+0000 7f044e1fd740 1 -- 172.21.15.23:0/3744170093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d43cd0 msgr2=0x55bc71d5b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.820+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3744170093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d43cd0 0x55bc71d5b550 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f043c0325c0 tx=0x7f043c009b20 comp rx=0 tx=0).stop 2024-03-24T10:43:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.820+0000 7f044e1fd740 1 -- 172.21.15.23:0/3744170093 shutdown_connections 2024-03-24T10:43:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.820+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3744170093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d43cd0 0x55bc71d5b550 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.820+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3744170093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d3af00 0x55bc71d41610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.820+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3744170093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d4f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.820+0000 7f044e1fd740 1 -- 172.21.15.23:0/3744170093 >> 172.21.15.23:0/3744170093 conn(0x55bc71d34f00 msgr2=0x55bc71d33ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.820+0000 7f044e1fd740 1 -- 172.21.15.23:0/3744170093 shutdown_connections 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.820+0000 7f044e1fd740 1 -- 172.21.15.23:0/3744170093 wait complete. 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.821+0000 7f044e1fd740 1 -- start start 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.821+0000 7f044e1fd740 1 Processor -- start 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.821+0000 7f044e1fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d4b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.821+0000 7f044e1fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d3af00 0x55bc71d3d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.821+0000 7f044e1fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d43cd0 0x55bc71d3d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.821+0000 7f044e1fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bc71d3de50 con 0x55bc71d3af00 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.821+0000 7f044e1fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bc71d3dfc0 con 0x55bc71d43cd0 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.821+0000 7f044e1fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bc71d3e180 con 0x55bc71d3a5e0 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.821+0000 7f044e1fd740 10 client.0 _ll_register_callbacks cb 0x7f0440005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.821+0000 7f04493ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d4b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.821+0000 7f04493ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d4b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:45292/0 (socket says 172.21.15.23:45292) 2024-03-24T10:43:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.822+0000 7f04493ba640 1 -- 172.21.15.23:0/3021528082 learned_addr learned my addr 172.21.15.23:0/3021528082 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.822+0000 7f0449bbb640 1 --2- 172.21.15.23:0/3021528082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d43cd0 0x55bc71d3d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.822+0000 7f0448bb9640 1 --2- 172.21.15.23:0/3021528082 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d3af00 0x55bc71d3d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.822+0000 7f04493ba640 1 -- 172.21.15.23:0/3021528082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d43cd0 msgr2=0x55bc71d3d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.822+0000 7f04493ba640 1 --2- 172.21.15.23:0/3021528082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d43cd0 0x55bc71d3d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.822+0000 7f04493ba640 1 -- 172.21.15.23:0/3021528082 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d3af00 msgr2=0x55bc71d3d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.822+0000 7f04493ba640 1 --2- 172.21.15.23:0/3021528082 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d3af00 0x55bc71d3d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.822+0000 7f04493ba640 1 -- 172.21.15.23:0/3021528082 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc71c9f310 con 0x55bc71d3a5e0 2024-03-24T10:43:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.822+0000 7f0448bb9640 1 --2- 172.21.15.23:0/3021528082 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d3af00 0x55bc71d3d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:43:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.822+0000 7f04493ba640 1 --2- 172.21.15.23:0/3021528082 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d4b910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0440011ca0 tx=0x7f0440004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af31963d8439a72c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.822+0000 7f042cff9640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:43:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.822+0000 7f042cff9640 1 -- 172.21.15.23:0/3021528082 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0440022040 con 0x55bc71d3a5e0 2024-03-24T10:43:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.822+0000 7f042cff9640 1 -- 172.21.15.23:0/3021528082 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0440003e40 con 0x55bc71d3a5e0 2024-03-24T10:43:04.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.822+0000 7f044e1fd740 1 -- 172.21.15.23:0/3021528082 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bc71cc5790 con 0x55bc71d3a5e0 2024-03-24T10:43:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.823+0000 7f042cff9640 1 -- 172.21.15.23:0/3021528082 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f044000f680 con 0x55bc71d3a5e0 2024-03-24T10:43:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.823+0000 7f044e1fd740 10 client.26281 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.823+0000 7f044e1fd740 1 -- 172.21.15.23:0/3021528082 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bc71c615a0 con 0x55bc71d3a5e0 2024-03-24T10:43:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.823+0000 7f042cff9640 1 -- 172.21.15.23:0/3021528082 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f044003e020 con 0x55bc71d3a5e0 2024-03-24T10:43:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.823+0000 7f042cff9640 1 -- 172.21.15.23:0/3021528082 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f04400319c0 con 0x55bc71d3a5e0 2024-03-24T10:43:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.823+0000 7f044e1fd740 10 client.26281 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.823+0000 7f044e1fd740 10 client.26281 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.823+0000 7f044e1fd740 20 client.26281 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.823+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3021528082 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc71dba080 0x55bc71dbc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.823+0000 7f044e1fd740 4 client.26281 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:43:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.823+0000 7f044e1fd740 1 -- 172.21.15.23:0/3021528082 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bc719dd550 con 0x55bc71dba080 2024-03-24T10:43:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.823+0000 7f0448bb9640 1 --2- 172.21.15.23:0/3021528082 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc71dba080 0x55bc71dbc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.824+0000 7f042cff9640 10 client.26281 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:04.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.824+0000 7f0448bb9640 1 --2- 172.21.15.23:0/3021528082 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc71dba080 0x55bc71dbc4a0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30880822237c74e8 server_cookie=98e23a1ff525959d in_seq=0 out_seq=0 2024-03-24T10:43:04.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f042cff9640 1 -- 172.21.15.23:0/3021528082 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bc719dd550 con 0x55bc71dba080 2024-03-24T10:43:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f042cff9640 10 client.26281 handle_command_reply: tid=0 2024-03-24T10:43:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f044e1fd740 1 client.26281 shutdown 2024-03-24T10:43:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f044e1fd740 1 -- 172.21.15.23:0/3021528082 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 msgr2=0x55bc71d4b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3021528082 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d4b910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0440011ca0 tx=0x7f0440004730 comp rx=0 tx=0).stop 2024-03-24T10:43:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f044e1fd740 1 -- 172.21.15.23:0/3021528082 shutdown_connections 2024-03-24T10:43:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3021528082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc71d43cd0 0x55bc71d3d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3021528082 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc71d3af00 0x55bc71d3d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3021528082 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc71dba080 0x55bc71dbc4a0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f044e1fd740 1 --2- 172.21.15.23:0/3021528082 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc71d3a5e0 0x55bc71d4b910 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f042cff9640 0 client.26281 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f044e1fd740 1 -- 172.21.15.23:0/3021528082 >> 172.21.15.23:0/3021528082 conn(0x55bc71d34f00 msgr2=0x55bc71d42130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f044e1fd740 1 -- 172.21.15.23:0/3021528082 shutdown_connections 2024-03-24T10:43:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f044e1fd740 1 -- 172.21.15.23:0/3021528082 wait complete. 2024-03-24T10:43:04.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.832+0000 7f044e1fd740 20 client.26281 trim_cache size 0 max 16384 2024-03-24T10:43:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:43:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:43:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:43:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:43:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.833+0000 7f044b61e640 1 -- 172.21.15.23:0/3247364646 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f04300a2350 msgr2=0x7f04300a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.833+0000 7f044b61e640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f04300a2350 0x7f04300a4810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0440009fc0 tx=0x7f0440005f90 comp rx=0 tx=0).stop 2024-03-24T10:43:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.833+0000 7f044b61e640 1 -- 172.21.15.23:0/3247364646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f044406a8d0 msgr2=0x7f044419a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.833+0000 7f044b61e640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f044406a8d0 0x7f044419a330 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f043400e9f0 tx=0x7f043400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.833+0000 7f044b61e640 1 -- 172.21.15.23:0/3247364646 shutdown_connections 2024-03-24T10:43:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.833+0000 7f044b61e640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f04300a2350 0x7f04300a4810 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.833+0000 7f044b61e640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0444107790 0x7f044419e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.833+0000 7f044b61e640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f044406a8d0 0x7f044419a330 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.833+0000 7f044b61e640 1 --2- 172.21.15.23:0/3247364646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0444069f10 0x7f0444199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:04.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.833+0000 7f044b61e640 1 -- 172.21.15.23:0/3247364646 >> 172.21.15.23:0/3247364646 conn(0x7f04440773e0 msgr2=0x7f04440fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:04.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.833+0000 7f044b61e640 1 -- 172.21.15.23:0/3247364646 shutdown_connections 2024-03-24T10:43:04.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:04.833+0000 7f044b61e640 1 -- 172.21.15.23:0/3247364646 wait complete. 2024-03-24T10:43:05.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 188306 12.61 MB/sec warmup 82 sec latency 10.462 ms 2024-03-24T10:43:05.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:05 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:43:05.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:05 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:43:05.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:05 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:43:05.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:05 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:43:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:05 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:43:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:05 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:43:06.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 190342 12.55 MB/sec warmup 83 sec latency 79.208 ms 2024-03-24T10:43:06.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:06 smithi117 ceph-mon[36368]: pgmap v93: 97 pgs: 97 active+clean; 105 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 243 op/s 2024-03-24T10:43:06.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:06 smithi116 ceph-mon[36248]: pgmap v93: 97 pgs: 97 active+clean; 105 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 243 op/s 2024-03-24T10:43:06.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:06 smithi023 ceph-mon[32466]: pgmap v93: 97 pgs: 97 active+clean; 105 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 243 op/s 2024-03-24T10:43:07.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 192223 12.55 MB/sec warmup 84 sec latency 20.223 ms 2024-03-24T10:43:08.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 195167 12.60 MB/sec warmup 85 sec latency 13.559 ms 2024-03-24T10:43:08.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:08 smithi117 ceph-mon[36368]: pgmap v94: 97 pgs: 97 active+clean; 115 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 346 op/s 2024-03-24T10:43:08.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:08 smithi116 ceph-mon[36248]: pgmap v94: 97 pgs: 97 active+clean; 115 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 346 op/s 2024-03-24T10:43:08.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:08 smithi023 ceph-mon[32466]: pgmap v94: 97 pgs: 97 active+clean; 115 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 346 op/s 2024-03-24T10:43:09.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 196705 12.53 MB/sec warmup 86 sec latency 59.991 ms 2024-03-24T10:43:10.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 199179 12.55 MB/sec warmup 87 sec latency 17.337 ms 2024-03-24T10:43:10.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:10 smithi117 ceph-mon[36368]: pgmap v95: 97 pgs: 97 active+clean; 115 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 238 op/s 2024-03-24T10:43:10.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:10 smithi116 ceph-mon[36248]: pgmap v95: 97 pgs: 97 active+clean; 115 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 238 op/s 2024-03-24T10:43:10.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:10 smithi023 ceph-mon[32466]: pgmap v95: 97 pgs: 97 active+clean; 115 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 238 op/s 2024-03-24T10:43:11.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 201558 12.55 MB/sec warmup 88 sec latency 97.128 ms 2024-03-24T10:43:12.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 203443 12.51 MB/sec warmup 89 sec latency 19.510 ms 2024-03-24T10:43:12.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:12 smithi117 ceph-mon[36368]: pgmap v96: 97 pgs: 97 active+clean; 125 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 336 op/s 2024-03-24T10:43:12.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:12 smithi116 ceph-mon[36248]: pgmap v96: 97 pgs: 97 active+clean; 125 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 336 op/s 2024-03-24T10:43:12.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:12 smithi023 ceph-mon[32466]: pgmap v96: 97 pgs: 97 active+clean; 125 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 336 op/s 2024-03-24T10:43:13.114 INFO:tasks.workunit.client.0.smithi023.stdout: 1 206191 12.56 MB/sec warmup 90 sec latency 16.023 ms 2024-03-24T10:43:13.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:13 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:43:13.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:13 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:43:13.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:13 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:43:14.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 208319 12.51 MB/sec warmup 91 sec latency 27.670 ms 2024-03-24T10:43:14.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:14 smithi117 ceph-mon[36368]: pgmap v97: 97 pgs: 97 active+clean; 125 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-24T10:43:14.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:14 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:43:14.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:14 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:43:14.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:14 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:43:14.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:14 smithi116 ceph-mon[36248]: pgmap v97: 97 pgs: 97 active+clean; 125 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-24T10:43:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:14 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:43:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:14 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:43:14.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:14 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:43:14.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:14 smithi023 ceph-mon[32466]: pgmap v97: 97 pgs: 97 active+clean; 125 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-24T10:43:14.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:14 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:43:14.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:14 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:43:14.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:14 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:43:15.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 210268 12.50 MB/sec warmup 92 sec latency 21.428 ms 2024-03-24T10:43:16.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 212828 12.54 MB/sec warmup 93 sec latency 116.043 ms 2024-03-24T10:43:16.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:16 smithi117 ceph-mon[36368]: pgmap v98: 97 pgs: 97 active+clean; 125 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-24T10:43:16.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:16 smithi116 ceph-mon[36248]: pgmap v98: 97 pgs: 97 active+clean; 125 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-24T10:43:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:16 smithi023 ceph-mon[32466]: pgmap v98: 97 pgs: 97 active+clean; 125 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-24T10:43:17.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 214595 12.49 MB/sec warmup 94 sec latency 22.334 ms 2024-03-24T10:43:17.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.926+0000 7f276749d640 1 -- 172.21.15.23:0/1774153592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27601076f0 msgr2=0x7f2760107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.926+0000 7f276749d640 1 --2- 172.21.15.23:0/1774153592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27601076f0 0x7f2760107ad0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f275c009930 tx=0x7f275c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:43:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.926+0000 7f276749d640 1 -- 172.21.15.23:0/1774153592 shutdown_connections 2024-03-24T10:43:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.926+0000 7f276749d640 1 --2- 172.21.15.23:0/1774153592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27601020a0 0x7f276010e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.926+0000 7f276749d640 1 --2- 172.21.15.23:0/1774153592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27601016e0 0x7f2760101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.926+0000 7f276749d640 1 --2- 172.21.15.23:0/1774153592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27601076f0 0x7f2760107ad0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.926+0000 7f276749d640 1 -- 172.21.15.23:0/1774153592 >> 172.21.15.23:0/1774153592 conn(0x7f27600fd1f0 msgr2=0x7f27600ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.926+0000 7f276749d640 1 -- 172.21.15.23:0/1774153592 shutdown_connections 2024-03-24T10:43:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.926+0000 7f276749d640 1 -- 172.21.15.23:0/1774153592 wait complete. 2024-03-24T10:43:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f276749d640 1 Processor -- start 2024-03-24T10:43:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f276749d640 1 -- start start 2024-03-24T10:43:17.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f276749d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27601016e0 0x7f2760197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f276749d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27601020a0 0x7f27601982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f276749d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27601076f0 0x7f276019c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f276749d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f276019cbf0 con 0x7f27601020a0 2024-03-24T10:43:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f276749d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f276019cd60 con 0x7f27601076f0 2024-03-24T10:43:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f276749d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f276019cf20 con 0x7f27601016e0 2024-03-24T10:43:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f2764a38640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27601020a0 0x7f27601982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f2765239640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27601016e0 0x7f2760197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f2764a38640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27601020a0 0x7f27601982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55052/0 (socket says 172.21.15.23:55052) 2024-03-24T10:43:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f2764a38640 1 -- 172.21.15.23:0/2912879238 learned_addr learned my addr 172.21.15.23:0/2912879238 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f2765a3a640 1 --2- 172.21.15.23:0/2912879238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27601076f0 0x7f276019c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f2764a38640 1 -- 172.21.15.23:0/2912879238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27601016e0 msgr2=0x7f2760197d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f2764a38640 1 --2- 172.21.15.23:0/2912879238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27601016e0 0x7f2760197d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f2764a38640 1 -- 172.21.15.23:0/2912879238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27601076f0 msgr2=0x7f276019c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:17.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f2764a38640 1 --2- 172.21.15.23:0/2912879238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27601076f0 0x7f276019c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f2764a38640 1 -- 172.21.15.23:0/2912879238 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275c009560 con 0x7f27601020a0 2024-03-24T10:43:17.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f2765239640 1 --2- 172.21.15.23:0/2912879238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27601016e0 0x7f2760197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:17.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.927+0000 7f2764a38640 1 --2- 172.21.15.23:0/2912879238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27601020a0 0x7f27601982c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f275000e9f0 tx=0x7f275000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:17.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.928+0000 7f27567fc640 1 -- 172.21.15.23:0/2912879238 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2750009720 con 0x7f27601020a0 2024-03-24T10:43:17.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.928+0000 7f27567fc640 1 -- 172.21.15.23:0/2912879238 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f275000cd80 con 0x7f27601020a0 2024-03-24T10:43:17.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.928+0000 7f27567fc640 1 -- 172.21.15.23:0/2912879238 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2750010690 con 0x7f27601020a0 2024-03-24T10:43:17.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.928+0000 7f276749d640 1 -- 172.21.15.23:0/2912879238 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f276019d200 con 0x7f27601020a0 2024-03-24T10:43:17.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.928+0000 7f276749d640 1 -- 172.21.15.23:0/2912879238 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27601a4a80 con 0x7f27601020a0 2024-03-24T10:43:17.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.930+0000 7f27567fc640 1 -- 172.21.15.23:0/2912879238 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f27500046c0 con 0x7f27601020a0 2024-03-24T10:43:17.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.930+0000 7f276a07e740 1 Processor -- start 2024-03-24T10:43:17.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.930+0000 7f276a07e740 1 -- start start 2024-03-24T10:43:17.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f27567fc640 1 --2- 172.21.15.23:0/2912879238 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f274c0a22d0 0x7f274c0a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f276a07e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x55630907f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f276a07e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309088790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2765a3a640 1 --2- 172.21.15.23:0/2912879238 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f274c0a22d0 0x7f274c0a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f276a07e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556309088cd0 0x55630908b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f276a07e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55630903a1b0 con 0x55630907ff00 2024-03-24T10:43:17.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f276a07e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556308fecf80 con 0x55630907f5e0 2024-03-24T10:43:17.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f276a07e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556308fea4b0 con 0x556309088cd0 2024-03-24T10:43:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f27567fc640 1 -- 172.21.15.23:0/2912879238 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f2750014070 con 0x7f27601020a0 2024-03-24T10:43:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2765239640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x55630907f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2764a38640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309088790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2765239640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x55630907f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:54582/0 (socket says 172.21.15.23:54582) 2024-03-24T10:43:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2765239640 1 -- 172.21.15.23:0/4192656535 learned_addr learned my addr 172.21.15.23:0/4192656535 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2764a38640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309088790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55068/0 (socket says 172.21.15.23:55068) 2024-03-24T10:43:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2765a3a640 1 --2- 172.21.15.23:0/4192656535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556309088cd0 0x55630908b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2765239640 1 -- 172.21.15.23:0/4192656535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556309088cd0 msgr2=0x55630908b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2765239640 1 --2- 172.21.15.23:0/4192656535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556309088cd0 0x55630908b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2765239640 1 -- 172.21.15.23:0/4192656535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 msgr2=0x556309088790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2765239640 1 --2- 172.21.15.23:0/4192656535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309088790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2765239640 1 -- 172.21.15.23:0/4192656535 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556308f7b810 con 0x55630907f5e0 2024-03-24T10:43:17.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2764a38640 1 --2- 172.21.15.23:0/4192656535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309088790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2765a3a640 1 --2- 172.21.15.23:0/2912879238 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f274c0a22d0 0x7f274c0a4790 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2758005fd0 tx=0x7f2758011b40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.931+0000 7f2765239640 1 --2- 172.21.15.23:0/4192656535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x55630907f9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f275c036040 tx=0x7f275c02c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cc30e8900d3a3f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.932+0000 7f274b7fe640 1 -- 172.21.15.23:0/4192656535 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c004af0 con 0x55630907f5e0 2024-03-24T10:43:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.932+0000 7f274b7fe640 1 -- 172.21.15.23:0/4192656535 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f275c004030 con 0x55630907f5e0 2024-03-24T10:43:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.932+0000 7f274b7fe640 1 -- 172.21.15.23:0/4192656535 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c0053e0 con 0x55630907f5e0 2024-03-24T10:43:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.933+0000 7f276a07e740 1 -- 172.21.15.23:0/4192656535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 msgr2=0x55630907f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.933+0000 7f276a07e740 1 --2- 172.21.15.23:0/4192656535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x55630907f9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f275c036040 tx=0x7f275c02c660 comp rx=0 tx=0).stop 2024-03-24T10:43:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.933+0000 7f276a07e740 1 -- 172.21.15.23:0/4192656535 shutdown_connections 2024-03-24T10:43:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.933+0000 7f276a07e740 1 --2- 172.21.15.23:0/4192656535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556309088cd0 0x55630908b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.933+0000 7f276a07e740 1 --2- 172.21.15.23:0/4192656535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309088790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.933+0000 7f276a07e740 1 --2- 172.21.15.23:0/4192656535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x55630907f9c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.933+0000 7f276a07e740 1 -- 172.21.15.23:0/4192656535 >> 172.21.15.23:0/4192656535 conn(0x556309079f00 msgr2=0x55630907a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:17.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.933+0000 7f276a07e740 1 -- 172.21.15.23:0/4192656535 shutdown_connections 2024-03-24T10:43:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.933+0000 7f276a07e740 1 -- 172.21.15.23:0/4192656535 wait complete. 2024-03-24T10:43:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.933+0000 7f276a07e740 1 -- start start 2024-03-24T10:43:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.933+0000 7f276a07e740 1 Processor -- start 2024-03-24T10:43:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f276a07e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x556309085ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f276a07e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630907ff00 0x556309086220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f276a07e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556309088cd0 0x556309086760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f276a07e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556308fe5c90 con 0x556309088cd0 2024-03-24T10:43:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f276a07e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556308f8ac50 con 0x55630907f5e0 2024-03-24T10:43:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f276a07e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556308cb9250 con 0x55630907ff00 2024-03-24T10:43:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f276a07e740 10 client.0 _ll_register_callbacks cb 0x556308fd9290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2765a3a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556309088cd0 0x556309086760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2765a3a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556309088cd0 0x556309086760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55082/0 (socket says 172.21.15.23:55082) 2024-03-24T10:43:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2765a3a640 1 -- 172.21.15.23:0/4063256929 learned_addr learned my addr 172.21.15.23:0/4063256929 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2765239640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x556309085ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2764a38640 1 --2- 172.21.15.23:0/4063256929 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630907ff00 0x556309086220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2765a3a640 1 -- 172.21.15.23:0/4063256929 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630907ff00 msgr2=0x556309086220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2765a3a640 1 --2- 172.21.15.23:0/4063256929 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630907ff00 0x556309086220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2765a3a640 1 -- 172.21.15.23:0/4063256929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 msgr2=0x556309085ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2765a3a640 1 --2- 172.21.15.23:0/4063256929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x556309085ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2765a3a640 1 -- 172.21.15.23:0/4063256929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f276010eeb0 con 0x556309088cd0 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2765a3a640 1 --2- 172.21.15.23:0/4063256929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556309088cd0 0x556309086760 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f275800ff40 tx=0x7f2758004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dbfd7fc64fb56b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2748ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2748ff9640 1 -- 172.21.15.23:0/4063256929 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2758004750 con 0x556309088cd0 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2748ff9640 1 -- 172.21.15.23:0/4063256929 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f2758020470 con 0x556309088cd0 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f276a07e740 1 -- 172.21.15.23:0/4063256929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55630900a790 con 0x556309088cd0 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.934+0000 7f2748ff9640 1 -- 172.21.15.23:0/4063256929 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2758020630 con 0x556309088cd0 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.935+0000 7f276a07e740 10 client.16443 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.935+0000 7f276a07e740 1 -- 172.21.15.23:0/4063256929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556308fe4310 con 0x556309088cd0 2024-03-24T10:43:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.935+0000 7f2748ff9640 1 -- 172.21.15.23:0/4063256929 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2758035080 con 0x556309088cd0 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.935+0000 7f2748ff9640 1 -- 172.21.15.23:0/4063256929 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f275802f030 con 0x556309088cd0 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.935+0000 7f276a07e740 10 client.16443 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.935+0000 7f276a07e740 10 client.16443 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.935+0000 7f276a07e740 20 client.16443 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.935+0000 7f276a07e740 1 --2- 172.21.15.23:0/4063256929 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x556309098c50 0x55630909b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.935+0000 7f276a07e740 4 client.16443 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.935+0000 7f276a07e740 1 -- 172.21.15.23:0/4063256929 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556308ce5600 con 0x556309098c50 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.935+0000 7f2765239640 1 --2- 172.21.15.23:0/4063256929 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x556309098c50 0x55630909b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.936+0000 7f2748ff9640 10 client.16443 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.936+0000 7f2765239640 1 --2- 172.21.15.23:0/4063256929 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x556309098c50 0x55630909b070 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ed6821dce97aa04 server_cookie=69b8af92113c2ae4 in_seq=0 out_seq=0 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.936+0000 7f2748ff9640 1 -- 172.21.15.23:0/4063256929 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556308ce5600 con 0x556309098c50 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.936+0000 7f2748ff9640 10 client.16443 handle_command_reply: tid=0 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.936+0000 7f276a07e740 1 client.16443 shutdown 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.936+0000 7f276a07e740 1 -- 172.21.15.23:0/4063256929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556309088cd0 msgr2=0x556309086760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.936+0000 7f276a07e740 1 --2- 172.21.15.23:0/4063256929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556309088cd0 0x556309086760 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f275800ff40 tx=0x7f2758004ab0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.937+0000 7f276a07e740 1 -- 172.21.15.23:0/4063256929 shutdown_connections 2024-03-24T10:43:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.937+0000 7f276a07e740 1 --2- 172.21.15.23:0/4063256929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556309088cd0 0x556309086760 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.937+0000 7f276a07e740 1 --2- 172.21.15.23:0/4063256929 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x556309098c50 0x55630909b070 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.937+0000 7f276a07e740 1 --2- 172.21.15.23:0/4063256929 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630907ff00 0x556309086220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.937+0000 7f276a07e740 1 --2- 172.21.15.23:0/4063256929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x556309085ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.937+0000 7f2748ff9640 0 client.16443 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.937+0000 7f276a07e740 1 -- 172.21.15.23:0/4063256929 >> 172.21.15.23:0/4063256929 conn(0x556309079f00 msgr2=0x55630907bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.937+0000 7f276a07e740 1 -- 172.21.15.23:0/4063256929 shutdown_connections 2024-03-24T10:43:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.937+0000 7f276a07e740 1 -- 172.21.15.23:0/4063256929 wait complete. 2024-03-24T10:43:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.937+0000 7f276a07e740 20 client.16443 trim_cache size 0 max 16384 2024-03-24T10:43:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.952+0000 7f276a07e740 1 Processor -- start 2024-03-24T10:43:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.952+0000 7f276a07e740 1 -- start start 2024-03-24T10:43:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f276a07e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630907f5e0 0x556309094340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f276a07e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309086580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f276a07e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556309088cd0 0x556309078d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f276a07e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556308f6b660 con 0x55630907ff00 2024-03-24T10:43:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f276a07e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5563090756c0 con 0x556309088cd0 2024-03-24T10:43:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f276a07e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556309081930 con 0x55630907f5e0 2024-03-24T10:43:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f2764a38640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309086580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f2764a38640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309086580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55090/0 (socket says 172.21.15.23:55090) 2024-03-24T10:43:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f2764a38640 1 -- 172.21.15.23:0/2047418809 learned_addr learned my addr 172.21.15.23:0/2047418809 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f2765a3a640 1 --2- 172.21.15.23:0/2047418809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556309088cd0 0x556309078d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f2765239640 1 --2- 172.21.15.23:0/2047418809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630907f5e0 0x556309094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f2764a38640 1 -- 172.21.15.23:0/2047418809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630907f5e0 msgr2=0x556309094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f2764a38640 1 --2- 172.21.15.23:0/2047418809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630907f5e0 0x556309094340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f2764a38640 1 -- 172.21.15.23:0/2047418809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556309088cd0 msgr2=0x556309078d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f2764a38640 1 --2- 172.21.15.23:0/2047418809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556309088cd0 0x556309078d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f2764a38640 1 -- 172.21.15.23:0/2047418809 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556308f80810 con 0x55630907ff00 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f2764a38640 1 --2- 172.21.15.23:0/2047418809 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309086580 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f27500cbc10 tx=0x7f2750004b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2845b535500e4f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f274affd640 1 -- 172.21.15.23:0/2047418809 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275009d070 con 0x55630907ff00 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f274affd640 1 -- 172.21.15.23:0/2047418809 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f275009e040 con 0x55630907ff00 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.953+0000 7f274affd640 1 -- 172.21.15.23:0/2047418809 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27500a3110 con 0x55630907ff00 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.954+0000 7f276a07e740 1 -- 172.21.15.23:0/2047418809 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 msgr2=0x556309086580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.954+0000 7f276a07e740 1 --2- 172.21.15.23:0/2047418809 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309086580 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f27500cbc10 tx=0x7f2750004b30 comp rx=0 tx=0).stop 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.954+0000 7f276a07e740 1 -- 172.21.15.23:0/2047418809 shutdown_connections 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.954+0000 7f276a07e740 1 --2- 172.21.15.23:0/2047418809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556309088cd0 0x556309078d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.954+0000 7f276a07e740 1 --2- 172.21.15.23:0/2047418809 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309086580 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.956 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.954+0000 7f276a07e740 1 --2- 172.21.15.23:0/2047418809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630907f5e0 0x556309094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.954+0000 7f276a07e740 1 -- 172.21.15.23:0/2047418809 >> 172.21.15.23:0/2047418809 conn(0x556309079f00 msgr2=0x556309078610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:17.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.954+0000 7f276a07e740 1 -- 172.21.15.23:0/2047418809 shutdown_connections 2024-03-24T10:43:17.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.954+0000 7f276a07e740 1 -- 172.21.15.23:0/2047418809 wait complete. 2024-03-24T10:43:17.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.954+0000 7f276a07e740 1 -- start start 2024-03-24T10:43:17.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.955+0000 7f276a07e740 1 Processor -- start 2024-03-24T10:43:17.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.955+0000 7f276a07e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x556309082950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.955+0000 7f276a07e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309082e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.955+0000 7f276a07e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556309088cd0 0x5563090833d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.955+0000 7f276a07e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5563090fdfb0 con 0x55630907ff00 2024-03-24T10:43:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.955+0000 7f276a07e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5563090fe0f0 con 0x55630907f5e0 2024-03-24T10:43:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.955+0000 7f276a07e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5563090fe230 con 0x556309088cd0 2024-03-24T10:43:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.955+0000 7f276a07e740 10 client.0 _ll_register_callbacks cb 0x7f2758009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.955+0000 7f2764a38640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309082e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.955+0000 7f2764a38640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309082e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55102/0 (socket says 172.21.15.23:55102) 2024-03-24T10:43:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.955+0000 7f2764a38640 1 -- 172.21.15.23:0/2714723610 learned_addr learned my addr 172.21.15.23:0/2714723610 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.955+0000 7f2765239640 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x556309082950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.955+0000 7f2765a3a640 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556309088cd0 0x5563090833d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.955+0000 7f2764a38640 1 -- 172.21.15.23:0/2714723610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556309088cd0 msgr2=0x5563090833d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f2764a38640 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556309088cd0 0x5563090833d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f2764a38640 1 -- 172.21.15.23:0/2714723610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 msgr2=0x556309082950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:17.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f2764a38640 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x556309082950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f2764a38640 1 -- 172.21.15.23:0/2714723610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f276019d200 con 0x55630907ff00 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f2765239640 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x556309082950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f2765a3a640 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556309088cd0 0x5563090833d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f2764a38640 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309082e90 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f275009ad90 tx=0x7f2750005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa6a3cffa002db5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f2748ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f2748ff9640 1 -- 172.21.15.23:0/2714723610 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275009d040 con 0x55630907ff00 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f2748ff9640 1 -- 172.21.15.23:0/2714723610 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f27500a3110 con 0x55630907ff00 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f276a07e740 1 -- 172.21.15.23:0/2714723610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f275c004030 con 0x55630907ff00 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f2748ff9640 1 -- 172.21.15.23:0/2714723610 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f27500a3310 con 0x55630907ff00 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f276a07e740 10 client.16449 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f276a07e740 1 -- 172.21.15.23:0/2714723610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556308fa65a0 con 0x55630907ff00 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f2748ff9640 1 -- 172.21.15.23:0/2714723610 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f275c004030 con 0x55630907ff00 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.956+0000 7f2748ff9640 1 -- 172.21.15.23:0/2714723610 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f27500cd040 con 0x55630907ff00 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.957+0000 7f276a07e740 10 client.16449 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.957+0000 7f276a07e740 10 client.16449 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:17.961 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.957+0000 7f276a07e740 20 client.16449 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:17.962 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.957+0000 7f276a07e740 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5563090ff890 0x556309101cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:17.962 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.957+0000 7f276a07e740 4 client.16449 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:43:17.962 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.957+0000 7f276a07e740 1 -- 172.21.15.23:0/2714723610 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556308d361b0 con 0x5563090ff890 2024-03-24T10:43:17.962 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.957+0000 7f2765239640 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5563090ff890 0x556309101cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:17.962 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.957+0000 7f2748ff9640 10 client.16449 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:17.962 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.957+0000 7f2765239640 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5563090ff890 0x556309101cb0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f145db08fd4ab539 server_cookie=531843b66fab5216 in_seq=0 out_seq=0 2024-03-24T10:43:17.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.965+0000 7f2748ff9640 1 -- 172.21.15.23:0/2714723610 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556308d361b0 con 0x5563090ff890 2024-03-24T10:43:17.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.965+0000 7f2748ff9640 10 client.16449 handle_command_reply: tid=0 2024-03-24T10:43:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.965+0000 7f276a07e740 1 client.16449 shutdown 2024-03-24T10:43:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.966+0000 7f276a07e740 1 -- 172.21.15.23:0/2714723610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 msgr2=0x556309082e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.966+0000 7f276a07e740 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309082e90 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f275009ad90 tx=0x7f2750005cc0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.966+0000 7f276a07e740 1 -- 172.21.15.23:0/2714723610 shutdown_connections 2024-03-24T10:43:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.966+0000 7f276a07e740 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5563090ff890 0x556309101cb0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.966+0000 7f276a07e740 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556309088cd0 0x5563090833d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.966+0000 7f2748ff9640 0 client.16449 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.966+0000 7f276a07e740 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55630907ff00 0x556309082e90 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.966+0000 7f276a07e740 1 --2- 172.21.15.23:0/2714723610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55630907f5e0 0x556309082950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.966+0000 7f276a07e740 1 -- 172.21.15.23:0/2714723610 >> 172.21.15.23:0/2714723610 conn(0x556309079f00 msgr2=0x556309078ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.966+0000 7f276a07e740 1 -- 172.21.15.23:0/2714723610 shutdown_connections 2024-03-24T10:43:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.966+0000 7f276a07e740 1 -- 172.21.15.23:0/2714723610 wait complete. 2024-03-24T10:43:17.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.966+0000 7f276a07e740 20 client.16449 trim_cache size 0 max 16384 2024-03-24T10:43:17.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:43:17.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:43:17.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:43:17.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:43:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.967+0000 7f276749d640 1 -- 172.21.15.23:0/2912879238 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f274c0a22d0 msgr2=0x7f274c0a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.967+0000 7f276749d640 1 --2- 172.21.15.23:0/2912879238 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f274c0a22d0 0x7f274c0a4790 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2758005fd0 tx=0x7f2758011b40 comp rx=0 tx=0).stop 2024-03-24T10:43:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.967+0000 7f276749d640 1 -- 172.21.15.23:0/2912879238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27601020a0 msgr2=0x7f27601982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.967+0000 7f276749d640 1 --2- 172.21.15.23:0/2912879238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27601020a0 0x7f27601982c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f275000e9f0 tx=0x7f275000edf0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.967+0000 7f276749d640 1 -- 172.21.15.23:0/2912879238 shutdown_connections 2024-03-24T10:43:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.967+0000 7f276749d640 1 --2- 172.21.15.23:0/2912879238 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f274c0a22d0 0x7f274c0a4790 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.967+0000 7f276749d640 1 --2- 172.21.15.23:0/2912879238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27601076f0 0x7f276019c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.967+0000 7f276749d640 1 --2- 172.21.15.23:0/2912879238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27601020a0 0x7f27601982c0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.967+0000 7f276749d640 1 --2- 172.21.15.23:0/2912879238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27601016e0 0x7f2760197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.967+0000 7f276749d640 1 -- 172.21.15.23:0/2912879238 >> 172.21.15.23:0/2912879238 conn(0x7f27600fd1f0 msgr2=0x7f27600fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:17.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.967+0000 7f276749d640 1 -- 172.21.15.23:0/2912879238 shutdown_connections 2024-03-24T10:43:17.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:17.967+0000 7f276749d640 1 -- 172.21.15.23:0/2912879238 wait complete. 2024-03-24T10:43:18.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 216990 12.50 MB/sec warmup 95 sec latency 9.847 ms 2024-03-24T10:43:18.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:18 smithi117 ceph-mon[36368]: pgmap v99: 97 pgs: 97 active+clean; 135 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.8 MiB/s wr, 305 op/s 2024-03-24T10:43:18.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:18 smithi116 ceph-mon[36248]: pgmap v99: 97 pgs: 97 active+clean; 135 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.8 MiB/s wr, 305 op/s 2024-03-24T10:43:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:18 smithi023 ceph-mon[32466]: pgmap v99: 97 pgs: 97 active+clean; 135 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.8 MiB/s wr, 305 op/s 2024-03-24T10:43:19.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 219973 12.55 MB/sec warmup 96 sec latency 13.231 ms 2024-03-24T10:43:20.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 221858 12.50 MB/sec warmup 97 sec latency 25.505 ms 2024-03-24T10:43:20.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:20 smithi117 ceph-mon[36368]: pgmap v100: 97 pgs: 97 active+clean; 135 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 201 op/s 2024-03-24T10:43:20.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:20 smithi116 ceph-mon[36248]: pgmap v100: 97 pgs: 97 active+clean; 135 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 201 op/s 2024-03-24T10:43:20.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:20 smithi023 ceph-mon[32466]: pgmap v100: 97 pgs: 97 active+clean; 135 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 201 op/s 2024-03-24T10:43:21.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 223976 12.51 MB/sec warmup 98 sec latency 130.531 ms 2024-03-24T10:43:22.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 226445 12.51 MB/sec warmup 99 sec latency 19.049 ms 2024-03-24T10:43:22.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:22 smithi117 ceph-mon[36368]: pgmap v101: 97 pgs: 97 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.8 MiB/s wr, 307 op/s 2024-03-24T10:43:22.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:22 smithi116 ceph-mon[36248]: pgmap v101: 97 pgs: 97 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.8 MiB/s wr, 307 op/s 2024-03-24T10:43:22.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:22 smithi023 ceph-mon[32466]: pgmap v101: 97 pgs: 97 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.8 MiB/s wr, 307 op/s 2024-03-24T10:43:23.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 228306 12.47 MB/sec warmup 100 sec latency 16.770 ms 2024-03-24T10:43:24.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 230988 12.51 MB/sec warmup 101 sec latency 11.338 ms 2024-03-24T10:43:24.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:24 smithi117 ceph-mon[36368]: pgmap v102: 97 pgs: 97 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 210 op/s 2024-03-24T10:43:24.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:24 smithi116 ceph-mon[36248]: pgmap v102: 97 pgs: 97 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 210 op/s 2024-03-24T10:43:24.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:24 smithi023 ceph-mon[32466]: pgmap v102: 97 pgs: 97 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 210 op/s 2024-03-24T10:43:25.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 233498 12.51 MB/sec warmup 102 sec latency 12.773 ms 2024-03-24T10:43:26.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 234942 12.46 MB/sec warmup 103 sec latency 61.957 ms 2024-03-24T10:43:26.550 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:43:26.550 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:43:26.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.661+0000 7f3c3959a640 1 -- 172.21.15.23:0/3109839792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c340692b0 msgr2=0x7f3c3410e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:26.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.661+0000 7f3c3959a640 1 --2- 172.21.15.23:0/3109839792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c340692b0 0x7f3c3410e550 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c009cd0 tx=0x7f3c2c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.661+0000 7f3c3959a640 1 -- 172.21.15.23:0/3109839792 shutdown_connections 2024-03-24T10:43:26.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.661+0000 7f3c3959a640 1 --2- 172.21.15.23:0/3109839792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c340692b0 0x7f3c3410e550 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.661+0000 7f3c3959a640 1 --2- 172.21.15.23:0/3109839792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c340688f0 0x7f3c34068d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.661+0000 7f3c3959a640 1 --2- 172.21.15.23:0/3109839792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c34107690 0x7f3c34107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.661+0000 7f3c3959a640 1 -- 172.21.15.23:0/3109839792 >> 172.21.15.23:0/3109839792 conn(0x7f3c34077710 msgr2=0x7f3c34077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:26.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.661+0000 7f3c3959a640 1 -- 172.21.15.23:0/3109839792 shutdown_connections 2024-03-24T10:43:26.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.661+0000 7f3c3959a640 1 -- 172.21.15.23:0/3109839792 wait complete. 2024-03-24T10:43:26.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.662+0000 7f3c3959a640 1 Processor -- start 2024-03-24T10:43:26.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.662+0000 7f3c3959a640 1 -- start start 2024-03-24T10:43:26.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.662+0000 7f3c3959a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c340688f0 0x7f3c34197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.662+0000 7f3c3959a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c340692b0 0x7f3c34198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.662+0000 7f3c3959a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c34107690 0x7f3c3419c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.662+0000 7f3c3959a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3419cbe0 con 0x7f3c340688f0 2024-03-24T10:43:26.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.662+0000 7f3c3959a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3419cd50 con 0x7f3c34107690 2024-03-24T10:43:26.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.662+0000 7f3c3959a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3419cf10 con 0x7f3c340692b0 2024-03-24T10:43:26.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.662+0000 7f3c32ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c340688f0 0x7f3c34197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.662+0000 7f3c32ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c340688f0 0x7f3c34197d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55122/0 (socket says 172.21.15.23:55122) 2024-03-24T10:43:26.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.662+0000 7f3c32ffd640 1 -- 172.21.15.23:0/3741515990 learned_addr learned my addr 172.21.15.23:0/3741515990 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:26.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.662+0000 7f3c327fc640 1 --2- 172.21.15.23:0/3741515990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c340692b0 0x7f3c34198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.662+0000 7f3c337fe640 1 --2- 172.21.15.23:0/3741515990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c34107690 0x7f3c3419c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.663+0000 7f3c32ffd640 1 -- 172.21.15.23:0/3741515990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c340692b0 msgr2=0x7f3c34198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:26.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.663+0000 7f3c32ffd640 1 --2- 172.21.15.23:0/3741515990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c340692b0 0x7f3c34198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.663+0000 7f3c32ffd640 1 -- 172.21.15.23:0/3741515990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c34107690 msgr2=0x7f3c3419c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:26.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.663+0000 7f3c32ffd640 1 --2- 172.21.15.23:0/3741515990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c34107690 0x7f3c3419c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.663+0000 7f3c32ffd640 1 -- 172.21.15.23:0/3741515990 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c2c009900 con 0x7f3c340688f0 2024-03-24T10:43:26.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.663+0000 7f3c32ffd640 1 --2- 172.21.15.23:0/3741515990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c340688f0 0x7f3c34197d20 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400e9f0 tx=0x7f3c2400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:26.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.663+0000 7f3c1bfff640 1 -- 172.21.15.23:0/3741515990 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c24009700 con 0x7f3c340688f0 2024-03-24T10:43:26.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.663+0000 7f3c3959a640 1 -- 172.21.15.23:0/3741515990 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c3419d1f0 con 0x7f3c340688f0 2024-03-24T10:43:26.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.663+0000 7f3c1bfff640 1 -- 172.21.15.23:0/3741515990 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c2400cd80 con 0x7f3c340688f0 2024-03-24T10:43:26.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.663+0000 7f3c1bfff640 1 -- 172.21.15.23:0/3741515990 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c24018920 con 0x7f3c340688f0 2024-03-24T10:43:26.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.663+0000 7f3c3959a640 1 -- 172.21.15.23:0/3741515990 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c341a4a70 con 0x7f3c340688f0 2024-03-24T10:43:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c3c17e740 1 Processor -- start 2024-03-24T10:43:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c1bfff640 1 -- 172.21.15.23:0/3741515990 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f3c24018a80 con 0x7f3c340688f0 2024-03-24T10:43:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c3c17e740 1 -- start start 2024-03-24T10:43:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c3c17e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 0x558c4f70b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c3c17e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f714190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c3c17e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f7146d0 0x558c4f716ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c3c17e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558c4f62ff30 con 0x558c4f70baf0 2024-03-24T10:43:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c3c17e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558c4f679810 con 0x558c4f7146d0 2024-03-24T10:43:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c3c17e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558c4f676d40 con 0x558c4f70b1d0 2024-03-24T10:43:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c32ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f714190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c32ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f714190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55132/0 (socket says 172.21.15.23:55132) 2024-03-24T10:43:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c32ffd640 1 -- 172.21.15.23:0/3291541293 learned_addr learned my addr 172.21.15.23:0/3291541293 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c337fe640 1 --2- 172.21.15.23:0/3291541293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 0x558c4f70b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c327fc640 1 --2- 172.21.15.23:0/3291541293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f7146d0 0x558c4f716ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c337fe640 1 -- 172.21.15.23:0/3291541293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f7146d0 msgr2=0x558c4f716ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c337fe640 1 --2- 172.21.15.23:0/3291541293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f7146d0 0x558c4f716ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c337fe640 1 -- 172.21.15.23:0/3291541293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 msgr2=0x558c4f714190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c337fe640 1 --2- 172.21.15.23:0/3291541293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f714190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c337fe640 1 -- 172.21.15.23:0/3291541293 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c4f633ea0 con 0x558c4f70b1d0 2024-03-24T10:43:26.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.666+0000 7f3c32ffd640 1 --2- 172.21.15.23:0/3291541293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f714190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:26.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.667+0000 7f3c1bfff640 1 --2- 172.21.15.23:0/3741515990 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3c1c0a2230 0x7f3c1c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.667+0000 7f3c32ffd640 1 --2- 172.21.15.23:0/3741515990 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3c1c0a2230 0x7f3c1c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.667+0000 7f3c337fe640 1 --2- 172.21.15.23:0/3291541293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 0x558c4f70b5b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c02c630 tx=0x7f3c2c032d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8126195153fa146e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:26.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.667+0000 7f3c197fa640 1 -- 172.21.15.23:0/3291541293 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c034440 con 0x558c4f70b1d0 2024-03-24T10:43:26.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.667+0000 7f3c1bfff640 1 -- 172.21.15.23:0/3741515990 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f3c24014070 con 0x7f3c340688f0 2024-03-24T10:43:26.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.667+0000 7f3c197fa640 1 -- 172.21.15.23:0/3291541293 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c2c0345a0 con 0x558c4f70b1d0 2024-03-24T10:43:26.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.667+0000 7f3c197fa640 1 -- 172.21.15.23:0/3291541293 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c003b70 con 0x558c4f70b1d0 2024-03-24T10:43:26.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 -- 172.21.15.23:0/3291541293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 msgr2=0x558c4f70b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:26.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/3291541293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 0x558c4f70b5b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c02c630 tx=0x7f3c2c032d70 comp rx=0 tx=0).stop 2024-03-24T10:43:26.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c32ffd640 1 --2- 172.21.15.23:0/3741515990 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3c1c0a2230 0x7f3c1c0a46f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3c24097280 tx=0x7f3c240d1cd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:26.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 -- 172.21.15.23:0/3291541293 shutdown_connections 2024-03-24T10:43:26.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/3291541293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f7146d0 0x558c4f716ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/3291541293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f714190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/3291541293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 0x558c4f70b5b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 -- 172.21.15.23:0/3291541293 >> 172.21.15.23:0/3291541293 conn(0x558c4f705900 msgr2=0x558c4f705ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 -- 172.21.15.23:0/3291541293 shutdown_connections 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 -- 172.21.15.23:0/3291541293 wait complete. 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 -- start start 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 Processor -- start 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 0x558c4f7116f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f711c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f7146d0 0x558c4f712170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558c4f672520 con 0x558c4f70baf0 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558c4f345430 con 0x558c4f7146d0 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558c4f301e00 con 0x558c4f70b1d0 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c3c17e740 10 client.0 _ll_register_callbacks cb 0x558c4f610d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c327fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f711c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c327fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f711c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55146/0 (socket says 172.21.15.23:55146) 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.668+0000 7f3c327fc640 1 -- 172.21.15.23:0/1368047496 learned_addr learned my addr 172.21.15.23:0/1368047496 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c337fe640 1 --2- 172.21.15.23:0/1368047496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 0x558c4f7116f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c337fe640 1 --2- 172.21.15.23:0/1368047496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f7146d0 0x558c4f712170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c327fc640 1 -- 172.21.15.23:0/1368047496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 msgr2=0x558c4f7116f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c327fc640 1 --2- 172.21.15.23:0/1368047496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 0x558c4f7116f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c327fc640 1 -- 172.21.15.23:0/1368047496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f7146d0 msgr2=0x558c4f712170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c327fc640 1 --2- 172.21.15.23:0/1368047496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f7146d0 0x558c4f712170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c327fc640 1 -- 172.21.15.23:0/1368047496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c3410ee50 con 0x558c4f70baf0 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c337fe640 1 --2- 172.21.15.23:0/1368047496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 0x558c4f7116f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c327fc640 1 --2- 172.21.15.23:0/1368047496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f711c30 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3c280126e0 tx=0x7f3c28012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e20b81ed3ae6cb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c16ff5640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c16ff5640 1 -- 172.21.15.23:0/1368047496 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c28017070 con 0x558c4f70baf0 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c16ff5640 1 -- 172.21.15.23:0/1368047496 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f3c2800f920 con 0x558c4f70baf0 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1368047496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558c4f3717e0 con 0x558c4f70baf0 2024-03-24T10:43:26.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c16ff5640 1 -- 172.21.15.23:0/1368047496 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3c2800fae0 con 0x558c4f70baf0 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c3c17e740 10 client.16461 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1368047496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558c4f670ba0 con 0x558c4f70baf0 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c16ff5640 1 -- 172.21.15.23:0/1368047496 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c28028080 con 0x558c4f70baf0 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.669+0000 7f3c16ff5640 1 -- 172.21.15.23:0/1368047496 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3c28022030 con 0x558c4f70baf0 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.670+0000 7f3c3c17e740 10 client.16461 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.670+0000 7f3c3c17e740 10 client.16461 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.670+0000 7f3c3c17e740 20 client.16461 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.670+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/1368047496 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558c4f724ca0 0x558c4f7270c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.670+0000 7f3c3c17e740 4 client.16461 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.670+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1368047496 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c4f3ae720 con 0x558c4f724ca0 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.670+0000 7f3c337fe640 1 --2- 172.21.15.23:0/1368047496 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558c4f724ca0 0x558c4f7270c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.670+0000 7f3c16ff5640 10 client.16461 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.670+0000 7f3c337fe640 1 --2- 172.21.15.23:0/1368047496 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558c4f724ca0 0x558c4f7270c0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=216f2b9a8d734fc1 server_cookie=d4be4d0ee0745e0c in_seq=0 out_seq=0 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c16ff5640 1 -- 172.21.15.23:0/1368047496 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f3c2c0559c0 con 0x558c4f724ca0 2024-03-24T10:43:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c16ff5640 10 client.16461 handle_command_reply: tid=0 2024-03-24T10:43:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c3c17e740 1 client.16461 shutdown 2024-03-24T10:43:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1368047496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 msgr2=0x558c4f711c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/1368047496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f711c30 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3c280126e0 tx=0x7f3c28012ba0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1368047496 shutdown_connections 2024-03-24T10:43:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/1368047496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f7146d0 0x558c4f712170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/1368047496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f711c30 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/1368047496 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558c4f724ca0 0x558c4f7270c0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/1368047496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 0x558c4f7116f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c16ff5640 0 client.16461 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1368047496 >> 172.21.15.23:0/1368047496 conn(0x558c4f705900 msgr2=0x558c4f707650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1368047496 shutdown_connections 2024-03-24T10:43:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1368047496 wait complete. 2024-03-24T10:43:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.671+0000 7f3c3c17e740 20 client.16461 trim_cache size 0 max 16384 2024-03-24T10:43:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c3c17e740 1 Processor -- start 2024-03-24T10:43:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c3c17e740 1 -- start start 2024-03-24T10:43:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c3c17e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f705900 0x558c4f713e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c3c17e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f70b1d0 0x558c4f71ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c3c17e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f72c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c3c17e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558c4f5f7430 con 0x558c4f70baf0 2024-03-24T10:43:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c3c17e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558c4f705730 con 0x558c4f70b1d0 2024-03-24T10:43:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c3c17e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558c4f701550 con 0x558c4f705900 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f72c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c327fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f70b1d0 0x558c4f71ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f72c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55156/0 (socket says 172.21.15.23:55156) 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c337fe640 1 -- 172.21.15.23:0/462056497 learned_addr learned my addr 172.21.15.23:0/462056497 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c337fe640 1 --2- 172.21.15.23:0/462056497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f705900 0x558c4f713e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c327fc640 1 -- 172.21.15.23:0/462056497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f705900 msgr2=0x558c4f713e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c327fc640 1 --2- 172.21.15.23:0/462056497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f705900 0x558c4f713e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c327fc640 1 -- 172.21.15.23:0/462056497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 msgr2=0x558c4f72c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c327fc640 1 --2- 172.21.15.23:0/462056497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f72c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c327fc640 1 -- 172.21.15.23:0/462056497 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c4f60cb20 con 0x558c4f70b1d0 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.690+0000 7f3c337fe640 1 --2- 172.21.15.23:0/462056497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f72c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.691+0000 7f3c327fc640 1 --2- 172.21.15.23:0/462056497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f70b1d0 0x558c4f71ff00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3c28000c00 tx=0x7f3c2800f810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17e26da74590c312 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.691+0000 7f3c18ff9640 1 -- 172.21.15.23:0/462056497 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c28017070 con 0x558c4f70b1d0 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.691+0000 7f3c18ff9640 1 -- 172.21.15.23:0/462056497 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558c4f3717e0 con 0x558c4f70b1d0 2024-03-24T10:43:26.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.691+0000 7f3c18ff9640 1 -- 172.21.15.23:0/462056497 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c280048a0 con 0x558c4f70b1d0 2024-03-24T10:43:26.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.691+0000 7f3c3c17e740 1 -- 172.21.15.23:0/462056497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f70b1d0 msgr2=0x558c4f71ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.691+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/462056497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f70b1d0 0x558c4f71ff00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3c28000c00 tx=0x7f3c2800f810 comp rx=0 tx=0).stop 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 -- 172.21.15.23:0/462056497 shutdown_connections 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/462056497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f70baf0 0x558c4f72c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/462056497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f70b1d0 0x558c4f71ff00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/462056497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f705900 0x558c4f713e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 -- 172.21.15.23:0/462056497 >> 172.21.15.23:0/462056497 conn(0x558c4f710330 msgr2=0x558c4f704b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 -- 172.21.15.23:0/462056497 shutdown_connections 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 -- 172.21.15.23:0/462056497 wait complete. 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 -- start start 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 Processor -- start 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f705900 0x558c4f71c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 0x558c4f718b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f70baf0 0x558c4f7190c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558c4f7197c0 con 0x558c4f705900 2024-03-24T10:43:26.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558c4f719930 con 0x558c4f70baf0 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558c4f78f3e0 con 0x558c4f70b1d0 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c3c17e740 10 client.0 _ll_register_callbacks cb 0x7f3c28009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f705900 0x558c4f71c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f705900 0x558c4f71c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55168/0 (socket says 172.21.15.23:55168) 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c337fe640 1 -- 172.21.15.23:0/1046996002 learned_addr learned my addr 172.21.15.23:0/1046996002 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c327fc640 1 --2- 172.21.15.23:0/1046996002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 0x558c4f718b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.692+0000 7f3c337fe640 1 --2- 172.21.15.23:0/1046996002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f70baf0 0x558c4f7190c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c337fe640 1 -- 172.21.15.23:0/1046996002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 msgr2=0x558c4f718b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c337fe640 1 --2- 172.21.15.23:0/1046996002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 0x558c4f718b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c337fe640 1 -- 172.21.15.23:0/1046996002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f70baf0 msgr2=0x558c4f7190c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c337fe640 1 --2- 172.21.15.23:0/1046996002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f70baf0 0x558c4f7190c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c337fe640 1 -- 172.21.15.23:0/1046996002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c4f633ea0 con 0x558c4f705900 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c337fe640 1 --2- 172.21.15.23:0/1046996002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f705900 0x558c4f71c7b0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c004af0 tx=0x7f3c2c041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e8f05d2c554720e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c16ff5640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c16ff5640 1 -- 172.21.15.23:0/1046996002 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c032970 con 0x558c4f705900 2024-03-24T10:43:26.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c16ff5640 1 -- 172.21.15.23:0/1046996002 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f3c2c034a90 con 0x558c4f705900 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1046996002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c2c0345a0 con 0x558c4f705900 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c16ff5640 1 -- 172.21.15.23:0/1046996002 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3c2c034c50 con 0x558c4f705900 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c3c17e740 10 client.16467 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1046996002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558c4f670ba0 con 0x558c4f705900 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c16ff5640 1 -- 172.21.15.23:0/1046996002 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c2c0345a0 con 0x558c4f705900 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c16ff5640 1 -- 172.21.15.23:0/1046996002 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3c2c004300 con 0x558c4f705900 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c3c17e740 10 client.16467 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c3c17e740 10 client.16467 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c3c17e740 20 client.16467 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.693+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/1046996002 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558c4f7146d0 0x558c4f792880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c3c17e740 4 client.16467 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1046996002 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558c4f3fd890 con 0x558c4f7146d0 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c327fc640 1 --2- 172.21.15.23:0/1046996002 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558c4f7146d0 0x558c4f792880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c16ff5640 10 client.16467 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c327fc640 1 --2- 172.21.15.23:0/1046996002 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558c4f7146d0 0x558c4f792880 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=400293505221cd61 server_cookie=eb24c717ca8656a0 in_seq=0 out_seq=0 2024-03-24T10:43:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c16ff5640 1 -- 172.21.15.23:0/1046996002 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558c4f3fd890 con 0x558c4f7146d0 2024-03-24T10:43:26.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c16ff5640 10 client.16467 handle_command_reply: tid=0 2024-03-24T10:43:26.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c3c17e740 1 client.16467 shutdown 2024-03-24T10:43:26.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1046996002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f705900 msgr2=0x558c4f71c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:26.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/1046996002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f705900 0x558c4f71c7b0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c004af0 tx=0x7f3c2c041040 comp rx=0 tx=0).stop 2024-03-24T10:43:26.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1046996002 shutdown_connections 2024-03-24T10:43:26.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/1046996002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4f70baf0 0x558c4f7190c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/1046996002 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558c4f7146d0 0x558c4f792880 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/1046996002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4f70b1d0 0x558c4f718b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c3c17e740 1 --2- 172.21.15.23:0/1046996002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558c4f705900 0x558c4f71c7b0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c16ff5640 0 client.16467 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:26.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1046996002 >> 172.21.15.23:0/1046996002 conn(0x558c4f710330 msgr2=0x558c4f706c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:26.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1046996002 shutdown_connections 2024-03-24T10:43:26.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.694+0000 7f3c3c17e740 1 -- 172.21.15.23:0/1046996002 wait complete. 2024-03-24T10:43:26.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.695+0000 7f3c3c17e740 20 client.16467 trim_cache size 0 max 16384 2024-03-24T10:43:26.701 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:43:26.701 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:43:26.701 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:43:26.701 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:43:26.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.695+0000 7f3c3959a640 1 -- 172.21.15.23:0/3741515990 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3c1c0a2230 msgr2=0x7f3c1c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:26.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.695+0000 7f3c3959a640 1 --2- 172.21.15.23:0/3741515990 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3c1c0a2230 0x7f3c1c0a46f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3c24097280 tx=0x7f3c240d1cd0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.695+0000 7f3c3959a640 1 -- 172.21.15.23:0/3741515990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c340688f0 msgr2=0x7f3c34197d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:26.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.695+0000 7f3c3959a640 1 --2- 172.21.15.23:0/3741515990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c340688f0 0x7f3c34197d20 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400e9f0 tx=0x7f3c2400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.695+0000 7f3c3959a640 1 -- 172.21.15.23:0/3741515990 shutdown_connections 2024-03-24T10:43:26.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.695+0000 7f3c3959a640 1 --2- 172.21.15.23:0/3741515990 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3c1c0a2230 0x7f3c1c0a46f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.695+0000 7f3c3959a640 1 --2- 172.21.15.23:0/3741515990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c34107690 0x7f3c3419c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.695+0000 7f3c3959a640 1 --2- 172.21.15.23:0/3741515990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c340692b0 0x7f3c34198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.695+0000 7f3c3959a640 1 --2- 172.21.15.23:0/3741515990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c340688f0 0x7f3c34197d20 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.695+0000 7f3c3959a640 1 -- 172.21.15.23:0/3741515990 >> 172.21.15.23:0/3741515990 conn(0x7f3c34077710 msgr2=0x7f3c340fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:26.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.695+0000 7f3c3959a640 1 -- 172.21.15.23:0/3741515990 shutdown_connections 2024-03-24T10:43:26.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.695+0000 7f3c3959a640 1 -- 172.21.15.23:0/3741515990 wait complete. 2024-03-24T10:43:26.706 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:43:26.706 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:43:26.731 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:26 smithi023 ceph-mon[32466]: pgmap v103: 97 pgs: 97 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 210 op/s 2024-03-24T10:43:26.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:26 smithi117 ceph-mon[36368]: pgmap v103: 97 pgs: 97 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 210 op/s 2024-03-24T10:43:26.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:26 smithi116 ceph-mon[36248]: pgmap v103: 97 pgs: 97 active+clean; 143 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 210 op/s 2024-03-24T10:43:26.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.818+0000 7f0a5de4e640 1 -- 172.21.15.23:0/3420525021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a58100cc0 msgr2=0x7f0a581063f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:26.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.818+0000 7f0a5de4e640 1 --2- 172.21.15.23:0/3420525021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a58100cc0 0x7f0a581063f0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c009930 tx=0x7f0a4c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:43:26.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.818+0000 7f0a5de4e640 1 -- 172.21.15.23:0/3420525021 shutdown_connections 2024-03-24T10:43:26.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.818+0000 7f0a5de4e640 1 --2- 172.21.15.23:0/3420525021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58106ad0 0x7f0a5810e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.818+0000 7f0a5de4e640 1 --2- 172.21.15.23:0/3420525021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a58100cc0 0x7f0a581063f0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.818+0000 7f0a5de4e640 1 --2- 172.21.15.23:0/3420525021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a58100310 0x7f0a581006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.818+0000 7f0a5de4e640 1 -- 172.21.15.23:0/3420525021 >> 172.21.15.23:0/3420525021 conn(0x7f0a58075920 msgr2=0x7f0a58075d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:26.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.818+0000 7f0a5de4e640 1 -- 172.21.15.23:0/3420525021 shutdown_connections 2024-03-24T10:43:26.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.818+0000 7f0a5de4e640 1 -- 172.21.15.23:0/3420525021 wait complete. 2024-03-24T10:43:26.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.818+0000 7f0a5de4e640 1 Processor -- start 2024-03-24T10:43:26.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.818+0000 7f0a5de4e640 1 -- start start 2024-03-24T10:43:26.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a5de4e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58100310 0x7f0a5819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a5de4e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a58106ad0 0x7f0a5819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a5de4e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5819e950 0x7f0a5819ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a5de4e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5819f340 con 0x7f0a5819e950 2024-03-24T10:43:26.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a5de4e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5819f4b0 con 0x7f0a58100310 2024-03-24T10:43:26.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a5de4e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5819f620 con 0x7f0a58106ad0 2024-03-24T10:43:26.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a57fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5819e950 0x7f0a5819ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a57fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5819e950 0x7f0a5819ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55190/0 (socket says 172.21.15.23:55190) 2024-03-24T10:43:26.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a57fff640 1 -- 172.21.15.23:0/2651708435 learned_addr learned my addr 172.21.15.23:0/2651708435 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:26.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a577fe640 1 --2- 172.21.15.23:0/2651708435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58100310 0x7f0a5819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a56ffd640 1 --2- 172.21.15.23:0/2651708435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a58106ad0 0x7f0a5819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a57fff640 1 -- 172.21.15.23:0/2651708435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a58106ad0 msgr2=0x7f0a5819a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:26.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a57fff640 1 --2- 172.21.15.23:0/2651708435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a58106ad0 0x7f0a5819a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a57fff640 1 -- 172.21.15.23:0/2651708435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58100310 msgr2=0x7f0a5819a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:26.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a57fff640 1 --2- 172.21.15.23:0/2651708435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58100310 0x7f0a5819a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:26.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a57fff640 1 -- 172.21.15.23:0/2651708435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a50009490 con 0x7f0a5819e950 2024-03-24T10:43:26.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a57fff640 1 --2- 172.21.15.23:0/2651708435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5819e950 0x7f0a5819ee00 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f0a5000e9b0 tx=0x7f0a5000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:26.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a54ff9640 1 -- 172.21.15.23:0/2651708435 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5000cd80 con 0x7f0a5819e950 2024-03-24T10:43:26.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a5de4e640 1 -- 172.21.15.23:0/2651708435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a4c009560 con 0x7f0a5819e950 2024-03-24T10:43:26.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.819+0000 7f0a54ff9640 1 -- 172.21.15.23:0/2651708435 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a500035f0 con 0x7f0a5819e950 2024-03-24T10:43:26.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.820+0000 7f0a54ff9640 1 -- 172.21.15.23:0/2651708435 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5000cee0 con 0x7f0a5819e950 2024-03-24T10:43:26.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.820+0000 7f0a5de4e640 1 -- 172.21.15.23:0/2651708435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a581a6ef0 con 0x7f0a5819e950 2024-03-24T10:43:26.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.820+0000 7f0a5de4e640 1 -- 172.21.15.23:0/2651708435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a58069840 con 0x7f0a5819e950 2024-03-24T10:43:26.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.822+0000 7f0a54ff9640 1 -- 172.21.15.23:0/2651708435 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f0a50003ad0 con 0x7f0a5819e950 2024-03-24T10:43:26.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.822+0000 7f0a54ff9640 1 --2- 172.21.15.23:0/2651708435 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a400a2230 0x7f0a400a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:26.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.822+0000 7f0a577fe640 1 --2- 172.21.15.23:0/2651708435 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a400a2230 0x7f0a400a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:26.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.822+0000 7f0a54ff9640 1 -- 172.21.15.23:0/2651708435 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0a50014070 con 0x7f0a5819e950 2024-03-24T10:43:26.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.823+0000 7f0a577fe640 1 --2- 172.21.15.23:0/2651708435 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a400a2230 0x7f0a400a46f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0a48005fd0 tx=0x7f0a48009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:26.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:26.827+0000 7f0a54ff9640 1 -- 172.21.15.23:0/2651708435 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0a50097100 con 0x7f0a5819e950 2024-03-24T10:43:27.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.061+0000 7f0a5de4e640 1 -- 172.21.15.23:0/2651708435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a58109da0 con 0x7f0a5819e950 2024-03-24T10:43:27.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.062+0000 7f0a54ff9640 1 -- 172.21.15.23:0/2651708435 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11869 (secure 0 0 0) 0x7f0a5009a7c0 con 0x7f0a5819e950 2024-03-24T10:43:27.062 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:43:27.063 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:41:49.922577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:43:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 25 2024-03-24T10:43:27.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.064+0000 7f0a5de4e640 1 -- 172.21.15.23:0/2651708435 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a400a2230 msgr2=0x7f0a400a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.064+0000 7f0a5de4e640 1 --2- 172.21.15.23:0/2651708435 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a400a2230 0x7f0a400a46f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0a48005fd0 tx=0x7f0a48009500 comp rx=0 tx=0).stop 2024-03-24T10:43:27.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.064+0000 7f0a5de4e640 1 -- 172.21.15.23:0/2651708435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5819e950 msgr2=0x7f0a5819ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.064+0000 7f0a5de4e640 1 --2- 172.21.15.23:0/2651708435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5819e950 0x7f0a5819ee00 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f0a5000e9b0 tx=0x7f0a5000ee70 comp rx=0 tx=0).stop 2024-03-24T10:43:27.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.064+0000 7f0a5de4e640 1 -- 172.21.15.23:0/2651708435 shutdown_connections 2024-03-24T10:43:27.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.064+0000 7f0a5de4e640 1 --2- 172.21.15.23:0/2651708435 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a400a2230 0x7f0a400a46f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.064+0000 7f0a5de4e640 1 --2- 172.21.15.23:0/2651708435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5819e950 0x7f0a5819ee00 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.064+0000 7f0a5de4e640 1 --2- 172.21.15.23:0/2651708435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a58106ad0 0x7f0a5819a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.064+0000 7f0a5de4e640 1 --2- 172.21.15.23:0/2651708435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a58100310 0x7f0a5819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.064+0000 7f0a5de4e640 1 -- 172.21.15.23:0/2651708435 >> 172.21.15.23:0/2651708435 conn(0x7f0a58075920 msgr2=0x7f0a580fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.064+0000 7f0a5de4e640 1 -- 172.21.15.23:0/2651708435 shutdown_connections 2024-03-24T10:43:27.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.064+0000 7f0a5de4e640 1 -- 172.21.15.23:0/2651708435 wait complete. 2024-03-24T10:43:27.075 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:43:27.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 237787 12.51 MB/sec warmup 104 sec latency 19.553 ms 2024-03-24T10:43:27.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.226+0000 7f8e3ab13640 1 -- 172.21.15.23:0/1858663915 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e341005a0 msgr2=0x7f8e34100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.226+0000 7f8e3ab13640 1 --2- 172.21.15.23:0/1858663915 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e341005a0 0x7f8e34100a20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009900 tx=0x7f8e2002bc30 comp rx=0 tx=0).stop 2024-03-24T10:43:27.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.226+0000 7f8e3ab13640 1 -- 172.21.15.23:0/1858663915 shutdown_connections 2024-03-24T10:43:27.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.226+0000 7f8e3ab13640 1 --2- 172.21.15.23:0/1858663915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e34100f60 0x7f8e3410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.226+0000 7f8e3ab13640 1 --2- 172.21.15.23:0/1858663915 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e341005a0 0x7f8e34100a20 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.226+0000 7f8e3ab13640 1 --2- 172.21.15.23:0/1858663915 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e34106660 0x7f8e34106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.226+0000 7f8e3ab13640 1 -- 172.21.15.23:0/1858663915 >> 172.21.15.23:0/1858663915 conn(0x7f8e34077710 msgr2=0x7f8e34077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.226+0000 7f8e3ab13640 1 -- 172.21.15.23:0/1858663915 shutdown_connections 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.226+0000 7f8e3ab13640 1 -- 172.21.15.23:0/1858663915 wait complete. 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.226+0000 7f8e3ab13640 1 Processor -- start 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.226+0000 7f8e3ab13640 1 -- start start 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e3ab13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e341005a0 0x7f8e3419c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e3ab13640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e34100f60 0x7f8e3419ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e3ab13640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e34106660 0x7f8e34196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e3ab13640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f8e341970c0 con 0x7f8e34106660 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e3ab13640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e34197230 con 0x7f8e341005a0 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e3ab13640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8e341973f0 con 0x7f8e34100f60 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e390b0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e34106660 0x7f8e34196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e390b0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e34106660 0x7f8e34196a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55200/0 (socket says 172.21.15.23:55200) 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e33fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e34100f60 0x7f8e3419ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e390b0640 1 -- 172.21.15.23:0/25290964 learned_addr learned my addr 172.21.15.23:0/25290964 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e388af640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e341005a0 0x7f8e3419c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e390b0640 1 -- 172.21.15.23:0/25290964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e34100f60 msgr2=0x7f8e3419ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e390b0640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e34100f60 0x7f8e3419ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e390b0640 1 -- 172.21.15.23:0/25290964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e341005a0 msgr2=0x7f8e3419c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e390b0640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e341005a0 0x7f8e3419c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e390b0640 1 -- 172.21.15.23:0/25290964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e20009560 con 0x7f8e34106660 2024-03-24T10:43:27.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e33fff640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e34100f60 0x7f8e3419ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:27.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e388af640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e341005a0 0x7f8e3419c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:27.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e390b0640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e34106660 0x7f8e34196a70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00eb10 tx=0x7f8e2c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e31ffb640 1 -- 172.21.15.23:0/25290964 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c004b20 con 0x7f8e34106660 2024-03-24T10:43:27.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e31ffb640 1 -- 172.21.15.23:0/25290964 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e2c004c80 con 0x7f8e34106660 2024-03-24T10:43:27.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e3ab13640 1 -- 172.21.15.23:0/25290964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e341976d0 con 0x7f8e34106660 2024-03-24T10:43:27.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.227+0000 7f8e31ffb640 1 -- 172.21.15.23:0/25290964 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c010650 con 0x7f8e34106660 2024-03-24T10:43:27.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.228+0000 7f8e3ab13640 1 -- 172.21.15.23:0/25290964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e341a3960 con 0x7f8e34106660 2024-03-24T10:43:27.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.230+0000 7f8e31ffb640 1 -- 172.21.15.23:0/25290964 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f8e2c005020 con 0x7f8e34106660 2024-03-24T10:43:27.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.230+0000 7f8e3d7d3740 1 Processor -- start 2024-03-24T10:43:27.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.230+0000 7f8e3d7d3740 1 -- start start 2024-03-24T10:43:27.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.230+0000 7f8e31ffb640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8e240a2230 0x7f8e240a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.230+0000 7f8e31ffb640 1 -- 172.21.15.23:0/25290964 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f8e2c014070 con 0x7f8e34106660 2024-03-24T10:43:27.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.230+0000 7f8e3d7d3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc66ee0 0x55c53cc672c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.230+0000 7f8e33fff640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8e240a2230 0x7f8e240a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.230+0000 7f8e3d7d3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc67890 0x55c53cc725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.230+0000 7f8e3d7d3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc72b00 0x55c53cc74ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.230+0000 7f8e3d7d3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c53cbd5840 con 0x55c53cc67890 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.230+0000 7f8e3d7d3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c53cbd2d70 con 0x55c53cc72b00 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.230+0000 7f8e3d7d3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c53cbce550 con 0x55c53cc66ee0 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e33fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc67890 0x55c53cc725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e33fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc67890 0x55c53cc725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55208/0 (socket says 172.21.15.23:55208) 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e33fff640 1 -- 172.21.15.23:0/2466972847 learned_addr learned my addr 172.21.15.23:0/2466972847 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e388af640 1 --2- 172.21.15.23:0/2466972847 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc66ee0 0x55c53cc672c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e390b0640 1 --2- 172.21.15.23:0/2466972847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc72b00 0x55c53cc74ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e388af640 1 -- 172.21.15.23:0/2466972847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc72b00 msgr2=0x55c53cc74ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e388af640 1 --2- 172.21.15.23:0/2466972847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc72b00 0x55c53cc74ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e388af640 1 -- 172.21.15.23:0/2466972847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc67890 msgr2=0x55c53cc725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e388af640 1 --2- 172.21.15.23:0/2466972847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc67890 0x55c53cc725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e388af640 1 -- 172.21.15.23:0/2466972847 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c53cb63de0 con 0x55c53cc66ee0 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e33fff640 1 --2- 172.21.15.23:0/2466972847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc67890 0x55c53cc725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e33fff640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8e240a2230 0x7f8e240a46f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8e20005ec0 tx=0x7f8e2002c770 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e388af640 1 --2- 172.21.15.23:0/2466972847 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc66ee0 0x55c53cc672c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8e28009f80 tx=0x7f8e280118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ed36ceef9de2822 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e1effd640 1 -- 172.21.15.23:0/2466972847 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e280040f0 con 0x55c53cc66ee0 2024-03-24T10:43:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e1effd640 1 -- 172.21.15.23:0/2466972847 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e28005500 con 0x55c53cc66ee0 2024-03-24T10:43:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.231+0000 7f8e1effd640 1 -- 172.21.15.23:0/2466972847 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e28004d00 con 0x55c53cc66ee0 2024-03-24T10:43:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.232+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/2466972847 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc66ee0 msgr2=0x55c53cc672c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.232+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/2466972847 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc66ee0 0x55c53cc672c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8e28009f80 tx=0x7f8e280118d0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.232+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/2466972847 shutdown_connections 2024-03-24T10:43:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.232+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/2466972847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc72b00 0x55c53cc74ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.232+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/2466972847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc67890 0x55c53cc725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.232+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/2466972847 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc66ee0 0x55c53cc672c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.232+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/2466972847 >> 172.21.15.23:0/2466972847 conn(0x55c53cc61b20 msgr2=0x55c53cc61f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.232+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/2466972847 shutdown_connections 2024-03-24T10:43:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.232+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/2466972847 wait complete. 2024-03-24T10:43:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.232+0000 7f8e3d7d3740 1 -- start start 2024-03-24T10:43:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.232+0000 7f8e3d7d3740 1 Processor -- start 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e3d7d3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc66ee0 0x55c53cc6fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e3d7d3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc67890 0x55c53cc70000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e3d7d3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc72b00 0x55c53cc70540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e3d7d3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c53c8a1430 con 0x55c53cc66ee0 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e3d7d3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c53c85de00 con 0x55c53cc67890 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e3d7d3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c53cb53590 con 0x55c53cc72b00 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e3d7d3740 10 client.0 _ll_register_callbacks cb 0x55c53cbc2fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e388af640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc66ee0 0x55c53cc6fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e388af640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc66ee0 0x55c53cc6fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55214/0 (socket says 172.21.15.23:55214) 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e388af640 1 -- 172.21.15.23:0/4040301774 learned_addr learned my addr 172.21.15.23:0/4040301774 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e390b0640 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc72b00 0x55c53cc70540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e33fff640 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc67890 0x55c53cc70000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e388af640 1 -- 172.21.15.23:0/4040301774 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc72b00 msgr2=0x55c53cc70540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e388af640 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc72b00 0x55c53cc70540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e388af640 1 -- 172.21.15.23:0/4040301774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc67890 msgr2=0x55c53cc70000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e388af640 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc67890 0x55c53cc70000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e388af640 1 -- 172.21.15.23:0/4040301774 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e28009be0 con 0x55c53cc66ee0 2024-03-24T10:43:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e390b0640 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc72b00 0x55c53cc70540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.233+0000 7f8e33fff640 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc67890 0x55c53cc70000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e388af640 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc66ee0 0x55c53cc6fac0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8e28004880 tx=0x7f8e28003880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b17cbdb0a322e9c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e13fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e13fff640 1 -- 172.21.15.23:0/4040301774 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e28004520 con 0x55c53cc66ee0 2024-03-24T10:43:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e13fff640 1 -- 172.21.15.23:0/4040301774 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f8e28003c80 con 0x55c53cc66ee0 2024-03-24T10:43:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/4040301774 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c53c8cd7e0 con 0x55c53cc66ee0 2024-03-24T10:43:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e13fff640 1 -- 172.21.15.23:0/4040301774 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8e28005670 con 0x55c53cc66ee0 2024-03-24T10:43:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e3d7d3740 10 client.16485 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/4040301774 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c53cbccbd0 con 0x55c53cc66ee0 2024-03-24T10:43:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e13fff640 1 -- 172.21.15.23:0/4040301774 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c53c8cd7e0 con 0x55c53cc66ee0 2024-03-24T10:43:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e13fff640 1 -- 172.21.15.23:0/4040301774 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8e28021040 con 0x55c53cc66ee0 2024-03-24T10:43:27.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e3d7d3740 10 client.16485 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:27.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e3d7d3740 10 client.16485 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:43:27.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e3d7d3740 20 client.16485 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:27.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55c53cc82ae0 0x55c53cc84ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e3d7d3740 4 client.16485 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:27.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.234+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/4040301774 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c53c90a720 con 0x55c53cc82ae0 2024-03-24T10:43:27.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.235+0000 7f8e33fff640 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55c53cc82ae0 0x55c53cc84ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.235+0000 7f8e13fff640 10 client.16485 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:43:27.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.235+0000 7f8e33fff640 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55c53cc82ae0 0x55c53cc84ef0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4362aaf27800cd51 server_cookie=5f7d52ef3babf047 in_seq=0 out_seq=0 2024-03-24T10:43:27.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e13fff640 1 -- 172.21.15.23:0/4040301774 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c53c90a720 con 0x55c53cc82ae0 2024-03-24T10:43:27.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e13fff640 10 client.16485 handle_command_reply: tid=0 2024-03-24T10:43:27.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e3d7d3740 1 client.16485 shutdown 2024-03-24T10:43:27.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/4040301774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc66ee0 msgr2=0x55c53cc6fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc66ee0 0x55c53cc6fac0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8e28004880 tx=0x7f8e28003880 comp rx=0 tx=0).stop 2024-03-24T10:43:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/4040301774 shutdown_connections 2024-03-24T10:43:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc72b00 0x55c53cc70540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55c53cc82ae0 0x55c53cc84ef0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc67890 0x55c53cc70000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/4040301774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc66ee0 0x55c53cc6fac0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e13fff640 0 client.16485 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:43:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/4040301774 >> 172.21.15.23:0/4040301774 conn(0x55c53cc61b20 msgr2=0x55c53cc63870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/4040301774 shutdown_connections 2024-03-24T10:43:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/4040301774 wait complete. 2024-03-24T10:43:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.237+0000 7f8e3d7d3740 20 client.16485 trim_cache size 0 max 16384 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.251+0000 7f8e3d7d3740 1 Processor -- start 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.251+0000 7f8e3d7d3740 1 -- start start 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.251+0000 7f8e3d7d3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc66ee0 0x55c53cc721f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.251+0000 7f8e3d7d3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc67890 0x55c53cc7e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.251+0000 7f8e3d7d3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc72b00 0x55c53cc8a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.251+0000 7f8e3d7d3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c53cc66d70 con 0x55c53cc72b00 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.251+0000 7f8e3d7d3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c53cc70a90 con 0x55c53cc66ee0 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.251+0000 7f8e3d7d3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c53cc5d210 con 0x55c53cc67890 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e390b0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc72b00 0x55c53cc8a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e390b0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc72b00 0x55c53cc8a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55216/0 (socket says 172.21.15.23:55216) 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e390b0640 1 -- 172.21.15.23:0/852069496 learned_addr learned my addr 172.21.15.23:0/852069496 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e388af640 1 --2- 172.21.15.23:0/852069496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc66ee0 0x55c53cc721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e33fff640 1 --2- 172.21.15.23:0/852069496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc67890 0x55c53cc7e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e390b0640 1 -- 172.21.15.23:0/852069496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc67890 msgr2=0x55c53cc7e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e390b0640 1 --2- 172.21.15.23:0/852069496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc67890 0x55c53cc7e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e390b0640 1 -- 172.21.15.23:0/852069496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc66ee0 msgr2=0x55c53cc721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e390b0640 1 --2- 172.21.15.23:0/852069496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc66ee0 0x55c53cc721f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e390b0640 1 -- 172.21.15.23:0/852069496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c53cb68c80 con 0x55c53cc72b00 2024-03-24T10:43:27.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e33fff640 1 --2- 172.21.15.23:0/852069496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc67890 0x55c53cc7e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:43:27.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e390b0640 1 --2- 172.21.15.23:0/852069496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc72b00 0x55c53cc8a7d0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c0cbe20 tx=0x7f8e2c0ce040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dbef6b65178357 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e1e7fc640 1 -- 172.21.15.23:0/852069496 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c0a3800 con 0x55c53cc72b00 2024-03-24T10:43:27.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e1e7fc640 1 -- 172.21.15.23:0/852069496 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e2c099360 con 0x55c53cc72b00 2024-03-24T10:43:27.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.252+0000 7f8e1e7fc640 1 -- 172.21.15.23:0/852069496 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c0a35d0 con 0x55c53cc72b00 2024-03-24T10:43:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.253+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/852069496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc72b00 msgr2=0x55c53cc8a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.253+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/852069496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc72b00 0x55c53cc8a7d0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c0cbe20 tx=0x7f8e2c0ce040 comp rx=0 tx=0).stop 2024-03-24T10:43:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.253+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/852069496 shutdown_connections 2024-03-24T10:43:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.253+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/852069496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc72b00 0x55c53cc8a7d0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.253+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/852069496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc67890 0x55c53cc7e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.253+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/852069496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc66ee0 0x55c53cc721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.253+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/852069496 >> 172.21.15.23:0/852069496 conn(0x55c53cc61b20 msgr2=0x55c53cc60cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.253+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/852069496 shutdown_connections 2024-03-24T10:43:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.253+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/852069496 wait complete. 2024-03-24T10:43:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.253+0000 7f8e3d7d3740 1 -- start start 2024-03-24T10:43:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.253+0000 7f8e3d7d3740 1 Processor -- start 2024-03-24T10:43:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e3d7d3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc66ee0 0x55c53cc76ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e3d7d3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc67890 0x55c53cc77000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e3d7d3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc72b00 0x55c53cce7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e3d7d3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c53cce8340 con 0x55c53cc67890 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e3d7d3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c53cce8480 con 0x55c53cc66ee0 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e3d7d3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c53cce85f0 con 0x55c53cc72b00 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e3d7d3740 10 client.0 _ll_register_callbacks cb 0x7f8e28005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e33fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc67890 0x55c53cc77000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e33fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc67890 0x55c53cc77000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55218/0 (socket says 172.21.15.23:55218) 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e33fff640 1 -- 172.21.15.23:0/3009837703 learned_addr learned my addr 172.21.15.23:0/3009837703 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e388af640 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc66ee0 0x55c53cc76ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e390b0640 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc72b00 0x55c53cce7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e33fff640 1 -- 172.21.15.23:0/3009837703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc72b00 msgr2=0x55c53cce7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e33fff640 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc72b00 0x55c53cce7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e33fff640 1 -- 172.21.15.23:0/3009837703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc66ee0 msgr2=0x55c53cc76ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e33fff640 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc66ee0 0x55c53cc76ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e33fff640 1 -- 172.21.15.23:0/3009837703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e341a3960 con 0x55c53cc67890 2024-03-24T10:43:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e390b0640 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc72b00 0x55c53cce7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e388af640 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc66ee0 0x55c53cc76ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e33fff640 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc67890 0x55c53cc77000 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8e2002c240 tx=0x7f8e2002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7925f5337173ffe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.254+0000 7f8e13fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e13fff640 1 -- 172.21.15.23:0/3009837703 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2005a070 con 0x55c53cc67890 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e13fff640 1 -- 172.21.15.23:0/3009837703 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f8e200053c0 con 0x55c53cc67890 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/3009837703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e28005500 con 0x55c53cc67890 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e13fff640 1 -- 172.21.15.23:0/3009837703 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8e20048070 con 0x55c53cc67890 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e3d7d3740 10 client.16491 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/3009837703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c53cb8e920 con 0x55c53cc67890 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e13fff640 1 -- 172.21.15.23:0/3009837703 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e28005500 con 0x55c53cc67890 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e13fff640 1 -- 172.21.15.23:0/3009837703 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8e2005d020 con 0x55c53cc67890 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e3d7d3740 10 client.16491 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e3d7d3740 10 client.16491 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e3d7d3740 20 client.16491 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55c53cce9ed0 0x55c53ccec2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e3d7d3740 4 client.16491 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:43:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/3009837703 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c53c959890 con 0x55c53cce9ed0 2024-03-24T10:43:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.255+0000 7f8e388af640 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55c53cce9ed0 0x55c53ccec2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.256+0000 7f8e13fff640 10 client.16491 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:43:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.256+0000 7f8e388af640 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55c53cce9ed0 0x55c53ccec2f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8604d8c338558b6 server_cookie=7bf150985003ad8 in_seq=0 out_seq=0 2024-03-24T10:43:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.256+0000 7f8e13fff640 1 -- 172.21.15.23:0/3009837703 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c53c959890 con 0x55c53cce9ed0 2024-03-24T10:43:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.256+0000 7f8e13fff640 10 client.16491 handle_command_reply: tid=0 2024-03-24T10:43:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.256+0000 7f8e3d7d3740 1 client.16491 shutdown 2024-03-24T10:43:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.256+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/3009837703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc67890 msgr2=0x55c53cc77000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.256+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc67890 0x55c53cc77000 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8e2002c240 tx=0x7f8e2002bc30 comp rx=0 tx=0).stop 2024-03-24T10:43:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.256+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/3009837703 shutdown_connections 2024-03-24T10:43:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.256+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c53cc72b00 0x55c53cce7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.256+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c53cc67890 0x55c53cc77000 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.256+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55c53cce9ed0 0x55c53ccec2f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.256+0000 7f8e3d7d3740 1 --2- 172.21.15.23:0/3009837703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c53cc66ee0 0x55c53cc76ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.256+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/3009837703 >> 172.21.15.23:0/3009837703 conn(0x55c53cc61b20 msgr2=0x55c53cc6f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.256+0000 7f8e13fff640 0 client.16491 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:43:27.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/3009837703 shutdown_connections 2024-03-24T10:43:27.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3d7d3740 1 -- 172.21.15.23:0/3009837703 wait complete. 2024-03-24T10:43:27.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3d7d3740 20 client.16491 trim_cache size 0 max 16384 2024-03-24T10:43:27.263 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:43:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3ab13640 1 -- 172.21.15.23:0/25290964 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8e240a2230 msgr2=0x7f8e240a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3ab13640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8e240a2230 0x7f8e240a46f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8e20005ec0 tx=0x7f8e2002c770 comp rx=0 tx=0).stop 2024-03-24T10:43:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3ab13640 1 -- 172.21.15.23:0/25290964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e34106660 msgr2=0x7f8e34196a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3ab13640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e34106660 0x7f8e34196a70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c00eb10 tx=0x7f8e2c00c2d0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3ab13640 1 -- 172.21.15.23:0/25290964 shutdown_connections 2024-03-24T10:43:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3ab13640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8e240a2230 0x7f8e240a46f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3ab13640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e34106660 0x7f8e34196a70 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3ab13640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e34100f60 0x7f8e3419ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3ab13640 1 --2- 172.21.15.23:0/25290964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e341005a0 0x7f8e3419c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3ab13640 1 -- 172.21.15.23:0/25290964 >> 172.21.15.23:0/25290964 conn(0x7f8e34077710 msgr2=0x7f8e340ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3ab13640 1 -- 172.21.15.23:0/25290964 shutdown_connections 2024-03-24T10:43:27.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.257+0000 7f8e3ab13640 1 -- 172.21.15.23:0/25290964 wait complete. 2024-03-24T10:43:27.268 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:43:27.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.379+0000 7f0d2b725640 1 -- 172.21.15.23:0/3492506474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d241005c0 msgr2=0x7f0d24100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.379+0000 7f0d2b725640 1 --2- 172.21.15.23:0/3492506474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d241005c0 0x7f0d24100a40 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009900 tx=0x7f0d1402bc30 comp rx=0 tx=0).stop 2024-03-24T10:43:27.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.379+0000 7f0d2b725640 1 -- 172.21.15.23:0/3492506474 shutdown_connections 2024-03-24T10:43:27.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.379+0000 7f0d2b725640 1 --2- 172.21.15.23:0/3492506474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d24100f80 0x7f0d2410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.379+0000 7f0d2b725640 1 --2- 172.21.15.23:0/3492506474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d241005c0 0x7f0d24100a40 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.379+0000 7f0d2b725640 1 --2- 172.21.15.23:0/3492506474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d24106680 0x7f0d24106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.379+0000 7f0d2b725640 1 -- 172.21.15.23:0/3492506474 >> 172.21.15.23:0/3492506474 conn(0x7f0d240776e0 msgr2=0x7f0d24077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.379+0000 7f0d2b725640 1 -- 172.21.15.23:0/3492506474 shutdown_connections 2024-03-24T10:43:27.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.379+0000 7f0d2b725640 1 -- 172.21.15.23:0/3492506474 wait complete. 2024-03-24T10:43:27.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.380+0000 7f0d2b725640 1 Processor -- start 2024-03-24T10:43:27.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.380+0000 7f0d2b725640 1 -- start start 2024-03-24T10:43:27.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.380+0000 7f0d2b725640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d241005c0 0x7f0d2410c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.380+0000 7f0d2b725640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d24100f80 0x7f0d2410c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.380+0000 7f0d2b725640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d24106680 0x7f0d241a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.380+0000 7f0d2b725640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0d241a31a0 con 0x7f0d24106680 2024-03-24T10:43:27.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.380+0000 7f0d2b725640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0d241a3310 con 0x7f0d24100f80 2024-03-24T10:43:27.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.380+0000 7f0d2b725640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0d241a34d0 con 0x7f0d241005c0 2024-03-24T10:43:27.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.380+0000 7f0d29cc2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d24106680 0x7f0d241a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.380+0000 7f0d29cc2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d24106680 0x7f0d241a2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55240/0 (socket says 172.21.15.23:55240) 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.380+0000 7f0d29cc2640 1 -- 172.21.15.23:0/570741418 learned_addr learned my addr 172.21.15.23:0/570741418 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.380+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/570741418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d24100f80 0x7f0d2410c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.380+0000 7f0d294c1640 1 --2- 172.21.15.23:0/570741418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d241005c0 0x7f0d2410c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.381+0000 7f0d29cc2640 1 -- 172.21.15.23:0/570741418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d241005c0 msgr2=0x7f0d2410c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.381+0000 7f0d29cc2640 1 --2- 172.21.15.23:0/570741418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d241005c0 0x7f0d2410c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.381+0000 7f0d29cc2640 1 -- 172.21.15.23:0/570741418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d24100f80 msgr2=0x7f0d2410c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.381+0000 7f0d29cc2640 1 --2- 172.21.15.23:0/570741418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d24100f80 0x7f0d2410c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.381+0000 7f0d29cc2640 1 -- 172.21.15.23:0/570741418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d14009560 con 0x7f0d24106680 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.381+0000 7f0d29cc2640 1 --2- 172.21.15.23:0/570741418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d24106680 0x7f0d241a2ba0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00eb10 tx=0x7f0d1c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.381+0000 7f0d1a7fc640 1 -- 172.21.15.23:0/570741418 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c004b20 con 0x7f0d24106680 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.381+0000 7f0d1a7fc640 1 -- 172.21.15.23:0/570741418 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d1c004c80 con 0x7f0d24106680 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.381+0000 7f0d2b725640 1 -- 172.21.15.23:0/570741418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d241a37b0 con 0x7f0d24106680 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.381+0000 7f0d1a7fc640 1 -- 172.21.15.23:0/570741418 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c010650 con 0x7f0d24106680 2024-03-24T10:43:27.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.381+0000 7f0d2b725640 1 -- 172.21.15.23:0/570741418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d241ab070 con 0x7f0d24106680 2024-03-24T10:43:27.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.383+0000 7f0d1a7fc640 1 -- 172.21.15.23:0/570741418 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f0d1c005020 con 0x7f0d24106680 2024-03-24T10:43:27.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d2e3dd740 1 Processor -- start 2024-03-24T10:43:27.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d2e3dd740 1 -- start start 2024-03-24T10:43:27.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d2e3dd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d2137463f0 0x55d2137467d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d2e3dd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d213746d10 0x55d21374f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d2e3dd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d21374f8f0 0x55d213751ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d2e3dd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d2136b4840 con 0x55d213746d10 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d2e3dd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d2136b1d70 con 0x55d21374f8f0 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d2e3dd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d2136ad550 con 0x55d2137463f0 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d1a7fc640 1 --2- 172.21.15.23:0/570741418 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0d100a2220 0x7f0d100a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d28cc0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d2137463f0 0x55d2137467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d28cc0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d2137463f0 0x55d2137467d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:47584/0 (socket says 172.21.15.23:47584) 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d28cc0640 1 -- 172.21.15.23:0/967363112 learned_addr learned my addr 172.21.15.23:0/967363112 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d294c1640 1 --2- 172.21.15.23:0/967363112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d213746d10 0x55d21374f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d29cc2640 1 --2- 172.21.15.23:0/570741418 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0d100a2220 0x7f0d100a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/967363112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d21374f8f0 0x55d213751ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d28cc0640 1 -- 172.21.15.23:0/967363112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d21374f8f0 msgr2=0x55d213751ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/967363112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d21374f8f0 0x55d213751ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d28cc0640 1 -- 172.21.15.23:0/967363112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d213746d10 msgr2=0x55d21374f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/967363112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d213746d10 0x55d21374f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.384+0000 7f0d28cc0640 1 -- 172.21.15.23:0/967363112 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d213642de0 con 0x55d2137463f0 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.385+0000 7f0d1a7fc640 1 -- 172.21.15.23:0/570741418 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0d1c014070 con 0x7f0d24106680 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.385+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/967363112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d2137463f0 0x55d2137467d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009900 tx=0x7f0d14005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b27f99b6e496904 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.385+0000 7f0d29cc2640 1 --2- 172.21.15.23:0/570741418 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0d100a2220 0x7f0d100a46e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c09ad40 tx=0x7f0d1c0c5700 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.385+0000 7f0d0f7fe640 1 -- 172.21.15.23:0/967363112 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d140046c0 con 0x55d2137463f0 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.385+0000 7f0d0f7fe640 1 -- 172.21.15.23:0/967363112 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d14004510 con 0x55d2137463f0 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.385+0000 7f0d0f7fe640 1 -- 172.21.15.23:0/967363112 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d140051e0 con 0x55d2137463f0 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.386+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/967363112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d2137463f0 msgr2=0x55d2137467d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.386+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/967363112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d2137463f0 0x55d2137467d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009900 tx=0x7f0d14005b50 comp rx=0 tx=0).stop 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.386+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/967363112 shutdown_connections 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.386+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/967363112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d21374f8f0 0x55d213751ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.386+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/967363112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d213746d10 0x55d21374f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.386+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/967363112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d2137463f0 0x55d2137467d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.386+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/967363112 >> 172.21.15.23:0/967363112 conn(0x55d213740b20 msgr2=0x55d213740f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.386+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/967363112 shutdown_connections 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.386+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/967363112 wait complete. 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.386+0000 7f0d2e3dd740 1 -- start start 2024-03-24T10:43:27.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.386+0000 7f0d2e3dd740 1 Processor -- start 2024-03-24T10:43:27.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d2e3dd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2137463f0 0x55d21374c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d2e3dd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d213746d10 0x55d21374cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d2e3dd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d21374f8f0 0x55d21374d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d2e3dd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d213380430 con 0x55d2137463f0 2024-03-24T10:43:27.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d2e3dd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d21333ce00 con 0x55d21374f8f0 2024-03-24T10:43:27.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d2e3dd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d213632590 con 0x55d213746d10 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d2e3dd740 10 client.0 _ll_register_callbacks cb 0x55d2136a1fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d28cc0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2137463f0 0x55d21374c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d28cc0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2137463f0 0x55d21374c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55256/0 (socket says 172.21.15.23:55256) 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d28cc0640 1 -- 172.21.15.23:0/1407748657 learned_addr learned my addr 172.21.15.23:0/1407748657 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d294c1640 1 --2- 172.21.15.23:0/1407748657 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d213746d10 0x55d21374cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/1407748657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d21374f8f0 0x55d21374d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d28cc0640 1 -- 172.21.15.23:0/1407748657 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d213746d10 msgr2=0x55d21374cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/1407748657 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d213746d10 0x55d21374cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d28cc0640 1 -- 172.21.15.23:0/1407748657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d21374f8f0 msgr2=0x55d21374d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/1407748657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d21374f8f0 0x55d21374d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d28cc0640 1 -- 172.21.15.23:0/1407748657 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d2410dd40 con 0x55d2137463f0 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/1407748657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2137463f0 0x55d21374c960 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f0d14004f60 tx=0x7f0d14005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e8557b3ae9fef8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d0cff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d0cff9640 1 -- 172.21.15.23:0/1407748657 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d14005370 con 0x55d2137463f0 2024-03-24T10:43:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d0cff9640 1 -- 172.21.15.23:0/1407748657 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0d140054d0 con 0x55d2137463f0 2024-03-24T10:43:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.387+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/1407748657 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d2133ac7e0 con 0x55d2137463f0 2024-03-24T10:43:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.388+0000 7f0d0cff9640 1 -- 172.21.15.23:0/1407748657 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0d14005690 con 0x55d2137463f0 2024-03-24T10:43:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.388+0000 7f0d2e3dd740 10 client.16503 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.388+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/1407748657 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d2136abbd0 con 0x55d2137463f0 2024-03-24T10:43:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.388+0000 7f0d0cff9640 1 -- 172.21.15.23:0/1407748657 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d2133ac7e0 con 0x55d2137463f0 2024-03-24T10:43:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.388+0000 7f0d0cff9640 1 -- 172.21.15.23:0/1407748657 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f0d1405a4c0 con 0x55d2137463f0 2024-03-24T10:43:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.388+0000 7f0d2e3dd740 10 client.16503 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.388+0000 7f0d2e3dd740 10 client.16503 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:43:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.388+0000 7f0d2e3dd740 20 client.16503 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.388+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/1407748657 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d21375f660 0x55d213761a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.388+0000 7f0d2e3dd740 4 client.16503 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.388+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/1407748657 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2133e9720 con 0x55d21375f660 2024-03-24T10:43:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.388+0000 7f0d294c1640 1 --2- 172.21.15.23:0/1407748657 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d21375f660 0x55d213761a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.389+0000 7f0d294c1640 1 --2- 172.21.15.23:0/1407748657 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d21375f660 0x55d213761a50 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ce12171fd313731e server_cookie=75767cbf9617ce96 in_seq=0 out_seq=0 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.389+0000 7f0d0cff9640 10 client.16503 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.389+0000 7f0d0cff9640 1 -- 172.21.15.23:0/1407748657 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d2133e9720 con 0x55d21375f660 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.389+0000 7f0d0cff9640 10 client.16503 handle_command_reply: tid=0 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.389+0000 7f0d2e3dd740 1 client.16503 shutdown 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.389+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/1407748657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2137463f0 msgr2=0x55d21374c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.389+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/1407748657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2137463f0 0x55d21374c960 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f0d14004f60 tx=0x7f0d14005060 comp rx=0 tx=0).stop 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.389+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/1407748657 shutdown_connections 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.389+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/1407748657 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d21375f660 0x55d213761a50 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.389+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/1407748657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d21374f8f0 0x55d21374d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.389+0000 7f0d0cff9640 0 client.16503 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.389+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/1407748657 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d213746d10 0x55d21374cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.389+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/1407748657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2137463f0 0x55d21374c960 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.389+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/1407748657 >> 172.21.15.23:0/1407748657 conn(0x55d213740b20 msgr2=0x55d213742870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.390+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/1407748657 shutdown_connections 2024-03-24T10:43:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.390+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/1407748657 wait complete. 2024-03-24T10:43:27.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.390+0000 7f0d2e3dd740 20 client.16503 trim_cache size 0 max 16384 2024-03-24T10:43:27.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.404+0000 7f0d2e3dd740 1 Processor -- start 2024-03-24T10:43:27.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.404+0000 7f0d2e3dd740 1 -- start start 2024-03-24T10:43:27.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.404+0000 7f0d2e3dd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d2137463f0 0x55d21374f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.404+0000 7f0d2e3dd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d213746d10 0x55d21375b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.404+0000 7f0d2e3dd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d21374f8f0 0x55d2137672e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.404+0000 7f0d2e3dd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d21375c460 con 0x55d21374f8f0 2024-03-24T10:43:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.404+0000 7f0d2e3dd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d213740650 con 0x55d213746d10 2024-03-24T10:43:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.404+0000 7f0d2e3dd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d21373c1f0 con 0x55d2137463f0 2024-03-24T10:43:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.404+0000 7f0d28cc0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d21374f8f0 0x55d2137672e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.404+0000 7f0d294c1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d213746d10 0x55d21375b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.404+0000 7f0d28cc0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d2137463f0 0x55d21374f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.404+0000 7f0d294c1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d213746d10 0x55d21375b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:54770/0 (socket says 172.21.15.23:54770) 2024-03-24T10:43:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.404+0000 7f0d294c1640 1 -- 172.21.15.23:0/215052810 learned_addr learned my addr 172.21.15.23:0/215052810 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.404+0000 7f0d28cc0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d2137463f0 0x55d21374f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:47600/0 (socket says 172.21.15.23:47600) 2024-03-24T10:43:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.405+0000 7f0d28cc0640 1 -- 172.21.15.23:0/215052810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d213746d10 msgr2=0x55d21375b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.405+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/215052810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d213746d10 0x55d21375b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.405+0000 7f0d28cc0640 1 -- 172.21.15.23:0/215052810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d21374f8f0 msgr2=0x55d2137672e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.405+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/215052810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d21374f8f0 0x55d2137672e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.405+0000 7f0d28cc0640 1 -- 172.21.15.23:0/215052810 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d213647c80 con 0x55d2137463f0 2024-03-24T10:43:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.405+0000 7f0d294c1640 1 --2- 172.21.15.23:0/215052810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d213746d10 0x55d21375b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.405+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/215052810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d2137463f0 0x55d21374f090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0d14001eb0 tx=0x7f0d1402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=576746fa1d2dd284 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.405+0000 7f0d0effd640 1 -- 172.21.15.23:0/215052810 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d14005d50 con 0x55d2137463f0 2024-03-24T10:43:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.405+0000 7f0d0effd640 1 -- 172.21.15.23:0/215052810 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d1405ab70 con 0x55d2137463f0 2024-03-24T10:43:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.405+0000 7f0d0effd640 1 -- 172.21.15.23:0/215052810 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1405a430 con 0x55d2137463f0 2024-03-24T10:43:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.406+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/215052810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d2137463f0 msgr2=0x55d21374f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.406+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/215052810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d2137463f0 0x55d21374f090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0d14001eb0 tx=0x7f0d1402bc30 comp rx=0 tx=0).stop 2024-03-24T10:43:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.406+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/215052810 shutdown_connections 2024-03-24T10:43:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.406+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/215052810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d21374f8f0 0x55d2137672e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.406+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/215052810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d213746d10 0x55d21375b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.406+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/215052810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d2137463f0 0x55d21374f090 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.406+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/215052810 >> 172.21.15.23:0/215052810 conn(0x55d213740b20 msgr2=0x55d2137470f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.406+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/215052810 shutdown_connections 2024-03-24T10:43:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.406+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/215052810 wait complete. 2024-03-24T10:43:27.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.406+0000 7f0d2e3dd740 1 -- start start 2024-03-24T10:43:27.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.406+0000 7f0d2e3dd740 1 Processor -- start 2024-03-24T10:43:27.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d2e3dd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2137463f0 0x55d213757780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d2e3dd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d213746d10 0x55d213752b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d2e3dd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d21374f8f0 0x55d213753060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d2e3dd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d2137535a0 con 0x55d21374f8f0 2024-03-24T10:43:27.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d2e3dd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d213753710 con 0x55d2137463f0 2024-03-24T10:43:27.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d2e3dd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d2137538d0 con 0x55d213746d10 2024-03-24T10:43:27.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d2e3dd740 10 client.0 _ll_register_callbacks cb 0x7f0d20009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:27.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d28cc0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d21374f8f0 0x55d213753060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d28cc0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d21374f8f0 0x55d213753060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55274/0 (socket says 172.21.15.23:55274) 2024-03-24T10:43:27.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d294c1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d213746d10 0x55d213752b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d28cc0640 1 -- 172.21.15.23:0/4270377244 learned_addr learned my addr 172.21.15.23:0/4270377244 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/4270377244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2137463f0 0x55d213757780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d28cc0640 1 -- 172.21.15.23:0/4270377244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d213746d10 msgr2=0x55d213752b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/4270377244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d213746d10 0x55d213752b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d28cc0640 1 -- 172.21.15.23:0/4270377244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2137463f0 msgr2=0x55d213757780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/4270377244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2137463f0 0x55d213757780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d28cc0640 1 -- 172.21.15.23:0/4270377244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d20009b00 con 0x55d21374f8f0 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d28cc0640 1 --2- 172.21.15.23:0/4270377244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d21374f8f0 0x55d213753060 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f0d1405a800 tx=0x7f0d1405a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea4bcbbc8347c15 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d0cff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d0cff9640 1 -- 172.21.15.23:0/4270377244 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d14005d50 con 0x55d21374f8f0 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d0cff9640 1 -- 172.21.15.23:0/4270377244 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0d14005370 con 0x55d21374f8f0 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.407+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/4270377244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d14004510 con 0x55d21374f8f0 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.408+0000 7f0d0cff9640 1 -- 172.21.15.23:0/4270377244 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0d14005530 con 0x55d21374f8f0 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.408+0000 7f0d2e3dd740 10 client.16509 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:27.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.408+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/4270377244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d2136abbd0 con 0x55d21374f8f0 2024-03-24T10:43:27.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.408+0000 7f0d0cff9640 1 -- 172.21.15.23:0/4270377244 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d14004510 con 0x55d21374f8f0 2024-03-24T10:43:27.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.408+0000 7f0d0cff9640 1 -- 172.21.15.23:0/4270377244 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f0d14061c90 con 0x55d21374f8f0 2024-03-24T10:43:27.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.408+0000 7f0d2e3dd740 10 client.16509 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:27.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.408+0000 7f0d2e3dd740 10 client.16509 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:43:27.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.408+0000 7f0d2e3dd740 20 client.16509 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:27.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.408+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/4270377244 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d2137c59f0 0x55d2137c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.408+0000 7f0d2e3dd740 4 client.16509 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:43:27.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.408+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/4270377244 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d213438890 con 0x55d2137c59f0 2024-03-24T10:43:27.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.408+0000 7f0d294c1640 1 --2- 172.21.15.23:0/4270377244 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d2137c59f0 0x55d2137c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d294c1640 1 --2- 172.21.15.23:0/4270377244 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d2137c59f0 0x55d2137c7e10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=331683a9c9e2c4d8 server_cookie=f4b1840191e0ba10 in_seq=0 out_seq=0 2024-03-24T10:43:27.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d0cff9640 10 client.16509 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:43:27.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d0cff9640 1 -- 172.21.15.23:0/4270377244 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d213438890 con 0x55d2137c59f0 2024-03-24T10:43:27.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d0cff9640 10 client.16509 handle_command_reply: tid=0 2024-03-24T10:43:27.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d2e3dd740 1 client.16509 shutdown 2024-03-24T10:43:27.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/4270377244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d21374f8f0 msgr2=0x55d213753060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/4270377244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d21374f8f0 0x55d213753060 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f0d1405a800 tx=0x7f0d1405a210 comp rx=0 tx=0).stop 2024-03-24T10:43:27.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/4270377244 shutdown_connections 2024-03-24T10:43:27.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/4270377244 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55d2137c59f0 0x55d2137c7e10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/4270377244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d21374f8f0 0x55d213753060 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d0cff9640 0 client.16509 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:43:27.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/4270377244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d213746d10 0x55d213752b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d2e3dd740 1 --2- 172.21.15.23:0/4270377244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2137463f0 0x55d213757780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/4270377244 >> 172.21.15.23:0/4270377244 conn(0x55d213740b20 msgr2=0x55d21373f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/4270377244 shutdown_connections 2024-03-24T10:43:27.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d2e3dd740 1 -- 172.21.15.23:0/4270377244 wait complete. 2024-03-24T10:43:27.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.409+0000 7f0d2e3dd740 20 client.16509 trim_cache size 0 max 16384 2024-03-24T10:43:27.416 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:43:27.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.410+0000 7f0d2b725640 1 -- 172.21.15.23:0/570741418 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0d100a2220 msgr2=0x7f0d100a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.410+0000 7f0d2b725640 1 --2- 172.21.15.23:0/570741418 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0d100a2220 0x7f0d100a46e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c09ad40 tx=0x7f0d1c0c5700 comp rx=0 tx=0).stop 2024-03-24T10:43:27.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.410+0000 7f0d2b725640 1 -- 172.21.15.23:0/570741418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d24106680 msgr2=0x7f0d241a2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.410+0000 7f0d2b725640 1 --2- 172.21.15.23:0/570741418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d24106680 0x7f0d241a2ba0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00eb10 tx=0x7f0d1c00c2d0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.410+0000 7f0d2b725640 1 -- 172.21.15.23:0/570741418 shutdown_connections 2024-03-24T10:43:27.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.410+0000 7f0d2b725640 1 --2- 172.21.15.23:0/570741418 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0d100a2220 0x7f0d100a46e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.410+0000 7f0d2b725640 1 --2- 172.21.15.23:0/570741418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d24106680 0x7f0d241a2ba0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.410+0000 7f0d2b725640 1 --2- 172.21.15.23:0/570741418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d24100f80 0x7f0d2410c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.410+0000 7f0d2b725640 1 --2- 172.21.15.23:0/570741418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d241005c0 0x7f0d2410c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.410+0000 7f0d2b725640 1 -- 172.21.15.23:0/570741418 >> 172.21.15.23:0/570741418 conn(0x7f0d240776e0 msgr2=0x7f0d240feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.410+0000 7f0d2b725640 1 -- 172.21.15.23:0/570741418 shutdown_connections 2024-03-24T10:43:27.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.410+0000 7f0d2b725640 1 -- 172.21.15.23:0/570741418 wait complete. 2024-03-24T10:43:27.421 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:43:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.527+0000 7f7e7c525640 1 -- 172.21.15.23:0/4087756495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e74100f80 msgr2=0x7f7e7410d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.527+0000 7f7e7c525640 1 --2- 172.21.15.23:0/4087756495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e74100f80 0x7f7e7410d490 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009cd0 tx=0x7f7e6c02c0e0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.527+0000 7f7e7c525640 1 -- 172.21.15.23:0/4087756495 shutdown_connections 2024-03-24T10:43:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.527+0000 7f7e7c525640 1 --2- 172.21.15.23:0/4087756495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e74100f80 0x7f7e7410d490 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.527+0000 7f7e7c525640 1 --2- 172.21.15.23:0/4087756495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e741005c0 0x7f7e74100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.527+0000 7f7e7c525640 1 --2- 172.21.15.23:0/4087756495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e74106660 0x7f7e74106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.527+0000 7f7e7c525640 1 -- 172.21.15.23:0/4087756495 >> 172.21.15.23:0/4087756495 conn(0x7f7e74076990 msgr2=0x7f7e74076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.527+0000 7f7e7c525640 1 -- 172.21.15.23:0/4087756495 shutdown_connections 2024-03-24T10:43:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.527+0000 7f7e7c525640 1 -- 172.21.15.23:0/4087756495 wait complete. 2024-03-24T10:43:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.528+0000 7f7e7c525640 1 Processor -- start 2024-03-24T10:43:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.528+0000 7f7e7c525640 1 -- start start 2024-03-24T10:43:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.528+0000 7f7e7c525640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e741005c0 0x7f7e74198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.528+0000 7f7e7c525640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e74100f80 0x7f7e74199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.528+0000 7f7e7c525640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e74106660 0x7f7e741a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.528+0000 7f7e7c525640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741998f0 con 0x7f7e741005c0 2024-03-24T10:43:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.528+0000 7f7e7c525640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741a40f0 con 0x7f7e74100f80 2024-03-24T10:43:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.528+0000 7f7e7c525640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741a4230 con 0x7f7e74106660 2024-03-24T10:43:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.528+0000 7f7e7a2c1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e741005c0 0x7f7e74198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.528+0000 7f7e7a2c1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e741005c0 0x7f7e74198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55292/0 (socket says 172.21.15.23:55292) 2024-03-24T10:43:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.528+0000 7f7e7a2c1640 1 -- 172.21.15.23:0/417389459 learned_addr learned my addr 172.21.15.23:0/417389459 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.528+0000 7f7e79ac0640 1 --2- 172.21.15.23:0/417389459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e74100f80 0x7f7e74199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.528+0000 7f7e7aac2640 1 --2- 172.21.15.23:0/417389459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e74106660 0x7f7e741a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.529+0000 7f7e7a2c1640 1 -- 172.21.15.23:0/417389459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e74106660 msgr2=0x7f7e741a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.529+0000 7f7e7a2c1640 1 --2- 172.21.15.23:0/417389459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e74106660 0x7f7e741a3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.529+0000 7f7e7a2c1640 1 -- 172.21.15.23:0/417389459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e74100f80 msgr2=0x7f7e74199070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.529+0000 7f7e7a2c1640 1 --2- 172.21.15.23:0/417389459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e74100f80 0x7f7e74199070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.529+0000 7f7e7a2c1640 1 -- 172.21.15.23:0/417389459 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e70009490 con 0x7f7e741005c0 2024-03-24T10:43:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.529+0000 7f7e7a2c1640 1 --2- 172.21.15.23:0/417389459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e741005c0 0x7f7e74198b30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000e9f0 tx=0x7f7e7000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.529+0000 7f7e6b7fe640 1 -- 172.21.15.23:0/417389459 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7000cd80 con 0x7f7e741005c0 2024-03-24T10:43:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.529+0000 7f7e6b7fe640 1 -- 172.21.15.23:0/417389459 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e70010910 con 0x7f7e741005c0 2024-03-24T10:43:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.529+0000 7f7e7c525640 1 -- 172.21.15.23:0/417389459 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e6c009900 con 0x7f7e741005c0 2024-03-24T10:43:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.529+0000 7f7e6b7fe640 1 -- 172.21.15.23:0/417389459 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e70018950 con 0x7f7e741005c0 2024-03-24T10:43:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.529+0000 7f7e7c525640 1 -- 172.21.15.23:0/417389459 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e741a4860 con 0x7f7e741005c0 2024-03-24T10:43:27.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.531+0000 7f7e6b7fe640 1 -- 172.21.15.23:0/417389459 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f7e70010a80 con 0x7f7e741005c0 2024-03-24T10:43:27.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.531+0000 7f7e7f1e8740 1 Processor -- start 2024-03-24T10:43:27.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e7f1e8740 1 -- start start 2024-03-24T10:43:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e7f1e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 0x55ced48617d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e7f1e8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced486a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e7f1e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced486a8f0 0x55ced486cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e7f1e8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ced47cf840 con 0x55ced4861d10 2024-03-24T10:43:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e7f1e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ced47ccd70 con 0x55ced486a8f0 2024-03-24T10:43:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e7f1e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ced47c8550 con 0x55ced48613f0 2024-03-24T10:43:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e6b7fe640 1 --2- 172.21.15.23:0/417389459 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7e600a22b0 0x7f7e600a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e7a2c1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced486a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e79ac0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced486a8f0 0x55ced486cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e79ac0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced486a8f0 0x55ced486cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:54808/0 (socket says 172.21.15.23:54808) 2024-03-24T10:43:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e7a2c1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced486a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55298/0 (socket says 172.21.15.23:55298) 2024-03-24T10:43:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e79ac0640 1 -- 172.21.15.23:0/4088028982 learned_addr learned my addr 172.21.15.23:0/4088028982 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e7aac2640 1 --2- 172.21.15.23:0/4088028982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 0x55ced48617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e7aac2640 1 --2- 172.21.15.23:0/417389459 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7e600a22b0 0x7f7e600a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e6b7fe640 1 -- 172.21.15.23:0/417389459 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f7e70014070 con 0x7f7e741005c0 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e79ac0640 1 -- 172.21.15.23:0/4088028982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 msgr2=0x55ced48617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e79ac0640 1 --2- 172.21.15.23:0/4088028982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 0x55ced48617d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e79ac0640 1 -- 172.21.15.23:0/4088028982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 msgr2=0x55ced486a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e79ac0640 1 --2- 172.21.15.23:0/4088028982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced486a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.532+0000 7f7e79ac0640 1 -- 172.21.15.23:0/4088028982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ced475dde0 con 0x55ced486a8f0 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.533+0000 7f7e79ac0640 1 --2- 172.21.15.23:0/4088028982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced486a8f0 0x55ced486cce0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009ca0 tx=0x7f7e64011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3dd70b24eb1b761 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.533+0000 7f7e7aac2640 1 --2- 172.21.15.23:0/417389459 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7e600a22b0 0x7f7e600a4770 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c005cc0 tx=0x7f7e6c005c50 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.533+0000 7f7e68bfe640 1 -- 172.21.15.23:0/4088028982 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e64019630 con 0x55ced486a8f0 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.533+0000 7f7e68bfe640 1 -- 172.21.15.23:0/4088028982 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e64009e50 con 0x55ced486a8f0 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.533+0000 7f7e68bfe640 1 -- 172.21.15.23:0/4088028982 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400ea80 con 0x55ced486a8f0 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/4088028982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced486a8f0 msgr2=0x55ced486cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/4088028982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced486a8f0 0x55ced486cce0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009ca0 tx=0x7f7e64011810 comp rx=0 tx=0).stop 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/4088028982 shutdown_connections 2024-03-24T10:43:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/4088028982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced486a8f0 0x55ced486cce0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/4088028982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced486a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/4088028982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 0x55ced48617d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/4088028982 >> 172.21.15.23:0/4088028982 conn(0x55ced485bb20 msgr2=0x55ced485bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/4088028982 shutdown_connections 2024-03-24T10:43:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/4088028982 wait complete. 2024-03-24T10:43:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 -- start start 2024-03-24T10:43:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 Processor -- start 2024-03-24T10:43:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 0x55ced4867900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced4867e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced486a8f0 0x55ced4868380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ced449b430 con 0x55ced4861d10 2024-03-24T10:43:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ced4457e00 con 0x55ced486a8f0 2024-03-24T10:43:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ced474d590 con 0x55ced48613f0 2024-03-24T10:43:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.534+0000 7f7e7f1e8740 10 client.0 _ll_register_callbacks cb 0x55ced47bcfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e79ac0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced4867e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e79ac0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced4867e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55302/0 (socket says 172.21.15.23:55302) 2024-03-24T10:43:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e79ac0640 1 -- 172.21.15.23:0/1760781387 learned_addr learned my addr 172.21.15.23:0/1760781387 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e7aac2640 1 --2- 172.21.15.23:0/1760781387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced486a8f0 0x55ced4868380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e7a2c1640 1 --2- 172.21.15.23:0/1760781387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 0x55ced4867900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e79ac0640 1 -- 172.21.15.23:0/1760781387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 msgr2=0x55ced4867900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e79ac0640 1 --2- 172.21.15.23:0/1760781387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 0x55ced4867900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e79ac0640 1 -- 172.21.15.23:0/1760781387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced486a8f0 msgr2=0x55ced4868380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e79ac0640 1 --2- 172.21.15.23:0/1760781387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced486a8f0 0x55ced4868380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e79ac0640 1 -- 172.21.15.23:0/1760781387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e640098d0 con 0x55ced4861d10 2024-03-24T10:43:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e79ac0640 1 --2- 172.21.15.23:0/1760781387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced4867e40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009c70 tx=0x7f7e64005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50b23f998a13483e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e5dffb640 1 -- 172.21.15.23:0/1760781387 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400e430 con 0x55ced4861d10 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e5dffb640 1 -- 172.21.15.23:0/1760781387 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f7e6400ec10 con 0x55ced4861d10 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/1760781387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ced44c77e0 con 0x55ced4861d10 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e5dffb640 1 -- 172.21.15.23:0/1760781387 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7e6400edd0 con 0x55ced4861d10 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e7f1e8740 10 client.16521 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/1760781387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ced47c6bd0 con 0x55ced4861d10 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.535+0000 7f7e5dffb640 1 -- 172.21.15.23:0/1760781387 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ced44c77e0 con 0x55ced4861d10 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.536+0000 7f7e5dffb640 1 -- 172.21.15.23:0/1760781387 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7e64019920 con 0x55ced4861d10 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.536+0000 7f7e7f1e8740 10 client.16521 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.536+0000 7f7e7f1e8740 10 client.16521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.536+0000 7f7e7f1e8740 20 client.16521 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.536+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/1760781387 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ced487a970 0x55ced487cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.536+0000 7f7e7f1e8740 4 client.16521 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.536+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/1760781387 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ced4504720 con 0x55ced487a970 2024-03-24T10:43:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.536+0000 7f7e7a2c1640 1 --2- 172.21.15.23:0/1760781387 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ced487a970 0x55ced487cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.536+0000 7f7e5dffb640 10 client.16521 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:27.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.536+0000 7f7e7a2c1640 1 --2- 172.21.15.23:0/1760781387 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ced487a970 0x55ced487cd90 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a44e7f35a9f10ac server_cookie=6f5c450567ff7e90 in_seq=0 out_seq=0 2024-03-24T10:43:27.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.537+0000 7f7e5dffb640 1 -- 172.21.15.23:0/1760781387 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ced4504720 con 0x55ced487a970 2024-03-24T10:43:27.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.537+0000 7f7e5dffb640 10 client.16521 handle_command_reply: tid=0 2024-03-24T10:43:27.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.537+0000 7f7e7f1e8740 1 client.16521 shutdown 2024-03-24T10:43:27.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.537+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/1760781387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 msgr2=0x55ced4867e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.537+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/1760781387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced4867e40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f7e64009c70 tx=0x7f7e64005720 comp rx=0 tx=0).stop 2024-03-24T10:43:27.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.537+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/1760781387 shutdown_connections 2024-03-24T10:43:27.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.537+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/1760781387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced486a8f0 0x55ced4868380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.537+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/1760781387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced4867e40 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.537+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/1760781387 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ced487a970 0x55ced487cd90 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.537+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/1760781387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 0x55ced4867900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.537+0000 7f7e5dffb640 0 client.16521 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:27.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.537+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/1760781387 >> 172.21.15.23:0/1760781387 conn(0x55ced485bb20 msgr2=0x55ced485d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.538+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/1760781387 shutdown_connections 2024-03-24T10:43:27.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.538+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/1760781387 wait complete. 2024-03-24T10:43:27.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.538+0000 7f7e7f1e8740 20 client.16521 trim_cache size 0 max 16384 2024-03-24T10:43:27.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.552+0000 7f7e7f1e8740 1 Processor -- start 2024-03-24T10:43:27.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.552+0000 7f7e7f1e8740 1 -- start start 2024-03-24T10:43:27.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.552+0000 7f7e7f1e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 0x55ced48761b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.552+0000 7f7e7f1e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced4861d10 0x55ced485ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.552+0000 7f7e7f1e8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced486a8f0 0x55ced4882640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.552+0000 7f7e7f1e8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ced48688d0 con 0x55ced486a8f0 2024-03-24T10:43:27.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.552+0000 7f7e7f1e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ced4877470 con 0x55ced4861d10 2024-03-24T10:43:27.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.552+0000 7f7e7f1e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ced48604e0 con 0x55ced48613f0 2024-03-24T10:43:27.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.552+0000 7f7e7aac2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced486a8f0 0x55ced4882640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.552+0000 7f7e7a2c1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 0x55ced48761b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.552+0000 7f7e7aac2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced486a8f0 0x55ced4882640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55304/0 (socket says 172.21.15.23:55304) 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.552+0000 7f7e7aac2640 1 -- 172.21.15.23:0/1991778085 learned_addr learned my addr 172.21.15.23:0/1991778085 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.552+0000 7f7e79ac0640 1 --2- 172.21.15.23:0/1991778085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced4861d10 0x55ced485ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e7aac2640 1 -- 172.21.15.23:0/1991778085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 msgr2=0x55ced48761b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e7aac2640 1 --2- 172.21.15.23:0/1991778085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 0x55ced48761b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e7aac2640 1 -- 172.21.15.23:0/1991778085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced4861d10 msgr2=0x55ced485ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e7aac2640 1 --2- 172.21.15.23:0/1991778085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced4861d10 0x55ced485ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e7aac2640 1 -- 172.21.15.23:0/1991778085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ced4762c80 con 0x55ced486a8f0 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e7a2c1640 1 --2- 172.21.15.23:0/1991778085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 0x55ced48761b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e7aac2640 1 --2- 172.21.15.23:0/1991778085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced486a8f0 0x55ced4882640 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009cd0 tx=0x7f7e6c009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405ba1ddfe526b23 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e5ffff640 1 -- 172.21.15.23:0/1991778085 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c0326b0 con 0x55ced486a8f0 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e5ffff640 1 -- 172.21.15.23:0/1991778085 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e6c049050 con 0x55ced486a8f0 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e5ffff640 1 -- 172.21.15.23:0/1991778085 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c004330 con 0x55ced486a8f0 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/1991778085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced486a8f0 msgr2=0x55ced4882640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/1991778085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced486a8f0 0x55ced4882640 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009cd0 tx=0x7f7e6c009360 comp rx=0 tx=0).stop 2024-03-24T10:43:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/1991778085 shutdown_connections 2024-03-24T10:43:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/1991778085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced486a8f0 0x55ced4882640 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/1991778085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced4861d10 0x55ced485ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/1991778085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced48613f0 0x55ced48761b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.553+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/1991778085 >> 172.21.15.23:0/1991778085 conn(0x55ced485bb20 msgr2=0x55ced485a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/1991778085 shutdown_connections 2024-03-24T10:43:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/1991778085 wait complete. 2024-03-24T10:43:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e7f1e8740 1 -- start start 2024-03-24T10:43:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e7f1e8740 1 Processor -- start 2024-03-24T10:43:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e7f1e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced48613f0 0x55ced486e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e7f1e8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced486e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e7f1e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced486a8f0 0x55ced486ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e7f1e8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ced48dfe10 con 0x55ced4861d10 2024-03-24T10:43:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e7f1e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ced48dff80 con 0x55ced48613f0 2024-03-24T10:43:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e7f1e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ced48e0140 con 0x55ced486a8f0 2024-03-24T10:43:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e7f1e8740 10 client.0 _ll_register_callbacks cb 0x7f7e6401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e79ac0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced486e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e79ac0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced486e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55312/0 (socket says 172.21.15.23:55312) 2024-03-24T10:43:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e7a2c1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced48613f0 0x55ced486e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e79ac0640 1 -- 172.21.15.23:0/2986323656 learned_addr learned my addr 172.21.15.23:0/2986323656 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e7aac2640 1 --2- 172.21.15.23:0/2986323656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced486a8f0 0x55ced486ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e79ac0640 1 -- 172.21.15.23:0/2986323656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced486a8f0 msgr2=0x55ced486ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e79ac0640 1 --2- 172.21.15.23:0/2986323656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced486a8f0 0x55ced486ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e79ac0640 1 -- 172.21.15.23:0/2986323656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced48613f0 msgr2=0x55ced486e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e79ac0640 1 --2- 172.21.15.23:0/2986323656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced48613f0 0x55ced486e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.554+0000 7f7e79ac0640 1 -- 172.21.15.23:0/2986323656 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e741a4860 con 0x55ced4861d10 2024-03-24T10:43:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e79ac0640 1 --2- 172.21.15.23:0/2986323656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced486e8e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f7e6401c040 tx=0x7f7e64009af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecbfc518914e4790 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:27.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e5dffb640 1 -- 172.21.15.23:0/2986323656 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400ea10 con 0x55ced4861d10 2024-03-24T10:43:27.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e5dffb640 1 -- 172.21.15.23:0/2986323656 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f7e6400eb70 con 0x55ced4861d10 2024-03-24T10:43:27.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/2986323656 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e64009e50 con 0x55ced4861d10 2024-03-24T10:43:27.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e5dffb640 1 -- 172.21.15.23:0/2986323656 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7e6400ed30 con 0x55ced4861d10 2024-03-24T10:43:27.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e7f1e8740 10 client.16527 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:27.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/2986323656 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x7f7e7410dd90 con 0x55ced4861d10 2024-03-24T10:43:27.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e5dffb640 1 -- 172.21.15.23:0/2986323656 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e64044020 con 0x55ced4861d10 2024-03-24T10:43:27.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e5dffb640 1 -- 172.21.15.23:0/2986323656 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7e640238e0 con 0x55ced4861d10 2024-03-24T10:43:27.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e7f1e8740 10 client.16527 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:27.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e7f1e8740 10 client.16527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:27.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e7f1e8740 20 client.16527 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:27.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/2986323656 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ced48e1590 0x55ced48e3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:27.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e7f1e8740 4 client.16527 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:43:27.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/2986323656 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ced4553890 con 0x55ced48e1590 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.555+0000 7f7e7a2c1640 1 --2- 172.21.15.23:0/2986323656 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ced48e1590 0x55ced48e3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e5dffb640 10 client.16527 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e7a2c1640 1 --2- 172.21.15.23:0/2986323656 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ced48e1590 0x55ced48e3980 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98df72418810f3c9 server_cookie=1074e3d992627be7 in_seq=0 out_seq=0 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e5dffb640 1 -- 172.21.15.23:0/2986323656 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ced4553890 con 0x55ced48e1590 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e5dffb640 10 client.16527 handle_command_reply: tid=0 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e7f1e8740 1 client.16527 shutdown 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/2986323656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 msgr2=0x55ced486e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/2986323656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced486e8e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f7e6401c040 tx=0x7f7e64009af0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/2986323656 shutdown_connections 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/2986323656 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ced48e1590 0x55ced48e3980 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/2986323656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ced486a8f0 0x55ced486ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/2986323656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ced4861d10 0x55ced486e8e0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e7f1e8740 1 --2- 172.21.15.23:0/2986323656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ced48613f0 0x55ced486e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/2986323656 >> 172.21.15.23:0/2986323656 conn(0x55ced485bb20 msgr2=0x55ced4868130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e5dffb640 0 client.16527 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:27.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/2986323656 shutdown_connections 2024-03-24T10:43:27.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e7f1e8740 1 -- 172.21.15.23:0/2986323656 wait complete. 2024-03-24T10:43:27.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.556+0000 7f7e7f1e8740 20 client.16527 trim_cache size 0 max 16384 2024-03-24T10:43:27.564 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:43:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.557+0000 7f7e7c525640 1 -- 172.21.15.23:0/417389459 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7e600a22b0 msgr2=0x7f7e600a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.557+0000 7f7e7c525640 1 --2- 172.21.15.23:0/417389459 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7e600a22b0 0x7f7e600a4770 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c005cc0 tx=0x7f7e6c005c50 comp rx=0 tx=0).stop 2024-03-24T10:43:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.557+0000 7f7e7c525640 1 -- 172.21.15.23:0/417389459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e741005c0 msgr2=0x7f7e74198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.557+0000 7f7e7c525640 1 --2- 172.21.15.23:0/417389459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e741005c0 0x7f7e74198b30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000e9f0 tx=0x7f7e7000eeb0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.557+0000 7f7e7c525640 1 -- 172.21.15.23:0/417389459 shutdown_connections 2024-03-24T10:43:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.557+0000 7f7e7c525640 1 --2- 172.21.15.23:0/417389459 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f7e600a22b0 0x7f7e600a4770 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.557+0000 7f7e7c525640 1 --2- 172.21.15.23:0/417389459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e74106660 0x7f7e741a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.557+0000 7f7e7c525640 1 --2- 172.21.15.23:0/417389459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e74100f80 0x7f7e74199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.557+0000 7f7e7c525640 1 --2- 172.21.15.23:0/417389459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e741005c0 0x7f7e74198b30 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.557+0000 7f7e7c525640 1 -- 172.21.15.23:0/417389459 >> 172.21.15.23:0/417389459 conn(0x7f7e74076990 msgr2=0x7f7e740fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.557+0000 7f7e7c525640 1 -- 172.21.15.23:0/417389459 shutdown_connections 2024-03-24T10:43:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:27.557+0000 7f7e7c525640 1 -- 172.21.15.23:0/417389459 wait complete. 2024-03-24T10:43:27.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:27 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2651708435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:43:27.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:27 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2651708435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:43:27.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:27 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2651708435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:43:28.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 239690 12.45 MB/sec warmup 105 sec latency 13.839 ms 2024-03-24T10:43:28.569 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:43:28.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:43:28.569 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:43:28.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.678+0000 7f8936e57640 1 -- 172.21.15.23:0/3615023695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8930102060 msgr2=0x7f893010e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.678+0000 7f8936e57640 1 --2- 172.21.15.23:0/3615023695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8930102060 0x7f893010e570 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8928009cd0 tx=0x7f892802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.678+0000 7f8936e57640 1 -- 172.21.15.23:0/3615023695 shutdown_connections 2024-03-24T10:43:28.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.678+0000 7f8936e57640 1 --2- 172.21.15.23:0/3615023695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8930102060 0x7f893010e570 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.678+0000 7f8936e57640 1 --2- 172.21.15.23:0/3615023695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89301016a0 0x7f8930101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.678+0000 7f8936e57640 1 --2- 172.21.15.23:0/3615023695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89301076b0 0x7f8930107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.678+0000 7f8936e57640 1 -- 172.21.15.23:0/3615023695 >> 172.21.15.23:0/3615023695 conn(0x7f89300fd1b0 msgr2=0x7f89300ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:28.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.678+0000 7f8936e57640 1 -- 172.21.15.23:0/3615023695 shutdown_connections 2024-03-24T10:43:28.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.678+0000 7f8936e57640 1 -- 172.21.15.23:0/3615023695 wait complete. 2024-03-24T10:43:28.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8936e57640 1 Processor -- start 2024-03-24T10:43:28.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8936e57640 1 -- start start 2024-03-24T10:43:28.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8936e57640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89301016a0 0x7f8930073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8936e57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8930102060 0x7f8930073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8936e57640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89301076b0 0x7f8930074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8936e57640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f89301a60d0 con 0x7f89301016a0 2024-03-24T10:43:28.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8936e57640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f89301a6240 con 0x7f8930102060 2024-03-24T10:43:28.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8936e57640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f89301a6400 con 0x7f89301076b0 2024-03-24T10:43:28.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8934bf3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89301016a0 0x7f8930073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8934bf3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89301016a0 0x7f8930073a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42470/0 (socket says 172.21.15.23:42470) 2024-03-24T10:43:28.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8934bf3640 1 -- 172.21.15.23:0/2678129667 learned_addr learned my addr 172.21.15.23:0/2678129667 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:28.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f89353f4640 1 --2- 172.21.15.23:0/2678129667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89301076b0 0x7f8930074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f892ffff640 1 --2- 172.21.15.23:0/2678129667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8930102060 0x7f8930073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8934bf3640 1 -- 172.21.15.23:0/2678129667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89301076b0 msgr2=0x7f8930074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8934bf3640 1 --2- 172.21.15.23:0/2678129667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89301076b0 0x7f8930074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8934bf3640 1 -- 172.21.15.23:0/2678129667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8930102060 msgr2=0x7f8930073f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8934bf3640 1 --2- 172.21.15.23:0/2678129667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8930102060 0x7f8930073f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.679+0000 7f8934bf3640 1 -- 172.21.15.23:0/2678129667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8928009900 con 0x7f89301016a0 2024-03-24T10:43:28.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.680+0000 7f8934bf3640 1 --2- 172.21.15.23:0/2678129667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89301016a0 0x7f8930073a10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f892400e9f0 tx=0x7f892400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:28.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.680+0000 7f892dffb640 1 -- 172.21.15.23:0/2678129667 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924009700 con 0x7f89301016a0 2024-03-24T10:43:28.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.680+0000 7f892dffb640 1 -- 172.21.15.23:0/2678129667 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f892400cd80 con 0x7f89301016a0 2024-03-24T10:43:28.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.680+0000 7f8936e57640 1 -- 172.21.15.23:0/2678129667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89301a66e0 con 0x7f89301016a0 2024-03-24T10:43:28.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.680+0000 7f892dffb640 1 -- 172.21.15.23:0/2678129667 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924018920 con 0x7f89301016a0 2024-03-24T10:43:28.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.680+0000 7f8936e57640 1 -- 172.21.15.23:0/2678129667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89301a6bf0 con 0x7f89301016a0 2024-03-24T10:43:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.682+0000 7f892dffb640 1 -- 172.21.15.23:0/2678129667 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f8924018a80 con 0x7f89301016a0 2024-03-24T10:43:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f8939a31740 1 Processor -- start 2024-03-24T10:43:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f8939a31740 1 -- start start 2024-03-24T10:43:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f892dffb640 1 --2- 172.21.15.23:0/2678129667 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f89200a2230 0x7f89200a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f8939a31740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fc760 0x56525f9fcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f8939a31740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fd080 0x56525fa05720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f8939a31740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525fa05c60 0x56525fa08050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f8939a31740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56525f96a530 con 0x56525f9fd080 2024-03-24T10:43:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f89353f4640 1 --2- 172.21.15.23:0/2678129667 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f89200a2230 0x7f89200a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f8939a31740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56525f967a60 con 0x56525fa05c60 2024-03-24T10:43:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f8939a31740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56525f963240 con 0x56525f9fc760 2024-03-24T10:43:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f892dffb640 1 -- 172.21.15.23:0/2678129667 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f8924014070 con 0x7f89301016a0 2024-03-24T10:43:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f892ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fd080 0x56525fa05720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f892ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fd080 0x56525fa05720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42484/0 (socket says 172.21.15.23:42484) 2024-03-24T10:43:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f892ffff640 1 -- 172.21.15.23:0/239108913 learned_addr learned my addr 172.21.15.23:0/239108913 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f8934bf3640 1 --2- 172.21.15.23:0/239108913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fc760 0x56525f9fcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f89353f4640 1 --2- 172.21.15.23:0/239108913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525fa05c60 0x56525fa08050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f892ffff640 1 -- 172.21.15.23:0/239108913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fc760 msgr2=0x56525f9fcb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f892ffff640 1 --2- 172.21.15.23:0/239108913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fc760 0x56525f9fcb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f892ffff640 1 -- 172.21.15.23:0/239108913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525fa05c60 msgr2=0x56525fa08050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f892ffff640 1 --2- 172.21.15.23:0/239108913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525fa05c60 0x56525fa08050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f892ffff640 1 -- 172.21.15.23:0/239108913 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56525f9618c0 con 0x56525f9fd080 2024-03-24T10:43:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f89353f4640 1 --2- 172.21.15.23:0/239108913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525fa05c60 0x56525fa08050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:43:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f8934bf3640 1 --2- 172.21.15.23:0/239108913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fc760 0x56525f9fcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.683+0000 7f89353f4640 1 --2- 172.21.15.23:0/2678129667 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f89200a2230 0x7f89200a46f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8928005cc0 tx=0x7f89280325d0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.684+0000 7f892ffff640 1 --2- 172.21.15.23:0/239108913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fd080 0x56525fa05720 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f891c009ca0 tx=0x7f891c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bcc26a1e0ca6bc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.684+0000 7f891affd640 1 -- 172.21.15.23:0/239108913 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891c019630 con 0x56525f9fd080 2024-03-24T10:43:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.684+0000 7f891affd640 1 -- 172.21.15.23:0/239108913 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f891c019c70 con 0x56525f9fd080 2024-03-24T10:43:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.684+0000 7f891affd640 1 -- 172.21.15.23:0/239108913 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891c00eb30 con 0x56525f9fd080 2024-03-24T10:43:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.684+0000 7f8939a31740 1 -- 172.21.15.23:0/239108913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fd080 msgr2=0x56525fa05720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.684+0000 7f8939a31740 1 --2- 172.21.15.23:0/239108913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fd080 0x56525fa05720 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f891c009ca0 tx=0x7f891c011810 comp rx=0 tx=0).stop 2024-03-24T10:43:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.684+0000 7f8939a31740 1 -- 172.21.15.23:0/239108913 shutdown_connections 2024-03-24T10:43:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 1 --2- 172.21.15.23:0/239108913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525fa05c60 0x56525fa08050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 1 --2- 172.21.15.23:0/239108913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fd080 0x56525fa05720 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 1 --2- 172.21.15.23:0/239108913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fc760 0x56525f9fcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 1 -- 172.21.15.23:0/239108913 >> 172.21.15.23:0/239108913 conn(0x56525f9f6e90 msgr2=0x56525f9f7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 1 -- 172.21.15.23:0/239108913 shutdown_connections 2024-03-24T10:43:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 1 -- 172.21.15.23:0/239108913 wait complete. 2024-03-24T10:43:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 1 -- start start 2024-03-24T10:43:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 1 Processor -- start 2024-03-24T10:43:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fc760 0x56525fa02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525f9fd080 0x56525fa03200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525fa05c60 0x56525fa03740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56525f907f80 con 0x56525f9fc760 2024-03-24T10:43:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56525f6364b0 con 0x56525f9fd080 2024-03-24T10:43:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56525f8e8ad0 con 0x56525fa05c60 2024-03-24T10:43:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.685+0000 7f8939a31740 10 client.0 _ll_register_callbacks cb 0x56525f91bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f8934bf3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fc760 0x56525fa02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f8934bf3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fc760 0x56525fa02cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42494/0 (socket says 172.21.15.23:42494) 2024-03-24T10:43:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f8934bf3640 1 -- 172.21.15.23:0/1532094800 learned_addr learned my addr 172.21.15.23:0/1532094800 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f892ffff640 1 --2- 172.21.15.23:0/1532094800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525f9fd080 0x56525fa03200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f89353f4640 1 --2- 172.21.15.23:0/1532094800 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525fa05c60 0x56525fa03740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f8934bf3640 1 -- 172.21.15.23:0/1532094800 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525fa05c60 msgr2=0x56525fa03740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f8934bf3640 1 --2- 172.21.15.23:0/1532094800 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525fa05c60 0x56525fa03740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f8934bf3640 1 -- 172.21.15.23:0/1532094800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525f9fd080 msgr2=0x56525fa03200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f8934bf3640 1 --2- 172.21.15.23:0/1532094800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525f9fd080 0x56525fa03200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f8934bf3640 1 -- 172.21.15.23:0/1532094800 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f891c0098d0 con 0x56525f9fc760 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f8934bf3640 1 --2- 172.21.15.23:0/1532094800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fc760 0x56525fa02cc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8924097800 tx=0x7f892409a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59b961d7ccd3417 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f890ffff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f890ffff640 1 -- 172.21.15.23:0/1532094800 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892409a280 con 0x56525f9fc760 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f890ffff640 1 -- 172.21.15.23:0/1532094800 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f892409a3e0 con 0x56525f9fc760 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f8939a31740 1 -- 172.21.15.23:0/1532094800 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56525f9877a0 con 0x56525f9fc760 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f890ffff640 1 -- 172.21.15.23:0/1532094800 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f892409a5a0 con 0x56525f9fc760 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f8939a31740 10 client.16539 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f8939a31740 1 -- 172.21.15.23:0/1532094800 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56525f9196d0 con 0x56525f9fc760 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.686+0000 7f890ffff640 1 -- 172.21.15.23:0/1532094800 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56525f9877a0 con 0x56525f9fc760 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.687+0000 7f890ffff640 1 -- 172.21.15.23:0/1532094800 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8924099cf0 con 0x56525f9fc760 2024-03-24T10:43:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.687+0000 7f8939a31740 10 client.16539 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:28.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.687+0000 7f8939a31740 10 client.16539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:28.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.687+0000 7f8939a31740 20 client.16539 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:28.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.687+0000 7f8939a31740 1 --2- 172.21.15.23:0/1532094800 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56525fa15bf0 0x56525fa18010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.687+0000 7f8939a31740 4 client.16539 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:28.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.687+0000 7f8939a31740 1 -- 172.21.15.23:0/1532094800 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56525f9591f0 con 0x56525fa15bf0 2024-03-24T10:43:28.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.687+0000 7f892ffff640 1 --2- 172.21.15.23:0/1532094800 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56525fa15bf0 0x56525fa18010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.687+0000 7f890ffff640 10 client.16539 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:28.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.687+0000 7f892ffff640 1 --2- 172.21.15.23:0/1532094800 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56525fa15bf0 0x56525fa18010 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70e44540f16e03d5 server_cookie=6970e4013f501fb0 in_seq=0 out_seq=0 2024-03-24T10:43:28.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.688+0000 7f890ffff640 1 -- 172.21.15.23:0/1532094800 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56525f9591f0 con 0x56525fa15bf0 2024-03-24T10:43:28.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.688+0000 7f890ffff640 10 client.16539 handle_command_reply: tid=0 2024-03-24T10:43:28.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.688+0000 7f8939a31740 1 client.16539 shutdown 2024-03-24T10:43:28.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.688+0000 7f8939a31740 1 -- 172.21.15.23:0/1532094800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fc760 msgr2=0x56525fa02cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.688+0000 7f8939a31740 1 --2- 172.21.15.23:0/1532094800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fc760 0x56525fa02cc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8924097800 tx=0x7f892409a8b0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.688+0000 7f8939a31740 1 -- 172.21.15.23:0/1532094800 shutdown_connections 2024-03-24T10:43:28.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.688+0000 7f8939a31740 1 --2- 172.21.15.23:0/1532094800 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56525fa15bf0 0x56525fa18010 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.688+0000 7f8939a31740 1 --2- 172.21.15.23:0/1532094800 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525fa05c60 0x56525fa03740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.688+0000 7f8939a31740 1 --2- 172.21.15.23:0/1532094800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525f9fd080 0x56525fa03200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.688+0000 7f8939a31740 1 --2- 172.21.15.23:0/1532094800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fc760 0x56525fa02cc0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.688+0000 7f890ffff640 0 client.16539 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:28.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.688+0000 7f8939a31740 1 -- 172.21.15.23:0/1532094800 >> 172.21.15.23:0/1532094800 conn(0x56525f9f6e90 msgr2=0x56525f9f8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:28.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.688+0000 7f8939a31740 1 -- 172.21.15.23:0/1532094800 shutdown_connections 2024-03-24T10:43:28.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.688+0000 7f8939a31740 1 -- 172.21.15.23:0/1532094800 wait complete. 2024-03-24T10:43:28.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.689+0000 7f8939a31740 20 client.16539 trim_cache size 0 max 16384 2024-03-24T10:43:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8939a31740 1 Processor -- start 2024-03-24T10:43:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8939a31740 1 -- start start 2024-03-24T10:43:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8939a31740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fc760 0x56525fa114f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8939a31740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fd080 0x56525fa03560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8939a31740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525fa05c60 0x56525fa1d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8939a31740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56525f9fc0b0 con 0x56525f9fc760 2024-03-24T10:43:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8939a31740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56525f9f6670 con 0x56525fa05c60 2024-03-24T10:43:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8939a31740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56525f9fbf00 con 0x56525f9fd080 2024-03-24T10:43:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8934bf3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fc760 0x56525fa114f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8934bf3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fc760 0x56525fa114f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42510/0 (socket says 172.21.15.23:42510) 2024-03-24T10:43:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8934bf3640 1 -- 172.21.15.23:0/3694786489 learned_addr learned my addr 172.21.15.23:0/3694786489 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f892ffff640 1 --2- 172.21.15.23:0/3694786489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fd080 0x56525fa03560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f89353f4640 1 --2- 172.21.15.23:0/3694786489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525fa05c60 0x56525fa1d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8934bf3640 1 -- 172.21.15.23:0/3694786489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fd080 msgr2=0x56525fa03560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8934bf3640 1 --2- 172.21.15.23:0/3694786489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fd080 0x56525fa03560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8934bf3640 1 -- 172.21.15.23:0/3694786489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525fa05c60 msgr2=0x56525fa1d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8934bf3640 1 --2- 172.21.15.23:0/3694786489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525fa05c60 0x56525fa1d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8934bf3640 1 -- 172.21.15.23:0/3694786489 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56525f8fdc60 con 0x56525f9fc760 2024-03-24T10:43:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.707+0000 7f8934bf3640 1 --2- 172.21.15.23:0/3694786489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fc760 0x56525fa114f0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f892409adc0 tx=0x7f8924000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3271b85cc63798ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.708+0000 7f891a7fc640 1 -- 172.21.15.23:0/3694786489 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89240970d0 con 0x56525f9fc760 2024-03-24T10:43:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.708+0000 7f891a7fc640 1 -- 172.21.15.23:0/3694786489 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89240b9250 con 0x56525f9fc760 2024-03-24T10:43:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.708+0000 7f891a7fc640 1 -- 172.21.15.23:0/3694786489 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89240b3ab0 con 0x56525f9fc760 2024-03-24T10:43:28.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.708+0000 7f8939a31740 1 -- 172.21.15.23:0/3694786489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fc760 msgr2=0x56525fa114f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.708+0000 7f8939a31740 1 --2- 172.21.15.23:0/3694786489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fc760 0x56525fa114f0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f892409adc0 tx=0x7f8924000bc0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.708+0000 7f8939a31740 1 -- 172.21.15.23:0/3694786489 shutdown_connections 2024-03-24T10:43:28.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.708+0000 7f8939a31740 1 --2- 172.21.15.23:0/3694786489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525fa05c60 0x56525fa1d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.708+0000 7f8939a31740 1 --2- 172.21.15.23:0/3694786489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fd080 0x56525fa03560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.708+0000 7f8939a31740 1 --2- 172.21.15.23:0/3694786489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525f9fc760 0x56525fa114f0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.708+0000 7f8939a31740 1 -- 172.21.15.23:0/3694786489 >> 172.21.15.23:0/3694786489 conn(0x56525f9f6e90 msgr2=0x56525f9f8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:28.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.708+0000 7f8939a31740 1 -- 172.21.15.23:0/3694786489 shutdown_connections 2024-03-24T10:43:28.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.708+0000 7f8939a31740 1 -- 172.21.15.23:0/3694786489 wait complete. 2024-03-24T10:43:28.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.709+0000 7f8939a31740 1 -- start start 2024-03-24T10:43:28.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.709+0000 7f8939a31740 1 Processor -- start 2024-03-24T10:43:28.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.709+0000 7f8939a31740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525f9fc760 0x56525fa0db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.709+0000 7f8939a31740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fd080 0x56525fa0aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.709+0000 7f8939a31740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525fa05c60 0x56525fa0b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.709+0000 7f8939a31740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56525fa0e280 con 0x56525fa05c60 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.709+0000 7f8939a31740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56525fa0b570 con 0x56525f9fc760 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.709+0000 7f8939a31740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56525fa0b730 con 0x56525f9fd080 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.709+0000 7f8939a31740 10 client.0 _ll_register_callbacks cb 0x7f892409ae30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.709+0000 7f89353f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525fa05c60 0x56525fa0b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.709+0000 7f89353f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525fa05c60 0x56525fa0b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42524/0 (socket says 172.21.15.23:42524) 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.709+0000 7f89353f4640 1 -- 172.21.15.23:0/276210976 learned_addr learned my addr 172.21.15.23:0/276210976 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.709+0000 7f8934bf3640 1 --2- 172.21.15.23:0/276210976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525f9fc760 0x56525fa0db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.709+0000 7f892ffff640 1 --2- 172.21.15.23:0/276210976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fd080 0x56525fa0aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f89353f4640 1 -- 172.21.15.23:0/276210976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fd080 msgr2=0x56525fa0aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f89353f4640 1 --2- 172.21.15.23:0/276210976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fd080 0x56525fa0aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f89353f4640 1 -- 172.21.15.23:0/276210976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525f9fc760 msgr2=0x56525fa0db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f89353f4640 1 --2- 172.21.15.23:0/276210976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525f9fc760 0x56525fa0db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f89353f4640 1 -- 172.21.15.23:0/276210976 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56525f9196d0 con 0x56525fa05c60 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f892ffff640 1 --2- 172.21.15.23:0/276210976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fd080 0x56525fa0aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f89353f4640 1 --2- 172.21.15.23:0/276210976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525fa05c60 0x56525fa0b030 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f8928009cd0 tx=0x7f892802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afbd57c4b0ad93b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f890ffff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f890ffff640 1 -- 172.21.15.23:0/276210976 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8928003d80 con 0x56525fa05c60 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f890ffff640 1 -- 172.21.15.23:0/276210976 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f8928004040 con 0x56525fa05c60 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f8939a31740 1 -- 172.21.15.23:0/276210976 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f891c019c70 con 0x56525fa05c60 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f890ffff640 1 -- 172.21.15.23:0/276210976 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8928004200 con 0x56525fa05c60 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f8939a31740 10 client.16545 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f8939a31740 1 -- 172.21.15.23:0/276210976 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56525f923970 con 0x56525fa05c60 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.710+0000 7f890ffff640 1 -- 172.21.15.23:0/276210976 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f891c019c70 con 0x56525fa05c60 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.711+0000 7f890ffff640 1 -- 172.21.15.23:0/276210976 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8928044a00 con 0x56525fa05c60 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.711+0000 7f8939a31740 10 client.16545 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.711+0000 7f8939a31740 10 client.16545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.711+0000 7f8939a31740 20 client.16545 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.711+0000 7f8939a31740 1 --2- 172.21.15.23:0/276210976 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56525fa7d420 0x56525fa7f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.711+0000 7f8939a31740 4 client.16545 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.711+0000 7f8939a31740 1 -- 172.21.15.23:0/276210976 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56525f662860 con 0x56525fa7d420 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.711+0000 7f8934bf3640 1 --2- 172.21.15.23:0/276210976 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56525fa7d420 0x56525fa7f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.711+0000 7f8934bf3640 1 --2- 172.21.15.23:0/276210976 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56525fa7d420 0x56525fa7f840 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2517a950d47bcc1d server_cookie=92ab7ca82660b17 in_seq=0 out_seq=0 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.711+0000 7f890ffff640 10 client.16545 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.712+0000 7f890ffff640 1 -- 172.21.15.23:0/276210976 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56525f662860 con 0x56525fa7d420 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.712+0000 7f890ffff640 10 client.16545 handle_command_reply: tid=0 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8939a31740 1 client.16545 shutdown 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8939a31740 1 -- 172.21.15.23:0/276210976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525fa05c60 msgr2=0x56525fa0b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8939a31740 1 --2- 172.21.15.23:0/276210976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525fa05c60 0x56525fa0b030 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f8928009cd0 tx=0x7f892802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8939a31740 1 -- 172.21.15.23:0/276210976 shutdown_connections 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8939a31740 1 --2- 172.21.15.23:0/276210976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56525fa05c60 0x56525fa0b030 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8939a31740 1 --2- 172.21.15.23:0/276210976 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56525fa7d420 0x56525fa7f840 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8939a31740 1 --2- 172.21.15.23:0/276210976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56525f9fd080 0x56525fa0aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8939a31740 1 --2- 172.21.15.23:0/276210976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56525f9fc760 0x56525fa0db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8939a31740 1 -- 172.21.15.23:0/276210976 >> 172.21.15.23:0/276210976 conn(0x56525f9f6e90 msgr2=0x56525f9f5660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f890ffff640 0 client.16545 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:28.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8939a31740 1 -- 172.21.15.23:0/276210976 shutdown_connections 2024-03-24T10:43:28.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8939a31740 1 -- 172.21.15.23:0/276210976 wait complete. 2024-03-24T10:43:28.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8939a31740 20 client.16545 trim_cache size 0 max 16384 2024-03-24T10:43:28.718 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:43:28.718 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:43:28.718 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "bdca8b6d-e38c-4178-a11b-737bc3855b0b", 2024-03-24T10:43:28.718 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:43:28.718 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:43:28.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8936e57640 1 -- 172.21.15.23:0/2678129667 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f89200a2230 msgr2=0x7f89200a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8936e57640 1 --2- 172.21.15.23:0/2678129667 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f89200a2230 0x7f89200a46f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8928005cc0 tx=0x7f89280325d0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8936e57640 1 -- 172.21.15.23:0/2678129667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89301016a0 msgr2=0x7f8930073a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.713+0000 7f8936e57640 1 --2- 172.21.15.23:0/2678129667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89301016a0 0x7f8930073a10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f892400e9f0 tx=0x7f892400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.714+0000 7f8936e57640 1 -- 172.21.15.23:0/2678129667 shutdown_connections 2024-03-24T10:43:28.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.714+0000 7f8936e57640 1 --2- 172.21.15.23:0/2678129667 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f89200a2230 0x7f89200a46f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.714+0000 7f8936e57640 1 --2- 172.21.15.23:0/2678129667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89301076b0 0x7f8930074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.714+0000 7f8936e57640 1 --2- 172.21.15.23:0/2678129667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8930102060 0x7f8930073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.714+0000 7f8936e57640 1 --2- 172.21.15.23:0/2678129667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89301016a0 0x7f8930073a10 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.714+0000 7f8936e57640 1 -- 172.21.15.23:0/2678129667 >> 172.21.15.23:0/2678129667 conn(0x7f89300fd1b0 msgr2=0x7f89300fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:28.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.714+0000 7f8936e57640 1 -- 172.21.15.23:0/2678129667 shutdown_connections 2024-03-24T10:43:28.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.714+0000 7f8936e57640 1 -- 172.21.15.23:0/2678129667 wait complete. 2024-03-24T10:43:28.725 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:43:28.725 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:43:28.749 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:28 smithi023 ceph-mon[32466]: pgmap v104: 97 pgs: 97 active+clean; 147 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.8 MiB/s wr, 311 op/s 2024-03-24T10:43:28.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:28 smithi117 ceph-mon[36368]: pgmap v104: 97 pgs: 97 active+clean; 147 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.8 MiB/s wr, 311 op/s 2024-03-24T10:43:28.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:28 smithi116 ceph-mon[36248]: pgmap v104: 97 pgs: 97 active+clean; 147 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.8 MiB/s wr, 311 op/s 2024-03-24T10:43:28.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.835+0000 7f68572d0640 1 -- 172.21.15.23:0/2451337648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68501016b0 msgr2=0x7f6850101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.835+0000 7f68572d0640 1 --2- 172.21.15.23:0/2451337648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68501016b0 0x7f6850101b30 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6844009930 tx=0x7f684402bc30 comp rx=0 tx=0).stop 2024-03-24T10:43:28.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.835+0000 7f68572d0640 1 -- 172.21.15.23:0/2451337648 shutdown_connections 2024-03-24T10:43:28.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.835+0000 7f68572d0640 1 --2- 172.21.15.23:0/2451337648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6850102070 0x7f685010e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.835+0000 7f68572d0640 1 --2- 172.21.15.23:0/2451337648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68501016b0 0x7f6850101b30 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.835+0000 7f68572d0640 1 --2- 172.21.15.23:0/2451337648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68501076c0 0x7f6850107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.835+0000 7f68572d0640 1 -- 172.21.15.23:0/2451337648 >> 172.21.15.23:0/2451337648 conn(0x7f68500fd140 msgr2=0x7f68500ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:28.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.835+0000 7f68572d0640 1 -- 172.21.15.23:0/2451337648 shutdown_connections 2024-03-24T10:43:28.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.835+0000 7f68572d0640 1 -- 172.21.15.23:0/2451337648 wait complete. 2024-03-24T10:43:28.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.835+0000 7f68572d0640 1 Processor -- start 2024-03-24T10:43:28.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.835+0000 7f68572d0640 1 -- start start 2024-03-24T10:43:28.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.835+0000 7f68572d0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68501016b0 0x7f6850197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.835+0000 7f68572d0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6850102070 0x7f6850198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f68572d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68501076c0 0x7f685019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f68572d0640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f685019cc60 con 0x7f6850102070 2024-03-24T10:43:28.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f68572d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f685019cdd0 con 0x7f68501076c0 2024-03-24T10:43:28.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f68572d0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f685019cf90 con 0x7f68501016b0 2024-03-24T10:43:28.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f685486b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6850102070 0x7f6850198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f685506c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68501016b0 0x7f6850197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f685486b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6850102070 0x7f6850198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42546/0 (socket says 172.21.15.23:42546) 2024-03-24T10:43:28.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f685486b640 1 -- 172.21.15.23:0/3848769633 learned_addr learned my addr 172.21.15.23:0/3848769633 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:28.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f685586d640 1 --2- 172.21.15.23:0/3848769633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68501076c0 0x7f685019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f685486b640 1 -- 172.21.15.23:0/3848769633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68501016b0 msgr2=0x7f6850197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f685486b640 1 --2- 172.21.15.23:0/3848769633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68501016b0 0x7f6850197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f685486b640 1 -- 172.21.15.23:0/3848769633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68501076c0 msgr2=0x7f685019c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f685486b640 1 --2- 172.21.15.23:0/3848769633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68501076c0 0x7f685019c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f685486b640 1 -- 172.21.15.23:0/3848769633 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6844009560 con 0x7f6850102070 2024-03-24T10:43:28.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f685486b640 1 --2- 172.21.15.23:0/3848769633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6850102070 0x7f6850198330 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f6844009230 tx=0x7f684402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:28.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f684e7fc640 1 -- 172.21.15.23:0/3848769633 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6844004350 con 0x7f6850102070 2024-03-24T10:43:28.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f684e7fc640 1 -- 172.21.15.23:0/3848769633 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68440044b0 con 0x7f6850102070 2024-03-24T10:43:28.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f684e7fc640 1 -- 172.21.15.23:0/3848769633 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6844003d00 con 0x7f6850102070 2024-03-24T10:43:28.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f68572d0640 1 -- 172.21.15.23:0/3848769633 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f685019d210 con 0x7f6850102070 2024-03-24T10:43:28.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.836+0000 7f68572d0640 1 -- 172.21.15.23:0/3848769633 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68501a4af0 con 0x7f6850102070 2024-03-24T10:43:28.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.839+0000 7f684e7fc640 1 -- 172.21.15.23:0/3848769633 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f6844004df0 con 0x7f6850102070 2024-03-24T10:43:28.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.839+0000 7f6859f82740 1 Processor -- start 2024-03-24T10:43:28.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.839+0000 7f6859f82740 1 -- start start 2024-03-24T10:43:28.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.839+0000 7f6859f82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0bdc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.839+0000 7f684e7fc640 1 --2- 172.21.15.23:0/3848769633 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f68380a2230 0x7f68380a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.839+0000 7f6859f82740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.839+0000 7f6859f82740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0be56d0 0x55b6f0be7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.839+0000 7f6859f82740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0b00f30 con 0x55b6f0be56d0 2024-03-24T10:43:28.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.839+0000 7f6859f82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0b4a810 con 0x55b6f0bdc1d0 2024-03-24T10:43:28.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.839+0000 7f6859f82740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0b47d40 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f685586d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0be56d0 0x55b6f0be7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f685506c640 1 --2- 172.21.15.23:0/3848769633 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f68380a2230 0x7f68380a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f685586d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0be56d0 0x55b6f0be7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42550/0 (socket says 172.21.15.23:42550) 2024-03-24T10:43:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f685586d640 1 -- 172.21.15.23:0/2107677766 learned_addr learned my addr 172.21.15.23:0/2107677766 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f685586d640 1 --2- 172.21.15.23:0/2107677766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0bdc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f684e7fc640 1 -- 172.21.15.23:0/3848769633 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f68440eff50 con 0x7f6850102070 2024-03-24T10:43:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f685486b640 1 --2- 172.21.15.23:0/2107677766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f685586d640 1 -- 172.21.15.23:0/2107677766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bdcaf0 msgr2=0x55b6f0be5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f685586d640 1 --2- 172.21.15.23:0/2107677766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f685586d640 1 -- 172.21.15.23:0/2107677766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 msgr2=0x55b6f0bdc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f685586d640 1 --2- 172.21.15.23:0/2107677766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0bdc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f685586d640 1 -- 172.21.15.23:0/2107677766 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6f0b04ea0 con 0x55b6f0be56d0 2024-03-24T10:43:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f685506c640 1 --2- 172.21.15.23:0/3848769633 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f68380a2230 0x7f68380a46f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6848009b50 tx=0x7f6848005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:28.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f685586d640 1 --2- 172.21.15.23:0/2107677766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0be56d0 0x55b6f0be7ac0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6840005f40 tx=0x7f6840009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1fb435a4631fbb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:28.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f68377fe640 1 -- 172.21.15.23:0/2107677766 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840004b20 con 0x55b6f0be56d0 2024-03-24T10:43:28.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f68377fe640 1 -- 172.21.15.23:0/2107677766 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6840003e50 con 0x55b6f0be56d0 2024-03-24T10:43:28.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.840+0000 7f68377fe640 1 -- 172.21.15.23:0/2107677766 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840004050 con 0x55b6f0be56d0 2024-03-24T10:43:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.841+0000 7f6859f82740 1 -- 172.21.15.23:0/2107677766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0be56d0 msgr2=0x55b6f0be7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.841+0000 7f6859f82740 1 --2- 172.21.15.23:0/2107677766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0be56d0 0x55b6f0be7ac0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6840005f40 tx=0x7f6840009b80 comp rx=0 tx=0).stop 2024-03-24T10:43:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.841+0000 7f6859f82740 1 -- 172.21.15.23:0/2107677766 shutdown_connections 2024-03-24T10:43:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.841+0000 7f6859f82740 1 --2- 172.21.15.23:0/2107677766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0be56d0 0x55b6f0be7ac0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.841+0000 7f6859f82740 1 --2- 172.21.15.23:0/2107677766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.841+0000 7f6859f82740 1 --2- 172.21.15.23:0/2107677766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0bdc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.841+0000 7f6859f82740 1 -- 172.21.15.23:0/2107677766 >> 172.21.15.23:0/2107677766 conn(0x55b6f0bd6900 msgr2=0x55b6f0bd6ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.841+0000 7f6859f82740 1 -- 172.21.15.23:0/2107677766 shutdown_connections 2024-03-24T10:43:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.841+0000 7f6859f82740 1 -- 172.21.15.23:0/2107677766 wait complete. 2024-03-24T10:43:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.841+0000 7f6859f82740 1 -- start start 2024-03-24T10:43:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f6859f82740 1 Processor -- start 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f6859f82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0be2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f6859f82740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f6859f82740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0be56d0 0x55b6f0be31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f6859f82740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0b43520 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f6859f82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0816430 con 0x55b6f0bdc1d0 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f6859f82740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b6f07d2e00 con 0x55b6f0be56d0 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f6859f82740 10 client.0 _ll_register_callbacks cb 0x55b6f0ae1d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f685486b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f685486b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42558/0 (socket says 172.21.15.23:42558) 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f685486b640 1 -- 172.21.15.23:0/322292853 learned_addr learned my addr 172.21.15.23:0/322292853 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f685506c640 1 --2- 172.21.15.23:0/322292853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0be2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f685586d640 1 --2- 172.21.15.23:0/322292853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0be56d0 0x55b6f0be31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f685486b640 1 -- 172.21.15.23:0/322292853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0be56d0 msgr2=0x55b6f0be31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f685486b640 1 --2- 172.21.15.23:0/322292853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0be56d0 0x55b6f0be31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f685486b640 1 -- 172.21.15.23:0/322292853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 msgr2=0x55b6f0be2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f685486b640 1 --2- 172.21.15.23:0/322292853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0be2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f685486b640 1 -- 172.21.15.23:0/322292853 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6840019040 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f685486b640 1 --2- 172.21.15.23:0/322292853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be2ca0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6844009430 tx=0x7f684402cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64878b57eb7dbd27 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.842+0000 7f6834ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f6834ff9640 1 -- 172.21.15.23:0/322292853 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6844003e60 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f6834ff9640 1 -- 172.21.15.23:0/322292853 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f68440035f0 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f6859f82740 1 -- 172.21.15.23:0/322292853 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b6f08427e0 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f6834ff9640 1 -- 172.21.15.23:0/322292853 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f68440bd150 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f6859f82740 10 client.16557 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f6859f82740 1 -- 172.21.15.23:0/322292853 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b6f0b41ba0 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f6834ff9640 1 -- 172.21.15.23:0/322292853 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b6f08427e0 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f6834ff9640 1 -- 172.21.15.23:0/322292853 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f684403c070 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f6859f82740 10 client.16557 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f6859f82740 10 client.16557 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f6859f82740 20 client.16557 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f6859f82740 1 --2- 172.21.15.23:0/322292853 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b6f0bf5910 0x55b6f0bf7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f6859f82740 4 client.16557 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:28.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f6859f82740 1 -- 172.21.15.23:0/322292853 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6f087f720 con 0x55b6f0bf5910 2024-03-24T10:43:28.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.843+0000 7f685506c640 1 --2- 172.21.15.23:0/322292853 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b6f0bf5910 0x55b6f0bf7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.844+0000 7f685506c640 1 --2- 172.21.15.23:0/322292853 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b6f0bf5910 0x55b6f0bf7d30 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eda51cbaaf71834f server_cookie=81bc605ae5c47a47 in_seq=0 out_seq=0 2024-03-24T10:43:28.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.844+0000 7f6834ff9640 10 client.16557 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:28.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.844+0000 7f6834ff9640 1 -- 172.21.15.23:0/322292853 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b6f087f720 con 0x55b6f0bf5910 2024-03-24T10:43:28.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.844+0000 7f6834ff9640 10 client.16557 handle_command_reply: tid=0 2024-03-24T10:43:28.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.845+0000 7f6859f82740 1 client.16557 shutdown 2024-03-24T10:43:28.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.845+0000 7f6859f82740 1 -- 172.21.15.23:0/322292853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 msgr2=0x55b6f0be2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.845+0000 7f6859f82740 1 --2- 172.21.15.23:0/322292853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be2ca0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6844009430 tx=0x7f684402cda0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.845+0000 7f6859f82740 1 -- 172.21.15.23:0/322292853 shutdown_connections 2024-03-24T10:43:28.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.845+0000 7f6859f82740 1 --2- 172.21.15.23:0/322292853 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b6f0bf5910 0x55b6f0bf7d30 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.845+0000 7f6859f82740 1 --2- 172.21.15.23:0/322292853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0be56d0 0x55b6f0be31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.845+0000 7f6859f82740 1 --2- 172.21.15.23:0/322292853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be2ca0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.845+0000 7f6859f82740 1 --2- 172.21.15.23:0/322292853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0be2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.845+0000 7f6859f82740 1 -- 172.21.15.23:0/322292853 >> 172.21.15.23:0/322292853 conn(0x55b6f0bd6900 msgr2=0x55b6f0bd8650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:28.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.845+0000 7f6834ff9640 0 client.16557 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:28.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.845+0000 7f6859f82740 1 -- 172.21.15.23:0/322292853 shutdown_connections 2024-03-24T10:43:28.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.845+0000 7f6859f82740 1 -- 172.21.15.23:0/322292853 wait complete. 2024-03-24T10:43:28.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.845+0000 7f6859f82740 20 client.16557 trim_cache size 0 max 16384 2024-03-24T10:43:28.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f6859f82740 1 Processor -- start 2024-03-24T10:43:28.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f6859f82740 1 -- start start 2024-03-24T10:43:28.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f6859f82740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bd6900 0x55b6f0be1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f6859f82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0bf0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f6859f82740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f6859f82740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0ac8430 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f6859f82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0bdb970 con 0x55b6f0bdc1d0 2024-03-24T10:43:28.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f6859f82740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0bdbc10 con 0x55b6f0bd6900 2024-03-24T10:43:28.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f685586d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f685586d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42574/0 (socket says 172.21.15.23:42574) 2024-03-24T10:43:28.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f685586d640 1 -- 172.21.15.23:0/3001600433 learned_addr learned my addr 172.21.15.23:0/3001600433 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:28.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f685486b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0bf0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f685506c640 1 --2- 172.21.15.23:0/3001600433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bd6900 0x55b6f0be1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f685586d640 1 -- 172.21.15.23:0/3001600433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bd6900 msgr2=0x55b6f0be1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f685586d640 1 --2- 172.21.15.23:0/3001600433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bd6900 0x55b6f0be1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f685586d640 1 -- 172.21.15.23:0/3001600433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 msgr2=0x55b6f0bf0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f685586d640 1 --2- 172.21.15.23:0/3001600433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0bf0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f685586d640 1 -- 172.21.15.23:0/3001600433 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6f0addb20 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.864+0000 7f685486b640 1 --2- 172.21.15.23:0/3001600433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0bf0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:28.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.865+0000 7f685586d640 1 --2- 172.21.15.23:0/3001600433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be3000 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f684000a010 tx=0x7f684000fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ccac4fe1571309 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:28.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.865+0000 7f6836ffd640 1 -- 172.21.15.23:0/3001600433 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840025040 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.865+0000 7f6836ffd640 1 -- 172.21.15.23:0/3001600433 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6840005bd0 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.865+0000 7f6836ffd640 1 -- 172.21.15.23:0/3001600433 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840005420 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 -- 172.21.15.23:0/3001600433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 msgr2=0x55b6f0be3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 --2- 172.21.15.23:0/3001600433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be3000 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f684000a010 tx=0x7f684000fc20 comp rx=0 tx=0).stop 2024-03-24T10:43:28.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 -- 172.21.15.23:0/3001600433 shutdown_connections 2024-03-24T10:43:28.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 --2- 172.21.15.23:0/3001600433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be3000 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 --2- 172.21.15.23:0/3001600433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0bf0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 --2- 172.21.15.23:0/3001600433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bd6900 0x55b6f0be1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 -- 172.21.15.23:0/3001600433 >> 172.21.15.23:0/3001600433 conn(0x55b6f0bde130 msgr2=0x55b6f0bd5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:28.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 -- 172.21.15.23:0/3001600433 shutdown_connections 2024-03-24T10:43:28.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 -- 172.21.15.23:0/3001600433 wait complete. 2024-03-24T10:43:28.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 -- start start 2024-03-24T10:43:28.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 Processor -- start 2024-03-24T10:43:28.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bd6900 0x55b6f0be8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0be94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0c5aca0 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0c5ae10 con 0x55b6f0bdc1d0 2024-03-24T10:43:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.866+0000 7f6859f82740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b6f0c5afd0 con 0x55b6f0bd6900 2024-03-24T10:43:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f6859f82740 10 client.0 _ll_register_callbacks cb 0x7f68440b8d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f685586d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f685486b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0be94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f685586d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42576/0 (socket says 172.21.15.23:42576) 2024-03-24T10:43:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f685586d640 1 -- 172.21.15.23:0/379908241 learned_addr learned my addr 172.21.15.23:0/379908241 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f685506c640 1 --2- 172.21.15.23:0/379908241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bd6900 0x55b6f0be8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f685586d640 1 -- 172.21.15.23:0/379908241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bd6900 msgr2=0x55b6f0be8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f685586d640 1 --2- 172.21.15.23:0/379908241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bd6900 0x55b6f0be8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f685586d640 1 -- 172.21.15.23:0/379908241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 msgr2=0x55b6f0be94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f685586d640 1 --2- 172.21.15.23:0/379908241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0be94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f685586d640 1 -- 172.21.15.23:0/379908241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6f0b41ba0 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f685506c640 1 --2- 172.21.15.23:0/379908241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bd6900 0x55b6f0be8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:28.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f685486b640 1 --2- 172.21.15.23:0/379908241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0be94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:28.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f685586d640 1 --2- 172.21.15.23:0/379908241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be9a10 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6840011600 tx=0x7f6840011770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5639a9b26e82c424 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:28.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f6834ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:28.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f6834ff9640 1 -- 172.21.15.23:0/379908241 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840025040 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f6834ff9640 1 -- 172.21.15.23:0/379908241 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f6840003970 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f6859f82740 1 -- 172.21.15.23:0/379908241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6840003e50 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f6834ff9640 1 -- 172.21.15.23:0/379908241 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f684000f850 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f6859f82740 10 client.16563 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:28.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.867+0000 7f6859f82740 1 -- 172.21.15.23:0/379908241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b6f0b04ea0 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f6834ff9640 1 -- 172.21.15.23:0/379908241 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6840003e50 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f6834ff9640 1 -- 172.21.15.23:0/379908241 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f684001d070 con 0x55b6f0bdcaf0 2024-03-24T10:43:28.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f6859f82740 10 client.16563 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:28.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f6859f82740 10 client.16563 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:28.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f6859f82740 20 client.16563 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f6859f82740 1 --2- 172.21.15.23:0/379908241 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b6f0be56d0 0x55b6f0c5e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f6859f82740 4 client.16563 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f6859f82740 1 -- 172.21.15.23:0/379908241 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b6f08ce890 con 0x55b6f0be56d0 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f685506c640 1 --2- 172.21.15.23:0/379908241 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b6f0be56d0 0x55b6f0c5e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f685506c640 1 --2- 172.21.15.23:0/379908241 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b6f0be56d0 0x55b6f0c5e500 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98ac13642fac70c server_cookie=4ea64a75542cfdbd in_seq=0 out_seq=0 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f6834ff9640 10 client.16563 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f6834ff9640 1 -- 172.21.15.23:0/379908241 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b6f08ce890 con 0x55b6f0be56d0 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f6834ff9640 10 client.16563 handle_command_reply: tid=0 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f6859f82740 1 client.16563 shutdown 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.868+0000 7f6859f82740 1 -- 172.21.15.23:0/379908241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 msgr2=0x55b6f0be9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f6859f82740 1 --2- 172.21.15.23:0/379908241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be9a10 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6840011600 tx=0x7f6840011770 comp rx=0 tx=0).stop 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f6859f82740 1 -- 172.21.15.23:0/379908241 shutdown_connections 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f6859f82740 1 --2- 172.21.15.23:0/379908241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6f0bdcaf0 0x55b6f0be9a10 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f6859f82740 1 --2- 172.21.15.23:0/379908241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6f0bdc1d0 0x55b6f0be94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f6859f82740 1 --2- 172.21.15.23:0/379908241 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b6f0be56d0 0x55b6f0c5e500 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f6859f82740 1 --2- 172.21.15.23:0/379908241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6f0bd6900 0x55b6f0be8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f6834ff9640 0 client.16563 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:28.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f6859f82740 1 -- 172.21.15.23:0/379908241 >> 172.21.15.23:0/379908241 conn(0x55b6f0bde130 msgr2=0x55b6f0bd7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:28.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f6859f82740 1 -- 172.21.15.23:0/379908241 shutdown_connections 2024-03-24T10:43:28.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f6859f82740 1 -- 172.21.15.23:0/379908241 wait complete. 2024-03-24T10:43:28.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f6859f82740 20 client.16563 trim_cache size 0 max 16384 2024-03-24T10:43:28.876 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:43:28.876 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:43:28.876 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:43:28.876 INFO:teuthology.orchestra.run.smithi023.stdout: "bdca8b6d-e38c-4178-a11b-737bc3855b0b": { 2024-03-24T10:43:28.876 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:43:28.876 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "bdca8b6d-e38c-4178-a11b-737bc3855b0b", 2024-03-24T10:43:28.876 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:43:28.877 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:43:28.877 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:43:28.877 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:43:28.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f68572d0640 1 -- 172.21.15.23:0/3848769633 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f68380a2230 msgr2=0x7f68380a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f68572d0640 1 --2- 172.21.15.23:0/3848769633 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f68380a2230 0x7f68380a46f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6848009b50 tx=0x7f6848005f90 comp rx=0 tx=0).stop 2024-03-24T10:43:28.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f68572d0640 1 -- 172.21.15.23:0/3848769633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6850102070 msgr2=0x7f6850198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:28.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f68572d0640 1 --2- 172.21.15.23:0/3848769633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6850102070 0x7f6850198330 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f6844009230 tx=0x7f684402c650 comp rx=0 tx=0).stop 2024-03-24T10:43:28.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f68572d0640 1 -- 172.21.15.23:0/3848769633 shutdown_connections 2024-03-24T10:43:28.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f68572d0640 1 --2- 172.21.15.23:0/3848769633 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f68380a2230 0x7f68380a46f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f68572d0640 1 --2- 172.21.15.23:0/3848769633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68501076c0 0x7f685019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f68572d0640 1 --2- 172.21.15.23:0/3848769633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6850102070 0x7f6850198330 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f68572d0640 1 --2- 172.21.15.23:0/3848769633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68501016b0 0x7f6850197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:28.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f68572d0640 1 -- 172.21.15.23:0/3848769633 >> 172.21.15.23:0/3848769633 conn(0x7f68500fd140 msgr2=0x7f68500fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:28.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f68572d0640 1 -- 172.21.15.23:0/3848769633 shutdown_connections 2024-03-24T10:43:28.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:28.869+0000 7f68572d0640 1 -- 172.21.15.23:0/3848769633 wait complete. 2024-03-24T10:43:28.880 INFO:tasks.cephfs.filesystem:scrub status for tag:bdca8b6d-e38c-4178-a11b-737bc3855b0b - {'path': '/', 'tag': 'bdca8b6d-e38c-4178-a11b-737bc3855b0b', 'options': 'recursive,force'} 2024-03-24T10:43:29.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 241847 12.47 MB/sec warmup 106 sec latency 9.141 ms 2024-03-24T10:43:29.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:29 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:43:29.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:29 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:43:29.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:29 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:43:29.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:29 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:43:29.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:29 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:43:29.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:29 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:43:29.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:29 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:43:29.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:29 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:43:29.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:29 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:43:29.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:29 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:43:29.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:29 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:43:29.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:29 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:43:29.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:29 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:43:29.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:29 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:43:29.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:29 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:43:29.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:29 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:43:29.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:29 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:43:29.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:29 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:43:30.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 244766 12.51 MB/sec warmup 107 sec latency 21.689 ms 2024-03-24T10:43:30.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:30 smithi117 ceph-mon[36368]: pgmap v105: 97 pgs: 97 active+clean; 147 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 207 op/s 2024-03-24T10:43:30.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:30 smithi116 ceph-mon[36248]: pgmap v105: 97 pgs: 97 active+clean; 147 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 207 op/s 2024-03-24T10:43:30.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:30 smithi023 ceph-mon[32466]: pgmap v105: 97 pgs: 97 active+clean; 147 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 207 op/s 2024-03-24T10:43:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.055+0000 7fbc28a07640 1 -- 172.21.15.23:0/3078138592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc24100860 msgr2=0x7fbc24100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.055+0000 7fbc28a07640 1 --2- 172.21.15.23:0/3078138592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc24100860 0x7fbc24100ce0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c009900 tx=0x7fbc0c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:43:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.055+0000 7fbc28a07640 1 -- 172.21.15.23:0/3078138592 shutdown_connections 2024-03-24T10:43:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.055+0000 7fbc28a07640 1 --2- 172.21.15.23:0/3078138592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc24101220 0x7fbc2410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.055+0000 7fbc28a07640 1 --2- 172.21.15.23:0/3078138592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc24100860 0x7fbc24100ce0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.055+0000 7fbc28a07640 1 --2- 172.21.15.23:0/3078138592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc241055b0 0x7fbc24105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.055+0000 7fbc28a07640 1 -- 172.21.15.23:0/3078138592 >> 172.21.15.23:0/3078138592 conn(0x7fbc24077460 msgr2=0x7fbc24077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.055+0000 7fbc28a07640 1 -- 172.21.15.23:0/3078138592 shutdown_connections 2024-03-24T10:43:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.055+0000 7fbc28a07640 1 -- 172.21.15.23:0/3078138592 wait complete. 2024-03-24T10:43:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc28a07640 1 Processor -- start 2024-03-24T10:43:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc28a07640 1 -- start start 2024-03-24T10:43:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc28a07640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc24100860 0x7fbc2419a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc28a07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc24101220 0x7fbc2419a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc28a07640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc241055b0 0x7fbc2419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc28a07640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2419ef20 con 0x7fbc24100860 2024-03-24T10:43:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc28a07640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2419f090 con 0x7fbc24101220 2024-03-24T10:43:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc28a07640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2419f250 con 0x7fbc241055b0 2024-03-24T10:43:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc24100860 0x7fbc2419a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc24100860 0x7fbc2419a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42586/0 (socket says 172.21.15.23:42586) 2024-03-24T10:43:31.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc2259c640 1 -- 172.21.15.23:0/293757622 learned_addr learned my addr 172.21.15.23:0/293757622 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc22d9d640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc241055b0 0x7fbc2419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc1bfff640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc24101220 0x7fbc2419a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc2259c640 1 -- 172.21.15.23:0/293757622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc241055b0 msgr2=0x7fbc2419e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc2259c640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc241055b0 0x7fbc2419e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc2259c640 1 -- 172.21.15.23:0/293757622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc24101220 msgr2=0x7fbc2419a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc2259c640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc24101220 0x7fbc2419a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc2259c640 1 -- 172.21.15.23:0/293757622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0c009560 con 0x7fbc24100860 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.056+0000 7fbc22d9d640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc241055b0 0x7fbc2419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.057+0000 7fbc2259c640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc24100860 0x7fbc2419a060 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400e9f0 tx=0x7fbc1400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.057+0000 7fbc1b7fe640 1 -- 172.21.15.23:0/293757622 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc14009720 con 0x7fbc24100860 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.057+0000 7fbc1b7fe640 1 -- 172.21.15.23:0/293757622 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc1400cd80 con 0x7fbc24100860 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.057+0000 7fbc28a07640 1 -- 172.21.15.23:0/293757622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc2419f530 con 0x7fbc24100860 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.057+0000 7fbc1b7fe640 1 -- 172.21.15.23:0/293757622 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc14010690 con 0x7fbc24100860 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.057+0000 7fbc28a07640 1 -- 172.21.15.23:0/293757622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc241a6db0 con 0x7fbc24100860 2024-03-24T10:43:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.059+0000 7fbc1b7fe640 1 -- 172.21.15.23:0/293757622 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fbc140046c0 con 0x7fbc24100860 2024-03-24T10:43:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.059+0000 7fbc2b5e8740 1 Processor -- start 2024-03-24T10:43:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.059+0000 7fbc1b7fe640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbc100a22d0 0x7fbc100a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc22d9d640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbc100a22d0 0x7fbc100a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc1b7fe640 1 -- 172.21.15.23:0/293757622 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fbc14014070 con 0x7fbc24100860 2024-03-24T10:43:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2b5e8740 1 -- start start 2024-03-24T10:43:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2b5e8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a45d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2b5e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 0x557f0a466790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2b5e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 0x557f0a4690c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2b5e8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557f0a4181b0 con 0x557f0a45d5e0 2024-03-24T10:43:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2b5e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557f0a3caf80 con 0x557f0a466cd0 2024-03-24T10:43:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2b5e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557f0a3c84b0 con 0x557f0a45df00 2024-03-24T10:43:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a45d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a45d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42602/0 (socket says 172.21.15.23:42602) 2024-03-24T10:43:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2259c640 1 -- 172.21.15.23:0/3000426857 learned_addr learned my addr 172.21.15.23:0/3000426857 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc22d9d640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbc100a22d0 0x7fbc100a4790 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009fd0 tx=0x7fbc1c009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc1bfff640 1 --2- 172.21.15.23:0/3000426857 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 0x557f0a466790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc22d9d640 1 --2- 172.21.15.23:0/3000426857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 0x557f0a4690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2259c640 1 -- 172.21.15.23:0/3000426857 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 msgr2=0x557f0a466790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2259c640 1 --2- 172.21.15.23:0/3000426857 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 0x557f0a466790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2259c640 1 -- 172.21.15.23:0/3000426857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 msgr2=0x557f0a4690c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2259c640 1 --2- 172.21.15.23:0/3000426857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 0x557f0a4690c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2259c640 1 -- 172.21.15.23:0/3000426857 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f0a359810 con 0x557f0a45d5e0 2024-03-24T10:43:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.060+0000 7fbc2259c640 1 --2- 172.21.15.23:0/3000426857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a45d9c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbc140a3000 tx=0x7fbc1409a770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd4ffbd1e735950 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.061+0000 7fbc18ff9640 1 -- 172.21.15.23:0/3000426857 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1409eee0 con 0x557f0a45d5e0 2024-03-24T10:43:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.061+0000 7fbc18ff9640 1 -- 172.21.15.23:0/3000426857 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc140a3530 con 0x557f0a45d5e0 2024-03-24T10:43:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.061+0000 7fbc18ff9640 1 -- 172.21.15.23:0/3000426857 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1409a1e0 con 0x557f0a45d5e0 2024-03-24T10:43:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.061+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/3000426857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 msgr2=0x557f0a45d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.061+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/3000426857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a45d9c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbc140a3000 tx=0x7fbc1409a770 comp rx=0 tx=0).stop 2024-03-24T10:43:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.061+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/3000426857 shutdown_connections 2024-03-24T10:43:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.061+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/3000426857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 0x557f0a4690c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.061+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/3000426857 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 0x557f0a466790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.061+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/3000426857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a45d9c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.061+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/3000426857 >> 172.21.15.23:0/3000426857 conn(0x557f0a457f00 msgr2=0x557f0a4582e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.061+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/3000426857 shutdown_connections 2024-03-24T10:43:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.061+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/3000426857 wait complete. 2024-03-24T10:43:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2b5e8740 1 -- start start 2024-03-24T10:43:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2b5e8740 1 Processor -- start 2024-03-24T10:43:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2b5e8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a463cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2b5e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a45df00 0x557f0a464230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2b5e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a466cd0 0x557f0a464770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2b5e8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557f0a3c3c90 con 0x557f0a45d5e0 2024-03-24T10:43:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2b5e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557f0a368c50 con 0x557f0a45df00 2024-03-24T10:43:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2b5e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557f0a097250 con 0x557f0a466cd0 2024-03-24T10:43:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2b5e8740 10 client.0 _ll_register_callbacks cb 0x557f0a3b7290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a463cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a463cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42618/0 (socket says 172.21.15.23:42618) 2024-03-24T10:43:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2259c640 1 -- 172.21.15.23:0/4115503673 learned_addr learned my addr 172.21.15.23:0/4115503673 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc1bfff640 1 --2- 172.21.15.23:0/4115503673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a45df00 0x557f0a464230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc22d9d640 1 --2- 172.21.15.23:0/4115503673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a466cd0 0x557f0a464770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2259c640 1 -- 172.21.15.23:0/4115503673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a466cd0 msgr2=0x557f0a464770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2259c640 1 --2- 172.21.15.23:0/4115503673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a466cd0 0x557f0a464770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2259c640 1 -- 172.21.15.23:0/4115503673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a45df00 msgr2=0x557f0a464230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2259c640 1 --2- 172.21.15.23:0/4115503673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a45df00 0x557f0a464230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.062+0000 7fbc2259c640 1 -- 172.21.15.23:0/4115503673 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc241a6db0 con 0x557f0a45d5e0 2024-03-24T10:43:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc2259c640 1 --2- 172.21.15.23:0/4115503673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a463cf0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fbc140971f0 tx=0x7fbc14097680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce825af0666d442 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc09ffb640 1 -- 172.21.15.23:0/4115503673 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc140a3320 con 0x557f0a45d5e0 2024-03-24T10:43:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc09ffb640 1 -- 172.21.15.23:0/4115503673 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fbc14099090 con 0x557f0a45d5e0 2024-03-24T10:43:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/4115503673 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557f0a3e8790 con 0x557f0a45d5e0 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc09ffb640 1 -- 172.21.15.23:0/4115503673 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbc1409a1e0 con 0x557f0a45d5e0 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc2b5e8740 10 client.16575 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/4115503673 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557f0a3c2310 con 0x557f0a45d5e0 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc09ffb640 1 -- 172.21.15.23:0/4115503673 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc14089080 con 0x557f0a45d5e0 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc09ffb640 1 -- 172.21.15.23:0/4115503673 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fbc140ce070 con 0x557f0a45d5e0 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc2b5e8740 10 client.16575 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc2b5e8740 10 client.16575 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc2b5e8740 20 client.16575 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/4115503673 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557f0a476920 0x557f0a478d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc2b5e8740 4 client.16575 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/4115503673 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f0a0c3600 con 0x557f0a476920 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.063+0000 7fbc1bfff640 1 --2- 172.21.15.23:0/4115503673 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557f0a476920 0x557f0a478d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.064+0000 7fbc1bfff640 1 --2- 172.21.15.23:0/4115503673 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557f0a476920 0x557f0a478d10 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7c16c35d84c5769 server_cookie=7873a1257ce48923 in_seq=0 out_seq=0 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.064+0000 7fbc09ffb640 10 client.16575 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc09ffb640 1 -- 172.21.15.23:0/4115503673 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557f0a0c3600 con 0x557f0a476920 2024-03-24T10:43:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc09ffb640 10 client.16575 handle_command_reply: tid=0 2024-03-24T10:43:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc2b5e8740 1 client.16575 shutdown 2024-03-24T10:43:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/4115503673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 msgr2=0x557f0a463cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/4115503673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a463cf0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fbc140971f0 tx=0x7fbc14097680 comp rx=0 tx=0).stop 2024-03-24T10:43:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/4115503673 shutdown_connections 2024-03-24T10:43:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/4115503673 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557f0a476920 0x557f0a478d10 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/4115503673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a466cd0 0x557f0a464770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/4115503673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a45df00 0x557f0a464230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc09ffb640 0 client.16575 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/4115503673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a463cf0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/4115503673 >> 172.21.15.23:0/4115503673 conn(0x557f0a457f00 msgr2=0x557f0a459c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/4115503673 shutdown_connections 2024-03-24T10:43:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/4115503673 wait complete. 2024-03-24T10:43:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.065+0000 7fbc2b5e8740 20 client.16575 trim_cache size 0 max 16384 2024-03-24T10:43:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.080+0000 7fbc2b5e8740 1 Processor -- start 2024-03-24T10:43:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.080+0000 7fbc2b5e8740 1 -- start start 2024-03-24T10:43:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.080+0000 7fbc2b5e8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a466420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.080+0000 7fbc2b5e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 0x557f0a4723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.080+0000 7fbc2b5e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 0x557f0a456d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.080+0000 7fbc2b5e8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557f0a349660 con 0x557f0a45d5e0 2024-03-24T10:43:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.080+0000 7fbc2b5e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557f0a45dcf0 con 0x557f0a466cd0 2024-03-24T10:43:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.080+0000 7fbc2b5e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557f0a458510 con 0x557f0a45df00 2024-03-24T10:43:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.080+0000 7fbc2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a466420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.080+0000 7fbc2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a466420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42632/0 (socket says 172.21.15.23:42632) 2024-03-24T10:43:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.080+0000 7fbc2259c640 1 -- 172.21.15.23:0/3741609116 learned_addr learned my addr 172.21.15.23:0/3741609116 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.081+0000 7fbc22d9d640 1 --2- 172.21.15.23:0/3741609116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 0x557f0a456d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.081+0000 7fbc1bfff640 1 --2- 172.21.15.23:0/3741609116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 0x557f0a4723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.081+0000 7fbc2259c640 1 -- 172.21.15.23:0/3741609116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 msgr2=0x557f0a4723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.081+0000 7fbc2259c640 1 --2- 172.21.15.23:0/3741609116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 0x557f0a4723a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.081+0000 7fbc2259c640 1 -- 172.21.15.23:0/3741609116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 msgr2=0x557f0a456d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.081+0000 7fbc2259c640 1 --2- 172.21.15.23:0/3741609116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 0x557f0a456d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.081+0000 7fbc2259c640 1 -- 172.21.15.23:0/3741609116 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f0a35e810 con 0x557f0a45d5e0 2024-03-24T10:43:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.081+0000 7fbc2259c640 1 --2- 172.21.15.23:0/3741609116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a466420 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400e9c0 tx=0x7fbc140ce320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c75d38c658eab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.081+0000 7fbc0bfff640 1 -- 172.21.15.23:0/3741609116 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1409ef50 con 0x557f0a45d5e0 2024-03-24T10:43:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.081+0000 7fbc0bfff640 1 -- 172.21.15.23:0/3741609116 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557f0a3e8790 con 0x557f0a45d5e0 2024-03-24T10:43:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.081+0000 7fbc0bfff640 1 -- 172.21.15.23:0/3741609116 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc140b3ac0 con 0x557f0a45d5e0 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/3741609116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 msgr2=0x557f0a466420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/3741609116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a466420 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400e9c0 tx=0x7fbc140ce320 comp rx=0 tx=0).stop 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/3741609116 shutdown_connections 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/3741609116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 0x557f0a456d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/3741609116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 0x557f0a4723a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/3741609116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a466420 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/3741609116 >> 172.21.15.23:0/3741609116 conn(0x557f0a457f00 msgr2=0x557f0a456610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/3741609116 shutdown_connections 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/3741609116 wait complete. 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 -- start start 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 Processor -- start 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a46ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 0x557f0a460650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 0x557f0a460c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557f0a461380 con 0x557f0a45d5e0 2024-03-24T10:43:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557f0a4614f0 con 0x557f0a466cd0 2024-03-24T10:43:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557f0a4dbc80 con 0x557f0a45df00 2024-03-24T10:43:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2b5e8740 10 client.0 _ll_register_callbacks cb 0x7fbc14005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a46ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.082+0000 7fbc2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a46ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42646/0 (socket says 172.21.15.23:42646) 2024-03-24T10:43:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc2259c640 1 -- 172.21.15.23:0/143881903 learned_addr learned my addr 172.21.15.23:0/143881903 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc1bfff640 1 --2- 172.21.15.23:0/143881903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 0x557f0a460650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc22d9d640 1 --2- 172.21.15.23:0/143881903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 0x557f0a460c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc2259c640 1 -- 172.21.15.23:0/143881903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 msgr2=0x557f0a460650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc2259c640 1 --2- 172.21.15.23:0/143881903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 0x557f0a460650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc2259c640 1 -- 172.21.15.23:0/143881903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 msgr2=0x557f0a460c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc2259c640 1 --2- 172.21.15.23:0/143881903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 0x557f0a460c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc2259c640 1 -- 172.21.15.23:0/143881903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f0a3c2310 con 0x557f0a45d5e0 2024-03-24T10:43:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc2259c640 1 --2- 172.21.15.23:0/143881903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a46ed70 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fbc140a3000 tx=0x7fbc1409a770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c37c9f2f4787a4f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc09ffb640 1 -- 172.21.15.23:0/143881903 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1409ef50 con 0x557f0a45d5e0 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/143881903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc140a3530 con 0x557f0a45d5e0 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc09ffb640 1 -- 172.21.15.23:0/143881903 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fbc14099f80 con 0x557f0a45d5e0 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc09ffb640 1 -- 172.21.15.23:0/143881903 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbc1409a8d0 con 0x557f0a45d5e0 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc2b5e8740 10 client.16581 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/143881903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557f0a3845a0 con 0x557f0a45d5e0 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc09ffb640 1 -- 172.21.15.23:0/143881903 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc140bf020 con 0x557f0a45d5e0 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.083+0000 7fbc09ffb640 1 -- 172.21.15.23:0/143881903 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fbc140ce040 con 0x557f0a45d5e0 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.084+0000 7fbc2b5e8740 10 client.16581 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.084+0000 7fbc2b5e8740 10 client.16581 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.084+0000 7fbc2b5e8740 20 client.16581 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.084+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/143881903 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557f0a4dcf20 0x557f0a4df310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.084+0000 7fbc2b5e8740 4 client.16581 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:43:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.084+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/143881903 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557f0a1141b0 con 0x557f0a4dcf20 2024-03-24T10:43:31.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.084+0000 7fbc1bfff640 1 --2- 172.21.15.23:0/143881903 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557f0a4dcf20 0x557f0a4df310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:31.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.084+0000 7fbc1bfff640 1 --2- 172.21.15.23:0/143881903 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557f0a4dcf20 0x557f0a4df310 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1b65b170f585d4 server_cookie=98e60bb9ea81fa25 in_seq=0 out_seq=0 2024-03-24T10:43:31.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.084+0000 7fbc09ffb640 10 client.16581 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.092+0000 7fbc09ffb640 1 -- 172.21.15.23:0/143881903 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557f0a1141b0 con 0x557f0a4dcf20 2024-03-24T10:43:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.092+0000 7fbc09ffb640 10 client.16581 handle_command_reply: tid=0 2024-03-24T10:43:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc2b5e8740 1 client.16581 shutdown 2024-03-24T10:43:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/143881903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 msgr2=0x557f0a46ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/143881903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a46ed70 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fbc140a3000 tx=0x7fbc1409a770 comp rx=0 tx=0).stop 2024-03-24T10:43:31.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/143881903 shutdown_connections 2024-03-24T10:43:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/143881903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557f0a466cd0 0x557f0a460c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/143881903 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557f0a4dcf20 0x557f0a4df310 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/143881903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557f0a45df00 0x557f0a460650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc2b5e8740 1 --2- 172.21.15.23:0/143881903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f0a45d5e0 0x557f0a46ed70 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc09ffb640 0 client.16581 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/143881903 >> 172.21.15.23:0/143881903 conn(0x557f0a457f00 msgr2=0x557f0a456ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/143881903 shutdown_connections 2024-03-24T10:43:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc2b5e8740 1 -- 172.21.15.23:0/143881903 wait complete. 2024-03-24T10:43:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc2b5e8740 20 client.16581 trim_cache size 0 max 16384 2024-03-24T10:43:31.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:43:31.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:43:31.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:43:31.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:43:31.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc28a07640 1 -- 172.21.15.23:0/293757622 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbc100a22d0 msgr2=0x7fbc100a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:31.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc28a07640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbc100a22d0 0x7fbc100a4790 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009fd0 tx=0x7fbc1c009500 comp rx=0 tx=0).stop 2024-03-24T10:43:31.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc28a07640 1 -- 172.21.15.23:0/293757622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc24100860 msgr2=0x7fbc2419a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:31.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc28a07640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc24100860 0x7fbc2419a060 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400e9f0 tx=0x7fbc1400edf0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc28a07640 1 -- 172.21.15.23:0/293757622 shutdown_connections 2024-03-24T10:43:31.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc28a07640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbc100a22d0 0x7fbc100a4790 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc28a07640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc241055b0 0x7fbc2419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc28a07640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc24101220 0x7fbc2419a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc28a07640 1 --2- 172.21.15.23:0/293757622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc24100860 0x7fbc2419a060 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:31.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.093+0000 7fbc28a07640 1 -- 172.21.15.23:0/293757622 >> 172.21.15.23:0/293757622 conn(0x7fbc24077460 msgr2=0x7fbc240feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:31.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.094+0000 7fbc28a07640 1 -- 172.21.15.23:0/293757622 shutdown_connections 2024-03-24T10:43:31.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:31.094+0000 7fbc28a07640 1 -- 172.21.15.23:0/293757622 wait complete. 2024-03-24T10:43:31.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 246380 12.46 MB/sec warmup 108 sec latency 56.862 ms 2024-03-24T10:43:32.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 248801 12.47 MB/sec warmup 109 sec latency 23.191 ms 2024-03-24T10:43:32.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:32 smithi117 ceph-mon[36368]: pgmap v106: 97 pgs: 97 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 308 op/s 2024-03-24T10:43:32.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:32 smithi116 ceph-mon[36248]: pgmap v106: 97 pgs: 97 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 308 op/s 2024-03-24T10:43:32.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:32 smithi023 ceph-mon[32466]: pgmap v106: 97 pgs: 97 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 308 op/s 2024-03-24T10:43:33.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 251669 12.49 MB/sec warmup 110 sec latency 19.932 ms 2024-03-24T10:43:34.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 253520 12.47 MB/sec warmup 111 sec latency 17.739 ms 2024-03-24T10:43:34.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:34 smithi117 ceph-mon[36368]: pgmap v107: 97 pgs: 97 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 202 op/s 2024-03-24T10:43:34.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:34 smithi116 ceph-mon[36248]: pgmap v107: 97 pgs: 97 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 202 op/s 2024-03-24T10:43:34.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:34 smithi023 ceph-mon[32466]: pgmap v107: 97 pgs: 97 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 202 op/s 2024-03-24T10:43:35.115 INFO:tasks.workunit.client.0.smithi023.stdout: 1 255987 12.48 MB/sec warmup 112 sec latency 15.049 ms 2024-03-24T10:43:35.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:35 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:43:35.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:35 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:43:35.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:35 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:43:35.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:35 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:43:35.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:35 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:43:35.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:35 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:43:36.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 258353 12.48 MB/sec warmup 113 sec latency 120.814 ms 2024-03-24T10:43:36.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:36 smithi117 ceph-mon[36368]: pgmap v108: 97 pgs: 97 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 202 op/s 2024-03-24T10:43:36.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:36 smithi116 ceph-mon[36248]: pgmap v108: 97 pgs: 97 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 202 op/s 2024-03-24T10:43:36.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:36 smithi023 ceph-mon[32466]: pgmap v108: 97 pgs: 97 active+clean; 150 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 202 op/s 2024-03-24T10:43:37.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 259959 12.44 MB/sec warmup 114 sec latency 26.295 ms 2024-03-24T10:43:38.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 262768 12.48 MB/sec warmup 115 sec latency 14.502 ms 2024-03-24T10:43:38.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:38 smithi116 ceph-mon[36248]: pgmap v109: 97 pgs: 97 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 303 op/s 2024-03-24T10:43:38.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:38 smithi023 ceph-mon[32466]: pgmap v109: 97 pgs: 97 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 303 op/s 2024-03-24T10:43:39.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:38 smithi117 ceph-mon[36368]: pgmap v109: 97 pgs: 97 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 303 op/s 2024-03-24T10:43:39.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 264867 12.44 MB/sec warmup 116 sec latency 14.207 ms 2024-03-24T10:43:40.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 266819 12.44 MB/sec warmup 117 sec latency 19.391 ms 2024-03-24T10:43:40.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:40 smithi116 ceph-mon[36248]: pgmap v110: 97 pgs: 97 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 202 op/s 2024-03-24T10:43:40.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:40 smithi023 ceph-mon[32466]: pgmap v110: 97 pgs: 97 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 202 op/s 2024-03-24T10:43:41.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:40 smithi117 ceph-mon[36368]: pgmap v110: 97 pgs: 97 active+clean; 158 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 202 op/s 2024-03-24T10:43:41.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 269736 12.48 MB/sec warmup 118 sec latency 14.371 ms 2024-03-24T10:43:42.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 271108 12.43 MB/sec warmup 119 sec latency 114.539 ms 2024-03-24T10:43:42.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:42 smithi116 ceph-mon[36248]: pgmap v111: 97 pgs: 97 active+clean; 165 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 306 op/s 2024-03-24T10:43:42.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:42 smithi023 ceph-mon[32466]: pgmap v111: 97 pgs: 97 active+clean; 165 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 306 op/s 2024-03-24T10:43:43.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:42 smithi117 ceph-mon[36368]: pgmap v111: 97 pgs: 97 active+clean; 165 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 306 op/s 2024-03-24T10:43:44.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 276155 12.59 MB/sec execute 1 sec latency 82.485 ms 2024-03-24T10:43:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.183+0000 7fb7c5415640 1 -- 172.21.15.23:0/4164744662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7c0100860 msgr2=0x7fb7c0100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.183+0000 7fb7c5415640 1 --2- 172.21.15.23:0/4164744662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7c0100860 0x7fb7c0100cc0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8009900 tx=0x7fb7a802bc30 comp rx=0 tx=0).stop 2024-03-24T10:43:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.183+0000 7fb7c5415640 1 -- 172.21.15.23:0/4164744662 shutdown_connections 2024-03-24T10:43:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.183+0000 7fb7c5415640 1 --2- 172.21.15.23:0/4164744662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c0101200 0x7fb7c010d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.183+0000 7fb7c5415640 1 --2- 172.21.15.23:0/4164744662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7c0100860 0x7fb7c0100cc0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.183+0000 7fb7c5415640 1 --2- 172.21.15.23:0/4164744662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7c0105590 0x7fb7c0105970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.183+0000 7fb7c5415640 1 -- 172.21.15.23:0/4164744662 >> 172.21.15.23:0/4164744662 conn(0x7fb7c00773e0 msgr2=0x7fb7c00777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.183+0000 7fb7c5415640 1 -- 172.21.15.23:0/4164744662 shutdown_connections 2024-03-24T10:43:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.183+0000 7fb7c5415640 1 -- 172.21.15.23:0/4164744662 wait complete. 2024-03-24T10:43:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.183+0000 7fb7c5415640 1 Processor -- start 2024-03-24T10:43:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.183+0000 7fb7c5415640 1 -- start start 2024-03-24T10:43:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7c5415640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7c0100860 0x7fb7c019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7c5415640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c0101200 0x7fb7c019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7c5415640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7c0105590 0x7fb7c0196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7c5415640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c01970d0 con 0x7fb7c0105590 2024-03-24T10:43:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7c5415640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c0197240 con 0x7fb7c0101200 2024-03-24T10:43:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7c5415640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c0197400 con 0x7fb7c0100860 2024-03-24T10:43:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7c0105590 0x7fb7c0196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7beffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7c0100860 0x7fb7c019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7c0105590 0x7fb7c0196a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36080/0 (socket says 172.21.15.23:36080) 2024-03-24T10:43:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7bf7fe640 1 -- 172.21.15.23:0/2364099472 learned_addr learned my addr 172.21.15.23:0/2364099472 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c0101200 0x7fb7c019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7beffd640 1 -- 172.21.15.23:0/2364099472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c0101200 msgr2=0x7fb7c019cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7beffd640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c0101200 0x7fb7c019cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7beffd640 1 -- 172.21.15.23:0/2364099472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7c0105590 msgr2=0x7fb7c0196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7beffd640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7c0105590 0x7fb7c0196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7beffd640 1 -- 172.21.15.23:0/2364099472 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a8009560 con 0x7fb7c0100860 2024-03-24T10:43:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c0101200 0x7fb7c019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7bf7fe640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7c0105590 0x7fb7c0196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7beffd640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7c0100860 0x7fb7c019c880 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009fd0 tx=0x7fb7b800b690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7b7fff640 1 -- 172.21.15.23:0/2364099472 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8013070 con 0x7fb7c0100860 2024-03-24T10:43:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.184+0000 7fb7c5415640 1 -- 172.21.15.23:0/2364099472 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c01976e0 con 0x7fb7c0100860 2024-03-24T10:43:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.185+0000 7fb7b7fff640 1 -- 172.21.15.23:0/2364099472 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b8017450 con 0x7fb7c0100860 2024-03-24T10:43:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.185+0000 7fb7b7fff640 1 -- 172.21.15.23:0/2364099472 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8005280 con 0x7fb7c0100860 2024-03-24T10:43:44.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.185+0000 7fb7c5415640 1 -- 172.21.15.23:0/2364099472 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c006a0f0 con 0x7fb7c0100860 2024-03-24T10:43:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.187+0000 7fb7b7fff640 1 -- 172.21.15.23:0/2364099472 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fb7b80175c0 con 0x7fb7c0100860 2024-03-24T10:43:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.187+0000 7fb7c7ff4740 1 Processor -- start 2024-03-24T10:43:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.187+0000 7fb7c7ff4740 1 -- start start 2024-03-24T10:43:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.187+0000 7fb7c7ff4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edb100 0x5612a3edb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.187+0000 7fb7c7ff4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edba20 0x5612a3ee8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.187+0000 7fb7c7ff4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 0x5612a3eeb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.187+0000 7fb7b7fff640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7ac0a22d0 0x7fb7ac0a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7c7ff4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5612a3e961b0 con 0x5612a3edba20 2024-03-24T10:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7c7ff4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5612a3e48f80 con 0x5612a3ee90e0 2024-03-24T10:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7c7ff4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5612a3e464b0 con 0x5612a3edb100 2024-03-24T10:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7bf7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edb100 0x5612a3edb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7bf7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edb100 0x5612a3edb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:32884/0 (socket says 172.21.15.23:32884) 2024-03-24T10:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7bf7fe640 1 -- 172.21.15.23:0/3099801993 learned_addr learned my addr 172.21.15.23:0/3099801993 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7be7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 0x5612a3eeb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7bf7fe640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7ac0a22d0 0x7fb7ac0a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7b7fff640 1 -- 172.21.15.23:0/2364099472 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fb7b80cfb70 con 0x7fb7c0100860 2024-03-24T10:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7beffd640 1 --2- 172.21.15.23:0/3099801993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edba20 0x5612a3ee8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7be7fc640 1 -- 172.21.15.23:0/3099801993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edb100 msgr2=0x5612a3edb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/3099801993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edb100 0x5612a3edb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7be7fc640 1 -- 172.21.15.23:0/3099801993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edba20 msgr2=0x5612a3ee8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/3099801993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edba20 0x5612a3ee8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7be7fc640 1 -- 172.21.15.23:0/3099801993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612a3dd7810 con 0x5612a3ee90e0 2024-03-24T10:43:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7bf7fe640 1 --2- 172.21.15.23:0/3099801993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edb100 0x5612a3edb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7bf7fe640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7ac0a22d0 0x7fb7ac0a4790 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0011f30 tx=0x7fb7b0005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.188+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/3099801993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 0x5612a3eeb4d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8000c00 tx=0x7fb7a802c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=149c3723e55fb333 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.189+0000 7fb7b57fa640 1 -- 172.21.15.23:0/3099801993 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a802c7e0 con 0x5612a3ee90e0 2024-03-24T10:43:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.189+0000 7fb7b57fa640 1 -- 172.21.15.23:0/3099801993 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7a8004510 con 0x5612a3ee90e0 2024-03-24T10:43:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.189+0000 7fb7b57fa640 1 -- 172.21.15.23:0/3099801993 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a80053b0 con 0x5612a3ee90e0 2024-03-24T10:43:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.189+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/3099801993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 msgr2=0x5612a3eeb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.189+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/3099801993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 0x5612a3eeb4d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8000c00 tx=0x7fb7a802c680 comp rx=0 tx=0).stop 2024-03-24T10:43:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/3099801993 shutdown_connections 2024-03-24T10:43:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/3099801993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 0x5612a3eeb4d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/3099801993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edba20 0x5612a3ee8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/3099801993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edb100 0x5612a3edb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/3099801993 >> 172.21.15.23:0/3099801993 conn(0x5612a3ed5f00 msgr2=0x5612a3ed62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/3099801993 shutdown_connections 2024-03-24T10:43:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/3099801993 wait complete. 2024-03-24T10:43:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 -- start start 2024-03-24T10:43:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 Processor -- start 2024-03-24T10:43:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edb100 0x5612a3ee60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edba20 0x5612a3ee6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 0x5612a3ee6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5612a3e41c90 con 0x5612a3edb100 2024-03-24T10:43:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5612a3de6c50 con 0x5612a3ee90e0 2024-03-24T10:43:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5612a3b15250 con 0x5612a3edba20 2024-03-24T10:43:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7c7ff4740 10 client.0 _ll_register_callbacks cb 0x5612a3e35290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.190+0000 7fb7beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edb100 0x5612a3ee60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edb100 0x5612a3ee60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36094/0 (socket says 172.21.15.23:36094) 2024-03-24T10:43:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7be7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edba20 0x5612a3ee6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7beffd640 1 -- 172.21.15.23:0/751431570 learned_addr learned my addr 172.21.15.23:0/751431570 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7bf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 0x5612a3ee6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7beffd640 1 -- 172.21.15.23:0/751431570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edba20 msgr2=0x5612a3ee6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7beffd640 1 --2- 172.21.15.23:0/751431570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edba20 0x5612a3ee6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7beffd640 1 -- 172.21.15.23:0/751431570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 msgr2=0x5612a3ee6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7beffd640 1 --2- 172.21.15.23:0/751431570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 0x5612a3ee6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7beffd640 1 -- 172.21.15.23:0/751431570 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c010ddc0 con 0x5612a3edb100 2024-03-24T10:43:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/751431570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edba20 0x5612a3ee6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7beffd640 1 --2- 172.21.15.23:0/751431570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edb100 0x5612a3ee60e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb7b809c310 tx=0x7fb7b809c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee03bb2db5e029ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7a67fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7a67fc640 1 -- 172.21.15.23:0/751431570 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b809d690 con 0x5612a3edb100 2024-03-24T10:43:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7a67fc640 1 -- 172.21.15.23:0/751431570 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fb7b809c610 con 0x5612a3edb100 2024-03-24T10:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/751431570 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5612a3e66790 con 0x5612a3edb100 2024-03-24T10:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7a67fc640 1 -- 172.21.15.23:0/751431570 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb7b809cd10 con 0x5612a3edb100 2024-03-24T10:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7c7ff4740 10 client.16593 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.191+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/751431570 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5612a3e40310 con 0x5612a3edb100 2024-03-24T10:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.192+0000 7fb7a67fc640 1 -- 172.21.15.23:0/751431570 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5612a3e66790 con 0x5612a3edb100 2024-03-24T10:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.192+0000 7fb7a67fc640 1 -- 172.21.15.23:0/751431570 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb7b80a1860 con 0x5612a3edb100 2024-03-24T10:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.192+0000 7fb7c7ff4740 10 client.16593 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.192+0000 7fb7c7ff4740 10 client.16593 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.192+0000 7fb7c7ff4740 20 client.16593 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.192+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/751431570 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5612a3ef8fd0 0x5612a3efb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.192+0000 7fb7c7ff4740 4 client.16593 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.192+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/751431570 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612a3b41600 con 0x5612a3ef8fd0 2024-03-24T10:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.192+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/751431570 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5612a3ef8fd0 0x5612a3efb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.192+0000 7fb7a67fc640 10 client.16593 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.192+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/751431570 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5612a3ef8fd0 0x5612a3efb430 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee5e7019baad3a83 server_cookie=12e1b5ccf546c947 in_seq=0 out_seq=0 2024-03-24T10:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.193+0000 7fb7a67fc640 1 -- 172.21.15.23:0/751431570 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5612a3b41600 con 0x5612a3ef8fd0 2024-03-24T10:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.193+0000 7fb7a67fc640 10 client.16593 handle_command_reply: tid=0 2024-03-24T10:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.193+0000 7fb7c7ff4740 1 client.16593 shutdown 2024-03-24T10:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.193+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/751431570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edb100 msgr2=0x5612a3ee60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.193+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/751431570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edb100 0x5612a3ee60e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb7b809c310 tx=0x7fb7b809c1d0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.193+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/751431570 shutdown_connections 2024-03-24T10:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.193+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/751431570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 0x5612a3ee6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.193+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/751431570 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5612a3ef8fd0 0x5612a3efb430 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.193+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/751431570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edba20 0x5612a3ee6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.193+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/751431570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edb100 0x5612a3ee60e0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.193+0000 7fb7a67fc640 0 client.16593 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.193+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/751431570 >> 172.21.15.23:0/751431570 conn(0x5612a3ed5f00 msgr2=0x5612a3ed7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.193+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/751431570 shutdown_connections 2024-03-24T10:43:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.193+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/751431570 wait complete. 2024-03-24T10:43:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.194+0000 7fb7c7ff4740 20 client.16593 trim_cache size 0 max 16384 2024-03-24T10:43:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.209+0000 7fb7c7ff4740 1 Processor -- start 2024-03-24T10:43:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.209+0000 7fb7c7ff4740 1 -- start start 2024-03-24T10:43:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.209+0000 7fb7c7ff4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edb100 0x5612a3ef4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.209+0000 7fb7c7ff4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edba20 0x5612a3ee6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.209+0000 7fb7c7ff4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 0x5612a3ed4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.209+0000 7fb7c7ff4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5612a3dc7660 con 0x5612a3edba20 2024-03-24T10:43:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.209+0000 7fb7c7ff4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5612a3ed6ad0 con 0x5612a3ee90e0 2024-03-24T10:43:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.209+0000 7fb7c7ff4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5612a3ed0160 con 0x5612a3edb100 2024-03-24T10:43:44.214 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.209+0000 7fb7be7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edba20 0x5612a3ee6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.209+0000 7fb7be7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edba20 0x5612a3ee6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36098/0 (socket says 172.21.15.23:36098) 2024-03-24T10:43:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.209+0000 7fb7be7fc640 1 -- 172.21.15.23:0/3990634329 learned_addr learned my addr 172.21.15.23:0/3990634329 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.209+0000 7fb7beffd640 1 --2- 172.21.15.23:0/3990634329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edb100 0x5612a3ef4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.209+0000 7fb7bf7fe640 1 --2- 172.21.15.23:0/3990634329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 0x5612a3ed4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.210+0000 7fb7be7fc640 1 -- 172.21.15.23:0/3990634329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edb100 msgr2=0x5612a3ef4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.210+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/3990634329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edb100 0x5612a3ef4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.210+0000 7fb7be7fc640 1 -- 172.21.15.23:0/3990634329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 msgr2=0x5612a3ed4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.210+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/3990634329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 0x5612a3ed4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.210+0000 7fb7be7fc640 1 -- 172.21.15.23:0/3990634329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612a3ddc810 con 0x5612a3edba20 2024-03-24T10:43:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.210+0000 7fb7bf7fe640 1 --2- 172.21.15.23:0/3990634329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 0x5612a3ed4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:43:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.210+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/3990634329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edba20 0x5612a3ee6980 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x5612a3ed7320 tx=0x7fb7a8041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d1728cf02ca136 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.210+0000 7fb7b4ff9640 1 -- 172.21.15.23:0/3990634329 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8004af0 con 0x5612a3edba20 2024-03-24T10:43:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.210+0000 7fb7b4ff9640 1 -- 172.21.15.23:0/3990634329 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7a8004df0 con 0x5612a3edba20 2024-03-24T10:43:44.215 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.210+0000 7fb7b4ff9640 1 -- 172.21.15.23:0/3990634329 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a80052f0 con 0x5612a3edba20 2024-03-24T10:43:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.211+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/3990634329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edba20 msgr2=0x5612a3ee6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.211+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/3990634329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edba20 0x5612a3ee6980 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x5612a3ed7320 tx=0x7fb7a8041040 comp rx=0 tx=0).stop 2024-03-24T10:43:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.211+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/3990634329 shutdown_connections 2024-03-24T10:43:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.211+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/3990634329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3ee90e0 0x5612a3ed4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.211+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/3990634329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edba20 0x5612a3ee6980 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.211+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/3990634329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3edb100 0x5612a3ef4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.211+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/3990634329 >> 172.21.15.23:0/3990634329 conn(0x5612a3ed5f00 msgr2=0x5612a3ed4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.211+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/3990634329 shutdown_connections 2024-03-24T10:43:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.211+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/3990634329 wait complete. 2024-03-24T10:43:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.211+0000 7fb7c7ff4740 1 -- start start 2024-03-24T10:43:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.211+0000 7fb7c7ff4740 1 Processor -- start 2024-03-24T10:43:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7c7ff4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edb100 0x5612a3ef0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7c7ff4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3edba20 0x5612a3ef10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.216 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7c7ff4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3ee90e0 0x5612a3eec260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7c7ff4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5612a3eec930 con 0x5612a3edb100 2024-03-24T10:43:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7c7ff4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5612a3eecaa0 con 0x5612a3edba20 2024-03-24T10:43:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7c7ff4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5612a3eecc60 con 0x5612a3ee90e0 2024-03-24T10:43:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7c7ff4740 10 client.0 _ll_register_callbacks cb 0x7fb7b80988b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edb100 0x5612a3ef0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edb100 0x5612a3ef0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36110/0 (socket says 172.21.15.23:36110) 2024-03-24T10:43:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/1830572325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3edba20 0x5612a3ef10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7beffd640 1 -- 172.21.15.23:0/1830572325 learned_addr learned my addr 172.21.15.23:0/1830572325 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7bf7fe640 1 --2- 172.21.15.23:0/1830572325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3ee90e0 0x5612a3eec260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7beffd640 1 -- 172.21.15.23:0/1830572325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3ee90e0 msgr2=0x5612a3eec260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7beffd640 1 --2- 172.21.15.23:0/1830572325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3ee90e0 0x5612a3eec260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7beffd640 1 -- 172.21.15.23:0/1830572325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3edba20 msgr2=0x5612a3ef10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7beffd640 1 --2- 172.21.15.23:0/1830572325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3edba20 0x5612a3ef10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7beffd640 1 -- 172.21.15.23:0/1830572325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612a3dd7810 con 0x5612a3edb100 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/1830572325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3edba20 0x5612a3ef10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7beffd640 1 --2- 172.21.15.23:0/1830572325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edb100 0x5612a3ef0b80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80cd8c0 tx=0x7fb7b8098960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be06e04ff2c8973c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7a67fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7a67fc640 1 -- 172.21.15.23:0/1830572325 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b809c3e0 con 0x5612a3edb100 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7a67fc640 1 -- 172.21.15.23:0/1830572325 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fb7b809c610 con 0x5612a3edb100 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/1830572325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7a8004510 con 0x5612a3edb100 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7a67fc640 1 -- 172.21.15.23:0/1830572325 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb7b809bf70 con 0x5612a3edb100 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7c7ff4740 10 client.16599 fetch_fsmap learned FSMap version 25 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.212+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/1830572325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5612a3e025a0 con 0x5612a3edb100 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.213+0000 7fb7a67fc640 1 -- 172.21.15.23:0/1830572325 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b80a8080 con 0x5612a3edb100 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.213+0000 7fb7a67fc640 1 -- 172.21.15.23:0/1830572325 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb7b8085b50 con 0x5612a3edb100 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.213+0000 7fb7c7ff4740 10 client.16599 fetch_fsmap finished waiting for FSMap version 25 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.213+0000 7fb7c7ff4740 10 client.16599 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.213+0000 7fb7c7ff4740 20 client.16599 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.213+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/1830572325 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5612a3f5f4e0 0x5612a3f61900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.213+0000 7fb7c7ff4740 4 client.16599 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:43:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.213+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/1830572325 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5612a3b921b0 con 0x5612a3f5f4e0 2024-03-24T10:43:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.213+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/1830572325 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5612a3f5f4e0 0x5612a3f61900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.213+0000 7fb7a67fc640 10 client.16599 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:44.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.213+0000 7fb7be7fc640 1 --2- 172.21.15.23:0/1830572325 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5612a3f5f4e0 0x5612a3f61900 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a5e5a784792e191 server_cookie=5bf8f6e6fc50f8b6 in_seq=0 out_seq=0 2024-03-24T10:43:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7a67fc640 1 -- 172.21.15.23:0/1830572325 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5612a3b921b0 con 0x5612a3f5f4e0 2024-03-24T10:43:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7a67fc640 10 client.16599 handle_command_reply: tid=0 2024-03-24T10:43:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7c7ff4740 1 client.16599 shutdown 2024-03-24T10:43:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/1830572325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edb100 msgr2=0x5612a3ef0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/1830572325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edb100 0x5612a3ef0b80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80cd8c0 tx=0x7fb7b8098960 comp rx=0 tx=0).stop 2024-03-24T10:43:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/1830572325 shutdown_connections 2024-03-24T10:43:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/1830572325 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5612a3f5f4e0 0x5612a3f61900 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/1830572325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5612a3ee90e0 0x5612a3eec260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/1830572325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5612a3edba20 0x5612a3ef10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7c7ff4740 1 --2- 172.21.15.23:0/1830572325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5612a3edb100 0x5612a3ef0b80 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7a67fc640 0 client.16599 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/1830572325 >> 172.21.15.23:0/1830572325 conn(0x5612a3ed5f00 msgr2=0x5612a3ed4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/1830572325 shutdown_connections 2024-03-24T10:43:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7c7ff4740 1 -- 172.21.15.23:0/1830572325 wait complete. 2024-03-24T10:43:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.222+0000 7fb7c7ff4740 20 client.16599 trim_cache size 0 max 16384 2024-03-24T10:43:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:43:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:43:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:43:44.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:43:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.223+0000 7fb7c5415640 1 -- 172.21.15.23:0/2364099472 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7ac0a22d0 msgr2=0x7fb7ac0a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.223+0000 7fb7c5415640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7ac0a22d0 0x7fb7ac0a4790 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0011f30 tx=0x7fb7b0005f90 comp rx=0 tx=0).stop 2024-03-24T10:43:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.223+0000 7fb7c5415640 1 -- 172.21.15.23:0/2364099472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7c0100860 msgr2=0x7fb7c019c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.223+0000 7fb7c5415640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7c0100860 0x7fb7c019c880 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009fd0 tx=0x7fb7b800b690 comp rx=0 tx=0).stop 2024-03-24T10:43:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.223+0000 7fb7c5415640 1 -- 172.21.15.23:0/2364099472 shutdown_connections 2024-03-24T10:43:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.223+0000 7fb7c5415640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb7ac0a22d0 0x7fb7ac0a4790 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.223+0000 7fb7c5415640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7c0105590 0x7fb7c0196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.223+0000 7fb7c5415640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7c0101200 0x7fb7c019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.223+0000 7fb7c5415640 1 --2- 172.21.15.23:0/2364099472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7c0100860 0x7fb7c019c880 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.223+0000 7fb7c5415640 1 -- 172.21.15.23:0/2364099472 >> 172.21.15.23:0/2364099472 conn(0x7fb7c00773e0 msgr2=0x7fb7c00fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.223+0000 7fb7c5415640 1 -- 172.21.15.23:0/2364099472 shutdown_connections 2024-03-24T10:43:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:44.223+0000 7fb7c5415640 1 -- 172.21.15.23:0/2364099472 wait complete. 2024-03-24T10:43:44.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:44 smithi116 ceph-mon[36248]: pgmap v112: 97 pgs: 97 active+clean; 165 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-03-24T10:43:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:44 smithi023 ceph-mon[32466]: pgmap v112: 97 pgs: 97 active+clean; 165 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-03-24T10:43:45.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:44 smithi117 ceph-mon[36368]: pgmap v112: 97 pgs: 97 active+clean; 165 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-03-24T10:43:45.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 278145 12.10 MB/sec execute 2 sec latency 13.063 ms 2024-03-24T10:43:46.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 280710 12.75 MB/sec execute 3 sec latency 17.558 ms 2024-03-24T10:43:46.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:46 smithi116 ceph-mon[36248]: pgmap v113: 97 pgs: 97 active+clean; 165 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-03-24T10:43:46.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:46 smithi023 ceph-mon[32466]: pgmap v113: 97 pgs: 97 active+clean; 165 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-03-24T10:43:47.005 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:46 smithi117 ceph-mon[36368]: pgmap v113: 97 pgs: 97 active+clean; 165 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-03-24T10:43:47.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 282800 11.55 MB/sec execute 4 sec latency 139.871 ms 2024-03-24T10:43:48.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 284784 11.66 MB/sec execute 5 sec latency 15.634 ms 2024-03-24T10:43:48.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:48 smithi116 ceph-mon[36248]: pgmap v114: 97 pgs: 97 active+clean; 169 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 307 op/s 2024-03-24T10:43:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:48 smithi023 ceph-mon[32466]: pgmap v114: 97 pgs: 97 active+clean; 169 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 307 op/s 2024-03-24T10:43:49.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:48 smithi117 ceph-mon[36368]: pgmap v114: 97 pgs: 97 active+clean; 169 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 307 op/s 2024-03-24T10:43:49.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 287633 12.70 MB/sec execute 6 sec latency 9.833 ms 2024-03-24T10:43:50.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 289891 12.06 MB/sec execute 7 sec latency 23.705 ms 2024-03-24T10:43:50.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:50 smithi116 ceph-mon[36248]: pgmap v115: 97 pgs: 97 active+clean; 169 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 206 op/s 2024-03-24T10:43:50.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:50 smithi023 ceph-mon[32466]: pgmap v115: 97 pgs: 97 active+clean; 169 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 206 op/s 2024-03-24T10:43:51.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:50 smithi117 ceph-mon[36368]: pgmap v115: 97 pgs: 97 active+clean; 169 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 206 op/s 2024-03-24T10:43:51.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 291819 12.05 MB/sec execute 8 sec latency 17.528 ms 2024-03-24T10:43:51.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:51 smithi023 ceph-mon[32466]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:43:52.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:51 smithi117 ceph-mon[36368]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:43:52.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:51 smithi116 ceph-mon[36248]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:43:52.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 294491 12.58 MB/sec execute 9 sec latency 63.526 ms 2024-03-24T10:43:52.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:52 smithi023 ceph-mon[32466]: pgmap v116: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 298 op/s 2024-03-24T10:43:53.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:52 smithi117 ceph-mon[36368]: pgmap v116: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 298 op/s 2024-03-24T10:43:53.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:52 smithi116 ceph-mon[36248]: pgmap v116: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 298 op/s 2024-03-24T10:43:53.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 296201 12.00 MB/sec execute 10 sec latency 20.468 ms 2024-03-24T10:43:54.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 298570 12.17 MB/sec execute 11 sec latency 14.889 ms 2024-03-24T10:43:54.804 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:54 smithi117 ceph-mon[36368]: pgmap v117: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 194 op/s 2024-03-24T10:43:54.819 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:54 smithi116 ceph-mon[36248]: pgmap v117: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 194 op/s 2024-03-24T10:43:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:54 smithi023 ceph-mon[32466]: pgmap v117: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 194 op/s 2024-03-24T10:43:55.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 301324 12.36 MB/sec execute 12 sec latency 23.462 ms 2024-03-24T10:43:56.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 303080 12.16 MB/sec execute 13 sec latency 18.813 ms 2024-03-24T10:43:56.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:56 smithi023 ceph-mon[32466]: pgmap v118: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 194 op/s 2024-03-24T10:43:57.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:56 smithi117 ceph-mon[36368]: pgmap v118: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 194 op/s 2024-03-24T10:43:57.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:56 smithi116 ceph-mon[36248]: pgmap v118: 97 pgs: 97 active+clean; 166 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 194 op/s 2024-03-24T10:43:57.116 INFO:tasks.workunit.client.0.smithi023.stdout: 1 305393 12.27 MB/sec execute 14 sec latency 119.413 ms 2024-03-24T10:43:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.310+0000 7f231249d640 1 -- 172.21.15.23:0/2846534770 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f230c100860 msgr2=0x7f230c100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.310+0000 7f231249d640 1 --2- 172.21.15.23:0/2846534770 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f230c100860 0x7f230c100ce0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f2300009900 tx=0x7f230002bc30 comp rx=0 tx=0).stop 2024-03-24T10:43:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.310+0000 7f231249d640 1 -- 172.21.15.23:0/2846534770 shutdown_connections 2024-03-24T10:43:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.310+0000 7f231249d640 1 --2- 172.21.15.23:0/2846534770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f230c101220 0x7f230c10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.310+0000 7f231249d640 1 --2- 172.21.15.23:0/2846534770 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f230c100860 0x7f230c100ce0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.310+0000 7f231249d640 1 --2- 172.21.15.23:0/2846534770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f230c1055b0 0x7f230c105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.310+0000 7f231249d640 1 -- 172.21.15.23:0/2846534770 >> 172.21.15.23:0/2846534770 conn(0x7f230c077460 msgr2=0x7f230c077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.310+0000 7f231249d640 1 -- 172.21.15.23:0/2846534770 shutdown_connections 2024-03-24T10:43:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.310+0000 7f231249d640 1 -- 172.21.15.23:0/2846534770 wait complete. 2024-03-24T10:43:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.311+0000 7f231249d640 1 Processor -- start 2024-03-24T10:43:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.311+0000 7f231249d640 1 -- start start 2024-03-24T10:43:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.311+0000 7f231249d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f230c100860 0x7f230c19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.311+0000 7f231249d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f230c101220 0x7f230c19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.311+0000 7f231249d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f230c1055b0 0x7f230c196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.311+0000 7f231249d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f230c1970a0 con 0x7f230c1055b0 2024-03-24T10:43:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.311+0000 7f231249d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f230c197210 con 0x7f230c101220 2024-03-24T10:43:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.311+0000 7f231249d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f230c1973d0 con 0x7f230c100860 2024-03-24T10:43:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.311+0000 7f2310a3a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f230c1055b0 0x7f230c196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.311+0000 7f2310a3a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f230c1055b0 0x7f230c196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52986/0 (socket says 172.21.15.23:52986) 2024-03-24T10:43:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.311+0000 7f230bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f230c100860 0x7f230c19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.311+0000 7f230b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f230c101220 0x7f230c19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.311+0000 7f2310a3a640 1 -- 172.21.15.23:0/3776132753 learned_addr learned my addr 172.21.15.23:0/3776132753 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.312+0000 7f2310a3a640 1 -- 172.21.15.23:0/3776132753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f230c100860 msgr2=0x7f230c19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.312+0000 7f2310a3a640 1 --2- 172.21.15.23:0/3776132753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f230c100860 0x7f230c19c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.312+0000 7f2310a3a640 1 -- 172.21.15.23:0/3776132753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f230c101220 msgr2=0x7f230c19cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.312+0000 7f2310a3a640 1 --2- 172.21.15.23:0/3776132753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f230c101220 0x7f230c19cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.312+0000 7f2310a3a640 1 -- 172.21.15.23:0/3776132753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2300009560 con 0x7f230c1055b0 2024-03-24T10:43:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.312+0000 7f230bfff640 1 --2- 172.21.15.23:0/3776132753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f230c100860 0x7f230c19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.312+0000 7f2310a3a640 1 --2- 172.21.15.23:0/3776132753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f230c1055b0 0x7f230c196a50 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f230400ea60 tx=0x7f230400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.312+0000 7f23097fa640 1 -- 172.21.15.23:0/3776132753 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230400ce40 con 0x7f230c1055b0 2024-03-24T10:43:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.312+0000 7f231249d640 1 -- 172.21.15.23:0/3776132753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f230c1976b0 con 0x7f230c1055b0 2024-03-24T10:43:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.312+0000 7f23097fa640 1 -- 172.21.15.23:0/3776132753 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2304005020 con 0x7f230c1055b0 2024-03-24T10:43:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.312+0000 7f23097fa640 1 -- 172.21.15.23:0/3776132753 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2304010690 con 0x7f230c1055b0 2024-03-24T10:43:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.312+0000 7f231249d640 1 -- 172.21.15.23:0/3776132753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f230c1a38f0 con 0x7f230c1055b0 2024-03-24T10:43:57.322 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.314+0000 7f23097fa640 1 -- 172.21.15.23:0/3776132753 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f2304005500 con 0x7f230c1055b0 2024-03-24T10:43:57.322 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f231507e740 1 Processor -- start 2024-03-24T10:43:57.322 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f231507e740 1 -- start start 2024-03-24T10:43:57.322 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f23097fa640 1 --2- 172.21.15.23:0/3776132753 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f22f40a22d0 0x7f22f40a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.322 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f231507e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e37f0 0x5617df3e3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.322 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f231507e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3e4110 0x5617df3ec9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.322 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f231507e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3ecee0 0x5617df3ef2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.322 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f231507e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5617df39c1b0 con 0x5617df3e37f0 2024-03-24T10:43:57.322 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f230b7fe640 1 --2- 172.21.15.23:0/3776132753 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f22f40a22d0 0x7f22f40a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.322 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f23097fa640 1 -- 172.21.15.23:0/3776132753 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f2304014070 con 0x7f230c1055b0 2024-03-24T10:43:57.322 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f231507e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617df34ef80 con 0x5617df3ecee0 2024-03-24T10:43:57.323 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f230bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e37f0 0x5617df3e3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.323 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f231507e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617df34c4b0 con 0x5617df3e4110 2024-03-24T10:43:57.323 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f230bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e37f0 0x5617df3e3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52998/0 (socket says 172.21.15.23:52998) 2024-03-24T10:43:57.323 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f230bfff640 1 -- 172.21.15.23:0/2815706396 learned_addr learned my addr 172.21.15.23:0/2815706396 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:57.323 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f2310a3a640 1 --2- 172.21.15.23:0/2815706396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3ecee0 0x5617df3ef2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.323 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f230b7fe640 1 --2- 172.21.15.23:0/2815706396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3e4110 0x5617df3ec9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.323 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.315+0000 7f230bfff640 1 -- 172.21.15.23:0/2815706396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3e4110 msgr2=0x5617df3ec9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:57.323 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.316+0000 7f230bfff640 1 --2- 172.21.15.23:0/2815706396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3e4110 0x5617df3ec9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.323 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.316+0000 7f230bfff640 1 -- 172.21.15.23:0/2815706396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3ecee0 msgr2=0x5617df3ef2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:57.323 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.316+0000 7f230bfff640 1 --2- 172.21.15.23:0/2815706396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3ecee0 0x5617df3ef2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.323 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.316+0000 7f230bfff640 1 -- 172.21.15.23:0/2815706396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617df2dd810 con 0x5617df3e37f0 2024-03-24T10:43:57.323 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.316+0000 7f230bfff640 1 --2- 172.21.15.23:0/2815706396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e37f0 0x5617df3e3bd0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0114f0 tx=0x7f22fc0119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d1c74a7ba184c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:57.323 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.316+0000 7f230b7fe640 1 --2- 172.21.15.23:0/3776132753 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f22f40a22d0 0x7f22f40a4790 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2300006010 tx=0x7f230002cc70 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.316+0000 7f22f27fc640 1 -- 172.21.15.23:0/2815706396 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc004020 con 0x5617df3e37f0 2024-03-24T10:43:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.316+0000 7f22f27fc640 1 -- 172.21.15.23:0/2815706396 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22fc004180 con 0x5617df3e37f0 2024-03-24T10:43:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.316+0000 7f22f27fc640 1 -- 172.21.15.23:0/2815706396 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc00f8b0 con 0x5617df3e37f0 2024-03-24T10:43:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.317+0000 7f231507e740 1 -- 172.21.15.23:0/2815706396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e37f0 msgr2=0x5617df3e3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.317+0000 7f231507e740 1 --2- 172.21.15.23:0/2815706396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e37f0 0x5617df3e3bd0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0114f0 tx=0x7f22fc0119b0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.317+0000 7f231507e740 1 -- 172.21.15.23:0/2815706396 shutdown_connections 2024-03-24T10:43:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.317+0000 7f231507e740 1 --2- 172.21.15.23:0/2815706396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3ecee0 0x5617df3ef2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.317+0000 7f231507e740 1 --2- 172.21.15.23:0/2815706396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3e4110 0x5617df3ec9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.317+0000 7f231507e740 1 --2- 172.21.15.23:0/2815706396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e37f0 0x5617df3e3bd0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.317+0000 7f231507e740 1 -- 172.21.15.23:0/2815706396 >> 172.21.15.23:0/2815706396 conn(0x5617df3de110 msgr2=0x5617df3de4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.317+0000 7f231507e740 1 -- 172.21.15.23:0/2815706396 shutdown_connections 2024-03-24T10:43:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.317+0000 7f231507e740 1 -- 172.21.15.23:0/2815706396 wait complete. 2024-03-24T10:43:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.317+0000 7f231507e740 1 -- start start 2024-03-24T10:43:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.317+0000 7f231507e740 1 Processor -- start 2024-03-24T10:43:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.317+0000 7f231507e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e37f0 0x5617df3e9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f231507e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e4110 0x5617df3ea480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f231507e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 0x5617df3ea9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f231507e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5617df347c90 con 0x5617df3e37f0 2024-03-24T10:43:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f231507e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617df2ecc50 con 0x5617df3e4110 2024-03-24T10:43:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f231507e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617df01b250 con 0x5617df3ecee0 2024-03-24T10:43:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f231507e740 10 client.0 _ll_register_callbacks cb 0x5617df33b290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f230bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e37f0 0x5617df3e9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f230bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e37f0 0x5617df3e9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53004/0 (socket says 172.21.15.23:53004) 2024-03-24T10:43:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f230bfff640 1 -- 172.21.15.23:0/2244088577 learned_addr learned my addr 172.21.15.23:0/2244088577 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f230b7fe640 1 --2- 172.21.15.23:0/2244088577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e4110 0x5617df3ea480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f2310a3a640 1 --2- 172.21.15.23:0/2244088577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 0x5617df3ea9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f230bfff640 1 -- 172.21.15.23:0/2244088577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 msgr2=0x5617df3ea9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:57.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f230bfff640 1 --2- 172.21.15.23:0/2244088577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 0x5617df3ea9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f230bfff640 1 -- 172.21.15.23:0/2244088577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e4110 msgr2=0x5617df3ea480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f230bfff640 1 --2- 172.21.15.23:0/2244088577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e4110 0x5617df3ea480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f230bfff640 1 -- 172.21.15.23:0/2244088577 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22fc009cc0 con 0x5617df3e37f0 2024-03-24T10:43:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f2310a3a640 1 --2- 172.21.15.23:0/2244088577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 0x5617df3ea9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:43:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f230bfff640 1 --2- 172.21.15.23:0/2244088577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e37f0 0x5617df3e9f40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0114c0 tx=0x7f22fc005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b349ec4caae412 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f22eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f22eb7fe640 1 -- 172.21.15.23:0/2244088577 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc0035f0 con 0x5617df3e37f0 2024-03-24T10:43:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f22eb7fe640 1 -- 172.21.15.23:0/2244088577 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f22fc003750 con 0x5617df3e37f0 2024-03-24T10:43:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f231507e740 1 -- 172.21.15.23:0/2244088577 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5617df36c790 con 0x5617df3e37f0 2024-03-24T10:43:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f22eb7fe640 1 -- 172.21.15.23:0/2244088577 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f22fc003910 con 0x5617df3e37f0 2024-03-24T10:43:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f231507e740 10 client.16611 fetch_fsmap learned FSMap version 26 2024-03-24T10:43:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.318+0000 7f231507e740 1 -- 172.21.15.23:0/2244088577 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5617df346310 con 0x5617df3e37f0 2024-03-24T10:43:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.319+0000 7f22eb7fe640 1 -- 172.21.15.23:0/2244088577 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5617df36c790 con 0x5617df3e37f0 2024-03-24T10:43:57.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.319+0000 7f22eb7fe640 1 -- 172.21.15.23:0/2244088577 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f22fc021070 con 0x5617df3e37f0 2024-03-24T10:43:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.319+0000 7f231507e740 10 client.16611 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:43:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.319+0000 7f231507e740 10 client.16611 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.319+0000 7f231507e740 20 client.16611 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.319+0000 7f231507e740 1 --2- 172.21.15.23:0/2244088577 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617df3fce00 0x5617df3ff230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.319+0000 7f231507e740 4 client.16611 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.319+0000 7f231507e740 1 -- 172.21.15.23:0/2244088577 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617df047600 con 0x5617df3fce00 2024-03-24T10:43:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.319+0000 7f230b7fe640 1 --2- 172.21.15.23:0/2244088577 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617df3fce00 0x5617df3ff230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.319+0000 7f22eb7fe640 10 client.16611 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.319+0000 7f230b7fe640 1 --2- 172.21.15.23:0/2244088577 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617df3fce00 0x5617df3ff230 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5487ba7763a86a59 server_cookie=21fb8ad28ac07e81 in_seq=0 out_seq=0 2024-03-24T10:43:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.320+0000 7f22eb7fe640 1 -- 172.21.15.23:0/2244088577 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5617df047600 con 0x5617df3fce00 2024-03-24T10:43:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.320+0000 7f22eb7fe640 10 client.16611 handle_command_reply: tid=0 2024-03-24T10:43:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.320+0000 7f231507e740 1 client.16611 shutdown 2024-03-24T10:43:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.320+0000 7f231507e740 1 -- 172.21.15.23:0/2244088577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e37f0 msgr2=0x5617df3e9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:57.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.320+0000 7f231507e740 1 --2- 172.21.15.23:0/2244088577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e37f0 0x5617df3e9f40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0114c0 tx=0x7f22fc005340 comp rx=0 tx=0).stop 2024-03-24T10:43:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.320+0000 7f231507e740 1 -- 172.21.15.23:0/2244088577 shutdown_connections 2024-03-24T10:43:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.320+0000 7f231507e740 1 --2- 172.21.15.23:0/2244088577 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617df3fce00 0x5617df3ff230 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.320+0000 7f231507e740 1 --2- 172.21.15.23:0/2244088577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 0x5617df3ea9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.320+0000 7f231507e740 1 --2- 172.21.15.23:0/2244088577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e4110 0x5617df3ea480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.320+0000 7f22eb7fe640 0 client.16611 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.320+0000 7f231507e740 1 --2- 172.21.15.23:0/2244088577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e37f0 0x5617df3e9f40 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.320+0000 7f231507e740 1 -- 172.21.15.23:0/2244088577 >> 172.21.15.23:0/2244088577 conn(0x5617df3de110 msgr2=0x5617df3dfe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.321+0000 7f231507e740 1 -- 172.21.15.23:0/2244088577 shutdown_connections 2024-03-24T10:43:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.321+0000 7f231507e740 1 -- 172.21.15.23:0/2244088577 wait complete. 2024-03-24T10:43:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.321+0000 7f231507e740 20 client.16611 trim_cache size 0 max 16384 2024-03-24T10:43:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.335+0000 7f231507e740 1 Processor -- start 2024-03-24T10:43:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.335+0000 7f231507e740 1 -- start start 2024-03-24T10:43:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.335+0000 7f231507e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e37f0 0x5617df3ec670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.335+0000 7f231507e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e4110 0x5617df3f85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.335+0000 7f231507e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 0x5617df3dad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.335+0000 7f231507e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5617df2cd660 con 0x5617df3e4110 2024-03-24T10:43:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.335+0000 7f231507e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617df3fcb30 con 0x5617df3e37f0 2024-03-24T10:43:57.340 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.335+0000 7f231507e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617df3ddc40 con 0x5617df3ecee0 2024-03-24T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.336+0000 7f230b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e4110 0x5617df3f85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.336+0000 7f230b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e4110 0x5617df3f85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53014/0 (socket says 172.21.15.23:53014) 2024-03-24T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.336+0000 7f230b7fe640 1 -- 172.21.15.23:0/2081000522 learned_addr learned my addr 172.21.15.23:0/2081000522 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.336+0000 7f230bfff640 1 --2- 172.21.15.23:0/2081000522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e37f0 0x5617df3ec670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.336+0000 7f2310a3a640 1 --2- 172.21.15.23:0/2081000522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 0x5617df3dad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.336+0000 7f230b7fe640 1 -- 172.21.15.23:0/2081000522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 msgr2=0x5617df3dad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.336+0000 7f230b7fe640 1 --2- 172.21.15.23:0/2081000522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 0x5617df3dad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.336+0000 7f230b7fe640 1 -- 172.21.15.23:0/2081000522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e37f0 msgr2=0x5617df3ec670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.336+0000 7f230b7fe640 1 --2- 172.21.15.23:0/2081000522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e37f0 0x5617df3ec670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.336+0000 7f230b7fe640 1 -- 172.21.15.23:0/2081000522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617df2e2810 con 0x5617df3e4110 2024-03-24T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.336+0000 7f230b7fe640 1 --2- 172.21.15.23:0/2081000522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e4110 0x5617df3f85b0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f230002cd80 tx=0x7f2300046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a305d8bd1db1194 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.336+0000 7f22f1ffb640 1 -- 172.21.15.23:0/2081000522 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300004990 con 0x5617df3e4110 2024-03-24T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.336+0000 7f22f1ffb640 1 -- 172.21.15.23:0/2081000522 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2300004af0 con 0x5617df3e4110 2024-03-24T10:43:57.341 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.336+0000 7f22f1ffb640 1 -- 172.21.15.23:0/2081000522 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300005440 con 0x5617df3e4110 2024-03-24T10:43:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.337+0000 7f231507e740 1 -- 172.21.15.23:0/2081000522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e4110 msgr2=0x5617df3f85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.337+0000 7f231507e740 1 --2- 172.21.15.23:0/2081000522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e4110 0x5617df3f85b0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f230002cd80 tx=0x7f2300046000 comp rx=0 tx=0).stop 2024-03-24T10:43:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.337+0000 7f231507e740 1 -- 172.21.15.23:0/2081000522 shutdown_connections 2024-03-24T10:43:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.337+0000 7f231507e740 1 --2- 172.21.15.23:0/2081000522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 0x5617df3dad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.337+0000 7f231507e740 1 --2- 172.21.15.23:0/2081000522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e4110 0x5617df3f85b0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.337+0000 7f231507e740 1 --2- 172.21.15.23:0/2081000522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e37f0 0x5617df3ec670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.337+0000 7f231507e740 1 -- 172.21.15.23:0/2081000522 >> 172.21.15.23:0/2081000522 conn(0x5617df3de110 msgr2=0x5617df3da610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.337+0000 7f231507e740 1 -- 172.21.15.23:0/2081000522 shutdown_connections 2024-03-24T10:43:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.337+0000 7f231507e740 1 -- 172.21.15.23:0/2081000522 wait complete. 2024-03-24T10:43:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.337+0000 7f231507e740 1 -- start start 2024-03-24T10:43:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.337+0000 7f231507e740 1 Processor -- start 2024-03-24T10:43:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f231507e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e37f0 0x5617df3f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f231507e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e4110 0x5617df3e65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f231507e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 0x5617df3e6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f231507e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5617df3e7040 con 0x5617df3e4110 2024-03-24T10:43:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f231507e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617df3e71b0 con 0x5617df3e37f0 2024-03-24T10:43:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f231507e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617df3e7370 con 0x5617df3ecee0 2024-03-24T10:43:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f231507e740 10 client.0 _ll_register_callbacks cb 0x7f230409ad70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f230b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e4110 0x5617df3e65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f230b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e4110 0x5617df3e65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53024/0 (socket says 172.21.15.23:53024) 2024-03-24T10:43:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f230b7fe640 1 -- 172.21.15.23:0/720196920 learned_addr learned my addr 172.21.15.23:0/720196920 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f230bfff640 1 --2- 172.21.15.23:0/720196920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e37f0 0x5617df3f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f2310a3a640 1 --2- 172.21.15.23:0/720196920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 0x5617df3e6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f230b7fe640 1 -- 172.21.15.23:0/720196920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 msgr2=0x5617df3e6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f230b7fe640 1 --2- 172.21.15.23:0/720196920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 0x5617df3e6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f230b7fe640 1 -- 172.21.15.23:0/720196920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e37f0 msgr2=0x5617df3f4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f230b7fe640 1 --2- 172.21.15.23:0/720196920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e37f0 0x5617df3f4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f230b7fe640 1 -- 172.21.15.23:0/720196920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2300049050 con 0x5617df3e4110 2024-03-24T10:43:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f230b7fe640 1 --2- 172.21.15.23:0/720196920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e4110 0x5617df3e65c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2300036040 tx=0x7f2300004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5086b14f33ac0708 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f22eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f22eb7fe640 1 -- 172.21.15.23:0/720196920 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23000053c0 con 0x5617df3e4110 2024-03-24T10:43:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f22eb7fe640 1 -- 172.21.15.23:0/720196920 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f2300004eb0 con 0x5617df3e4110 2024-03-24T10:43:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.338+0000 7f231507e740 1 -- 172.21.15.23:0/720196920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22fc004180 con 0x5617df3e4110 2024-03-24T10:43:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.339+0000 7f22eb7fe640 1 -- 172.21.15.23:0/720196920 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2300005520 con 0x5617df3e4110 2024-03-24T10:43:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.339+0000 7f231507e740 10 client.16617 fetch_fsmap learned FSMap version 26 2024-03-24T10:43:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.339+0000 7f231507e740 1 -- 172.21.15.23:0/720196920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5617df3085a0 con 0x5617df3e4110 2024-03-24T10:43:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.339+0000 7f22eb7fe640 1 -- 172.21.15.23:0/720196920 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2300068020 con 0x5617df3e4110 2024-03-24T10:43:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.339+0000 7f22eb7fe640 1 -- 172.21.15.23:0/720196920 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f230005a070 con 0x5617df3e4110 2024-03-24T10:43:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.339+0000 7f231507e740 10 client.16617 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:43:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.339+0000 7f231507e740 10 client.16617 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.339+0000 7f231507e740 20 client.16617 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.339+0000 7f231507e740 1 --2- 172.21.15.23:0/720196920 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617df463460 0x5617df465880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.339+0000 7f231507e740 4 client.16617 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:43:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.339+0000 7f231507e740 1 -- 172.21.15.23:0/720196920 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5617df0981b0 con 0x5617df463460 2024-03-24T10:43:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.339+0000 7f230bfff640 1 --2- 172.21.15.23:0/720196920 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617df463460 0x5617df465880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.340+0000 7f22eb7fe640 10 client.16617 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.340+0000 7f230bfff640 1 --2- 172.21.15.23:0/720196920 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617df463460 0x5617df465880 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=447ca70a1583b3bd server_cookie=365cb2e92659ad69 in_seq=0 out_seq=0 2024-03-24T10:43:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.348+0000 7f22eb7fe640 1 -- 172.21.15.23:0/720196920 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5617df0981b0 con 0x5617df463460 2024-03-24T10:43:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.348+0000 7f22eb7fe640 10 client.16617 handle_command_reply: tid=0 2024-03-24T10:43:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.348+0000 7f231507e740 1 client.16617 shutdown 2024-03-24T10:43:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.348+0000 7f231507e740 1 -- 172.21.15.23:0/720196920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e4110 msgr2=0x5617df3e65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.348+0000 7f231507e740 1 --2- 172.21.15.23:0/720196920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e4110 0x5617df3e65c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2300036040 tx=0x7f2300004740 comp rx=0 tx=0).stop 2024-03-24T10:43:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231507e740 1 -- 172.21.15.23:0/720196920 shutdown_connections 2024-03-24T10:43:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231507e740 1 --2- 172.21.15.23:0/720196920 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617df463460 0x5617df465880 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231507e740 1 --2- 172.21.15.23:0/720196920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617df3ecee0 0x5617df3e6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231507e740 1 --2- 172.21.15.23:0/720196920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617df3e4110 0x5617df3e65c0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231507e740 1 --2- 172.21.15.23:0/720196920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617df3e37f0 0x5617df3f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f22eb7fe640 0 client.16617 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231507e740 1 -- 172.21.15.23:0/720196920 >> 172.21.15.23:0/720196920 conn(0x5617df3de110 msgr2=0x5617df3e8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231507e740 1 -- 172.21.15.23:0/720196920 shutdown_connections 2024-03-24T10:43:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231507e740 1 -- 172.21.15.23:0/720196920 wait complete. 2024-03-24T10:43:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231507e740 20 client.16617 trim_cache size 0 max 16384 2024-03-24T10:43:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:43:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:43:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:43:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:43:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231249d640 1 -- 172.21.15.23:0/3776132753 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f22f40a22d0 msgr2=0x7f22f40a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231249d640 1 --2- 172.21.15.23:0/3776132753 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f22f40a22d0 0x7f22f40a4790 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2300006010 tx=0x7f230002cc70 comp rx=0 tx=0).stop 2024-03-24T10:43:57.351 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231249d640 1 -- 172.21.15.23:0/3776132753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f230c1055b0 msgr2=0x7f230c196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231249d640 1 --2- 172.21.15.23:0/3776132753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f230c1055b0 0x7f230c196a50 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f230400ea60 tx=0x7f230400ef20 comp rx=0 tx=0).stop 2024-03-24T10:43:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231249d640 1 -- 172.21.15.23:0/3776132753 shutdown_connections 2024-03-24T10:43:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231249d640 1 --2- 172.21.15.23:0/3776132753 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f22f40a22d0 0x7f22f40a4790 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231249d640 1 --2- 172.21.15.23:0/3776132753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f230c1055b0 0x7f230c196a50 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231249d640 1 --2- 172.21.15.23:0/3776132753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f230c101220 0x7f230c19cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231249d640 1 --2- 172.21.15.23:0/3776132753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f230c100860 0x7f230c19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231249d640 1 -- 172.21.15.23:0/3776132753 >> 172.21.15.23:0/3776132753 conn(0x7f230c077460 msgr2=0x7f230c0fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231249d640 1 -- 172.21.15.23:0/3776132753 shutdown_connections 2024-03-24T10:43:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:43:57.349+0000 7f231249d640 1 -- 172.21.15.23:0/3776132753 wait complete. 2024-03-24T10:43:58.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 307624 12.00 MB/sec execute 15 sec latency 36.525 ms 2024-03-24T10:43:58.880 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:43:58.880 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:43:58.904 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:58 smithi023 ceph-mon[32466]: pgmap v119: 97 pgs: 97 active+clean; 172 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 293 op/s 2024-03-24T10:43:58.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.991+0000 7f75b6c9d640 1 -- 172.21.15.23:0/3592201467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b0100f60 msgr2=0x7f75b010d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:58.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.991+0000 7f75b6c9d640 1 --2- 172.21.15.23:0/3592201467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b0100f60 0x7f75b010d420 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7598009900 tx=0x7f759802bc30 comp rx=0 tx=0).stop 2024-03-24T10:43:58.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.991+0000 7f75b6c9d640 1 -- 172.21.15.23:0/3592201467 shutdown_connections 2024-03-24T10:43:58.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.991+0000 7f75b6c9d640 1 --2- 172.21.15.23:0/3592201467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b0100f60 0x7f75b010d420 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:58.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.991+0000 7f75b6c9d640 1 --2- 172.21.15.23:0/3592201467 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75b01005a0 0x7f75b0100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:58.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.991+0000 7f75b6c9d640 1 --2- 172.21.15.23:0/3592201467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75b0106660 0x7f75b0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:58.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.991+0000 7f75b6c9d640 1 -- 172.21.15.23:0/3592201467 >> 172.21.15.23:0/3592201467 conn(0x7f75b0077710 msgr2=0x7f75b0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:58.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.991+0000 7f75b6c9d640 1 -- 172.21.15.23:0/3592201467 shutdown_connections 2024-03-24T10:43:58.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.991+0000 7f75b6c9d640 1 -- 172.21.15.23:0/3592201467 wait complete. 2024-03-24T10:43:58.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.992+0000 7f75b6c9d640 1 Processor -- start 2024-03-24T10:43:58.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.992+0000 7f75b6c9d640 1 -- start start 2024-03-24T10:43:58.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.992+0000 7f75b6c9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b01005a0 0x7f75b019d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:58.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.992+0000 7f75b6c9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75b0100f60 0x7f75b019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.992+0000 7f75b6c9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75b0106660 0x7f75b0197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.992+0000 7f75b6c9d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f75b0198160 con 0x7f75b01005a0 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.992+0000 7f75b6c9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f75b01982d0 con 0x7f75b0106660 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.992+0000 7f75b6c9d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f75b0198490 con 0x7f75b0100f60 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.992+0000 7f75b4a39640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b01005a0 0x7f75b019d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.992+0000 7f75affff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75b0100f60 0x7f75b019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.992+0000 7f75b4a39640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b01005a0 0x7f75b019d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45152/0 (socket says 172.21.15.23:45152) 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.992+0000 7f75b4a39640 1 -- 172.21.15.23:0/3377495207 learned_addr learned my addr 172.21.15.23:0/3377495207 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.992+0000 7f75b523a640 1 --2- 172.21.15.23:0/3377495207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75b0106660 0x7f75b0197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.993+0000 7f75b4a39640 1 -- 172.21.15.23:0/3377495207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75b0100f60 msgr2=0x7f75b019de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.993+0000 7f75b4a39640 1 --2- 172.21.15.23:0/3377495207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75b0100f60 0x7f75b019de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.993+0000 7f75b4a39640 1 -- 172.21.15.23:0/3377495207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75b0106660 msgr2=0x7f75b0197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.993+0000 7f75b4a39640 1 --2- 172.21.15.23:0/3377495207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75b0106660 0x7f75b0197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.993+0000 7f75b4a39640 1 -- 172.21.15.23:0/3377495207 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7598009560 con 0x7f75b01005a0 2024-03-24T10:43:58.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.993+0000 7f75b4a39640 1 --2- 172.21.15.23:0/3377495207 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b01005a0 0x7f75b019d8c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f75a800d950 tx=0x7f75a800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:58.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.993+0000 7f75adffb640 1 -- 172.21.15.23:0/3377495207 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a8009a30 con 0x7f75b01005a0 2024-03-24T10:43:58.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.993+0000 7f75adffb640 1 -- 172.21.15.23:0/3377495207 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a8009b90 con 0x7f75b01005a0 2024-03-24T10:43:58.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.993+0000 7f75adffb640 1 -- 172.21.15.23:0/3377495207 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a8004a00 con 0x7f75b01005a0 2024-03-24T10:43:58.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.993+0000 7f75b6c9d640 1 -- 172.21.15.23:0/3377495207 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b0198770 con 0x7f75b01005a0 2024-03-24T10:43:58.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.993+0000 7f75b6c9d640 1 -- 172.21.15.23:0/3377495207 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b01a4960 con 0x7f75b01005a0 2024-03-24T10:43:59.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.995+0000 7f75adffb640 1 -- 172.21.15.23:0/3377495207 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f75a8009d00 con 0x7f75b01005a0 2024-03-24T10:43:59.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75b9882740 1 Processor -- start 2024-03-24T10:43:59.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75b9882740 1 -- start start 2024-03-24T10:43:59.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75b9882740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee1041d0 0x55ffee1045b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75b9882740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 0x55ffee10d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75b9882740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee10fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75adffb640 1 --2- 172.21.15.23:0/3377495207 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f759c0a2230 0x7f759c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75b9882740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ffee028f30 con 0x55ffee10d6d0 2024-03-24T10:43:59.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75b9882740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ffee072810 con 0x55ffee1041d0 2024-03-24T10:43:59.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75b9882740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ffee06fd40 con 0x55ffee104af0 2024-03-24T10:43:59.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75b523a640 1 --2- 172.21.15.23:0/3377495207 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f759c0a2230 0x7f759c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75affff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee10fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75b4a39640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 0x55ffee10d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75affff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee10fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45156/0 (socket says 172.21.15.23:45156) 2024-03-24T10:43:59.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75affff640 1 -- 172.21.15.23:0/538010843 learned_addr learned my addr 172.21.15.23:0/538010843 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75b523a640 1 --2- 172.21.15.23:0/538010843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee1041d0 0x55ffee1045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.996+0000 7f75adffb640 1 -- 172.21.15.23:0/3377495207 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f75a80cd9e0 con 0x7f75b01005a0 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.997+0000 7f75affff640 1 -- 172.21.15.23:0/538010843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 msgr2=0x55ffee10d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.997+0000 7f75affff640 1 --2- 172.21.15.23:0/538010843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 0x55ffee10d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.997+0000 7f75affff640 1 -- 172.21.15.23:0/538010843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee1041d0 msgr2=0x55ffee1045b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.997+0000 7f75affff640 1 --2- 172.21.15.23:0/538010843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee1041d0 0x55ffee1045b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.997+0000 7f75affff640 1 -- 172.21.15.23:0/538010843 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffee02cea0 con 0x55ffee10d6d0 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.997+0000 7f75b523a640 1 --2- 172.21.15.23:0/3377495207 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f759c0a2230 0x7f759c0a46f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7598005ab0 tx=0x7f759802c810 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.997+0000 7f75affff640 1 --2- 172.21.15.23:0/538010843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee10fac0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009fd0 tx=0x7f75a00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=699787d4105073d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.997+0000 7f75a6ffd640 1 -- 172.21.15.23:0/538010843 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0004d20 con 0x55ffee10d6d0 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.997+0000 7f75a6ffd640 1 -- 172.21.15.23:0/538010843 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a0004e80 con 0x55ffee10d6d0 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.997+0000 7f75a6ffd640 1 -- 172.21.15.23:0/538010843 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0005220 con 0x55ffee10d6d0 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 -- 172.21.15.23:0/538010843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 msgr2=0x55ffee10fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 --2- 172.21.15.23:0/538010843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee10fac0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009fd0 tx=0x7f75a00118d0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 -- 172.21.15.23:0/538010843 shutdown_connections 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 --2- 172.21.15.23:0/538010843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee10fac0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 --2- 172.21.15.23:0/538010843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 0x55ffee10d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 --2- 172.21.15.23:0/538010843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee1041d0 0x55ffee1045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 -- 172.21.15.23:0/538010843 >> 172.21.15.23:0/538010843 conn(0x55ffee0fe900 msgr2=0x55ffee0fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 -- 172.21.15.23:0/538010843 shutdown_connections 2024-03-24T10:43:59.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 -- 172.21.15.23:0/538010843 wait complete. 2024-03-24T10:43:59.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 -- start start 2024-03-24T10:43:59.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 Processor -- start 2024-03-24T10:43:59.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee1041d0 0x55ffee10a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 0x55ffee10ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee10d6d0 0x55ffee10b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ffee06b520 con 0x55ffee1041d0 2024-03-24T10:43:59.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ffedd3e430 con 0x55ffee10d6d0 2024-03-24T10:43:59.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.998+0000 7f75b9882740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ffedcfae00 con 0x55ffee104af0 2024-03-24T10:43:59.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75b9882740 10 client.0 _ll_register_callbacks cb 0x55ffee009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:59.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75b4a39640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee1041d0 0x55ffee10a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75affff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 0x55ffee10ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75b4a39640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee1041d0 0x55ffee10a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45158/0 (socket says 172.21.15.23:45158) 2024-03-24T10:43:59.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75b4a39640 1 -- 172.21.15.23:0/1270042940 learned_addr learned my addr 172.21.15.23:0/1270042940 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75b523a640 1 --2- 172.21.15.23:0/1270042940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee10d6d0 0x55ffee10b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75b4a39640 1 -- 172.21.15.23:0/1270042940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 msgr2=0x55ffee10ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75b4a39640 1 --2- 172.21.15.23:0/1270042940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 0x55ffee10ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75b4a39640 1 -- 172.21.15.23:0/1270042940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee10d6d0 msgr2=0x55ffee10b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75b4a39640 1 --2- 172.21.15.23:0/1270042940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee10d6d0 0x55ffee10b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75b4a39640 1 -- 172.21.15.23:0/1270042940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a0009c00 con 0x55ffee1041d0 2024-03-24T10:43:59.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75b4a39640 1 --2- 172.21.15.23:0/1270042940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee1041d0 0x55ffee10a6d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f75a8096da0 tx=0x7f75a806a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582bd9e4ed2e8c3c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f758ffff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:59.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f758ffff640 1 -- 172.21.15.23:0/1270042940 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a809a3d0 con 0x55ffee1041d0 2024-03-24T10:43:59.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f758ffff640 1 -- 172.21.15.23:0/1270042940 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f75a809e520 con 0x55ffee1041d0 2024-03-24T10:43:59.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75b9882740 1 -- 172.21.15.23:0/1270042940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ffedda7720 con 0x55ffee1041d0 2024-03-24T10:43:59.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f758ffff640 1 -- 172.21.15.23:0/1270042940 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f75a80a2b70 con 0x55ffee1041d0 2024-03-24T10:43:59.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75b9882740 10 client.16629 fetch_fsmap learned FSMap version 26 2024-03-24T10:43:59.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:58.999+0000 7f75b9882740 1 -- 172.21.15.23:0/1270042940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ffee069ba0 con 0x55ffee1041d0 2024-03-24T10:43:59.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.000+0000 7f758ffff640 1 -- 172.21.15.23:0/1270042940 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ffedda7720 con 0x55ffee1041d0 2024-03-24T10:43:59.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.000+0000 7f758ffff640 1 -- 172.21.15.23:0/1270042940 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f75a80690d0 con 0x55ffee1041d0 2024-03-24T10:43:59.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.000+0000 7f75b9882740 10 client.16629 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:43:59.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.000+0000 7f75b9882740 10 client.16629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:59.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.000+0000 7f75b9882740 20 client.16629 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:59.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.000+0000 7f75b9882740 1 --2- 172.21.15.23:0/1270042940 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ffee11da10 0x55ffee11fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.000+0000 7f75b9882740 4 client.16629 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:59.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.000+0000 7f75b9882740 1 -- 172.21.15.23:0/1270042940 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffeddf6890 con 0x55ffee11da10 2024-03-24T10:43:59.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.000+0000 7f75affff640 1 --2- 172.21.15.23:0/1270042940 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ffee11da10 0x55ffee11fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.000+0000 7f758ffff640 10 client.16629 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.000+0000 7f75affff640 1 --2- 172.21.15.23:0/1270042940 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ffee11da10 0x55ffee11fe00 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71ae91ebb1236d6d server_cookie=ab2268751368cd8d in_seq=0 out_seq=0 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f758ffff640 1 -- 172.21.15.23:0/1270042940 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ffeddf6890 con 0x55ffee11da10 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f758ffff640 10 client.16629 handle_command_reply: tid=0 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f75b9882740 1 client.16629 shutdown 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f75b9882740 1 -- 172.21.15.23:0/1270042940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee1041d0 msgr2=0x55ffee10a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f75b9882740 1 --2- 172.21.15.23:0/1270042940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee1041d0 0x55ffee10a6d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f75a8096da0 tx=0x7f75a806a230 comp rx=0 tx=0).stop 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f75b9882740 1 -- 172.21.15.23:0/1270042940 shutdown_connections 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f75b9882740 1 --2- 172.21.15.23:0/1270042940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee10d6d0 0x55ffee10b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f75b9882740 1 --2- 172.21.15.23:0/1270042940 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ffee11da10 0x55ffee11fe00 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f75b9882740 1 --2- 172.21.15.23:0/1270042940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 0x55ffee10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f75b9882740 1 --2- 172.21.15.23:0/1270042940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee1041d0 0x55ffee10a6d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f758ffff640 0 client.16629 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f75b9882740 1 -- 172.21.15.23:0/1270042940 >> 172.21.15.23:0/1270042940 conn(0x55ffee0fe900 msgr2=0x55ffee0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f75b9882740 1 -- 172.21.15.23:0/1270042940 shutdown_connections 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f75b9882740 1 -- 172.21.15.23:0/1270042940 wait complete. 2024-03-24T10:43:59.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.001+0000 7f75b9882740 20 client.16629 trim_cache size 0 max 16384 2024-03-24T10:43:59.009 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:58 smithi117 ceph-mon[36368]: pgmap v119: 97 pgs: 97 active+clean; 172 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 293 op/s 2024-03-24T10:43:59.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:58 smithi116 ceph-mon[36248]: pgmap v119: 97 pgs: 97 active+clean; 172 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 293 op/s 2024-03-24T10:43:59.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b9882740 1 Processor -- start 2024-03-24T10:43:59.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b9882740 1 -- start start 2024-03-24T10:43:59.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b9882740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee1041d0 0x55ffee118f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b9882740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 0x55ffee10af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b9882740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee0fd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b9882740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ffedff0430 con 0x55ffee10d6d0 2024-03-24T10:43:59.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b9882740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ffee11d820 con 0x55ffee1041d0 2024-03-24T10:43:59.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b9882740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ffee103e60 con 0x55ffee104af0 2024-03-24T10:43:59.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b523a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee0fd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b523a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee0fd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45160/0 (socket says 172.21.15.23:45160) 2024-03-24T10:43:59.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b523a640 1 -- 172.21.15.23:0/696158444 learned_addr learned my addr 172.21.15.23:0/696158444 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75affff640 1 --2- 172.21.15.23:0/696158444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 0x55ffee10af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b4a39640 1 --2- 172.21.15.23:0/696158444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee1041d0 0x55ffee118f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b523a640 1 -- 172.21.15.23:0/696158444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 msgr2=0x55ffee10af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b523a640 1 --2- 172.21.15.23:0/696158444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 0x55ffee10af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b523a640 1 -- 172.21.15.23:0/696158444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee1041d0 msgr2=0x55ffee118f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b523a640 1 --2- 172.21.15.23:0/696158444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee1041d0 0x55ffee118f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b523a640 1 -- 172.21.15.23:0/696158444 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffee005b20 con 0x55ffee10d6d0 2024-03-24T10:43:59.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.019+0000 7f75b523a640 1 --2- 172.21.15.23:0/696158444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee0fd9e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7598009900 tx=0x7f759804b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9938d2e55e5acf8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.020+0000 7f75a67fc640 1 -- 172.21.15.23:0/696158444 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7598048070 con 0x55ffee10d6d0 2024-03-24T10:43:59.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.020+0000 7f75a67fc640 1 -- 172.21.15.23:0/696158444 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7598004aa0 con 0x55ffee10d6d0 2024-03-24T10:43:59.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.020+0000 7f75a67fc640 1 -- 172.21.15.23:0/696158444 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75980039c0 con 0x55ffee10d6d0 2024-03-24T10:43:59.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.020+0000 7f75b9882740 1 -- 172.21.15.23:0/696158444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 msgr2=0x55ffee0fd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.020+0000 7f75b9882740 1 --2- 172.21.15.23:0/696158444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee0fd9e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7598009900 tx=0x7f759804b000 comp rx=0 tx=0).stop 2024-03-24T10:43:59.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.020+0000 7f75b9882740 1 -- 172.21.15.23:0/696158444 shutdown_connections 2024-03-24T10:43:59.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.020+0000 7f75b9882740 1 --2- 172.21.15.23:0/696158444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee0fd9e0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.020+0000 7f75b9882740 1 --2- 172.21.15.23:0/696158444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee104af0 0x55ffee10af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.020+0000 7f75b9882740 1 --2- 172.21.15.23:0/696158444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee1041d0 0x55ffee118f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.020+0000 7f75b9882740 1 -- 172.21.15.23:0/696158444 >> 172.21.15.23:0/696158444 conn(0x55ffee0fe900 msgr2=0x55ffee0fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.020+0000 7f75b9882740 1 -- 172.21.15.23:0/696158444 shutdown_connections 2024-03-24T10:43:59.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.020+0000 7f75b9882740 1 -- 172.21.15.23:0/696158444 wait complete. 2024-03-24T10:43:59.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.020+0000 7f75b9882740 1 -- start start 2024-03-24T10:43:59.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b9882740 1 Processor -- start 2024-03-24T10:43:59.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b9882740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee1041d0 0x55ffee1157f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b9882740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee104af0 0x55ffee110bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b9882740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee1110f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b9882740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ffee1117c0 con 0x55ffee10d6d0 2024-03-24T10:43:59.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b9882740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ffee111930 con 0x55ffee104af0 2024-03-24T10:43:59.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b9882740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ffee182d10 con 0x55ffee1041d0 2024-03-24T10:43:59.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b9882740 10 client.0 _ll_register_callbacks cb 0x7f75a8005240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:59.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b523a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee1110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b523a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee1110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45168/0 (socket says 172.21.15.23:45168) 2024-03-24T10:43:59.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b523a640 1 -- 172.21.15.23:0/3338357408 learned_addr learned my addr 172.21.15.23:0/3338357408 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b4a39640 1 --2- 172.21.15.23:0/3338357408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee1041d0 0x55ffee1157f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75affff640 1 --2- 172.21.15.23:0/3338357408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee104af0 0x55ffee110bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b523a640 1 -- 172.21.15.23:0/3338357408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee1041d0 msgr2=0x55ffee1157f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b523a640 1 --2- 172.21.15.23:0/3338357408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee1041d0 0x55ffee1157f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b523a640 1 -- 172.21.15.23:0/3338357408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee104af0 msgr2=0x55ffee110bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b523a640 1 --2- 172.21.15.23:0/3338357408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee104af0 0x55ffee110bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.021+0000 7f75b523a640 1 -- 172.21.15.23:0/3338357408 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b010dd20 con 0x55ffee10d6d0 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f75b523a640 1 --2- 172.21.15.23:0/3338357408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee1110f0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f759802cad0 tx=0x7f7598009300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2701d2b8355bbc7f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f758ffff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f758ffff640 1 -- 172.21.15.23:0/3338357408 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7598048070 con 0x55ffee10d6d0 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f758ffff640 1 -- 172.21.15.23:0/3338357408 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f7598003ab0 con 0x55ffee10d6d0 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f75b9882740 1 -- 172.21.15.23:0/3338357408 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ffedd6a7e0 con 0x55ffee10d6d0 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f758ffff640 1 -- 172.21.15.23:0/3338357408 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7598003c70 con 0x55ffee10d6d0 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f75b9882740 10 client.16635 fetch_fsmap learned FSMap version 26 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f75b9882740 1 -- 172.21.15.23:0/3338357408 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ffee02b700 con 0x55ffee10d6d0 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f758ffff640 1 -- 172.21.15.23:0/3338357408 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ffedd6a7e0 con 0x55ffee10d6d0 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f758ffff640 1 -- 172.21.15.23:0/3338357408 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f7598034510 con 0x55ffee10d6d0 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f75b9882740 10 client.16635 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f75b9882740 10 client.16635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f75b9882740 20 client.16635 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f75b9882740 1 --2- 172.21.15.23:0/3338357408 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ffee184030 0x55ffee186420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f75b9882740 4 client.16635 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:43:59.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.022+0000 7f75b9882740 1 -- 172.21.15.23:0/3338357408 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ffedd16700 con 0x55ffee184030 2024-03-24T10:43:59.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f75b4a39640 1 --2- 172.21.15.23:0/3338357408 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ffee184030 0x55ffee186420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f75b4a39640 1 --2- 172.21.15.23:0/3338357408 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ffee184030 0x55ffee186420 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca56aaa4d6621c3 server_cookie=be2a3b309481ff6b in_seq=0 out_seq=0 2024-03-24T10:43:59.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f758ffff640 10 client.16635 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:59.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f758ffff640 1 -- 172.21.15.23:0/3338357408 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ffedd16700 con 0x55ffee184030 2024-03-24T10:43:59.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f758ffff640 10 client.16635 handle_command_reply: tid=0 2024-03-24T10:43:59.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f75b9882740 1 client.16635 shutdown 2024-03-24T10:43:59.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f75b9882740 1 -- 172.21.15.23:0/3338357408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 msgr2=0x55ffee1110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f75b9882740 1 --2- 172.21.15.23:0/3338357408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee1110f0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f759802cad0 tx=0x7f7598009300 comp rx=0 tx=0).stop 2024-03-24T10:43:59.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f75b9882740 1 -- 172.21.15.23:0/3338357408 shutdown_connections 2024-03-24T10:43:59.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f75b9882740 1 --2- 172.21.15.23:0/3338357408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ffee10d6d0 0x55ffee1110f0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f75b9882740 1 --2- 172.21.15.23:0/3338357408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ffee104af0 0x55ffee110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f75b9882740 1 --2- 172.21.15.23:0/3338357408 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ffee184030 0x55ffee186420 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f75b9882740 1 --2- 172.21.15.23:0/3338357408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffee1041d0 0x55ffee1157f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f758ffff640 0 client.16635 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:59.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f75b9882740 1 -- 172.21.15.23:0/3338357408 >> 172.21.15.23:0/3338357408 conn(0x55ffee0fe900 msgr2=0x55ffee0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f75b9882740 1 -- 172.21.15.23:0/3338357408 shutdown_connections 2024-03-24T10:43:59.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.023+0000 7f75b9882740 1 -- 172.21.15.23:0/3338357408 wait complete. 2024-03-24T10:43:59.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.024+0000 7f75b9882740 20 client.16635 trim_cache size 0 max 16384 2024-03-24T10:43:59.030 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:43:59.030 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:43:59.030 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:43:59.030 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:43:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.024+0000 7f75b6c9d640 1 -- 172.21.15.23:0/3377495207 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f759c0a2230 msgr2=0x7f759c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.024+0000 7f75b6c9d640 1 --2- 172.21.15.23:0/3377495207 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f759c0a2230 0x7f759c0a46f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7598005ab0 tx=0x7f759802c810 comp rx=0 tx=0).stop 2024-03-24T10:43:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.024+0000 7f75b6c9d640 1 -- 172.21.15.23:0/3377495207 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b01005a0 msgr2=0x7f75b019d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.024+0000 7f75b6c9d640 1 --2- 172.21.15.23:0/3377495207 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b01005a0 0x7f75b019d8c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f75a800d950 tx=0x7f75a800de10 comp rx=0 tx=0).stop 2024-03-24T10:43:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.024+0000 7f75b6c9d640 1 -- 172.21.15.23:0/3377495207 shutdown_connections 2024-03-24T10:43:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.024+0000 7f75b6c9d640 1 --2- 172.21.15.23:0/3377495207 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f759c0a2230 0x7f759c0a46f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.024+0000 7f75b6c9d640 1 --2- 172.21.15.23:0/3377495207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75b0106660 0x7f75b0197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.024+0000 7f75b6c9d640 1 --2- 172.21.15.23:0/3377495207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75b0100f60 0x7f75b019de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.024+0000 7f75b6c9d640 1 --2- 172.21.15.23:0/3377495207 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b01005a0 0x7f75b019d8c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.024+0000 7f75b6c9d640 1 -- 172.21.15.23:0/3377495207 >> 172.21.15.23:0/3377495207 conn(0x7f75b0077710 msgr2=0x7f75b00ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.024+0000 7f75b6c9d640 1 -- 172.21.15.23:0/3377495207 shutdown_connections 2024-03-24T10:43:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.024+0000 7f75b6c9d640 1 -- 172.21.15.23:0/3377495207 wait complete. 2024-03-24T10:43:59.035 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:43:59.035 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:43:59.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 309672 12.07 MB/sec execute 16 sec latency 14.516 ms 2024-03-24T10:43:59.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.179+0000 7f6e60eaa640 1 -- 172.21.15.23:0/1971553045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c1076d0 msgr2=0x7f6e5c107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.179+0000 7f6e60eaa640 1 --2- 172.21.15.23:0/1971553045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c1076d0 0x7f6e5c107ab0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009930 tx=0x7f6e4c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:43:59.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.179+0000 7f6e60eaa640 1 -- 172.21.15.23:0/1971553045 shutdown_connections 2024-03-24T10:43:59.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.179+0000 7f6e60eaa640 1 --2- 172.21.15.23:0/1971553045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e5c102080 0x7f6e5c10e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.179+0000 7f6e60eaa640 1 --2- 172.21.15.23:0/1971553045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e5c1016c0 0x7f6e5c101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.179+0000 7f6e60eaa640 1 --2- 172.21.15.23:0/1971553045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c1076d0 0x7f6e5c107ab0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.179+0000 7f6e60eaa640 1 -- 172.21.15.23:0/1971553045 >> 172.21.15.23:0/1971553045 conn(0x7f6e5c0fd1b0 msgr2=0x7f6e5c0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.179+0000 7f6e60eaa640 1 -- 172.21.15.23:0/1971553045 shutdown_connections 2024-03-24T10:43:59.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.179+0000 7f6e60eaa640 1 -- 172.21.15.23:0/1971553045 wait complete. 2024-03-24T10:43:59.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.180+0000 7f6e60eaa640 1 Processor -- start 2024-03-24T10:43:59.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.180+0000 7f6e60eaa640 1 -- start start 2024-03-24T10:43:59.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.180+0000 7f6e60eaa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e5c1016c0 0x7f6e5c197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.180+0000 7f6e60eaa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e5c102080 0x7f6e5c198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.180+0000 7f6e60eaa640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c1076d0 0x7f6e5c19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.180+0000 7f6e60eaa640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c19cc80 con 0x7f6e5c1076d0 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.180+0000 7f6e60eaa640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c19cdf0 con 0x7f6e5c102080 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.180+0000 7f6e60eaa640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c19cfb0 con 0x7f6e5c1016c0 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.180+0000 7f6e5ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c1076d0 0x7f6e5c19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.180+0000 7f6e5ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c1076d0 0x7f6e5c19c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45192/0 (socket says 172.21.15.23:45192) 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.180+0000 7f6e5ad9d640 1 -- 172.21.15.23:0/458164212 learned_addr learned my addr 172.21.15.23:0/458164212 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.180+0000 7f6e59d9b640 1 --2- 172.21.15.23:0/458164212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e5c102080 0x7f6e5c198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.180+0000 7f6e5a59c640 1 --2- 172.21.15.23:0/458164212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e5c1016c0 0x7f6e5c197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.181+0000 7f6e5ad9d640 1 -- 172.21.15.23:0/458164212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e5c1016c0 msgr2=0x7f6e5c197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.181+0000 7f6e5ad9d640 1 --2- 172.21.15.23:0/458164212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e5c1016c0 0x7f6e5c197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.181+0000 7f6e5ad9d640 1 -- 172.21.15.23:0/458164212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e5c102080 msgr2=0x7f6e5c198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.181+0000 7f6e5ad9d640 1 --2- 172.21.15.23:0/458164212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e5c102080 0x7f6e5c198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.181+0000 7f6e5ad9d640 1 -- 172.21.15.23:0/458164212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4c009560 con 0x7f6e5c1076d0 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.181+0000 7f6e5ad9d640 1 --2- 172.21.15.23:0/458164212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c1076d0 0x7f6e5c19c680 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400eb10 tx=0x7f6e5400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.181+0000 7f6e4b7fe640 1 -- 172.21.15.23:0/458164212 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e54004b20 con 0x7f6e5c1076d0 2024-03-24T10:43:59.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.181+0000 7f6e4b7fe640 1 -- 172.21.15.23:0/458164212 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e54004c80 con 0x7f6e5c1076d0 2024-03-24T10:43:59.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.181+0000 7f6e60eaa640 1 -- 172.21.15.23:0/458164212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e5c19d290 con 0x7f6e5c1076d0 2024-03-24T10:43:59.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.181+0000 7f6e4b7fe640 1 -- 172.21.15.23:0/458164212 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e54010650 con 0x7f6e5c1076d0 2024-03-24T10:43:59.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.181+0000 7f6e60eaa640 1 -- 172.21.15.23:0/458164212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e5c1a4b10 con 0x7f6e5c1076d0 2024-03-24T10:43:59.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.182+0000 7f6e60eaa640 1 -- 172.21.15.23:0/458164212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e5c068950 con 0x7f6e5c1076d0 2024-03-24T10:43:59.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.183+0000 7f6e4b7fe640 1 -- 172.21.15.23:0/458164212 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f6e54005020 con 0x7f6e5c1076d0 2024-03-24T10:43:59.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.184+0000 7f6e4b7fe640 1 --2- 172.21.15.23:0/458164212 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e440a2230 0x7f6e440a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.184+0000 7f6e5a59c640 1 --2- 172.21.15.23:0/458164212 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e440a2230 0x7f6e440a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.184+0000 7f6e4b7fe640 1 -- 172.21.15.23:0/458164212 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f6e54014070 con 0x7f6e5c1076d0 2024-03-24T10:43:59.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.184+0000 7f6e5a59c640 1 --2- 172.21.15.23:0/458164212 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e440a2230 0x7f6e440a46f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c005ab0 tx=0x7f6e4c005fd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.188+0000 7f6e4b7fe640 1 -- 172.21.15.23:0/458164212 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6e54097260 con 0x7f6e5c1076d0 2024-03-24T10:43:59.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.415+0000 7f6e60eaa640 1 -- 172.21.15.23:0/458164212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e5c199110 con 0x7f6e5c1076d0 2024-03-24T10:43:59.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.415+0000 7f6e4b7fe640 1 -- 172.21.15.23:0/458164212 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11867 (secure 0 0 0) 0x7f6e5409a920 con 0x7f6e5c1076d0 2024-03-24T10:43:59.416 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:43:59.416 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:43:49.923797+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24539,24503,24505]},"id":1}]} 2024-03-24T10:43:59.416 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 26 2024-03-24T10:43:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.417+0000 7f6e60eaa640 1 -- 172.21.15.23:0/458164212 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e440a2230 msgr2=0x7f6e440a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.417+0000 7f6e60eaa640 1 --2- 172.21.15.23:0/458164212 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e440a2230 0x7f6e440a46f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c005ab0 tx=0x7f6e4c005fd0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.417+0000 7f6e60eaa640 1 -- 172.21.15.23:0/458164212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c1076d0 msgr2=0x7f6e5c19c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.417+0000 7f6e60eaa640 1 --2- 172.21.15.23:0/458164212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c1076d0 0x7f6e5c19c680 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400eb10 tx=0x7f6e5400c2d0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.417+0000 7f6e60eaa640 1 -- 172.21.15.23:0/458164212 shutdown_connections 2024-03-24T10:43:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.417+0000 7f6e60eaa640 1 --2- 172.21.15.23:0/458164212 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e440a2230 0x7f6e440a46f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.417+0000 7f6e60eaa640 1 --2- 172.21.15.23:0/458164212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c1076d0 0x7f6e5c19c680 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.417+0000 7f6e60eaa640 1 --2- 172.21.15.23:0/458164212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e5c102080 0x7f6e5c198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.417+0000 7f6e60eaa640 1 --2- 172.21.15.23:0/458164212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e5c1016c0 0x7f6e5c197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.417+0000 7f6e60eaa640 1 -- 172.21.15.23:0/458164212 >> 172.21.15.23:0/458164212 conn(0x7f6e5c0fd1b0 msgr2=0x7f6e5c0fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.418+0000 7f6e60eaa640 1 -- 172.21.15.23:0/458164212 shutdown_connections 2024-03-24T10:43:59.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.418+0000 7f6e60eaa640 1 -- 172.21.15.23:0/458164212 wait complete. 2024-03-24T10:43:59.428 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:43:59.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.588+0000 7fef0feb1640 1 -- 172.21.15.23:0/1421774070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef081032a0 msgr2=0x7fef0810b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.588+0000 7fef0feb1640 1 --2- 172.21.15.23:0/1421774070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef081032a0 0x7fef0810b780 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7feef8009900 tx=0x7feef802bcd0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.588+0000 7fef0feb1640 1 -- 172.21.15.23:0/1421774070 shutdown_connections 2024-03-24T10:43:59.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.588+0000 7fef0feb1640 1 --2- 172.21.15.23:0/1421774070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef0810bd50 0x7fef0810e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.588+0000 7fef0feb1640 1 --2- 172.21.15.23:0/1421774070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef081032a0 0x7fef0810b780 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.588+0000 7fef0feb1640 1 --2- 172.21.15.23:0/1421774070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef08102980 0x7fef08102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.588+0000 7fef0feb1640 1 -- 172.21.15.23:0/1421774070 >> 172.21.15.23:0/1421774070 conn(0x7fef08077770 msgr2=0x7fef08077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.588+0000 7fef0feb1640 1 -- 172.21.15.23:0/1421774070 shutdown_connections 2024-03-24T10:43:59.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.588+0000 7fef0feb1640 1 -- 172.21.15.23:0/1421774070 wait complete. 2024-03-24T10:43:59.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0feb1640 1 Processor -- start 2024-03-24T10:43:59.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0feb1640 1 -- start start 2024-03-24T10:43:59.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0feb1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef08102980 0x7fef08199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0feb1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef081032a0 0x7fef0819a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0feb1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef0810bd50 0x7fef0819e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0feb1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fef0819ee30 con 0x7fef081032a0 2024-03-24T10:43:59.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0feb1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fef0819efa0 con 0x7fef08102980 2024-03-24T10:43:59.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0feb1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fef0819f160 con 0x7fef0810bd50 2024-03-24T10:43:59.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0d44c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef081032a0 0x7fef0819a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0d44c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef081032a0 0x7fef0819a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45216/0 (socket says 172.21.15.23:45216) 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0d44c640 1 -- 172.21.15.23:0/1540307972 learned_addr learned my addr 172.21.15.23:0/1540307972 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0dc4d640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef08102980 0x7fef08199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0e44e640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef0810bd50 0x7fef0819e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0dc4d640 1 -- 172.21.15.23:0/1540307972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef0810bd50 msgr2=0x7fef0819e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0dc4d640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef0810bd50 0x7fef0819e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0dc4d640 1 -- 172.21.15.23:0/1540307972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef081032a0 msgr2=0x7fef0819a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0dc4d640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef081032a0 0x7fef0819a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0dc4d640 1 -- 172.21.15.23:0/1540307972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef8009560 con 0x7fef08102980 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.589+0000 7fef0d44c640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef081032a0 0x7fef0819a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.590+0000 7fef0dc4d640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef08102980 0x7fef08199fc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fef0400e9f0 tx=0x7fef0400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.590+0000 7feefeffd640 1 -- 172.21.15.23:0/1540307972 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0400ccc0 con 0x7fef08102980 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.590+0000 7feefeffd640 1 -- 172.21.15.23:0/1540307972 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef0400ce20 con 0x7fef08102980 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.590+0000 7fef0feb1640 1 -- 172.21.15.23:0/1540307972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef0819f440 con 0x7fef08102980 2024-03-24T10:43:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.590+0000 7feefeffd640 1 -- 172.21.15.23:0/1540307972 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef04018920 con 0x7fef08102980 2024-03-24T10:43:59.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.590+0000 7fef0feb1640 1 -- 172.21.15.23:0/1540307972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef081a6cc0 con 0x7fef08102980 2024-03-24T10:43:59.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.592+0000 7feefeffd640 1 -- 172.21.15.23:0/1540307972 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fef04018a80 con 0x7fef08102980 2024-03-24T10:43:59.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.592+0000 7fef12b68740 1 Processor -- start 2024-03-24T10:43:59.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.592+0000 7fef12b68740 1 -- start start 2024-03-24T10:43:59.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef12b68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12d73f0 0x55cde12d77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef12b68740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12e03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef12b68740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12e08f0 0x55cde12e2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef12b68740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55cde1245840 con 0x55cde12d7d10 2024-03-24T10:43:59.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef12b68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cde1242d70 con 0x55cde12d73f0 2024-03-24T10:43:59.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef12b68740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cde123e550 con 0x55cde12e08f0 2024-03-24T10:43:59.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7feefeffd640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feef40a22b0 0x7feef40a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef0dc4d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12e03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef0e44e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12d73f0 0x55cde12d77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef0d44c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12e08f0 0x55cde12e2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef0e44e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12d73f0 0x55cde12d77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:48794/0 (socket says 172.21.15.23:48794) 2024-03-24T10:43:59.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef0d44c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12e08f0 0x55cde12e2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:34602/0 (socket says 172.21.15.23:34602) 2024-03-24T10:43:59.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef0e44e640 1 -- 172.21.15.23:0/752858338 learned_addr learned my addr 172.21.15.23:0/752858338 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef0e44e640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feef40a22b0 0x7feef40a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7feefeffd640 1 -- 172.21.15.23:0/1540307972 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fef04014070 con 0x7fef08102980 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef0d44c640 1 -- 172.21.15.23:0/752858338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12d73f0 msgr2=0x55cde12d77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef0d44c640 1 --2- 172.21.15.23:0/752858338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12d73f0 0x55cde12d77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef0d44c640 1 -- 172.21.15.23:0/752858338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 msgr2=0x55cde12e03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef0d44c640 1 --2- 172.21.15.23:0/752858338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12e03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef0d44c640 1 -- 172.21.15.23:0/752858338 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cde11d3de0 con 0x55cde12e08f0 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef0e44e640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feef40a22b0 0x7feef40a4770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fef00005c40 tx=0x7fef00009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.593+0000 7fef0d44c640 1 --2- 172.21.15.23:0/752858338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12e08f0 0x55cde12e2ce0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7feef8005ec0 tx=0x7feef802c780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc34ca29aab96082 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.594+0000 7feef3fff640 1 -- 172.21.15.23:0/752858338 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef802c880 con 0x55cde12e08f0 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.594+0000 7feef3fff640 1 -- 172.21.15.23:0/752858338 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feef802c9e0 con 0x55cde12e08f0 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.594+0000 7feef3fff640 1 -- 172.21.15.23:0/752858338 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef80053f0 con 0x55cde12e08f0 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.594+0000 7fef12b68740 1 -- 172.21.15.23:0/752858338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12e08f0 msgr2=0x55cde12e2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.594+0000 7fef12b68740 1 --2- 172.21.15.23:0/752858338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12e08f0 0x55cde12e2ce0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7feef8005ec0 tx=0x7feef802c780 comp rx=0 tx=0).stop 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.594+0000 7fef12b68740 1 -- 172.21.15.23:0/752858338 shutdown_connections 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.594+0000 7fef12b68740 1 --2- 172.21.15.23:0/752858338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12e08f0 0x55cde12e2ce0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.594+0000 7fef12b68740 1 --2- 172.21.15.23:0/752858338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12e03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.594+0000 7fef12b68740 1 --2- 172.21.15.23:0/752858338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12d73f0 0x55cde12d77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.594+0000 7fef12b68740 1 -- 172.21.15.23:0/752858338 >> 172.21.15.23:0/752858338 conn(0x55cde12d1b20 msgr2=0x55cde12d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.594+0000 7fef12b68740 1 -- 172.21.15.23:0/752858338 shutdown_connections 2024-03-24T10:43:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef12b68740 1 -- 172.21.15.23:0/752858338 wait complete. 2024-03-24T10:43:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef12b68740 1 -- start start 2024-03-24T10:43:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef12b68740 1 Processor -- start 2024-03-24T10:43:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef12b68740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12d73f0 0x55cde12dd910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef12b68740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12dde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef12b68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12e08f0 0x55cde12de390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef12b68740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55cde0f11430 con 0x55cde12d7d10 2024-03-24T10:43:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef12b68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cde0ecde00 con 0x55cde12e08f0 2024-03-24T10:43:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef12b68740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cde11c3590 con 0x55cde12d73f0 2024-03-24T10:43:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef12b68740 10 client.0 _ll_register_callbacks cb 0x55cde1232fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef0d44c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12dde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef0d44c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12dde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45224/0 (socket says 172.21.15.23:45224) 2024-03-24T10:43:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef0d44c640 1 -- 172.21.15.23:0/3308191235 learned_addr learned my addr 172.21.15.23:0/3308191235 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef0dc4d640 1 --2- 172.21.15.23:0/3308191235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12d73f0 0x55cde12dd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.595+0000 7fef0e44e640 1 --2- 172.21.15.23:0/3308191235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12e08f0 0x55cde12de390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.596+0000 7fef0d44c640 1 -- 172.21.15.23:0/3308191235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12d73f0 msgr2=0x55cde12dd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.596+0000 7fef0d44c640 1 --2- 172.21.15.23:0/3308191235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12d73f0 0x55cde12dd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.596+0000 7fef0d44c640 1 -- 172.21.15.23:0/3308191235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12e08f0 msgr2=0x55cde12de390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.596+0000 7fef0d44c640 1 --2- 172.21.15.23:0/3308191235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12e08f0 0x55cde12de390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.596+0000 7fef0d44c640 1 -- 172.21.15.23:0/3308191235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef0810ee40 con 0x55cde12d7d10 2024-03-24T10:43:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.596+0000 7fef0d44c640 1 --2- 172.21.15.23:0/3308191235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12dde50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7feef8036040 tx=0x7feef80092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d304c1166e31bcf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.596+0000 7feef17fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.596+0000 7feef17fa640 1 -- 172.21.15.23:0/3308191235 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8005680 con 0x55cde12d7d10 2024-03-24T10:43:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.596+0000 7feef17fa640 1 -- 172.21.15.23:0/3308191235 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7feef804a450 con 0x55cde12d7d10 2024-03-24T10:43:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.596+0000 7fef12b68740 1 -- 172.21.15.23:0/3308191235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cde0f3d7e0 con 0x55cde12d7d10 2024-03-24T10:43:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.596+0000 7feef17fa640 1 -- 172.21.15.23:0/3308191235 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7feef804a610 con 0x55cde12d7d10 2024-03-24T10:43:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.596+0000 7fef12b68740 10 client.16653 fetch_fsmap learned FSMap version 26 2024-03-24T10:43:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.596+0000 7fef12b68740 1 -- 172.21.15.23:0/3308191235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cde123cbd0 con 0x55cde12d7d10 2024-03-24T10:43:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.596+0000 7feef17fa640 1 -- 172.21.15.23:0/3308191235 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55cde0f3d7e0 con 0x55cde12d7d10 2024-03-24T10:43:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.597+0000 7feef17fa640 1 -- 172.21.15.23:0/3308191235 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7feef8052070 con 0x55cde12d7d10 2024-03-24T10:43:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.597+0000 7fef12b68740 10 client.16653 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:43:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.597+0000 7fef12b68740 10 client.16653 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:43:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.597+0000 7fef12b68740 20 client.16653 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.597+0000 7fef12b68740 1 --2- 172.21.15.23:0/3308191235 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cde12f0430 0x55cde12f2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.597+0000 7fef12b68740 4 client.16653 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.597+0000 7fef12b68740 1 -- 172.21.15.23:0/3308191235 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cde0f7a720 con 0x55cde12f0430 2024-03-24T10:43:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.597+0000 7fef0dc4d640 1 --2- 172.21.15.23:0/3308191235 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cde12f0430 0x55cde12f2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.597+0000 7fef0dc4d640 1 --2- 172.21.15.23:0/3308191235 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cde12f0430 0x55cde12f2820 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ce7f0b54b388e83a server_cookie=431bfae5377b00a1 in_seq=0 out_seq=0 2024-03-24T10:43:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.597+0000 7feef17fa640 10 client.16653 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:43:59.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.598+0000 7feef17fa640 1 -- 172.21.15.23:0/3308191235 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cde0f7a720 con 0x55cde12f0430 2024-03-24T10:43:59.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.598+0000 7feef17fa640 10 client.16653 handle_command_reply: tid=0 2024-03-24T10:43:59.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.599+0000 7fef12b68740 1 client.16653 shutdown 2024-03-24T10:43:59.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.599+0000 7fef12b68740 1 -- 172.21.15.23:0/3308191235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 msgr2=0x55cde12dde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.599+0000 7fef12b68740 1 --2- 172.21.15.23:0/3308191235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12dde50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7feef8036040 tx=0x7feef80092d0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.599+0000 7fef12b68740 1 -- 172.21.15.23:0/3308191235 shutdown_connections 2024-03-24T10:43:59.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.599+0000 7fef12b68740 1 --2- 172.21.15.23:0/3308191235 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cde12f0430 0x55cde12f2820 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.599+0000 7fef12b68740 1 --2- 172.21.15.23:0/3308191235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12e08f0 0x55cde12de390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.599+0000 7fef12b68740 1 --2- 172.21.15.23:0/3308191235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12dde50 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.599+0000 7feef17fa640 0 client.16653 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:43:59.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.599+0000 7fef12b68740 1 --2- 172.21.15.23:0/3308191235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12d73f0 0x55cde12dd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.599+0000 7fef12b68740 1 -- 172.21.15.23:0/3308191235 >> 172.21.15.23:0/3308191235 conn(0x55cde12d1b20 msgr2=0x55cde12d3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.599+0000 7fef12b68740 1 -- 172.21.15.23:0/3308191235 shutdown_connections 2024-03-24T10:43:59.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.599+0000 7fef12b68740 1 -- 172.21.15.23:0/3308191235 wait complete. 2024-03-24T10:43:59.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.599+0000 7fef12b68740 20 client.16653 trim_cache size 0 max 16384 2024-03-24T10:43:59.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.614+0000 7fef12b68740 1 Processor -- start 2024-03-24T10:43:59.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.614+0000 7fef12b68740 1 -- start start 2024-03-24T10:43:59.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.614+0000 7fef12b68740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d73f0 0x55cde12ec000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.614+0000 7fef12b68740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12d7d10 0x55cde12d0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.614+0000 7fef12b68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12e08f0 0x55cde12f8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.614+0000 7fef12b68740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55cde12ed2d0 con 0x55cde12d73f0 2024-03-24T10:43:59.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7fef12b68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cde12dd700 con 0x55cde12e08f0 2024-03-24T10:43:59.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7fef12b68740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cde12d6f70 con 0x55cde12d7d10 2024-03-24T10:43:59.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7fef0dc4d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d73f0 0x55cde12ec000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7fef0dc4d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d73f0 0x55cde12ec000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45226/0 (socket says 172.21.15.23:45226) 2024-03-24T10:43:59.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7fef0dc4d640 1 -- 172.21.15.23:0/1440703949 learned_addr learned my addr 172.21.15.23:0/1440703949 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7fef0d44c640 1 --2- 172.21.15.23:0/1440703949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12d7d10 0x55cde12d0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7fef0e44e640 1 --2- 172.21.15.23:0/1440703949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12e08f0 0x55cde12f8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7fef0dc4d640 1 -- 172.21.15.23:0/1440703949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12d7d10 msgr2=0x55cde12d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7fef0dc4d640 1 --2- 172.21.15.23:0/1440703949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12d7d10 0x55cde12d0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7fef0dc4d640 1 -- 172.21.15.23:0/1440703949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12e08f0 msgr2=0x55cde12f8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7fef0dc4d640 1 --2- 172.21.15.23:0/1440703949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12e08f0 0x55cde12f8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7fef0dc4d640 1 -- 172.21.15.23:0/1440703949 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cde11d8c80 con 0x55cde12d73f0 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7fef0dc4d640 1 --2- 172.21.15.23:0/1440703949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d73f0 0x55cde12ec000 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fef0400c6a0 tx=0x7fef0406cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e3aee4ba4fb8fb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7feef37fe640 1 -- 172.21.15.23:0/1440703949 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef040a3450 con 0x55cde12d73f0 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7feef37fe640 1 -- 172.21.15.23:0/1440703949 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef04098e00 con 0x55cde12d73f0 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.615+0000 7feef37fe640 1 -- 172.21.15.23:0/1440703949 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef040cbc90 con 0x55cde12d73f0 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.616+0000 7fef12b68740 1 -- 172.21.15.23:0/1440703949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d73f0 msgr2=0x55cde12ec000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.616+0000 7fef12b68740 1 --2- 172.21.15.23:0/1440703949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d73f0 0x55cde12ec000 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fef0400c6a0 tx=0x7fef0406cd20 comp rx=0 tx=0).stop 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.616+0000 7fef12b68740 1 -- 172.21.15.23:0/1440703949 shutdown_connections 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.616+0000 7fef12b68740 1 --2- 172.21.15.23:0/1440703949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12e08f0 0x55cde12f8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.616+0000 7fef12b68740 1 --2- 172.21.15.23:0/1440703949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12d7d10 0x55cde12d0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.616+0000 7fef12b68740 1 --2- 172.21.15.23:0/1440703949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d73f0 0x55cde12ec000 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.616+0000 7fef12b68740 1 -- 172.21.15.23:0/1440703949 >> 172.21.15.23:0/1440703949 conn(0x55cde12d1b20 msgr2=0x55cde12d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.616+0000 7fef12b68740 1 -- 172.21.15.23:0/1440703949 shutdown_connections 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.616+0000 7fef12b68740 1 -- 172.21.15.23:0/1440703949 wait complete. 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.616+0000 7fef12b68740 1 -- start start 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.616+0000 7fef12b68740 1 Processor -- start 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef12b68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12d73f0 0x55cde12e8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef12b68740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12e5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef12b68740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12e08f0 0x55cde12e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef12b68740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55cde12e8bc0 con 0x55cde12d7d10 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef12b68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cde12e8d30 con 0x55cde12d73f0 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef12b68740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cde12e6080 con 0x55cde12e08f0 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef12b68740 10 client.0 _ll_register_callbacks cb 0x7fef04010e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef0d44c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12e5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef0d44c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12e5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45234/0 (socket says 172.21.15.23:45234) 2024-03-24T10:43:59.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef0d44c640 1 -- 172.21.15.23:0/1605930326 learned_addr learned my addr 172.21.15.23:0/1605930326 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef0dc4d640 1 --2- 172.21.15.23:0/1605930326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12d73f0 0x55cde12e8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef0e44e640 1 --2- 172.21.15.23:0/1605930326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12e08f0 0x55cde12e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef0d44c640 1 -- 172.21.15.23:0/1605930326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12e08f0 msgr2=0x55cde12e5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef0d44c640 1 --2- 172.21.15.23:0/1605930326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12e08f0 0x55cde12e5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef0d44c640 1 -- 172.21.15.23:0/1605930326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12d73f0 msgr2=0x55cde12e8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef0d44c640 1 --2- 172.21.15.23:0/1605930326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12d73f0 0x55cde12e8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef0d44c640 1 -- 172.21.15.23:0/1605930326 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef040cd040 con 0x55cde12d7d10 2024-03-24T10:43:59.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef0d44c640 1 --2- 172.21.15.23:0/1605930326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12e5600 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x55cde12d7100 tx=0x7feef802c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a9e55ea37bc323e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7feef17fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:59.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7feef17fa640 1 -- 172.21.15.23:0/1605930326 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef805d690 con 0x55cde12d7d10 2024-03-24T10:43:59.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7feef17fa640 1 -- 172.21.15.23:0/1605930326 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7feef8000e40 con 0x55cde12d7d10 2024-03-24T10:43:59.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.617+0000 7fef12b68740 1 -- 172.21.15.23:0/1605930326 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feef802c9e0 con 0x55cde12d7d10 2024-03-24T10:43:59.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.618+0000 7feef17fa640 1 -- 172.21.15.23:0/1605930326 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7feef805d7f0 con 0x55cde12d7d10 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.618+0000 7fef12b68740 10 client.16659 fetch_fsmap learned FSMap version 26 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.618+0000 7fef12b68740 1 -- 172.21.15.23:0/1605930326 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cde11fe920 con 0x55cde12d7d10 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.618+0000 7feef17fa640 1 -- 172.21.15.23:0/1605930326 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feef802c9e0 con 0x55cde12d7d10 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.618+0000 7feef17fa640 1 -- 172.21.15.23:0/1605930326 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7feef8052040 con 0x55cde12d7d10 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.618+0000 7fef12b68740 10 client.16659 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.618+0000 7fef12b68740 10 client.16659 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.618+0000 7fef12b68740 20 client.16659 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.618+0000 7fef12b68740 1 --2- 172.21.15.23:0/1605930326 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cde13575a0 0x55cde13599c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.618+0000 7fef12b68740 4 client.16659 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.618+0000 7fef12b68740 1 -- 172.21.15.23:0/1605930326 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cde0fc9890 con 0x55cde13575a0 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.618+0000 7fef0dc4d640 1 --2- 172.21.15.23:0/1605930326 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cde13575a0 0x55cde13599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.618+0000 7feef17fa640 10 client.16659 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.618+0000 7fef0dc4d640 1 --2- 172.21.15.23:0/1605930326 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cde13575a0 0x55cde13599c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=13970e93f622142 server_cookie=da88787e811f00b7 in_seq=0 out_seq=0 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7feef17fa640 1 -- 172.21.15.23:0/1605930326 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cde0fc9890 con 0x55cde13575a0 2024-03-24T10:43:59.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7feef17fa640 10 client.16659 handle_command_reply: tid=0 2024-03-24T10:43:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef12b68740 1 client.16659 shutdown 2024-03-24T10:43:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef12b68740 1 -- 172.21.15.23:0/1605930326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 msgr2=0x55cde12e5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef12b68740 1 --2- 172.21.15.23:0/1605930326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12e5600 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x55cde12d7100 tx=0x7feef802c880 comp rx=0 tx=0).stop 2024-03-24T10:43:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef12b68740 1 -- 172.21.15.23:0/1605930326 shutdown_connections 2024-03-24T10:43:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef12b68740 1 --2- 172.21.15.23:0/1605930326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cde12e08f0 0x55cde12e5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef12b68740 1 --2- 172.21.15.23:0/1605930326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cde12d7d10 0x55cde12e5600 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef12b68740 1 --2- 172.21.15.23:0/1605930326 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cde13575a0 0x55cde13599c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef12b68740 1 --2- 172.21.15.23:0/1605930326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cde12d73f0 0x55cde12e8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7feef17fa640 0 client.16659 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:43:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef12b68740 1 -- 172.21.15.23:0/1605930326 >> 172.21.15.23:0/1605930326 conn(0x55cde12d1b20 msgr2=0x55cde12de120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef12b68740 1 -- 172.21.15.23:0/1605930326 shutdown_connections 2024-03-24T10:43:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef12b68740 1 -- 172.21.15.23:0/1605930326 wait complete. 2024-03-24T10:43:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef12b68740 20 client.16659 trim_cache size 0 max 16384 2024-03-24T10:43:59.626 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:43:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef0feb1640 1 -- 172.21.15.23:0/1540307972 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feef40a22b0 msgr2=0x7feef40a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef0feb1640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feef40a22b0 0x7feef40a4770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fef00005c40 tx=0x7fef00009500 comp rx=0 tx=0).stop 2024-03-24T10:43:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef0feb1640 1 -- 172.21.15.23:0/1540307972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef08102980 msgr2=0x7fef08199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.619+0000 7fef0feb1640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef08102980 0x7fef08199fc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fef0400e9f0 tx=0x7fef0400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.620+0000 7fef0feb1640 1 -- 172.21.15.23:0/1540307972 shutdown_connections 2024-03-24T10:43:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.620+0000 7fef0feb1640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feef40a22b0 0x7feef40a4770 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.620+0000 7fef0feb1640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef0810bd50 0x7fef0819e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.620+0000 7fef0feb1640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef081032a0 0x7fef0819a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.620+0000 7fef0feb1640 1 --2- 172.21.15.23:0/1540307972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef08102980 0x7fef08199fc0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.620+0000 7fef0feb1640 1 -- 172.21.15.23:0/1540307972 >> 172.21.15.23:0/1540307972 conn(0x7fef08077770 msgr2=0x7fef080fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.620+0000 7fef0feb1640 1 -- 172.21.15.23:0/1540307972 shutdown_connections 2024-03-24T10:43:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.620+0000 7fef0feb1640 1 -- 172.21.15.23:0/1540307972 wait complete. 2024-03-24T10:43:59.630 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:43:59.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.736+0000 7f38b3815640 1 -- 172.21.15.23:0/803788894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38ac100f60 msgr2=0x7f38ac10d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.736+0000 7f38b3815640 1 --2- 172.21.15.23:0/803788894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38ac100f60 0x7f38ac10d420 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f38a4009ca0 tx=0x7f38a402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.736+0000 7f38b3815640 1 -- 172.21.15.23:0/803788894 shutdown_connections 2024-03-24T10:43:59.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.736+0000 7f38b3815640 1 --2- 172.21.15.23:0/803788894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38ac100f60 0x7f38ac10d420 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.736+0000 7f38b3815640 1 --2- 172.21.15.23:0/803788894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac1005a0 0x7f38ac100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.736+0000 7f38b3815640 1 --2- 172.21.15.23:0/803788894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38ac106660 0x7f38ac106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.736+0000 7f38b3815640 1 -- 172.21.15.23:0/803788894 >> 172.21.15.23:0/803788894 conn(0x7f38ac077710 msgr2=0x7f38ac077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.736+0000 7f38b3815640 1 -- 172.21.15.23:0/803788894 shutdown_connections 2024-03-24T10:43:59.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.736+0000 7f38b3815640 1 -- 172.21.15.23:0/803788894 wait complete. 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b3815640 1 Processor -- start 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b3815640 1 -- start start 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b3815640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac1005a0 0x7f38ac19d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b3815640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38ac100f60 0x7f38ac19ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b3815640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38ac106660 0x7f38ac197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b3815640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac1980e0 con 0x7f38ac100f60 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b3815640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac198250 con 0x7f38ac1005a0 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b3815640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac198410 con 0x7f38ac106660 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b0db0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38ac100f60 0x7f38ac19ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b15b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac1005a0 0x7f38ac19d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b0db0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38ac100f60 0x7f38ac19ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45266/0 (socket says 172.21.15.23:45266) 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b0db0640 1 -- 172.21.15.23:0/3541597618 learned_addr learned my addr 172.21.15.23:0/3541597618 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b1db2640 1 --2- 172.21.15.23:0/3541597618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38ac106660 0x7f38ac197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b0db0640 1 -- 172.21.15.23:0/3541597618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38ac106660 msgr2=0x7f38ac197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b0db0640 1 --2- 172.21.15.23:0/3541597618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38ac106660 0x7f38ac197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b0db0640 1 -- 172.21.15.23:0/3541597618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac1005a0 msgr2=0x7f38ac19d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b0db0640 1 --2- 172.21.15.23:0/3541597618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac1005a0 0x7f38ac19d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b0db0640 1 -- 172.21.15.23:0/3541597618 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a4009900 con 0x7f38ac100f60 2024-03-24T10:43:59.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.737+0000 7f38b0db0640 1 --2- 172.21.15.23:0/3541597618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38ac100f60 0x7f38ac19ddd0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e9f0 tx=0x7f38a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.738+0000 7f389e7fc640 1 -- 172.21.15.23:0/3541597618 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a8009700 con 0x7f38ac100f60 2024-03-24T10:43:59.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.738+0000 7f389e7fc640 1 -- 172.21.15.23:0/3541597618 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38a800cd80 con 0x7f38ac100f60 2024-03-24T10:43:59.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.738+0000 7f38b3815640 1 -- 172.21.15.23:0/3541597618 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38ac1986f0 con 0x7f38ac100f60 2024-03-24T10:43:59.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.738+0000 7f389e7fc640 1 -- 172.21.15.23:0/3541597618 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a8018920 con 0x7f38ac100f60 2024-03-24T10:43:59.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.738+0000 7f38b3815640 1 -- 172.21.15.23:0/3541597618 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38ac1a4930 con 0x7f38ac100f60 2024-03-24T10:43:59.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.740+0000 7f389e7fc640 1 -- 172.21.15.23:0/3541597618 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f38a8018a80 con 0x7f38ac100f60 2024-03-24T10:43:59.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.740+0000 7f38b63f2740 1 Processor -- start 2024-03-24T10:43:59.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b63f2740 1 -- start start 2024-03-24T10:43:59.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b63f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0283f0 0x5588ac0287d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b63f2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac028d10 0x5588ac0313b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b63f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0318f0 0x5588ac033ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b63f2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5588abf96840 con 0x5588ac028d10 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b63f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5588abf93d70 con 0x5588ac0283f0 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b63f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5588abf8f550 con 0x5588ac0318f0 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f389e7fc640 1 --2- 172.21.15.23:0/3541597618 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f38980a22b0 0x7f38980a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b15b1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac028d10 0x5588ac0313b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b1db2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0283f0 0x5588ac0287d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b15b1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac028d10 0x5588ac0313b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45280/0 (socket says 172.21.15.23:45280) 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b15b1640 1 -- 172.21.15.23:0/1184662900 learned_addr learned my addr 172.21.15.23:0/1184662900 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b0db0640 1 --2- 172.21.15.23:0/1184662900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0318f0 0x5588ac033ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b1db2640 1 --2- 172.21.15.23:0/3541597618 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f38980a22b0 0x7f38980a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f389e7fc640 1 -- 172.21.15.23:0/3541597618 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f38a8014070 con 0x7f38ac100f60 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b1db2640 1 -- 172.21.15.23:0/1184662900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0318f0 msgr2=0x5588ac033ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b1db2640 1 --2- 172.21.15.23:0/1184662900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0318f0 0x5588ac033ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b1db2640 1 -- 172.21.15.23:0/1184662900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac028d10 msgr2=0x5588ac0313b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b1db2640 1 --2- 172.21.15.23:0/1184662900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac028d10 0x5588ac0313b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b1db2640 1 -- 172.21.15.23:0/1184662900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588abf24de0 con 0x5588ac0283f0 2024-03-24T10:43:59.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.741+0000 7f38b15b1640 1 --2- 172.21.15.23:0/1184662900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac028d10 0x5588ac0313b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:59.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.742+0000 7f38b1db2640 1 --2- 172.21.15.23:0/1184662900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0283f0 0x5588ac0287d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f38a402c700 tx=0x7f38a4034440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d635d3a776e9444 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.742+0000 7f38b1db2640 1 --2- 172.21.15.23:0/3541597618 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f38980a22b0 0x7f38980a4770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f38a4048000 tx=0x7f38a4049000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.742+0000 7f38977fe640 1 -- 172.21.15.23:0/1184662900 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4034c90 con 0x5588ac0283f0 2024-03-24T10:43:59.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.742+0000 7f38977fe640 1 -- 172.21.15.23:0/1184662900 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38a4003970 con 0x5588ac0283f0 2024-03-24T10:43:59.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.742+0000 7f38977fe640 1 -- 172.21.15.23:0/1184662900 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4004280 con 0x5588ac0283f0 2024-03-24T10:43:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 -- 172.21.15.23:0/1184662900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0283f0 msgr2=0x5588ac0287d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 --2- 172.21.15.23:0/1184662900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0283f0 0x5588ac0287d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f38a402c700 tx=0x7f38a4034440 comp rx=0 tx=0).stop 2024-03-24T10:43:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 -- 172.21.15.23:0/1184662900 shutdown_connections 2024-03-24T10:43:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 --2- 172.21.15.23:0/1184662900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0318f0 0x5588ac033ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 --2- 172.21.15.23:0/1184662900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac028d10 0x5588ac0313b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 --2- 172.21.15.23:0/1184662900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0283f0 0x5588ac0287d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 -- 172.21.15.23:0/1184662900 >> 172.21.15.23:0/1184662900 conn(0x5588ac022b20 msgr2=0x5588ac022f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 -- 172.21.15.23:0/1184662900 shutdown_connections 2024-03-24T10:43:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 -- 172.21.15.23:0/1184662900 wait complete. 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 -- start start 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 Processor -- start 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0283f0 0x5588ac02e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac028d10 0x5588ac02ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac0318f0 0x5588ac02f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5588abc62430 con 0x5588ac0318f0 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5588abc1ee00 con 0x5588ac028d10 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.743+0000 7f38b63f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5588abf14590 con 0x5588ac0283f0 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b63f2740 10 client.0 _ll_register_callbacks cb 0x5588abf83fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b1db2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac0318f0 0x5588ac02f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b15b1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0283f0 0x5588ac02e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b0db0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac028d10 0x5588ac02ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b15b1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0283f0 0x5588ac02e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:34666/0 (socket says 172.21.15.23:34666) 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b15b1640 1 -- 172.21.15.23:0/3727584487 learned_addr learned my addr 172.21.15.23:0/3727584487 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b1db2640 1 -- 172.21.15.23:0/3727584487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0283f0 msgr2=0x5588ac02e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b1db2640 1 --2- 172.21.15.23:0/3727584487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0283f0 0x5588ac02e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b1db2640 1 -- 172.21.15.23:0/3727584487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac028d10 msgr2=0x5588ac02ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b1db2640 1 --2- 172.21.15.23:0/3727584487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac028d10 0x5588ac02ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b1db2640 1 -- 172.21.15.23:0/3727584487 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38ac10dd20 con 0x5588ac0318f0 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b15b1640 1 --2- 172.21.15.23:0/3727584487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0283f0 0x5588ac02e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b1db2640 1 --2- 172.21.15.23:0/3727584487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac0318f0 0x5588ac02f470 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f38a4004af0 tx=0x7f38a40328d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5aac1b0c48e7c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f3894ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f3894ff9640 1 -- 172.21.15.23:0/3727584487 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4034ea0 con 0x5588ac0318f0 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b63f2740 1 -- 172.21.15.23:0/3727584487 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5588abc8e7e0 con 0x5588ac0318f0 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f3894ff9640 1 -- 172.21.15.23:0/3727584487 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f38a4004040 con 0x5588ac0318f0 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f3894ff9640 1 -- 172.21.15.23:0/3727584487 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f38a4009540 con 0x5588ac0318f0 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b63f2740 10 client.16671 fetch_fsmap learned FSMap version 26 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.744+0000 7f38b63f2740 1 -- 172.21.15.23:0/3727584487 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5588abf8dbd0 con 0x5588ac0318f0 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.745+0000 7f3894ff9640 1 -- 172.21.15.23:0/3727584487 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5588abc8e7e0 con 0x5588ac0318f0 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.745+0000 7f3894ff9640 1 -- 172.21.15.23:0/3727584487 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f38a404c880 con 0x5588ac0318f0 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.745+0000 7f38b63f2740 10 client.16671 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:43:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.745+0000 7f38b63f2740 10 client.16671 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:43:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.745+0000 7f38b63f2740 20 client.16671 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.745+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3727584487 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5588ac0417d0 0x5588ac043bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.745+0000 7f38b63f2740 4 client.16671 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.745+0000 7f38b63f2740 1 -- 172.21.15.23:0/3727584487 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588abccb720 con 0x5588ac0417d0 2024-03-24T10:43:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.745+0000 7f38b15b1640 1 --2- 172.21.15.23:0/3727584487 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5588ac0417d0 0x5588ac043bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.746+0000 7f38b15b1640 1 --2- 172.21.15.23:0/3727584487 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5588ac0417d0 0x5588ac043bf0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fdbc440baf41a5e0 server_cookie=2912e12c88af89cb in_seq=0 out_seq=0 2024-03-24T10:43:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.746+0000 7f3894ff9640 10 client.16671 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:43:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.746+0000 7f3894ff9640 1 -- 172.21.15.23:0/3727584487 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5588abccb720 con 0x5588ac0417d0 2024-03-24T10:43:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.746+0000 7f3894ff9640 10 client.16671 handle_command_reply: tid=0 2024-03-24T10:43:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.746+0000 7f38b63f2740 1 client.16671 shutdown 2024-03-24T10:43:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.746+0000 7f38b63f2740 1 -- 172.21.15.23:0/3727584487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac0318f0 msgr2=0x5588ac02f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.746+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3727584487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac0318f0 0x5588ac02f470 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f38a4004af0 tx=0x7f38a40328d0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.746+0000 7f38b63f2740 1 -- 172.21.15.23:0/3727584487 shutdown_connections 2024-03-24T10:43:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.746+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3727584487 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5588ac0417d0 0x5588ac043bf0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.746+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3727584487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac0318f0 0x5588ac02f470 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.746+0000 7f3894ff9640 0 client.16671 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:43:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.746+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3727584487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac028d10 0x5588ac02ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.746+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3727584487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0283f0 0x5588ac02e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.746+0000 7f38b63f2740 1 -- 172.21.15.23:0/3727584487 >> 172.21.15.23:0/3727584487 conn(0x5588ac022b20 msgr2=0x5588ac024870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.747+0000 7f38b63f2740 1 -- 172.21.15.23:0/3727584487 shutdown_connections 2024-03-24T10:43:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.747+0000 7f38b63f2740 1 -- 172.21.15.23:0/3727584487 wait complete. 2024-03-24T10:43:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.747+0000 7f38b63f2740 20 client.16671 trim_cache size 0 max 16384 2024-03-24T10:43:59.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.760+0000 7f38b63f2740 1 Processor -- start 2024-03-24T10:43:59.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b63f2740 1 -- start start 2024-03-24T10:43:59.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b63f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0283f0 0x5588ac02e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b63f2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac028d10 0x5588ac03d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b63f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0318f0 0x5588ac0494c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b63f2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5588ac034330 con 0x5588ac028d10 2024-03-24T10:43:59.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b63f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5588ac02e810 con 0x5588ac0283f0 2024-03-24T10:43:59.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b63f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5588ac044270 con 0x5588ac0318f0 2024-03-24T10:43:59.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b0db0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac028d10 0x5588ac03d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b0db0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac028d10 0x5588ac03d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45300/0 (socket says 172.21.15.23:45300) 2024-03-24T10:43:59.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b0db0640 1 -- 172.21.15.23:0/3441706133 learned_addr learned my addr 172.21.15.23:0/3441706133 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b1db2640 1 --2- 172.21.15.23:0/3441706133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0318f0 0x5588ac0494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b15b1640 1 --2- 172.21.15.23:0/3441706133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0283f0 0x5588ac02e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b0db0640 1 -- 172.21.15.23:0/3441706133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0318f0 msgr2=0x5588ac0494c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b0db0640 1 --2- 172.21.15.23:0/3441706133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0318f0 0x5588ac0494c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b0db0640 1 -- 172.21.15.23:0/3441706133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0283f0 msgr2=0x5588ac02e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b0db0640 1 --2- 172.21.15.23:0/3441706133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0283f0 0x5588ac02e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b0db0640 1 -- 172.21.15.23:0/3441706133 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588abf29c80 con 0x5588ac028d10 2024-03-24T10:43:59.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f38b0db0640 1 --2- 172.21.15.23:0/3441706133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac028d10 0x5588ac03d120 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f38a809add0 tx=0x7f38a809ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898ca17e394c810e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.761+0000 7f3896ffd640 1 -- 172.21.15.23:0/3441706133 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a80a3530 con 0x5588ac028d10 2024-03-24T10:43:59.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.762+0000 7f3896ffd640 1 -- 172.21.15.23:0/3441706133 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38a8010b80 con 0x5588ac028d10 2024-03-24T10:43:59.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.762+0000 7f3896ffd640 1 -- 172.21.15.23:0/3441706133 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a809a4f0 con 0x5588ac028d10 2024-03-24T10:43:59.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.762+0000 7f38b63f2740 1 -- 172.21.15.23:0/3441706133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac028d10 msgr2=0x5588ac03d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.762+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3441706133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac028d10 0x5588ac03d120 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f38a809add0 tx=0x7f38a809ad40 comp rx=0 tx=0).stop 2024-03-24T10:43:59.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.762+0000 7f38b63f2740 1 -- 172.21.15.23:0/3441706133 shutdown_connections 2024-03-24T10:43:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.762+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3441706133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac0318f0 0x5588ac0494c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.762+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3441706133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac028d10 0x5588ac03d120 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.762+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3441706133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0283f0 0x5588ac02e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.762+0000 7f38b63f2740 1 -- 172.21.15.23:0/3441706133 >> 172.21.15.23:0/3441706133 conn(0x5588ac022b20 msgr2=0x5588ac0309a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.762+0000 7f38b63f2740 1 -- 172.21.15.23:0/3441706133 shutdown_connections 2024-03-24T10:43:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.762+0000 7f38b63f2740 1 -- 172.21.15.23:0/3441706133 wait complete. 2024-03-24T10:43:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b63f2740 1 -- start start 2024-03-24T10:43:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b63f2740 1 Processor -- start 2024-03-24T10:43:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b63f2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac0283f0 0x5588ac035080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b63f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac028d10 0x5588ac0355c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b63f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0318f0 0x5588ac035b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b63f2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5588ac0a6af0 con 0x5588ac0283f0 2024-03-24T10:43:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b63f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5588ac0a6c30 con 0x5588ac0318f0 2024-03-24T10:43:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b63f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5588ac0a6d70 con 0x5588ac028d10 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b63f2740 10 client.0 _ll_register_callbacks cb 0x7f38a0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b15b1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac0283f0 0x5588ac035080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b15b1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac0283f0 0x5588ac035080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45310/0 (socket says 172.21.15.23:45310) 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b15b1640 1 -- 172.21.15.23:0/3229097282 learned_addr learned my addr 172.21.15.23:0/3229097282 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b0db0640 1 --2- 172.21.15.23:0/3229097282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac028d10 0x5588ac0355c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b1db2640 1 --2- 172.21.15.23:0/3229097282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0318f0 0x5588ac035b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b15b1640 1 -- 172.21.15.23:0/3229097282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac028d10 msgr2=0x5588ac0355c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b15b1640 1 --2- 172.21.15.23:0/3229097282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac028d10 0x5588ac0355c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b15b1640 1 -- 172.21.15.23:0/3229097282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0318f0 msgr2=0x5588ac035b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b15b1640 1 --2- 172.21.15.23:0/3229097282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0318f0 0x5588ac035b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.763+0000 7f38b15b1640 1 -- 172.21.15.23:0/3229097282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38ac1a4930 con 0x5588ac0283f0 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f38b15b1640 1 --2- 172.21.15.23:0/3229097282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac0283f0 0x5588ac035080 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f38a0005eb0 tx=0x7f38a0012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62be15e7d02978f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f3894ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f3894ff9640 1 -- 172.21.15.23:0/3229097282 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a0017070 con 0x5588ac0283f0 2024-03-24T10:43:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f3894ff9640 1 -- 172.21.15.23:0/3229097282 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f38a0004230 con 0x5588ac0283f0 2024-03-24T10:43:59.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f38b63f2740 1 -- 172.21.15.23:0/3229097282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38a4003970 con 0x5588ac0283f0 2024-03-24T10:43:59.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f3894ff9640 1 -- 172.21.15.23:0/3229097282 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f38a00043f0 con 0x5588ac0283f0 2024-03-24T10:43:59.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f38b63f2740 10 client.16677 fetch_fsmap learned FSMap version 26 2024-03-24T10:43:59.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f38b63f2740 1 -- 172.21.15.23:0/3229097282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5588abf4f920 con 0x5588ac0283f0 2024-03-24T10:43:59.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f3894ff9640 1 -- 172.21.15.23:0/3229097282 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38a4003970 con 0x5588ac0283f0 2024-03-24T10:43:59.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f3894ff9640 1 -- 172.21.15.23:0/3229097282 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f38a0020680 con 0x5588ac0283f0 2024-03-24T10:43:59.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f38b63f2740 10 client.16677 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:43:59.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f38b63f2740 10 client.16677 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:43:59.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f38b63f2740 20 client.16677 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:59.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3229097282 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5588ac0a85c0 0x5588ac0aa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f38b63f2740 4 client.16677 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:43:59.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f38b63f2740 1 -- 172.21.15.23:0/3229097282 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5588abd1a890 con 0x5588ac0a85c0 2024-03-24T10:43:59.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.764+0000 7f38b0db0640 1 --2- 172.21.15.23:0/3229097282 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5588ac0a85c0 0x5588ac0aa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f3894ff9640 10 client.16677 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f38b0db0640 1 --2- 172.21.15.23:0/3229097282 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5588ac0a85c0 0x5588ac0aa9e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=26de8d671eb1400 server_cookie=26db0ae7d1e64892 in_seq=0 out_seq=0 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f3894ff9640 1 -- 172.21.15.23:0/3229097282 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5588abd1a890 con 0x5588ac0a85c0 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f3894ff9640 10 client.16677 handle_command_reply: tid=0 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f38b63f2740 1 client.16677 shutdown 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f38b63f2740 1 -- 172.21.15.23:0/3229097282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac0283f0 msgr2=0x5588ac035080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3229097282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac0283f0 0x5588ac035080 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f38a0005eb0 tx=0x7f38a0012ba0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f38b63f2740 1 -- 172.21.15.23:0/3229097282 shutdown_connections 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3229097282 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5588ac0a85c0 0x5588ac0aa9e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3229097282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588ac0318f0 0x5588ac035b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3229097282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588ac028d10 0x5588ac0355c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f3894ff9640 0 client.16677 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f38b63f2740 1 --2- 172.21.15.23:0/3229097282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588ac0283f0 0x5588ac035080 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f38b63f2740 1 -- 172.21.15.23:0/3229097282 >> 172.21.15.23:0/3229097282 conn(0x5588ac022b20 msgr2=0x5588ac021430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f38b63f2740 1 -- 172.21.15.23:0/3229097282 shutdown_connections 2024-03-24T10:43:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f38b63f2740 1 -- 172.21.15.23:0/3229097282 wait complete. 2024-03-24T10:43:59.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.765+0000 7f38b63f2740 20 client.16677 trim_cache size 0 max 16384 2024-03-24T10:43:59.772 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:43:59.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.766+0000 7f38b3815640 1 -- 172.21.15.23:0/3541597618 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f38980a22b0 msgr2=0x7f38980a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.766+0000 7f38b3815640 1 --2- 172.21.15.23:0/3541597618 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f38980a22b0 0x7f38980a4770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f38a4048000 tx=0x7f38a4049000 comp rx=0 tx=0).stop 2024-03-24T10:43:59.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.766+0000 7f38b3815640 1 -- 172.21.15.23:0/3541597618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38ac100f60 msgr2=0x7f38ac19ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.766+0000 7f38b3815640 1 --2- 172.21.15.23:0/3541597618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38ac100f60 0x7f38ac19ddd0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e9f0 tx=0x7f38a800eeb0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.766+0000 7f38b3815640 1 -- 172.21.15.23:0/3541597618 shutdown_connections 2024-03-24T10:43:59.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.766+0000 7f38b3815640 1 --2- 172.21.15.23:0/3541597618 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f38980a22b0 0x7f38980a4770 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.766+0000 7f38b3815640 1 --2- 172.21.15.23:0/3541597618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38ac106660 0x7f38ac197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.766+0000 7f38b3815640 1 --2- 172.21.15.23:0/3541597618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38ac100f60 0x7f38ac19ddd0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.766+0000 7f38b3815640 1 --2- 172.21.15.23:0/3541597618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38ac1005a0 0x7f38ac19d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.766+0000 7f38b3815640 1 -- 172.21.15.23:0/3541597618 >> 172.21.15.23:0/3541597618 conn(0x7f38ac077710 msgr2=0x7f38ac0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.766+0000 7f38b3815640 1 -- 172.21.15.23:0/3541597618 shutdown_connections 2024-03-24T10:43:59.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.766+0000 7f38b3815640 1 -- 172.21.15.23:0/3541597618 wait complete. 2024-03-24T10:43:59.775 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:43:59 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/458164212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:43:59.777 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:43:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.884+0000 7f059f181640 1 -- 172.21.15.23:0/2795796855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0598100f80 msgr2=0x7f059810d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.884+0000 7f059f181640 1 --2- 172.21.15.23:0/2795796855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0598100f80 0x7f059810d490 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0590009cd0 tx=0x7f059002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.884+0000 7f059f181640 1 -- 172.21.15.23:0/2795796855 shutdown_connections 2024-03-24T10:43:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.884+0000 7f059f181640 1 --2- 172.21.15.23:0/2795796855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0598100f80 0x7f059810d490 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.884+0000 7f059f181640 1 --2- 172.21.15.23:0/2795796855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05981005c0 0x7f0598100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.884+0000 7f059f181640 1 --2- 172.21.15.23:0/2795796855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0598106660 0x7f0598106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.884+0000 7f059f181640 1 -- 172.21.15.23:0/2795796855 >> 172.21.15.23:0/2795796855 conn(0x7f0598076990 msgr2=0x7f0598076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.884+0000 7f059f181640 1 -- 172.21.15.23:0/2795796855 shutdown_connections 2024-03-24T10:43:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.884+0000 7f059f181640 1 -- 172.21.15.23:0/2795796855 wait complete. 2024-03-24T10:43:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.884+0000 7f059f181640 1 Processor -- start 2024-03-24T10:43:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.884+0000 7f059f181640 1 -- start start 2024-03-24T10:43:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f059f181640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05981005c0 0x7f0598198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f059f181640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0598100f80 0x7f0598199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f059f181640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0598106660 0x7f05981a3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f059f181640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0598199870 con 0x7f0598100f80 2024-03-24T10:43:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f059f181640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f05981a4090 con 0x7f0598106660 2024-03-24T10:43:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f059f181640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f05981a41d0 con 0x7f05981005c0 2024-03-24T10:43:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f059cf1d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05981005c0 0x7f0598198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f0597fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0598100f80 0x7f0598199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f0597fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0598100f80 0x7f0598199010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45324/0 (socket says 172.21.15.23:45324) 2024-03-24T10:43:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f059cf1d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05981005c0 0x7f0598198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:34678/0 (socket says 172.21.15.23:34678) 2024-03-24T10:43:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f0597fff640 1 -- 172.21.15.23:0/1890867226 learned_addr learned my addr 172.21.15.23:0/1890867226 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f059d71e640 1 --2- 172.21.15.23:0/1890867226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0598106660 0x7f05981a3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f0597fff640 1 -- 172.21.15.23:0/1890867226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05981005c0 msgr2=0x7f0598198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f0597fff640 1 --2- 172.21.15.23:0/1890867226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05981005c0 0x7f0598198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f0597fff640 1 -- 172.21.15.23:0/1890867226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0598106660 msgr2=0x7f05981a3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f0597fff640 1 --2- 172.21.15.23:0/1890867226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0598106660 0x7f05981a3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f0597fff640 1 -- 172.21.15.23:0/1890867226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0590009900 con 0x7f0598100f80 2024-03-24T10:43:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f0597fff640 1 --2- 172.21.15.23:0/1890867226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0598100f80 0x7f0598199010 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f058400e9f0 tx=0x7f058400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f0595ffb640 1 -- 172.21.15.23:0/1890867226 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0584009700 con 0x7f0598100f80 2024-03-24T10:43:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f059f181640 1 -- 172.21.15.23:0/1890867226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05981a4430 con 0x7f0598100f80 2024-03-24T10:43:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f0595ffb640 1 -- 172.21.15.23:0/1890867226 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f058400cd80 con 0x7f0598100f80 2024-03-24T10:43:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.885+0000 7f0595ffb640 1 -- 172.21.15.23:0/1890867226 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0584018920 con 0x7f0598100f80 2024-03-24T10:43:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.886+0000 7f059f181640 1 -- 172.21.15.23:0/1890867226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05981a4940 con 0x7f0598100f80 2024-03-24T10:43:59.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f0595ffb640 1 -- 172.21.15.23:0/1890867226 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f0584018a80 con 0x7f0598100f80 2024-03-24T10:43:59.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f05a19fd740 1 Processor -- start 2024-03-24T10:43:59.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f05a19fd740 1 -- start start 2024-03-24T10:43:59.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f05a19fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a814d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f05a19fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a81563b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f05a19fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 0x5601a8158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f05a19fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5601a80bb840 con 0x5601a814dd10 2024-03-24T10:43:59.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f05a19fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5601a80b8d70 con 0x5601a814d3f0 2024-03-24T10:43:59.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f05a19fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5601a80b4550 con 0x5601a81568f0 2024-03-24T10:43:59.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f0595ffb640 1 --2- 172.21.15.23:0/1890867226 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f05880a2230 0x7f05880a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f059cf1d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a81563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f059cf1d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a81563b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45336/0 (socket says 172.21.15.23:45336) 2024-03-24T10:43:59.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f059cf1d640 1 -- 172.21.15.23:0/1945252801 learned_addr learned my addr 172.21.15.23:0/1945252801 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f059d71e640 1 --2- 172.21.15.23:0/1945252801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a814d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f0597fff640 1 --2- 172.21.15.23:0/1945252801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 0x5601a8158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f059d71e640 1 --2- 172.21.15.23:0/1890867226 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f05880a2230 0x7f05880a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.888+0000 7f0595ffb640 1 -- 172.21.15.23:0/1890867226 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0584014070 con 0x7f0598100f80 2024-03-24T10:43:59.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.889+0000 7f059cf1d640 1 -- 172.21.15.23:0/1945252801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 msgr2=0x5601a8158ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.889+0000 7f059cf1d640 1 --2- 172.21.15.23:0/1945252801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 0x5601a8158ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.889+0000 7f059cf1d640 1 -- 172.21.15.23:0/1945252801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 msgr2=0x5601a814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.889+0000 7f059cf1d640 1 --2- 172.21.15.23:0/1945252801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a814d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.889+0000 7f059cf1d640 1 -- 172.21.15.23:0/1945252801 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601a8049de0 con 0x5601a814dd10 2024-03-24T10:43:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.889+0000 7f0597fff640 1 --2- 172.21.15.23:0/1945252801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 0x5601a8158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:43:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.889+0000 7f059cf1d640 1 --2- 172.21.15.23:0/1945252801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a81563b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f058c009c20 tx=0x7f058c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f52ca944b9aeafc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.889+0000 7f059d71e640 1 --2- 172.21.15.23:0/1890867226 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f05880a2230 0x7f05880a46f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0590005cc0 tx=0x7f0590005c50 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.889+0000 7f0582ffd640 1 -- 172.21.15.23:0/1945252801 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c004220 con 0x5601a814dd10 2024-03-24T10:43:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.889+0000 7f0582ffd640 1 -- 172.21.15.23:0/1945252801 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f058c009e90 con 0x5601a814dd10 2024-03-24T10:43:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.889+0000 7f0582ffd640 1 -- 172.21.15.23:0/1945252801 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c019690 con 0x5601a814dd10 2024-03-24T10:43:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.890+0000 7f05a19fd740 1 -- 172.21.15.23:0/1945252801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 msgr2=0x5601a81563b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.890+0000 7f05a19fd740 1 --2- 172.21.15.23:0/1945252801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a81563b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f058c009c20 tx=0x7f058c011810 comp rx=0 tx=0).stop 2024-03-24T10:43:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.890+0000 7f05a19fd740 1 -- 172.21.15.23:0/1945252801 shutdown_connections 2024-03-24T10:43:59.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.890+0000 7f05a19fd740 1 --2- 172.21.15.23:0/1945252801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 0x5601a8158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.890+0000 7f05a19fd740 1 --2- 172.21.15.23:0/1945252801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a81563b0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.890+0000 7f05a19fd740 1 --2- 172.21.15.23:0/1945252801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a814d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.890+0000 7f05a19fd740 1 -- 172.21.15.23:0/1945252801 >> 172.21.15.23:0/1945252801 conn(0x5601a8147b20 msgr2=0x5601a8147f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.890+0000 7f05a19fd740 1 -- 172.21.15.23:0/1945252801 shutdown_connections 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.890+0000 7f05a19fd740 1 -- 172.21.15.23:0/1945252801 wait complete. 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.890+0000 7f05a19fd740 1 -- start start 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.890+0000 7f05a19fd740 1 Processor -- start 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f05a19fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a8153940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f05a19fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a814dd10 0x5601a8153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f05a19fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a81568f0 0x5601a81543c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f05a19fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5601a7d87430 con 0x5601a81568f0 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f05a19fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5601a7d43e00 con 0x5601a814d3f0 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f05a19fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5601a8039590 con 0x5601a814dd10 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f05a19fd740 10 client.0 _ll_register_callbacks cb 0x5601a80a8fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:59.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f059d71e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a81568f0 0x5601a81543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f059cf1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a8153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f059d71e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a81568f0 0x5601a81543c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45340/0 (socket says 172.21.15.23:45340) 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f059d71e640 1 -- 172.21.15.23:0/4068069834 learned_addr learned my addr 172.21.15.23:0/4068069834 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f0597fff640 1 --2- 172.21.15.23:0/4068069834 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a814dd10 0x5601a8153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f059d71e640 1 -- 172.21.15.23:0/4068069834 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a814dd10 msgr2=0x5601a8153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f059d71e640 1 --2- 172.21.15.23:0/4068069834 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a814dd10 0x5601a8153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f059d71e640 1 -- 172.21.15.23:0/4068069834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 msgr2=0x5601a8153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f059d71e640 1 --2- 172.21.15.23:0/4068069834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a8153940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f059d71e640 1 -- 172.21.15.23:0/4068069834 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f058c009850 con 0x5601a81568f0 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f059d71e640 1 --2- 172.21.15.23:0/4068069834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a81568f0 0x5601a81543c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0590036000 tx=0x7f05900095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f95f72076cf49e99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f0577fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f0577fff640 1 -- 172.21.15.23:0/4068069834 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0590034560 con 0x5601a81568f0 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f0577fff640 1 -- 172.21.15.23:0/4068069834 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0590003b70 con 0x5601a81568f0 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f05a19fd740 1 -- 172.21.15.23:0/4068069834 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5601a7db37e0 con 0x5601a81568f0 2024-03-24T10:43:59.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.891+0000 7f0577fff640 1 -- 172.21.15.23:0/4068069834 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0590003d30 con 0x5601a81568f0 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.892+0000 7f05a19fd740 10 client.16689 fetch_fsmap learned FSMap version 26 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.892+0000 7f05a19fd740 1 -- 172.21.15.23:0/4068069834 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5601a80b2bd0 con 0x5601a81568f0 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.892+0000 7f0577fff640 1 -- 172.21.15.23:0/4068069834 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f059005c020 con 0x5601a81568f0 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.892+0000 7f0577fff640 1 -- 172.21.15.23:0/4068069834 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f059004c070 con 0x5601a81568f0 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.892+0000 7f05a19fd740 10 client.16689 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.892+0000 7f05a19fd740 10 client.16689 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.892+0000 7f05a19fd740 20 client.16689 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.892+0000 7f05a19fd740 1 --2- 172.21.15.23:0/4068069834 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5601a8166a00 0x5601a8168e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.892+0000 7f05a19fd740 4 client.16689 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.892+0000 7f05a19fd740 1 -- 172.21.15.23:0/4068069834 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601a7df0720 con 0x5601a8166a00 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.892+0000 7f059cf1d640 1 --2- 172.21.15.23:0/4068069834 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5601a8166a00 0x5601a8168e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.893+0000 7f059cf1d640 1 --2- 172.21.15.23:0/4068069834 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5601a8166a00 0x5601a8168e20 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8257fe46b4cb639 server_cookie=903a40267fac9e1d in_seq=0 out_seq=0 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.893+0000 7f0577fff640 10 client.16689 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.893+0000 7f0577fff640 1 -- 172.21.15.23:0/4068069834 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5601a7df0720 con 0x5601a8166a00 2024-03-24T10:43:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.893+0000 7f0577fff640 10 client.16689 handle_command_reply: tid=0 2024-03-24T10:43:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.893+0000 7f05a19fd740 1 client.16689 shutdown 2024-03-24T10:43:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.894+0000 7f05a19fd740 1 -- 172.21.15.23:0/4068069834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a81568f0 msgr2=0x5601a81543c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.894+0000 7f05a19fd740 1 --2- 172.21.15.23:0/4068069834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a81568f0 0x5601a81543c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0590036000 tx=0x7f05900095c0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.894+0000 7f05a19fd740 1 -- 172.21.15.23:0/4068069834 shutdown_connections 2024-03-24T10:43:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.894+0000 7f05a19fd740 1 --2- 172.21.15.23:0/4068069834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a81568f0 0x5601a81543c0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.894+0000 7f05a19fd740 1 --2- 172.21.15.23:0/4068069834 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5601a8166a00 0x5601a8168e20 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.894+0000 7f05a19fd740 1 --2- 172.21.15.23:0/4068069834 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a814dd10 0x5601a8153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.894+0000 7f05a19fd740 1 --2- 172.21.15.23:0/4068069834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a8153940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.894+0000 7f0577fff640 0 client.16689 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.894+0000 7f05a19fd740 1 -- 172.21.15.23:0/4068069834 >> 172.21.15.23:0/4068069834 conn(0x5601a8147b20 msgr2=0x5601a8149870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.894+0000 7f05a19fd740 1 -- 172.21.15.23:0/4068069834 shutdown_connections 2024-03-24T10:43:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.894+0000 7f05a19fd740 1 -- 172.21.15.23:0/4068069834 wait complete. 2024-03-24T10:43:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.894+0000 7f05a19fd740 20 client.16689 trim_cache size 0 max 16384 2024-03-24T10:43:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.908+0000 7f05a19fd740 1 Processor -- start 2024-03-24T10:43:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.908+0000 7f05a19fd740 1 -- start start 2024-03-24T10:43:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.908+0000 7f05a19fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a8156070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.908+0000 7f05a19fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a81541e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.908+0000 7f05a19fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 0x5601a816e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.908+0000 7f05a19fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5601a8166700 con 0x5601a814dd10 2024-03-24T10:43:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.908+0000 7f05a19fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5601a814cd20 con 0x5601a814d3f0 2024-03-24T10:43:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.908+0000 7f05a19fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5601a8163490 con 0x5601a81568f0 2024-03-24T10:43:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.909+0000 7f0597fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a81541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.909+0000 7f0597fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a81541e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45348/0 (socket says 172.21.15.23:45348) 2024-03-24T10:43:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.909+0000 7f0597fff640 1 -- 172.21.15.23:0/2834263509 learned_addr learned my addr 172.21.15.23:0/2834263509 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.909+0000 7f059cf1d640 1 --2- 172.21.15.23:0/2834263509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a8156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.909+0000 7f059d71e640 1 --2- 172.21.15.23:0/2834263509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 0x5601a816e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.909+0000 7f0597fff640 1 -- 172.21.15.23:0/2834263509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 msgr2=0x5601a816e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.909+0000 7f0597fff640 1 --2- 172.21.15.23:0/2834263509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 0x5601a816e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.909+0000 7f0597fff640 1 -- 172.21.15.23:0/2834263509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 msgr2=0x5601a8156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.909+0000 7f0597fff640 1 --2- 172.21.15.23:0/2834263509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a8156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.909+0000 7f0597fff640 1 -- 172.21.15.23:0/2834263509 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601a804ec80 con 0x5601a814dd10 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.909+0000 7f0597fff640 1 --2- 172.21.15.23:0/2834263509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a81541e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f058400c6a0 tx=0x7f058406cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef863b14ed0c9ae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.909+0000 7f05827fc640 1 -- 172.21.15.23:0/2834263509 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058409ef60 con 0x5601a814dd10 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.909+0000 7f05827fc640 1 -- 172.21.15.23:0/2834263509 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f058409d070 con 0x5601a814dd10 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.909+0000 7f05827fc640 1 -- 172.21.15.23:0/2834263509 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0584097530 con 0x5601a814dd10 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.910+0000 7f05a19fd740 1 -- 172.21.15.23:0/2834263509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 msgr2=0x5601a81541e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.910+0000 7f05a19fd740 1 --2- 172.21.15.23:0/2834263509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a81541e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f058400c6a0 tx=0x7f058406cf40 comp rx=0 tx=0).stop 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.910+0000 7f05a19fd740 1 -- 172.21.15.23:0/2834263509 shutdown_connections 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.910+0000 7f05a19fd740 1 --2- 172.21.15.23:0/2834263509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 0x5601a816e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.910+0000 7f05a19fd740 1 --2- 172.21.15.23:0/2834263509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a81541e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.910+0000 7f05a19fd740 1 --2- 172.21.15.23:0/2834263509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a8156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.910+0000 7f05a19fd740 1 -- 172.21.15.23:0/2834263509 >> 172.21.15.23:0/2834263509 conn(0x5601a8147b20 msgr2=0x5601a8148e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.910+0000 7f05a19fd740 1 -- 172.21.15.23:0/2834263509 shutdown_connections 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.910+0000 7f05a19fd740 1 -- 172.21.15.23:0/2834263509 wait complete. 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.910+0000 7f05a19fd740 1 -- start start 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.910+0000 7f05a19fd740 1 Processor -- start 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f05a19fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a815e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f05a19fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a815b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f05a19fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 0x5601a815bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f05a19fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5601a815ecf0 con 0x5601a814dd10 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f05a19fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5601a815ee60 con 0x5601a814d3f0 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f05a19fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5601a815f020 con 0x5601a81568f0 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f05a19fd740 10 client.0 _ll_register_callbacks cb 0x7f058c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f0597fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a815b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f0597fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a815b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45364/0 (socket says 172.21.15.23:45364) 2024-03-24T10:43:59.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f0597fff640 1 -- 172.21.15.23:0/1270123608 learned_addr learned my addr 172.21.15.23:0/1270123608 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f059cf1d640 1 --2- 172.21.15.23:0/1270123608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a815e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f059d71e640 1 --2- 172.21.15.23:0/1270123608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 0x5601a815bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f0597fff640 1 -- 172.21.15.23:0/1270123608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 msgr2=0x5601a815bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f0597fff640 1 --2- 172.21.15.23:0/1270123608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 0x5601a815bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f0597fff640 1 -- 172.21.15.23:0/1270123608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 msgr2=0x5601a815e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f0597fff640 1 --2- 172.21.15.23:0/1270123608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a815e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f0597fff640 1 -- 172.21.15.23:0/1270123608 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05981a4940 con 0x5601a814dd10 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f0597fff640 1 --2- 172.21.15.23:0/1270123608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a815b780 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0584097890 tx=0x7f0584010f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec43ee4c8e97cc19 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f0577fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f0577fff640 1 -- 172.21.15.23:0/1270123608 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058409ef60 con 0x5601a814dd10 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f0577fff640 1 -- 172.21.15.23:0/1270123608 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0584097530 con 0x5601a814dd10 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f05a19fd740 1 -- 172.21.15.23:0/1270123608 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f058c009e90 con 0x5601a814dd10 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f0577fff640 1 -- 172.21.15.23:0/1270123608 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f05840976f0 con 0x5601a814dd10 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f05a19fd740 10 client.16695 fetch_fsmap learned FSMap version 26 2024-03-24T10:43:59.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.911+0000 7f05a19fd740 1 -- 172.21.15.23:0/1270123608 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5601a8074920 con 0x5601a814dd10 2024-03-24T10:43:59.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.912+0000 7f0577fff640 1 -- 172.21.15.23:0/1270123608 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f058c009e90 con 0x5601a814dd10 2024-03-24T10:43:59.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.912+0000 7f0577fff640 1 -- 172.21.15.23:0/1270123608 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f0584099d80 con 0x5601a814dd10 2024-03-24T10:43:59.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.912+0000 7f05a19fd740 10 client.16695 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:43:59.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.912+0000 7f05a19fd740 10 client.16695 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:43:59.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.912+0000 7f05a19fd740 20 client.16695 populate_metadata read hostname 'smithi023' 2024-03-24T10:43:59.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.912+0000 7f05a19fd740 1 --2- 172.21.15.23:0/1270123608 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5601a81cdc50 0x5601a81d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:43:59.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.912+0000 7f05a19fd740 4 client.16695 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:43:59.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.912+0000 7f05a19fd740 1 -- 172.21.15.23:0/1270123608 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5601a7e3f890 con 0x5601a81cdc50 2024-03-24T10:43:59.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.912+0000 7f059cf1d640 1 --2- 172.21.15.23:0/1270123608 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5601a81cdc50 0x5601a81d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:43:59.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.912+0000 7f0577fff640 10 client.16695 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:59.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.912+0000 7f059cf1d640 1 --2- 172.21.15.23:0/1270123608 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5601a81cdc50 0x5601a81d0070 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46f065f085a8d862 server_cookie=6f7d41c152dcc10a in_seq=0 out_seq=0 2024-03-24T10:43:59.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.912+0000 7f0577fff640 1 -- 172.21.15.23:0/1270123608 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5601a7e3f890 con 0x5601a81cdc50 2024-03-24T10:43:59.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.912+0000 7f0577fff640 10 client.16695 handle_command_reply: tid=0 2024-03-24T10:43:59.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.912+0000 7f05a19fd740 1 client.16695 shutdown 2024-03-24T10:43:59.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f05a19fd740 1 -- 172.21.15.23:0/1270123608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 msgr2=0x5601a815b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f05a19fd740 1 --2- 172.21.15.23:0/1270123608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a815b780 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0584097890 tx=0x7f0584010f50 comp rx=0 tx=0).stop 2024-03-24T10:43:59.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f05a19fd740 1 -- 172.21.15.23:0/1270123608 shutdown_connections 2024-03-24T10:43:59.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f05a19fd740 1 --2- 172.21.15.23:0/1270123608 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5601a81cdc50 0x5601a81d0070 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f05a19fd740 1 --2- 172.21.15.23:0/1270123608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601a81568f0 0x5601a815bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f05a19fd740 1 --2- 172.21.15.23:0/1270123608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5601a814dd10 0x5601a815b780 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f0577fff640 0 client.16695 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:43:59.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f05a19fd740 1 --2- 172.21.15.23:0/1270123608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601a814d3f0 0x5601a815e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f05a19fd740 1 -- 172.21.15.23:0/1270123608 >> 172.21.15.23:0/1270123608 conn(0x5601a8147b20 msgr2=0x5601a8151de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f05a19fd740 1 -- 172.21.15.23:0/1270123608 shutdown_connections 2024-03-24T10:43:59.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f05a19fd740 1 -- 172.21.15.23:0/1270123608 wait complete. 2024-03-24T10:43:59.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f05a19fd740 20 client.16695 trim_cache size 0 max 16384 2024-03-24T10:43:59.919 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:43:59.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f059f181640 1 -- 172.21.15.23:0/1890867226 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f05880a2230 msgr2=0x7f05880a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f059f181640 1 --2- 172.21.15.23:0/1890867226 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f05880a2230 0x7f05880a46f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0590005cc0 tx=0x7f0590005c50 comp rx=0 tx=0).stop 2024-03-24T10:43:59.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f059f181640 1 -- 172.21.15.23:0/1890867226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0598100f80 msgr2=0x7f0598199010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:43:59.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f059f181640 1 --2- 172.21.15.23:0/1890867226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0598100f80 0x7f0598199010 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f058400e9f0 tx=0x7f058400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f059f181640 1 -- 172.21.15.23:0/1890867226 shutdown_connections 2024-03-24T10:43:59.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f059f181640 1 --2- 172.21.15.23:0/1890867226 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f05880a2230 0x7f05880a46f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f059f181640 1 --2- 172.21.15.23:0/1890867226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0598106660 0x7f05981a3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f059f181640 1 --2- 172.21.15.23:0/1890867226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0598100f80 0x7f0598199010 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f059f181640 1 --2- 172.21.15.23:0/1890867226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05981005c0 0x7f0598198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:43:59.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.913+0000 7f059f181640 1 -- 172.21.15.23:0/1890867226 >> 172.21.15.23:0/1890867226 conn(0x7f0598076990 msgr2=0x7f05980fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:43:59.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.914+0000 7f059f181640 1 -- 172.21.15.23:0/1890867226 shutdown_connections 2024-03-24T10:43:59.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:43:59.914+0000 7f059f181640 1 -- 172.21.15.23:0/1890867226 wait complete. 2024-03-24T10:44:00.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:43:59 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/458164212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:44:00.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:43:59 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/458164212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:44:00.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 312496 12.37 MB/sec execute 17 sec latency 22.867 ms 2024-03-24T10:44:00.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:00 smithi023 ceph-mon[32466]: pgmap v120: 97 pgs: 97 active+clean; 172 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 190 op/s 2024-03-24T10:44:00.924 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:44:00.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:44:00.925 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:44:01.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:00 smithi117 ceph-mon[36368]: pgmap v120: 97 pgs: 97 active+clean; 172 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 190 op/s 2024-03-24T10:44:01.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:00 smithi116 ceph-mon[36248]: pgmap v120: 97 pgs: 97 active+clean; 172 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 190 op/s 2024-03-24T10:44:01.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.033+0000 7fce55a8d640 1 -- 172.21.15.23:0/3950217319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce501005a0 msgr2=0x7fce50100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.033+0000 7fce55a8d640 1 --2- 172.21.15.23:0/3950217319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce501005a0 0x7fce50100a20 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fce44009900 tx=0x7fce4402bc30 comp rx=0 tx=0).stop 2024-03-24T10:44:01.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.033+0000 7fce55a8d640 1 -- 172.21.15.23:0/3950217319 shutdown_connections 2024-03-24T10:44:01.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.033+0000 7fce55a8d640 1 --2- 172.21.15.23:0/3950217319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce50100f60 0x7fce5010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.033+0000 7fce55a8d640 1 --2- 172.21.15.23:0/3950217319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce501005a0 0x7fce50100a20 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.033+0000 7fce55a8d640 1 --2- 172.21.15.23:0/3950217319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce50106660 0x7fce50106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.033+0000 7fce55a8d640 1 -- 172.21.15.23:0/3950217319 >> 172.21.15.23:0/3950217319 conn(0x7fce50077710 msgr2=0x7fce50077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:01.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.033+0000 7fce55a8d640 1 -- 172.21.15.23:0/3950217319 shutdown_connections 2024-03-24T10:44:01.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.033+0000 7fce55a8d640 1 -- 172.21.15.23:0/3950217319 wait complete. 2024-03-24T10:44:01.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.033+0000 7fce55a8d640 1 Processor -- start 2024-03-24T10:44:01.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.033+0000 7fce55a8d640 1 -- start start 2024-03-24T10:44:01.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce55a8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce501005a0 0x7fce5019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce55a8d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce50100f60 0x7fce5019ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce55a8d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce50106660 0x7fce50196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce55a8d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fce501970d0 con 0x7fce50106660 2024-03-24T10:44:01.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce55a8d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fce50197240 con 0x7fce501005a0 2024-03-24T10:44:01.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce55a8d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fce50197400 con 0x7fce50100f60 2024-03-24T10:44:01.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce50106660 0x7fce50196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce50106660 0x7fce50196a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45374/0 (socket says 172.21.15.23:45374) 2024-03-24T10:44:01.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4ffff640 1 -- 172.21.15.23:0/4286659726 learned_addr learned my addr 172.21.15.23:0/4286659726 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:01.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4f7fe640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce501005a0 0x7fce5019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4effd640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce50100f60 0x7fce5019ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4ffff640 1 -- 172.21.15.23:0/4286659726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce50100f60 msgr2=0x7fce5019ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4ffff640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce50100f60 0x7fce5019ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4ffff640 1 -- 172.21.15.23:0/4286659726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce501005a0 msgr2=0x7fce5019c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4ffff640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce501005a0 0x7fce5019c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4ffff640 1 -- 172.21.15.23:0/4286659726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce44009560 con 0x7fce50106660 2024-03-24T10:44:01.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4effd640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce50100f60 0x7fce5019ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:44:01.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4ffff640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce50106660 0x7fce50196a80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fce4800eb10 tx=0x7fce4800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:01.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4cff9640 1 -- 172.21.15.23:0/4286659726 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48004b20 con 0x7fce50106660 2024-03-24T10:44:01.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4cff9640 1 -- 172.21.15.23:0/4286659726 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce48004c80 con 0x7fce50106660 2024-03-24T10:44:01.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce55a8d640 1 -- 172.21.15.23:0/4286659726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce501976e0 con 0x7fce50106660 2024-03-24T10:44:01.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce4cff9640 1 -- 172.21.15.23:0/4286659726 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48010650 con 0x7fce50106660 2024-03-24T10:44:01.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.034+0000 7fce55a8d640 1 -- 172.21.15.23:0/4286659726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce501a3970 con 0x7fce50106660 2024-03-24T10:44:01.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce4cff9640 1 -- 172.21.15.23:0/4286659726 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fce48005020 con 0x7fce50106660 2024-03-24T10:44:01.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce5866a740 1 Processor -- start 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce5866a740 1 -- start start 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce5866a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756410 0x55bd397567f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce5866a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756d30 0x55bd39761930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce5866a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd39764260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce5866a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bd396c4530 con 0x55bd39756d30 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce5866a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bd396c1a60 con 0x55bd39756410 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce5866a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bd396bd240 con 0x55bd39761e70 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce4cff9640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fce380a2230 0x7fce380a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce4f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756d30 0x55bd39761930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce4f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756d30 0x55bd39761930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45390/0 (socket says 172.21.15.23:45390) 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce4f7fe640 1 -- 172.21.15.23:0/2762312461 learned_addr learned my addr 172.21.15.23:0/2762312461 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce4effd640 1 --2- 172.21.15.23:0/2762312461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756410 0x55bd397567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce4effd640 1 --2- 172.21.15.23:0/2762312461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd39764260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.037+0000 7fce4ffff640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fce380a2230 0x7fce380a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.038+0000 7fce4cff9640 1 -- 172.21.15.23:0/4286659726 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fce48014070 con 0x7fce50106660 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.038+0000 7fce4f7fe640 1 -- 172.21.15.23:0/2762312461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 msgr2=0x55bd39764260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.038+0000 7fce4f7fe640 1 --2- 172.21.15.23:0/2762312461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd39764260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.038+0000 7fce4f7fe640 1 -- 172.21.15.23:0/2762312461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756410 msgr2=0x55bd397567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.038+0000 7fce4f7fe640 1 --2- 172.21.15.23:0/2762312461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756410 0x55bd397567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.038+0000 7fce4f7fe640 1 -- 172.21.15.23:0/2762312461 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd396bb8c0 con 0x55bd39756d30 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.038+0000 7fce4effd640 1 --2- 172.21.15.23:0/2762312461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd39764260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.038+0000 7fce4f7fe640 1 --2- 172.21.15.23:0/2762312461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756d30 0x55bd39761930 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fce40009c20 tx=0x7fce40011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc7e91cf413e997 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.038+0000 7fce4ffff640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fce380a2230 0x7fce380a46f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fce480cc450 tx=0x7fce4806d0b0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.038+0000 7fce35ffb640 1 -- 172.21.15.23:0/2762312461 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce40004230 con 0x55bd39756d30 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.038+0000 7fce35ffb640 1 -- 172.21.15.23:0/2762312461 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce40009e90 con 0x55bd39756d30 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.038+0000 7fce35ffb640 1 -- 172.21.15.23:0/2762312461 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce40004d40 con 0x55bd39756d30 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.039+0000 7fce5866a740 1 -- 172.21.15.23:0/2762312461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756d30 msgr2=0x55bd39761930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.039+0000 7fce5866a740 1 --2- 172.21.15.23:0/2762312461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756d30 0x55bd39761930 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fce40009c20 tx=0x7fce40011810 comp rx=0 tx=0).stop 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.039+0000 7fce5866a740 1 -- 172.21.15.23:0/2762312461 shutdown_connections 2024-03-24T10:44:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.039+0000 7fce5866a740 1 --2- 172.21.15.23:0/2762312461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd39764260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.039+0000 7fce5866a740 1 --2- 172.21.15.23:0/2762312461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756d30 0x55bd39761930 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.039+0000 7fce5866a740 1 --2- 172.21.15.23:0/2762312461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756410 0x55bd397567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.039+0000 7fce5866a740 1 -- 172.21.15.23:0/2762312461 >> 172.21.15.23:0/2762312461 conn(0x55bd39750e90 msgr2=0x55bd39751270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.039+0000 7fce5866a740 1 -- 172.21.15.23:0/2762312461 shutdown_connections 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.039+0000 7fce5866a740 1 -- 172.21.15.23:0/2762312461 wait complete. 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.039+0000 7fce5866a740 1 -- start start 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.039+0000 7fce5866a740 1 Processor -- start 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce5866a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756410 0x55bd3975ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce5866a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756d30 0x55bd3975f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce5866a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd3975f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce5866a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bd39661f80 con 0x55bd39756410 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce5866a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bd393904b0 con 0x55bd39756d30 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce5866a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bd39642ad0 con 0x55bd39761e70 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce5866a740 10 client.0 _ll_register_callbacks cb 0x55bd39675b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce4effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756410 0x55bd3975ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce4effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756410 0x55bd3975ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45394/0 (socket says 172.21.15.23:45394) 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce4effd640 1 -- 172.21.15.23:0/3786400913 learned_addr learned my addr 172.21.15.23:0/3786400913 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce4f7fe640 1 --2- 172.21.15.23:0/3786400913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756d30 0x55bd3975f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce4effd640 1 --2- 172.21.15.23:0/3786400913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd3975f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce4effd640 1 -- 172.21.15.23:0/3786400913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 msgr2=0x55bd3975f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce4effd640 1 --2- 172.21.15.23:0/3786400913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd3975f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce4effd640 1 -- 172.21.15.23:0/3786400913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756d30 msgr2=0x55bd3975f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce4effd640 1 --2- 172.21.15.23:0/3786400913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756d30 0x55bd3975f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce4effd640 1 -- 172.21.15.23:0/3786400913 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce40009850 con 0x55bd39756410 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce4effd640 1 --2- 172.21.15.23:0/3786400913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756410 0x55bd3975ee10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fce4402c170 tx=0x7fce4402c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78edc07f1e3ca7a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce2effd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce2effd640 1 -- 172.21.15.23:0/3786400913 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44004250 con 0x55bd39756410 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce2effd640 1 -- 172.21.15.23:0/3786400913 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fce440043b0 con 0x55bd39756410 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce5866a740 1 -- 172.21.15.23:0/3786400913 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bd396e17a0 con 0x55bd39756410 2024-03-24T10:44:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce2effd640 1 -- 172.21.15.23:0/3786400913 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fce44004df0 con 0x55bd39756410 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce5866a740 10 client.16707 fetch_fsmap learned FSMap version 26 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.040+0000 7fce5866a740 1 -- 172.21.15.23:0/3786400913 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bd396736d0 con 0x55bd39756410 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.041+0000 7fce2effd640 1 -- 172.21.15.23:0/3786400913 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bd396e17a0 con 0x55bd39756410 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.041+0000 7fce2effd640 1 -- 172.21.15.23:0/3786400913 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7fce44003990 con 0x55bd39756410 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.041+0000 7fce5866a740 10 client.16707 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.041+0000 7fce5866a740 10 client.16707 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.041+0000 7fce5866a740 20 client.16707 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.041+0000 7fce5866a740 1 --2- 172.21.15.23:0/3786400913 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd39771ec0 0x55bd397742e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.041+0000 7fce5866a740 4 client.16707 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.041+0000 7fce5866a740 1 -- 172.21.15.23:0/3786400913 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd396b31f0 con 0x55bd39771ec0 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.041+0000 7fce4f7fe640 1 --2- 172.21.15.23:0/3786400913 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd39771ec0 0x55bd397742e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.041+0000 7fce2effd640 10 client.16707 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.041+0000 7fce4f7fe640 1 --2- 172.21.15.23:0/3786400913 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd39771ec0 0x55bd397742e0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78fbbe9ecb523b4e server_cookie=80b65c1485d76aff in_seq=0 out_seq=0 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.042+0000 7fce2effd640 1 -- 172.21.15.23:0/3786400913 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bd396b31f0 con 0x55bd39771ec0 2024-03-24T10:44:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.042+0000 7fce2effd640 10 client.16707 handle_command_reply: tid=0 2024-03-24T10:44:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.042+0000 7fce5866a740 1 client.16707 shutdown 2024-03-24T10:44:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.042+0000 7fce5866a740 1 -- 172.21.15.23:0/3786400913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756410 msgr2=0x55bd3975ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.042+0000 7fce5866a740 1 --2- 172.21.15.23:0/3786400913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756410 0x55bd3975ee10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fce4402c170 tx=0x7fce4402c810 comp rx=0 tx=0).stop 2024-03-24T10:44:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.042+0000 7fce5866a740 1 -- 172.21.15.23:0/3786400913 shutdown_connections 2024-03-24T10:44:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.042+0000 7fce5866a740 1 --2- 172.21.15.23:0/3786400913 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd39771ec0 0x55bd397742e0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.042+0000 7fce5866a740 1 --2- 172.21.15.23:0/3786400913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd3975f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.042+0000 7fce5866a740 1 --2- 172.21.15.23:0/3786400913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756d30 0x55bd3975f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.042+0000 7fce5866a740 1 --2- 172.21.15.23:0/3786400913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756410 0x55bd3975ee10 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.042+0000 7fce2effd640 0 client.16707 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.042+0000 7fce5866a740 1 -- 172.21.15.23:0/3786400913 >> 172.21.15.23:0/3786400913 conn(0x55bd39750e90 msgr2=0x55bd39752be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.042+0000 7fce5866a740 1 -- 172.21.15.23:0/3786400913 shutdown_connections 2024-03-24T10:44:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.042+0000 7fce5866a740 1 -- 172.21.15.23:0/3786400913 wait complete. 2024-03-24T10:44:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.043+0000 7fce5866a740 20 client.16707 trim_cache size 0 max 16384 2024-03-24T10:44:01.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.060+0000 7fce5866a740 1 Processor -- start 2024-03-24T10:44:01.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.060+0000 7fce5866a740 1 -- start start 2024-03-24T10:44:01.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.060+0000 7fce5866a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756410 0x55bd39761540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.060+0000 7fce5866a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756d30 0x55bd3976d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.060+0000 7fce5866a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd39779bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.060+0000 7fce5866a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bd3974f350 con 0x55bd39756410 2024-03-24T10:44:01.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.060+0000 7fce5866a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bd39751c70 con 0x55bd39756d30 2024-03-24T10:44:01.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.060+0000 7fce5866a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bd3974c620 con 0x55bd39761e70 2024-03-24T10:44:01.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.060+0000 7fce4effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756410 0x55bd39761540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.060+0000 7fce4effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756410 0x55bd39761540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45404/0 (socket says 172.21.15.23:45404) 2024-03-24T10:44:01.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.060+0000 7fce4f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756d30 0x55bd3976d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.060+0000 7fce4effd640 1 -- 172.21.15.23:0/1202842128 learned_addr learned my addr 172.21.15.23:0/1202842128 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:01.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.060+0000 7fce4effd640 1 --2- 172.21.15.23:0/1202842128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd39779bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.061+0000 7fce4effd640 1 -- 172.21.15.23:0/1202842128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 msgr2=0x55bd39779bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.061+0000 7fce4effd640 1 --2- 172.21.15.23:0/1202842128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd39779bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.061+0000 7fce4effd640 1 -- 172.21.15.23:0/1202842128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756d30 msgr2=0x55bd3976d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.061+0000 7fce4effd640 1 --2- 172.21.15.23:0/1202842128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756d30 0x55bd3976d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.061+0000 7fce4effd640 1 -- 172.21.15.23:0/1202842128 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd39657c60 con 0x55bd39756410 2024-03-24T10:44:01.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.061+0000 7fce4f7fe640 1 --2- 172.21.15.23:0/1202842128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756d30 0x55bd3976d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:01.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.061+0000 7fce4effd640 1 --2- 172.21.15.23:0/1202842128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756410 0x55bd39761540 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fce44005c40 tx=0x7fce4402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81453cc3cc1deb27 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:01.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.061+0000 7fce357fa640 1 -- 172.21.15.23:0/1202842128 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44004250 con 0x55bd39756410 2024-03-24T10:44:01.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.061+0000 7fce357fa640 1 -- 172.21.15.23:0/1202842128 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce44050590 con 0x55bd39756410 2024-03-24T10:44:01.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.061+0000 7fce357fa640 1 -- 172.21.15.23:0/1202842128 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce440507e0 con 0x55bd39756410 2024-03-24T10:44:01.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.062+0000 7fce5866a740 1 -- 172.21.15.23:0/1202842128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756410 msgr2=0x55bd39761540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.062+0000 7fce5866a740 1 --2- 172.21.15.23:0/1202842128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756410 0x55bd39761540 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fce44005c40 tx=0x7fce4402bc30 comp rx=0 tx=0).stop 2024-03-24T10:44:01.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.062+0000 7fce5866a740 1 -- 172.21.15.23:0/1202842128 shutdown_connections 2024-03-24T10:44:01.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.062+0000 7fce5866a740 1 --2- 172.21.15.23:0/1202842128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd39779bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.062+0000 7fce5866a740 1 --2- 172.21.15.23:0/1202842128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756d30 0x55bd3976d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.062+0000 7fce5866a740 1 --2- 172.21.15.23:0/1202842128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756410 0x55bd39761540 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.062+0000 7fce5866a740 1 -- 172.21.15.23:0/1202842128 >> 172.21.15.23:0/1202842128 conn(0x55bd39750e90 msgr2=0x55bd3974fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.062+0000 7fce5866a740 1 -- 172.21.15.23:0/1202842128 shutdown_connections 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.062+0000 7fce5866a740 1 -- 172.21.15.23:0/1202842128 wait complete. 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.062+0000 7fce5866a740 1 -- start start 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.062+0000 7fce5866a740 1 Processor -- start 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce5866a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756410 0x55bd39766770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce5866a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756d30 0x55bd39766cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce5866a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd397671f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce5866a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bd397d8230 con 0x55bd39756d30 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce5866a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bd397d8370 con 0x55bd39756410 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce5866a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bd397d84e0 con 0x55bd39761e70 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce5866a740 10 client.0 _ll_register_callbacks cb 0x7fce44004b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce4f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756d30 0x55bd39766cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce4effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756410 0x55bd39766770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce4f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756d30 0x55bd39766cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45412/0 (socket says 172.21.15.23:45412) 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce4f7fe640 1 -- 172.21.15.23:0/2932476664 learned_addr learned my addr 172.21.15.23:0/2932476664 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce4effd640 1 --2- 172.21.15.23:0/2932476664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd397671f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce4f7fe640 1 -- 172.21.15.23:0/2932476664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 msgr2=0x55bd397671f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce4f7fe640 1 --2- 172.21.15.23:0/2932476664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd397671f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce4f7fe640 1 -- 172.21.15.23:0/2932476664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756410 msgr2=0x55bd39766770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce4f7fe640 1 --2- 172.21.15.23:0/2932476664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756410 0x55bd39766770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce4f7fe640 1 -- 172.21.15.23:0/2932476664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce4001d040 con 0x55bd39756d30 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce4effd640 1 --2- 172.21.15.23:0/2932476664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd397671f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce4f7fe640 1 --2- 172.21.15.23:0/2932476664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756d30 0x55bd39766cb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fce400048f0 tx=0x7fce40011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1478d9bbc283412 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce2effd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.063+0000 7fce2effd640 1 -- 172.21.15.23:0/2932476664 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce40019680 con 0x55bd39756d30 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.064+0000 7fce2effd640 1 -- 172.21.15.23:0/2932476664 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fce400197e0 con 0x55bd39756d30 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.064+0000 7fce5866a740 1 -- 172.21.15.23:0/2932476664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce40009e90 con 0x55bd39756d30 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.064+0000 7fce2effd640 1 -- 172.21.15.23:0/2932476664 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fce400199a0 con 0x55bd39756d30 2024-03-24T10:44:01.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.064+0000 7fce5866a740 10 client.16713 fetch_fsmap learned FSMap version 26 2024-03-24T10:44:01.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.064+0000 7fce5866a740 1 -- 172.21.15.23:0/2932476664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bd396736d0 con 0x55bd39756d30 2024-03-24T10:44:01.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.064+0000 7fce2effd640 1 -- 172.21.15.23:0/2932476664 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce4003b080 con 0x55bd39756d30 2024-03-24T10:44:01.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.064+0000 7fce2effd640 1 -- 172.21.15.23:0/2932476664 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7fce40035030 con 0x55bd39756d30 2024-03-24T10:44:01.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.064+0000 7fce5866a740 10 client.16713 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:44:01.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.064+0000 7fce5866a740 10 client.16713 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:01.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.064+0000 7fce5866a740 20 client.16713 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:01.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.064+0000 7fce5866a740 1 --2- 172.21.15.23:0/2932476664 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd397d9dd0 0x55bd397dc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.064+0000 7fce5866a740 4 client.16713 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:44:01.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.064+0000 7fce5866a740 1 -- 172.21.15.23:0/2932476664 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bd393bc860 con 0x55bd397d9dd0 2024-03-24T10:44:01.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.064+0000 7fce4effd640 1 --2- 172.21.15.23:0/2932476664 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd397d9dd0 0x55bd397dc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.065+0000 7fce2effd640 10 client.16713 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:01.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.065+0000 7fce4effd640 1 --2- 172.21.15.23:0/2932476664 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd397d9dd0 0x55bd397dc1f0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dfc8f16883a9279 server_cookie=16dda7f1090337c5 in_seq=0 out_seq=0 2024-03-24T10:44:01.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce2effd640 1 -- 172.21.15.23:0/2932476664 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bd393bc860 con 0x55bd397d9dd0 2024-03-24T10:44:01.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce2effd640 10 client.16713 handle_command_reply: tid=0 2024-03-24T10:44:01.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce5866a740 1 client.16713 shutdown 2024-03-24T10:44:01.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce5866a740 1 -- 172.21.15.23:0/2932476664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756d30 msgr2=0x55bd39766cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce5866a740 1 --2- 172.21.15.23:0/2932476664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756d30 0x55bd39766cb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fce400048f0 tx=0x7fce40011e50 comp rx=0 tx=0).stop 2024-03-24T10:44:01.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce5866a740 1 -- 172.21.15.23:0/2932476664 shutdown_connections 2024-03-24T10:44:01.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce5866a740 1 --2- 172.21.15.23:0/2932476664 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bd397d9dd0 0x55bd397dc1f0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce5866a740 1 --2- 172.21.15.23:0/2932476664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd39761e70 0x55bd397671f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce5866a740 1 --2- 172.21.15.23:0/2932476664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bd39756d30 0x55bd39766cb0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce5866a740 1 --2- 172.21.15.23:0/2932476664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bd39756410 0x55bd39766770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce2effd640 0 client.16713 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:01.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce5866a740 1 -- 172.21.15.23:0/2932476664 >> 172.21.15.23:0/2932476664 conn(0x55bd39750e90 msgr2=0x55bd397522e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:01.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce5866a740 1 -- 172.21.15.23:0/2932476664 shutdown_connections 2024-03-24T10:44:01.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce5866a740 1 -- 172.21.15.23:0/2932476664 wait complete. 2024-03-24T10:44:01.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.066+0000 7fce5866a740 20 client.16713 trim_cache size 0 max 16384 2024-03-24T10:44:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:44:01.072 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:44:01.072 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "1a1b79b5-9f21-410a-9912-58bc73307afc", 2024-03-24T10:44:01.072 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:44:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:44:01.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.067+0000 7fce55a8d640 1 -- 172.21.15.23:0/4286659726 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fce380a2230 msgr2=0x7fce380a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.067+0000 7fce55a8d640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fce380a2230 0x7fce380a46f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fce480cc450 tx=0x7fce4806d0b0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.067+0000 7fce55a8d640 1 -- 172.21.15.23:0/4286659726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce50106660 msgr2=0x7fce50196a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.067+0000 7fce55a8d640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce50106660 0x7fce50196a80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fce4800eb10 tx=0x7fce4800c2d0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.067+0000 7fce55a8d640 1 -- 172.21.15.23:0/4286659726 shutdown_connections 2024-03-24T10:44:01.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.067+0000 7fce55a8d640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fce380a2230 0x7fce380a46f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.067+0000 7fce55a8d640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce50106660 0x7fce50196a80 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.067+0000 7fce55a8d640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce50100f60 0x7fce5019ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.067+0000 7fce55a8d640 1 --2- 172.21.15.23:0/4286659726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce501005a0 0x7fce5019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.067+0000 7fce55a8d640 1 -- 172.21.15.23:0/4286659726 >> 172.21.15.23:0/4286659726 conn(0x7fce50077710 msgr2=0x7fce500ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:01.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.067+0000 7fce55a8d640 1 -- 172.21.15.23:0/4286659726 shutdown_connections 2024-03-24T10:44:01.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.067+0000 7fce55a8d640 1 -- 172.21.15.23:0/4286659726 wait complete. 2024-03-24T10:44:01.078 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:44:01.078 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:44:01.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 313608 11.97 MB/sec execute 18 sec latency 23.982 ms 2024-03-24T10:44:01.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.186+0000 7f05f7f02640 1 -- 172.21.15.23:0/1706181249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f0100500 msgr2=0x7f05f01008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.186+0000 7f05f7f02640 1 --2- 172.21.15.23:0/1706181249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f0100500 0x7f05f01008e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f05e8009900 tx=0x7f05e802bc30 comp rx=0 tx=0).stop 2024-03-24T10:44:01.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.186+0000 7f05f7f02640 1 -- 172.21.15.23:0/1706181249 shutdown_connections 2024-03-24T10:44:01.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.186+0000 7f05f7f02640 1 --2- 172.21.15.23:0/1706181249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05f010e0b0 0x7f05f01104a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.186+0000 7f05f7f02640 1 --2- 172.21.15.23:0/1706181249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05f0100eb0 0x7f05f010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.186+0000 7f05f7f02640 1 --2- 172.21.15.23:0/1706181249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f0100500 0x7f05f01008e0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.186+0000 7f05f7f02640 1 -- 172.21.15.23:0/1706181249 >> 172.21.15.23:0/1706181249 conn(0x7f05f0077770 msgr2=0x7f05f0077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:01.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.186+0000 7f05f7f02640 1 -- 172.21.15.23:0/1706181249 shutdown_connections 2024-03-24T10:44:01.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.186+0000 7f05f7f02640 1 -- 172.21.15.23:0/1706181249 wait complete. 2024-03-24T10:44:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f7f02640 1 Processor -- start 2024-03-24T10:44:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f7f02640 1 -- start start 2024-03-24T10:44:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f7f02640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05f0100500 0x7f05f010d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f7f02640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f0100eb0 0x7f05f010ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f7f02640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05f010e0b0 0x7f05f010bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f7f02640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f05f010c730 con 0x7f05f0100eb0 2024-03-24T10:44:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f7f02640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f05f010c8a0 con 0x7f05f0100500 2024-03-24T10:44:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f7f02640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f05f01aa920 con 0x7f05f010e0b0 2024-03-24T10:44:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f549d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f0100eb0 0x7f05f010ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f549d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f0100eb0 0x7f05f010ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45432/0 (socket says 172.21.15.23:45432) 2024-03-24T10:44:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f649f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05f010e0b0 0x7f05f010bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f549d640 1 -- 172.21.15.23:0/486409417 learned_addr learned my addr 172.21.15.23:0/486409417 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f5c9e640 1 --2- 172.21.15.23:0/486409417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05f0100500 0x7f05f010d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f649f640 1 -- 172.21.15.23:0/486409417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05f0100500 msgr2=0x7f05f010d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f649f640 1 --2- 172.21.15.23:0/486409417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05f0100500 0x7f05f010d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f649f640 1 -- 172.21.15.23:0/486409417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f0100eb0 msgr2=0x7f05f010ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f649f640 1 --2- 172.21.15.23:0/486409417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f0100eb0 0x7f05f010ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.187+0000 7f05f649f640 1 -- 172.21.15.23:0/486409417 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e8009560 con 0x7f05f010e0b0 2024-03-24T10:44:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.188+0000 7f05f649f640 1 --2- 172.21.15.23:0/486409417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05f010e0b0 0x7f05f010bfa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f05e0005fd0 tx=0x7f05e000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.188+0000 7f05eeffd640 1 -- 172.21.15.23:0/486409417 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e000cdf0 con 0x7f05f010e0b0 2024-03-24T10:44:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.188+0000 7f05f7f02640 1 -- 172.21.15.23:0/486409417 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05f01aac00 con 0x7f05f010e0b0 2024-03-24T10:44:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.188+0000 7f05eeffd640 1 -- 172.21.15.23:0/486409417 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05e0005020 con 0x7f05f010e0b0 2024-03-24T10:44:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.188+0000 7f05eeffd640 1 -- 172.21.15.23:0/486409417 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e0010690 con 0x7f05f010e0b0 2024-03-24T10:44:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.188+0000 7f05f7f02640 1 -- 172.21.15.23:0/486409417 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05f01ab160 con 0x7f05f010e0b0 2024-03-24T10:44:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.190+0000 7f05eeffd640 1 -- 172.21.15.23:0/486409417 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f05e0005190 con 0x7f05f010e0b0 2024-03-24T10:44:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.190+0000 7f05fabc4740 1 Processor -- start 2024-03-24T10:44:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05fabc4740 1 -- start start 2024-03-24T10:44:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05fabc4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bb1d0 0x561a5f4bb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05fabc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05fabc4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4c46d0 0x561a5f4c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05fabc4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x561a5f3dff30 con 0x561a5f4c46d0 2024-03-24T10:44:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05fabc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561a5f429810 con 0x561a5f4bbaf0 2024-03-24T10:44:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05fabc4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561a5f426d40 con 0x561a5f4bb1d0 2024-03-24T10:44:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05eeffd640 1 --2- 172.21.15.23:0/486409417 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f05d80a2230 0x7f05d80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05f549d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bb1d0 0x561a5f4bb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05f5c9e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05f549d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bb1d0 0x561a5f4bb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:34794/0 (socket says 172.21.15.23:34794) 2024-03-24T10:44:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05f549d640 1 -- 172.21.15.23:0/2650995061 learned_addr learned my addr 172.21.15.23:0/2650995061 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05f649f640 1 --2- 172.21.15.23:0/486409417 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f05d80a2230 0x7f05d80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05f549d640 1 --2- 172.21.15.23:0/2650995061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4c46d0 0x561a5f4c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05eeffd640 1 -- 172.21.15.23:0/486409417 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f05e0014070 con 0x7f05f010e0b0 2024-03-24T10:44:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05f5c9e640 1 -- 172.21.15.23:0/2650995061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bb1d0 msgr2=0x561a5f4bb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05f5c9e640 1 --2- 172.21.15.23:0/2650995061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bb1d0 0x561a5f4bb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05f5c9e640 1 -- 172.21.15.23:0/2650995061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4c46d0 msgr2=0x561a5f4c6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05f5c9e640 1 --2- 172.21.15.23:0/2650995061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4c46d0 0x561a5f4c6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05f5c9e640 1 -- 172.21.15.23:0/2650995061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a5f3e3ea0 con 0x561a5f4bbaf0 2024-03-24T10:44:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.191+0000 7f05f649f640 1 --2- 172.21.15.23:0/486409417 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f05d80a2230 0x7f05d80a46f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f05e00cbe40 tx=0x7f05e0005470 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.192+0000 7f05f5c9e640 1 --2- 172.21.15.23:0/2650995061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c4190 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f05e8000c00 tx=0x7f05e802c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7d486e38478842 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.192+0000 7f05d7fff640 1 -- 172.21.15.23:0/2650995061 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e8004740 con 0x561a5f4bbaf0 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.192+0000 7f05d7fff640 1 -- 172.21.15.23:0/2650995061 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05e80048a0 con 0x561a5f4bbaf0 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.192+0000 7f05d7fff640 1 -- 172.21.15.23:0/2650995061 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e8005050 con 0x561a5f4bbaf0 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.192+0000 7f05fabc4740 1 -- 172.21.15.23:0/2650995061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4bbaf0 msgr2=0x561a5f4c4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.192+0000 7f05fabc4740 1 --2- 172.21.15.23:0/2650995061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c4190 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f05e8000c00 tx=0x7f05e802c6a0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 -- 172.21.15.23:0/2650995061 shutdown_connections 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 --2- 172.21.15.23:0/2650995061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4c46d0 0x561a5f4c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 --2- 172.21.15.23:0/2650995061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c4190 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 --2- 172.21.15.23:0/2650995061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bb1d0 0x561a5f4bb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 -- 172.21.15.23:0/2650995061 >> 172.21.15.23:0/2650995061 conn(0x561a5f4b5900 msgr2=0x561a5f4b5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 -- 172.21.15.23:0/2650995061 shutdown_connections 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 -- 172.21.15.23:0/2650995061 wait complete. 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 -- start start 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 Processor -- start 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4c46d0 0x561a5f4c2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x561a5f422520 con 0x561a5f4bb1d0 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561a5f0f5430 con 0x561a5f4c46d0 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.193+0000 7f05fabc4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561a5f0b1e00 con 0x561a5f4bbaf0 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05fabc4740 10 client.0 _ll_register_callbacks cb 0x561a5f3c0d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05f549d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05f549d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45454/0 (socket says 172.21.15.23:45454) 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05f549d640 1 -- 172.21.15.23:0/1868884014 learned_addr learned my addr 172.21.15.23:0/1868884014 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05f5c9e640 1 --2- 172.21.15.23:0/1868884014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05f549d640 1 --2- 172.21.15.23:0/1868884014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4c46d0 0x561a5f4c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05f549d640 1 -- 172.21.15.23:0/1868884014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 msgr2=0x561a5f4c1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05f549d640 1 --2- 172.21.15.23:0/1868884014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05f549d640 1 -- 172.21.15.23:0/1868884014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4c46d0 msgr2=0x561a5f4c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05f549d640 1 --2- 172.21.15.23:0/1868884014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4c46d0 0x561a5f4c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05f549d640 1 -- 172.21.15.23:0/1868884014 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05f01110d0 con 0x561a5f4bb1d0 2024-03-24T10:44:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05f549d640 1 --2- 172.21.15.23:0/1868884014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c1700 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f05e4011b00 tx=0x7f05e4009c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e4d41bccb1b755 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05d57fa640 1 -- 172.21.15.23:0/1868884014 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e4024070 con 0x561a5f4bb1d0 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05d57fa640 1 -- 172.21.15.23:0/1868884014 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f05e400fc50 con 0x561a5f4bb1d0 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05fabc4740 1 -- 172.21.15.23:0/1868884014 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561a5f1217e0 con 0x561a5f4bb1d0 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05d57fa640 1 -- 172.21.15.23:0/1868884014 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f05e4016040 con 0x561a5f4bb1d0 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05fabc4740 10 client.16725 fetch_fsmap learned FSMap version 26 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.194+0000 7f05fabc4740 1 -- 172.21.15.23:0/1868884014 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561a5f420ba0 con 0x561a5f4bb1d0 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.195+0000 7f05d57fa640 1 -- 172.21.15.23:0/1868884014 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561a5f1217e0 con 0x561a5f4bb1d0 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.195+0000 7f05d57fa640 1 -- 172.21.15.23:0/1868884014 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f05e4027020 con 0x561a5f4bb1d0 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.195+0000 7f05fabc4740 10 client.16725 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.195+0000 7f05fabc4740 10 client.16725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.195+0000 7f05fabc4740 20 client.16725 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.195+0000 7f05fabc4740 1 --2- 172.21.15.23:0/1868884014 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561a5f4d4b10 0x561a5f4d6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.195+0000 7f05fabc4740 4 client.16725 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.195+0000 7f05fabc4740 1 -- 172.21.15.23:0/1868884014 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a5f15e720 con 0x561a5f4d4b10 2024-03-24T10:44:01.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.195+0000 7f05f5c9e640 1 --2- 172.21.15.23:0/1868884014 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561a5f4d4b10 0x561a5f4d6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.195+0000 7f05f5c9e640 1 --2- 172.21.15.23:0/1868884014 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561a5f4d4b10 0x561a5f4d6f50 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bce1c25e9fbb9b66 server_cookie=a7050df8dd9eebd7 in_seq=0 out_seq=0 2024-03-24T10:44:01.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.195+0000 7f05d57fa640 10 client.16725 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:01.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.196+0000 7f05d57fa640 1 -- 172.21.15.23:0/1868884014 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561a5f15e720 con 0x561a5f4d4b10 2024-03-24T10:44:01.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.196+0000 7f05d57fa640 10 client.16725 handle_command_reply: tid=0 2024-03-24T10:44:01.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.196+0000 7f05fabc4740 1 client.16725 shutdown 2024-03-24T10:44:01.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.197+0000 7f05fabc4740 1 -- 172.21.15.23:0/1868884014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4bb1d0 msgr2=0x561a5f4c1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.197+0000 7f05fabc4740 1 --2- 172.21.15.23:0/1868884014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c1700 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f05e4011b00 tx=0x7f05e4009c40 comp rx=0 tx=0).stop 2024-03-24T10:44:01.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.197+0000 7f05fabc4740 1 -- 172.21.15.23:0/1868884014 shutdown_connections 2024-03-24T10:44:01.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.197+0000 7f05fabc4740 1 --2- 172.21.15.23:0/1868884014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4c46d0 0x561a5f4c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.197+0000 7f05fabc4740 1 --2- 172.21.15.23:0/1868884014 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561a5f4d4b10 0x561a5f4d6f50 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.197+0000 7f05fabc4740 1 --2- 172.21.15.23:0/1868884014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.197+0000 7f05fabc4740 1 --2- 172.21.15.23:0/1868884014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c1700 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.197+0000 7f05fabc4740 1 -- 172.21.15.23:0/1868884014 >> 172.21.15.23:0/1868884014 conn(0x561a5f4b5900 msgr2=0x561a5f4b4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:01.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.197+0000 7f05d57fa640 0 client.16725 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:01.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.197+0000 7f05fabc4740 1 -- 172.21.15.23:0/1868884014 shutdown_connections 2024-03-24T10:44:01.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.197+0000 7f05fabc4740 1 -- 172.21.15.23:0/1868884014 wait complete. 2024-03-24T10:44:01.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.197+0000 7f05fabc4740 20 client.16725 trim_cache size 0 max 16384 2024-03-24T10:44:01.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.214+0000 7f05fabc4740 1 Processor -- start 2024-03-24T10:44:01.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.214+0000 7f05fabc4740 1 -- start start 2024-03-24T10:44:01.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.214+0000 7f05fabc4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05fabc4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05fabc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4c46d0 0x561a5f4cff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05fabc4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x561a5f3a7430 con 0x561a5f4bb1d0 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05fabc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561a5f4d15b0 con 0x561a5f4c46d0 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05fabc4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561a5f4d47e0 con 0x561a5f4bbaf0 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05f549d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05f5c9e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05f549d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45460/0 (socket says 172.21.15.23:45460) 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05f549d640 1 -- 172.21.15.23:0/2839201402 learned_addr learned my addr 172.21.15.23:0/2839201402 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05f549d640 1 --2- 172.21.15.23:0/2839201402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4c46d0 0x561a5f4cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05f549d640 1 -- 172.21.15.23:0/2839201402 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 msgr2=0x561a5f4c0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05f549d640 1 --2- 172.21.15.23:0/2839201402 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05f549d640 1 -- 172.21.15.23:0/2839201402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4c46d0 msgr2=0x561a5f4cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05f549d640 1 --2- 172.21.15.23:0/2839201402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4c46d0 0x561a5f4cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05f549d640 1 -- 172.21.15.23:0/2839201402 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a5f3bcb20 con 0x561a5f4bb1d0 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05f5c9e640 1 --2- 172.21.15.23:0/2839201402 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:01.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05f549d640 1 --2- 172.21.15.23:0/2839201402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c3e30 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f05e4012fd0 tx=0x7f05e4005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291fe21db3ed2282 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:01.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05d77fe640 1 -- 172.21.15.23:0/2839201402 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e4016070 con 0x561a5f4bb1d0 2024-03-24T10:44:01.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05d77fe640 1 -- 172.21.15.23:0/2839201402 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05e40055f0 con 0x561a5f4bb1d0 2024-03-24T10:44:01.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.215+0000 7f05d77fe640 1 -- 172.21.15.23:0/2839201402 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e4024040 con 0x561a5f4bb1d0 2024-03-24T10:44:01.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.216+0000 7f05fabc4740 1 -- 172.21.15.23:0/2839201402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4bb1d0 msgr2=0x561a5f4c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.216+0000 7f05fabc4740 1 --2- 172.21.15.23:0/2839201402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c3e30 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f05e4012fd0 tx=0x7f05e4005d30 comp rx=0 tx=0).stop 2024-03-24T10:44:01.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.216+0000 7f05fabc4740 1 -- 172.21.15.23:0/2839201402 shutdown_connections 2024-03-24T10:44:01.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.216+0000 7f05fabc4740 1 --2- 172.21.15.23:0/2839201402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4c46d0 0x561a5f4cff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.216+0000 7f05fabc4740 1 --2- 172.21.15.23:0/2839201402 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.216+0000 7f05fabc4740 1 --2- 172.21.15.23:0/2839201402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c3e30 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.216+0000 7f05fabc4740 1 -- 172.21.15.23:0/2839201402 >> 172.21.15.23:0/2839201402 conn(0x561a5f4b5900 msgr2=0x561a5f4b4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:01.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.216+0000 7f05fabc4740 1 -- 172.21.15.23:0/2839201402 shutdown_connections 2024-03-24T10:44:01.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.216+0000 7f05fabc4740 1 -- 172.21.15.23:0/2839201402 wait complete. 2024-03-24T10:44:01.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.216+0000 7f05fabc4740 1 -- start start 2024-03-24T10:44:01.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.216+0000 7f05fabc4740 1 Processor -- start 2024-03-24T10:44:01.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05fabc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05fabc4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05fabc4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4c46d0 0x561a5f4c8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05fabc4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x561a5f539e50 con 0x561a5f4c46d0 2024-03-24T10:44:01.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05fabc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561a5f539f90 con 0x561a5f4bb1d0 2024-03-24T10:44:01.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05fabc4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561a5f53a0d0 con 0x561a5f4bbaf0 2024-03-24T10:44:01.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05fabc4740 10 client.0 _ll_register_callbacks cb 0x7f05e40118f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:01.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05f549d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4c46d0 0x561a5f4c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05f549d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4c46d0 0x561a5f4c8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45462/0 (socket says 172.21.15.23:45462) 2024-03-24T10:44:01.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05f549d640 1 -- 172.21.15.23:0/3342183669 learned_addr learned my addr 172.21.15.23:0/3342183669 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:01.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05f5c9e640 1 --2- 172.21.15.23:0/3342183669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05f549d640 1 --2- 172.21.15.23:0/3342183669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05f549d640 1 -- 172.21.15.23:0/3342183669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 msgr2=0x561a5f4c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05f549d640 1 --2- 172.21.15.23:0/3342183669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05f549d640 1 -- 172.21.15.23:0/3342183669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4bb1d0 msgr2=0x561a5f4c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:01.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05f549d640 1 --2- 172.21.15.23:0/3342183669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05f549d640 1 -- 172.21.15.23:0/3342183669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a5f3e3ea0 con 0x561a5f4c46d0 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05f549d640 1 --2- 172.21.15.23:0/3342183669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4c46d0 0x561a5f4c8930 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f05e4011690 tx=0x7f05e4009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bbed51dac3852c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05d57fa640 1 -- 172.21.15.23:0/3342183669 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e4016070 con 0x561a5f4c46d0 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05d57fa640 1 -- 172.21.15.23:0/3342183669 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f05e4000e40 con 0x561a5f4c46d0 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.217+0000 7f05fabc4740 1 -- 172.21.15.23:0/3342183669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05e80048a0 con 0x561a5f4c46d0 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.218+0000 7f05d57fa640 1 -- 172.21.15.23:0/3342183669 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f05e4005110 con 0x561a5f4c46d0 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.218+0000 7f05fabc4740 10 client.16731 fetch_fsmap learned FSMap version 26 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.218+0000 7f05fabc4740 1 -- 172.21.15.23:0/3342183669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561a5f420ba0 con 0x561a5f4c46d0 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.218+0000 7f05d57fa640 1 -- 172.21.15.23:0/3342183669 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05e80048a0 con 0x561a5f4c46d0 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.218+0000 7f05d57fa640 1 -- 172.21.15.23:0/3342183669 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7f05e4044070 con 0x561a5f4c46d0 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.218+0000 7f05fabc4740 10 client.16731 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.218+0000 7f05fabc4740 10 client.16731 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.218+0000 7f05fabc4740 20 client.16731 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.218+0000 7f05fabc4740 1 --2- 172.21.15.23:0/3342183669 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561a5f53b560 0x561a5f53d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:01.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.218+0000 7f05fabc4740 4 client.16731 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.218+0000 7f05fabc4740 1 -- 172.21.15.23:0/3342183669 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561a5f1ad890 con 0x561a5f53b560 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.218+0000 7f05f5c9e640 1 --2- 172.21.15.23:0/3342183669 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561a5f53b560 0x561a5f53d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.218+0000 7f05f5c9e640 1 --2- 172.21.15.23:0/3342183669 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561a5f53b560 0x561a5f53d980 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e40c907dd995b8d5 server_cookie=e661764377b47876 in_seq=0 out_seq=0 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.218+0000 7f05d57fa640 10 client.16731 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05d57fa640 1 -- 172.21.15.23:0/3342183669 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561a5f1ad890 con 0x561a5f53b560 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05d57fa640 10 client.16731 handle_command_reply: tid=0 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05fabc4740 1 client.16731 shutdown 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05fabc4740 1 -- 172.21.15.23:0/3342183669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4c46d0 msgr2=0x561a5f4c8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05fabc4740 1 --2- 172.21.15.23:0/3342183669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4c46d0 0x561a5f4c8930 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f05e4011690 tx=0x7f05e4009860 comp rx=0 tx=0).stop 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05fabc4740 1 -- 172.21.15.23:0/3342183669 shutdown_connections 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05fabc4740 1 --2- 172.21.15.23:0/3342183669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a5f4c46d0 0x561a5f4c8930 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05fabc4740 1 --2- 172.21.15.23:0/3342183669 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561a5f53b560 0x561a5f53d980 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05fabc4740 1 --2- 172.21.15.23:0/3342183669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561a5f4bbaf0 0x561a5f4c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05d57fa640 0 client.16731 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:01.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05fabc4740 1 --2- 172.21.15.23:0/3342183669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561a5f4bb1d0 0x561a5f4c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05fabc4740 1 -- 172.21.15.23:0/3342183669 >> 172.21.15.23:0/3342183669 conn(0x561a5f4b5900 msgr2=0x561a5f4c16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:01.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05fabc4740 1 -- 172.21.15.23:0/3342183669 shutdown_connections 2024-03-24T10:44:01.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05fabc4740 1 -- 172.21.15.23:0/3342183669 wait complete. 2024-03-24T10:44:01.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05fabc4740 20 client.16731 trim_cache size 0 max 16384 2024-03-24T10:44:01.226 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:44:01.226 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:44:01.226 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:44:01.226 INFO:teuthology.orchestra.run.smithi023.stdout: "1a1b79b5-9f21-410a-9912-58bc73307afc": { 2024-03-24T10:44:01.226 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:44:01.226 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "1a1b79b5-9f21-410a-9912-58bc73307afc", 2024-03-24T10:44:01.227 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:44:01.227 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:44:01.227 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:44:01.227 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:44:01.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05f7f02640 1 -- 172.21.15.23:0/486409417 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f05d80a2230 msgr2=0x7f05d80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05f7f02640 1 --2- 172.21.15.23:0/486409417 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f05d80a2230 0x7f05d80a46f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f05e00cbe40 tx=0x7f05e0005470 comp rx=0 tx=0).stop 2024-03-24T10:44:01.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05f7f02640 1 -- 172.21.15.23:0/486409417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05f010e0b0 msgr2=0x7f05f010bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:01.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.219+0000 7f05f7f02640 1 --2- 172.21.15.23:0/486409417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05f010e0b0 0x7f05f010bfa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f05e0005fd0 tx=0x7f05e000ef10 comp rx=0 tx=0).stop 2024-03-24T10:44:01.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.220+0000 7f05f7f02640 1 -- 172.21.15.23:0/486409417 shutdown_connections 2024-03-24T10:44:01.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.220+0000 7f05f7f02640 1 --2- 172.21.15.23:0/486409417 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f05d80a2230 0x7f05d80a46f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.220+0000 7f05f7f02640 1 --2- 172.21.15.23:0/486409417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05f010e0b0 0x7f05f010bfa0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.220+0000 7f05f7f02640 1 --2- 172.21.15.23:0/486409417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f0100eb0 0x7f05f010ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.220+0000 7f05f7f02640 1 --2- 172.21.15.23:0/486409417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05f0100500 0x7f05f010d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:01.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.220+0000 7f05f7f02640 1 -- 172.21.15.23:0/486409417 >> 172.21.15.23:0/486409417 conn(0x7f05f0077770 msgr2=0x7f05f010f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:01.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.220+0000 7f05f7f02640 1 -- 172.21.15.23:0/486409417 shutdown_connections 2024-03-24T10:44:01.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:01.220+0000 7f05f7f02640 1 -- 172.21.15.23:0/486409417 wait complete. 2024-03-24T10:44:01.230 INFO:tasks.cephfs.filesystem:scrub status for tag:1a1b79b5-9f21-410a-9912-58bc73307afc - {'path': '/', 'tag': '1a1b79b5-9f21-410a-9912-58bc73307afc', 'options': 'recursive,force'} 2024-03-24T10:44:02.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 315945 12.06 MB/sec execute 19 sec latency 146.999 ms 2024-03-24T10:44:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:02 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:44:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:02 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:02 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:44:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:02 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:44:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:02 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:02 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:02.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:02 smithi023 ceph-mon[32466]: pgmap v121: 97 pgs: 97 active+clean; 177 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 325 op/s 2024-03-24T10:44:03.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:02 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:44:03.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:02 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:03.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:02 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:44:03.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:02 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:44:03.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:02 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:03.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:02 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:03.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:02 smithi117 ceph-mon[36368]: pgmap v121: 97 pgs: 97 active+clean; 177 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 325 op/s 2024-03-24T10:44:03.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:02 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:44:03.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:02 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:03.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:02 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:44:03.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:02 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:44:03.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:02 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:03.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:02 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:03.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:02 smithi116 ceph-mon[36248]: pgmap v121: 97 pgs: 97 active+clean; 177 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 325 op/s 2024-03-24T10:44:03.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 318051 11.82 MB/sec execute 20 sec latency 24.429 ms 2024-03-24T10:44:04.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 319944 11.85 MB/sec execute 21 sec latency 13.623 ms 2024-03-24T10:44:04.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:04 smithi023 ceph-mon[32466]: pgmap v122: 97 pgs: 97 active+clean; 177 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 233 op/s 2024-03-24T10:44:05.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:04 smithi117 ceph-mon[36368]: pgmap v122: 97 pgs: 97 active+clean; 177 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 233 op/s 2024-03-24T10:44:05.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:04 smithi116 ceph-mon[36248]: pgmap v122: 97 pgs: 97 active+clean; 177 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 233 op/s 2024-03-24T10:44:05.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 322789 12.09 MB/sec execute 22 sec latency 22.192 ms 2024-03-24T10:44:06.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 324656 11.87 MB/sec execute 23 sec latency 33.085 ms 2024-03-24T10:44:06.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:06 smithi023 ceph-mon[32466]: pgmap v123: 97 pgs: 97 active+clean; 177 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 233 op/s 2024-03-24T10:44:07.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:06 smithi117 ceph-mon[36368]: pgmap v123: 97 pgs: 97 active+clean; 177 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 233 op/s 2024-03-24T10:44:07.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:06 smithi116 ceph-mon[36248]: pgmap v123: 97 pgs: 97 active+clean; 177 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 233 op/s 2024-03-24T10:44:07.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 326803 11.95 MB/sec execute 24 sec latency 79.916 ms 2024-03-24T10:44:08.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 329319 11.97 MB/sec execute 25 sec latency 18.150 ms 2024-03-24T10:44:08.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:08 smithi023 ceph-mon[32466]: pgmap v124: 97 pgs: 97 active+clean; 183 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.6 MiB/s wr, 337 op/s 2024-03-24T10:44:09.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:08 smithi117 ceph-mon[36368]: pgmap v124: 97 pgs: 97 active+clean; 183 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.6 MiB/s wr, 337 op/s 2024-03-24T10:44:09.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:08 smithi116 ceph-mon[36248]: pgmap v124: 97 pgs: 97 active+clean; 183 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.6 MiB/s wr, 337 op/s 2024-03-24T10:44:09.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 331117 11.85 MB/sec execute 26 sec latency 26.435 ms 2024-03-24T10:44:10.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 333846 12.04 MB/sec execute 27 sec latency 13.625 ms 2024-03-24T10:44:10.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.441+0000 7fdcee79a640 1 -- 172.21.15.23:0/2872992222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdce81066a0 msgr2=0x7fdce8106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.441+0000 7fdcee79a640 1 --2- 172.21.15.23:0/2872992222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdce81066a0 0x7fdce8106a80 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009900 tx=0x7fdcd802bc30 comp rx=0 tx=0).stop 2024-03-24T10:44:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.441+0000 7fdcee79a640 1 -- 172.21.15.23:0/2872992222 shutdown_connections 2024-03-24T10:44:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.441+0000 7fdcee79a640 1 --2- 172.21.15.23:0/2872992222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdce8101200 0x7fdce810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.441+0000 7fdcee79a640 1 --2- 172.21.15.23:0/2872992222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdce8100860 0x7fdce8100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.441+0000 7fdcee79a640 1 --2- 172.21.15.23:0/2872992222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdce81066a0 0x7fdce8106a80 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.441+0000 7fdcee79a640 1 -- 172.21.15.23:0/2872992222 >> 172.21.15.23:0/2872992222 conn(0x7fdce8077470 msgr2=0x7fdce8077880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.441+0000 7fdcee79a640 1 -- 172.21.15.23:0/2872992222 shutdown_connections 2024-03-24T10:44:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.441+0000 7fdcee79a640 1 -- 172.21.15.23:0/2872992222 wait complete. 2024-03-24T10:44:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdcee79a640 1 Processor -- start 2024-03-24T10:44:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdcee79a640 1 -- start start 2024-03-24T10:44:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdcee79a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdce8100860 0x7fdce8102280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdcee79a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdce8101200 0x7fdce81027c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdcee79a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdce81066a0 0x7fdce8102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdcee79a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fdce81a6090 con 0x7fdce8100860 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdcee79a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdce81a6200 con 0x7fdce8101200 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdcee79a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdce81a63c0 con 0x7fdce81066a0 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdce7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdce8100860 0x7fdce8102280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdce7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdce8100860 0x7fdce8102280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54230/0 (socket says 172.21.15.23:54230) 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdce7fff640 1 -- 172.21.15.23:0/2482609442 learned_addr learned my addr 172.21.15.23:0/2482609442 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdce77fe640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdce8101200 0x7fdce81027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdcecd37640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdce81066a0 0x7fdce8102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdce7fff640 1 -- 172.21.15.23:0/2482609442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdce81066a0 msgr2=0x7fdce8102d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.442+0000 7fdce7fff640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdce81066a0 0x7fdce8102d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.443+0000 7fdce7fff640 1 -- 172.21.15.23:0/2482609442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdce8101200 msgr2=0x7fdce81027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.443+0000 7fdce7fff640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdce8101200 0x7fdce81027c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.443+0000 7fdce7fff640 1 -- 172.21.15.23:0/2482609442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce0009840 con 0x7fdce8100860 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.443+0000 7fdce77fe640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdce8101200 0x7fdce81027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.443+0000 7fdce7fff640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdce8100860 0x7fdce8102280 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8005940 tx=0x7fdcd802c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.443+0000 7fdce57fa640 1 -- 172.21.15.23:0/2482609442 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd80042d0 con 0x7fdce8100860 2024-03-24T10:44:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.443+0000 7fdce57fa640 1 -- 172.21.15.23:0/2482609442 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcd8004430 con 0x7fdce8100860 2024-03-24T10:44:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.443+0000 7fdce57fa640 1 -- 172.21.15.23:0/2482609442 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8003c50 con 0x7fdce8100860 2024-03-24T10:44:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.443+0000 7fdcee79a640 1 -- 172.21.15.23:0/2482609442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcd8009560 con 0x7fdce8100860 2024-03-24T10:44:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.443+0000 7fdcee79a640 1 -- 172.21.15.23:0/2482609442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdce81a69c0 con 0x7fdce8100860 2024-03-24T10:44:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.445+0000 7fdce57fa640 1 -- 172.21.15.23:0/2482609442 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fdcd80054f0 con 0x7fdce8100860 2024-03-24T10:44:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdcf137e740 1 Processor -- start 2024-03-24T10:44:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdcf137e740 1 -- start start 2024-03-24T10:44:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdcf137e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b85e0 0x5640cb8b89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdcf137e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8b8f00 0x5640cb8c1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdcf137e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8c1cd0 0x5640cb8c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdcf137e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5640cb8731b0 con 0x5640cb8b8f00 2024-03-24T10:44:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdcf137e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5640cb825f80 con 0x5640cb8c1cd0 2024-03-24T10:44:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdcf137e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5640cb8234b0 con 0x5640cb8b85e0 2024-03-24T10:44:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdce7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8b8f00 0x5640cb8c1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdce57fa640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdcd00a22d0 0x7fdcd00a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdce77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8c1cd0 0x5640cb8c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdce7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8b8f00 0x5640cb8c1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54236/0 (socket says 172.21.15.23:54236) 2024-03-24T10:44:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdce7fff640 1 -- 172.21.15.23:0/2232199460 learned_addr learned my addr 172.21.15.23:0/2232199460 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdcecd37640 1 --2- 172.21.15.23:0/2232199460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b85e0 0x5640cb8b89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdcecd37640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdcd00a22d0 0x7fdcd00a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdce57fa640 1 -- 172.21.15.23:0/2482609442 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fdcd80f0030 con 0x7fdce8100860 2024-03-24T10:44:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdcecd37640 1 -- 172.21.15.23:0/2232199460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8c1cd0 msgr2=0x5640cb8c40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdcecd37640 1 --2- 172.21.15.23:0/2232199460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8c1cd0 0x5640cb8c40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdcecd37640 1 -- 172.21.15.23:0/2232199460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8b8f00 msgr2=0x5640cb8c1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.446+0000 7fdcecd37640 1 --2- 172.21.15.23:0/2232199460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8b8f00 0x5640cb8c1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.447+0000 7fdcecd37640 1 -- 172.21.15.23:0/2232199460 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640cb7b4810 con 0x5640cb8b85e0 2024-03-24T10:44:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.447+0000 7fdce77fe640 1 --2- 172.21.15.23:0/2232199460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8c1cd0 0x5640cb8c40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.447+0000 7fdcecd37640 1 --2- 172.21.15.23:0/2232199460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b85e0 0x5640cb8b89c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdce0011e20 tx=0x7fdce000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c10a0605ee07e4e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.447+0000 7fdcecd37640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdcd00a22d0 0x7fdcd00a4790 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdce000fc90 tx=0x7fdce001e000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.447+0000 7fdcce7fc640 1 -- 172.21.15.23:0/2232199460 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce001dcb0 con 0x5640cb8b85e0 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.447+0000 7fdcce7fc640 1 -- 172.21.15.23:0/2232199460 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdce0009e40 con 0x5640cb8b85e0 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.447+0000 7fdcce7fc640 1 -- 172.21.15.23:0/2232199460 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce001e660 con 0x5640cb8b85e0 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 -- 172.21.15.23:0/2232199460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b85e0 msgr2=0x5640cb8b89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 --2- 172.21.15.23:0/2232199460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b85e0 0x5640cb8b89c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdce0011e20 tx=0x7fdce000f790 comp rx=0 tx=0).stop 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 -- 172.21.15.23:0/2232199460 shutdown_connections 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 --2- 172.21.15.23:0/2232199460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8c1cd0 0x5640cb8c40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 --2- 172.21.15.23:0/2232199460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8b8f00 0x5640cb8c1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 --2- 172.21.15.23:0/2232199460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b85e0 0x5640cb8b89c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 -- 172.21.15.23:0/2232199460 >> 172.21.15.23:0/2232199460 conn(0x5640cb8b2f00 msgr2=0x5640cb8b32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 -- 172.21.15.23:0/2232199460 shutdown_connections 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 -- 172.21.15.23:0/2232199460 wait complete. 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 -- start start 2024-03-24T10:44:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 Processor -- start 2024-03-24T10:44:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8b85e0 0x5640cb8bed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b8f00 0x5640cb8bf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8c1cd0 0x5640cb8bf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5640cb81ec90 con 0x5640cb8b85e0 2024-03-24T10:44:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.448+0000 7fdcf137e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5640cb7c3c50 con 0x5640cb8b8f00 2024-03-24T10:44:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdcf137e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5640cb4f2250 con 0x5640cb8c1cd0 2024-03-24T10:44:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdcf137e740 10 client.0 _ll_register_callbacks cb 0x5640cb812290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdce7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8b85e0 0x5640cb8bed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdce7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8b85e0 0x5640cb8bed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54240/0 (socket says 172.21.15.23:54240) 2024-03-24T10:44:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdce7fff640 1 -- 172.21.15.23:0/4101829533 learned_addr learned my addr 172.21.15.23:0/4101829533 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdce77fe640 1 --2- 172.21.15.23:0/4101829533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b8f00 0x5640cb8bf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdcecd37640 1 --2- 172.21.15.23:0/4101829533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8c1cd0 0x5640cb8bf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdce7fff640 1 -- 172.21.15.23:0/4101829533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8c1cd0 msgr2=0x5640cb8bf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdce7fff640 1 --2- 172.21.15.23:0/4101829533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8c1cd0 0x5640cb8bf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdce7fff640 1 -- 172.21.15.23:0/4101829533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b8f00 msgr2=0x5640cb8bf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdce7fff640 1 --2- 172.21.15.23:0/4101829533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b8f00 0x5640cb8bf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdce7fff640 1 -- 172.21.15.23:0/4101829533 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdce0019040 con 0x5640cb8b85e0 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdce7fff640 1 --2- 172.21.15.23:0/4101829533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8b85e0 0x5640cb8bed10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8000c00 tx=0x7fdcd80bc730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2167727297785229 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdcc77fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdcc77fe640 1 -- 172.21.15.23:0/4101829533 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd80bcb30 con 0x5640cb8b85e0 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdcc77fe640 1 -- 172.21.15.23:0/4101829533 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fdcd80c0c80 con 0x5640cb8b85e0 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdcf137e740 1 -- 172.21.15.23:0/4101829533 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5640cb843790 con 0x5640cb8b85e0 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdcc77fe640 1 -- 172.21.15.23:0/4101829533 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdcd80c52d0 con 0x5640cb8b85e0 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdcf137e740 10 client.16743 fetch_fsmap learned FSMap version 26 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.449+0000 7fdcf137e740 1 -- 172.21.15.23:0/4101829533 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5640cb81d310 con 0x5640cb8b85e0 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.450+0000 7fdcc77fe640 1 -- 172.21.15.23:0/4101829533 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5640cb843790 con 0x5640cb8b85e0 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.450+0000 7fdcc77fe640 1 -- 172.21.15.23:0/4101829533 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7fdcd803c070 con 0x5640cb8b85e0 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.450+0000 7fdcf137e740 10 client.16743 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:44:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.450+0000 7fdcf137e740 10 client.16743 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:10.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.450+0000 7fdcf137e740 20 client.16743 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:10.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.450+0000 7fdcf137e740 1 --2- 172.21.15.23:0/4101829533 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5640cb8d1900 0x5640cb8d3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.450+0000 7fdcf137e740 4 client.16743 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:44:10.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.450+0000 7fdcf137e740 1 -- 172.21.15.23:0/4101829533 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640cb51e600 con 0x5640cb8d1900 2024-03-24T10:44:10.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.450+0000 7fdce77fe640 1 --2- 172.21.15.23:0/4101829533 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5640cb8d1900 0x5640cb8d3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.450+0000 7fdcc77fe640 10 client.16743 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:10.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.450+0000 7fdce77fe640 1 --2- 172.21.15.23:0/4101829533 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5640cb8d1900 0x5640cb8d3d40 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be2aebc6ef145d5 server_cookie=2058ff08e43bb66f in_seq=0 out_seq=0 2024-03-24T10:44:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.451+0000 7fdcc77fe640 1 -- 172.21.15.23:0/4101829533 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5640cb51e600 con 0x5640cb8d1900 2024-03-24T10:44:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.451+0000 7fdcc77fe640 10 client.16743 handle_command_reply: tid=0 2024-03-24T10:44:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.451+0000 7fdcf137e740 1 client.16743 shutdown 2024-03-24T10:44:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.451+0000 7fdcf137e740 1 -- 172.21.15.23:0/4101829533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8b85e0 msgr2=0x5640cb8bed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.451+0000 7fdcf137e740 1 --2- 172.21.15.23:0/4101829533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8b85e0 0x5640cb8bed10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8000c00 tx=0x7fdcd80bc730 comp rx=0 tx=0).stop 2024-03-24T10:44:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.452+0000 7fdcf137e740 1 -- 172.21.15.23:0/4101829533 shutdown_connections 2024-03-24T10:44:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.452+0000 7fdcf137e740 1 --2- 172.21.15.23:0/4101829533 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5640cb8d1900 0x5640cb8d3d40 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.452+0000 7fdcf137e740 1 --2- 172.21.15.23:0/4101829533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8c1cd0 0x5640cb8bf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.452+0000 7fdcf137e740 1 --2- 172.21.15.23:0/4101829533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b8f00 0x5640cb8bf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.452+0000 7fdcc77fe640 0 client.16743 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.452+0000 7fdcf137e740 1 --2- 172.21.15.23:0/4101829533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8b85e0 0x5640cb8bed10 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.452+0000 7fdcf137e740 1 -- 172.21.15.23:0/4101829533 >> 172.21.15.23:0/4101829533 conn(0x5640cb8b2f00 msgr2=0x5640cb8b4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.452+0000 7fdcf137e740 1 -- 172.21.15.23:0/4101829533 shutdown_connections 2024-03-24T10:44:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.452+0000 7fdcf137e740 1 -- 172.21.15.23:0/4101829533 wait complete. 2024-03-24T10:44:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.452+0000 7fdcf137e740 20 client.16743 trim_cache size 0 max 16384 2024-03-24T10:44:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcf137e740 1 Processor -- start 2024-03-24T10:44:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcf137e740 1 -- start start 2024-03-24T10:44:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcf137e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b85e0 0x5640cb8cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcf137e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b8f00 0x5640cb8bf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcf137e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8c1cd0 0x5640cb8b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcf137e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5640cb7a4660 con 0x5640cb8c1cd0 2024-03-24T10:44:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcf137e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5640cb8b3f00 con 0x5640cb8b85e0 2024-03-24T10:44:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcf137e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5640cb8b1420 con 0x5640cb8b8f00 2024-03-24T10:44:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcecd37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8c1cd0 0x5640cb8b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcecd37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8c1cd0 0x5640cb8b1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54244/0 (socket says 172.21.15.23:54244) 2024-03-24T10:44:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcecd37640 1 -- 172.21.15.23:0/3922862287 learned_addr learned my addr 172.21.15.23:0/3922862287 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdce7fff640 1 --2- 172.21.15.23:0/3922862287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b85e0 0x5640cb8cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdce77fe640 1 --2- 172.21.15.23:0/3922862287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b8f00 0x5640cb8bf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcecd37640 1 -- 172.21.15.23:0/3922862287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b8f00 msgr2=0x5640cb8bf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcecd37640 1 --2- 172.21.15.23:0/3922862287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b8f00 0x5640cb8bf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcecd37640 1 -- 172.21.15.23:0/3922862287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b85e0 msgr2=0x5640cb8cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcecd37640 1 --2- 172.21.15.23:0/3922862287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b85e0 0x5640cb8cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.467+0000 7fdcecd37640 1 -- 172.21.15.23:0/3922862287 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640cb7b9810 con 0x5640cb8c1cd0 2024-03-24T10:44:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.468+0000 7fdcecd37640 1 --2- 172.21.15.23:0/3922862287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8c1cd0 0x5640cb8b1d20 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdce0011e20 tx=0x7fdce0009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c25a98108a1b977 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.468+0000 7fdccdffb640 1 -- 172.21.15.23:0/3922862287 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce001dcb0 con 0x5640cb8c1cd0 2024-03-24T10:44:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.468+0000 7fdccdffb640 1 -- 172.21.15.23:0/3922862287 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdce001de10 con 0x5640cb8c1cd0 2024-03-24T10:44:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.468+0000 7fdccdffb640 1 -- 172.21.15.23:0/3922862287 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce00267f0 con 0x5640cb8c1cd0 2024-03-24T10:44:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.468+0000 7fdcf137e740 1 -- 172.21.15.23:0/3922862287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8c1cd0 msgr2=0x5640cb8b1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.468+0000 7fdcf137e740 1 --2- 172.21.15.23:0/3922862287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8c1cd0 0x5640cb8b1d20 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdce0011e20 tx=0x7fdce0009570 comp rx=0 tx=0).stop 2024-03-24T10:44:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.468+0000 7fdcf137e740 1 -- 172.21.15.23:0/3922862287 shutdown_connections 2024-03-24T10:44:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.468+0000 7fdcf137e740 1 --2- 172.21.15.23:0/3922862287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8c1cd0 0x5640cb8b1d20 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.468+0000 7fdcf137e740 1 --2- 172.21.15.23:0/3922862287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b8f00 0x5640cb8bf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.468+0000 7fdcf137e740 1 --2- 172.21.15.23:0/3922862287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b85e0 0x5640cb8cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.468+0000 7fdcf137e740 1 -- 172.21.15.23:0/3922862287 >> 172.21.15.23:0/3922862287 conn(0x5640cb8b2f00 msgr2=0x5640cb8b1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdcf137e740 1 -- 172.21.15.23:0/3922862287 shutdown_connections 2024-03-24T10:44:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdcf137e740 1 -- 172.21.15.23:0/3922862287 wait complete. 2024-03-24T10:44:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdcf137e740 1 -- start start 2024-03-24T10:44:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdcf137e740 1 Processor -- start 2024-03-24T10:44:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdcf137e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b85e0 0x5640cb8c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdcf137e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b8f00 0x5640cb8bb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdcf137e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8c1cd0 0x5640cb8bbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdcf137e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5640cb8bc3c0 con 0x5640cb8c1cd0 2024-03-24T10:44:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdcf137e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5640cb8bc530 con 0x5640cb8b8f00 2024-03-24T10:44:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdcf137e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5640cb936cb0 con 0x5640cb8b85e0 2024-03-24T10:44:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdcf137e740 10 client.0 _ll_register_callbacks cb 0x7fdcd8005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdcecd37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8c1cd0 0x5640cb8bbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdcecd37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8c1cd0 0x5640cb8bbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54252/0 (socket says 172.21.15.23:54252) 2024-03-24T10:44:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdcecd37640 1 -- 172.21.15.23:0/2685508477 learned_addr learned my addr 172.21.15.23:0/2685508477 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdce77fe640 1 --2- 172.21.15.23:0/2685508477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b8f00 0x5640cb8bb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.469+0000 7fdce7fff640 1 --2- 172.21.15.23:0/2685508477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b85e0 0x5640cb8c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcecd37640 1 -- 172.21.15.23:0/2685508477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b85e0 msgr2=0x5640cb8c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcecd37640 1 --2- 172.21.15.23:0/2685508477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b85e0 0x5640cb8c9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcecd37640 1 -- 172.21.15.23:0/2685508477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b8f00 msgr2=0x5640cb8bb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcecd37640 1 --2- 172.21.15.23:0/2685508477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b8f00 0x5640cb8bb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcecd37640 1 -- 172.21.15.23:0/2685508477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640cb7b4810 con 0x5640cb8c1cd0 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcecd37640 1 --2- 172.21.15.23:0/2685508477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8c1cd0 0x5640cb8bbcf0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fdce0009df0 tx=0x7fdce0004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c289716790c61674 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcc77fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcc77fe640 1 -- 172.21.15.23:0/2685508477 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce001dcb0 con 0x5640cb8c1cd0 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcc77fe640 1 -- 172.21.15.23:0/2685508477 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fdce0026450 con 0x5640cb8c1cd0 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcf137e740 1 -- 172.21.15.23:0/2685508477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdce0009e40 con 0x5640cb8c1cd0 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcc77fe640 1 -- 172.21.15.23:0/2685508477 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdce0026610 con 0x5640cb8c1cd0 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcf137e740 10 client.16749 fetch_fsmap learned FSMap version 26 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcf137e740 1 -- 172.21.15.23:0/2685508477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5640cb7df5a0 con 0x5640cb8c1cd0 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcc77fe640 1 -- 172.21.15.23:0/2685508477 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdce0009e40 con 0x5640cb8c1cd0 2024-03-24T10:44:10.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcc77fe640 1 -- 172.21.15.23:0/2685508477 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7fdce0024070 con 0x5640cb8c1cd0 2024-03-24T10:44:10.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcf137e740 10 client.16749 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:44:10.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcf137e740 10 client.16749 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:10.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcf137e740 20 client.16749 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:10.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.470+0000 7fdcf137e740 1 --2- 172.21.15.23:0/2685508477 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5640cb9380f0 0x5640cb93a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:10.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.471+0000 7fdcf137e740 4 client.16749 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:44:10.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.471+0000 7fdcf137e740 1 -- 172.21.15.23:0/2685508477 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640cb56f1b0 con 0x5640cb9380f0 2024-03-24T10:44:10.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.471+0000 7fdce7fff640 1 --2- 172.21.15.23:0/2685508477 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5640cb9380f0 0x5640cb93a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:10.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.471+0000 7fdce7fff640 1 --2- 172.21.15.23:0/2685508477 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5640cb9380f0 0x5640cb93a510 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=369a0588995e83c server_cookie=ef063b38481ecf26 in_seq=0 out_seq=0 2024-03-24T10:44:10.476 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.471+0000 7fdcc77fe640 10 client.16749 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.480+0000 7fdcc77fe640 1 -- 172.21.15.23:0/2685508477 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5640cb56f1b0 con 0x5640cb9380f0 2024-03-24T10:44:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.480+0000 7fdcc77fe640 10 client.16749 handle_command_reply: tid=0 2024-03-24T10:44:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.480+0000 7fdcf137e740 1 client.16749 shutdown 2024-03-24T10:44:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.481+0000 7fdcf137e740 1 -- 172.21.15.23:0/2685508477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8c1cd0 msgr2=0x5640cb8bbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.481+0000 7fdcf137e740 1 --2- 172.21.15.23:0/2685508477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8c1cd0 0x5640cb8bbcf0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fdce0009df0 tx=0x7fdce0004d60 comp rx=0 tx=0).stop 2024-03-24T10:44:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.481+0000 7fdcf137e740 1 -- 172.21.15.23:0/2685508477 shutdown_connections 2024-03-24T10:44:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.481+0000 7fdcf137e740 1 --2- 172.21.15.23:0/2685508477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640cb8c1cd0 0x5640cb8bbcf0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.481+0000 7fdcf137e740 1 --2- 172.21.15.23:0/2685508477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5640cb8b8f00 0x5640cb8bb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.481+0000 7fdcf137e740 1 --2- 172.21.15.23:0/2685508477 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5640cb9380f0 0x5640cb93a510 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.481+0000 7fdcf137e740 1 --2- 172.21.15.23:0/2685508477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5640cb8b85e0 0x5640cb8c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.481+0000 7fdcf137e740 1 -- 172.21.15.23:0/2685508477 >> 172.21.15.23:0/2685508477 conn(0x5640cb8b2f00 msgr2=0x5640cb8b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.481+0000 7fdcc77fe640 0 client.16749 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.481+0000 7fdcf137e740 1 -- 172.21.15.23:0/2685508477 shutdown_connections 2024-03-24T10:44:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.481+0000 7fdcf137e740 1 -- 172.21.15.23:0/2685508477 wait complete. 2024-03-24T10:44:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.481+0000 7fdcf137e740 20 client.16749 trim_cache size 0 max 16384 2024-03-24T10:44:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:44:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:44:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:44:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:44:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.482+0000 7fdcee79a640 1 -- 172.21.15.23:0/2482609442 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdcd00a22d0 msgr2=0x7fdcd00a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.482+0000 7fdcee79a640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdcd00a22d0 0x7fdcd00a4790 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdce000fc90 tx=0x7fdce001e000 comp rx=0 tx=0).stop 2024-03-24T10:44:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.482+0000 7fdcee79a640 1 -- 172.21.15.23:0/2482609442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdce8100860 msgr2=0x7fdce8102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.482+0000 7fdcee79a640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdce8100860 0x7fdce8102280 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8005940 tx=0x7fdcd802c670 comp rx=0 tx=0).stop 2024-03-24T10:44:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.482+0000 7fdcee79a640 1 -- 172.21.15.23:0/2482609442 shutdown_connections 2024-03-24T10:44:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.482+0000 7fdcee79a640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdcd00a22d0 0x7fdcd00a4790 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.482+0000 7fdcee79a640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdce81066a0 0x7fdce8102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.482+0000 7fdcee79a640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdce8101200 0x7fdce81027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.482+0000 7fdcee79a640 1 --2- 172.21.15.23:0/2482609442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdce8100860 0x7fdce8102280 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.482+0000 7fdcee79a640 1 -- 172.21.15.23:0/2482609442 >> 172.21.15.23:0/2482609442 conn(0x7fdce8077470 msgr2=0x7fdce806a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.482+0000 7fdcee79a640 1 -- 172.21.15.23:0/2482609442 shutdown_connections 2024-03-24T10:44:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:10.482+0000 7fdcee79a640 1 -- 172.21.15.23:0/2482609442 wait complete. 2024-03-24T10:44:10.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:10 smithi023 ceph-mon[32466]: pgmap v125: 97 pgs: 97 active+clean; 183 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 237 op/s 2024-03-24T10:44:10.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:10 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:44:10.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:10 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:44:11.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:10 smithi117 ceph-mon[36368]: pgmap v125: 97 pgs: 97 active+clean; 183 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 237 op/s 2024-03-24T10:44:11.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:10 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:44:11.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:10 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:44:11.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:10 smithi116 ceph-mon[36248]: pgmap v125: 97 pgs: 97 active+clean; 183 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 237 op/s 2024-03-24T10:44:11.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:10 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:44:11.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:10 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:44:11.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 336323 12.03 MB/sec execute 28 sec latency 16.971 ms 2024-03-24T10:44:12.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 337767 11.88 MB/sec execute 29 sec latency 67.024 ms 2024-03-24T10:44:12.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:12 smithi023 ceph-mon[32466]: pgmap v126: 97 pgs: 97 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.7 MiB/s wr, 341 op/s 2024-03-24T10:44:13.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:12 smithi117 ceph-mon[36368]: pgmap v126: 97 pgs: 97 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.7 MiB/s wr, 341 op/s 2024-03-24T10:44:13.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:12 smithi116 ceph-mon[36248]: pgmap v126: 97 pgs: 97 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.7 MiB/s wr, 341 op/s 2024-03-24T10:44:13.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 340608 12.06 MB/sec execute 30 sec latency 17.799 ms 2024-03-24T10:44:14.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 342213 11.89 MB/sec execute 31 sec latency 13.137 ms 2024-03-24T10:44:14.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:14 smithi023 ceph-mon[32466]: pgmap v127: 97 pgs: 97 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 206 op/s 2024-03-24T10:44:14.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:14 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:44:15.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:14 smithi117 ceph-mon[36368]: pgmap v127: 97 pgs: 97 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 206 op/s 2024-03-24T10:44:15.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:14 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:44:15.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:14 smithi116 ceph-mon[36248]: pgmap v127: 97 pgs: 97 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 206 op/s 2024-03-24T10:44:15.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:14 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:44:15.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 344613 11.95 MB/sec execute 32 sec latency 14.730 ms 2024-03-24T10:44:15.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:15 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:44:15.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:15 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:44:15.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:15 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:44:16.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:15 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:44:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:15 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:44:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:15 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:44:16.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:15 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:44:16.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:15 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:44:16.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:15 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:44:16.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 347089 11.97 MB/sec execute 33 sec latency 20.834 ms 2024-03-24T10:44:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:16 smithi023 ceph-mon[32466]: pgmap v128: 97 pgs: 97 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 206 op/s 2024-03-24T10:44:17.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:16 smithi117 ceph-mon[36368]: pgmap v128: 97 pgs: 97 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 206 op/s 2024-03-24T10:44:17.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:16 smithi116 ceph-mon[36248]: pgmap v128: 97 pgs: 97 active+clean; 190 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 206 op/s 2024-03-24T10:44:17.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 348648 11.87 MB/sec execute 34 sec latency 74.196 ms 2024-03-24T10:44:18.117 INFO:tasks.workunit.client.0.smithi023.stdout: 1 351383 12.01 MB/sec execute 35 sec latency 28.865 ms 2024-03-24T10:44:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:18 smithi023 ceph-mon[32466]: pgmap v129: 97 pgs: 97 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.6 MiB/s wr, 300 op/s 2024-03-24T10:44:19.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:18 smithi117 ceph-mon[36368]: pgmap v129: 97 pgs: 97 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.6 MiB/s wr, 300 op/s 2024-03-24T10:44:19.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:18 smithi116 ceph-mon[36248]: pgmap v129: 97 pgs: 97 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.6 MiB/s wr, 300 op/s 2024-03-24T10:44:19.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 353507 11.88 MB/sec execute 36 sec latency 15.640 ms 2024-03-24T10:44:20.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 355422 11.89 MB/sec execute 37 sec latency 18.224 ms 2024-03-24T10:44:20.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:20 smithi023 ceph-mon[32466]: pgmap v130: 97 pgs: 97 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 197 op/s 2024-03-24T10:44:21.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:20 smithi117 ceph-mon[36368]: pgmap v130: 97 pgs: 97 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 197 op/s 2024-03-24T10:44:21.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:20 smithi116 ceph-mon[36248]: pgmap v130: 97 pgs: 97 active+clean; 196 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 197 op/s 2024-03-24T10:44:21.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 358220 12.03 MB/sec execute 38 sec latency 16.657 ms 2024-03-24T10:44:22.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 359941 11.90 MB/sec execute 39 sec latency 43.286 ms 2024-03-24T10:44:22.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:22 smithi023 ceph-mon[32466]: pgmap v131: 97 pgs: 97 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 303 op/s 2024-03-24T10:44:23.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:22 smithi117 ceph-mon[36368]: pgmap v131: 97 pgs: 97 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 303 op/s 2024-03-24T10:44:23.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:22 smithi116 ceph-mon[36248]: pgmap v131: 97 pgs: 97 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 303 op/s 2024-03-24T10:44:23.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 362183 11.95 MB/sec execute 40 sec latency 94.844 ms 2024-03-24T10:44:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.570+0000 7fe43fa1e640 1 -- 172.21.15.23:0/2827800624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4381016e0 msgr2=0x7fe438101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.570+0000 7fe43fa1e640 1 --2- 172.21.15.23:0/2827800624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4381016e0 0x7fe438101b60 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fe428009930 tx=0x7fe42802bc30 comp rx=0 tx=0).stop 2024-03-24T10:44:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.570+0000 7fe43fa1e640 1 -- 172.21.15.23:0/2827800624 shutdown_connections 2024-03-24T10:44:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.570+0000 7fe43fa1e640 1 --2- 172.21.15.23:0/2827800624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4381020a0 0x7fe43810e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.570+0000 7fe43fa1e640 1 --2- 172.21.15.23:0/2827800624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4381016e0 0x7fe438101b60 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.570+0000 7fe43fa1e640 1 --2- 172.21.15.23:0/2827800624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4381076f0 0x7fe438107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.570+0000 7fe43fa1e640 1 -- 172.21.15.23:0/2827800624 >> 172.21.15.23:0/2827800624 conn(0x7fe4380fd210 msgr2=0x7fe4380ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.570+0000 7fe43fa1e640 1 -- 172.21.15.23:0/2827800624 shutdown_connections 2024-03-24T10:44:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.570+0000 7fe43fa1e640 1 -- 172.21.15.23:0/2827800624 wait complete. 2024-03-24T10:44:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.570+0000 7fe43fa1e640 1 Processor -- start 2024-03-24T10:44:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.570+0000 7fe43fa1e640 1 -- start start 2024-03-24T10:44:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43fa1e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4381016e0 0x7fe438073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43fa1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4381020a0 0x7fe438073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43fa1e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4381076f0 0x7fe438074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43fa1e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe4381a60b0 con 0x7fe4381076f0 2024-03-24T10:44:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43fa1e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe4381a6220 con 0x7fe4381020a0 2024-03-24T10:44:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43fa1e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe4381a63e0 con 0x7fe4381016e0 2024-03-24T10:44:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43dfbb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4381076f0 0x7fe438074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43cfb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4381020a0 0x7fe438073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43dfbb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4381076f0 0x7fe438074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49184/0 (socket says 172.21.15.23:49184) 2024-03-24T10:44:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43d7ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4381016e0 0x7fe438073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43dfbb640 1 -- 172.21.15.23:0/3308088312 learned_addr learned my addr 172.21.15.23:0/3308088312 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43dfbb640 1 -- 172.21.15.23:0/3308088312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4381016e0 msgr2=0x7fe438073a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43dfbb640 1 --2- 172.21.15.23:0/3308088312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4381016e0 0x7fe438073a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43dfbb640 1 -- 172.21.15.23:0/3308088312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4381020a0 msgr2=0x7fe438073f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43dfbb640 1 --2- 172.21.15.23:0/3308088312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4381020a0 0x7fe438073f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43dfbb640 1 -- 172.21.15.23:0/3308088312 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe428009560 con 0x7fe4381076f0 2024-03-24T10:44:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43d7ba640 1 --2- 172.21.15.23:0/3308088312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4381016e0 0x7fe438073a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43dfbb640 1 --2- 172.21.15.23:0/3308088312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4381076f0 0x7fe438074490 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe43000eb10 tx=0x7fe43000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe42e7fc640 1 -- 172.21.15.23:0/3308088312 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe430004b20 con 0x7fe4381076f0 2024-03-24T10:44:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe42e7fc640 1 -- 172.21.15.23:0/3308088312 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe430004c80 con 0x7fe4381076f0 2024-03-24T10:44:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43fa1e640 1 -- 172.21.15.23:0/3308088312 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4381a66c0 con 0x7fe4381076f0 2024-03-24T10:44:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe42e7fc640 1 -- 172.21.15.23:0/3308088312 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe430010650 con 0x7fe4381076f0 2024-03-24T10:44:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.571+0000 7fe43fa1e640 1 -- 172.21.15.23:0/3308088312 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4381a6b50 con 0x7fe4381076f0 2024-03-24T10:44:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.573+0000 7fe42e7fc640 1 -- 172.21.15.23:0/3308088312 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fe430005020 con 0x7fe4381076f0 2024-03-24T10:44:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.574+0000 7fe4425fd740 1 Processor -- start 2024-03-24T10:44:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.574+0000 7fe4425fd740 1 -- start start 2024-03-24T10:44:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.574+0000 7fe42e7fc640 1 --2- 172.21.15.23:0/3308088312 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe4240a22d0 0x7fe4240a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.574+0000 7fe4425fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adf5e0 0x558573adf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.574+0000 7fe4425fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adff00 0x558573ae8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.574+0000 7fe4425fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573aeb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.574+0000 7fe4425fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558573a9a1b0 con 0x558573ae8cd0 2024-03-24T10:44:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.574+0000 7fe4425fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558573a4cf80 con 0x558573adff00 2024-03-24T10:44:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.574+0000 7fe4425fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558573a4a4b0 con 0x558573adf5e0 2024-03-24T10:44:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.574+0000 7fe43cfb9640 1 --2- 172.21.15.23:0/3308088312 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe4240a22d0 0x7fe4240a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.574+0000 7fe42e7fc640 1 -- 172.21.15.23:0/3308088312 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe430014070 con 0x7fe4381076f0 2024-03-24T10:44:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.574+0000 7fe43d7ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adf5e0 0x558573adf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.574+0000 7fe43dfbb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573aeb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.574+0000 7fe43d7ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adf5e0 0x558573adf9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:44330/0 (socket says 172.21.15.23:44330) 2024-03-24T10:44:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.575+0000 7fe43dfbb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573aeb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49186/0 (socket says 172.21.15.23:49186) 2024-03-24T10:44:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.575+0000 7fe43d7ba640 1 -- 172.21.15.23:0/1004440611 learned_addr learned my addr 172.21.15.23:0/1004440611 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.575+0000 7fe43cfb9640 1 --2- 172.21.15.23:0/1004440611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adff00 0x558573ae8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.575+0000 7fe43d7ba640 1 -- 172.21.15.23:0/1004440611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adff00 msgr2=0x558573ae8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.575+0000 7fe43d7ba640 1 --2- 172.21.15.23:0/1004440611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adff00 0x558573ae8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.575+0000 7fe43d7ba640 1 -- 172.21.15.23:0/1004440611 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 msgr2=0x558573aeb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.575+0000 7fe43d7ba640 1 --2- 172.21.15.23:0/1004440611 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573aeb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.575+0000 7fe43d7ba640 1 -- 172.21.15.23:0/1004440611 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585739db810 con 0x558573adf5e0 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.575+0000 7fe43cfb9640 1 --2- 172.21.15.23:0/3308088312 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe4240a22d0 0x7fe4240a4790 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe428005ab0 tx=0x7fe42802c640 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.575+0000 7fe43d7ba640 1 --2- 172.21.15.23:0/1004440611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adf5e0 0x558573adf9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe4340114f0 tx=0x7fe4340119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3a6635212ef77d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.575+0000 7fe4237fe640 1 -- 172.21.15.23:0/1004440611 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43400f5c0 con 0x558573adf5e0 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.575+0000 7fe4237fe640 1 -- 172.21.15.23:0/1004440611 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe43400fc00 con 0x558573adf5e0 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.575+0000 7fe4237fe640 1 -- 172.21.15.23:0/1004440611 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe434019cd0 con 0x558573adf5e0 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.576+0000 7fe4425fd740 1 -- 172.21.15.23:0/1004440611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adf5e0 msgr2=0x558573adf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.576+0000 7fe4425fd740 1 --2- 172.21.15.23:0/1004440611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adf5e0 0x558573adf9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe4340114f0 tx=0x7fe4340119b0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.576+0000 7fe4425fd740 1 -- 172.21.15.23:0/1004440611 shutdown_connections 2024-03-24T10:44:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.576+0000 7fe4425fd740 1 --2- 172.21.15.23:0/1004440611 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573aeb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.576+0000 7fe4425fd740 1 --2- 172.21.15.23:0/1004440611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adff00 0x558573ae8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.576+0000 7fe4425fd740 1 --2- 172.21.15.23:0/1004440611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adf5e0 0x558573adf9c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.576+0000 7fe4425fd740 1 -- 172.21.15.23:0/1004440611 >> 172.21.15.23:0/1004440611 conn(0x558573ad9f00 msgr2=0x558573ada2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.576+0000 7fe4425fd740 1 -- 172.21.15.23:0/1004440611 shutdown_connections 2024-03-24T10:44:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.576+0000 7fe4425fd740 1 -- 172.21.15.23:0/1004440611 wait complete. 2024-03-24T10:44:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.576+0000 7fe4425fd740 1 -- start start 2024-03-24T10:44:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.576+0000 7fe4425fd740 1 Processor -- start 2024-03-24T10:44:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe4425fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adf5e0 0x558573ae5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe4425fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adff00 0x558573ae6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe4425fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573ae67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe4425fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558573a45c90 con 0x558573ae8cd0 2024-03-24T10:44:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe4425fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5585739eac50 con 0x558573adf5e0 2024-03-24T10:44:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe4425fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558573719250 con 0x558573adff00 2024-03-24T10:44:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe4425fd740 10 client.0 _ll_register_callbacks cb 0x558573a39290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe43dfbb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573ae67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe43dfbb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573ae67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49200/0 (socket says 172.21.15.23:49200) 2024-03-24T10:44:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe43dfbb640 1 -- 172.21.15.23:0/4223668921 learned_addr learned my addr 172.21.15.23:0/4223668921 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe43cfb9640 1 --2- 172.21.15.23:0/4223668921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adff00 0x558573ae6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe43d7ba640 1 --2- 172.21.15.23:0/4223668921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adf5e0 0x558573ae5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe43dfbb640 1 -- 172.21.15.23:0/4223668921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adff00 msgr2=0x558573ae6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe43dfbb640 1 --2- 172.21.15.23:0/4223668921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adff00 0x558573ae6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe43dfbb640 1 -- 172.21.15.23:0/4223668921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adf5e0 msgr2=0x558573ae5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe43dfbb640 1 --2- 172.21.15.23:0/4223668921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adf5e0 0x558573ae5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe43dfbb640 1 -- 172.21.15.23:0/4223668921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe434009cc0 con 0x558573ae8cd0 2024-03-24T10:44:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe43cfb9640 1 --2- 172.21.15.23:0/4223668921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adff00 0x558573ae6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe43dfbb640 1 --2- 172.21.15.23:0/4223668921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573ae67d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe43009aff0 tx=0x7fe430049e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f4930e5df55621 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe420ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.577+0000 7fe420ff9640 1 -- 172.21.15.23:0/4223668921 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43009f1b0 con 0x558573ae8cd0 2024-03-24T10:44:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.578+0000 7fe420ff9640 1 -- 172.21.15.23:0/4223668921 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe4300a3800 con 0x558573ae8cd0 2024-03-24T10:44:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.578+0000 7fe4425fd740 1 -- 172.21.15.23:0/4223668921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558573a6a790 con 0x558573ae8cd0 2024-03-24T10:44:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.578+0000 7fe420ff9640 1 -- 172.21.15.23:0/4223668921 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe4300976d0 con 0x558573ae8cd0 2024-03-24T10:44:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.578+0000 7fe4425fd740 10 client.16761 fetch_fsmap learned FSMap version 26 2024-03-24T10:44:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.578+0000 7fe4425fd740 1 -- 172.21.15.23:0/4223668921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558573a44310 con 0x558573ae8cd0 2024-03-24T10:44:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.578+0000 7fe420ff9640 1 -- 172.21.15.23:0/4223668921 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558573a6a790 con 0x558573ae8cd0 2024-03-24T10:44:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.578+0000 7fe420ff9640 1 -- 172.21.15.23:0/4223668921 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7fe4300b3020 con 0x558573ae8cd0 2024-03-24T10:44:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.578+0000 7fe4425fd740 10 client.16761 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:44:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.578+0000 7fe4425fd740 10 client.16761 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.578+0000 7fe4425fd740 20 client.16761 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.578+0000 7fe4425fd740 1 --2- 172.21.15.23:0/4223668921 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558573af8c80 0x558573afb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.578+0000 7fe4425fd740 4 client.16761 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:44:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.578+0000 7fe4425fd740 1 -- 172.21.15.23:0/4223668921 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558573745600 con 0x558573af8c80 2024-03-24T10:44:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.578+0000 7fe43d7ba640 1 --2- 172.21.15.23:0/4223668921 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558573af8c80 0x558573afb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.579+0000 7fe420ff9640 10 client.16761 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.579+0000 7fe43d7ba640 1 --2- 172.21.15.23:0/4223668921 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558573af8c80 0x558573afb070 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0041e975cdacdf9 server_cookie=a3b98e014750446b in_seq=0 out_seq=0 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe420ff9640 1 -- 172.21.15.23:0/4223668921 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558573745600 con 0x558573af8c80 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe420ff9640 10 client.16761 handle_command_reply: tid=0 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe4425fd740 1 client.16761 shutdown 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe4425fd740 1 -- 172.21.15.23:0/4223668921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 msgr2=0x558573ae67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe4425fd740 1 --2- 172.21.15.23:0/4223668921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573ae67d0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe43009aff0 tx=0x7fe430049e10 comp rx=0 tx=0).stop 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe4425fd740 1 -- 172.21.15.23:0/4223668921 shutdown_connections 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe4425fd740 1 --2- 172.21.15.23:0/4223668921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573ae67d0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe4425fd740 1 --2- 172.21.15.23:0/4223668921 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558573af8c80 0x558573afb070 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe4425fd740 1 --2- 172.21.15.23:0/4223668921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adff00 0x558573ae6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe4425fd740 1 --2- 172.21.15.23:0/4223668921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adf5e0 0x558573ae5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe420ff9640 0 client.16761 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe4425fd740 1 -- 172.21.15.23:0/4223668921 >> 172.21.15.23:0/4223668921 conn(0x558573ad9f00 msgr2=0x558573adbc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe4425fd740 1 -- 172.21.15.23:0/4223668921 shutdown_connections 2024-03-24T10:44:23.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe4425fd740 1 -- 172.21.15.23:0/4223668921 wait complete. 2024-03-24T10:44:23.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.580+0000 7fe4425fd740 20 client.16761 trim_cache size 0 max 16384 2024-03-24T10:44:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe4425fd740 1 Processor -- start 2024-03-24T10:44:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe4425fd740 1 -- start start 2024-03-24T10:44:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe4425fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adf5e0 0x558573af43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe4425fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adff00 0x558573ada2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe4425fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573ae65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe4425fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5585739cb660 con 0x558573ae8cd0 2024-03-24T10:44:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe4425fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558573adaf00 con 0x558573adf5e0 2024-03-24T10:44:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe4425fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558573afb3a0 con 0x558573adff00 2024-03-24T10:44:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe43dfbb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573ae65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe43dfbb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573ae65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49216/0 (socket says 172.21.15.23:49216) 2024-03-24T10:44:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe43cfb9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adff00 0x558573ada2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe43dfbb640 1 -- 172.21.15.23:0/3405253659 learned_addr learned my addr 172.21.15.23:0/3405253659 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe43d7ba640 1 --2- 172.21.15.23:0/3405253659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adf5e0 0x558573af43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe43dfbb640 1 -- 172.21.15.23:0/3405253659 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adff00 msgr2=0x558573ada2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe43dfbb640 1 --2- 172.21.15.23:0/3405253659 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adff00 0x558573ada2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe43dfbb640 1 -- 172.21.15.23:0/3405253659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adf5e0 msgr2=0x558573af43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe43dfbb640 1 --2- 172.21.15.23:0/3405253659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adf5e0 0x558573af43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.596+0000 7fe43dfbb640 1 -- 172.21.15.23:0/3405253659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585739e0810 con 0x558573ae8cd0 2024-03-24T10:44:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.597+0000 7fe43dfbb640 1 --2- 172.21.15.23:0/3405253659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573ae65f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe430004af0 tx=0x7fe43009b1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1039eb21d410b020 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.597+0000 7fe422ffd640 1 -- 172.21.15.23:0/3405253659 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43009a9c0 con 0x558573ae8cd0 2024-03-24T10:44:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.597+0000 7fe422ffd640 1 -- 172.21.15.23:0/3405253659 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4300a3800 con 0x558573ae8cd0 2024-03-24T10:44:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.597+0000 7fe422ffd640 1 -- 172.21.15.23:0/3405253659 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43009a4b0 con 0x558573ae8cd0 2024-03-24T10:44:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.597+0000 7fe4425fd740 1 -- 172.21.15.23:0/3405253659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 msgr2=0x558573ae65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.597+0000 7fe4425fd740 1 --2- 172.21.15.23:0/3405253659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573ae65f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe430004af0 tx=0x7fe43009b1e0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.597+0000 7fe4425fd740 1 -- 172.21.15.23:0/3405253659 shutdown_connections 2024-03-24T10:44:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.597+0000 7fe4425fd740 1 --2- 172.21.15.23:0/3405253659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573ae8cd0 0x558573ae65f0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.597+0000 7fe4425fd740 1 --2- 172.21.15.23:0/3405253659 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573adff00 0x558573ada2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.597+0000 7fe4425fd740 1 --2- 172.21.15.23:0/3405253659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adf5e0 0x558573af43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.597+0000 7fe4425fd740 1 -- 172.21.15.23:0/3405253659 >> 172.21.15.23:0/3405253659 conn(0x558573ad9f00 msgr2=0x558573ad8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.597+0000 7fe4425fd740 1 -- 172.21.15.23:0/3405253659 shutdown_connections 2024-03-24T10:44:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.597+0000 7fe4425fd740 1 -- 172.21.15.23:0/3405253659 wait complete. 2024-03-24T10:44:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe4425fd740 1 -- start start 2024-03-24T10:44:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe4425fd740 1 Processor -- start 2024-03-24T10:44:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe4425fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573adf5e0 0x558573ae2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe4425fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adff00 0x558573ae2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe4425fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573ae8cd0 0x558573ae33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe4425fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558573b5dfb0 con 0x558573adf5e0 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe4425fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558573b5e0f0 con 0x558573adff00 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe4425fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558573b5e280 con 0x558573ae8cd0 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe4425fd740 10 client.0 _ll_register_callbacks cb 0x7fe43009aee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe43d7ba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573adf5e0 0x558573ae2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe43d7ba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573adf5e0 0x558573ae2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49224/0 (socket says 172.21.15.23:49224) 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe43d7ba640 1 -- 172.21.15.23:0/3978969945 learned_addr learned my addr 172.21.15.23:0/3978969945 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe43cfb9640 1 --2- 172.21.15.23:0/3978969945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adff00 0x558573ae2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe43dfbb640 1 --2- 172.21.15.23:0/3978969945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573ae8cd0 0x558573ae33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe43d7ba640 1 -- 172.21.15.23:0/3978969945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573ae8cd0 msgr2=0x558573ae33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe43d7ba640 1 --2- 172.21.15.23:0/3978969945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573ae8cd0 0x558573ae33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe43d7ba640 1 -- 172.21.15.23:0/3978969945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adff00 msgr2=0x558573ae2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe43d7ba640 1 --2- 172.21.15.23:0/3978969945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adff00 0x558573ae2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.598+0000 7fe43d7ba640 1 -- 172.21.15.23:0/3978969945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558573a44310 con 0x558573adf5e0 2024-03-24T10:44:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe43d7ba640 1 --2- 172.21.15.23:0/3978969945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573adf5e0 0x558573ae2940 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe43401b000 tx=0x7fe434005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e863f0825100355 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe420ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe420ff9640 1 -- 172.21.15.23:0/3978969945 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe434009b50 con 0x558573adf5e0 2024-03-24T10:44:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe4425fd740 1 -- 172.21.15.23:0/3978969945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe43400fc00 con 0x558573adf5e0 2024-03-24T10:44:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe420ff9640 1 -- 172.21.15.23:0/3978969945 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fe434011030 con 0x558573adf5e0 2024-03-24T10:44:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe420ff9640 1 -- 172.21.15.23:0/3978969945 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe434009310 con 0x558573adf5e0 2024-03-24T10:44:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe4425fd740 10 client.16767 fetch_fsmap learned FSMap version 26 2024-03-24T10:44:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe4425fd740 1 -- 172.21.15.23:0/3978969945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558573a065a0 con 0x558573adf5e0 2024-03-24T10:44:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe420ff9640 1 -- 172.21.15.23:0/3978969945 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe43400fc00 con 0x558573adf5e0 2024-03-24T10:44:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe420ff9640 1 -- 172.21.15.23:0/3978969945 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) ==== 8342+0+0 (secure 0 0 0) 0x7fe4340050b0 con 0x558573adf5e0 2024-03-24T10:44:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe4425fd740 10 client.16767 fetch_fsmap finished waiting for FSMap version 26 2024-03-24T10:44:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe4425fd740 10 client.16767 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe4425fd740 20 client.16767 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe4425fd740 1 --2- 172.21.15.23:0/3978969945 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558573b5fa80 0x558573b61ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe4425fd740 4 client.16767 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:44:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.599+0000 7fe4425fd740 1 -- 172.21.15.23:0/3978969945 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5585737961b0 con 0x558573b5fa80 2024-03-24T10:44:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.600+0000 7fe43cfb9640 1 --2- 172.21.15.23:0/3978969945 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558573b5fa80 0x558573b61ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.600+0000 7fe420ff9640 10 client.16767 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.600+0000 7fe43cfb9640 1 --2- 172.21.15.23:0/3978969945 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558573b5fa80 0x558573b61ea0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46a928947cab0781 server_cookie=14fdd920f77cfdb9 in_seq=0 out_seq=0 2024-03-24T10:44:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe420ff9640 1 -- 172.21.15.23:0/3978969945 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5585737961b0 con 0x558573b5fa80 2024-03-24T10:44:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe420ff9640 10 client.16767 handle_command_reply: tid=0 2024-03-24T10:44:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe4425fd740 1 client.16767 shutdown 2024-03-24T10:44:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe4425fd740 1 -- 172.21.15.23:0/3978969945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573adf5e0 msgr2=0x558573ae2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe4425fd740 1 --2- 172.21.15.23:0/3978969945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573adf5e0 0x558573ae2940 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe43401b000 tx=0x7fe434005fa0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe4425fd740 1 -- 172.21.15.23:0/3978969945 shutdown_connections 2024-03-24T10:44:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe4425fd740 1 --2- 172.21.15.23:0/3978969945 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558573b5fa80 0x558573b61ea0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe4425fd740 1 --2- 172.21.15.23:0/3978969945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558573ae8cd0 0x558573ae33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe4425fd740 1 --2- 172.21.15.23:0/3978969945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558573adff00 0x558573ae2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe420ff9640 0 client.16767 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe4425fd740 1 --2- 172.21.15.23:0/3978969945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558573adf5e0 0x558573ae2940 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe4425fd740 1 -- 172.21.15.23:0/3978969945 >> 172.21.15.23:0/3978969945 conn(0x558573ad9f00 msgr2=0x558573ad8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:23.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe4425fd740 1 -- 172.21.15.23:0/3978969945 shutdown_connections 2024-03-24T10:44:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe4425fd740 1 -- 172.21.15.23:0/3978969945 wait complete. 2024-03-24T10:44:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.609+0000 7fe4425fd740 20 client.16767 trim_cache size 0 max 16384 2024-03-24T10:44:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:44:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:44:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:44:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:44:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.610+0000 7fe43fa1e640 1 -- 172.21.15.23:0/3308088312 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe4240a22d0 msgr2=0x7fe4240a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.610+0000 7fe43fa1e640 1 --2- 172.21.15.23:0/3308088312 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe4240a22d0 0x7fe4240a4790 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe428005ab0 tx=0x7fe42802c640 comp rx=0 tx=0).stop 2024-03-24T10:44:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.610+0000 7fe43fa1e640 1 -- 172.21.15.23:0/3308088312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4381076f0 msgr2=0x7fe438074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.610+0000 7fe43fa1e640 1 --2- 172.21.15.23:0/3308088312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4381076f0 0x7fe438074490 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe43000eb10 tx=0x7fe43000c2d0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.610+0000 7fe43fa1e640 1 -- 172.21.15.23:0/3308088312 shutdown_connections 2024-03-24T10:44:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.610+0000 7fe43fa1e640 1 --2- 172.21.15.23:0/3308088312 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe4240a22d0 0x7fe4240a4790 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.610+0000 7fe43fa1e640 1 --2- 172.21.15.23:0/3308088312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4381076f0 0x7fe438074490 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.610+0000 7fe43fa1e640 1 --2- 172.21.15.23:0/3308088312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4381020a0 0x7fe438073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.610+0000 7fe43fa1e640 1 --2- 172.21.15.23:0/3308088312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4381016e0 0x7fe438073a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.610+0000 7fe43fa1e640 1 -- 172.21.15.23:0/3308088312 >> 172.21.15.23:0/3308088312 conn(0x7fe4380fd210 msgr2=0x7fe4380fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.610+0000 7fe43fa1e640 1 -- 172.21.15.23:0/3308088312 shutdown_connections 2024-03-24T10:44:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:23.610+0000 7fe43fa1e640 1 -- 172.21.15.23:0/3308088312 wait complete. 2024-03-24T10:44:24.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 364558 11.93 MB/sec execute 41 sec latency 17.625 ms 2024-03-24T10:44:24.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:24 smithi023 ceph-mon[32466]: pgmap v132: 97 pgs: 97 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-24T10:44:25.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:24 smithi117 ceph-mon[36368]: pgmap v132: 97 pgs: 97 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-24T10:44:25.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:24 smithi116 ceph-mon[36248]: pgmap v132: 97 pgs: 97 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-24T10:44:25.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 366318 11.86 MB/sec execute 42 sec latency 17.598 ms 2024-03-24T10:44:26.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 369203 12.00 MB/sec execute 43 sec latency 8.182 ms 2024-03-24T10:44:26.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:26 smithi023 ceph-mon[32466]: pgmap v133: 97 pgs: 97 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-24T10:44:26.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:26 smithi023 ceph-mon[32466]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:44:27.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:26 smithi117 ceph-mon[36368]: pgmap v133: 97 pgs: 97 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-24T10:44:27.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:26 smithi117 ceph-mon[36368]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:44:27.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:26 smithi116 ceph-mon[36248]: pgmap v133: 97 pgs: 97 active+clean; 203 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-24T10:44:27.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:26 smithi116 ceph-mon[36248]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:44:27.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 371279 11.89 MB/sec execute 44 sec latency 28.226 ms 2024-03-24T10:44:28.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 372966 11.90 MB/sec execute 45 sec latency 56.181 ms 2024-03-24T10:44:28.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:28 smithi023 ceph-mon[32466]: pgmap v134: 97 pgs: 97 active+clean; 209 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.7 MiB/s wr, 301 op/s 2024-03-24T10:44:29.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:28 smithi117 ceph-mon[36368]: pgmap v134: 97 pgs: 97 active+clean; 209 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.7 MiB/s wr, 301 op/s 2024-03-24T10:44:29.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:28 smithi116 ceph-mon[36248]: pgmap v134: 97 pgs: 97 active+clean; 209 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.7 MiB/s wr, 301 op/s 2024-03-24T10:44:29.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 375372 11.91 MB/sec execute 46 sec latency 44.396 ms 2024-03-24T10:44:30.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 377162 11.84 MB/sec execute 47 sec latency 18.308 ms 2024-03-24T10:44:30.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:30 smithi023 ceph-mon[32466]: pgmap v135: 97 pgs: 97 active+clean; 209 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 208 op/s 2024-03-24T10:44:31.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:30 smithi117 ceph-mon[36368]: pgmap v135: 97 pgs: 97 active+clean; 209 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 208 op/s 2024-03-24T10:44:31.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:30 smithi116 ceph-mon[36248]: pgmap v135: 97 pgs: 97 active+clean; 209 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 208 op/s 2024-03-24T10:44:31.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 379986 11.95 MB/sec execute 48 sec latency 9.490 ms 2024-03-24T10:44:31.231 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:44:31.231 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:44:31.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.338+0000 7f834593a640 1 -- 172.21.15.23:0/3096761252 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83400692d0 msgr2=0x7f834010e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.338+0000 7f834593a640 1 --2- 172.21.15.23:0/3096761252 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83400692d0 0x7f834010e570 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f8338009cd0 tx=0x7f833802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.338+0000 7f834593a640 1 -- 172.21.15.23:0/3096761252 shutdown_connections 2024-03-24T10:44:31.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.338+0000 7f834593a640 1 --2- 172.21.15.23:0/3096761252 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83400692d0 0x7f834010e570 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.338+0000 7f834593a640 1 --2- 172.21.15.23:0/3096761252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8340068910 0x7f8340068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.338+0000 7f834593a640 1 --2- 172.21.15.23:0/3096761252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8340107760 0x7f8340107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.338+0000 7f834593a640 1 -- 172.21.15.23:0/3096761252 >> 172.21.15.23:0/3096761252 conn(0x7f8340077710 msgr2=0x7f8340077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.338+0000 7f834593a640 1 -- 172.21.15.23:0/3096761252 shutdown_connections 2024-03-24T10:44:31.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.338+0000 7f834593a640 1 -- 172.21.15.23:0/3096761252 wait complete. 2024-03-24T10:44:31.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.339+0000 7f834593a640 1 Processor -- start 2024-03-24T10:44:31.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.339+0000 7f834593a640 1 -- start start 2024-03-24T10:44:31.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.339+0000 7f834593a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8340068910 0x7f834019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.339+0000 7f834593a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83400692d0 0x7f834019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.339+0000 7f834593a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8340107760 0x7f834019f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.339+0000 7f834593a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f834019ffb0 con 0x7f83400692d0 2024-03-24T10:44:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.339+0000 7f834593a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f83401a0120 con 0x7f8340068910 2024-03-24T10:44:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f834593a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f83401a02e0 con 0x7f8340107760 2024-03-24T10:44:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f83367fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83400692d0 0x7f834019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f833effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8340068910 0x7f834019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f83367fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83400692d0 0x7f834019efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42398/0 (socket says 172.21.15.23:42398) 2024-03-24T10:44:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f83367fc640 1 -- 172.21.15.23:0/3293621251 learned_addr learned my addr 172.21.15.23:0/3293621251 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f833f7fe640 1 --2- 172.21.15.23:0/3293621251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8340107760 0x7f834019f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f83367fc640 1 -- 172.21.15.23:0/3293621251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8340107760 msgr2=0x7f834019f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f83367fc640 1 --2- 172.21.15.23:0/3293621251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8340107760 0x7f834019f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f83367fc640 1 -- 172.21.15.23:0/3293621251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8340068910 msgr2=0x7f834019eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f83367fc640 1 --2- 172.21.15.23:0/3293621251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8340068910 0x7f834019eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f83367fc640 1 -- 172.21.15.23:0/3293621251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8338009900 con 0x7f83400692d0 2024-03-24T10:44:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f833effd640 1 --2- 172.21.15.23:0/3293621251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8340068910 0x7f834019eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f83367fc640 1 --2- 172.21.15.23:0/3293621251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83400692d0 0x7f834019efe0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f832800e9f0 tx=0x7f832800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f8344938640 1 -- 172.21.15.23:0/3293621251 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8328009720 con 0x7f83400692d0 2024-03-24T10:44:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f8344938640 1 -- 172.21.15.23:0/3293621251 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f832800cd80 con 0x7f83400692d0 2024-03-24T10:44:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f834593a640 1 -- 172.21.15.23:0/3293621251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83401a0560 con 0x7f83400692d0 2024-03-24T10:44:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f8344938640 1 -- 172.21.15.23:0/3293621251 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8328010690 con 0x7f83400692d0 2024-03-24T10:44:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.340+0000 7f834593a640 1 -- 172.21.15.23:0/3293621251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83401a6ba0 con 0x7f83400692d0 2024-03-24T10:44:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.343+0000 7f8344938640 1 -- 172.21.15.23:0/3293621251 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f83280046c0 con 0x7f83400692d0 2024-03-24T10:44:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.343+0000 7f8344938640 1 --2- 172.21.15.23:0/3293621251 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f832c0a2230 0x7f832c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.343+0000 7f83485eb740 1 Processor -- start 2024-03-24T10:44:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.343+0000 7f833effd640 1 --2- 172.21.15.23:0/3293621251 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f832c0a2230 0x7f832c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.343+0000 7f8344938640 1 -- 172.21.15.23:0/3293621251 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f8328014070 con 0x7f83400692d0 2024-03-24T10:44:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.343+0000 7f83485eb740 1 -- start start 2024-03-24T10:44:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83485eb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac68fb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83485eb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac68fbaf0 0x556ac6904190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83485eb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac69046d0 0x556ac6906ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83485eb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556ac681ff30 con 0x556ac68fb1d0 2024-03-24T10:44:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83485eb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556ac6869810 con 0x556ac68fbaf0 2024-03-24T10:44:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83485eb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556ac6866d40 con 0x556ac69046d0 2024-03-24T10:44:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83367fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac68fbaf0 0x556ac6904190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83367fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac68fbaf0 0x556ac6904190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:40480/0 (socket says 172.21.15.23:40480) 2024-03-24T10:44:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83367fc640 1 -- 172.21.15.23:0/301999038 learned_addr learned my addr 172.21.15.23:0/301999038 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f833effd640 1 --2- 172.21.15.23:0/301999038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac68fb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f833f7fe640 1 --2- 172.21.15.23:0/301999038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac69046d0 0x556ac6906ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f833effd640 1 --2- 172.21.15.23:0/3293621251 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f832c0a2230 0x7f832c0a46f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8330009fc0 tx=0x7f8330005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83367fc640 1 -- 172.21.15.23:0/301999038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac69046d0 msgr2=0x556ac6906ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83367fc640 1 --2- 172.21.15.23:0/301999038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac69046d0 0x556ac6906ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83367fc640 1 -- 172.21.15.23:0/301999038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 msgr2=0x556ac68fb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83367fc640 1 --2- 172.21.15.23:0/301999038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac68fb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83367fc640 1 -- 172.21.15.23:0/301999038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ac6823ea0 con 0x556ac68fbaf0 2024-03-24T10:44:31.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.344+0000 7f83367fc640 1 --2- 172.21.15.23:0/301999038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac68fbaf0 0x556ac6904190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f83280977d0 tx=0x7f832809b500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4e4ede5cfebeb2b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.345+0000 7f8335ffb640 1 -- 172.21.15.23:0/301999038 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832809d030 con 0x556ac68fbaf0 2024-03-24T10:44:31.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.345+0000 7f8335ffb640 1 -- 172.21.15.23:0/301999038 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83280a35a0 con 0x556ac68fbaf0 2024-03-24T10:44:31.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.345+0000 7f8335ffb640 1 -- 172.21.15.23:0/301999038 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83280a3370 con 0x556ac68fbaf0 2024-03-24T10:44:31.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.345+0000 7f83485eb740 1 -- 172.21.15.23:0/301999038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac68fbaf0 msgr2=0x556ac6904190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.345+0000 7f83485eb740 1 --2- 172.21.15.23:0/301999038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac68fbaf0 0x556ac6904190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f83280977d0 tx=0x7f832809b500 comp rx=0 tx=0).stop 2024-03-24T10:44:31.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.345+0000 7f83485eb740 1 -- 172.21.15.23:0/301999038 shutdown_connections 2024-03-24T10:44:31.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.345+0000 7f83485eb740 1 --2- 172.21.15.23:0/301999038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac69046d0 0x556ac6906ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.345+0000 7f83485eb740 1 --2- 172.21.15.23:0/301999038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac68fbaf0 0x556ac6904190 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.345+0000 7f83485eb740 1 --2- 172.21.15.23:0/301999038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac68fb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.345+0000 7f83485eb740 1 -- 172.21.15.23:0/301999038 >> 172.21.15.23:0/301999038 conn(0x556ac68f5900 msgr2=0x556ac68f5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.345+0000 7f83485eb740 1 -- 172.21.15.23:0/301999038 shutdown_connections 2024-03-24T10:44:31.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.345+0000 7f83485eb740 1 -- 172.21.15.23:0/301999038 wait complete. 2024-03-24T10:44:31.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.346+0000 7f83485eb740 1 -- start start 2024-03-24T10:44:31.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.346+0000 7f83485eb740 1 Processor -- start 2024-03-24T10:44:31.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.346+0000 7f83485eb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac6901780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.346+0000 7f83485eb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac6901cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.346+0000 7f83485eb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac69046d0 0x556ac6902200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.346+0000 7f83485eb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556ac6862520 con 0x556ac68fb1d0 2024-03-24T10:44:31.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.346+0000 7f83485eb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556ac6535430 con 0x556ac69046d0 2024-03-24T10:44:31.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.346+0000 7f83485eb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556ac64f1e00 con 0x556ac68fbaf0 2024-03-24T10:44:31.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.347+0000 7f83485eb740 10 client.0 _ll_register_callbacks cb 0x556ac6800d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:31.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.347+0000 7f833effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac6901780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.347+0000 7f833effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac6901780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42428/0 (socket says 172.21.15.23:42428) 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.347+0000 7f833effd640 1 -- 172.21.15.23:0/4030715465 learned_addr learned my addr 172.21.15.23:0/4030715465 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.347+0000 7f83367fc640 1 --2- 172.21.15.23:0/4030715465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac6901cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.347+0000 7f833f7fe640 1 --2- 172.21.15.23:0/4030715465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac69046d0 0x556ac6902200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.347+0000 7f83367fc640 1 -- 172.21.15.23:0/4030715465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac69046d0 msgr2=0x556ac6902200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.347+0000 7f83367fc640 1 --2- 172.21.15.23:0/4030715465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac69046d0 0x556ac6902200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.347+0000 7f83367fc640 1 -- 172.21.15.23:0/4030715465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 msgr2=0x556ac6901780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.347+0000 7f83367fc640 1 --2- 172.21.15.23:0/4030715465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac6901780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.347+0000 7f83367fc640 1 -- 172.21.15.23:0/4030715465 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83401a6ba0 con 0x556ac68fbaf0 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.347+0000 7f833effd640 1 --2- 172.21.15.23:0/4030715465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac6901780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.347+0000 7f83367fc640 1 --2- 172.21.15.23:0/4030715465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac6901cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x556ac6907110 tx=0x7f8328004b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7069e096ac8d41b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.347+0000 7f831effd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f831effd640 1 -- 172.21.15.23:0/4030715465 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83280a9070 con 0x556ac68fbaf0 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f831effd640 1 -- 172.21.15.23:0/4030715465 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f832809e070 con 0x556ac68fbaf0 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f83485eb740 1 -- 172.21.15.23:0/4030715465 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556ac65617e0 con 0x556ac68fbaf0 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f831effd640 1 -- 172.21.15.23:0/4030715465 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f832809d040 con 0x556ac68fbaf0 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f83485eb740 10 client.26629 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f83485eb740 1 -- 172.21.15.23:0/4030715465 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556ac6860ba0 con 0x556ac68fbaf0 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f831effd640 1 -- 172.21.15.23:0/4030715465 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556ac65617e0 con 0x556ac68fbaf0 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f831effd640 1 -- 172.21.15.23:0/4030715465 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f83280ac070 con 0x556ac68fbaf0 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f83485eb740 10 client.26629 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f83485eb740 10 client.26629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f83485eb740 20 client.26629 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f83485eb740 1 --2- 172.21.15.23:0/4030715465 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x556ac6914820 0x556ac6916c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f83485eb740 4 client.26629 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f83485eb740 1 -- 172.21.15.23:0/4030715465 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ac659e720 con 0x556ac6914820 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.348+0000 7f833effd640 1 --2- 172.21.15.23:0/4030715465 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x556ac6914820 0x556ac6916c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.349+0000 7f831effd640 10 client.26629 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.349+0000 7f833effd640 1 --2- 172.21.15.23:0/4030715465 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x556ac6914820 0x556ac6916c30 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4fa71aa98927553 server_cookie=afde917ebacc8ba7 in_seq=0 out_seq=0 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f831effd640 1 -- 172.21.15.23:0/4030715465 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556ac659e720 con 0x556ac6914820 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f831effd640 10 client.26629 handle_command_reply: tid=0 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f83485eb740 1 client.26629 shutdown 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f83485eb740 1 -- 172.21.15.23:0/4030715465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 msgr2=0x556ac6901cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f83485eb740 1 --2- 172.21.15.23:0/4030715465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac6901cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x556ac6907110 tx=0x7f8328004b30 comp rx=0 tx=0).stop 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f83485eb740 1 -- 172.21.15.23:0/4030715465 shutdown_connections 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f83485eb740 1 --2- 172.21.15.23:0/4030715465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac69046d0 0x556ac6902200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f83485eb740 1 --2- 172.21.15.23:0/4030715465 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x556ac6914820 0x556ac6916c30 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f83485eb740 1 --2- 172.21.15.23:0/4030715465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac6901cc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f83485eb740 1 --2- 172.21.15.23:0/4030715465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac6901780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f83485eb740 1 -- 172.21.15.23:0/4030715465 >> 172.21.15.23:0/4030715465 conn(0x556ac68f5900 msgr2=0x556ac68f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f831effd640 0 client.26629 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f83485eb740 1 -- 172.21.15.23:0/4030715465 shutdown_connections 2024-03-24T10:44:31.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f83485eb740 1 -- 172.21.15.23:0/4030715465 wait complete. 2024-03-24T10:44:31.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.350+0000 7f83485eb740 20 client.26629 trim_cache size 0 max 16384 2024-03-24T10:44:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83485eb740 1 Processor -- start 2024-03-24T10:44:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83485eb740 1 -- start start 2024-03-24T10:44:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83485eb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac6903eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83485eb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac690ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83485eb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac69046d0 0x556ac69008e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83485eb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556ac67e7430 con 0x556ac68fb1d0 2024-03-24T10:44:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83485eb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556ac68f5430 con 0x556ac69046d0 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83485eb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556ac68f11d0 con 0x556ac68fbaf0 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f833effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac6903eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83367fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac690ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83367fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac690ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39494/0 (socket says 172.21.15.23:39494) 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83367fc640 1 -- 172.21.15.23:0/2831652439 learned_addr learned my addr 172.21.15.23:0/2831652439 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f833f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac69046d0 0x556ac69008e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83367fc640 1 -- 172.21.15.23:0/2831652439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac69046d0 msgr2=0x556ac69008e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83367fc640 1 --2- 172.21.15.23:0/2831652439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac69046d0 0x556ac69008e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83367fc640 1 -- 172.21.15.23:0/2831652439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 msgr2=0x556ac6903eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83367fc640 1 --2- 172.21.15.23:0/2831652439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac6903eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f83367fc640 1 -- 172.21.15.23:0/2831652439 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ac67fcb20 con 0x556ac68fbaf0 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.371+0000 7f833effd640 1 --2- 172.21.15.23:0/2831652439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac6903eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.372+0000 7f83367fc640 1 --2- 172.21.15.23:0/2831652439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac690ff60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f83280cbc10 tx=0x7f832809f0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3d62c7694ad4a01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.372+0000 7f83357fa640 1 -- 172.21.15.23:0/2831652439 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83280a9020 con 0x556ac68fbaf0 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.372+0000 7f83357fa640 1 -- 172.21.15.23:0/2831652439 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83280b9300 con 0x556ac68fbaf0 2024-03-24T10:44:31.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.372+0000 7f83357fa640 1 -- 172.21.15.23:0/2831652439 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832809d030 con 0x556ac68fbaf0 2024-03-24T10:44:31.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.373+0000 7f83485eb740 1 -- 172.21.15.23:0/2831652439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 msgr2=0x556ac690ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.373+0000 7f83485eb740 1 --2- 172.21.15.23:0/2831652439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac690ff60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f83280cbc10 tx=0x7f832809f0d0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.373+0000 7f83485eb740 1 -- 172.21.15.23:0/2831652439 shutdown_connections 2024-03-24T10:44:31.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.373+0000 7f83485eb740 1 --2- 172.21.15.23:0/2831652439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac69046d0 0x556ac69008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.373+0000 7f83485eb740 1 --2- 172.21.15.23:0/2831652439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac690ff60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.373+0000 7f83485eb740 1 --2- 172.21.15.23:0/2831652439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac68fb1d0 0x556ac6903eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.373+0000 7f83485eb740 1 -- 172.21.15.23:0/2831652439 >> 172.21.15.23:0/2831652439 conn(0x556ac68f5900 msgr2=0x556ac68f42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.373+0000 7f83485eb740 1 -- 172.21.15.23:0/2831652439 shutdown_connections 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.373+0000 7f83485eb740 1 -- 172.21.15.23:0/2831652439 wait complete. 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.373+0000 7f83485eb740 1 -- start start 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.373+0000 7f83485eb740 1 Processor -- start 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f83485eb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac68fb1d0 0x556ac690c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f83485eb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac68fdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f83485eb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac69046d0 0x556ac68fe320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f83485eb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556ac690cc60 con 0x556ac69046d0 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f83485eb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556ac690cdd0 con 0x556ac68fb1d0 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f83485eb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556ac68fe860 con 0x556ac68fbaf0 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f83485eb740 10 client.0 _ll_register_callbacks cb 0x7f8330009db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f833f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac69046d0 0x556ac68fe320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f83367fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac68fdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f833f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac69046d0 0x556ac68fe320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42434/0 (socket says 172.21.15.23:42434) 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f833f7fe640 1 -- 172.21.15.23:0/2155506710 learned_addr learned my addr 172.21.15.23:0/2155506710 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f833effd640 1 --2- 172.21.15.23:0/2155506710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac68fb1d0 0x556ac690c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f833f7fe640 1 -- 172.21.15.23:0/2155506710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 msgr2=0x556ac68fdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f833f7fe640 1 --2- 172.21.15.23:0/2155506710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac68fdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f833f7fe640 1 -- 172.21.15.23:0/2155506710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac68fb1d0 msgr2=0x556ac690c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f833f7fe640 1 --2- 172.21.15.23:0/2155506710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac68fb1d0 0x556ac690c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f833f7fe640 1 -- 172.21.15.23:0/2155506710 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8330017050 con 0x556ac69046d0 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f83367fc640 1 --2- 172.21.15.23:0/2155506710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac68fdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f833f7fe640 1 --2- 172.21.15.23:0/2155506710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac69046d0 0x556ac68fe320 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f8338009cd0 tx=0x7f83380346e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10d3b6ed33f6bd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.374+0000 7f831effd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f831effd640 1 -- 172.21.15.23:0/2155506710 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338045070 con 0x556ac69046d0 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f831effd640 1 -- 172.21.15.23:0/2155506710 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f83380326a0 con 0x556ac69046d0 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f83485eb740 1 -- 172.21.15.23:0/2155506710 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83280a35a0 con 0x556ac69046d0 2024-03-24T10:44:31.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f831effd640 1 -- 172.21.15.23:0/2155506710 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8338032860 con 0x556ac69046d0 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f83485eb740 10 client.16785 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f83485eb740 1 -- 172.21.15.23:0/2155506710 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556ac6860ba0 con 0x556ac69046d0 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f831effd640 1 -- 172.21.15.23:0/2155506710 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8338056020 con 0x556ac69046d0 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f831effd640 1 -- 172.21.15.23:0/2155506710 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f83380044b0 con 0x556ac69046d0 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f83485eb740 10 client.16785 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f83485eb740 10 client.16785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f83485eb740 20 client.16785 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f83485eb740 1 --2- 172.21.15.23:0/2155506710 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x556ac697ae50 0x556ac697d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f83485eb740 4 client.16785 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f83485eb740 1 -- 172.21.15.23:0/2155506710 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556ac65ed890 con 0x556ac697ae50 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.375+0000 7f833effd640 1 --2- 172.21.15.23:0/2155506710 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x556ac697ae50 0x556ac697d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f833effd640 1 --2- 172.21.15.23:0/2155506710 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x556ac697ae50 0x556ac697d270 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eccff9f8e10c7206 server_cookie=9699d06dbe20efd2 in_seq=0 out_seq=0 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f831effd640 10 client.16785 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f831effd640 1 -- 172.21.15.23:0/2155506710 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556ac65ed890 con 0x556ac697ae50 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f831effd640 10 client.16785 handle_command_reply: tid=0 2024-03-24T10:44:31.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f83485eb740 1 client.16785 shutdown 2024-03-24T10:44:31.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f83485eb740 1 -- 172.21.15.23:0/2155506710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac69046d0 msgr2=0x556ac68fe320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f83485eb740 1 --2- 172.21.15.23:0/2155506710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac69046d0 0x556ac68fe320 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f8338009cd0 tx=0x7f83380346e0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f83485eb740 1 -- 172.21.15.23:0/2155506710 shutdown_connections 2024-03-24T10:44:31.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f83485eb740 1 --2- 172.21.15.23:0/2155506710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556ac69046d0 0x556ac68fe320 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f83485eb740 1 --2- 172.21.15.23:0/2155506710 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x556ac697ae50 0x556ac697d270 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f83485eb740 1 --2- 172.21.15.23:0/2155506710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ac68fbaf0 0x556ac68fdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f83485eb740 1 --2- 172.21.15.23:0/2155506710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ac68fb1d0 0x556ac690c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f83485eb740 1 -- 172.21.15.23:0/2155506710 >> 172.21.15.23:0/2155506710 conn(0x556ac68f5900 msgr2=0x556ac68f4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f831effd640 0 client.16785 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:31.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f83485eb740 1 -- 172.21.15.23:0/2155506710 shutdown_connections 2024-03-24T10:44:31.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.376+0000 7f83485eb740 1 -- 172.21.15.23:0/2155506710 wait complete. 2024-03-24T10:44:31.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.377+0000 7f83485eb740 20 client.16785 trim_cache size 0 max 16384 2024-03-24T10:44:31.382 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:44:31.382 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:44:31.382 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:44:31.382 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:44:31.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.377+0000 7f834593a640 1 -- 172.21.15.23:0/3293621251 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f832c0a2230 msgr2=0x7f832c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.377+0000 7f834593a640 1 --2- 172.21.15.23:0/3293621251 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f832c0a2230 0x7f832c0a46f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8330009fc0 tx=0x7f8330005f90 comp rx=0 tx=0).stop 2024-03-24T10:44:31.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.377+0000 7f834593a640 1 -- 172.21.15.23:0/3293621251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83400692d0 msgr2=0x7f834019efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.377+0000 7f834593a640 1 --2- 172.21.15.23:0/3293621251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83400692d0 0x7f834019efe0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f832800e9f0 tx=0x7f832800edf0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.377+0000 7f834593a640 1 -- 172.21.15.23:0/3293621251 shutdown_connections 2024-03-24T10:44:31.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.377+0000 7f834593a640 1 --2- 172.21.15.23:0/3293621251 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f832c0a2230 0x7f832c0a46f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.377+0000 7f834593a640 1 --2- 172.21.15.23:0/3293621251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8340107760 0x7f834019f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.377+0000 7f834593a640 1 --2- 172.21.15.23:0/3293621251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83400692d0 0x7f834019efe0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.377+0000 7f834593a640 1 --2- 172.21.15.23:0/3293621251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8340068910 0x7f834019eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.377+0000 7f834593a640 1 -- 172.21.15.23:0/3293621251 >> 172.21.15.23:0/3293621251 conn(0x7f8340077710 msgr2=0x7f83401031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.377+0000 7f834593a640 1 -- 172.21.15.23:0/3293621251 shutdown_connections 2024-03-24T10:44:31.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.377+0000 7f834593a640 1 -- 172.21.15.23:0/3293621251 wait complete. 2024-03-24T10:44:31.391 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:44:31.391 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:44:31.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.503+0000 7f1e6988d640 1 -- 172.21.15.23:0/4151329744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e64106fd0 msgr2=0x7f1e64100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.503+0000 7f1e6988d640 1 --2- 172.21.15.23:0/4151329744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e64106fd0 0x7f1e64100610 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009900 tx=0x7f1e4c02bcd0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.503+0000 7f1e6988d640 1 -- 172.21.15.23:0/4151329744 shutdown_connections 2024-03-24T10:44:31.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.503+0000 7f1e6988d640 1 --2- 172.21.15.23:0/4151329744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e64100d40 0x7f1e6410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.503+0000 7f1e6988d640 1 --2- 172.21.15.23:0/4151329744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e64106fd0 0x7f1e64100610 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.503+0000 7f1e6988d640 1 --2- 172.21.15.23:0/4151329744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e641066b0 0x7f1e64106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.503+0000 7f1e6988d640 1 -- 172.21.15.23:0/4151329744 >> 172.21.15.23:0/4151329744 conn(0x7f1e64077710 msgr2=0x7f1e64077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.503+0000 7f1e6988d640 1 -- 172.21.15.23:0/4151329744 shutdown_connections 2024-03-24T10:44:31.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.503+0000 7f1e6988d640 1 -- 172.21.15.23:0/4151329744 wait complete. 2024-03-24T10:44:31.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.504+0000 7f1e6988d640 1 Processor -- start 2024-03-24T10:44:31.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.504+0000 7f1e6988d640 1 -- start start 2024-03-24T10:44:31.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.504+0000 7f1e6988d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e64100d40 0x7f1e64197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.504+0000 7f1e6988d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e641066b0 0x7f1e64198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.504+0000 7f1e6988d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e64106fd0 0x7f1e6419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.504+0000 7f1e6988d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6419ccb0 con 0x7f1e641066b0 2024-03-24T10:44:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.504+0000 7f1e6988d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6419ce20 con 0x7f1e64106fd0 2024-03-24T10:44:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.504+0000 7f1e6988d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6419cfe0 con 0x7f1e64100d40 2024-03-24T10:44:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.504+0000 7f1e5bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e641066b0 0x7f1e64198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.504+0000 7f1e5bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e641066b0 0x7f1e64198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42452/0 (socket says 172.21.15.23:42452) 2024-03-24T10:44:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.504+0000 7f1e5bfff640 1 -- 172.21.15.23:0/1424339254 learned_addr learned my addr 172.21.15.23:0/1424339254 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.504+0000 7f1e62ffd640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e64100d40 0x7f1e64197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.504+0000 7f1e637fe640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e64106fd0 0x7f1e6419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.505+0000 7f1e62ffd640 1 -- 172.21.15.23:0/1424339254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e64106fd0 msgr2=0x7f1e6419c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.505+0000 7f1e62ffd640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e64106fd0 0x7f1e6419c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.505+0000 7f1e62ffd640 1 -- 172.21.15.23:0/1424339254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e641066b0 msgr2=0x7f1e64198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.505+0000 7f1e62ffd640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e641066b0 0x7f1e64198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.505+0000 7f1e62ffd640 1 -- 172.21.15.23:0/1424339254 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e4c009560 con 0x7f1e64100d40 2024-03-24T10:44:31.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.505+0000 7f1e5bfff640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e641066b0 0x7f1e64198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:31.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.505+0000 7f1e62ffd640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e64100d40 0x7f1e64197df0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400e9c0 tx=0x7f1e5400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.505+0000 7f1e6888b640 1 -- 172.21.15.23:0/1424339254 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5400ccc0 con 0x7f1e64100d40 2024-03-24T10:44:31.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.505+0000 7f1e6888b640 1 -- 172.21.15.23:0/1424339254 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e5400ce20 con 0x7f1e64100d40 2024-03-24T10:44:31.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.505+0000 7f1e6988d640 1 -- 172.21.15.23:0/1424339254 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e6419d2c0 con 0x7f1e64100d40 2024-03-24T10:44:31.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.505+0000 7f1e6888b640 1 -- 172.21.15.23:0/1424339254 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54010430 con 0x7f1e64100d40 2024-03-24T10:44:31.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.505+0000 7f1e6988d640 1 -- 172.21.15.23:0/1424339254 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e641a4b40 con 0x7f1e64100d40 2024-03-24T10:44:31.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.506+0000 7f1e6988d640 1 -- 172.21.15.23:0/1424339254 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e64068500 con 0x7f1e64100d40 2024-03-24T10:44:31.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.507+0000 7f1e6888b640 1 -- 172.21.15.23:0/1424339254 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f1e54004000 con 0x7f1e64100d40 2024-03-24T10:44:31.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.508+0000 7f1e6888b640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1e500a2230 0x7f1e500a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.508+0000 7f1e5bfff640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1e500a2230 0x7f1e500a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.508+0000 7f1e6888b640 1 -- 172.21.15.23:0/1424339254 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f1e54014070 con 0x7f1e64100d40 2024-03-24T10:44:31.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.508+0000 7f1e5bfff640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1e500a2230 0x7f1e500a46f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c005ec0 tx=0x7f1e4c005a20 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.511+0000 7f1e6888b640 1 -- 172.21.15.23:0/1424339254 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1e54096fd0 con 0x7f1e64100d40 2024-03-24T10:44:31.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.756+0000 7f1e6988d640 1 -- 172.21.15.23:0/1424339254 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e64199140 con 0x7f1e64100d40 2024-03-24T10:44:31.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.757+0000 7f1e6888b640 1 -- 172.21.15.23:0/1424339254 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11866 (secure 0 0 0) 0x7f1e5409a690 con 0x7f1e64100d40 2024-03-24T10:44:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:44:31.757 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:44:24.924154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24505,24539,24503]},"id":1}]} 2024-03-24T10:44:31.758 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 27 2024-03-24T10:44:31.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.759+0000 7f1e6988d640 1 -- 172.21.15.23:0/1424339254 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1e500a2230 msgr2=0x7f1e500a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.759+0000 7f1e6988d640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1e500a2230 0x7f1e500a46f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c005ec0 tx=0x7f1e4c005a20 comp rx=0 tx=0).stop 2024-03-24T10:44:31.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.759+0000 7f1e6988d640 1 -- 172.21.15.23:0/1424339254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e64100d40 msgr2=0x7f1e64197df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.759+0000 7f1e6988d640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e64100d40 0x7f1e64197df0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400e9c0 tx=0x7f1e5400ee80 comp rx=0 tx=0).stop 2024-03-24T10:44:31.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.760+0000 7f1e6988d640 1 -- 172.21.15.23:0/1424339254 shutdown_connections 2024-03-24T10:44:31.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.760+0000 7f1e6988d640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1e500a2230 0x7f1e500a46f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.760+0000 7f1e6988d640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e64106fd0 0x7f1e6419c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.760+0000 7f1e6988d640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e641066b0 0x7f1e64198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.760+0000 7f1e6988d640 1 --2- 172.21.15.23:0/1424339254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e64100d40 0x7f1e64197df0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.760+0000 7f1e6988d640 1 -- 172.21.15.23:0/1424339254 >> 172.21.15.23:0/1424339254 conn(0x7f1e64077710 msgr2=0x7f1e64068b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.760+0000 7f1e6988d640 1 -- 172.21.15.23:0/1424339254 shutdown_connections 2024-03-24T10:44:31.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.760+0000 7f1e6988d640 1 -- 172.21.15.23:0/1424339254 wait complete. 2024-03-24T10:44:31.774 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:44:31.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.884+0000 7f439c215640 1 -- 172.21.15.23:0/2984674877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4394102060 msgr2=0x7f439410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.884+0000 7f439c215640 1 --2- 172.21.15.23:0/2984674877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4394102060 0x7f439410e570 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f438c009cd0 tx=0x7f438c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.884+0000 7f439c215640 1 -- 172.21.15.23:0/2984674877 shutdown_connections 2024-03-24T10:44:31.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.884+0000 7f439c215640 1 --2- 172.21.15.23:0/2984674877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4394102060 0x7f439410e570 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.884+0000 7f439c215640 1 --2- 172.21.15.23:0/2984674877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43941016a0 0x7f4394101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.884+0000 7f439c215640 1 --2- 172.21.15.23:0/2984674877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43941076b0 0x7f4394107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.884+0000 7f439c215640 1 -- 172.21.15.23:0/2984674877 >> 172.21.15.23:0/2984674877 conn(0x7f43940fd1b0 msgr2=0x7f43940ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.884+0000 7f439c215640 1 -- 172.21.15.23:0/2984674877 shutdown_connections 2024-03-24T10:44:31.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.884+0000 7f439c215640 1 -- 172.21.15.23:0/2984674877 wait complete. 2024-03-24T10:44:31.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.885+0000 7f439c215640 1 Processor -- start 2024-03-24T10:44:31.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.885+0000 7f439c215640 1 -- start start 2024-03-24T10:44:31.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.885+0000 7f439c215640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43941016a0 0x7f4394197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.885+0000 7f439c215640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4394102060 0x7f4394198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.885+0000 7f439c215640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43941076b0 0x7f439419c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.885+0000 7f439c215640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f439419cc00 con 0x7f4394102060 2024-03-24T10:44:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.885+0000 7f439c215640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f439419cd70 con 0x7f43941076b0 2024-03-24T10:44:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.885+0000 7f439c215640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f439419cf30 con 0x7f43941016a0 2024-03-24T10:44:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f4399fb1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43941016a0 0x7f4394197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f43997b0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4394102060 0x7f4394198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f4399fb1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43941016a0 0x7f4394197d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39544/0 (socket says 172.21.15.23:39544) 2024-03-24T10:44:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f4399fb1640 1 -- 172.21.15.23:0/1083473483 learned_addr learned my addr 172.21.15.23:0/1083473483 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f43997b0640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4394102060 0x7f4394198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42472/0 (socket says 172.21.15.23:42472) 2024-03-24T10:44:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f439a7b2640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43941076b0 0x7f439419c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f4399fb1640 1 -- 172.21.15.23:0/1083473483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43941076b0 msgr2=0x7f439419c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f4399fb1640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43941076b0 0x7f439419c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f4399fb1640 1 -- 172.21.15.23:0/1083473483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4394102060 msgr2=0x7f4394198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f4399fb1640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4394102060 0x7f4394198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f4399fb1640 1 -- 172.21.15.23:0/1083473483 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f438c009900 con 0x7f43941016a0 2024-03-24T10:44:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f43997b0640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4394102060 0x7f4394198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f4399fb1640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43941016a0 0x7f4394197d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f439000e9f0 tx=0x7f439000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f438affd640 1 -- 172.21.15.23:0/1083473483 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4390009700 con 0x7f43941016a0 2024-03-24T10:44:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f439c215640 1 -- 172.21.15.23:0/1083473483 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f439419d210 con 0x7f43941016a0 2024-03-24T10:44:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f439c215640 1 -- 172.21.15.23:0/1083473483 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4394073b20 con 0x7f43941016a0 2024-03-24T10:44:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f438affd640 1 -- 172.21.15.23:0/1083473483 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f439000cd80 con 0x7f43941016a0 2024-03-24T10:44:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.886+0000 7f438affd640 1 -- 172.21.15.23:0/1083473483 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4390018920 con 0x7f43941016a0 2024-03-24T10:44:31.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.889+0000 7f438affd640 1 -- 172.21.15.23:0/1083473483 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f4390018a80 con 0x7f43941016a0 2024-03-24T10:44:31.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.889+0000 7f439edf4740 1 Processor -- start 2024-03-24T10:44:31.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.889+0000 7f439edf4740 1 -- start start 2024-03-24T10:44:31.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.889+0000 7f439edf4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fc3f0 0x55abe24fc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.889+0000 7f439edf4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fcd10 0x55abe25053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.889+0000 7f439edf4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 0x55abe2507ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.889+0000 7f439edf4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55abe246a840 con 0x55abe24fcd10 2024-03-24T10:44:31.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.889+0000 7f439edf4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55abe2467d70 con 0x55abe25058f0 2024-03-24T10:44:31.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.889+0000 7f439edf4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55abe2463550 con 0x55abe24fc3f0 2024-03-24T10:44:31.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f438affd640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f43800a2230 0x7f43800a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f4399fb1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fcd10 0x55abe25053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f43997b0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 0x55abe2507ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f4399fb1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fcd10 0x55abe25053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42476/0 (socket says 172.21.15.23:42476) 2024-03-24T10:44:31.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f4399fb1640 1 -- 172.21.15.23:0/1516388938 learned_addr learned my addr 172.21.15.23:0/1516388938 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:31.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f439a7b2640 1 --2- 172.21.15.23:0/1516388938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fc3f0 0x55abe24fc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f439a7b2640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f43800a2230 0x7f43800a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f438affd640 1 -- 172.21.15.23:0/1083473483 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f4390014070 con 0x7f43941016a0 2024-03-24T10:44:31.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f4399fb1640 1 -- 172.21.15.23:0/1516388938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fc3f0 msgr2=0x55abe24fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f4399fb1640 1 --2- 172.21.15.23:0/1516388938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fc3f0 0x55abe24fc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f4399fb1640 1 -- 172.21.15.23:0/1516388938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 msgr2=0x55abe2507ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f4399fb1640 1 --2- 172.21.15.23:0/1516388938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 0x55abe2507ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f4399fb1640 1 -- 172.21.15.23:0/1516388938 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abe23f8de0 con 0x55abe24fcd10 2024-03-24T10:44:31.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f43997b0640 1 --2- 172.21.15.23:0/1516388938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 0x55abe2507ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:31.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f439a7b2640 1 --2- 172.21.15.23:0/1516388938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fc3f0 0x55abe24fc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:31.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f4399fb1640 1 --2- 172.21.15.23:0/1516388938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fcd10 0x55abe25053b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f439009ab30 tx=0x7f439009ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06d78aaa46776a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f439a7b2640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f43800a2230 0x7f43800a46f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f438c005cc0 tx=0x7f438c005c50 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f437ffff640 1 -- 172.21.15.23:0/1516388938 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439009eed0 con 0x55abe24fcd10 2024-03-24T10:44:31.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f437ffff640 1 -- 172.21.15.23:0/1516388938 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43900a3520 con 0x55abe24fcd10 2024-03-24T10:44:31.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.890+0000 7f437ffff640 1 -- 172.21.15.23:0/1516388938 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43900cbfa0 con 0x55abe24fcd10 2024-03-24T10:44:31.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.891+0000 7f439edf4740 1 -- 172.21.15.23:0/1516388938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fcd10 msgr2=0x55abe25053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.891+0000 7f439edf4740 1 --2- 172.21.15.23:0/1516388938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fcd10 0x55abe25053b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f439009ab30 tx=0x7f439009ad30 comp rx=0 tx=0).stop 2024-03-24T10:44:31.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.891+0000 7f439edf4740 1 -- 172.21.15.23:0/1516388938 shutdown_connections 2024-03-24T10:44:31.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.891+0000 7f439edf4740 1 --2- 172.21.15.23:0/1516388938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 0x55abe2507ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.891+0000 7f439edf4740 1 --2- 172.21.15.23:0/1516388938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fcd10 0x55abe25053b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.891+0000 7f439edf4740 1 --2- 172.21.15.23:0/1516388938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fc3f0 0x55abe24fc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.891+0000 7f439edf4740 1 -- 172.21.15.23:0/1516388938 >> 172.21.15.23:0/1516388938 conn(0x55abe24f6b20 msgr2=0x55abe24f6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.891+0000 7f439edf4740 1 -- 172.21.15.23:0/1516388938 shutdown_connections 2024-03-24T10:44:31.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.891+0000 7f439edf4740 1 -- 172.21.15.23:0/1516388938 wait complete. 2024-03-24T10:44:31.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.891+0000 7f439edf4740 1 -- start start 2024-03-24T10:44:31.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f439edf4740 1 Processor -- start 2024-03-24T10:44:31.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f439edf4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fc3f0 0x55abe25029c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f439edf4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fcd10 0x55abe2502f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f439edf4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 0x55abe2503440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f439edf4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55abe2136430 con 0x55abe24fc3f0 2024-03-24T10:44:31.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f439edf4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55abe20f2e00 con 0x55abe25058f0 2024-03-24T10:44:31.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f439edf4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55abe23e8590 con 0x55abe24fcd10 2024-03-24T10:44:31.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f439edf4740 10 client.0 _ll_register_callbacks cb 0x55abe2457fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:31.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f4399fb1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fc3f0 0x55abe25029c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f4399fb1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fc3f0 0x55abe25029c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42488/0 (socket says 172.21.15.23:42488) 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f4399fb1640 1 -- 172.21.15.23:0/1056561436 learned_addr learned my addr 172.21.15.23:0/1056561436 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f43997b0640 1 --2- 172.21.15.23:0/1056561436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fcd10 0x55abe2502f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f439a7b2640 1 --2- 172.21.15.23:0/1056561436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 0x55abe2503440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f43997b0640 1 -- 172.21.15.23:0/1056561436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 msgr2=0x55abe2503440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f43997b0640 1 --2- 172.21.15.23:0/1056561436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 0x55abe2503440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f43997b0640 1 -- 172.21.15.23:0/1056561436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fc3f0 msgr2=0x55abe25029c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f43997b0640 1 --2- 172.21.15.23:0/1056561436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fc3f0 0x55abe25029c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f43997b0640 1 -- 172.21.15.23:0/1056561436 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4394073b20 con 0x55abe24fcd10 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.892+0000 7f4399fb1640 1 --2- 172.21.15.23:0/1056561436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fc3f0 0x55abe25029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.893+0000 7f43997b0640 1 --2- 172.21.15.23:0/1056561436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fcd10 0x55abe2502f00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f43940ff800 tx=0x7f438400b690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9ddfc3bb2411db9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.893+0000 7f437d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.893+0000 7f437d7fa640 1 -- 172.21.15.23:0/1056561436 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384015070 con 0x55abe24fcd10 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.893+0000 7f439edf4740 1 -- 172.21.15.23:0/1056561436 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55abe21627e0 con 0x55abe24fcd10 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.893+0000 7f437d7fa640 1 -- 172.21.15.23:0/1056561436 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f4384004460 con 0x55abe24fcd10 2024-03-24T10:44:31.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.893+0000 7f437d7fa640 1 -- 172.21.15.23:0/1056561436 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f438400be80 con 0x55abe24fcd10 2024-03-24T10:44:31.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.893+0000 7f439edf4740 10 client.26653 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:31.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.893+0000 7f439edf4740 1 -- 172.21.15.23:0/1056561436 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55abe2461bd0 con 0x55abe24fcd10 2024-03-24T10:44:31.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.893+0000 7f437d7fa640 1 -- 172.21.15.23:0/1056561436 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55abe21627e0 con 0x55abe24fcd10 2024-03-24T10:44:31.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.894+0000 7f437d7fa640 1 -- 172.21.15.23:0/1056561436 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f438401b890 con 0x55abe24fcd10 2024-03-24T10:44:31.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.894+0000 7f439edf4740 10 client.26653 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:31.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.894+0000 7f439edf4740 10 client.26653 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:44:31.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.894+0000 7f439edf4740 20 client.26653 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:31.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.894+0000 7f439edf4740 1 --2- 172.21.15.23:0/1056561436 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55abe2515a90 0x55abe2517eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.894+0000 7f439edf4740 4 client.26653 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:44:31.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.894+0000 7f439edf4740 1 -- 172.21.15.23:0/1056561436 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abe219f720 con 0x55abe2515a90 2024-03-24T10:44:31.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.894+0000 7f4399fb1640 1 --2- 172.21.15.23:0/1056561436 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55abe2515a90 0x55abe2517eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.894+0000 7f4399fb1640 1 --2- 172.21.15.23:0/1056561436 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55abe2515a90 0x55abe2517eb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7e8afc87b368615c server_cookie=bb0c14d340290cb1 in_seq=0 out_seq=0 2024-03-24T10:44:31.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.894+0000 7f437d7fa640 10 client.26653 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:44:31.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f437d7fa640 1 -- 172.21.15.23:0/1056561436 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55abe219f720 con 0x55abe2515a90 2024-03-24T10:44:31.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f437d7fa640 10 client.26653 handle_command_reply: tid=0 2024-03-24T10:44:31.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f439edf4740 1 client.26653 shutdown 2024-03-24T10:44:31.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f439edf4740 1 -- 172.21.15.23:0/1056561436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fcd10 msgr2=0x55abe2502f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f439edf4740 1 --2- 172.21.15.23:0/1056561436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fcd10 0x55abe2502f00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f43940ff800 tx=0x7f438400b690 comp rx=0 tx=0).stop 2024-03-24T10:44:31.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f439edf4740 1 -- 172.21.15.23:0/1056561436 shutdown_connections 2024-03-24T10:44:31.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f439edf4740 1 --2- 172.21.15.23:0/1056561436 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55abe2515a90 0x55abe2517eb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f439edf4740 1 --2- 172.21.15.23:0/1056561436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 0x55abe2503440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f439edf4740 1 --2- 172.21.15.23:0/1056561436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fcd10 0x55abe2502f00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f439edf4740 1 --2- 172.21.15.23:0/1056561436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fc3f0 0x55abe25029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f437d7fa640 0 client.26653 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:44:31.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f439edf4740 1 -- 172.21.15.23:0/1056561436 >> 172.21.15.23:0/1056561436 conn(0x55abe24f6b20 msgr2=0x55abe24f8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f439edf4740 1 -- 172.21.15.23:0/1056561436 shutdown_connections 2024-03-24T10:44:31.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f439edf4740 1 -- 172.21.15.23:0/1056561436 wait complete. 2024-03-24T10:44:31.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.897+0000 7f439edf4740 20 client.26653 trim_cache size 0 max 16384 2024-03-24T10:44:31.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.911+0000 7f439edf4740 1 Processor -- start 2024-03-24T10:44:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.911+0000 7f439edf4740 1 -- start start 2024-03-24T10:44:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.911+0000 7f439edf4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fc3f0 0x55abe251d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.911+0000 7f439edf4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe24fcd10 0x55abe250d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.911+0000 7f439edf4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe25058f0 0x55abe25047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.911+0000 7f439edf4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55abe24f8e50 con 0x55abe24fc3f0 2024-03-24T10:44:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.911+0000 7f439edf4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55abe2503d10 con 0x55abe24fcd10 2024-03-24T10:44:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.911+0000 7f439edf4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55abe24fc050 con 0x55abe25058f0 2024-03-24T10:44:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f4399fb1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fc3f0 0x55abe251d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f439a7b2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe25058f0 0x55abe25047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f43997b0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe24fcd10 0x55abe250d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f439a7b2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe25058f0 0x55abe25047f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39566/0 (socket says 172.21.15.23:39566) 2024-03-24T10:44:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f4399fb1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fc3f0 0x55abe251d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42498/0 (socket says 172.21.15.23:42498) 2024-03-24T10:44:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f439a7b2640 1 -- 172.21.15.23:0/1188664034 learned_addr learned my addr 172.21.15.23:0/1188664034 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f439a7b2640 1 -- 172.21.15.23:0/1188664034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe24fcd10 msgr2=0x55abe250d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f439a7b2640 1 --2- 172.21.15.23:0/1188664034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe24fcd10 0x55abe250d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f439a7b2640 1 -- 172.21.15.23:0/1188664034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fc3f0 msgr2=0x55abe251d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f439a7b2640 1 --2- 172.21.15.23:0/1188664034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fc3f0 0x55abe251d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f439a7b2640 1 -- 172.21.15.23:0/1188664034 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abe23fdc80 con 0x55abe25058f0 2024-03-24T10:44:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f439a7b2640 1 --2- 172.21.15.23:0/1188664034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe25058f0 0x55abe25047f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f438c009cd0 tx=0x7f438c009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=718a9210bacea8a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f437f7fe640 1 -- 172.21.15.23:0/1188664034 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c034c90 con 0x55abe25058f0 2024-03-24T10:44:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f437f7fe640 1 -- 172.21.15.23:0/1188664034 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f438c049050 con 0x55abe25058f0 2024-03-24T10:44:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.912+0000 7f437f7fe640 1 -- 172.21.15.23:0/1188664034 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c0042a0 con 0x55abe25058f0 2024-03-24T10:44:31.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.913+0000 7f439edf4740 1 -- 172.21.15.23:0/1188664034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe25058f0 msgr2=0x55abe25047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.913+0000 7f439edf4740 1 --2- 172.21.15.23:0/1188664034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe25058f0 0x55abe25047f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f438c009cd0 tx=0x7f438c009360 comp rx=0 tx=0).stop 2024-03-24T10:44:31.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.913+0000 7f439edf4740 1 -- 172.21.15.23:0/1188664034 shutdown_connections 2024-03-24T10:44:31.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.913+0000 7f439edf4740 1 --2- 172.21.15.23:0/1188664034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe25058f0 0x55abe25047f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.913+0000 7f439edf4740 1 --2- 172.21.15.23:0/1188664034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe24fcd10 0x55abe250d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.913+0000 7f439edf4740 1 --2- 172.21.15.23:0/1188664034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fc3f0 0x55abe251d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.913+0000 7f439edf4740 1 -- 172.21.15.23:0/1188664034 >> 172.21.15.23:0/1188664034 conn(0x55abe24f6b20 msgr2=0x55abe25040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.913+0000 7f439edf4740 1 -- 172.21.15.23:0/1188664034 shutdown_connections 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.913+0000 7f439edf4740 1 -- 172.21.15.23:0/1188664034 wait complete. 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.913+0000 7f439edf4740 1 -- start start 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f439edf4740 1 Processor -- start 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f439edf4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fc3f0 0x55abe2509970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f439edf4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fcd10 0x55abe2509eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f439edf4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 0x55abe2574da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f439edf4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55abe25752e0 con 0x55abe24fcd10 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f439edf4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55abe2575450 con 0x55abe25058f0 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f439edf4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55abe2575610 con 0x55abe24fc3f0 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f439edf4740 10 client.0 _ll_register_callbacks cb 0x7f4384009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f43997b0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fcd10 0x55abe2509eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f4399fb1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fc3f0 0x55abe2509970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f43997b0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fcd10 0x55abe2509eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42508/0 (socket says 172.21.15.23:42508) 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f43997b0640 1 -- 172.21.15.23:0/2303428652 learned_addr learned my addr 172.21.15.23:0/2303428652 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f439a7b2640 1 --2- 172.21.15.23:0/2303428652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 0x55abe2574da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f43997b0640 1 -- 172.21.15.23:0/2303428652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fc3f0 msgr2=0x55abe2509970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f43997b0640 1 --2- 172.21.15.23:0/2303428652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fc3f0 0x55abe2509970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f43997b0640 1 -- 172.21.15.23:0/2303428652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 msgr2=0x55abe2574da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f43997b0640 1 --2- 172.21.15.23:0/2303428652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 0x55abe2574da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f43997b0640 1 -- 172.21.15.23:0/2303428652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f438c030070 con 0x55abe24fcd10 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.914+0000 7f4399fb1640 1 --2- 172.21.15.23:0/2303428652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fc3f0 0x55abe2509970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f43997b0640 1 --2- 172.21.15.23:0/2303428652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fcd10 0x55abe2509eb0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f438401bdc0 tx=0x7f43840042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5115679ce436a427 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f437d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f437d7fa640 1 -- 172.21.15.23:0/2303428652 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384015040 con 0x55abe24fcd10 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f437d7fa640 1 -- 172.21.15.23:0/2303428652 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f438401be40 con 0x55abe24fcd10 2024-03-24T10:44:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f439edf4740 1 -- 172.21.15.23:0/2303428652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43900a3520 con 0x55abe24fcd10 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f437d7fa640 1 -- 172.21.15.23:0/2303428652 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f438401b850 con 0x55abe24fcd10 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f439edf4740 10 client.16809 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f439edf4740 1 -- 172.21.15.23:0/2303428652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55abe2423920 con 0x55abe24fcd10 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f437d7fa640 1 -- 172.21.15.23:0/2303428652 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43900a3520 con 0x55abe24fcd10 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f437d7fa640 1 -- 172.21.15.23:0/2303428652 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f4384003790 con 0x55abe24fcd10 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f439edf4740 10 client.16809 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f439edf4740 10 client.16809 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f439edf4740 20 client.16809 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f439edf4740 1 --2- 172.21.15.23:0/2303428652 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55abe257cf60 0x55abe257f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f439edf4740 4 client.16809 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f439edf4740 1 -- 172.21.15.23:0/2303428652 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55abe21ee890 con 0x55abe257cf60 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.915+0000 7f4399fb1640 1 --2- 172.21.15.23:0/2303428652 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55abe257cf60 0x55abe257f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.916+0000 7f437d7fa640 10 client.16809 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.916+0000 7f4399fb1640 1 --2- 172.21.15.23:0/2303428652 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55abe257cf60 0x55abe257f380 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8f66b9a384b2b266 server_cookie=6ea06c2611b86eb4 in_seq=0 out_seq=0 2024-03-24T10:44:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.916+0000 7f437d7fa640 1 -- 172.21.15.23:0/2303428652 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55abe21ee890 con 0x55abe257cf60 2024-03-24T10:44:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.916+0000 7f437d7fa640 10 client.16809 handle_command_reply: tid=0 2024-03-24T10:44:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.916+0000 7f439edf4740 1 client.16809 shutdown 2024-03-24T10:44:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439edf4740 1 -- 172.21.15.23:0/2303428652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fcd10 msgr2=0x55abe2509eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439edf4740 1 --2- 172.21.15.23:0/2303428652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fcd10 0x55abe2509eb0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f438401bdc0 tx=0x7f43840042e0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439edf4740 1 -- 172.21.15.23:0/2303428652 shutdown_connections 2024-03-24T10:44:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439edf4740 1 --2- 172.21.15.23:0/2303428652 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55abe257cf60 0x55abe257f380 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439edf4740 1 --2- 172.21.15.23:0/2303428652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55abe25058f0 0x55abe2574da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439edf4740 1 --2- 172.21.15.23:0/2303428652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abe24fcd10 0x55abe2509eb0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439edf4740 1 --2- 172.21.15.23:0/2303428652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55abe24fc3f0 0x55abe2509970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f437d7fa640 0 client.16809 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:44:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439edf4740 1 -- 172.21.15.23:0/2303428652 >> 172.21.15.23:0/2303428652 conn(0x55abe24f6b20 msgr2=0x55abe24f5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439edf4740 1 -- 172.21.15.23:0/2303428652 shutdown_connections 2024-03-24T10:44:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439edf4740 1 -- 172.21.15.23:0/2303428652 wait complete. 2024-03-24T10:44:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439edf4740 20 client.16809 trim_cache size 0 max 16384 2024-03-24T10:44:31.923 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:44:31.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439c215640 1 -- 172.21.15.23:0/1083473483 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f43800a2230 msgr2=0x7f43800a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439c215640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f43800a2230 0x7f43800a46f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f438c005cc0 tx=0x7f438c005c50 comp rx=0 tx=0).stop 2024-03-24T10:44:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439c215640 1 -- 172.21.15.23:0/1083473483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43941016a0 msgr2=0x7f4394197d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439c215640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43941016a0 0x7f4394197d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f439000e9f0 tx=0x7f439000eeb0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439c215640 1 -- 172.21.15.23:0/1083473483 shutdown_connections 2024-03-24T10:44:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439c215640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f43800a2230 0x7f43800a46f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439c215640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43941076b0 0x7f439419c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439c215640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4394102060 0x7f4394198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439c215640 1 --2- 172.21.15.23:0/1083473483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43941016a0 0x7f4394197d40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439c215640 1 -- 172.21.15.23:0/1083473483 >> 172.21.15.23:0/1083473483 conn(0x7f43940fd1b0 msgr2=0x7f43940fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439c215640 1 -- 172.21.15.23:0/1083473483 shutdown_connections 2024-03-24T10:44:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:31.917+0000 7f439c215640 1 -- 172.21.15.23:0/1083473483 wait complete. 2024-03-24T10:44:31.928 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:44:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.041+0000 7f5f3e3b5640 1 -- 172.21.15.23:0/3735527649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f38107690 msgr2=0x7f5f38107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.041+0000 7f5f3e3b5640 1 --2- 172.21.15.23:0/3735527649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f38107690 0x7f5f38107a70 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5f28009930 tx=0x7f5f2802bc30 comp rx=0 tx=0).stop 2024-03-24T10:44:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.041+0000 7f5f3e3b5640 1 -- 172.21.15.23:0/3735527649 shutdown_connections 2024-03-24T10:44:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.041+0000 7f5f3e3b5640 1 --2- 172.21.15.23:0/3735527649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f380692b0 0x7f5f3810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.041+0000 7f5f3e3b5640 1 --2- 172.21.15.23:0/3735527649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f380688f0 0x7f5f38068d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.041+0000 7f5f3e3b5640 1 --2- 172.21.15.23:0/3735527649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f38107690 0x7f5f38107a70 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.041+0000 7f5f3e3b5640 1 -- 172.21.15.23:0/3735527649 >> 172.21.15.23:0/3735527649 conn(0x7f5f38077710 msgr2=0x7f5f38077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.041+0000 7f5f3e3b5640 1 -- 172.21.15.23:0/3735527649 shutdown_connections 2024-03-24T10:44:32.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.041+0000 7f5f3e3b5640 1 -- 172.21.15.23:0/3735527649 wait complete. 2024-03-24T10:44:32.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.042+0000 7f5f3e3b5640 1 Processor -- start 2024-03-24T10:44:32.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.042+0000 7f5f3e3b5640 1 -- start start 2024-03-24T10:44:32.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.042+0000 7f5f3e3b5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f380688f0 0x7f5f38197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.042+0000 7f5f3e3b5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f380692b0 0x7f5f381982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.042+0000 7f5f3e3b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f38107690 0x7f5f3819c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.042+0000 7f5f3e3b5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3819cc20 con 0x7f5f380692b0 2024-03-24T10:44:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.042+0000 7f5f3e3b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3819cd90 con 0x7f5f38107690 2024-03-24T10:44:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.042+0000 7f5f3e3b5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3819cf50 con 0x7f5f380688f0 2024-03-24T10:44:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.042+0000 7f5f377fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f380692b0 0x7f5f381982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.042+0000 7f5f377fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f380692b0 0x7f5f381982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42528/0 (socket says 172.21.15.23:42528) 2024-03-24T10:44:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.042+0000 7f5f377fe640 1 -- 172.21.15.23:0/3701909902 learned_addr learned my addr 172.21.15.23:0/3701909902 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.042+0000 7f5f37fff640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f380688f0 0x7f5f38197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.042+0000 7f5f3c952640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f38107690 0x7f5f3819c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.043+0000 7f5f377fe640 1 -- 172.21.15.23:0/3701909902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f380688f0 msgr2=0x7f5f38197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.043+0000 7f5f377fe640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f380688f0 0x7f5f38197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.043+0000 7f5f377fe640 1 -- 172.21.15.23:0/3701909902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f38107690 msgr2=0x7f5f3819c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.043+0000 7f5f377fe640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f38107690 0x7f5f3819c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.043+0000 7f5f377fe640 1 -- 172.21.15.23:0/3701909902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f28009560 con 0x7f5f380692b0 2024-03-24T10:44:32.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.043+0000 7f5f37fff640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f380688f0 0x7f5f38197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:32.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.043+0000 7f5f377fe640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f380692b0 0x7f5f381982a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f5f2000e9f0 tx=0x7f5f2000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:32.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.043+0000 7f5f357fa640 1 -- 172.21.15.23:0/3701909902 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f20009700 con 0x7f5f380692b0 2024-03-24T10:44:32.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.043+0000 7f5f3e3b5640 1 -- 172.21.15.23:0/3701909902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f3819d230 con 0x7f5f380692b0 2024-03-24T10:44:32.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.043+0000 7f5f357fa640 1 -- 172.21.15.23:0/3701909902 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f2000cd80 con 0x7f5f380692b0 2024-03-24T10:44:32.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.043+0000 7f5f357fa640 1 -- 172.21.15.23:0/3701909902 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f20018920 con 0x7f5f380692b0 2024-03-24T10:44:32.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.043+0000 7f5f3e3b5640 1 -- 172.21.15.23:0/3701909902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f381a4ab0 con 0x7f5f380692b0 2024-03-24T10:44:32.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.045+0000 7f5f357fa640 1 -- 172.21.15.23:0/3701909902 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f5f20018a80 con 0x7f5f380692b0 2024-03-24T10:44:32.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f40f99740 1 Processor -- start 2024-03-24T10:44:32.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f40f99740 1 -- start start 2024-03-24T10:44:32.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f40f99740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 0x556049dce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f40f99740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049dd73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f357fa640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f5f240a2230 0x7f5f240a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f40f99740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dd78f0 0x556049dd9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f40f99740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556049d3c840 con 0x556049dced10 2024-03-24T10:44:32.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f40f99740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556049d39d70 con 0x556049dce3f0 2024-03-24T10:44:32.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f40f99740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556049d35550 con 0x556049dd78f0 2024-03-24T10:44:32.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f37fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049dd73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f377fe640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f5f240a2230 0x7f5f240a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f37fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049dd73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42542/0 (socket says 172.21.15.23:42542) 2024-03-24T10:44:32.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f37fff640 1 -- 172.21.15.23:0/3371928821 learned_addr learned my addr 172.21.15.23:0/3371928821 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:32.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f3c952640 1 --2- 172.21.15.23:0/3371928821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 0x556049dce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f3c952640 1 --2- 172.21.15.23:0/3371928821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dd78f0 0x556049dd9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f37fff640 1 -- 172.21.15.23:0/3371928821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dd78f0 msgr2=0x556049dd9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f37fff640 1 --2- 172.21.15.23:0/3371928821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dd78f0 0x556049dd9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f37fff640 1 -- 172.21.15.23:0/3371928821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 msgr2=0x556049dce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f37fff640 1 --2- 172.21.15.23:0/3371928821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 0x556049dce7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f37fff640 1 -- 172.21.15.23:0/3371928821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556049ccade0 con 0x556049dced10 2024-03-24T10:44:32.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.046+0000 7f5f3c952640 1 --2- 172.21.15.23:0/3371928821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 0x556049dce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:32.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.047+0000 7f5f357fa640 1 -- 172.21.15.23:0/3701909902 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f5f20014070 con 0x7f5f380692b0 2024-03-24T10:44:32.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.047+0000 7f5f37fff640 1 --2- 172.21.15.23:0/3371928821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049dd73b0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5f28036040 tx=0x7f5f2802c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeee0865df1d601c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:32.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.047+0000 7f5f377fe640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f5f240a2230 0x7f5f240a46f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5f20097300 tx=0x7f5f20097230 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:32.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.047+0000 7f5f2e7fc640 1 -- 172.21.15.23:0/3371928821 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2802c7e0 con 0x556049dced10 2024-03-24T10:44:32.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.047+0000 7f5f2e7fc640 1 -- 172.21.15.23:0/3371928821 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f28004510 con 0x556049dced10 2024-03-24T10:44:32.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.047+0000 7f5f2e7fc640 1 -- 172.21.15.23:0/3371928821 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f280053d0 con 0x556049dced10 2024-03-24T10:44:32.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.048+0000 7f5f40f99740 1 -- 172.21.15.23:0/3371928821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 msgr2=0x556049dd73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.048+0000 7f5f40f99740 1 --2- 172.21.15.23:0/3371928821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049dd73b0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5f28036040 tx=0x7f5f2802c660 comp rx=0 tx=0).stop 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.048+0000 7f5f40f99740 1 -- 172.21.15.23:0/3371928821 shutdown_connections 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.048+0000 7f5f40f99740 1 --2- 172.21.15.23:0/3371928821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dd78f0 0x556049dd9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.048+0000 7f5f40f99740 1 --2- 172.21.15.23:0/3371928821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049dd73b0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.048+0000 7f5f40f99740 1 --2- 172.21.15.23:0/3371928821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 0x556049dce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.048+0000 7f5f40f99740 1 -- 172.21.15.23:0/3371928821 >> 172.21.15.23:0/3371928821 conn(0x556049dc8b20 msgr2=0x556049dc8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.048+0000 7f5f40f99740 1 -- 172.21.15.23:0/3371928821 shutdown_connections 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.048+0000 7f5f40f99740 1 -- 172.21.15.23:0/3371928821 wait complete. 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.048+0000 7f5f40f99740 1 -- start start 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.048+0000 7f5f40f99740 1 Processor -- start 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f40f99740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 0x556049dd4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f40f99740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049dd4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f40f99740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dd78f0 0x556049dd53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f40f99740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556049a08430 con 0x556049dced10 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f40f99740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5560499c4e00 con 0x556049dce3f0 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f40f99740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556049cba590 con 0x556049dd78f0 2024-03-24T10:44:32.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f40f99740 10 client.0 _ll_register_callbacks cb 0x556049d29fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f37fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049dd4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f37fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049dd4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42550/0 (socket says 172.21.15.23:42550) 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f37fff640 1 -- 172.21.15.23:0/403731735 learned_addr learned my addr 172.21.15.23:0/403731735 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f3c952640 1 --2- 172.21.15.23:0/403731735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 0x556049dd4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f3c952640 1 --2- 172.21.15.23:0/403731735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dd78f0 0x556049dd53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f37fff640 1 -- 172.21.15.23:0/403731735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dd78f0 msgr2=0x556049dd53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f37fff640 1 --2- 172.21.15.23:0/403731735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dd78f0 0x556049dd53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f37fff640 1 -- 172.21.15.23:0/403731735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 msgr2=0x556049dd4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f37fff640 1 --2- 172.21.15.23:0/403731735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 0x556049dd4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f37fff640 1 -- 172.21.15.23:0/403731735 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f3810ee50 con 0x556049dced10 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f37fff640 1 --2- 172.21.15.23:0/403731735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049dd4e90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5f28009900 tx=0x7f5f28004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dea46c9e7f43181 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f177fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f177fe640 1 -- 172.21.15.23:0/403731735 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2803f640 con 0x556049dced10 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f177fe640 1 -- 172.21.15.23:0/403731735 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f5f2804d070 con 0x556049dced10 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.049+0000 7f5f40f99740 1 -- 172.21.15.23:0/403731735 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556049a347e0 con 0x556049dced10 2024-03-24T10:44:32.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.050+0000 7f5f177fe640 1 -- 172.21.15.23:0/403731735 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5f2802cbe0 con 0x556049dced10 2024-03-24T10:44:32.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.050+0000 7f5f40f99740 10 client.16821 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:32.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.050+0000 7f5f40f99740 1 -- 172.21.15.23:0/403731735 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556049d33bd0 con 0x556049dced10 2024-03-24T10:44:32.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.050+0000 7f5f177fe640 1 -- 172.21.15.23:0/403731735 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556049a347e0 con 0x556049dced10 2024-03-24T10:44:32.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.050+0000 7f5f177fe640 1 -- 172.21.15.23:0/403731735 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f5f2803c050 con 0x556049dced10 2024-03-24T10:44:32.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.050+0000 7f5f40f99740 10 client.16821 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:32.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.050+0000 7f5f40f99740 10 client.16821 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:44:32.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.050+0000 7f5f40f99740 20 client.16821 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:32.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.050+0000 7f5f40f99740 1 --2- 172.21.15.23:0/403731735 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556049de75f0 0x556049de9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.050+0000 7f5f40f99740 4 client.16821 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:44:32.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.050+0000 7f5f40f99740 1 -- 172.21.15.23:0/403731735 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556049a71720 con 0x556049de75f0 2024-03-24T10:44:32.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.050+0000 7f5f3c952640 1 --2- 172.21.15.23:0/403731735 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556049de75f0 0x556049de9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.051+0000 7f5f177fe640 10 client.16821 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:44:32.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.051+0000 7f5f3c952640 1 --2- 172.21.15.23:0/403731735 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556049de75f0 0x556049de9a10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d8a424cc9e5f7ce server_cookie=b48ffedf610ed182 in_seq=0 out_seq=0 2024-03-24T10:44:32.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.051+0000 7f5f177fe640 1 -- 172.21.15.23:0/403731735 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556049a71720 con 0x556049de75f0 2024-03-24T10:44:32.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.051+0000 7f5f177fe640 10 client.16821 handle_command_reply: tid=0 2024-03-24T10:44:32.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.051+0000 7f5f40f99740 1 client.16821 shutdown 2024-03-24T10:44:32.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.051+0000 7f5f40f99740 1 -- 172.21.15.23:0/403731735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 msgr2=0x556049dd4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.051+0000 7f5f40f99740 1 --2- 172.21.15.23:0/403731735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049dd4e90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5f28009900 tx=0x7f5f28004f70 comp rx=0 tx=0).stop 2024-03-24T10:44:32.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.051+0000 7f5f40f99740 1 -- 172.21.15.23:0/403731735 shutdown_connections 2024-03-24T10:44:32.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.051+0000 7f5f40f99740 1 --2- 172.21.15.23:0/403731735 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556049de75f0 0x556049de9a10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.051+0000 7f5f40f99740 1 --2- 172.21.15.23:0/403731735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dd78f0 0x556049dd53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.051+0000 7f5f40f99740 1 --2- 172.21.15.23:0/403731735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049dd4e90 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.051+0000 7f5f177fe640 0 client.16821 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:44:32.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.051+0000 7f5f40f99740 1 --2- 172.21.15.23:0/403731735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 0x556049dd4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.051+0000 7f5f40f99740 1 -- 172.21.15.23:0/403731735 >> 172.21.15.23:0/403731735 conn(0x556049dc8b20 msgr2=0x556049dca870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:32.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.052+0000 7f5f40f99740 1 -- 172.21.15.23:0/403731735 shutdown_connections 2024-03-24T10:44:32.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.052+0000 7f5f40f99740 1 -- 172.21.15.23:0/403731735 wait complete. 2024-03-24T10:44:32.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.052+0000 7f5f40f99740 20 client.16821 trim_cache size 0 max 16384 2024-03-24T10:44:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f40f99740 1 Processor -- start 2024-03-24T10:44:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f40f99740 1 -- start start 2024-03-24T10:44:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f40f99740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 0x556049de3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f40f99740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dced10 0x556049def320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f40f99740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dd78f0 0x556049ddf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f40f99740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556049dc9b20 con 0x556049dd78f0 2024-03-24T10:44:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f40f99740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556049dc43e0 con 0x556049dce3f0 2024-03-24T10:44:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f40f99740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556049dcd510 con 0x556049dced10 2024-03-24T10:44:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f37fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dced10 0x556049def320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f3c952640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 0x556049de3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f37fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dced10 0x556049def320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39624/0 (socket says 172.21.15.23:39624) 2024-03-24T10:44:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f37fff640 1 -- 172.21.15.23:0/2419269755 learned_addr learned my addr 172.21.15.23:0/2419269755 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f3c952640 1 --2- 172.21.15.23:0/2419269755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dd78f0 0x556049ddf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f3c952640 1 -- 172.21.15.23:0/2419269755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dced10 msgr2=0x556049def320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f3c952640 1 --2- 172.21.15.23:0/2419269755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dced10 0x556049def320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f3c952640 1 -- 172.21.15.23:0/2419269755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dd78f0 msgr2=0x556049ddf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f3c952640 1 --2- 172.21.15.23:0/2419269755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dd78f0 0x556049ddf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f3c952640 1 -- 172.21.15.23:0/2419269755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556049ccfc80 con 0x556049dce3f0 2024-03-24T10:44:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.066+0000 7f5f37fff640 1 --2- 172.21.15.23:0/2419269755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dced10 0x556049def320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.067+0000 7f5f3c952640 1 --2- 172.21.15.23:0/2419269755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 0x556049de3120 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x556049dea290 tx=0x7f5f3000fc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=571fb78e6a1a5aa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.067+0000 7f5f2dffb640 1 -- 172.21.15.23:0/2419269755 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f30018040 con 0x556049dce3f0 2024-03-24T10:44:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.067+0000 7f5f2dffb640 1 -- 172.21.15.23:0/2419269755 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f3002c020 con 0x556049dce3f0 2024-03-24T10:44:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.067+0000 7f5f2dffb640 1 -- 172.21.15.23:0/2419269755 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f300048a0 con 0x556049dce3f0 2024-03-24T10:44:32.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.067+0000 7f5f40f99740 1 -- 172.21.15.23:0/2419269755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 msgr2=0x556049de3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.067+0000 7f5f40f99740 1 --2- 172.21.15.23:0/2419269755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 0x556049de3120 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x556049dea290 tx=0x7f5f3000fc30 comp rx=0 tx=0).stop 2024-03-24T10:44:32.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.067+0000 7f5f40f99740 1 -- 172.21.15.23:0/2419269755 shutdown_connections 2024-03-24T10:44:32.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.067+0000 7f5f40f99740 1 --2- 172.21.15.23:0/2419269755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dd78f0 0x556049ddf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.067+0000 7f5f40f99740 1 --2- 172.21.15.23:0/2419269755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dced10 0x556049def320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.067+0000 7f5f40f99740 1 --2- 172.21.15.23:0/2419269755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dce3f0 0x556049de3120 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.067+0000 7f5f40f99740 1 -- 172.21.15.23:0/2419269755 >> 172.21.15.23:0/2419269755 conn(0x556049dc8b20 msgr2=0x556049dd4910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.068+0000 7f5f40f99740 1 -- 172.21.15.23:0/2419269755 shutdown_connections 2024-03-24T10:44:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.068+0000 7f5f40f99740 1 -- 172.21.15.23:0/2419269755 wait complete. 2024-03-24T10:44:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.068+0000 7f5f40f99740 1 -- start start 2024-03-24T10:44:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.068+0000 7f5f40f99740 1 Processor -- start 2024-03-24T10:44:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.068+0000 7f5f40f99740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dce3f0 0x556049ddf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.068+0000 7f5f40f99740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049ddba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f40f99740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dd78f0 0x556049ddbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f40f99740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556049ddff60 con 0x556049dced10 2024-03-24T10:44:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f40f99740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556049ddc490 con 0x556049dd78f0 2024-03-24T10:44:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f40f99740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556049ddc620 con 0x556049dce3f0 2024-03-24T10:44:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f40f99740 10 client.0 _ll_register_callbacks cb 0x7f5f30009b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f37fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049ddba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f37fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049ddba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42562/0 (socket says 172.21.15.23:42562) 2024-03-24T10:44:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f37fff640 1 -- 172.21.15.23:0/268434582 learned_addr learned my addr 172.21.15.23:0/268434582 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f3c952640 1 --2- 172.21.15.23:0/268434582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dce3f0 0x556049ddf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f3c952640 1 --2- 172.21.15.23:0/268434582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dd78f0 0x556049ddbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f37fff640 1 -- 172.21.15.23:0/268434582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dce3f0 msgr2=0x556049ddf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f37fff640 1 --2- 172.21.15.23:0/268434582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dce3f0 0x556049ddf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f37fff640 1 -- 172.21.15.23:0/268434582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dd78f0 msgr2=0x556049ddbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f37fff640 1 --2- 172.21.15.23:0/268434582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dd78f0 0x556049ddbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f37fff640 1 -- 172.21.15.23:0/268434582 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556049d33bd0 con 0x556049dced10 2024-03-24T10:44:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f37fff640 1 --2- 172.21.15.23:0/268434582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049ddba10 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x556049a71740 tx=0x7f5f28005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aaddec9af452801 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f177fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f177fe640 1 -- 172.21.15.23:0/268434582 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f28034ae0 con 0x556049dced10 2024-03-24T10:44:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f177fe640 1 -- 172.21.15.23:0/268434582 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f5f2804d070 con 0x556049dced10 2024-03-24T10:44:32.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f40f99740 1 -- 172.21.15.23:0/268434582 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f28004510 con 0x556049dced10 2024-03-24T10:44:32.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f177fe640 1 -- 172.21.15.23:0/268434582 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5f28004df0 con 0x556049dced10 2024-03-24T10:44:32.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f40f99740 10 client.16827 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:32.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.069+0000 7f5f40f99740 1 -- 172.21.15.23:0/268434582 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x7f5f3001b040 con 0x556049dced10 2024-03-24T10:44:32.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.070+0000 7f5f177fe640 1 -- 172.21.15.23:0/268434582 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f28004510 con 0x556049dced10 2024-03-24T10:44:32.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.070+0000 7f5f177fe640 1 -- 172.21.15.23:0/268434582 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f5f2803c040 con 0x556049dced10 2024-03-24T10:44:32.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.070+0000 7f5f40f99740 10 client.16827 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:32.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.070+0000 7f5f40f99740 10 client.16827 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:44:32.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.070+0000 7f5f40f99740 20 client.16827 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:32.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.070+0000 7f5f40f99740 1 --2- 172.21.15.23:0/268434582 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556049e4e9d0 0x556049e50df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.070+0000 7f5f40f99740 4 client.16827 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:44:32.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.070+0000 7f5f40f99740 1 -- 172.21.15.23:0/268434582 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556049ac0890 con 0x556049e4e9d0 2024-03-24T10:44:32.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.070+0000 7f5f3c952640 1 --2- 172.21.15.23:0/268434582 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556049e4e9d0 0x556049e50df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.070+0000 7f5f177fe640 10 client.16827 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:44:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.070+0000 7f5f3c952640 1 --2- 172.21.15.23:0/268434582 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556049e4e9d0 0x556049e50df0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=134e8b0ec5dd768 server_cookie=d0d158007c8f1854 in_seq=0 out_seq=0 2024-03-24T10:44:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f177fe640 1 -- 172.21.15.23:0/268434582 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556049ac0890 con 0x556049e4e9d0 2024-03-24T10:44:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f177fe640 10 client.16827 handle_command_reply: tid=0 2024-03-24T10:44:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f40f99740 1 client.16827 shutdown 2024-03-24T10:44:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f40f99740 1 -- 172.21.15.23:0/268434582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 msgr2=0x556049ddba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f40f99740 1 --2- 172.21.15.23:0/268434582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049ddba10 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x556049a71740 tx=0x7f5f28005d90 comp rx=0 tx=0).stop 2024-03-24T10:44:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f40f99740 1 -- 172.21.15.23:0/268434582 shutdown_connections 2024-03-24T10:44:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f40f99740 1 --2- 172.21.15.23:0/268434582 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556049e4e9d0 0x556049e50df0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f40f99740 1 --2- 172.21.15.23:0/268434582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556049dd78f0 0x556049ddbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f177fe640 0 client.16827 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:44:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f40f99740 1 --2- 172.21.15.23:0/268434582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556049dced10 0x556049ddba10 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f40f99740 1 --2- 172.21.15.23:0/268434582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556049dce3f0 0x556049ddf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f40f99740 1 -- 172.21.15.23:0/268434582 >> 172.21.15.23:0/268434582 conn(0x556049dc8b20 msgr2=0x556049dd5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:32.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f40f99740 1 -- 172.21.15.23:0/268434582 shutdown_connections 2024-03-24T10:44:32.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f40f99740 1 -- 172.21.15.23:0/268434582 wait complete. 2024-03-24T10:44:32.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f40f99740 20 client.16827 trim_cache size 0 max 16384 2024-03-24T10:44:32.078 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:44:32.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f3e3b5640 1 -- 172.21.15.23:0/3701909902 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f5f240a2230 msgr2=0x7f5f240a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f3e3b5640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f5f240a2230 0x7f5f240a46f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5f20097300 tx=0x7f5f20097230 comp rx=0 tx=0).stop 2024-03-24T10:44:32.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f3e3b5640 1 -- 172.21.15.23:0/3701909902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f380692b0 msgr2=0x7f5f381982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.071+0000 7f5f3e3b5640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f380692b0 0x7f5f381982a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f5f2000e9f0 tx=0x7f5f2000eeb0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.072+0000 7f5f3e3b5640 1 -- 172.21.15.23:0/3701909902 shutdown_connections 2024-03-24T10:44:32.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.072+0000 7f5f3e3b5640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f5f240a2230 0x7f5f240a46f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.072+0000 7f5f3e3b5640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f38107690 0x7f5f3819c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.072+0000 7f5f3e3b5640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f380692b0 0x7f5f381982a0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.072+0000 7f5f3e3b5640 1 --2- 172.21.15.23:0/3701909902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f380688f0 0x7f5f38197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.072+0000 7f5f3e3b5640 1 -- 172.21.15.23:0/3701909902 >> 172.21.15.23:0/3701909902 conn(0x7f5f38077710 msgr2=0x7f5f380fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:32.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.072+0000 7f5f3e3b5640 1 -- 172.21.15.23:0/3701909902 shutdown_connections 2024-03-24T10:44:32.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.072+0000 7f5f3e3b5640 1 -- 172.21.15.23:0/3701909902 wait complete. 2024-03-24T10:44:32.083 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:44:32.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 382456 11.95 MB/sec execute 49 sec latency 18.957 ms 2024-03-24T10:44:32.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.194+0000 7feeb6f6c640 1 -- 172.21.15.23:0/1750663472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeb0106660 msgr2=0x7feeb0106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.194+0000 7feeb6f6c640 1 --2- 172.21.15.23:0/1750663472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeb0106660 0x7feeb0106a40 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7feea4009900 tx=0x7feea402bc30 comp rx=0 tx=0).stop 2024-03-24T10:44:32.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.194+0000 7feeb6f6c640 1 -- 172.21.15.23:0/1750663472 shutdown_connections 2024-03-24T10:44:32.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.194+0000 7feeb6f6c640 1 --2- 172.21.15.23:0/1750663472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feeb0100f60 0x7feeb010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.194+0000 7feeb6f6c640 1 --2- 172.21.15.23:0/1750663472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeb01005a0 0x7feeb0100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.194+0000 7feeb6f6c640 1 --2- 172.21.15.23:0/1750663472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeb0106660 0x7feeb0106a40 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.194+0000 7feeb6f6c640 1 -- 172.21.15.23:0/1750663472 >> 172.21.15.23:0/1750663472 conn(0x7feeb0077710 msgr2=0x7feeb0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:32.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.194+0000 7feeb6f6c640 1 -- 172.21.15.23:0/1750663472 shutdown_connections 2024-03-24T10:44:32.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.194+0000 7feeb6f6c640 1 -- 172.21.15.23:0/1750663472 wait complete. 2024-03-24T10:44:32.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.194+0000 7feeb6f6c640 1 Processor -- start 2024-03-24T10:44:32.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.194+0000 7feeb6f6c640 1 -- start start 2024-03-24T10:44:32.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb6f6c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeb01005a0 0x7feeb0102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb6f6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feeb0100f60 0x7feeb0102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb6f6c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeb0106660 0x7feeb0102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb6f6c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7feeb01a6040 con 0x7feeb0106660 2024-03-24T10:44:32.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb6f6c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feeb01a61b0 con 0x7feeb0100f60 2024-03-24T10:44:32.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb6f6c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feeb01a6370 con 0x7feeb01005a0 2024-03-24T10:44:32.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb4d08640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeb01005a0 0x7feeb0102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb5509640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeb0106660 0x7feeb0102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb4d08640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeb01005a0 0x7feeb0102210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39644/0 (socket says 172.21.15.23:39644) 2024-03-24T10:44:32.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb4d08640 1 -- 172.21.15.23:0/618827681 learned_addr learned my addr 172.21.15.23:0/618827681 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeaffff640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feeb0100f60 0x7feeb0102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb5509640 1 -- 172.21.15.23:0/618827681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeb01005a0 msgr2=0x7feeb0102210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb5509640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeb01005a0 0x7feeb0102210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb5509640 1 -- 172.21.15.23:0/618827681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feeb0100f60 msgr2=0x7feeb0102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb5509640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feeb0100f60 0x7feeb0102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb5509640 1 -- 172.21.15.23:0/618827681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea4009560 con 0x7feeb0106660 2024-03-24T10:44:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeaffff640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feeb0100f60 0x7feeb0102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb4d08640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeb01005a0 0x7feeb0102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.195+0000 7feeb5509640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeb0106660 0x7feeb0102c90 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7feea8009fd0 tx=0x7feea800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.196+0000 7feeadffb640 1 -- 172.21.15.23:0/618827681 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea800ce20 con 0x7feeb0106660 2024-03-24T10:44:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.196+0000 7feeadffb640 1 -- 172.21.15.23:0/618827681 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea8005020 con 0x7feeb0106660 2024-03-24T10:44:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.196+0000 7feeb6f6c640 1 -- 172.21.15.23:0/618827681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb01a6650 con 0x7feeb0106660 2024-03-24T10:44:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.196+0000 7feeadffb640 1 -- 172.21.15.23:0/618827681 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8010690 con 0x7feeb0106660 2024-03-24T10:44:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.196+0000 7feeb6f6c640 1 -- 172.21.15.23:0/618827681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb01a6b60 con 0x7feeb0106660 2024-03-24T10:44:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.198+0000 7feeadffb640 1 -- 172.21.15.23:0/618827681 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7feea8005500 con 0x7feeb0106660 2024-03-24T10:44:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.198+0000 7feeb9c26740 1 Processor -- start 2024-03-24T10:44:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.198+0000 7feeb9c26740 1 -- start start 2024-03-24T10:44:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeb9c26740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a63153f0 0x5573a63157d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeb9c26740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a6315d10 0x5573a631e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeb9c26740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 0x5573a6320ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeb9c26740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5573a6283840 con 0x5573a63153f0 2024-03-24T10:44:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeb9c26740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5573a6280d70 con 0x5573a6315d10 2024-03-24T10:44:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeb9c26740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5573a627c550 con 0x5573a631e8f0 2024-03-24T10:44:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeaffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a63153f0 0x5573a63157d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeaffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a63153f0 0x5573a63157d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42576/0 (socket says 172.21.15.23:42576) 2024-03-24T10:44:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeb4d08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a6315d10 0x5573a631e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeaffff640 1 -- 172.21.15.23:0/1867761486 learned_addr learned my addr 172.21.15.23:0/1867761486 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeaffff640 1 --2- 172.21.15.23:0/1867761486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 0x5573a6320ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeadffb640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feea00a22b0 0x7feea00a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeaffff640 1 -- 172.21.15.23:0/1867761486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 msgr2=0x5573a6320ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeb5509640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feea00a22b0 0x7feea00a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeaffff640 1 --2- 172.21.15.23:0/1867761486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 0x5573a6320ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeaffff640 1 -- 172.21.15.23:0/1867761486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a6315d10 msgr2=0x5573a631e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeaffff640 1 --2- 172.21.15.23:0/1867761486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a6315d10 0x5573a631e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeaffff640 1 -- 172.21.15.23:0/1867761486 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573a6211de0 con 0x5573a63153f0 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeb4d08640 1 --2- 172.21.15.23:0/1867761486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a6315d10 0x5573a631e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeadffb640 1 -- 172.21.15.23:0/618827681 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7feea8014070 con 0x7feeb0106660 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.199+0000 7feeaffff640 1 --2- 172.21.15.23:0/1867761486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a63153f0 0x5573a63157d0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fee9c013c20 tx=0x7fee9c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2decd85f79301c83 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.200+0000 7fee9affd640 1 -- 172.21.15.23:0/1867761486 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c013dd0 con 0x5573a63153f0 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.200+0000 7fee9affd640 1 -- 172.21.15.23:0/1867761486 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee9c004510 con 0x5573a63153f0 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.200+0000 7fee9affd640 1 -- 172.21.15.23:0/1867761486 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c003850 con 0x5573a63153f0 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.201+0000 7feeb5509640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feea00a22b0 0x7feea00a4770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7feea8097ae0 tx=0x7feea8097430 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.201+0000 7feeb9c26740 1 -- 172.21.15.23:0/1867761486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a63153f0 msgr2=0x5573a63157d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.201+0000 7feeb9c26740 1 --2- 172.21.15.23:0/1867761486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a63153f0 0x5573a63157d0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fee9c013c20 tx=0x7fee9c009b30 comp rx=0 tx=0).stop 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.201+0000 7feeb9c26740 1 -- 172.21.15.23:0/1867761486 shutdown_connections 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.201+0000 7feeb9c26740 1 --2- 172.21.15.23:0/1867761486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 0x5573a6320ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.201+0000 7feeb9c26740 1 --2- 172.21.15.23:0/1867761486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a6315d10 0x5573a631e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.201+0000 7feeb9c26740 1 --2- 172.21.15.23:0/1867761486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a63153f0 0x5573a63157d0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.201+0000 7feeb9c26740 1 -- 172.21.15.23:0/1867761486 >> 172.21.15.23:0/1867761486 conn(0x5573a630fb20 msgr2=0x5573a630ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:32.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.201+0000 7feeb9c26740 1 -- 172.21.15.23:0/1867761486 shutdown_connections 2024-03-24T10:44:32.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.201+0000 7feeb9c26740 1 -- 172.21.15.23:0/1867761486 wait complete. 2024-03-24T10:44:32.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.201+0000 7feeb9c26740 1 -- start start 2024-03-24T10:44:32.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.201+0000 7feeb9c26740 1 Processor -- start 2024-03-24T10:44:32.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb9c26740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a63153f0 0x5573a631b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb9c26740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a6315d10 0x5573a631bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb9c26740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 0x5573a631c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb9c26740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5573a5f4f430 con 0x5573a6315d10 2024-03-24T10:44:32.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb9c26740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5573a5f0be00 con 0x5573a63153f0 2024-03-24T10:44:32.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb9c26740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5573a6201590 con 0x5573a631e8f0 2024-03-24T10:44:32.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb9c26740 10 client.0 _ll_register_callbacks cb 0x5573a6270fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:32.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb4d08640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a6315d10 0x5573a631bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb4d08640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a6315d10 0x5573a631bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42592/0 (socket says 172.21.15.23:42592) 2024-03-24T10:44:32.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb4d08640 1 -- 172.21.15.23:0/3938548430 learned_addr learned my addr 172.21.15.23:0/3938548430 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:32.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeaffff640 1 --2- 172.21.15.23:0/3938548430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a63153f0 0x5573a631b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeaffff640 1 --2- 172.21.15.23:0/3938548430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 0x5573a631c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb4d08640 1 -- 172.21.15.23:0/3938548430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 msgr2=0x5573a631c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb4d08640 1 --2- 172.21.15.23:0/3938548430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 0x5573a631c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb4d08640 1 -- 172.21.15.23:0/3938548430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a63153f0 msgr2=0x5573a631b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb4d08640 1 --2- 172.21.15.23:0/3938548430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a63153f0 0x5573a631b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb4d08640 1 -- 172.21.15.23:0/3938548430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee9c01b040 con 0x5573a6315d10 2024-03-24T10:44:32.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb4d08640 1 --2- 172.21.15.23:0/3938548430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a6315d10 0x5573a631bed0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7feea4006010 tx=0x7feea402cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c91d44481c820e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:32.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7fee8ffff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:32.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7fee8ffff640 1 -- 172.21.15.23:0/3938548430 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4004410 con 0x5573a6315d10 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7fee8ffff640 1 -- 172.21.15.23:0/3938548430 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7feea4004df0 con 0x5573a6315d10 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb9c26740 1 -- 172.21.15.23:0/3938548430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5573a5f7b7e0 con 0x5573a6315d10 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7fee8ffff640 1 -- 172.21.15.23:0/3938548430 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feea4004fb0 con 0x5573a6315d10 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb9c26740 10 client.16839 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.202+0000 7feeb9c26740 1 -- 172.21.15.23:0/3938548430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5573a627abd0 con 0x5573a6315d10 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.203+0000 7fee8ffff640 1 -- 172.21.15.23:0/3938548430 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5573a5f7b7e0 con 0x5573a6315d10 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.203+0000 7fee8ffff640 1 -- 172.21.15.23:0/3938548430 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7feea403c040 con 0x5573a6315d10 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.203+0000 7feeb9c26740 10 client.16839 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.203+0000 7feeb9c26740 10 client.16839 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.203+0000 7feeb9c26740 20 client.16839 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.203+0000 7feeb9c26740 1 --2- 172.21.15.23:0/3938548430 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5573a632e8d0 0x5573a6330cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.203+0000 7feeb9c26740 4 client.16839 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.203+0000 7feeb9c26740 1 -- 172.21.15.23:0/3938548430 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573a5fb8720 con 0x5573a632e8d0 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.203+0000 7feeaffff640 1 --2- 172.21.15.23:0/3938548430 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5573a632e8d0 0x5573a6330cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.203+0000 7fee8ffff640 10 client.16839 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:32.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.203+0000 7feeaffff640 1 --2- 172.21.15.23:0/3938548430 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5573a632e8d0 0x5573a6330cf0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b9c7d50cf5d6cdc server_cookie=35be9d10137ce83 in_seq=0 out_seq=0 2024-03-24T10:44:32.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.204+0000 7fee8ffff640 1 -- 172.21.15.23:0/3938548430 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5573a5fb8720 con 0x5573a632e8d0 2024-03-24T10:44:32.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.204+0000 7fee8ffff640 10 client.16839 handle_command_reply: tid=0 2024-03-24T10:44:32.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.204+0000 7feeb9c26740 1 client.16839 shutdown 2024-03-24T10:44:32.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.204+0000 7feeb9c26740 1 -- 172.21.15.23:0/3938548430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a6315d10 msgr2=0x5573a631bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.204+0000 7feeb9c26740 1 --2- 172.21.15.23:0/3938548430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a6315d10 0x5573a631bed0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7feea4006010 tx=0x7feea402cc70 comp rx=0 tx=0).stop 2024-03-24T10:44:32.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.204+0000 7feeb9c26740 1 -- 172.21.15.23:0/3938548430 shutdown_connections 2024-03-24T10:44:32.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.204+0000 7feeb9c26740 1 --2- 172.21.15.23:0/3938548430 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5573a632e8d0 0x5573a6330cf0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.204+0000 7feeb9c26740 1 --2- 172.21.15.23:0/3938548430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 0x5573a631c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.204+0000 7feeb9c26740 1 --2- 172.21.15.23:0/3938548430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a6315d10 0x5573a631bed0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.204+0000 7fee8ffff640 0 client.16839 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:32.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.204+0000 7feeb9c26740 1 --2- 172.21.15.23:0/3938548430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a63153f0 0x5573a631b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.204+0000 7feeb9c26740 1 -- 172.21.15.23:0/3938548430 >> 172.21.15.23:0/3938548430 conn(0x5573a630fb20 msgr2=0x5573a6311870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:32.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.204+0000 7feeb9c26740 1 -- 172.21.15.23:0/3938548430 shutdown_connections 2024-03-24T10:44:32.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.205+0000 7feeb9c26740 1 -- 172.21.15.23:0/3938548430 wait complete. 2024-03-24T10:44:32.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.205+0000 7feeb9c26740 20 client.16839 trim_cache size 0 max 16384 2024-03-24T10:44:32.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.218+0000 7feeb9c26740 1 Processor -- start 2024-03-24T10:44:32.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.218+0000 7feeb9c26740 1 -- start start 2024-03-24T10:44:32.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.218+0000 7feeb9c26740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a63153f0 0x5573a63160f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.218+0000 7feeb9c26740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a6315d10 0x5573a632a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.218+0000 7feeb9c26740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 0x5573a631c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.218+0000 7feeb9c26740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5573a5f7b7e0 con 0x5573a6315d10 2024-03-24T10:44:32.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.218+0000 7feeb9c26740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5573a630b450 con 0x5573a63153f0 2024-03-24T10:44:32.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.218+0000 7feeb9c26740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5573a632b470 con 0x5573a631e8f0 2024-03-24T10:44:32.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7feeb4d08640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a6315d10 0x5573a632a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7feeb4d08640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a6315d10 0x5573a632a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42608/0 (socket says 172.21.15.23:42608) 2024-03-24T10:44:32.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7feeb4d08640 1 -- 172.21.15.23:0/2499834843 learned_addr learned my addr 172.21.15.23:0/2499834843 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:32.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7feeaffff640 1 --2- 172.21.15.23:0/2499834843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a63153f0 0x5573a63160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7feeaffff640 1 --2- 172.21.15.23:0/2499834843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 0x5573a631c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7feeb4d08640 1 -- 172.21.15.23:0/2499834843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 msgr2=0x5573a631c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7feeb4d08640 1 --2- 172.21.15.23:0/2499834843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 0x5573a631c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7feeb4d08640 1 -- 172.21.15.23:0/2499834843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a63153f0 msgr2=0x5573a63160f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7feeb4d08640 1 --2- 172.21.15.23:0/2499834843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a63153f0 0x5573a63160f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7feeb4d08640 1 -- 172.21.15.23:0/2499834843 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573a6216c80 con 0x5573a6315d10 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7feeaffff640 1 --2- 172.21.15.23:0/2499834843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 0x5573a631c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7feeb4d08640 1 --2- 172.21.15.23:0/2499834843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a6315d10 0x5573a632a120 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7feea4000c00 tx=0x7feea402cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5622044cce0a0435 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7fee9a7fc640 1 -- 172.21.15.23:0/2499834843 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4003fe0 con 0x5573a6315d10 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7fee9a7fc640 1 -- 172.21.15.23:0/2499834843 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea4003e60 con 0x5573a6315d10 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.219+0000 7fee9a7fc640 1 -- 172.21.15.23:0/2499834843 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4053c40 con 0x5573a6315d10 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.220+0000 7feeb9c26740 1 -- 172.21.15.23:0/2499834843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a6315d10 msgr2=0x5573a632a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.220+0000 7feeb9c26740 1 --2- 172.21.15.23:0/2499834843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a6315d10 0x5573a632a120 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7feea4000c00 tx=0x7feea402cbc0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.220+0000 7feeb9c26740 1 -- 172.21.15.23:0/2499834843 shutdown_connections 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.220+0000 7feeb9c26740 1 --2- 172.21.15.23:0/2499834843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a631e8f0 0x5573a631c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.220+0000 7feeb9c26740 1 --2- 172.21.15.23:0/2499834843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a6315d10 0x5573a632a120 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.220+0000 7feeb9c26740 1 --2- 172.21.15.23:0/2499834843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a63153f0 0x5573a63160f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.220+0000 7feeb9c26740 1 -- 172.21.15.23:0/2499834843 >> 172.21.15.23:0/2499834843 conn(0x5573a630fb20 msgr2=0x5573a63117e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:32.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.220+0000 7feeb9c26740 1 -- 172.21.15.23:0/2499834843 shutdown_connections 2024-03-24T10:44:32.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.220+0000 7feeb9c26740 1 -- 172.21.15.23:0/2499834843 wait complete. 2024-03-24T10:44:32.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.220+0000 7feeb9c26740 1 -- start start 2024-03-24T10:44:32.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.220+0000 7feeb9c26740 1 Processor -- start 2024-03-24T10:44:32.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeb9c26740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a63153f0 0x5573a63223e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeb9c26740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a6315d10 0x5573a6322920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeb9c26740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a631e8f0 0x5573a6393c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeb9c26740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5573a63231f0 con 0x5573a63153f0 2024-03-24T10:44:32.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeb9c26740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5573a6394170 con 0x5573a631e8f0 2024-03-24T10:44:32.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeb9c26740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5573a63942b0 con 0x5573a6315d10 2024-03-24T10:44:32.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeb9c26740 10 client.0 _ll_register_callbacks cb 0x7feea4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:32.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeaffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a63153f0 0x5573a63223e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeaffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a63153f0 0x5573a63223e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42616/0 (socket says 172.21.15.23:42616) 2024-03-24T10:44:32.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeaffff640 1 -- 172.21.15.23:0/1910616532 learned_addr learned my addr 172.21.15.23:0/1910616532 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:32.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeb4d08640 1 --2- 172.21.15.23:0/1910616532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a6315d10 0x5573a6322920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeaffff640 1 --2- 172.21.15.23:0/1910616532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a631e8f0 0x5573a6393c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeaffff640 1 -- 172.21.15.23:0/1910616532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a6315d10 msgr2=0x5573a6322920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeaffff640 1 --2- 172.21.15.23:0/1910616532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a6315d10 0x5573a6322920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeaffff640 1 -- 172.21.15.23:0/1910616532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a631e8f0 msgr2=0x5573a6393c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeaffff640 1 --2- 172.21.15.23:0/1910616532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a631e8f0 0x5573a6393c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.221+0000 7feeaffff640 1 -- 172.21.15.23:0/1910616532 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573a627abd0 con 0x5573a63153f0 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7feeaffff640 1 --2- 172.21.15.23:0/1910616532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a63153f0 0x5573a63223e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00fee0 tx=0x7fee9c025040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce639ecfa3137aac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7fee8ffff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7fee8ffff640 1 -- 172.21.15.23:0/1910616532 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c02fd50 con 0x5573a63153f0 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7fee8ffff640 1 -- 172.21.15.23:0/1910616532 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fee9c02f2d0 con 0x5573a63153f0 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7feeb9c26740 1 -- 172.21.15.23:0/1910616532 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5573a6007890 con 0x5573a63153f0 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7fee8ffff640 1 -- 172.21.15.23:0/1910616532 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fee9c02f490 con 0x5573a63153f0 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7feeb9c26740 10 client.16845 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7feeb9c26740 1 -- 172.21.15.23:0/1910616532 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5573a623c920 con 0x5573a63153f0 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7fee8ffff640 1 -- 172.21.15.23:0/1910616532 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5573a6007890 con 0x5573a63153f0 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7fee8ffff640 1 -- 172.21.15.23:0/1910616532 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fee9c004680 con 0x5573a63153f0 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7feeb9c26740 10 client.16845 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7feeb9c26740 10 client.16845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7feeb9c26740 20 client.16845 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:32.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7feeb9c26740 1 --2- 172.21.15.23:0/1910616532 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5573a6395980 0x5573a6397d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7feeb9c26740 4 client.16845 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:44:32.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7feeb9c26740 1 -- 172.21.15.23:0/1910616532 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5573a5f27700 con 0x5573a6395980 2024-03-24T10:44:32.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.222+0000 7feeb4d08640 1 --2- 172.21.15.23:0/1910616532 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5573a6395980 0x5573a6397d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7feeb4d08640 1 --2- 172.21.15.23:0/1910616532 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5573a6395980 0x5573a6397d70 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12f4b179b18cceb7 server_cookie=114e35e3862f51aa in_seq=0 out_seq=0 2024-03-24T10:44:32.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7fee8ffff640 10 client.16845 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:32.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7fee8ffff640 1 -- 172.21.15.23:0/1910616532 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5573a5f27700 con 0x5573a6395980 2024-03-24T10:44:32.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7fee8ffff640 10 client.16845 handle_command_reply: tid=0 2024-03-24T10:44:32.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7feeb9c26740 1 client.16845 shutdown 2024-03-24T10:44:32.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7feeb9c26740 1 -- 172.21.15.23:0/1910616532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a63153f0 msgr2=0x5573a63223e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7feeb9c26740 1 --2- 172.21.15.23:0/1910616532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a63153f0 0x5573a63223e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00fee0 tx=0x7fee9c025040 comp rx=0 tx=0).stop 2024-03-24T10:44:32.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7feeb9c26740 1 -- 172.21.15.23:0/1910616532 shutdown_connections 2024-03-24T10:44:32.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7feeb9c26740 1 --2- 172.21.15.23:0/1910616532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5573a631e8f0 0x5573a6393c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7feeb9c26740 1 --2- 172.21.15.23:0/1910616532 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5573a6395980 0x5573a6397d70 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7feeb9c26740 1 --2- 172.21.15.23:0/1910616532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5573a6315d10 0x5573a6322920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7feeb9c26740 1 --2- 172.21.15.23:0/1910616532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573a63153f0 0x5573a63223e0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7fee8ffff640 0 client.16845 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:32.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7feeb9c26740 1 -- 172.21.15.23:0/1910616532 >> 172.21.15.23:0/1910616532 conn(0x5573a630fb20 msgr2=0x5573a630e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:32.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7feeb9c26740 1 -- 172.21.15.23:0/1910616532 shutdown_connections 2024-03-24T10:44:32.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7feeb9c26740 1 -- 172.21.15.23:0/1910616532 wait complete. 2024-03-24T10:44:32.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.223+0000 7feeb9c26740 20 client.16845 trim_cache size 0 max 16384 2024-03-24T10:44:32.230 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:44:32.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.224+0000 7feeb6f6c640 1 -- 172.21.15.23:0/618827681 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feea00a22b0 msgr2=0x7feea00a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.224+0000 7feeb6f6c640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feea00a22b0 0x7feea00a4770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7feea8097ae0 tx=0x7feea8097430 comp rx=0 tx=0).stop 2024-03-24T10:44:32.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.224+0000 7feeb6f6c640 1 -- 172.21.15.23:0/618827681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeb0106660 msgr2=0x7feeb0102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.224+0000 7feeb6f6c640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeb0106660 0x7feeb0102c90 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7feea8009fd0 tx=0x7feea800ef10 comp rx=0 tx=0).stop 2024-03-24T10:44:32.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.224+0000 7feeb6f6c640 1 -- 172.21.15.23:0/618827681 shutdown_connections 2024-03-24T10:44:32.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.224+0000 7feeb6f6c640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7feea00a22b0 0x7feea00a4770 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.224+0000 7feeb6f6c640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeb0106660 0x7feeb0102c90 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.224+0000 7feeb6f6c640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feeb0100f60 0x7feeb0102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.224+0000 7feeb6f6c640 1 --2- 172.21.15.23:0/618827681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeb01005a0 0x7feeb0102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.224+0000 7feeb6f6c640 1 -- 172.21.15.23:0/618827681 >> 172.21.15.23:0/618827681 conn(0x7feeb0077710 msgr2=0x7feeb00ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:32.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.224+0000 7feeb6f6c640 1 -- 172.21.15.23:0/618827681 shutdown_connections 2024-03-24T10:44:32.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:32.224+0000 7feeb6f6c640 1 -- 172.21.15.23:0/618827681 wait complete. 2024-03-24T10:44:32.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:32 smithi023 ceph-mon[32466]: pgmap v136: 97 pgs: 97 active+clean; 216 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 307 op/s 2024-03-24T10:44:32.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:32 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1424339254' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:44:33.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:32 smithi117 ceph-mon[36368]: pgmap v136: 97 pgs: 97 active+clean; 216 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 307 op/s 2024-03-24T10:44:33.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:32 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1424339254' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:44:33.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:32 smithi116 ceph-mon[36248]: pgmap v136: 97 pgs: 97 active+clean; 216 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 307 op/s 2024-03-24T10:44:33.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:32 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1424339254' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:44:33.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 383894 11.86 MB/sec execute 50 sec latency 93.879 ms 2024-03-24T10:44:33.236 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:44:33.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:44:33.237 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:44:33.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.349+0000 7f11ce33e640 1 -- 172.21.15.23:0/1375181744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c81076d0 msgr2=0x7f11c8107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.349+0000 7f11ce33e640 1 --2- 172.21.15.23:0/1375181744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c81076d0 0x7f11c8107ab0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f11b8009930 tx=0x7f11b802bc30 comp rx=0 tx=0).stop 2024-03-24T10:44:33.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.349+0000 7f11ce33e640 1 -- 172.21.15.23:0/1375181744 shutdown_connections 2024-03-24T10:44:33.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.349+0000 7f11ce33e640 1 --2- 172.21.15.23:0/1375181744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11c8102080 0x7f11c810e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.349+0000 7f11ce33e640 1 --2- 172.21.15.23:0/1375181744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11c81016c0 0x7f11c8101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.349+0000 7f11ce33e640 1 --2- 172.21.15.23:0/1375181744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c81076d0 0x7f11c8107ab0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.349+0000 7f11ce33e640 1 -- 172.21.15.23:0/1375181744 >> 172.21.15.23:0/1375181744 conn(0x7f11c80fd150 msgr2=0x7f11c80ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.349+0000 7f11ce33e640 1 -- 172.21.15.23:0/1375181744 shutdown_connections 2024-03-24T10:44:33.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.349+0000 7f11ce33e640 1 -- 172.21.15.23:0/1375181744 wait complete. 2024-03-24T10:44:33.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.350+0000 7f11ce33e640 1 Processor -- start 2024-03-24T10:44:33.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.350+0000 7f11ce33e640 1 -- start start 2024-03-24T10:44:33.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.350+0000 7f11ce33e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c81016c0 0x7f11c8197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.350+0000 7f11ce33e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11c8102080 0x7f11c8198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.350+0000 7f11ce33e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11c81076d0 0x7f11c819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.350+0000 7f11ce33e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f11c819cc50 con 0x7f11c81016c0 2024-03-24T10:44:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.350+0000 7f11ce33e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11c819cdc0 con 0x7f11c81076d0 2024-03-24T10:44:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.350+0000 7f11ce33e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f11c819cf80 con 0x7f11c8102080 2024-03-24T10:44:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.350+0000 7f11c7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c81016c0 0x7f11c8197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.350+0000 7f11c77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11c8102080 0x7f11c8198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.350+0000 7f11c7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c81016c0 0x7f11c8197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42628/0 (socket says 172.21.15.23:42628) 2024-03-24T10:44:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.350+0000 7f11c7fff640 1 -- 172.21.15.23:0/2519799746 learned_addr learned my addr 172.21.15.23:0/2519799746 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.350+0000 7f11cc8db640 1 --2- 172.21.15.23:0/2519799746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11c81076d0 0x7f11c819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.351+0000 7f11c7fff640 1 -- 172.21.15.23:0/2519799746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11c8102080 msgr2=0x7f11c8198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.351+0000 7f11c7fff640 1 --2- 172.21.15.23:0/2519799746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11c8102080 0x7f11c8198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.351+0000 7f11c7fff640 1 -- 172.21.15.23:0/2519799746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11c81076d0 msgr2=0x7f11c819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.351+0000 7f11c7fff640 1 --2- 172.21.15.23:0/2519799746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11c81076d0 0x7f11c819c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.351+0000 7f11c7fff640 1 -- 172.21.15.23:0/2519799746 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b8009560 con 0x7f11c81016c0 2024-03-24T10:44:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.351+0000 7f11c7fff640 1 --2- 172.21.15.23:0/2519799746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c81016c0 0x7f11c8197de0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f11b8009230 tx=0x7f11b802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.351+0000 7f11c57fa640 1 -- 172.21.15.23:0/2519799746 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8004350 con 0x7f11c81016c0 2024-03-24T10:44:33.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.351+0000 7f11c57fa640 1 -- 172.21.15.23:0/2519799746 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11b80044b0 con 0x7f11c81016c0 2024-03-24T10:44:33.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.351+0000 7f11c57fa640 1 -- 172.21.15.23:0/2519799746 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8003d00 con 0x7f11c81016c0 2024-03-24T10:44:33.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.351+0000 7f11ce33e640 1 -- 172.21.15.23:0/2519799746 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c819d200 con 0x7f11c81016c0 2024-03-24T10:44:33.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.351+0000 7f11ce33e640 1 -- 172.21.15.23:0/2519799746 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c81a4ae0 con 0x7f11c81016c0 2024-03-24T10:44:33.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.353+0000 7f11c57fa640 1 -- 172.21.15.23:0/2519799746 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f11b8004df0 con 0x7f11c81016c0 2024-03-24T10:44:33.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11d0ff6740 1 Processor -- start 2024-03-24T10:44:33.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11d0ff6740 1 -- start start 2024-03-24T10:44:33.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11c57fa640 1 --2- 172.21.15.23:0/2519799746 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f11b00a2230 0x7f11b00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11d0ff6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363c760 0x55d6b363cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11cc8db640 1 --2- 172.21.15.23:0/2519799746 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f11b00a2230 0x7f11b00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11d0ff6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363d080 0x55d6b3645720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11c57fa640 1 -- 172.21.15.23:0/2519799746 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f11b80eff50 con 0x7f11c81016c0 2024-03-24T10:44:33.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11d0ff6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b3645c60 0x55d6b3648050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11d0ff6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d6b35aa530 con 0x55d6b3645c60 2024-03-24T10:44:33.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11d0ff6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d6b35a7a60 con 0x55d6b363d080 2024-03-24T10:44:33.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11d0ff6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d6b35a3240 con 0x55d6b363c760 2024-03-24T10:44:33.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11cc8db640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b3645c60 0x55d6b3648050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11c7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363c760 0x55d6b363cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363d080 0x55d6b3645720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11c7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363c760 0x55d6b363cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39704/0 (socket says 172.21.15.23:39704) 2024-03-24T10:44:33.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11c7fff640 1 -- 172.21.15.23:0/601130637 learned_addr learned my addr 172.21.15.23:0/601130637 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:33.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11cc8db640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b3645c60 0x55d6b3648050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42632/0 (socket says 172.21.15.23:42632) 2024-03-24T10:44:33.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.354+0000 7f11c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363d080 0x55d6b3645720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:40700/0 (socket says 172.21.15.23:40700) 2024-03-24T10:44:33.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.355+0000 7f11c7fff640 1 -- 172.21.15.23:0/601130637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363d080 msgr2=0x55d6b3645720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.355+0000 7f11c7fff640 1 --2- 172.21.15.23:0/601130637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363d080 0x55d6b3645720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.355+0000 7f11c7fff640 1 -- 172.21.15.23:0/601130637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b3645c60 msgr2=0x55d6b3648050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.355+0000 7f11c7fff640 1 --2- 172.21.15.23:0/601130637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b3645c60 0x55d6b3648050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.355+0000 7f11c7fff640 1 -- 172.21.15.23:0/601130637 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6b35a18c0 con 0x55d6b363c760 2024-03-24T10:44:33.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.355+0000 7f11cc8db640 1 --2- 172.21.15.23:0/2519799746 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f11b00a2230 0x7f11b00a46f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f11c0005fd0 tx=0x7f11c0013ae0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.355+0000 7f11c7fff640 1 --2- 172.21.15.23:0/601130637 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363c760 0x55d6b363cb40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f11b80b9420 tx=0x7f11b802bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5429d29fbae01f2e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.355+0000 7f11ae7fc640 1 -- 172.21.15.23:0/601130637 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b80bca50 con 0x55d6b363c760 2024-03-24T10:44:33.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.355+0000 7f11ae7fc640 1 -- 172.21.15.23:0/601130637 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11b80c0ba0 con 0x55d6b363c760 2024-03-24T10:44:33.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.355+0000 7f11ae7fc640 1 -- 172.21.15.23:0/601130637 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b80b90c0 con 0x55d6b363c760 2024-03-24T10:44:33.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.356+0000 7f11d0ff6740 1 -- 172.21.15.23:0/601130637 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363c760 msgr2=0x55d6b363cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.356+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/601130637 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363c760 0x55d6b363cb40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f11b80b9420 tx=0x7f11b802bc30 comp rx=0 tx=0).stop 2024-03-24T10:44:33.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.356+0000 7f11d0ff6740 1 -- 172.21.15.23:0/601130637 shutdown_connections 2024-03-24T10:44:33.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.356+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/601130637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b3645c60 0x55d6b3648050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.356+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/601130637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363d080 0x55d6b3645720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.356+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/601130637 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363c760 0x55d6b363cb40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.356+0000 7f11d0ff6740 1 -- 172.21.15.23:0/601130637 >> 172.21.15.23:0/601130637 conn(0x55d6b3636e90 msgr2=0x55d6b3637270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.356+0000 7f11d0ff6740 1 -- 172.21.15.23:0/601130637 shutdown_connections 2024-03-24T10:44:33.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.356+0000 7f11d0ff6740 1 -- 172.21.15.23:0/601130637 wait complete. 2024-03-24T10:44:33.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.356+0000 7f11d0ff6740 1 -- start start 2024-03-24T10:44:33.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.356+0000 7f11d0ff6740 1 Processor -- start 2024-03-24T10:44:33.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11d0ff6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363c760 0x55d6b3642c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11d0ff6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b363d080 0x55d6b36431c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11d0ff6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b3645c60 0x55d6b3643700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11d0ff6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d6b3547f80 con 0x55d6b363d080 2024-03-24T10:44:33.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11d0ff6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d6b32764b0 con 0x55d6b363c760 2024-03-24T10:44:33.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11d0ff6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d6b3528ad0 con 0x55d6b3645c60 2024-03-24T10:44:33.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11d0ff6740 10 client.0 _ll_register_callbacks cb 0x55d6b355bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:33.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11c77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b363d080 0x55d6b36431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363c760 0x55d6b3642c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11c77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b363d080 0x55d6b36431c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42640/0 (socket says 172.21.15.23:42640) 2024-03-24T10:44:33.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363c760 0x55d6b3642c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:40708/0 (socket says 172.21.15.23:40708) 2024-03-24T10:44:33.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11cc8db640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b3645c60 0x55d6b3643700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11c77fe640 1 -- 172.21.15.23:0/4023662710 learned_addr learned my addr 172.21.15.23:0/4023662710 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:33.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11c77fe640 1 -- 172.21.15.23:0/4023662710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b3645c60 msgr2=0x55d6b3643700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11c77fe640 1 --2- 172.21.15.23:0/4023662710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b3645c60 0x55d6b3643700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11c77fe640 1 -- 172.21.15.23:0/4023662710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363c760 msgr2=0x55d6b3642c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11c77fe640 1 --2- 172.21.15.23:0/4023662710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363c760 0x55d6b3642c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11c77fe640 1 -- 172.21.15.23:0/4023662710 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c81a4ae0 con 0x55d6b363d080 2024-03-24T10:44:33.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11c7fff640 1 --2- 172.21.15.23:0/4023662710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363c760 0x55d6b3642c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11c77fe640 1 --2- 172.21.15.23:0/4023662710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b363d080 0x55d6b36431c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f11bc012720 tx=0x7f11bc012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732ccf4f390b59de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11a77fe640 1 -- 172.21.15.23:0/4023662710 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc017070 con 0x55d6b363d080 2024-03-24T10:44:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11a77fe640 1 -- 172.21.15.23:0/4023662710 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f11bc00f920 con 0x55d6b363d080 2024-03-24T10:44:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11d0ff6740 1 -- 172.21.15.23:0/4023662710 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d6b35c77a0 con 0x55d6b363d080 2024-03-24T10:44:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.357+0000 7f11a77fe640 1 -- 172.21.15.23:0/4023662710 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f11bc00fb20 con 0x55d6b363d080 2024-03-24T10:44:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.358+0000 7f11d0ff6740 10 client.16857 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.358+0000 7f11d0ff6740 1 -- 172.21.15.23:0/4023662710 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d6b35596d0 con 0x55d6b363d080 2024-03-24T10:44:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.358+0000 7f11a77fe640 1 -- 172.21.15.23:0/4023662710 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d6b35c77a0 con 0x55d6b363d080 2024-03-24T10:44:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.358+0000 7f11a77fe640 1 -- 172.21.15.23:0/4023662710 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f11bc01c470 con 0x55d6b363d080 2024-03-24T10:44:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.358+0000 7f11d0ff6740 10 client.16857 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.358+0000 7f11d0ff6740 10 client.16857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.358+0000 7f11d0ff6740 20 client.16857 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:33.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.358+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/4023662710 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d6b3655b80 0x55d6b3657fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.358+0000 7f11d0ff6740 4 client.16857 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.358+0000 7f11d0ff6740 1 -- 172.21.15.23:0/4023662710 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6b35991f0 con 0x55d6b3655b80 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.358+0000 7f11c7fff640 1 --2- 172.21.15.23:0/4023662710 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d6b3655b80 0x55d6b3657fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.358+0000 7f11a77fe640 10 client.16857 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.358+0000 7f11c7fff640 1 --2- 172.21.15.23:0/4023662710 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d6b3655b80 0x55d6b3657fc0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89f361a16b52363b server_cookie=4a905bdbecbd74ad in_seq=0 out_seq=0 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.359+0000 7f11a77fe640 1 -- 172.21.15.23:0/4023662710 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d6b35991f0 con 0x55d6b3655b80 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.359+0000 7f11a77fe640 10 client.16857 handle_command_reply: tid=0 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.359+0000 7f11d0ff6740 1 client.16857 shutdown 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.359+0000 7f11d0ff6740 1 -- 172.21.15.23:0/4023662710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b363d080 msgr2=0x55d6b36431c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.359+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/4023662710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b363d080 0x55d6b36431c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f11bc012720 tx=0x7f11bc012be0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.359+0000 7f11d0ff6740 1 -- 172.21.15.23:0/4023662710 shutdown_connections 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.359+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/4023662710 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d6b3655b80 0x55d6b3657fc0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.359+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/4023662710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b3645c60 0x55d6b3643700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.359+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/4023662710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b363d080 0x55d6b36431c0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.359+0000 7f11a77fe640 0 client.16857 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:33.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.359+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/4023662710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363c760 0x55d6b3642c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.359+0000 7f11d0ff6740 1 -- 172.21.15.23:0/4023662710 >> 172.21.15.23:0/4023662710 conn(0x55d6b3636e90 msgr2=0x55d6b3638be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.359+0000 7f11d0ff6740 1 -- 172.21.15.23:0/4023662710 shutdown_connections 2024-03-24T10:44:33.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.360+0000 7f11d0ff6740 1 -- 172.21.15.23:0/4023662710 wait complete. 2024-03-24T10:44:33.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.360+0000 7f11d0ff6740 20 client.16857 trim_cache size 0 max 16384 2024-03-24T10:44:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.377+0000 7f11d0ff6740 1 Processor -- start 2024-03-24T10:44:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.377+0000 7f11d0ff6740 1 -- start start 2024-03-24T10:44:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11d0ff6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b3636e90 0x55d6b36453b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11d0ff6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b363c760 0x55d6b3643520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11d0ff6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363d080 0x55d6b365d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11d0ff6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d6b363c1a0 con 0x55d6b363c760 2024-03-24T10:44:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11d0ff6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d6b363c3c0 con 0x55d6b3636e90 2024-03-24T10:44:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11d0ff6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d6b3632620 con 0x55d6b363d080 2024-03-24T10:44:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11c77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b363c760 0x55d6b3643520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11c77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b363c760 0x55d6b3643520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42644/0 (socket says 172.21.15.23:42644) 2024-03-24T10:44:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11c77fe640 1 -- 172.21.15.23:0/128013036 learned_addr learned my addr 172.21.15.23:0/128013036 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11cc8db640 1 --2- 172.21.15.23:0/128013036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363d080 0x55d6b365d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11c7fff640 1 --2- 172.21.15.23:0/128013036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b3636e90 0x55d6b36453b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11c77fe640 1 -- 172.21.15.23:0/128013036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363d080 msgr2=0x55d6b365d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11c77fe640 1 --2- 172.21.15.23:0/128013036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363d080 0x55d6b365d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11c77fe640 1 -- 172.21.15.23:0/128013036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b3636e90 msgr2=0x55d6b36453b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11c77fe640 1 --2- 172.21.15.23:0/128013036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b3636e90 0x55d6b36453b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11c77fe640 1 -- 172.21.15.23:0/128013036 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6b353dc60 con 0x55d6b363c760 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11c77fe640 1 --2- 172.21.15.23:0/128013036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b363c760 0x55d6b3643520 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f11bc01c880 tx=0x7f11bc0096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2849c7cee6935b76 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11adffb640 1 -- 172.21.15.23:0/128013036 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc017040 con 0x55d6b363c760 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11adffb640 1 -- 172.21.15.23:0/128013036 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11bc00f920 con 0x55d6b363c760 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.378+0000 7f11adffb640 1 -- 172.21.15.23:0/128013036 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc005470 con 0x55d6b363c760 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.379+0000 7f11d0ff6740 1 -- 172.21.15.23:0/128013036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b363c760 msgr2=0x55d6b3643520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.379+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/128013036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b363c760 0x55d6b3643520 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f11bc01c880 tx=0x7f11bc0096c0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.379+0000 7f11d0ff6740 1 -- 172.21.15.23:0/128013036 shutdown_connections 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.379+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/128013036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363d080 0x55d6b365d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.379+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/128013036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b363c760 0x55d6b3643520 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.379+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/128013036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b3636e90 0x55d6b36453b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.379+0000 7f11d0ff6740 1 -- 172.21.15.23:0/128013036 >> 172.21.15.23:0/128013036 conn(0x55d6b36418c0 msgr2=0x55d6b3638190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.379+0000 7f11d0ff6740 1 -- 172.21.15.23:0/128013036 shutdown_connections 2024-03-24T10:44:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.379+0000 7f11d0ff6740 1 -- 172.21.15.23:0/128013036 wait complete. 2024-03-24T10:44:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.380+0000 7f11d0ff6740 1 -- start start 2024-03-24T10:44:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.380+0000 7f11d0ff6740 1 Processor -- start 2024-03-24T10:44:33.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.380+0000 7f11d0ff6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b3636e90 0x55d6b364dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.380+0000 7f11d0ff6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363c760 0x55d6b364a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.380+0000 7f11d0ff6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363d080 0x55d6b364aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.380+0000 7f11d0ff6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d6b364b5d0 con 0x55d6b3636e90 2024-03-24T10:44:33.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.380+0000 7f11d0ff6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d6b364b740 con 0x55d6b363c760 2024-03-24T10:44:33.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.380+0000 7f11d0ff6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d6b36bbf10 con 0x55d6b363d080 2024-03-24T10:44:33.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.380+0000 7f11d0ff6740 10 client.0 _ll_register_callbacks cb 0x7f11bc009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:33.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.380+0000 7f11c7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b3636e90 0x55d6b364dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.380+0000 7f11c7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b3636e90 0x55d6b364dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42656/0 (socket says 172.21.15.23:42656) 2024-03-24T10:44:33.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.380+0000 7f11c7fff640 1 -- 172.21.15.23:0/461262732 learned_addr learned my addr 172.21.15.23:0/461262732 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:33.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.380+0000 7f11cc8db640 1 --2- 172.21.15.23:0/461262732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363d080 0x55d6b364aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.380+0000 7f11c77fe640 1 --2- 172.21.15.23:0/461262732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363c760 0x55d6b364a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11c7fff640 1 -- 172.21.15.23:0/461262732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363d080 msgr2=0x55d6b364aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11c7fff640 1 --2- 172.21.15.23:0/461262732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363d080 0x55d6b364aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11c7fff640 1 -- 172.21.15.23:0/461262732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363c760 msgr2=0x55d6b364a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11c7fff640 1 --2- 172.21.15.23:0/461262732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363c760 0x55d6b364a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11c7fff640 1 -- 172.21.15.23:0/461262732 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6b35596d0 con 0x55d6b3636e90 2024-03-24T10:44:33.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11c7fff640 1 --2- 172.21.15.23:0/461262732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b3636e90 0x55d6b364dcf0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f11b80ed860 tx=0x7f11b80bc950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44f475c306ee20b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:33.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11a77fe640 1 -- 172.21.15.23:0/461262732 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b80056f0 con 0x55d6b3636e90 2024-03-24T10:44:33.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11a77fe640 1 -- 172.21.15.23:0/461262732 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f11b80bca50 con 0x55d6b3636e90 2024-03-24T10:44:33.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11d0ff6740 1 -- 172.21.15.23:0/461262732 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11b80c0ba0 con 0x55d6b3636e90 2024-03-24T10:44:33.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11a77fe640 1 -- 172.21.15.23:0/461262732 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f11b80c51f0 con 0x55d6b3636e90 2024-03-24T10:44:33.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11d0ff6740 10 client.16863 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:33.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11d0ff6740 1 -- 172.21.15.23:0/461262732 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d6b3563970 con 0x55d6b3636e90 2024-03-24T10:44:33.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11a77fe640 1 -- 172.21.15.23:0/461262732 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11b80c0ba0 con 0x55d6b3636e90 2024-03-24T10:44:33.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11a77fe640 1 -- 172.21.15.23:0/461262732 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f11b803c070 con 0x55d6b3636e90 2024-03-24T10:44:33.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11d0ff6740 10 client.16863 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:33.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11d0ff6740 10 client.16863 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:33.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11d0ff6740 20 client.16863 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:33.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/461262732 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d6b3645c60 0x55d6b36bf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11d0ff6740 4 client.16863 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:44:33.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.381+0000 7f11d0ff6740 1 -- 172.21.15.23:0/461262732 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d6b32a2860 con 0x55d6b3645c60 2024-03-24T10:44:33.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.382+0000 7f11c77fe640 1 --2- 172.21.15.23:0/461262732 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d6b3645c60 0x55d6b36bf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.382+0000 7f11a77fe640 10 client.16863 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:33.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.382+0000 7f11c77fe640 1 --2- 172.21.15.23:0/461262732 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d6b3645c60 0x55d6b36bf340 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57d868752203f4e server_cookie=96204b23ebcb89f1 in_seq=0 out_seq=0 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.383+0000 7f11a77fe640 1 -- 172.21.15.23:0/461262732 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d6b32a2860 con 0x55d6b3645c60 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.383+0000 7f11a77fe640 10 client.16863 handle_command_reply: tid=0 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.383+0000 7f11d0ff6740 1 client.16863 shutdown 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11d0ff6740 1 -- 172.21.15.23:0/461262732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b3636e90 msgr2=0x55d6b364dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/461262732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b3636e90 0x55d6b364dcf0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f11b80ed860 tx=0x7f11b80bc950 comp rx=0 tx=0).stop 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11d0ff6740 1 -- 172.21.15.23:0/461262732 shutdown_connections 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/461262732 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d6b3645c60 0x55d6b36bf340 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/461262732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d6b363d080 0x55d6b364aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/461262732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d6b363c760 0x55d6b364a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11d0ff6740 1 --2- 172.21.15.23:0/461262732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6b3636e90 0x55d6b364dcf0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11d0ff6740 1 -- 172.21.15.23:0/461262732 >> 172.21.15.23:0/461262732 conn(0x55d6b36418c0 msgr2=0x55d6b3638a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11a77fe640 0 client.16863 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11d0ff6740 1 -- 172.21.15.23:0/461262732 shutdown_connections 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11d0ff6740 1 -- 172.21.15.23:0/461262732 wait complete. 2024-03-24T10:44:33.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11d0ff6740 20 client.16863 trim_cache size 0 max 16384 2024-03-24T10:44:33.393 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:44:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:44:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "c847940a-d3fd-491e-b607-078aeb9a8a6f", 2024-03-24T10:44:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:44:33.393 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:44:33.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11ce33e640 1 -- 172.21.15.23:0/2519799746 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f11b00a2230 msgr2=0x7f11b00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11ce33e640 1 --2- 172.21.15.23:0/2519799746 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f11b00a2230 0x7f11b00a46f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f11c0005fd0 tx=0x7f11c0013ae0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11ce33e640 1 -- 172.21.15.23:0/2519799746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c81016c0 msgr2=0x7f11c8197de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.384+0000 7f11ce33e640 1 --2- 172.21.15.23:0/2519799746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c81016c0 0x7f11c8197de0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f11b8009230 tx=0x7f11b802c650 comp rx=0 tx=0).stop 2024-03-24T10:44:33.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.385+0000 7f11ce33e640 1 -- 172.21.15.23:0/2519799746 shutdown_connections 2024-03-24T10:44:33.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.385+0000 7f11ce33e640 1 --2- 172.21.15.23:0/2519799746 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f11b00a2230 0x7f11b00a46f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.385+0000 7f11ce33e640 1 --2- 172.21.15.23:0/2519799746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11c81076d0 0x7f11c819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.385+0000 7f11ce33e640 1 --2- 172.21.15.23:0/2519799746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11c8102080 0x7f11c8198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.385+0000 7f11ce33e640 1 --2- 172.21.15.23:0/2519799746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11c81016c0 0x7f11c8197de0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.385+0000 7f11ce33e640 1 -- 172.21.15.23:0/2519799746 >> 172.21.15.23:0/2519799746 conn(0x7f11c80fd150 msgr2=0x7f11c80fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.385+0000 7f11ce33e640 1 -- 172.21.15.23:0/2519799746 shutdown_connections 2024-03-24T10:44:33.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.385+0000 7f11ce33e640 1 -- 172.21.15.23:0/2519799746 wait complete. 2024-03-24T10:44:33.396 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:44:33.396 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:44:33.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.504+0000 7f3f94b8a640 1 -- 172.21.15.23:0/944177318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f90101350 msgr2=0x7f3f90101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.504+0000 7f3f94b8a640 1 --2- 172.21.15.23:0/944177318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f90101350 0x7f3f90101730 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f3f80009930 tx=0x7f3f8002bc30 comp rx=0 tx=0).stop 2024-03-24T10:44:33.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.504+0000 7f3f94b8a640 1 -- 172.21.15.23:0/944177318 shutdown_connections 2024-03-24T10:44:33.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.504+0000 7f3f94b8a640 1 --2- 172.21.15.23:0/944177318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f901059d0 0x7f3f9010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.504+0000 7f3f94b8a640 1 --2- 172.21.15.23:0/944177318 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f90101d00 0x7f3f901052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.504+0000 7f3f94b8a640 1 --2- 172.21.15.23:0/944177318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f90101350 0x7f3f90101730 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.504+0000 7f3f94b8a640 1 -- 172.21.15.23:0/944177318 >> 172.21.15.23:0/944177318 conn(0x7f3f90075950 msgr2=0x7f3f90075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.504+0000 7f3f94b8a640 1 -- 172.21.15.23:0/944177318 shutdown_connections 2024-03-24T10:44:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.504+0000 7f3f94b8a640 1 -- 172.21.15.23:0/944177318 wait complete. 2024-03-24T10:44:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.504+0000 7f3f94b8a640 1 Processor -- start 2024-03-24T10:44:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.504+0000 7f3f94b8a640 1 -- start start 2024-03-24T10:44:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f94b8a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f90101350 0x7f3f90198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f94b8a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f901059d0 0x7f3f901990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f94b8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f90199650 0x7f3f901a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f94b8a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f3f901a42d0 con 0x7f3f901059d0 2024-03-24T10:44:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f94b8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3f901a4440 con 0x7f3f90199650 2024-03-24T10:44:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f94b8a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3f901a4600 con 0x7f3f90101350 2024-03-24T10:44:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f8dd9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f901059d0 0x7f3f901990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f8dd9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f901059d0 0x7f3f901990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42664/0 (socket says 172.21.15.23:42664) 2024-03-24T10:44:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f8dd9b640 1 -- 172.21.15.23:0/617176262 learned_addr learned my addr 172.21.15.23:0/617176262 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f8e59c640 1 --2- 172.21.15.23:0/617176262 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f90101350 0x7f3f90198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f8ed9d640 1 --2- 172.21.15.23:0/617176262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f90199650 0x7f3f901a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f8dd9b640 1 -- 172.21.15.23:0/617176262 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f90101350 msgr2=0x7f3f90198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f8dd9b640 1 --2- 172.21.15.23:0/617176262 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f90101350 0x7f3f90198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f8dd9b640 1 -- 172.21.15.23:0/617176262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f90199650 msgr2=0x7f3f901a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f8dd9b640 1 --2- 172.21.15.23:0/617176262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f90199650 0x7f3f901a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f8dd9b640 1 -- 172.21.15.23:0/617176262 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f84009490 con 0x7f3f901059d0 2024-03-24T10:44:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f8dd9b640 1 --2- 172.21.15.23:0/617176262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f901059d0 0x7f3f901990c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400e970 tx=0x7f3f8400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f7f7fe640 1 -- 172.21.15.23:0/617176262 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8400cd80 con 0x7f3f901059d0 2024-03-24T10:44:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f94b8a640 1 -- 172.21.15.23:0/617176262 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f80009560 con 0x7f3f901059d0 2024-03-24T10:44:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f7f7fe640 1 -- 172.21.15.23:0/617176262 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f840035f0 con 0x7f3f901059d0 2024-03-24T10:44:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.505+0000 7f3f7f7fe640 1 -- 172.21.15.23:0/617176262 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84010690 con 0x7f3f901059d0 2024-03-24T10:44:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.506+0000 7f3f94b8a640 1 -- 172.21.15.23:0/617176262 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f901a4c60 con 0x7f3f901059d0 2024-03-24T10:44:33.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.508+0000 7f3f7f7fe640 1 -- 172.21.15.23:0/617176262 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f3f84003760 con 0x7f3f901059d0 2024-03-24T10:44:33.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.508+0000 7f3f97768740 1 Processor -- start 2024-03-24T10:44:33.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f97768740 1 -- start start 2024-03-24T10:44:33.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f97768740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1a1d0 0x558793b1a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f97768740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b1aaf0 0x558793b23190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f97768740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b236d0 0x558793b25ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f97768740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558793a3ef30 con 0x558793b1a1d0 2024-03-24T10:44:33.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f97768740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558793a88810 con 0x558793b236d0 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f97768740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558793a85d40 con 0x558793b1aaf0 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f8ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1a1d0 0x558793b1a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f8ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1a1d0 0x558793b1a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42676/0 (socket says 172.21.15.23:42676) 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f8e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b1aaf0 0x558793b23190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f8ed9d640 1 -- 172.21.15.23:0/3703094800 learned_addr learned my addr 172.21.15.23:0/3703094800 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f8dd9b640 1 --2- 172.21.15.23:0/3703094800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b236d0 0x558793b25ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f7f7fe640 1 --2- 172.21.15.23:0/617176262 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3f780a2230 0x7f3f780a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f8e59c640 1 -- 172.21.15.23:0/3703094800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b236d0 msgr2=0x558793b25ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f8e59c640 1 --2- 172.21.15.23:0/3703094800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b236d0 0x558793b25ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f8e59c640 1 -- 172.21.15.23:0/3703094800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1a1d0 msgr2=0x558793b1a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f8e59c640 1 --2- 172.21.15.23:0/3703094800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1a1d0 0x558793b1a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f8e59c640 1 -- 172.21.15.23:0/3703094800 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558793a42ea0 con 0x558793b1aaf0 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f8ed9d640 1 --2- 172.21.15.23:0/3703094800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1a1d0 0x558793b1a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.509+0000 7f3f8ed9d640 1 --2- 172.21.15.23:0/617176262 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3f780a2230 0x7f3f780a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.510+0000 7f3f7f7fe640 1 -- 172.21.15.23:0/617176262 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f3f84014070 con 0x7f3f901059d0 2024-03-24T10:44:33.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.510+0000 7f3f8e59c640 1 --2- 172.21.15.23:0/3703094800 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b1aaf0 0x558793b23190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3f80009230 tx=0x7f3f8002c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5507e9b2a0e142ea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.510+0000 7f3f7cff9640 1 -- 172.21.15.23:0/3703094800 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f80004800 con 0x558793b1aaf0 2024-03-24T10:44:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.510+0000 7f3f7cff9640 1 -- 172.21.15.23:0/3703094800 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f80004960 con 0x558793b1aaf0 2024-03-24T10:44:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.510+0000 7f3f7cff9640 1 -- 172.21.15.23:0/3703094800 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f80004ff0 con 0x558793b1aaf0 2024-03-24T10:44:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 -- 172.21.15.23:0/3703094800 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b1aaf0 msgr2=0x558793b23190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 --2- 172.21.15.23:0/3703094800 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b1aaf0 0x558793b23190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3f80009230 tx=0x7f3f8002c5d0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f8ed9d640 1 --2- 172.21.15.23:0/617176262 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3f780a2230 0x7f3f780a46f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3f880107d0 tx=0x7f3f88004a40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 -- 172.21.15.23:0/3703094800 shutdown_connections 2024-03-24T10:44:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 --2- 172.21.15.23:0/3703094800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b236d0 0x558793b25ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 --2- 172.21.15.23:0/3703094800 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b1aaf0 0x558793b23190 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 --2- 172.21.15.23:0/3703094800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1a1d0 0x558793b1a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 -- 172.21.15.23:0/3703094800 >> 172.21.15.23:0/3703094800 conn(0x558793b14900 msgr2=0x558793b14ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 -- 172.21.15.23:0/3703094800 shutdown_connections 2024-03-24T10:44:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 -- 172.21.15.23:0/3703094800 wait complete. 2024-03-24T10:44:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 -- start start 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 Processor -- start 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1a1d0 0x558793b20770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b1aaf0 0x558793b20cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b236d0 0x558793b211f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558793a81520 con 0x558793b1a1d0 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558793754430 con 0x558793b1aaf0 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.511+0000 7f3f97768740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558793710e00 con 0x558793b236d0 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f8e59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1a1d0 0x558793b20770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f97768740 10 client.0 _ll_register_callbacks cb 0x558793a1fd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f8e59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1a1d0 0x558793b20770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42690/0 (socket says 172.21.15.23:42690) 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f8e59c640 1 -- 172.21.15.23:0/967078117 learned_addr learned my addr 172.21.15.23:0/967078117 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f8ed9d640 1 --2- 172.21.15.23:0/967078117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b236d0 0x558793b211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f8dd9b640 1 --2- 172.21.15.23:0/967078117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b1aaf0 0x558793b20cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f8e59c640 1 -- 172.21.15.23:0/967078117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b236d0 msgr2=0x558793b211f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f8e59c640 1 --2- 172.21.15.23:0/967078117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b236d0 0x558793b211f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f8e59c640 1 -- 172.21.15.23:0/967078117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b1aaf0 msgr2=0x558793b20cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f8e59c640 1 --2- 172.21.15.23:0/967078117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b1aaf0 0x558793b20cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f8e59c640 1 -- 172.21.15.23:0/967078117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f9010dd20 con 0x558793b1a1d0 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f8dd9b640 1 --2- 172.21.15.23:0/967078117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b1aaf0 0x558793b20cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f8e59c640 1 --2- 172.21.15.23:0/967078117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1a1d0 0x558793b20770 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f3f8002c170 tx=0x7f3f8002c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60b92d18b7d7393 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f75ffb640 1 -- 172.21.15.23:0/967078117 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f80051070 con 0x558793b1a1d0 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f75ffb640 1 -- 172.21.15.23:0/967078117 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f3f80005000 con 0x558793b1a1d0 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f97768740 1 -- 172.21.15.23:0/967078117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5587937807e0 con 0x558793b1a1d0 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f75ffb640 1 -- 172.21.15.23:0/967078117 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3f800051c0 con 0x558793b1a1d0 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f97768740 10 client.16875 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.512+0000 7f3f97768740 1 -- 172.21.15.23:0/967078117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558793a7fba0 con 0x558793b1a1d0 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.513+0000 7f3f75ffb640 1 -- 172.21.15.23:0/967078117 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5587937807e0 con 0x558793b1a1d0 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.513+0000 7f3f75ffb640 1 -- 172.21.15.23:0/967078117 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f3f80054020 con 0x558793b1a1d0 2024-03-24T10:44:33.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.513+0000 7f3f97768740 10 client.16875 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:33.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.513+0000 7f3f97768740 10 client.16875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:33.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.513+0000 7f3f97768740 20 client.16875 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:33.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.513+0000 7f3f97768740 1 --2- 172.21.15.23:0/967078117 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558793b33880 0x558793b35c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.513+0000 7f3f97768740 4 client.16875 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:44:33.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.513+0000 7f3f97768740 1 -- 172.21.15.23:0/967078117 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587937bd720 con 0x558793b33880 2024-03-24T10:44:33.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.513+0000 7f3f8dd9b640 1 --2- 172.21.15.23:0/967078117 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558793b33880 0x558793b35c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.513+0000 7f3f8dd9b640 1 --2- 172.21.15.23:0/967078117 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558793b33880 0x558793b35c90 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=557ec28a943279d4 server_cookie=2e8a63be450bc1f5 in_seq=0 out_seq=0 2024-03-24T10:44:33.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.513+0000 7f3f75ffb640 10 client.16875 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:33.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.514+0000 7f3f75ffb640 1 -- 172.21.15.23:0/967078117 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5587937bd720 con 0x558793b33880 2024-03-24T10:44:33.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.514+0000 7f3f75ffb640 10 client.16875 handle_command_reply: tid=0 2024-03-24T10:44:33.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.514+0000 7f3f97768740 1 client.16875 shutdown 2024-03-24T10:44:33.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.515+0000 7f3f97768740 1 -- 172.21.15.23:0/967078117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1a1d0 msgr2=0x558793b20770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.515+0000 7f3f97768740 1 --2- 172.21.15.23:0/967078117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1a1d0 0x558793b20770 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f3f8002c170 tx=0x7f3f8002c740 comp rx=0 tx=0).stop 2024-03-24T10:44:33.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.515+0000 7f3f97768740 1 -- 172.21.15.23:0/967078117 shutdown_connections 2024-03-24T10:44:33.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.515+0000 7f3f97768740 1 --2- 172.21.15.23:0/967078117 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558793b33880 0x558793b35c90 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.515+0000 7f3f97768740 1 --2- 172.21.15.23:0/967078117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b236d0 0x558793b211f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.515+0000 7f3f97768740 1 --2- 172.21.15.23:0/967078117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b1aaf0 0x558793b20cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.515+0000 7f3f75ffb640 0 client.16875 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:33.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.515+0000 7f3f97768740 1 --2- 172.21.15.23:0/967078117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1a1d0 0x558793b20770 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.515+0000 7f3f97768740 1 -- 172.21.15.23:0/967078117 >> 172.21.15.23:0/967078117 conn(0x558793b14900 msgr2=0x558793b13b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.515+0000 7f3f97768740 1 -- 172.21.15.23:0/967078117 shutdown_connections 2024-03-24T10:44:33.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.515+0000 7f3f97768740 1 -- 172.21.15.23:0/967078117 wait complete. 2024-03-24T10:44:33.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.515+0000 7f3f97768740 20 client.16875 trim_cache size 0 max 16384 2024-03-24T10:44:33.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.533+0000 7f3f97768740 1 Processor -- start 2024-03-24T10:44:33.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.533+0000 7f3f97768740 1 -- start start 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.533+0000 7f3f97768740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b1a1d0 0x558793b22ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.533+0000 7f3f97768740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1aaf0 0x558793b2ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.533+0000 7f3f97768740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b236d0 0x558793b139e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.533+0000 7f3f97768740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558793a06430 con 0x558793b1aaf0 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.533+0000 7f3f97768740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558793b15900 con 0x558793b1a1d0 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.533+0000 7f3f97768740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558793b10550 con 0x558793b236d0 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f8dd9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1aaf0 0x558793b2ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f8ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b236d0 0x558793b139e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f8e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b1a1d0 0x558793b22ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f8dd9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1aaf0 0x558793b2ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42706/0 (socket says 172.21.15.23:42706) 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f8ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b236d0 0x558793b139e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39786/0 (socket says 172.21.15.23:39786) 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f8dd9b640 1 -- 172.21.15.23:0/919006433 learned_addr learned my addr 172.21.15.23:0/919006433 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f8dd9b640 1 -- 172.21.15.23:0/919006433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b236d0 msgr2=0x558793b139e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f8dd9b640 1 --2- 172.21.15.23:0/919006433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b236d0 0x558793b139e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f8dd9b640 1 -- 172.21.15.23:0/919006433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b1a1d0 msgr2=0x558793b22ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f8dd9b640 1 --2- 172.21.15.23:0/919006433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b1a1d0 0x558793b22ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f8dd9b640 1 -- 172.21.15.23:0/919006433 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558793a1bb20 con 0x558793b1aaf0 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f8dd9b640 1 --2- 172.21.15.23:0/919006433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1aaf0 0x558793b2ef00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3f84097320 tx=0x7f3f840971e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb6f163aa68b47c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f77fff640 1 -- 172.21.15.23:0/919006433 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f840d1e20 con 0x558793b1aaf0 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f77fff640 1 -- 172.21.15.23:0/919006433 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f840a3660 con 0x558793b1aaf0 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.534+0000 7f3f77fff640 1 -- 172.21.15.23:0/919006433 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84097530 con 0x558793b1aaf0 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 -- 172.21.15.23:0/919006433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1aaf0 msgr2=0x558793b2ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 --2- 172.21.15.23:0/919006433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1aaf0 0x558793b2ef00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3f84097320 tx=0x7f3f840971e0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 -- 172.21.15.23:0/919006433 shutdown_connections 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 --2- 172.21.15.23:0/919006433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b236d0 0x558793b139e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 --2- 172.21.15.23:0/919006433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b1aaf0 0x558793b2ef00 secure :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3f84097320 tx=0x7f3f840971e0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 --2- 172.21.15.23:0/919006433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b1a1d0 0x558793b22ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 -- 172.21.15.23:0/919006433 >> 172.21.15.23:0/919006433 conn(0x558793b14900 msgr2=0x558793b132d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 -- 172.21.15.23:0/919006433 shutdown_connections 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 -- 172.21.15.23:0/919006433 wait complete. 2024-03-24T10:44:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 -- start start 2024-03-24T10:44:33.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 Processor -- start 2024-03-24T10:44:33.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b1a1d0 0x558793b2b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b236d0 0x558793b28500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b28a40 0x558793b28ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558793b99ba0 con 0x558793b236d0 2024-03-24T10:44:33.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558793b99ce0 con 0x558793b28a40 2024-03-24T10:44:33.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.535+0000 7f3f97768740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558793b99e20 con 0x558793b1a1d0 2024-03-24T10:44:33.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f97768740 10 client.0 _ll_register_callbacks cb 0x7f3f88009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:33.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f8dd9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b236d0 0x558793b28500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f8dd9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b236d0 0x558793b28500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42708/0 (socket says 172.21.15.23:42708) 2024-03-24T10:44:33.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f8dd9b640 1 -- 172.21.15.23:0/848390419 learned_addr learned my addr 172.21.15.23:0/848390419 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:33.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f8ed9d640 1 --2- 172.21.15.23:0/848390419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b28a40 0x558793b28ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f8e59c640 1 --2- 172.21.15.23:0/848390419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b1a1d0 0x558793b2b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f8dd9b640 1 -- 172.21.15.23:0/848390419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b1a1d0 msgr2=0x558793b2b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f8dd9b640 1 --2- 172.21.15.23:0/848390419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b1a1d0 0x558793b2b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f8dd9b640 1 -- 172.21.15.23:0/848390419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b28a40 msgr2=0x558793b28ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f8dd9b640 1 --2- 172.21.15.23:0/848390419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b28a40 0x558793b28ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f8dd9b640 1 -- 172.21.15.23:0/848390419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f901a4c60 con 0x558793b236d0 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f8dd9b640 1 --2- 172.21.15.23:0/848390419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b236d0 0x558793b28500 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f3f84097320 tx=0x7f3f840d1b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc714d60f373259 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f75ffb640 1 -- 172.21.15.23:0/848390419 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8409f010 con 0x558793b236d0 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f75ffb640 1 -- 172.21.15.23:0/848390419 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f3f840991c0 con 0x558793b236d0 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f97768740 1 -- 172.21.15.23:0/848390419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f80004960 con 0x558793b236d0 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f75ffb640 1 -- 172.21.15.23:0/848390419 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3f840b9310 con 0x558793b236d0 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f97768740 10 client.16881 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.536+0000 7f3f97768740 1 -- 172.21.15.23:0/848390419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558793a41700 con 0x558793b236d0 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.537+0000 7f3f75ffb640 1 -- 172.21.15.23:0/848390419 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f8407d020 con 0x558793b236d0 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.537+0000 7f3f75ffb640 1 -- 172.21.15.23:0/848390419 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f3f8401f060 con 0x558793b236d0 2024-03-24T10:44:33.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.537+0000 7f3f97768740 10 client.16881 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:33.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.537+0000 7f3f97768740 10 client.16881 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:33.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.537+0000 7f3f97768740 20 client.16881 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:33.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.537+0000 7f3f97768740 1 --2- 172.21.15.23:0/848390419 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558793b9b2c0 0x558793b9d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.537+0000 7f3f97768740 4 client.16881 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:44:33.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.537+0000 7f3f97768740 1 -- 172.21.15.23:0/848390419 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55879380c890 con 0x558793b9b2c0 2024-03-24T10:44:33.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.537+0000 7f3f8e59c640 1 --2- 172.21.15.23:0/848390419 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558793b9b2c0 0x558793b9d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.537+0000 7f3f75ffb640 10 client.16881 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:33.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.537+0000 7f3f8e59c640 1 --2- 172.21.15.23:0/848390419 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558793b9b2c0 0x558793b9d6e0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83522c010249592 server_cookie=3f067468056fe3a8 in_seq=0 out_seq=0 2024-03-24T10:44:33.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f75ffb640 1 -- 172.21.15.23:0/848390419 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55879380c890 con 0x558793b9b2c0 2024-03-24T10:44:33.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f75ffb640 10 client.16881 handle_command_reply: tid=0 2024-03-24T10:44:33.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f97768740 1 client.16881 shutdown 2024-03-24T10:44:33.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f97768740 1 -- 172.21.15.23:0/848390419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b236d0 msgr2=0x558793b28500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f97768740 1 --2- 172.21.15.23:0/848390419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b236d0 0x558793b28500 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f3f84097320 tx=0x7f3f840d1b80 comp rx=0 tx=0).stop 2024-03-24T10:44:33.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f97768740 1 -- 172.21.15.23:0/848390419 shutdown_connections 2024-03-24T10:44:33.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f97768740 1 --2- 172.21.15.23:0/848390419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558793b28a40 0x558793b28ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f97768740 1 --2- 172.21.15.23:0/848390419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558793b236d0 0x558793b28500 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f97768740 1 --2- 172.21.15.23:0/848390419 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x558793b9b2c0 0x558793b9d6e0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f97768740 1 --2- 172.21.15.23:0/848390419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558793b1a1d0 0x558793b2b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f75ffb640 0 client.16881 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:33.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f97768740 1 -- 172.21.15.23:0/848390419 >> 172.21.15.23:0/848390419 conn(0x558793b14900 msgr2=0x558793b13b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f97768740 1 -- 172.21.15.23:0/848390419 shutdown_connections 2024-03-24T10:44:33.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f97768740 1 -- 172.21.15.23:0/848390419 wait complete. 2024-03-24T10:44:33.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f97768740 20 client.16881 trim_cache size 0 max 16384 2024-03-24T10:44:33.544 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:44:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:44:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:44:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: "c847940a-d3fd-491e-b607-078aeb9a8a6f": { 2024-03-24T10:44:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:44:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "c847940a-d3fd-491e-b607-078aeb9a8a6f", 2024-03-24T10:44:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:44:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:44:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:44:33.545 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:44:33.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f94b8a640 1 -- 172.21.15.23:0/617176262 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3f780a2230 msgr2=0x7f3f780a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f94b8a640 1 --2- 172.21.15.23:0/617176262 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3f780a2230 0x7f3f780a46f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3f880107d0 tx=0x7f3f88004a40 comp rx=0 tx=0).stop 2024-03-24T10:44:33.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f94b8a640 1 -- 172.21.15.23:0/617176262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f901059d0 msgr2=0x7f3f901990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f94b8a640 1 --2- 172.21.15.23:0/617176262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f901059d0 0x7f3f901990c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400e970 tx=0x7f3f8400ee30 comp rx=0 tx=0).stop 2024-03-24T10:44:33.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f94b8a640 1 -- 172.21.15.23:0/617176262 shutdown_connections 2024-03-24T10:44:33.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f94b8a640 1 --2- 172.21.15.23:0/617176262 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3f780a2230 0x7f3f780a46f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f94b8a640 1 --2- 172.21.15.23:0/617176262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f90199650 0x7f3f901a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f94b8a640 1 --2- 172.21.15.23:0/617176262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f901059d0 0x7f3f901990c0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f94b8a640 1 --2- 172.21.15.23:0/617176262 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f90101350 0x7f3f90198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f94b8a640 1 -- 172.21.15.23:0/617176262 >> 172.21.15.23:0/617176262 conn(0x7f3f90075950 msgr2=0x7f3f900fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f94b8a640 1 -- 172.21.15.23:0/617176262 shutdown_connections 2024-03-24T10:44:33.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:44:33.538+0000 7f3f94b8a640 1 -- 172.21.15.23:0/617176262 wait complete. 2024-03-24T10:44:33.549 INFO:tasks.cephfs.filesystem:scrub status for tag:c847940a-d3fd-491e-b607-078aeb9a8a6f - {'path': '/', 'tag': 'c847940a-d3fd-491e-b607-078aeb9a8a6f', 'options': 'recursive,force'} 2024-03-24T10:44:34.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 386813 11.97 MB/sec execute 51 sec latency 10.916 ms 2024-03-24T10:44:34.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:34 smithi023 ceph-mon[32466]: pgmap v137: 97 pgs: 97 active+clean; 216 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 201 op/s 2024-03-24T10:44:34.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:34 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:44:34.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:34 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:34.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:34 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:44:34.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:34 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:44:34.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:34 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:34.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:34 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:35.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:34 smithi117 ceph-mon[36368]: pgmap v137: 97 pgs: 97 active+clean; 216 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 201 op/s 2024-03-24T10:44:35.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:34 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:44:35.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:34 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:35.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:34 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:44:35.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:34 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:44:35.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:34 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:35.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:34 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:35.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:34 smithi116 ceph-mon[36248]: pgmap v137: 97 pgs: 97 active+clean; 216 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 201 op/s 2024-03-24T10:44:35.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:34 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:44:35.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:34 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:35.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:34 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:44:35.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:34 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:44:35.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:34 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:35.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:34 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:44:35.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 389022 11.90 MB/sec execute 52 sec latency 22.346 ms 2024-03-24T10:44:36.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 390978 11.91 MB/sec execute 53 sec latency 18.577 ms 2024-03-24T10:44:36.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.703+0000 7f1b4d381640 1 -- 172.21.15.23:0/1013947417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b481076d0 msgr2=0x7f1b48107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.703+0000 7f1b4d381640 1 --2- 172.21.15.23:0/1013947417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b481076d0 0x7f1b48107ab0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1b38009890 tx=0x7f1b3802bb70 comp rx=0 tx=0).stop 2024-03-24T10:44:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.703+0000 7f1b4d381640 1 -- 172.21.15.23:0/1013947417 shutdown_connections 2024-03-24T10:44:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.703+0000 7f1b4d381640 1 --2- 172.21.15.23:0/1013947417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b48102080 0x7f1b4810e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.703+0000 7f1b4d381640 1 --2- 172.21.15.23:0/1013947417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b481016c0 0x7f1b48101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.703+0000 7f1b4d381640 1 --2- 172.21.15.23:0/1013947417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b481076d0 0x7f1b48107ab0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.703+0000 7f1b4d381640 1 -- 172.21.15.23:0/1013947417 >> 172.21.15.23:0/1013947417 conn(0x7f1b480fd1d0 msgr2=0x7f1b480ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.704+0000 7f1b4d381640 1 -- 172.21.15.23:0/1013947417 shutdown_connections 2024-03-24T10:44:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.704+0000 7f1b4d381640 1 -- 172.21.15.23:0/1013947417 wait complete. 2024-03-24T10:44:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.704+0000 7f1b4d381640 1 Processor -- start 2024-03-24T10:44:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.704+0000 7f1b4d381640 1 -- start start 2024-03-24T10:44:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.704+0000 7f1b4d381640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b481016c0 0x7f1b4819d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.704+0000 7f1b4d381640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b48102080 0x7f1b4819de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.704+0000 7f1b4d381640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b481076d0 0x7f1b48197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.704+0000 7f1b4d381640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f1b481980f0 con 0x7f1b48102080 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.704+0000 7f1b4d381640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1b48198260 con 0x7f1b481076d0 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.704+0000 7f1b4d381640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1b48198420 con 0x7f1b481016c0 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.704+0000 7f1b3ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b48102080 0x7f1b4819de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.704+0000 7f1b3ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b48102080 0x7f1b4819de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42718/0 (socket says 172.21.15.23:42718) 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.704+0000 7f1b3ffff640 1 -- 172.21.15.23:0/954107114 learned_addr learned my addr 172.21.15.23:0/954107114 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.705+0000 7f1b477fe640 1 --2- 172.21.15.23:0/954107114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b481076d0 0x7f1b48197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.705+0000 7f1b46ffd640 1 --2- 172.21.15.23:0/954107114 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b481016c0 0x7f1b4819d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.705+0000 7f1b3ffff640 1 -- 172.21.15.23:0/954107114 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b481016c0 msgr2=0x7f1b4819d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.705+0000 7f1b3ffff640 1 --2- 172.21.15.23:0/954107114 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b481016c0 0x7f1b4819d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.705+0000 7f1b3ffff640 1 -- 172.21.15.23:0/954107114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b481076d0 msgr2=0x7f1b48197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.705+0000 7f1b3ffff640 1 --2- 172.21.15.23:0/954107114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b481076d0 0x7f1b48197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.705+0000 7f1b3ffff640 1 -- 172.21.15.23:0/954107114 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b30009560 con 0x7f1b48102080 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.705+0000 7f1b3ffff640 1 --2- 172.21.15.23:0/954107114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b48102080 0x7f1b4819de30 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1b3000e9c0 tx=0x7f1b3000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.705+0000 7f1b44ff9640 1 -- 172.21.15.23:0/954107114 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3000cde0 con 0x7f1b48102080 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.705+0000 7f1b44ff9640 1 -- 172.21.15.23:0/954107114 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b300035f0 con 0x7f1b48102080 2024-03-24T10:44:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.705+0000 7f1b44ff9640 1 -- 172.21.15.23:0/954107114 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b30010670 con 0x7f1b48102080 2024-03-24T10:44:36.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.705+0000 7f1b4d381640 1 -- 172.21.15.23:0/954107114 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b38009490 con 0x7f1b48102080 2024-03-24T10:44:36.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.705+0000 7f1b4d381640 1 -- 172.21.15.23:0/954107114 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b481a47b0 con 0x7f1b48102080 2024-03-24T10:44:36.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.708+0000 7f1b44ff9640 1 -- 172.21.15.23:0/954107114 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f1b30003ad0 con 0x7f1b48102080 2024-03-24T10:44:36.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.708+0000 7f1b4fbfd740 1 Processor -- start 2024-03-24T10:44:36.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.708+0000 7f1b4fbfd740 1 -- start start 2024-03-24T10:44:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b44ff9640 1 --2- 172.21.15.23:0/954107114 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1b340a2350 0x7f1b340a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b4fbfd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b774380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b4fbfd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7748c0 0x55fb2b781ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b4fbfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7820e0 0x55fb2b7844d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b4fbfd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fb2b72f1b0 con 0x55fb2b7748c0 2024-03-24T10:44:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b4fbfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fb2b6e1f80 con 0x55fb2b7820e0 2024-03-24T10:44:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b4fbfd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fb2b6df4b0 con 0x55fb2b773fa0 2024-03-24T10:44:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b3ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7748c0 0x55fb2b781ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b44ff9640 1 -- 172.21.15.23:0/954107114 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f1b30014070 con 0x7f1b48102080 2024-03-24T10:44:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b477fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b774380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b3ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7748c0 0x55fb2b781ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42728/0 (socket says 172.21.15.23:42728) 2024-03-24T10:44:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b3ffff640 1 -- 172.21.15.23:0/6367942 learned_addr learned my addr 172.21.15.23:0/6367942 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b46ffd640 1 --2- 172.21.15.23:0/954107114 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1b340a2350 0x7f1b340a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b477fe640 1 --2- 172.21.15.23:0/6367942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7820e0 0x55fb2b7844d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b477fe640 1 -- 172.21.15.23:0/6367942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7820e0 msgr2=0x55fb2b7844d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b477fe640 1 --2- 172.21.15.23:0/6367942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7820e0 0x55fb2b7844d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b477fe640 1 -- 172.21.15.23:0/6367942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7748c0 msgr2=0x55fb2b781ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b477fe640 1 --2- 172.21.15.23:0/6367942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7748c0 0x55fb2b781ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b477fe640 1 -- 172.21.15.23:0/6367942 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb2b670810 con 0x55fb2b773fa0 2024-03-24T10:44:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b3ffff640 1 --2- 172.21.15.23:0/6367942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7748c0 0x55fb2b781ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.709+0000 7f1b477fe640 1 --2- 172.21.15.23:0/6367942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b774380 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1b400119f0 tx=0x7f1b40004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb4c697ade13cf94 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.710+0000 7f1b46ffd640 1 --2- 172.21.15.23:0/954107114 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1b340a2350 0x7f1b340a4810 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1b48198be0 tx=0x7f1b380058d0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.710+0000 7f1b3d7fa640 1 -- 172.21.15.23:0/6367942 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b40011050 con 0x55fb2b773fa0 2024-03-24T10:44:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.710+0000 7f1b3d7fa640 1 -- 172.21.15.23:0/6367942 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b40003e50 con 0x55fb2b773fa0 2024-03-24T10:44:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.710+0000 7f1b3d7fa640 1 -- 172.21.15.23:0/6367942 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4000fe80 con 0x55fb2b773fa0 2024-03-24T10:44:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.710+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/6367942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 msgr2=0x55fb2b774380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.710+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/6367942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b774380 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1b400119f0 tx=0x7f1b40004ab0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/6367942 shutdown_connections 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/6367942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7820e0 0x55fb2b7844d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/6367942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7748c0 0x55fb2b781ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/6367942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b774380 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/6367942 >> 172.21.15.23:0/6367942 conn(0x55fb2b76ef00 msgr2=0x55fb2b76f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/6367942 shutdown_connections 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/6367942 wait complete. 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 -- start start 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 Processor -- start 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b77f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7748c0 0x55fb2b77f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fb2b6dac90 con 0x55fb2b7820e0 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fb2b67fc50 con 0x55fb2b7748c0 2024-03-24T10:44:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fb2b3ae250 con 0x55fb2b773fa0 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b4fbfd740 10 client.0 _ll_register_callbacks cb 0x55fb2b6ce290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b477fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b46ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b77f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b3ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7748c0 0x55fb2b77f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b46ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b77f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39836/0 (socket says 172.21.15.23:39836) 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b477fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42744/0 (socket says 172.21.15.23:42744) 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.711+0000 7f1b46ffd640 1 -- 172.21.15.23:0/2814440668 learned_addr learned my addr 172.21.15.23:0/2814440668 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.712+0000 7f1b477fe640 1 -- 172.21.15.23:0/2814440668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 msgr2=0x55fb2b77f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.712+0000 7f1b477fe640 1 --2- 172.21.15.23:0/2814440668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b77f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.712+0000 7f1b477fe640 1 -- 172.21.15.23:0/2814440668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7748c0 msgr2=0x55fb2b77f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.712+0000 7f1b477fe640 1 --2- 172.21.15.23:0/2814440668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7748c0 0x55fb2b77f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.712+0000 7f1b477fe640 1 -- 172.21.15.23:0/2814440668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b40019040 con 0x55fb2b7820e0 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.712+0000 7f1b477fe640 1 --2- 172.21.15.23:0/2814440668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77fbb0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1b40005fd0 tx=0x7f1b4000fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f094a7619ca392f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.712+0000 7f1b2e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.712+0000 7f1b2e7fc640 1 -- 172.21.15.23:0/2814440668 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b40003b80 con 0x55fb2b7820e0 2024-03-24T10:44:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.712+0000 7f1b2e7fc640 1 -- 172.21.15.23:0/2814440668 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f1b40004b20 con 0x55fb2b7820e0 2024-03-24T10:44:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.712+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2814440668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fb2b6ff790 con 0x55fb2b7820e0 2024-03-24T10:44:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.712+0000 7f1b2e7fc640 1 -- 172.21.15.23:0/2814440668 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1b40003ce0 con 0x55fb2b7820e0 2024-03-24T10:44:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.712+0000 7f1b4fbfd740 10 client.16893 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.712+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2814440668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fb2b6d9310 con 0x55fb2b7820e0 2024-03-24T10:44:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.713+0000 7f1b2e7fc640 1 -- 172.21.15.23:0/2814440668 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fb2b6ff790 con 0x55fb2b7820e0 2024-03-24T10:44:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.713+0000 7f1b2e7fc640 1 -- 172.21.15.23:0/2814440668 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f1b4001f770 con 0x55fb2b7820e0 2024-03-24T10:44:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.713+0000 7f1b4fbfd740 10 client.16893 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.713+0000 7f1b4fbfd740 10 client.16893 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.713+0000 7f1b4fbfd740 20 client.16893 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.713+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2814440668 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fb2b791ef0 0x55fb2b7942e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.713+0000 7f1b4fbfd740 4 client.16893 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:44:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.713+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2814440668 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb2b3da600 con 0x55fb2b791ef0 2024-03-24T10:44:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.713+0000 7f1b46ffd640 1 --2- 172.21.15.23:0/2814440668 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fb2b791ef0 0x55fb2b7942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.714+0000 7f1b2e7fc640 10 client.16893 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.714+0000 7f1b46ffd640 1 --2- 172.21.15.23:0/2814440668 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fb2b791ef0 0x55fb2b7942e0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc7bae4be5f4a835 server_cookie=440b480c71228582 in_seq=0 out_seq=0 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b2e7fc640 1 -- 172.21.15.23:0/2814440668 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fb2b3da600 con 0x55fb2b791ef0 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b2e7fc640 10 client.16893 handle_command_reply: tid=0 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b4fbfd740 1 client.16893 shutdown 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2814440668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 msgr2=0x55fb2b77fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2814440668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77fbb0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1b40005fd0 tx=0x7f1b4000fd10 comp rx=0 tx=0).stop 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2814440668 shutdown_connections 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2814440668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77fbb0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2814440668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7748c0 0x55fb2b77f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2814440668 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fb2b791ef0 0x55fb2b7942e0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2814440668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b77f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b2e7fc640 0 client.16893 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2814440668 >> 172.21.15.23:0/2814440668 conn(0x55fb2b76ef00 msgr2=0x55fb2b770c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2814440668 shutdown_connections 2024-03-24T10:44:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2814440668 wait complete. 2024-03-24T10:44:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.715+0000 7f1b4fbfd740 20 client.16893 trim_cache size 0 max 16384 2024-03-24T10:44:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.733+0000 7f1b4fbfd740 1 Processor -- start 2024-03-24T10:44:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.733+0000 7f1b4fbfd740 1 -- start start 2024-03-24T10:44:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.733+0000 7f1b4fbfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b773fa0 0x55fb2b781860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.733+0000 7f1b4fbfd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b7748c0 0x55fb2b78d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.733+0000 7f1b4fbfd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.733+0000 7f1b4fbfd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fb2b660660 con 0x55fb2b7820e0 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.733+0000 7f1b4fbfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fb2b779fa0 con 0x55fb2b773fa0 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.733+0000 7f1b4fbfd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fb2b78dcf0 con 0x55fb2b7748c0 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.733+0000 7f1b477fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.733+0000 7f1b477fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42758/0 (socket says 172.21.15.23:42758) 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.733+0000 7f1b3ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b7748c0 0x55fb2b78d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.733+0000 7f1b477fe640 1 -- 172.21.15.23:0/2596979055 learned_addr learned my addr 172.21.15.23:0/2596979055 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.733+0000 7f1b46ffd640 1 --2- 172.21.15.23:0/2596979055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b773fa0 0x55fb2b781860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.733+0000 7f1b3ffff640 1 -- 172.21.15.23:0/2596979055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b773fa0 msgr2=0x55fb2b781860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.734+0000 7f1b3ffff640 1 --2- 172.21.15.23:0/2596979055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b773fa0 0x55fb2b781860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.734+0000 7f1b3ffff640 1 -- 172.21.15.23:0/2596979055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 msgr2=0x55fb2b77f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.734+0000 7f1b3ffff640 1 --2- 172.21.15.23:0/2596979055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.734+0000 7f1b3ffff640 1 -- 172.21.15.23:0/2596979055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb2b675810 con 0x55fb2b7748c0 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.734+0000 7f1b477fe640 1 --2- 172.21.15.23:0/2596979055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.734+0000 7f1b3ffff640 1 --2- 172.21.15.23:0/2596979055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b7748c0 0x55fb2b78d7b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1b30097810 tx=0x7f1b30003ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16eb42c9832ec3e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.734+0000 7f1b3cff9640 1 -- 172.21.15.23:0/2596979055 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b300b3d50 con 0x55fb2b7748c0 2024-03-24T10:44:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.734+0000 7f1b3cff9640 1 -- 172.21.15.23:0/2596979055 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b300aee90 con 0x55fb2b7748c0 2024-03-24T10:44:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.734+0000 7f1b3cff9640 1 -- 172.21.15.23:0/2596979055 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b300a3650 con 0x55fb2b7748c0 2024-03-24T10:44:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2596979055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b7748c0 msgr2=0x55fb2b78d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2596979055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b7748c0 0x55fb2b78d7b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1b30097810 tx=0x7f1b30003ed0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2596979055 shutdown_connections 2024-03-24T10:44:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2596979055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2596979055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b7748c0 0x55fb2b78d7b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2596979055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b773fa0 0x55fb2b781860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2596979055 >> 172.21.15.23:0/2596979055 conn(0x55fb2b76ef00 msgr2=0x55fb2b76f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2596979055 shutdown_connections 2024-03-24T10:44:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2596979055 wait complete. 2024-03-24T10:44:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 -- start start 2024-03-24T10:44:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 Processor -- start 2024-03-24T10:44:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b77bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7748c0 0x55fb2b77c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fb2b78a610 con 0x55fb2b7820e0 2024-03-24T10:44:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fb2b7f7140 con 0x55fb2b7748c0 2024-03-24T10:44:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fb2b7f7280 con 0x55fb2b773fa0 2024-03-24T10:44:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.735+0000 7f1b4fbfd740 10 client.0 _ll_register_callbacks cb 0x7f1b300046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b477fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b46ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b77bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b477fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42772/0 (socket says 172.21.15.23:42772) 2024-03-24T10:44:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b477fe640 1 -- 172.21.15.23:0/2835133587 learned_addr learned my addr 172.21.15.23:0/2835133587 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b3ffff640 1 --2- 172.21.15.23:0/2835133587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7748c0 0x55fb2b77c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b477fe640 1 -- 172.21.15.23:0/2835133587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 msgr2=0x55fb2b77bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b477fe640 1 --2- 172.21.15.23:0/2835133587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b77bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b477fe640 1 -- 172.21.15.23:0/2835133587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7748c0 msgr2=0x55fb2b77c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b477fe640 1 --2- 172.21.15.23:0/2835133587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7748c0 0x55fb2b77c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b477fe640 1 -- 172.21.15.23:0/2835133587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb2b6d9310 con 0x55fb2b7820e0 2024-03-24T10:44:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b46ffd640 1 --2- 172.21.15.23:0/2835133587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b77bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b477fe640 1 --2- 172.21.15.23:0/2835133587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77c620 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f1b4001fb00 tx=0x7f1b4000ffb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2dc86272c6a21e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b2e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b2e7fc640 1 -- 172.21.15.23:0/2835133587 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b40003cc0 con 0x55fb2b7820e0 2024-03-24T10:44:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b2e7fc640 1 -- 172.21.15.23:0/2835133587 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f1b4001f650 con 0x55fb2b7820e0 2024-03-24T10:44:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2835133587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b40003e50 con 0x55fb2b7820e0 2024-03-24T10:44:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b2e7fc640 1 -- 172.21.15.23:0/2835133587 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1b4001f830 con 0x55fb2b7820e0 2024-03-24T10:44:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b4fbfd740 10 client.16899 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.736+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2835133587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x7f1b38009490 con 0x55fb2b7820e0 2024-03-24T10:44:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.737+0000 7f1b2e7fc640 1 -- 172.21.15.23:0/2835133587 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b40003e50 con 0x55fb2b7820e0 2024-03-24T10:44:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.737+0000 7f1b2e7fc640 1 -- 172.21.15.23:0/2835133587 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f1b40027560 con 0x55fb2b7820e0 2024-03-24T10:44:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.737+0000 7f1b4fbfd740 10 client.16899 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.737+0000 7f1b4fbfd740 10 client.16899 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.737+0000 7f1b4fbfd740 20 client.16899 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.737+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2835133587 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fb2b7f8680 0x55fb2b7faa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.737+0000 7f1b4fbfd740 4 client.16899 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:44:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.737+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2835133587 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fb2b42b1b0 con 0x55fb2b7f8680 2024-03-24T10:44:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.737+0000 7f1b46ffd640 1 --2- 172.21.15.23:0/2835133587 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fb2b7f8680 0x55fb2b7faa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.737+0000 7f1b46ffd640 1 --2- 172.21.15.23:0/2835133587 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fb2b7f8680 0x55fb2b7faa70 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f12a15983f3b1118 server_cookie=27c16c57f85b0584 in_seq=0 out_seq=0 2024-03-24T10:44:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.737+0000 7f1b2e7fc640 10 client.16899 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.746+0000 7f1b2e7fc640 1 -- 172.21.15.23:0/2835133587 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fb2b42b1b0 con 0x55fb2b7f8680 2024-03-24T10:44:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.746+0000 7f1b2e7fc640 10 client.16899 handle_command_reply: tid=0 2024-03-24T10:44:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.746+0000 7f1b4fbfd740 1 client.16899 shutdown 2024-03-24T10:44:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.746+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2835133587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 msgr2=0x55fb2b77c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.746+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2835133587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77c620 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f1b4001fb00 tx=0x7f1b4000ffb0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.746+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2835133587 shutdown_connections 2024-03-24T10:44:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.746+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2835133587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb2b7820e0 0x55fb2b77c620 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.746+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2835133587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb2b7748c0 0x55fb2b77c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.746+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2835133587 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55fb2b7f8680 0x55fb2b7faa70 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.746+0000 7f1b4fbfd740 1 --2- 172.21.15.23:0/2835133587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb2b773fa0 0x55fb2b77bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.746+0000 7f1b2e7fc640 0 client.16899 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.746+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2835133587 >> 172.21.15.23:0/2835133587 conn(0x55fb2b76ef00 msgr2=0x55fb2b76d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.746+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2835133587 shutdown_connections 2024-03-24T10:44:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.746+0000 7f1b4fbfd740 1 -- 172.21.15.23:0/2835133587 wait complete. 2024-03-24T10:44:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.747+0000 7f1b4fbfd740 20 client.16899 trim_cache size 0 max 16384 2024-03-24T10:44:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:44:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:44:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:44:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:44:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.747+0000 7f1b4d381640 1 -- 172.21.15.23:0/954107114 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1b340a2350 msgr2=0x7f1b340a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.747+0000 7f1b4d381640 1 --2- 172.21.15.23:0/954107114 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1b340a2350 0x7f1b340a4810 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1b48198be0 tx=0x7f1b380058d0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.747+0000 7f1b4d381640 1 -- 172.21.15.23:0/954107114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b48102080 msgr2=0x7f1b4819de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.747+0000 7f1b4d381640 1 --2- 172.21.15.23:0/954107114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b48102080 0x7f1b4819de30 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1b3000e9c0 tx=0x7f1b3000ee80 comp rx=0 tx=0).stop 2024-03-24T10:44:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.747+0000 7f1b4d381640 1 -- 172.21.15.23:0/954107114 shutdown_connections 2024-03-24T10:44:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.747+0000 7f1b4d381640 1 --2- 172.21.15.23:0/954107114 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1b340a2350 0x7f1b340a4810 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.747+0000 7f1b4d381640 1 --2- 172.21.15.23:0/954107114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b481076d0 0x7f1b48197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.747+0000 7f1b4d381640 1 --2- 172.21.15.23:0/954107114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b48102080 0x7f1b4819de30 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.747+0000 7f1b4d381640 1 --2- 172.21.15.23:0/954107114 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b481016c0 0x7f1b4819d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.747+0000 7f1b4d381640 1 -- 172.21.15.23:0/954107114 >> 172.21.15.23:0/954107114 conn(0x7f1b480fd1d0 msgr2=0x7f1b480fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.747+0000 7f1b4d381640 1 -- 172.21.15.23:0/954107114 shutdown_connections 2024-03-24T10:44:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:36.747+0000 7f1b4d381640 1 -- 172.21.15.23:0/954107114 wait complete. 2024-03-24T10:44:36.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:36 smithi023 ceph-mon[32466]: pgmap v138: 97 pgs: 97 active+clean; 216 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 201 op/s 2024-03-24T10:44:37.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:36 smithi117 ceph-mon[36368]: pgmap v138: 97 pgs: 97 active+clean; 216 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 201 op/s 2024-03-24T10:44:37.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:36 smithi116 ceph-mon[36248]: pgmap v138: 97 pgs: 97 active+clean; 216 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 201 op/s 2024-03-24T10:44:37.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 393904 12.02 MB/sec execute 54 sec latency 15.108 ms 2024-03-24T10:44:38.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 395255 11.91 MB/sec execute 55 sec latency 103.291 ms 2024-03-24T10:44:38.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:38 smithi023 ceph-mon[32466]: pgmap v139: 97 pgs: 97 active+clean; 223 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.7 MiB/s wr, 305 op/s 2024-03-24T10:44:39.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:38 smithi117 ceph-mon[36368]: pgmap v139: 97 pgs: 97 active+clean; 223 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.7 MiB/s wr, 305 op/s 2024-03-24T10:44:39.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:38 smithi116 ceph-mon[36248]: pgmap v139: 97 pgs: 97 active+clean; 223 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.7 MiB/s wr, 305 op/s 2024-03-24T10:44:39.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 397758 11.95 MB/sec execute 56 sec latency 14.661 ms 2024-03-24T10:44:40.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 400322 11.96 MB/sec execute 57 sec latency 13.294 ms 2024-03-24T10:44:40.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:40 smithi023 ceph-mon[32466]: pgmap v140: 97 pgs: 97 active+clean; 223 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 203 op/s 2024-03-24T10:44:40.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:40 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:44:40.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:40 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:44:41.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:40 smithi117 ceph-mon[36368]: pgmap v140: 97 pgs: 97 active+clean; 223 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 203 op/s 2024-03-24T10:44:41.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:40 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:44:41.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:40 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:44:41.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:40 smithi116 ceph-mon[36248]: pgmap v140: 97 pgs: 97 active+clean; 223 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 203 op/s 2024-03-24T10:44:41.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:40 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:44:41.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:40 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:44:41.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 402311 11.96 MB/sec execute 58 sec latency 26.021 ms 2024-03-24T10:44:42.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 404872 11.99 MB/sec execute 59 sec latency 7.927 ms 2024-03-24T10:44:42.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:42 smithi023 ceph-mon[32466]: pgmap v141: 97 pgs: 97 active+clean; 229 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 304 op/s 2024-03-24T10:44:43.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:42 smithi117 ceph-mon[36368]: pgmap v141: 97 pgs: 97 active+clean; 229 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 304 op/s 2024-03-24T10:44:43.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:42 smithi116 ceph-mon[36248]: pgmap v141: 97 pgs: 97 active+clean; 229 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 304 op/s 2024-03-24T10:44:43.118 INFO:tasks.workunit.client.0.smithi023.stdout: 1 406836 11.91 MB/sec execute 60 sec latency 73.826 ms 2024-03-24T10:44:44.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 408749 11.92 MB/sec execute 61 sec latency 18.293 ms 2024-03-24T10:44:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:44 smithi023 ceph-mon[32466]: pgmap v142: 97 pgs: 97 active+clean; 229 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-03-24T10:44:45.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:44 smithi117 ceph-mon[36368]: pgmap v142: 97 pgs: 97 active+clean; 229 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-03-24T10:44:45.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:44 smithi116 ceph-mon[36248]: pgmap v142: 97 pgs: 97 active+clean; 229 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-03-24T10:44:45.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 411726 12.02 MB/sec execute 62 sec latency 7.376 ms 2024-03-24T10:44:46.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 413853 11.95 MB/sec execute 63 sec latency 28.632 ms 2024-03-24T10:44:46.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:46 smithi023 ceph-mon[32466]: pgmap v143: 97 pgs: 97 active+clean; 229 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-03-24T10:44:47.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:46 smithi117 ceph-mon[36368]: pgmap v143: 97 pgs: 97 active+clean; 229 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-03-24T10:44:47.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:46 smithi116 ceph-mon[36248]: pgmap v143: 97 pgs: 97 active+clean; 229 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-03-24T10:44:47.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 415763 11.96 MB/sec execute 64 sec latency 13.021 ms 2024-03-24T10:44:48.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 418214 11.97 MB/sec execute 65 sec latency 119.124 ms 2024-03-24T10:44:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:48 smithi023 ceph-mon[32466]: pgmap v144: 97 pgs: 97 active+clean; 235 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 308 op/s 2024-03-24T10:44:49.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:48 smithi117 ceph-mon[36368]: pgmap v144: 97 pgs: 97 active+clean; 235 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 308 op/s 2024-03-24T10:44:49.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:48 smithi116 ceph-mon[36248]: pgmap v144: 97 pgs: 97 active+clean; 235 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 308 op/s 2024-03-24T10:44:49.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 419836 11.91 MB/sec execute 66 sec latency 41.832 ms 2024-03-24T10:44:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.840+0000 7fce0311e640 1 -- 172.21.15.23:0/1082132143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdfc100d10 msgr2=0x7fcdfc10d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.840+0000 7fce0311e640 1 --2- 172.21.15.23:0/1082132143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdfc100d10 0x7fcdfc10d4b0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009900 tx=0x7fcde402bcd0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.840+0000 7fce0311e640 1 -- 172.21.15.23:0/1082132143 shutdown_connections 2024-03-24T10:44:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.840+0000 7fce0311e640 1 --2- 172.21.15.23:0/1082132143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdfc100d10 0x7fcdfc10d4b0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.840+0000 7fce0311e640 1 --2- 172.21.15.23:0/1082132143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdfc106fa0 0x7fcdfc1005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.840+0000 7fce0311e640 1 --2- 172.21.15.23:0/1082132143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc106680 0x7fcdfc106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.840+0000 7fce0311e640 1 -- 172.21.15.23:0/1082132143 >> 172.21.15.23:0/1082132143 conn(0x7fcdfc077740 msgr2=0x7fcdfc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.840+0000 7fce0311e640 1 -- 172.21.15.23:0/1082132143 shutdown_connections 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.840+0000 7fce0311e640 1 -- 172.21.15.23:0/1082132143 wait complete. 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce0311e640 1 Processor -- start 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce0311e640 1 -- start start 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce0311e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc100d10 0x7fcdfc19d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce0311e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdfc106680 0x7fcdfc19de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce0311e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdfc106fa0 0x7fcdfc197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce0311e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc198160 con 0x7fcdfc106fa0 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce0311e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc1982d0 con 0x7fcdfc106680 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce0311e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcdfc198490 con 0x7fcdfc100d10 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce00eba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc100d10 0x7fcdfc19d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce016bb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdfc106fa0 0x7fcdfc197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce00eba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc100d10 0x7fcdfc19d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:42534/0 (socket says 172.21.15.23:42534) 2024-03-24T10:44:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce00eba640 1 -- 172.21.15.23:0/3798187492 learned_addr learned my addr 172.21.15.23:0/3798187492 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fcdfbfff640 1 --2- 172.21.15.23:0/3798187492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdfc106680 0x7fcdfc19de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce016bb640 1 -- 172.21.15.23:0/3798187492 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc100d10 msgr2=0x7fcdfc19d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce016bb640 1 --2- 172.21.15.23:0/3798187492 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc100d10 0x7fcdfc19d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce016bb640 1 -- 172.21.15.23:0/3798187492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdfc106680 msgr2=0x7fcdfc19de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce016bb640 1 --2- 172.21.15.23:0/3798187492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdfc106680 0x7fcdfc19de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce016bb640 1 -- 172.21.15.23:0/3798187492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde4009560 con 0x7fcdfc106fa0 2024-03-24T10:44:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.841+0000 7fce00eba640 1 --2- 172.21.15.23:0/3798187492 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc100d10 0x7fcdfc19d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.842+0000 7fce016bb640 1 --2- 172.21.15.23:0/3798187492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdfc106fa0 0x7fcdfc197b10 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009230 tx=0x7fcde402c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.842+0000 7fcdf9ffb640 1 -- 172.21.15.23:0/3798187492 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde40042d0 con 0x7fcdfc106fa0 2024-03-24T10:44:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.842+0000 7fcdf9ffb640 1 -- 172.21.15.23:0/3798187492 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcde4004430 con 0x7fcdfc106fa0 2024-03-24T10:44:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.842+0000 7fce0311e640 1 -- 172.21.15.23:0/3798187492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdfc198710 con 0x7fcdfc106fa0 2024-03-24T10:44:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.842+0000 7fcdf9ffb640 1 -- 172.21.15.23:0/3798187492 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde4003b20 con 0x7fcdfc106fa0 2024-03-24T10:44:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.842+0000 7fce0311e640 1 -- 172.21.15.23:0/3798187492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdfc1a49b0 con 0x7fcdfc106fa0 2024-03-24T10:44:49.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.844+0000 7fcdf9ffb640 1 -- 172.21.15.23:0/3798187492 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fcde4003c80 con 0x7fcdfc106fa0 2024-03-24T10:44:49.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.844+0000 7fce05ddc740 1 Processor -- start 2024-03-24T10:44:49.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce05ddc740 1 -- start start 2024-03-24T10:44:49.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce05ddc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bf5e0 0x5566168bf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce05ddc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bff00 0x5566168c8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce05ddc740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fcdf9ffb640 1 --2- 172.21.15.23:0/3798187492 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcde80a22d0 0x7fcde80a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce05ddc740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55661687a1b0 con 0x5566168c8cd0 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce05ddc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55661682cf80 con 0x5566168bff00 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce05ddc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55661682a4b0 con 0x5566168bf5e0 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce016bb640 1 --2- 172.21.15.23:0/3798187492 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcde80a22d0 0x7fcde80a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fcdfbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bf5e0 0x5566168bf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce00eba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bff00 0x5566168c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce00eba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bff00 0x5566168c8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:49012/0 (socket says 172.21.15.23:49012) 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce00eba640 1 -- 172.21.15.23:0/810635572 learned_addr learned my addr 172.21.15.23:0/810635572 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fcdf9ffb640 1 -- 172.21.15.23:0/3798187492 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fcde40f0540 con 0x7fcdfc106fa0 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fcdfbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bf5e0 0x5566168bf9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:42538/0 (socket says 172.21.15.23:42538) 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fcdfbfff640 1 --2- 172.21.15.23:0/810635572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce00eba640 1 -- 172.21.15.23:0/810635572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bf5e0 msgr2=0x5566168bf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce00eba640 1 --2- 172.21.15.23:0/810635572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bf5e0 0x5566168bf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce00eba640 1 -- 172.21.15.23:0/810635572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 msgr2=0x5566168cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce00eba640 1 --2- 172.21.15.23:0/810635572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce00eba640 1 -- 172.21.15.23:0/810635572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566167bb810 con 0x5566168bff00 2024-03-24T10:44:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fcdfbfff640 1 --2- 172.21.15.23:0/810635572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:44:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce00eba640 1 --2- 172.21.15.23:0/810635572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bff00 0x5566168c8790 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40115f0 tx=0x7fcdf4011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cce2ef335a03a07 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.845+0000 7fce016bb640 1 --2- 172.21.15.23:0/3798187492 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcde80a22d0 0x7fcde80a4790 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcde4041000 tx=0x7fcde402c7e0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.846+0000 7fcdf2ffd640 1 -- 172.21.15.23:0/810635572 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4005020 con 0x5566168bff00 2024-03-24T10:44:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.846+0000 7fcdf2ffd640 1 -- 172.21.15.23:0/810635572 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdf4005660 con 0x5566168bff00 2024-03-24T10:44:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.846+0000 7fcdf2ffd640 1 -- 172.21.15.23:0/810635572 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf400f960 con 0x5566168bff00 2024-03-24T10:44:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.846+0000 7fce05ddc740 1 -- 172.21.15.23:0/810635572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bff00 msgr2=0x5566168c8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.846+0000 7fce05ddc740 1 --2- 172.21.15.23:0/810635572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bff00 0x5566168c8790 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40115f0 tx=0x7fcdf4011ab0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.846+0000 7fce05ddc740 1 -- 172.21.15.23:0/810635572 shutdown_connections 2024-03-24T10:44:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.846+0000 7fce05ddc740 1 --2- 172.21.15.23:0/810635572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.846+0000 7fce05ddc740 1 --2- 172.21.15.23:0/810635572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bff00 0x5566168c8790 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.846+0000 7fce05ddc740 1 --2- 172.21.15.23:0/810635572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bf5e0 0x5566168bf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.846+0000 7fce05ddc740 1 -- 172.21.15.23:0/810635572 >> 172.21.15.23:0/810635572 conn(0x5566168b9f00 msgr2=0x5566168ba2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fce05ddc740 1 -- 172.21.15.23:0/810635572 shutdown_connections 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fce05ddc740 1 -- 172.21.15.23:0/810635572 wait complete. 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fce05ddc740 1 -- start start 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fce05ddc740 1 Processor -- start 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fce05ddc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 0x5566168c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fce05ddc740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168bff00 0x5566168c6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fce05ddc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168c8cd0 0x5566168c6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fce05ddc740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556616825c90 con 0x5566168bff00 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fce05ddc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5566167cac50 con 0x5566168bf5e0 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fce05ddc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5566164f9250 con 0x5566168c8cd0 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fce05ddc740 10 client.0 _ll_register_callbacks cb 0x556616819290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fce00eba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168bff00 0x5566168c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fce00eba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168bff00 0x5566168c6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39726/0 (socket says 172.21.15.23:39726) 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fce00eba640 1 -- 172.21.15.23:0/198175681 learned_addr learned my addr 172.21.15.23:0/198175681 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.847+0000 7fcdfbfff640 1 --2- 172.21.15.23:0/198175681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 0x5566168c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fcdfbfff640 1 --2- 172.21.15.23:0/198175681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168c8cd0 0x5566168c6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fce00eba640 1 -- 172.21.15.23:0/198175681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168c8cd0 msgr2=0x5566168c6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fce00eba640 1 --2- 172.21.15.23:0/198175681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168c8cd0 0x5566168c6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fce00eba640 1 -- 172.21.15.23:0/198175681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 msgr2=0x5566168c5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fce00eba640 1 --2- 172.21.15.23:0/198175681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 0x5566168c5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fce00eba640 1 -- 172.21.15.23:0/198175681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdf4009d40 con 0x5566168bff00 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fce00eba640 1 --2- 172.21.15.23:0/198175681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168bff00 0x5566168c6240 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4005220 tx=0x7fcdf4004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc16840fe459cb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fcddbfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fcddbfff640 1 -- 172.21.15.23:0/198175681 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4004b60 con 0x5566168bff00 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fcddbfff640 1 -- 172.21.15.23:0/198175681 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fcdf4004030 con 0x5566168bff00 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fce05ddc740 1 -- 172.21.15.23:0/198175681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55661684a790 con 0x5566168bff00 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fcddbfff640 1 -- 172.21.15.23:0/198175681 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcdf40041f0 con 0x5566168bff00 2024-03-24T10:44:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fce05ddc740 10 client.16911 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fce05ddc740 1 -- 172.21.15.23:0/198175681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556616824310 con 0x5566168bff00 2024-03-24T10:44:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.848+0000 7fcddbfff640 1 -- 172.21.15.23:0/198175681 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55661684a790 con 0x5566168bff00 2024-03-24T10:44:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.849+0000 7fcddbfff640 1 -- 172.21.15.23:0/198175681 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fcdf402a020 con 0x5566168bff00 2024-03-24T10:44:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.849+0000 7fce05ddc740 10 client.16911 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.849+0000 7fce05ddc740 10 client.16911 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.849+0000 7fce05ddc740 20 client.16911 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.849+0000 7fce05ddc740 1 --2- 172.21.15.23:0/198175681 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5566168d8b60 0x5566168dafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.849+0000 7fce05ddc740 4 client.16911 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:44:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.849+0000 7fce05ddc740 1 -- 172.21.15.23:0/198175681 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556616525600 con 0x5566168d8b60 2024-03-24T10:44:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.849+0000 7fcdfbfff640 1 --2- 172.21.15.23:0/198175681 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5566168d8b60 0x5566168dafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.849+0000 7fcdfbfff640 1 --2- 172.21.15.23:0/198175681 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5566168d8b60 0x5566168dafc0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d01eb14add9c06a6 server_cookie=d969c0e5764aa257 in_seq=0 out_seq=0 2024-03-24T10:44:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.849+0000 7fcddbfff640 10 client.16911 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.850+0000 7fcddbfff640 1 -- 172.21.15.23:0/198175681 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556616525600 con 0x5566168d8b60 2024-03-24T10:44:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.850+0000 7fcddbfff640 10 client.16911 handle_command_reply: tid=0 2024-03-24T10:44:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.850+0000 7fce05ddc740 1 client.16911 shutdown 2024-03-24T10:44:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.850+0000 7fce05ddc740 1 -- 172.21.15.23:0/198175681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168bff00 msgr2=0x5566168c6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.850+0000 7fce05ddc740 1 --2- 172.21.15.23:0/198175681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168bff00 0x5566168c6240 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4005220 tx=0x7fcdf4004d50 comp rx=0 tx=0).stop 2024-03-24T10:44:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.850+0000 7fce05ddc740 1 -- 172.21.15.23:0/198175681 shutdown_connections 2024-03-24T10:44:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.850+0000 7fce05ddc740 1 --2- 172.21.15.23:0/198175681 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5566168d8b60 0x5566168dafc0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.850+0000 7fce05ddc740 1 --2- 172.21.15.23:0/198175681 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168c8cd0 0x5566168c6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.850+0000 7fce05ddc740 1 --2- 172.21.15.23:0/198175681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168bff00 0x5566168c6240 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.850+0000 7fcddbfff640 0 client.16911 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.850+0000 7fce05ddc740 1 --2- 172.21.15.23:0/198175681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 0x5566168c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.850+0000 7fce05ddc740 1 -- 172.21.15.23:0/198175681 >> 172.21.15.23:0/198175681 conn(0x5566168b9f00 msgr2=0x5566168bbc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.851+0000 7fce05ddc740 1 -- 172.21.15.23:0/198175681 shutdown_connections 2024-03-24T10:44:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.851+0000 7fce05ddc740 1 -- 172.21.15.23:0/198175681 wait complete. 2024-03-24T10:44:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.851+0000 7fce05ddc740 20 client.16911 trim_cache size 0 max 16384 2024-03-24T10:44:49.867 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.866+0000 7fce05ddc740 1 Processor -- start 2024-03-24T10:44:49.867 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.866+0000 7fce05ddc740 1 -- start start 2024-03-24T10:44:49.867 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.866+0000 7fce05ddc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 0x5566168c8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.867 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.866+0000 7fce05ddc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bff00 0x5566168d43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.867 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.866+0000 7fce05ddc740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168e0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.866+0000 7fce05ddc740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5566167ab660 con 0x5566168c8cd0 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.866+0000 7fce05ddc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5566168d5650 con 0x5566168bf5e0 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.866+0000 7fce05ddc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5566168baf00 con 0x5566168bff00 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.866+0000 7fcdfbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168e0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.866+0000 7fce00eba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bff00 0x5566168d43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.866+0000 7fce00eba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bff00 0x5566168d43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:42550/0 (socket says 172.21.15.23:42550) 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.867+0000 7fce00eba640 1 -- 172.21.15.23:0/1583064621 learned_addr learned my addr 172.21.15.23:0/1583064621 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.867+0000 7fcdfbfff640 1 --2- 172.21.15.23:0/1583064621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 0x5566168c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.867+0000 7fce00eba640 1 -- 172.21.15.23:0/1583064621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 msgr2=0x5566168c8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.867+0000 7fce00eba640 1 --2- 172.21.15.23:0/1583064621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 0x5566168c8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.867+0000 7fce00eba640 1 -- 172.21.15.23:0/1583064621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 msgr2=0x5566168e0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.867+0000 7fce00eba640 1 --2- 172.21.15.23:0/1583064621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168e0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.867+0000 7fce00eba640 1 -- 172.21.15.23:0/1583064621 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566167c0810 con 0x5566168bff00 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.867+0000 7fcdfbfff640 1 --2- 172.21.15.23:0/1583064621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168e0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.867+0000 7fce00eba640 1 --2- 172.21.15.23:0/1583064621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bff00 0x5566168d43a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4003870 tx=0x7fcdf4011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd5b16def90c1c98 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.867+0000 7fcdf27fc640 1 -- 172.21.15.23:0/1583064621 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4005250 con 0x5566168bff00 2024-03-24T10:44:49.869 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.867+0000 7fcdf27fc640 1 -- 172.21.15.23:0/1583064621 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdf40038f0 con 0x5566168bff00 2024-03-24T10:44:49.869 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.867+0000 7fcdf27fc640 1 -- 172.21.15.23:0/1583064621 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4004000 con 0x5566168bff00 2024-03-24T10:44:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.868+0000 7fce05ddc740 1 -- 172.21.15.23:0/1583064621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bff00 msgr2=0x5566168d43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.868+0000 7fce05ddc740 1 --2- 172.21.15.23:0/1583064621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bff00 0x5566168d43a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4003870 tx=0x7fcdf4011ab0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.868+0000 7fce05ddc740 1 -- 172.21.15.23:0/1583064621 shutdown_connections 2024-03-24T10:44:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.868+0000 7fce05ddc740 1 --2- 172.21.15.23:0/1583064621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168e0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.868+0000 7fce05ddc740 1 --2- 172.21.15.23:0/1583064621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bff00 0x5566168d43a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.868+0000 7fce05ddc740 1 --2- 172.21.15.23:0/1583064621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 0x5566168c8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.868+0000 7fce05ddc740 1 -- 172.21.15.23:0/1583064621 >> 172.21.15.23:0/1583064621 conn(0x5566168b9f00 msgr2=0x5566168c0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.868+0000 7fce05ddc740 1 -- 172.21.15.23:0/1583064621 shutdown_connections 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.868+0000 7fce05ddc740 1 -- 172.21.15.23:0/1583064621 wait complete. 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.869+0000 7fce05ddc740 1 -- start start 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.869+0000 7fce05ddc740 1 Processor -- start 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.869+0000 7fce05ddc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 0x5566168d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.869+0000 7fce05ddc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bff00 0x5566168cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.869+0000 7fce05ddc740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.869+0000 7fce05ddc740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5566168cca10 con 0x5566168c8cd0 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.869+0000 7fce05ddc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5566168ccb80 con 0x5566168bf5e0 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.869+0000 7fce05ddc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5566168ccd40 con 0x5566168bff00 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.869+0000 7fce05ddc740 10 client.0 _ll_register_callbacks cb 0x7fcdec009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.869+0000 7fcdfbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.869+0000 7fcdfbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168cc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39738/0 (socket says 172.21.15.23:39738) 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.869+0000 7fcdfbfff640 1 -- 172.21.15.23:0/1443502712 learned_addr learned my addr 172.21.15.23:0/1443502712 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.869+0000 7fce00eba640 1 --2- 172.21.15.23:0/1443502712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bff00 0x5566168cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.869+0000 7fcdfbfff640 1 --2- 172.21.15.23:0/1443502712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 0x5566168d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.870+0000 7fcdfbfff640 1 -- 172.21.15.23:0/1443502712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bff00 msgr2=0x5566168cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.870+0000 7fcdfbfff640 1 --2- 172.21.15.23:0/1443502712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bff00 0x5566168cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.870+0000 7fcdfbfff640 1 -- 172.21.15.23:0/1443502712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 msgr2=0x5566168d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.870+0000 7fcdfbfff640 1 --2- 172.21.15.23:0/1443502712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 0x5566168d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.870+0000 7fcdfbfff640 1 -- 172.21.15.23:0/1443502712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdec019040 con 0x5566168c8cd0 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.870+0000 7fcdfbfff640 1 --2- 172.21.15.23:0/1443502712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168cc4d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc1073a0 tx=0x7fcdec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46ab45e0127a78c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.870+0000 7fcddbfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.870+0000 7fcddbfff640 1 -- 172.21.15.23:0/1443502712 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec016040 con 0x5566168c8cd0 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.870+0000 7fcddbfff640 1 -- 172.21.15.23:0/1443502712 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fcdec004b20 con 0x5566168c8cd0 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.870+0000 7fce05ddc740 1 -- 172.21.15.23:0/1443502712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdf4005660 con 0x5566168c8cd0 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.870+0000 7fcddbfff640 1 -- 172.21.15.23:0/1443502712 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcdec004ce0 con 0x5566168c8cd0 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.870+0000 7fce05ddc740 10 client.16917 fetch_fsmap learned FSMap version 27 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.870+0000 7fce05ddc740 1 -- 172.21.15.23:0/1443502712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556616824310 con 0x5566168c8cd0 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.870+0000 7fcddbfff640 1 -- 172.21.15.23:0/1443502712 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdf4005660 con 0x5566168c8cd0 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.871+0000 7fcddbfff640 1 -- 172.21.15.23:0/1443502712 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fcdec022470 con 0x5566168c8cd0 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.871+0000 7fce05ddc740 10 client.16917 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:44:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.871+0000 7fce05ddc740 10 client.16917 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:44:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.871+0000 7fce05ddc740 20 client.16917 populate_metadata read hostname 'smithi023' 2024-03-24T10:44:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.871+0000 7fce05ddc740 1 --2- 172.21.15.23:0/1443502712 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55661693f4e0 0x556616941900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.871+0000 7fce05ddc740 4 client.16917 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:44:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.871+0000 7fce05ddc740 1 -- 172.21.15.23:0/1443502712 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5566165761b0 con 0x55661693f4e0 2024-03-24T10:44:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.871+0000 7fce00eba640 1 --2- 172.21.15.23:0/1443502712 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55661693f4e0 0x556616941900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.871+0000 7fcddbfff640 10 client.16917 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.871+0000 7fce00eba640 1 --2- 172.21.15.23:0/1443502712 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55661693f4e0 0x556616941900 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=918b629465bedd43 server_cookie=d42ae0394525a73e in_seq=0 out_seq=0 2024-03-24T10:44:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.880+0000 7fcddbfff640 1 -- 172.21.15.23:0/1443502712 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5566165761b0 con 0x55661693f4e0 2024-03-24T10:44:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.880+0000 7fcddbfff640 10 client.16917 handle_command_reply: tid=0 2024-03-24T10:44:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.880+0000 7fce05ddc740 1 client.16917 shutdown 2024-03-24T10:44:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce05ddc740 1 -- 172.21.15.23:0/1443502712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 msgr2=0x5566168cc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce05ddc740 1 --2- 172.21.15.23:0/1443502712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168cc4d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc1073a0 tx=0x7fcdec004a40 comp rx=0 tx=0).stop 2024-03-24T10:44:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce05ddc740 1 -- 172.21.15.23:0/1443502712 shutdown_connections 2024-03-24T10:44:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce05ddc740 1 --2- 172.21.15.23:0/1443502712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5566168c8cd0 0x5566168cc4d0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce05ddc740 1 --2- 172.21.15.23:0/1443502712 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55661693f4e0 0x556616941900 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce05ddc740 1 --2- 172.21.15.23:0/1443502712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566168bff00 0x5566168cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce05ddc740 1 --2- 172.21.15.23:0/1443502712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566168bf5e0 0x5566168d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fcddbfff640 0 client.16917 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:44:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce05ddc740 1 -- 172.21.15.23:0/1443502712 >> 172.21.15.23:0/1443502712 conn(0x5566168b9f00 msgr2=0x5566168b8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce05ddc740 1 -- 172.21.15.23:0/1443502712 shutdown_connections 2024-03-24T10:44:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce05ddc740 1 -- 172.21.15.23:0/1443502712 wait complete. 2024-03-24T10:44:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce05ddc740 20 client.16917 trim_cache size 0 max 16384 2024-03-24T10:44:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:44:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:44:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:44:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:44:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce0311e640 1 -- 172.21.15.23:0/3798187492 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcde80a22d0 msgr2=0x7fcde80a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce0311e640 1 --2- 172.21.15.23:0/3798187492 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcde80a22d0 0x7fcde80a4790 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcde4041000 tx=0x7fcde402c7e0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce0311e640 1 -- 172.21.15.23:0/3798187492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdfc106fa0 msgr2=0x7fcdfc197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.881+0000 7fce0311e640 1 --2- 172.21.15.23:0/3798187492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdfc106fa0 0x7fcdfc197b10 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009230 tx=0x7fcde402c670 comp rx=0 tx=0).stop 2024-03-24T10:44:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.882+0000 7fce0311e640 1 -- 172.21.15.23:0/3798187492 shutdown_connections 2024-03-24T10:44:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.882+0000 7fce0311e640 1 --2- 172.21.15.23:0/3798187492 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcde80a22d0 0x7fcde80a4790 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.882+0000 7fce0311e640 1 --2- 172.21.15.23:0/3798187492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdfc106fa0 0x7fcdfc197b10 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.884 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.882+0000 7fce0311e640 1 --2- 172.21.15.23:0/3798187492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdfc106680 0x7fcdfc19de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.884 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.882+0000 7fce0311e640 1 --2- 172.21.15.23:0/3798187492 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdfc100d10 0x7fcdfc19d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.884 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.882+0000 7fce0311e640 1 -- 172.21.15.23:0/3798187492 >> 172.21.15.23:0/3798187492 conn(0x7fcdfc077740 msgr2=0x7fcdfc0ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:49.884 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.882+0000 7fce0311e640 1 -- 172.21.15.23:0/3798187492 shutdown_connections 2024-03-24T10:44:49.884 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:44:49.882+0000 7fce0311e640 1 -- 172.21.15.23:0/3798187492 wait complete. 2024-03-24T10:44:50.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 422501 11.99 MB/sec execute 67 sec latency 13.143 ms 2024-03-24T10:44:50.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:50 smithi023 ceph-mon[32466]: pgmap v145: 97 pgs: 97 active+clean; 235 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-03-24T10:44:51.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:50 smithi117 ceph-mon[36368]: pgmap v145: 97 pgs: 97 active+clean; 235 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-03-24T10:44:51.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:50 smithi116 ceph-mon[36248]: pgmap v145: 97 pgs: 97 active+clean; 235 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-03-24T10:44:51.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 424601 11.92 MB/sec execute 68 sec latency 17.049 ms 2024-03-24T10:44:52.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 426548 11.93 MB/sec execute 69 sec latency 17.305 ms 2024-03-24T10:44:52.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:52 smithi023 ceph-mon[32466]: pgmap v146: 97 pgs: 97 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 307 op/s 2024-03-24T10:44:53.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:52 smithi117 ceph-mon[36368]: pgmap v146: 97 pgs: 97 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 307 op/s 2024-03-24T10:44:53.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:52 smithi116 ceph-mon[36248]: pgmap v146: 97 pgs: 97 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 307 op/s 2024-03-24T10:44:53.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 429036 11.95 MB/sec execute 70 sec latency 66.607 ms 2024-03-24T10:44:54.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 430632 11.92 MB/sec execute 71 sec latency 28.632 ms 2024-03-24T10:44:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:54 smithi023 ceph-mon[32466]: pgmap v147: 97 pgs: 97 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 206 op/s 2024-03-24T10:44:55.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:54 smithi117 ceph-mon[36368]: pgmap v147: 97 pgs: 97 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 206 op/s 2024-03-24T10:44:55.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:54 smithi116 ceph-mon[36248]: pgmap v147: 97 pgs: 97 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 206 op/s 2024-03-24T10:44:55.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 433135 11.95 MB/sec execute 72 sec latency 13.664 ms 2024-03-24T10:44:56.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 435542 11.94 MB/sec execute 73 sec latency 21.064 ms 2024-03-24T10:44:56.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:56 smithi023 ceph-mon[32466]: pgmap v148: 97 pgs: 97 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 206 op/s 2024-03-24T10:44:57.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:56 smithi117 ceph-mon[36368]: pgmap v148: 97 pgs: 97 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 206 op/s 2024-03-24T10:44:57.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:56 smithi116 ceph-mon[36248]: pgmap v148: 97 pgs: 97 active+clean; 241 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 206 op/s 2024-03-24T10:44:57.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 437386 11.91 MB/sec execute 74 sec latency 20.927 ms 2024-03-24T10:44:58.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 439978 11.96 MB/sec execute 75 sec latency 134.274 ms 2024-03-24T10:44:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:44:58 smithi023 ceph-mon[32466]: pgmap v149: 97 pgs: 97 active+clean; 248 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 309 op/s 2024-03-24T10:44:59.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:44:58 smithi117 ceph-mon[36368]: pgmap v149: 97 pgs: 97 active+clean; 248 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 309 op/s 2024-03-24T10:44:59.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:44:58 smithi116 ceph-mon[36248]: pgmap v149: 97 pgs: 97 active+clean; 248 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 309 op/s 2024-03-24T10:44:59.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 441631 11.90 MB/sec execute 76 sec latency 23.403 ms 2024-03-24T10:45:00.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 444006 11.92 MB/sec execute 77 sec latency 7.953 ms 2024-03-24T10:45:00.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:00 smithi023 ceph-mon[32466]: pgmap v150: 97 pgs: 97 active+clean; 248 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 205 op/s 2024-03-24T10:45:01.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:00 smithi117 ceph-mon[36368]: pgmap v150: 97 pgs: 97 active+clean; 248 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 205 op/s 2024-03-24T10:45:01.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:00 smithi116 ceph-mon[36248]: pgmap v150: 97 pgs: 97 active+clean; 248 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 205 op/s 2024-03-24T10:45:01.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 445751 11.86 MB/sec execute 78 sec latency 25.077 ms 2024-03-24T10:45:02.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 447736 11.87 MB/sec execute 79 sec latency 14.259 ms 2024-03-24T10:45:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:02 smithi023 ceph-mon[32466]: pgmap v151: 97 pgs: 97 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.7 MiB/s wr, 332 op/s 2024-03-24T10:45:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.971+0000 7fd88d2d0640 1 -- 172.21.15.23:0/2665413801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8881055b0 msgr2=0x7fd888105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.971+0000 7fd88d2d0640 1 --2- 172.21.15.23:0/2665413801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8881055b0 0x7fd888105990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fd880009900 tx=0x7fd88002bc30 comp rx=0 tx=0).stop 2024-03-24T10:45:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.971+0000 7fd88d2d0640 1 -- 172.21.15.23:0/2665413801 shutdown_connections 2024-03-24T10:45:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.971+0000 7fd88d2d0640 1 --2- 172.21.15.23:0/2665413801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd888101220 0x7fd88810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.971+0000 7fd88d2d0640 1 --2- 172.21.15.23:0/2665413801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd888100860 0x7fd888100ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.971+0000 7fd88d2d0640 1 --2- 172.21.15.23:0/2665413801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8881055b0 0x7fd888105990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.971+0000 7fd88d2d0640 1 -- 172.21.15.23:0/2665413801 >> 172.21.15.23:0/2665413801 conn(0x7fd888077460 msgr2=0x7fd888077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.971+0000 7fd88d2d0640 1 -- 172.21.15.23:0/2665413801 shutdown_connections 2024-03-24T10:45:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.971+0000 7fd88d2d0640 1 -- 172.21.15.23:0/2665413801 wait complete. 2024-03-24T10:45:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.971+0000 7fd88d2d0640 1 Processor -- start 2024-03-24T10:45:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd88d2d0640 1 -- start start 2024-03-24T10:45:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd88d2d0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd888100860 0x7fd888197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd88d2d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd888101220 0x7fd888198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd88d2d0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8881055b0 0x7fd88819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd88d2d0640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd88819cc50 con 0x7fd8881055b0 2024-03-24T10:45:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd88d2d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd88819cdc0 con 0x7fd888101220 2024-03-24T10:45:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd88d2d0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd88819cf80 con 0x7fd888100860 2024-03-24T10:45:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd8877fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8881055b0 0x7fd88819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd888100860 0x7fd888197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd8877fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8881055b0 0x7fd88819c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52570/0 (socket says 172.21.15.23:52570) 2024-03-24T10:45:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd888100860 0x7fd888197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:43178/0 (socket says 172.21.15.23:43178) 2024-03-24T10:45:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd8877fe640 1 -- 172.21.15.23:0/3336359263 learned_addr learned my addr 172.21.15.23:0/3336359263 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd8867fc640 1 --2- 172.21.15.23:0/3336359263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd888101220 0x7fd888198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd886ffd640 1 -- 172.21.15.23:0/3336359263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd888101220 msgr2=0x7fd888198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd886ffd640 1 --2- 172.21.15.23:0/3336359263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd888101220 0x7fd888198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd886ffd640 1 -- 172.21.15.23:0/3336359263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8881055b0 msgr2=0x7fd88819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd886ffd640 1 --2- 172.21.15.23:0/3336359263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8881055b0 0x7fd88819c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd886ffd640 1 -- 172.21.15.23:0/3336359263 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd880009560 con 0x7fd888100860 2024-03-24T10:45:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.972+0000 7fd8877fe640 1 --2- 172.21.15.23:0/3336359263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8881055b0 0x7fd88819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.973+0000 7fd886ffd640 1 --2- 172.21.15.23:0/3336359263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd888100860 0x7fd888197de0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd880036040 tx=0x7fd88002c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.973+0000 7fd87ffff640 1 -- 172.21.15.23:0/3336359263 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880004350 con 0x7fd888100860 2024-03-24T10:45:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.973+0000 7fd87ffff640 1 -- 172.21.15.23:0/3336359263 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8800044b0 con 0x7fd888100860 2024-03-24T10:45:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.973+0000 7fd88d2d0640 1 -- 172.21.15.23:0/3336359263 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd88819d200 con 0x7fd888100860 2024-03-24T10:45:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.973+0000 7fd87ffff640 1 -- 172.21.15.23:0/3336359263 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880003cc0 con 0x7fd888100860 2024-03-24T10:45:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.973+0000 7fd88d2d0640 1 -- 172.21.15.23:0/3336359263 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8881a4ae0 con 0x7fd888100860 2024-03-24T10:45:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.975+0000 7fd87ffff640 1 -- 172.21.15.23:0/3336359263 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fd880005560 con 0x7fd888100860 2024-03-24T10:45:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd88ff82740 1 Processor -- start 2024-03-24T10:45:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd87ffff640 1 --2- 172.21.15.23:0/3336359263 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd8740a22d0 0x7fd8740a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd88ff82740 1 -- start start 2024-03-24T10:45:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd8867fc640 1 --2- 172.21.15.23:0/3336359263 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd8740a22d0 0x7fd8740a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd87ffff640 1 -- 172.21.15.23:0/3336359263 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fd8800eff80 con 0x7fd888100860 2024-03-24T10:45:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd88ff82740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc875e0 0x56210fc879c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd88ff82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc87f00 0x56210fc90790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd88ff82740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc90cd0 0x56210fc930c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd88ff82740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56210fc421b0 con 0x56210fc875e0 2024-03-24T10:45:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd88ff82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56210fbf4f80 con 0x56210fc87f00 2024-03-24T10:45:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd88ff82740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56210fbf24b0 con 0x56210fc90cd0 2024-03-24T10:45:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc875e0 0x56210fc879c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc875e0 0x56210fc879c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52582/0 (socket says 172.21.15.23:52582) 2024-03-24T10:45:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd886ffd640 1 -- 172.21.15.23:0/3103160309 learned_addr learned my addr 172.21.15.23:0/3103160309 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd8877fe640 1 --2- 172.21.15.23:0/3103160309 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc90cd0 0x56210fc930c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd8867fc640 1 --2- 172.21.15.23:0/3103160309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc87f00 0x56210fc90790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd8867fc640 1 --2- 172.21.15.23:0/3336359263 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd8740a22d0 0x7fd8740a4790 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd888077b30 tx=0x7fd870005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd8877fe640 1 -- 172.21.15.23:0/3103160309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc87f00 msgr2=0x56210fc90790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd8877fe640 1 --2- 172.21.15.23:0/3103160309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc87f00 0x56210fc90790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd8877fe640 1 -- 172.21.15.23:0/3103160309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc875e0 msgr2=0x56210fc879c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd8877fe640 1 --2- 172.21.15.23:0/3103160309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc875e0 0x56210fc879c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.976+0000 7fd8877fe640 1 -- 172.21.15.23:0/3103160309 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56210fb83810 con 0x56210fc90cd0 2024-03-24T10:45:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.977+0000 7fd886ffd640 1 --2- 172.21.15.23:0/3103160309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc875e0 0x56210fc879c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.977+0000 7fd8867fc640 1 --2- 172.21.15.23:0/3103160309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc87f00 0x56210fc90790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.977+0000 7fd8877fe640 1 --2- 172.21.15.23:0/3103160309 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc90cd0 0x56210fc930c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd8780049d0 tx=0x7fd878011c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1271718b6220014 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.977+0000 7fd87d7fa640 1 -- 172.21.15.23:0/3103160309 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd878011310 con 0x56210fc90cd0 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.977+0000 7fd87d7fa640 1 -- 172.21.15.23:0/3103160309 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd87800f9a0 con 0x56210fc90cd0 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.977+0000 7fd87d7fa640 1 -- 172.21.15.23:0/3103160309 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd878005660 con 0x56210fc90cd0 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 -- 172.21.15.23:0/3103160309 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc90cd0 msgr2=0x56210fc930c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 --2- 172.21.15.23:0/3103160309 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc90cd0 0x56210fc930c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd8780049d0 tx=0x7fd878011c90 comp rx=0 tx=0).stop 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 -- 172.21.15.23:0/3103160309 shutdown_connections 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 --2- 172.21.15.23:0/3103160309 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc90cd0 0x56210fc930c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 --2- 172.21.15.23:0/3103160309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc87f00 0x56210fc90790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 --2- 172.21.15.23:0/3103160309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc875e0 0x56210fc879c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 -- 172.21.15.23:0/3103160309 >> 172.21.15.23:0/3103160309 conn(0x56210fc81f00 msgr2=0x56210fc822e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 -- 172.21.15.23:0/3103160309 shutdown_connections 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 -- 172.21.15.23:0/3103160309 wait complete. 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 -- start start 2024-03-24T10:45:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 Processor -- start 2024-03-24T10:45:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc875e0 0x56210fc8dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc87f00 0x56210fc8e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc90cd0 0x56210fc8e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56210fbedc90 con 0x56210fc87f00 2024-03-24T10:45:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56210fb92c50 con 0x56210fc90cd0 2024-03-24T10:45:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56210f8c1250 con 0x56210fc875e0 2024-03-24T10:45:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd88ff82740 10 client.0 _ll_register_callbacks cb 0x56210fbe1290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd8867fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc87f00 0x56210fc8e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc875e0 0x56210fc8dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd8877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc90cd0 0x56210fc8e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd8867fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc87f00 0x56210fc8e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52584/0 (socket says 172.21.15.23:52584) 2024-03-24T10:45:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.978+0000 7fd8867fc640 1 -- 172.21.15.23:0/3277724395 learned_addr learned my addr 172.21.15.23:0/3277724395 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd8867fc640 1 -- 172.21.15.23:0/3277724395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc875e0 msgr2=0x56210fc8dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd8867fc640 1 --2- 172.21.15.23:0/3277724395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc875e0 0x56210fc8dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd8867fc640 1 -- 172.21.15.23:0/3277724395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc90cd0 msgr2=0x56210fc8e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd8867fc640 1 --2- 172.21.15.23:0/3277724395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc90cd0 0x56210fc8e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd8867fc640 1 -- 172.21.15.23:0/3277724395 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8780114e0 con 0x56210fc87f00 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd886ffd640 1 --2- 172.21.15.23:0/3277724395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc875e0 0x56210fc8dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd8867fc640 1 --2- 172.21.15.23:0/3277724395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc87f00 0x56210fc8e150 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd8700097b0 tx=0x7fd870004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9717fbccf2cad53 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd86e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd86e7fc640 1 -- 172.21.15.23:0/3277724395 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd870021070 con 0x56210fc87f00 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd86e7fc640 1 -- 172.21.15.23:0/3277724395 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fd87001bb60 con 0x56210fc87f00 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd88ff82740 1 -- 172.21.15.23:0/3277724395 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56210fc12790 con 0x56210fc87f00 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd86e7fc640 1 -- 172.21.15.23:0/3277724395 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd87001bd60 con 0x56210fc87f00 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd88ff82740 10 client.16929 fetch_fsmap learned FSMap version 27 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd88ff82740 1 -- 172.21.15.23:0/3277724395 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56210fbec310 con 0x56210fc87f00 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.979+0000 7fd86e7fc640 1 -- 172.21.15.23:0/3277724395 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd870032020 con 0x56210fc87f00 2024-03-24T10:45:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.980+0000 7fd86e7fc640 1 -- 172.21.15.23:0/3277724395 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fd870025430 con 0x56210fc87f00 2024-03-24T10:45:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.980+0000 7fd88ff82740 10 client.16929 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:45:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.980+0000 7fd88ff82740 10 client.16929 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.980+0000 7fd88ff82740 20 client.16929 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.980+0000 7fd88ff82740 1 --2- 172.21.15.23:0/3277724395 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56210fca0b20 0x56210fca2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.980+0000 7fd88ff82740 4 client.16929 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.980+0000 7fd88ff82740 1 -- 172.21.15.23:0/3277724395 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56210f8ed600 con 0x56210fca0b20 2024-03-24T10:45:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.980+0000 7fd886ffd640 1 --2- 172.21.15.23:0/3277724395 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56210fca0b20 0x56210fca2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.980+0000 7fd86e7fc640 10 client.16929 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.980+0000 7fd886ffd640 1 --2- 172.21.15.23:0/3277724395 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56210fca0b20 0x56210fca2f60 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8101826f1030d0f5 server_cookie=c4002ef24016c17d in_seq=0 out_seq=0 2024-03-24T10:45:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd86e7fc640 1 -- 172.21.15.23:0/3277724395 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56210f8ed600 con 0x56210fca0b20 2024-03-24T10:45:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd86e7fc640 10 client.16929 handle_command_reply: tid=0 2024-03-24T10:45:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd88ff82740 1 client.16929 shutdown 2024-03-24T10:45:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd88ff82740 1 -- 172.21.15.23:0/3277724395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc87f00 msgr2=0x56210fc8e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd88ff82740 1 --2- 172.21.15.23:0/3277724395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc87f00 0x56210fc8e150 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd8700097b0 tx=0x7fd870004730 comp rx=0 tx=0).stop 2024-03-24T10:45:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd88ff82740 1 -- 172.21.15.23:0/3277724395 shutdown_connections 2024-03-24T10:45:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd88ff82740 1 --2- 172.21.15.23:0/3277724395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc90cd0 0x56210fc8e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd88ff82740 1 --2- 172.21.15.23:0/3277724395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc87f00 0x56210fc8e150 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd88ff82740 1 --2- 172.21.15.23:0/3277724395 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56210fca0b20 0x56210fca2f60 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd88ff82740 1 --2- 172.21.15.23:0/3277724395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc875e0 0x56210fc8dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd86e7fc640 0 client.16929 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd88ff82740 1 -- 172.21.15.23:0/3277724395 >> 172.21.15.23:0/3277724395 conn(0x56210fc81f00 msgr2=0x56210fc83c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd88ff82740 1 -- 172.21.15.23:0/3277724395 shutdown_connections 2024-03-24T10:45:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd88ff82740 1 -- 172.21.15.23:0/3277724395 wait complete. 2024-03-24T10:45:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.981+0000 7fd88ff82740 20 client.16929 trim_cache size 0 max 16384 2024-03-24T10:45:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.997+0000 7fd88ff82740 1 Processor -- start 2024-03-24T10:45:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.997+0000 7fd88ff82740 1 -- start start 2024-03-24T10:45:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd88ff82740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc875e0 0x56210fc90340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd88ff82740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc87f00 0x56210fc9c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd88ff82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc90cd0 0x56210fc8cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd88ff82740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56210fb73660 con 0x56210fc87f00 2024-03-24T10:45:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd88ff82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56210fc82f00 con 0x56210fc90cd0 2024-03-24T10:45:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd88ff82740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56210fc7d6c0 con 0x56210fc875e0 2024-03-24T10:45:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd8867fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc87f00 0x56210fc9c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc875e0 0x56210fc90340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd8867fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc87f00 0x56210fc9c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52596/0 (socket says 172.21.15.23:52596) 2024-03-24T10:45:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd8867fc640 1 -- 172.21.15.23:0/512898159 learned_addr learned my addr 172.21.15.23:0/512898159 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd8877fe640 1 --2- 172.21.15.23:0/512898159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc90cd0 0x56210fc8cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd886ffd640 1 -- 172.21.15.23:0/512898159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc90cd0 msgr2=0x56210fc8cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd886ffd640 1 --2- 172.21.15.23:0/512898159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc90cd0 0x56210fc8cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd886ffd640 1 -- 172.21.15.23:0/512898159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc87f00 msgr2=0x56210fc9c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd886ffd640 1 --2- 172.21.15.23:0/512898159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc87f00 0x56210fc9c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd886ffd640 1 -- 172.21.15.23:0/512898159 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56210fb88810 con 0x56210fc875e0 2024-03-24T10:45:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd8877fe640 1 --2- 172.21.15.23:0/512898159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc90cd0 0x56210fc8cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:45:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd8867fc640 1 --2- 172.21.15.23:0/512898159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc87f00 0x56210fc9c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd886ffd640 1 --2- 172.21.15.23:0/512898159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc875e0 0x56210fc90340 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd8800b9450 tx=0x7fd8800a7040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc15ce7fa6fc512f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.998+0000 7fd87cff9640 1 -- 172.21.15.23:0/512898159 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8800bca80 con 0x56210fc875e0 2024-03-24T10:45:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.999+0000 7fd87cff9640 1 -- 172.21.15.23:0/512898159 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8800c0bd0 con 0x56210fc875e0 2024-03-24T10:45:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.999+0000 7fd87cff9640 1 -- 172.21.15.23:0/512898159 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8800c4ff0 con 0x56210fc875e0 2024-03-24T10:45:03.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.999+0000 7fd88ff82740 1 -- 172.21.15.23:0/512898159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc875e0 msgr2=0x56210fc90340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.999+0000 7fd88ff82740 1 --2- 172.21.15.23:0/512898159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc875e0 0x56210fc90340 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd8800b9450 tx=0x7fd8800a7040 comp rx=0 tx=0).stop 2024-03-24T10:45:03.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.999+0000 7fd88ff82740 1 -- 172.21.15.23:0/512898159 shutdown_connections 2024-03-24T10:45:03.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.999+0000 7fd88ff82740 1 --2- 172.21.15.23:0/512898159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc90cd0 0x56210fc8cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.999+0000 7fd88ff82740 1 --2- 172.21.15.23:0/512898159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc87f00 0x56210fc9c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.999+0000 7fd88ff82740 1 --2- 172.21.15.23:0/512898159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc875e0 0x56210fc90340 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.999+0000 7fd88ff82740 1 -- 172.21.15.23:0/512898159 >> 172.21.15.23:0/512898159 conn(0x56210fc81f00 msgr2=0x56210fc822e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:03.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.999+0000 7fd88ff82740 1 -- 172.21.15.23:0/512898159 shutdown_connections 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:02.999+0000 7fd88ff82740 1 -- 172.21.15.23:0/512898159 wait complete. 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd88ff82740 1 -- start start 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd88ff82740 1 Processor -- start 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd88ff82740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc875e0 0x56210fc98860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd88ff82740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc87f00 0x56210fc98da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd88ff82740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc90cd0 0x56210fc93eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd88ff82740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56210fc945b0 con 0x56210fc875e0 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd88ff82740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56210fc94720 con 0x56210fc87f00 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd88ff82740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56210fc948e0 con 0x56210fc90cd0 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd88ff82740 10 client.0 _ll_register_callbacks cb 0x7fd87001c9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc875e0 0x56210fc98860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc875e0 0x56210fc98860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52608/0 (socket says 172.21.15.23:52608) 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd886ffd640 1 -- 172.21.15.23:0/2423180090 learned_addr learned my addr 172.21.15.23:0/2423180090 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd8877fe640 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc90cd0 0x56210fc93eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd8867fc640 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc87f00 0x56210fc98da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd886ffd640 1 -- 172.21.15.23:0/2423180090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc90cd0 msgr2=0x56210fc93eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd886ffd640 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc90cd0 0x56210fc93eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd886ffd640 1 -- 172.21.15.23:0/2423180090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc87f00 msgr2=0x56210fc98da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd886ffd640 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc87f00 0x56210fc98da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd886ffd640 1 -- 172.21.15.23:0/2423180090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56210fbec310 con 0x56210fc875e0 2024-03-24T10:45:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd8867fc640 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc87f00 0x56210fc98da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:45:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.000+0000 7fd8877fe640 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc90cd0 0x56210fc93eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd886ffd640 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc875e0 0x56210fc98860 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd880005ec0 tx=0x7fd880003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b0df86dd84c407 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd86e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd86e7fc640 1 -- 172.21.15.23:0/2423180090 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8800a0070 con 0x56210fc875e0 2024-03-24T10:45:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd86e7fc640 1 -- 172.21.15.23:0/2423180090 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fd8800bad80 con 0x56210fc875e0 2024-03-24T10:45:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd88ff82740 1 -- 172.21.15.23:0/2423180090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd87800f9a0 con 0x56210fc875e0 2024-03-24T10:45:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd86e7fc640 1 -- 172.21.15.23:0/2423180090 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd8800c4d90 con 0x56210fc875e0 2024-03-24T10:45:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd88ff82740 10 client.16935 fetch_fsmap learned FSMap version 27 2024-03-24T10:45:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd88ff82740 1 -- 172.21.15.23:0/2423180090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x7fd88819d200 con 0x56210fc875e0 2024-03-24T10:45:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd86e7fc640 1 -- 172.21.15.23:0/2423180090 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd87800f9a0 con 0x56210fc875e0 2024-03-24T10:45:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd86e7fc640 1 -- 172.21.15.23:0/2423180090 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fd8800a3070 con 0x56210fc875e0 2024-03-24T10:45:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd88ff82740 10 client.16935 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:45:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd88ff82740 10 client.16935 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd88ff82740 20 client.16935 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd88ff82740 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56210fd0b280 0x56210fd0d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd88ff82740 4 client.16935 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:45:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd88ff82740 1 -- 172.21.15.23:0/2423180090 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56210f93e1b0 con 0x56210fd0b280 2024-03-24T10:45:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.001+0000 7fd8867fc640 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56210fd0b280 0x56210fd0d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.002+0000 7fd86e7fc640 10 client.16935 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:03.002+0000 7fd8867fc640 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56210fd0b280 0x56210fd0d6a0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4948a1f15a07ff8 server_cookie=8a602784b4069348 in_seq=0 out_seq=0 2024-03-24T10:45:03.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:02 smithi117 ceph-mon[36368]: pgmap v151: 97 pgs: 97 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.7 MiB/s wr, 332 op/s 2024-03-24T10:45:03.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:02 smithi116 ceph-mon[36248]: pgmap v151: 97 pgs: 97 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.7 MiB/s wr, 332 op/s 2024-03-24T10:45:03.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 450310 11.89 MB/sec execute 80 sec latency 55.879 ms 2024-03-24T10:45:03.550 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:45:03.551 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:45:03.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.659+0000 7f90d238a640 1 -- 172.21.15.23:0/2823153858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90cc106660 msgr2=0x7f90cc106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.659+0000 7f90d238a640 1 --2- 172.21.15.23:0/2823153858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90cc106660 0x7f90cc106a40 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f90bc009900 tx=0x7f90bc02bc30 comp rx=0 tx=0).stop 2024-03-24T10:45:03.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.659+0000 7f90d238a640 1 -- 172.21.15.23:0/2823153858 shutdown_connections 2024-03-24T10:45:03.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.659+0000 7f90d238a640 1 --2- 172.21.15.23:0/2823153858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90cc100f60 0x7f90cc10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.659+0000 7f90d238a640 1 --2- 172.21.15.23:0/2823153858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90cc1005a0 0x7f90cc100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.659+0000 7f90d238a640 1 --2- 172.21.15.23:0/2823153858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90cc106660 0x7f90cc106a40 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.659+0000 7f90d238a640 1 -- 172.21.15.23:0/2823153858 >> 172.21.15.23:0/2823153858 conn(0x7f90cc077710 msgr2=0x7f90cc077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:03.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.660+0000 7f90d238a640 1 -- 172.21.15.23:0/2823153858 shutdown_connections 2024-03-24T10:45:03.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.660+0000 7f90d238a640 1 -- 172.21.15.23:0/2823153858 wait complete. 2024-03-24T10:45:03.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.660+0000 7f90d238a640 1 Processor -- start 2024-03-24T10:45:03.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.660+0000 7f90d238a640 1 -- start start 2024-03-24T10:45:03.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.660+0000 7f90d238a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90cc1005a0 0x7f90cc198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.660+0000 7f90d238a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90cc100f60 0x7f90cc1990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.660+0000 7f90d238a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90cc106660 0x7f90cc1a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.660+0000 7f90d238a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f90cc199940 con 0x7f90cc1005a0 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.660+0000 7f90d238a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f90cc1a4140 con 0x7f90cc100f60 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.660+0000 7f90d238a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f90cc1a4280 con 0x7f90cc106660 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90cbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90cc1005a0 0x7f90cc198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90cbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90cc1005a0 0x7f90cc198b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52624/0 (socket says 172.21.15.23:52624) 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90cbfff640 1 -- 172.21.15.23:0/2557661260 learned_addr learned my addr 172.21.15.23:0/2557661260 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90cb7fe640 1 --2- 172.21.15.23:0/2557661260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90cc100f60 0x7f90cc1990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90d0927640 1 --2- 172.21.15.23:0/2557661260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90cc106660 0x7f90cc1a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90cbfff640 1 -- 172.21.15.23:0/2557661260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90cc106660 msgr2=0x7f90cc1a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90cbfff640 1 --2- 172.21.15.23:0/2557661260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90cc106660 0x7f90cc1a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90cbfff640 1 -- 172.21.15.23:0/2557661260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90cc100f60 msgr2=0x7f90cc1990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90cbfff640 1 --2- 172.21.15.23:0/2557661260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90cc100f60 0x7f90cc1990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90cbfff640 1 -- 172.21.15.23:0/2557661260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90bc009560 con 0x7f90cc1005a0 2024-03-24T10:45:03.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90cbfff640 1 --2- 172.21.15.23:0/2557661260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90cc1005a0 0x7f90cc198b80 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f90bc006010 tx=0x7f90bc02c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:03.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90c97fa640 1 -- 172.21.15.23:0/2557661260 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc0043d0 con 0x7f90cc1005a0 2024-03-24T10:45:03.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90c97fa640 1 -- 172.21.15.23:0/2557661260 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90bc0052d0 con 0x7f90cc1005a0 2024-03-24T10:45:03.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90d238a640 1 -- 172.21.15.23:0/2557661260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90cc1a4480 con 0x7f90cc1005a0 2024-03-24T10:45:03.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90c97fa640 1 -- 172.21.15.23:0/2557661260 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc003c50 con 0x7f90cc1005a0 2024-03-24T10:45:03.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.661+0000 7f90d238a640 1 -- 172.21.15.23:0/2557661260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90cc1a4930 con 0x7f90cc1005a0 2024-03-24T10:45:03.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.663+0000 7f90c97fa640 1 -- 172.21.15.23:0/2557661260 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f90bc004df0 con 0x7f90cc1005a0 2024-03-24T10:45:03.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90d4f6c740 1 Processor -- start 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90d4f6c740 1 -- start start 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90d4f6c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8fec1d0 0x55ebc8fec5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90d4f6c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fecaf0 0x55ebc8ff5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90d4f6c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8ff56d0 0x55ebc8ff7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90d4f6c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ebc8f10f30 con 0x55ebc8ff56d0 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90d4f6c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ebc8f5a810 con 0x55ebc8fec1d0 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90d4f6c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ebc8f57d40 con 0x55ebc8fecaf0 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90c97fa640 1 --2- 172.21.15.23:0/2557661260 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f90b40a2230 0x7f90b40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90d0927640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8fec1d0 0x55ebc8fec5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90d0927640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8fec1d0 0x55ebc8fec5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:50884/0 (socket says 172.21.15.23:50884) 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90d0927640 1 -- 172.21.15.23:0/3520502570 learned_addr learned my addr 172.21.15.23:0/3520502570 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90cbfff640 1 --2- 172.21.15.23:0/3520502570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fecaf0 0x55ebc8ff5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90cb7fe640 1 --2- 172.21.15.23:0/2557661260 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f90b40a2230 0x7f90b40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90d0927640 1 --2- 172.21.15.23:0/3520502570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8ff56d0 0x55ebc8ff7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.664+0000 7f90c97fa640 1 -- 172.21.15.23:0/2557661260 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f90bc0f0240 con 0x7f90cc1005a0 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.665+0000 7f90d0927640 1 -- 172.21.15.23:0/3520502570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fecaf0 msgr2=0x55ebc8ff5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.665+0000 7f90d0927640 1 --2- 172.21.15.23:0/3520502570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fecaf0 0x55ebc8ff5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.665+0000 7f90d0927640 1 -- 172.21.15.23:0/3520502570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8ff56d0 msgr2=0x55ebc8ff7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.665+0000 7f90d0927640 1 --2- 172.21.15.23:0/3520502570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8ff56d0 0x55ebc8ff7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.665+0000 7f90d0927640 1 -- 172.21.15.23:0/3520502570 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc8f14ea0 con 0x55ebc8fec1d0 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.665+0000 7f90cb7fe640 1 --2- 172.21.15.23:0/2557661260 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f90b40a2230 0x7f90b40a46f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f90cc0693c0 tx=0x7f90c0005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.665+0000 7f90d0927640 1 --2- 172.21.15.23:0/3520502570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8fec1d0 0x55ebc8fec5b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f90c4012a20 tx=0x7f90c4009a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc19a03f9c6c32b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.665+0000 7f90b27fc640 1 -- 172.21.15.23:0/3520502570 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c4004b20 con 0x55ebc8fec1d0 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.665+0000 7f90b27fc640 1 -- 172.21.15.23:0/3520502570 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90c4003e50 con 0x55ebc8fec1d0 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.665+0000 7f90b27fc640 1 -- 172.21.15.23:0/3520502570 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c4004030 con 0x55ebc8fec1d0 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.666+0000 7f90d4f6c740 1 -- 172.21.15.23:0/3520502570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8fec1d0 msgr2=0x55ebc8fec5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.666+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/3520502570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8fec1d0 0x55ebc8fec5b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f90c4012a20 tx=0x7f90c4009a50 comp rx=0 tx=0).stop 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.666+0000 7f90d4f6c740 1 -- 172.21.15.23:0/3520502570 shutdown_connections 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.666+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/3520502570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8ff56d0 0x55ebc8ff7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.666+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/3520502570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fecaf0 0x55ebc8ff5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.666+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/3520502570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8fec1d0 0x55ebc8fec5b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.666+0000 7f90d4f6c740 1 -- 172.21.15.23:0/3520502570 >> 172.21.15.23:0/3520502570 conn(0x55ebc8fe6900 msgr2=0x55ebc8fe6ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:03.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.666+0000 7f90d4f6c740 1 -- 172.21.15.23:0/3520502570 shutdown_connections 2024-03-24T10:45:03.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.666+0000 7f90d4f6c740 1 -- 172.21.15.23:0/3520502570 wait complete. 2024-03-24T10:45:03.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.666+0000 7f90d4f6c740 1 -- start start 2024-03-24T10:45:03.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.666+0000 7f90d4f6c740 1 Processor -- start 2024-03-24T10:45:03.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90d4f6c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fec1d0 0x55ebc8ff2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90d4f6c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8fecaf0 0x55ebc8ff2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90d4f6c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8ff56d0 0x55ebc8ff3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90d4f6c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ebc8f53520 con 0x55ebc8fec1d0 2024-03-24T10:45:03.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90d4f6c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ebc8c26430 con 0x55ebc8fecaf0 2024-03-24T10:45:03.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90d4f6c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ebc8be2e00 con 0x55ebc8ff56d0 2024-03-24T10:45:03.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90cbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fec1d0 0x55ebc8ff2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90d4f6c740 10 client.0 _ll_register_callbacks cb 0x55ebc8ef1d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90cbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fec1d0 0x55ebc8ff2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52638/0 (socket says 172.21.15.23:52638) 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90cbfff640 1 -- 172.21.15.23:0/2490352152 learned_addr learned my addr 172.21.15.23:0/2490352152 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90d0927640 1 --2- 172.21.15.23:0/2490352152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8ff56d0 0x55ebc8ff3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90cb7fe640 1 --2- 172.21.15.23:0/2490352152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8fecaf0 0x55ebc8ff2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90cbfff640 1 -- 172.21.15.23:0/2490352152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8ff56d0 msgr2=0x55ebc8ff3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90cbfff640 1 --2- 172.21.15.23:0/2490352152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8ff56d0 0x55ebc8ff3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90cbfff640 1 -- 172.21.15.23:0/2490352152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8fecaf0 msgr2=0x55ebc8ff2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90cbfff640 1 --2- 172.21.15.23:0/2490352152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8fecaf0 0x55ebc8ff2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90cbfff640 1 -- 172.21.15.23:0/2490352152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90c401a040 con 0x55ebc8fec1d0 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90cbfff640 1 --2- 172.21.15.23:0/2490352152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fec1d0 0x55ebc8ff2790 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f90bc03afd0 tx=0x7f90bc0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c265a00232558b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.667+0000 7f90ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.669+0000 7f90ab7fe640 1 -- 172.21.15.23:0/2490352152 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc0bcce0 con 0x55ebc8fec1d0 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.669+0000 7f90ab7fe640 1 -- 172.21.15.23:0/2490352152 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f90bc0c0e30 con 0x55ebc8fec1d0 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.669+0000 7f90d4f6c740 1 -- 172.21.15.23:0/2490352152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ebc8c527e0 con 0x55ebc8fec1d0 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.670+0000 7f90ab7fe640 1 -- 172.21.15.23:0/2490352152 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f90bc0c5480 con 0x55ebc8fec1d0 2024-03-24T10:45:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.670+0000 7f90d4f6c740 10 client.16947 fetch_fsmap learned FSMap version 27 2024-03-24T10:45:03.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.670+0000 7f90d4f6c740 1 -- 172.21.15.23:0/2490352152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ebc8f51ba0 con 0x55ebc8fec1d0 2024-03-24T10:45:03.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.670+0000 7f90ab7fe640 1 -- 172.21.15.23:0/2490352152 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ebc8c527e0 con 0x55ebc8fec1d0 2024-03-24T10:45:03.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.670+0000 7f90ab7fe640 1 -- 172.21.15.23:0/2490352152 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f90bc03c070 con 0x55ebc8fec1d0 2024-03-24T10:45:03.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.670+0000 7f90d4f6c740 10 client.16947 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:45:03.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.670+0000 7f90d4f6c740 10 client.16947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:03.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.670+0000 7f90d4f6c740 20 client.16947 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:03.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.670+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/2490352152 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ebc90059a0 0x55ebc9007db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.670+0000 7f90d4f6c740 4 client.16947 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:03.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.670+0000 7f90d4f6c740 1 -- 172.21.15.23:0/2490352152 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebc8c8f720 con 0x55ebc90059a0 2024-03-24T10:45:03.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.670+0000 7f90cb7fe640 1 --2- 172.21.15.23:0/2490352152 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ebc90059a0 0x55ebc9007db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.671+0000 7f90cb7fe640 1 --2- 172.21.15.23:0/2490352152 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ebc90059a0 0x55ebc9007db0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6392931fc9a953fc server_cookie=f5349df9ce1065ec in_seq=0 out_seq=0 2024-03-24T10:45:03.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:03.671+0000 7f90ab7fe640 10 client.16947 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:04.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 451588 11.83 MB/sec execute 81 sec latency 62.130 ms 2024-03-24T10:45:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd86e7fc640 1 -- 172.21.15.23:0/2423180090 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56210f93e1b0 con 0x56210fd0b280 2024-03-24T10:45:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd86e7fc640 10 client.16935 handle_command_reply: tid=0 2024-03-24T10:45:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd88ff82740 1 client.16935 shutdown 2024-03-24T10:45:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd88ff82740 1 -- 172.21.15.23:0/2423180090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc875e0 msgr2=0x56210fc98860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd88ff82740 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc875e0 0x56210fc98860 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd880005ec0 tx=0x7fd880003760 comp rx=0 tx=0).stop 2024-03-24T10:45:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd88ff82740 1 -- 172.21.15.23:0/2423180090 shutdown_connections 2024-03-24T10:45:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd88ff82740 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56210fd0b280 0x56210fd0d6a0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd88ff82740 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56210fc90cd0 0x56210fc93eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd86e7fc640 0 client.16935 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd88ff82740 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56210fc87f00 0x56210fc98da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd88ff82740 1 --2- 172.21.15.23:0/2423180090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56210fc875e0 0x56210fc98860 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd88ff82740 1 -- 172.21.15.23:0/2423180090 >> 172.21.15.23:0/2423180090 conn(0x56210fc81f00 msgr2=0x56210fc80ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd88ff82740 1 -- 172.21.15.23:0/2423180090 shutdown_connections 2024-03-24T10:45:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd88ff82740 1 -- 172.21.15.23:0/2423180090 wait complete. 2024-03-24T10:45:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.928+0000 7fd88ff82740 20 client.16935 trim_cache size 0 max 16384 2024-03-24T10:45:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:45:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:45:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:45:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:45:04.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90ab7fe640 1 -- 172.21.15.23:0/2490352152 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ebc8c8f720 con 0x55ebc90059a0 2024-03-24T10:45:04.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90ab7fe640 10 client.16947 handle_command_reply: tid=0 2024-03-24T10:45:04.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90d4f6c740 1 client.16947 shutdown 2024-03-24T10:45:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.929+0000 7fd88d2d0640 1 -- 172.21.15.23:0/3336359263 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd8740a22d0 msgr2=0x7fd8740a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.929+0000 7fd88d2d0640 1 --2- 172.21.15.23:0/3336359263 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd8740a22d0 0x7fd8740a4790 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd888077b30 tx=0x7fd870005f90 comp rx=0 tx=0).stop 2024-03-24T10:45:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.929+0000 7fd88d2d0640 1 -- 172.21.15.23:0/3336359263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd888100860 msgr2=0x7fd888197de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.929+0000 7fd88d2d0640 1 --2- 172.21.15.23:0/3336359263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd888100860 0x7fd888197de0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd880036040 tx=0x7fd88002c680 comp rx=0 tx=0).stop 2024-03-24T10:45:04.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90d4f6c740 1 -- 172.21.15.23:0/2490352152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fec1d0 msgr2=0x55ebc8ff2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:04.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/2490352152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fec1d0 0x55ebc8ff2790 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f90bc03afd0 tx=0x7f90bc0036f0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90d4f6c740 1 -- 172.21.15.23:0/2490352152 shutdown_connections 2024-03-24T10:45:04.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/2490352152 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ebc90059a0 0x55ebc9007db0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.929+0000 7fd88d2d0640 1 -- 172.21.15.23:0/3336359263 shutdown_connections 2024-03-24T10:45:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.929+0000 7fd88d2d0640 1 --2- 172.21.15.23:0/3336359263 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd8740a22d0 0x7fd8740a4790 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.929+0000 7fd88d2d0640 1 --2- 172.21.15.23:0/3336359263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8881055b0 0x7fd88819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.929+0000 7fd88d2d0640 1 --2- 172.21.15.23:0/3336359263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd888101220 0x7fd888198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.929+0000 7fd88d2d0640 1 --2- 172.21.15.23:0/3336359263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd888100860 0x7fd888197de0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.929+0000 7fd88d2d0640 1 -- 172.21.15.23:0/3336359263 >> 172.21.15.23:0/3336359263 conn(0x7fd888077460 msgr2=0x7fd888069010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:04.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/2490352152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8ff56d0 0x55ebc8ff3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/2490352152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8fecaf0 0x55ebc8ff2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/2490352152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fec1d0 0x55ebc8ff2790 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90d4f6c740 1 -- 172.21.15.23:0/2490352152 >> 172.21.15.23:0/2490352152 conn(0x55ebc8fe6900 msgr2=0x55ebc8fe8650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:04.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90ab7fe640 0 client.16947 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:04.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90d4f6c740 1 -- 172.21.15.23:0/2490352152 shutdown_connections 2024-03-24T10:45:04.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90d4f6c740 1 -- 172.21.15.23:0/2490352152 wait complete. 2024-03-24T10:45:04.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.929+0000 7f90d4f6c740 20 client.16947 trim_cache size 0 max 16384 2024-03-24T10:45:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.929+0000 7fd88d2d0640 1 -- 172.21.15.23:0/3336359263 shutdown_connections 2024-03-24T10:45:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:04.929+0000 7fd88d2d0640 1 -- 172.21.15.23:0/3336359263 wait complete. 2024-03-24T10:45:04.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.947+0000 7f90d4f6c740 1 Processor -- start 2024-03-24T10:45:04.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.947+0000 7f90d4f6c740 1 -- start start 2024-03-24T10:45:04.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.947+0000 7f90d4f6c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fec1d0 0x55ebc9000f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:04.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.947+0000 7f90d4f6c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fecaf0 0x55ebc8ff3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:04.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.947+0000 7f90d4f6c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8ff56d0 0x55ebc8fe59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:04.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.947+0000 7f90d4f6c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ebc8ed8430 con 0x55ebc8fecaf0 2024-03-24T10:45:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.947+0000 7f90d4f6c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ebc90014d0 con 0x55ebc8ff56d0 2024-03-24T10:45:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.947+0000 7f90d4f6c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ebc8fe8c90 con 0x55ebc8fec1d0 2024-03-24T10:45:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.947+0000 7f90cb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fecaf0 0x55ebc8ff3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.947+0000 7f90cb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fecaf0 0x55ebc8ff3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52644/0 (socket says 172.21.15.23:52644) 2024-03-24T10:45:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.947+0000 7f90cbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fec1d0 0x55ebc9000f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.947+0000 7f90cb7fe640 1 -- 172.21.15.23:0/1888488146 learned_addr learned my addr 172.21.15.23:0/1888488146 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.947+0000 7f90d0927640 1 --2- 172.21.15.23:0/1888488146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8ff56d0 0x55ebc8fe59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.948+0000 7f90cbfff640 1 -- 172.21.15.23:0/1888488146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8ff56d0 msgr2=0x55ebc8fe59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.948+0000 7f90cbfff640 1 --2- 172.21.15.23:0/1888488146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8ff56d0 0x55ebc8fe59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.948+0000 7f90cbfff640 1 -- 172.21.15.23:0/1888488146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fecaf0 msgr2=0x55ebc8ff3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.948+0000 7f90cbfff640 1 --2- 172.21.15.23:0/1888488146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fecaf0 0x55ebc8ff3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.948+0000 7f90cbfff640 1 -- 172.21.15.23:0/1888488146 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc8eedb20 con 0x55ebc8fec1d0 2024-03-24T10:45:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.948+0000 7f90cb7fe640 1 --2- 172.21.15.23:0/1888488146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fecaf0 0x55ebc8ff3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:04.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.948+0000 7f90cbfff640 1 --2- 172.21.15.23:0/1888488146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fec1d0 0x55ebc9000f60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f90bc005940 tx=0x7f90bc005700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b547d34827e5bf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:04.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.948+0000 7f90b1ffb640 1 -- 172.21.15.23:0/1888488146 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc0bc640 con 0x55ebc8fec1d0 2024-03-24T10:45:04.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.948+0000 7f90b1ffb640 1 -- 172.21.15.23:0/1888488146 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90bc0c0e30 con 0x55ebc8fec1d0 2024-03-24T10:45:04.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.948+0000 7f90b1ffb640 1 -- 172.21.15.23:0/1888488146 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc003ed0 con 0x55ebc8fec1d0 2024-03-24T10:45:04.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.949+0000 7f90d4f6c740 1 -- 172.21.15.23:0/1888488146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fec1d0 msgr2=0x55ebc9000f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:04.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.949+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/1888488146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fec1d0 0x55ebc9000f60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f90bc005940 tx=0x7f90bc005700 comp rx=0 tx=0).stop 2024-03-24T10:45:04.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.949+0000 7f90d4f6c740 1 -- 172.21.15.23:0/1888488146 shutdown_connections 2024-03-24T10:45:04.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.949+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/1888488146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8ff56d0 0x55ebc8fe59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.949+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/1888488146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fecaf0 0x55ebc8ff3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.949+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/1888488146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fec1d0 0x55ebc9000f60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.949+0000 7f90d4f6c740 1 -- 172.21.15.23:0/1888488146 >> 172.21.15.23:0/1888488146 conn(0x55ebc8fe6900 msgr2=0x55ebc8fe52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:04.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.949+0000 7f90d4f6c740 1 -- 172.21.15.23:0/1888488146 shutdown_connections 2024-03-24T10:45:04.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.949+0000 7f90d4f6c740 1 -- 172.21.15.23:0/1888488146 wait complete. 2024-03-24T10:45:04.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.949+0000 7f90d4f6c740 1 -- start start 2024-03-24T10:45:04.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.949+0000 7f90d4f6c740 1 Processor -- start 2024-03-24T10:45:04.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.949+0000 7f90d4f6c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fec1d0 0x55ebc8ffd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:04.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90d4f6c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fecaf0 0x55ebc8feef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:04.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90d4f6c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8ff56d0 0x55ebc8fef4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:04.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90d4f6c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ebc8fefa00 con 0x55ebc8fec1d0 2024-03-24T10:45:04.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90d4f6c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ebc8fefb70 con 0x55ebc8ff56d0 2024-03-24T10:45:04.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90d4f6c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ebc8fefd30 con 0x55ebc8fecaf0 2024-03-24T10:45:04.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90d4f6c740 10 client.0 _ll_register_callbacks cb 0x7f90bc0b8fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:04.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90cbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fec1d0 0x55ebc8ffd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:04.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90cbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fec1d0 0x55ebc8ffd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52646/0 (socket says 172.21.15.23:52646) 2024-03-24T10:45:04.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90cbfff640 1 -- 172.21.15.23:0/3143438748 learned_addr learned my addr 172.21.15.23:0/3143438748 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:04.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90cb7fe640 1 --2- 172.21.15.23:0/3143438748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fecaf0 0x55ebc8feef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:04.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90d0927640 1 --2- 172.21.15.23:0/3143438748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8ff56d0 0x55ebc8fef4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90cbfff640 1 -- 172.21.15.23:0/3143438748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fecaf0 msgr2=0x55ebc8feef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90cbfff640 1 --2- 172.21.15.23:0/3143438748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fecaf0 0x55ebc8feef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90cbfff640 1 -- 172.21.15.23:0/3143438748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8ff56d0 msgr2=0x55ebc8fef4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90cbfff640 1 --2- 172.21.15.23:0/3143438748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8ff56d0 0x55ebc8fef4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90cbfff640 1 -- 172.21.15.23:0/3143438748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90c0017070 con 0x55ebc8fec1d0 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90cbfff640 1 --2- 172.21.15.23:0/3143438748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fec1d0 0x55ebc8ffd780 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0c4f50 tx=0x7f90bc0c5360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c5521bba0e10e30 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90ab7fe640 1 -- 172.21.15.23:0/3143438748 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc003eb0 con 0x55ebc8fec1d0 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90ab7fe640 1 -- 172.21.15.23:0/3143438748 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f90bc0bcce0 con 0x55ebc8fec1d0 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90d4f6c740 1 -- 172.21.15.23:0/3143438748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90c4003e50 con 0x55ebc8fec1d0 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90ab7fe640 1 -- 172.21.15.23:0/3143438748 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f90bc0bc640 con 0x55ebc8fec1d0 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90d4f6c740 10 client.16953 fetch_fsmap learned FSMap version 27 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.950+0000 7f90d4f6c740 1 -- 172.21.15.23:0/3143438748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ebc8f51ba0 con 0x55ebc8fec1d0 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.951+0000 7f90ab7fe640 1 -- 172.21.15.23:0/3143438748 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90c4003e50 con 0x55ebc8fec1d0 2024-03-24T10:45:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.951+0000 7f90ab7fe640 1 -- 172.21.15.23:0/3143438748 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f90bc03c070 con 0x55ebc8fec1d0 2024-03-24T10:45:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.951+0000 7f90d4f6c740 10 client.16953 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:45:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.951+0000 7f90d4f6c740 10 client.16953 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.951+0000 7f90d4f6c740 20 client.16953 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.951+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/3143438748 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ebc906bd00 0x55ebc906e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.951+0000 7f90d4f6c740 4 client.16953 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:45:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.951+0000 7f90d4f6c740 1 -- 172.21.15.23:0/3143438748 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ebc8cde890 con 0x55ebc906bd00 2024-03-24T10:45:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.951+0000 7f90cb7fe640 1 --2- 172.21.15.23:0/3143438748 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ebc906bd00 0x55ebc906e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.951+0000 7f90ab7fe640 10 client.16953 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.951+0000 7f90cb7fe640 1 --2- 172.21.15.23:0/3143438748 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ebc906bd00 0x55ebc906e0f0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f37f3c617eb12496 server_cookie=d0d7903cad220db2 in_seq=0 out_seq=0 2024-03-24T10:45:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90ab7fe640 1 -- 172.21.15.23:0/3143438748 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ebc8cde890 con 0x55ebc906bd00 2024-03-24T10:45:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90ab7fe640 10 client.16953 handle_command_reply: tid=0 2024-03-24T10:45:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d4f6c740 1 client.16953 shutdown 2024-03-24T10:45:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d4f6c740 1 -- 172.21.15.23:0/3143438748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fec1d0 msgr2=0x55ebc8ffd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/3143438748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fec1d0 0x55ebc8ffd780 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0c4f50 tx=0x7f90bc0c5360 comp rx=0 tx=0).stop 2024-03-24T10:45:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d4f6c740 1 -- 172.21.15.23:0/3143438748 shutdown_connections 2024-03-24T10:45:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/3143438748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ebc8ff56d0 0x55ebc8fef4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/3143438748 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ebc906bd00 0x55ebc906e0f0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/3143438748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebc8fecaf0 0x55ebc8feef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d4f6c740 1 --2- 172.21.15.23:0/3143438748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebc8fec1d0 0x55ebc8ffd780 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90ab7fe640 0 client.16953 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d4f6c740 1 -- 172.21.15.23:0/3143438748 >> 172.21.15.23:0/3143438748 conn(0x55ebc8fe6900 msgr2=0x55ebc8fe5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d4f6c740 1 -- 172.21.15.23:0/3143438748 shutdown_connections 2024-03-24T10:45:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d4f6c740 1 -- 172.21.15.23:0/3143438748 wait complete. 2024-03-24T10:45:04.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d4f6c740 20 client.16953 trim_cache size 0 max 16384 2024-03-24T10:45:04.959 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:45:04.959 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:45:04.960 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:45:04.960 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:45:04.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d238a640 1 -- 172.21.15.23:0/2557661260 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f90b40a2230 msgr2=0x7f90b40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:04.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d238a640 1 --2- 172.21.15.23:0/2557661260 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f90b40a2230 0x7f90b40a46f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f90cc0693c0 tx=0x7f90c0005f90 comp rx=0 tx=0).stop 2024-03-24T10:45:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d238a640 1 -- 172.21.15.23:0/2557661260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90cc1005a0 msgr2=0x7f90cc198b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.952+0000 7f90d238a640 1 --2- 172.21.15.23:0/2557661260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90cc1005a0 0x7f90cc198b80 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f90bc006010 tx=0x7f90bc02c700 comp rx=0 tx=0).stop 2024-03-24T10:45:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.953+0000 7f90d238a640 1 -- 172.21.15.23:0/2557661260 shutdown_connections 2024-03-24T10:45:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.953+0000 7f90d238a640 1 --2- 172.21.15.23:0/2557661260 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f90b40a2230 0x7f90b40a46f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.953+0000 7f90d238a640 1 --2- 172.21.15.23:0/2557661260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90cc106660 0x7f90cc1a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.953+0000 7f90d238a640 1 --2- 172.21.15.23:0/2557661260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90cc100f60 0x7f90cc1990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.953+0000 7f90d238a640 1 --2- 172.21.15.23:0/2557661260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90cc1005a0 0x7f90cc198b80 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.953+0000 7f90d238a640 1 -- 172.21.15.23:0/2557661260 >> 172.21.15.23:0/2557661260 conn(0x7f90cc077710 msgr2=0x7f90cc0ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.953+0000 7f90d238a640 1 -- 172.21.15.23:0/2557661260 shutdown_connections 2024-03-24T10:45:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:04.953+0000 7f90d238a640 1 -- 172.21.15.23:0/2557661260 wait complete. 2024-03-24T10:45:04.962 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:04 smithi023 ceph-mon[32466]: pgmap v152: 97 pgs: 97 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-03-24T10:45:04.963 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:45:04.964 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:45:05.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:04 smithi117 ceph-mon[36368]: pgmap v152: 97 pgs: 97 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-03-24T10:45:05.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:04 smithi116 ceph-mon[36248]: pgmap v152: 97 pgs: 97 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-03-24T10:45:05.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.070+0000 7f911d84e640 1 -- 172.21.15.23:0/467688662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9118106fd0 msgr2=0x7f9118100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.070+0000 7f911d84e640 1 --2- 172.21.15.23:0/467688662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9118106fd0 0x7f9118100610 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9100009900 tx=0x7f910002bcd0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.070+0000 7f911d84e640 1 -- 172.21.15.23:0/467688662 shutdown_connections 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.070+0000 7f911d84e640 1 --2- 172.21.15.23:0/467688662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118100d40 0x7f911810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.070+0000 7f911d84e640 1 --2- 172.21.15.23:0/467688662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9118106fd0 0x7f9118100610 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.070+0000 7f911d84e640 1 --2- 172.21.15.23:0/467688662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91181066b0 0x7f9118106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.070+0000 7f911d84e640 1 -- 172.21.15.23:0/467688662 >> 172.21.15.23:0/467688662 conn(0x7f9118077710 msgr2=0x7f9118077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.070+0000 7f911d84e640 1 -- 172.21.15.23:0/467688662 shutdown_connections 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.070+0000 7f911d84e640 1 -- 172.21.15.23:0/467688662 wait complete. 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f911d84e640 1 Processor -- start 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f911d84e640 1 -- start start 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f911d84e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9118100d40 0x7f9118199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f911d84e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91181066b0 0x7f911819a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f911d84e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118106fd0 0x7f911819e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f911d84e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f911819ee60 con 0x7f91181066b0 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f911d84e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f911819efd0 con 0x7f9118106fd0 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f911d84e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f911819f190 con 0x7f9118100d40 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f9116ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9118100d40 0x7f9118199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f91167fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91181066b0 0x7f911819a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f9116ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9118100d40 0x7f9118199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:43286/0 (socket says 172.21.15.23:43286) 2024-03-24T10:45:05.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f9116ffd640 1 -- 172.21.15.23:0/2919391037 learned_addr learned my addr 172.21.15.23:0/2919391037 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f91177fe640 1 --2- 172.21.15.23:0/2919391037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118106fd0 0x7f911819e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f91167fc640 1 -- 172.21.15.23:0/2919391037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9118100d40 msgr2=0x7f9118199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f91167fc640 1 --2- 172.21.15.23:0/2919391037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9118100d40 0x7f9118199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.071+0000 7f91167fc640 1 -- 172.21.15.23:0/2919391037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118106fd0 msgr2=0x7f911819e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.072+0000 7f91167fc640 1 --2- 172.21.15.23:0/2919391037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118106fd0 0x7f911819e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.072+0000 7f91167fc640 1 -- 172.21.15.23:0/2919391037 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911000b6a0 con 0x7f91181066b0 2024-03-24T10:45:05.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.072+0000 7f91167fc640 1 --2- 172.21.15.23:0/2919391037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91181066b0 0x7f911819a4e0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f910002c210 tx=0x7f910002c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.072+0000 7f911c84c640 1 -- 172.21.15.23:0/2919391037 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9100004350 con 0x7f91181066b0 2024-03-24T10:45:05.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.072+0000 7f911d84e640 1 -- 172.21.15.23:0/2919391037 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9100009560 con 0x7f91181066b0 2024-03-24T10:45:05.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.072+0000 7f911c84c640 1 -- 172.21.15.23:0/2919391037 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91000044b0 con 0x7f91181066b0 2024-03-24T10:45:05.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.072+0000 7f911c84c640 1 -- 172.21.15.23:0/2919391037 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9100003da0 con 0x7f91181066b0 2024-03-24T10:45:05.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.072+0000 7f911d84e640 1 -- 172.21.15.23:0/2919391037 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91181a6b10 con 0x7f91181066b0 2024-03-24T10:45:05.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.073+0000 7f911d84e640 1 -- 172.21.15.23:0/2919391037 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9118068500 con 0x7f91181066b0 2024-03-24T10:45:05.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.074+0000 7f911c84c640 1 -- 172.21.15.23:0/2919391037 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f9100004df0 con 0x7f91181066b0 2024-03-24T10:45:05.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.075+0000 7f911c84c640 1 --2- 172.21.15.23:0/2919391037 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f91040a2230 0x7f91040a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.075+0000 7f9116ffd640 1 --2- 172.21.15.23:0/2919391037 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f91040a2230 0x7f91040a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.075+0000 7f911c84c640 1 -- 172.21.15.23:0/2919391037 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f91000f05a0 con 0x7f91181066b0 2024-03-24T10:45:05.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.075+0000 7f9116ffd640 1 --2- 172.21.15.23:0/2919391037 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f91040a2230 0x7f91040a46f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f911000b3c0 tx=0x7f911000d040 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.079+0000 7f911c84c640 1 -- 172.21.15.23:0/2919391037 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f91000b9340 con 0x7f91181066b0 2024-03-24T10:45:05.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 454400 11.89 MB/sec execute 82 sec latency 11.230 ms 2024-03-24T10:45:05.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.310+0000 7f911d84e640 1 -- 172.21.15.23:0/2919391037 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9118108ba0 con 0x7f91181066b0 2024-03-24T10:45:05.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.311+0000 7f911c84c640 1 -- 172.21.15.23:0/2919391037 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11866 (secure 0 0 0) 0x7f910002ce60 con 0x7f91181066b0 2024-03-24T10:45:05.311 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:45:05.311 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:44:24.924154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/977076441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":977076441},{"type":"v1","addr":"172.21.15.117:6837","nonce":977076441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24505,24539,24503]},"id":1}]} 2024-03-24T10:45:05.312 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 27 2024-03-24T10:45:05.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.313+0000 7f911d84e640 1 -- 172.21.15.23:0/2919391037 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f91040a2230 msgr2=0x7f91040a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.313+0000 7f911d84e640 1 --2- 172.21.15.23:0/2919391037 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f91040a2230 0x7f91040a46f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f911000b3c0 tx=0x7f911000d040 comp rx=0 tx=0).stop 2024-03-24T10:45:05.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.313+0000 7f911d84e640 1 -- 172.21.15.23:0/2919391037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91181066b0 msgr2=0x7f911819a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.313+0000 7f911d84e640 1 --2- 172.21.15.23:0/2919391037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91181066b0 0x7f911819a4e0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f910002c210 tx=0x7f910002c720 comp rx=0 tx=0).stop 2024-03-24T10:45:05.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.313+0000 7f911d84e640 1 -- 172.21.15.23:0/2919391037 shutdown_connections 2024-03-24T10:45:05.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.313+0000 7f911d84e640 1 --2- 172.21.15.23:0/2919391037 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f91040a2230 0x7f91040a46f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.313+0000 7f911d84e640 1 --2- 172.21.15.23:0/2919391037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118106fd0 0x7f911819e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.313+0000 7f911d84e640 1 --2- 172.21.15.23:0/2919391037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91181066b0 0x7f911819a4e0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.313+0000 7f911d84e640 1 --2- 172.21.15.23:0/2919391037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9118100d40 0x7f9118199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.313+0000 7f911d84e640 1 -- 172.21.15.23:0/2919391037 >> 172.21.15.23:0/2919391037 conn(0x7f9118077710 msgr2=0x7f91180ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.313+0000 7f911d84e640 1 -- 172.21.15.23:0/2919391037 shutdown_connections 2024-03-24T10:45:05.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.313+0000 7f911d84e640 1 -- 172.21.15.23:0/2919391037 wait complete. 2024-03-24T10:45:05.324 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:45:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.436+0000 7f1252f25640 1 -- 172.21.15.23:0/456791452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f124c100570 msgr2=0x7f124c10e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.436+0000 7f1252f25640 1 --2- 172.21.15.23:0/456791452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f124c100570 0x7f124c10e530 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f1244009cd0 tx=0x7f124402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.436+0000 7f1252f25640 1 -- 172.21.15.23:0/456791452 shutdown_connections 2024-03-24T10:45:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.436+0000 7f1252f25640 1 --2- 172.21.15.23:0/456791452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f124c100570 0x7f124c10e530 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.436+0000 7f1252f25640 1 --2- 172.21.15.23:0/456791452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f124c0ffbb0 0x7f124c100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.436+0000 7f1252f25640 1 --2- 172.21.15.23:0/456791452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f124c1076b0 0x7f124c107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.436+0000 7f1252f25640 1 -- 172.21.15.23:0/456791452 >> 172.21.15.23:0/456791452 conn(0x7f124c076b30 msgr2=0x7f124c0fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.436+0000 7f1252f25640 1 -- 172.21.15.23:0/456791452 shutdown_connections 2024-03-24T10:45:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.436+0000 7f1252f25640 1 -- 172.21.15.23:0/456791452 wait complete. 2024-03-24T10:45:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.437+0000 7f1252f25640 1 Processor -- start 2024-03-24T10:45:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.437+0000 7f1252f25640 1 -- start start 2024-03-24T10:45:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.437+0000 7f1252f25640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f124c0ffbb0 0x7f124c199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.437+0000 7f1252f25640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f124c100570 0x7f124c19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.437+0000 7f1252f25640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f124c1076b0 0x7f124c19e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.437+0000 7f1252f25640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f124c19ede0 con 0x7f124c0ffbb0 2024-03-24T10:45:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.437+0000 7f1252f25640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f124c19ef50 con 0x7f124c100570 2024-03-24T10:45:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.437+0000 7f1252f25640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f124c19f110 con 0x7f124c1076b0 2024-03-24T10:45:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.437+0000 7f1250cc1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f124c0ffbb0 0x7f124c199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.437+0000 7f1250cc1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f124c0ffbb0 0x7f124c199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52676/0 (socket says 172.21.15.23:52676) 2024-03-24T10:45:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.437+0000 7f124bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f124c100570 0x7f124c19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.437+0000 7f1250cc1640 1 -- 172.21.15.23:0/3764939150 learned_addr learned my addr 172.21.15.23:0/3764939150 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.437+0000 7f12514c2640 1 --2- 172.21.15.23:0/3764939150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f124c1076b0 0x7f124c19e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.438+0000 7f1250cc1640 1 -- 172.21.15.23:0/3764939150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f124c1076b0 msgr2=0x7f124c19e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.438+0000 7f1250cc1640 1 --2- 172.21.15.23:0/3764939150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f124c1076b0 0x7f124c19e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.438+0000 7f1250cc1640 1 -- 172.21.15.23:0/3764939150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f124c100570 msgr2=0x7f124c19a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.438+0000 7f1250cc1640 1 --2- 172.21.15.23:0/3764939150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f124c100570 0x7f124c19a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.438+0000 7f1250cc1640 1 -- 172.21.15.23:0/3764939150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1244009900 con 0x7f124c0ffbb0 2024-03-24T10:45:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.438+0000 7f124bfff640 1 --2- 172.21.15.23:0/3764939150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f124c100570 0x7f124c19a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.438+0000 7f1250cc1640 1 --2- 172.21.15.23:0/3764939150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f124c0ffbb0 0x7f124c199f20 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f124000e9f0 tx=0x7f124000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.438+0000 7f1249ffb640 1 -- 172.21.15.23:0/3764939150 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1240009700 con 0x7f124c0ffbb0 2024-03-24T10:45:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.438+0000 7f1249ffb640 1 -- 172.21.15.23:0/3764939150 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f124000cd80 con 0x7f124c0ffbb0 2024-03-24T10:45:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.438+0000 7f1252f25640 1 -- 172.21.15.23:0/3764939150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f124c19f3f0 con 0x7f124c0ffbb0 2024-03-24T10:45:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.438+0000 7f1249ffb640 1 -- 172.21.15.23:0/3764939150 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1240018920 con 0x7f124c0ffbb0 2024-03-24T10:45:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.438+0000 7f1252f25640 1 -- 172.21.15.23:0/3764939150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f124c1a6c70 con 0x7f124c0ffbb0 2024-03-24T10:45:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.440+0000 7f1249ffb640 1 -- 172.21.15.23:0/3764939150 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f1240018a80 con 0x7f124c0ffbb0 2024-03-24T10:45:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.441+0000 7f1249ffb640 1 --2- 172.21.15.23:0/3764939150 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f123c0a22b0 0x7f123c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.441+0000 7f1249ffb640 1 -- 172.21.15.23:0/3764939150 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f1240014070 con 0x7f124c0ffbb0 2024-03-24T10:45:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.441+0000 7f124bfff640 1 --2- 172.21.15.23:0/3764939150 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f123c0a22b0 0x7f123c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.441+0000 7f1255be7740 1 Processor -- start 2024-03-24T10:45:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f1255be7740 1 -- start start 2024-03-24T10:45:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f124bfff640 1 --2- 172.21.15.23:0/3764939150 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f123c0a22b0 0x7f123c0a4770 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f124c10e270 tx=0x7f1238005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f1255be7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb4273f0 0x55a3fb4277d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f1255be7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb427d10 0x55a3fb4303b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f1255be7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 0x55a3fb432ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f1255be7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a3fb395840 con 0x55a3fb427d10 2024-03-24T10:45:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f1255be7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a3fb392d70 con 0x55a3fb4273f0 2024-03-24T10:45:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f1255be7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a3fb38e550 con 0x55a3fb4308f0 2024-03-24T10:45:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f124bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb427d10 0x55a3fb4303b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f1250cc1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb4273f0 0x55a3fb4277d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f12514c2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 0x55a3fb432ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f12514c2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 0x55a3fb432ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:43312/0 (socket says 172.21.15.23:43312) 2024-03-24T10:45:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f124bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb427d10 0x55a3fb4303b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52688/0 (socket says 172.21.15.23:52688) 2024-03-24T10:45:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f1250cc1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb4273f0 0x55a3fb4277d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:50968/0 (socket says 172.21.15.23:50968) 2024-03-24T10:45:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f12514c2640 1 -- 172.21.15.23:0/1809140284 learned_addr learned my addr 172.21.15.23:0/1809140284 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f12514c2640 1 -- 172.21.15.23:0/1809140284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb4273f0 msgr2=0x55a3fb4277d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f12514c2640 1 --2- 172.21.15.23:0/1809140284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb4273f0 0x55a3fb4277d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f12514c2640 1 -- 172.21.15.23:0/1809140284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb427d10 msgr2=0x55a3fb4303b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f12514c2640 1 --2- 172.21.15.23:0/1809140284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb427d10 0x55a3fb4303b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f12514c2640 1 -- 172.21.15.23:0/1809140284 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3fb323de0 con 0x55a3fb4308f0 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.442+0000 7f12514c2640 1 --2- 172.21.15.23:0/1809140284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 0x55a3fb432ce0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f124402c700 tx=0x7f1244032d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14e0d66196b233c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.443+0000 7f1236ffd640 1 -- 172.21.15.23:0/1809140284 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1244034440 con 0x55a3fb4308f0 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.443+0000 7f1236ffd640 1 -- 172.21.15.23:0/1809140284 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12440345a0 con 0x55a3fb4308f0 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.443+0000 7f1236ffd640 1 -- 172.21.15.23:0/1809140284 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1244003bd0 con 0x55a3fb4308f0 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.443+0000 7f1255be7740 1 -- 172.21.15.23:0/1809140284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 msgr2=0x55a3fb432ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.443+0000 7f1255be7740 1 --2- 172.21.15.23:0/1809140284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 0x55a3fb432ce0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f124402c700 tx=0x7f1244032d70 comp rx=0 tx=0).stop 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.443+0000 7f1255be7740 1 -- 172.21.15.23:0/1809140284 shutdown_connections 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.443+0000 7f1255be7740 1 --2- 172.21.15.23:0/1809140284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 0x55a3fb432ce0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.443+0000 7f1255be7740 1 --2- 172.21.15.23:0/1809140284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb427d10 0x55a3fb4303b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.443+0000 7f1255be7740 1 --2- 172.21.15.23:0/1809140284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb4273f0 0x55a3fb4277d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.443+0000 7f1255be7740 1 -- 172.21.15.23:0/1809140284 >> 172.21.15.23:0/1809140284 conn(0x55a3fb421b20 msgr2=0x55a3fb421f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.443+0000 7f1255be7740 1 -- 172.21.15.23:0/1809140284 shutdown_connections 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.443+0000 7f1255be7740 1 -- 172.21.15.23:0/1809140284 wait complete. 2024-03-24T10:45:05.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f1255be7740 1 -- start start 2024-03-24T10:45:05.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f1255be7740 1 Processor -- start 2024-03-24T10:45:05.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f1255be7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb4273f0 0x55a3fb42d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f1255be7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb427d10 0x55a3fb42de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f1255be7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4308f0 0x55a3fb42e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f1255be7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a3fb061430 con 0x55a3fb4308f0 2024-03-24T10:45:05.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f1255be7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a3fb01de00 con 0x55a3fb4273f0 2024-03-24T10:45:05.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f1255be7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a3fb313590 con 0x55a3fb427d10 2024-03-24T10:45:05.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f1255be7740 10 client.0 _ll_register_callbacks cb 0x55a3fb382fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f12514c2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4308f0 0x55a3fb42e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f1250cc1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb4273f0 0x55a3fb42d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f124bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb427d10 0x55a3fb42de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f12514c2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4308f0 0x55a3fb42e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52700/0 (socket says 172.21.15.23:52700) 2024-03-24T10:45:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f1250cc1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb4273f0 0x55a3fb42d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:50984/0 (socket says 172.21.15.23:50984) 2024-03-24T10:45:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f12514c2640 1 -- 172.21.15.23:0/3980906590 learned_addr learned my addr 172.21.15.23:0/3980906590 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.444+0000 7f124bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb427d10 0x55a3fb42de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:43314/0 (socket says 172.21.15.23:43314) 2024-03-24T10:45:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f12514c2640 1 -- 172.21.15.23:0/3980906590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb427d10 msgr2=0x55a3fb42de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f12514c2640 1 --2- 172.21.15.23:0/3980906590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb427d10 0x55a3fb42de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f12514c2640 1 -- 172.21.15.23:0/3980906590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb4273f0 msgr2=0x55a3fb42d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f12514c2640 1 --2- 172.21.15.23:0/3980906590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb4273f0 0x55a3fb42d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f12514c2640 1 -- 172.21.15.23:0/3980906590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f124c10edc0 con 0x55a3fb4308f0 2024-03-24T10:45:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f1250cc1640 1 --2- 172.21.15.23:0/3980906590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb4273f0 0x55a3fb42d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:05.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f124bfff640 1 --2- 172.21.15.23:0/3980906590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb427d10 0x55a3fb42de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f12514c2640 1 --2- 172.21.15.23:0/3980906590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4308f0 0x55a3fb42e3b0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f124402c630 tx=0x7f12440093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9153da87eb46af5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f122bfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f122bfff640 1 -- 172.21.15.23:0/3980906590 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1244034ea0 con 0x55a3fb4308f0 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f122bfff640 1 -- 172.21.15.23:0/3980906590 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f1244003970 con 0x55a3fb4308f0 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f1255be7740 1 -- 172.21.15.23:0/3980906590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a3fb08d7e0 con 0x55a3fb4308f0 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f122bfff640 1 -- 172.21.15.23:0/3980906590 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1244003b30 con 0x55a3fb4308f0 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f1255be7740 10 client.16971 fetch_fsmap learned FSMap version 27 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f1255be7740 1 -- 172.21.15.23:0/3980906590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a3fb38cbd0 con 0x55a3fb4308f0 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.445+0000 7f122bfff640 1 -- 172.21.15.23:0/3980906590 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a3fb08d7e0 con 0x55a3fb4308f0 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.446+0000 7f122bfff640 1 -- 172.21.15.23:0/3980906590 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f124404a450 con 0x55a3fb4308f0 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.446+0000 7f1255be7740 10 client.16971 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.446+0000 7f1255be7740 10 client.16971 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.446+0000 7f1255be7740 20 client.16971 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.446+0000 7f1255be7740 1 --2- 172.21.15.23:0/3980906590 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55a3fb4405c0 0x55a3fb4429e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.446+0000 7f1255be7740 4 client.16971 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:05.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.446+0000 7f1255be7740 1 -- 172.21.15.23:0/3980906590 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3fb0ca720 con 0x55a3fb4405c0 2024-03-24T10:45:05.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.446+0000 7f1250cc1640 1 --2- 172.21.15.23:0/3980906590 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55a3fb4405c0 0x55a3fb4429e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.446+0000 7f122bfff640 10 client.16971 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:45:05.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.446+0000 7f1250cc1640 1 --2- 172.21.15.23:0/3980906590 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55a3fb4405c0 0x55a3fb4429e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6a63b7be47db0c4b server_cookie=b7891898353bbaa in_seq=0 out_seq=0 2024-03-24T10:45:05.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.447+0000 7f122bfff640 1 -- 172.21.15.23:0/3980906590 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a3fb0ca720 con 0x55a3fb4405c0 2024-03-24T10:45:05.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.447+0000 7f122bfff640 10 client.16971 handle_command_reply: tid=0 2024-03-24T10:45:05.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.447+0000 7f1255be7740 1 client.16971 shutdown 2024-03-24T10:45:05.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.447+0000 7f1255be7740 1 -- 172.21.15.23:0/3980906590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4308f0 msgr2=0x55a3fb42e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.447+0000 7f1255be7740 1 --2- 172.21.15.23:0/3980906590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4308f0 0x55a3fb42e3b0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f124402c630 tx=0x7f12440093e0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.448+0000 7f1255be7740 1 -- 172.21.15.23:0/3980906590 shutdown_connections 2024-03-24T10:45:05.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.448+0000 7f1255be7740 1 --2- 172.21.15.23:0/3980906590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4308f0 0x55a3fb42e3b0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.448+0000 7f1255be7740 1 --2- 172.21.15.23:0/3980906590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb427d10 0x55a3fb42de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.448+0000 7f1255be7740 1 --2- 172.21.15.23:0/3980906590 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55a3fb4405c0 0x55a3fb4429e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.448+0000 7f1255be7740 1 --2- 172.21.15.23:0/3980906590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb4273f0 0x55a3fb42d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.448+0000 7f122bfff640 0 client.16971 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:45:05.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.448+0000 7f1255be7740 1 -- 172.21.15.23:0/3980906590 >> 172.21.15.23:0/3980906590 conn(0x55a3fb421b20 msgr2=0x55a3fb423870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.448+0000 7f1255be7740 1 -- 172.21.15.23:0/3980906590 shutdown_connections 2024-03-24T10:45:05.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.448+0000 7f1255be7740 1 -- 172.21.15.23:0/3980906590 wait complete. 2024-03-24T10:45:05.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.448+0000 7f1255be7740 20 client.16971 trim_cache size 0 max 16384 2024-03-24T10:45:05.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1255be7740 1 Processor -- start 2024-03-24T10:45:05.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1255be7740 1 -- start start 2024-03-24T10:45:05.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1255be7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4273f0 0x55a3fb430060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1255be7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb427d10 0x55a3fb43c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1255be7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 0x55a3fb4482d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1255be7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a3fb41d250 con 0x55a3fb4273f0 2024-03-24T10:45:05.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1255be7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a3fb08d7e0 con 0x55a3fb427d10 2024-03-24T10:45:05.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1255be7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a3fb43d480 con 0x55a3fb4308f0 2024-03-24T10:45:05.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1250cc1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4273f0 0x55a3fb430060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1250cc1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4273f0 0x55a3fb430060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52702/0 (socket says 172.21.15.23:52702) 2024-03-24T10:45:05.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1250cc1640 1 -- 172.21.15.23:0/2692133818 learned_addr learned my addr 172.21.15.23:0/2692133818 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f124bfff640 1 --2- 172.21.15.23:0/2692133818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb427d10 0x55a3fb43c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f12514c2640 1 --2- 172.21.15.23:0/2692133818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 0x55a3fb4482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1250cc1640 1 -- 172.21.15.23:0/2692133818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 msgr2=0x55a3fb4482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1250cc1640 1 --2- 172.21.15.23:0/2692133818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 0x55a3fb4482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1250cc1640 1 -- 172.21.15.23:0/2692133818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb427d10 msgr2=0x55a3fb43c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1250cc1640 1 --2- 172.21.15.23:0/2692133818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb427d10 0x55a3fb43c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1250cc1640 1 -- 172.21.15.23:0/2692133818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3fb328c80 con 0x55a3fb4273f0 2024-03-24T10:45:05.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.463+0000 7f1250cc1640 1 --2- 172.21.15.23:0/2692133818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4273f0 0x55a3fb430060 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f1240097200 tx=0x7f124001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2ffe1774e740ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.464+0000 7f12367fc640 1 -- 172.21.15.23:0/2692133818 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12400b9260 con 0x55a3fb4273f0 2024-03-24T10:45:05.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.464+0000 7f12367fc640 1 -- 172.21.15.23:0/2692133818 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12400a3490 con 0x55a3fb4273f0 2024-03-24T10:45:05.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.464+0000 7f12367fc640 1 -- 172.21.15.23:0/2692133818 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124009a1f0 con 0x55a3fb4273f0 2024-03-24T10:45:05.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.464+0000 7f1255be7740 1 -- 172.21.15.23:0/2692133818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4273f0 msgr2=0x55a3fb430060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.464+0000 7f1255be7740 1 --2- 172.21.15.23:0/2692133818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4273f0 0x55a3fb430060 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f1240097200 tx=0x7f124001a040 comp rx=0 tx=0).stop 2024-03-24T10:45:05.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.464+0000 7f1255be7740 1 -- 172.21.15.23:0/2692133818 shutdown_connections 2024-03-24T10:45:05.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.464+0000 7f1255be7740 1 --2- 172.21.15.23:0/2692133818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 0x55a3fb4482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.464+0000 7f1255be7740 1 --2- 172.21.15.23:0/2692133818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb427d10 0x55a3fb43c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.464+0000 7f1255be7740 1 --2- 172.21.15.23:0/2692133818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4273f0 0x55a3fb430060 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.464+0000 7f1255be7740 1 -- 172.21.15.23:0/2692133818 >> 172.21.15.23:0/2692133818 conn(0x55a3fb421b20 msgr2=0x55a3fb421f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.464+0000 7f1255be7740 1 -- 172.21.15.23:0/2692133818 shutdown_connections 2024-03-24T10:45:05.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.464+0000 7f1255be7740 1 -- 172.21.15.23:0/2692133818 wait complete. 2024-03-24T10:45:05.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.465+0000 7f1255be7740 1 -- start start 2024-03-24T10:45:05.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.465+0000 7f1255be7740 1 Processor -- start 2024-03-24T10:45:05.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.465+0000 7f1255be7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4273f0 0x55a3fb438a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.465+0000 7f1255be7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb427d10 0x55a3fb433db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.465+0000 7f1255be7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 0x55a3fb4342f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.465+0000 7f1255be7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a3fb4349f0 con 0x55a3fb4273f0 2024-03-24T10:45:05.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.465+0000 7f1255be7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a3fb434b60 con 0x55a3fb427d10 2024-03-24T10:45:05.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.465+0000 7f1255be7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a3fb4a5900 con 0x55a3fb4308f0 2024-03-24T10:45:05.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.465+0000 7f1255be7740 10 client.0 _ll_register_callbacks cb 0x7f1238009db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:05.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.465+0000 7f1250cc1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4273f0 0x55a3fb438a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.465+0000 7f1250cc1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4273f0 0x55a3fb438a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52712/0 (socket says 172.21.15.23:52712) 2024-03-24T10:45:05.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.465+0000 7f1250cc1640 1 -- 172.21.15.23:0/2976373088 learned_addr learned my addr 172.21.15.23:0/2976373088 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.465+0000 7f124bfff640 1 --2- 172.21.15.23:0/2976373088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb427d10 0x55a3fb433db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.465+0000 7f12514c2640 1 --2- 172.21.15.23:0/2976373088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 0x55a3fb4342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1250cc1640 1 -- 172.21.15.23:0/2976373088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 msgr2=0x55a3fb4342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1250cc1640 1 --2- 172.21.15.23:0/2976373088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 0x55a3fb4342f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1250cc1640 1 -- 172.21.15.23:0/2976373088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb427d10 msgr2=0x55a3fb433db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1250cc1640 1 --2- 172.21.15.23:0/2976373088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb427d10 0x55a3fb433db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1250cc1640 1 -- 172.21.15.23:0/2976373088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f124c1a6c70 con 0x55a3fb4273f0 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1250cc1640 1 --2- 172.21.15.23:0/2976373088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4273f0 0x55a3fb438a40 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f124009ad30 tx=0x7f124009ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbadce6350ddb307 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f122bfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f122bfff640 1 -- 172.21.15.23:0/2976373088 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12400cbcd0 con 0x55a3fb4273f0 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f122bfff640 1 -- 172.21.15.23:0/2976373088 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f12400cbe30 con 0x55a3fb4273f0 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1255be7740 1 -- 172.21.15.23:0/2976373088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12440345a0 con 0x55a3fb4273f0 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f122bfff640 1 -- 172.21.15.23:0/2976373088 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1240097410 con 0x55a3fb4273f0 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1255be7740 10 client.16977 fetch_fsmap learned FSMap version 27 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1255be7740 1 -- 172.21.15.23:0/2976373088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a3fb34e920 con 0x55a3fb4273f0 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f122bfff640 1 -- 172.21.15.23:0/2976373088 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12440345a0 con 0x55a3fb4273f0 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f122bfff640 1 -- 172.21.15.23:0/2976373088 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f1240099e20 con 0x55a3fb4273f0 2024-03-24T10:45:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1255be7740 10 client.16977 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:45:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1255be7740 10 client.16977 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:45:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1255be7740 20 client.16977 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1255be7740 1 --2- 172.21.15.23:0/2976373088 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55a3fb4a6d00 0x55a3fb4a90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1255be7740 4 client.16977 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:45:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.466+0000 7f1255be7740 1 -- 172.21.15.23:0/2976373088 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a3fb119890 con 0x55a3fb4a6d00 2024-03-24T10:45:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f124bfff640 1 --2- 172.21.15.23:0/2976373088 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55a3fb4a6d00 0x55a3fb4a90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f124bfff640 1 --2- 172.21.15.23:0/2976373088 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55a3fb4a6d00 0x55a3fb4a90f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8e4333f0030330d8 server_cookie=cd4cf8f8bbaa8468 in_seq=0 out_seq=0 2024-03-24T10:45:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f122bfff640 10 client.16977 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:45:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f122bfff640 1 -- 172.21.15.23:0/2976373088 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a3fb119890 con 0x55a3fb4a6d00 2024-03-24T10:45:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f122bfff640 10 client.16977 handle_command_reply: tid=0 2024-03-24T10:45:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f1255be7740 1 client.16977 shutdown 2024-03-24T10:45:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f1255be7740 1 -- 172.21.15.23:0/2976373088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4273f0 msgr2=0x55a3fb438a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f1255be7740 1 --2- 172.21.15.23:0/2976373088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4273f0 0x55a3fb438a40 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f124009ad30 tx=0x7f124009ef70 comp rx=0 tx=0).stop 2024-03-24T10:45:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f1255be7740 1 -- 172.21.15.23:0/2976373088 shutdown_connections 2024-03-24T10:45:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f1255be7740 1 --2- 172.21.15.23:0/2976373088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fb4308f0 0x55a3fb4342f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f1255be7740 1 --2- 172.21.15.23:0/2976373088 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55a3fb4a6d00 0x55a3fb4a90f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f1255be7740 1 --2- 172.21.15.23:0/2976373088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fb427d10 0x55a3fb433db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f1255be7740 1 --2- 172.21.15.23:0/2976373088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a3fb4273f0 0x55a3fb438a40 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f122bfff640 0 client.16977 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:45:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.467+0000 7f1255be7740 1 -- 172.21.15.23:0/2976373088 >> 172.21.15.23:0/2976373088 conn(0x55a3fb421b20 msgr2=0x55a3fb420430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.468+0000 7f1255be7740 1 -- 172.21.15.23:0/2976373088 shutdown_connections 2024-03-24T10:45:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.468+0000 7f1255be7740 1 -- 172.21.15.23:0/2976373088 wait complete. 2024-03-24T10:45:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.468+0000 7f1255be7740 20 client.16977 trim_cache size 0 max 16384 2024-03-24T10:45:05.474 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:45:05.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.468+0000 7f1252f25640 1 -- 172.21.15.23:0/3764939150 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f123c0a22b0 msgr2=0x7f123c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.468+0000 7f1252f25640 1 --2- 172.21.15.23:0/3764939150 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f123c0a22b0 0x7f123c0a4770 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f124c10e270 tx=0x7f1238005f90 comp rx=0 tx=0).stop 2024-03-24T10:45:05.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.468+0000 7f1252f25640 1 -- 172.21.15.23:0/3764939150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f124c0ffbb0 msgr2=0x7f124c199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.468+0000 7f1252f25640 1 --2- 172.21.15.23:0/3764939150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f124c0ffbb0 0x7f124c199f20 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f124000e9f0 tx=0x7f124000eeb0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.468+0000 7f1252f25640 1 -- 172.21.15.23:0/3764939150 shutdown_connections 2024-03-24T10:45:05.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.468+0000 7f1252f25640 1 --2- 172.21.15.23:0/3764939150 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f123c0a22b0 0x7f123c0a4770 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.468+0000 7f1252f25640 1 --2- 172.21.15.23:0/3764939150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f124c1076b0 0x7f124c19e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.468+0000 7f1252f25640 1 --2- 172.21.15.23:0/3764939150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f124c100570 0x7f124c19a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.468+0000 7f1252f25640 1 --2- 172.21.15.23:0/3764939150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f124c0ffbb0 0x7f124c199f20 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.468+0000 7f1252f25640 1 -- 172.21.15.23:0/3764939150 >> 172.21.15.23:0/3764939150 conn(0x7f124c076b30 msgr2=0x7f124c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.468+0000 7f1252f25640 1 -- 172.21.15.23:0/3764939150 shutdown_connections 2024-03-24T10:45:05.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.469+0000 7f1252f25640 1 -- 172.21.15.23:0/3764939150 wait complete. 2024-03-24T10:45:05.479 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:45:05.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.589+0000 7f0cfe655640 1 -- 172.21.15.23:0/3245766790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf8100f60 msgr2=0x7f0cf810d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.589+0000 7f0cfe655640 1 --2- 172.21.15.23:0/3245766790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf8100f60 0x7f0cf810d420 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0009ca0 tx=0x7f0cf002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.589+0000 7f0cfe655640 1 -- 172.21.15.23:0/3245766790 shutdown_connections 2024-03-24T10:45:05.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.589+0000 7f0cfe655640 1 --2- 172.21.15.23:0/3245766790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf8100f60 0x7f0cf810d420 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.589+0000 7f0cfe655640 1 --2- 172.21.15.23:0/3245766790 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cf81005a0 0x7f0cf8100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.589+0000 7f0cfe655640 1 --2- 172.21.15.23:0/3245766790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf8106660 0x7f0cf8106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.589+0000 7f0cfe655640 1 -- 172.21.15.23:0/3245766790 >> 172.21.15.23:0/3245766790 conn(0x7f0cf8077710 msgr2=0x7f0cf8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.589+0000 7f0cfe655640 1 -- 172.21.15.23:0/3245766790 shutdown_connections 2024-03-24T10:45:05.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.589+0000 7f0cfe655640 1 -- 172.21.15.23:0/3245766790 wait complete. 2024-03-24T10:45:05.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.589+0000 7f0cfe655640 1 Processor -- start 2024-03-24T10:45:05.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.589+0000 7f0cfe655640 1 -- start start 2024-03-24T10:45:05.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cfe655640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf81005a0 0x7f0cf819d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cfe655640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf8100f60 0x7f0cf819ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cfe655640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cf8106660 0x7f0cf8197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cfe655640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf8198080 con 0x7f0cf8100f60 2024-03-24T10:45:05.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cfe655640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf81981f0 con 0x7f0cf81005a0 2024-03-24T10:45:05.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cfe655640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf81983b0 con 0x7f0cf8106660 2024-03-24T10:45:05.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cf77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf8100f60 0x7f0cf819ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf81005a0 0x7f0cf819d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cfcbf2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cf8106660 0x7f0cf8197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cf77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf8100f60 0x7f0cf819ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52720/0 (socket says 172.21.15.23:52720) 2024-03-24T10:45:05.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf81005a0 0x7f0cf819d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:51014/0 (socket says 172.21.15.23:51014) 2024-03-24T10:45:05.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cf77fe640 1 -- 172.21.15.23:0/2277674608 learned_addr learned my addr 172.21.15.23:0/2277674608 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cf77fe640 1 -- 172.21.15.23:0/2277674608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cf8106660 msgr2=0x7f0cf8197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cf77fe640 1 --2- 172.21.15.23:0/2277674608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cf8106660 0x7f0cf8197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cf77fe640 1 -- 172.21.15.23:0/2277674608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf81005a0 msgr2=0x7f0cf819d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cf77fe640 1 --2- 172.21.15.23:0/2277674608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf81005a0 0x7f0cf819d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cf77fe640 1 -- 172.21.15.23:0/2277674608 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf0009900 con 0x7f0cf8100f60 2024-03-24T10:45:05.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cf7fff640 1 --2- 172.21.15.23:0/2277674608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf81005a0 0x7f0cf819d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:05.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.590+0000 7f0cf77fe640 1 --2- 172.21.15.23:0/2277674608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf8100f60 0x7f0cf819ddc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00e9f0 tx=0x7f0cec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.591+0000 7f0cf57fa640 1 -- 172.21.15.23:0/2277674608 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cec009720 con 0x7f0cf8100f60 2024-03-24T10:45:05.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.591+0000 7f0cf57fa640 1 -- 172.21.15.23:0/2277674608 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cec00cd80 con 0x7f0cf8100f60 2024-03-24T10:45:05.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.591+0000 7f0cfe655640 1 -- 172.21.15.23:0/2277674608 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cf8198690 con 0x7f0cf8100f60 2024-03-24T10:45:05.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.591+0000 7f0cf57fa640 1 -- 172.21.15.23:0/2277674608 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cec010690 con 0x7f0cf8100f60 2024-03-24T10:45:05.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.591+0000 7f0cfe655640 1 -- 172.21.15.23:0/2277674608 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cf81a4920 con 0x7f0cf8100f60 2024-03-24T10:45:05.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.593+0000 7f0cf57fa640 1 -- 172.21.15.23:0/2277674608 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f0cec0046c0 con 0x7f0cf8100f60 2024-03-24T10:45:05.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.593+0000 7f0d01235740 1 Processor -- start 2024-03-24T10:45:05.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.593+0000 7f0d01235740 1 -- start start 2024-03-24T10:45:05.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0d01235740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204f63f0 0x556d204f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0d01235740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d204ff3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cf57fa640 1 --2- 172.21.15.23:0/2277674608 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0ce00a22b0 0x7f0ce00a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0d01235740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204ff8f0 0x556d20501ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0d01235740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556d20464840 con 0x556d204ff8f0 2024-03-24T10:45:05.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0d01235740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556d20461d70 con 0x556d204f6d10 2024-03-24T10:45:05.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0d01235740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556d2045d550 con 0x556d204f63f0 2024-03-24T10:45:05.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d204ff3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cf77fe640 1 --2- 172.21.15.23:0/2277674608 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0ce00a22b0 0x7f0ce00a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cfcbf2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204ff8f0 0x556d20501ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d204ff3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:51026/0 (socket says 172.21.15.23:51026) 2024-03-24T10:45:05.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cf7fff640 1 -- 172.21.15.23:0/2788067161 learned_addr learned my addr 172.21.15.23:0/2788067161 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cfcbf2640 1 --2- 172.21.15.23:0/2788067161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204f63f0 0x556d204f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cf57fa640 1 -- 172.21.15.23:0/2277674608 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0cec014070 con 0x7f0cf8100f60 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cf7fff640 1 -- 172.21.15.23:0/2788067161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204f63f0 msgr2=0x556d204f67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cf7fff640 1 --2- 172.21.15.23:0/2788067161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204f63f0 0x556d204f67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cf7fff640 1 -- 172.21.15.23:0/2788067161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204ff8f0 msgr2=0x556d20501ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cf7fff640 1 --2- 172.21.15.23:0/2788067161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204ff8f0 0x556d20501ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cf7fff640 1 -- 172.21.15.23:0/2788067161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d203f2de0 con 0x556d204f6d10 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cfcbf2640 1 --2- 172.21.15.23:0/2788067161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204f63f0 0x556d204f67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cf77fe640 1 --2- 172.21.15.23:0/2277674608 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0ce00a22b0 0x7f0ce00a4770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0cec097260 tx=0x7f0cec0b1040 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.594+0000 7f0cf7fff640 1 --2- 172.21.15.23:0/2788067161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d204ff3b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80114f0 tx=0x7f0ce80119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=724a752b237c9ce8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.595+0000 7f0cde7fc640 1 -- 172.21.15.23:0/2788067161 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8004030 con 0x556d204f6d10 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.595+0000 7f0cde7fc640 1 -- 172.21.15.23:0/2788067161 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ce80035f0 con 0x556d204f6d10 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.595+0000 7f0cde7fc640 1 -- 172.21.15.23:0/2788067161 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce800f8f0 con 0x556d204f6d10 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.595+0000 7f0d01235740 1 -- 172.21.15.23:0/2788067161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 msgr2=0x556d204ff3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.595+0000 7f0d01235740 1 --2- 172.21.15.23:0/2788067161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d204ff3b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80114f0 tx=0x7f0ce80119b0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.595+0000 7f0d01235740 1 -- 172.21.15.23:0/2788067161 shutdown_connections 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.595+0000 7f0d01235740 1 --2- 172.21.15.23:0/2788067161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204ff8f0 0x556d20501ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.595+0000 7f0d01235740 1 --2- 172.21.15.23:0/2788067161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d204ff3b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.595+0000 7f0d01235740 1 --2- 172.21.15.23:0/2788067161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204f63f0 0x556d204f67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.595+0000 7f0d01235740 1 -- 172.21.15.23:0/2788067161 >> 172.21.15.23:0/2788067161 conn(0x556d204f0b20 msgr2=0x556d204f0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.595+0000 7f0d01235740 1 -- 172.21.15.23:0/2788067161 shutdown_connections 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.595+0000 7f0d01235740 1 -- 172.21.15.23:0/2788067161 wait complete. 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.596+0000 7f0d01235740 1 -- start start 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.596+0000 7f0d01235740 1 Processor -- start 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.596+0000 7f0d01235740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204f63f0 0x556d204fc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.596+0000 7f0d01235740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d204fced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.596+0000 7f0d01235740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204ff8f0 0x556d204fd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.596+0000 7f0d01235740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556d20130430 con 0x556d204ff8f0 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.596+0000 7f0d01235740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556d200ece00 con 0x556d204f6d10 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.596+0000 7f0d01235740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556d203e2590 con 0x556d204f63f0 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.596+0000 7f0d01235740 10 client.0 _ll_register_callbacks cb 0x556d20451fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.596+0000 7f0cfcbf2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204f63f0 0x556d204fc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.596+0000 7f0cf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d204fced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.596+0000 7f0cfcbf2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204ff8f0 0x556d204fd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.596+0000 7f0cf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d204fced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:51038/0 (socket says 172.21.15.23:51038) 2024-03-24T10:45:05.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.596+0000 7f0cf7fff640 1 -- 172.21.15.23:0/428550397 learned_addr learned my addr 172.21.15.23:0/428550397 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0cfcbf2640 1 -- 172.21.15.23:0/428550397 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204f63f0 msgr2=0x556d204fc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0cfcbf2640 1 --2- 172.21.15.23:0/428550397 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204f63f0 0x556d204fc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0cfcbf2640 1 -- 172.21.15.23:0/428550397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 msgr2=0x556d204fced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0cfcbf2640 1 --2- 172.21.15.23:0/428550397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d204fced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0cfcbf2640 1 -- 172.21.15.23:0/428550397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ce8009cc0 con 0x556d204ff8f0 2024-03-24T10:45:05.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0cf7fff640 1 --2- 172.21.15.23:0/428550397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d204fced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:05.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0cfcbf2640 1 --2- 172.21.15.23:0/428550397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204ff8f0 0x556d204fd410 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f0cf002c700 tx=0x7f0cf002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e517fc0b7eec7af4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0cd77fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:05.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0cd77fe640 1 -- 172.21.15.23:0/428550397 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0035ea0 con 0x556d204ff8f0 2024-03-24T10:45:05.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0cd77fe640 1 -- 172.21.15.23:0/428550397 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0cf0032ab0 con 0x556d204ff8f0 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0d01235740 1 -- 172.21.15.23:0/428550397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556d2015c7e0 con 0x556d204ff8f0 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0cd77fe640 1 -- 172.21.15.23:0/428550397 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0cf0003970 con 0x556d204ff8f0 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0d01235740 10 client.16989 fetch_fsmap learned FSMap version 27 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0d01235740 1 -- 172.21.15.23:0/428550397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556d2045bbd0 con 0x556d204ff8f0 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.597+0000 7f0cd77fe640 1 -- 172.21.15.23:0/428550397 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556d2015c7e0 con 0x556d204ff8f0 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.598+0000 7f0cd77fe640 1 -- 172.21.15.23:0/428550397 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f0cf00046b0 con 0x556d204ff8f0 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.598+0000 7f0d01235740 10 client.16989 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.598+0000 7f0d01235740 10 client.16989 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.598+0000 7f0d01235740 20 client.16989 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.598+0000 7f0d01235740 1 --2- 172.21.15.23:0/428550397 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556d2050f9d0 0x556d20511dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.598+0000 7f0d01235740 4 client.16989 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.598+0000 7f0d01235740 1 -- 172.21.15.23:0/428550397 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d20199720 con 0x556d2050f9d0 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.598+0000 7f0cf7fff640 1 --2- 172.21.15.23:0/428550397 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556d2050f9d0 0x556d20511dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.598+0000 7f0cf7fff640 1 --2- 172.21.15.23:0/428550397 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556d2050f9d0 0x556d20511dc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=40bb304a7fd529a server_cookie=24e75f5d10afd7b in_seq=0 out_seq=0 2024-03-24T10:45:05.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.598+0000 7f0cd77fe640 10 client.16989 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:45:05.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.599+0000 7f0cd77fe640 1 -- 172.21.15.23:0/428550397 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556d20199720 con 0x556d2050f9d0 2024-03-24T10:45:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.599+0000 7f0cd77fe640 10 client.16989 handle_command_reply: tid=0 2024-03-24T10:45:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.599+0000 7f0d01235740 1 client.16989 shutdown 2024-03-24T10:45:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.599+0000 7f0d01235740 1 -- 172.21.15.23:0/428550397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204ff8f0 msgr2=0x556d204fd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.599+0000 7f0d01235740 1 --2- 172.21.15.23:0/428550397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204ff8f0 0x556d204fd410 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f0cf002c700 tx=0x7f0cf002e040 comp rx=0 tx=0).stop 2024-03-24T10:45:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.599+0000 7f0d01235740 1 -- 172.21.15.23:0/428550397 shutdown_connections 2024-03-24T10:45:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.599+0000 7f0d01235740 1 --2- 172.21.15.23:0/428550397 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556d2050f9d0 0x556d20511dc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.599+0000 7f0d01235740 1 --2- 172.21.15.23:0/428550397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204ff8f0 0x556d204fd410 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.599+0000 7f0d01235740 1 --2- 172.21.15.23:0/428550397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d204fced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.599+0000 7f0cd77fe640 0 client.16989 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:45:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.599+0000 7f0d01235740 1 --2- 172.21.15.23:0/428550397 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204f63f0 0x556d204fc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.599+0000 7f0d01235740 1 -- 172.21.15.23:0/428550397 >> 172.21.15.23:0/428550397 conn(0x556d204f0b20 msgr2=0x556d204f2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.600+0000 7f0d01235740 1 -- 172.21.15.23:0/428550397 shutdown_connections 2024-03-24T10:45:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.600+0000 7f0d01235740 1 -- 172.21.15.23:0/428550397 wait complete. 2024-03-24T10:45:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.600+0000 7f0d01235740 20 client.16989 trim_cache size 0 max 16384 2024-03-24T10:45:05.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.613+0000 7f0d01235740 1 Processor -- start 2024-03-24T10:45:05.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.613+0000 7f0d01235740 1 -- start start 2024-03-24T10:45:05.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.613+0000 7f0d01235740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204f63f0 0x556d204ff0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.613+0000 7f0d01235740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d2050b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.613+0000 7f0d01235740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204ff8f0 0x556d205176b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.613+0000 7f0d01235740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556d2050c4d0 con 0x556d204f63f0 2024-03-24T10:45:05.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.613+0000 7f0d01235740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556d2050b6c0 con 0x556d204f6d10 2024-03-24T10:45:05.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.613+0000 7f0d01235740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556d204f0650 con 0x556d204ff8f0 2024-03-24T10:45:05.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.614+0000 7f0cfcbf2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204f63f0 0x556d204ff0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.614+0000 7f0cfcbf2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204f63f0 0x556d204ff0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52738/0 (socket says 172.21.15.23:52738) 2024-03-24T10:45:05.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.614+0000 7f0cfcbf2640 1 -- 172.21.15.23:0/1004838890 learned_addr learned my addr 172.21.15.23:0/1004838890 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.614+0000 7f0cf7fff640 1 --2- 172.21.15.23:0/1004838890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d2050b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.614+0000 7f0cfcbf2640 1 --2- 172.21.15.23:0/1004838890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204ff8f0 0x556d205176b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.614+0000 7f0cfcbf2640 1 -- 172.21.15.23:0/1004838890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204ff8f0 msgr2=0x556d205176b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.614+0000 7f0cfcbf2640 1 --2- 172.21.15.23:0/1004838890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204ff8f0 0x556d205176b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.614+0000 7f0cfcbf2640 1 -- 172.21.15.23:0/1004838890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 msgr2=0x556d2050b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.614+0000 7f0cfcbf2640 1 --2- 172.21.15.23:0/1004838890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d2050b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.614+0000 7f0cfcbf2640 1 -- 172.21.15.23:0/1004838890 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d203f7c80 con 0x556d204f63f0 2024-03-24T10:45:05.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.614+0000 7f0cfcbf2640 1 --2- 172.21.15.23:0/1004838890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204f63f0 0x556d204ff0c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0009890 tx=0x7f0cf0009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02cefdd8bbf0424 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.614+0000 7f0cddffb640 1 -- 172.21.15.23:0/1004838890 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0035b10 con 0x556d204f63f0 2024-03-24T10:45:05.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.614+0000 7f0cddffb640 1 -- 172.21.15.23:0/1004838890 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cf0049070 con 0x556d204f63f0 2024-03-24T10:45:05.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.614+0000 7f0cddffb640 1 -- 172.21.15.23:0/1004838890 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0003970 con 0x556d204f63f0 2024-03-24T10:45:05.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.615+0000 7f0d01235740 1 -- 172.21.15.23:0/1004838890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204f63f0 msgr2=0x556d204ff0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.615+0000 7f0d01235740 1 --2- 172.21.15.23:0/1004838890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204f63f0 0x556d204ff0c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0009890 tx=0x7f0cf0009800 comp rx=0 tx=0).stop 2024-03-24T10:45:05.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.615+0000 7f0d01235740 1 -- 172.21.15.23:0/1004838890 shutdown_connections 2024-03-24T10:45:05.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.615+0000 7f0d01235740 1 --2- 172.21.15.23:0/1004838890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204ff8f0 0x556d205176b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.615+0000 7f0d01235740 1 --2- 172.21.15.23:0/1004838890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f6d10 0x556d2050b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.615+0000 7f0d01235740 1 --2- 172.21.15.23:0/1004838890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204f63f0 0x556d204ff0c0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.615+0000 7f0d01235740 1 -- 172.21.15.23:0/1004838890 >> 172.21.15.23:0/1004838890 conn(0x556d204f0b20 msgr2=0x556d204f70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.615+0000 7f0d01235740 1 -- 172.21.15.23:0/1004838890 shutdown_connections 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.615+0000 7f0d01235740 1 -- 172.21.15.23:0/1004838890 wait complete. 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.615+0000 7f0d01235740 1 -- start start 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.615+0000 7f0d01235740 1 Processor -- start 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0d01235740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f63f0 0x556d20507a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0d01235740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204f6d10 0x556d204f9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0d01235740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204ff8f0 0x556d204f9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0d01235740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x556d204f9e60 con 0x556d204f6d10 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0d01235740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556d204f9fd0 con 0x556d204f63f0 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0d01235740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556d20574ce0 con 0x556d204ff8f0 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0d01235740 10 client.0 _ll_register_callbacks cb 0x7f0cec005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cf7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204f6d10 0x556d204f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cf7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204f6d10 0x556d204f9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52742/0 (socket says 172.21.15.23:52742) 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cf7fff640 1 -- 172.21.15.23:0/4173231380 learned_addr learned my addr 172.21.15.23:0/4173231380 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cfcbf2640 1 --2- 172.21.15.23:0/4173231380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f63f0 0x556d20507a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cfcbf2640 1 --2- 172.21.15.23:0/4173231380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204ff8f0 0x556d204f9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cf7fff640 1 -- 172.21.15.23:0/4173231380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204ff8f0 msgr2=0x556d204f9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cf7fff640 1 --2- 172.21.15.23:0/4173231380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204ff8f0 0x556d204f9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cf7fff640 1 -- 172.21.15.23:0/4173231380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f63f0 msgr2=0x556d20507a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cf7fff640 1 --2- 172.21.15.23:0/4173231380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f63f0 0x556d20507a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cf7fff640 1 -- 172.21.15.23:0/4173231380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d2045bbd0 con 0x556d204f6d10 2024-03-24T10:45:05.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cf7fff640 1 --2- 172.21.15.23:0/4173231380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204f6d10 0x556d204f9220 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8014040 tx=0x7f0ce8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136c514a0ae8b2f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cd77fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:05.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cd77fe640 1 -- 172.21.15.23:0/4173231380 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8011040 con 0x556d204f6d10 2024-03-24T10:45:05.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cd77fe640 1 -- 172.21.15.23:0/4173231380 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f0ce80111a0 con 0x556d204f6d10 2024-03-24T10:45:05.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0d01235740 1 -- 172.21.15.23:0/4173231380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ce80035f0 con 0x556d204f6d10 2024-03-24T10:45:05.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0cd77fe640 1 -- 172.21.15.23:0/4173231380 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0ce8011360 con 0x556d204f6d10 2024-03-24T10:45:05.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.616+0000 7f0d01235740 10 client.16995 fetch_fsmap learned FSMap version 27 2024-03-24T10:45:05.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.617+0000 7f0d01235740 1 -- 172.21.15.23:0/4173231380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556d2041d920 con 0x556d204f6d10 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.617+0000 7f0cd77fe640 1 -- 172.21.15.23:0/4173231380 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ce80035f0 con 0x556d204f6d10 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.617+0000 7f0cd77fe640 1 -- 172.21.15.23:0/4173231380 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f0ce8019450 con 0x556d204f6d10 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.617+0000 7f0d01235740 10 client.16995 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.617+0000 7f0d01235740 10 client.16995 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.617+0000 7f0d01235740 20 client.16995 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.617+0000 7f0d01235740 1 --2- 172.21.15.23:0/4173231380 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556d205763b0 0x556d205787d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.617+0000 7f0d01235740 4 client.16995 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.617+0000 7f0d01235740 1 -- 172.21.15.23:0/4173231380 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556d201e8890 con 0x556d205763b0 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.617+0000 7f0cfcbf2640 1 --2- 172.21.15.23:0/4173231380 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556d205763b0 0x556d205787d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.617+0000 7f0cfcbf2640 1 --2- 172.21.15.23:0/4173231380 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556d205763b0 0x556d205787d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d9582b32719ac387 server_cookie=c05b6d426359b0e0 in_seq=0 out_seq=0 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.617+0000 7f0cd77fe640 10 client.16995 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0cd77fe640 1 -- 172.21.15.23:0/4173231380 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556d201e8890 con 0x556d205763b0 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0cd77fe640 10 client.16995 handle_command_reply: tid=0 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0d01235740 1 client.16995 shutdown 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0d01235740 1 -- 172.21.15.23:0/4173231380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204f6d10 msgr2=0x556d204f9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0d01235740 1 --2- 172.21.15.23:0/4173231380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204f6d10 0x556d204f9220 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8014040 tx=0x7f0ce8003b90 comp rx=0 tx=0).stop 2024-03-24T10:45:05.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0d01235740 1 -- 172.21.15.23:0/4173231380 shutdown_connections 2024-03-24T10:45:05.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0d01235740 1 --2- 172.21.15.23:0/4173231380 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x556d205763b0 0x556d205787d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0d01235740 1 --2- 172.21.15.23:0/4173231380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556d204ff8f0 0x556d204f9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0d01235740 1 --2- 172.21.15.23:0/4173231380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d204f6d10 0x556d204f9220 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0cd77fe640 0 client.16995 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:45:05.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0d01235740 1 --2- 172.21.15.23:0/4173231380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556d204f63f0 0x556d20507a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0d01235740 1 -- 172.21.15.23:0/4173231380 >> 172.21.15.23:0/4173231380 conn(0x556d204f0b20 msgr2=0x556d204ef430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0d01235740 1 -- 172.21.15.23:0/4173231380 shutdown_connections 2024-03-24T10:45:05.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0d01235740 1 -- 172.21.15.23:0/4173231380 wait complete. 2024-03-24T10:45:05.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0d01235740 20 client.16995 trim_cache size 0 max 16384 2024-03-24T10:45:05.624 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:45:05.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0cfe655640 1 -- 172.21.15.23:0/2277674608 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0ce00a22b0 msgr2=0x7f0ce00a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0cfe655640 1 --2- 172.21.15.23:0/2277674608 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0ce00a22b0 0x7f0ce00a4770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0cec097260 tx=0x7f0cec0b1040 comp rx=0 tx=0).stop 2024-03-24T10:45:05.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0cfe655640 1 -- 172.21.15.23:0/2277674608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf8100f60 msgr2=0x7f0cf819ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.618+0000 7f0cfe655640 1 --2- 172.21.15.23:0/2277674608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf8100f60 0x7f0cf819ddc0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00e9f0 tx=0x7f0cec00edf0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.619+0000 7f0cfe655640 1 -- 172.21.15.23:0/2277674608 shutdown_connections 2024-03-24T10:45:05.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.619+0000 7f0cfe655640 1 --2- 172.21.15.23:0/2277674608 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0ce00a22b0 0x7f0ce00a4770 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.619+0000 7f0cfe655640 1 --2- 172.21.15.23:0/2277674608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cf8106660 0x7f0cf8197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.619+0000 7f0cfe655640 1 --2- 172.21.15.23:0/2277674608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cf8100f60 0x7f0cf819ddc0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.619+0000 7f0cfe655640 1 --2- 172.21.15.23:0/2277674608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cf81005a0 0x7f0cf819d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.619+0000 7f0cfe655640 1 -- 172.21.15.23:0/2277674608 >> 172.21.15.23:0/2277674608 conn(0x7f0cf8077710 msgr2=0x7f0cf8104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.619+0000 7f0cfe655640 1 -- 172.21.15.23:0/2277674608 shutdown_connections 2024-03-24T10:45:05.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.619+0000 7f0cfe655640 1 -- 172.21.15.23:0/2277674608 wait complete. 2024-03-24T10:45:05.630 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:45:05.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.738+0000 7fda0d0eb640 1 -- 172.21.15.23:0/3407228320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda081005a0 msgr2=0x7fda08100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.738+0000 7fda0d0eb640 1 --2- 172.21.15.23:0/3407228320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda081005a0 0x7fda08100a20 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009900 tx=0x7fd9f002bc30 comp rx=0 tx=0).stop 2024-03-24T10:45:05.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.738+0000 7fda0d0eb640 1 -- 172.21.15.23:0/3407228320 shutdown_connections 2024-03-24T10:45:05.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.738+0000 7fda0d0eb640 1 --2- 172.21.15.23:0/3407228320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda08100f60 0x7fda0810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.738+0000 7fda0d0eb640 1 --2- 172.21.15.23:0/3407228320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda081005a0 0x7fda08100a20 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.738+0000 7fda0d0eb640 1 --2- 172.21.15.23:0/3407228320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda08106660 0x7fda08106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.738+0000 7fda0d0eb640 1 -- 172.21.15.23:0/3407228320 >> 172.21.15.23:0/3407228320 conn(0x7fda08077710 msgr2=0x7fda08077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.738+0000 7fda0d0eb640 1 -- 172.21.15.23:0/3407228320 shutdown_connections 2024-03-24T10:45:05.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.738+0000 7fda0d0eb640 1 -- 172.21.15.23:0/3407228320 wait complete. 2024-03-24T10:45:05.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.739+0000 7fda0d0eb640 1 Processor -- start 2024-03-24T10:45:05.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.739+0000 7fda0d0eb640 1 -- start start 2024-03-24T10:45:05.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0d0eb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda081005a0 0x7fda0819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0d0eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda08100f60 0x7fda0819ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0d0eb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda08106660 0x7fda08196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0d0eb640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fda081970c0 con 0x7fda08106660 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0d0eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fda08197230 con 0x7fda08100f60 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0d0eb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fda081973f0 con 0x7fda081005a0 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0759e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda08106660 0x7fda08196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0759e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda08106660 0x7fda08196a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52752/0 (socket says 172.21.15.23:52752) 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0759e640 1 -- 172.21.15.23:0/2655158501 learned_addr learned my addr 172.21.15.23:0/2655158501 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda06d9d640 1 --2- 172.21.15.23:0/2655158501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda081005a0 0x7fda0819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fd9fffff640 1 --2- 172.21.15.23:0/2655158501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda08100f60 0x7fda0819ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0759e640 1 -- 172.21.15.23:0/2655158501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda081005a0 msgr2=0x7fda0819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0759e640 1 --2- 172.21.15.23:0/2655158501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda081005a0 0x7fda0819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0759e640 1 -- 172.21.15.23:0/2655158501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda08100f60 msgr2=0x7fda0819ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0759e640 1 --2- 172.21.15.23:0/2655158501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda08100f60 0x7fda0819ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0759e640 1 -- 172.21.15.23:0/2655158501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f0009560 con 0x7fda08106660 2024-03-24T10:45:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0759e640 1 --2- 172.21.15.23:0/2655158501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda08106660 0x7fda08196a70 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fda0000eb10 tx=0x7fda0000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda04d99640 1 -- 172.21.15.23:0/2655158501 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00004b20 con 0x7fda08106660 2024-03-24T10:45:05.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0d0eb640 1 -- 172.21.15.23:0/2655158501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda081976d0 con 0x7fda08106660 2024-03-24T10:45:05.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda04d99640 1 -- 172.21.15.23:0/2655158501 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda00004c80 con 0x7fda08106660 2024-03-24T10:45:05.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda04d99640 1 -- 172.21.15.23:0/2655158501 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda00010650 con 0x7fda08106660 2024-03-24T10:45:05.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.740+0000 7fda0d0eb640 1 -- 172.21.15.23:0/2655158501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda081a3960 con 0x7fda08106660 2024-03-24T10:45:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.743+0000 7fda04d99640 1 -- 172.21.15.23:0/2655158501 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fda00005020 con 0x7fda08106660 2024-03-24T10:45:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.743+0000 7fda0fda7740 1 Processor -- start 2024-03-24T10:45:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.743+0000 7fda0fda7740 1 -- start start 2024-03-24T10:45:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fda0fda7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645893f0 0x55b1645897d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fda0fda7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b1645923b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fda0fda7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645928f0 0x55b164594ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fda0fda7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b1644f7840 con 0x55b1645928f0 2024-03-24T10:45:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fda0fda7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b1644f4d70 con 0x55b164589d10 2024-03-24T10:45:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fda0fda7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b1644f0550 con 0x55b1645893f0 2024-03-24T10:45:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fd9fffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645893f0 0x55b1645897d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fda04d99640 1 --2- 172.21.15.23:0/2655158501 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9f40a2230 0x7fd9f40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fd9fffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645893f0 0x55b1645897d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:43386/0 (socket says 172.21.15.23:43386) 2024-03-24T10:45:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fd9fffff640 1 -- 172.21.15.23:0/3300565369 learned_addr learned my addr 172.21.15.23:0/3300565369 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fda06d9d640 1 --2- 172.21.15.23:0/3300565369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b1645923b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fd9fffff640 1 --2- 172.21.15.23:0/3300565369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645928f0 0x55b164594ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fda0759e640 1 --2- 172.21.15.23:0/2655158501 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9f40a2230 0x7fd9f40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fd9fffff640 1 -- 172.21.15.23:0/3300565369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 msgr2=0x55b1645923b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fd9fffff640 1 --2- 172.21.15.23:0/3300565369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b1645923b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fd9fffff640 1 -- 172.21.15.23:0/3300565369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645928f0 msgr2=0x55b164594ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fd9fffff640 1 --2- 172.21.15.23:0/3300565369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645928f0 0x55b164594ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fda04d99640 1 -- 172.21.15.23:0/2655158501 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fda00014070 con 0x7fda08106660 2024-03-24T10:45:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fd9fffff640 1 -- 172.21.15.23:0/3300565369 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b164485de0 con 0x55b1645893f0 2024-03-24T10:45:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.744+0000 7fd9fffff640 1 --2- 172.21.15.23:0/3300565369 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645893f0 0x55b1645897d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009900 tx=0x7fd9f0005e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd0137c601e7e46b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.745+0000 7fda0759e640 1 --2- 172.21.15.23:0/2655158501 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9f40a2230 0x7fd9f40a46f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fda000049d0 tx=0x7fda00007850 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.745+0000 7fd9fd7fa640 1 -- 172.21.15.23:0/3300565369 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f002dea0 con 0x55b1645893f0 2024-03-24T10:45:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.745+0000 7fd9fd7fa640 1 -- 172.21.15.23:0/3300565369 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9f002c5d0 con 0x55b1645893f0 2024-03-24T10:45:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.745+0000 7fd9fd7fa640 1 -- 172.21.15.23:0/3300565369 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0003aa0 con 0x55b1645893f0 2024-03-24T10:45:05.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.745+0000 7fda0fda7740 1 -- 172.21.15.23:0/3300565369 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645893f0 msgr2=0x55b1645897d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.745+0000 7fda0fda7740 1 --2- 172.21.15.23:0/3300565369 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645893f0 0x55b1645897d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009900 tx=0x7fd9f0005e20 comp rx=0 tx=0).stop 2024-03-24T10:45:05.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.745+0000 7fda0fda7740 1 -- 172.21.15.23:0/3300565369 shutdown_connections 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.745+0000 7fda0fda7740 1 --2- 172.21.15.23:0/3300565369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645928f0 0x55b164594ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.745+0000 7fda0fda7740 1 --2- 172.21.15.23:0/3300565369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b1645923b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.745+0000 7fda0fda7740 1 --2- 172.21.15.23:0/3300565369 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645893f0 0x55b1645897d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.745+0000 7fda0fda7740 1 -- 172.21.15.23:0/3300565369 >> 172.21.15.23:0/3300565369 conn(0x55b164583b20 msgr2=0x55b164583f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.745+0000 7fda0fda7740 1 -- 172.21.15.23:0/3300565369 shutdown_connections 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.745+0000 7fda0fda7740 1 -- 172.21.15.23:0/3300565369 wait complete. 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fda0fda7740 1 -- start start 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fda0fda7740 1 Processor -- start 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fda0fda7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645893f0 0x55b164595790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fda0fda7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b164595cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fda0fda7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645928f0 0x55b1645a0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fda0fda7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b1641c3430 con 0x55b1645893f0 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fda0fda7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b16417fe00 con 0x55b164589d10 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fda0fda7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b164475590 con 0x55b1645928f0 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fda0fda7740 10 client.0 _ll_register_callbacks cb 0x55b1644e4fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fd9fffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645893f0 0x55b164595790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fd9fffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645893f0 0x55b164595790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52768/0 (socket says 172.21.15.23:52768) 2024-03-24T10:45:05.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fd9fffff640 1 -- 172.21.15.23:0/1626407898 learned_addr learned my addr 172.21.15.23:0/1626407898 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fda06d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b164595cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fd9fffff640 1 --2- 172.21.15.23:0/1626407898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645928f0 0x55b1645a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.746+0000 7fd9fffff640 1 -- 172.21.15.23:0/1626407898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645928f0 msgr2=0x55b1645a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fd9fffff640 1 --2- 172.21.15.23:0/1626407898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645928f0 0x55b1645a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fd9fffff640 1 -- 172.21.15.23:0/1626407898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 msgr2=0x55b164595cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fd9fffff640 1 --2- 172.21.15.23:0/1626407898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b164595cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fd9fffff640 1 -- 172.21.15.23:0/1626407898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda0810dd20 con 0x55b1645893f0 2024-03-24T10:45:05.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fd9fffff640 1 --2- 172.21.15.23:0/1626407898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645893f0 0x55b164595790 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0036040 tx=0x7fd9f0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59c8e937c3a6ecc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fd9ee7fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:05.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fd9ee7fc640 1 -- 172.21.15.23:0/1626407898 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0005280 con 0x55b1645893f0 2024-03-24T10:45:05.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fd9ee7fc640 1 -- 172.21.15.23:0/1626407898 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fd9f00053e0 con 0x55b1645893f0 2024-03-24T10:45:05.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fda0fda7740 1 -- 172.21.15.23:0/1626407898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b16422c720 con 0x55b1645893f0 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fd9ee7fc640 1 -- 172.21.15.23:0/1626407898 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd9f00055a0 con 0x55b1645893f0 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fda0fda7740 10 client.17007 fetch_fsmap learned FSMap version 27 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fda0fda7740 1 -- 172.21.15.23:0/1626407898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b1644eebd0 con 0x55b1645893f0 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fd9ee7fc640 1 -- 172.21.15.23:0/1626407898 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9f0060020 con 0x55b1645893f0 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fd9ee7fc640 1 -- 172.21.15.23:0/1626407898 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fd9f004b050 con 0x55b1645893f0 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fda0fda7740 10 client.17007 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fda0fda7740 10 client.17007 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fda0fda7740 20 client.17007 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fda0fda7740 1 --2- 172.21.15.23:0/1626407898 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b1645a48d0 0x55b1645a6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fda0fda7740 4 client.17007 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.747+0000 7fda0fda7740 1 -- 172.21.15.23:0/1626407898 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b16427b890 con 0x55b1645a48d0 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.748+0000 7fda06d9d640 1 --2- 172.21.15.23:0/1626407898 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b1645a48d0 0x55b1645a6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.748+0000 7fd9ee7fc640 10 client.17007 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.748+0000 7fda06d9d640 1 --2- 172.21.15.23:0/1626407898 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b1645a48d0 0x55b1645a6cc0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f64b86c1d7d36b8 server_cookie=57633651877ac84b in_seq=0 out_seq=0 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fd9ee7fc640 1 -- 172.21.15.23:0/1626407898 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b16427b890 con 0x55b1645a48d0 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fd9ee7fc640 10 client.17007 handle_command_reply: tid=0 2024-03-24T10:45:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fda0fda7740 1 client.17007 shutdown 2024-03-24T10:45:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fda0fda7740 1 -- 172.21.15.23:0/1626407898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645893f0 msgr2=0x55b164595790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fda0fda7740 1 --2- 172.21.15.23:0/1626407898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645893f0 0x55b164595790 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0036040 tx=0x7fd9f0009260 comp rx=0 tx=0).stop 2024-03-24T10:45:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fda0fda7740 1 -- 172.21.15.23:0/1626407898 shutdown_connections 2024-03-24T10:45:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fda0fda7740 1 --2- 172.21.15.23:0/1626407898 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b1645a48d0 0x55b1645a6cc0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fda0fda7740 1 --2- 172.21.15.23:0/1626407898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645928f0 0x55b1645a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fda0fda7740 1 --2- 172.21.15.23:0/1626407898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b164595cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fd9ee7fc640 0 client.17007 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fda0fda7740 1 --2- 172.21.15.23:0/1626407898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645893f0 0x55b164595790 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fda0fda7740 1 -- 172.21.15.23:0/1626407898 >> 172.21.15.23:0/1626407898 conn(0x55b164583b20 msgr2=0x55b164585870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fda0fda7740 1 -- 172.21.15.23:0/1626407898 shutdown_connections 2024-03-24T10:45:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fda0fda7740 1 -- 172.21.15.23:0/1626407898 wait complete. 2024-03-24T10:45:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.749+0000 7fda0fda7740 20 client.17007 trim_cache size 0 max 16384 2024-03-24T10:45:05.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.764+0000 7fda0fda7740 1 Processor -- start 2024-03-24T10:45:05.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.765+0000 7fda0fda7740 1 -- start start 2024-03-24T10:45:05.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.765+0000 7fda0fda7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645893f0 0x55b1645ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.765+0000 7fda0fda7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b164582b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.765+0000 7fda0fda7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645928f0 0x55b16458ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.765+0000 7fda0fda7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b16458c100 con 0x55b1645893f0 2024-03-24T10:45:05.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.765+0000 7fda0fda7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b164574e20 con 0x55b164589d10 2024-03-24T10:45:05.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.765+0000 7fda0fda7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b1645a7340 con 0x55b1645928f0 2024-03-24T10:45:05.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.765+0000 7fd9fffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645893f0 0x55b1645ac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.765+0000 7fd9fffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645893f0 0x55b1645ac5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52770/0 (socket says 172.21.15.23:52770) 2024-03-24T10:45:05.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.765+0000 7fd9fffff640 1 -- 172.21.15.23:0/451749455 learned_addr learned my addr 172.21.15.23:0/451749455 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.765+0000 7fd9fffff640 1 --2- 172.21.15.23:0/451749455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645928f0 0x55b16458ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.766+0000 7fd9fffff640 1 -- 172.21.15.23:0/451749455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645928f0 msgr2=0x55b16458ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.766+0000 7fd9fffff640 1 --2- 172.21.15.23:0/451749455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645928f0 0x55b16458ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.766+0000 7fd9fffff640 1 -- 172.21.15.23:0/451749455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 msgr2=0x55b164582b40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T10:45:05.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.766+0000 7fd9fffff640 1 --2- 172.21.15.23:0/451749455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b164582b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.766+0000 7fd9fffff640 1 -- 172.21.15.23:0/451749455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b16448ac80 con 0x55b1645893f0 2024-03-24T10:45:05.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.766+0000 7fd9fffff640 1 --2- 172.21.15.23:0/451749455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645893f0 0x55b1645ac5d0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fd9f002c8b0 tx=0x7fd9f0004c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35bd4d3d676c673 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.766+0000 7fd9fcff9640 1 -- 172.21.15.23:0/451749455 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f005cc70 con 0x55b1645893f0 2024-03-24T10:45:05.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.767+0000 7fd9fcff9640 1 -- 172.21.15.23:0/451749455 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9f0060070 con 0x55b1645893f0 2024-03-24T10:45:05.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.767+0000 7fd9fcff9640 1 -- 172.21.15.23:0/451749455 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f00057c0 con 0x55b1645893f0 2024-03-24T10:45:05.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.767+0000 7fda0fda7740 1 -- 172.21.15.23:0/451749455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645893f0 msgr2=0x55b1645ac5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.767+0000 7fda0fda7740 1 --2- 172.21.15.23:0/451749455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645893f0 0x55b1645ac5d0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fd9f002c8b0 tx=0x7fd9f0004c10 comp rx=0 tx=0).stop 2024-03-24T10:45:05.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.768+0000 7fda0fda7740 1 -- 172.21.15.23:0/451749455 shutdown_connections 2024-03-24T10:45:05.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.768+0000 7fda0fda7740 1 --2- 172.21.15.23:0/451749455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645928f0 0x55b16458ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.768+0000 7fda0fda7740 1 --2- 172.21.15.23:0/451749455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b164582b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.768+0000 7fda0fda7740 1 --2- 172.21.15.23:0/451749455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645893f0 0x55b1645ac5d0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.768+0000 7fda0fda7740 1 -- 172.21.15.23:0/451749455 >> 172.21.15.23:0/451749455 conn(0x55b164583b20 msgr2=0x55b164582430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.768+0000 7fda0fda7740 1 -- 172.21.15.23:0/451749455 shutdown_connections 2024-03-24T10:45:05.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.768+0000 7fda0fda7740 1 -- 172.21.15.23:0/451749455 wait complete. 2024-03-24T10:45:05.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.768+0000 7fda0fda7740 1 -- start start 2024-03-24T10:45:05.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.768+0000 7fda0fda7740 1 Processor -- start 2024-03-24T10:45:05.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.769+0000 7fda0fda7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645893f0 0x55b164600fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.769+0000 7fda0fda7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b164601500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.769+0000 7fda0fda7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645928f0 0x55b164601a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.769+0000 7fda0fda7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b164604e40 con 0x55b1645928f0 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.769+0000 7fda0fda7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b16459a630 con 0x55b164589d10 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.769+0000 7fda0fda7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b16459a7a0 con 0x55b1645893f0 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.769+0000 7fda0fda7740 10 client.0 _ll_register_callbacks cb 0x7fd9f0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.769+0000 7fd9fffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645928f0 0x55b164601a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.769+0000 7fd9fffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645928f0 0x55b164601a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52774/0 (socket says 172.21.15.23:52774) 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.769+0000 7fd9fffff640 1 -- 172.21.15.23:0/1971886396 learned_addr learned my addr 172.21.15.23:0/1971886396 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.770+0000 7fd9fffff640 1 --2- 172.21.15.23:0/1971886396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645893f0 0x55b164600fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.770+0000 7fda06d9d640 1 --2- 172.21.15.23:0/1971886396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b164601500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.770+0000 7fd9fffff640 1 -- 172.21.15.23:0/1971886396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645893f0 msgr2=0x55b164600fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.770+0000 7fd9fffff640 1 --2- 172.21.15.23:0/1971886396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645893f0 0x55b164600fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.770+0000 7fd9fffff640 1 -- 172.21.15.23:0/1971886396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 msgr2=0x55b164601500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.770+0000 7fd9fffff640 1 --2- 172.21.15.23:0/1971886396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b164601500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.770+0000 7fd9fffff640 1 -- 172.21.15.23:0/1971886396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1644eebd0 con 0x55b1645928f0 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.770+0000 7fd9fffff640 1 --2- 172.21.15.23:0/1971886396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645928f0 0x55b164601a40 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0045040 tx=0x7fd9f002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e93699c45a7fdc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.770+0000 7fd9ee7fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:05.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.770+0000 7fd9ee7fc640 1 -- 172.21.15.23:0/1971886396 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0005490 con 0x55b1645928f0 2024-03-24T10:45:05.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.770+0000 7fda0fda7740 1 -- 172.21.15.23:0/1971886396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b1641ef7e0 con 0x55b1645928f0 2024-03-24T10:45:05.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.770+0000 7fd9ee7fc640 1 -- 172.21.15.23:0/1971886396 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7fd9f005ce40 con 0x55b1645928f0 2024-03-24T10:45:05.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.771+0000 7fd9ee7fc640 1 -- 172.21.15.23:0/1971886396 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd9f0004740 con 0x55b1645928f0 2024-03-24T10:45:05.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.771+0000 7fda0fda7740 10 client.17013 fetch_fsmap learned FSMap version 27 2024-03-24T10:45:05.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.771+0000 7fda0fda7740 1 -- 172.21.15.23:0/1971886396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b1644b0920 con 0x55b1645928f0 2024-03-24T10:45:05.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.771+0000 7fd9ee7fc640 1 -- 172.21.15.23:0/1971886396 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b1641ef7e0 con 0x55b1645928f0 2024-03-24T10:45:05.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.771+0000 7fd9ee7fc640 1 -- 172.21.15.23:0/1971886396 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7fd9f0063070 con 0x55b1645928f0 2024-03-24T10:45:05.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.771+0000 7fda0fda7740 10 client.17013 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:45:05.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.771+0000 7fda0fda7740 10 client.17013 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:05.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.772+0000 7fda0fda7740 20 client.17013 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:05.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.772+0000 7fda0fda7740 1 --2- 172.21.15.23:0/1971886396 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b16459bae0 0x55b164624810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.772+0000 7fda0fda7740 4 client.17013 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:45:05.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.772+0000 7fda0fda7740 1 -- 172.21.15.23:0/1971886396 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b16419b700 con 0x55b16459bae0 2024-03-24T10:45:05.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.772+0000 7fda06d9d640 1 --2- 172.21.15.23:0/1971886396 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b16459bae0 0x55b164624810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.773+0000 7fd9ee7fc640 10 client.17013 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:05.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.773+0000 7fda06d9d640 1 --2- 172.21.15.23:0/1971886396 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b16459bae0 0x55b164624810 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95ed271631903556 server_cookie=e3517558e49bf063 in_seq=0 out_seq=0 2024-03-24T10:45:05.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.773+0000 7fd9ee7fc640 1 -- 172.21.15.23:0/1971886396 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b16419b700 con 0x55b16459bae0 2024-03-24T10:45:05.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.773+0000 7fd9ee7fc640 10 client.17013 handle_command_reply: tid=0 2024-03-24T10:45:05.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.773+0000 7fda0fda7740 1 client.17013 shutdown 2024-03-24T10:45:05.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.773+0000 7fda0fda7740 1 -- 172.21.15.23:0/1971886396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645928f0 msgr2=0x55b164601a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.773+0000 7fda0fda7740 1 --2- 172.21.15.23:0/1971886396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645928f0 0x55b164601a40 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0045040 tx=0x7fd9f002bc30 comp rx=0 tx=0).stop 2024-03-24T10:45:05.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0fda7740 1 -- 172.21.15.23:0/1971886396 shutdown_connections 2024-03-24T10:45:05.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0fda7740 1 --2- 172.21.15.23:0/1971886396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1645928f0 0x55b164601a40 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0fda7740 1 --2- 172.21.15.23:0/1971886396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b164589d10 0x55b164601500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0fda7740 1 --2- 172.21.15.23:0/1971886396 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b16459bae0 0x55b164624810 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0fda7740 1 --2- 172.21.15.23:0/1971886396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1645893f0 0x55b164600fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0fda7740 1 -- 172.21.15.23:0/1971886396 >> 172.21.15.23:0/1971886396 conn(0x55b164583b20 msgr2=0x55b1645a08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fd9ee7fc640 0 client.17013 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:05.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0fda7740 1 -- 172.21.15.23:0/1971886396 shutdown_connections 2024-03-24T10:45:05.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0fda7740 1 -- 172.21.15.23:0/1971886396 wait complete. 2024-03-24T10:45:05.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0fda7740 20 client.17013 trim_cache size 0 max 16384 2024-03-24T10:45:05.776 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:45:05.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0d0eb640 1 -- 172.21.15.23:0/2655158501 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9f40a2230 msgr2=0x7fd9f40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0d0eb640 1 --2- 172.21.15.23:0/2655158501 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9f40a2230 0x7fd9f40a46f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fda000049d0 tx=0x7fda00007850 comp rx=0 tx=0).stop 2024-03-24T10:45:05.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0d0eb640 1 -- 172.21.15.23:0/2655158501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda08106660 msgr2=0x7fda08196a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0d0eb640 1 --2- 172.21.15.23:0/2655158501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda08106660 0x7fda08196a70 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fda0000eb10 tx=0x7fda0000c2d0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0d0eb640 1 -- 172.21.15.23:0/2655158501 shutdown_connections 2024-03-24T10:45:05.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0d0eb640 1 --2- 172.21.15.23:0/2655158501 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd9f40a2230 0x7fd9f40a46f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0d0eb640 1 --2- 172.21.15.23:0/2655158501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda08106660 0x7fda08196a70 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0d0eb640 1 --2- 172.21.15.23:0/2655158501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda08100f60 0x7fda0819ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.774+0000 7fda0d0eb640 1 --2- 172.21.15.23:0/2655158501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda081005a0 0x7fda0819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.775+0000 7fda0d0eb640 1 -- 172.21.15.23:0/2655158501 >> 172.21.15.23:0/2655158501 conn(0x7fda08077710 msgr2=0x7fda080ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.775+0000 7fda0d0eb640 1 -- 172.21.15.23:0/2655158501 shutdown_connections 2024-03-24T10:45:05.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:05.775+0000 7fda0d0eb640 1 -- 172.21.15.23:0/2655158501 wait complete. 2024-03-24T10:45:05.884 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2919391037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:45:05.884 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: *** Caught signal (Segmentation fault) ** 2024-03-24T10:45:05.885 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: in thread 7f562d490700 thread_name:md_log_replay 2024-03-24T10:45:05.885 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev) 2024-03-24T10:45:05.885 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f563e851d20] 2024-03-24T10:45:05.885 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 2: /usr/bin/ceph-mds(+0x5487ef) [0x55ac4ea457ef] 2024-03-24T10:45:05.885 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55ac4ea49233] 2024-03-24T10:45:05.885 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 4: (MDLog::_replay_thread()+0x617) [0x55ac4eaa58f7] 2024-03-24T10:45:05.885 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55ac4eaac041] 2024-03-24T10:45:05.885 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f563e8471ca] 2024-03-24T10:45:05.885 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 7: clone() 2024-03-24T10:45:05.885 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 2024-03-24T10:45:05.503+0000 7f562d490700 -1 *** Caught signal (Segmentation fault) ** 2024-03-24T10:45:05.885 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: in thread 7f562d490700 thread_name:md_log_replay 2024-03-24T10:45:05.885 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 2024-03-24T10:45:05.885 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev) 2024-03-24T10:45:05.885 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f563e851d20] 2024-03-24T10:45:05.886 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 2: /usr/bin/ceph-mds(+0x5487ef) [0x55ac4ea457ef] 2024-03-24T10:45:05.886 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55ac4ea49233] 2024-03-24T10:45:05.886 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 4: (MDLog::_replay_thread()+0x617) [0x55ac4eaa58f7] 2024-03-24T10:45:05.886 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55ac4eaac041] 2024-03-24T10:45:05.886 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f563e8471ca] 2024-03-24T10:45:05.886 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 7: clone() 2024-03-24T10:45:05.886 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T10:45:05.886 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 2024-03-24T10:45:05.886 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 0> 2024-03-24T10:45:05.503+0000 7f562d490700 -1 *** Caught signal (Segmentation fault) ** 2024-03-24T10:45:05.886 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: in thread 7f562d490700 thread_name:md_log_replay 2024-03-24T10:45:05.886 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 2024-03-24T10:45:05.886 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev) 2024-03-24T10:45:05.886 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f563e851d20] 2024-03-24T10:45:05.886 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 2: /usr/bin/ceph-mds(+0x5487ef) [0x55ac4ea457ef] 2024-03-24T10:45:05.887 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55ac4ea49233] 2024-03-24T10:45:05.887 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 4: (MDLog::_replay_thread()+0x617) [0x55ac4eaa58f7] 2024-03-24T10:45:05.887 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55ac4eaac041] 2024-03-24T10:45:05.887 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f563e8471ca] 2024-03-24T10:45:05.887 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 7: clone() 2024-03-24T10:45:05.887 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T10:45:05.887 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 2024-03-24T10:45:05.887 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 0> 2024-03-24T10:45:05.503+0000 7f562d490700 -1 *** Caught signal (Segmentation fault) ** 2024-03-24T10:45:05.887 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: in thread 7f562d490700 thread_name:md_log_replay 2024-03-24T10:45:05.887 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 2024-03-24T10:45:05.887 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev) 2024-03-24T10:45:05.887 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f563e851d20] 2024-03-24T10:45:05.888 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 2: /usr/bin/ceph-mds(+0x5487ef) [0x55ac4ea457ef] 2024-03-24T10:45:05.888 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55ac4ea49233] 2024-03-24T10:45:05.888 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 4: (MDLog::_replay_thread()+0x617) [0x55ac4eaa58f7] 2024-03-24T10:45:05.888 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55ac4eaac041] 2024-03-24T10:45:05.888 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f563e8471ca] 2024-03-24T10:45:05.888 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 7: clone() 2024-03-24T10:45:05.888 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T10:45:05.888 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[67121]: 2024-03-24T10:45:05.888 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:05 smithi117 podman[71003]: 2024-03-24 10:45:05.568726876 +0000 UTC m=+0.027802489 container died 9797d79d19738e49e516851e7d757974cf46528a64e728a4fda9afd1c6c8e9f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:45:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:05 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2919391037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:45:06.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:05 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2919391037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:45:06.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 456640 11.85 MB/sec execute 83 sec latency 16.985 ms 2024-03-24T10:45:06.533 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:06 smithi117 podman[71003]: 2024-03-24 10:45:06.277908805 +0000 UTC m=+0.736984392 container remove 9797d79d19738e49e516851e7d757974cf46528a64e728a4fda9afd1c6c8e9f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:45:06.533 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:06 smithi117 systemd[1]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.c.service: Main process exited, code=exited, status=139/n/a 2024-03-24T10:45:06.790 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:45:06.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:45:06.790 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:45:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.901+0000 7f55d155c640 1 -- 172.21.15.23:0/3113928279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55cc107760 msgr2=0x7f55cc107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.901+0000 7f55d155c640 1 --2- 172.21.15.23:0/3113928279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55cc107760 0x7f55cc107b40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f55bc009890 tx=0x7f55bc02bb70 comp rx=0 tx=0).stop 2024-03-24T10:45:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.901+0000 7f55d155c640 1 -- 172.21.15.23:0/3113928279 shutdown_connections 2024-03-24T10:45:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.901+0000 7f55d155c640 1 --2- 172.21.15.23:0/3113928279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55cc0692d0 0x7f55cc10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.901+0000 7f55d155c640 1 --2- 172.21.15.23:0/3113928279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc068910 0x7f55cc068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.901+0000 7f55d155c640 1 --2- 172.21.15.23:0/3113928279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55cc107760 0x7f55cc107b40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.901+0000 7f55d155c640 1 -- 172.21.15.23:0/3113928279 >> 172.21.15.23:0/3113928279 conn(0x7f55cc077710 msgr2=0x7f55cc077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.901+0000 7f55d155c640 1 -- 172.21.15.23:0/3113928279 shutdown_connections 2024-03-24T10:45:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.901+0000 7f55d155c640 1 -- 172.21.15.23:0/3113928279 wait complete. 2024-03-24T10:45:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.901+0000 7f55d155c640 1 Processor -- start 2024-03-24T10:45:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55d155c640 1 -- start start 2024-03-24T10:45:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55d155c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55cc068910 0x7f55cc197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55d155c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc0692d0 0x7f55cc1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55d155c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55cc107760 0x7f55cc19c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55d155c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc19cbe0 con 0x7f55cc068910 2024-03-24T10:45:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55d155c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc19cd50 con 0x7f55cc0692d0 2024-03-24T10:45:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55d155c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc19cf10 con 0x7f55cc107760 2024-03-24T10:45:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55caffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55cc068910 0x7f55cc197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55caffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55cc068910 0x7f55cc197d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52802/0 (socket says 172.21.15.23:52802) 2024-03-24T10:45:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55caffd640 1 -- 172.21.15.23:0/3223754660 learned_addr learned my addr 172.21.15.23:0/3223754660 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55c3fff640 1 --2- 172.21.15.23:0/3223754660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc0692d0 0x7f55cc1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55cb7fe640 1 --2- 172.21.15.23:0/3223754660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55cc107760 0x7f55cc19c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55caffd640 1 -- 172.21.15.23:0/3223754660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55cc107760 msgr2=0x7f55cc19c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55caffd640 1 --2- 172.21.15.23:0/3223754660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55cc107760 0x7f55cc19c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55caffd640 1 -- 172.21.15.23:0/3223754660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc0692d0 msgr2=0x7f55cc1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55caffd640 1 --2- 172.21.15.23:0/3223754660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc0692d0 0x7f55cc1982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55caffd640 1 -- 172.21.15.23:0/3223754660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c4009900 con 0x7f55cc068910 2024-03-24T10:45:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.902+0000 7f55caffd640 1 --2- 172.21.15.23:0/3223754660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55cc068910 0x7f55cc197d70 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f55bc009860 tx=0x7f55bc004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.903+0000 7f55c8ff9640 1 -- 172.21.15.23:0/3223754660 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55bc034950 con 0x7f55cc068910 2024-03-24T10:45:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.903+0000 7f55c8ff9640 1 -- 172.21.15.23:0/3223754660 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55bc02c9f0 con 0x7f55cc068910 2024-03-24T10:45:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.903+0000 7f55d155c640 1 -- 172.21.15.23:0/3223754660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55bc009490 con 0x7f55cc068910 2024-03-24T10:45:06.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.903+0000 7f55c8ff9640 1 -- 172.21.15.23:0/3223754660 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55bc0035f0 con 0x7f55cc068910 2024-03-24T10:45:06.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.903+0000 7f55d155c640 1 -- 172.21.15.23:0/3223754660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55cc100500 con 0x7f55cc068910 2024-03-24T10:45:06.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.905+0000 7f55c8ff9640 1 -- 172.21.15.23:0/3223754660 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f55bc03d620 con 0x7f55cc068910 2024-03-24T10:45:06.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.905+0000 7f55d4210740 1 Processor -- start 2024-03-24T10:45:06.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55d4210740 1 -- start start 2024-03-24T10:45:06.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55d4210740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ee760 0x55682f2eeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55d4210740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ef080 0x55682f2f7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55d4210740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 0x55682f2fa050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55d4210740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55682f25c530 con 0x55682f2ee760 2024-03-24T10:45:06.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55d4210740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55682f259a60 con 0x55682f2f7c60 2024-03-24T10:45:06.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55d4210740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55682f255240 con 0x55682f2ef080 2024-03-24T10:45:06.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55cb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ee760 0x55682f2eeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55cb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ee760 0x55682f2eeb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52806/0 (socket says 172.21.15.23:52806) 2024-03-24T10:45:06.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55cb7fe640 1 -- 172.21.15.23:0/947164177 learned_addr learned my addr 172.21.15.23:0/947164177 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:06.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55c3fff640 1 --2- 172.21.15.23:0/947164177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 0x55682f2fa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55caffd640 1 --2- 172.21.15.23:0/947164177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ef080 0x55682f2f7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55c8ff9640 1 --2- 172.21.15.23:0/3223754660 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f55b80a2230 0x7f55b80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55c3fff640 1 -- 172.21.15.23:0/947164177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ef080 msgr2=0x55682f2f7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:06.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55c3fff640 1 --2- 172.21.15.23:0/947164177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ef080 0x55682f2f7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55c3fff640 1 -- 172.21.15.23:0/947164177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ee760 msgr2=0x55682f2eeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:06.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55c3fff640 1 --2- 172.21.15.23:0/947164177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ee760 0x55682f2eeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55c3fff640 1 -- 172.21.15.23:0/947164177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55682f2538c0 con 0x55682f2f7c60 2024-03-24T10:45:06.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55cb7fe640 1 --2- 172.21.15.23:0/947164177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ee760 0x55682f2eeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:06.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.906+0000 7f55cb7fe640 1 --2- 172.21.15.23:0/3223754660 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f55b80a2230 0x7f55b80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.907+0000 7f55c8ff9640 1 -- 172.21.15.23:0/3223754660 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f55bc0f3bf0 con 0x7f55cc068910 2024-03-24T10:45:06.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.907+0000 7f55c3fff640 1 --2- 172.21.15.23:0/947164177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 0x55682f2fa050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f55b4009bf0 tx=0x7f55b4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=575786a58934fc87 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:06.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.907+0000 7f55c17fa640 1 -- 172.21.15.23:0/947164177 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4004050 con 0x55682f2f7c60 2024-03-24T10:45:06.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.907+0000 7f55cb7fe640 1 --2- 172.21.15.23:0/3223754660 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f55b80a2230 0x7f55b80a46f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f55c4005c40 tx=0x7f55c4004a40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:06.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.907+0000 7f55c17fa640 1 -- 172.21.15.23:0/947164177 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55b4005500 con 0x55682f2f7c60 2024-03-24T10:45:06.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.907+0000 7f55c17fa640 1 -- 172.21.15.23:0/947164177 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4004cd0 con 0x55682f2f7c60 2024-03-24T10:45:06.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 -- 172.21.15.23:0/947164177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 msgr2=0x55682f2fa050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:06.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 --2- 172.21.15.23:0/947164177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 0x55682f2fa050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f55b4009bf0 tx=0x7f55b4011810 comp rx=0 tx=0).stop 2024-03-24T10:45:06.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 -- 172.21.15.23:0/947164177 shutdown_connections 2024-03-24T10:45:06.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 --2- 172.21.15.23:0/947164177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 0x55682f2fa050 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 --2- 172.21.15.23:0/947164177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ef080 0x55682f2f7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 --2- 172.21.15.23:0/947164177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ee760 0x55682f2eeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 -- 172.21.15.23:0/947164177 >> 172.21.15.23:0/947164177 conn(0x55682f2e8e90 msgr2=0x55682f2e9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:06.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 -- 172.21.15.23:0/947164177 shutdown_connections 2024-03-24T10:45:06.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 -- 172.21.15.23:0/947164177 wait complete. 2024-03-24T10:45:06.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 -- start start 2024-03-24T10:45:06.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 Processor -- start 2024-03-24T10:45:06.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ee760 0x55682f2f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f2f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 0x55682f2f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55682f1f9f80 con 0x55682f2ef080 2024-03-24T10:45:06.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55682ef284b0 con 0x55682f2f7c60 2024-03-24T10:45:06.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55682f1daad0 con 0x55682f2ee760 2024-03-24T10:45:06.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55d4210740 10 client.0 _ll_register_callbacks cb 0x55682f20db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:06.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55c3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f2f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.908+0000 7f55c3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f2f51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52816/0 (socket says 172.21.15.23:52816) 2024-03-24T10:45:06.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55c3fff640 1 -- 172.21.15.23:0/715607232 learned_addr learned my addr 172.21.15.23:0/715607232 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:06.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55caffd640 1 --2- 172.21.15.23:0/715607232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ee760 0x55682f2f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55cb7fe640 1 --2- 172.21.15.23:0/715607232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 0x55682f2f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55c3fff640 1 -- 172.21.15.23:0/715607232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ee760 msgr2=0x55682f2f4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55c3fff640 1 --2- 172.21.15.23:0/715607232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ee760 0x55682f2f4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55c3fff640 1 -- 172.21.15.23:0/715607232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 msgr2=0x55682f2f56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55c3fff640 1 --2- 172.21.15.23:0/715607232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 0x55682f2f56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55c3fff640 1 -- 172.21.15.23:0/715607232 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55b4009850 con 0x55682f2ef080 2024-03-24T10:45:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55c3fff640 1 --2- 172.21.15.23:0/715607232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f2f51b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f55b401b040 tx=0x7f55b4009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9218a8afd9753447 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55b27fc640 1 -- 172.21.15.23:0/715607232 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4003a00 con 0x55682f2ef080 2024-03-24T10:45:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55b27fc640 1 -- 172.21.15.23:0/715607232 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f55b40046c0 con 0x55682f2ef080 2024-03-24T10:45:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55d4210740 1 -- 172.21.15.23:0/715607232 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55682f2797a0 con 0x55682f2ef080 2024-03-24T10:45:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55b27fc640 1 -- 172.21.15.23:0/715607232 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f55b4003b60 con 0x55682f2ef080 2024-03-24T10:45:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55d4210740 10 client.17025 fetch_fsmap learned FSMap version 27 2024-03-24T10:45:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55d4210740 1 -- 172.21.15.23:0/715607232 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55682f20b6d0 con 0x55682f2ef080 2024-03-24T10:45:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.909+0000 7f55b27fc640 1 -- 172.21.15.23:0/715607232 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55682f2797a0 con 0x55682f2ef080 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.910+0000 7f55b27fc640 1 -- 172.21.15.23:0/715607232 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f55b4021040 con 0x55682f2ef080 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.910+0000 7f55d4210740 10 client.17025 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.910+0000 7f55d4210740 10 client.17025 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.910+0000 7f55d4210740 20 client.17025 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.910+0000 7f55d4210740 1 --2- 172.21.15.23:0/715607232 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55682f307cb0 0x55682f30a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.910+0000 7f55d4210740 4 client.17025 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.910+0000 7f55d4210740 1 -- 172.21.15.23:0/715607232 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55682f24b1f0 con 0x55682f307cb0 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.910+0000 7f55caffd640 1 --2- 172.21.15.23:0/715607232 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55682f307cb0 0x55682f30a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.910+0000 7f55caffd640 1 --2- 172.21.15.23:0/715607232 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55682f307cb0 0x55682f30a0d0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afd1893a0bcd1a4d server_cookie=7cdcee1ea5b87a49 in_seq=0 out_seq=0 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.910+0000 7f55b27fc640 10 client.17025 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55b27fc640 1 -- 172.21.15.23:0/715607232 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55682f24b1f0 con 0x55682f307cb0 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55b27fc640 10 client.17025 handle_command_reply: tid=0 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55d4210740 1 client.17025 shutdown 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55d4210740 1 -- 172.21.15.23:0/715607232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 msgr2=0x55682f2f51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:06.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55d4210740 1 --2- 172.21.15.23:0/715607232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f2f51b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f55b401b040 tx=0x7f55b4009fd0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55d4210740 1 -- 172.21.15.23:0/715607232 shutdown_connections 2024-03-24T10:45:06.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55d4210740 1 --2- 172.21.15.23:0/715607232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 0x55682f2f56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55d4210740 1 --2- 172.21.15.23:0/715607232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f2f51b0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55d4210740 1 --2- 172.21.15.23:0/715607232 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55682f307cb0 0x55682f30a0d0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55d4210740 1 --2- 172.21.15.23:0/715607232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ee760 0x55682f2f4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55b27fc640 0 client.17025 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:06.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55d4210740 1 -- 172.21.15.23:0/715607232 >> 172.21.15.23:0/715607232 conn(0x55682f2e8e90 msgr2=0x55682f2eabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:06.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55d4210740 1 -- 172.21.15.23:0/715607232 shutdown_connections 2024-03-24T10:45:06.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55d4210740 1 -- 172.21.15.23:0/715607232 wait complete. 2024-03-24T10:45:06.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.911+0000 7f55d4210740 20 client.17025 trim_cache size 0 max 16384 2024-03-24T10:45:06.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55d4210740 1 Processor -- start 2024-03-24T10:45:06.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55d4210740 1 -- start start 2024-03-24T10:45:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55d4210740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ee760 0x55682f2f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55d4210740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f3034f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55d4210740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 0x55682f30f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55d4210740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55682f3047f0 con 0x55682f2ef080 2024-03-24T10:45:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55d4210740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55682f3079e0 con 0x55682f2f7c60 2024-03-24T10:45:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55d4210740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55682f2e4620 con 0x55682f2ee760 2024-03-24T10:45:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55c3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f3034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55c3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f3034f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52826/0 (socket says 172.21.15.23:52826) 2024-03-24T10:45:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55caffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ee760 0x55682f2f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55c3fff640 1 -- 172.21.15.23:0/1169095841 learned_addr learned my addr 172.21.15.23:0/1169095841 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55cb7fe640 1 --2- 172.21.15.23:0/1169095841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 0x55682f30f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55c3fff640 1 -- 172.21.15.23:0/1169095841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ee760 msgr2=0x55682f2f73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55c3fff640 1 --2- 172.21.15.23:0/1169095841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ee760 0x55682f2f73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55c3fff640 1 -- 172.21.15.23:0/1169095841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 msgr2=0x55682f30f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55c3fff640 1 --2- 172.21.15.23:0/1169095841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 0x55682f30f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55c3fff640 1 -- 172.21.15.23:0/1169095841 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55682f1efc60 con 0x55682f2ef080 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.930+0000 7f55cb7fe640 1 --2- 172.21.15.23:0/1169095841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 0x55682f30f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.931+0000 7f55c3fff640 1 --2- 172.21.15.23:0/1169095841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f3034f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f55b4005f10 tx=0x7f55b4005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591edeff0f0a54b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.931+0000 7f55c0ff9640 1 -- 172.21.15.23:0/1169095841 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4023870 con 0x55682f2ef080 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.931+0000 7f55c0ff9640 1 -- 172.21.15.23:0/1169095841 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55b4023a60 con 0x55682f2ef080 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.931+0000 7f55c0ff9640 1 -- 172.21.15.23:0/1169095841 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4003c90 con 0x55682f2ef080 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.931+0000 7f55d4210740 1 -- 172.21.15.23:0/1169095841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 msgr2=0x55682f3034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.931+0000 7f55d4210740 1 --2- 172.21.15.23:0/1169095841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f3034f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f55b4005f10 tx=0x7f55b4005f40 comp rx=0 tx=0).stop 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.931+0000 7f55d4210740 1 -- 172.21.15.23:0/1169095841 shutdown_connections 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.931+0000 7f55d4210740 1 --2- 172.21.15.23:0/1169095841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2f7c60 0x55682f30f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.931+0000 7f55d4210740 1 --2- 172.21.15.23:0/1169095841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f3034f0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.931+0000 7f55d4210740 1 --2- 172.21.15.23:0/1169095841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2ee760 0x55682f2f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.931+0000 7f55d4210740 1 -- 172.21.15.23:0/1169095841 >> 172.21.15.23:0/1169095841 conn(0x55682f2e8e90 msgr2=0x55682f2eab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55d4210740 1 -- 172.21.15.23:0/1169095841 shutdown_connections 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55d4210740 1 -- 172.21.15.23:0/1169095841 wait complete. 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55d4210740 1 -- start start 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55d4210740 1 Processor -- start 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55d4210740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2ee760 0x55682f2ffd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55d4210740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f2fc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55d4210740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2f7c60 0x55682f2fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55d4210740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55682f2fcd30 con 0x55682f2ef080 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55d4210740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55682f2fcea0 con 0x55682f2ee760 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55d4210740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55682f36e020 con 0x55682f2f7c60 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55d4210740 10 client.0 _ll_register_callbacks cb 0x7f55b4009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55c3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f2fc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55c3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f2fc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52838/0 (socket says 172.21.15.23:52838) 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55c3fff640 1 -- 172.21.15.23:0/128043964 learned_addr learned my addr 172.21.15.23:0/128043964 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55caffd640 1 --2- 172.21.15.23:0/128043964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2ee760 0x55682f2ffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55cb7fe640 1 --2- 172.21.15.23:0/128043964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2f7c60 0x55682f2fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55c3fff640 1 -- 172.21.15.23:0/128043964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2f7c60 msgr2=0x55682f2fc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55c3fff640 1 --2- 172.21.15.23:0/128043964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2f7c60 0x55682f2fc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55c3fff640 1 -- 172.21.15.23:0/128043964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2ee760 msgr2=0x55682f2ffd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55c3fff640 1 --2- 172.21.15.23:0/128043964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2ee760 0x55682f2ffd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.932+0000 7f55c3fff640 1 -- 172.21.15.23:0/128043964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55682f20b6d0 con 0x55682f2ef080 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55c3fff640 1 --2- 172.21.15.23:0/128043964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f2fc0f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f55b4005f10 tx=0x7f55b4004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1877bf9259d0444 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55b27fc640 1 -- 172.21.15.23:0/128043964 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b40045d0 con 0x55682f2ef080 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55b27fc640 1 -- 172.21.15.23:0/128043964 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f55b4000e40 con 0x55682f2ef080 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55d4210740 1 -- 172.21.15.23:0/128043964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55682f2f4aa0 con 0x55682f2ef080 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55b27fc640 1 -- 172.21.15.23:0/128043964 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f55b4004730 con 0x55682f2ef080 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55d4210740 10 client.17031 fetch_fsmap learned FSMap version 27 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55d4210740 1 -- 172.21.15.23:0/128043964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55682f215970 con 0x55682f2ef080 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55b27fc640 1 -- 172.21.15.23:0/128043964 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55b4046080 con 0x55682f2ef080 2024-03-24T10:45:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55b27fc640 1 -- 172.21.15.23:0/128043964 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f55b4040030 con 0x55682f2ef080 2024-03-24T10:45:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55d4210740 10 client.17031 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:45:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55d4210740 10 client.17031 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55d4210740 20 client.17031 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55d4210740 1 --2- 172.21.15.23:0/128043964 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55682f36f3a0 0x55682f3717c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55d4210740 4 client.17031 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:45:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.933+0000 7f55d4210740 1 -- 172.21.15.23:0/128043964 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f55b4005500 con 0x55682f36f3a0 2024-03-24T10:45:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.934+0000 7f55caffd640 1 --2- 172.21.15.23:0/128043964 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55682f36f3a0 0x55682f3717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.934+0000 7f55caffd640 1 --2- 172.21.15.23:0/128043964 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55682f36f3a0 0x55682f3717c0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d521e215f2a36b0 server_cookie=6f8fbb74cc7ac980 in_seq=0 out_seq=0 2024-03-24T10:45:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.934+0000 7f55b27fc640 10 client.17031 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:06.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55b27fc640 1 -- 172.21.15.23:0/128043964 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f55b4005500 con 0x55682f36f3a0 2024-03-24T10:45:06.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55b27fc640 10 client.17031 handle_command_reply: tid=0 2024-03-24T10:45:06.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55d4210740 1 client.17031 shutdown 2024-03-24T10:45:06.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55d4210740 1 -- 172.21.15.23:0/128043964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 msgr2=0x55682f2fc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:06.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55d4210740 1 --2- 172.21.15.23:0/128043964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f2fc0f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f55b4005f10 tx=0x7f55b4004360 comp rx=0 tx=0).stop 2024-03-24T10:45:06.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55d4210740 1 -- 172.21.15.23:0/128043964 shutdown_connections 2024-03-24T10:45:06.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55d4210740 1 --2- 172.21.15.23:0/128043964 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55682f36f3a0 0x55682f3717c0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55d4210740 1 --2- 172.21.15.23:0/128043964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682f2f7c60 0x55682f2fc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55d4210740 1 --2- 172.21.15.23:0/128043964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55682f2ef080 0x55682f2fc0f0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55d4210740 1 --2- 172.21.15.23:0/128043964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682f2ee760 0x55682f2ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55b27fc640 0 client.17031 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:06.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55d4210740 1 -- 172.21.15.23:0/128043964 >> 172.21.15.23:0/128043964 conn(0x55682f2e8e90 msgr2=0x55682f2f4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:06.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55d4210740 1 -- 172.21.15.23:0/128043964 shutdown_connections 2024-03-24T10:45:06.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55d4210740 1 -- 172.21.15.23:0/128043964 wait complete. 2024-03-24T10:45:06.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.935+0000 7f55d4210740 20 client.17031 trim_cache size 0 max 16384 2024-03-24T10:45:06.941 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:45:06.942 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:45:06.942 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "bba9ea44-4e52-43fd-a0b7-bbf643597fcc", 2024-03-24T10:45:06.942 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:45:06.942 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:45:06.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.936+0000 7f55d155c640 1 -- 172.21.15.23:0/3223754660 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f55b80a2230 msgr2=0x7f55b80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:06.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.936+0000 7f55d155c640 1 --2- 172.21.15.23:0/3223754660 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f55b80a2230 0x7f55b80a46f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f55c4005c40 tx=0x7f55c4004a40 comp rx=0 tx=0).stop 2024-03-24T10:45:06.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.936+0000 7f55d155c640 1 -- 172.21.15.23:0/3223754660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55cc068910 msgr2=0x7f55cc197d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:06.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.936+0000 7f55d155c640 1 --2- 172.21.15.23:0/3223754660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55cc068910 0x7f55cc197d70 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f55bc009860 tx=0x7f55bc004530 comp rx=0 tx=0).stop 2024-03-24T10:45:06.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.936+0000 7f55d155c640 1 -- 172.21.15.23:0/3223754660 shutdown_connections 2024-03-24T10:45:06.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.936+0000 7f55d155c640 1 --2- 172.21.15.23:0/3223754660 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f55b80a2230 0x7f55b80a46f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.936+0000 7f55d155c640 1 --2- 172.21.15.23:0/3223754660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55cc107760 0x7f55cc19c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.936+0000 7f55d155c640 1 --2- 172.21.15.23:0/3223754660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55cc0692d0 0x7f55cc1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.936+0000 7f55d155c640 1 --2- 172.21.15.23:0/3223754660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55cc068910 0x7f55cc197d70 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.936+0000 7f55d155c640 1 -- 172.21.15.23:0/3223754660 >> 172.21.15.23:0/3223754660 conn(0x7f55cc077710 msgr2=0x7f55cc0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:06.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.936+0000 7f55d155c640 1 -- 172.21.15.23:0/3223754660 shutdown_connections 2024-03-24T10:45:06.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:06.936+0000 7f55d155c640 1 -- 172.21.15.23:0/3223754660 wait complete. 2024-03-24T10:45:06.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:06 smithi023 ceph-mon[32466]: pgmap v153: 97 pgs: 97 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-03-24T10:45:06.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:45:06.947 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:45:07.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:06 smithi117 ceph-mon[36368]: pgmap v153: 97 pgs: 97 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-03-24T10:45:07.003 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:06 smithi117 systemd[1]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.c.service: Failed with result 'exit-code'. 2024-03-24T10:45:07.003 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:06 smithi117 systemd[1]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.c.service: Consumed 1.304s CPU time. 2024-03-24T10:45:07.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:06 smithi116 ceph-mon[36248]: pgmap v153: 97 pgs: 97 active+clean; 254 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-03-24T10:45:07.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.056+0000 7f8620c4d640 1 -- 172.21.15.23:0/3267961033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f861c106660 msgr2=0x7f861c106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.056+0000 7f8620c4d640 1 --2- 172.21.15.23:0/3267961033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f861c106660 0x7f861c106a40 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f860c009900 tx=0x7f860c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:45:07.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.056+0000 7f8620c4d640 1 -- 172.21.15.23:0/3267961033 shutdown_connections 2024-03-24T10:45:07.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.056+0000 7f8620c4d640 1 --2- 172.21.15.23:0/3267961033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f861c100f60 0x7f861c10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.056+0000 7f8620c4d640 1 --2- 172.21.15.23:0/3267961033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f861c1005a0 0x7f861c100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.056+0000 7f8620c4d640 1 --2- 172.21.15.23:0/3267961033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f861c106660 0x7f861c106a40 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.056+0000 7f8620c4d640 1 -- 172.21.15.23:0/3267961033 >> 172.21.15.23:0/3267961033 conn(0x7f861c077710 msgr2=0x7f861c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:07.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.056+0000 7f8620c4d640 1 -- 172.21.15.23:0/3267961033 shutdown_connections 2024-03-24T10:45:07.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.056+0000 7f8620c4d640 1 -- 172.21.15.23:0/3267961033 wait complete. 2024-03-24T10:45:07.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8620c4d640 1 Processor -- start 2024-03-24T10:45:07.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8620c4d640 1 -- start start 2024-03-24T10:45:07.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8620c4d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f861c1005a0 0x7f861c19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8620c4d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f861c100f60 0x7f861c19cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8620c4d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f861c106660 0x7f861c196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8620c4d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f861c197080 con 0x7f861c100f60 2024-03-24T10:45:07.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8620c4d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f861c1971f0 con 0x7f861c1005a0 2024-03-24T10:45:07.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8620c4d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f861c1973b0 con 0x7f861c106660 2024-03-24T10:45:07.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8613fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f861c100f60 0x7f861c19cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8613fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f861c100f60 0x7f861c19cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52842/0 (socket says 172.21.15.23:52842) 2024-03-24T10:45:07.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8613fff640 1 -- 172.21.15.23:0/2134518541 learned_addr learned my addr 172.21.15.23:0/2134518541 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:07.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f861a59c640 1 --2- 172.21.15.23:0/2134518541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f861c1005a0 0x7f861c19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f861ad9d640 1 --2- 172.21.15.23:0/2134518541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f861c106660 0x7f861c196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8613fff640 1 -- 172.21.15.23:0/2134518541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f861c106660 msgr2=0x7f861c196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8613fff640 1 --2- 172.21.15.23:0/2134518541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f861c106660 0x7f861c196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8613fff640 1 -- 172.21.15.23:0/2134518541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f861c1005a0 msgr2=0x7f861c19c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8613fff640 1 --2- 172.21.15.23:0/2134518541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f861c1005a0 0x7f861c19c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8613fff640 1 -- 172.21.15.23:0/2134518541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860c009560 con 0x7f861c100f60 2024-03-24T10:45:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.057+0000 7f8613fff640 1 --2- 172.21.15.23:0/2134518541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f861c100f60 0x7f861c19cdc0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f860400e9f0 tx=0x7f860400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.058+0000 7f86137fe640 1 -- 172.21.15.23:0/2134518541 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604009700 con 0x7f861c100f60 2024-03-24T10:45:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.058+0000 7f86137fe640 1 -- 172.21.15.23:0/2134518541 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f860400cd80 con 0x7f861c100f60 2024-03-24T10:45:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.058+0000 7f8620c4d640 1 -- 172.21.15.23:0/2134518541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f861c197690 con 0x7f861c100f60 2024-03-24T10:45:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.058+0000 7f86137fe640 1 -- 172.21.15.23:0/2134518541 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604018920 con 0x7f861c100f60 2024-03-24T10:45:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.058+0000 7f8620c4d640 1 -- 172.21.15.23:0/2134518541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f861c1a3920 con 0x7f861c100f60 2024-03-24T10:45:07.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.060+0000 7f86137fe640 1 -- 172.21.15.23:0/2134518541 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f8604018a80 con 0x7f861c100f60 2024-03-24T10:45:07.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.060+0000 7f8623820740 1 Processor -- start 2024-03-24T10:45:07.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.060+0000 7f8623820740 1 -- start start 2024-03-24T10:45:07.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.060+0000 7f8623820740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e299ebd0 0x55c4e299efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.060+0000 7f86137fe640 1 --2- 172.21.15.23:0/2134518541 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f86080a2230 0x7f86080a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.060+0000 7f8623820740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29aa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.060+0000 7f8623820740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e29aa8e0 0x55c4e29accd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.060+0000 7f8623820740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c4e28c3f30 con 0x55c4e299f580 2024-03-24T10:45:07.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.060+0000 7f8623820740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c4e290d810 con 0x55c4e29aa8e0 2024-03-24T10:45:07.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.060+0000 7f8623820740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c4e290ad40 con 0x55c4e299ebd0 2024-03-24T10:45:07.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f861a59c640 1 --2- 172.21.15.23:0/2134518541 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f86080a2230 0x7f86080a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f861ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e299ebd0 0x55c4e299efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f86137fe640 1 -- 172.21.15.23:0/2134518541 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f8604014070 con 0x7f861c100f60 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f8613fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29aa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f861ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e299ebd0 0x55c4e299efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:43490/0 (socket says 172.21.15.23:43490) 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f861ad9d640 1 -- 172.21.15.23:0/1798651572 learned_addr learned my addr 172.21.15.23:0/1798651572 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f861ad9d640 1 --2- 172.21.15.23:0/1798651572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e29aa8e0 0x55c4e29accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f861ad9d640 1 -- 172.21.15.23:0/1798651572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e29aa8e0 msgr2=0x55c4e29accd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f861ad9d640 1 --2- 172.21.15.23:0/1798651572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e29aa8e0 0x55c4e29accd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f861ad9d640 1 -- 172.21.15.23:0/1798651572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 msgr2=0x55c4e29aa3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f861ad9d640 1 --2- 172.21.15.23:0/1798651572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29aa3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f861ad9d640 1 -- 172.21.15.23:0/1798651572 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4e28c7ea0 con 0x55c4e299ebd0 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f861a59c640 1 --2- 172.21.15.23:0/2134518541 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f86080a2230 0x7f86080a46f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f860c005ab0 tx=0x7f860c005a20 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f861ad9d640 1 --2- 172.21.15.23:0/1798651572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e299ebd0 0x55c4e299efb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8614005c40 tx=0x7f8614005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34dd8295a1daf93d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f8610ff9640 1 -- 172.21.15.23:0/1798651572 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8614011130 con 0x55c4e299ebd0 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f8610ff9640 1 -- 172.21.15.23:0/1798651572 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8614011290 con 0x55c4e299ebd0 2024-03-24T10:45:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.061+0000 7f8610ff9640 1 -- 172.21.15.23:0/1798651572 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861400fb70 con 0x55c4e299ebd0 2024-03-24T10:45:07.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.062+0000 7f8623820740 1 -- 172.21.15.23:0/1798651572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e299ebd0 msgr2=0x55c4e299efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.062+0000 7f8623820740 1 --2- 172.21.15.23:0/1798651572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e299ebd0 0x55c4e299efb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8614005c40 tx=0x7f8614005bd0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.062+0000 7f8623820740 1 -- 172.21.15.23:0/1798651572 shutdown_connections 2024-03-24T10:45:07.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.062+0000 7f8623820740 1 --2- 172.21.15.23:0/1798651572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e29aa8e0 0x55c4e29accd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.062+0000 7f8623820740 1 --2- 172.21.15.23:0/1798651572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29aa3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.062+0000 7f8623820740 1 --2- 172.21.15.23:0/1798651572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e299ebd0 0x55c4e299efb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.062+0000 7f8623820740 1 -- 172.21.15.23:0/1798651572 >> 172.21.15.23:0/1798651572 conn(0x55c4e2999900 msgr2=0x55c4e2999ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:07.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.062+0000 7f8623820740 1 -- 172.21.15.23:0/1798651572 shutdown_connections 2024-03-24T10:45:07.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.062+0000 7f8623820740 1 -- 172.21.15.23:0/1798651572 wait complete. 2024-03-24T10:45:07.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.062+0000 7f8623820740 1 -- start start 2024-03-24T10:45:07.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.062+0000 7f8623820740 1 Processor -- start 2024-03-24T10:45:07.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f8623820740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 0x55c4e29a7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f8623820740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e299f580 0x55c4e29a7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f8623820740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e29aa8e0 0x55c4e29a8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f8623820740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c4e2906520 con 0x55c4e29aa8e0 2024-03-24T10:45:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f8623820740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c4e25d9430 con 0x55c4e299ebd0 2024-03-24T10:45:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f8623820740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c4e2595e00 con 0x55c4e299f580 2024-03-24T10:45:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f8623820740 10 client.0 _ll_register_callbacks cb 0x55c4e28a4d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f861ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e29aa8e0 0x55c4e29a8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f861ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e29aa8e0 0x55c4e29a8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52862/0 (socket says 172.21.15.23:52862) 2024-03-24T10:45:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f861ad9d640 1 -- 172.21.15.23:0/2145740957 learned_addr learned my addr 172.21.15.23:0/2145740957 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f8613fff640 1 --2- 172.21.15.23:0/2145740957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e299f580 0x55c4e29a7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f861a59c640 1 --2- 172.21.15.23:0/2145740957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 0x55c4e29a7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f861ad9d640 1 -- 172.21.15.23:0/2145740957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e299f580 msgr2=0x55c4e29a7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f861ad9d640 1 --2- 172.21.15.23:0/2145740957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e299f580 0x55c4e29a7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f861ad9d640 1 -- 172.21.15.23:0/2145740957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 msgr2=0x55c4e29a7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:07.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f861ad9d640 1 --2- 172.21.15.23:0/2145740957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 0x55c4e29a7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f861ad9d640 1 -- 172.21.15.23:0/2145740957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8614019040 con 0x55c4e29aa8e0 2024-03-24T10:45:07.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f861ad9d640 1 --2- 172.21.15.23:0/2145740957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e29aa8e0 0x55c4e29a8390 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8614005f40 tx=0x7f8614011400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5c77e12213cffa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:07.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f8601ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:07.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f8601ffb640 1 -- 172.21.15.23:0/2145740957 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8614025070 con 0x55c4e29aa8e0 2024-03-24T10:45:07.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f8601ffb640 1 -- 172.21.15.23:0/2145740957 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f861400f950 con 0x55c4e29aa8e0 2024-03-24T10:45:07.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.063+0000 7f8623820740 1 -- 172.21.15.23:0/2145740957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c4e26057e0 con 0x55c4e29aa8e0 2024-03-24T10:45:07.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.064+0000 7f8601ffb640 1 -- 172.21.15.23:0/2145740957 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f861400fb30 con 0x55c4e29aa8e0 2024-03-24T10:45:07.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.064+0000 7f8623820740 10 client.17043 fetch_fsmap learned FSMap version 27 2024-03-24T10:45:07.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.064+0000 7f8623820740 1 -- 172.21.15.23:0/2145740957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c4e2904ba0 con 0x55c4e29aa8e0 2024-03-24T10:45:07.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.064+0000 7f8601ffb640 1 -- 172.21.15.23:0/2145740957 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8614036080 con 0x55c4e29aa8e0 2024-03-24T10:45:07.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.064+0000 7f8601ffb640 1 -- 172.21.15.23:0/2145740957 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f8614030030 con 0x55c4e29aa8e0 2024-03-24T10:45:07.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.064+0000 7f8623820740 10 client.17043 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:45:07.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.064+0000 7f8623820740 10 client.17043 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:07.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.064+0000 7f8623820740 20 client.17043 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.064+0000 7f8623820740 1 --2- 172.21.15.23:0/2145740957 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c4e29bad20 0x55c4e29bd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.064+0000 7f8623820740 4 client.17043 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.064+0000 7f8623820740 1 -- 172.21.15.23:0/2145740957 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4e2642720 con 0x55c4e29bad20 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.064+0000 7f861a59c640 1 --2- 172.21.15.23:0/2145740957 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c4e29bad20 0x55c4e29bd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.065+0000 7f861a59c640 1 --2- 172.21.15.23:0/2145740957 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c4e29bad20 0x55c4e29bd130 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6e23a59a1437327 server_cookie=4982950792033cab in_seq=0 out_seq=0 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.065+0000 7f8601ffb640 10 client.17043 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.065+0000 7f8601ffb640 1 -- 172.21.15.23:0/2145740957 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c4e2642720 con 0x55c4e29bad20 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.065+0000 7f8601ffb640 10 client.17043 handle_command_reply: tid=0 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.066+0000 7f8623820740 1 client.17043 shutdown 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.066+0000 7f8623820740 1 -- 172.21.15.23:0/2145740957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e29aa8e0 msgr2=0x55c4e29a8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.066+0000 7f8623820740 1 --2- 172.21.15.23:0/2145740957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e29aa8e0 0x55c4e29a8390 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8614005f40 tx=0x7f8614011400 comp rx=0 tx=0).stop 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.066+0000 7f8623820740 1 -- 172.21.15.23:0/2145740957 shutdown_connections 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.066+0000 7f8623820740 1 --2- 172.21.15.23:0/2145740957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e29aa8e0 0x55c4e29a8390 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.066+0000 7f8623820740 1 --2- 172.21.15.23:0/2145740957 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c4e29bad20 0x55c4e29bd130 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.066+0000 7f8623820740 1 --2- 172.21.15.23:0/2145740957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e299f580 0x55c4e29a7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.066+0000 7f8601ffb640 0 client.17043 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:07.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.066+0000 7f8623820740 1 --2- 172.21.15.23:0/2145740957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 0x55c4e29a7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.066+0000 7f8623820740 1 -- 172.21.15.23:0/2145740957 >> 172.21.15.23:0/2145740957 conn(0x55c4e2999900 msgr2=0x55c4e299b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:07.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.066+0000 7f8623820740 1 -- 172.21.15.23:0/2145740957 shutdown_connections 2024-03-24T10:45:07.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.066+0000 7f8623820740 1 -- 172.21.15.23:0/2145740957 wait complete. 2024-03-24T10:45:07.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.066+0000 7f8623820740 20 client.17043 trim_cache size 0 max 16384 2024-03-24T10:45:07.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.083+0000 7f8623820740 1 Processor -- start 2024-03-24T10:45:07.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.083+0000 7f8623820740 1 -- start start 2024-03-24T10:45:07.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8623820740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 0x55c4e29a71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8623820740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8623820740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e29aa8e0 0x55c4e29c29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8623820740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c4e288b430 con 0x55c4e299f580 2024-03-24T10:45:07.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8623820740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c4e29a7730 con 0x55c4e299ebd0 2024-03-24T10:45:07.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8623820740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c4e29a8870 con 0x55c4e29aa8e0 2024-03-24T10:45:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8613fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8613fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29b61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52864/0 (socket says 172.21.15.23:52864) 2024-03-24T10:45:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8613fff640 1 -- 172.21.15.23:0/1347399979 learned_addr learned my addr 172.21.15.23:0/1347399979 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f861a59c640 1 --2- 172.21.15.23:0/1347399979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 0x55c4e29a71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f861ad9d640 1 --2- 172.21.15.23:0/1347399979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e29aa8e0 0x55c4e29c29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8613fff640 1 -- 172.21.15.23:0/1347399979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e29aa8e0 msgr2=0x55c4e29c29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8613fff640 1 --2- 172.21.15.23:0/1347399979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e29aa8e0 0x55c4e29c29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8613fff640 1 -- 172.21.15.23:0/1347399979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 msgr2=0x55c4e29a71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8613fff640 1 --2- 172.21.15.23:0/1347399979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 0x55c4e29a71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8613fff640 1 -- 172.21.15.23:0/1347399979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4e28a0b20 con 0x55c4e299f580 2024-03-24T10:45:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8613fff640 1 --2- 172.21.15.23:0/1347399979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29b61b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f860400c6a0 tx=0x7f860409bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cbbc2f6230b405a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8603fff640 1 -- 172.21.15.23:0/1347399979 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860409b7c0 con 0x55c4e299f580 2024-03-24T10:45:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8603fff640 1 -- 172.21.15.23:0/1347399979 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f860409c160 con 0x55c4e299f580 2024-03-24T10:45:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.084+0000 7f8603fff640 1 -- 172.21.15.23:0/1347399979 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604010ca0 con 0x55c4e299f580 2024-03-24T10:45:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 -- 172.21.15.23:0/1347399979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 msgr2=0x55c4e29b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 --2- 172.21.15.23:0/1347399979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29b61b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f860400c6a0 tx=0x7f860409bf60 comp rx=0 tx=0).stop 2024-03-24T10:45:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 -- 172.21.15.23:0/1347399979 shutdown_connections 2024-03-24T10:45:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 --2- 172.21.15.23:0/1347399979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e29aa8e0 0x55c4e29c29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 --2- 172.21.15.23:0/1347399979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29b61b0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 --2- 172.21.15.23:0/1347399979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 0x55c4e29a71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 -- 172.21.15.23:0/1347399979 >> 172.21.15.23:0/1347399979 conn(0x55c4e2999900 msgr2=0x55c4e29a8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 -- 172.21.15.23:0/1347399979 shutdown_connections 2024-03-24T10:45:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 -- 172.21.15.23:0/1347399979 wait complete. 2024-03-24T10:45:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 -- start start 2024-03-24T10:45:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 Processor -- start 2024-03-24T10:45:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 0x55c4e29a0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e29aa8e0 0x55c4e29b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55c4e29a13b0 con 0x55c4e299f580 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c4e29a1520 con 0x55c4e299ebd0 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c4e29b1d20 con 0x55c4e29aa8e0 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8623820740 10 client.0 _ll_register_callbacks cb 0x7f8604010e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8613fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.085+0000 7f8613fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52880/0 (socket says 172.21.15.23:52880) 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8613fff640 1 -- 172.21.15.23:0/372314510 learned_addr learned my addr 172.21.15.23:0/372314510 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f861a59c640 1 --2- 172.21.15.23:0/372314510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 0x55c4e29a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f861ad9d640 1 --2- 172.21.15.23:0/372314510 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e29aa8e0 0x55c4e29b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8613fff640 1 -- 172.21.15.23:0/372314510 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e29aa8e0 msgr2=0x55c4e29b17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8613fff640 1 --2- 172.21.15.23:0/372314510 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e29aa8e0 0x55c4e29b17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8613fff640 1 -- 172.21.15.23:0/372314510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 msgr2=0x55c4e29a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8613fff640 1 --2- 172.21.15.23:0/372314510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 0x55c4e29a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8613fff640 1 -- 172.21.15.23:0/372314510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f861c1a3920 con 0x55c4e299f580 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8613fff640 1 --2- 172.21.15.23:0/372314510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29a0c50 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f86040cc190 tx=0x7f860409ade0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c678b7b3251f6d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8601ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8601ffb640 1 -- 172.21.15.23:0/372314510 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86040a35d0 con 0x55c4e299f580 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8601ffb640 1 -- 172.21.15.23:0/372314510 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8527+0+0 (secure 0 0 0) 0x7f860409ef80 con 0x55c4e299f580 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8623820740 1 -- 172.21.15.23:0/372314510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8614011290 con 0x55c4e299f580 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8601ffb640 1 -- 172.21.15.23:0/372314510 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8604010c80 con 0x55c4e299f580 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8623820740 10 client.17049 fetch_fsmap learned FSMap version 27 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8623820740 1 -- 172.21.15.23:0/372314510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c4e28c6700 con 0x55c4e299f580 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8601ffb640 1 -- 172.21.15.23:0/372314510 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86040b7080 con 0x55c4e299f580 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.086+0000 7f8601ffb640 1 -- 172.21.15.23:0/372314510 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) ==== 8338+0+0 (secure 0 0 0) 0x7f860409a330 con 0x55c4e299f580 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 10 client.17049 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 10 client.17049 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 20 client.17049 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 1 --2- 172.21.15.23:0/372314510 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c4e2a23400 0x55c4e2a257f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 4 client.17049 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 1 -- 172.21.15.23:0/372314510 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c4e2691890 con 0x55c4e2a23400 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f861a59c640 1 --2- 172.21.15.23:0/372314510 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c4e2a23400 0x55c4e2a257f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8601ffb640 10 client.17049 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f861a59c640 1 --2- 172.21.15.23:0/372314510 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c4e2a23400 0x55c4e2a257f0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58a60ec1998ea9f6 server_cookie=1d28a2c9390c8a4f in_seq=0 out_seq=0 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8601ffb640 1 -- 172.21.15.23:0/372314510 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c4e2691890 con 0x55c4e2a23400 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8601ffb640 10 client.17049 handle_command_reply: tid=0 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 1 client.17049 shutdown 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 1 -- 172.21.15.23:0/372314510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 msgr2=0x55c4e29a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 1 --2- 172.21.15.23:0/372314510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29a0c50 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f86040cc190 tx=0x7f860409ade0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 1 -- 172.21.15.23:0/372314510 shutdown_connections 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 1 --2- 172.21.15.23:0/372314510 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55c4e2a23400 0x55c4e2a257f0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 1 --2- 172.21.15.23:0/372314510 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4e29aa8e0 0x55c4e29b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 1 --2- 172.21.15.23:0/372314510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4e299f580 0x55c4e29a0c50 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 1 --2- 172.21.15.23:0/372314510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4e299ebd0 0x55c4e29a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8601ffb640 0 client.17049 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:07.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 1 -- 172.21.15.23:0/372314510 >> 172.21.15.23:0/372314510 conn(0x55c4e2999900 msgr2=0x55c4e299a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:07.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 1 -- 172.21.15.23:0/372314510 shutdown_connections 2024-03-24T10:45:07.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.087+0000 7f8623820740 1 -- 172.21.15.23:0/372314510 wait complete. 2024-03-24T10:45:07.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.088+0000 7f8623820740 20 client.17049 trim_cache size 0 max 16384 2024-03-24T10:45:07.095 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:45:07.095 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:45:07.095 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:45:07.095 INFO:teuthology.orchestra.run.smithi023.stdout: "bba9ea44-4e52-43fd-a0b7-bbf643597fcc": { 2024-03-24T10:45:07.095 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:45:07.095 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "bba9ea44-4e52-43fd-a0b7-bbf643597fcc", 2024-03-24T10:45:07.095 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:45:07.096 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:45:07.096 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:45:07.096 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:45:07.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.088+0000 7f8620c4d640 1 -- 172.21.15.23:0/2134518541 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f86080a2230 msgr2=0x7f86080a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.088+0000 7f8620c4d640 1 --2- 172.21.15.23:0/2134518541 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f86080a2230 0x7f86080a46f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f860c005ab0 tx=0x7f860c005a20 comp rx=0 tx=0).stop 2024-03-24T10:45:07.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.088+0000 7f8620c4d640 1 -- 172.21.15.23:0/2134518541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f861c100f60 msgr2=0x7f861c19cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.088+0000 7f8620c4d640 1 --2- 172.21.15.23:0/2134518541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f861c100f60 0x7f861c19cdc0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f860400e9f0 tx=0x7f860400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.088+0000 7f8620c4d640 1 -- 172.21.15.23:0/2134518541 shutdown_connections 2024-03-24T10:45:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.088+0000 7f8620c4d640 1 --2- 172.21.15.23:0/2134518541 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f86080a2230 0x7f86080a46f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.088+0000 7f8620c4d640 1 --2- 172.21.15.23:0/2134518541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f861c106660 0x7f861c196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.088+0000 7f8620c4d640 1 --2- 172.21.15.23:0/2134518541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f861c100f60 0x7f861c19cdc0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.088+0000 7f8620c4d640 1 --2- 172.21.15.23:0/2134518541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f861c1005a0 0x7f861c19c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.088+0000 7f8620c4d640 1 -- 172.21.15.23:0/2134518541 >> 172.21.15.23:0/2134518541 conn(0x7f861c077710 msgr2=0x7f861c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.088+0000 7f8620c4d640 1 -- 172.21.15.23:0/2134518541 shutdown_connections 2024-03-24T10:45:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:07.088+0000 7f8620c4d640 1 -- 172.21.15.23:0/2134518541 wait complete. 2024-03-24T10:45:07.099 INFO:tasks.cephfs.filesystem:scrub status for tag:bba9ea44-4e52-43fd-a0b7-bbf643597fcc - {'path': '/', 'tag': 'bba9ea44-4e52-43fd-a0b7-bbf643597fcc', 'options': 'recursive,force'} 2024-03-24T10:45:07.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 458344 11.84 MB/sec execute 84 sec latency 75.942 ms 2024-03-24T10:45:08.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:07 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:45:08.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:07 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:08.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:07 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:45:08.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:07 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:45:08.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:07 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:45:08.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:07 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:08.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:07 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:45:08.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:07 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:45:08.119 INFO:tasks.workunit.client.0.smithi023.stdout: 1 460720 12.03 MB/sec execute 85 sec latency 59.278 ms 2024-03-24T10:45:08.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:07 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:45:08.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:07 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:08.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:07 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:45:08.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:07 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:45:09.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:08 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:09.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:08 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:09.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:08 smithi117 ceph-mon[36368]: pgmap v154: 97 pgs: 97 active+clean; 261 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.7 MiB/s wr, 328 op/s 2024-03-24T10:45:09.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:08 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:09.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:08 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:09.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:08 smithi116 ceph-mon[36248]: pgmap v154: 97 pgs: 97 active+clean; 261 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.7 MiB/s wr, 328 op/s 2024-03-24T10:45:09.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 462343 12.03 MB/sec execute 86 sec latency 85.264 ms 2024-03-24T10:45:09.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:08 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:09.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:08 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:09.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:08 smithi023 ceph-mon[32466]: pgmap v154: 97 pgs: 97 active+clean; 261 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.7 MiB/s wr, 328 op/s 2024-03-24T10:45:10.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 464924 12.04 MB/sec execute 87 sec latency 14.532 ms 2024-03-24T10:45:11.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:10 smithi117 ceph-mon[36368]: pgmap v155: 97 pgs: 97 active+clean; 261 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 225 op/s 2024-03-24T10:45:11.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:10 smithi116 ceph-mon[36248]: pgmap v155: 97 pgs: 97 active+clean; 261 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 225 op/s 2024-03-24T10:45:11.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 466976 12.03 MB/sec execute 88 sec latency 21.727 ms 2024-03-24T10:45:11.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:10 smithi023 ceph-mon[32466]: pgmap v155: 97 pgs: 97 active+clean; 261 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.1 MiB/s wr, 225 op/s 2024-03-24T10:45:12.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 469456 12.05 MB/sec execute 89 sec latency 19.641 ms 2024-03-24T10:45:13.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:12 smithi117 ceph-mon[36368]: pgmap v156: 97 pgs: 97 active+clean; 269 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.7 MiB/s wr, 322 op/s 2024-03-24T10:45:13.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:12 smithi116 ceph-mon[36248]: pgmap v156: 97 pgs: 97 active+clean; 269 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.7 MiB/s wr, 322 op/s 2024-03-24T10:45:13.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 471482 12.01 MB/sec execute 90 sec latency 19.300 ms 2024-03-24T10:45:13.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:12 smithi023 ceph-mon[32466]: pgmap v156: 97 pgs: 97 active+clean; 269 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.7 MiB/s wr, 322 op/s 2024-03-24T10:45:14.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 473135 12.00 MB/sec execute 91 sec latency 75.669 ms 2024-03-24T10:45:15.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:14 smithi117 ceph-mon[36368]: pgmap v157: 97 pgs: 97 active+clean; 269 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 196 op/s 2024-03-24T10:45:15.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:14 smithi116 ceph-mon[36248]: pgmap v157: 97 pgs: 97 active+clean; 269 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 196 op/s 2024-03-24T10:45:15.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 475135 11.96 MB/sec execute 92 sec latency 27.996 ms 2024-03-24T10:45:15.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:14 smithi023 ceph-mon[32466]: pgmap v157: 97 pgs: 97 active+clean; 269 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 196 op/s 2024-03-24T10:45:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:15 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:45:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:15 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:45:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:15 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:45:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:15 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:45:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:15 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:45:16.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:15 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:45:16.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:15 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:45:16.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:15 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:45:16.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:15 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:45:16.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:15 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:45:16.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:15 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:45:16.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:15 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:45:16.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 477156 11.97 MB/sec execute 93 sec latency 18.812 ms 2024-03-24T10:45:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:15 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:45:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:15 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:45:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:15 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:45:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:15 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:45:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:15 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:45:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:15 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:45:16.980 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:16 smithi117 ceph-mon[36368]: pgmap v158: 97 pgs: 97 active+clean; 269 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 196 op/s 2024-03-24T10:45:16.980 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:16 smithi117 systemd[1]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-03-24T10:45:16.980 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:16 smithi117 systemd[1]: Stopped Ceph mds.c for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:45:16.980 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:16 smithi117 systemd[1]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.c.service: Consumed 1.304s CPU time. 2024-03-24T10:45:16.980 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:16 smithi117 systemd[1]: Starting Ceph mds.c for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:45:17.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:16 smithi116 ceph-mon[36248]: pgmap v158: 97 pgs: 97 active+clean; 269 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 196 op/s 2024-03-24T10:45:17.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 479741 12.01 MB/sec execute 94 sec latency 16.576 ms 2024-03-24T10:45:17.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:16 smithi023 ceph-mon[32466]: pgmap v158: 97 pgs: 97 active+clean; 269 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 196 op/s 2024-03-24T10:45:17.252 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:16 smithi117 podman[71230]: 2024-03-24 10:45:16.885878026 +0000 UTC m=+0.020363658 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:45:17.252 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:17 smithi117 podman[71230]: 2024-03-24 10:45:17.036656989 +0000 UTC m=+0.171142643 container create e994541783aeff6baed12288e24342bf8fbb92290f562dbe3dd85a2d0100d7c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T10:45:17.705 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:17 smithi117 podman[71230]: 2024-03-24 10:45:17.289428846 +0000 UTC m=+0.423914459 container init e994541783aeff6baed12288e24342bf8fbb92290f562dbe3dd85a2d0100d7c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:45:17.705 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:17 smithi117 podman[71230]: 2024-03-24 10:45:17.292188737 +0000 UTC m=+0.426674358 container start e994541783aeff6baed12288e24342bf8fbb92290f562dbe3dd85a2d0100d7c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:45:17.705 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:17 smithi117 ceph-mds[71248]: -- 172.21.15.117:0/174903403 <== mon.1 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56404436a1a0 con 0x564044341000 2024-03-24T10:45:17.705 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:17 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: starting mds.c at 2024-03-24T10:45:17.705 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:17 smithi117 bash[71230]: e994541783aeff6baed12288e24342bf8fbb92290f562dbe3dd85a2d0100d7c5 2024-03-24T10:45:17.705 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:45:17 smithi117 systemd[1]: Started Ceph mds.c for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:45:18.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:17 smithi117 ceph-mon[36368]: daemon mds.c restarted 2024-03-24T10:45:18.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:17 smithi116 ceph-mon[36248]: daemon mds.c restarted 2024-03-24T10:45:18.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.020+0000 7f3192247640 1 -- 172.21.15.23:0/4089375701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f318c100860 msgr2=0x7f318c100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.020+0000 7f3192247640 1 --2- 172.21.15.23:0/4089375701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f318c100860 0x7f318c100ce0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3180009900 tx=0x7f318002bc30 comp rx=0 tx=0).stop 2024-03-24T10:45:18.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.020+0000 7f3192247640 1 -- 172.21.15.23:0/4089375701 shutdown_connections 2024-03-24T10:45:18.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.020+0000 7f3192247640 1 --2- 172.21.15.23:0/4089375701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c101220 0x7f318c10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.020+0000 7f3192247640 1 --2- 172.21.15.23:0/4089375701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f318c100860 0x7f318c100ce0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.020+0000 7f3192247640 1 --2- 172.21.15.23:0/4089375701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f318c1055b0 0x7f318c105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.020+0000 7f3192247640 1 -- 172.21.15.23:0/4089375701 >> 172.21.15.23:0/4089375701 conn(0x7f318c077460 msgr2=0x7f318c077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:18.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.020+0000 7f3192247640 1 -- 172.21.15.23:0/4089375701 shutdown_connections 2024-03-24T10:45:18.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.020+0000 7f3192247640 1 -- 172.21.15.23:0/4089375701 wait complete. 2024-03-24T10:45:18.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.020+0000 7f3192247640 1 Processor -- start 2024-03-24T10:45:18.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.020+0000 7f3192247640 1 -- start start 2024-03-24T10:45:18.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f3192247640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f318c100860 0x7f318c198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f3192247640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c101220 0x7f318c199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f3192247640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f318c1055b0 0x7f318c1a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f3192247640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f318c1998c0 con 0x7f318c100860 2024-03-24T10:45:18.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f3192247640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f318c1a40f0 con 0x7f318c101220 2024-03-24T10:45:18.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f3192247640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f318c1a4230 con 0x7f318c1055b0 2024-03-24T10:45:18.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f318b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f318c100860 0x7f318c198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f318b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f318c100860 0x7f318c198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35526/0 (socket says 172.21.15.23:35526) 2024-03-24T10:45:18.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f318b7fe640 1 -- 172.21.15.23:0/2235443954 learned_addr learned my addr 172.21.15.23:0/2235443954 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f318affd640 1 --2- 172.21.15.23:0/2235443954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c101220 0x7f318c199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f318bfff640 1 --2- 172.21.15.23:0/2235443954 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f318c1055b0 0x7f318c1a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f318b7fe640 1 -- 172.21.15.23:0/2235443954 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f318c1055b0 msgr2=0x7f318c1a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f318b7fe640 1 --2- 172.21.15.23:0/2235443954 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f318c1055b0 0x7f318c1a3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f318b7fe640 1 -- 172.21.15.23:0/2235443954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c101220 msgr2=0x7f318c199070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f318b7fe640 1 --2- 172.21.15.23:0/2235443954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c101220 0x7f318c199070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f318b7fe640 1 -- 172.21.15.23:0/2235443954 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3180009560 con 0x7f318c100860 2024-03-24T10:45:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f318b7fe640 1 --2- 172.21.15.23:0/2235443954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f318c100860 0x7f318c198b30 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f317c00ea20 tx=0x7f317c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f3188ff9640 1 -- 172.21.15.23:0/2235443954 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c009720 con 0x7f318c100860 2024-03-24T10:45:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f3188ff9640 1 -- 172.21.15.23:0/2235443954 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f317c00cd80 con 0x7f318c100860 2024-03-24T10:45:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f3192247640 1 -- 172.21.15.23:0/2235443954 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f318c1a4490 con 0x7f318c100860 2024-03-24T10:45:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f3188ff9640 1 -- 172.21.15.23:0/2235443954 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c010490 con 0x7f318c100860 2024-03-24T10:45:18.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.021+0000 7f3192247640 1 -- 172.21.15.23:0/2235443954 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f318c1a4920 con 0x7f318c100860 2024-03-24T10:45:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.024+0000 7f3194e28740 1 Processor -- start 2024-03-24T10:45:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.024+0000 7f3188ff9640 1 -- 172.21.15.23:0/2235443954 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f317c005020 con 0x7f318c100860 2024-03-24T10:45:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.024+0000 7f3194e28740 1 -- start start 2024-03-24T10:45:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f3194e28740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0c5e0 0x563858f0c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f3194e28740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f0cf00 0x563858f15790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f3194e28740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f15cd0 0x563858f180c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f3194e28740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563858ec71b0 con 0x563858f15cd0 2024-03-24T10:45:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f3194e28740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563858e79f80 con 0x563858f0cf00 2024-03-24T10:45:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f3194e28740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563858e774b0 con 0x563858f0c5e0 2024-03-24T10:45:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f318affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f15cd0 0x563858f180c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f318affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f15cd0 0x563858f180c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35534/0 (socket says 172.21.15.23:35534) 2024-03-24T10:45:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f318affd640 1 -- 172.21.15.23:0/1228807520 learned_addr learned my addr 172.21.15.23:0/1228807520 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f318b7fe640 1 --2- 172.21.15.23:0/1228807520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f0cf00 0x563858f15790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f318bfff640 1 --2- 172.21.15.23:0/1228807520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0c5e0 0x563858f0c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f3188ff9640 1 --2- 172.21.15.23:0/2235443954 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f31740a22d0 0x7f31740a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f318affd640 1 -- 172.21.15.23:0/1228807520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0c5e0 msgr2=0x563858f0c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f318affd640 1 --2- 172.21.15.23:0/1228807520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0c5e0 0x563858f0c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f318affd640 1 -- 172.21.15.23:0/1228807520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f0cf00 msgr2=0x563858f15790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f318affd640 1 --2- 172.21.15.23:0/1228807520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f0cf00 0x563858f15790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f318affd640 1 -- 172.21.15.23:0/1228807520 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563858e08810 con 0x563858f15cd0 2024-03-24T10:45:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f318bfff640 1 --2- 172.21.15.23:0/2235443954 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f31740a22d0 0x7f31740a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f3188ff9640 1 -- 172.21.15.23:0/2235443954 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f317c014070 con 0x7f318c100860 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f318affd640 1 --2- 172.21.15.23:0/1228807520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f15cd0 0x563858f180c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f3180009900 tx=0x7f318002c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=967d3c539614908b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.025+0000 7f3171ffb640 1 -- 172.21.15.23:0/1228807520 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318002c910 con 0x563858f15cd0 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.026+0000 7f3171ffb640 1 -- 172.21.15.23:0/1228807520 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f318002ca70 con 0x563858f15cd0 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.026+0000 7f3171ffb640 1 -- 172.21.15.23:0/1228807520 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3180005170 con 0x563858f15cd0 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.026+0000 7f318bfff640 1 --2- 172.21.15.23:0/2235443954 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f31740a22d0 0x7f31740a4790 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3184005fd0 tx=0x7f3184004a40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.026+0000 7f3194e28740 1 -- 172.21.15.23:0/1228807520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f15cd0 msgr2=0x563858f180c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.026+0000 7f3194e28740 1 --2- 172.21.15.23:0/1228807520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f15cd0 0x563858f180c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f3180009900 tx=0x7f318002c6a0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.026+0000 7f3194e28740 1 -- 172.21.15.23:0/1228807520 shutdown_connections 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.026+0000 7f3194e28740 1 --2- 172.21.15.23:0/1228807520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f15cd0 0x563858f180c0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.026+0000 7f3194e28740 1 --2- 172.21.15.23:0/1228807520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f0cf00 0x563858f15790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.026+0000 7f3194e28740 1 --2- 172.21.15.23:0/1228807520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0c5e0 0x563858f0c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.026+0000 7f3194e28740 1 -- 172.21.15.23:0/1228807520 >> 172.21.15.23:0/1228807520 conn(0x563858f06f00 msgr2=0x563858f072e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.026+0000 7f3194e28740 1 -- 172.21.15.23:0/1228807520 shutdown_connections 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.026+0000 7f3194e28740 1 -- 172.21.15.23:0/1228807520 wait complete. 2024-03-24T10:45:18.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.027+0000 7f3194e28740 1 -- start start 2024-03-24T10:45:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.027+0000 7f3194e28740 1 Processor -- start 2024-03-24T10:45:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.027+0000 7f3194e28740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0c5e0 0x563858f12cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.027+0000 7f3194e28740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0cf00 0x563858f13230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.027+0000 7f3194e28740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 0x563858f13770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.027+0000 7f3194e28740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563858e72c90 con 0x563858f0c5e0 2024-03-24T10:45:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.027+0000 7f3194e28740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563858e17c50 con 0x563858f15cd0 2024-03-24T10:45:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.027+0000 7f3194e28740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563858b46250 con 0x563858f0cf00 2024-03-24T10:45:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.027+0000 7f3194e28740 10 client.0 _ll_register_callbacks cb 0x563858e66290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.027+0000 7f318b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0c5e0 0x563858f12cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.027+0000 7f318b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0c5e0 0x563858f12cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35550/0 (socket says 172.21.15.23:35550) 2024-03-24T10:45:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.027+0000 7f318b7fe640 1 -- 172.21.15.23:0/568722052 learned_addr learned my addr 172.21.15.23:0/568722052 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.027+0000 7f318affd640 1 --2- 172.21.15.23:0/568722052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0cf00 0x563858f13230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.027+0000 7f318bfff640 1 --2- 172.21.15.23:0/568722052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 0x563858f13770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f318b7fe640 1 -- 172.21.15.23:0/568722052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0cf00 msgr2=0x563858f13230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f318b7fe640 1 --2- 172.21.15.23:0/568722052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0cf00 0x563858f13230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f318b7fe640 1 -- 172.21.15.23:0/568722052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 msgr2=0x563858f13770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f318b7fe640 1 --2- 172.21.15.23:0/568722052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 0x563858f13770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f318b7fe640 1 -- 172.21.15.23:0/568722052 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f318c10ddf0 con 0x563858f0c5e0 2024-03-24T10:45:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f318b7fe640 1 --2- 172.21.15.23:0/568722052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0c5e0 0x563858f12cf0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f317c0cc180 tx=0x7f317c00ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b740f09f3402d5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f316affd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f316affd640 1 -- 172.21.15.23:0/568722052 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c09a780 con 0x563858f0c5e0 2024-03-24T10:45:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f316affd640 1 -- 172.21.15.23:0/568722052 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f317c09b520 con 0x563858f0c5e0 2024-03-24T10:45:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f3194e28740 1 -- 172.21.15.23:0/568722052 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563858e97790 con 0x563858f0c5e0 2024-03-24T10:45:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f316affd640 1 -- 172.21.15.23:0/568722052 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f317c0970c0 con 0x563858f0c5e0 2024-03-24T10:45:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f3194e28740 10 client.17064 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f3194e28740 1 -- 172.21.15.23:0/568722052 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563858e71310 con 0x563858f0c5e0 2024-03-24T10:45:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f316affd640 1 -- 172.21.15.23:0/568722052 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563858e97790 con 0x563858f0c5e0 2024-03-24T10:45:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.028+0000 7f316affd640 1 -- 172.21.15.23:0/568722052 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f317c01f060 con 0x563858f0c5e0 2024-03-24T10:45:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.029+0000 7f3194e28740 10 client.17064 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.029+0000 7f3194e28740 10 client.17064 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.029+0000 7f3194e28740 20 client.17064 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.029+0000 7f3194e28740 1 --2- 172.21.15.23:0/568722052 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563858f25bd0 0x563858f27fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.029+0000 7f3194e28740 4 client.17064 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.029+0000 7f3194e28740 1 -- 172.21.15.23:0/568722052 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563858b72600 con 0x563858f25bd0 2024-03-24T10:45:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.029+0000 7f318affd640 1 --2- 172.21.15.23:0/568722052 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563858f25bd0 0x563858f27fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.029+0000 7f316affd640 10 client.17064 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.029+0000 7f318affd640 1 --2- 172.21.15.23:0/568722052 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563858f25bd0 0x563858f27fc0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c636ae3c46fa6d26 server_cookie=b94006c4cffb8f28 in_seq=0 out_seq=0 2024-03-24T10:45:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f316affd640 1 -- 172.21.15.23:0/568722052 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563858b72600 con 0x563858f25bd0 2024-03-24T10:45:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f316affd640 10 client.17064 handle_command_reply: tid=0 2024-03-24T10:45:18.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f3194e28740 1 client.17064 shutdown 2024-03-24T10:45:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f3194e28740 1 -- 172.21.15.23:0/568722052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0c5e0 msgr2=0x563858f12cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f3194e28740 1 --2- 172.21.15.23:0/568722052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0c5e0 0x563858f12cf0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f317c0cc180 tx=0x7f317c00ef50 comp rx=0 tx=0).stop 2024-03-24T10:45:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f3194e28740 1 -- 172.21.15.23:0/568722052 shutdown_connections 2024-03-24T10:45:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f3194e28740 1 --2- 172.21.15.23:0/568722052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 0x563858f13770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f3194e28740 1 --2- 172.21.15.23:0/568722052 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563858f25bd0 0x563858f27fc0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f3194e28740 1 --2- 172.21.15.23:0/568722052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0cf00 0x563858f13230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f3194e28740 1 --2- 172.21.15.23:0/568722052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0c5e0 0x563858f12cf0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f3194e28740 1 -- 172.21.15.23:0/568722052 >> 172.21.15.23:0/568722052 conn(0x563858f06f00 msgr2=0x563858f08c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f316affd640 0 client.17064 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f3194e28740 1 -- 172.21.15.23:0/568722052 shutdown_connections 2024-03-24T10:45:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f3194e28740 1 -- 172.21.15.23:0/568722052 wait complete. 2024-03-24T10:45:18.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.030+0000 7f3194e28740 20 client.17064 trim_cache size 0 max 16384 2024-03-24T10:45:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f3194e28740 1 Processor -- start 2024-03-24T10:45:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f3194e28740 1 -- start start 2024-03-24T10:45:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f3194e28740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0c5e0 0x563858f0d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f3194e28740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0cf00 0x563858f21340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f3194e28740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 0x563858f05d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f3194e28740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563858df8660 con 0x563858f0cf00 2024-03-24T10:45:18.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f3194e28740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563858f180c0 con 0x563858f15cd0 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f3194e28740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563858f21880 con 0x563858f0c5e0 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f318affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0cf00 0x563858f21340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f318affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0cf00 0x563858f21340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35560/0 (socket says 172.21.15.23:35560) 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f318affd640 1 -- 172.21.15.23:0/3784737918 learned_addr learned my addr 172.21.15.23:0/3784737918 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f318b7fe640 1 --2- 172.21.15.23:0/3784737918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0c5e0 0x563858f0d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f318bfff640 1 --2- 172.21.15.23:0/3784737918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 0x563858f05d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f318affd640 1 -- 172.21.15.23:0/3784737918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0c5e0 msgr2=0x563858f0d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f318affd640 1 --2- 172.21.15.23:0/3784737918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0c5e0 0x563858f0d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f318affd640 1 -- 172.21.15.23:0/3784737918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 msgr2=0x563858f05d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f318affd640 1 --2- 172.21.15.23:0/3784737918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 0x563858f05d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.045+0000 7f318affd640 1 -- 172.21.15.23:0/3784737918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563858e0d810 con 0x563858f0cf00 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.046+0000 7f318affd640 1 --2- 172.21.15.23:0/3784737918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0cf00 0x563858f21340 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x563858f08090 tx=0x7f31800045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8081040f2a63fae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.046+0000 7f31717fa640 1 -- 172.21.15.23:0/3784737918 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3180004740 con 0x563858f0cf00 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.046+0000 7f31717fa640 1 -- 172.21.15.23:0/3784737918 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31800048a0 con 0x563858f0cf00 2024-03-24T10:45:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.046+0000 7f31717fa640 1 -- 172.21.15.23:0/3784737918 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3180005010 con 0x563858f0cf00 2024-03-24T10:45:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.046+0000 7f3194e28740 1 -- 172.21.15.23:0/3784737918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0cf00 msgr2=0x563858f21340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.046+0000 7f3194e28740 1 --2- 172.21.15.23:0/3784737918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0cf00 0x563858f21340 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x563858f08090 tx=0x7f31800045e0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.046+0000 7f3194e28740 1 -- 172.21.15.23:0/3784737918 shutdown_connections 2024-03-24T10:45:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.046+0000 7f3194e28740 1 --2- 172.21.15.23:0/3784737918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 0x563858f05d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.046+0000 7f3194e28740 1 --2- 172.21.15.23:0/3784737918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0cf00 0x563858f21340 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.046+0000 7f3194e28740 1 --2- 172.21.15.23:0/3784737918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0c5e0 0x563858f0d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.046+0000 7f3194e28740 1 -- 172.21.15.23:0/3784737918 >> 172.21.15.23:0/3784737918 conn(0x563858f06f00 msgr2=0x563858f05610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.046+0000 7f3194e28740 1 -- 172.21.15.23:0/3784737918 shutdown_connections 2024-03-24T10:45:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.046+0000 7f3194e28740 1 -- 172.21.15.23:0/3784737918 wait complete. 2024-03-24T10:45:18.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.047+0000 7f3194e28740 1 -- start start 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.047+0000 7f3194e28740 1 Processor -- start 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.047+0000 7f3194e28740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0c5e0 0x563858f1dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.047+0000 7f3194e28740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0cf00 0x563858f0f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.047+0000 7f3194e28740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 0x563858f0fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.047+0000 7f3194e28740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563858f10220 con 0x563858f0c5e0 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.047+0000 7f3194e28740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563858f10390 con 0x563858f15cd0 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.047+0000 7f3194e28740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563858f8f2e0 con 0x563858f0cf00 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.047+0000 7f3194e28740 10 client.0 _ll_register_callbacks cb 0x7f317c00e970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.047+0000 7f318b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0c5e0 0x563858f1dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.047+0000 7f318b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0c5e0 0x563858f1dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35562/0 (socket says 172.21.15.23:35562) 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.047+0000 7f318b7fe640 1 -- 172.21.15.23:0/3623520632 learned_addr learned my addr 172.21.15.23:0/3623520632 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.047+0000 7f318affd640 1 --2- 172.21.15.23:0/3623520632 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0cf00 0x563858f0f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.047+0000 7f318bfff640 1 --2- 172.21.15.23:0/3623520632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 0x563858f0fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f318b7fe640 1 -- 172.21.15.23:0/3623520632 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0cf00 msgr2=0x563858f0f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:18.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f318b7fe640 1 --2- 172.21.15.23:0/3623520632 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0cf00 0x563858f0f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f318b7fe640 1 -- 172.21.15.23:0/3623520632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 msgr2=0x563858f0fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f318b7fe640 1 --2- 172.21.15.23:0/3623520632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 0x563858f0fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f318b7fe640 1 -- 172.21.15.23:0/3623520632 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563858e08810 con 0x563858f0c5e0 2024-03-24T10:45:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f318bfff640 1 --2- 172.21.15.23:0/3623520632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 0x563858f0fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:45:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f318b7fe640 1 --2- 172.21.15.23:0/3623520632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0c5e0 0x563858f1dc60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f317c09adb0 tx=0x7f317c09a520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5bda2cbf34924c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f316affd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f316affd640 1 -- 172.21.15.23:0/3623520632 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c0a3400 con 0x563858f0c5e0 2024-03-24T10:45:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f316affd640 1 -- 172.21.15.23:0/3623520632 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f317c0b9340 con 0x563858f0c5e0 2024-03-24T10:45:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f3194e28740 1 -- 172.21.15.23:0/3623520632 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563858f0db40 con 0x563858f0c5e0 2024-03-24T10:45:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f316affd640 1 -- 172.21.15.23:0/3623520632 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f317c005490 con 0x563858f0c5e0 2024-03-24T10:45:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f3194e28740 10 client.17070 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f3194e28740 1 -- 172.21.15.23:0/3623520632 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563858e335a0 con 0x563858f0c5e0 2024-03-24T10:45:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f316affd640 1 -- 172.21.15.23:0/3623520632 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563858f0db40 con 0x563858f0c5e0 2024-03-24T10:45:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f316affd640 1 -- 172.21.15.23:0/3623520632 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f317c01f040 con 0x563858f0c5e0 2024-03-24T10:45:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f3194e28740 10 client.17070 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f3194e28740 10 client.17070 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f3194e28740 20 client.17070 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f3194e28740 1 --2- 172.21.15.23:0/3623520632 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563858f90510 0x563858f92900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f3194e28740 4 client.17070 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:45:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.048+0000 7f3194e28740 1 -- 172.21.15.23:0/3623520632 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f318002ca70 con 0x563858f90510 2024-03-24T10:45:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.049+0000 7f318affd640 1 --2- 172.21.15.23:0/3623520632 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563858f90510 0x563858f92900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.049+0000 7f318affd640 1 --2- 172.21.15.23:0/3623520632 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563858f90510 0x563858f92900 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b16fabcfef35dd52 server_cookie=6a8818c27004653e in_seq=0 out_seq=0 2024-03-24T10:45:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.049+0000 7f316affd640 10 client.17070 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:18.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f316affd640 1 -- 172.21.15.23:0/3623520632 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f318002ca70 con 0x563858f90510 2024-03-24T10:45:18.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f316affd640 10 client.17070 handle_command_reply: tid=0 2024-03-24T10:45:18.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f3194e28740 1 client.17070 shutdown 2024-03-24T10:45:18.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f3194e28740 1 -- 172.21.15.23:0/3623520632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0c5e0 msgr2=0x563858f1dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f3194e28740 1 --2- 172.21.15.23:0/3623520632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0c5e0 0x563858f1dc60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f317c09adb0 tx=0x7f317c09a520 comp rx=0 tx=0).stop 2024-03-24T10:45:18.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f3194e28740 1 -- 172.21.15.23:0/3623520632 shutdown_connections 2024-03-24T10:45:18.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f3194e28740 1 --2- 172.21.15.23:0/3623520632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563858f15cd0 0x563858f0fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f3194e28740 1 --2- 172.21.15.23:0/3623520632 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563858f90510 0x563858f92900 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f3194e28740 1 --2- 172.21.15.23:0/3623520632 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563858f0cf00 0x563858f0f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f3194e28740 1 --2- 172.21.15.23:0/3623520632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563858f0c5e0 0x563858f1dc60 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f316affd640 0 client.17070 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:18.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f3194e28740 1 -- 172.21.15.23:0/3623520632 >> 172.21.15.23:0/3623520632 conn(0x563858f06f00 msgr2=0x563858f05ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:18.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f3194e28740 1 -- 172.21.15.23:0/3623520632 shutdown_connections 2024-03-24T10:45:18.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f3194e28740 1 -- 172.21.15.23:0/3623520632 wait complete. 2024-03-24T10:45:18.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.059+0000 7f3194e28740 20 client.17070 trim_cache size 0 max 16384 2024-03-24T10:45:18.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:45:18.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:45:18.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:45:18.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:45:18.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.060+0000 7f3192247640 1 -- 172.21.15.23:0/2235443954 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f31740a22d0 msgr2=0x7f31740a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.060+0000 7f3192247640 1 --2- 172.21.15.23:0/2235443954 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f31740a22d0 0x7f31740a4790 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3184005fd0 tx=0x7f3184004a40 comp rx=0 tx=0).stop 2024-03-24T10:45:18.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.060+0000 7f3192247640 1 -- 172.21.15.23:0/2235443954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f318c100860 msgr2=0x7f318c198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.060+0000 7f3192247640 1 --2- 172.21.15.23:0/2235443954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f318c100860 0x7f318c198b30 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f317c00ea20 tx=0x7f317c00ee20 comp rx=0 tx=0).stop 2024-03-24T10:45:18.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.060+0000 7f3192247640 1 -- 172.21.15.23:0/2235443954 shutdown_connections 2024-03-24T10:45:18.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.060+0000 7f3192247640 1 --2- 172.21.15.23:0/2235443954 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f31740a22d0 0x7f31740a4790 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.060+0000 7f3192247640 1 --2- 172.21.15.23:0/2235443954 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f318c1055b0 0x7f318c1a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.060+0000 7f3192247640 1 --2- 172.21.15.23:0/2235443954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c101220 0x7f318c199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.063 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.060+0000 7f3192247640 1 --2- 172.21.15.23:0/2235443954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f318c100860 0x7f318c198b30 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.063 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.060+0000 7f3192247640 1 -- 172.21.15.23:0/2235443954 >> 172.21.15.23:0/2235443954 conn(0x7f318c077460 msgr2=0x7f318c0fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:18.063 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.060+0000 7f3192247640 1 -- 172.21.15.23:0/2235443954 shutdown_connections 2024-03-24T10:45:18.063 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:18.060+0000 7f3192247640 1 -- 172.21.15.23:0/2235443954 wait complete. 2024-03-24T10:45:18.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 481187 11.95 MB/sec execute 95 sec latency 36.951 ms 2024-03-24T10:45:18.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:17 smithi023 ceph-mon[32466]: daemon mds.c restarted 2024-03-24T10:45:19.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:18 smithi117 ceph-mon[36368]: pgmap v159: 97 pgs: 97 active+clean; 275 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 289 op/s 2024-03-24T10:45:19.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:18 smithi117 ceph-mon[36368]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T10:45:19.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:18 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.117:6836/2955742470,v1:172.21.15.117:6837/2955742470] up:boot 2024-03-24T10:45:19.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:18 smithi117 ceph-mon[36368]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T10:45:19.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:18 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:45:19.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:18 smithi117 ceph-mon[36368]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:45:19.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:18 smithi116 ceph-mon[36248]: pgmap v159: 97 pgs: 97 active+clean; 275 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 289 op/s 2024-03-24T10:45:19.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:18 smithi116 ceph-mon[36248]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T10:45:19.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:18 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.117:6836/2955742470,v1:172.21.15.117:6837/2955742470] up:boot 2024-03-24T10:45:19.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:18 smithi116 ceph-mon[36248]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T10:45:19.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:18 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:45:19.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:18 smithi116 ceph-mon[36248]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:45:19.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 483223 11.96 MB/sec execute 96 sec latency 58.844 ms 2024-03-24T10:45:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:18 smithi023 ceph-mon[32466]: pgmap v159: 97 pgs: 97 active+clean; 275 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.8 MiB/s wr, 289 op/s 2024-03-24T10:45:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:18 smithi023 ceph-mon[32466]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T10:45:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:18 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.117:6836/2955742470,v1:172.21.15.117:6837/2955742470] up:boot 2024-03-24T10:45:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:18 smithi023 ceph-mon[32466]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T10:45:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:18 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:45:19.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:18 smithi023 ceph-mon[32466]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:45:20.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 484749 11.90 MB/sec execute 97 sec latency 17.524 ms 2024-03-24T10:45:21.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:20 smithi117 ceph-mon[36368]: pgmap v161: 97 pgs: 97 active+clean; 275 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 228 op/s 2024-03-24T10:45:21.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:20 smithi116 ceph-mon[36248]: pgmap v161: 97 pgs: 97 active+clean; 275 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 228 op/s 2024-03-24T10:45:21.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 487135 11.92 MB/sec execute 98 sec latency 15.264 ms 2024-03-24T10:45:21.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:20 smithi023 ceph-mon[32466]: pgmap v161: 97 pgs: 97 active+clean; 275 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 228 op/s 2024-03-24T10:45:22.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 488853 11.86 MB/sec execute 99 sec latency 24.082 ms 2024-03-24T10:45:23.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:22 smithi117 ceph-mon[36368]: pgmap v162: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 219 op/s 2024-03-24T10:45:23.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:22 smithi116 ceph-mon[36248]: pgmap v162: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 219 op/s 2024-03-24T10:45:23.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 490917 11.88 MB/sec execute 100 sec latency 16.989 ms 2024-03-24T10:45:23.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:22 smithi023 ceph-mon[32466]: pgmap v162: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 219 op/s 2024-03-24T10:45:24.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 492449 11.82 MB/sec execute 101 sec latency 135.525 ms 2024-03-24T10:45:25.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:24 smithi117 ceph-mon[36368]: pgmap v163: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 219 op/s 2024-03-24T10:45:25.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:24 smithi116 ceph-mon[36248]: pgmap v163: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 219 op/s 2024-03-24T10:45:25.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 494518 11.83 MB/sec execute 102 sec latency 17.302 ms 2024-03-24T10:45:25.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:24 smithi023 ceph-mon[32466]: pgmap v163: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 219 op/s 2024-03-24T10:45:26.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 496830 11.84 MB/sec execute 103 sec latency 18.502 ms 2024-03-24T10:45:27.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:26 smithi117 ceph-mon[36368]: pgmap v164: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 219 op/s 2024-03-24T10:45:27.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:26 smithi116 ceph-mon[36248]: pgmap v164: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 219 op/s 2024-03-24T10:45:27.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 498658 11.81 MB/sec execute 104 sec latency 26.505 ms 2024-03-24T10:45:27.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:26 smithi023 ceph-mon[32466]: pgmap v164: 97 pgs: 97 active+clean; 280 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 219 op/s 2024-03-24T10:45:28.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 501154 11.85 MB/sec execute 105 sec latency 17.434 ms 2024-03-24T10:45:29.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:28 smithi117 ceph-mon[36368]: pgmap v165: 97 pgs: 97 active+clean; 287 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 213 op/s 2024-03-24T10:45:29.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:28 smithi116 ceph-mon[36248]: pgmap v165: 97 pgs: 97 active+clean; 287 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 213 op/s 2024-03-24T10:45:29.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 502199 11.77 MB/sec execute 106 sec latency 150.382 ms 2024-03-24T10:45:29.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:28 smithi023 ceph-mon[32466]: pgmap v165: 97 pgs: 97 active+clean; 287 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 213 op/s 2024-03-24T10:45:30.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 504700 11.81 MB/sec execute 107 sec latency 18.324 ms 2024-03-24T10:45:31.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:30 smithi117 ceph-mon[36368]: pgmap v166: 97 pgs: 97 active+clean; 287 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 182 op/s 2024-03-24T10:45:31.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:30 smithi116 ceph-mon[36248]: pgmap v166: 97 pgs: 97 active+clean; 287 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 182 op/s 2024-03-24T10:45:31.120 INFO:tasks.workunit.client.0.smithi023.stdout: 1 506195 11.75 MB/sec execute 108 sec latency 25.442 ms 2024-03-24T10:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.148+0000 7ff8830af640 1 -- 172.21.15.23:0/2099733972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff87c0ffff0 msgr2=0x7ff87c1003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.148+0000 7ff8830af640 1 --2- 172.21.15.23:0/2099733972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff87c0ffff0 0x7ff87c1003d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ff870009930 tx=0x7ff87002bc30 comp rx=0 tx=0).stop 2024-03-24T10:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.148+0000 7ff8830af640 1 -- 172.21.15.23:0/2099733972 shutdown_connections 2024-03-24T10:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.148+0000 7ff8830af640 1 --2- 172.21.15.23:0/2099733972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff87c1050c0 0x7ff87c10cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.148+0000 7ff8830af640 1 --2- 172.21.15.23:0/2099733972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff87c1009a0 0x7ff87c1049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.148+0000 7ff8830af640 1 --2- 172.21.15.23:0/2099733972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff87c0ffff0 0x7ff87c1003d0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.148+0000 7ff8830af640 1 -- 172.21.15.23:0/2099733972 >> 172.21.15.23:0/2099733972 conn(0x7ff87c076b60 msgr2=0x7ff87c0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.148+0000 7ff8830af640 1 -- 172.21.15.23:0/2099733972 shutdown_connections 2024-03-24T10:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.148+0000 7ff8830af640 1 -- 172.21.15.23:0/2099733972 wait complete. 2024-03-24T10:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.149+0000 7ff8830af640 1 Processor -- start 2024-03-24T10:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.149+0000 7ff8830af640 1 -- start start 2024-03-24T10:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.149+0000 7ff8830af640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff87c0ffff0 0x7ff87c199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.149+0000 7ff8830af640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff87c1050c0 0x7ff87c19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.149+0000 7ff8830af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff87c19e7e0 0x7ff87c19ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.149+0000 7ff8830af640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff87c19f1d0 con 0x7ff87c1050c0 2024-03-24T10:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.149+0000 7ff8830af640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff87c19f340 con 0x7ff87c19e7e0 2024-03-24T10:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.149+0000 7ff8830af640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff87c19f4b0 con 0x7ff87c0ffff0 2024-03-24T10:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.149+0000 7ff87bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff87c1050c0 0x7ff87c19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.149+0000 7ff87bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff87c1050c0 0x7ff87c19a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34630/0 (socket says 172.21.15.23:34630) 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.149+0000 7ff88164c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff87c19e7e0 0x7ff87c19ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.149+0000 7ff87bfff640 1 -- 172.21.15.23:0/2071504317 learned_addr learned my addr 172.21.15.23:0/2071504317 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.149+0000 7ff880e4b640 1 --2- 172.21.15.23:0/2071504317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff87c0ffff0 0x7ff87c199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.150+0000 7ff87bfff640 1 -- 172.21.15.23:0/2071504317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff87c0ffff0 msgr2=0x7ff87c199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.150+0000 7ff87bfff640 1 --2- 172.21.15.23:0/2071504317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff87c0ffff0 0x7ff87c199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.150+0000 7ff87bfff640 1 -- 172.21.15.23:0/2071504317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff87c19e7e0 msgr2=0x7ff87c19ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.150+0000 7ff87bfff640 1 --2- 172.21.15.23:0/2071504317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff87c19e7e0 0x7ff87c19ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.150+0000 7ff87bfff640 1 -- 172.21.15.23:0/2071504317 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff870009560 con 0x7ff87c1050c0 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.150+0000 7ff880e4b640 1 --2- 172.21.15.23:0/2071504317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff87c0ffff0 0x7ff87c199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.150+0000 7ff87bfff640 1 --2- 172.21.15.23:0/2071504317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff87c1050c0 0x7ff87c19a460 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff86800e9c0 tx=0x7ff86800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.150+0000 7ff879ffb640 1 -- 172.21.15.23:0/2071504317 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868009700 con 0x7ff87c1050c0 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.150+0000 7ff879ffb640 1 -- 172.21.15.23:0/2071504317 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff86800cd80 con 0x7ff87c1050c0 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.150+0000 7ff8830af640 1 -- 172.21.15.23:0/2071504317 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff87c1a6a90 con 0x7ff87c1050c0 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.150+0000 7ff879ffb640 1 -- 172.21.15.23:0/2071504317 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868010430 con 0x7ff87c1050c0 2024-03-24T10:45:31.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.150+0000 7ff8830af640 1 -- 172.21.15.23:0/2071504317 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff87c1a6f90 con 0x7ff87c1050c0 2024-03-24T10:45:31.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff879ffb640 1 -- 172.21.15.23:0/2071504317 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7ff868005000 con 0x7ff87c1050c0 2024-03-24T10:45:31.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff885d69740 1 Processor -- start 2024-03-24T10:45:31.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff885d69740 1 -- start start 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff879ffb640 1 --2- 172.21.15.23:0/2071504317 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff86c0a2350 0x7ff86c0a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff885d69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883f5e0 0x55d51883f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff885d69740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883ff00 0x55d518848790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff885d69740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d51884b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff885d69740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d5187fa1b0 con 0x55d518848cd0 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff885d69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d5187acf80 con 0x55d51883f5e0 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff885d69740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d5187aa4b0 con 0x55d51883ff00 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff879ffb640 1 -- 172.21.15.23:0/2071504317 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7ff868014070 con 0x7ff87c1050c0 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff88164c640 1 --2- 172.21.15.23:0/2071504317 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff86c0a2350 0x7ff86c0a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff880e4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883f5e0 0x55d51883f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff87bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883ff00 0x55d518848790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff880e4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883f5e0 0x55d51883f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:37750/0 (socket says 172.21.15.23:37750) 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff880e4b640 1 -- 172.21.15.23:0/2431519158 learned_addr learned my addr 172.21.15.23:0/2431519158 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.153+0000 7ff88164c640 1 --2- 172.21.15.23:0/2431519158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d51884b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.154+0000 7ff87bfff640 1 -- 172.21.15.23:0/2431519158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883f5e0 msgr2=0x55d51883f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:31.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.154+0000 7ff87bfff640 1 --2- 172.21.15.23:0/2431519158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883f5e0 0x55d51883f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.154+0000 7ff87bfff640 1 -- 172.21.15.23:0/2431519158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 msgr2=0x55d51884b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.154+0000 7ff87bfff640 1 --2- 172.21.15.23:0/2431519158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d51884b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.154+0000 7ff87bfff640 1 -- 172.21.15.23:0/2431519158 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d51873b810 con 0x55d51883ff00 2024-03-24T10:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.154+0000 7ff88164c640 1 --2- 172.21.15.23:0/2071504317 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff86c0a2350 0x7ff86c0a4810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff87400a380 tx=0x7ff87400aea0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.154+0000 7ff87bfff640 1 --2- 172.21.15.23:0/2431519158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883ff00 0x55d518848790 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff8680cbb50 tx=0x7ff86809a810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c673788a74197cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.154+0000 7ff866ffd640 1 -- 172.21.15.23:0/2431519158 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8680196b0 con 0x55d51883ff00 2024-03-24T10:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.154+0000 7ff866ffd640 1 -- 172.21.15.23:0/2431519158 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff86809b4f0 con 0x55d51883ff00 2024-03-24T10:45:31.156 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.154+0000 7ff866ffd640 1 -- 172.21.15.23:0/2431519158 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8680cbdd0 con 0x55d51883ff00 2024-03-24T10:45:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.155+0000 7ff885d69740 1 -- 172.21.15.23:0/2431519158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883ff00 msgr2=0x55d518848790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.155+0000 7ff885d69740 1 --2- 172.21.15.23:0/2431519158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883ff00 0x55d518848790 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff8680cbb50 tx=0x7ff86809a810 comp rx=0 tx=0).stop 2024-03-24T10:45:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.155+0000 7ff885d69740 1 -- 172.21.15.23:0/2431519158 shutdown_connections 2024-03-24T10:45:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.155+0000 7ff885d69740 1 --2- 172.21.15.23:0/2431519158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d51884b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.155+0000 7ff885d69740 1 --2- 172.21.15.23:0/2431519158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883ff00 0x55d518848790 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.155+0000 7ff885d69740 1 --2- 172.21.15.23:0/2431519158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883f5e0 0x55d51883f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.155+0000 7ff885d69740 1 -- 172.21.15.23:0/2431519158 >> 172.21.15.23:0/2431519158 conn(0x55d518839f00 msgr2=0x55d51883a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.155+0000 7ff885d69740 1 -- 172.21.15.23:0/2431519158 shutdown_connections 2024-03-24T10:45:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.155+0000 7ff885d69740 1 -- 172.21.15.23:0/2431519158 wait complete. 2024-03-24T10:45:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.155+0000 7ff885d69740 1 -- start start 2024-03-24T10:45:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.155+0000 7ff885d69740 1 Processor -- start 2024-03-24T10:45:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff885d69740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883f5e0 0x55d518845d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff885d69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883ff00 0x55d518846290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff885d69740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d5188467d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff885d69740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d5187a5c90 con 0x55d518848cd0 2024-03-24T10:45:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff885d69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d51874ac50 con 0x55d51883ff00 2024-03-24T10:45:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff885d69740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d518479250 con 0x55d51883f5e0 2024-03-24T10:45:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff885d69740 10 client.0 _ll_register_callbacks cb 0x55d518799290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:31.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff88164c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d5188467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff87bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883ff00 0x55d518846290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff880e4b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883f5e0 0x55d518845d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff88164c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d5188467d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34652/0 (socket says 172.21.15.23:34652) 2024-03-24T10:45:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff88164c640 1 -- 172.21.15.23:0/2631765679 learned_addr learned my addr 172.21.15.23:0/2631765679 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff88164c640 1 -- 172.21.15.23:0/2631765679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883f5e0 msgr2=0x55d518845d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff88164c640 1 --2- 172.21.15.23:0/2631765679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883f5e0 0x55d518845d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff88164c640 1 -- 172.21.15.23:0/2631765679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883ff00 msgr2=0x55d518846290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff88164c640 1 --2- 172.21.15.23:0/2631765679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883ff00 0x55d518846290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff88164c640 1 -- 172.21.15.23:0/2631765679 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff87c1a6f90 con 0x55d518848cd0 2024-03-24T10:45:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff87bfff640 1 --2- 172.21.15.23:0/2631765679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883ff00 0x55d518846290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff880e4b640 1 --2- 172.21.15.23:0/2631765679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883f5e0 0x55d518845d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff88164c640 1 --2- 172.21.15.23:0/2631765679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d5188467d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff87400cde0 tx=0x7ff87400a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f3d5605eacca09 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff85bfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff85bfff640 1 -- 172.21.15.23:0/2631765679 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff874023070 con 0x55d518848cd0 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff85bfff640 1 -- 172.21.15.23:0/2631765679 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7ff874004b20 con 0x55d518848cd0 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.156+0000 7ff885d69740 1 -- 172.21.15.23:0/2631765679 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d5187ca790 con 0x55d518848cd0 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.157+0000 7ff85bfff640 1 -- 172.21.15.23:0/2631765679 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff874004ce0 con 0x55d518848cd0 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.157+0000 7ff885d69740 10 client.17082 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.157+0000 7ff885d69740 1 -- 172.21.15.23:0/2631765679 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d5187a4310 con 0x55d518848cd0 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.157+0000 7ff85bfff640 1 -- 172.21.15.23:0/2631765679 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d5187ca790 con 0x55d518848cd0 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.157+0000 7ff85bfff640 1 -- 172.21.15.23:0/2631765679 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7ff8740056a0 con 0x55d518848cd0 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.157+0000 7ff885d69740 10 client.17082 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.157+0000 7ff885d69740 10 client.17082 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.157+0000 7ff885d69740 20 client.17082 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.157+0000 7ff885d69740 1 --2- 172.21.15.23:0/2631765679 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d518858c70 0x55d51885b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.157+0000 7ff885d69740 4 client.17082 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.157+0000 7ff885d69740 1 -- 172.21.15.23:0/2631765679 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5184a5600 con 0x55d518858c70 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.157+0000 7ff880e4b640 1 --2- 172.21.15.23:0/2631765679 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d518858c70 0x55d51885b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.158+0000 7ff880e4b640 1 --2- 172.21.15.23:0/2631765679 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d518858c70 0x55d51885b090 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd60a7a6b3bec150 server_cookie=e3534bf955f81c4a in_seq=0 out_seq=0 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.158+0000 7ff85bfff640 10 client.17082 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.158+0000 7ff85bfff640 1 -- 172.21.15.23:0/2631765679 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d5184a5600 con 0x55d518858c70 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.159+0000 7ff85bfff640 10 client.17082 handle_command_reply: tid=0 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.159+0000 7ff885d69740 1 client.17082 shutdown 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.159+0000 7ff885d69740 1 -- 172.21.15.23:0/2631765679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 msgr2=0x55d5188467d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.159+0000 7ff885d69740 1 --2- 172.21.15.23:0/2631765679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d5188467d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff87400cde0 tx=0x7ff87400a610 comp rx=0 tx=0).stop 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.159+0000 7ff885d69740 1 -- 172.21.15.23:0/2631765679 shutdown_connections 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.159+0000 7ff885d69740 1 --2- 172.21.15.23:0/2631765679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d5188467d0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.159+0000 7ff885d69740 1 --2- 172.21.15.23:0/2631765679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883ff00 0x55d518846290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.159+0000 7ff885d69740 1 --2- 172.21.15.23:0/2631765679 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d518858c70 0x55d51885b090 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.159+0000 7ff885d69740 1 --2- 172.21.15.23:0/2631765679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883f5e0 0x55d518845d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.159+0000 7ff85bfff640 0 client.17082 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.159+0000 7ff885d69740 1 -- 172.21.15.23:0/2631765679 >> 172.21.15.23:0/2631765679 conn(0x55d518839f00 msgr2=0x55d51883bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.159+0000 7ff885d69740 1 -- 172.21.15.23:0/2631765679 shutdown_connections 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.159+0000 7ff885d69740 1 -- 172.21.15.23:0/2631765679 wait complete. 2024-03-24T10:45:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.159+0000 7ff885d69740 20 client.17082 trim_cache size 0 max 16384 2024-03-24T10:45:31.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:30 smithi023 ceph-mon[32466]: pgmap v166: 97 pgs: 97 active+clean; 287 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 182 op/s 2024-03-24T10:45:31.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.174+0000 7ff885d69740 1 Processor -- start 2024-03-24T10:45:31.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.174+0000 7ff885d69740 1 -- start start 2024-03-24T10:45:31.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.174+0000 7ff885d69740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d51883f5e0 0x55d518848480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.174+0000 7ff885d69740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883ff00 0x55d5188543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.174+0000 7ff885d69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d518848cd0 0x55d518860980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.174+0000 7ff885d69740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d51872b660 con 0x55d51883f5e0 2024-03-24T10:45:31.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.174+0000 7ff885d69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d5188383f0 con 0x55d518848cd0 2024-03-24T10:45:31.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.174+0000 7ff885d69740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d5188556a0 con 0x55d51883ff00 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.174+0000 7ff880e4b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d51883f5e0 0x55d518848480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.174+0000 7ff880e4b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d51883f5e0 0x55d518848480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34666/0 (socket says 172.21.15.23:34666) 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.174+0000 7ff880e4b640 1 -- 172.21.15.23:0/668758078 learned_addr learned my addr 172.21.15.23:0/668758078 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.174+0000 7ff87bfff640 1 --2- 172.21.15.23:0/668758078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883ff00 0x55d5188543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.174+0000 7ff88164c640 1 --2- 172.21.15.23:0/668758078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d518848cd0 0x55d518860980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.175+0000 7ff880e4b640 1 -- 172.21.15.23:0/668758078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883ff00 msgr2=0x55d5188543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.175+0000 7ff880e4b640 1 --2- 172.21.15.23:0/668758078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883ff00 0x55d5188543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.175+0000 7ff880e4b640 1 -- 172.21.15.23:0/668758078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d518848cd0 msgr2=0x55d518860980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.175+0000 7ff880e4b640 1 --2- 172.21.15.23:0/668758078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d518848cd0 0x55d518860980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.175+0000 7ff880e4b640 1 -- 172.21.15.23:0/668758078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d518740810 con 0x55d51883f5e0 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.175+0000 7ff88164c640 1 --2- 172.21.15.23:0/668758078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d518848cd0 0x55d518860980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.175+0000 7ff880e4b640 1 --2- 172.21.15.23:0/668758078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d51883f5e0 0x55d518848480 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ff870000c00 tx=0x7ff870009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=281c41028d6fbeaa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.175+0000 7ff8667fc640 1 -- 172.21.15.23:0/668758078 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87002c5d0 con 0x55d51883f5e0 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.175+0000 7ff8667fc640 1 -- 172.21.15.23:0/668758078 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff870004040 con 0x55d51883f5e0 2024-03-24T10:45:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.175+0000 7ff8667fc640 1 -- 172.21.15.23:0/668758078 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff870004df0 con 0x55d51883f5e0 2024-03-24T10:45:31.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.176+0000 7ff885d69740 1 -- 172.21.15.23:0/668758078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d51883f5e0 msgr2=0x55d518848480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:31.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.176+0000 7ff885d69740 1 --2- 172.21.15.23:0/668758078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d51883f5e0 0x55d518848480 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ff870000c00 tx=0x7ff870009260 comp rx=0 tx=0).stop 2024-03-24T10:45:31.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.176+0000 7ff885d69740 1 -- 172.21.15.23:0/668758078 shutdown_connections 2024-03-24T10:45:31.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.176+0000 7ff885d69740 1 --2- 172.21.15.23:0/668758078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d518848cd0 0x55d518860980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.176+0000 7ff885d69740 1 --2- 172.21.15.23:0/668758078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883ff00 0x55d5188543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.176+0000 7ff885d69740 1 --2- 172.21.15.23:0/668758078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d51883f5e0 0x55d518848480 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.176+0000 7ff885d69740 1 -- 172.21.15.23:0/668758078 >> 172.21.15.23:0/668758078 conn(0x55d518839f00 msgr2=0x55d5188410f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:31.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.176+0000 7ff885d69740 1 -- 172.21.15.23:0/668758078 shutdown_connections 2024-03-24T10:45:31.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.176+0000 7ff885d69740 1 -- 172.21.15.23:0/668758078 wait complete. 2024-03-24T10:45:31.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.176+0000 7ff885d69740 1 -- start start 2024-03-24T10:45:31.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.176+0000 7ff885d69740 1 Processor -- start 2024-03-24T10:45:31.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.176+0000 7ff885d69740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883f5e0 0x55d518843040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.176+0000 7ff885d69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883ff00 0x55d518843580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff885d69740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d518841fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff885d69740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d518843ac0 con 0x55d518848cd0 2024-03-24T10:45:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff885d69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d518843c30 con 0x55d51883ff00 2024-03-24T10:45:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff885d69740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d518843df0 con 0x55d51883f5e0 2024-03-24T10:45:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff885d69740 10 client.0 _ll_register_callbacks cb 0x7ff87400e650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff88164c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d518841fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff88164c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d518841fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34678/0 (socket says 172.21.15.23:34678) 2024-03-24T10:45:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff88164c640 1 -- 172.21.15.23:0/519939808 learned_addr learned my addr 172.21.15.23:0/519939808 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff87bfff640 1 --2- 172.21.15.23:0/519939808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883ff00 0x55d518843580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff880e4b640 1 --2- 172.21.15.23:0/519939808 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883f5e0 0x55d518843040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff88164c640 1 -- 172.21.15.23:0/519939808 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883f5e0 msgr2=0x55d518843040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff88164c640 1 --2- 172.21.15.23:0/519939808 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883f5e0 0x55d518843040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff88164c640 1 -- 172.21.15.23:0/519939808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883ff00 msgr2=0x55d518843580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff88164c640 1 --2- 172.21.15.23:0/519939808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883ff00 0x55d518843580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff88164c640 1 -- 172.21.15.23:0/519939808 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff87c10d4a0 con 0x55d518848cd0 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff88164c640 1 --2- 172.21.15.23:0/519939808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d518841fc0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ff87400cde0 tx=0x7ff874027610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=404cf57cf7def40a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff85bfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff85bfff640 1 -- 172.21.15.23:0/519939808 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff874023040 con 0x55d518848cd0 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff85bfff640 1 -- 172.21.15.23:0/519939808 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7ff87400e920 con 0x55d518848cd0 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff885d69740 1 -- 172.21.15.23:0/519939808 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff86809b4f0 con 0x55d518848cd0 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff85bfff640 1 -- 172.21.15.23:0/519939808 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff874004950 con 0x55d518848cd0 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff885d69740 10 client.17088 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.177+0000 7ff885d69740 1 -- 172.21.15.23:0/519939808 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d5187665a0 con 0x55d518848cd0 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.178+0000 7ff85bfff640 1 -- 172.21.15.23:0/519939808 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff86809b4f0 con 0x55d518848cd0 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.178+0000 7ff85bfff640 1 -- 172.21.15.23:0/519939808 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7ff8740309f0 con 0x55d518848cd0 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.178+0000 7ff885d69740 10 client.17088 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.178+0000 7ff885d69740 10 client.17088 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.178+0000 7ff885d69740 20 client.17088 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:31.182 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.178+0000 7ff885d69740 1 --2- 172.21.15.23:0/519939808 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d5188be930 0x55d5188c0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:31.183 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.178+0000 7ff885d69740 4 client.17088 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:45:31.183 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.178+0000 7ff885d69740 1 -- 172.21.15.23:0/519939808 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d5184f61b0 con 0x55d5188be930 2024-03-24T10:45:31.183 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.178+0000 7ff880e4b640 1 --2- 172.21.15.23:0/519939808 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d5188be930 0x55d5188c0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:31.183 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.178+0000 7ff85bfff640 10 client.17088 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:31.183 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.178+0000 7ff880e4b640 1 --2- 172.21.15.23:0/519939808 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d5188be930 0x55d5188c0d50 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edbf46041a220f29 server_cookie=d75c95c4216a040d in_seq=0 out_seq=0 2024-03-24T10:45:31.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.188+0000 7ff85bfff640 1 -- 172.21.15.23:0/519939808 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d5184f61b0 con 0x55d5188be930 2024-03-24T10:45:31.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.188+0000 7ff85bfff640 10 client.17088 handle_command_reply: tid=0 2024-03-24T10:45:31.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.188+0000 7ff885d69740 1 client.17088 shutdown 2024-03-24T10:45:31.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.188+0000 7ff885d69740 1 -- 172.21.15.23:0/519939808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 msgr2=0x55d518841fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:31.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.188+0000 7ff885d69740 1 --2- 172.21.15.23:0/519939808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d518841fc0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ff87400cde0 tx=0x7ff874027610 comp rx=0 tx=0).stop 2024-03-24T10:45:31.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.188+0000 7ff885d69740 1 -- 172.21.15.23:0/519939808 shutdown_connections 2024-03-24T10:45:31.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.188+0000 7ff885d69740 1 --2- 172.21.15.23:0/519939808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d518848cd0 0x55d518841fc0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.188+0000 7ff885d69740 1 --2- 172.21.15.23:0/519939808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d51883ff00 0x55d518843580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.188+0000 7ff885d69740 1 --2- 172.21.15.23:0/519939808 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d5188be930 0x55d5188c0d50 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.188+0000 7ff885d69740 1 --2- 172.21.15.23:0/519939808 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d51883f5e0 0x55d518843040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.188+0000 7ff885d69740 1 -- 172.21.15.23:0/519939808 >> 172.21.15.23:0/519939808 conn(0x55d518839f00 msgr2=0x55d5188445e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:31.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.188+0000 7ff85bfff640 0 client.17088 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:31.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.188+0000 7ff885d69740 1 -- 172.21.15.23:0/519939808 shutdown_connections 2024-03-24T10:45:31.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.188+0000 7ff885d69740 1 -- 172.21.15.23:0/519939808 wait complete. 2024-03-24T10:45:31.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.189+0000 7ff885d69740 20 client.17088 trim_cache size 0 max 16384 2024-03-24T10:45:31.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:45:31.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:45:31.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:45:31.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.189+0000 7ff8830af640 1 -- 172.21.15.23:0/2071504317 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff86c0a2350 msgr2=0x7ff86c0a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.189+0000 7ff8830af640 1 --2- 172.21.15.23:0/2071504317 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff86c0a2350 0x7ff86c0a4810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff87400a380 tx=0x7ff87400aea0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.189+0000 7ff8830af640 1 -- 172.21.15.23:0/2071504317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff87c1050c0 msgr2=0x7ff87c19a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.189+0000 7ff8830af640 1 --2- 172.21.15.23:0/2071504317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff87c1050c0 0x7ff87c19a460 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff86800e9c0 tx=0x7ff86800ee80 comp rx=0 tx=0).stop 2024-03-24T10:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.189+0000 7ff8830af640 1 -- 172.21.15.23:0/2071504317 shutdown_connections 2024-03-24T10:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.189+0000 7ff8830af640 1 --2- 172.21.15.23:0/2071504317 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff86c0a2350 0x7ff86c0a4810 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.189+0000 7ff8830af640 1 --2- 172.21.15.23:0/2071504317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff87c19e7e0 0x7ff87c19ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.189+0000 7ff8830af640 1 --2- 172.21.15.23:0/2071504317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff87c1050c0 0x7ff87c19a460 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.189+0000 7ff8830af640 1 --2- 172.21.15.23:0/2071504317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff87c0ffff0 0x7ff87c199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:31.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.189+0000 7ff8830af640 1 -- 172.21.15.23:0/2071504317 >> 172.21.15.23:0/2071504317 conn(0x7ff87c076b60 msgr2=0x7ff87c1047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.189+0000 7ff8830af640 1 -- 172.21.15.23:0/2071504317 shutdown_connections 2024-03-24T10:45:31.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:31.189+0000 7ff8830af640 1 -- 172.21.15.23:0/2071504317 wait complete. 2024-03-24T10:45:32.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 508574 11.77 MB/sec execute 109 sec latency 17.670 ms 2024-03-24T10:45:33.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:32 smithi117 ceph-mon[36368]: pgmap v167: 97 pgs: 97 active+clean; 293 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 266 op/s 2024-03-24T10:45:33.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:32 smithi116 ceph-mon[36248]: pgmap v167: 97 pgs: 97 active+clean; 293 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 266 op/s 2024-03-24T10:45:33.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 510503 11.74 MB/sec execute 110 sec latency 18.389 ms 2024-03-24T10:45:33.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:32 smithi023 ceph-mon[32466]: pgmap v167: 97 pgs: 97 active+clean; 293 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 266 op/s 2024-03-24T10:45:34.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 512200 11.73 MB/sec execute 111 sec latency 75.519 ms 2024-03-24T10:45:35.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:34 smithi117 ceph-mon[36368]: pgmap v168: 97 pgs: 97 active+clean; 293 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 176 op/s 2024-03-24T10:45:35.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:34 smithi116 ceph-mon[36248]: pgmap v168: 97 pgs: 97 active+clean; 293 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 176 op/s 2024-03-24T10:45:35.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 514327 11.73 MB/sec execute 112 sec latency 28.153 ms 2024-03-24T10:45:35.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:34 smithi023 ceph-mon[32466]: pgmap v168: 97 pgs: 97 active+clean; 293 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 176 op/s 2024-03-24T10:45:36.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 516193 11.71 MB/sec execute 113 sec latency 18.901 ms 2024-03-24T10:45:37.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:36 smithi117 ceph-mon[36368]: pgmap v169: 97 pgs: 97 active+clean; 293 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 176 op/s 2024-03-24T10:45:37.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:36 smithi116 ceph-mon[36248]: pgmap v169: 97 pgs: 97 active+clean; 293 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 176 op/s 2024-03-24T10:45:37.100 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:45:37.100 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:45:37.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 518665 11.75 MB/sec execute 114 sec latency 19.408 ms 2024-03-24T10:45:37.124 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:36 smithi023 ceph-mon[32466]: pgmap v169: 97 pgs: 97 active+clean; 293 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 176 op/s 2024-03-24T10:45:37.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.211+0000 7f76b1d19640 1 -- 172.21.15.23:0/2690902951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76ac106660 msgr2=0x7f76ac106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.211+0000 7f76b1d19640 1 --2- 172.21.15.23:0/2690902951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76ac106660 0x7f76ac106a40 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f769c009900 tx=0x7f769c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:45:37.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.211+0000 7f76b1d19640 1 -- 172.21.15.23:0/2690902951 shutdown_connections 2024-03-24T10:45:37.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.211+0000 7f76b1d19640 1 --2- 172.21.15.23:0/2690902951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76ac100f60 0x7f76ac10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.211+0000 7f76b1d19640 1 --2- 172.21.15.23:0/2690902951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76ac1005a0 0x7f76ac100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.211+0000 7f76b1d19640 1 --2- 172.21.15.23:0/2690902951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76ac106660 0x7f76ac106a40 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.211+0000 7f76b1d19640 1 -- 172.21.15.23:0/2690902951 >> 172.21.15.23:0/2690902951 conn(0x7f76ac077710 msgr2=0x7f76ac077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.211+0000 7f76b1d19640 1 -- 172.21.15.23:0/2690902951 shutdown_connections 2024-03-24T10:45:37.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.211+0000 7f76b1d19640 1 -- 172.21.15.23:0/2690902951 wait complete. 2024-03-24T10:45:37.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.212+0000 7f76b1d19640 1 Processor -- start 2024-03-24T10:45:37.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.212+0000 7f76b1d19640 1 -- start start 2024-03-24T10:45:37.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.212+0000 7f76b1d19640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76ac1005a0 0x7f76ac19d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.212+0000 7f76b1d19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76ac100f60 0x7f76ac19dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76b1d19640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76ac106660 0x7f76ac197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76b1d19640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f76ac1980a0 con 0x7f76ac106660 2024-03-24T10:45:37.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76b1d19640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f76ac198210 con 0x7f76ac100f60 2024-03-24T10:45:37.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76b1d19640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f76ac1983d0 con 0x7f76ac1005a0 2024-03-24T10:45:37.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76aaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76ac100f60 0x7f76ac19dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76abfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76ac106660 0x7f76ac197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76aaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76ac100f60 0x7f76ac19dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:37786/0 (socket says 172.21.15.23:37786) 2024-03-24T10:45:37.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76aaffd640 1 -- 172.21.15.23:0/1567378498 learned_addr learned my addr 172.21.15.23:0/1567378498 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76ab7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76ac1005a0 0x7f76ac19d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76aaffd640 1 -- 172.21.15.23:0/1567378498 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76ac1005a0 msgr2=0x7f76ac19d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76aaffd640 1 --2- 172.21.15.23:0/1567378498 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76ac1005a0 0x7f76ac19d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76aaffd640 1 -- 172.21.15.23:0/1567378498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76ac106660 msgr2=0x7f76ac197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76aaffd640 1 --2- 172.21.15.23:0/1567378498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76ac106660 0x7f76ac197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76aaffd640 1 -- 172.21.15.23:0/1567378498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f769c009560 con 0x7f76ac100f60 2024-03-24T10:45:37.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76abfff640 1 --2- 172.21.15.23:0/1567378498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76ac106660 0x7f76ac197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:37.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76aaffd640 1 --2- 172.21.15.23:0/1567378498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76ac100f60 0x7f76ac19dde0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f76a000e9f0 tx=0x7f76a000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76a8ff9640 1 -- 172.21.15.23:0/1567378498 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a0009700 con 0x7f76ac100f60 2024-03-24T10:45:37.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.213+0000 7f76b1d19640 1 -- 172.21.15.23:0/1567378498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76ac1986b0 con 0x7f76ac100f60 2024-03-24T10:45:37.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.214+0000 7f76a8ff9640 1 -- 172.21.15.23:0/1567378498 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76a000cd80 con 0x7f76ac100f60 2024-03-24T10:45:37.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.214+0000 7f76a8ff9640 1 -- 172.21.15.23:0/1567378498 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a0018920 con 0x7f76ac100f60 2024-03-24T10:45:37.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.214+0000 7f76b1d19640 1 -- 172.21.15.23:0/1567378498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76ac1a4940 con 0x7f76ac100f60 2024-03-24T10:45:37.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.216+0000 7f76b49cb740 1 Processor -- start 2024-03-24T10:45:37.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.216+0000 7f76b49cb740 1 -- start start 2024-03-24T10:45:37.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76a8ff9640 1 -- 172.21.15.23:0/1567378498 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f76a0018a80 con 0x7f76ac100f60 2024-03-24T10:45:37.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76b49cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058a1d0 0x55f68058a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76b49cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058aaf0 0x55f680593190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76b49cb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f680595ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76b49cb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f6804aef30 con 0x55f6805936d0 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76b49cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f6804f8810 con 0x55f68058a1d0 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76b49cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f6804f5d40 con 0x55f68058aaf0 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76aaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f680595ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76ab7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058aaf0 0x55f680593190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76aaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f680595ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34704/0 (socket says 172.21.15.23:34704) 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76aaffd640 1 -- 172.21.15.23:0/3170878868 learned_addr learned my addr 172.21.15.23:0/3170878868 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76abfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058a1d0 0x55f68058a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76aaffd640 1 -- 172.21.15.23:0/3170878868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058aaf0 msgr2=0x55f680593190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76aaffd640 1 --2- 172.21.15.23:0/3170878868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058aaf0 0x55f680593190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76aaffd640 1 -- 172.21.15.23:0/3170878868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058a1d0 msgr2=0x55f68058a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76aaffd640 1 --2- 172.21.15.23:0/3170878868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058a1d0 0x55f68058a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76a8ff9640 1 --2- 172.21.15.23:0/1567378498 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f76940a2230 0x7f76940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76aaffd640 1 -- 172.21.15.23:0/3170878868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6804b2ea0 con 0x55f6805936d0 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76ab7fe640 1 --2- 172.21.15.23:0/3170878868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058aaf0 0x55f680593190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:37.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76abfff640 1 --2- 172.21.15.23:0/3170878868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058a1d0 0x55f68058a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:37.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76abfff640 1 --2- 172.21.15.23:0/1567378498 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f76940a2230 0x7f76940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76a8ff9640 1 -- 172.21.15.23:0/1567378498 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f76a0014070 con 0x7f76ac100f60 2024-03-24T10:45:37.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.217+0000 7f76aaffd640 1 --2- 172.21.15.23:0/3170878868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f680595ac0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f76a00cbae0 tx=0x7f76a00d66c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529978936911b076 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.218+0000 7f7691ffb640 1 -- 172.21.15.23:0/3170878868 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a00d1880 con 0x55f6805936d0 2024-03-24T10:45:37.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.218+0000 7f7691ffb640 1 -- 172.21.15.23:0/3170878868 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76a009ac90 con 0x55f6805936d0 2024-03-24T10:45:37.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.218+0000 7f7691ffb640 1 -- 172.21.15.23:0/3170878868 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a00a35b0 con 0x55f6805936d0 2024-03-24T10:45:37.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.218+0000 7f76b49cb740 1 -- 172.21.15.23:0/3170878868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 msgr2=0x55f680595ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.218+0000 7f76abfff640 1 --2- 172.21.15.23:0/1567378498 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f76940a2230 0x7f76940a46f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f76a4009fd0 tx=0x7f76a4009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.218+0000 7f76b49cb740 1 --2- 172.21.15.23:0/3170878868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f680595ac0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f76a00cbae0 tx=0x7f76a00d66c0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.218+0000 7f76b49cb740 1 -- 172.21.15.23:0/3170878868 shutdown_connections 2024-03-24T10:45:37.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.218+0000 7f76b49cb740 1 --2- 172.21.15.23:0/3170878868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f680595ac0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.218+0000 7f76b49cb740 1 --2- 172.21.15.23:0/3170878868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058aaf0 0x55f680593190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.218+0000 7f76b49cb740 1 --2- 172.21.15.23:0/3170878868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058a1d0 0x55f68058a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.218+0000 7f76b49cb740 1 -- 172.21.15.23:0/3170878868 >> 172.21.15.23:0/3170878868 conn(0x55f680584900 msgr2=0x55f680584ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.218+0000 7f76b49cb740 1 -- 172.21.15.23:0/3170878868 shutdown_connections 2024-03-24T10:45:37.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.218+0000 7f76b49cb740 1 -- 172.21.15.23:0/3170878868 wait complete. 2024-03-24T10:45:37.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.219+0000 7f76b49cb740 1 -- start start 2024-03-24T10:45:37.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.219+0000 7f76b49cb740 1 Processor -- start 2024-03-24T10:45:37.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.219+0000 7f76b49cb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f68058a1d0 0x55f680590760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.219+0000 7f76b49cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 0x55f680590ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.219+0000 7f76b49cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6805936d0 0x55f6805911e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.219+0000 7f76b49cb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f6804f1520 con 0x55f68058a1d0 2024-03-24T10:45:37.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.219+0000 7f76b49cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f6801c4430 con 0x55f68058aaf0 2024-03-24T10:45:37.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.219+0000 7f76b49cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f680180e00 con 0x55f6805936d0 2024-03-24T10:45:37.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.219+0000 7f76b49cb740 10 client.0 _ll_register_callbacks cb 0x55f68048fd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:37.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.219+0000 7f76ab7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f68058a1d0 0x55f680590760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f76ab7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f68058a1d0 0x55f680590760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34708/0 (socket says 172.21.15.23:34708) 2024-03-24T10:45:37.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f76abfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6805936d0 0x55f6805911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f76aaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 0x55f680590ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f76ab7fe640 1 -- 172.21.15.23:0/2497373807 learned_addr learned my addr 172.21.15.23:0/2497373807 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f76ab7fe640 1 -- 172.21.15.23:0/2497373807 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6805936d0 msgr2=0x55f6805911e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f76ab7fe640 1 --2- 172.21.15.23:0/2497373807 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6805936d0 0x55f6805911e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f76ab7fe640 1 -- 172.21.15.23:0/2497373807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 msgr2=0x55f680590ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f76ab7fe640 1 --2- 172.21.15.23:0/2497373807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 0x55f680590ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f76ab7fe640 1 -- 172.21.15.23:0/2497373807 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76ac1a4940 con 0x55f68058a1d0 2024-03-24T10:45:37.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f76ab7fe640 1 --2- 172.21.15.23:0/2497373807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f68058a1d0 0x55f680590760 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f769c009900 tx=0x7f769c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7428b644690b7cb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f768affd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:37.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f768affd640 1 -- 172.21.15.23:0/2497373807 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769c004040 con 0x55f68058a1d0 2024-03-24T10:45:37.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f768affd640 1 -- 172.21.15.23:0/2497373807 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f769c0041a0 con 0x55f68058a1d0 2024-03-24T10:45:37.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f76b49cb740 1 -- 172.21.15.23:0/2497373807 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f6801f07e0 con 0x55f68058a1d0 2024-03-24T10:45:37.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f768affd640 1 -- 172.21.15.23:0/2497373807 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f769c004360 con 0x55f68058a1d0 2024-03-24T10:45:37.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f76b49cb740 10 client.17100 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:37.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.220+0000 7f76b49cb740 1 -- 172.21.15.23:0/2497373807 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f6804efba0 con 0x55f68058a1d0 2024-03-24T10:45:37.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.221+0000 7f768affd640 1 -- 172.21.15.23:0/2497373807 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f6801f07e0 con 0x55f68058a1d0 2024-03-24T10:45:37.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.221+0000 7f768affd640 1 -- 172.21.15.23:0/2497373807 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f769c03c040 con 0x55f68058a1d0 2024-03-24T10:45:37.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.221+0000 7f76b49cb740 10 client.17100 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:37.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.221+0000 7f76b49cb740 10 client.17100 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.221+0000 7f76b49cb740 20 client.17100 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.221+0000 7f76b49cb740 1 --2- 172.21.15.23:0/2497373807 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f6805a7f30 0x55f6805aa370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.221+0000 7f76b49cb740 4 client.17100 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.221+0000 7f76b49cb740 1 -- 172.21.15.23:0/2497373807 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f68022d720 con 0x55f6805a7f30 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.221+0000 7f76aaffd640 1 --2- 172.21.15.23:0/2497373807 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f6805a7f30 0x55f6805aa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.221+0000 7f76aaffd640 1 --2- 172.21.15.23:0/2497373807 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f6805a7f30 0x55f6805aa370 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8514fd7435fd995 server_cookie=97877ce3a2152ce2 in_seq=0 out_seq=0 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.221+0000 7f768affd640 10 client.17100 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f768affd640 1 -- 172.21.15.23:0/2497373807 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f68022d720 con 0x55f6805a7f30 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f768affd640 10 client.17100 handle_command_reply: tid=0 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f76b49cb740 1 client.17100 shutdown 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f76b49cb740 1 -- 172.21.15.23:0/2497373807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f68058a1d0 msgr2=0x55f680590760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f76b49cb740 1 --2- 172.21.15.23:0/2497373807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f68058a1d0 0x55f680590760 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f769c009900 tx=0x7f769c0048f0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f76b49cb740 1 -- 172.21.15.23:0/2497373807 shutdown_connections 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f76b49cb740 1 --2- 172.21.15.23:0/2497373807 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f6805a7f30 0x55f6805aa370 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f76b49cb740 1 --2- 172.21.15.23:0/2497373807 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6805936d0 0x55f6805911e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f76b49cb740 1 --2- 172.21.15.23:0/2497373807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 0x55f680590ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f768affd640 0 client.17100 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:37.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f76b49cb740 1 --2- 172.21.15.23:0/2497373807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f68058a1d0 0x55f680590760 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f76b49cb740 1 -- 172.21.15.23:0/2497373807 >> 172.21.15.23:0/2497373807 conn(0x55f680584900 msgr2=0x55f680583b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f76b49cb740 1 -- 172.21.15.23:0/2497373807 shutdown_connections 2024-03-24T10:45:37.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f76b49cb740 1 -- 172.21.15.23:0/2497373807 wait complete. 2024-03-24T10:45:37.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.222+0000 7f76b49cb740 20 client.17100 trim_cache size 0 max 16384 2024-03-24T10:45:37.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.242+0000 7f76b49cb740 1 Processor -- start 2024-03-24T10:45:37.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.242+0000 7f76b49cb740 1 -- start start 2024-03-24T10:45:37.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.242+0000 7f76b49cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058a1d0 0x55f680592e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.242+0000 7f76b49cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 0x55f680583b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.242+0000 7f76b49cb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f680591000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.242+0000 7f76b49cb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f680476430 con 0x55f6805936d0 2024-03-24T10:45:37.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.242+0000 7f76b49cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f680580550 con 0x55f68058aaf0 2024-03-24T10:45:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.242+0000 7f76b49cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f68058ecb0 con 0x55f68058a1d0 2024-03-24T10:45:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.242+0000 7f76abfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f680591000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.242+0000 7f76ab7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058a1d0 0x55f680592e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.242+0000 7f76aaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 0x55f680583b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.242+0000 7f76ab7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058a1d0 0x55f680592e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:40444/0 (socket says 172.21.15.23:40444) 2024-03-24T10:45:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.242+0000 7f76abfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f680591000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34716/0 (socket says 172.21.15.23:34716) 2024-03-24T10:45:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.242+0000 7f76ab7fe640 1 -- 172.21.15.23:0/1220760885 learned_addr learned my addr 172.21.15.23:0/1220760885 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.243+0000 7f76abfff640 1 -- 172.21.15.23:0/1220760885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058a1d0 msgr2=0x55f680592e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.243+0000 7f76abfff640 1 --2- 172.21.15.23:0/1220760885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058a1d0 0x55f680592e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.243+0000 7f76abfff640 1 -- 172.21.15.23:0/1220760885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 msgr2=0x55f680583b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.243+0000 7f76abfff640 1 --2- 172.21.15.23:0/1220760885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 0x55f680583b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.243+0000 7f76abfff640 1 -- 172.21.15.23:0/1220760885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f68048bb20 con 0x55f6805936d0 2024-03-24T10:45:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.243+0000 7f76ab7fe640 1 --2- 172.21.15.23:0/1220760885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058a1d0 0x55f680592e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.243+0000 7f76aaffd640 1 --2- 172.21.15.23:0/1220760885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 0x55f680583b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:37.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.243+0000 7f76abfff640 1 --2- 172.21.15.23:0/1220760885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f680591000 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f76a4011510 tx=0x7f76a401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a9232c17e77088 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.243+0000 7f76917fa640 1 -- 172.21.15.23:0/1220760885 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a4024040 con 0x55f6805936d0 2024-03-24T10:45:37.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.243+0000 7f76917fa640 1 -- 172.21.15.23:0/1220760885 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76a401e5b0 con 0x55f6805936d0 2024-03-24T10:45:37.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.243+0000 7f76917fa640 1 -- 172.21.15.23:0/1220760885 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a4016070 con 0x55f6805936d0 2024-03-24T10:45:37.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.244+0000 7f76b49cb740 1 -- 172.21.15.23:0/1220760885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 msgr2=0x55f680591000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.244+0000 7f76b49cb740 1 --2- 172.21.15.23:0/1220760885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f680591000 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f76a4011510 tx=0x7f76a401a000 comp rx=0 tx=0).stop 2024-03-24T10:45:37.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.244+0000 7f76b49cb740 1 -- 172.21.15.23:0/1220760885 shutdown_connections 2024-03-24T10:45:37.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.244+0000 7f76b49cb740 1 --2- 172.21.15.23:0/1220760885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f680591000 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.244+0000 7f76b49cb740 1 --2- 172.21.15.23:0/1220760885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 0x55f680583b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.244+0000 7f76b49cb740 1 --2- 172.21.15.23:0/1220760885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058a1d0 0x55f680592e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.244+0000 7f76b49cb740 1 -- 172.21.15.23:0/1220760885 >> 172.21.15.23:0/1220760885 conn(0x55f680584900 msgr2=0x55f680585b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.244+0000 7f76b49cb740 1 -- 172.21.15.23:0/1220760885 shutdown_connections 2024-03-24T10:45:37.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.244+0000 7f76b49cb740 1 -- 172.21.15.23:0/1220760885 wait complete. 2024-03-24T10:45:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.244+0000 7f76b49cb740 1 -- start start 2024-03-24T10:45:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.244+0000 7f76b49cb740 1 Processor -- start 2024-03-24T10:45:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76b49cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058a1d0 0x55f68059fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76b49cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 0x55f68059bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76b49cb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f68059c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76b49cb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f68059cb80 con 0x55f6805936d0 2024-03-24T10:45:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76b49cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f68059ccf0 con 0x55f68058aaf0 2024-03-24T10:45:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76b49cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f68060e2c0 con 0x55f68058a1d0 2024-03-24T10:45:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76b49cb740 10 client.0 _ll_register_callbacks cb 0x7f769c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76abfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f68059c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76abfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f68059c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34728/0 (socket says 172.21.15.23:34728) 2024-03-24T10:45:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76abfff640 1 -- 172.21.15.23:0/2791486022 learned_addr learned my addr 172.21.15.23:0/2791486022 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76ab7fe640 1 --2- 172.21.15.23:0/2791486022 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058a1d0 0x55f68059fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76aaffd640 1 --2- 172.21.15.23:0/2791486022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 0x55f68059bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76abfff640 1 -- 172.21.15.23:0/2791486022 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058a1d0 msgr2=0x55f68059fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76abfff640 1 --2- 172.21.15.23:0/2791486022 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058a1d0 0x55f68059fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76abfff640 1 -- 172.21.15.23:0/2791486022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 msgr2=0x55f68059bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76abfff640 1 --2- 172.21.15.23:0/2791486022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 0x55f68059bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76abfff640 1 -- 172.21.15.23:0/2791486022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6804efba0 con 0x55f6805936d0 2024-03-24T10:45:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76abfff640 1 --2- 172.21.15.23:0/2791486022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f68059c500 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f76a401ee20 tx=0x7f76a401e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46e28a7a6a8d0e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f768affd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f768affd640 1 -- 172.21.15.23:0/2791486022 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a4016040 con 0x55f6805936d0 2024-03-24T10:45:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f768affd640 1 -- 172.21.15.23:0/2791486022 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f76a4004c40 con 0x55f6805936d0 2024-03-24T10:45:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.245+0000 7f76b49cb740 1 -- 172.21.15.23:0/2791486022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76a009ac90 con 0x55f6805936d0 2024-03-24T10:45:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.246+0000 7f768affd640 1 -- 172.21.15.23:0/2791486022 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f76a4004e00 con 0x55f6805936d0 2024-03-24T10:45:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.246+0000 7f76b49cb740 10 client.17106 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.246+0000 7f76b49cb740 1 -- 172.21.15.23:0/2791486022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f6804b2ea0 con 0x55f6805936d0 2024-03-24T10:45:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.246+0000 7f768affd640 1 -- 172.21.15.23:0/2791486022 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76a009ac90 con 0x55f6805936d0 2024-03-24T10:45:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.246+0000 7f768affd640 1 -- 172.21.15.23:0/2791486022 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f76a40285c0 con 0x55f6805936d0 2024-03-24T10:45:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.246+0000 7f76b49cb740 10 client.17106 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.246+0000 7f76b49cb740 10 client.17106 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.246+0000 7f76b49cb740 20 client.17106 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.246+0000 7f76b49cb740 1 --2- 172.21.15.23:0/2791486022 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f68060f980 0x55f680611da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.246+0000 7f76b49cb740 4 client.17106 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:45:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.246+0000 7f76b49cb740 1 -- 172.21.15.23:0/2791486022 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f68027c890 con 0x55f68060f980 2024-03-24T10:45:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.246+0000 7f76ab7fe640 1 --2- 172.21.15.23:0/2791486022 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f68060f980 0x55f680611da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f76ab7fe640 1 --2- 172.21.15.23:0/2791486022 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f68060f980 0x55f680611da0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e6bc836529e20e3 server_cookie=45bff83b65854032 in_seq=0 out_seq=0 2024-03-24T10:45:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f768affd640 10 client.17106 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f768affd640 1 -- 172.21.15.23:0/2791486022 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f68027c890 con 0x55f68060f980 2024-03-24T10:45:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f768affd640 10 client.17106 handle_command_reply: tid=0 2024-03-24T10:45:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f76b49cb740 1 client.17106 shutdown 2024-03-24T10:45:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f76b49cb740 1 -- 172.21.15.23:0/2791486022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 msgr2=0x55f68059c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f76b49cb740 1 --2- 172.21.15.23:0/2791486022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f68059c500 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f76a401ee20 tx=0x7f76a401e720 comp rx=0 tx=0).stop 2024-03-24T10:45:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f76b49cb740 1 -- 172.21.15.23:0/2791486022 shutdown_connections 2024-03-24T10:45:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f76b49cb740 1 --2- 172.21.15.23:0/2791486022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f6805936d0 0x55f68059c500 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f76b49cb740 1 --2- 172.21.15.23:0/2791486022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f68058aaf0 0x55f68059bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f76b49cb740 1 --2- 172.21.15.23:0/2791486022 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f68060f980 0x55f680611da0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f76b49cb740 1 --2- 172.21.15.23:0/2791486022 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f68058a1d0 0x55f68059fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f768affd640 0 client.17106 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f76b49cb740 1 -- 172.21.15.23:0/2791486022 >> 172.21.15.23:0/2791486022 conn(0x55f680584900 msgr2=0x55f6805839e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f76b49cb740 1 -- 172.21.15.23:0/2791486022 shutdown_connections 2024-03-24T10:45:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f76b49cb740 1 -- 172.21.15.23:0/2791486022 wait complete. 2024-03-24T10:45:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.247+0000 7f76b49cb740 20 client.17106 trim_cache size 0 max 16384 2024-03-24T10:45:37.255 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:45:37.255 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:45:37.255 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:45:37.255 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:45:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.248+0000 7f76b1d19640 1 -- 172.21.15.23:0/1567378498 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f76940a2230 msgr2=0x7f76940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.248+0000 7f76b1d19640 1 --2- 172.21.15.23:0/1567378498 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f76940a2230 0x7f76940a46f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f76a4009fd0 tx=0x7f76a4009500 comp rx=0 tx=0).stop 2024-03-24T10:45:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.248+0000 7f76b1d19640 1 -- 172.21.15.23:0/1567378498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76ac100f60 msgr2=0x7f76ac19dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.248+0000 7f76b1d19640 1 --2- 172.21.15.23:0/1567378498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76ac100f60 0x7f76ac19dde0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f76a000e9f0 tx=0x7f76a000eeb0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.248+0000 7f76b1d19640 1 -- 172.21.15.23:0/1567378498 shutdown_connections 2024-03-24T10:45:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.248+0000 7f76b1d19640 1 --2- 172.21.15.23:0/1567378498 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f76940a2230 0x7f76940a46f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.248+0000 7f76b1d19640 1 --2- 172.21.15.23:0/1567378498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76ac106660 0x7f76ac197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.248+0000 7f76b1d19640 1 --2- 172.21.15.23:0/1567378498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76ac100f60 0x7f76ac19dde0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.248+0000 7f76b1d19640 1 --2- 172.21.15.23:0/1567378498 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76ac1005a0 0x7f76ac19d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.248+0000 7f76b1d19640 1 -- 172.21.15.23:0/1567378498 >> 172.21.15.23:0/1567378498 conn(0x7f76ac077710 msgr2=0x7f76ac0ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.248+0000 7f76b1d19640 1 -- 172.21.15.23:0/1567378498 shutdown_connections 2024-03-24T10:45:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.248+0000 7f76b1d19640 1 -- 172.21.15.23:0/1567378498 wait complete. 2024-03-24T10:45:37.259 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:45:37.259 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:45:37.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.407+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/3470630525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f8106fd0 msgr2=0x7fb0f8100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.407+0000 7fb0ff1f4640 1 --2- 172.21.15.23:0/3470630525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f8106fd0 0x7fb0f8100610 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009900 tx=0x7fb0e402bcd0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.407+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/3470630525 shutdown_connections 2024-03-24T10:45:37.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.407+0000 7fb0ff1f4640 1 --2- 172.21.15.23:0/3470630525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0f8100d40 0x7fb0f810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.407+0000 7fb0ff1f4640 1 --2- 172.21.15.23:0/3470630525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f8106fd0 0x7fb0f8100610 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.407+0000 7fb0ff1f4640 1 --2- 172.21.15.23:0/3470630525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0f81066b0 0x7fb0f8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.407+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/3470630525 >> 172.21.15.23:0/3470630525 conn(0x7fb0f8077710 msgr2=0x7fb0f8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.407+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/3470630525 shutdown_connections 2024-03-24T10:45:37.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.407+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/3470630525 wait complete. 2024-03-24T10:45:37.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0ff1f4640 1 Processor -- start 2024-03-24T10:45:37.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0ff1f4640 1 -- start start 2024-03-24T10:45:37.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0ff1f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f8100d40 0x7fb0f8197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0ff1f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0f81066b0 0x7fb0f81982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0ff1f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0f8106fd0 0x7fb0f819c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0ff1f4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f819cc40 con 0x7fb0f8100d40 2024-03-24T10:45:37.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0ff1f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f819cdb0 con 0x7fb0f8106fd0 2024-03-24T10:45:37.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0ff1f4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f819cf70 con 0x7fb0f81066b0 2024-03-24T10:45:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0fcf90640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f8100d40 0x7fb0f8197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0fcf90640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f8100d40 0x7fb0f8197d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34744/0 (socket says 172.21.15.23:34744) 2024-03-24T10:45:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0f7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0f81066b0 0x7fb0f81982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0fcf90640 1 -- 172.21.15.23:0/4236678515 learned_addr learned my addr 172.21.15.23:0/4236678515 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0fd791640 1 --2- 172.21.15.23:0/4236678515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0f8106fd0 0x7fb0f819c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0fcf90640 1 -- 172.21.15.23:0/4236678515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0f81066b0 msgr2=0x7fb0f81982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0fcf90640 1 --2- 172.21.15.23:0/4236678515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0f81066b0 0x7fb0f81982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0fcf90640 1 -- 172.21.15.23:0/4236678515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0f8106fd0 msgr2=0x7fb0f819c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0fcf90640 1 --2- 172.21.15.23:0/4236678515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0f8106fd0 0x7fb0f819c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0fcf90640 1 -- 172.21.15.23:0/4236678515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e4009560 con 0x7fb0f8100d40 2024-03-24T10:45:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.408+0000 7fb0f7fff640 1 --2- 172.21.15.23:0/4236678515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0f81066b0 0x7fb0f81982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.409+0000 7fb0fcf90640 1 --2- 172.21.15.23:0/4236678515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f8100d40 0x7fb0f8197d80 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00e9c0 tx=0x7fb0ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.409+0000 7fb0f5ffb640 1 -- 172.21.15.23:0/4236678515 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec009700 con 0x7fb0f8100d40 2024-03-24T10:45:37.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.409+0000 7fb0f5ffb640 1 -- 172.21.15.23:0/4236678515 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0ec00cd80 con 0x7fb0f8100d40 2024-03-24T10:45:37.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.409+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/4236678515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0f819d250 con 0x7fb0f8100d40 2024-03-24T10:45:37.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.409+0000 7fb0f5ffb640 1 -- 172.21.15.23:0/4236678515 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec010430 con 0x7fb0f8100d40 2024-03-24T10:45:37.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.409+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/4236678515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0f81a4ad0 con 0x7fb0f8100d40 2024-03-24T10:45:37.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.410+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/4236678515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0f8068500 con 0x7fb0f8100d40 2024-03-24T10:45:37.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.411+0000 7fb0f5ffb640 1 -- 172.21.15.23:0/4236678515 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fb0ec003680 con 0x7fb0f8100d40 2024-03-24T10:45:37.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.412+0000 7fb0f5ffb640 1 --2- 172.21.15.23:0/4236678515 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb0e80a2230 0x7fb0e80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.412+0000 7fb0f7fff640 1 --2- 172.21.15.23:0/4236678515 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb0e80a2230 0x7fb0e80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.412+0000 7fb0f5ffb640 1 -- 172.21.15.23:0/4236678515 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fb0ec014070 con 0x7fb0f8100d40 2024-03-24T10:45:37.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.412+0000 7fb0f7fff640 1 --2- 172.21.15.23:0/4236678515 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb0e80a2230 0x7fb0e80a46f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4000c00 tx=0x7fb0e4005fd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.417+0000 7fb0f5ffb640 1 -- 172.21.15.23:0/4236678515 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb0ec0970d0 con 0x7fb0f8100d40 2024-03-24T10:45:37.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.645+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/4236678515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0f81990d0 con 0x7fb0f8100d40 2024-03-24T10:45:37.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.646+0000 7fb0f5ffb640 1 -- 172.21.15.23:0/4236678515 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11869 (secure 0 0 0) 0x7fb0ec09a790 con 0x7fb0f8100d40 2024-03-24T10:45:37.646 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:45:37.646 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:45:17.709481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26933":{"gid":26933,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/2955742470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":2955742470},{"type":"v1","addr":"172.21.15.117:6837","nonce":2955742470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24539,24503,24505]},"id":1}]} 2024-03-24T10:45:37.646 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 29 2024-03-24T10:45:37.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.648+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/4236678515 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb0e80a2230 msgr2=0x7fb0e80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.648+0000 7fb0ff1f4640 1 --2- 172.21.15.23:0/4236678515 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb0e80a2230 0x7fb0e80a46f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4000c00 tx=0x7fb0e4005fd0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.648+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/4236678515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f8100d40 msgr2=0x7fb0f8197d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.648+0000 7fb0ff1f4640 1 --2- 172.21.15.23:0/4236678515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f8100d40 0x7fb0f8197d80 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00e9c0 tx=0x7fb0ec00ee80 comp rx=0 tx=0).stop 2024-03-24T10:45:37.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.648+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/4236678515 shutdown_connections 2024-03-24T10:45:37.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.648+0000 7fb0ff1f4640 1 --2- 172.21.15.23:0/4236678515 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb0e80a2230 0x7fb0e80a46f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.648+0000 7fb0ff1f4640 1 --2- 172.21.15.23:0/4236678515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0f8106fd0 0x7fb0f819c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.648+0000 7fb0ff1f4640 1 --2- 172.21.15.23:0/4236678515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0f81066b0 0x7fb0f81982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.648+0000 7fb0ff1f4640 1 --2- 172.21.15.23:0/4236678515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f8100d40 0x7fb0f8197d80 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.648+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/4236678515 >> 172.21.15.23:0/4236678515 conn(0x7fb0f8077710 msgr2=0x7fb0f80ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.648+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/4236678515 shutdown_connections 2024-03-24T10:45:37.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.648+0000 7fb0ff1f4640 1 -- 172.21.15.23:0/4236678515 wait complete. 2024-03-24T10:45:37.660 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:45:37.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.774+0000 7f99e0e9d640 1 -- 172.21.15.23:0/428443367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99dc1076d0 msgr2=0x7f99dc107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.774+0000 7f99e0e9d640 1 --2- 172.21.15.23:0/428443367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99dc1076d0 0x7f99dc107ab0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f99cc009930 tx=0x7f99cc02bc30 comp rx=0 tx=0).stop 2024-03-24T10:45:37.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.774+0000 7f99e0e9d640 1 -- 172.21.15.23:0/428443367 shutdown_connections 2024-03-24T10:45:37.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.774+0000 7f99e0e9d640 1 --2- 172.21.15.23:0/428443367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99dc1005b0 0x7f99dc10e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.774+0000 7f99e0e9d640 1 --2- 172.21.15.23:0/428443367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99dc0ffc10 0x7f99dc100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.774+0000 7f99e0e9d640 1 --2- 172.21.15.23:0/428443367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99dc1076d0 0x7f99dc107ab0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.774+0000 7f99e0e9d640 1 -- 172.21.15.23:0/428443367 >> 172.21.15.23:0/428443367 conn(0x7f99dc076bd0 msgr2=0x7f99dc0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.774+0000 7f99e0e9d640 1 -- 172.21.15.23:0/428443367 shutdown_connections 2024-03-24T10:45:37.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.774+0000 7f99e0e9d640 1 -- 172.21.15.23:0/428443367 wait complete. 2024-03-24T10:45:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.774+0000 7f99e0e9d640 1 Processor -- start 2024-03-24T10:45:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99e0e9d640 1 -- start start 2024-03-24T10:45:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99e0e9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99dc0ffc10 0x7f99dc197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99e0e9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99dc1005b0 0x7f99dc1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99e0e9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99dc1076d0 0x7f99dc19c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99e0e9d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f99dc19cbe0 con 0x7f99dc1005b0 2024-03-24T10:45:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99e0e9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99dc19cd50 con 0x7f99dc0ffc10 2024-03-24T10:45:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99e0e9d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f99dc19cf10 con 0x7f99dc1076d0 2024-03-24T10:45:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99d1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99dc1005b0 0x7f99dc1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99d1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99dc1005b0 0x7f99dc1982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46922/0 (socket says 172.21.15.23:46922) 2024-03-24T10:45:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99da59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99dc0ffc10 0x7f99dc197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99d1d9b640 1 -- 172.21.15.23:0/3358475377 learned_addr learned my addr 172.21.15.23:0/3358475377 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99dad9d640 1 --2- 172.21.15.23:0/3358475377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99dc1076d0 0x7f99dc19c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99d1d9b640 1 -- 172.21.15.23:0/3358475377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99dc1076d0 msgr2=0x7f99dc19c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99d1d9b640 1 --2- 172.21.15.23:0/3358475377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99dc1076d0 0x7f99dc19c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99d1d9b640 1 -- 172.21.15.23:0/3358475377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99dc0ffc10 msgr2=0x7f99dc197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99d1d9b640 1 --2- 172.21.15.23:0/3358475377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99dc0ffc10 0x7f99dc197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99d1d9b640 1 -- 172.21.15.23:0/3358475377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99cc009560 con 0x7f99dc1005b0 2024-03-24T10:45:37.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99da59c640 1 --2- 172.21.15.23:0/3358475377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99dc0ffc10 0x7f99dc197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:37.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.775+0000 7f99d1d9b640 1 --2- 172.21.15.23:0/3358475377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99dc1005b0 0x7f99dc1982b0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f99c400e9f0 tx=0x7f99c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.776+0000 7f99d3fff640 1 -- 172.21.15.23:0/3358475377 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c4009700 con 0x7f99dc1005b0 2024-03-24T10:45:37.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.776+0000 7f99d3fff640 1 -- 172.21.15.23:0/3358475377 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99c400cd80 con 0x7f99dc1005b0 2024-03-24T10:45:37.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.776+0000 7f99e0e9d640 1 -- 172.21.15.23:0/3358475377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99dc19d1f0 con 0x7f99dc1005b0 2024-03-24T10:45:37.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.776+0000 7f99d3fff640 1 -- 172.21.15.23:0/3358475377 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c4018920 con 0x7f99dc1005b0 2024-03-24T10:45:37.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.776+0000 7f99e0e9d640 1 -- 172.21.15.23:0/3358475377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99dc1a4a70 con 0x7f99dc1005b0 2024-03-24T10:45:37.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.778+0000 7f99d3fff640 1 -- 172.21.15.23:0/3358475377 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f99c4018a80 con 0x7f99dc1005b0 2024-03-24T10:45:37.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.778+0000 7f99e3722740 1 Processor -- start 2024-03-24T10:45:37.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.778+0000 7f99e3722740 1 -- start start 2024-03-24T10:45:37.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99d3fff640 1 --2- 172.21.15.23:0/3358475377 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f99c80a2230 0x7f99c80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99e3722740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39350a0 0x55f5c3935480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99e3722740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c39359c0 0x55f5c39427c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99e3722740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c3942d00 0x55f5c39450f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99e3722740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f5c38a3840 con 0x55f5c39350a0 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99e3722740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f5c38a0d70 con 0x55f5c39359c0 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99e3722740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f5c389c550 con 0x55f5c3942d00 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99da59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39350a0 0x55f5c3935480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99dad9d640 1 --2- 172.21.15.23:0/3358475377 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f99c80a2230 0x7f99c80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99da59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39350a0 0x55f5c3935480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46936/0 (socket says 172.21.15.23:46936) 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99da59c640 1 -- 172.21.15.23:0/127599080 learned_addr learned my addr 172.21.15.23:0/127599080 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99d1d9b640 1 --2- 172.21.15.23:0/127599080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c39359c0 0x55f5c39427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99dad9d640 1 --2- 172.21.15.23:0/127599080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c3942d00 0x55f5c39450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99d3fff640 1 -- 172.21.15.23:0/3358475377 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f99c4014070 con 0x7f99dc1005b0 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99da59c640 1 -- 172.21.15.23:0/127599080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c3942d00 msgr2=0x55f5c39450f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99da59c640 1 --2- 172.21.15.23:0/127599080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c3942d00 0x55f5c39450f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99da59c640 1 -- 172.21.15.23:0/127599080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c39359c0 msgr2=0x55f5c39427c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99da59c640 1 --2- 172.21.15.23:0/127599080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c39359c0 0x55f5c39427c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99da59c640 1 -- 172.21.15.23:0/127599080 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5c3831de0 con 0x55f5c39350a0 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99dad9d640 1 --2- 172.21.15.23:0/3358475377 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f99c80a2230 0x7f99c80a46f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f99d4009fd0 tx=0x7f99d4009de0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.779+0000 7f99da59c640 1 --2- 172.21.15.23:0/127599080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39350a0 0x55f5c3935480 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f99cc036040 tx=0x7f99cc02c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7adb7412cfd70cc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.780+0000 7f99d0d99640 1 -- 172.21.15.23:0/127599080 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99cc02c7e0 con 0x55f5c39350a0 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.780+0000 7f99d0d99640 1 -- 172.21.15.23:0/127599080 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99cc004510 con 0x55f5c39350a0 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.780+0000 7f99d0d99640 1 -- 172.21.15.23:0/127599080 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99cc005410 con 0x55f5c39350a0 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.780+0000 7f99e3722740 1 -- 172.21.15.23:0/127599080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39350a0 msgr2=0x55f5c3935480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.780+0000 7f99e3722740 1 --2- 172.21.15.23:0/127599080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39350a0 0x55f5c3935480 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f99cc036040 tx=0x7f99cc02c660 comp rx=0 tx=0).stop 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.780+0000 7f99e3722740 1 -- 172.21.15.23:0/127599080 shutdown_connections 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.780+0000 7f99e3722740 1 --2- 172.21.15.23:0/127599080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c3942d00 0x55f5c39450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.780+0000 7f99e3722740 1 --2- 172.21.15.23:0/127599080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c39359c0 0x55f5c39427c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.780+0000 7f99e3722740 1 --2- 172.21.15.23:0/127599080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39350a0 0x55f5c3935480 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.780+0000 7f99e3722740 1 -- 172.21.15.23:0/127599080 >> 172.21.15.23:0/127599080 conn(0x55f5c392fb20 msgr2=0x55f5c392ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.780+0000 7f99e3722740 1 -- 172.21.15.23:0/127599080 shutdown_connections 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.780+0000 7f99e3722740 1 -- 172.21.15.23:0/127599080 wait complete. 2024-03-24T10:45:37.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99e3722740 1 -- start start 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99e3722740 1 Processor -- start 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99e3722740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 0x55f5c393fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99e3722740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39359c0 0x55f5c3940240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99e3722740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c3942d00 0x55f5c3940780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99e3722740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f5c356f430 con 0x55f5c39359c0 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99e3722740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f5c352be00 con 0x55f5c3942d00 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99e3722740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f5c3821590 con 0x55f5c39350a0 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99e3722740 10 client.0 _ll_register_callbacks cb 0x55f5c3890fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99d1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39359c0 0x55f5c3940240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99d1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39359c0 0x55f5c3940240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46950/0 (socket says 172.21.15.23:46950) 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99d1d9b640 1 -- 172.21.15.23:0/1381972907 learned_addr learned my addr 172.21.15.23:0/1381972907 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99dad9d640 1 --2- 172.21.15.23:0/1381972907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c3942d00 0x55f5c3940780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99da59c640 1 --2- 172.21.15.23:0/1381972907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 0x55f5c393fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99d1d9b640 1 -- 172.21.15.23:0/1381972907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 msgr2=0x55f5c393fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99d1d9b640 1 --2- 172.21.15.23:0/1381972907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 0x55f5c393fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99d1d9b640 1 -- 172.21.15.23:0/1381972907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c3942d00 msgr2=0x55f5c3940780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99d1d9b640 1 --2- 172.21.15.23:0/1381972907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c3942d00 0x55f5c3940780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99d1d9b640 1 -- 172.21.15.23:0/1381972907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99dc10ee90 con 0x55f5c39359c0 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.781+0000 7f99da59c640 1 --2- 172.21.15.23:0/1381972907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 0x55f5c393fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99d1d9b640 1 --2- 172.21.15.23:0/1381972907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39359c0 0x55f5c3940240 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f99c40cbc00 tx=0x7f99c4064620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da53077f18d9d9cb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99c1ffb640 1 -- 172.21.15.23:0/1381972907 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c409aea0 con 0x55f5c39359c0 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99c1ffb640 1 -- 172.21.15.23:0/1381972907 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f99c409ef40 con 0x55f5c39359c0 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99e3722740 1 -- 172.21.15.23:0/1381972907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f5c359b7e0 con 0x55f5c39359c0 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99c1ffb640 1 -- 172.21.15.23:0/1381972907 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f99c40a3590 con 0x55f5c39359c0 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99e3722740 10 client.17124 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99e3722740 1 -- 172.21.15.23:0/1381972907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f5c389abd0 con 0x55f5c39359c0 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99c1ffb640 1 -- 172.21.15.23:0/1381972907 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f5c359b7e0 con 0x55f5c39359c0 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99c1ffb640 1 -- 172.21.15.23:0/1381972907 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f99c40ce040 con 0x55f5c39359c0 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99e3722740 10 client.17124 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:37.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99e3722740 10 client.17124 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:45:37.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99e3722740 20 client.17124 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:37.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99e3722740 1 --2- 172.21.15.23:0/1381972907 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f5c3952c80 0x55f5c3955100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99e3722740 4 client.17124 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:37.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99e3722740 1 -- 172.21.15.23:0/1381972907 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5c35d8720 con 0x55f5c3952c80 2024-03-24T10:45:37.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.782+0000 7f99da59c640 1 --2- 172.21.15.23:0/1381972907 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f5c3952c80 0x55f5c3955100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.783+0000 7f99da59c640 1 --2- 172.21.15.23:0/1381972907 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f5c3952c80 0x55f5c3955100 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7e373aedbf209c82 server_cookie=25e034e72de229ec in_seq=0 out_seq=0 2024-03-24T10:45:37.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.783+0000 7f99c1ffb640 10 client.17124 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:45:37.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.784+0000 7f99c1ffb640 1 -- 172.21.15.23:0/1381972907 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f5c35d8720 con 0x55f5c3952c80 2024-03-24T10:45:37.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.784+0000 7f99c1ffb640 10 client.17124 handle_command_reply: tid=0 2024-03-24T10:45:37.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.784+0000 7f99e3722740 1 client.17124 shutdown 2024-03-24T10:45:37.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.785+0000 7f99e3722740 1 -- 172.21.15.23:0/1381972907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39359c0 msgr2=0x55f5c3940240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.785+0000 7f99e3722740 1 --2- 172.21.15.23:0/1381972907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39359c0 0x55f5c3940240 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f99c40cbc00 tx=0x7f99c4064620 comp rx=0 tx=0).stop 2024-03-24T10:45:37.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.785+0000 7f99e3722740 1 -- 172.21.15.23:0/1381972907 shutdown_connections 2024-03-24T10:45:37.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.785+0000 7f99e3722740 1 --2- 172.21.15.23:0/1381972907 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f5c3952c80 0x55f5c3955100 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.785+0000 7f99e3722740 1 --2- 172.21.15.23:0/1381972907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c3942d00 0x55f5c3940780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.785+0000 7f99e3722740 1 --2- 172.21.15.23:0/1381972907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39359c0 0x55f5c3940240 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.785+0000 7f99c1ffb640 0 client.17124 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:45:37.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.785+0000 7f99e3722740 1 --2- 172.21.15.23:0/1381972907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 0x55f5c393fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.785+0000 7f99e3722740 1 -- 172.21.15.23:0/1381972907 >> 172.21.15.23:0/1381972907 conn(0x55f5c392fb20 msgr2=0x55f5c3931870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.785+0000 7f99e3722740 1 -- 172.21.15.23:0/1381972907 shutdown_connections 2024-03-24T10:45:37.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.785+0000 7f99e3722740 1 -- 172.21.15.23:0/1381972907 wait complete. 2024-03-24T10:45:37.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.785+0000 7f99e3722740 20 client.17124 trim_cache size 0 max 16384 2024-03-24T10:45:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.799+0000 7f99e3722740 1 Processor -- start 2024-03-24T10:45:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.799+0000 7f99e3722740 1 -- start start 2024-03-24T10:45:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99e3722740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 0x55f5c392ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99e3722740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c39359c0 0x55f5c394e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99e3722740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c3942d00 0x55f5c395aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99e3722740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f5c3952a90 con 0x55f5c3942d00 2024-03-24T10:45:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99e3722740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f5c392b090 con 0x55f5c39359c0 2024-03-24T10:45:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99e3722740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f5c3934710 con 0x55f5c39350a0 2024-03-24T10:45:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99dad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c3942d00 0x55f5c395aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99dad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c3942d00 0x55f5c395aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46964/0 (socket says 172.21.15.23:46964) 2024-03-24T10:45:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99dad9d640 1 -- 172.21.15.23:0/2844260182 learned_addr learned my addr 172.21.15.23:0/2844260182 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99da59c640 1 --2- 172.21.15.23:0/2844260182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 0x55f5c392ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99d1d9b640 1 --2- 172.21.15.23:0/2844260182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c39359c0 0x55f5c394e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99dad9d640 1 -- 172.21.15.23:0/2844260182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 msgr2=0x55f5c392ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99dad9d640 1 --2- 172.21.15.23:0/2844260182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 0x55f5c392ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99dad9d640 1 -- 172.21.15.23:0/2844260182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c39359c0 msgr2=0x55f5c394e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99dad9d640 1 --2- 172.21.15.23:0/2844260182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c39359c0 0x55f5c394e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99dad9d640 1 -- 172.21.15.23:0/2844260182 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5c3836c80 con 0x55f5c3942d00 2024-03-24T10:45:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99da59c640 1 --2- 172.21.15.23:0/2844260182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 0x55f5c392ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99dad9d640 1 --2- 172.21.15.23:0/2844260182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c3942d00 0x55f5c395aa10 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f99d4013630 tx=0x7f99d4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28eebdae00454b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99c3fff640 1 -- 172.21.15.23:0/2844260182 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d4003ab0 con 0x55f5c3942d00 2024-03-24T10:45:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99c3fff640 1 -- 172.21.15.23:0/2844260182 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99d4003c10 con 0x55f5c3942d00 2024-03-24T10:45:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.800+0000 7f99c3fff640 1 -- 172.21.15.23:0/2844260182 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d40125b0 con 0x55f5c3942d00 2024-03-24T10:45:37.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.801+0000 7f99e3722740 1 -- 172.21.15.23:0/2844260182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c3942d00 msgr2=0x55f5c395aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.801+0000 7f99e3722740 1 --2- 172.21.15.23:0/2844260182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c3942d00 0x55f5c395aa10 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f99d4013630 tx=0x7f99d4009500 comp rx=0 tx=0).stop 2024-03-24T10:45:37.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.801+0000 7f99e3722740 1 -- 172.21.15.23:0/2844260182 shutdown_connections 2024-03-24T10:45:37.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.801+0000 7f99e3722740 1 --2- 172.21.15.23:0/2844260182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c3942d00 0x55f5c395aa10 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.801+0000 7f99e3722740 1 --2- 172.21.15.23:0/2844260182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c39359c0 0x55f5c394e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.801+0000 7f99e3722740 1 --2- 172.21.15.23:0/2844260182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 0x55f5c392ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.801+0000 7f99e3722740 1 -- 172.21.15.23:0/2844260182 >> 172.21.15.23:0/2844260182 conn(0x55f5c392fb20 msgr2=0x55f5c392ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.801+0000 7f99e3722740 1 -- 172.21.15.23:0/2844260182 shutdown_connections 2024-03-24T10:45:37.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.801+0000 7f99e3722740 1 -- 172.21.15.23:0/2844260182 wait complete. 2024-03-24T10:45:37.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.801+0000 7f99e3722740 1 -- start start 2024-03-24T10:45:37.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99e3722740 1 Processor -- start 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99e3722740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 0x55f5c394ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99e3722740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39359c0 0x55f5c39471e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99e3722740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c3942d00 0x55f5c3947720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99e3722740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f5c3947dc0 con 0x55f5c39359c0 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99e3722740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f5c3947f30 con 0x55f5c3942d00 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99e3722740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f5c39b9050 con 0x55f5c39350a0 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99e3722740 10 client.0 _ll_register_callbacks cb 0x7f99c400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99d1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39359c0 0x55f5c39471e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99d1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39359c0 0x55f5c39471e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46968/0 (socket says 172.21.15.23:46968) 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99d1d9b640 1 -- 172.21.15.23:0/4208633910 learned_addr learned my addr 172.21.15.23:0/4208633910 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99da59c640 1 --2- 172.21.15.23:0/4208633910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 0x55f5c394ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99dad9d640 1 --2- 172.21.15.23:0/4208633910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c3942d00 0x55f5c3947720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99d1d9b640 1 -- 172.21.15.23:0/4208633910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 msgr2=0x55f5c394ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99d1d9b640 1 --2- 172.21.15.23:0/4208633910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 0x55f5c394ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99d1d9b640 1 -- 172.21.15.23:0/4208633910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c3942d00 msgr2=0x55f5c3947720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99d1d9b640 1 --2- 172.21.15.23:0/4208633910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c3942d00 0x55f5c3947720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99d1d9b640 1 -- 172.21.15.23:0/4208633910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5c3831de0 con 0x55f5c39359c0 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99d1d9b640 1 --2- 172.21.15.23:0/4208633910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39359c0 0x55f5c39471e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f99c4010f00 tx=0x7f99c4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c305e73e6531549 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.802+0000 7f99c1ffb640 1 -- 172.21.15.23:0/4208633910 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c40b2070 con 0x55f5c39359c0 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.803+0000 7f99c1ffb640 1 -- 172.21.15.23:0/4208633910 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f99c40aee40 con 0x55f5c39359c0 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.803+0000 7f99e3722740 1 -- 172.21.15.23:0/4208633910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99cc004510 con 0x55f5c39359c0 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.803+0000 7f99c1ffb640 1 -- 172.21.15.23:0/4208633910 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f99c409ef40 con 0x55f5c39359c0 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.803+0000 7f99e3722740 10 client.17130 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.803+0000 7f99e3722740 1 -- 172.21.15.23:0/4208633910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x7f99d401b040 con 0x55f5c39359c0 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.803+0000 7f99c1ffb640 1 -- 172.21.15.23:0/4208633910 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99cc004510 con 0x55f5c39359c0 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.803+0000 7f99c1ffb640 1 -- 172.21.15.23:0/4208633910 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f99c409a4b0 con 0x55f5c39359c0 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.803+0000 7f99e3722740 10 client.17130 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.803+0000 7f99e3722740 10 client.17130 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.803+0000 7f99e3722740 20 client.17130 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:37.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.803+0000 7f99e3722740 1 --2- 172.21.15.23:0/4208633910 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f5c39ba4d0 0x55f5c39bc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.803+0000 7f99e3722740 4 client.17130 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.803+0000 7f99e3722740 1 -- 172.21.15.23:0/4208633910 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f5c3627890 con 0x55f5c39ba4d0 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.803+0000 7f99da59c640 1 --2- 172.21.15.23:0/4208633910 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f5c39ba4d0 0x55f5c39bc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99c1ffb640 10 client.17130 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99da59c640 1 --2- 172.21.15.23:0/4208633910 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f5c39ba4d0 0x55f5c39bc8f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3f4cfb7e30f3324c server_cookie=ed0f39ffd118894d in_seq=0 out_seq=0 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99c1ffb640 1 -- 172.21.15.23:0/4208633910 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f5c3627890 con 0x55f5c39ba4d0 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99c1ffb640 10 client.17130 handle_command_reply: tid=0 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99e3722740 1 client.17130 shutdown 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99e3722740 1 -- 172.21.15.23:0/4208633910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39359c0 msgr2=0x55f5c39471e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99e3722740 1 --2- 172.21.15.23:0/4208633910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39359c0 0x55f5c39471e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f99c4010f00 tx=0x7f99c4000bc0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99e3722740 1 -- 172.21.15.23:0/4208633910 shutdown_connections 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99e3722740 1 --2- 172.21.15.23:0/4208633910 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55f5c39ba4d0 0x55f5c39bc8f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99e3722740 1 --2- 172.21.15.23:0/4208633910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5c3942d00 0x55f5c3947720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99e3722740 1 --2- 172.21.15.23:0/4208633910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5c39359c0 0x55f5c39471e0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99e3722740 1 --2- 172.21.15.23:0/4208633910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5c39350a0 0x55f5c394ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99e3722740 1 -- 172.21.15.23:0/4208633910 >> 172.21.15.23:0/4208633910 conn(0x55f5c392fb20 msgr2=0x55f5c3930f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99c1ffb640 0 client.17130 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:45:37.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99e3722740 1 -- 172.21.15.23:0/4208633910 shutdown_connections 2024-03-24T10:45:37.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99e3722740 1 -- 172.21.15.23:0/4208633910 wait complete. 2024-03-24T10:45:37.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.804+0000 7f99e3722740 20 client.17130 trim_cache size 0 max 16384 2024-03-24T10:45:37.810 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:45:37.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.805+0000 7f99e0e9d640 1 -- 172.21.15.23:0/3358475377 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f99c80a2230 msgr2=0x7f99c80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.805+0000 7f99e0e9d640 1 --2- 172.21.15.23:0/3358475377 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f99c80a2230 0x7f99c80a46f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f99d4009fd0 tx=0x7f99d4009de0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.805+0000 7f99e0e9d640 1 -- 172.21.15.23:0/3358475377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99dc1005b0 msgr2=0x7f99dc1982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.805+0000 7f99e0e9d640 1 --2- 172.21.15.23:0/3358475377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99dc1005b0 0x7f99dc1982b0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f99c400e9f0 tx=0x7f99c400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.805+0000 7f99e0e9d640 1 -- 172.21.15.23:0/3358475377 shutdown_connections 2024-03-24T10:45:37.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.805+0000 7f99e0e9d640 1 --2- 172.21.15.23:0/3358475377 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f99c80a2230 0x7f99c80a46f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.805+0000 7f99e0e9d640 1 --2- 172.21.15.23:0/3358475377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99dc1076d0 0x7f99dc19c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.805+0000 7f99e0e9d640 1 --2- 172.21.15.23:0/3358475377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99dc1005b0 0x7f99dc1982b0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.805+0000 7f99e0e9d640 1 --2- 172.21.15.23:0/3358475377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99dc0ffc10 0x7f99dc197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.805+0000 7f99e0e9d640 1 -- 172.21.15.23:0/3358475377 >> 172.21.15.23:0/3358475377 conn(0x7f99dc076bd0 msgr2=0x7f99dc0fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.805+0000 7f99e0e9d640 1 -- 172.21.15.23:0/3358475377 shutdown_connections 2024-03-24T10:45:37.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.805+0000 7f99e0e9d640 1 -- 172.21.15.23:0/3358475377 wait complete. 2024-03-24T10:45:37.815 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:45:37.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:37 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4236678515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:45:37.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.942+0000 7ffbde5a0640 1 -- 172.21.15.23:0/1631771529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd8106660 msgr2=0x7ffbd8106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.942+0000 7ffbde5a0640 1 --2- 172.21.15.23:0/1631771529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd8106660 0x7ffbd8106a40 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009900 tx=0x7ffbc802bc30 comp rx=0 tx=0).stop 2024-03-24T10:45:37.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.942+0000 7ffbde5a0640 1 -- 172.21.15.23:0/1631771529 shutdown_connections 2024-03-24T10:45:37.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.942+0000 7ffbde5a0640 1 --2- 172.21.15.23:0/1631771529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbd8100f60 0x7ffbd810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.942+0000 7ffbde5a0640 1 --2- 172.21.15.23:0/1631771529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd81005a0 0x7ffbd8100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.942+0000 7ffbde5a0640 1 --2- 172.21.15.23:0/1631771529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd8106660 0x7ffbd8106a40 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.942+0000 7ffbde5a0640 1 -- 172.21.15.23:0/1631771529 >> 172.21.15.23:0/1631771529 conn(0x7ffbd8077710 msgr2=0x7ffbd8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.942+0000 7ffbde5a0640 1 -- 172.21.15.23:0/1631771529 shutdown_connections 2024-03-24T10:45:37.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.942+0000 7ffbde5a0640 1 -- 172.21.15.23:0/1631771529 wait complete. 2024-03-24T10:45:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.943+0000 7ffbde5a0640 1 Processor -- start 2024-03-24T10:45:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.943+0000 7ffbde5a0640 1 -- start start 2024-03-24T10:45:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.943+0000 7ffbde5a0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd81005a0 0x7ffbd8102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.943+0000 7ffbde5a0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbd8100f60 0x7ffbd8102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.943+0000 7ffbde5a0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd8106660 0x7ffbd8102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.943+0000 7ffbde5a0640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd81a6070 con 0x7ffbd8106660 2024-03-24T10:45:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.943+0000 7ffbde5a0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd81a61e0 con 0x7ffbd81005a0 2024-03-24T10:45:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.943+0000 7ffbde5a0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd81a63a0 con 0x7ffbd8100f60 2024-03-24T10:45:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbdcb3d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd8106660 0x7ffbd8102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbdcb3d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd8106660 0x7ffbd8102c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46978/0 (socket says 172.21.15.23:46978) 2024-03-24T10:45:37.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbd7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd81005a0 0x7ffbd8102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbdcb3d640 1 -- 172.21.15.23:0/730686669 learned_addr learned my addr 172.21.15.23:0/730686669 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbd77fe640 1 --2- 172.21.15.23:0/730686669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbd8100f60 0x7ffbd8102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbdcb3d640 1 -- 172.21.15.23:0/730686669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbd8100f60 msgr2=0x7ffbd8102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbdcb3d640 1 --2- 172.21.15.23:0/730686669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbd8100f60 0x7ffbd8102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbdcb3d640 1 -- 172.21.15.23:0/730686669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd81005a0 msgr2=0x7ffbd8102210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbdcb3d640 1 --2- 172.21.15.23:0/730686669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd81005a0 0x7ffbd8102210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbdcb3d640 1 -- 172.21.15.23:0/730686669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc8009560 con 0x7ffbd8106660 2024-03-24T10:45:37.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbdcb3d640 1 --2- 172.21.15.23:0/730686669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd8106660 0x7ffbd8102c90 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000eb10 tx=0x7ffbd000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbd57fa640 1 -- 172.21.15.23:0/730686669 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0004b20 con 0x7ffbd8106660 2024-03-24T10:45:37.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbde5a0640 1 -- 172.21.15.23:0/730686669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd81a6680 con 0x7ffbd8106660 2024-03-24T10:45:37.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbde5a0640 1 -- 172.21.15.23:0/730686669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd81a6be0 con 0x7ffbd8106660 2024-03-24T10:45:37.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbd57fa640 1 -- 172.21.15.23:0/730686669 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbd0004c80 con 0x7ffbd8106660 2024-03-24T10:45:37.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.944+0000 7ffbd57fa640 1 -- 172.21.15.23:0/730686669 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0018630 con 0x7ffbd8106660 2024-03-24T10:45:37.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.947+0000 7ffbd57fa640 1 -- 172.21.15.23:0/730686669 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7ffbd0018790 con 0x7ffbd8106660 2024-03-24T10:45:37.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.947+0000 7ffbe1176740 1 Processor -- start 2024-03-24T10:45:37.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.947+0000 7ffbe1176740 1 -- start start 2024-03-24T10:45:37.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.947+0000 7ffbe1176740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f8161d3f0 0x555f8161d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.947+0000 7ffbe1176740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161dd10 0x555f816263b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.947+0000 7ffbe1176740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f816268f0 0x555f81628ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.947+0000 7ffbe1176740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555f8158b840 con 0x555f816268f0 2024-03-24T10:45:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.947+0000 7ffbe1176740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555f81588d70 con 0x555f8161dd10 2024-03-24T10:45:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.947+0000 7ffbe1176740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555f81584550 con 0x555f8161d3f0 2024-03-24T10:45:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbd77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f816268f0 0x555f81628ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbd7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161dd10 0x555f816263b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbd77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f816268f0 0x555f81628ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46980/0 (socket says 172.21.15.23:46980) 2024-03-24T10:45:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbd77fe640 1 -- 172.21.15.23:0/3584778446 learned_addr learned my addr 172.21.15.23:0/3584778446 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbd77fe640 1 --2- 172.21.15.23:0/3584778446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f8161d3f0 0x555f8161d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbd57fa640 1 --2- 172.21.15.23:0/730686669 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ffbc00a2260 0x7ffbc00a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbd7fff640 1 -- 172.21.15.23:0/3584778446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f8161d3f0 msgr2=0x555f8161d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbd7fff640 1 --2- 172.21.15.23:0/3584778446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f8161d3f0 0x555f8161d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbd7fff640 1 -- 172.21.15.23:0/3584778446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f816268f0 msgr2=0x555f81628ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbd7fff640 1 --2- 172.21.15.23:0/3584778446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f816268f0 0x555f81628ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbd7fff640 1 -- 172.21.15.23:0/3584778446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f81519de0 con 0x555f8161dd10 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbd77fe640 1 --2- 172.21.15.23:0/3584778446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f816268f0 0x555f81628ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbd57fa640 1 -- 172.21.15.23:0/730686669 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7ffbd0014070 con 0x7ffbd8106660 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbdcb3d640 1 --2- 172.21.15.23:0/730686669 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ffbc00a2260 0x7ffbc00a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbd7fff640 1 --2- 172.21.15.23:0/3584778446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161dd10 0x555f816263b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8000c00 tx=0x7ffbc802c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=798e09a5d20080ea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbdcb3d640 1 --2- 172.21.15.23:0/730686669 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ffbc00a2260 0x7ffbc00a4720 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00cbd00 tx=0x7ffbd0009200 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbbe7fc640 1 -- 172.21.15.23:0/3584778446 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc802c910 con 0x555f8161dd10 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbbe7fc640 1 -- 172.21.15.23:0/3584778446 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc802ca70 con 0x555f8161dd10 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.948+0000 7ffbbe7fc640 1 -- 172.21.15.23:0/3584778446 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc80050e0 con 0x555f8161dd10 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.949+0000 7ffbe1176740 1 -- 172.21.15.23:0/3584778446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161dd10 msgr2=0x555f816263b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.949+0000 7ffbe1176740 1 --2- 172.21.15.23:0/3584778446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161dd10 0x555f816263b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8000c00 tx=0x7ffbc802c6a0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.949+0000 7ffbe1176740 1 -- 172.21.15.23:0/3584778446 shutdown_connections 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.949+0000 7ffbe1176740 1 --2- 172.21.15.23:0/3584778446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f816268f0 0x555f81628ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.949+0000 7ffbe1176740 1 --2- 172.21.15.23:0/3584778446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161dd10 0x555f816263b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.949+0000 7ffbe1176740 1 --2- 172.21.15.23:0/3584778446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f8161d3f0 0x555f8161d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.949+0000 7ffbe1176740 1 -- 172.21.15.23:0/3584778446 >> 172.21.15.23:0/3584778446 conn(0x555f81617b20 msgr2=0x555f81617f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.949+0000 7ffbe1176740 1 -- 172.21.15.23:0/3584778446 shutdown_connections 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.949+0000 7ffbe1176740 1 -- 172.21.15.23:0/3584778446 wait complete. 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.949+0000 7ffbe1176740 1 -- start start 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbe1176740 1 Processor -- start 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbe1176740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161d3f0 0x555f816238b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbe1176740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f8161dd10 0x555f81623df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbe1176740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f816268f0 0x555f81624330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbe1176740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555f81257430 con 0x555f8161d3f0 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbe1176740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555f81213e00 con 0x555f816268f0 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbe1176740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555f81509590 con 0x555f8161dd10 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbe1176740 10 client.0 _ll_register_callbacks cb 0x555f81578fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbd77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161d3f0 0x555f816238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbd7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f8161dd10 0x555f81623df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbd7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f8161dd10 0x555f81623df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:50356/0 (socket says 172.21.15.23:50356) 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbd7fff640 1 -- 172.21.15.23:0/4049796059 learned_addr learned my addr 172.21.15.23:0/4049796059 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbd77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161d3f0 0x555f816238b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46992/0 (socket says 172.21.15.23:46992) 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbd77fe640 1 --2- 172.21.15.23:0/4049796059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f816268f0 0x555f81624330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbd7fff640 1 -- 172.21.15.23:0/4049796059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f816268f0 msgr2=0x555f81624330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbd7fff640 1 --2- 172.21.15.23:0/4049796059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f816268f0 0x555f81624330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbd7fff640 1 -- 172.21.15.23:0/4049796059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161d3f0 msgr2=0x555f816238b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbd7fff640 1 --2- 172.21.15.23:0/4049796059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161d3f0 0x555f816238b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.950+0000 7ffbd7fff640 1 -- 172.21.15.23:0/4049796059 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd810dd20 con 0x555f8161dd10 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.951+0000 7ffbd7fff640 1 --2- 172.21.15.23:0/4049796059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f8161dd10 0x555f81623df0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8036040 tx=0x7ffbc8004470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adc19d27cd65a4fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.951+0000 7ffbb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.951+0000 7ffbb77fe640 1 -- 172.21.15.23:0/4049796059 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8005270 con 0x555f8161dd10 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.951+0000 7ffbb77fe640 1 -- 172.21.15.23:0/4049796059 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7ffbc80053d0 con 0x555f8161dd10 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.951+0000 7ffbe1176740 1 -- 172.21.15.23:0/4049796059 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555f812837e0 con 0x555f8161dd10 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.951+0000 7ffbb77fe640 1 -- 172.21.15.23:0/4049796059 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffbc8005590 con 0x555f8161dd10 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.951+0000 7ffbe1176740 10 client.26995 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:37.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.951+0000 7ffbe1176740 1 -- 172.21.15.23:0/4049796059 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555f81582bd0 con 0x555f8161dd10 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.951+0000 7ffbb77fe640 1 -- 172.21.15.23:0/4049796059 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555f812837e0 con 0x555f8161dd10 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.951+0000 7ffbb77fe640 1 -- 172.21.15.23:0/4049796059 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7ffbc803c050 con 0x555f8161dd10 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.951+0000 7ffbe1176740 10 client.26995 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.951+0000 7ffbe1176740 10 client.26995 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.951+0000 7ffbe1176740 20 client.26995 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.951+0000 7ffbe1176740 1 --2- 172.21.15.23:0/4049796059 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x555f81636650 0x555f81638a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.952+0000 7ffbe1176740 4 client.26995 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.952+0000 7ffbe1176740 1 -- 172.21.15.23:0/4049796059 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f812c0720 con 0x555f81636650 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.952+0000 7ffbd77fe640 1 --2- 172.21.15.23:0/4049796059 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x555f81636650 0x555f81638a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.952+0000 7ffbb77fe640 10 client.26995 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.952+0000 7ffbd77fe640 1 --2- 172.21.15.23:0/4049796059 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x555f81636650 0x555f81638a70 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=741240bfe46ab6ce server_cookie=ca04ed95eb03cf12 in_seq=0 out_seq=0 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbb77fe640 1 -- 172.21.15.23:0/4049796059 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555f812c0720 con 0x555f81636650 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbb77fe640 10 client.26995 handle_command_reply: tid=0 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbe1176740 1 client.26995 shutdown 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbe1176740 1 -- 172.21.15.23:0/4049796059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f8161dd10 msgr2=0x555f81623df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbe1176740 1 --2- 172.21.15.23:0/4049796059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f8161dd10 0x555f81623df0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8036040 tx=0x7ffbc8004470 comp rx=0 tx=0).stop 2024-03-24T10:45:37.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbe1176740 1 -- 172.21.15.23:0/4049796059 shutdown_connections 2024-03-24T10:45:37.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbe1176740 1 --2- 172.21.15.23:0/4049796059 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x555f81636650 0x555f81638a70 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbe1176740 1 --2- 172.21.15.23:0/4049796059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f816268f0 0x555f81624330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbe1176740 1 --2- 172.21.15.23:0/4049796059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f8161dd10 0x555f81623df0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbb77fe640 0 client.26995 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:45:37.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbe1176740 1 --2- 172.21.15.23:0/4049796059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161d3f0 0x555f816238b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbe1176740 1 -- 172.21.15.23:0/4049796059 >> 172.21.15.23:0/4049796059 conn(0x555f81617b20 msgr2=0x555f81619870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbe1176740 1 -- 172.21.15.23:0/4049796059 shutdown_connections 2024-03-24T10:45:37.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbe1176740 1 -- 172.21.15.23:0/4049796059 wait complete. 2024-03-24T10:45:37.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.953+0000 7ffbe1176740 20 client.26995 trim_cache size 0 max 16384 2024-03-24T10:45:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.967+0000 7ffbe1176740 1 Processor -- start 2024-03-24T10:45:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.967+0000 7ffbe1176740 1 -- start start 2024-03-24T10:45:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.967+0000 7ffbe1176740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161d3f0 0x555f81632180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.967+0000 7ffbe1176740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161dd10 0x555f8163e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.967+0000 7ffbe1176740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f816268f0 0x555f8162e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.967+0000 7ffbe1176740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555f81618b20 con 0x555f8161dd10 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.967+0000 7ffbe1176740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555f81633410 con 0x555f8161d3f0 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.967+0000 7ffbe1176740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555f8161ce60 con 0x555f816268f0 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.967+0000 7ffbd7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161dd10 0x555f8163e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.967+0000 7ffbd7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161dd10 0x555f8163e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47006/0 (socket says 172.21.15.23:47006) 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.967+0000 7ffbd7fff640 1 -- 172.21.15.23:0/1970545879 learned_addr learned my addr 172.21.15.23:0/1970545879 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.967+0000 7ffbd77fe640 1 --2- 172.21.15.23:0/1970545879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161d3f0 0x555f81632180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.968+0000 7ffbd77fe640 1 --2- 172.21.15.23:0/1970545879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f816268f0 0x555f8162e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.968+0000 7ffbd7fff640 1 -- 172.21.15.23:0/1970545879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f816268f0 msgr2=0x555f8162e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.968+0000 7ffbd7fff640 1 --2- 172.21.15.23:0/1970545879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f816268f0 0x555f8162e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.968+0000 7ffbd7fff640 1 -- 172.21.15.23:0/1970545879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161d3f0 msgr2=0x555f81632180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.968+0000 7ffbd7fff640 1 --2- 172.21.15.23:0/1970545879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161d3f0 0x555f81632180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.968+0000 7ffbd7fff640 1 -- 172.21.15.23:0/1970545879 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f8151ec80 con 0x555f8161dd10 2024-03-24T10:45:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.968+0000 7ffbd7fff640 1 --2- 172.21.15.23:0/1970545879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161dd10 0x555f8163e340 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ffbc803f700 tx=0x7ffbc8009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42dead2346f9ca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.968+0000 7ffbbdffb640 1 -- 172.21.15.23:0/1970545879 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8061070 con 0x555f8161dd10 2024-03-24T10:45:37.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.968+0000 7ffbbdffb640 1 -- 172.21.15.23:0/1970545879 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc80044e0 con 0x555f8161dd10 2024-03-24T10:45:37.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.968+0000 7ffbbdffb640 1 -- 172.21.15.23:0/1970545879 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc80053f0 con 0x555f8161dd10 2024-03-24T10:45:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.969+0000 7ffbe1176740 1 -- 172.21.15.23:0/1970545879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161dd10 msgr2=0x555f8163e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.969+0000 7ffbe1176740 1 --2- 172.21.15.23:0/1970545879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161dd10 0x555f8163e340 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ffbc803f700 tx=0x7ffbc8009880 comp rx=0 tx=0).stop 2024-03-24T10:45:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.969+0000 7ffbe1176740 1 -- 172.21.15.23:0/1970545879 shutdown_connections 2024-03-24T10:45:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.969+0000 7ffbe1176740 1 --2- 172.21.15.23:0/1970545879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f816268f0 0x555f8162e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.969+0000 7ffbe1176740 1 --2- 172.21.15.23:0/1970545879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161dd10 0x555f8163e340 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.969+0000 7ffbe1176740 1 --2- 172.21.15.23:0/1970545879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161d3f0 0x555f81632180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.969+0000 7ffbe1176740 1 -- 172.21.15.23:0/1970545879 >> 172.21.15.23:0/1970545879 conn(0x555f81617b20 msgr2=0x555f81616430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.969+0000 7ffbe1176740 1 -- 172.21.15.23:0/1970545879 shutdown_connections 2024-03-24T10:45:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.969+0000 7ffbe1176740 1 -- 172.21.15.23:0/1970545879 wait complete. 2024-03-24T10:45:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.969+0000 7ffbe1176740 1 -- start start 2024-03-24T10:45:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbe1176740 1 Processor -- start 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbe1176740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161d3f0 0x555f81620c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbe1176740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161dd10 0x555f816211c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbe1176740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f816268f0 0x555f8169b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbe1176740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555f81621960 con 0x555f8161dd10 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbe1176740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555f8169bee0 con 0x555f8161d3f0 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbe1176740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555f8169c0a0 con 0x555f816268f0 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbe1176740 10 client.0 _ll_register_callbacks cb 0x7ffbcc009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbd7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161dd10 0x555f816211c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbd7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161dd10 0x555f816211c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47022/0 (socket says 172.21.15.23:47022) 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbd7fff640 1 -- 172.21.15.23:0/2675414024 learned_addr learned my addr 172.21.15.23:0/2675414024 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbd77fe640 1 --2- 172.21.15.23:0/2675414024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161d3f0 0x555f81620c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbd77fe640 1 --2- 172.21.15.23:0/2675414024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f816268f0 0x555f8169b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbd7fff640 1 -- 172.21.15.23:0/2675414024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f816268f0 msgr2=0x555f8169b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbd7fff640 1 --2- 172.21.15.23:0/2675414024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f816268f0 0x555f8169b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbd7fff640 1 -- 172.21.15.23:0/2675414024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161d3f0 msgr2=0x555f81620c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbd7fff640 1 --2- 172.21.15.23:0/2675414024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161d3f0 0x555f81620c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbd7fff640 1 -- 172.21.15.23:0/2675414024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f81582bd0 con 0x555f8161dd10 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbd7fff640 1 --2- 172.21.15.23:0/2675414024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161dd10 0x555f816211c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ffbc803f590 tx=0x7ffbc8005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65affa1a71db8bb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.970+0000 7ffbb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbb77fe640 1 -- 172.21.15.23:0/2675414024 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8061070 con 0x555f8161dd10 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbe1176740 1 -- 172.21.15.23:0/2675414024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbc802ca70 con 0x555f8161dd10 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbb77fe640 1 -- 172.21.15.23:0/2675414024 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7ffbc803f940 con 0x555f8161dd10 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbb77fe640 1 -- 172.21.15.23:0/2675414024 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffbc8005d50 con 0x555f8161dd10 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbe1176740 10 client.17148 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbe1176740 1 -- 172.21.15.23:0/2675414024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555f81544920 con 0x555f8161dd10 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbb77fe640 1 -- 172.21.15.23:0/2675414024 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc802ca70 con 0x555f8161dd10 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbb77fe640 1 -- 172.21.15.23:0/2675414024 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7ffbc803c040 con 0x555f8161dd10 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbe1176740 10 client.17148 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbe1176740 10 client.17148 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbe1176740 20 client.17148 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbe1176740 1 --2- 172.21.15.23:0/2675414024 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x555f8169d500 0x555f8169f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbe1176740 4 client.17148 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbe1176740 1 -- 172.21.15.23:0/2675414024 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555f8130f890 con 0x555f8169d500 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.971+0000 7ffbd77fe640 1 --2- 172.21.15.23:0/2675414024 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x555f8169d500 0x555f8169f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbd77fe640 1 --2- 172.21.15.23:0/2675414024 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x555f8169d500 0x555f8169f920 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=90b9d3d0ed1d8d69 server_cookie=6ea68b713580db68 in_seq=0 out_seq=0 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbb77fe640 10 client.17148 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbb77fe640 1 -- 172.21.15.23:0/2675414024 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555f8130f890 con 0x555f8169d500 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbb77fe640 10 client.17148 handle_command_reply: tid=0 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbe1176740 1 client.17148 shutdown 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbe1176740 1 -- 172.21.15.23:0/2675414024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161dd10 msgr2=0x555f816211c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbe1176740 1 --2- 172.21.15.23:0/2675414024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161dd10 0x555f816211c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ffbc803f590 tx=0x7ffbc8005760 comp rx=0 tx=0).stop 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbe1176740 1 -- 172.21.15.23:0/2675414024 shutdown_connections 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbe1176740 1 --2- 172.21.15.23:0/2675414024 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x555f8169d500 0x555f8169f920 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbe1176740 1 --2- 172.21.15.23:0/2675414024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f816268f0 0x555f8169b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbe1176740 1 --2- 172.21.15.23:0/2675414024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f8161dd10 0x555f816211c0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbb77fe640 0 client.17148 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:45:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbe1176740 1 --2- 172.21.15.23:0/2675414024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f8161d3f0 0x555f81620c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbe1176740 1 -- 172.21.15.23:0/2675414024 >> 172.21.15.23:0/2675414024 conn(0x555f81617b20 msgr2=0x555f816240e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbe1176740 1 -- 172.21.15.23:0/2675414024 shutdown_connections 2024-03-24T10:45:37.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbe1176740 1 -- 172.21.15.23:0/2675414024 wait complete. 2024-03-24T10:45:37.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.972+0000 7ffbe1176740 20 client.17148 trim_cache size 0 max 16384 2024-03-24T10:45:37.978 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:45:37.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.973+0000 7ffbde5a0640 1 -- 172.21.15.23:0/730686669 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ffbc00a2260 msgr2=0x7ffbc00a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.973+0000 7ffbde5a0640 1 --2- 172.21.15.23:0/730686669 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ffbc00a2260 0x7ffbc00a4720 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00cbd00 tx=0x7ffbd0009200 comp rx=0 tx=0).stop 2024-03-24T10:45:37.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.973+0000 7ffbde5a0640 1 -- 172.21.15.23:0/730686669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd8106660 msgr2=0x7ffbd8102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:37.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.973+0000 7ffbde5a0640 1 --2- 172.21.15.23:0/730686669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd8106660 0x7ffbd8102c90 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000eb10 tx=0x7ffbd000c2d0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.973+0000 7ffbde5a0640 1 -- 172.21.15.23:0/730686669 shutdown_connections 2024-03-24T10:45:37.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.973+0000 7ffbde5a0640 1 --2- 172.21.15.23:0/730686669 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ffbc00a2260 0x7ffbc00a4720 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.973+0000 7ffbde5a0640 1 --2- 172.21.15.23:0/730686669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd8106660 0x7ffbd8102c90 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.973+0000 7ffbde5a0640 1 --2- 172.21.15.23:0/730686669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbd8100f60 0x7ffbd8102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.973+0000 7ffbde5a0640 1 --2- 172.21.15.23:0/730686669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd81005a0 0x7ffbd8102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:37.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.973+0000 7ffbde5a0640 1 -- 172.21.15.23:0/730686669 >> 172.21.15.23:0/730686669 conn(0x7ffbd8077710 msgr2=0x7ffbd80ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:37.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.973+0000 7ffbde5a0640 1 -- 172.21.15.23:0/730686669 shutdown_connections 2024-03-24T10:45:37.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:37.973+0000 7ffbde5a0640 1 -- 172.21.15.23:0/730686669 wait complete. 2024-03-24T10:45:37.984 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:45:38.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:37 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4236678515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:45:38.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:37 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4236678515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:45:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.093+0000 7f6e28993640 1 -- 172.21.15.23:0/1112754606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e241038b0 msgr2=0x7f6e24103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.093+0000 7f6e28993640 1 --2- 172.21.15.23:0/1112754606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e241038b0 0x7f6e24103d30 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6e18009930 tx=0x7f6e1802bc30 comp rx=0 tx=0).stop 2024-03-24T10:45:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.094+0000 7f6e28993640 1 -- 172.21.15.23:0/1112754606 shutdown_connections 2024-03-24T10:45:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.094+0000 7f6e28993640 1 --2- 172.21.15.23:0/1112754606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e24104270 0x7f6e24110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.094+0000 7f6e28993640 1 --2- 172.21.15.23:0/1112754606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e241038b0 0x7f6e24103d30 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.094+0000 7f6e28993640 1 --2- 172.21.15.23:0/1112754606 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e241098c0 0x7f6e24109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.094+0000 7f6e28993640 1 -- 172.21.15.23:0/1112754606 >> 172.21.15.23:0/1112754606 conn(0x7f6e240ff360 msgr2=0x7f6e24101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.094+0000 7f6e28993640 1 -- 172.21.15.23:0/1112754606 shutdown_connections 2024-03-24T10:45:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.094+0000 7f6e28993640 1 -- 172.21.15.23:0/1112754606 wait complete. 2024-03-24T10:45:38.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.094+0000 7f6e28993640 1 Processor -- start 2024-03-24T10:45:38.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.094+0000 7f6e28993640 1 -- start start 2024-03-24T10:45:38.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.094+0000 7f6e28993640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e241038b0 0x7f6e24199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e28993640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e24104270 0x7f6e2419a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e28993640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e241098c0 0x7f6e2419e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e28993640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2419ee70 con 0x7f6e241038b0 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e28993640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2419efe0 con 0x7f6e24104270 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e28993640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2419f1a0 con 0x7f6e241098c0 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e241038b0 0x7f6e24199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e241038b0 0x7f6e24199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47036/0 (socket says 172.21.15.23:47036) 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e2259c640 1 -- 172.21.15.23:0/2318050999 learned_addr learned my addr 172.21.15.23:0/2318050999 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e21d9b640 1 --2- 172.21.15.23:0/2318050999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e24104270 0x7f6e2419a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e22d9d640 1 --2- 172.21.15.23:0/2318050999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e241098c0 0x7f6e2419e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e2259c640 1 -- 172.21.15.23:0/2318050999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e241098c0 msgr2=0x7f6e2419e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e2259c640 1 --2- 172.21.15.23:0/2318050999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e241098c0 0x7f6e2419e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e2259c640 1 -- 172.21.15.23:0/2318050999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e24104270 msgr2=0x7f6e2419a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e2259c640 1 --2- 172.21.15.23:0/2318050999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e24104270 0x7f6e2419a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e2259c640 1 -- 172.21.15.23:0/2318050999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e18009560 con 0x7f6e241038b0 2024-03-24T10:45:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e2259c640 1 --2- 172.21.15.23:0/2318050999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e241038b0 0x7f6e24199fb0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400e9f0 tx=0x7f6e1400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:38.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e137fe640 1 -- 172.21.15.23:0/2318050999 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14009720 con 0x7f6e241038b0 2024-03-24T10:45:38.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e137fe640 1 -- 172.21.15.23:0/2318050999 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e1400cd80 con 0x7f6e241038b0 2024-03-24T10:45:38.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e28993640 1 -- 172.21.15.23:0/2318050999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e2419f480 con 0x7f6e241038b0 2024-03-24T10:45:38.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e137fe640 1 -- 172.21.15.23:0/2318050999 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14010690 con 0x7f6e241038b0 2024-03-24T10:45:38.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.095+0000 7f6e28993640 1 -- 172.21.15.23:0/2318050999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e241a6d00 con 0x7f6e241038b0 2024-03-24T10:45:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.098+0000 7f6e137fe640 1 -- 172.21.15.23:0/2318050999 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f6e140046c0 con 0x7f6e241038b0 2024-03-24T10:45:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.098+0000 7f6e2b57b740 1 Processor -- start 2024-03-24T10:45:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.098+0000 7f6e2b57b740 1 -- start start 2024-03-24T10:45:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.098+0000 7f6e2b57b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 0x555b1ee637d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.098+0000 7f6e2b57b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee63d10 0x555b1ee6c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.098+0000 7f6e2b57b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee6c8f0 0x555b1ee6ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.098+0000 7f6e137fe640 1 --2- 172.21.15.23:0/2318050999 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e0c0a2230 0x7f6e0c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.098+0000 7f6e2b57b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555b1edd1840 con 0x555b1ee63d10 2024-03-24T10:45:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.098+0000 7f6e2b57b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555b1edced70 con 0x555b1ee6c8f0 2024-03-24T10:45:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.098+0000 7f6e2b57b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555b1edca550 con 0x555b1ee633f0 2024-03-24T10:45:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.098+0000 7f6e2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee63d10 0x555b1ee6c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.098+0000 7f6e22d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 0x555b1ee637d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.098+0000 7f6e22d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 0x555b1ee637d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:50414/0 (socket says 172.21.15.23:50414) 2024-03-24T10:45:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.098+0000 7f6e22d9d640 1 -- 172.21.15.23:0/3222848154 learned_addr learned my addr 172.21.15.23:0/3222848154 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e21d9b640 1 --2- 172.21.15.23:0/3222848154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee6c8f0 0x555b1ee6ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e22d9d640 1 --2- 172.21.15.23:0/2318050999 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e0c0a2230 0x7f6e0c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e137fe640 1 -- 172.21.15.23:0/2318050999 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f6e14014070 con 0x7f6e241038b0 2024-03-24T10:45:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e2259c640 1 -- 172.21.15.23:0/3222848154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 msgr2=0x555b1ee637d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e2259c640 1 --2- 172.21.15.23:0/3222848154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 0x555b1ee637d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e2259c640 1 -- 172.21.15.23:0/3222848154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee6c8f0 msgr2=0x555b1ee6ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e2259c640 1 --2- 172.21.15.23:0/3222848154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee6c8f0 0x555b1ee6ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e2259c640 1 -- 172.21.15.23:0/3222848154 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b1ed5fde0 con 0x555b1ee63d10 2024-03-24T10:45:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e21d9b640 1 --2- 172.21.15.23:0/3222848154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee6c8f0 0x555b1ee6ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e22d9d640 1 --2- 172.21.15.23:0/3222848154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 0x555b1ee637d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e2259c640 1 --2- 172.21.15.23:0/3222848154 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee63d10 0x555b1ee6c3b0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6e140cbaf0 tx=0x7f6e1406ce10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a64d3281932cde0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e22d9d640 1 --2- 172.21.15.23:0/2318050999 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e0c0a2230 0x7f6e0c0a46f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c005f40 tx=0x7f6e1c009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e10ff9640 1 -- 172.21.15.23:0/3222848154 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1409e070 con 0x555b1ee63d10 2024-03-24T10:45:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e10ff9640 1 -- 172.21.15.23:0/3222848154 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e1409a6f0 con 0x555b1ee63d10 2024-03-24T10:45:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.099+0000 7f6e10ff9640 1 -- 172.21.15.23:0/3222848154 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14097400 con 0x555b1ee63d10 2024-03-24T10:45:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.100+0000 7f6e2b57b740 1 -- 172.21.15.23:0/3222848154 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee63d10 msgr2=0x555b1ee6c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.100+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/3222848154 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee63d10 0x555b1ee6c3b0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6e140cbaf0 tx=0x7f6e1406ce10 comp rx=0 tx=0).stop 2024-03-24T10:45:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.100+0000 7f6e2b57b740 1 -- 172.21.15.23:0/3222848154 shutdown_connections 2024-03-24T10:45:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.100+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/3222848154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee6c8f0 0x555b1ee6ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.100+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/3222848154 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee63d10 0x555b1ee6c3b0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.100+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/3222848154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 0x555b1ee637d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.100+0000 7f6e2b57b740 1 -- 172.21.15.23:0/3222848154 >> 172.21.15.23:0/3222848154 conn(0x555b1ee5db20 msgr2=0x555b1ee5df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:38.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.100+0000 7f6e2b57b740 1 -- 172.21.15.23:0/3222848154 shutdown_connections 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.100+0000 7f6e2b57b740 1 -- 172.21.15.23:0/3222848154 wait complete. 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.100+0000 7f6e2b57b740 1 -- start start 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.100+0000 7f6e2b57b740 1 Processor -- start 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2b57b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee633f0 0x555b1ee69a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2b57b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee63d10 0x555b1ee69f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2b57b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee6c8f0 0x555b1ee6a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2b57b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555b1ea9d430 con 0x555b1ee633f0 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2b57b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555b1ea59e00 con 0x555b1ee63d10 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2b57b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555b1ed4f590 con 0x555b1ee6c8f0 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2b57b740 10 client.0 _ll_register_callbacks cb 0x555b1edbefc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee633f0 0x555b1ee69a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee633f0 0x555b1ee69a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47054/0 (socket says 172.21.15.23:47054) 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e22d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee6c8f0 0x555b1ee6a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2259c640 1 -- 172.21.15.23:0/3124069526 learned_addr learned my addr 172.21.15.23:0/3124069526 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e21d9b640 1 --2- 172.21.15.23:0/3124069526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee63d10 0x555b1ee69f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2259c640 1 -- 172.21.15.23:0/3124069526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee6c8f0 msgr2=0x555b1ee6a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2259c640 1 --2- 172.21.15.23:0/3124069526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee6c8f0 0x555b1ee6a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2259c640 1 -- 172.21.15.23:0/3124069526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee63d10 msgr2=0x555b1ee69f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2259c640 1 --2- 172.21.15.23:0/3124069526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee63d10 0x555b1ee69f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2259c640 1 -- 172.21.15.23:0/3124069526 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e241a6d00 con 0x555b1ee633f0 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2259c640 1 --2- 172.21.15.23:0/3124069526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee633f0 0x555b1ee69a10 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6e140cc0f0 tx=0x7f6e140cc010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dcf115193b857ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e09ffb640 1 -- 172.21.15.23:0/3124069526 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e140c3ea0 con 0x555b1ee633f0 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e09ffb640 1 -- 172.21.15.23:0/3124069526 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f6e1409ad90 con 0x555b1ee633f0 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.101+0000 7f6e2b57b740 1 -- 172.21.15.23:0/3124069526 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555b1eac97e0 con 0x555b1ee633f0 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.102+0000 7f6e09ffb640 1 -- 172.21.15.23:0/3124069526 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6e140b9070 con 0x555b1ee633f0 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.102+0000 7f6e2b57b740 10 client.17160 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.102+0000 7f6e2b57b740 1 -- 172.21.15.23:0/3124069526 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555b1edc8bd0 con 0x555b1ee633f0 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.102+0000 7f6e09ffb640 1 -- 172.21.15.23:0/3124069526 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555b1eac97e0 con 0x555b1ee633f0 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.102+0000 7f6e09ffb640 1 -- 172.21.15.23:0/3124069526 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f6e1406ad20 con 0x555b1ee633f0 2024-03-24T10:45:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.102+0000 7f6e2b57b740 10 client.17160 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.102+0000 7f6e2b57b740 10 client.17160 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.102+0000 7f6e2b57b740 20 client.17160 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.102+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/3124069526 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555b1ee7c560 0x555b1ee7e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.102+0000 7f6e2b57b740 4 client.17160 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.102+0000 7f6e2b57b740 1 -- 172.21.15.23:0/3124069526 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b1eb06720 con 0x555b1ee7c560 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.102+0000 7f6e21d9b640 1 --2- 172.21.15.23:0/3124069526 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555b1ee7c560 0x555b1ee7e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.103+0000 7f6e21d9b640 1 --2- 172.21.15.23:0/3124069526 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555b1ee7c560 0x555b1ee7e990 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6515b761551f54f8 server_cookie=22d17887413a9309 in_seq=0 out_seq=0 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.103+0000 7f6e09ffb640 10 client.17160 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.103+0000 7f6e09ffb640 1 -- 172.21.15.23:0/3124069526 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555b1eb06720 con 0x555b1ee7c560 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.103+0000 7f6e09ffb640 10 client.17160 handle_command_reply: tid=0 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.104+0000 7f6e2b57b740 1 client.17160 shutdown 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.104+0000 7f6e2b57b740 1 -- 172.21.15.23:0/3124069526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee633f0 msgr2=0x555b1ee69a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.104+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/3124069526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee633f0 0x555b1ee69a10 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6e140cc0f0 tx=0x7f6e140cc010 comp rx=0 tx=0).stop 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.104+0000 7f6e2b57b740 1 -- 172.21.15.23:0/3124069526 shutdown_connections 2024-03-24T10:45:38.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.104+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/3124069526 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555b1ee7c560 0x555b1ee7e990 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.104+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/3124069526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee6c8f0 0x555b1ee6a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.104+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/3124069526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee63d10 0x555b1ee69f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.104+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/3124069526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee633f0 0x555b1ee69a10 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.104+0000 7f6e09ffb640 0 client.17160 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:38.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.104+0000 7f6e2b57b740 1 -- 172.21.15.23:0/3124069526 >> 172.21.15.23:0/3124069526 conn(0x555b1ee5db20 msgr2=0x555b1ee5f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:38.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.104+0000 7f6e2b57b740 1 -- 172.21.15.23:0/3124069526 shutdown_connections 2024-03-24T10:45:38.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.104+0000 7f6e2b57b740 1 -- 172.21.15.23:0/3124069526 wait complete. 2024-03-24T10:45:38.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.104+0000 7f6e2b57b740 20 client.17160 trim_cache size 0 max 16384 2024-03-24T10:45:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e2b57b740 1 Processor -- start 2024-03-24T10:45:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e2b57b740 1 -- start start 2024-03-24T10:45:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e2b57b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 0x555b1ee78180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e2b57b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee63d10 0x555b1ee5ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e2b57b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee6c8f0 0x555b1ee841a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e2b57b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555b1eb06720 con 0x555b1ee6c8f0 2024-03-24T10:45:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e2b57b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555b1eac97e0 con 0x555b1ee63d10 2024-03-24T10:45:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e2b57b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555b1ee59210 con 0x555b1ee633f0 2024-03-24T10:45:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e22d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee6c8f0 0x555b1ee841a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e22d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee6c8f0 0x555b1ee841a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47068/0 (socket says 172.21.15.23:47068) 2024-03-24T10:45:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e22d9d640 1 -- 172.21.15.23:0/2522091176 learned_addr learned my addr 172.21.15.23:0/2522091176 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e21d9b640 1 --2- 172.21.15.23:0/2522091176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee63d10 0x555b1ee5ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e2259c640 1 --2- 172.21.15.23:0/2522091176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 0x555b1ee78180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e22d9d640 1 -- 172.21.15.23:0/2522091176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 msgr2=0x555b1ee78180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e22d9d640 1 --2- 172.21.15.23:0/2522091176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 0x555b1ee78180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e22d9d640 1 -- 172.21.15.23:0/2522091176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee63d10 msgr2=0x555b1ee5ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e22d9d640 1 --2- 172.21.15.23:0/2522091176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee63d10 0x555b1ee5ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e22d9d640 1 -- 172.21.15.23:0/2522091176 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b1ee5d3c0 con 0x555b1ee6c8f0 2024-03-24T10:45:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.118+0000 7f6e22d9d640 1 --2- 172.21.15.23:0/2522091176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee6c8f0 0x555b1ee841a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009fd0 tx=0x7f6e1c013070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4328ebc75aa7e04 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.119+0000 7f6e0bfff640 1 -- 172.21.15.23:0/2522091176 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c025040 con 0x555b1ee6c8f0 2024-03-24T10:45:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.119+0000 7f6e0bfff640 1 -- 172.21.15.23:0/2522091176 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e1c003e50 con 0x555b1ee6c8f0 2024-03-24T10:45:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.119+0000 7f6e0bfff640 1 -- 172.21.15.23:0/2522091176 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c005020 con 0x555b1ee6c8f0 2024-03-24T10:45:38.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.119+0000 7f6e2b57b740 1 -- 172.21.15.23:0/2522091176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee6c8f0 msgr2=0x555b1ee841a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:38.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.119+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/2522091176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee6c8f0 0x555b1ee841a0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009fd0 tx=0x7f6e1c013070 comp rx=0 tx=0).stop 2024-03-24T10:45:38.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.119+0000 7f6e2b57b740 1 -- 172.21.15.23:0/2522091176 shutdown_connections 2024-03-24T10:45:38.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.119+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/2522091176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee6c8f0 0x555b1ee841a0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.119+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/2522091176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee63d10 0x555b1ee5ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.119+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/2522091176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 0x555b1ee78180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.119+0000 7f6e2b57b740 1 -- 172.21.15.23:0/2522091176 >> 172.21.15.23:0/2522091176 conn(0x555b1ee5db20 msgr2=0x555b1ee699d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:38.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.119+0000 7f6e2b57b740 1 -- 172.21.15.23:0/2522091176 shutdown_connections 2024-03-24T10:45:38.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.119+0000 7f6e2b57b740 1 -- 172.21.15.23:0/2522091176 wait complete. 2024-03-24T10:45:38.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.120+0000 7f6e2b57b740 1 -- start start 2024-03-24T10:45:38.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.120+0000 7f6e2b57b740 1 Processor -- start 2024-03-24T10:45:38.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.120+0000 7f6e2b57b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 0x555b1ee74bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.120+0000 7f6e2b57b740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee63d10 0x555b1ee70f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.120+0000 7f6e2b57b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee6c8f0 0x555b1ee71670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.120+0000 7f6e2b57b740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555b1ee71d40 con 0x555b1ee63d10 2024-03-24T10:45:38.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.120+0000 7f6e2b57b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555b1eee27e0 con 0x555b1ee6c8f0 2024-03-24T10:45:38.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.120+0000 7f6e2b57b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555b1eee2920 con 0x555b1ee633f0 2024-03-24T10:45:38.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.120+0000 7f6e2b57b740 10 client.0 _ll_register_callbacks cb 0x7f6e14005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:38.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.120+0000 7f6e21d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee63d10 0x555b1ee70f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.120+0000 7f6e21d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee63d10 0x555b1ee70f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47080/0 (socket says 172.21.15.23:47080) 2024-03-24T10:45:38.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.120+0000 7f6e21d9b640 1 -- 172.21.15.23:0/815859725 learned_addr learned my addr 172.21.15.23:0/815859725 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:38.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.120+0000 7f6e2259c640 1 --2- 172.21.15.23:0/815859725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 0x555b1ee74bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.120+0000 7f6e22d9d640 1 --2- 172.21.15.23:0/815859725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee6c8f0 0x555b1ee71670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.121+0000 7f6e21d9b640 1 -- 172.21.15.23:0/815859725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 msgr2=0x555b1ee74bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:38.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.121+0000 7f6e21d9b640 1 --2- 172.21.15.23:0/815859725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 0x555b1ee74bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.121+0000 7f6e21d9b640 1 -- 172.21.15.23:0/815859725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee6c8f0 msgr2=0x555b1ee71670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:38.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.121+0000 7f6e21d9b640 1 --2- 172.21.15.23:0/815859725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee6c8f0 0x555b1ee71670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.121+0000 7f6e21d9b640 1 -- 172.21.15.23:0/815859725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e1c011e70 con 0x555b1ee63d10 2024-03-24T10:45:38.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 520192 11.70 MB/sec execute 115 sec latency 22.689 ms 2024-03-24T10:45:38.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.121+0000 7f6e21d9b640 1 --2- 172.21.15.23:0/815859725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee63d10 0x555b1ee70f30 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6e18036040 tx=0x7f6e18004a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3677674d58614012 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.121+0000 7f6e09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.121+0000 7f6e09ffb640 1 -- 172.21.15.23:0/815859725 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e18046040 con 0x555b1ee63d10 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.121+0000 7f6e09ffb640 1 -- 172.21.15.23:0/815859725 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f6e1804ad20 con 0x555b1ee63d10 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.121+0000 7f6e2b57b740 1 -- 172.21.15.23:0/815859725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e1409a6f0 con 0x555b1ee63d10 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.121+0000 7f6e09ffb640 1 -- 172.21.15.23:0/815859725 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6e18004b20 con 0x555b1ee63d10 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.121+0000 7f6e2b57b740 10 client.17166 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.121+0000 7f6e2b57b740 1 -- 172.21.15.23:0/815859725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555b1ed64c80 con 0x555b1ee63d10 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.121+0000 7f6e09ffb640 1 -- 172.21.15.23:0/815859725 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e1409a6f0 con 0x555b1ee63d10 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.122+0000 7f6e09ffb640 1 -- 172.21.15.23:0/815859725 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f6e1803c040 con 0x555b1ee63d10 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.122+0000 7f6e2b57b740 10 client.17166 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.122+0000 7f6e2b57b740 10 client.17166 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.122+0000 7f6e2b57b740 20 client.17166 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.122+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/815859725 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555b1eee4110 0x555b1eee6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.122+0000 7f6e2b57b740 4 client.17166 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.122+0000 7f6e2b57b740 1 -- 172.21.15.23:0/815859725 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555b1eb55890 con 0x555b1eee4110 2024-03-24T10:45:38.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.122+0000 7f6e2259c640 1 --2- 172.21.15.23:0/815859725 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555b1eee4110 0x555b1eee6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.122+0000 7f6e09ffb640 10 client.17166 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.122+0000 7f6e2259c640 1 --2- 172.21.15.23:0/815859725 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555b1eee4110 0x555b1eee6530 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3840ccbc6b8060fd server_cookie=4119c2cb0023030 in_seq=0 out_seq=0 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.122+0000 7f6e09ffb640 1 -- 172.21.15.23:0/815859725 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555b1eb55890 con 0x555b1eee4110 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.122+0000 7f6e09ffb640 10 client.17166 handle_command_reply: tid=0 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e2b57b740 1 client.17166 shutdown 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e2b57b740 1 -- 172.21.15.23:0/815859725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee63d10 msgr2=0x555b1ee70f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/815859725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee63d10 0x555b1ee70f30 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6e18036040 tx=0x7f6e18004a30 comp rx=0 tx=0).stop 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e2b57b740 1 -- 172.21.15.23:0/815859725 shutdown_connections 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/815859725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b1ee6c8f0 0x555b1ee71670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/815859725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b1ee63d10 0x555b1ee70f30 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/815859725 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555b1eee4110 0x555b1eee6530 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e2b57b740 1 --2- 172.21.15.23:0/815859725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b1ee633f0 0x555b1ee74bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e2b57b740 1 -- 172.21.15.23:0/815859725 >> 172.21.15.23:0/815859725 conn(0x555b1ee5db20 msgr2=0x555b1ee5cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e09ffb640 0 client.17166 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:38.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e2b57b740 1 -- 172.21.15.23:0/815859725 shutdown_connections 2024-03-24T10:45:38.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e2b57b740 1 -- 172.21.15.23:0/815859725 wait complete. 2024-03-24T10:45:38.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e2b57b740 20 client.17166 trim_cache size 0 max 16384 2024-03-24T10:45:38.129 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:45:38.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e28993640 1 -- 172.21.15.23:0/2318050999 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e0c0a2230 msgr2=0x7f6e0c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:38.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e28993640 1 --2- 172.21.15.23:0/2318050999 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e0c0a2230 0x7f6e0c0a46f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c005f40 tx=0x7f6e1c009500 comp rx=0 tx=0).stop 2024-03-24T10:45:38.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e28993640 1 -- 172.21.15.23:0/2318050999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e241038b0 msgr2=0x7f6e24199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:38.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e28993640 1 --2- 172.21.15.23:0/2318050999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e241038b0 0x7f6e24199fb0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400e9f0 tx=0x7f6e1400edf0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e28993640 1 -- 172.21.15.23:0/2318050999 shutdown_connections 2024-03-24T10:45:38.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e28993640 1 --2- 172.21.15.23:0/2318050999 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6e0c0a2230 0x7f6e0c0a46f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e28993640 1 --2- 172.21.15.23:0/2318050999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e241098c0 0x7f6e2419e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e28993640 1 --2- 172.21.15.23:0/2318050999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e24104270 0x7f6e2419a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e28993640 1 --2- 172.21.15.23:0/2318050999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e241038b0 0x7f6e24199fb0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:38.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.123+0000 7f6e28993640 1 -- 172.21.15.23:0/2318050999 >> 172.21.15.23:0/2318050999 conn(0x7f6e240ff360 msgr2=0x7f6e240ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:38.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.124+0000 7f6e28993640 1 -- 172.21.15.23:0/2318050999 shutdown_connections 2024-03-24T10:45:38.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:38.124+0000 7f6e28993640 1 -- 172.21.15.23:0/2318050999 wait complete. 2024-03-24T10:45:39.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:38 smithi117 ceph-mon[36368]: pgmap v170: 97 pgs: 97 active+clean; 299 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 265 op/s 2024-03-24T10:45:39.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:38 smithi116 ceph-mon[36248]: pgmap v170: 97 pgs: 97 active+clean; 299 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 265 op/s 2024-03-24T10:45:39.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 522180 11.71 MB/sec execute 116 sec latency 163.639 ms 2024-03-24T10:45:39.134 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:45:39.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:45:39.135 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:45:39.158 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:38 smithi023 ceph-mon[32466]: pgmap v170: 97 pgs: 97 active+clean; 299 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.4 MiB/s wr, 265 op/s 2024-03-24T10:45:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.246+0000 7f2fa02eb640 1 -- 172.21.15.23:0/2703215064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f98106660 msgr2=0x7f2f98106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.246+0000 7f2fa02eb640 1 --2- 172.21.15.23:0/2703215064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f98106660 0x7f2f98106a40 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009900 tx=0x7f2f9402bc30 comp rx=0 tx=0).stop 2024-03-24T10:45:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.246+0000 7f2fa02eb640 1 -- 172.21.15.23:0/2703215064 shutdown_connections 2024-03-24T10:45:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.246+0000 7f2fa02eb640 1 --2- 172.21.15.23:0/2703215064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f98100f60 0x7f2f9810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.246+0000 7f2fa02eb640 1 --2- 172.21.15.23:0/2703215064 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f981005a0 0x7f2f98100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.246+0000 7f2fa02eb640 1 --2- 172.21.15.23:0/2703215064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f98106660 0x7f2f98106a40 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.246+0000 7f2fa02eb640 1 -- 172.21.15.23:0/2703215064 >> 172.21.15.23:0/2703215064 conn(0x7f2f98077710 msgr2=0x7f2f98077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.246+0000 7f2fa02eb640 1 -- 172.21.15.23:0/2703215064 shutdown_connections 2024-03-24T10:45:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.246+0000 7f2fa02eb640 1 -- 172.21.15.23:0/2703215064 wait complete. 2024-03-24T10:45:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2fa02eb640 1 Processor -- start 2024-03-24T10:45:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2fa02eb640 1 -- start start 2024-03-24T10:45:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2fa02eb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f981005a0 0x7f2f9819d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2fa02eb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f98100f60 0x7f2f9819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2fa02eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f98106660 0x7f2f98197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2fa02eb640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2f98198100 con 0x7f2f981005a0 2024-03-24T10:45:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2fa02eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2f98198270 con 0x7f2f98106660 2024-03-24T10:45:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2fa02eb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2f98198430 con 0x7f2f98100f60 2024-03-24T10:45:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2f9e087640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f981005a0 0x7f2f9819d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2f9e087640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f981005a0 0x7f2f9819d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47084/0 (socket says 172.21.15.23:47084) 2024-03-24T10:45:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2f9e087640 1 -- 172.21.15.23:0/1532458889 learned_addr learned my addr 172.21.15.23:0/1532458889 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2f9e888640 1 --2- 172.21.15.23:0/1532458889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f98106660 0x7f2f98197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2f9d886640 1 --2- 172.21.15.23:0/1532458889 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f98100f60 0x7f2f9819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2f9e087640 1 -- 172.21.15.23:0/1532458889 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f98100f60 msgr2=0x7f2f9819de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2f9e087640 1 --2- 172.21.15.23:0/1532458889 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f98100f60 0x7f2f9819de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2f9e087640 1 -- 172.21.15.23:0/1532458889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f98106660 msgr2=0x7f2f98197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2f9e087640 1 --2- 172.21.15.23:0/1532458889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f98106660 0x7f2f98197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.247+0000 7f2f9e087640 1 -- 172.21.15.23:0/1532458889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f94009560 con 0x7f2f981005a0 2024-03-24T10:45:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.248+0000 7f2f9e087640 1 --2- 172.21.15.23:0/1532458889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f981005a0 0x7f2f9819d900 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f2f94005940 tx=0x7f2f9402c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.248+0000 7f2f8f7fe640 1 -- 172.21.15.23:0/1532458889 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f940042d0 con 0x7f2f981005a0 2024-03-24T10:45:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.248+0000 7f2f8f7fe640 1 -- 172.21.15.23:0/1532458889 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f94004430 con 0x7f2f981005a0 2024-03-24T10:45:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.248+0000 7f2fa02eb640 1 -- 172.21.15.23:0/1532458889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f981986b0 con 0x7f2f981005a0 2024-03-24T10:45:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.248+0000 7f2f8f7fe640 1 -- 172.21.15.23:0/1532458889 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f94003c50 con 0x7f2f981005a0 2024-03-24T10:45:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.248+0000 7f2fa02eb640 1 -- 172.21.15.23:0/1532458889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f981a49a0 con 0x7f2f981005a0 2024-03-24T10:45:39.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.250+0000 7f2f8f7fe640 1 -- 172.21.15.23:0/1532458889 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f2f940054f0 con 0x7f2f981005a0 2024-03-24T10:45:39.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.250+0000 7f2fa2fa0740 1 Processor -- start 2024-03-24T10:45:39.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2fa2fa0740 1 -- start start 2024-03-24T10:45:39.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2fa2fa0740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c36a760 0x56464c36ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2fa2fa0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36b080 0x56464c373720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2fa2fa0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c373c60 0x56464c376050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2fa2fa0740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56464c2d8530 con 0x56464c36a760 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2fa2fa0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56464c2d5a60 con 0x56464c373c60 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2fa2fa0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56464c2d1240 con 0x56464c36b080 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2f8f7fe640 1 --2- 172.21.15.23:0/1532458889 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2f840a2230 0x7f2f840a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2f9d886640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c373c60 0x56464c376050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2f9e888640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c36a760 0x56464c36ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2f9e087640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36b080 0x56464c373720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2f9e888640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c36a760 0x56464c36ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47098/0 (socket says 172.21.15.23:47098) 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2f9d886640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c373c60 0x56464c376050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:52682/0 (socket says 172.21.15.23:52682) 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2f9e888640 1 -- 172.21.15.23:0/2138584288 learned_addr learned my addr 172.21.15.23:0/2138584288 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2f9e888640 1 --2- 172.21.15.23:0/1532458889 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2f840a2230 0x7f2f840a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2f8f7fe640 1 -- 172.21.15.23:0/1532458889 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f2f940f0030 con 0x7f2f981005a0 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2f9d886640 1 -- 172.21.15.23:0/2138584288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36b080 msgr2=0x56464c373720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2f9d886640 1 --2- 172.21.15.23:0/2138584288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36b080 0x56464c373720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2f9d886640 1 -- 172.21.15.23:0/2138584288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c36a760 msgr2=0x56464c36ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2f9d886640 1 --2- 172.21.15.23:0/2138584288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c36a760 0x56464c36ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.251+0000 7f2f9d886640 1 -- 172.21.15.23:0/2138584288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56464c2cf8c0 con 0x56464c373c60 2024-03-24T10:45:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.252+0000 7f2f9e888640 1 --2- 172.21.15.23:0/1532458889 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2f840a2230 0x7f2f840a46f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2f900099e0 tx=0x7f2f90009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.252+0000 7f2f9d886640 1 --2- 172.21.15.23:0/2138584288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c373c60 0x56464c376050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2f981992a0 tx=0x7f2f88011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9306de6503cc5a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.252+0000 7f2f8cbfe640 1 -- 172.21.15.23:0/2138584288 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f88004230 con 0x56464c373c60 2024-03-24T10:45:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.252+0000 7f2f8cbfe640 1 -- 172.21.15.23:0/2138584288 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f88009e10 con 0x56464c373c60 2024-03-24T10:45:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.252+0000 7f2f8cbfe640 1 -- 172.21.15.23:0/2138584288 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f88004c30 con 0x56464c373c60 2024-03-24T10:45:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/2138584288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c373c60 msgr2=0x56464c376050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/2138584288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c373c60 0x56464c376050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2f981992a0 tx=0x7f2f88011810 comp rx=0 tx=0).stop 2024-03-24T10:45:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/2138584288 shutdown_connections 2024-03-24T10:45:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/2138584288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c373c60 0x56464c376050 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/2138584288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36b080 0x56464c373720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/2138584288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c36a760 0x56464c36ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/2138584288 >> 172.21.15.23:0/2138584288 conn(0x56464c364e90 msgr2=0x56464c365270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/2138584288 shutdown_connections 2024-03-24T10:45:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/2138584288 wait complete. 2024-03-24T10:45:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 -- start start 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 Processor -- start 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36a760 0x56464c370c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c36b080 0x56464c3711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c373c60 0x56464c3716f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56464c275f80 con 0x56464c36b080 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56464bfa44b0 con 0x56464c373c60 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56464c256ad0 con 0x56464c36a760 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.253+0000 7f2fa2fa0740 10 client.0 _ll_register_callbacks cb 0x56464c289b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f9d886640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c36b080 0x56464c3711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f9d886640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c36b080 0x56464c3711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47102/0 (socket says 172.21.15.23:47102) 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f9d886640 1 -- 172.21.15.23:0/1638870408 learned_addr learned my addr 172.21.15.23:0/1638870408 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f9e888640 1 --2- 172.21.15.23:0/1638870408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c373c60 0x56464c3716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f9e087640 1 --2- 172.21.15.23:0/1638870408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36a760 0x56464c370c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f9d886640 1 -- 172.21.15.23:0/1638870408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36a760 msgr2=0x56464c370c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f9d886640 1 --2- 172.21.15.23:0/1638870408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36a760 0x56464c370c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f9d886640 1 -- 172.21.15.23:0/1638870408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c373c60 msgr2=0x56464c3716f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f9d886640 1 --2- 172.21.15.23:0/1638870408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c373c60 0x56464c3716f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f9d886640 1 -- 172.21.15.23:0/1638870408 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f88009830 con 0x56464c36b080 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f9e888640 1 --2- 172.21.15.23:0/1638870408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c373c60 0x56464c3716f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f9d886640 1 --2- 172.21.15.23:0/1638870408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c36b080 0x56464c3711b0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f2f88011fd0 tx=0x7f2f88019ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ec8b5b6fb381be server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f81ffb640 1 -- 172.21.15.23:0/1638870408 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f88005680 con 0x56464c36b080 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f81ffb640 1 -- 172.21.15.23:0/1638870408 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f2f880046c0 con 0x56464c36b080 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/1638870408 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56464c2f57a0 con 0x56464c36b080 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2f81ffb640 1 -- 172.21.15.23:0/1638870408 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2f88024470 con 0x56464c36b080 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2fa2fa0740 10 client.17178 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.254+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/1638870408 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56464c2876d0 con 0x56464c36b080 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.255+0000 7f2f81ffb640 1 -- 172.21.15.23:0/1638870408 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56464c2f57a0 con 0x56464c36b080 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.255+0000 7f2f81ffb640 1 -- 172.21.15.23:0/1638870408 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f2f88023700 con 0x56464c36b080 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.255+0000 7f2fa2fa0740 10 client.17178 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.255+0000 7f2fa2fa0740 10 client.17178 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.255+0000 7f2fa2fa0740 20 client.17178 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.255+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/1638870408 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56464c383de0 0x56464c386200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.255+0000 7f2fa2fa0740 4 client.17178 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.255+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/1638870408 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56464c2c71f0 con 0x56464c383de0 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.255+0000 7f2f9e087640 1 --2- 172.21.15.23:0/1638870408 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56464c383de0 0x56464c386200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.255+0000 7f2f9e087640 1 --2- 172.21.15.23:0/1638870408 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56464c383de0 0x56464c386200 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d41d27973bb3625c server_cookie=cf19a9ebff624724 in_seq=0 out_seq=0 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.255+0000 7f2f81ffb640 10 client.17178 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.256+0000 7f2f81ffb640 1 -- 172.21.15.23:0/1638870408 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56464c2c71f0 con 0x56464c383de0 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.256+0000 7f2f81ffb640 10 client.17178 handle_command_reply: tid=0 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.256+0000 7f2fa2fa0740 1 client.17178 shutdown 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.257+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/1638870408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c36b080 msgr2=0x56464c3711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.257+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/1638870408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c36b080 0x56464c3711b0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f2f88011fd0 tx=0x7f2f88019ae0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.257+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/1638870408 shutdown_connections 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.257+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/1638870408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c373c60 0x56464c3716f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.257+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/1638870408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c36b080 0x56464c3711b0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.257+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/1638870408 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56464c383de0 0x56464c386200 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.257+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/1638870408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36a760 0x56464c370c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.257+0000 7f2f81ffb640 0 client.17178 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.257+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/1638870408 >> 172.21.15.23:0/1638870408 conn(0x56464c364e90 msgr2=0x56464c366be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:39.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.257+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/1638870408 shutdown_connections 2024-03-24T10:45:39.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.257+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/1638870408 wait complete. 2024-03-24T10:45:39.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.257+0000 7f2fa2fa0740 20 client.17178 trim_cache size 0 max 16384 2024-03-24T10:45:39.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.274+0000 7f2fa2fa0740 1 Processor -- start 2024-03-24T10:45:39.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.274+0000 7f2fa2fa0740 1 -- start start 2024-03-24T10:45:39.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2fa2fa0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36a760 0x56464c3733a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2fa2fa0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c36b080 0x56464c37f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2fa2fa0740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c373c60 0x56464c38bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2fa2fa0740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56464c364670 con 0x56464c373c60 2024-03-24T10:45:39.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2fa2fa0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56464c360620 con 0x56464c36b080 2024-03-24T10:45:39.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2fa2fa0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56464c365e90 con 0x56464c36a760 2024-03-24T10:45:39.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2f9e888640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c373c60 0x56464c38bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2f9e888640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c373c60 0x56464c38bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47110/0 (socket says 172.21.15.23:47110) 2024-03-24T10:45:39.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2f9e888640 1 -- 172.21.15.23:0/823577829 learned_addr learned my addr 172.21.15.23:0/823577829 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:39.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2f9e087640 1 --2- 172.21.15.23:0/823577829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36a760 0x56464c3733a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2f9d886640 1 --2- 172.21.15.23:0/823577829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c36b080 0x56464c37f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2f9e888640 1 -- 172.21.15.23:0/823577829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36a760 msgr2=0x56464c3733a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2f9e888640 1 --2- 172.21.15.23:0/823577829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36a760 0x56464c3733a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2f9e888640 1 -- 172.21.15.23:0/823577829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c36b080 msgr2=0x56464c37f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2f9e888640 1 --2- 172.21.15.23:0/823577829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c36b080 0x56464c37f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2f9e888640 1 -- 172.21.15.23:0/823577829 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56464c26bc60 con 0x56464c373c60 2024-03-24T10:45:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2f9d886640 1 --2- 172.21.15.23:0/823577829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c36b080 0x56464c37f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2f9e888640 1 --2- 172.21.15.23:0/823577829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c373c60 0x56464c38bb10 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2f90011a50 tx=0x7f2f90008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9e261b8938cd15 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.275+0000 7f2f83fff640 1 -- 172.21.15.23:0/823577829 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f90026040 con 0x56464c373c60 2024-03-24T10:45:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.276+0000 7f2f83fff640 1 -- 172.21.15.23:0/823577829 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f900045c0 con 0x56464c373c60 2024-03-24T10:45:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.276+0000 7f2f83fff640 1 -- 172.21.15.23:0/823577829 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f900058e0 con 0x56464c373c60 2024-03-24T10:45:39.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.276+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/823577829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c373c60 msgr2=0x56464c38bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.276+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/823577829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c373c60 0x56464c38bb10 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2f90011a50 tx=0x7f2f90008000 comp rx=0 tx=0).stop 2024-03-24T10:45:39.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.276+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/823577829 shutdown_connections 2024-03-24T10:45:39.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.276+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/823577829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c373c60 0x56464c38bb10 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.276+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/823577829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c36b080 0x56464c37f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.276+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/823577829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36a760 0x56464c3733a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.276+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/823577829 >> 172.21.15.23:0/823577829 conn(0x56464c364e90 msgr2=0x56464c366a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.276+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/823577829 shutdown_connections 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.276+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/823577829 wait complete. 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2fa2fa0740 1 -- start start 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2fa2fa0740 1 Processor -- start 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2fa2fa0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c36a760 0x56464c378af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2fa2fa0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36b080 0x56464c379030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2fa2fa0740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c373c60 0x56464c379570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2fa2fa0740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56464c3ea1d0 con 0x56464c373c60 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2fa2fa0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56464c3ea340 con 0x56464c36a760 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2fa2fa0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56464c3ea500 con 0x56464c36b080 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2fa2fa0740 10 client.0 _ll_register_callbacks cb 0x7f2f88009a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2f9e888640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c373c60 0x56464c379570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2f9e888640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c373c60 0x56464c379570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47122/0 (socket says 172.21.15.23:47122) 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2f9e087640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c36a760 0x56464c378af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2f9e888640 1 -- 172.21.15.23:0/3776851098 learned_addr learned my addr 172.21.15.23:0/3776851098 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2f9d886640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36b080 0x56464c379030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2f9e888640 1 -- 172.21.15.23:0/3776851098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36b080 msgr2=0x56464c379030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2f9e888640 1 --2- 172.21.15.23:0/3776851098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36b080 0x56464c379030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2f9e888640 1 -- 172.21.15.23:0/3776851098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c36a760 msgr2=0x56464c378af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2f9e888640 1 --2- 172.21.15.23:0/3776851098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c36a760 0x56464c378af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.277+0000 7f2f9e888640 1 -- 172.21.15.23:0/3776851098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56464c2876d0 con 0x56464c373c60 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2f9e888640 1 --2- 172.21.15.23:0/3776851098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c373c60 0x56464c379570 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000a010 tx=0x7f2f90005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c447af1beb570a45 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2f81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2f81ffb640 1 -- 172.21.15.23:0/3776851098 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f90005860 con 0x56464c373c60 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/3776851098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f88009e10 con 0x56464c373c60 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2f81ffb640 1 -- 172.21.15.23:0/3776851098 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f2f90004830 con 0x56464c373c60 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2f81ffb640 1 -- 172.21.15.23:0/3776851098 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2f900059c0 con 0x56464c373c60 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2fa2fa0740 10 client.17184 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/3776851098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x7f2f90019040 con 0x56464c373c60 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2f81ffb640 1 -- 172.21.15.23:0/3776851098 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f9003f020 con 0x56464c373c60 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2f81ffb640 1 -- 172.21.15.23:0/3776851098 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f2f9002a830 con 0x56464c373c60 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2fa2fa0740 10 client.17184 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2fa2fa0740 10 client.17184 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2fa2fa0740 20 client.17184 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/3776851098 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56464c3ebc30 0x56464c3ee050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2fa2fa0740 4 client.17184 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.278+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/3776851098 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56464bfd0860 con 0x56464c3ebc30 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.279+0000 7f2f9e087640 1 --2- 172.21.15.23:0/3776851098 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56464c3ebc30 0x56464c3ee050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.279+0000 7f2f9e087640 1 --2- 172.21.15.23:0/3776851098 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56464c3ebc30 0x56464c3ee050 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a8b2dbbe4a3cfd1 server_cookie=2b267191253dca5 in_seq=0 out_seq=0 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.279+0000 7f2f81ffb640 10 client.17184 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2f81ffb640 1 -- 172.21.15.23:0/3776851098 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56464bfd0860 con 0x56464c3ebc30 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2f81ffb640 10 client.17184 handle_command_reply: tid=0 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2fa2fa0740 1 client.17184 shutdown 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/3776851098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c373c60 msgr2=0x56464c379570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/3776851098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c373c60 0x56464c379570 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000a010 tx=0x7f2f90005bd0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/3776851098 shutdown_connections 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/3776851098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56464c373c60 0x56464c379570 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/3776851098 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56464c3ebc30 0x56464c3ee050 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/3776851098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56464c36b080 0x56464c379030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2fa2fa0740 1 --2- 172.21.15.23:0/3776851098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56464c36a760 0x56464c378af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/3776851098 >> 172.21.15.23:0/3776851098 conn(0x56464c364e90 msgr2=0x56464c363660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2f81ffb640 0 client.17184 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/3776851098 shutdown_connections 2024-03-24T10:45:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2fa2fa0740 1 -- 172.21.15.23:0/3776851098 wait complete. 2024-03-24T10:45:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.280+0000 7f2fa2fa0740 20 client.17184 trim_cache size 0 max 16384 2024-03-24T10:45:39.285 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:45:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:45:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "cbb0919c-826e-4039-81c0-4f7482c65b30", 2024-03-24T10:45:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:45:39.285 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:45:39.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.281+0000 7f2fa02eb640 1 -- 172.21.15.23:0/1532458889 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2f840a2230 msgr2=0x7f2f840a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.281+0000 7f2fa02eb640 1 --2- 172.21.15.23:0/1532458889 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2f840a2230 0x7f2f840a46f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2f900099e0 tx=0x7f2f90009500 comp rx=0 tx=0).stop 2024-03-24T10:45:39.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.281+0000 7f2fa02eb640 1 -- 172.21.15.23:0/1532458889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f981005a0 msgr2=0x7f2f9819d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.281+0000 7f2fa02eb640 1 --2- 172.21.15.23:0/1532458889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f981005a0 0x7f2f9819d900 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f2f94005940 tx=0x7f2f9402c670 comp rx=0 tx=0).stop 2024-03-24T10:45:39.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.281+0000 7f2fa02eb640 1 -- 172.21.15.23:0/1532458889 shutdown_connections 2024-03-24T10:45:39.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.281+0000 7f2fa02eb640 1 --2- 172.21.15.23:0/1532458889 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2f840a2230 0x7f2f840a46f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.281+0000 7f2fa02eb640 1 --2- 172.21.15.23:0/1532458889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f98106660 0x7f2f98197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.281+0000 7f2fa02eb640 1 --2- 172.21.15.23:0/1532458889 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f98100f60 0x7f2f9819de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.281+0000 7f2fa02eb640 1 --2- 172.21.15.23:0/1532458889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f981005a0 0x7f2f9819d900 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.281+0000 7f2fa02eb640 1 -- 172.21.15.23:0/1532458889 >> 172.21.15.23:0/1532458889 conn(0x7f2f98077710 msgr2=0x7f2f981047a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:39.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.281+0000 7f2fa02eb640 1 -- 172.21.15.23:0/1532458889 shutdown_connections 2024-03-24T10:45:39.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.281+0000 7f2fa02eb640 1 -- 172.21.15.23:0/1532458889 wait complete. 2024-03-24T10:45:39.292 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:45:39.292 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:45:39.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.439+0000 7f5eb21e4640 1 -- 172.21.15.23:0/3538294796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac106660 msgr2=0x7f5eac106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.439+0000 7f5eb21e4640 1 --2- 172.21.15.23:0/3538294796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac106660 0x7f5eac106a40 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0009900 tx=0x7f5ea002bc30 comp rx=0 tx=0).stop 2024-03-24T10:45:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.439+0000 7f5eb21e4640 1 -- 172.21.15.23:0/3538294796 shutdown_connections 2024-03-24T10:45:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.439+0000 7f5eb21e4640 1 --2- 172.21.15.23:0/3538294796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5eac100f60 0x7f5eac10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.439+0000 7f5eb21e4640 1 --2- 172.21.15.23:0/3538294796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eac1005a0 0x7f5eac100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.439+0000 7f5eb21e4640 1 --2- 172.21.15.23:0/3538294796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac106660 0x7f5eac106a40 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.439+0000 7f5eb21e4640 1 -- 172.21.15.23:0/3538294796 >> 172.21.15.23:0/3538294796 conn(0x7f5eac077710 msgr2=0x7f5eac077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.439+0000 7f5eb21e4640 1 -- 172.21.15.23:0/3538294796 shutdown_connections 2024-03-24T10:45:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.439+0000 7f5eb21e4640 1 -- 172.21.15.23:0/3538294796 wait complete. 2024-03-24T10:45:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.440+0000 7f5eb21e4640 1 Processor -- start 2024-03-24T10:45:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.440+0000 7f5eb21e4640 1 -- start start 2024-03-24T10:45:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.440+0000 7f5eb21e4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac1005a0 0x7f5eac19d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.440+0000 7f5eb21e4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eac100f60 0x7f5eac19de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.440+0000 7f5eb21e4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5eac106660 0x7f5eac197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eb21e4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f5eac1980f0 con 0x7f5eac1005a0 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eb21e4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5eac198260 con 0x7f5eac100f60 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eb21e4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5eac198420 con 0x7f5eac106660 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eab7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac1005a0 0x7f5eac19d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eab7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac1005a0 0x7f5eac19d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47148/0 (socket says 172.21.15.23:47148) 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eab7fe640 1 -- 172.21.15.23:0/39748926 learned_addr learned my addr 172.21.15.23:0/39748926 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eabfff640 1 --2- 172.21.15.23:0/39748926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5eac106660 0x7f5eac197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eaaffd640 1 --2- 172.21.15.23:0/39748926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eac100f60 0x7f5eac19de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eab7fe640 1 -- 172.21.15.23:0/39748926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5eac106660 msgr2=0x7f5eac197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eab7fe640 1 --2- 172.21.15.23:0/39748926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5eac106660 0x7f5eac197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eab7fe640 1 -- 172.21.15.23:0/39748926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eac100f60 msgr2=0x7f5eac19de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eab7fe640 1 --2- 172.21.15.23:0/39748926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eac100f60 0x7f5eac19de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eab7fe640 1 -- 172.21.15.23:0/39748926 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ea0009560 con 0x7f5eac1005a0 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eab7fe640 1 --2- 172.21.15.23:0/39748926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac1005a0 0x7f5eac19d8f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0005940 tx=0x7f5ea002c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5ea8ff9640 1 -- 172.21.15.23:0/39748926 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea00043d0 con 0x7f5eac1005a0 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eb21e4640 1 -- 172.21.15.23:0/39748926 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eac1986a0 con 0x7f5eac1005a0 2024-03-24T10:45:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5ea8ff9640 1 -- 172.21.15.23:0/39748926 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ea00052d0 con 0x7f5eac1005a0 2024-03-24T10:45:39.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5ea8ff9640 1 -- 172.21.15.23:0/39748926 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0003cc0 con 0x7f5eac1005a0 2024-03-24T10:45:39.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.441+0000 7f5eb21e4640 1 -- 172.21.15.23:0/39748926 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eac1a4990 con 0x7f5eac1005a0 2024-03-24T10:45:39.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.443+0000 7f5ea8ff9640 1 -- 172.21.15.23:0/39748926 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f5ea0005440 con 0x7f5eac1005a0 2024-03-24T10:45:39.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eb4dbd740 1 Processor -- start 2024-03-24T10:45:39.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5ea8ff9640 1 --2- 172.21.15.23:0/39748926 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f5e940a2230 0x7f5e940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eb4dbd740 1 -- start start 2024-03-24T10:45:39.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eabfff640 1 --2- 172.21.15.23:0/39748926 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f5e940a2230 0x7f5e940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5ea8ff9640 1 -- 172.21.15.23:0/39748926 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f5ea00f0110 con 0x7f5eac1005a0 2024-03-24T10:45:39.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eb4dbd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d581725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eb4dbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d58172af0 0x557d5817b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eb4dbd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5817b6d0 0x557d5817dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eb4dbd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557d58096f30 con 0x557d5817b6d0 2024-03-24T10:45:39.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eb4dbd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557d580e0810 con 0x557d58172af0 2024-03-24T10:45:39.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eb4dbd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557d580ddd40 con 0x557d581721d0 2024-03-24T10:45:39.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eaaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d58172af0 0x557d5817b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eab7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d581725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eaaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d58172af0 0x557d5817b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:52738/0 (socket says 172.21.15.23:52738) 2024-03-24T10:45:39.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eab7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d581725b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:50510/0 (socket says 172.21.15.23:50510) 2024-03-24T10:45:39.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eaaffd640 1 -- 172.21.15.23:0/421787816 learned_addr learned my addr 172.21.15.23:0/421787816 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:39.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.444+0000 7f5eabfff640 1 --2- 172.21.15.23:0/421787816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5817b6d0 0x557d5817dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.445+0000 7f5eabfff640 1 --2- 172.21.15.23:0/39748926 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f5e940a2230 0x7f5e940a46f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4005fd0 tx=0x7f5ea4011810 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:39.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.445+0000 7f5eaaffd640 1 -- 172.21.15.23:0/421787816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 msgr2=0x557d581725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.445+0000 7f5eaaffd640 1 --2- 172.21.15.23:0/421787816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d581725b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.445+0000 7f5eaaffd640 1 -- 172.21.15.23:0/421787816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5817b6d0 msgr2=0x557d5817dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.445+0000 7f5eaaffd640 1 --2- 172.21.15.23:0/421787816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5817b6d0 0x557d5817dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.445+0000 7f5eaaffd640 1 -- 172.21.15.23:0/421787816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d5809aea0 con 0x557d58172af0 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.445+0000 7f5eab7fe640 1 --2- 172.21.15.23:0/421787816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d581725b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.445+0000 7f5eabfff640 1 --2- 172.21.15.23:0/421787816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5817b6d0 0x557d5817dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.445+0000 7f5eaaffd640 1 --2- 172.21.15.23:0/421787816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d58172af0 0x557d5817b190 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c00bcd0 tx=0x7f5e9c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=553c630e8a266bb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.445+0000 7f5e91ffb640 1 -- 172.21.15.23:0/421787816 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c01c680 con 0x557d58172af0 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.445+0000 7f5e91ffb640 1 -- 172.21.15.23:0/421787816 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e9c015070 con 0x557d58172af0 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.445+0000 7f5e91ffb640 1 -- 172.21.15.23:0/421787816 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c011ac0 con 0x557d58172af0 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.446+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/421787816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d58172af0 msgr2=0x557d5817b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.446+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/421787816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d58172af0 0x557d5817b190 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c00bcd0 tx=0x7f5e9c0107a0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.446+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/421787816 shutdown_connections 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.446+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/421787816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5817b6d0 0x557d5817dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.446+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/421787816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d58172af0 0x557d5817b190 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.446+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/421787816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d581725b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.446+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/421787816 >> 172.21.15.23:0/421787816 conn(0x557d5816c900 msgr2=0x557d5816cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.446+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/421787816 shutdown_connections 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.446+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/421787816 wait complete. 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.446+0000 7f5eb4dbd740 1 -- start start 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eb4dbd740 1 Processor -- start 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eb4dbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d581721d0 0x557d58185620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eb4dbd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d58172af0 0x557d58185b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eb4dbd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d5817b6d0 0x557d5817fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eb4dbd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557d580d9520 con 0x557d58172af0 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eb4dbd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557d57dac430 con 0x557d581721d0 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eb4dbd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557d57d68e00 con 0x557d5817b6d0 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eb4dbd740 10 client.0 _ll_register_callbacks cb 0x557d58077d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eaaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d58172af0 0x557d58185b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eaaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d58172af0 0x557d58185b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47154/0 (socket says 172.21.15.23:47154) 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eaaffd640 1 -- 172.21.15.23:0/156739386 learned_addr learned my addr 172.21.15.23:0/156739386 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eab7fe640 1 --2- 172.21.15.23:0/156739386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d581721d0 0x557d58185620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eabfff640 1 --2- 172.21.15.23:0/156739386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d5817b6d0 0x557d5817fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eaaffd640 1 -- 172.21.15.23:0/156739386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d5817b6d0 msgr2=0x557d5817fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eaaffd640 1 --2- 172.21.15.23:0/156739386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d5817b6d0 0x557d5817fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eaaffd640 1 -- 172.21.15.23:0/156739386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d581721d0 msgr2=0x557d58185620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eaaffd640 1 --2- 172.21.15.23:0/156739386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d581721d0 0x557d58185620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eaaffd640 1 -- 172.21.15.23:0/156739386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e9c00b900 con 0x557d58172af0 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5eaaffd640 1 --2- 172.21.15.23:0/156739386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d58172af0 0x557d58185b60 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c011010 tx=0x7f5e9c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1405761e95061681 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5e8affd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.447+0000 7f5e8affd640 1 -- 172.21.15.23:0/156739386 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c021070 con 0x557d58172af0 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.448+0000 7f5e8affd640 1 -- 172.21.15.23:0/156739386 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f5e9c017070 con 0x557d58172af0 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.448+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/156739386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557d57dd87e0 con 0x557d58172af0 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.448+0000 7f5e8affd640 1 -- 172.21.15.23:0/156739386 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5e9c0057d0 con 0x557d58172af0 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.448+0000 7f5eb4dbd740 10 client.17196 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.448+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/156739386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557d580d7ba0 con 0x557d58172af0 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.448+0000 7f5e8affd640 1 -- 172.21.15.23:0/156739386 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557d57dd87e0 con 0x557d58172af0 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.448+0000 7f5e8affd640 1 -- 172.21.15.23:0/156739386 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f5e9c026500 con 0x557d58172af0 2024-03-24T10:45:39.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.448+0000 7f5eb4dbd740 10 client.17196 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:39.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.448+0000 7f5eb4dbd740 10 client.17196 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:39.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.448+0000 7f5eb4dbd740 20 client.17196 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:39.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.448+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/156739386 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557d5818ff20 0x557d58192340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.448+0000 7f5eb4dbd740 4 client.17196 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:39.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.448+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/156739386 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d57e15720 con 0x557d5818ff20 2024-03-24T10:45:39.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.448+0000 7f5eab7fe640 1 --2- 172.21.15.23:0/156739386 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557d5818ff20 0x557d58192340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.449+0000 7f5e8affd640 10 client.17196 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:39.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.449+0000 7f5eab7fe640 1 --2- 172.21.15.23:0/156739386 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557d5818ff20 0x557d58192340 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efc70b82aa70e34f server_cookie=5de207c5cd51f83 in_seq=0 out_seq=0 2024-03-24T10:45:39.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.449+0000 7f5e8affd640 1 -- 172.21.15.23:0/156739386 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557d57e15720 con 0x557d5818ff20 2024-03-24T10:45:39.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.449+0000 7f5e8affd640 10 client.17196 handle_command_reply: tid=0 2024-03-24T10:45:39.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.449+0000 7f5eb4dbd740 1 client.17196 shutdown 2024-03-24T10:45:39.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.450+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/156739386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d58172af0 msgr2=0x557d58185b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.450+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/156739386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d58172af0 0x557d58185b60 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c011010 tx=0x7f5e9c011a30 comp rx=0 tx=0).stop 2024-03-24T10:45:39.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.450+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/156739386 shutdown_connections 2024-03-24T10:45:39.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.450+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/156739386 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557d5818ff20 0x557d58192340 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.450+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/156739386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d5817b6d0 0x557d5817fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.450+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/156739386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d58172af0 0x557d58185b60 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.450+0000 7f5e8affd640 0 client.17196 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:39.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.450+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/156739386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d581721d0 0x557d58185620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.450+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/156739386 >> 172.21.15.23:0/156739386 conn(0x557d5816c900 msgr2=0x557d5816bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:39.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.450+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/156739386 shutdown_connections 2024-03-24T10:45:39.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.450+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/156739386 wait complete. 2024-03-24T10:45:39.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.450+0000 7f5eb4dbd740 20 client.17196 trim_cache size 0 max 16384 2024-03-24T10:45:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.470+0000 7f5eb4dbd740 1 Processor -- start 2024-03-24T10:45:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.470+0000 7f5eb4dbd740 1 -- start start 2024-03-24T10:45:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.470+0000 7f5eb4dbd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d5817f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.470+0000 7f5eb4dbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d58172af0 0x557d58185880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.470+0000 7f5eb4dbd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5817b6d0 0x557d58197c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.470+0000 7f5eb4dbd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557d5805e430 con 0x557d5817b6d0 2024-03-24T10:45:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.470+0000 7f5eb4dbd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557d5817e080 con 0x557d58172af0 2024-03-24T10:45:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.470+0000 7f5eb4dbd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557d58171e30 con 0x557d581721d0 2024-03-24T10:45:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.470+0000 7f5eabfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5817b6d0 0x557d58197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.470+0000 7f5eabfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5817b6d0 0x557d58197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47160/0 (socket says 172.21.15.23:47160) 2024-03-24T10:45:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.470+0000 7f5eabfff640 1 -- 172.21.15.23:0/1461807520 learned_addr learned my addr 172.21.15.23:0/1461807520 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.470+0000 7f5eab7fe640 1 --2- 172.21.15.23:0/1461807520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d5817f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.470+0000 7f5eaaffd640 1 --2- 172.21.15.23:0/1461807520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d58172af0 0x557d58185880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.471+0000 7f5eabfff640 1 -- 172.21.15.23:0/1461807520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 msgr2=0x557d5817f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.471+0000 7f5eabfff640 1 --2- 172.21.15.23:0/1461807520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d5817f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.471+0000 7f5eabfff640 1 -- 172.21.15.23:0/1461807520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d58172af0 msgr2=0x557d58185880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.471+0000 7f5eabfff640 1 --2- 172.21.15.23:0/1461807520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d58172af0 0x557d58185880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.471+0000 7f5eabfff640 1 -- 172.21.15.23:0/1461807520 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d58073b20 con 0x557d5817b6d0 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.471+0000 7f5eab7fe640 1 --2- 172.21.15.23:0/1461807520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d5817f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.471+0000 7f5eabfff640 1 --2- 172.21.15.23:0/1461807520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5817b6d0 0x557d58197c50 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4003e00 tx=0x7f5ea401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b8fac1dbcc2526 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.471+0000 7f5e917fa640 1 -- 172.21.15.23:0/1461807520 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4023040 con 0x557d5817b6d0 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.471+0000 7f5e917fa640 1 -- 172.21.15.23:0/1461807520 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ea401eb20 con 0x557d5817b6d0 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.471+0000 7f5e917fa640 1 -- 172.21.15.23:0/1461807520 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4027cb0 con 0x557d5817b6d0 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/1461807520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5817b6d0 msgr2=0x557d58197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/1461807520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5817b6d0 0x557d58197c50 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4003e00 tx=0x7f5ea401c000 comp rx=0 tx=0).stop 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/1461807520 shutdown_connections 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/1461807520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d5817b6d0 0x557d58197c50 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/1461807520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d58172af0 0x557d58185880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/1461807520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d5817f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/1461807520 >> 172.21.15.23:0/1461807520 conn(0x557d5816c900 msgr2=0x557d58185170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:39.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/1461807520 shutdown_connections 2024-03-24T10:45:39.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/1461807520 wait complete. 2024-03-24T10:45:39.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 -- start start 2024-03-24T10:45:39.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 Processor -- start 2024-03-24T10:45:39.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d58189ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d58172af0 0x557d5818a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5817b6d0 0x557d581f5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557d5818a900 con 0x557d58172af0 2024-03-24T10:45:39.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557d581f57d0 con 0x557d5817b6d0 2024-03-24T10:45:39.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.472+0000 7f5eb4dbd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557d581f5940 con 0x557d581721d0 2024-03-24T10:45:39.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eb4dbd740 10 client.0 _ll_register_callbacks cb 0x7f5ea4009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:39.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eaaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d58172af0 0x557d5818a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eaaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d58172af0 0x557d5818a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47166/0 (socket says 172.21.15.23:47166) 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eaaffd640 1 -- 172.21.15.23:0/770293518 learned_addr learned my addr 172.21.15.23:0/770293518 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eab7fe640 1 --2- 172.21.15.23:0/770293518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d58189ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eabfff640 1 --2- 172.21.15.23:0/770293518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5817b6d0 0x557d581f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eaaffd640 1 -- 172.21.15.23:0/770293518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 msgr2=0x557d58189ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eaaffd640 1 --2- 172.21.15.23:0/770293518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d58189ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eaaffd640 1 -- 172.21.15.23:0/770293518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5817b6d0 msgr2=0x557d581f5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eaaffd640 1 --2- 172.21.15.23:0/770293518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5817b6d0 0x557d581f5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eaaffd640 1 -- 172.21.15.23:0/770293518 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eac1986a0 con 0x557d58172af0 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eab7fe640 1 --2- 172.21.15.23:0/770293518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d58189ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eaaffd640 1 --2- 172.21.15.23:0/770293518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d58172af0 0x557d5818a1e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c006010 tx=0x7f5e9c026ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e530db9d40848f9c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5e8affd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5e8affd640 1 -- 172.21.15.23:0/770293518 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c021040 con 0x557d58172af0 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5e8affd640 1 -- 172.21.15.23:0/770293518 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f5e9c017040 con 0x557d58172af0 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/770293518 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e9c015070 con 0x557d58172af0 2024-03-24T10:45:39.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5e8affd640 1 -- 172.21.15.23:0/770293518 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5e9c011440 con 0x557d58172af0 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eb4dbd740 10 client.17202 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.473+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/770293518 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x7f5ea4003bc0 con 0x557d58172af0 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.474+0000 7f5e8affd640 1 -- 172.21.15.23:0/770293518 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e9c015070 con 0x557d58172af0 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.474+0000 7f5e8affd640 1 -- 172.21.15.23:0/770293518 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f5e9c005450 con 0x557d58172af0 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.474+0000 7f5eb4dbd740 10 client.17202 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.474+0000 7f5eb4dbd740 10 client.17202 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.474+0000 7f5eb4dbd740 20 client.17202 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.474+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/770293518 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557d581f6ee0 0x557d581f9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.474+0000 7f5eb4dbd740 4 client.17202 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.474+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/770293518 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557d57e64890 con 0x557d581f6ee0 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.474+0000 7f5eab7fe640 1 --2- 172.21.15.23:0/770293518 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557d581f6ee0 0x557d581f9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.474+0000 7f5eab7fe640 1 --2- 172.21.15.23:0/770293518 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557d581f6ee0 0x557d581f9300 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=112f44c0a7d8b46c server_cookie=248bb3638d2e6e0c in_seq=0 out_seq=0 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.474+0000 7f5e8affd640 10 client.17202 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5e8affd640 1 -- 172.21.15.23:0/770293518 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557d57e64890 con 0x557d581f6ee0 2024-03-24T10:45:39.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5e8affd640 10 client.17202 handle_command_reply: tid=0 2024-03-24T10:45:39.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb4dbd740 1 client.17202 shutdown 2024-03-24T10:45:39.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/770293518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d58172af0 msgr2=0x557d5818a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/770293518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d58172af0 0x557d5818a1e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c006010 tx=0x7f5e9c026ce0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/770293518 shutdown_connections 2024-03-24T10:45:39.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/770293518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557d5817b6d0 0x557d581f5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/770293518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557d58172af0 0x557d5818a1e0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/770293518 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557d581f6ee0 0x557d581f9300 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb4dbd740 1 --2- 172.21.15.23:0/770293518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d581721d0 0x557d58189ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5e8affd640 0 client.17202 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:39.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/770293518 >> 172.21.15.23:0/770293518 conn(0x557d5816c900 msgr2=0x557d5816b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:39.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/770293518 shutdown_connections 2024-03-24T10:45:39.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb4dbd740 1 -- 172.21.15.23:0/770293518 wait complete. 2024-03-24T10:45:39.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb4dbd740 20 client.17202 trim_cache size 0 max 16384 2024-03-24T10:45:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:45:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:45:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:45:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "cbb0919c-826e-4039-81c0-4f7482c65b30": { 2024-03-24T10:45:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:45:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "cbb0919c-826e-4039-81c0-4f7482c65b30", 2024-03-24T10:45:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:45:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:45:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:45:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:45:39.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb21e4640 1 -- 172.21.15.23:0/39748926 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f5e940a2230 msgr2=0x7f5e940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb21e4640 1 --2- 172.21.15.23:0/39748926 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f5e940a2230 0x7f5e940a46f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4005fd0 tx=0x7f5ea4011810 comp rx=0 tx=0).stop 2024-03-24T10:45:39.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb21e4640 1 -- 172.21.15.23:0/39748926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac1005a0 msgr2=0x7f5eac19d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:39.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb21e4640 1 --2- 172.21.15.23:0/39748926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac1005a0 0x7f5eac19d8f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0005940 tx=0x7f5ea002c700 comp rx=0 tx=0).stop 2024-03-24T10:45:39.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb21e4640 1 -- 172.21.15.23:0/39748926 shutdown_connections 2024-03-24T10:45:39.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb21e4640 1 --2- 172.21.15.23:0/39748926 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f5e940a2230 0x7f5e940a46f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb21e4640 1 --2- 172.21.15.23:0/39748926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5eac106660 0x7f5eac197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb21e4640 1 --2- 172.21.15.23:0/39748926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eac100f60 0x7f5eac19de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb21e4640 1 --2- 172.21.15.23:0/39748926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac1005a0 0x7f5eac19d8f0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:39.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.475+0000 7f5eb21e4640 1 -- 172.21.15.23:0/39748926 >> 172.21.15.23:0/39748926 conn(0x7f5eac077710 msgr2=0x7f5eac0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:39.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.476+0000 7f5eb21e4640 1 -- 172.21.15.23:0/39748926 shutdown_connections 2024-03-24T10:45:39.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:45:39.476+0000 7f5eb21e4640 1 -- 172.21.15.23:0/39748926 wait complete. 2024-03-24T10:45:39.487 INFO:tasks.cephfs.filesystem:scrub status for tag:cbb0919c-826e-4039-81c0-4f7482c65b30 - {'path': '/', 'tag': 'cbb0919c-826e-4039-81c0-4f7482c65b30', 'options': 'recursive,force'} 2024-03-24T10:45:40.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 523648 11.66 MB/sec execute 117 sec latency 43.462 ms 2024-03-24T10:45:41.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:40 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:45:41.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:40 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:41.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:40 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:45:41.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:40 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:45:41.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:40 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:41.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:40 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:41.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:40 smithi117 ceph-mon[36368]: pgmap v171: 97 pgs: 97 active+clean; 299 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-03-24T10:45:41.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:40 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:45:41.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:40 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:41.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:40 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:45:41.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:40 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:45:41.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:40 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:41.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:40 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:41.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:40 smithi116 ceph-mon[36248]: pgmap v171: 97 pgs: 97 active+clean; 299 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-03-24T10:45:41.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 525997 11.68 MB/sec execute 118 sec latency 18.870 ms 2024-03-24T10:45:41.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:40 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:45:41.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:40 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:41.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:40 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:45:41.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:40 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:45:41.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:40 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:41.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:40 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:45:41.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:40 smithi023 ceph-mon[32466]: pgmap v171: 97 pgs: 97 active+clean; 299 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-03-24T10:45:42.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 527345 11.63 MB/sec execute 119 sec latency 18.327 ms 2024-03-24T10:45:43.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:42 smithi117 ceph-mon[36368]: pgmap v172: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.9 MiB/s wr, 266 op/s 2024-03-24T10:45:43.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:42 smithi116 ceph-mon[36248]: pgmap v172: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.9 MiB/s wr, 266 op/s 2024-03-24T10:45:43.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 529680 11.65 MB/sec execute 120 sec latency 17.703 ms 2024-03-24T10:45:43.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:42 smithi023 ceph-mon[32466]: pgmap v172: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.9 MiB/s wr, 266 op/s 2024-03-24T10:45:44.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 530975 11.60 MB/sec execute 121 sec latency 129.133 ms 2024-03-24T10:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.277+0000 7fb918f1e640 1 -- 172.21.15.23:0/3096744557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb914074400 msgr2=0x7fb91410da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.277+0000 7fb918f1e640 1 --2- 172.21.15.23:0/3096744557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb914074400 0x7fb91410da20 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009900 tx=0x7fb8fc02bcd0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.277+0000 7fb918f1e640 1 -- 172.21.15.23:0/3096744557 shutdown_connections 2024-03-24T10:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.277+0000 7fb918f1e640 1 --2- 172.21.15.23:0/3096744557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb91410df60 0x7fb914110370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.277+0000 7fb918f1e640 1 --2- 172.21.15.23:0/3096744557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb914074400 0x7fb91410da20 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.277+0000 7fb918f1e640 1 --2- 172.21.15.23:0/3096744557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb914073ae0 0x7fb914073ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.277+0000 7fb918f1e640 1 -- 172.21.15.23:0/3096744557 >> 172.21.15.23:0/3096744557 conn(0x7fb9140fd1d0 msgr2=0x7fb9140ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.277+0000 7fb918f1e640 1 -- 172.21.15.23:0/3096744557 shutdown_connections 2024-03-24T10:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.277+0000 7fb918f1e640 1 -- 172.21.15.23:0/3096744557 wait complete. 2024-03-24T10:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb918f1e640 1 Processor -- start 2024-03-24T10:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb918f1e640 1 -- start start 2024-03-24T10:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb918f1e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb914073ae0 0x7fb914199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb918f1e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb914074400 0x7fb91419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb918f1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb91410df60 0x7fb91419e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb918f1e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb91419edb0 con 0x7fb914074400 2024-03-24T10:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb918f1e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb91419ef20 con 0x7fb91410df60 2024-03-24T10:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb918f1e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb91419f0e0 con 0x7fb914073ae0 2024-03-24T10:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb91259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb914074400 0x7fb91419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb91259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb914074400 0x7fb91419a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47196/0 (socket says 172.21.15.23:47196) 2024-03-24T10:45:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb91259c640 1 -- 172.21.15.23:0/888243497 learned_addr learned my addr 172.21.15.23:0/888243497 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb90bfff640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb914073ae0 0x7fb914199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb912d9d640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb91410df60 0x7fb91419e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb91259c640 1 -- 172.21.15.23:0/888243497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb914073ae0 msgr2=0x7fb914199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb91259c640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb914073ae0 0x7fb914199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb91259c640 1 -- 172.21.15.23:0/888243497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb91410df60 msgr2=0x7fb91419e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb91259c640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb91410df60 0x7fb91419e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb91259c640 1 -- 172.21.15.23:0/888243497 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8fc009560 con 0x7fb914074400 2024-03-24T10:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb90bfff640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb914073ae0 0x7fb914199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.278+0000 7fb91259c640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb914074400 0x7fb91419a480 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc005940 tx=0x7fb8fc02c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.279+0000 7fb90b7fe640 1 -- 172.21.15.23:0/888243497 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc004350 con 0x7fb914074400 2024-03-24T10:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.279+0000 7fb90b7fe640 1 -- 172.21.15.23:0/888243497 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8fc0044b0 con 0x7fb914074400 2024-03-24T10:45:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.279+0000 7fb918f1e640 1 -- 172.21.15.23:0/888243497 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb91419f360 con 0x7fb914074400 2024-03-24T10:45:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.279+0000 7fb90b7fe640 1 -- 172.21.15.23:0/888243497 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc003d30 con 0x7fb914074400 2024-03-24T10:45:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.279+0000 7fb918f1e640 1 -- 172.21.15.23:0/888243497 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9141a6c40 con 0x7fb914074400 2024-03-24T10:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.281+0000 7fb90b7fe640 1 -- 172.21.15.23:0/888243497 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fb8fc004df0 con 0x7fb914074400 2024-03-24T10:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.281+0000 7fb91bbd9740 1 Processor -- start 2024-03-24T10:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.281+0000 7fb91bbd9740 1 -- start start 2024-03-24T10:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb91bbd9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b15e0 0x55610f7b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb91bbd9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7b1f00 0x55610f7ba790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb91bbd9740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7bacd0 0x55610f7bd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb91bbd9740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55610f76c1b0 con 0x55610f7bacd0 2024-03-24T10:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb91bbd9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55610f71ef80 con 0x55610f7b15e0 2024-03-24T10:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb91bbd9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55610f71c4b0 con 0x55610f7b1f00 2024-03-24T10:45:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb90b7fe640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb9000a22d0 0x7fb9000a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb91259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7bacd0 0x55610f7bd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb90bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7b1f00 0x55610f7ba790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb912d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b15e0 0x55610f7b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb90bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7b1f00 0x55610f7ba790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:50544/0 (socket says 172.21.15.23:50544) 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb90bfff640 1 -- 172.21.15.23:0/2843303390 learned_addr learned my addr 172.21.15.23:0/2843303390 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb912d9d640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb9000a22d0 0x7fb9000a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb90b7fe640 1 -- 172.21.15.23:0/888243497 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fb8fc0f0620 con 0x7fb914074400 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb90bfff640 1 -- 172.21.15.23:0/2843303390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b15e0 msgr2=0x55610f7b19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb90bfff640 1 --2- 172.21.15.23:0/2843303390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b15e0 0x55610f7b19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb90bfff640 1 -- 172.21.15.23:0/2843303390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7bacd0 msgr2=0x55610f7bd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb90bfff640 1 --2- 172.21.15.23:0/2843303390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7bacd0 0x55610f7bd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb90bfff640 1 -- 172.21.15.23:0/2843303390 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55610f6ad810 con 0x55610f7b1f00 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb912d9d640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb9000a22d0 0x7fb9000a4790 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb91406adc0 tx=0x7fb90c009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.282+0000 7fb90bfff640 1 --2- 172.21.15.23:0/2843303390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7b1f00 0x55610f7ba790 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb9040115f0 tx=0x7fb9040119f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7727c19949914557 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.283+0000 7fb908ff9640 1 -- 172.21.15.23:0/2843303390 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90400f560 con 0x55610f7b1f00 2024-03-24T10:45:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.283+0000 7fb908ff9640 1 -- 172.21.15.23:0/2843303390 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb90400fba0 con 0x55610f7b1f00 2024-03-24T10:45:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.283+0000 7fb908ff9640 1 -- 172.21.15.23:0/2843303390 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904019c20 con 0x55610f7b1f00 2024-03-24T10:45:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.283+0000 7fb91bbd9740 1 -- 172.21.15.23:0/2843303390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7b1f00 msgr2=0x55610f7ba790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.283+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/2843303390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7b1f00 0x55610f7ba790 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb9040115f0 tx=0x7fb9040119f0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.283+0000 7fb91bbd9740 1 -- 172.21.15.23:0/2843303390 shutdown_connections 2024-03-24T10:45:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.283+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/2843303390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7bacd0 0x55610f7bd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.283+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/2843303390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7b1f00 0x55610f7ba790 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.283+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/2843303390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b15e0 0x55610f7b19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.283+0000 7fb91bbd9740 1 -- 172.21.15.23:0/2843303390 >> 172.21.15.23:0/2843303390 conn(0x55610f7abf00 msgr2=0x55610f7ac2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.283+0000 7fb91bbd9740 1 -- 172.21.15.23:0/2843303390 shutdown_connections 2024-03-24T10:45:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.283+0000 7fb91bbd9740 1 -- 172.21.15.23:0/2843303390 wait complete. 2024-03-24T10:45:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb91bbd9740 1 -- start start 2024-03-24T10:45:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb91bbd9740 1 Processor -- start 2024-03-24T10:45:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb91bbd9740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b15e0 0x55610f7b7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb91bbd9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b1f00 0x55610f7b8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb91bbd9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7bacd0 0x55610f7b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb91bbd9740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55610f717c90 con 0x55610f7b15e0 2024-03-24T10:45:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb91bbd9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55610f6bcc50 con 0x55610f7b1f00 2024-03-24T10:45:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb91bbd9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55610f3eb250 con 0x55610f7bacd0 2024-03-24T10:45:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb91bbd9740 10 client.0 _ll_register_callbacks cb 0x55610f70b290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb90bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b15e0 0x55610f7b7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb90bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b15e0 0x55610f7b7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47214/0 (socket says 172.21.15.23:47214) 2024-03-24T10:45:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb90bfff640 1 -- 172.21.15.23:0/3934972273 learned_addr learned my addr 172.21.15.23:0/3934972273 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb91259c640 1 --2- 172.21.15.23:0/3934972273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b1f00 0x55610f7b8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb912d9d640 1 --2- 172.21.15.23:0/3934972273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7bacd0 0x55610f7b87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb90bfff640 1 -- 172.21.15.23:0/3934972273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7bacd0 msgr2=0x55610f7b87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb90bfff640 1 --2- 172.21.15.23:0/3934972273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7bacd0 0x55610f7b87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb90bfff640 1 -- 172.21.15.23:0/3934972273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b1f00 msgr2=0x55610f7b8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb90bfff640 1 --2- 172.21.15.23:0/3934972273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b1f00 0x55610f7b8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.284+0000 7fb90bfff640 1 -- 172.21.15.23:0/3934972273 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb904009d40 con 0x55610f7b15e0 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb90bfff640 1 --2- 172.21.15.23:0/3934972273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b15e0 0x55610f7b7d40 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fb90401c040 tx=0x7fb9040196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ad2dba0279237e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb8f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb8f9ffb640 1 -- 172.21.15.23:0/3934972273 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904004350 con 0x55610f7b15e0 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb8f9ffb640 1 -- 172.21.15.23:0/3934972273 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fb9040044b0 con 0x55610f7b15e0 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb91bbd9740 1 -- 172.21.15.23:0/3934972273 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55610f73c790 con 0x55610f7b15e0 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb8f9ffb640 1 -- 172.21.15.23:0/3934972273 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb90400f560 con 0x55610f7b15e0 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb91bbd9740 10 client.17214 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb91bbd9740 1 -- 172.21.15.23:0/3934972273 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55610f716310 con 0x55610f7b15e0 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb8f9ffb640 1 -- 172.21.15.23:0/3934972273 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55610f73c790 con 0x55610f7b15e0 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb8f9ffb640 1 -- 172.21.15.23:0/3934972273 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fb904003860 con 0x55610f7b15e0 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb91bbd9740 10 client.17214 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb91bbd9740 10 client.17214 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb91bbd9740 20 client.17214 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/3934972273 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55610f7cab90 0x55610f7ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb91bbd9740 4 client.17214 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.285+0000 7fb91bbd9740 1 -- 172.21.15.23:0/3934972273 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55610f417600 con 0x55610f7cab90 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.286+0000 7fb91259c640 1 --2- 172.21.15.23:0/3934972273 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55610f7cab90 0x55610f7ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.286+0000 7fb91259c640 1 --2- 172.21.15.23:0/3934972273 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55610f7cab90 0x55610f7ccfc0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4e42311a7e77202 server_cookie=5f674dcd70ae0e5c in_seq=0 out_seq=0 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.286+0000 7fb8f9ffb640 10 client.17214 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb8f9ffb640 1 -- 172.21.15.23:0/3934972273 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55610f417600 con 0x55610f7cab90 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb8f9ffb640 10 client.17214 handle_command_reply: tid=0 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb91bbd9740 1 client.17214 shutdown 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb91bbd9740 1 -- 172.21.15.23:0/3934972273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b15e0 msgr2=0x55610f7b7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/3934972273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b15e0 0x55610f7b7d40 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fb90401c040 tx=0x7fb9040196c0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb91bbd9740 1 -- 172.21.15.23:0/3934972273 shutdown_connections 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/3934972273 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55610f7cab90 0x55610f7ccfc0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/3934972273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7bacd0 0x55610f7b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/3934972273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b1f00 0x55610f7b8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb8f9ffb640 0 client.17214 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/3934972273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b15e0 0x55610f7b7d40 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb91bbd9740 1 -- 172.21.15.23:0/3934972273 >> 172.21.15.23:0/3934972273 conn(0x55610f7abf00 msgr2=0x55610f7adc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb91bbd9740 1 -- 172.21.15.23:0/3934972273 shutdown_connections 2024-03-24T10:45:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb91bbd9740 1 -- 172.21.15.23:0/3934972273 wait complete. 2024-03-24T10:45:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.287+0000 7fb91bbd9740 20 client.17214 trim_cache size 0 max 16384 2024-03-24T10:45:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.302+0000 7fb91bbd9740 1 Processor -- start 2024-03-24T10:45:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb91bbd9740 1 -- start start 2024-03-24T10:45:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb91bbd9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7b15e0 0x55610f7ba470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb91bbd9740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b1f00 0x55610f7c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb91bbd9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7bacd0 0x55610f7d28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb91bbd9740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55610f69d660 con 0x55610f7b1f00 2024-03-24T10:45:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb91bbd9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55610f7aa3f0 con 0x55610f7bacd0 2024-03-24T10:45:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb91bbd9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55610f7b1030 con 0x55610f7b15e0 2024-03-24T10:45:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb91259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b1f00 0x55610f7c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb90bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7b15e0 0x55610f7ba470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb91259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b1f00 0x55610f7c63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47226/0 (socket says 172.21.15.23:47226) 2024-03-24T10:45:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb91259c640 1 -- 172.21.15.23:0/217805054 learned_addr learned my addr 172.21.15.23:0/217805054 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb912d9d640 1 --2- 172.21.15.23:0/217805054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7bacd0 0x55610f7d28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb90bfff640 1 -- 172.21.15.23:0/217805054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7bacd0 msgr2=0x55610f7d28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb90bfff640 1 --2- 172.21.15.23:0/217805054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7bacd0 0x55610f7d28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb90bfff640 1 -- 172.21.15.23:0/217805054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b1f00 msgr2=0x55610f7c63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb90bfff640 1 --2- 172.21.15.23:0/217805054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b1f00 0x55610f7c63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb90bfff640 1 -- 172.21.15.23:0/217805054 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55610f6b2810 con 0x55610f7b15e0 2024-03-24T10:45:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb91259c640 1 --2- 172.21.15.23:0/217805054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b1f00 0x55610f7c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.303+0000 7fb90bfff640 1 --2- 172.21.15.23:0/217805054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7b15e0 0x55610f7ba470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb904003de0 tx=0x7fb904019640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f07ff4d0ff4d663 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.304+0000 7fb8fbfff640 1 -- 172.21.15.23:0/217805054 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904015030 con 0x55610f7b15e0 2024-03-24T10:45:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.304+0000 7fb8fbfff640 1 -- 172.21.15.23:0/217805054 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb904003e60 con 0x55610f7b15e0 2024-03-24T10:45:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.304+0000 7fb8fbfff640 1 -- 172.21.15.23:0/217805054 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904003990 con 0x55610f7b15e0 2024-03-24T10:45:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.304+0000 7fb91bbd9740 1 -- 172.21.15.23:0/217805054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7b15e0 msgr2=0x55610f7ba470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.304+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/217805054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7b15e0 0x55610f7ba470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb904003de0 tx=0x7fb904019640 comp rx=0 tx=0).stop 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.304+0000 7fb91bbd9740 1 -- 172.21.15.23:0/217805054 shutdown_connections 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.304+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/217805054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7bacd0 0x55610f7d28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.304+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/217805054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b1f00 0x55610f7c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.304+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/217805054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7b15e0 0x55610f7ba470 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.304+0000 7fb91bbd9740 1 -- 172.21.15.23:0/217805054 >> 172.21.15.23:0/217805054 conn(0x55610f7abf00 msgr2=0x55610f7b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.304+0000 7fb91bbd9740 1 -- 172.21.15.23:0/217805054 shutdown_connections 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.304+0000 7fb91bbd9740 1 -- 172.21.15.23:0/217805054 wait complete. 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.304+0000 7fb91bbd9740 1 -- start start 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91bbd9740 1 Processor -- start 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91bbd9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b15e0 0x55610f7c2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91bbd9740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b1f00 0x55610f7bdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91bbd9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7bacd0 0x55610f7be4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91bbd9740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55610f7bea10 con 0x55610f7b1f00 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91bbd9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55610f7beb80 con 0x55610f7b15e0 2024-03-24T10:45:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91bbd9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55610f7bed40 con 0x55610f7bacd0 2024-03-24T10:45:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91bbd9740 10 client.0 _ll_register_callbacks cb 0x7fb90401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b1f00 0x55610f7bdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b1f00 0x55610f7bdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47238/0 (socket says 172.21.15.23:47238) 2024-03-24T10:45:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb90bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b15e0 0x55610f7c2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91259c640 1 -- 172.21.15.23:0/1056918375 learned_addr learned my addr 172.21.15.23:0/1056918375 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb912d9d640 1 --2- 172.21.15.23:0/1056918375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7bacd0 0x55610f7be4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91259c640 1 -- 172.21.15.23:0/1056918375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7bacd0 msgr2=0x55610f7be4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91259c640 1 --2- 172.21.15.23:0/1056918375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7bacd0 0x55610f7be4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91259c640 1 -- 172.21.15.23:0/1056918375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b15e0 msgr2=0x55610f7c2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91259c640 1 --2- 172.21.15.23:0/1056918375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b15e0 0x55610f7c2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91259c640 1 -- 172.21.15.23:0/1056918375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9141a6c40 con 0x55610f7b1f00 2024-03-24T10:45:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb91259c640 1 --2- 172.21.15.23:0/1056918375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b1f00 0x55610f7bdf90 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc006010 tx=0x7fb8fc0902b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4129c2ad35ef90 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.305+0000 7fb8f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb8f9ffb640 1 -- 172.21.15.23:0/1056918375 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc0bb6e0 con 0x55610f7b1f00 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb91bbd9740 1 -- 172.21.15.23:0/1056918375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55610f7b7b70 con 0x55610f7b1f00 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb8f9ffb640 1 -- 172.21.15.23:0/1056918375 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fb8fc02ce40 con 0x55610f7b1f00 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb8f9ffb640 1 -- 172.21.15.23:0/1056918375 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb8fc0bd820 con 0x55610f7b1f00 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb91bbd9740 10 client.17220 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb91bbd9740 1 -- 172.21.15.23:0/1056918375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55610f716310 con 0x55610f7b1f00 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb8f9ffb640 1 -- 172.21.15.23:0/1056918375 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55610f7b7b70 con 0x55610f7b1f00 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb8f9ffb640 1 -- 172.21.15.23:0/1056918375 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fb8fc093d20 con 0x55610f7b1f00 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb91bbd9740 10 client.17220 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb91bbd9740 10 client.17220 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb91bbd9740 20 client.17220 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/1056918375 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55610f8311d0 0x55610f8335f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb91bbd9740 4 client.17220 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:45:44.311 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb91bbd9740 1 -- 172.21.15.23:0/1056918375 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fb90400fba0 con 0x55610f8311d0 2024-03-24T10:45:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.306+0000 7fb90bfff640 1 --2- 172.21.15.23:0/1056918375 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55610f8311d0 0x55610f8335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.307+0000 7fb90bfff640 1 --2- 172.21.15.23:0/1056918375 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55610f8311d0 0x55610f8335f0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98474a2e7f93a4af server_cookie=3e433fc3e9127dd2 in_seq=0 out_seq=0 2024-03-24T10:45:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.307+0000 7fb8f9ffb640 10 client.17220 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.316+0000 7fb8f9ffb640 1 -- 172.21.15.23:0/1056918375 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fb90400fba0 con 0x55610f8311d0 2024-03-24T10:45:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.316+0000 7fb8f9ffb640 10 client.17220 handle_command_reply: tid=0 2024-03-24T10:45:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.316+0000 7fb91bbd9740 1 client.17220 shutdown 2024-03-24T10:45:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb91bbd9740 1 -- 172.21.15.23:0/1056918375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b1f00 msgr2=0x55610f7bdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/1056918375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b1f00 0x55610f7bdf90 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc006010 tx=0x7fb8fc0902b0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb91bbd9740 1 -- 172.21.15.23:0/1056918375 shutdown_connections 2024-03-24T10:45:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/1056918375 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55610f8311d0 0x55610f8335f0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/1056918375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55610f7bacd0 0x55610f7be4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/1056918375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55610f7b1f00 0x55610f7bdf90 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb8f9ffb640 0 client.17220 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb91bbd9740 1 --2- 172.21.15.23:0/1056918375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55610f7b15e0 0x55610f7c2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb91bbd9740 1 -- 172.21.15.23:0/1056918375 >> 172.21.15.23:0/1056918375 conn(0x55610f7abf00 msgr2=0x55610f7b7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb91bbd9740 1 -- 172.21.15.23:0/1056918375 shutdown_connections 2024-03-24T10:45:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb91bbd9740 1 -- 172.21.15.23:0/1056918375 wait complete. 2024-03-24T10:45:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb91bbd9740 20 client.17220 trim_cache size 0 max 16384 2024-03-24T10:45:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:45:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:45:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:45:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:45:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb918f1e640 1 -- 172.21.15.23:0/888243497 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb9000a22d0 msgr2=0x7fb9000a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb918f1e640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb9000a22d0 0x7fb9000a4790 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb91406adc0 tx=0x7fb90c009500 comp rx=0 tx=0).stop 2024-03-24T10:45:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb918f1e640 1 -- 172.21.15.23:0/888243497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb914074400 msgr2=0x7fb91419a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb918f1e640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb914074400 0x7fb91419a480 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc005940 tx=0x7fb8fc02c710 comp rx=0 tx=0).stop 2024-03-24T10:45:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb918f1e640 1 -- 172.21.15.23:0/888243497 shutdown_connections 2024-03-24T10:45:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb918f1e640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb9000a22d0 0x7fb9000a4790 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb918f1e640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb91410df60 0x7fb91419e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb918f1e640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb914074400 0x7fb91419a480 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb918f1e640 1 --2- 172.21.15.23:0/888243497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb914073ae0 0x7fb914199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb918f1e640 1 -- 172.21.15.23:0/888243497 >> 172.21.15.23:0/888243497 conn(0x7fb9140fd1d0 msgr2=0x7fb9140fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb918f1e640 1 -- 172.21.15.23:0/888243497 shutdown_connections 2024-03-24T10:45:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:44.317+0000 7fb918f1e640 1 -- 172.21.15.23:0/888243497 wait complete. 2024-03-24T10:45:45.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:44 smithi117 ceph-mon[36368]: pgmap v173: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.5 MiB/s wr, 177 op/s 2024-03-24T10:45:45.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:44 smithi116 ceph-mon[36248]: pgmap v173: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.5 MiB/s wr, 177 op/s 2024-03-24T10:45:45.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 533227 11.62 MB/sec execute 122 sec latency 56.759 ms 2024-03-24T10:45:45.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:44 smithi023 ceph-mon[32466]: pgmap v173: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.5 MiB/s wr, 177 op/s 2024-03-24T10:45:46.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:45 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:45:46.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:45 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:45:46.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:45 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:45:46.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:45 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:45:46.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 534666 11.56 MB/sec execute 123 sec latency 26.788 ms 2024-03-24T10:45:46.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:45 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:45:46.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:45 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:45:47.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:46 smithi117 ceph-mon[36368]: pgmap v174: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.5 MiB/s wr, 177 op/s 2024-03-24T10:45:47.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:46 smithi116 ceph-mon[36248]: pgmap v174: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.5 MiB/s wr, 177 op/s 2024-03-24T10:45:47.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 536915 11.58 MB/sec execute 124 sec latency 18.729 ms 2024-03-24T10:45:47.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:46 smithi023 ceph-mon[32466]: pgmap v174: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.5 MiB/s wr, 177 op/s 2024-03-24T10:45:48.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 538886 11.55 MB/sec execute 125 sec latency 27.323 ms 2024-03-24T10:45:49.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:48 smithi117 ceph-mon[36368]: pgmap v175: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 269 op/s 2024-03-24T10:45:49.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:48 smithi116 ceph-mon[36248]: pgmap v175: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 269 op/s 2024-03-24T10:45:49.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 540780 11.55 MB/sec execute 126 sec latency 56.924 ms 2024-03-24T10:45:49.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:48 smithi023 ceph-mon[32466]: pgmap v175: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 269 op/s 2024-03-24T10:45:50.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 542550 11.52 MB/sec execute 127 sec latency 55.787 ms 2024-03-24T10:45:51.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:50 smithi117 ceph-mon[36368]: pgmap v176: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 181 op/s 2024-03-24T10:45:51.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:50 smithi116 ceph-mon[36248]: pgmap v176: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 181 op/s 2024-03-24T10:45:51.121 INFO:tasks.workunit.client.0.smithi023.stdout: 1 544580 11.53 MB/sec execute 128 sec latency 18.343 ms 2024-03-24T10:45:51.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:50 smithi023 ceph-mon[32466]: pgmap v176: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.5 MiB/s wr, 181 op/s 2024-03-24T10:45:52.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 547050 11.57 MB/sec execute 129 sec latency 17.825 ms 2024-03-24T10:45:53.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:52 smithi116 ceph-mon[36248]: pgmap v177: 97 pgs: 97 active+clean; 338 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.0 MiB/s wr, 277 op/s 2024-03-24T10:45:53.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 548579 11.53 MB/sec execute 130 sec latency 19.203 ms 2024-03-24T10:45:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:52 smithi023 ceph-mon[32466]: pgmap v177: 97 pgs: 97 active+clean; 338 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.0 MiB/s wr, 277 op/s 2024-03-24T10:45:53.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:52 smithi117 ceph-mon[36368]: pgmap v177: 97 pgs: 97 active+clean; 338 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.0 MiB/s wr, 277 op/s 2024-03-24T10:45:54.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 550874 11.55 MB/sec execute 131 sec latency 17.504 ms 2024-03-24T10:45:55.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:54 smithi116 ceph-mon[36248]: pgmap v178: 97 pgs: 97 active+clean; 338 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-24T10:45:55.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:54 smithi117 ceph-mon[36368]: pgmap v178: 97 pgs: 97 active+clean; 338 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-24T10:45:55.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 552172 11.50 MB/sec execute 132 sec latency 130.785 ms 2024-03-24T10:45:55.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:54 smithi023 ceph-mon[32466]: pgmap v178: 97 pgs: 97 active+clean; 338 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-24T10:45:56.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 554486 11.52 MB/sec execute 133 sec latency 16.025 ms 2024-03-24T10:45:57.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:56 smithi116 ceph-mon[36248]: pgmap v179: 97 pgs: 97 active+clean; 338 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-24T10:45:57.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 556077 11.47 MB/sec execute 134 sec latency 21.037 ms 2024-03-24T10:45:57.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:56 smithi023 ceph-mon[32466]: pgmap v179: 97 pgs: 97 active+clean; 338 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-24T10:45:57.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:56 smithi117 ceph-mon[36368]: pgmap v179: 97 pgs: 97 active+clean; 338 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-24T10:45:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.406+0000 7f10053e2640 1 -- 172.21.15.23:0/2380459553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1000101220 msgr2=0x7f100010d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.406+0000 7f10053e2640 1 --2- 172.21.15.23:0/2380459553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1000101220 0x7f100010d440 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009ca0 tx=0x7f0ff802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.406+0000 7f10053e2640 1 -- 172.21.15.23:0/2380459553 shutdown_connections 2024-03-24T10:45:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.406+0000 7f10053e2640 1 --2- 172.21.15.23:0/2380459553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1000101220 0x7f100010d440 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.406+0000 7f10053e2640 1 --2- 172.21.15.23:0/2380459553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1000100860 0x7f1000100ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.406+0000 7f10053e2640 1 --2- 172.21.15.23:0/2380459553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10001055b0 0x7f1000105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.406+0000 7f10053e2640 1 -- 172.21.15.23:0/2380459553 >> 172.21.15.23:0/2380459553 conn(0x7f1000077460 msgr2=0x7f1000077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.406+0000 7f10053e2640 1 -- 172.21.15.23:0/2380459553 shutdown_connections 2024-03-24T10:45:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.406+0000 7f10053e2640 1 -- 172.21.15.23:0/2380459553 wait complete. 2024-03-24T10:45:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f10053e2640 1 Processor -- start 2024-03-24T10:45:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f10053e2640 1 -- start start 2024-03-24T10:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f10053e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1000100860 0x7f1000198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f10053e2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1000101220 0x7f1000199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f10053e2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10001055b0 0x7f10001a3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f10053e2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f1000199890 con 0x7f10001055b0 2024-03-24T10:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f10053e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f10001a40b0 con 0x7f1000100860 2024-03-24T10:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f10053e2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f10001a41f0 con 0x7f1000101220 2024-03-24T10:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f0fff7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10001055b0 0x7f10001a3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f0fff7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10001055b0 0x7f10001a3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49014/0 (socket says 172.21.15.23:49014) 2024-03-24T10:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f0fff7fe640 1 -- 172.21.15.23:0/4210692644 learned_addr learned my addr 172.21.15.23:0/4210692644 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f0ffeffd640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1000100860 0x7f1000198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f0ff7fff640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1000101220 0x7f1000199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f0fff7fe640 1 -- 172.21.15.23:0/4210692644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1000101220 msgr2=0x7f1000199030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f0fff7fe640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1000101220 0x7f1000199030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f0fff7fe640 1 -- 172.21.15.23:0/4210692644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1000100860 msgr2=0x7f1000198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f0fff7fe640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1000100860 0x7f1000198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f0fff7fe640 1 -- 172.21.15.23:0/4210692644 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff8009900 con 0x7f10001055b0 2024-03-24T10:45:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.407+0000 7f0ffeffd640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1000100860 0x7f1000198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.408+0000 7f0fff7fe640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10001055b0 0x7f10001a3b70 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8005e30 tx=0x7f0ff8035aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.408+0000 7f0ffcff9640 1 -- 172.21.15.23:0/4210692644 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8035ea0 con 0x7f10001055b0 2024-03-24T10:45:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.408+0000 7f10053e2640 1 -- 172.21.15.23:0/4210692644 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10001a43f0 con 0x7f10001055b0 2024-03-24T10:45:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.408+0000 7f0ffcff9640 1 -- 172.21.15.23:0/4210692644 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ff8003e50 con 0x7f10001055b0 2024-03-24T10:45:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.408+0000 7f0ffcff9640 1 -- 172.21.15.23:0/4210692644 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8004830 con 0x7f10001055b0 2024-03-24T10:45:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.408+0000 7f10053e2640 1 -- 172.21.15.23:0/4210692644 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10001a48f0 con 0x7f10001055b0 2024-03-24T10:45:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.410+0000 7f0ffcff9640 1 -- 172.21.15.23:0/4210692644 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f0ff803b040 con 0x7f10001055b0 2024-03-24T10:45:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.410+0000 7f0ffcff9640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0fec0a22d0 0x7f0fec0a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.410+0000 7f0ffcff9640 1 -- 172.21.15.23:0/4210692644 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f0ff80f0840 con 0x7f10001055b0 2024-03-24T10:45:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.410+0000 7f0ffeffd640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0fec0a22d0 0x7f0fec0a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.411+0000 7f1007fc4740 1 Processor -- start 2024-03-24T10:45:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.411+0000 7f1007fc4740 1 -- start start 2024-03-24T10:45:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.411+0000 7f0ffeffd640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0fec0a22d0 0x7f0fec0a4790 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009fc0 tx=0x7f0ff0005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.411+0000 7f1007fc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 0x55967c1fd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.411+0000 7f1007fc4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fdf00 0x55967c206790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.411+0000 7f1007fc4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c206cd0 0x55967c2090c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.411+0000 7f1007fc4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55967c1b81b0 con 0x55967c1fdf00 2024-03-24T10:45:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.411+0000 7f1007fc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55967c16af80 con 0x55967c1fd5e0 2024-03-24T10:45:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.411+0000 7f1007fc4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55967c1684b0 con 0x55967c206cd0 2024-03-24T10:45:57.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.411+0000 7f0ff7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fdf00 0x55967c206790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.411+0000 7f0ff7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fdf00 0x55967c206790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49022/0 (socket says 172.21.15.23:49022) 2024-03-24T10:45:57.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.411+0000 7f0ffeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 0x55967c1fd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.411+0000 7f0ff7fff640 1 -- 172.21.15.23:0/3688462342 learned_addr learned my addr 172.21.15.23:0/3688462342 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:57.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.411+0000 7f0fff7fe640 1 --2- 172.21.15.23:0/3688462342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c206cd0 0x55967c2090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.412+0000 7f0ff7fff640 1 -- 172.21.15.23:0/3688462342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c206cd0 msgr2=0x55967c2090c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:57.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.412+0000 7f0ff7fff640 1 --2- 172.21.15.23:0/3688462342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c206cd0 0x55967c2090c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.412+0000 7f0ff7fff640 1 -- 172.21.15.23:0/3688462342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 msgr2=0x55967c1fd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.412+0000 7f0ff7fff640 1 --2- 172.21.15.23:0/3688462342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 0x55967c1fd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.412+0000 7f0ff7fff640 1 -- 172.21.15.23:0/3688462342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55967c0f9810 con 0x55967c1fdf00 2024-03-24T10:45:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.412+0000 7f0ffeffd640 1 --2- 172.21.15.23:0/3688462342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 0x55967c1fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.412+0000 7f0ff7fff640 1 --2- 172.21.15.23:0/3688462342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fdf00 0x55967c206790 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009ca0 tx=0x7f0fe8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e657d4da87531512 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.412+0000 7f0ff57fa640 1 -- 172.21.15.23:0/3688462342 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8019630 con 0x55967c1fdf00 2024-03-24T10:45:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.412+0000 7f0ff57fa640 1 -- 172.21.15.23:0/3688462342 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fe8019c70 con 0x55967c1fdf00 2024-03-24T10:45:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.412+0000 7f0ff57fa640 1 -- 172.21.15.23:0/3688462342 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe800eb60 con 0x55967c1fdf00 2024-03-24T10:45:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 -- 172.21.15.23:0/3688462342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fdf00 msgr2=0x55967c206790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 --2- 172.21.15.23:0/3688462342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fdf00 0x55967c206790 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009ca0 tx=0x7f0fe8011810 comp rx=0 tx=0).stop 2024-03-24T10:45:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 -- 172.21.15.23:0/3688462342 shutdown_connections 2024-03-24T10:45:57.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 --2- 172.21.15.23:0/3688462342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c206cd0 0x55967c2090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 --2- 172.21.15.23:0/3688462342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fdf00 0x55967c206790 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 --2- 172.21.15.23:0/3688462342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 0x55967c1fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 -- 172.21.15.23:0/3688462342 >> 172.21.15.23:0/3688462342 conn(0x55967c1f7f00 msgr2=0x55967c1f82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 -- 172.21.15.23:0/3688462342 shutdown_connections 2024-03-24T10:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 -- 172.21.15.23:0/3688462342 wait complete. 2024-03-24T10:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 -- start start 2024-03-24T10:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 Processor -- start 2024-03-24T10:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fd5e0 0x55967c203cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c1fdf00 0x55967c204230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c206cd0 0x55967c204770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55967c163c90 con 0x55967c1fd5e0 2024-03-24T10:45:57.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55967c108c50 con 0x55967c206cd0 2024-03-24T10:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55967be37250 con 0x55967c1fdf00 2024-03-24T10:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f1007fc4740 10 client.0 _ll_register_callbacks cb 0x55967c157290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.413+0000 7f0ffeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fd5e0 0x55967c203cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0ff7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c1fdf00 0x55967c204230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0ffeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fd5e0 0x55967c203cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49034/0 (socket says 172.21.15.23:49034) 2024-03-24T10:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0ffeffd640 1 -- 172.21.15.23:0/138170974 learned_addr learned my addr 172.21.15.23:0/138170974 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0fff7fe640 1 --2- 172.21.15.23:0/138170974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c206cd0 0x55967c204770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0ffeffd640 1 -- 172.21.15.23:0/138170974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c1fdf00 msgr2=0x55967c204230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0ffeffd640 1 --2- 172.21.15.23:0/138170974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c1fdf00 0x55967c204230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0ffeffd640 1 -- 172.21.15.23:0/138170974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c206cd0 msgr2=0x55967c204770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0ffeffd640 1 --2- 172.21.15.23:0/138170974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c206cd0 0x55967c204770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0ffeffd640 1 -- 172.21.15.23:0/138170974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fe80098d0 con 0x55967c1fd5e0 2024-03-24T10:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0ff7fff640 1 --2- 172.21.15.23:0/138170974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c1fdf00 0x55967c204230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0fff7fe640 1 --2- 172.21.15.23:0/138170974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c206cd0 0x55967c204770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0ffeffd640 1 --2- 172.21.15.23:0/138170974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fd5e0 0x55967c203cf0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0003730 tx=0x7f0ff0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f98e8956137e86fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0fe67fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0fe67fc640 1 -- 172.21.15.23:0/138170974 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0022070 con 0x55967c1fd5e0 2024-03-24T10:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0fe67fc640 1 -- 172.21.15.23:0/138170974 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f0ff000f8c0 con 0x55967c1fd5e0 2024-03-24T10:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f1007fc4740 1 -- 172.21.15.23:0/138170974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55967c188790 con 0x55967c1fd5e0 2024-03-24T10:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f0fe67fc640 1 -- 172.21.15.23:0/138170974 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0ff000fa80 con 0x55967c1fd5e0 2024-03-24T10:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f1007fc4740 10 client.17232 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.414+0000 7f1007fc4740 1 -- 172.21.15.23:0/138170974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55967c162310 con 0x55967c1fd5e0 2024-03-24T10:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.415+0000 7f0fe67fc640 1 -- 172.21.15.23:0/138170974 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ff0033020 con 0x55967c1fd5e0 2024-03-24T10:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.415+0000 7f0fe67fc640 1 -- 172.21.15.23:0/138170974 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f0ff0026410 con 0x55967c1fd5e0 2024-03-24T10:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.415+0000 7f1007fc4740 10 client.17232 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.415+0000 7f1007fc4740 10 client.17232 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.415+0000 7f1007fc4740 20 client.17232 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.415+0000 7f1007fc4740 1 --2- 172.21.15.23:0/138170974 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55967c216be0 0x55967c219020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.415+0000 7f1007fc4740 4 client.17232 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.415+0000 7f1007fc4740 1 -- 172.21.15.23:0/138170974 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55967be63600 con 0x55967c216be0 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.415+0000 7f0ff7fff640 1 --2- 172.21.15.23:0/138170974 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55967c216be0 0x55967c219020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.416+0000 7f0fe67fc640 10 client.17232 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.416+0000 7f0ff7fff640 1 --2- 172.21.15.23:0/138170974 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55967c216be0 0x55967c219020 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a60446482d3e151 server_cookie=1ab832097ad9b04e in_seq=0 out_seq=0 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.416+0000 7f0fe67fc640 1 -- 172.21.15.23:0/138170974 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55967be63600 con 0x55967c216be0 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.416+0000 7f0fe67fc640 10 client.17232 handle_command_reply: tid=0 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.416+0000 7f1007fc4740 1 client.17232 shutdown 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.417+0000 7f1007fc4740 1 -- 172.21.15.23:0/138170974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fd5e0 msgr2=0x55967c203cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.417+0000 7f1007fc4740 1 --2- 172.21.15.23:0/138170974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fd5e0 0x55967c203cf0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0003730 tx=0x7f0ff0004730 comp rx=0 tx=0).stop 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.417+0000 7f1007fc4740 1 -- 172.21.15.23:0/138170974 shutdown_connections 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.417+0000 7f1007fc4740 1 --2- 172.21.15.23:0/138170974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c206cd0 0x55967c204770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.417+0000 7f1007fc4740 1 --2- 172.21.15.23:0/138170974 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55967c216be0 0x55967c219020 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.417+0000 7f1007fc4740 1 --2- 172.21.15.23:0/138170974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c1fdf00 0x55967c204230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.417+0000 7f1007fc4740 1 --2- 172.21.15.23:0/138170974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fd5e0 0x55967c203cf0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.417+0000 7f0fe67fc640 0 client.17232 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.417+0000 7f1007fc4740 1 -- 172.21.15.23:0/138170974 >> 172.21.15.23:0/138170974 conn(0x55967c1f7f00 msgr2=0x55967c1f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.417+0000 7f1007fc4740 1 -- 172.21.15.23:0/138170974 shutdown_connections 2024-03-24T10:45:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.417+0000 7f1007fc4740 1 -- 172.21.15.23:0/138170974 wait complete. 2024-03-24T10:45:57.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.417+0000 7f1007fc4740 20 client.17232 trim_cache size 0 max 16384 2024-03-24T10:45:57.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.432+0000 7f1007fc4740 1 Processor -- start 2024-03-24T10:45:57.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.432+0000 7f1007fc4740 1 -- start start 2024-03-24T10:45:57.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.432+0000 7f1007fc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 0x55967c1f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.432+0000 7f1007fc4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c1fdf00 0x55967c2123a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.432+0000 7f1007fc4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c206cd0 0x55967c204590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.432+0000 7f1007fc4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55967c0e9660 con 0x55967c206cd0 2024-03-24T10:45:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.432+0000 7f1007fc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55967c2067f0 con 0x55967c1fd5e0 2024-03-24T10:45:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.432+0000 7f1007fc4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55967c1fcaa0 con 0x55967c1fdf00 2024-03-24T10:45:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.433+0000 7f0fff7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c206cd0 0x55967c204590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.433+0000 7f0fff7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c206cd0 0x55967c204590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49044/0 (socket says 172.21.15.23:49044) 2024-03-24T10:45:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.433+0000 7f0fff7fe640 1 -- 172.21.15.23:0/1179494559 learned_addr learned my addr 172.21.15.23:0/1179494559 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.433+0000 7f0ffeffd640 1 --2- 172.21.15.23:0/1179494559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 0x55967c1f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.433+0000 7f0ff7fff640 1 --2- 172.21.15.23:0/1179494559 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c1fdf00 0x55967c2123a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.433+0000 7f0fff7fe640 1 -- 172.21.15.23:0/1179494559 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c1fdf00 msgr2=0x55967c2123a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.433+0000 7f0fff7fe640 1 --2- 172.21.15.23:0/1179494559 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c1fdf00 0x55967c2123a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.434 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.433+0000 7f0fff7fe640 1 -- 172.21.15.23:0/1179494559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 msgr2=0x55967c1f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:57.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.433+0000 7f0fff7fe640 1 --2- 172.21.15.23:0/1179494559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 0x55967c1f2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.433+0000 7f0fff7fe640 1 -- 172.21.15.23:0/1179494559 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55967c0fe810 con 0x55967c206cd0 2024-03-24T10:45:57.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.433+0000 7f0fff7fe640 1 --2- 172.21.15.23:0/1179494559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c206cd0 0x55967c204590 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0ff80ee590 tx=0x7f0ff80b9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba7691d8931dc6a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:57.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.433+0000 7f0ff4ff9640 1 -- 172.21.15.23:0/1179494559 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff80c1490 con 0x55967c206cd0 2024-03-24T10:45:57.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.433+0000 7f0ff4ff9640 1 -- 172.21.15.23:0/1179494559 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ff80c5ae0 con 0x55967c206cd0 2024-03-24T10:45:57.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.433+0000 7f0ff4ff9640 1 -- 172.21.15.23:0/1179494559 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff80b9bd0 con 0x55967c206cd0 2024-03-24T10:45:57.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.434+0000 7f1007fc4740 1 -- 172.21.15.23:0/1179494559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c206cd0 msgr2=0x55967c204590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:57.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.434+0000 7f1007fc4740 1 --2- 172.21.15.23:0/1179494559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c206cd0 0x55967c204590 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0ff80ee590 tx=0x7f0ff80b9040 comp rx=0 tx=0).stop 2024-03-24T10:45:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.434+0000 7f1007fc4740 1 -- 172.21.15.23:0/1179494559 shutdown_connections 2024-03-24T10:45:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.434+0000 7f1007fc4740 1 --2- 172.21.15.23:0/1179494559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c206cd0 0x55967c204590 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.434+0000 7f1007fc4740 1 --2- 172.21.15.23:0/1179494559 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c1fdf00 0x55967c2123a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.434+0000 7f1007fc4740 1 --2- 172.21.15.23:0/1179494559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 0x55967c1f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.434+0000 7f1007fc4740 1 -- 172.21.15.23:0/1179494559 >> 172.21.15.23:0/1179494559 conn(0x55967c1f7f00 msgr2=0x55967c1f82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.434+0000 7f1007fc4740 1 -- 172.21.15.23:0/1179494559 shutdown_connections 2024-03-24T10:45:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.434+0000 7f1007fc4740 1 -- 172.21.15.23:0/1179494559 wait complete. 2024-03-24T10:45:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.434+0000 7f1007fc4740 1 -- start start 2024-03-24T10:45:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f1007fc4740 1 Processor -- start 2024-03-24T10:45:57.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f1007fc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 0x55967c20ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f1007fc4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fdf00 0x55967c200580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f1007fc4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c206cd0 0x55967c200ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f1007fc4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55967c201000 con 0x55967c1fdf00 2024-03-24T10:45:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f1007fc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55967c201170 con 0x55967c1fd5e0 2024-03-24T10:45:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f1007fc4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55967c201330 con 0x55967c206cd0 2024-03-24T10:45:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f1007fc4740 10 client.0 _ll_register_callbacks cb 0x7f0ff0004130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f0ff7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fdf00 0x55967c200580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f0ff7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fdf00 0x55967c200580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49056/0 (socket says 172.21.15.23:49056) 2024-03-24T10:45:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f0ff7fff640 1 -- 172.21.15.23:0/2032386677 learned_addr learned my addr 172.21.15.23:0/2032386677 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:45:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f0ffeffd640 1 --2- 172.21.15.23:0/2032386677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 0x55967c20ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f0fff7fe640 1 --2- 172.21.15.23:0/2032386677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c206cd0 0x55967c200ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f0ff7fff640 1 -- 172.21.15.23:0/2032386677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c206cd0 msgr2=0x55967c200ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f0ff7fff640 1 --2- 172.21.15.23:0/2032386677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c206cd0 0x55967c200ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f0ff7fff640 1 -- 172.21.15.23:0/2032386677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 msgr2=0x55967c20ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f0ff7fff640 1 --2- 172.21.15.23:0/2032386677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 0x55967c20ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f0ff7fff640 1 -- 172.21.15.23:0/2032386677 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10001a48f0 con 0x55967c1fdf00 2024-03-24T10:45:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f0ff7fff640 1 --2- 172.21.15.23:0/2032386677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fdf00 0x55967c200580 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8013010 tx=0x7f0fe800e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f03e357bc75db269 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.435+0000 7f0fe67fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:45:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f0fe67fc640 1 -- 172.21.15.23:0/2032386677 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8029040 con 0x55967c1fdf00 2024-03-24T10:45:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f0fe67fc640 1 -- 172.21.15.23:0/2032386677 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f0fe802bc30 con 0x55967c1fdf00 2024-03-24T10:45:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f1007fc4740 1 -- 172.21.15.23:0/2032386677 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55967c1feb40 con 0x55967c1fdf00 2024-03-24T10:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f0fe67fc640 1 -- 172.21.15.23:0/2032386677 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0fe800e6b0 con 0x55967c1fdf00 2024-03-24T10:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f1007fc4740 10 client.17238 fetch_fsmap learned FSMap version 29 2024-03-24T10:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f1007fc4740 1 -- 172.21.15.23:0/2032386677 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55967c1245a0 con 0x55967c1fdf00 2024-03-24T10:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f0fe67fc640 1 -- 172.21.15.23:0/2032386677 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55967c1feb40 con 0x55967c1fdf00 2024-03-24T10:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f0fe67fc640 1 -- 172.21.15.23:0/2032386677 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f0fe80050b0 con 0x55967c1fdf00 2024-03-24T10:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f1007fc4740 10 client.17238 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f1007fc4740 10 client.17238 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f1007fc4740 20 client.17238 populate_metadata read hostname 'smithi023' 2024-03-24T10:45:57.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f1007fc4740 1 --2- 172.21.15.23:0/2032386677 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55967c27d480 0x55967c27f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f1007fc4740 4 client.17238 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f1007fc4740 1 -- 172.21.15.23:0/2032386677 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f0fe8019c70 con 0x55967c27d480 2024-03-24T10:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.436+0000 7f0ffeffd640 1 --2- 172.21.15.23:0/2032386677 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55967c27d480 0x55967c27f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.437+0000 7f0ffeffd640 1 --2- 172.21.15.23:0/2032386677 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55967c27d480 0x55967c27f8a0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98388c7f7040430a server_cookie=81179f69e76960f4 in_seq=0 out_seq=0 2024-03-24T10:45:57.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.437+0000 7f0fe67fc640 10 client.17238 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f0fe67fc640 1 -- 172.21.15.23:0/2032386677 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f0fe8019c70 con 0x55967c27d480 2024-03-24T10:45:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f0fe67fc640 10 client.17238 handle_command_reply: tid=0 2024-03-24T10:45:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f1007fc4740 1 client.17238 shutdown 2024-03-24T10:45:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f1007fc4740 1 -- 172.21.15.23:0/2032386677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fdf00 msgr2=0x55967c200580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f1007fc4740 1 --2- 172.21.15.23:0/2032386677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fdf00 0x55967c200580 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8013010 tx=0x7f0fe800e820 comp rx=0 tx=0).stop 2024-03-24T10:45:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f1007fc4740 1 -- 172.21.15.23:0/2032386677 shutdown_connections 2024-03-24T10:45:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f1007fc4740 1 --2- 172.21.15.23:0/2032386677 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55967c27d480 0x55967c27f8a0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f1007fc4740 1 --2- 172.21.15.23:0/2032386677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55967c206cd0 0x55967c200ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f1007fc4740 1 --2- 172.21.15.23:0/2032386677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55967c1fdf00 0x55967c200580 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f0fe67fc640 0 client.17238 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:45:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f1007fc4740 1 --2- 172.21.15.23:0/2032386677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55967c1fd5e0 0x55967c20ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f1007fc4740 1 -- 172.21.15.23:0/2032386677 >> 172.21.15.23:0/2032386677 conn(0x55967c1f7f00 msgr2=0x55967c1f6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f1007fc4740 1 -- 172.21.15.23:0/2032386677 shutdown_connections 2024-03-24T10:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f1007fc4740 1 -- 172.21.15.23:0/2032386677 wait complete. 2024-03-24T10:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.447+0000 7f1007fc4740 20 client.17238 trim_cache size 0 max 16384 2024-03-24T10:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:45:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:45:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.448+0000 7f10053e2640 1 -- 172.21.15.23:0/4210692644 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0fec0a22d0 msgr2=0x7f0fec0a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.448+0000 7f10053e2640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0fec0a22d0 0x7f0fec0a4790 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009fc0 tx=0x7f0ff0005f90 comp rx=0 tx=0).stop 2024-03-24T10:45:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.448+0000 7f10053e2640 1 -- 172.21.15.23:0/4210692644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10001055b0 msgr2=0x7f10001a3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.448+0000 7f10053e2640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10001055b0 0x7f10001a3b70 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8005e30 tx=0x7f0ff8035aa0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.448+0000 7f10053e2640 1 -- 172.21.15.23:0/4210692644 shutdown_connections 2024-03-24T10:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.448+0000 7f10053e2640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0fec0a22d0 0x7f0fec0a4790 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.448+0000 7f10053e2640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10001055b0 0x7f10001a3b70 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.448+0000 7f10053e2640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1000101220 0x7f1000199030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.448+0000 7f10053e2640 1 --2- 172.21.15.23:0/4210692644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1000100860 0x7f1000198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.448+0000 7f10053e2640 1 -- 172.21.15.23:0/4210692644 >> 172.21.15.23:0/4210692644 conn(0x7f1000077460 msgr2=0x7f10000ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.448+0000 7f10053e2640 1 -- 172.21.15.23:0/4210692644 shutdown_connections 2024-03-24T10:45:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:45:57.448+0000 7f10053e2640 1 -- 172.21.15.23:0/4210692644 wait complete. 2024-03-24T10:45:58.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 558275 11.49 MB/sec execute 135 sec latency 17.652 ms 2024-03-24T10:45:59.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:45:58 smithi116 ceph-mon[36248]: pgmap v180: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 272 op/s 2024-03-24T10:45:59.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 560140 11.46 MB/sec execute 136 sec latency 36.288 ms 2024-03-24T10:45:59.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:45:58 smithi023 ceph-mon[32466]: pgmap v180: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 272 op/s 2024-03-24T10:45:59.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:45:58 smithi117 ceph-mon[36368]: pgmap v180: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 272 op/s 2024-03-24T10:46:00.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 561795 11.46 MB/sec execute 137 sec latency 80.314 ms 2024-03-24T10:46:01.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:00 smithi116 ceph-mon[36248]: pgmap v181: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 180 op/s 2024-03-24T10:46:01.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 563520 11.42 MB/sec execute 138 sec latency 25.688 ms 2024-03-24T10:46:01.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:00 smithi023 ceph-mon[32466]: pgmap v181: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 180 op/s 2024-03-24T10:46:01.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:00 smithi117 ceph-mon[36368]: pgmap v181: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 180 op/s 2024-03-24T10:46:02.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 565187 11.43 MB/sec execute 139 sec latency 23.207 ms 2024-03-24T10:46:03.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:02 smithi116 ceph-mon[36248]: pgmap v182: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 298 op/s 2024-03-24T10:46:03.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 566935 11.39 MB/sec execute 140 sec latency 19.019 ms 2024-03-24T10:46:03.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:02 smithi023 ceph-mon[32466]: pgmap v182: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 298 op/s 2024-03-24T10:46:03.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:02 smithi117 ceph-mon[36368]: pgmap v182: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 298 op/s 2024-03-24T10:46:04.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 568932 11.41 MB/sec execute 141 sec latency 22.555 ms 2024-03-24T10:46:05.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:04 smithi116 ceph-mon[36248]: pgmap v183: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 202 op/s 2024-03-24T10:46:05.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 570485 11.37 MB/sec execute 142 sec latency 96.781 ms 2024-03-24T10:46:05.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:04 smithi023 ceph-mon[32466]: pgmap v183: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 202 op/s 2024-03-24T10:46:05.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:04 smithi117 ceph-mon[36368]: pgmap v183: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 202 op/s 2024-03-24T10:46:06.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 572554 11.38 MB/sec execute 143 sec latency 27.438 ms 2024-03-24T10:46:07.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 574463 11.36 MB/sec execute 144 sec latency 25.587 ms 2024-03-24T10:46:07.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:06 smithi023 ceph-mon[32466]: pgmap v184: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 202 op/s 2024-03-24T10:46:07.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:06 smithi117 ceph-mon[36368]: pgmap v184: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 202 op/s 2024-03-24T10:46:07.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:06 smithi116 ceph-mon[36248]: pgmap v184: 97 pgs: 97 active+clean; 349 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.7 MiB/s wr, 202 op/s 2024-03-24T10:46:08.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 576308 11.36 MB/sec execute 145 sec latency 22.913 ms 2024-03-24T10:46:09.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 578448 11.36 MB/sec execute 146 sec latency 17.753 ms 2024-03-24T10:46:09.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:08 smithi023 ceph-mon[32466]: pgmap v185: 97 pgs: 97 active+clean; 357 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.2 MiB/s wr, 292 op/s 2024-03-24T10:46:09.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:08 smithi117 ceph-mon[36368]: pgmap v185: 97 pgs: 97 active+clean; 357 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.2 MiB/s wr, 292 op/s 2024-03-24T10:46:09.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:08 smithi116 ceph-mon[36248]: pgmap v185: 97 pgs: 97 active+clean; 357 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.2 MiB/s wr, 292 op/s 2024-03-24T10:46:09.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:46:09.488 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:46:09.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.611+0000 7f35080ed640 1 -- 172.21.15.23:0/3843492763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35001005b0 msgr2=0x7f350010e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:09.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.611+0000 7f35080ed640 1 --2- 172.21.15.23:0/3843492763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35001005b0 0x7f350010e590 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009860 tx=0x7f34fc02bb70 comp rx=0 tx=0).stop 2024-03-24T10:46:09.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.612+0000 7f35080ed640 1 -- 172.21.15.23:0/3843492763 shutdown_connections 2024-03-24T10:46:09.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.612+0000 7f35080ed640 1 --2- 172.21.15.23:0/3843492763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35001005b0 0x7f350010e590 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.612+0000 7f35080ed640 1 --2- 172.21.15.23:0/3843492763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35000ffc10 0x7f3500100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.612+0000 7f35080ed640 1 --2- 172.21.15.23:0/3843492763 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35001076d0 0x7f3500107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.612+0000 7f35080ed640 1 -- 172.21.15.23:0/3843492763 >> 172.21.15.23:0/3843492763 conn(0x7f3500076bd0 msgr2=0x7f35000fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:09.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.615+0000 7f35080ed640 1 -- 172.21.15.23:0/3843492763 shutdown_connections 2024-03-24T10:46:09.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.615+0000 7f35080ed640 1 -- 172.21.15.23:0/3843492763 wait complete. 2024-03-24T10:46:09.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.616+0000 7f35080ed640 1 Processor -- start 2024-03-24T10:46:09.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.616+0000 7f35080ed640 1 -- start start 2024-03-24T10:46:09.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.617+0000 7f35080ed640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35000ffc10 0x7f35001ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.617+0000 7f35080ed640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35001005b0 0x7f35001acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.617+0000 7f35080ed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35001076d0 0x7f35001a6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.617+0000 7f35080ed640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f35001a70e0 con 0x7f35001005b0 2024-03-24T10:46:09.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.617+0000 7f35080ed640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f35001a7250 con 0x7f35001076d0 2024-03-24T10:46:09.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.617+0000 7f35080ed640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f35001a7410 con 0x7f35000ffc10 2024-03-24T10:46:09.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.617+0000 7f350668a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35001076d0 0x7f35001a6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.618+0000 7f350668a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35001076d0 0x7f35001a6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:41724/0 (socket says 172.21.15.23:41724) 2024-03-24T10:46:09.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.618+0000 7f350668a640 1 -- 172.21.15.23:0/3116446155 learned_addr learned my addr 172.21.15.23:0/3116446155 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:09.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.618+0000 7f3505e89640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35000ffc10 0x7f35001ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.618+0000 7f350668a640 1 -- 172.21.15.23:0/3116446155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35000ffc10 msgr2=0x7f35001ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:09.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.618+0000 7f350668a640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35000ffc10 0x7f35001ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.618+0000 7f350668a640 1 -- 172.21.15.23:0/3116446155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35001005b0 msgr2=0x7f35001acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:09.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.618+0000 7f3505688640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35001005b0 0x7f35001acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.619+0000 7f350668a640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35001005b0 0x7f35001acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.619+0000 7f350668a640 1 -- 172.21.15.23:0/3116446155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34fc009490 con 0x7f35001076d0 2024-03-24T10:46:09.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.619+0000 7f3505688640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35001005b0 0x7f35001acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:46:09.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.620+0000 7f350668a640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35001076d0 0x7f35001a6a90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f34fc02c180 tx=0x7f34fc0045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:09.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.622+0000 7f34f6ffd640 1 -- 172.21.15.23:0/3116446155 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34fc034a10 con 0x7f35001076d0 2024-03-24T10:46:09.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.623+0000 7f34f6ffd640 1 -- 172.21.15.23:0/3116446155 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34fc02c9f0 con 0x7f35001076d0 2024-03-24T10:46:09.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.623+0000 7f34f6ffd640 1 -- 172.21.15.23:0/3116446155 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34fc0035f0 con 0x7f35001076d0 2024-03-24T10:46:09.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.623+0000 7f35080ed640 1 -- 172.21.15.23:0/3116446155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35001a7690 con 0x7f35001076d0 2024-03-24T10:46:09.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.623+0000 7f35080ed640 1 -- 172.21.15.23:0/3116446155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35001a7910 con 0x7f35001076d0 2024-03-24T10:46:09.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f34f6ffd640 1 -- 172.21.15.23:0/3116446155 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f34fc03b040 con 0x7f35001076d0 2024-03-24T10:46:09.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f350adae740 1 Processor -- start 2024-03-24T10:46:09.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f350adae740 1 -- start start 2024-03-24T10:46:09.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f350adae740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e7701d0 0x55783e7705b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f350adae740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e770af0 0x55783e779190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f350adae740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 0x55783e77bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f350adae740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55783e694f30 con 0x55783e7796d0 2024-03-24T10:46:09.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f350adae740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55783e6de810 con 0x55783e770af0 2024-03-24T10:46:09.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f350adae740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55783e6dbd40 con 0x55783e7701d0 2024-03-24T10:46:09.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f34f6ffd640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34ec0a22f0 0x7f34ec0a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f3505688640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 0x55783e77bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f3505e89640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e770af0 0x55783e779190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f3505e89640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e770af0 0x55783e779190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:41740/0 (socket says 172.21.15.23:41740) 2024-03-24T10:46:09.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f3505688640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 0x55783e77bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56098/0 (socket says 172.21.15.23:56098) 2024-03-24T10:46:09.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.627+0000 7f350668a640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34ec0a22f0 0x7f34ec0a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.626+0000 7f3505e89640 1 -- 172.21.15.23:0/1153749587 learned_addr learned my addr 172.21.15.23:0/1153749587 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:09.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.627+0000 7f3505688640 1 --2- 172.21.15.23:0/1153749587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e7701d0 0x55783e7705b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.627+0000 7f34f6ffd640 1 -- 172.21.15.23:0/3116446155 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f34fc0efa80 con 0x7f35001076d0 2024-03-24T10:46:09.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.627+0000 7f3505688640 1 -- 172.21.15.23:0/1153749587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e7701d0 msgr2=0x55783e7705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:09.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.627+0000 7f3505688640 1 --2- 172.21.15.23:0/1153749587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e7701d0 0x55783e7705b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.627+0000 7f3505688640 1 -- 172.21.15.23:0/1153749587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e770af0 msgr2=0x55783e779190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:09.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.627+0000 7f3505688640 1 --2- 172.21.15.23:0/1153749587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e770af0 0x55783e779190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.627+0000 7f3505688640 1 -- 172.21.15.23:0/1153749587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55783e698ea0 con 0x55783e7796d0 2024-03-24T10:46:09.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.627+0000 7f3505688640 1 --2- 172.21.15.23:0/1153749587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 0x55783e77bac0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f35001a8230 tx=0x7f34f000c190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1f5b8258e600bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:09.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.627+0000 7f350668a640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34ec0a22f0 0x7f34ec0a47b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f34fc0ed390 tx=0x7f34fc0a7040 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:09.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.627+0000 7f34ebfff640 1 -- 172.21.15.23:0/1153749587 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0018470 con 0x55783e7796d0 2024-03-24T10:46:09.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.627+0000 7f34ebfff640 1 -- 172.21.15.23:0/1153749587 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f0018ab0 con 0x55783e7796d0 2024-03-24T10:46:09.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.627+0000 7f34ebfff640 1 -- 172.21.15.23:0/1153749587 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0018470 con 0x55783e7796d0 2024-03-24T10:46:09.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.628+0000 7f350adae740 1 -- 172.21.15.23:0/1153749587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 msgr2=0x55783e77bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:09.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.628+0000 7f350adae740 1 --2- 172.21.15.23:0/1153749587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 0x55783e77bac0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f35001a8230 tx=0x7f34f000c190 comp rx=0 tx=0).stop 2024-03-24T10:46:09.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.628+0000 7f350adae740 1 -- 172.21.15.23:0/1153749587 shutdown_connections 2024-03-24T10:46:09.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.628+0000 7f350adae740 1 --2- 172.21.15.23:0/1153749587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 0x55783e77bac0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.628+0000 7f350adae740 1 --2- 172.21.15.23:0/1153749587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e770af0 0x55783e779190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.628+0000 7f350adae740 1 --2- 172.21.15.23:0/1153749587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e7701d0 0x55783e7705b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.628+0000 7f350adae740 1 -- 172.21.15.23:0/1153749587 >> 172.21.15.23:0/1153749587 conn(0x55783e76a900 msgr2=0x55783e76ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:09.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.628+0000 7f350adae740 1 -- 172.21.15.23:0/1153749587 shutdown_connections 2024-03-24T10:46:09.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.628+0000 7f350adae740 1 -- 172.21.15.23:0/1153749587 wait complete. 2024-03-24T10:46:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.629+0000 7f350adae740 1 -- start start 2024-03-24T10:46:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.629+0000 7f350adae740 1 Processor -- start 2024-03-24T10:46:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.629+0000 7f350adae740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7701d0 0x55783e776770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.629+0000 7f350adae740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e776cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.629+0000 7f350adae740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 0x55783e7771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f3505e89640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e776cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f350adae740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55783e6d7520 con 0x55783e7796d0 2024-03-24T10:46:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f350adae740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55783e3aa430 con 0x55783e7701d0 2024-03-24T10:46:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f3505688640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7701d0 0x55783e776770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f3505688640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7701d0 0x55783e776770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:41744/0 (socket says 172.21.15.23:41744) 2024-03-24T10:46:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f3505688640 1 -- 172.21.15.23:0/3520788513 learned_addr learned my addr 172.21.15.23:0/3520788513 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f350adae740 1 -- 172.21.15.23:0/3520788513 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55783e366e00 con 0x55783e770af0 2024-03-24T10:46:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f350adae740 10 client.0 _ll_register_callbacks cb 0x55783e675d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f3505688640 1 --2- 172.21.15.23:0/3520788513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 0x55783e7771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f3505e89640 1 -- 172.21.15.23:0/3520788513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7701d0 msgr2=0x55783e776770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f3505e89640 1 --2- 172.21.15.23:0/3520788513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7701d0 0x55783e776770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f3505e89640 1 -- 172.21.15.23:0/3520788513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 msgr2=0x55783e7771f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f3505e89640 1 --2- 172.21.15.23:0/3520788513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 0x55783e7771f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f3505e89640 1 -- 172.21.15.23:0/3520788513 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f0015300 con 0x55783e770af0 2024-03-24T10:46:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f3505688640 1 --2- 172.21.15.23:0/3520788513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7701d0 0x55783e776770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f3505e89640 1 --2- 172.21.15.23:0/3520788513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e776cb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f34f8013480 tx=0x7f34f8013940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa1bb9115bba2f3d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.630+0000 7f34e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:46:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f34e97fa640 1 -- 172.21.15.23:0/3520788513 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f8018070 con 0x55783e770af0 2024-03-24T10:46:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f350adae740 1 -- 172.21.15.23:0/3520788513 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55783e3d67e0 con 0x55783e770af0 2024-03-24T10:46:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f34e97fa640 1 -- 172.21.15.23:0/3520788513 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f34f800fbb0 con 0x55783e770af0 2024-03-24T10:46:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f34e97fa640 1 -- 172.21.15.23:0/3520788513 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f34f800fd70 con 0x55783e770af0 2024-03-24T10:46:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f350adae740 10 client.27103 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f350adae740 1 -- 172.21.15.23:0/3520788513 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55783e6d5ba0 con 0x55783e770af0 2024-03-24T10:46:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f34e97fa640 1 -- 172.21.15.23:0/3520788513 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f8029080 con 0x55783e770af0 2024-03-24T10:46:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f34e97fa640 1 -- 172.21.15.23:0/3520788513 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f34f8023030 con 0x55783e770af0 2024-03-24T10:46:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f350adae740 10 client.27103 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f350adae740 10 client.27103 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f350adae740 20 client.27103 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f350adae740 1 --2- 172.21.15.23:0/3520788513 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55783e789c40 0x55783e78c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f350adae740 4 client.27103 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f350adae740 1 -- 172.21.15.23:0/3520788513 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55783e413720 con 0x55783e789c40 2024-03-24T10:46:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.631+0000 7f3505688640 1 --2- 172.21.15.23:0/3520788513 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55783e789c40 0x55783e78c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.632+0000 7f34e97fa640 10 client.27103 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.632+0000 7f3505688640 1 --2- 172.21.15.23:0/3520788513 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55783e789c40 0x55783e78c060 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a238def5fe8b08b3 server_cookie=499550fcad9cb59 in_seq=0 out_seq=0 2024-03-24T10:46:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f34e97fa640 1 -- 172.21.15.23:0/3520788513 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55783e413720 con 0x55783e789c40 2024-03-24T10:46:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f34e97fa640 10 client.27103 handle_command_reply: tid=0 2024-03-24T10:46:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f350adae740 1 client.27103 shutdown 2024-03-24T10:46:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f350adae740 1 -- 172.21.15.23:0/3520788513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 msgr2=0x55783e776cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f350adae740 1 --2- 172.21.15.23:0/3520788513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e776cb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f34f8013480 tx=0x7f34f8013940 comp rx=0 tx=0).stop 2024-03-24T10:46:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f350adae740 1 -- 172.21.15.23:0/3520788513 shutdown_connections 2024-03-24T10:46:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f350adae740 1 --2- 172.21.15.23:0/3520788513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 0x55783e7771f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f350adae740 1 --2- 172.21.15.23:0/3520788513 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55783e789c40 0x55783e78c060 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f350adae740 1 --2- 172.21.15.23:0/3520788513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e776cb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f350adae740 1 --2- 172.21.15.23:0/3520788513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7701d0 0x55783e776770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f34e97fa640 0 client.27103 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f350adae740 1 -- 172.21.15.23:0/3520788513 >> 172.21.15.23:0/3520788513 conn(0x55783e76a900 msgr2=0x55783e769b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f350adae740 1 -- 172.21.15.23:0/3520788513 shutdown_connections 2024-03-24T10:46:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f350adae740 1 -- 172.21.15.23:0/3520788513 wait complete. 2024-03-24T10:46:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.633+0000 7f350adae740 20 client.27103 trim_cache size 0 max 16384 2024-03-24T10:46:09.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.650+0000 7f350adae740 1 Processor -- start 2024-03-24T10:46:09.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.650+0000 7f350adae740 1 -- start start 2024-03-24T10:46:09.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f350adae740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7701d0 0x55783e784f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f350adae740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e777010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f350adae740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7796d0 0x55783e791970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f350adae740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55783e65c430 con 0x55783e7701d0 2024-03-24T10:46:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f350adae740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55783e764f70 con 0x55783e7796d0 2024-03-24T10:46:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f350adae740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55783e766550 con 0x55783e770af0 2024-03-24T10:46:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f3505688640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7701d0 0x55783e784f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f3505688640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7701d0 0x55783e784f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56124/0 (socket says 172.21.15.23:56124) 2024-03-24T10:46:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f3505688640 1 -- 172.21.15.23:0/1505588664 learned_addr learned my addr 172.21.15.23:0/1505588664 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f3505e89640 1 --2- 172.21.15.23:0/1505588664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e777010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f3505688640 1 --2- 172.21.15.23:0/1505588664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7796d0 0x55783e791970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f3505688640 1 -- 172.21.15.23:0/1505588664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 msgr2=0x55783e777010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f3505688640 1 --2- 172.21.15.23:0/1505588664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e777010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f3505688640 1 -- 172.21.15.23:0/1505588664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7796d0 msgr2=0x55783e791970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f3505688640 1 --2- 172.21.15.23:0/1505588664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7796d0 0x55783e791970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f3505688640 1 -- 172.21.15.23:0/1505588664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55783e671b20 con 0x55783e7701d0 2024-03-24T10:46:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f3505e89640 1 --2- 172.21.15.23:0/1505588664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e777010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:09.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.651+0000 7f3505688640 1 --2- 172.21.15.23:0/1505588664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7701d0 0x55783e784f60 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f34f0008010 tx=0x7f34f00047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d406823f1e8b4d2f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:09.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.652+0000 7f34eb7fe640 1 -- 172.21.15.23:0/1505588664 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0018470 con 0x55783e7701d0 2024-03-24T10:46:09.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.652+0000 7f34eb7fe640 1 -- 172.21.15.23:0/1505588664 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f002b070 con 0x55783e7701d0 2024-03-24T10:46:09.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.652+0000 7f34eb7fe640 1 -- 172.21.15.23:0/1505588664 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0025670 con 0x55783e7701d0 2024-03-24T10:46:09.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.652+0000 7f350adae740 1 -- 172.21.15.23:0/1505588664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7701d0 msgr2=0x55783e784f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:09.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.652+0000 7f350adae740 1 --2- 172.21.15.23:0/1505588664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7701d0 0x55783e784f60 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f34f0008010 tx=0x7f34f00047d0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.653+0000 7f350adae740 1 -- 172.21.15.23:0/1505588664 shutdown_connections 2024-03-24T10:46:09.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.653+0000 7f350adae740 1 --2- 172.21.15.23:0/1505588664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7796d0 0x55783e791970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.653+0000 7f350adae740 1 --2- 172.21.15.23:0/1505588664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e777010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.653+0000 7f350adae740 1 --2- 172.21.15.23:0/1505588664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7701d0 0x55783e784f60 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.653+0000 7f350adae740 1 -- 172.21.15.23:0/1505588664 >> 172.21.15.23:0/1505588664 conn(0x55783e76a900 msgr2=0x55783e769b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.653+0000 7f350adae740 1 -- 172.21.15.23:0/1505588664 shutdown_connections 2024-03-24T10:46:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.653+0000 7f350adae740 1 -- 172.21.15.23:0/1505588664 wait complete. 2024-03-24T10:46:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.653+0000 7f350adae740 1 -- start start 2024-03-24T10:46:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.653+0000 7f350adae740 1 Processor -- start 2024-03-24T10:46:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f350adae740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7701d0 0x55783e77d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f350adae740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e77dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f350adae740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 0x55783e7e8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f350adae740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55783e7e9520 con 0x55783e7796d0 2024-03-24T10:46:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f350adae740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55783e7e9660 con 0x55783e7701d0 2024-03-24T10:46:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f350adae740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55783e7e9820 con 0x55783e770af0 2024-03-24T10:46:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f350adae740 10 client.0 _ll_register_callbacks cb 0x7f34f8013e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f3505688640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 0x55783e7e8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f3505688640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7701d0 0x55783e77d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f3505e89640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e77dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f3505688640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7701d0 0x55783e77d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:41764/0 (socket says 172.21.15.23:41764) 2024-03-24T10:46:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f3505688640 1 -- 172.21.15.23:0/2766160068 learned_addr learned my addr 172.21.15.23:0/2766160068 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f3505e89640 1 -- 172.21.15.23:0/2766160068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7701d0 msgr2=0x55783e77d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f3505e89640 1 --2- 172.21.15.23:0/2766160068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7701d0 0x55783e77d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f3505e89640 1 -- 172.21.15.23:0/2766160068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 msgr2=0x55783e7e8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f3505e89640 1 --2- 172.21.15.23:0/2766160068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 0x55783e7e8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f3505e89640 1 -- 172.21.15.23:0/2766160068 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55783e6d5ba0 con 0x55783e770af0 2024-03-24T10:46:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f3505e89640 1 --2- 172.21.15.23:0/2766160068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e77dc90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f34f8000f80 tx=0x7f34f800fde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cf04d2b94f49487 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.654+0000 7f34e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:46:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f34e97fa640 1 -- 172.21.15.23:0/2766160068 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f8018040 con 0x55783e770af0 2024-03-24T10:46:09.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f34e97fa640 1 -- 172.21.15.23:0/2766160068 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f34f8009280 con 0x55783e770af0 2024-03-24T10:46:09.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f350adae740 1 -- 172.21.15.23:0/2766160068 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34f0018ab0 con 0x55783e770af0 2024-03-24T10:46:09.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f34e97fa640 1 -- 172.21.15.23:0/2766160068 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f34f8004bf0 con 0x55783e770af0 2024-03-24T10:46:09.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f350adae740 10 client.27109 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:09.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f350adae740 1 -- 172.21.15.23:0/2766160068 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55783e698ea0 con 0x55783e770af0 2024-03-24T10:46:09.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f34e97fa640 1 -- 172.21.15.23:0/2766160068 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f0018ab0 con 0x55783e770af0 2024-03-24T10:46:09.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f34e97fa640 1 -- 172.21.15.23:0/2766160068 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f34f8005190 con 0x55783e770af0 2024-03-24T10:46:09.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f350adae740 10 client.27109 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:09.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f350adae740 10 client.27109 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:09.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f350adae740 20 client.27109 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:09.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f350adae740 1 --2- 172.21.15.23:0/2766160068 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55783e7f1170 0x55783e7f3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f350adae740 4 client.27109 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:46:09.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f350adae740 1 -- 172.21.15.23:0/2766160068 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55783e462890 con 0x55783e7f1170 2024-03-24T10:46:09.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.655+0000 7f3505688640 1 --2- 172.21.15.23:0/2766160068 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55783e7f1170 0x55783e7f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f34e97fa640 10 client.27109 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f3505688640 1 --2- 172.21.15.23:0/2766160068 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55783e7f1170 0x55783e7f3590 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e31c5f5f2b0de34 server_cookie=fd0973c1fafb91c4 in_seq=0 out_seq=0 2024-03-24T10:46:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f34e97fa640 1 -- 172.21.15.23:0/2766160068 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55783e462890 con 0x55783e7f1170 2024-03-24T10:46:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f34e97fa640 10 client.27109 handle_command_reply: tid=0 2024-03-24T10:46:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f350adae740 1 client.27109 shutdown 2024-03-24T10:46:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f350adae740 1 -- 172.21.15.23:0/2766160068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 msgr2=0x55783e77dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f350adae740 1 --2- 172.21.15.23:0/2766160068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e77dc90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f34f8000f80 tx=0x7f34f800fde0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f350adae740 1 -- 172.21.15.23:0/2766160068 shutdown_connections 2024-03-24T10:46:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f350adae740 1 --2- 172.21.15.23:0/2766160068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55783e7796d0 0x55783e7e8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f350adae740 1 --2- 172.21.15.23:0/2766160068 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55783e7f1170 0x55783e7f3590 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f350adae740 1 --2- 172.21.15.23:0/2766160068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55783e770af0 0x55783e77dc90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f350adae740 1 --2- 172.21.15.23:0/2766160068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55783e7701d0 0x55783e77d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f34e97fa640 0 client.27109 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f350adae740 1 -- 172.21.15.23:0/2766160068 >> 172.21.15.23:0/2766160068 conn(0x55783e76a900 msgr2=0x55783e76b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f350adae740 1 -- 172.21.15.23:0/2766160068 shutdown_connections 2024-03-24T10:46:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f350adae740 1 -- 172.21.15.23:0/2766160068 wait complete. 2024-03-24T10:46:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.656+0000 7f350adae740 20 client.27109 trim_cache size 0 max 16384 2024-03-24T10:46:09.663 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:46:09.663 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:46:09.663 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:46:09.663 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:46:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.657+0000 7f35080ed640 1 -- 172.21.15.23:0/3116446155 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34ec0a22f0 msgr2=0x7f34ec0a47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.657+0000 7f35080ed640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34ec0a22f0 0x7f34ec0a47b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f34fc0ed390 tx=0x7f34fc0a7040 comp rx=0 tx=0).stop 2024-03-24T10:46:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.657+0000 7f35080ed640 1 -- 172.21.15.23:0/3116446155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35001076d0 msgr2=0x7f35001a6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.657+0000 7f35080ed640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35001076d0 0x7f35001a6a90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f34fc02c180 tx=0x7f34fc0045e0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.657+0000 7f35080ed640 1 -- 172.21.15.23:0/3116446155 shutdown_connections 2024-03-24T10:46:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.657+0000 7f35080ed640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34ec0a22f0 0x7f34ec0a47b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.657+0000 7f35080ed640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35001076d0 0x7f35001a6a90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.657+0000 7f35080ed640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35001005b0 0x7f35001acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.657+0000 7f35080ed640 1 --2- 172.21.15.23:0/3116446155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35000ffc10 0x7f35001ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.657+0000 7f35080ed640 1 -- 172.21.15.23:0/3116446155 >> 172.21.15.23:0/3116446155 conn(0x7f3500076bd0 msgr2=0x7f35000fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.657+0000 7f35080ed640 1 -- 172.21.15.23:0/3116446155 shutdown_connections 2024-03-24T10:46:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.657+0000 7f35080ed640 1 -- 172.21.15.23:0/3116446155 wait complete. 2024-03-24T10:46:09.668 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:46:09.668 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:46:09.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.778+0000 7fd35d75c640 1 -- 172.21.15.23:0/4027716122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581008f0 msgr2=0x7fd358100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:09.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.778+0000 7fd35d75c640 1 --2- 172.21.15.23:0/4027716122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581008f0 0x7fd358100d70 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fd34c009930 tx=0x7fd34c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:46:09.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.778+0000 7fd35d75c640 1 -- 172.21.15.23:0/4027716122 shutdown_connections 2024-03-24T10:46:09.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.778+0000 7fd35d75c640 1 --2- 172.21.15.23:0/4027716122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3581012b0 0x7fd35810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.778+0000 7fd35d75c640 1 --2- 172.21.15.23:0/4027716122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581008f0 0x7fd358100d70 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.778+0000 7fd35d75c640 1 --2- 172.21.15.23:0/4027716122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd358106750 0x7fd358106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.778+0000 7fd35d75c640 1 -- 172.21.15.23:0/4027716122 >> 172.21.15.23:0/4027716122 conn(0x7fd3580775f0 msgr2=0x7fd358077a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:09.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.779+0000 7fd35d75c640 1 -- 172.21.15.23:0/4027716122 shutdown_connections 2024-03-24T10:46:09.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.779+0000 7fd35d75c640 1 -- 172.21.15.23:0/4027716122 wait complete. 2024-03-24T10:46:09.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.779+0000 7fd35d75c640 1 Processor -- start 2024-03-24T10:46:09.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.779+0000 7fd35d75c640 1 -- start start 2024-03-24T10:46:09.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.779+0000 7fd35d75c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3581008f0 0x7fd358197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.779+0000 7fd35d75c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581012b0 0x7fd3581982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.779+0000 7fd35d75c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd358106750 0x7fd35819c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd35d75c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd35819cc20 con 0x7fd3581012b0 2024-03-24T10:46:09.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd35d75c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd35819cd90 con 0x7fd3581008f0 2024-03-24T10:46:09.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd35d75c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd35819cf50 con 0x7fd358106750 2024-03-24T10:46:09.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd3567fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581012b0 0x7fd3581982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd3567fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581012b0 0x7fd3581982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56148/0 (socket says 172.21.15.23:56148) 2024-03-24T10:46:09.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd3567fc640 1 -- 172.21.15.23:0/3021266768 learned_addr learned my addr 172.21.15.23:0/3021266768 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:09.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd356ffd640 1 --2- 172.21.15.23:0/3021266768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3581008f0 0x7fd358197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd3577fe640 1 --2- 172.21.15.23:0/3021266768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd358106750 0x7fd35819c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd3567fc640 1 -- 172.21.15.23:0/3021266768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd358106750 msgr2=0x7fd35819c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:09.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd3567fc640 1 --2- 172.21.15.23:0/3021266768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd358106750 0x7fd35819c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd3567fc640 1 -- 172.21.15.23:0/3021266768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3581008f0 msgr2=0x7fd358197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:09.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd3567fc640 1 --2- 172.21.15.23:0/3021266768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3581008f0 0x7fd358197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:09.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd3567fc640 1 -- 172.21.15.23:0/3021266768 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd34c009560 con 0x7fd3581012b0 2024-03-24T10:46:09.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd3567fc640 1 --2- 172.21.15.23:0/3021266768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581012b0 0x7fd3581982a0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fd34c009230 tx=0x7fd34c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:09.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd33ffff640 1 -- 172.21.15.23:0/3021266768 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c004350 con 0x7fd3581012b0 2024-03-24T10:46:09.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd35d75c640 1 -- 172.21.15.23:0/3021266768 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd35819d1d0 con 0x7fd3581012b0 2024-03-24T10:46:09.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd33ffff640 1 -- 172.21.15.23:0/3021266768 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd34c0044b0 con 0x7fd3581012b0 2024-03-24T10:46:09.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd33ffff640 1 -- 172.21.15.23:0/3021266768 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c003d00 con 0x7fd3581012b0 2024-03-24T10:46:09.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.780+0000 7fd35d75c640 1 -- 172.21.15.23:0/3021266768 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3581a4ab0 con 0x7fd3581012b0 2024-03-24T10:46:09.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.781+0000 7fd35d75c640 1 -- 172.21.15.23:0/3021266768 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd358101d10 con 0x7fd3581012b0 2024-03-24T10:46:09.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.782+0000 7fd33ffff640 1 -- 172.21.15.23:0/3021266768 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fd34c004df0 con 0x7fd3581012b0 2024-03-24T10:46:09.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.783+0000 7fd33ffff640 1 --2- 172.21.15.23:0/3021266768 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd3400a2230 0x7fd3400a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:09.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.783+0000 7fd356ffd640 1 --2- 172.21.15.23:0/3021266768 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd3400a2230 0x7fd3400a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:09.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.783+0000 7fd33ffff640 1 -- 172.21.15.23:0/3021266768 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fd34c0f06c0 con 0x7fd3581012b0 2024-03-24T10:46:09.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.784+0000 7fd356ffd640 1 --2- 172.21.15.23:0/3021266768 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd3400a2230 0x7fd3400a46f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd348009b50 tx=0x7fd348005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:09.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:09.787+0000 7fd33ffff640 1 -- 172.21.15.23:0/3021266768 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd34c0b9460 con 0x7fd3581012b0 2024-03-24T10:46:10.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.029+0000 7fd35d75c640 1 -- 172.21.15.23:0/3021266768 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3581990b0 con 0x7fd3581012b0 2024-03-24T10:46:10.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.030+0000 7fd33ffff640 1 -- 172.21.15.23:0/3021266768 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11869 (secure 0 0 0) 0x7fd34c003e60 con 0x7fd3581012b0 2024-03-24T10:46:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:46:10.031 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:45:17.709481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26933":{"gid":26933,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/2955742470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":2955742470},{"type":"v1","addr":"172.21.15.117:6837","nonce":2955742470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24539,24503,24505]},"id":1}]} 2024-03-24T10:46:10.031 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 29 2024-03-24T10:46:10.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.033+0000 7fd35d75c640 1 -- 172.21.15.23:0/3021266768 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd3400a2230 msgr2=0x7fd3400a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.033+0000 7fd35d75c640 1 --2- 172.21.15.23:0/3021266768 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd3400a2230 0x7fd3400a46f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd348009b50 tx=0x7fd348005f90 comp rx=0 tx=0).stop 2024-03-24T10:46:10.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.033+0000 7fd35d75c640 1 -- 172.21.15.23:0/3021266768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581012b0 msgr2=0x7fd3581982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.033+0000 7fd35d75c640 1 --2- 172.21.15.23:0/3021266768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581012b0 0x7fd3581982a0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fd34c009230 tx=0x7fd34c02c650 comp rx=0 tx=0).stop 2024-03-24T10:46:10.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.033+0000 7fd35d75c640 1 -- 172.21.15.23:0/3021266768 shutdown_connections 2024-03-24T10:46:10.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.033+0000 7fd35d75c640 1 --2- 172.21.15.23:0/3021266768 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd3400a2230 0x7fd3400a46f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.033+0000 7fd35d75c640 1 --2- 172.21.15.23:0/3021266768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd358106750 0x7fd35819c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.033+0000 7fd35d75c640 1 --2- 172.21.15.23:0/3021266768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581012b0 0x7fd3581982a0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.033+0000 7fd35d75c640 1 --2- 172.21.15.23:0/3021266768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3581008f0 0x7fd358197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.033+0000 7fd35d75c640 1 -- 172.21.15.23:0/3021266768 >> 172.21.15.23:0/3021266768 conn(0x7fd3580775f0 msgr2=0x7fd3580ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.034+0000 7fd35d75c640 1 -- 172.21.15.23:0/3021266768 shutdown_connections 2024-03-24T10:46:10.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.034+0000 7fd35d75c640 1 -- 172.21.15.23:0/3021266768 wait complete. 2024-03-24T10:46:10.051 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:46:10.122 INFO:tasks.workunit.client.0.smithi023.stdout: 1 579845 11.34 MB/sec execute 147 sec latency 90.405 ms 2024-03-24T10:46:10.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.172+0000 7f39c4c10640 1 -- 172.21.15.23:0/3135779793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39c0107760 msgr2=0x7f39c0107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.172+0000 7f39c4c10640 1 --2- 172.21.15.23:0/3135779793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39c0107760 0x7f39c0107b40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009890 tx=0x7f39b002bb70 comp rx=0 tx=0).stop 2024-03-24T10:46:10.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.172+0000 7f39c4c10640 1 -- 172.21.15.23:0/3135779793 shutdown_connections 2024-03-24T10:46:10.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.172+0000 7f39c4c10640 1 --2- 172.21.15.23:0/3135779793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c00692d0 0x7f39c010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.172+0000 7f39c4c10640 1 --2- 172.21.15.23:0/3135779793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39c0068910 0x7f39c0068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.172+0000 7f39c4c10640 1 --2- 172.21.15.23:0/3135779793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39c0107760 0x7f39c0107b40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.172+0000 7f39c4c10640 1 -- 172.21.15.23:0/3135779793 >> 172.21.15.23:0/3135779793 conn(0x7f39c0077710 msgr2=0x7f39c0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.172+0000 7f39c4c10640 1 -- 172.21.15.23:0/3135779793 shutdown_connections 2024-03-24T10:46:10.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.172+0000 7f39c4c10640 1 -- 172.21.15.23:0/3135779793 wait complete. 2024-03-24T10:46:10.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39c4c10640 1 Processor -- start 2024-03-24T10:46:10.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39c4c10640 1 -- start start 2024-03-24T10:46:10.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39c4c10640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c0068910 0x7f39c0197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39c4c10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39c00692d0 0x7f39c01982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39c4c10640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39c0107760 0x7f39c019c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39c4c10640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f39c019cc30 con 0x7f39c0068910 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39c4c10640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f39c019cda0 con 0x7f39c00692d0 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39c4c10640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f39c019cf60 con 0x7f39c0107760 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39be59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c0068910 0x7f39c0197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39bdd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39c00692d0 0x7f39c01982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39be59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c0068910 0x7f39c0197d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56160/0 (socket says 172.21.15.23:56160) 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39be59c640 1 -- 172.21.15.23:0/1048744617 learned_addr learned my addr 172.21.15.23:0/1048744617 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39bed9d640 1 --2- 172.21.15.23:0/1048744617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39c0107760 0x7f39c019c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39be59c640 1 -- 172.21.15.23:0/1048744617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39c0107760 msgr2=0x7f39c019c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39be59c640 1 --2- 172.21.15.23:0/1048744617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39c0107760 0x7f39c019c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39be59c640 1 -- 172.21.15.23:0/1048744617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39c00692d0 msgr2=0x7f39c01982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39be59c640 1 --2- 172.21.15.23:0/1048744617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39c00692d0 0x7f39c01982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.173+0000 7f39be59c640 1 -- 172.21.15.23:0/1048744617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b8009840 con 0x7f39c0068910 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.174+0000 7f39be59c640 1 --2- 172.21.15.23:0/1048744617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c0068910 0x7f39c0197d70 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009860 tx=0x7f39b0004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.174+0000 7f39af7fe640 1 -- 172.21.15.23:0/1048744617 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b0034950 con 0x7f39c0068910 2024-03-24T10:46:10.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.174+0000 7f39c4c10640 1 -- 172.21.15.23:0/1048744617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b0009490 con 0x7f39c0068910 2024-03-24T10:46:10.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.174+0000 7f39af7fe640 1 -- 172.21.15.23:0/1048744617 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39b002c9f0 con 0x7f39c0068910 2024-03-24T10:46:10.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.174+0000 7f39af7fe640 1 -- 172.21.15.23:0/1048744617 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b0003e70 con 0x7f39c0068910 2024-03-24T10:46:10.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.174+0000 7f39c4c10640 1 -- 172.21.15.23:0/1048744617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c019d410 con 0x7f39c0068910 2024-03-24T10:46:10.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.176+0000 7f39af7fe640 1 -- 172.21.15.23:0/1048744617 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f39b003b040 con 0x7f39c0068910 2024-03-24T10:46:10.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.176+0000 7f39c77ea740 1 Processor -- start 2024-03-24T10:46:10.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.176+0000 7f39c77ea740 1 -- start start 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39c77ea740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb077953f0 0x55fb077957d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39c77ea740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb07795d10 0x55fb0779e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39c77ea740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb0779e8f0 0x55fb077a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39c77ea740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fb07703840 con 0x55fb0779e8f0 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39c77ea740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fb07700d70 con 0x55fb077953f0 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39c77ea740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fb076fc550 con 0x55fb07795d10 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39af7fe640 1 --2- 172.21.15.23:0/1048744617 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f39a80a2230 0x7f39a80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39be59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb07795d10 0x55fb0779e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39be59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb07795d10 0x55fb0779e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:60716/0 (socket says 172.21.15.23:60716) 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39be59c640 1 -- 172.21.15.23:0/339635831 learned_addr learned my addr 172.21.15.23:0/339635831 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39bed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb077953f0 0x55fb077957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39bed9d640 1 --2- 172.21.15.23:0/1048744617 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f39a80a2230 0x7f39a80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39bdd9b640 1 --2- 172.21.15.23:0/339635831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb0779e8f0 0x55fb077a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39af7fe640 1 -- 172.21.15.23:0/1048744617 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f39b00efa90 con 0x7f39c0068910 2024-03-24T10:46:10.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39be59c640 1 -- 172.21.15.23:0/339635831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb077953f0 msgr2=0x55fb077957d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39be59c640 1 --2- 172.21.15.23:0/339635831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb077953f0 0x55fb077957d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39be59c640 1 -- 172.21.15.23:0/339635831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb0779e8f0 msgr2=0x55fb077a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39be59c640 1 --2- 172.21.15.23:0/339635831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb0779e8f0 0x55fb077a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39be59c640 1 -- 172.21.15.23:0/339635831 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb07691de0 con 0x55fb07795d10 2024-03-24T10:46:10.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.177+0000 7f39bdd9b640 1 --2- 172.21.15.23:0/339635831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb0779e8f0 0x55fb077a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:46:10.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.178+0000 7f39be59c640 1 --2- 172.21.15.23:0/339635831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb07795d10 0x55fb0779e3b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f39b0087670 tx=0x7f39b008f6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c8baaac38b700e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.178+0000 7f39bed9d640 1 --2- 172.21.15.23:0/1048744617 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f39a80a2230 0x7f39a80a46f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f39b8009b80 tx=0x7f39b8009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.178+0000 7f39acff9640 1 -- 172.21.15.23:0/339635831 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b002cb60 con 0x55fb07795d10 2024-03-24T10:46:10.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.178+0000 7f39acff9640 1 -- 172.21.15.23:0/339635831 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39b0087e40 con 0x55fb07795d10 2024-03-24T10:46:10.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.178+0000 7f39acff9640 1 -- 172.21.15.23:0/339635831 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b00c0770 con 0x55fb07795d10 2024-03-24T10:46:10.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.179+0000 7f39c77ea740 1 -- 172.21.15.23:0/339635831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb07795d10 msgr2=0x55fb0779e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.179+0000 7f39c77ea740 1 --2- 172.21.15.23:0/339635831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb07795d10 0x55fb0779e3b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f39b0087670 tx=0x7f39b008f6c0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.179+0000 7f39c77ea740 1 -- 172.21.15.23:0/339635831 shutdown_connections 2024-03-24T10:46:10.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.179+0000 7f39c77ea740 1 --2- 172.21.15.23:0/339635831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb0779e8f0 0x55fb077a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.179+0000 7f39c77ea740 1 --2- 172.21.15.23:0/339635831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb07795d10 0x55fb0779e3b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.179+0000 7f39c77ea740 1 --2- 172.21.15.23:0/339635831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb077953f0 0x55fb077957d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.179+0000 7f39c77ea740 1 -- 172.21.15.23:0/339635831 >> 172.21.15.23:0/339635831 conn(0x55fb0778fb20 msgr2=0x55fb0778ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.179+0000 7f39c77ea740 1 -- 172.21.15.23:0/339635831 shutdown_connections 2024-03-24T10:46:10.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.179+0000 7f39c77ea740 1 -- 172.21.15.23:0/339635831 wait complete. 2024-03-24T10:46:10.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.179+0000 7f39c77ea740 1 -- start start 2024-03-24T10:46:10.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.179+0000 7f39c77ea740 1 Processor -- start 2024-03-24T10:46:10.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39c77ea740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb077953f0 0x55fb0779b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39c77ea740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb0779be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39c77ea740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb0779e8f0 0x55fb0779c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39c77ea740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fb073cf430 con 0x55fb077953f0 2024-03-24T10:46:10.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39c77ea740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fb0738be00 con 0x55fb07795d10 2024-03-24T10:46:10.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39c77ea740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fb07681590 con 0x55fb0779e8f0 2024-03-24T10:46:10.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39c77ea740 10 client.0 _ll_register_callbacks cb 0x55fb076f0fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:10.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39be59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb077953f0 0x55fb0779b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39be59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb077953f0 0x55fb0779b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56184/0 (socket says 172.21.15.23:56184) 2024-03-24T10:46:10.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39be59c640 1 -- 172.21.15.23:0/3771255172 learned_addr learned my addr 172.21.15.23:0/3771255172 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39bdd9b640 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb0779be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39bed9d640 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb0779e8f0 0x55fb0779c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39be59c640 1 -- 172.21.15.23:0/3771255172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb0779e8f0 msgr2=0x55fb0779c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39be59c640 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb0779e8f0 0x55fb0779c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39be59c640 1 -- 172.21.15.23:0/3771255172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 msgr2=0x55fb0779be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39be59c640 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb0779be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39be59c640 1 -- 172.21.15.23:0/3771255172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c019d410 con 0x55fb077953f0 2024-03-24T10:46:10.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39bdd9b640 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb0779be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39bed9d640 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb0779e8f0 0x55fb0779c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39be59c640 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb077953f0 0x55fb0779b8d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f39b00b8f60 tx=0x7f39b002cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d07363e34df05f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39a5ffb640 1 -- 172.21.15.23:0/3771255172 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b00bc590 con 0x55fb077953f0 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.180+0000 7f39a5ffb640 1 -- 172.21.15.23:0/3771255172 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f39b00bcc90 con 0x55fb077953f0 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.181+0000 7f39c77ea740 1 -- 172.21.15.23:0/3771255172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fb073fb7e0 con 0x55fb077953f0 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.181+0000 7f39a5ffb640 1 -- 172.21.15.23:0/3771255172 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f39b00c4d30 con 0x55fb077953f0 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.181+0000 7f39c77ea740 10 client.17268 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.181+0000 7f39c77ea740 1 -- 172.21.15.23:0/3771255172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fb076fabd0 con 0x55fb077953f0 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.181+0000 7f39a5ffb640 1 -- 172.21.15.23:0/3771255172 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fb073fb7e0 con 0x55fb077953f0 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.181+0000 7f39a5ffb640 1 -- 172.21.15.23:0/3771255172 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f39b00bb9e0 con 0x55fb077953f0 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.181+0000 7f39c77ea740 10 client.17268 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.181+0000 7f39c77ea740 10 client.17268 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.181+0000 7f39c77ea740 20 client.17268 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:10.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.181+0000 7f39c77ea740 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55fb077ae1c0 0x55fb077b0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.181+0000 7f39c77ea740 4 client.17268 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.181+0000 7f39c77ea740 1 -- 172.21.15.23:0/3771255172 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb07438720 con 0x55fb077ae1c0 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.181+0000 7f39bdd9b640 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55fb077ae1c0 0x55fb077b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.182+0000 7f39a5ffb640 10 client.17268 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.182+0000 7f39bdd9b640 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55fb077ae1c0 0x55fb077b0600 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9df4006884c00545 server_cookie=69f1f80ef1e31641 in_seq=0 out_seq=0 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.182+0000 7f39a5ffb640 1 -- 172.21.15.23:0/3771255172 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fb07438720 con 0x55fb077ae1c0 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.182+0000 7f39a5ffb640 10 client.17268 handle_command_reply: tid=0 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.183+0000 7f39c77ea740 1 client.17268 shutdown 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.183+0000 7f39c77ea740 1 -- 172.21.15.23:0/3771255172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb077953f0 msgr2=0x55fb0779b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.183+0000 7f39c77ea740 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb077953f0 0x55fb0779b8d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f39b00b8f60 tx=0x7f39b002cc50 comp rx=0 tx=0).stop 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.183+0000 7f39c77ea740 1 -- 172.21.15.23:0/3771255172 shutdown_connections 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.183+0000 7f39c77ea740 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb0779e8f0 0x55fb0779c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.183+0000 7f39c77ea740 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55fb077ae1c0 0x55fb077b0600 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.183+0000 7f39c77ea740 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb0779be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.183+0000 7f39c77ea740 1 --2- 172.21.15.23:0/3771255172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb077953f0 0x55fb0779b8d0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.183+0000 7f39a5ffb640 0 client.17268 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:46:10.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.183+0000 7f39c77ea740 1 -- 172.21.15.23:0/3771255172 >> 172.21.15.23:0/3771255172 conn(0x55fb0778fb20 msgr2=0x55fb07791870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.183+0000 7f39c77ea740 1 -- 172.21.15.23:0/3771255172 shutdown_connections 2024-03-24T10:46:10.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.183+0000 7f39c77ea740 1 -- 172.21.15.23:0/3771255172 wait complete. 2024-03-24T10:46:10.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.183+0000 7f39c77ea740 20 client.17268 trim_cache size 0 max 16384 2024-03-24T10:46:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39c77ea740 1 Processor -- start 2024-03-24T10:46:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39c77ea740 1 -- start start 2024-03-24T10:46:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39c77ea740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb077953f0 0x55fb0779e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39c77ea740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb077a9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39c77ea740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb0779e8f0 0x55fb0779aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39c77ea740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fb0778b250 con 0x55fb0779e8f0 2024-03-24T10:46:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39c77ea740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fb07794510 con 0x55fb07795d10 2024-03-24T10:46:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39c77ea740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fb077b0c80 con 0x55fb077953f0 2024-03-24T10:46:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39bed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb0779e8f0 0x55fb0779aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39bdd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb077a9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39be59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb077953f0 0x55fb0779e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39bdd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb077a9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:41832/0 (socket says 172.21.15.23:41832) 2024-03-24T10:46:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39bdd9b640 1 -- 172.21.15.23:0/2114553319 learned_addr learned my addr 172.21.15.23:0/2114553319 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39be59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb077953f0 0x55fb0779e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:60724/0 (socket says 172.21.15.23:60724) 2024-03-24T10:46:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.197+0000 7f39bdd9b640 1 -- 172.21.15.23:0/2114553319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb077953f0 msgr2=0x55fb0779e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.198+0000 7f39bdd9b640 1 --2- 172.21.15.23:0/2114553319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb077953f0 0x55fb0779e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.198+0000 7f39bdd9b640 1 -- 172.21.15.23:0/2114553319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb0779e8f0 msgr2=0x55fb0779aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.198+0000 7f39bdd9b640 1 --2- 172.21.15.23:0/2114553319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb0779e8f0 0x55fb0779aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.198+0000 7f39bdd9b640 1 -- 172.21.15.23:0/2114553319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb07696c80 con 0x55fb07795d10 2024-03-24T10:46:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.198+0000 7f39bdd9b640 1 --2- 172.21.15.23:0/2114553319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb077a9eb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x55fb07790cb0 tx=0x7f39b4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dff2ca47f9b4c704 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.198+0000 7f39a7fff640 1 -- 172.21.15.23:0/2114553319 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b4018040 con 0x55fb07795d10 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.198+0000 7f39a7fff640 1 -- 172.21.15.23:0/2114553319 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39b4024070 con 0x55fb07795d10 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.198+0000 7f39a7fff640 1 -- 172.21.15.23:0/2114553319 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b4020340 con 0x55fb07795d10 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.199+0000 7f39c77ea740 1 -- 172.21.15.23:0/2114553319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 msgr2=0x55fb077a9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.199+0000 7f39c77ea740 1 --2- 172.21.15.23:0/2114553319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb077a9eb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x55fb07790cb0 tx=0x7f39b4009260 comp rx=0 tx=0).stop 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.199+0000 7f39c77ea740 1 -- 172.21.15.23:0/2114553319 shutdown_connections 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.199+0000 7f39c77ea740 1 --2- 172.21.15.23:0/2114553319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb0779e8f0 0x55fb0779aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.199+0000 7f39c77ea740 1 --2- 172.21.15.23:0/2114553319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb077a9eb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.199+0000 7f39c77ea740 1 --2- 172.21.15.23:0/2114553319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb077953f0 0x55fb0779e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.199+0000 7f39c77ea740 1 -- 172.21.15.23:0/2114553319 >> 172.21.15.23:0/2114553319 conn(0x55fb0778fb20 msgr2=0x55fb0778e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.199+0000 7f39c77ea740 1 -- 172.21.15.23:0/2114553319 shutdown_connections 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.199+0000 7f39c77ea740 1 -- 172.21.15.23:0/2114553319 wait complete. 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.199+0000 7f39c77ea740 1 -- start start 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.199+0000 7f39c77ea740 1 Processor -- start 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39c77ea740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb077953f0 0x55fb077a6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39c77ea740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb07797f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39c77ea740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb0779e8f0 0x55fb07798490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39c77ea740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55fb077a6b60 con 0x55fb077953f0 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39c77ea740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fb077a6cd0 con 0x55fb07795d10 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39c77ea740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fb077989d0 con 0x55fb0779e8f0 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39c77ea740 10 client.0 _ll_register_callbacks cb 0x7f39b0034ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39bdd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb07797f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39be59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb077953f0 0x55fb077a6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39bdd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb07797f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:41840/0 (socket says 172.21.15.23:41840) 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39bdd9b640 1 -- 172.21.15.23:0/3462462247 learned_addr learned my addr 172.21.15.23:0/3462462247 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39bed9d640 1 --2- 172.21.15.23:0/3462462247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb0779e8f0 0x55fb07798490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39bdd9b640 1 -- 172.21.15.23:0/3462462247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb0779e8f0 msgr2=0x55fb07798490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39bdd9b640 1 --2- 172.21.15.23:0/3462462247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb0779e8f0 0x55fb07798490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39bdd9b640 1 -- 172.21.15.23:0/3462462247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb077953f0 msgr2=0x55fb077a6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39bdd9b640 1 --2- 172.21.15.23:0/3462462247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb077953f0 0x55fb077a6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39bdd9b640 1 -- 172.21.15.23:0/3462462247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b40146f0 con 0x55fb07795d10 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.200+0000 7f39be59c640 1 --2- 172.21.15.23:0/3462462247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb077953f0 0x55fb077a6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39bdd9b640 1 --2- 172.21.15.23:0/3462462247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb07797f50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f39b4012540 tx=0x7f39b40140d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0cab88b970c217b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39a5ffb640 1 -- 172.21.15.23:0/3462462247 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b4018030 con 0x55fb07795d10 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39c77ea740 1 -- 172.21.15.23:0/3462462247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39b0087e40 con 0x55fb07795d10 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39a5ffb640 1 -- 172.21.15.23:0/3462462247 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f39b4004050 con 0x55fb07795d10 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39a5ffb640 1 -- 172.21.15.23:0/3462462247 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f39b4004210 con 0x55fb07795d10 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39c77ea740 10 client.27161 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39c77ea740 1 -- 172.21.15.23:0/3462462247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fb076bc920 con 0x55fb07795d10 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39a5ffb640 1 -- 172.21.15.23:0/3462462247 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39b0087e40 con 0x55fb07795d10 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39a5ffb640 1 -- 172.21.15.23:0/3462462247 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f39b4012a40 con 0x55fb07795d10 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39c77ea740 10 client.27161 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39c77ea740 10 client.27161 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39c77ea740 20 client.27161 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39c77ea740 1 --2- 172.21.15.23:0/3462462247 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55fb078147c0 0x55fb07816bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39c77ea740 4 client.27161 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:46:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.201+0000 7f39c77ea740 1 -- 172.21.15.23:0/3462462247 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fb07487890 con 0x55fb078147c0 2024-03-24T10:46:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.202+0000 7f39be59c640 1 --2- 172.21.15.23:0/3462462247 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55fb078147c0 0x55fb07816bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.202+0000 7f39be59c640 1 --2- 172.21.15.23:0/3462462247 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55fb078147c0 0x55fb07816bb0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d380f25e9f4fd441 server_cookie=555709e1b538df77 in_seq=0 out_seq=0 2024-03-24T10:46:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.202+0000 7f39a5ffb640 10 client.27161 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:46:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.203+0000 7f39a5ffb640 1 -- 172.21.15.23:0/3462462247 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fb07487890 con 0x55fb078147c0 2024-03-24T10:46:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.203+0000 7f39a5ffb640 10 client.27161 handle_command_reply: tid=0 2024-03-24T10:46:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.203+0000 7f39c77ea740 1 client.27161 shutdown 2024-03-24T10:46:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.203+0000 7f39c77ea740 1 -- 172.21.15.23:0/3462462247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 msgr2=0x55fb07797f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.203+0000 7f39c77ea740 1 --2- 172.21.15.23:0/3462462247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb07797f50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f39b4012540 tx=0x7f39b40140d0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.203+0000 7f39c77ea740 1 -- 172.21.15.23:0/3462462247 shutdown_connections 2024-03-24T10:46:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.203+0000 7f39c77ea740 1 --2- 172.21.15.23:0/3462462247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fb0779e8f0 0x55fb07798490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.203+0000 7f39c77ea740 1 --2- 172.21.15.23:0/3462462247 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55fb078147c0 0x55fb07816bb0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.203+0000 7f39c77ea740 1 --2- 172.21.15.23:0/3462462247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fb07795d10 0x55fb07797f50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.203+0000 7f39c77ea740 1 --2- 172.21.15.23:0/3462462247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fb077953f0 0x55fb077a6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.203+0000 7f39a5ffb640 0 client.27161 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:46:10.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.203+0000 7f39c77ea740 1 -- 172.21.15.23:0/3462462247 >> 172.21.15.23:0/3462462247 conn(0x55fb0778fb20 msgr2=0x55fb0778e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.203+0000 7f39c77ea740 1 -- 172.21.15.23:0/3462462247 shutdown_connections 2024-03-24T10:46:10.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.203+0000 7f39c77ea740 1 -- 172.21.15.23:0/3462462247 wait complete. 2024-03-24T10:46:10.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.204+0000 7f39c77ea740 20 client.27161 trim_cache size 0 max 16384 2024-03-24T10:46:10.208 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:46:10.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.204+0000 7f39c4c10640 1 -- 172.21.15.23:0/1048744617 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f39a80a2230 msgr2=0x7f39a80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.204+0000 7f39c4c10640 1 --2- 172.21.15.23:0/1048744617 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f39a80a2230 0x7f39a80a46f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f39b8009b80 tx=0x7f39b8009500 comp rx=0 tx=0).stop 2024-03-24T10:46:10.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.204+0000 7f39c4c10640 1 -- 172.21.15.23:0/1048744617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c0068910 msgr2=0x7f39c0197d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.204+0000 7f39c4c10640 1 --2- 172.21.15.23:0/1048744617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c0068910 0x7f39c0197d70 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009860 tx=0x7f39b0004530 comp rx=0 tx=0).stop 2024-03-24T10:46:10.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.204+0000 7f39c4c10640 1 -- 172.21.15.23:0/1048744617 shutdown_connections 2024-03-24T10:46:10.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.204+0000 7f39c4c10640 1 --2- 172.21.15.23:0/1048744617 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f39a80a2230 0x7f39a80a46f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.204+0000 7f39c4c10640 1 --2- 172.21.15.23:0/1048744617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39c0107760 0x7f39c019c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.204+0000 7f39c4c10640 1 --2- 172.21.15.23:0/1048744617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39c00692d0 0x7f39c01982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.204+0000 7f39c4c10640 1 --2- 172.21.15.23:0/1048744617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c0068910 0x7f39c0197d70 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.204+0000 7f39c4c10640 1 -- 172.21.15.23:0/1048744617 >> 172.21.15.23:0/1048744617 conn(0x7f39c0077710 msgr2=0x7f39c00fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.204+0000 7f39c4c10640 1 -- 172.21.15.23:0/1048744617 shutdown_connections 2024-03-24T10:46:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.204+0000 7f39c4c10640 1 -- 172.21.15.23:0/1048744617 wait complete. 2024-03-24T10:46:10.215 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:46:10.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.328+0000 7f25429d0640 1 -- 172.21.15.23:0/813824746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f253c107760 msgr2=0x7f253c107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.328+0000 7f25429d0640 1 --2- 172.21.15.23:0/813824746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f253c107760 0x7f253c107b40 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f252c009930 tx=0x7f252c02bc20 comp rx=0 tx=0).stop 2024-03-24T10:46:10.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.328+0000 7f25429d0640 1 -- 172.21.15.23:0/813824746 shutdown_connections 2024-03-24T10:46:10.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.328+0000 7f25429d0640 1 --2- 172.21.15.23:0/813824746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f253c0692d0 0x7f253c10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.328+0000 7f25429d0640 1 --2- 172.21.15.23:0/813824746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f253c068910 0x7f253c068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.328+0000 7f25429d0640 1 --2- 172.21.15.23:0/813824746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f253c107760 0x7f253c107b40 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.328+0000 7f25429d0640 1 -- 172.21.15.23:0/813824746 >> 172.21.15.23:0/813824746 conn(0x7f253c077710 msgr2=0x7f253c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.328+0000 7f25429d0640 1 -- 172.21.15.23:0/813824746 shutdown_connections 2024-03-24T10:46:10.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.328+0000 7f25429d0640 1 -- 172.21.15.23:0/813824746 wait complete. 2024-03-24T10:46:10.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.329+0000 7f25429d0640 1 Processor -- start 2024-03-24T10:46:10.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.329+0000 7f25429d0640 1 -- start start 2024-03-24T10:46:10.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.329+0000 7f25429d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f253c068910 0x7f253c197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.329+0000 7f25429d0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f253c0692d0 0x7f253c1982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.329+0000 7f253bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f253c068910 0x7f253c197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.329+0000 7f253b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f253c0692d0 0x7f253c1982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.329+0000 7f253bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f253c068910 0x7f253c197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:41866/0 (socket says 172.21.15.23:41866) 2024-03-24T10:46:10.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.329+0000 7f253bfff640 1 -- 172.21.15.23:0/1408783111 learned_addr learned my addr 172.21.15.23:0/1408783111 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.329+0000 7f25429d0640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f253c107760 0x7f253c19c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.329+0000 7f25429d0640 1 -- 172.21.15.23:0/1408783111 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f253c19cc00 con 0x7f253c0692d0 2024-03-24T10:46:10.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.329+0000 7f25429d0640 1 -- 172.21.15.23:0/1408783111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f253c19cd70 con 0x7f253c068910 2024-03-24T10:46:10.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.329+0000 7f25429d0640 1 -- 172.21.15.23:0/1408783111 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f253c19cf30 con 0x7f253c107760 2024-03-24T10:46:10.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.329+0000 7f2540f6d640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f253c107760 0x7f253c19c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.330+0000 7f253b7fe640 1 -- 172.21.15.23:0/1408783111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f253c107760 msgr2=0x7f253c19c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.330+0000 7f253b7fe640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f253c107760 0x7f253c19c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.330+0000 7f253b7fe640 1 -- 172.21.15.23:0/1408783111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f253c068910 msgr2=0x7f253c197d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.330+0000 7f253b7fe640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f253c068910 0x7f253c197d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.330+0000 7f253b7fe640 1 -- 172.21.15.23:0/1408783111 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252c009560 con 0x7f253c0692d0 2024-03-24T10:46:10.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.330+0000 7f253bfff640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f253c068910 0x7f253c197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:10.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.330+0000 7f253b7fe640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f253c0692d0 0x7f253c1982d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f253000e9f0 tx=0x7f253000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.330+0000 7f25397fa640 1 -- 172.21.15.23:0/1408783111 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253000ccc0 con 0x7f253c0692d0 2024-03-24T10:46:10.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.330+0000 7f25429d0640 1 -- 172.21.15.23:0/1408783111 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253c19d210 con 0x7f253c0692d0 2024-03-24T10:46:10.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.330+0000 7f25397fa640 1 -- 172.21.15.23:0/1408783111 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f253000ce20 con 0x7f253c0692d0 2024-03-24T10:46:10.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.330+0000 7f25397fa640 1 -- 172.21.15.23:0/1408783111 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2530004a00 con 0x7f253c0692d0 2024-03-24T10:46:10.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.330+0000 7f25429d0640 1 -- 172.21.15.23:0/1408783111 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253c1a4a90 con 0x7f253c0692d0 2024-03-24T10:46:10.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.332+0000 7f25397fa640 1 -- 172.21.15.23:0/1408783111 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f25300050b0 con 0x7f253c0692d0 2024-03-24T10:46:10.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f25455b5740 1 Processor -- start 2024-03-24T10:46:10.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f25397fa640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f25240a22b0 0x7f25240a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f25455b5740 1 -- start start 2024-03-24T10:46:10.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f25397fa640 1 -- 172.21.15.23:0/1408783111 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f2530014070 con 0x7f253c0692d0 2024-03-24T10:46:10.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f2540f6d640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f25240a22b0 0x7f25240a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f25455b5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba217d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f25455b5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba21d10 0x56408ba2a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f25455b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba2a8f0 0x56408ba2cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f25455b5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56408b98f840 con 0x56408ba213f0 2024-03-24T10:46:10.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f25455b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56408b98cd70 con 0x56408ba2a8f0 2024-03-24T10:46:10.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f25455b5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56408b988550 con 0x56408ba21d10 2024-03-24T10:46:10.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f253bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba217d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f253bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba217d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56240/0 (socket says 172.21.15.23:56240) 2024-03-24T10:46:10.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f253bfff640 1 -- 172.21.15.23:0/3612753913 learned_addr learned my addr 172.21.15.23:0/3612753913 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f2540f6d640 1 --2- 172.21.15.23:0/3612753913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba2a8f0 0x56408ba2cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.333+0000 7f253b7fe640 1 --2- 172.21.15.23:0/3612753913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba21d10 0x56408ba2a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.334+0000 7f253bfff640 1 -- 172.21.15.23:0/3612753913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba21d10 msgr2=0x56408ba2a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.334+0000 7f253bfff640 1 --2- 172.21.15.23:0/3612753913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba21d10 0x56408ba2a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.334+0000 7f253bfff640 1 -- 172.21.15.23:0/3612753913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba2a8f0 msgr2=0x56408ba2cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.334+0000 7f253bfff640 1 --2- 172.21.15.23:0/3612753913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba2a8f0 0x56408ba2cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.334+0000 7f253bfff640 1 -- 172.21.15.23:0/3612753913 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56408b91dde0 con 0x56408ba213f0 2024-03-24T10:46:10.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.334+0000 7f253b7fe640 1 --2- 172.21.15.23:0/3612753913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba21d10 0x56408ba2a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:10.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.334+0000 7f2540f6d640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f25240a22b0 0x7f25240a4770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2534005fd0 tx=0x7f2534011bc0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.334+0000 7f253bfff640 1 --2- 172.21.15.23:0/3612753913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba217d0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f252c009430 tx=0x7f252c02ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3bfa72207f081fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.334+0000 7f25227fc640 1 -- 172.21.15.23:0/3612753913 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c0046c0 con 0x56408ba213f0 2024-03-24T10:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.334+0000 7f25227fc640 1 -- 172.21.15.23:0/3612753913 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f252c004510 con 0x56408ba213f0 2024-03-24T10:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.334+0000 7f25227fc640 1 -- 172.21.15.23:0/3612753913 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c0053a0 con 0x56408ba213f0 2024-03-24T10:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.335+0000 7f25455b5740 1 -- 172.21.15.23:0/3612753913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 msgr2=0x56408ba217d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.335+0000 7f25455b5740 1 --2- 172.21.15.23:0/3612753913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba217d0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f252c009430 tx=0x7f252c02ccc0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.335+0000 7f25455b5740 1 -- 172.21.15.23:0/3612753913 shutdown_connections 2024-03-24T10:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.335+0000 7f25455b5740 1 --2- 172.21.15.23:0/3612753913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba2a8f0 0x56408ba2cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.335+0000 7f25455b5740 1 --2- 172.21.15.23:0/3612753913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba21d10 0x56408ba2a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.335+0000 7f25455b5740 1 --2- 172.21.15.23:0/3612753913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba217d0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.335+0000 7f25455b5740 1 -- 172.21.15.23:0/3612753913 >> 172.21.15.23:0/3612753913 conn(0x56408ba1bb20 msgr2=0x56408ba1bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.335+0000 7f25455b5740 1 -- 172.21.15.23:0/3612753913 shutdown_connections 2024-03-24T10:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.335+0000 7f25455b5740 1 -- 172.21.15.23:0/3612753913 wait complete. 2024-03-24T10:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.335+0000 7f25455b5740 1 -- start start 2024-03-24T10:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.335+0000 7f25455b5740 1 Processor -- start 2024-03-24T10:46:10.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f25455b5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba27970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f25455b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba21d10 0x56408ba27eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f25455b5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 0x56408ba283f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f25455b5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56408b65b430 con 0x56408ba213f0 2024-03-24T10:46:10.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f25455b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56408b617e00 con 0x56408ba21d10 2024-03-24T10:46:10.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f25455b5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56408b90d590 con 0x56408ba2a8f0 2024-03-24T10:46:10.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f25455b5740 10 client.0 _ll_register_callbacks cb 0x56408b97cfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:10.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f253bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba27970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f253bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba27970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56254/0 (socket says 172.21.15.23:56254) 2024-03-24T10:46:10.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f253bfff640 1 -- 172.21.15.23:0/2431109331 learned_addr learned my addr 172.21.15.23:0/2431109331 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f2540f6d640 1 --2- 172.21.15.23:0/2431109331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 0x56408ba283f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f253b7fe640 1 --2- 172.21.15.23:0/2431109331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba21d10 0x56408ba27eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f253bfff640 1 -- 172.21.15.23:0/2431109331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 msgr2=0x56408ba283f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f253bfff640 1 --2- 172.21.15.23:0/2431109331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 0x56408ba283f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f253bfff640 1 -- 172.21.15.23:0/2431109331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba21d10 msgr2=0x56408ba27eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f253bfff640 1 --2- 172.21.15.23:0/2431109331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba21d10 0x56408ba27eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f253bfff640 1 -- 172.21.15.23:0/2431109331 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253c10ee70 con 0x56408ba213f0 2024-03-24T10:46:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f253b7fe640 1 --2- 172.21.15.23:0/2431109331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba21d10 0x56408ba27eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:46:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f253bfff640 1 --2- 172.21.15.23:0/2431109331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba27970 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f252c036040 tx=0x7f252c004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e468058b4291d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f251b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f251b7fe640 1 -- 172.21.15.23:0/2431109331 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c0055f0 con 0x56408ba213f0 2024-03-24T10:46:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f251b7fe640 1 -- 172.21.15.23:0/2431109331 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f252c04d070 con 0x56408ba213f0 2024-03-24T10:46:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.336+0000 7f25455b5740 1 -- 172.21.15.23:0/2431109331 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56408b6877e0 con 0x56408ba213f0 2024-03-24T10:46:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.337+0000 7f251b7fe640 1 -- 172.21.15.23:0/2431109331 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f252c03f440 con 0x56408ba213f0 2024-03-24T10:46:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.337+0000 7f25455b5740 10 client.17286 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.337+0000 7f25455b5740 1 -- 172.21.15.23:0/2431109331 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56408b986bd0 con 0x56408ba213f0 2024-03-24T10:46:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.337+0000 7f251b7fe640 1 -- 172.21.15.23:0/2431109331 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56408b6877e0 con 0x56408ba213f0 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.337+0000 7f251b7fe640 1 -- 172.21.15.23:0/2431109331 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f252c03c050 con 0x56408ba213f0 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.337+0000 7f25455b5740 10 client.17286 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.337+0000 7f25455b5740 10 client.17286 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.337+0000 7f25455b5740 20 client.17286 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.337+0000 7f25455b5740 1 --2- 172.21.15.23:0/2431109331 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56408ba3a5b0 0x56408ba3c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.337+0000 7f25455b5740 4 client.17286 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.337+0000 7f25455b5740 1 -- 172.21.15.23:0/2431109331 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56408b6c4720 con 0x56408ba3a5b0 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.337+0000 7f253b7fe640 1 --2- 172.21.15.23:0/2431109331 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56408ba3a5b0 0x56408ba3c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.338+0000 7f253b7fe640 1 --2- 172.21.15.23:0/2431109331 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56408ba3a5b0 0x56408ba3c9c0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ef8dc35f49f81ff7 server_cookie=d8515599a653fae5 in_seq=0 out_seq=0 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.338+0000 7f251b7fe640 10 client.17286 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.338+0000 7f251b7fe640 1 -- 172.21.15.23:0/2431109331 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56408b6c4720 con 0x56408ba3a5b0 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.338+0000 7f251b7fe640 10 client.17286 handle_command_reply: tid=0 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.339+0000 7f25455b5740 1 client.17286 shutdown 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.339+0000 7f25455b5740 1 -- 172.21.15.23:0/2431109331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 msgr2=0x56408ba27970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.339+0000 7f25455b5740 1 --2- 172.21.15.23:0/2431109331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba27970 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f252c036040 tx=0x7f252c004f00 comp rx=0 tx=0).stop 2024-03-24T10:46:10.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.339+0000 7f25455b5740 1 -- 172.21.15.23:0/2431109331 shutdown_connections 2024-03-24T10:46:10.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.339+0000 7f25455b5740 1 --2- 172.21.15.23:0/2431109331 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56408ba3a5b0 0x56408ba3c9c0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.339+0000 7f25455b5740 1 --2- 172.21.15.23:0/2431109331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 0x56408ba283f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.339+0000 7f25455b5740 1 --2- 172.21.15.23:0/2431109331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba21d10 0x56408ba27eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.339+0000 7f25455b5740 1 --2- 172.21.15.23:0/2431109331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba27970 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.339+0000 7f25455b5740 1 -- 172.21.15.23:0/2431109331 >> 172.21.15.23:0/2431109331 conn(0x56408ba1bb20 msgr2=0x56408ba1d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.339+0000 7f251b7fe640 0 client.17286 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:46:10.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.339+0000 7f25455b5740 1 -- 172.21.15.23:0/2431109331 shutdown_connections 2024-03-24T10:46:10.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.339+0000 7f25455b5740 1 -- 172.21.15.23:0/2431109331 wait complete. 2024-03-24T10:46:10.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.339+0000 7f25455b5740 20 client.17286 trim_cache size 0 max 16384 2024-03-24T10:46:10.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.354+0000 7f25455b5740 1 Processor -- start 2024-03-24T10:46:10.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.354+0000 7f25455b5740 1 -- start start 2024-03-24T10:46:10.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.354+0000 7f25455b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba213f0 0x56408ba2a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.354+0000 7f25455b5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba21d10 0x56408ba36120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.354+0000 7f25455b5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 0x56408ba1d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.354+0000 7f25455b5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56408b6877e0 con 0x56408ba21d10 2024-03-24T10:46:10.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.354+0000 7f25455b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56408ba1cb20 con 0x56408ba213f0 2024-03-24T10:46:10.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.354+0000 7f25455b5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56408ba1b650 con 0x56408ba2a8f0 2024-03-24T10:46:10.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.354+0000 7f253b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba21d10 0x56408ba36120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.354+0000 7f253b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba21d10 0x56408ba36120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56256/0 (socket says 172.21.15.23:56256) 2024-03-24T10:46:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.354+0000 7f253bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba213f0 0x56408ba2a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.354+0000 7f253b7fe640 1 -- 172.21.15.23:0/1686963209 learned_addr learned my addr 172.21.15.23:0/1686963209 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.354+0000 7f2540f6d640 1 --2- 172.21.15.23:0/1686963209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 0x56408ba1d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.355+0000 7f2540f6d640 1 -- 172.21.15.23:0/1686963209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba213f0 msgr2=0x56408ba2a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.355+0000 7f2540f6d640 1 --2- 172.21.15.23:0/1686963209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba213f0 0x56408ba2a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.355+0000 7f2540f6d640 1 -- 172.21.15.23:0/1686963209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba21d10 msgr2=0x56408ba36120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.355+0000 7f2540f6d640 1 --2- 172.21.15.23:0/1686963209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba21d10 0x56408ba36120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.355+0000 7f2540f6d640 1 -- 172.21.15.23:0/1686963209 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56408b922c80 con 0x56408ba2a8f0 2024-03-24T10:46:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.355+0000 7f2540f6d640 1 --2- 172.21.15.23:0/1686963209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 0x56408ba1d800 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f253401d000 tx=0x7f253401f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ee505f54041d1d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.355+0000 7f2521ffb640 1 -- 172.21.15.23:0/1686963209 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534016050 con 0x56408ba2a8f0 2024-03-24T10:46:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.355+0000 7f2521ffb640 1 -- 172.21.15.23:0/1686963209 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2534020e70 con 0x56408ba2a8f0 2024-03-24T10:46:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.355+0000 7f2521ffb640 1 -- 172.21.15.23:0/1686963209 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25340056e0 con 0x56408ba2a8f0 2024-03-24T10:46:10.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.356+0000 7f25455b5740 1 -- 172.21.15.23:0/1686963209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 msgr2=0x56408ba1d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.356+0000 7f25455b5740 1 --2- 172.21.15.23:0/1686963209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 0x56408ba1d800 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f253401d000 tx=0x7f253401f000 comp rx=0 tx=0).stop 2024-03-24T10:46:10.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.356+0000 7f25455b5740 1 -- 172.21.15.23:0/1686963209 shutdown_connections 2024-03-24T10:46:10.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.356+0000 7f25455b5740 1 --2- 172.21.15.23:0/1686963209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 0x56408ba1d800 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.356+0000 7f25455b5740 1 --2- 172.21.15.23:0/1686963209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba21d10 0x56408ba36120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.356+0000 7f25455b5740 1 --2- 172.21.15.23:0/1686963209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba213f0 0x56408ba2a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.356+0000 7f25455b5740 1 -- 172.21.15.23:0/1686963209 >> 172.21.15.23:0/1686963209 conn(0x56408ba1bb20 msgr2=0x56408ba1bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.356+0000 7f25455b5740 1 -- 172.21.15.23:0/1686963209 shutdown_connections 2024-03-24T10:46:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.356+0000 7f25455b5740 1 -- 172.21.15.23:0/1686963209 wait complete. 2024-03-24T10:46:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.356+0000 7f25455b5740 1 -- start start 2024-03-24T10:46:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.356+0000 7f25455b5740 1 Processor -- start 2024-03-24T10:46:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.356+0000 7f25455b5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba32750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.356+0000 7f25455b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba21d10 0x56408ba2f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f25455b5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 0x56408ba2fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f25455b5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56408ba301a0 con 0x56408ba213f0 2024-03-24T10:46:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f25455b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56408ba30310 con 0x56408ba21d10 2024-03-24T10:46:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f25455b5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56408ba304d0 con 0x56408ba2a8f0 2024-03-24T10:46:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f25455b5740 10 client.0 _ll_register_callbacks cb 0x7f2534011e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f253bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba32750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f253bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba32750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56260/0 (socket says 172.21.15.23:56260) 2024-03-24T10:46:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f253bfff640 1 -- 172.21.15.23:0/1319431978 learned_addr learned my addr 172.21.15.23:0/1319431978 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f253b7fe640 1 --2- 172.21.15.23:0/1319431978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba21d10 0x56408ba2f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f2540f6d640 1 --2- 172.21.15.23:0/1319431978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 0x56408ba2fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f253bfff640 1 -- 172.21.15.23:0/1319431978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 msgr2=0x56408ba2fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f253bfff640 1 --2- 172.21.15.23:0/1319431978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 0x56408ba2fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f253bfff640 1 -- 172.21.15.23:0/1319431978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba21d10 msgr2=0x56408ba2f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f253bfff640 1 --2- 172.21.15.23:0/1319431978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba21d10 0x56408ba2f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f253bfff640 1 -- 172.21.15.23:0/1319431978 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253400fa40 con 0x56408ba213f0 2024-03-24T10:46:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f253b7fe640 1 --2- 172.21.15.23:0/1319431978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba21d10 0x56408ba2f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f253bfff640 1 --2- 172.21.15.23:0/1319431978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba32750 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f252c05a920 tx=0x7f252c005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c3c4174df129d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f251b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f251b7fe640 1 -- 172.21.15.23:0/1319431978 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c0055f0 con 0x56408ba213f0 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f251b7fe640 1 -- 172.21.15.23:0/1319431978 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f252c05ae40 con 0x56408ba213f0 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f25455b5740 1 -- 172.21.15.23:0/1319431978 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f252c004510 con 0x56408ba213f0 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.357+0000 7f251b7fe640 1 -- 172.21.15.23:0/1319431978 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f252c02cc40 con 0x56408ba213f0 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.358+0000 7f25455b5740 10 client.17292 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.358+0000 7f25455b5740 1 -- 172.21.15.23:0/1319431978 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56408b948920 con 0x56408ba213f0 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.358+0000 7f251b7fe640 1 -- 172.21.15.23:0/1319431978 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f252c004510 con 0x56408ba213f0 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.358+0000 7f251b7fe640 1 -- 172.21.15.23:0/1319431978 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f252c03c040 con 0x56408ba213f0 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.358+0000 7f25455b5740 10 client.17292 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.358+0000 7f25455b5740 10 client.17292 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.358+0000 7f25455b5740 20 client.17292 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.358+0000 7f25455b5740 1 --2- 172.21.15.23:0/1319431978 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56408baa1a40 0x56408baa3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.358+0000 7f25455b5740 4 client.17292 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.358+0000 7f25455b5740 1 -- 172.21.15.23:0/1319431978 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56408b713890 con 0x56408baa1a40 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.358+0000 7f253b7fe640 1 --2- 172.21.15.23:0/1319431978 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56408baa1a40 0x56408baa3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.358+0000 7f253b7fe640 1 --2- 172.21.15.23:0/1319431978 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56408baa1a40 0x56408baa3e60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ce048abc6051f3cd server_cookie=fa94d05c95a76368 in_seq=0 out_seq=0 2024-03-24T10:46:10.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.358+0000 7f251b7fe640 10 client.17292 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:46:10.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f251b7fe640 1 -- 172.21.15.23:0/1319431978 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56408b713890 con 0x56408baa1a40 2024-03-24T10:46:10.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f251b7fe640 10 client.17292 handle_command_reply: tid=0 2024-03-24T10:46:10.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25455b5740 1 client.17292 shutdown 2024-03-24T10:46:10.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25455b5740 1 -- 172.21.15.23:0/1319431978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 msgr2=0x56408ba32750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25455b5740 1 --2- 172.21.15.23:0/1319431978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba32750 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f252c05a920 tx=0x7f252c005b50 comp rx=0 tx=0).stop 2024-03-24T10:46:10.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25455b5740 1 -- 172.21.15.23:0/1319431978 shutdown_connections 2024-03-24T10:46:10.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25455b5740 1 --2- 172.21.15.23:0/1319431978 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56408baa1a40 0x56408baa3e60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25455b5740 1 --2- 172.21.15.23:0/1319431978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56408ba2a8f0 0x56408ba2fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25455b5740 1 --2- 172.21.15.23:0/1319431978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56408ba21d10 0x56408ba2f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25455b5740 1 --2- 172.21.15.23:0/1319431978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408ba213f0 0x56408ba32750 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f251b7fe640 0 client.17292 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:46:10.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25455b5740 1 -- 172.21.15.23:0/1319431978 >> 172.21.15.23:0/1319431978 conn(0x56408ba1bb20 msgr2=0x56408ba1acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25455b5740 1 -- 172.21.15.23:0/1319431978 shutdown_connections 2024-03-24T10:46:10.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25455b5740 1 -- 172.21.15.23:0/1319431978 wait complete. 2024-03-24T10:46:10.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25455b5740 20 client.17292 trim_cache size 0 max 16384 2024-03-24T10:46:10.369 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:46:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25429d0640 1 -- 172.21.15.23:0/1408783111 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f25240a22b0 msgr2=0x7f25240a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25429d0640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f25240a22b0 0x7f25240a4770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2534005fd0 tx=0x7f2534011bc0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25429d0640 1 -- 172.21.15.23:0/1408783111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f253c0692d0 msgr2=0x7f253c1982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.359+0000 7f25429d0640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f253c0692d0 0x7f253c1982d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f253000e9f0 tx=0x7f253000eeb0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.360+0000 7f25429d0640 1 -- 172.21.15.23:0/1408783111 shutdown_connections 2024-03-24T10:46:10.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.360+0000 7f25429d0640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f25240a22b0 0x7f25240a4770 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.360+0000 7f25429d0640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f253c107760 0x7f253c19c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.360+0000 7f25429d0640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f253c0692d0 0x7f253c1982d0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.360+0000 7f25429d0640 1 --2- 172.21.15.23:0/1408783111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f253c068910 0x7f253c197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.360+0000 7f25429d0640 1 -- 172.21.15.23:0/1408783111 >> 172.21.15.23:0/1408783111 conn(0x7f253c077710 msgr2=0x7f253c0fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.360+0000 7f25429d0640 1 -- 172.21.15.23:0/1408783111 shutdown_connections 2024-03-24T10:46:10.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.360+0000 7f25429d0640 1 -- 172.21.15.23:0/1408783111 wait complete. 2024-03-24T10:46:10.372 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:46:10.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 -- 172.21.15.23:0/4212654193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8730106660 msgr2=0x7f8730106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 --2- 172.21.15.23:0/4212654193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8730106660 0x7f8730106a40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f872c009930 tx=0x7f872c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:46:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 -- 172.21.15.23:0/4212654193 shutdown_connections 2024-03-24T10:46:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 --2- 172.21.15.23:0/4212654193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8730100f60 0x7f873010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 --2- 172.21.15.23:0/4212654193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87301005a0 0x7f8730100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 --2- 172.21.15.23:0/4212654193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8730106660 0x7f8730106a40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 -- 172.21.15.23:0/4212654193 >> 172.21.15.23:0/4212654193 conn(0x7f8730077710 msgr2=0x7f8730077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 -- 172.21.15.23:0/4212654193 shutdown_connections 2024-03-24T10:46:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 -- 172.21.15.23:0/4212654193 wait complete. 2024-03-24T10:46:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 Processor -- start 2024-03-24T10:46:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 -- start start 2024-03-24T10:46:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87301005a0 0x7f873019da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8730100f60 0x7f873019dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8730106660 0x7f8730197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f8730198350 con 0x7f8730106660 2024-03-24T10:46:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f87301984c0 con 0x7f8730100f60 2024-03-24T10:46:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.481+0000 7f873846f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8730198680 con 0x7f87301005a0 2024-03-24T10:46:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f873620b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87301005a0 0x7f873019da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f873620b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87301005a0 0x7f873019da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:60808/0 (socket says 172.21.15.23:60808) 2024-03-24T10:46:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f8736a0c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8730106660 0x7f8730197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f873620b640 1 -- 172.21.15.23:0/3716776282 learned_addr learned my addr 172.21.15.23:0/3716776282 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f8735a0a640 1 --2- 172.21.15.23:0/3716776282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8730100f60 0x7f873019dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f8736a0c640 1 -- 172.21.15.23:0/3716776282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87301005a0 msgr2=0x7f873019da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f8736a0c640 1 --2- 172.21.15.23:0/3716776282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87301005a0 0x7f873019da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f8736a0c640 1 -- 172.21.15.23:0/3716776282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8730100f60 msgr2=0x7f873019dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f8736a0c640 1 --2- 172.21.15.23:0/3716776282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8730100f60 0x7f873019dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f8736a0c640 1 -- 172.21.15.23:0/3716776282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872c009560 con 0x7f8730106660 2024-03-24T10:46:10.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f873620b640 1 --2- 172.21.15.23:0/3716776282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87301005a0 0x7f873019da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:10.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f8736a0c640 1 --2- 172.21.15.23:0/3716776282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8730106660 0x7f8730197bc0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8728009fd0 tx=0x7f872800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f87277fe640 1 -- 172.21.15.23:0/3716776282 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872800ce20 con 0x7f8730106660 2024-03-24T10:46:10.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f873846f640 1 -- 172.21.15.23:0/3716776282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8730198960 con 0x7f8730106660 2024-03-24T10:46:10.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f87277fe640 1 -- 172.21.15.23:0/3716776282 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8728005020 con 0x7f8730106660 2024-03-24T10:46:10.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f87277fe640 1 -- 172.21.15.23:0/3716776282 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8728010690 con 0x7f8730106660 2024-03-24T10:46:10.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.482+0000 7f873846f640 1 -- 172.21.15.23:0/3716776282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87301a49e0 con 0x7f8730106660 2024-03-24T10:46:10.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.485+0000 7f87277fe640 1 -- 172.21.15.23:0/3716776282 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f8728005500 con 0x7f8730106660 2024-03-24T10:46:10.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.486+0000 7f873b054740 1 Processor -- start 2024-03-24T10:46:10.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.486+0000 7f87277fe640 1 --2- 172.21.15.23:0/3716776282 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f871c0a2230 0x7f871c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.486+0000 7f8735a0a640 1 --2- 172.21.15.23:0/3716776282 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f871c0a2230 0x7f871c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.486+0000 7f87277fe640 1 -- 172.21.15.23:0/3716776282 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f8728014070 con 0x7f8730106660 2024-03-24T10:46:10.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f873b054740 1 -- start start 2024-03-24T10:46:10.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f873b054740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf43f0 0x555f2eaf47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f873b054740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eaf4d10 0x555f2eafd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f873b054740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eafd8f0 0x555f2eaffce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f873b054740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555f2ea62840 con 0x555f2eaf43f0 2024-03-24T10:46:10.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f873b054740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555f2ea5fd70 con 0x555f2eaf4d10 2024-03-24T10:46:10.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f873b054740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555f2ea5b550 con 0x555f2eafd8f0 2024-03-24T10:46:10.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f8735a0a640 1 --2- 172.21.15.23:0/3716776282 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f871c0a2230 0x7f871c0a46f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8730199300 tx=0x7f8720005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f873620b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf43f0 0x555f2eaf47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f873620b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf43f0 0x555f2eaf47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56292/0 (socket says 172.21.15.23:56292) 2024-03-24T10:46:10.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f873620b640 1 -- 172.21.15.23:0/182639169 learned_addr learned my addr 172.21.15.23:0/182639169 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f8736a0c640 1 --2- 172.21.15.23:0/182639169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eafd8f0 0x555f2eaffce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f8735a0a640 1 --2- 172.21.15.23:0/182639169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eaf4d10 0x555f2eafd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f873620b640 1 -- 172.21.15.23:0/182639169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eafd8f0 msgr2=0x555f2eaffce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f873620b640 1 --2- 172.21.15.23:0/182639169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eafd8f0 0x555f2eaffce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.487+0000 7f873620b640 1 -- 172.21.15.23:0/182639169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eaf4d10 msgr2=0x555f2eafd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.488+0000 7f873620b640 1 --2- 172.21.15.23:0/182639169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eaf4d10 0x555f2eafd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.488+0000 7f873620b640 1 -- 172.21.15.23:0/182639169 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f2e9f0de0 con 0x555f2eaf43f0 2024-03-24T10:46:10.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.488+0000 7f8736a0c640 1 --2- 172.21.15.23:0/182639169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eafd8f0 0x555f2eaffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:10.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.488+0000 7f8735a0a640 1 --2- 172.21.15.23:0/182639169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eaf4d10 0x555f2eafd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:10.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.488+0000 7f873620b640 1 --2- 172.21.15.23:0/182639169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf43f0 0x555f2eaf47d0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f872c009430 tx=0x7f872c02c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554f3ddbb590d3b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.488+0000 7f8724bfe640 1 -- 172.21.15.23:0/182639169 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c02c7e0 con 0x555f2eaf43f0 2024-03-24T10:46:10.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.488+0000 7f8724bfe640 1 -- 172.21.15.23:0/182639169 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f872c004510 con 0x555f2eaf43f0 2024-03-24T10:46:10.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.488+0000 7f8724bfe640 1 -- 172.21.15.23:0/182639169 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c005480 con 0x555f2eaf43f0 2024-03-24T10:46:10.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.489+0000 7f873b054740 1 -- 172.21.15.23:0/182639169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf43f0 msgr2=0x555f2eaf47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.489+0000 7f873b054740 1 --2- 172.21.15.23:0/182639169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf43f0 0x555f2eaf47d0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f872c009430 tx=0x7f872c02c6e0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.489+0000 7f873b054740 1 -- 172.21.15.23:0/182639169 shutdown_connections 2024-03-24T10:46:10.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.489+0000 7f873b054740 1 --2- 172.21.15.23:0/182639169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eafd8f0 0x555f2eaffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.489+0000 7f873b054740 1 --2- 172.21.15.23:0/182639169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eaf4d10 0x555f2eafd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.489+0000 7f873b054740 1 --2- 172.21.15.23:0/182639169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf43f0 0x555f2eaf47d0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.489+0000 7f873b054740 1 -- 172.21.15.23:0/182639169 >> 172.21.15.23:0/182639169 conn(0x555f2eaeeb20 msgr2=0x555f2eaeef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.489+0000 7f873b054740 1 -- 172.21.15.23:0/182639169 shutdown_connections 2024-03-24T10:46:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.489+0000 7f873b054740 1 -- 172.21.15.23:0/182639169 wait complete. 2024-03-24T10:46:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.489+0000 7f873b054740 1 -- start start 2024-03-24T10:46:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.489+0000 7f873b054740 1 Processor -- start 2024-03-24T10:46:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873b054740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eaf43f0 0x555f2eafa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873b054740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eafae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873b054740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eafd8f0 0x555f2eafb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873b054740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555f2e72e430 con 0x555f2eaf4d10 2024-03-24T10:46:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873b054740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555f2e6eae00 con 0x555f2eafd8f0 2024-03-24T10:46:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873b054740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555f2e9e0590 con 0x555f2eaf43f0 2024-03-24T10:46:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873b054740 10 client.0 _ll_register_callbacks cb 0x555f2ea4ffc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f8735a0a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eafae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873620b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eaf43f0 0x555f2eafa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f8735a0a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eafae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56294/0 (socket says 172.21.15.23:56294) 2024-03-24T10:46:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f8735a0a640 1 -- 172.21.15.23:0/2823530703 learned_addr learned my addr 172.21.15.23:0/2823530703 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f8736a0c640 1 --2- 172.21.15.23:0/2823530703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eafd8f0 0x555f2eafb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873620b640 1 -- 172.21.15.23:0/2823530703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eafd8f0 msgr2=0x555f2eafb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873620b640 1 --2- 172.21.15.23:0/2823530703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eafd8f0 0x555f2eafb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873620b640 1 -- 172.21.15.23:0/2823530703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 msgr2=0x555f2eafae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873620b640 1 --2- 172.21.15.23:0/2823530703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eafae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873620b640 1 -- 172.21.15.23:0/2823530703 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f873010dd20 con 0x555f2eaf43f0 2024-03-24T10:46:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f8735a0a640 1 --2- 172.21.15.23:0/2823530703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eafae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873620b640 1 --2- 172.21.15.23:0/2823530703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eaf43f0 0x555f2eafa940 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f872c009900 tx=0x7f872c004fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=934c143e825c3833 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f8719ffb640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:46:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f8719ffb640 1 -- 172.21.15.23:0/2823530703 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c03f440 con 0x555f2eaf43f0 2024-03-24T10:46:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f8719ffb640 1 -- 172.21.15.23:0/2823530703 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f872c04d070 con 0x555f2eaf43f0 2024-03-24T10:46:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.490+0000 7f873b054740 1 -- 172.21.15.23:0/2823530703 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555f2e75a7e0 con 0x555f2eaf43f0 2024-03-24T10:46:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.491+0000 7f8719ffb640 1 -- 172.21.15.23:0/2823530703 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f872c03f5a0 con 0x555f2eaf43f0 2024-03-24T10:46:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.491+0000 7f873b054740 10 client.27163 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.491+0000 7f873b054740 1 -- 172.21.15.23:0/2823530703 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555f2ea59bd0 con 0x555f2eaf43f0 2024-03-24T10:46:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.491+0000 7f8719ffb640 1 -- 172.21.15.23:0/2823530703 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555f2e75a7e0 con 0x555f2eaf43f0 2024-03-24T10:46:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.491+0000 7f8719ffb640 1 -- 172.21.15.23:0/2823530703 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f872c03c050 con 0x555f2eaf43f0 2024-03-24T10:46:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.491+0000 7f873b054740 10 client.27163 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.491+0000 7f873b054740 10 client.27163 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.491+0000 7f873b054740 20 client.27163 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.491+0000 7f873b054740 1 --2- 172.21.15.23:0/2823530703 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555f2eb11a30 0x555f2eb13e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.491+0000 7f873b054740 4 client.27163 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.491+0000 7f873b054740 1 -- 172.21.15.23:0/2823530703 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f2e797720 con 0x555f2eb11a30 2024-03-24T10:46:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.491+0000 7f8735a0a640 1 --2- 172.21.15.23:0/2823530703 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555f2eb11a30 0x555f2eb13e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.492+0000 7f8719ffb640 10 client.27163 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.492+0000 7f8735a0a640 1 --2- 172.21.15.23:0/2823530703 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555f2eb11a30 0x555f2eb13e90 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2bf99b4170c3e29 server_cookie=ccfc84daf8fddb0f in_seq=0 out_seq=0 2024-03-24T10:46:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f8719ffb640 1 -- 172.21.15.23:0/2823530703 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555f2e797720 con 0x555f2eb11a30 2024-03-24T10:46:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f8719ffb640 10 client.27163 handle_command_reply: tid=0 2024-03-24T10:46:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f873b054740 1 client.27163 shutdown 2024-03-24T10:46:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f873b054740 1 -- 172.21.15.23:0/2823530703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eaf43f0 msgr2=0x555f2eafa940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f873b054740 1 --2- 172.21.15.23:0/2823530703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eaf43f0 0x555f2eafa940 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f872c009900 tx=0x7f872c004fe0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f873b054740 1 -- 172.21.15.23:0/2823530703 shutdown_connections 2024-03-24T10:46:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f873b054740 1 --2- 172.21.15.23:0/2823530703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eafd8f0 0x555f2eafb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f873b054740 1 --2- 172.21.15.23:0/2823530703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eafae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f873b054740 1 --2- 172.21.15.23:0/2823530703 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555f2eb11a30 0x555f2eb13e90 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f873b054740 1 --2- 172.21.15.23:0/2823530703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eaf43f0 0x555f2eafa940 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f8719ffb640 0 client.27163 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:10.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f873b054740 1 -- 172.21.15.23:0/2823530703 >> 172.21.15.23:0/2823530703 conn(0x555f2eaeeb20 msgr2=0x555f2eaf0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f873b054740 1 -- 172.21.15.23:0/2823530703 shutdown_connections 2024-03-24T10:46:10.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f873b054740 1 -- 172.21.15.23:0/2823530703 wait complete. 2024-03-24T10:46:10.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.493+0000 7f873b054740 20 client.27163 trim_cache size 0 max 16384 2024-03-24T10:46:10.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.508+0000 7f873b054740 1 Processor -- start 2024-03-24T10:46:10.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.508+0000 7f873b054740 1 -- start start 2024-03-24T10:46:10.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.508+0000 7f873b054740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eaf43f0 0x555f2eafd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.508+0000 7f873b054740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eb0d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.508+0000 7f873b054740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eafd8f0 0x555f2eaedcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.508+0000 7f873b054740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555f2eaf37c0 con 0x555f2eaf4d10 2024-03-24T10:46:10.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.508+0000 7f873b054740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555f2eaefb20 con 0x555f2eaf43f0 2024-03-24T10:46:10.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.508+0000 7f873b054740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555f2eb14510 con 0x555f2eafd8f0 2024-03-24T10:46:10.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.508+0000 7f8735a0a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eb0d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.508+0000 7f873620b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eaf43f0 0x555f2eafd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.509+0000 7f8735a0a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eb0d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56300/0 (socket says 172.21.15.23:56300) 2024-03-24T10:46:10.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.509+0000 7f8736a0c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eafd8f0 0x555f2eaedcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.509+0000 7f8735a0a640 1 -- 172.21.15.23:0/1990736116 learned_addr learned my addr 172.21.15.23:0/1990736116 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.509+0000 7f8735a0a640 1 -- 172.21.15.23:0/1990736116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eafd8f0 msgr2=0x555f2eaedcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.509+0000 7f8735a0a640 1 --2- 172.21.15.23:0/1990736116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eafd8f0 0x555f2eaedcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.509+0000 7f8735a0a640 1 -- 172.21.15.23:0/1990736116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eaf43f0 msgr2=0x555f2eafd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.509+0000 7f8735a0a640 1 --2- 172.21.15.23:0/1990736116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eaf43f0 0x555f2eafd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.509+0000 7f8735a0a640 1 -- 172.21.15.23:0/1990736116 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f2e9f5c80 con 0x555f2eaf4d10 2024-03-24T10:46:10.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.509+0000 7f873620b640 1 --2- 172.21.15.23:0/1990736116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eaf43f0 0x555f2eafd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:10.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.509+0000 7f8736a0c640 1 --2- 172.21.15.23:0/1990736116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eafd8f0 0x555f2eaedcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:10.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.509+0000 7f8735a0a640 1 --2- 172.21.15.23:0/1990736116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eb0d530 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f8720009710 tx=0x7f8720014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f695ef4e80d02743 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.509+0000 7f871bfff640 1 -- 172.21.15.23:0/1990736116 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8720024040 con 0x555f2eaf4d10 2024-03-24T10:46:10.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.509+0000 7f871bfff640 1 -- 172.21.15.23:0/1990736116 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f872002ad90 con 0x555f2eaf4d10 2024-03-24T10:46:10.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.509+0000 7f871bfff640 1 -- 172.21.15.23:0/1990736116 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872000e610 con 0x555f2eaf4d10 2024-03-24T10:46:10.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.510+0000 7f873b054740 1 -- 172.21.15.23:0/1990736116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 msgr2=0x555f2eb0d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.510+0000 7f873b054740 1 --2- 172.21.15.23:0/1990736116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eb0d530 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f8720009710 tx=0x7f8720014000 comp rx=0 tx=0).stop 2024-03-24T10:46:10.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.510+0000 7f873b054740 1 -- 172.21.15.23:0/1990736116 shutdown_connections 2024-03-24T10:46:10.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.510+0000 7f873b054740 1 --2- 172.21.15.23:0/1990736116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eafd8f0 0x555f2eaedcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.510+0000 7f873b054740 1 --2- 172.21.15.23:0/1990736116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eb0d530 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.510+0000 7f873b054740 1 --2- 172.21.15.23:0/1990736116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eaf43f0 0x555f2eafd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.510+0000 7f873b054740 1 -- 172.21.15.23:0/1990736116 >> 172.21.15.23:0/1990736116 conn(0x555f2eaeeb20 msgr2=0x555f2eaeef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.510+0000 7f873b054740 1 -- 172.21.15.23:0/1990736116 shutdown_connections 2024-03-24T10:46:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.510+0000 7f873b054740 1 -- 172.21.15.23:0/1990736116 wait complete. 2024-03-24T10:46:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.510+0000 7f873b054740 1 -- start start 2024-03-24T10:46:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.510+0000 7f873b054740 1 Processor -- start 2024-03-24T10:46:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f873b054740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eaf43f0 0x555f2eb09b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f873b054740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eb06b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f873b054740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eafd8f0 0x555f2eb07070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f873b054740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555f2eb0a0a0 con 0x555f2eaf4d10 2024-03-24T10:46:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f873b054740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555f2eb0a210 con 0x555f2eafd8f0 2024-03-24T10:46:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f873b054740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555f2eb0a380 con 0x555f2eaf43f0 2024-03-24T10:46:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f873b054740 10 client.0 _ll_register_callbacks cb 0x7f8720009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f8735a0a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eb06b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f8735a0a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eb06b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56310/0 (socket says 172.21.15.23:56310) 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f8735a0a640 1 -- 172.21.15.23:0/3773117034 learned_addr learned my addr 172.21.15.23:0/3773117034 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f873620b640 1 --2- 172.21.15.23:0/3773117034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eaf43f0 0x555f2eb09b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f8736a0c640 1 --2- 172.21.15.23:0/3773117034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eafd8f0 0x555f2eb07070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f8735a0a640 1 -- 172.21.15.23:0/3773117034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eaf43f0 msgr2=0x555f2eb09b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f8735a0a640 1 --2- 172.21.15.23:0/3773117034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eaf43f0 0x555f2eb09b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f8735a0a640 1 -- 172.21.15.23:0/3773117034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eafd8f0 msgr2=0x555f2eb07070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f8735a0a640 1 --2- 172.21.15.23:0/3773117034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eafd8f0 0x555f2eb07070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f8735a0a640 1 -- 172.21.15.23:0/3773117034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f2ea59bd0 con 0x555f2eaf4d10 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f8735a0a640 1 --2- 172.21.15.23:0/3773117034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eb06b30 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f8720003940 tx=0x7f872002af80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e68279ff667c5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f8719ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f8719ffb640 1 -- 172.21.15.23:0/3773117034 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8720024050 con 0x555f2eaf4d10 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f8719ffb640 1 -- 172.21.15.23:0/3773117034 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f872002a620 con 0x555f2eaf4d10 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.511+0000 7f873b054740 1 -- 172.21.15.23:0/3773117034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f872c004510 con 0x555f2eaf4d10 2024-03-24T10:46:10.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.512+0000 7f8719ffb640 1 -- 172.21.15.23:0/3773117034 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f872002a7e0 con 0x555f2eaf4d10 2024-03-24T10:46:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.512+0000 7f873b054740 10 client.17310 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.512+0000 7f873b054740 1 -- 172.21.15.23:0/3773117034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x7f8730198960 con 0x555f2eaf4d10 2024-03-24T10:46:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.512+0000 7f8719ffb640 1 -- 172.21.15.23:0/3773117034 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f872c004510 con 0x555f2eaf4d10 2024-03-24T10:46:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.512+0000 7f8719ffb640 1 -- 172.21.15.23:0/3773117034 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f8720039020 con 0x555f2eaf4d10 2024-03-24T10:46:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.512+0000 7f873b054740 10 client.17310 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.512+0000 7f873b054740 10 client.17310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.512+0000 7f873b054740 20 client.17310 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.512+0000 7f873b054740 1 --2- 172.21.15.23:0/3773117034 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555f2eb78ea0 0x555f2eb7b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.512+0000 7f873b054740 4 client.17310 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:46:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.512+0000 7f873b054740 1 -- 172.21.15.23:0/3773117034 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555f2e7e6890 con 0x555f2eb78ea0 2024-03-24T10:46:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.512+0000 7f873620b640 1 --2- 172.21.15.23:0/3773117034 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555f2eb78ea0 0x555f2eb7b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f8719ffb640 10 client.17310 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f873620b640 1 --2- 172.21.15.23:0/3773117034 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555f2eb78ea0 0x555f2eb7b2c0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5661b422f7ab6a server_cookie=8256ab5986bea59c in_seq=0 out_seq=0 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f8719ffb640 1 -- 172.21.15.23:0/3773117034 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555f2e7e6890 con 0x555f2eb78ea0 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f8719ffb640 10 client.17310 handle_command_reply: tid=0 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f873b054740 1 client.17310 shutdown 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f873b054740 1 -- 172.21.15.23:0/3773117034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 msgr2=0x555f2eb06b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f873b054740 1 --2- 172.21.15.23:0/3773117034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eb06b30 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f8720003940 tx=0x7f872002af80 comp rx=0 tx=0).stop 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f873b054740 1 -- 172.21.15.23:0/3773117034 shutdown_connections 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f873b054740 1 --2- 172.21.15.23:0/3773117034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f2eafd8f0 0x555f2eb07070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f873b054740 1 --2- 172.21.15.23:0/3773117034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555f2eaf4d10 0x555f2eb06b30 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f873b054740 1 --2- 172.21.15.23:0/3773117034 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555f2eb78ea0 0x555f2eb7b2c0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f873b054740 1 --2- 172.21.15.23:0/3773117034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f2eaf43f0 0x555f2eb09b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f8719ffb640 0 client.17310 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f873b054740 1 -- 172.21.15.23:0/3773117034 >> 172.21.15.23:0/3773117034 conn(0x555f2eaeeb20 msgr2=0x555f2eafb170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f873b054740 1 -- 172.21.15.23:0/3773117034 shutdown_connections 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f873b054740 1 -- 172.21.15.23:0/3773117034 wait complete. 2024-03-24T10:46:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.513+0000 7f873b054740 20 client.17310 trim_cache size 0 max 16384 2024-03-24T10:46:10.520 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:46:10.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.514+0000 7f873846f640 1 -- 172.21.15.23:0/3716776282 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f871c0a2230 msgr2=0x7f871c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.514+0000 7f873846f640 1 --2- 172.21.15.23:0/3716776282 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f871c0a2230 0x7f871c0a46f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8730199300 tx=0x7f8720005f90 comp rx=0 tx=0).stop 2024-03-24T10:46:10.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.514+0000 7f873846f640 1 -- 172.21.15.23:0/3716776282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8730106660 msgr2=0x7f8730197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.514+0000 7f873846f640 1 --2- 172.21.15.23:0/3716776282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8730106660 0x7f8730197bc0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8728009fd0 tx=0x7f872800ef10 comp rx=0 tx=0).stop 2024-03-24T10:46:10.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.514+0000 7f873846f640 1 -- 172.21.15.23:0/3716776282 shutdown_connections 2024-03-24T10:46:10.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.514+0000 7f873846f640 1 --2- 172.21.15.23:0/3716776282 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f871c0a2230 0x7f871c0a46f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.514+0000 7f873846f640 1 --2- 172.21.15.23:0/3716776282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8730106660 0x7f8730197bc0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.514+0000 7f873846f640 1 --2- 172.21.15.23:0/3716776282 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8730100f60 0x7f873019dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.514+0000 7f873846f640 1 --2- 172.21.15.23:0/3716776282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87301005a0 0x7f873019da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.514+0000 7f873846f640 1 -- 172.21.15.23:0/3716776282 >> 172.21.15.23:0/3716776282 conn(0x7f8730077710 msgr2=0x7f87300ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.514+0000 7f873846f640 1 -- 172.21.15.23:0/3716776282 shutdown_connections 2024-03-24T10:46:10.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:10.514+0000 7f873846f640 1 -- 172.21.15.23:0/3716776282 wait complete. 2024-03-24T10:46:10.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.544+0000 7fbb905a1640 1 -- 172.21.15.23:0/646703295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb88068d90 msgr2=0x7fbb880691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.544+0000 7fbb905a1640 1 --2- 172.21.15.23:0/646703295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb88068d90 0x7fbb880691f0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009900 tx=0x7fbb7802bc30 comp rx=0 tx=0).stop 2024-03-24T10:46:10.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.544+0000 7fbb905a1640 1 -- 172.21.15.23:0/646703295 shutdown_connections 2024-03-24T10:46:10.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.544+0000 7fbb905a1640 1 --2- 172.21.15.23:0/646703295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb88069730 0x7fbb8810e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.544+0000 7fbb905a1640 1 --2- 172.21.15.23:0/646703295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb88068d90 0x7fbb880691f0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.544+0000 7fbb905a1640 1 --2- 172.21.15.23:0/646703295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb88107800 0x7fbb88107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.544+0000 7fbb905a1640 1 -- 172.21.15.23:0/646703295 >> 172.21.15.23:0/646703295 conn(0x7fbb88077460 msgr2=0x7fbb88077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.544+0000 7fbb905a1640 1 -- 172.21.15.23:0/646703295 shutdown_connections 2024-03-24T10:46:10.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.544+0000 7fbb905a1640 1 -- 172.21.15.23:0/646703295 wait complete. 2024-03-24T10:46:10.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.544+0000 7fbb905a1640 1 Processor -- start 2024-03-24T10:46:10.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.544+0000 7fbb905a1640 1 -- start start 2024-03-24T10:46:10.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb905a1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb88068d90 0x7fbb88197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb905a1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb88069730 0x7fbb881982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb905a1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb88107800 0x7fbb8819c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb905a1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8819cc20 con 0x7fbb88068d90 2024-03-24T10:46:10.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb905a1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8819cd90 con 0x7fbb88107800 2024-03-24T10:46:10.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb905a1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8819cf50 con 0x7fbb88069730 2024-03-24T10:46:10.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb8e33d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb88068d90 0x7fbb88197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb8e33d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb88068d90 0x7fbb88197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56334/0 (socket says 172.21.15.23:56334) 2024-03-24T10:46:10.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb8e33d640 1 -- 172.21.15.23:0/1314795493 learned_addr learned my addr 172.21.15.23:0/1314795493 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb8eb3e640 1 --2- 172.21.15.23:0/1314795493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb88107800 0x7fbb8819c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb8db3c640 1 --2- 172.21.15.23:0/1314795493 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb88069730 0x7fbb881982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb8e33d640 1 -- 172.21.15.23:0/1314795493 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb88069730 msgr2=0x7fbb881982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb8e33d640 1 --2- 172.21.15.23:0/1314795493 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb88069730 0x7fbb881982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb8e33d640 1 -- 172.21.15.23:0/1314795493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb88107800 msgr2=0x7fbb8819c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb8e33d640 1 --2- 172.21.15.23:0/1314795493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb88107800 0x7fbb8819c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb8e33d640 1 -- 172.21.15.23:0/1314795493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb78009560 con 0x7fbb88068d90 2024-03-24T10:46:10.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb8e33d640 1 --2- 172.21.15.23:0/1314795493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb88068d90 0x7fbb88197d60 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbb8400e9f0 tx=0x7fbb8400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb7f7fe640 1 -- 172.21.15.23:0/1314795493 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb84009700 con 0x7fbb88068d90 2024-03-24T10:46:10.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb905a1640 1 -- 172.21.15.23:0/1314795493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb8819d230 con 0x7fbb88068d90 2024-03-24T10:46:10.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb7f7fe640 1 -- 172.21.15.23:0/1314795493 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb8400cd80 con 0x7fbb88068d90 2024-03-24T10:46:10.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb7f7fe640 1 -- 172.21.15.23:0/1314795493 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb84018920 con 0x7fbb88068d90 2024-03-24T10:46:10.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.545+0000 7fbb905a1640 1 -- 172.21.15.23:0/1314795493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb881a4ab0 con 0x7fbb88068d90 2024-03-24T10:46:10.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.548+0000 7fbb7f7fe640 1 -- 172.21.15.23:0/1314795493 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fbb84018a80 con 0x7fbb88068d90 2024-03-24T10:46:10.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.548+0000 7fbb93186740 1 Processor -- start 2024-03-24T10:46:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb93186740 1 -- start start 2024-03-24T10:46:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb93186740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0a5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb93186740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a59e0 0x5621ad0a5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb93186740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0b7370 0x5621ad0b9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb93186740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5621ad0601b0 con 0x5621ad0a59e0 2024-03-24T10:46:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb93186740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5621ad012f80 con 0x5621ad0b7370 2024-03-24T10:46:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb93186740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5621ad0104b0 con 0x5621ad0a5030 2024-03-24T10:46:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb8eb3e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb7f7fe640 1 --2- 172.21.15.23:0/1314795493 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbb740a22d0 0x7fbb740a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb8eb3e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0a5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:60850/0 (socket says 172.21.15.23:60850) 2024-03-24T10:46:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb8db3c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0b7370 0x5621ad0b9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb8eb3e640 1 -- 172.21.15.23:0/4218043084 learned_addr learned my addr 172.21.15.23:0/4218043084 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb8e33d640 1 --2- 172.21.15.23:0/4218043084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a59e0 0x5621ad0a5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb8eb3e640 1 --2- 172.21.15.23:0/1314795493 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbb740a22d0 0x7fbb740a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb8eb3e640 1 -- 172.21.15.23:0/4218043084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0b7370 msgr2=0x5621ad0b9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb8eb3e640 1 --2- 172.21.15.23:0/4218043084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0b7370 0x5621ad0b9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb8eb3e640 1 -- 172.21.15.23:0/4218043084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a59e0 msgr2=0x5621ad0a5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb8eb3e640 1 --2- 172.21.15.23:0/4218043084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a59e0 0x5621ad0a5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb8eb3e640 1 -- 172.21.15.23:0/4218043084 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621acfa1810 con 0x5621ad0a5030 2024-03-24T10:46:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.549+0000 7fbb7f7fe640 1 -- 172.21.15.23:0/1314795493 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fbb84014070 con 0x7fbb88068d90 2024-03-24T10:46:10.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.550+0000 7fbb8eb3e640 1 --2- 172.21.15.23:0/4218043084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0a5410 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbb80013b30 tx=0x7fbb8000f3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12661d5b928aa379 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.550+0000 7fbb8eb3e640 1 --2- 172.21.15.23:0/1314795493 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbb740a22d0 0x7fbb740a4790 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbb80013000 tx=0x7fbb8001e000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.550+0000 7fbb7cbfe640 1 -- 172.21.15.23:0/4218043084 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb80010630 con 0x5621ad0a5030 2024-03-24T10:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.550+0000 7fbb7cbfe640 1 -- 172.21.15.23:0/4218043084 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb8000fe70 con 0x5621ad0a5030 2024-03-24T10:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.550+0000 7fbb7cbfe640 1 -- 172.21.15.23:0/4218043084 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8000ea90 con 0x5621ad0a5030 2024-03-24T10:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.551+0000 7fbb93186740 1 -- 172.21.15.23:0/4218043084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 msgr2=0x5621ad0a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.551+0000 7fbb93186740 1 --2- 172.21.15.23:0/4218043084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0a5410 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbb80013b30 tx=0x7fbb8000f3c0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.551+0000 7fbb93186740 1 -- 172.21.15.23:0/4218043084 shutdown_connections 2024-03-24T10:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.551+0000 7fbb93186740 1 --2- 172.21.15.23:0/4218043084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0b7370 0x5621ad0b9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.551+0000 7fbb93186740 1 --2- 172.21.15.23:0/4218043084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a59e0 0x5621ad0a5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.551+0000 7fbb93186740 1 --2- 172.21.15.23:0/4218043084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0a5410 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.551+0000 7fbb93186740 1 -- 172.21.15.23:0/4218043084 >> 172.21.15.23:0/4218043084 conn(0x5621ad09ff00 msgr2=0x5621ad0a02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.551+0000 7fbb93186740 1 -- 172.21.15.23:0/4218043084 shutdown_connections 2024-03-24T10:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.551+0000 7fbb93186740 1 -- 172.21.15.23:0/4218043084 wait complete. 2024-03-24T10:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.551+0000 7fbb93186740 1 -- start start 2024-03-24T10:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.551+0000 7fbb93186740 1 Processor -- start 2024-03-24T10:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb93186740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0a5030 0x5621ad0a77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb93186740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a59e0 0x5621ad0a7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb93186740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0b7370 0x5621ad0a8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb93186740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5621ad00bc90 con 0x5621ad0a59e0 2024-03-24T10:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb93186740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5621acfb0c50 con 0x5621ad0a5030 2024-03-24T10:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb93186740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5621accdf250 con 0x5621ad0b7370 2024-03-24T10:46:10.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb8db3c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a59e0 0x5621ad0a7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb93186740 10 client.0 _ll_register_callbacks cb 0x5621acfff290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:10.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb8db3c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a59e0 0x5621ad0a7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56366/0 (socket says 172.21.15.23:56366) 2024-03-24T10:46:10.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb8db3c640 1 -- 172.21.15.23:0/968703540 learned_addr learned my addr 172.21.15.23:0/968703540 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb8e33d640 1 --2- 172.21.15.23:0/968703540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0a5030 0x5621ad0a77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb8eb3e640 1 --2- 172.21.15.23:0/968703540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0b7370 0x5621ad0a8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb8db3c640 1 -- 172.21.15.23:0/968703540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0b7370 msgr2=0x5621ad0a8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb8db3c640 1 --2- 172.21.15.23:0/968703540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0b7370 0x5621ad0a8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb8db3c640 1 -- 172.21.15.23:0/968703540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0a5030 msgr2=0x5621ad0a77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb8db3c640 1 --2- 172.21.15.23:0/968703540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0a5030 0x5621ad0a77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb8db3c640 1 -- 172.21.15.23:0/968703540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb800136e0 con 0x5621ad0a59e0 2024-03-24T10:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb8e33d640 1 --2- 172.21.15.23:0/968703540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0a5030 0x5621ad0a77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb8db3c640 1 --2- 172.21.15.23:0/968703540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a59e0 0x5621ad0a7d20 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbb7802c170 tx=0x7fbb7802ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d095325aa3fe104 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb71ffb640 1 -- 172.21.15.23:0/968703540 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7802cc40 con 0x5621ad0a59e0 2024-03-24T10:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb71ffb640 1 -- 172.21.15.23:0/968703540 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fbb78004250 con 0x5621ad0a59e0 2024-03-24T10:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb93186740 1 -- 172.21.15.23:0/968703540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5621ad030790 con 0x5621ad0a59e0 2024-03-24T10:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.552+0000 7fbb71ffb640 1 -- 172.21.15.23:0/968703540 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb78004410 con 0x5621ad0a59e0 2024-03-24T10:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.553+0000 7fbb93186740 10 client.17322 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.553+0000 7fbb93186740 1 -- 172.21.15.23:0/968703540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5621ad00a310 con 0x5621ad0a59e0 2024-03-24T10:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.553+0000 7fbb71ffb640 1 -- 172.21.15.23:0/968703540 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5621ad030790 con 0x5621ad0a59e0 2024-03-24T10:46:10.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.553+0000 7fbb71ffb640 1 -- 172.21.15.23:0/968703540 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fbb7803c040 con 0x5621ad0a59e0 2024-03-24T10:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.553+0000 7fbb93186740 10 client.17322 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.553+0000 7fbb93186740 10 client.17322 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.553+0000 7fbb93186740 20 client.17322 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.553+0000 7fbb93186740 1 --2- 172.21.15.23:0/968703540 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5621ad0c7330 0x5621ad0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.553+0000 7fbb93186740 4 client.17322 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.553+0000 7fbb93186740 1 -- 172.21.15.23:0/968703540 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621acd0b600 con 0x5621ad0c7330 2024-03-24T10:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.553+0000 7fbb8e33d640 1 --2- 172.21.15.23:0/968703540 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5621ad0c7330 0x5621ad0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.553+0000 7fbb71ffb640 10 client.17322 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.553+0000 7fbb8e33d640 1 --2- 172.21.15.23:0/968703540 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5621ad0c7330 0x5621ad0c9770 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=488949ff388e31b6 server_cookie=eac02b46b1b558cf in_seq=0 out_seq=0 2024-03-24T10:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.554+0000 7fbb71ffb640 1 -- 172.21.15.23:0/968703540 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5621acd0b600 con 0x5621ad0c7330 2024-03-24T10:46:10.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.554+0000 7fbb71ffb640 10 client.17322 handle_command_reply: tid=0 2024-03-24T10:46:10.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.554+0000 7fbb93186740 1 client.17322 shutdown 2024-03-24T10:46:10.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.554+0000 7fbb93186740 1 -- 172.21.15.23:0/968703540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a59e0 msgr2=0x5621ad0a7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.554+0000 7fbb93186740 1 --2- 172.21.15.23:0/968703540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a59e0 0x5621ad0a7d20 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbb7802c170 tx=0x7fbb7802ca10 comp rx=0 tx=0).stop 2024-03-24T10:46:10.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.555+0000 7fbb93186740 1 -- 172.21.15.23:0/968703540 shutdown_connections 2024-03-24T10:46:10.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.555+0000 7fbb93186740 1 --2- 172.21.15.23:0/968703540 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5621ad0c7330 0x5621ad0c9770 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.555+0000 7fbb93186740 1 --2- 172.21.15.23:0/968703540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0b7370 0x5621ad0a8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.555+0000 7fbb93186740 1 --2- 172.21.15.23:0/968703540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a59e0 0x5621ad0a7d20 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.555+0000 7fbb93186740 1 --2- 172.21.15.23:0/968703540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0a5030 0x5621ad0a77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.555+0000 7fbb71ffb640 0 client.17322 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:10.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.555+0000 7fbb93186740 1 -- 172.21.15.23:0/968703540 >> 172.21.15.23:0/968703540 conn(0x5621ad09ff00 msgr2=0x5621ad0a1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.555+0000 7fbb93186740 1 -- 172.21.15.23:0/968703540 shutdown_connections 2024-03-24T10:46:10.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.555+0000 7fbb93186740 1 -- 172.21.15.23:0/968703540 wait complete. 2024-03-24T10:46:10.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.555+0000 7fbb93186740 20 client.17322 trim_cache size 0 max 16384 2024-03-24T10:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.572+0000 7fbb93186740 1 Processor -- start 2024-03-24T10:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.572+0000 7fbb93186740 1 -- start start 2024-03-24T10:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.572+0000 7fbb93186740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a6920 0x5621ad0c2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.573 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.572+0000 7fbb93186740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad09ff00 0x5621ad0a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.572+0000 7fbb93186740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0a8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.572+0000 7fbb93186740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5621acf91660 con 0x5621ad0a6920 2024-03-24T10:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.572+0000 7fbb93186740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5621ad0c3dd0 con 0x5621ad09ff00 2024-03-24T10:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.572+0000 7fbb93186740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5621ad09b6f0 con 0x5621ad0a5030 2024-03-24T10:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.572+0000 7fbb8e33d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a6920 0x5621ad0c2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.572+0000 7fbb8eb3e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0a8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.572+0000 7fbb8db3c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad09ff00 0x5621ad0a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.572+0000 7fbb8eb3e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0a8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:60862/0 (socket says 172.21.15.23:60862) 2024-03-24T10:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.572+0000 7fbb8e33d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a6920 0x5621ad0c2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56382/0 (socket says 172.21.15.23:56382) 2024-03-24T10:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.572+0000 7fbb8eb3e640 1 -- 172.21.15.23:0/2022840732 learned_addr learned my addr 172.21.15.23:0/2022840732 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.573+0000 7fbb8eb3e640 1 -- 172.21.15.23:0/2022840732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad09ff00 msgr2=0x5621ad0a02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.573+0000 7fbb8eb3e640 1 --2- 172.21.15.23:0/2022840732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad09ff00 0x5621ad0a02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.573+0000 7fbb8eb3e640 1 -- 172.21.15.23:0/2022840732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a6920 msgr2=0x5621ad0c2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.573+0000 7fbb8eb3e640 1 --2- 172.21.15.23:0/2022840732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a6920 0x5621ad0c2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.573+0000 7fbb8eb3e640 1 -- 172.21.15.23:0/2022840732 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621acfa6810 con 0x5621ad0a5030 2024-03-24T10:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.573+0000 7fbb8e33d640 1 --2- 172.21.15.23:0/2022840732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a6920 0x5621ad0c2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.573+0000 7fbb8eb3e640 1 --2- 172.21.15.23:0/2022840732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0a8080 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbb80012000 tx=0x7fbb8000e5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62f123717ed9010b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.573+0000 7fbb73fff640 1 -- 172.21.15.23:0/2022840732 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8002d030 con 0x5621ad0a5030 2024-03-24T10:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.573+0000 7fbb73fff640 1 -- 172.21.15.23:0/2022840732 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb800134c0 con 0x5621ad0a5030 2024-03-24T10:46:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.573+0000 7fbb73fff640 1 -- 172.21.15.23:0/2022840732 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb80010b70 con 0x5621ad0a5030 2024-03-24T10:46:10.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 -- 172.21.15.23:0/2022840732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 msgr2=0x5621ad0a8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 --2- 172.21.15.23:0/2022840732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0a8080 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbb80012000 tx=0x7fbb8000e5f0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 -- 172.21.15.23:0/2022840732 shutdown_connections 2024-03-24T10:46:10.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 --2- 172.21.15.23:0/2022840732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0a8080 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 --2- 172.21.15.23:0/2022840732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad09ff00 0x5621ad0a02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 --2- 172.21.15.23:0/2022840732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad0a6920 0x5621ad0c2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 -- 172.21.15.23:0/2022840732 >> 172.21.15.23:0/2022840732 conn(0x5621ad0b9940 msgr2=0x5621ad09eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 -- 172.21.15.23:0/2022840732 shutdown_connections 2024-03-24T10:46:10.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 -- 172.21.15.23:0/2022840732 wait complete. 2024-03-24T10:46:10.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 -- start start 2024-03-24T10:46:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 Processor -- start 2024-03-24T10:46:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad09ff00 0x5621ad0b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0b5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0a6920 0x5621ad0b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5621ad0bd220 con 0x5621ad09ff00 2024-03-24T10:46:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5621ad0bd390 con 0x5621ad0a6920 2024-03-24T10:46:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5621ad0bd550 con 0x5621ad0a5030 2024-03-24T10:46:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb93186740 10 client.0 _ll_register_callbacks cb 0x7fbb78009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.574+0000 7fbb8e33d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad09ff00 0x5621ad0b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb8e33d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad09ff00 0x5621ad0b4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56388/0 (socket says 172.21.15.23:56388) 2024-03-24T10:46:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb8e33d640 1 -- 172.21.15.23:0/115338886 learned_addr learned my addr 172.21.15.23:0/115338886 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb8db3c640 1 --2- 172.21.15.23:0/115338886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb8eb3e640 1 --2- 172.21.15.23:0/115338886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0a6920 0x5621ad0b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb8e33d640 1 -- 172.21.15.23:0/115338886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 msgr2=0x5621ad0b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb8e33d640 1 --2- 172.21.15.23:0/115338886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0b5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb8e33d640 1 -- 172.21.15.23:0/115338886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0a6920 msgr2=0x5621ad0b5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb8e33d640 1 --2- 172.21.15.23:0/115338886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0a6920 0x5621ad0b5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb8e33d640 1 -- 172.21.15.23:0/115338886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb881a4ab0 con 0x5621ad09ff00 2024-03-24T10:46:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb8e33d640 1 --2- 172.21.15.23:0/115338886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad09ff00 0x5621ad0b4fc0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fbb840cc0d0 tx=0x7fbb840d0690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36bd9a6558d14d4a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb71ffb640 1 -- 172.21.15.23:0/115338886 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8409ae20 con 0x5621ad09ff00 2024-03-24T10:46:10.581 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb71ffb640 1 -- 172.21.15.23:0/115338886 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fbb84097490 con 0x5621ad09ff00 2024-03-24T10:46:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb93186740 1 -- 172.21.15.23:0/115338886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb8000fe70 con 0x5621ad09ff00 2024-03-24T10:46:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb71ffb640 1 -- 172.21.15.23:0/115338886 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb84097650 con 0x5621ad09ff00 2024-03-24T10:46:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb93186740 10 client.17328 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb93186740 1 -- 172.21.15.23:0/115338886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x7fbb80022070 con 0x5621ad09ff00 2024-03-24T10:46:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.575+0000 7fbb71ffb640 1 -- 172.21.15.23:0/115338886 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb840b9020 con 0x5621ad09ff00 2024-03-24T10:46:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.576+0000 7fbb71ffb640 1 -- 172.21.15.23:0/115338886 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fbb84099ce0 con 0x5621ad09ff00 2024-03-24T10:46:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.576+0000 7fbb93186740 10 client.17328 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.576+0000 7fbb93186740 10 client.17328 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.576+0000 7fbb93186740 20 client.17328 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:10.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.576+0000 7fbb93186740 1 --2- 172.21.15.23:0/115338886 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5621ad0a59e0 0x5621ad130200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.576+0000 7fbb93186740 4 client.17328 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:46:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.576+0000 7fbb93186740 1 -- 172.21.15.23:0/115338886 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5621acd5c1b0 con 0x5621ad0a59e0 2024-03-24T10:46:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.576+0000 7fbb8db3c640 1 --2- 172.21.15.23:0/115338886 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5621ad0a59e0 0x5621ad130200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.576+0000 7fbb71ffb640 10 client.17328 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:10.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.576+0000 7fbb8db3c640 1 --2- 172.21.15.23:0/115338886 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5621ad0a59e0 0x5621ad130200 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78071e15d675b7b9 server_cookie=7eb0bfa0912891bd in_seq=0 out_seq=0 2024-03-24T10:46:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb71ffb640 1 -- 172.21.15.23:0/115338886 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5621acd5c1b0 con 0x5621ad0a59e0 2024-03-24T10:46:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb71ffb640 10 client.17328 handle_command_reply: tid=0 2024-03-24T10:46:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb93186740 1 client.17328 shutdown 2024-03-24T10:46:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb93186740 1 -- 172.21.15.23:0/115338886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad09ff00 msgr2=0x5621ad0b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb93186740 1 --2- 172.21.15.23:0/115338886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad09ff00 0x5621ad0b4fc0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fbb840cc0d0 tx=0x7fbb840d0690 comp rx=0 tx=0).stop 2024-03-24T10:46:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb93186740 1 -- 172.21.15.23:0/115338886 shutdown_connections 2024-03-24T10:46:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb93186740 1 --2- 172.21.15.23:0/115338886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621ad0a6920 0x5621ad0b5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb93186740 1 --2- 172.21.15.23:0/115338886 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5621ad0a59e0 0x5621ad130200 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb93186740 1 --2- 172.21.15.23:0/115338886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621ad0a5030 0x5621ad0b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb93186740 1 --2- 172.21.15.23:0/115338886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5621ad09ff00 0x5621ad0b4fc0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb71ffb640 0 client.17328 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb93186740 1 -- 172.21.15.23:0/115338886 >> 172.21.15.23:0/115338886 conn(0x5621ad0b9940 msgr2=0x5621ad09eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb93186740 1 -- 172.21.15.23:0/115338886 shutdown_connections 2024-03-24T10:46:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb93186740 1 -- 172.21.15.23:0/115338886 wait complete. 2024-03-24T10:46:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.587+0000 7fbb93186740 20 client.17328 trim_cache size 0 max 16384 2024-03-24T10:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:46:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.588+0000 7fbb905a1640 1 -- 172.21.15.23:0/1314795493 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbb740a22d0 msgr2=0x7fbb740a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.588+0000 7fbb905a1640 1 --2- 172.21.15.23:0/1314795493 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbb740a22d0 0x7fbb740a4790 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbb80013000 tx=0x7fbb8001e000 comp rx=0 tx=0).stop 2024-03-24T10:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.588+0000 7fbb905a1640 1 -- 172.21.15.23:0/1314795493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb88068d90 msgr2=0x7fbb88197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.588+0000 7fbb905a1640 1 --2- 172.21.15.23:0/1314795493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb88068d90 0x7fbb88197d60 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbb8400e9f0 tx=0x7fbb8400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.588+0000 7fbb905a1640 1 -- 172.21.15.23:0/1314795493 shutdown_connections 2024-03-24T10:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.588+0000 7fbb905a1640 1 --2- 172.21.15.23:0/1314795493 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbb740a22d0 0x7fbb740a4790 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.588+0000 7fbb905a1640 1 --2- 172.21.15.23:0/1314795493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb88107800 0x7fbb8819c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.588+0000 7fbb905a1640 1 --2- 172.21.15.23:0/1314795493 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb88069730 0x7fbb881982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.588+0000 7fbb905a1640 1 --2- 172.21.15.23:0/1314795493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb88068d90 0x7fbb88197d60 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.588+0000 7fbb905a1640 1 -- 172.21.15.23:0/1314795493 >> 172.21.15.23:0/1314795493 conn(0x7fbb88077460 msgr2=0x7fbb880fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.588+0000 7fbb905a1640 1 -- 172.21.15.23:0/1314795493 shutdown_connections 2024-03-24T10:46:10.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:10.588+0000 7fbb905a1640 1 -- 172.21.15.23:0/1314795493 wait complete. 2024-03-24T10:46:11.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 581983 11.34 MB/sec execute 148 sec latency 22.008 ms 2024-03-24T10:46:11.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:10 smithi023 ceph-mon[32466]: pgmap v186: 97 pgs: 97 active+clean; 357 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-03-24T10:46:11.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:10 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3021266768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:46:11.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:10 smithi117 ceph-mon[36368]: pgmap v186: 97 pgs: 97 active+clean; 357 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-03-24T10:46:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:10 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3021266768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:46:11.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:10 smithi116 ceph-mon[36248]: pgmap v186: 97 pgs: 97 active+clean; 357 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.0 MiB/s wr, 207 op/s 2024-03-24T10:46:11.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:10 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3021266768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:46:11.526 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:46:11.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:46:11.527 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:46:11.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.640+0000 7fac1b1b5640 1 -- 172.21.15.23:0/3910596488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac141005a0 msgr2=0x7fac14100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.640+0000 7fac1b1b5640 1 --2- 172.21.15.23:0/3910596488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac141005a0 0x7fac14100a20 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fac00009900 tx=0x7fac0002bc30 comp rx=0 tx=0).stop 2024-03-24T10:46:11.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.640+0000 7fac1b1b5640 1 -- 172.21.15.23:0/3910596488 shutdown_connections 2024-03-24T10:46:11.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.640+0000 7fac1b1b5640 1 --2- 172.21.15.23:0/3910596488 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac14100f60 0x7fac1410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.640+0000 7fac1b1b5640 1 --2- 172.21.15.23:0/3910596488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac141005a0 0x7fac14100a20 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.640+0000 7fac1b1b5640 1 --2- 172.21.15.23:0/3910596488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac14106660 0x7fac14106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.640+0000 7fac1b1b5640 1 -- 172.21.15.23:0/3910596488 >> 172.21.15.23:0/3910596488 conn(0x7fac14077710 msgr2=0x7fac14077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:11.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.640+0000 7fac1b1b5640 1 -- 172.21.15.23:0/3910596488 shutdown_connections 2024-03-24T10:46:11.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.640+0000 7fac1b1b5640 1 -- 172.21.15.23:0/3910596488 wait complete. 2024-03-24T10:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.641+0000 7fac1b1b5640 1 Processor -- start 2024-03-24T10:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.641+0000 7fac1b1b5640 1 -- start start 2024-03-24T10:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.641+0000 7fac1b1b5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac141005a0 0x7fac1419d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.641+0000 7fac1b1b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac14100f60 0x7fac1419de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.641+0000 7fac1b1b5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac14106660 0x7fac14197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.641+0000 7fac1b1b5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fac14198120 con 0x7fac14106660 2024-03-24T10:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.641+0000 7fac1b1b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fac14198290 con 0x7fac14100f60 2024-03-24T10:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.641+0000 7fac1b1b5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fac14198450 con 0x7fac141005a0 2024-03-24T10:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.641+0000 7fac19752640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac14106660 0x7fac14197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.641+0000 7fac13fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac14100f60 0x7fac1419de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.641+0000 7fac19752640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac14106660 0x7fac14197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56396/0 (socket says 172.21.15.23:56396) 2024-03-24T10:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.641+0000 7fac19752640 1 -- 172.21.15.23:0/4200318153 learned_addr learned my addr 172.21.15.23:0/4200318153 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.641+0000 7fac18f51640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac141005a0 0x7fac1419d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.642+0000 7fac19752640 1 -- 172.21.15.23:0/4200318153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac141005a0 msgr2=0x7fac1419d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.642+0000 7fac19752640 1 --2- 172.21.15.23:0/4200318153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac141005a0 0x7fac1419d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.642+0000 7fac19752640 1 -- 172.21.15.23:0/4200318153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac14100f60 msgr2=0x7fac1419de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.642+0000 7fac19752640 1 --2- 172.21.15.23:0/4200318153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac14100f60 0x7fac1419de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.642+0000 7fac19752640 1 -- 172.21.15.23:0/4200318153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac00009560 con 0x7fac14106660 2024-03-24T10:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.642+0000 7fac19752640 1 --2- 172.21.15.23:0/4200318153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac14106660 0x7fac14197ad0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fac0c00eb10 tx=0x7fac0c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.642+0000 7fac11ffb640 1 -- 172.21.15.23:0/4200318153 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c004b20 con 0x7fac14106660 2024-03-24T10:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.642+0000 7fac1b1b5640 1 -- 172.21.15.23:0/4200318153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac14198730 con 0x7fac14106660 2024-03-24T10:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.642+0000 7fac11ffb640 1 -- 172.21.15.23:0/4200318153 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac0c004c80 con 0x7fac14106660 2024-03-24T10:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.642+0000 7fac11ffb640 1 -- 172.21.15.23:0/4200318153 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c010650 con 0x7fac14106660 2024-03-24T10:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.642+0000 7fac1b1b5640 1 -- 172.21.15.23:0/4200318153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac141a49c0 con 0x7fac14106660 2024-03-24T10:46:11.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.644+0000 7fac11ffb640 1 -- 172.21.15.23:0/4200318153 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fac0c005020 con 0x7fac14106660 2024-03-24T10:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac11ffb640 1 --2- 172.21.15.23:0/4200318153 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fac040a2230 0x7fac040a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac1dd96740 1 Processor -- start 2024-03-24T10:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac1dd96740 1 -- start start 2024-03-24T10:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac18f51640 1 --2- 172.21.15.23:0/4200318153 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fac040a2230 0x7fac040a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac11ffb640 1 -- 172.21.15.23:0/4200318153 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fac0c014070 con 0x7fac14106660 2024-03-24T10:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac1dd96740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9a760 0x5624a0a9ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac1dd96740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9b080 0x5624a0aa3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac1dd96740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0aa6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac1dd96740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5624a0a08530 con 0x5624a0aa3c60 2024-03-24T10:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac1dd96740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5624a0a05a60 con 0x5624a0a9b080 2024-03-24T10:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac1dd96740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5624a0a01240 con 0x5624a0a9a760 2024-03-24T10:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac19752640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0aa6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac18f51640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9a760 0x5624a0a9ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac19752640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0aa6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56402/0 (socket says 172.21.15.23:56402) 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac19752640 1 -- 172.21.15.23:0/279323479 learned_addr learned my addr 172.21.15.23:0/279323479 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac13fff640 1 --2- 172.21.15.23:0/279323479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9b080 0x5624a0aa3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.645+0000 7fac18f51640 1 --2- 172.21.15.23:0/4200318153 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fac040a2230 0x7fac040a46f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fac08009c20 tx=0x7fac08005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.646+0000 7fac18f51640 1 -- 172.21.15.23:0/279323479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9b080 msgr2=0x5624a0aa3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.646+0000 7fac18f51640 1 --2- 172.21.15.23:0/279323479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9b080 0x5624a0aa3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.646+0000 7fac18f51640 1 -- 172.21.15.23:0/279323479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 msgr2=0x5624a0aa6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.646+0000 7fac18f51640 1 --2- 172.21.15.23:0/279323479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0aa6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.646+0000 7fac18f51640 1 -- 172.21.15.23:0/279323479 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624a09ff8c0 con 0x5624a0a9a760 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.646+0000 7fac18f51640 1 --2- 172.21.15.23:0/279323479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9a760 0x5624a0a9ab40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fac08016000 tx=0x7fac08019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc66397ae4f9f0b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.646+0000 7fabfeffd640 1 -- 172.21.15.23:0/279323479 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac08005220 con 0x5624a0a9a760 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.646+0000 7fabfeffd640 1 -- 172.21.15.23:0/279323479 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac080047a0 con 0x5624a0a9a760 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.646+0000 7fabfeffd640 1 -- 172.21.15.23:0/279323479 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0801b8c0 con 0x5624a0a9a760 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.647+0000 7fac1dd96740 1 -- 172.21.15.23:0/279323479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9a760 msgr2=0x5624a0a9ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.647+0000 7fac1dd96740 1 --2- 172.21.15.23:0/279323479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9a760 0x5624a0a9ab40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fac08016000 tx=0x7fac08019000 comp rx=0 tx=0).stop 2024-03-24T10:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.647+0000 7fac1dd96740 1 -- 172.21.15.23:0/279323479 shutdown_connections 2024-03-24T10:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.647+0000 7fac1dd96740 1 --2- 172.21.15.23:0/279323479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0aa6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.647+0000 7fac1dd96740 1 --2- 172.21.15.23:0/279323479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9b080 0x5624a0aa3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.647+0000 7fac1dd96740 1 --2- 172.21.15.23:0/279323479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9a760 0x5624a0a9ab40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.647+0000 7fac1dd96740 1 -- 172.21.15.23:0/279323479 >> 172.21.15.23:0/279323479 conn(0x5624a0a94e90 msgr2=0x5624a0a95270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.647+0000 7fac1dd96740 1 -- 172.21.15.23:0/279323479 shutdown_connections 2024-03-24T10:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.647+0000 7fac1dd96740 1 -- 172.21.15.23:0/279323479 wait complete. 2024-03-24T10:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.647+0000 7fac1dd96740 1 -- start start 2024-03-24T10:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac1dd96740 1 Processor -- start 2024-03-24T10:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac1dd96740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9a760 0x5624a0aa0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac1dd96740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9b080 0x5624a0aa11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac1dd96740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0aa1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac1dd96740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5624a09a5f80 con 0x5624a0aa3c60 2024-03-24T10:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac1dd96740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5624a06d44b0 con 0x5624a0a9b080 2024-03-24T10:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac1dd96740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5624a0986ad0 con 0x5624a0a9a760 2024-03-24T10:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac1dd96740 10 client.0 _ll_register_callbacks cb 0x5624a09b9b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac19752640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0aa1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac18f51640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9a760 0x5624a0aa0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac18f51640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9a760 0x5624a0aa0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:60906/0 (socket says 172.21.15.23:60906) 2024-03-24T10:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac13fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9b080 0x5624a0aa11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac18f51640 1 -- 172.21.15.23:0/1268791914 learned_addr learned my addr 172.21.15.23:0/1268791914 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac18f51640 1 -- 172.21.15.23:0/1268791914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9b080 msgr2=0x5624a0aa11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac18f51640 1 --2- 172.21.15.23:0/1268791914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9b080 0x5624a0aa11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac18f51640 1 -- 172.21.15.23:0/1268791914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 msgr2=0x5624a0aa1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac18f51640 1 --2- 172.21.15.23:0/1268791914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0aa1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.648+0000 7fac18f51640 1 -- 172.21.15.23:0/1268791914 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac08017050 con 0x5624a0a9a760 2024-03-24T10:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fac18f51640 1 --2- 172.21.15.23:0/1268791914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9a760 0x5624a0aa0cb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fac08004f90 tx=0x7fac08004e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42953aba1c3172a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fabf3fff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fabf3fff640 1 -- 172.21.15.23:0/1268791914 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac080044d0 con 0x5624a0a9a760 2024-03-24T10:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fac1dd96740 1 -- 172.21.15.23:0/1268791914 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5624a0a257a0 con 0x5624a0a9a760 2024-03-24T10:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fabf3fff640 1 -- 172.21.15.23:0/1268791914 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fac08005000 con 0x5624a0a9a760 2024-03-24T10:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fabf3fff640 1 -- 172.21.15.23:0/1268791914 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fac080051c0 con 0x5624a0a9a760 2024-03-24T10:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fac1dd96740 10 client.27199 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fac1dd96740 1 -- 172.21.15.23:0/1268791914 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5624a09b76d0 con 0x5624a0a9a760 2024-03-24T10:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fabf3fff640 1 -- 172.21.15.23:0/1268791914 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5624a0a257a0 con 0x5624a0a9a760 2024-03-24T10:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fabf3fff640 1 -- 172.21.15.23:0/1268791914 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fac0801b600 con 0x5624a0a9a760 2024-03-24T10:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fac1dd96740 10 client.27199 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fac1dd96740 10 client.27199 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fac1dd96740 20 client.27199 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fac1dd96740 1 --2- 172.21.15.23:0/1268791914 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5624a0ab3c90 0x5624a0ab6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fac1dd96740 4 client.27199 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.649+0000 7fac1dd96740 1 -- 172.21.15.23:0/1268791914 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624a09f71f0 con 0x5624a0ab3c90 2024-03-24T10:46:11.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.650+0000 7fac13fff640 1 --2- 172.21.15.23:0/1268791914 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5624a0ab3c90 0x5624a0ab6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.650+0000 7fac13fff640 1 --2- 172.21.15.23:0/1268791914 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5624a0ab3c90 0x5624a0ab6080 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7e219429a4bc2a7 server_cookie=122f271862a32ae3 in_seq=0 out_seq=0 2024-03-24T10:46:11.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.650+0000 7fabf3fff640 10 client.27199 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:11.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fabf3fff640 1 -- 172.21.15.23:0/1268791914 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5624a09f71f0 con 0x5624a0ab3c90 2024-03-24T10:46:11.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fabf3fff640 10 client.27199 handle_command_reply: tid=0 2024-03-24T10:46:11.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fac1dd96740 1 client.27199 shutdown 2024-03-24T10:46:11.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fac1dd96740 1 -- 172.21.15.23:0/1268791914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9a760 msgr2=0x5624a0aa0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fac1dd96740 1 --2- 172.21.15.23:0/1268791914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9a760 0x5624a0aa0cb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fac08004f90 tx=0x7fac08004e20 comp rx=0 tx=0).stop 2024-03-24T10:46:11.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fac1dd96740 1 -- 172.21.15.23:0/1268791914 shutdown_connections 2024-03-24T10:46:11.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fac1dd96740 1 --2- 172.21.15.23:0/1268791914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0aa1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fac1dd96740 1 --2- 172.21.15.23:0/1268791914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9b080 0x5624a0aa11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fac1dd96740 1 --2- 172.21.15.23:0/1268791914 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5624a0ab3c90 0x5624a0ab6080 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fac1dd96740 1 --2- 172.21.15.23:0/1268791914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9a760 0x5624a0aa0cb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fabf3fff640 0 client.27199 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:11.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fac1dd96740 1 -- 172.21.15.23:0/1268791914 >> 172.21.15.23:0/1268791914 conn(0x5624a0a94e90 msgr2=0x5624a0a96be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:11.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fac1dd96740 1 -- 172.21.15.23:0/1268791914 shutdown_connections 2024-03-24T10:46:11.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fac1dd96740 1 -- 172.21.15.23:0/1268791914 wait complete. 2024-03-24T10:46:11.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.651+0000 7fac1dd96740 20 client.27199 trim_cache size 0 max 16384 2024-03-24T10:46:11.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.671+0000 7fac1dd96740 1 Processor -- start 2024-03-24T10:46:11.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.671+0000 7fac1dd96740 1 -- start start 2024-03-24T10:46:11.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.671+0000 7fac1dd96740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9a760 0x5624a0aa33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.671+0000 7fac1dd96740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9b080 0x5624a0aaf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.671+0000 7fac1dd96740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0abb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.671+0000 7fac1dd96740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5624a0a9a1a0 con 0x5624a0aa3c60 2024-03-24T10:46:11.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.671+0000 7fac1dd96740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5624a0aa0aa0 con 0x5624a0a9a760 2024-03-24T10:46:11.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.671+0000 7fac1dd96740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5624a0a95e90 con 0x5624a0a9b080 2024-03-24T10:46:11.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.671+0000 7fac19752640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0abb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.671+0000 7fac18f51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9a760 0x5624a0aa33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.671+0000 7fac13fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9b080 0x5624a0aaf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.671+0000 7fac19752640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0abb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56418/0 (socket says 172.21.15.23:56418) 2024-03-24T10:46:11.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.671+0000 7fac19752640 1 -- 172.21.15.23:0/999035702 learned_addr learned my addr 172.21.15.23:0/999035702 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:11.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.672+0000 7fac18f51640 1 -- 172.21.15.23:0/999035702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9b080 msgr2=0x5624a0aaf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.672+0000 7fac18f51640 1 --2- 172.21.15.23:0/999035702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9b080 0x5624a0aaf4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.672+0000 7fac18f51640 1 -- 172.21.15.23:0/999035702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 msgr2=0x5624a0abb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.672+0000 7fac18f51640 1 --2- 172.21.15.23:0/999035702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0abb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.672+0000 7fac18f51640 1 -- 172.21.15.23:0/999035702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624a099bc60 con 0x5624a0a9a760 2024-03-24T10:46:11.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.672+0000 7fac13fff640 1 --2- 172.21.15.23:0/999035702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9b080 0x5624a0aaf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:11.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.672+0000 7fac19752640 1 --2- 172.21.15.23:0/999035702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0abb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:11.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.672+0000 7fac18f51640 1 --2- 172.21.15.23:0/999035702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9a760 0x5624a0aa33e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac08011d40 tx=0x7fac08004cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=579630832c3eb0f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:11.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.672+0000 7fabfe7fc640 1 -- 172.21.15.23:0/999035702 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0801be00 con 0x5624a0a9a760 2024-03-24T10:46:11.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.672+0000 7fabfe7fc640 1 -- 172.21.15.23:0/999035702 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac0801dcb0 con 0x5624a0a9a760 2024-03-24T10:46:11.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.672+0000 7fabfe7fc640 1 -- 172.21.15.23:0/999035702 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac08004530 con 0x5624a0a9a760 2024-03-24T10:46:11.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.673+0000 7fac1dd96740 1 -- 172.21.15.23:0/999035702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9a760 msgr2=0x5624a0aa33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.673+0000 7fac1dd96740 1 --2- 172.21.15.23:0/999035702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9a760 0x5624a0aa33e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac08011d40 tx=0x7fac08004cd0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.673+0000 7fac1dd96740 1 -- 172.21.15.23:0/999035702 shutdown_connections 2024-03-24T10:46:11.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.673+0000 7fac1dd96740 1 --2- 172.21.15.23:0/999035702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0aa3c60 0x5624a0abb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.673+0000 7fac1dd96740 1 --2- 172.21.15.23:0/999035702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0a9b080 0x5624a0aaf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.673+0000 7fac1dd96740 1 --2- 172.21.15.23:0/999035702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9a760 0x5624a0aa33e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.673+0000 7fac1dd96740 1 -- 172.21.15.23:0/999035702 >> 172.21.15.23:0/999035702 conn(0x5624a0a94e90 msgr2=0x5624a0a96210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.673+0000 7fac1dd96740 1 -- 172.21.15.23:0/999035702 shutdown_connections 2024-03-24T10:46:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.673+0000 7fac1dd96740 1 -- 172.21.15.23:0/999035702 wait complete. 2024-03-24T10:46:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.673+0000 7fac1dd96740 1 -- start start 2024-03-24T10:46:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.673+0000 7fac1dd96740 1 Processor -- start 2024-03-24T10:46:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.673+0000 7fac1dd96740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9a760 0x5624a0a9db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.673+0000 7fac1dd96740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0a9b080 0x5624a0a9e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac1dd96740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0aa3c60 0x5624a0b1d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac1dd96740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5624a0a9e8a0 con 0x5624a0a9b080 2024-03-24T10:46:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac1dd96740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5624a0a9ea10 con 0x5624a0a9a760 2024-03-24T10:46:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac1dd96740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5624a0b1d8c0 con 0x5624a0aa3c60 2024-03-24T10:46:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac1dd96740 10 client.0 _ll_register_callbacks cb 0x7fac08009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac13fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0a9b080 0x5624a0a9e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac13fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0a9b080 0x5624a0a9e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56424/0 (socket says 172.21.15.23:56424) 2024-03-24T10:46:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac13fff640 1 -- 172.21.15.23:0/224826624 learned_addr learned my addr 172.21.15.23:0/224826624 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac18f51640 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9a760 0x5624a0a9db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac19752640 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0aa3c60 0x5624a0b1d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac13fff640 1 -- 172.21.15.23:0/224826624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0aa3c60 msgr2=0x5624a0b1d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac13fff640 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0aa3c60 0x5624a0b1d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac13fff640 1 -- 172.21.15.23:0/224826624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9a760 msgr2=0x5624a0a9db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac13fff640 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9a760 0x5624a0a9db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac13fff640 1 -- 172.21.15.23:0/224826624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac1410dd20 con 0x5624a0a9b080 2024-03-24T10:46:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac18f51640 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9a760 0x5624a0a9db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac19752640 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0aa3c60 0x5624a0b1d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:46:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac13fff640 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0a9b080 0x5624a0a9e0d0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fac00009900 tx=0x7fac00005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99554e20f9a5e15d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fabf3fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fabf3fff640 1 -- 172.21.15.23:0/224826624 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0004ad20 con 0x5624a0a9b080 2024-03-24T10:46:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fabf3fff640 1 -- 172.21.15.23:0/224826624 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fac00046040 con 0x5624a0a9b080 2024-03-24T10:46:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac1dd96740 1 -- 172.21.15.23:0/224826624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5624a0700860 con 0x5624a0a9b080 2024-03-24T10:46:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fabf3fff640 1 -- 172.21.15.23:0/224826624 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fac0004ae80 con 0x5624a0a9b080 2024-03-24T10:46:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac1dd96740 10 client.17346 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.674+0000 7fac1dd96740 1 -- 172.21.15.23:0/224826624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x7fac141a49c0 con 0x5624a0a9b080 2024-03-24T10:46:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.675+0000 7fabf3fff640 1 -- 172.21.15.23:0/224826624 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5624a0700860 con 0x5624a0a9b080 2024-03-24T10:46:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.675+0000 7fabf3fff640 1 -- 172.21.15.23:0/224826624 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fac00053020 con 0x5624a0a9b080 2024-03-24T10:46:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.675+0000 7fac1dd96740 10 client.17346 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.675+0000 7fac1dd96740 10 client.17346 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.675+0000 7fac1dd96740 20 client.17346 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.675+0000 7fac1dd96740 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5624a0b1f230 0x5624a0b21650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.675+0000 7fac1dd96740 4 client.17346 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:46:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.675+0000 7fac1dd96740 1 -- 172.21.15.23:0/224826624 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5624a0751410 con 0x5624a0b1f230 2024-03-24T10:46:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.675+0000 7fac18f51640 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5624a0b1f230 0x5624a0b21650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.675+0000 7fac18f51640 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5624a0b1f230 0x5624a0b21650 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22f622a70ee2206f server_cookie=78d965782dc5c28a in_seq=0 out_seq=0 2024-03-24T10:46:11.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.675+0000 7fabf3fff640 10 client.17346 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:11.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fabf3fff640 1 -- 172.21.15.23:0/224826624 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5624a0751410 con 0x5624a0b1f230 2024-03-24T10:46:11.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fabf3fff640 10 client.17346 handle_command_reply: tid=0 2024-03-24T10:46:11.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fac1dd96740 1 client.17346 shutdown 2024-03-24T10:46:11.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fac1dd96740 1 -- 172.21.15.23:0/224826624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0a9b080 msgr2=0x5624a0a9e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fac1dd96740 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0a9b080 0x5624a0a9e0d0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fac00009900 tx=0x7fac00005fd0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fac1dd96740 1 -- 172.21.15.23:0/224826624 shutdown_connections 2024-03-24T10:46:11.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fac1dd96740 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5624a0b1f230 0x5624a0b21650 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fac1dd96740 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5624a0aa3c60 0x5624a0b1d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fabf3fff640 0 client.17346 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:11.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fac1dd96740 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624a0a9b080 0x5624a0a9e0d0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fac1dd96740 1 --2- 172.21.15.23:0/224826624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5624a0a9a760 0x5624a0a9db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fac1dd96740 1 -- 172.21.15.23:0/224826624 >> 172.21.15.23:0/224826624 conn(0x5624a0a94e90 msgr2=0x5624a0aa14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:11.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fac1dd96740 1 -- 172.21.15.23:0/224826624 shutdown_connections 2024-03-24T10:46:11.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fac1dd96740 1 -- 172.21.15.23:0/224826624 wait complete. 2024-03-24T10:46:11.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.677+0000 7fac1dd96740 20 client.17346 trim_cache size 0 max 16384 2024-03-24T10:46:11.683 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:46:11.683 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:46:11.683 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "bbc26982-599f-4398-b15e-e1177a25b88a", 2024-03-24T10:46:11.683 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:46:11.683 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:46:11.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.678+0000 7fac1b1b5640 1 -- 172.21.15.23:0/4200318153 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fac040a2230 msgr2=0x7fac040a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.678+0000 7fac1b1b5640 1 --2- 172.21.15.23:0/4200318153 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fac040a2230 0x7fac040a46f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fac08009c20 tx=0x7fac08005f90 comp rx=0 tx=0).stop 2024-03-24T10:46:11.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.678+0000 7fac1b1b5640 1 -- 172.21.15.23:0/4200318153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac14106660 msgr2=0x7fac14197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.678+0000 7fac1b1b5640 1 --2- 172.21.15.23:0/4200318153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac14106660 0x7fac14197ad0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fac0c00eb10 tx=0x7fac0c00c2d0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.678+0000 7fac1b1b5640 1 -- 172.21.15.23:0/4200318153 shutdown_connections 2024-03-24T10:46:11.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.678+0000 7fac1b1b5640 1 --2- 172.21.15.23:0/4200318153 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fac040a2230 0x7fac040a46f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.678+0000 7fac1b1b5640 1 --2- 172.21.15.23:0/4200318153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac14106660 0x7fac14197ad0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.678+0000 7fac1b1b5640 1 --2- 172.21.15.23:0/4200318153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac14100f60 0x7fac1419de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.678+0000 7fac1b1b5640 1 --2- 172.21.15.23:0/4200318153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac141005a0 0x7fac1419d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.678+0000 7fac1b1b5640 1 -- 172.21.15.23:0/4200318153 >> 172.21.15.23:0/4200318153 conn(0x7fac14077710 msgr2=0x7fac14068af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:11.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.678+0000 7fac1b1b5640 1 -- 172.21.15.23:0/4200318153 shutdown_connections 2024-03-24T10:46:11.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.678+0000 7fac1b1b5640 1 -- 172.21.15.23:0/4200318153 wait complete. 2024-03-24T10:46:11.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:46:11.689 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:46:11.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.796+0000 7f4fdc7b5640 1 -- 172.21.15.23:0/1941944603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fd4100630 msgr2=0x7f4fd4100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.796+0000 7f4fdc7b5640 1 --2- 172.21.15.23:0/1941944603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fd4100630 0x7f4fd4100ab0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009930 tx=0x7f4fc802bc30 comp rx=0 tx=0).stop 2024-03-24T10:46:11.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.796+0000 7f4fdc7b5640 1 -- 172.21.15.23:0/1941944603 shutdown_connections 2024-03-24T10:46:11.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.796+0000 7f4fdc7b5640 1 --2- 172.21.15.23:0/1941944603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4fd4100ff0 0x7f4fd410d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.796+0000 7f4fdc7b5640 1 --2- 172.21.15.23:0/1941944603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fd4100630 0x7f4fd4100ab0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.796+0000 7f4fdc7b5640 1 --2- 172.21.15.23:0/1941944603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd4106640 0x7f4fd4106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.796+0000 7f4fdc7b5640 1 -- 172.21.15.23:0/1941944603 >> 172.21.15.23:0/1941944603 conn(0x7f4fd4077770 msgr2=0x7f4fd4077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:11.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.797+0000 7f4fdc7b5640 1 -- 172.21.15.23:0/1941944603 shutdown_connections 2024-03-24T10:46:11.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.797+0000 7f4fdc7b5640 1 -- 172.21.15.23:0/1941944603 wait complete. 2024-03-24T10:46:11.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.797+0000 7f4fdc7b5640 1 Processor -- start 2024-03-24T10:46:11.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.797+0000 7f4fdc7b5640 1 -- start start 2024-03-24T10:46:11.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.797+0000 7f4fdc7b5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fd4100630 0x7f4fd4197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.797+0000 7f4fdc7b5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4fd4100ff0 0x7f4fd4198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.797+0000 7f4fdc7b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd4106640 0x7f4fd419c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.797+0000 7f4fdc7b5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd419cc30 con 0x7f4fd4100630 2024-03-24T10:46:11.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.797+0000 7f4fdc7b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd419cda0 con 0x7f4fd4106640 2024-03-24T10:46:11.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.797+0000 7f4fdc7b5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd419cf60 con 0x7f4fd4100ff0 2024-03-24T10:46:11.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.797+0000 7f4fda551640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fd4100630 0x7f4fd4197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.797+0000 7f4fda551640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fd4100630 0x7f4fd4197dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56442/0 (socket says 172.21.15.23:56442) 2024-03-24T10:46:11.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.797+0000 7f4fda551640 1 -- 172.21.15.23:0/4034408109 learned_addr learned my addr 172.21.15.23:0/4034408109 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:11.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.798+0000 7f4fdad52640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd4106640 0x7f4fd419c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.798+0000 7f4fd9d50640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4fd4100ff0 0x7f4fd4198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.798+0000 7f4fda551640 1 -- 172.21.15.23:0/4034408109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4fd4100ff0 msgr2=0x7f4fd4198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.798+0000 7f4fda551640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4fd4100ff0 0x7f4fd4198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.798+0000 7f4fda551640 1 -- 172.21.15.23:0/4034408109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd4106640 msgr2=0x7f4fd419c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.798+0000 7f4fda551640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd4106640 0x7f4fd419c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.798+0000 7f4fda551640 1 -- 172.21.15.23:0/4034408109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc8009560 con 0x7f4fd4100630 2024-03-24T10:46:11.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.798+0000 7f4fdad52640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd4106640 0x7f4fd419c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:11.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.798+0000 7f4fda551640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fd4100630 0x7f4fd4197dc0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00e9f0 tx=0x7f4fcc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:11.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.798+0000 7f4fd37fe640 1 -- 172.21.15.23:0/4034408109 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc009700 con 0x7f4fd4100630 2024-03-24T10:46:11.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.798+0000 7f4fdc7b5640 1 -- 172.21.15.23:0/4034408109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd419d240 con 0x7f4fd4100630 2024-03-24T10:46:11.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.798+0000 7f4fd37fe640 1 -- 172.21.15.23:0/4034408109 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fcc00cd80 con 0x7f4fd4100630 2024-03-24T10:46:11.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.798+0000 7f4fd37fe640 1 -- 172.21.15.23:0/4034408109 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc018920 con 0x7f4fd4100630 2024-03-24T10:46:11.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.798+0000 7f4fdc7b5640 1 -- 172.21.15.23:0/4034408109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd41a4ac0 con 0x7f4fd4100630 2024-03-24T10:46:11.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.801+0000 7f4fd37fe640 1 -- 172.21.15.23:0/4034408109 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f4fcc018a80 con 0x7f4fd4100630 2024-03-24T10:46:11.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.801+0000 7f4fdf397740 1 Processor -- start 2024-03-24T10:46:11.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.801+0000 7f4fdf397740 1 -- start start 2024-03-24T10:46:11.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.801+0000 7f4fdf397740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 0x5622a37425b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.801+0000 7f4fdf397740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a374b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.801+0000 7f4fdf397740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a374b6d0 0x5622a374dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.801+0000 7f4fdf397740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5622a3666f30 con 0x5622a3742af0 2024-03-24T10:46:11.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.801+0000 7f4fdf397740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5622a36b0810 con 0x5622a37421d0 2024-03-24T10:46:11.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.801+0000 7f4fdf397740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5622a36add40 con 0x5622a374b6d0 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fda551640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a374b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fda551640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a374b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56446/0 (socket says 172.21.15.23:56446) 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fda551640 1 -- 172.21.15.23:0/4167599151 learned_addr learned my addr 172.21.15.23:0/4167599151 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fdad52640 1 --2- 172.21.15.23:0/4167599151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 0x5622a37425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fd9d50640 1 --2- 172.21.15.23:0/4167599151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a374b6d0 0x5622a374dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fd37fe640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f4fbc0a2230 0x7f4fbc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fdad52640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f4fbc0a2230 0x7f4fbc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fd37fe640 1 -- 172.21.15.23:0/4034408109 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f4fcc014070 con 0x7f4fd4100630 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fda551640 1 -- 172.21.15.23:0/4167599151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a374b6d0 msgr2=0x5622a374dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fda551640 1 --2- 172.21.15.23:0/4167599151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a374b6d0 0x5622a374dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fda551640 1 -- 172.21.15.23:0/4167599151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 msgr2=0x5622a37425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fda551640 1 --2- 172.21.15.23:0/4167599151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 0x5622a37425b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fda551640 1 -- 172.21.15.23:0/4167599151 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622a366aea0 con 0x5622a3742af0 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fd9d50640 1 --2- 172.21.15.23:0/4167599151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a374b6d0 0x5622a374dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:11.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fdad52640 1 --2- 172.21.15.23:0/4167599151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 0x5622a37425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:11.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fda551640 1 --2- 172.21.15.23:0/4167599151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a374b190 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc09ab20 tx=0x7f4fcc010f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f287de53220e87e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:11.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fd0ff9640 1 -- 172.21.15.23:0/4167599151 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc0d18e0 con 0x5622a3742af0 2024-03-24T10:46:11.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fd0ff9640 1 -- 172.21.15.23:0/4167599151 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fcc09ae20 con 0x5622a3742af0 2024-03-24T10:46:11.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.802+0000 7f4fd0ff9640 1 -- 172.21.15.23:0/4167599151 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc097490 con 0x5622a3742af0 2024-03-24T10:46:11.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.803+0000 7f4fdf397740 1 -- 172.21.15.23:0/4167599151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 msgr2=0x5622a374b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.803+0000 7f4fdad52640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f4fbc0a2230 0x7f4fbc0a46f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40049d0 tx=0x7f4fc4004ab0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:11.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.803+0000 7f4fdf397740 1 --2- 172.21.15.23:0/4167599151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a374b190 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc09ab20 tx=0x7f4fcc010f50 comp rx=0 tx=0).stop 2024-03-24T10:46:11.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.803+0000 7f4fdf397740 1 -- 172.21.15.23:0/4167599151 shutdown_connections 2024-03-24T10:46:11.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.803+0000 7f4fdf397740 1 --2- 172.21.15.23:0/4167599151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a374b6d0 0x5622a374dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.803+0000 7f4fdf397740 1 --2- 172.21.15.23:0/4167599151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a374b190 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.803+0000 7f4fdf397740 1 --2- 172.21.15.23:0/4167599151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 0x5622a37425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.803+0000 7f4fdf397740 1 -- 172.21.15.23:0/4167599151 >> 172.21.15.23:0/4167599151 conn(0x5622a373c900 msgr2=0x5622a373cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:11.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.803+0000 7f4fdf397740 1 -- 172.21.15.23:0/4167599151 shutdown_connections 2024-03-24T10:46:11.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.803+0000 7f4fdf397740 1 -- 172.21.15.23:0/4167599151 wait complete. 2024-03-24T10:46:11.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fdf397740 1 -- start start 2024-03-24T10:46:11.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fdf397740 1 Processor -- start 2024-03-24T10:46:11.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fdf397740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a37421d0 0x5622a3748840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fdf397740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a3748d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fdf397740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a374b6d0 0x5622a37492c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fdf397740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5622a36a9520 con 0x5622a3742af0 2024-03-24T10:46:11.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fdf397740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5622a337c430 con 0x5622a374b6d0 2024-03-24T10:46:11.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fdf397740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5622a3338e00 con 0x5622a37421d0 2024-03-24T10:46:11.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fdf397740 10 client.0 _ll_register_callbacks cb 0x5622a3647d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:11.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fd9d50640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a3748d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fd9d50640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a3748d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56454/0 (socket says 172.21.15.23:56454) 2024-03-24T10:46:11.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fd9d50640 1 -- 172.21.15.23:0/2156687695 learned_addr learned my addr 172.21.15.23:0/2156687695 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:11.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fdad52640 1 --2- 172.21.15.23:0/2156687695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a374b6d0 0x5622a37492c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fda551640 1 --2- 172.21.15.23:0/2156687695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a37421d0 0x5622a3748840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fd9d50640 1 -- 172.21.15.23:0/2156687695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a37421d0 msgr2=0x5622a3748840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fd9d50640 1 --2- 172.21.15.23:0/2156687695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a37421d0 0x5622a3748840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fd9d50640 1 -- 172.21.15.23:0/2156687695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a374b6d0 msgr2=0x5622a37492c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fd9d50640 1 --2- 172.21.15.23:0/2156687695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a374b6d0 0x5622a37492c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.804+0000 7f4fd9d50640 1 -- 172.21.15.23:0/2156687695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd41a4ac0 con 0x5622a3742af0 2024-03-24T10:46:11.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fd9d50640 1 --2- 172.21.15.23:0/2156687695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a3748d80 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009900 tx=0x7f4fc802ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af75a47deb34e20 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:11.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:11.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fb9ffb640 1 -- 172.21.15.23:0/2156687695 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8004040 con 0x5622a3742af0 2024-03-24T10:46:11.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fb9ffb640 1 -- 172.21.15.23:0/2156687695 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f4fc80041a0 con 0x5622a3742af0 2024-03-24T10:46:11.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fdf397740 1 -- 172.21.15.23:0/2156687695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5622a33a87e0 con 0x5622a3742af0 2024-03-24T10:46:11.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fb9ffb640 1 -- 172.21.15.23:0/2156687695 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4fc8004360 con 0x5622a3742af0 2024-03-24T10:46:11.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fdf397740 10 client.17358 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:11.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fdf397740 1 -- 172.21.15.23:0/2156687695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5622a36a7ba0 con 0x5622a3742af0 2024-03-24T10:46:11.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fb9ffb640 1 -- 172.21.15.23:0/2156687695 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5622a33a87e0 con 0x5622a3742af0 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fb9ffb640 1 -- 172.21.15.23:0/2156687695 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f4fc803c040 con 0x5622a3742af0 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fdf397740 10 client.17358 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fdf397740 10 client.17358 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fdf397740 20 client.17358 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fdf397740 1 --2- 172.21.15.23:0/2156687695 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5622a375bc50 0x5622a375e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fdf397740 4 client.17358 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.805+0000 7f4fdf397740 1 -- 172.21.15.23:0/2156687695 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622a33e5720 con 0x5622a375bc50 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.806+0000 7f4fda551640 1 --2- 172.21.15.23:0/2156687695 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5622a375bc50 0x5622a375e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.806+0000 7f4fda551640 1 --2- 172.21.15.23:0/2156687695 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5622a375bc50 0x5622a375e070 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86aac7ac518d1165 server_cookie=86b31f5b98d538a9 in_seq=0 out_seq=0 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.806+0000 7f4fb9ffb640 10 client.17358 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fb9ffb640 1 -- 172.21.15.23:0/2156687695 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5622a33e5720 con 0x5622a375bc50 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fb9ffb640 10 client.17358 handle_command_reply: tid=0 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fdf397740 1 client.17358 shutdown 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fdf397740 1 -- 172.21.15.23:0/2156687695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 msgr2=0x5622a3748d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fdf397740 1 --2- 172.21.15.23:0/2156687695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a3748d80 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009900 tx=0x7f4fc802ca10 comp rx=0 tx=0).stop 2024-03-24T10:46:11.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fdf397740 1 -- 172.21.15.23:0/2156687695 shutdown_connections 2024-03-24T10:46:11.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fdf397740 1 --2- 172.21.15.23:0/2156687695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a374b6d0 0x5622a37492c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fdf397740 1 --2- 172.21.15.23:0/2156687695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a3748d80 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fdf397740 1 --2- 172.21.15.23:0/2156687695 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5622a375bc50 0x5622a375e070 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fdf397740 1 --2- 172.21.15.23:0/2156687695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a37421d0 0x5622a3748840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fdf397740 1 -- 172.21.15.23:0/2156687695 >> 172.21.15.23:0/2156687695 conn(0x5622a373c900 msgr2=0x5622a373e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:11.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fb9ffb640 0 client.17358 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:11.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fdf397740 1 -- 172.21.15.23:0/2156687695 shutdown_connections 2024-03-24T10:46:11.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fdf397740 1 -- 172.21.15.23:0/2156687695 wait complete. 2024-03-24T10:46:11.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.807+0000 7f4fdf397740 20 client.17358 trim_cache size 0 max 16384 2024-03-24T10:46:11.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.825+0000 7f4fdf397740 1 Processor -- start 2024-03-24T10:46:11.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fdf397740 1 -- start start 2024-03-24T10:46:11.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fdf397740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 0x5622a3743010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fdf397740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a3742af0 0x5622a3756f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fdf397740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a374b6d0 0x5622a37490e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fdf397740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5622a362e430 con 0x5622a374b6d0 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fdf397740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5622a33e5750 con 0x5622a37421d0 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fdf397740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5622a373d700 con 0x5622a3742af0 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fdad52640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a374b6d0 0x5622a37490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fd9d50640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a3742af0 0x5622a3756f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fda551640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 0x5622a3743010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fd9d50640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a3742af0 0x5622a3756f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:60974/0 (socket says 172.21.15.23:60974) 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fd9d50640 1 -- 172.21.15.23:0/832627921 learned_addr learned my addr 172.21.15.23:0/832627921 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fdad52640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a374b6d0 0x5622a37490e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56460/0 (socket says 172.21.15.23:56460) 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fd9d50640 1 -- 172.21.15.23:0/832627921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 msgr2=0x5622a3743010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fd9d50640 1 --2- 172.21.15.23:0/832627921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 0x5622a3743010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fd9d50640 1 -- 172.21.15.23:0/832627921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a374b6d0 msgr2=0x5622a37490e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fd9d50640 1 --2- 172.21.15.23:0/832627921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a374b6d0 0x5622a37490e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fd9d50640 1 -- 172.21.15.23:0/832627921 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622a3643b20 con 0x5622a3742af0 2024-03-24T10:46:11.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.826+0000 7f4fdad52640 1 --2- 172.21.15.23:0/832627921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a374b6d0 0x5622a37490e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:11.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.827+0000 7f4fd9d50640 1 --2- 172.21.15.23:0/832627921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a3742af0 0x5622a3756f00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8000c00 tx=0x7f4fc8005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85bad30f4204040d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:11.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.827+0000 7f4fbbfff640 1 -- 172.21.15.23:0/832627921 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc802c5d0 con 0x5622a3742af0 2024-03-24T10:46:11.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.827+0000 7f4fbbfff640 1 -- 172.21.15.23:0/832627921 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fc80042e0 con 0x5622a3742af0 2024-03-24T10:46:11.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.827+0000 7f4fbbfff640 1 -- 172.21.15.23:0/832627921 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8005050 con 0x5622a3742af0 2024-03-24T10:46:11.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 -- 172.21.15.23:0/832627921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a3742af0 msgr2=0x5622a3756f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 --2- 172.21.15.23:0/832627921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a3742af0 0x5622a3756f00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8000c00 tx=0x7f4fc8005a20 comp rx=0 tx=0).stop 2024-03-24T10:46:11.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 -- 172.21.15.23:0/832627921 shutdown_connections 2024-03-24T10:46:11.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 --2- 172.21.15.23:0/832627921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a374b6d0 0x5622a37490e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 --2- 172.21.15.23:0/832627921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a3742af0 0x5622a3756f00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 --2- 172.21.15.23:0/832627921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 0x5622a3743010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 -- 172.21.15.23:0/832627921 >> 172.21.15.23:0/832627921 conn(0x5622a373c900 msgr2=0x5622a373bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:11.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 -- 172.21.15.23:0/832627921 shutdown_connections 2024-03-24T10:46:11.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 -- 172.21.15.23:0/832627921 wait complete. 2024-03-24T10:46:11.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 -- start start 2024-03-24T10:46:11.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 Processor -- start 2024-03-24T10:46:11.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 0x5622a374eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a374f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a374b6d0 0x5622a374f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5622a3753d70 con 0x5622a3742af0 2024-03-24T10:46:11.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5622a37c0f70 con 0x5622a37421d0 2024-03-24T10:46:11.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5622a37c10e0 con 0x5622a374b6d0 2024-03-24T10:46:11.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.828+0000 7f4fdf397740 10 client.0 _ll_register_callbacks cb 0x7f4fc8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:11.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fd9d50640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a374f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fd9d50640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a374f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56468/0 (socket says 172.21.15.23:56468) 2024-03-24T10:46:11.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fd9d50640 1 -- 172.21.15.23:0/865367765 learned_addr learned my addr 172.21.15.23:0/865367765 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:11.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fda551640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 0x5622a374eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fdad52640 1 --2- 172.21.15.23:0/865367765 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a374b6d0 0x5622a374f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fd9d50640 1 -- 172.21.15.23:0/865367765 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a374b6d0 msgr2=0x5622a374f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fd9d50640 1 --2- 172.21.15.23:0/865367765 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a374b6d0 0x5622a374f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fd9d50640 1 -- 172.21.15.23:0/865367765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 msgr2=0x5622a374eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:11.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fd9d50640 1 --2- 172.21.15.23:0/865367765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 0x5622a374eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fd9d50640 1 -- 172.21.15.23:0/865367765 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc40120d0 con 0x5622a3742af0 2024-03-24T10:46:11.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fd9d50640 1 --2- 172.21.15.23:0/865367765 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a374f2e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8000c00 tx=0x7f4fc8004b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6769aef1f9141a46 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:11.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fb9ffb640 1 -- 172.21.15.23:0/865367765 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8003c50 con 0x5622a3742af0 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fdf397740 1 -- 172.21.15.23:0/865367765 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fcc09ae20 con 0x5622a3742af0 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fb9ffb640 1 -- 172.21.15.23:0/865367765 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f4fc8000e40 con 0x5622a3742af0 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fb9ffb640 1 -- 172.21.15.23:0/865367765 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4fc80035f0 con 0x5622a3742af0 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fdf397740 10 client.17364 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.829+0000 7f4fdf397740 1 -- 172.21.15.23:0/865367765 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5622a36a7ba0 con 0x5622a3742af0 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.830+0000 7f4fb9ffb640 1 -- 172.21.15.23:0/865367765 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fcc09ae20 con 0x5622a3742af0 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.830+0000 7f4fb9ffb640 1 -- 172.21.15.23:0/865367765 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f4fc803c040 con 0x5622a3742af0 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.830+0000 7f4fdf397740 10 client.17364 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.830+0000 7f4fdf397740 10 client.17364 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.830+0000 7f4fdf397740 20 client.17364 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.830+0000 7f4fdf397740 1 --2- 172.21.15.23:0/865367765 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5622a37c2550 0x5622a37c4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.830+0000 7f4fdf397740 4 client.17364 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.830+0000 7f4fdf397740 1 -- 172.21.15.23:0/865367765 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5622a3434890 con 0x5622a37c2550 2024-03-24T10:46:11.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.830+0000 7f4fda551640 1 --2- 172.21.15.23:0/865367765 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5622a37c2550 0x5622a37c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:11.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.830+0000 7f4fda551640 1 --2- 172.21.15.23:0/865367765 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5622a37c2550 0x5622a37c4970 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=756c938600e1f76d server_cookie=b99a0d057b345fa9 in_seq=0 out_seq=0 2024-03-24T10:46:11.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.830+0000 7f4fb9ffb640 10 client.17364 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:11.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.830+0000 7f4fb9ffb640 1 -- 172.21.15.23:0/865367765 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5622a3434890 con 0x5622a37c2550 2024-03-24T10:46:11.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.830+0000 7f4fb9ffb640 10 client.17364 handle_command_reply: tid=0 2024-03-24T10:46:11.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.830+0000 7f4fdf397740 1 client.17364 shutdown 2024-03-24T10:46:11.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdf397740 1 -- 172.21.15.23:0/865367765 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 msgr2=0x5622a374f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdf397740 1 --2- 172.21.15.23:0/865367765 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a374f2e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8000c00 tx=0x7f4fc8004b10 comp rx=0 tx=0).stop 2024-03-24T10:46:11.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdf397740 1 -- 172.21.15.23:0/865367765 shutdown_connections 2024-03-24T10:46:11.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdf397740 1 --2- 172.21.15.23:0/865367765 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5622a37c2550 0x5622a37c4970 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdf397740 1 --2- 172.21.15.23:0/865367765 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5622a374b6d0 0x5622a374f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdf397740 1 --2- 172.21.15.23:0/865367765 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622a3742af0 0x5622a374f2e0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdf397740 1 --2- 172.21.15.23:0/865367765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5622a37421d0 0x5622a374eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fb9ffb640 0 client.17364 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:11.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdf397740 1 -- 172.21.15.23:0/865367765 >> 172.21.15.23:0/865367765 conn(0x5622a373c900 msgr2=0x5622a373dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:11.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdf397740 1 -- 172.21.15.23:0/865367765 shutdown_connections 2024-03-24T10:46:11.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdf397740 1 -- 172.21.15.23:0/865367765 wait complete. 2024-03-24T10:46:11.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdf397740 20 client.17364 trim_cache size 0 max 16384 2024-03-24T10:46:11.838 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:46:11.838 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:46:11.838 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:46:11.838 INFO:teuthology.orchestra.run.smithi023.stdout: "bbc26982-599f-4398-b15e-e1177a25b88a": { 2024-03-24T10:46:11.838 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:46:11.838 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "bbc26982-599f-4398-b15e-e1177a25b88a", 2024-03-24T10:46:11.839 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:46:11.839 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:46:11.839 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:46:11.839 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:46:11.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdc7b5640 1 -- 172.21.15.23:0/4034408109 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f4fbc0a2230 msgr2=0x7f4fbc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdc7b5640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f4fbc0a2230 0x7f4fbc0a46f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40049d0 tx=0x7f4fc4004ab0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdc7b5640 1 -- 172.21.15.23:0/4034408109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fd4100630 msgr2=0x7f4fd4197dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:11.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdc7b5640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fd4100630 0x7f4fd4197dc0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00e9f0 tx=0x7f4fcc00eeb0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdc7b5640 1 -- 172.21.15.23:0/4034408109 shutdown_connections 2024-03-24T10:46:11.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdc7b5640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f4fbc0a2230 0x7f4fbc0a46f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdc7b5640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fd4106640 0x7f4fd419c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdc7b5640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4fd4100ff0 0x7f4fd4198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdc7b5640 1 --2- 172.21.15.23:0/4034408109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fd4100630 0x7f4fd4197dc0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:11.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdc7b5640 1 -- 172.21.15.23:0/4034408109 >> 172.21.15.23:0/4034408109 conn(0x7f4fd4077770 msgr2=0x7f4fd40ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:11.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdc7b5640 1 -- 172.21.15.23:0/4034408109 shutdown_connections 2024-03-24T10:46:11.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:11.831+0000 7f4fdc7b5640 1 -- 172.21.15.23:0/4034408109 wait complete. 2024-03-24T10:46:11.842 INFO:tasks.cephfs.filesystem:scrub status for tag:bbc26982-599f-4398-b15e-e1177a25b88a - {'path': '/', 'tag': 'bbc26982-599f-4398-b15e-e1177a25b88a', 'options': 'recursive,force'} 2024-03-24T10:46:12.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 583758 11.32 MB/sec execute 149 sec latency 20.049 ms 2024-03-24T10:46:13.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 584403 11.27 MB/sec execute 150 sec latency 31.936 ms 2024-03-24T10:46:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:12 smithi023 ceph-mon[32466]: pgmap v187: 97 pgs: 97 active+clean; 364 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.6 MiB/s wr, 298 op/s 2024-03-24T10:46:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:12 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:46:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:12 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:12 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:46:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:12 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:46:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:12 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:13.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:12 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:13.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:12 smithi117 ceph-mon[36368]: pgmap v187: 97 pgs: 97 active+clean; 364 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.6 MiB/s wr, 298 op/s 2024-03-24T10:46:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:12 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:46:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:12 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:12 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:46:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:12 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:46:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:12 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:13.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:12 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:13.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:12 smithi116 ceph-mon[36248]: pgmap v187: 97 pgs: 97 active+clean; 364 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.6 MiB/s wr, 298 op/s 2024-03-24T10:46:13.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:12 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:46:13.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:12 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:12 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:46:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:12 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:46:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:12 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:13.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:12 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:14.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 586538 11.28 MB/sec execute 151 sec latency 16.966 ms 2024-03-24T10:46:15.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 587847 11.25 MB/sec execute 152 sec latency 128.429 ms 2024-03-24T10:46:15.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:14 smithi023 ceph-mon[32466]: pgmap v188: 97 pgs: 97 active+clean; 364 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 180 op/s 2024-03-24T10:46:15.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:14 smithi117 ceph-mon[36368]: pgmap v188: 97 pgs: 97 active+clean; 364 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 180 op/s 2024-03-24T10:46:15.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:14 smithi116 ceph-mon[36248]: pgmap v188: 97 pgs: 97 active+clean; 364 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 180 op/s 2024-03-24T10:46:16.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 590095 11.26 MB/sec execute 153 sec latency 15.138 ms 2024-03-24T10:46:16.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:15 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:46:16.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:15 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:46:16.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:15 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:46:17.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 591713 11.23 MB/sec execute 154 sec latency 28.764 ms 2024-03-24T10:46:17.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:16 smithi023 ceph-mon[32466]: pgmap v189: 97 pgs: 97 active+clean; 364 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 180 op/s 2024-03-24T10:46:17.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:16 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:46:17.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:16 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:46:17.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:16 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:46:17.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:16 smithi117 ceph-mon[36368]: pgmap v189: 97 pgs: 97 active+clean; 364 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 180 op/s 2024-03-24T10:46:17.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:16 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:46:17.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:16 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:46:17.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:16 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:46:17.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:16 smithi116 ceph-mon[36248]: pgmap v189: 97 pgs: 97 active+clean; 364 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 180 op/s 2024-03-24T10:46:17.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:16 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:46:17.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:16 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:46:17.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:16 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:46:18.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 593784 11.24 MB/sec execute 155 sec latency 14.879 ms 2024-03-24T10:46:19.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 595765 11.22 MB/sec execute 156 sec latency 24.559 ms 2024-03-24T10:46:19.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:18 smithi023 ceph-mon[32466]: pgmap v190: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 265 op/s 2024-03-24T10:46:19.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:18 smithi117 ceph-mon[36368]: pgmap v190: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 265 op/s 2024-03-24T10:46:19.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:18 smithi116 ceph-mon[36248]: pgmap v190: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 265 op/s 2024-03-24T10:46:20.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 597444 11.22 MB/sec execute 157 sec latency 63.589 ms 2024-03-24T10:46:21.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 599608 11.22 MB/sec execute 158 sec latency 19.922 ms 2024-03-24T10:46:21.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:20 smithi023 ceph-mon[32466]: pgmap v191: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 175 op/s 2024-03-24T10:46:21.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:20 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:46:21.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:20 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:46:21.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:20 smithi117 ceph-mon[36368]: pgmap v191: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 175 op/s 2024-03-24T10:46:21.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:20 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:46:21.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:20 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:46:21.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:20 smithi116 ceph-mon[36248]: pgmap v191: 97 pgs: 97 active+clean; 371 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 175 op/s 2024-03-24T10:46:21.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:20 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:46:21.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:20 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:46:22.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 601179 11.20 MB/sec execute 159 sec latency 39.584 ms 2024-03-24T10:46:22.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:22 smithi117 ceph-mon[36368]: pgmap v192: 97 pgs: 97 active+clean; 378 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 265 op/s 2024-03-24T10:46:22.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:22 smithi116 ceph-mon[36248]: pgmap v192: 97 pgs: 97 active+clean; 378 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 265 op/s 2024-03-24T10:46:22.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:22 smithi023 ceph-mon[32466]: pgmap v192: 97 pgs: 97 active+clean; 378 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 265 op/s 2024-03-24T10:46:23.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 603448 11.21 MB/sec execute 160 sec latency 17.043 ms 2024-03-24T10:46:23.677 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.675+0000 7fb9a0b1e640 1 -- 172.21.15.23:0/2349185684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb99c1076e0 msgr2=0x7fb99c107ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.675+0000 7fb9a0b1e640 1 --2- 172.21.15.23:0/2349185684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb99c1076e0 0x7fb99c107ac0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb98c009930 tx=0x7fb98c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:46:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.675+0000 7fb9a0b1e640 1 -- 172.21.15.23:0/2349185684 shutdown_connections 2024-03-24T10:46:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.675+0000 7fb9a0b1e640 1 --2- 172.21.15.23:0/2349185684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99c069750 0x7fb99c10e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.675+0000 7fb9a0b1e640 1 --2- 172.21.15.23:0/2349185684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb99c068d90 0x7fb99c069210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.675+0000 7fb9a0b1e640 1 --2- 172.21.15.23:0/2349185684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb99c1076e0 0x7fb99c107ac0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.675+0000 7fb9a0b1e640 1 -- 172.21.15.23:0/2349185684 >> 172.21.15.23:0/2349185684 conn(0x7fb99c077460 msgr2=0x7fb99c077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.675+0000 7fb9a0b1e640 1 -- 172.21.15.23:0/2349185684 shutdown_connections 2024-03-24T10:46:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.675+0000 7fb9a0b1e640 1 -- 172.21.15.23:0/2349185684 wait complete. 2024-03-24T10:46:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.676+0000 7fb9a0b1e640 1 Processor -- start 2024-03-24T10:46:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.676+0000 7fb9a0b1e640 1 -- start start 2024-03-24T10:46:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.676+0000 7fb9a0b1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99c068d90 0x7fb99c199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.676+0000 7fb9a0b1e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb99c069750 0x7fb99c19a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.676+0000 7fb9a0b1e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb99c1076e0 0x7fb99c19e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.676+0000 7fb9a0b1e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c19edc0 con 0x7fb99c1076e0 2024-03-24T10:46:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.676+0000 7fb9a0b1e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c19ef30 con 0x7fb99c068d90 2024-03-24T10:46:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.676+0000 7fb9a0b1e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c19f0f0 con 0x7fb99c069750 2024-03-24T10:46:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.676+0000 7fb99ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb99c1076e0 0x7fb99c19e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.676+0000 7fb999d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb99c069750 0x7fb99c19a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.676+0000 7fb99ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb99c1076e0 0x7fb99c19e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48962/0 (socket says 172.21.15.23:48962) 2024-03-24T10:46:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.676+0000 7fb99a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99c068d90 0x7fb99c199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.676+0000 7fb99ad9d640 1 -- 172.21.15.23:0/308311803 learned_addr learned my addr 172.21.15.23:0/308311803 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.677+0000 7fb99ad9d640 1 -- 172.21.15.23:0/308311803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb99c069750 msgr2=0x7fb99c19a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.677+0000 7fb99ad9d640 1 --2- 172.21.15.23:0/308311803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb99c069750 0x7fb99c19a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.677+0000 7fb99ad9d640 1 -- 172.21.15.23:0/308311803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99c068d90 msgr2=0x7fb99c199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.677+0000 7fb99ad9d640 1 --2- 172.21.15.23:0/308311803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99c068d90 0x7fb99c199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.677+0000 7fb99ad9d640 1 -- 172.21.15.23:0/308311803 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb98c009560 con 0x7fb99c1076e0 2024-03-24T10:46:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.677+0000 7fb99a59c640 1 --2- 172.21.15.23:0/308311803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99c068d90 0x7fb99c199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.677+0000 7fb99ad9d640 1 --2- 172.21.15.23:0/308311803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb99c1076e0 0x7fb99c19e7c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb99400ea60 tx=0x7fb99400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.677+0000 7fb98b7fe640 1 -- 172.21.15.23:0/308311803 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99400ce40 con 0x7fb99c1076e0 2024-03-24T10:46:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.677+0000 7fb9a0b1e640 1 -- 172.21.15.23:0/308311803 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb99c19f3d0 con 0x7fb99c1076e0 2024-03-24T10:46:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.677+0000 7fb98b7fe640 1 -- 172.21.15.23:0/308311803 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb994005020 con 0x7fb99c1076e0 2024-03-24T10:46:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.677+0000 7fb98b7fe640 1 -- 172.21.15.23:0/308311803 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb994010690 con 0x7fb99c1076e0 2024-03-24T10:46:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.677+0000 7fb9a0b1e640 1 -- 172.21.15.23:0/308311803 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb99c1a6c50 con 0x7fb99c1076e0 2024-03-24T10:46:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.679+0000 7fb98b7fe640 1 -- 172.21.15.23:0/308311803 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fb994005500 con 0x7fb99c1076e0 2024-03-24T10:46:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.680+0000 7fb9a37dc740 1 Processor -- start 2024-03-24T10:46:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.680+0000 7fb9a37dc740 1 -- start start 2024-03-24T10:46:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.680+0000 7fb9a37dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc65e0 0x557166dc69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.680+0000 7fb9a37dc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166dcf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.680+0000 7fb9a37dc740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dcfcd0 0x557166dd20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.680+0000 7fb9a37dc740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557166d811b0 con 0x557166dcfcd0 2024-03-24T10:46:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.680+0000 7fb9a37dc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557166d33f80 con 0x557166dc65e0 2024-03-24T10:46:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.680+0000 7fb9a37dc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557166d314b0 con 0x557166dc6f00 2024-03-24T10:46:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.680+0000 7fb98b7fe640 1 --2- 172.21.15.23:0/308311803 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb9840a22d0 0x7fb9840a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.680+0000 7fb99a59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166dcf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.680+0000 7fb999d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc65e0 0x557166dc69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.680+0000 7fb99a59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166dcf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:52312/0 (socket says 172.21.15.23:52312) 2024-03-24T10:46:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.680+0000 7fb999d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc65e0 0x557166dc69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:36306/0 (socket says 172.21.15.23:36306) 2024-03-24T10:46:23.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.680+0000 7fb99a59c640 1 -- 172.21.15.23:0/1046368567 learned_addr learned my addr 172.21.15.23:0/1046368567 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.681+0000 7fb999d9b640 1 --2- 172.21.15.23:0/1046368567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dcfcd0 0x557166dd20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.681+0000 7fb99ad9d640 1 --2- 172.21.15.23:0/308311803 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb9840a22d0 0x7fb9840a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.681+0000 7fb98b7fe640 1 -- 172.21.15.23:0/308311803 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fb994014070 con 0x7fb99c1076e0 2024-03-24T10:46:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.681+0000 7fb999d9b640 1 -- 172.21.15.23:0/1046368567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 msgr2=0x557166dcf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.681+0000 7fb999d9b640 1 --2- 172.21.15.23:0/1046368567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166dcf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.681+0000 7fb999d9b640 1 -- 172.21.15.23:0/1046368567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dcfcd0 msgr2=0x557166dd20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.681+0000 7fb999d9b640 1 --2- 172.21.15.23:0/1046368567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dcfcd0 0x557166dd20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.681+0000 7fb999d9b640 1 -- 172.21.15.23:0/1046368567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557166cc2810 con 0x557166dc65e0 2024-03-24T10:46:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.681+0000 7fb999d9b640 1 --2- 172.21.15.23:0/1046368567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc65e0 0x557166dc69c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb990009fd0 tx=0x7fb9900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2276a0702a645706 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.681+0000 7fb99ad9d640 1 --2- 172.21.15.23:0/308311803 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb9840a22d0 0x7fb9840a4790 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb99409ac00 tx=0x7fb99400ef90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.681+0000 7fb988ff9640 1 -- 172.21.15.23:0/1046368567 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990004050 con 0x557166dc65e0 2024-03-24T10:46:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.681+0000 7fb988ff9640 1 -- 172.21.15.23:0/1046368567 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb99000fc70 con 0x557166dc65e0 2024-03-24T10:46:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.681+0000 7fb988ff9640 1 -- 172.21.15.23:0/1046368567 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990003830 con 0x557166dc65e0 2024-03-24T10:46:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.682+0000 7fb9a37dc740 1 -- 172.21.15.23:0/1046368567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc65e0 msgr2=0x557166dc69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.682+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/1046368567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc65e0 0x557166dc69c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb990009fd0 tx=0x7fb9900046c0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.682+0000 7fb9a37dc740 1 -- 172.21.15.23:0/1046368567 shutdown_connections 2024-03-24T10:46:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.682+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/1046368567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dcfcd0 0x557166dd20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.682+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/1046368567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166dcf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.682+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/1046368567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc65e0 0x557166dc69c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.682+0000 7fb9a37dc740 1 -- 172.21.15.23:0/1046368567 >> 172.21.15.23:0/1046368567 conn(0x557166dc0f00 msgr2=0x557166dc12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.682+0000 7fb9a37dc740 1 -- 172.21.15.23:0/1046368567 shutdown_connections 2024-03-24T10:46:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.682+0000 7fb9a37dc740 1 -- 172.21.15.23:0/1046368567 wait complete. 2024-03-24T10:46:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.682+0000 7fb9a37dc740 1 -- start start 2024-03-24T10:46:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.682+0000 7fb9a37dc740 1 Processor -- start 2024-03-24T10:46:23.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb9a37dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc65e0 0x557166dccd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb9a37dc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166dcd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb9a37dc740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dcfcd0 0x557166dcd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb9a37dc740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557166d2cc90 con 0x557166dcfcd0 2024-03-24T10:46:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb9a37dc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557166cd1c50 con 0x557166dc65e0 2024-03-24T10:46:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb9a37dc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557166a00250 con 0x557166dc6f00 2024-03-24T10:46:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb9a37dc740 10 client.0 _ll_register_callbacks cb 0x557166d20290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb999d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dcfcd0 0x557166dcd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb999d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dcfcd0 0x557166dcd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48984/0 (socket says 172.21.15.23:48984) 2024-03-24T10:46:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb999d9b640 1 -- 172.21.15.23:0/1485155816 learned_addr learned my addr 172.21.15.23:0/1485155816 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb99a59c640 1 --2- 172.21.15.23:0/1485155816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166dcd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb999d9b640 1 --2- 172.21.15.23:0/1485155816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc65e0 0x557166dccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb99a59c640 1 -- 172.21.15.23:0/1485155816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc65e0 msgr2=0x557166dccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb99a59c640 1 --2- 172.21.15.23:0/1485155816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc65e0 0x557166dccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb99a59c640 1 -- 172.21.15.23:0/1485155816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dcfcd0 msgr2=0x557166dcd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb99a59c640 1 --2- 172.21.15.23:0/1485155816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dcfcd0 0x557166dcd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb99a59c640 1 -- 172.21.15.23:0/1485155816 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb990019040 con 0x557166dc6f00 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb999d9b640 1 --2- 172.21.15.23:0/1485155816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dcfcd0 0x557166dcd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb99a59c640 1 --2- 172.21.15.23:0/1485155816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166dcd290 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb98c005e00 tx=0x7fb98c02cd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c23b736c1142f8f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb981ffb640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb981ffb640 1 -- 172.21.15.23:0/1485155816 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c004350 con 0x557166dc6f00 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb981ffb640 1 -- 172.21.15.23:0/1485155816 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fb98c0044b0 con 0x557166dc6f00 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.683+0000 7fb9a37dc740 1 -- 172.21.15.23:0/1485155816 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557166d51790 con 0x557166dc6f00 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.684+0000 7fb981ffb640 1 -- 172.21.15.23:0/1485155816 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb98c004df0 con 0x557166dc6f00 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.684+0000 7fb9a37dc740 10 client.27235 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.684+0000 7fb9a37dc740 1 -- 172.21.15.23:0/1485155816 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557166d2b310 con 0x557166dc6f00 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.684+0000 7fb981ffb640 1 -- 172.21.15.23:0/1485155816 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557166d51790 con 0x557166dc6f00 2024-03-24T10:46:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.684+0000 7fb981ffb640 1 -- 172.21.15.23:0/1485155816 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fb98c050020 con 0x557166dc6f00 2024-03-24T10:46:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.684+0000 7fb9a37dc740 10 client.27235 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.684+0000 7fb9a37dc740 10 client.27235 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.684+0000 7fb9a37dc740 20 client.27235 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.684+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/1485155816 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557166ddfc20 0x557166de2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.684+0000 7fb9a37dc740 4 client.27235 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.684+0000 7fb9a37dc740 1 -- 172.21.15.23:0/1485155816 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557166a2c600 con 0x557166ddfc20 2024-03-24T10:46:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.684+0000 7fb999d9b640 1 --2- 172.21.15.23:0/1485155816 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557166ddfc20 0x557166de2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.685+0000 7fb981ffb640 10 client.27235 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.685+0000 7fb999d9b640 1 --2- 172.21.15.23:0/1485155816 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557166ddfc20 0x557166de2040 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e52cea463ae14772 server_cookie=873fd74c43a2763e in_seq=0 out_seq=0 2024-03-24T10:46:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.685+0000 7fb981ffb640 1 -- 172.21.15.23:0/1485155816 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557166a2c600 con 0x557166ddfc20 2024-03-24T10:46:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.685+0000 7fb981ffb640 10 client.27235 handle_command_reply: tid=0 2024-03-24T10:46:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.685+0000 7fb9a37dc740 1 client.27235 shutdown 2024-03-24T10:46:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.685+0000 7fb9a37dc740 1 -- 172.21.15.23:0/1485155816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 msgr2=0x557166dcd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.685+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/1485155816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166dcd290 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb98c005e00 tx=0x7fb98c02cd40 comp rx=0 tx=0).stop 2024-03-24T10:46:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.685+0000 7fb9a37dc740 1 -- 172.21.15.23:0/1485155816 shutdown_connections 2024-03-24T10:46:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.685+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/1485155816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dcfcd0 0x557166dcd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.685+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/1485155816 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557166ddfc20 0x557166de2040 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.685+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/1485155816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166dcd290 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.685+0000 7fb981ffb640 0 client.27235 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.685+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/1485155816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc65e0 0x557166dccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.685+0000 7fb9a37dc740 1 -- 172.21.15.23:0/1485155816 >> 172.21.15.23:0/1485155816 conn(0x557166dc0f00 msgr2=0x557166dc2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.686+0000 7fb9a37dc740 1 -- 172.21.15.23:0/1485155816 shutdown_connections 2024-03-24T10:46:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.686+0000 7fb9a37dc740 1 -- 172.21.15.23:0/1485155816 wait complete. 2024-03-24T10:46:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.686+0000 7fb9a37dc740 20 client.27235 trim_cache size 0 max 16384 2024-03-24T10:46:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.700+0000 7fb9a37dc740 1 Processor -- start 2024-03-24T10:46:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.700+0000 7fb9a37dc740 1 -- start start 2024-03-24T10:46:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.700+0000 7fb9a37dc740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dc65e0 0x557166dcf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.700+0000 7fb9a37dc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166ddb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.700+0000 7fb9a37dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dcfcd0 0x557166dcd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.700+0000 7fb9a37dc740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557166cb2660 con 0x557166dc65e0 2024-03-24T10:46:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.700+0000 7fb9a37dc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557166de26a0 con 0x557166dcfcd0 2024-03-24T10:46:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.700+0000 7fb9a37dc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557166dbc6c0 con 0x557166dc6f00 2024-03-24T10:46:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb999d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dc65e0 0x557166dcf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb999d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dc65e0 0x557166dcf480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48996/0 (socket says 172.21.15.23:48996) 2024-03-24T10:46:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb999d9b640 1 -- 172.21.15.23:0/3418131520 learned_addr learned my addr 172.21.15.23:0/3418131520 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb99a59c640 1 --2- 172.21.15.23:0/3418131520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166ddb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb999d9b640 1 --2- 172.21.15.23:0/3418131520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dcfcd0 0x557166dcd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb999d9b640 1 -- 172.21.15.23:0/3418131520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 msgr2=0x557166ddb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb999d9b640 1 --2- 172.21.15.23:0/3418131520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166ddb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb999d9b640 1 -- 172.21.15.23:0/3418131520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dcfcd0 msgr2=0x557166dcd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb999d9b640 1 --2- 172.21.15.23:0/3418131520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dcfcd0 0x557166dcd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb999d9b640 1 -- 172.21.15.23:0/3418131520 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557166cc7810 con 0x557166dc65e0 2024-03-24T10:46:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb99a59c640 1 --2- 172.21.15.23:0/3418131520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166ddb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb999d9b640 1 --2- 172.21.15.23:0/3418131520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dc65e0 0x557166dcf480 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb9900058c0 tx=0x7fb990005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8e6f08e4008234 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb983fff640 1 -- 172.21.15.23:0/3418131520 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990011110 con 0x557166dc65e0 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb983fff640 1 -- 172.21.15.23:0/3418131520 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9900055f0 con 0x557166dc65e0 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.701+0000 7fb983fff640 1 -- 172.21.15.23:0/3418131520 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990032040 con 0x557166dc65e0 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 -- 172.21.15.23:0/3418131520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dc65e0 msgr2=0x557166dcf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/3418131520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dc65e0 0x557166dcf480 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb9900058c0 tx=0x7fb990005f90 comp rx=0 tx=0).stop 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 -- 172.21.15.23:0/3418131520 shutdown_connections 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/3418131520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dcfcd0 0x557166dcd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/3418131520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dc6f00 0x557166ddb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/3418131520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dc65e0 0x557166dcf480 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 -- 172.21.15.23:0/3418131520 >> 172.21.15.23:0/3418131520 conn(0x557166dc0f00 msgr2=0x557166dc8050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 -- 172.21.15.23:0/3418131520 shutdown_connections 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 -- 172.21.15.23:0/3418131520 wait complete. 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 -- start start 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 Processor -- start 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dc65e0 0x557166dd79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc6f00 0x557166dc9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dcfcd0 0x557166dc9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557166dd7f10 con 0x557166dc65e0 2024-03-24T10:46:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557166dd8080 con 0x557166dc6f00 2024-03-24T10:46:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557166dd8240 con 0x557166dcfcd0 2024-03-24T10:46:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb9a37dc740 10 client.0 _ll_register_callbacks cb 0x7fb98c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb999d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dc65e0 0x557166dd79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb999d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dc65e0 0x557166dd79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49002/0 (socket says 172.21.15.23:49002) 2024-03-24T10:46:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb999d9b640 1 -- 172.21.15.23:0/826595470 learned_addr learned my addr 172.21.15.23:0/826595470 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.702+0000 7fb99a59c640 1 --2- 172.21.15.23:0/826595470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc6f00 0x557166dc9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb999d9b640 1 --2- 172.21.15.23:0/826595470 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dcfcd0 0x557166dc9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb999d9b640 1 -- 172.21.15.23:0/826595470 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dcfcd0 msgr2=0x557166dc9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb999d9b640 1 --2- 172.21.15.23:0/826595470 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dcfcd0 0x557166dc9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb999d9b640 1 -- 172.21.15.23:0/826595470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc6f00 msgr2=0x557166dc9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb999d9b640 1 --2- 172.21.15.23:0/826595470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc6f00 0x557166dc9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb999d9b640 1 -- 172.21.15.23:0/826595470 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557166d2b310 con 0x557166dc65e0 2024-03-24T10:46:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb99a59c640 1 --2- 172.21.15.23:0/826595470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc6f00 0x557166dc9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb999d9b640 1 --2- 172.21.15.23:0/826595470 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dc65e0 0x557166dd79d0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb9900057c0 tx=0x7fb990003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96f1563121136ba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb981ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb981ffb640 1 -- 172.21.15.23:0/826595470 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990011110 con 0x557166dc65e0 2024-03-24T10:46:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb981ffb640 1 -- 172.21.15.23:0/826595470 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fb990011270 con 0x557166dc65e0 2024-03-24T10:46:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb9a37dc740 1 -- 172.21.15.23:0/826595470 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb99000fc70 con 0x557166dc65e0 2024-03-24T10:46:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb981ffb640 1 -- 172.21.15.23:0/826595470 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb9900037b0 con 0x557166dc65e0 2024-03-24T10:46:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb9a37dc740 10 client.17382 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb9a37dc740 1 -- 172.21.15.23:0/826595470 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557166cc2810 con 0x557166dc65e0 2024-03-24T10:46:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.703+0000 7fb981ffb640 1 -- 172.21.15.23:0/826595470 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb99000fc70 con 0x557166dc65e0 2024-03-24T10:46:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.704+0000 7fb981ffb640 1 -- 172.21.15.23:0/826595470 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fb9900323b0 con 0x557166dc65e0 2024-03-24T10:46:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.704+0000 7fb9a37dc740 10 client.17382 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.704+0000 7fb9a37dc740 10 client.17382 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:23.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.704+0000 7fb9a37dc740 20 client.17382 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.704+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/826595470 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557166e45cd0 0x557166e480f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.704+0000 7fb9a37dc740 4 client.17382 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:46:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.704+0000 7fb9a37dc740 1 -- 172.21.15.23:0/826595470 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557166a7d1b0 con 0x557166e45cd0 2024-03-24T10:46:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.704+0000 7fb99a59c640 1 --2- 172.21.15.23:0/826595470 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557166e45cd0 0x557166e480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.704+0000 7fb99a59c640 1 --2- 172.21.15.23:0/826595470 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557166e45cd0 0x557166e480f0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82148ab91cef1573 server_cookie=69cd61c5ae68085c in_seq=0 out_seq=0 2024-03-24T10:46:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.704+0000 7fb981ffb640 10 client.17382 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb981ffb640 1 -- 172.21.15.23:0/826595470 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557166a7d1b0 con 0x557166e45cd0 2024-03-24T10:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb981ffb640 10 client.17382 handle_command_reply: tid=0 2024-03-24T10:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a37dc740 1 client.17382 shutdown 2024-03-24T10:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a37dc740 1 -- 172.21.15.23:0/826595470 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dc65e0 msgr2=0x557166dd79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/826595470 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dc65e0 0x557166dd79d0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb9900057c0 tx=0x7fb990003c80 comp rx=0 tx=0).stop 2024-03-24T10:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a37dc740 1 -- 172.21.15.23:0/826595470 shutdown_connections 2024-03-24T10:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/826595470 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557166e45cd0 0x557166e480f0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/826595470 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557166dcfcd0 0x557166dc9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb981ffb640 0 client.17382 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/826595470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557166dc6f00 0x557166dc9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a37dc740 1 --2- 172.21.15.23:0/826595470 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557166dc65e0 0x557166dd79d0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a37dc740 1 -- 172.21.15.23:0/826595470 >> 172.21.15.23:0/826595470 conn(0x557166dc0f00 msgr2=0x557166dbf610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a37dc740 1 -- 172.21.15.23:0/826595470 shutdown_connections 2024-03-24T10:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a37dc740 1 -- 172.21.15.23:0/826595470 wait complete. 2024-03-24T10:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a37dc740 20 client.17382 trim_cache size 0 max 16384 2024-03-24T10:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:46:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:46:23.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a0b1e640 1 -- 172.21.15.23:0/308311803 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb9840a22d0 msgr2=0x7fb9840a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:23.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a0b1e640 1 --2- 172.21.15.23:0/308311803 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb9840a22d0 0x7fb9840a4790 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb99409ac00 tx=0x7fb99400ef90 comp rx=0 tx=0).stop 2024-03-24T10:46:23.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a0b1e640 1 -- 172.21.15.23:0/308311803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb99c1076e0 msgr2=0x7fb99c19e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:23.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.715+0000 7fb9a0b1e640 1 --2- 172.21.15.23:0/308311803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb99c1076e0 0x7fb99c19e7c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fb99400ea60 tx=0x7fb99400ef20 comp rx=0 tx=0).stop 2024-03-24T10:46:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.716+0000 7fb9a0b1e640 1 -- 172.21.15.23:0/308311803 shutdown_connections 2024-03-24T10:46:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.716+0000 7fb9a0b1e640 1 --2- 172.21.15.23:0/308311803 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb9840a22d0 0x7fb9840a4790 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.716+0000 7fb9a0b1e640 1 --2- 172.21.15.23:0/308311803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb99c1076e0 0x7fb99c19e7c0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.716+0000 7fb9a0b1e640 1 --2- 172.21.15.23:0/308311803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb99c069750 0x7fb99c19a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.716+0000 7fb9a0b1e640 1 --2- 172.21.15.23:0/308311803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99c068d90 0x7fb99c199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.716+0000 7fb9a0b1e640 1 -- 172.21.15.23:0/308311803 >> 172.21.15.23:0/308311803 conn(0x7fb99c077460 msgr2=0x7fb99c10cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.716+0000 7fb9a0b1e640 1 -- 172.21.15.23:0/308311803 shutdown_connections 2024-03-24T10:46:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:23.716+0000 7fb9a0b1e640 1 -- 172.21.15.23:0/308311803 wait complete. 2024-03-24T10:46:24.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 605238 11.21 MB/sec execute 161 sec latency 17.632 ms 2024-03-24T10:46:24.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:24 smithi117 ceph-mon[36368]: pgmap v193: 97 pgs: 97 active+clean; 378 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 175 op/s 2024-03-24T10:46:24.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:24 smithi116 ceph-mon[36248]: pgmap v193: 97 pgs: 97 active+clean; 378 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 175 op/s 2024-03-24T10:46:24.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:24 smithi023 ceph-mon[32466]: pgmap v193: 97 pgs: 97 active+clean; 378 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 175 op/s 2024-03-24T10:46:25.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 607178 11.20 MB/sec execute 162 sec latency 70.176 ms 2024-03-24T10:46:26.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 608779 11.19 MB/sec execute 163 sec latency 32.423 ms 2024-03-24T10:46:26.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:26 smithi117 ceph-mon[36368]: pgmap v194: 97 pgs: 97 active+clean; 378 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 175 op/s 2024-03-24T10:46:26.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:26 smithi116 ceph-mon[36248]: pgmap v194: 97 pgs: 97 active+clean; 378 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 175 op/s 2024-03-24T10:46:26.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:26 smithi023 ceph-mon[32466]: pgmap v194: 97 pgs: 97 active+clean; 378 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 175 op/s 2024-03-24T10:46:27.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 611136 11.20 MB/sec execute 164 sec latency 16.121 ms 2024-03-24T10:46:28.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 612453 11.17 MB/sec execute 165 sec latency 36.839 ms 2024-03-24T10:46:28.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:28 smithi117 ceph-mon[36368]: pgmap v195: 97 pgs: 97 active+clean; 385 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 264 op/s 2024-03-24T10:46:28.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:28 smithi116 ceph-mon[36248]: pgmap v195: 97 pgs: 97 active+clean; 385 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 264 op/s 2024-03-24T10:46:28.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:28 smithi023 ceph-mon[32466]: pgmap v195: 97 pgs: 97 active+clean; 385 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 264 op/s 2024-03-24T10:46:29.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 614707 11.18 MB/sec execute 166 sec latency 16.924 ms 2024-03-24T10:46:30.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 616086 11.15 MB/sec execute 167 sec latency 22.432 ms 2024-03-24T10:46:30.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:30 smithi117 ceph-mon[36368]: pgmap v196: 97 pgs: 97 active+clean; 385 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 179 op/s 2024-03-24T10:46:30.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:30 smithi116 ceph-mon[36248]: pgmap v196: 97 pgs: 97 active+clean; 385 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 179 op/s 2024-03-24T10:46:30.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:30 smithi023 ceph-mon[32466]: pgmap v196: 97 pgs: 97 active+clean; 385 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 179 op/s 2024-03-24T10:46:31.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 618230 11.17 MB/sec execute 168 sec latency 98.282 ms 2024-03-24T10:46:32.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 619632 11.13 MB/sec execute 169 sec latency 31.168 ms 2024-03-24T10:46:32.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:32 smithi117 ceph-mon[36368]: pgmap v197: 97 pgs: 97 active+clean; 391 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 267 op/s 2024-03-24T10:46:32.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:32 smithi116 ceph-mon[36248]: pgmap v197: 97 pgs: 97 active+clean; 391 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 267 op/s 2024-03-24T10:46:32.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:32 smithi023 ceph-mon[32466]: pgmap v197: 97 pgs: 97 active+clean; 391 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 267 op/s 2024-03-24T10:46:33.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 621938 11.15 MB/sec execute 170 sec latency 16.244 ms 2024-03-24T10:46:34.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 623618 11.12 MB/sec execute 171 sec latency 20.875 ms 2024-03-24T10:46:34.719 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:34 smithi023 ceph-mon[32466]: pgmap v198: 97 pgs: 97 active+clean; 391 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-03-24T10:46:34.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:34 smithi117 ceph-mon[36368]: pgmap v198: 97 pgs: 97 active+clean; 391 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-03-24T10:46:34.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:34 smithi116 ceph-mon[36248]: pgmap v198: 97 pgs: 97 active+clean; 391 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-03-24T10:46:35.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 625621 11.13 MB/sec execute 172 sec latency 16.126 ms 2024-03-24T10:46:36.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 626841 11.10 MB/sec execute 173 sec latency 138.703 ms 2024-03-24T10:46:36.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:36 smithi117 ceph-mon[36368]: pgmap v199: 97 pgs: 97 active+clean; 391 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-03-24T10:46:36.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:36 smithi116 ceph-mon[36248]: pgmap v199: 97 pgs: 97 active+clean; 391 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-03-24T10:46:36.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.804+0000 7fe628faf640 1 -- 172.21.15.23:0/2039689620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6241020a0 msgr2=0x7fe62410e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.804+0000 7fe628faf640 1 --2- 172.21.15.23:0/2039689620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6241020a0 0x7fe62410e5b0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009cd0 tx=0x7fe61c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.804+0000 7fe628faf640 1 -- 172.21.15.23:0/2039689620 shutdown_connections 2024-03-24T10:46:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.804+0000 7fe628faf640 1 --2- 172.21.15.23:0/2039689620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6241020a0 0x7fe62410e5b0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.804+0000 7fe628faf640 1 --2- 172.21.15.23:0/2039689620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6241016e0 0x7fe624101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.804+0000 7fe628faf640 1 --2- 172.21.15.23:0/2039689620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6241076f0 0x7fe624107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.804+0000 7fe628faf640 1 -- 172.21.15.23:0/2039689620 >> 172.21.15.23:0/2039689620 conn(0x7fe6240fd210 msgr2=0x7fe6240ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.804+0000 7fe628faf640 1 -- 172.21.15.23:0/2039689620 shutdown_connections 2024-03-24T10:46:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.804+0000 7fe628faf640 1 -- 172.21.15.23:0/2039689620 wait complete. 2024-03-24T10:46:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.805+0000 7fe628faf640 1 Processor -- start 2024-03-24T10:46:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.805+0000 7fe628faf640 1 -- start start 2024-03-24T10:46:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.805+0000 7fe628faf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6241016e0 0x7fe62419d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.805+0000 7fe628faf640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6241020a0 0x7fe62419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.805+0000 7fe628faf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6241076f0 0x7fe624197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.805+0000 7fe628faf640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe624198150 con 0x7fe6241020a0 2024-03-24T10:46:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.805+0000 7fe628faf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe6241982c0 con 0x7fe6241016e0 2024-03-24T10:46:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.805+0000 7fe628faf640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe624198480 con 0x7fe6241076f0 2024-03-24T10:46:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.805+0000 7fe621d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6241020a0 0x7fe62419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.805+0000 7fe621d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6241020a0 0x7fe62419de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32998/0 (socket says 172.21.15.23:32998) 2024-03-24T10:46:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.805+0000 7fe621d9b640 1 -- 172.21.15.23:0/4266522255 learned_addr learned my addr 172.21.15.23:0/4266522255 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.805+0000 7fe62259c640 1 --2- 172.21.15.23:0/4266522255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6241016e0 0x7fe62419d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.805+0000 7fe622d9d640 1 --2- 172.21.15.23:0/4266522255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6241076f0 0x7fe624197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.806+0000 7fe621d9b640 1 -- 172.21.15.23:0/4266522255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6241076f0 msgr2=0x7fe624197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.806+0000 7fe621d9b640 1 --2- 172.21.15.23:0/4266522255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6241076f0 0x7fe624197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.806+0000 7fe621d9b640 1 -- 172.21.15.23:0/4266522255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6241016e0 msgr2=0x7fe62419d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.806+0000 7fe621d9b640 1 --2- 172.21.15.23:0/4266522255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6241016e0 0x7fe62419d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.806+0000 7fe621d9b640 1 -- 172.21.15.23:0/4266522255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe61c009900 con 0x7fe6241020a0 2024-03-24T10:46:36.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.806+0000 7fe621d9b640 1 --2- 172.21.15.23:0/4266522255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6241020a0 0x7fe62419de40 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe61800e9f0 tx=0x7fe61800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:36.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.806+0000 7fe6137fe640 1 -- 172.21.15.23:0/4266522255 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe618009700 con 0x7fe6241020a0 2024-03-24T10:46:36.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.806+0000 7fe628faf640 1 -- 172.21.15.23:0/4266522255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe624198760 con 0x7fe6241020a0 2024-03-24T10:46:36.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.806+0000 7fe6137fe640 1 -- 172.21.15.23:0/4266522255 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe61800cd80 con 0x7fe6241020a0 2024-03-24T10:46:36.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.806+0000 7fe6137fe640 1 -- 172.21.15.23:0/4266522255 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe618018920 con 0x7fe6241020a0 2024-03-24T10:46:36.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.806+0000 7fe628faf640 1 -- 172.21.15.23:0/4266522255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6241a49a0 con 0x7fe6241020a0 2024-03-24T10:46:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.808+0000 7fe6137fe640 1 -- 172.21.15.23:0/4266522255 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fe618018a80 con 0x7fe6241020a0 2024-03-24T10:46:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe6137fe640 1 --2- 172.21.15.23:0/4266522255 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe60c0a22d0 0x7fe60c0a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe62bb8e740 1 Processor -- start 2024-03-24T10:46:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe62259c640 1 --2- 172.21.15.23:0/4266522255 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe60c0a22d0 0x7fe60c0a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe6137fe640 1 -- 172.21.15.23:0/4266522255 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fe618014070 con 0x7fe6241020a0 2024-03-24T10:46:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe62bb8e740 1 -- start start 2024-03-24T10:46:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe62bb8e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce95e0 0x56104dce99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe62bb8e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dce9f00 0x56104dcf2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe62bb8e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dcf2cd0 0x56104dcf50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe62bb8e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56104dca41b0 con 0x56104dce95e0 2024-03-24T10:46:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe62bb8e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56104dc56f80 con 0x56104dce9f00 2024-03-24T10:46:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe62bb8e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56104dc544b0 con 0x56104dcf2cd0 2024-03-24T10:46:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe621d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dce9f00 0x56104dcf2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe622d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dcf2cd0 0x56104dcf50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe622d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dcf2cd0 0x56104dcf50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:50130/0 (socket says 172.21.15.23:50130) 2024-03-24T10:46:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe622d9d640 1 -- 172.21.15.23:0/3832510294 learned_addr learned my addr 172.21.15.23:0/3832510294 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe62259c640 1 --2- 172.21.15.23:0/3832510294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce95e0 0x56104dce99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.809+0000 7fe62259c640 1 --2- 172.21.15.23:0/4266522255 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe60c0a22d0 0x7fe60c0a4790 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe6240ff990 tx=0x7fe614005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.810+0000 7fe622d9d640 1 -- 172.21.15.23:0/3832510294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dce9f00 msgr2=0x56104dcf2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.810+0000 7fe622d9d640 1 --2- 172.21.15.23:0/3832510294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dce9f00 0x56104dcf2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.810+0000 7fe622d9d640 1 -- 172.21.15.23:0/3832510294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce95e0 msgr2=0x56104dce99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.810+0000 7fe622d9d640 1 --2- 172.21.15.23:0/3832510294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce95e0 0x56104dce99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.810+0000 7fe622d9d640 1 -- 172.21.15.23:0/3832510294 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56104dbe5810 con 0x56104dcf2cd0 2024-03-24T10:46:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.810+0000 7fe622d9d640 1 --2- 172.21.15.23:0/3832510294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dcf2cd0 0x56104dcf50c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe61c02c700 tx=0x7fe61c032d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94f3cb6ccf93f8d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.810+0000 7fe610ff9640 1 -- 172.21.15.23:0/3832510294 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c034440 con 0x56104dcf2cd0 2024-03-24T10:46:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.810+0000 7fe610ff9640 1 -- 172.21.15.23:0/3832510294 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe61c0345a0 con 0x56104dcf2cd0 2024-03-24T10:46:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.810+0000 7fe610ff9640 1 -- 172.21.15.23:0/3832510294 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c003bd0 con 0x56104dcf2cd0 2024-03-24T10:46:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 -- 172.21.15.23:0/3832510294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dcf2cd0 msgr2=0x56104dcf50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/3832510294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dcf2cd0 0x56104dcf50c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe61c02c700 tx=0x7fe61c032d70 comp rx=0 tx=0).stop 2024-03-24T10:46:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 -- 172.21.15.23:0/3832510294 shutdown_connections 2024-03-24T10:46:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/3832510294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dcf2cd0 0x56104dcf50c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/3832510294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dce9f00 0x56104dcf2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/3832510294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce95e0 0x56104dce99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 -- 172.21.15.23:0/3832510294 >> 172.21.15.23:0/3832510294 conn(0x56104dce3f00 msgr2=0x56104dce42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 -- 172.21.15.23:0/3832510294 shutdown_connections 2024-03-24T10:46:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 -- 172.21.15.23:0/3832510294 wait complete. 2024-03-24T10:46:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 -- start start 2024-03-24T10:46:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 Processor -- start 2024-03-24T10:46:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce95e0 0x56104dcefcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce9f00 0x56104dcf0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 0x56104dcf0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56104dc4fc90 con 0x56104dce9f00 2024-03-24T10:46:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56104dbf4c50 con 0x56104dcf2cd0 2024-03-24T10:46:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.811+0000 7fe62bb8e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56104d923250 con 0x56104dce95e0 2024-03-24T10:46:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe62bb8e740 10 client.0 _ll_register_callbacks cb 0x56104dc43290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe621d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce9f00 0x56104dcf0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe62259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce95e0 0x56104dcefcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe621d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce9f00 0x56104dcf0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33018/0 (socket says 172.21.15.23:33018) 2024-03-24T10:46:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe621d9b640 1 -- 172.21.15.23:0/385627268 learned_addr learned my addr 172.21.15.23:0/385627268 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe622d9d640 1 --2- 172.21.15.23:0/385627268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 0x56104dcf0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe621d9b640 1 -- 172.21.15.23:0/385627268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce95e0 msgr2=0x56104dcefcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe621d9b640 1 --2- 172.21.15.23:0/385627268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce95e0 0x56104dcefcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe621d9b640 1 -- 172.21.15.23:0/385627268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 msgr2=0x56104dcf0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe621d9b640 1 --2- 172.21.15.23:0/385627268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 0x56104dcf0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe621d9b640 1 -- 172.21.15.23:0/385627268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe62410eeb0 con 0x56104dce9f00 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe62259c640 1 --2- 172.21.15.23:0/385627268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce95e0 0x56104dcefcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe621d9b640 1 --2- 172.21.15.23:0/385627268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce9f00 0x56104dcf0230 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe61809abc0 tx=0x7fe61809ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=907b66ff6962e41a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe609ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe609ffb640 1 -- 172.21.15.23:0/385627268 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe618010c40 con 0x56104dce9f00 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe609ffb640 1 -- 172.21.15.23:0/385627268 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fe61809a310 con 0x56104dce9f00 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe62bb8e740 1 -- 172.21.15.23:0/385627268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56104dc74790 con 0x56104dce9f00 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe609ffb640 1 -- 172.21.15.23:0/385627268 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe6180b9300 con 0x56104dce9f00 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe62bb8e740 10 client.17394 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.812+0000 7fe62bb8e740 1 -- 172.21.15.23:0/385627268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56104dc4e310 con 0x56104dce9f00 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.813+0000 7fe609ffb640 1 -- 172.21.15.23:0/385627268 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6180a7080 con 0x56104dce9f00 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.813+0000 7fe609ffb640 1 -- 172.21.15.23:0/385627268 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fe6180b9020 con 0x56104dce9f00 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.813+0000 7fe62bb8e740 10 client.17394 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.813+0000 7fe62bb8e740 10 client.17394 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.813+0000 7fe62bb8e740 20 client.17394 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.813+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/385627268 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56104dd02c00 0x56104dd05010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.813+0000 7fe62bb8e740 4 client.17394 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.813+0000 7fe62bb8e740 1 -- 172.21.15.23:0/385627268 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56104d94f600 con 0x56104dd02c00 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.813+0000 7fe62259c640 1 --2- 172.21.15.23:0/385627268 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56104dd02c00 0x56104dd05010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.813+0000 7fe62259c640 1 --2- 172.21.15.23:0/385627268 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56104dd02c00 0x56104dd05010 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=927f6e590e4fd28a server_cookie=e2b41e58b39bc9c2 in_seq=0 out_seq=0 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.813+0000 7fe609ffb640 10 client.17394 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe609ffb640 1 -- 172.21.15.23:0/385627268 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56104d94f600 con 0x56104dd02c00 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe609ffb640 10 client.17394 handle_command_reply: tid=0 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe62bb8e740 1 client.17394 shutdown 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe62bb8e740 1 -- 172.21.15.23:0/385627268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce9f00 msgr2=0x56104dcf0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/385627268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce9f00 0x56104dcf0230 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe61809abc0 tx=0x7fe61809ef60 comp rx=0 tx=0).stop 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe62bb8e740 1 -- 172.21.15.23:0/385627268 shutdown_connections 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/385627268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 0x56104dcf0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/385627268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce9f00 0x56104dcf0230 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/385627268 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56104dd02c00 0x56104dd05010 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/385627268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce95e0 0x56104dcefcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe609ffb640 0 client.17394 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe62bb8e740 1 -- 172.21.15.23:0/385627268 >> 172.21.15.23:0/385627268 conn(0x56104dce3f00 msgr2=0x56104dce5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe62bb8e740 1 -- 172.21.15.23:0/385627268 shutdown_connections 2024-03-24T10:46:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe62bb8e740 1 -- 172.21.15.23:0/385627268 wait complete. 2024-03-24T10:46:36.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.814+0000 7fe62bb8e740 20 client.17394 trim_cache size 0 max 16384 2024-03-24T10:46:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62bb8e740 1 Processor -- start 2024-03-24T10:46:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62bb8e740 1 -- start start 2024-03-24T10:46:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62bb8e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce95e0 0x56104dcfe340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62bb8e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce9f00 0x56104dcf0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62bb8e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 0x56104dce5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62bb8e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56104dbd5660 con 0x56104dce9f00 2024-03-24T10:46:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62bb8e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56104dce50c0 con 0x56104dcf2cd0 2024-03-24T10:46:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62bb8e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56104dd05690 con 0x56104dce95e0 2024-03-24T10:46:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe621d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce9f00 0x56104dcf0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce95e0 0x56104dcfe340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe621d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce9f00 0x56104dcf0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33030/0 (socket says 172.21.15.23:33030) 2024-03-24T10:46:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce95e0 0x56104dcfe340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:50140/0 (socket says 172.21.15.23:50140) 2024-03-24T10:46:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe621d9b640 1 -- 172.21.15.23:0/609633094 learned_addr learned my addr 172.21.15.23:0/609633094 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe622d9d640 1 --2- 172.21.15.23:0/609633094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 0x56104dce5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62259c640 1 -- 172.21.15.23:0/609633094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 msgr2=0x56104dce5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62259c640 1 --2- 172.21.15.23:0/609633094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 0x56104dce5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62259c640 1 -- 172.21.15.23:0/609633094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce9f00 msgr2=0x56104dcf0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62259c640 1 --2- 172.21.15.23:0/609633094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce9f00 0x56104dcf0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62259c640 1 -- 172.21.15.23:0/609633094 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56104dbea810 con 0x56104dce95e0 2024-03-24T10:46:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe621d9b640 1 --2- 172.21.15.23:0/609633094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce9f00 0x56104dcf0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.830+0000 7fe62259c640 1 --2- 172.21.15.23:0/609633094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce95e0 0x56104dcfe340 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x56104dd05860 tx=0x7fe614014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=848301d4d7d09864 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.831+0000 7fe60bfff640 1 -- 172.21.15.23:0/609633094 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe614024040 con 0x56104dce95e0 2024-03-24T10:46:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.831+0000 7fe60bfff640 1 -- 172.21.15.23:0/609633094 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe61402ad90 con 0x56104dce95e0 2024-03-24T10:46:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.831+0000 7fe60bfff640 1 -- 172.21.15.23:0/609633094 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61400e3b0 con 0x56104dce95e0 2024-03-24T10:46:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.831+0000 7fe62bb8e740 1 -- 172.21.15.23:0/609633094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce95e0 msgr2=0x56104dcfe340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:36.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.831+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/609633094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce95e0 0x56104dcfe340 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x56104dd05860 tx=0x7fe614014000 comp rx=0 tx=0).stop 2024-03-24T10:46:36.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.831+0000 7fe62bb8e740 1 -- 172.21.15.23:0/609633094 shutdown_connections 2024-03-24T10:46:36.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.831+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/609633094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 0x56104dce5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.831+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/609633094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce9f00 0x56104dcf0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.831+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/609633094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce95e0 0x56104dcfe340 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.831+0000 7fe62bb8e740 1 -- 172.21.15.23:0/609633094 >> 172.21.15.23:0/609633094 conn(0x56104dce3f00 msgr2=0x56104dce5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:36.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.831+0000 7fe62bb8e740 1 -- 172.21.15.23:0/609633094 shutdown_connections 2024-03-24T10:46:36.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.831+0000 7fe62bb8e740 1 -- 172.21.15.23:0/609633094 wait complete. 2024-03-24T10:46:36.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62bb8e740 1 -- start start 2024-03-24T10:46:36.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62bb8e740 1 Processor -- start 2024-03-24T10:46:36.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62bb8e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce95e0 0x56104dcecc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62bb8e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce9f00 0x56104dced170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62bb8e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 0x56104dd67f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62bb8e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56104dceda40 con 0x56104dce95e0 2024-03-24T10:46:36.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62bb8e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56104dd68490 con 0x56104dcf2cd0 2024-03-24T10:46:36.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62bb8e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56104dd685d0 con 0x56104dce9f00 2024-03-24T10:46:36.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62bb8e740 10 client.0 _ll_register_callbacks cb 0x7fe61809ad20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:36.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce95e0 0x56104dcecc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce95e0 0x56104dcecc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33036/0 (socket says 172.21.15.23:33036) 2024-03-24T10:46:36.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62259c640 1 -- 172.21.15.23:0/1839473845 learned_addr learned my addr 172.21.15.23:0/1839473845 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:36.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe621d9b640 1 --2- 172.21.15.23:0/1839473845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce9f00 0x56104dced170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe622d9d640 1 --2- 172.21.15.23:0/1839473845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 0x56104dd67f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62259c640 1 -- 172.21.15.23:0/1839473845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce9f00 msgr2=0x56104dced170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:36.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62259c640 1 --2- 172.21.15.23:0/1839473845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce9f00 0x56104dced170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62259c640 1 -- 172.21.15.23:0/1839473845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 msgr2=0x56104dd67f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:36.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62259c640 1 --2- 172.21.15.23:0/1839473845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 0x56104dd67f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.832+0000 7fe62259c640 1 -- 172.21.15.23:0/1839473845 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56104dc4e310 con 0x56104dce95e0 2024-03-24T10:46:36.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe62259c640 1 --2- 172.21.15.23:0/1839473845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce95e0 0x56104dcecc30 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe61402ac60 tx=0x7fe61402ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddff21c2ba09ff92 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:36.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe609ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:36.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe609ffb640 1 -- 172.21.15.23:0/1839473845 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe614024070 con 0x56104dce95e0 2024-03-24T10:46:36.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe609ffb640 1 -- 172.21.15.23:0/1839473845 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fe61401e6b0 con 0x56104dce95e0 2024-03-24T10:46:36.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe62bb8e740 1 -- 172.21.15.23:0/1839473845 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe61c0345a0 con 0x56104dce95e0 2024-03-24T10:46:36.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe609ffb640 1 -- 172.21.15.23:0/1839473845 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe61401e8b0 con 0x56104dce95e0 2024-03-24T10:46:36.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe62bb8e740 10 client.17400 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:36.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe62bb8e740 1 -- 172.21.15.23:0/1839473845 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x7fe614017070 con 0x56104dce95e0 2024-03-24T10:46:36.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe609ffb640 1 -- 172.21.15.23:0/1839473845 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe61c0345a0 con 0x56104dce95e0 2024-03-24T10:46:36.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe609ffb640 1 -- 172.21.15.23:0/1839473845 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fe61403c610 con 0x56104dce95e0 2024-03-24T10:46:36.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe62bb8e740 10 client.17400 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:36.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe62bb8e740 10 client.17400 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:36.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe62bb8e740 20 client.17400 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:36.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/1839473845 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56104dd69da0 0x56104dd6c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:36.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe62bb8e740 4 client.17400 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:46:36.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe62bb8e740 1 -- 172.21.15.23:0/1839473845 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56104d9a01b0 con 0x56104dd69da0 2024-03-24T10:46:36.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.833+0000 7fe621d9b640 1 --2- 172.21.15.23:0/1839473845 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56104dd69da0 0x56104dd6c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:36.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.834+0000 7fe621d9b640 1 --2- 172.21.15.23:0/1839473845 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56104dd69da0 0x56104dd6c1c0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3319b0a9ead4c6db server_cookie=8f934c16e0862277 in_seq=0 out_seq=0 2024-03-24T10:46:36.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.834+0000 7fe609ffb640 10 client.17400 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:36.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.844+0000 7fe609ffb640 1 -- 172.21.15.23:0/1839473845 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56104d9a01b0 con 0x56104dd69da0 2024-03-24T10:46:36.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.844+0000 7fe609ffb640 10 client.17400 handle_command_reply: tid=0 2024-03-24T10:46:36.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.844+0000 7fe62bb8e740 1 client.17400 shutdown 2024-03-24T10:46:36.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.844+0000 7fe62bb8e740 1 -- 172.21.15.23:0/1839473845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce95e0 msgr2=0x56104dcecc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:36.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.844+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/1839473845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce95e0 0x56104dcecc30 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe61402ac60 tx=0x7fe61402ac90 comp rx=0 tx=0).stop 2024-03-24T10:46:36.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe62bb8e740 1 -- 172.21.15.23:0/1839473845 shutdown_connections 2024-03-24T10:46:36.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/1839473845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56104dcf2cd0 0x56104dd67f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/1839473845 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56104dd69da0 0x56104dd6c1c0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/1839473845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56104dce9f00 0x56104dced170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe62bb8e740 1 --2- 172.21.15.23:0/1839473845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56104dce95e0 0x56104dcecc30 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe609ffb640 0 client.17400 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:36.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe62bb8e740 1 -- 172.21.15.23:0/1839473845 >> 172.21.15.23:0/1839473845 conn(0x56104dce3f00 msgr2=0x56104dce2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:36.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe62bb8e740 1 -- 172.21.15.23:0/1839473845 shutdown_connections 2024-03-24T10:46:36.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe62bb8e740 1 -- 172.21.15.23:0/1839473845 wait complete. 2024-03-24T10:46:36.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe62bb8e740 20 client.17400 trim_cache size 0 max 16384 2024-03-24T10:46:36.847 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:46:36.847 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:46:36.847 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:46:36.847 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:46:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe628faf640 1 -- 172.21.15.23:0/4266522255 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe60c0a22d0 msgr2=0x7fe60c0a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe628faf640 1 --2- 172.21.15.23:0/4266522255 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe60c0a22d0 0x7fe60c0a4790 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe6240ff990 tx=0x7fe614005f90 comp rx=0 tx=0).stop 2024-03-24T10:46:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe628faf640 1 -- 172.21.15.23:0/4266522255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6241020a0 msgr2=0x7fe62419de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe628faf640 1 --2- 172.21.15.23:0/4266522255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6241020a0 0x7fe62419de40 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe61800e9f0 tx=0x7fe61800eeb0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe628faf640 1 -- 172.21.15.23:0/4266522255 shutdown_connections 2024-03-24T10:46:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe628faf640 1 --2- 172.21.15.23:0/4266522255 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe60c0a22d0 0x7fe60c0a4790 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe628faf640 1 --2- 172.21.15.23:0/4266522255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6241076f0 0x7fe624197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe628faf640 1 --2- 172.21.15.23:0/4266522255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6241020a0 0x7fe62419de40 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe628faf640 1 --2- 172.21.15.23:0/4266522255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6241016e0 0x7fe62419d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe628faf640 1 -- 172.21.15.23:0/4266522255 >> 172.21.15.23:0/4266522255 conn(0x7fe6240fd210 msgr2=0x7fe6240fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe628faf640 1 -- 172.21.15.23:0/4266522255 shutdown_connections 2024-03-24T10:46:36.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:36.845+0000 7fe628faf640 1 -- 172.21.15.23:0/4266522255 wait complete. 2024-03-24T10:46:36.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:36 smithi023 ceph-mon[32466]: pgmap v199: 97 pgs: 97 active+clean; 391 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-03-24T10:46:37.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 629095 11.11 MB/sec execute 174 sec latency 33.033 ms 2024-03-24T10:46:38.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 631019 11.09 MB/sec execute 175 sec latency 21.976 ms 2024-03-24T10:46:38.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:38 smithi117 ceph-mon[36368]: pgmap v200: 97 pgs: 97 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 260 op/s 2024-03-24T10:46:38.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:38 smithi116 ceph-mon[36248]: pgmap v200: 97 pgs: 97 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 260 op/s 2024-03-24T10:46:38.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:38 smithi023 ceph-mon[32466]: pgmap v200: 97 pgs: 97 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 260 op/s 2024-03-24T10:46:39.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 632972 11.10 MB/sec execute 176 sec latency 17.883 ms 2024-03-24T10:46:40.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 635175 11.10 MB/sec execute 177 sec latency 20.351 ms 2024-03-24T10:46:40.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:40 smithi117 ceph-mon[36368]: pgmap v201: 97 pgs: 97 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 170 op/s 2024-03-24T10:46:40.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:40 smithi116 ceph-mon[36248]: pgmap v201: 97 pgs: 97 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 170 op/s 2024-03-24T10:46:40.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:40 smithi023 ceph-mon[32466]: pgmap v201: 97 pgs: 97 active+clean; 366 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 170 op/s 2024-03-24T10:46:41.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 636769 11.09 MB/sec execute 178 sec latency 41.769 ms 2024-03-24T10:46:41.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:46:41.843 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:46:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.953+0000 7fe6c3e1e640 1 -- 172.21.15.23:0/494032784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc107760 msgr2=0x7fe6bc107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.953+0000 7fe6c3e1e640 1 --2- 172.21.15.23:0/494032784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc107760 0x7fe6bc107b40 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009930 tx=0x7fe6b802bc30 comp rx=0 tx=0).stop 2024-03-24T10:46:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.953+0000 7fe6c3e1e640 1 -- 172.21.15.23:0/494032784 shutdown_connections 2024-03-24T10:46:41.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.953+0000 7fe6c3e1e640 1 --2- 172.21.15.23:0/494032784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6bc0692d0 0x7fe6bc10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.953+0000 7fe6c3e1e640 1 --2- 172.21.15.23:0/494032784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc068910 0x7fe6bc068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.953+0000 7fe6c3e1e640 1 --2- 172.21.15.23:0/494032784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc107760 0x7fe6bc107b40 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.953+0000 7fe6c3e1e640 1 -- 172.21.15.23:0/494032784 >> 172.21.15.23:0/494032784 conn(0x7fe6bc077710 msgr2=0x7fe6bc077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:41.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.953+0000 7fe6c3e1e640 1 -- 172.21.15.23:0/494032784 shutdown_connections 2024-03-24T10:46:41.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.953+0000 7fe6c3e1e640 1 -- 172.21.15.23:0/494032784 wait complete. 2024-03-24T10:46:41.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c3e1e640 1 Processor -- start 2024-03-24T10:46:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c3e1e640 1 -- start start 2024-03-24T10:46:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c3e1e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc068910 0x7fe6bc19da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c3e1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc0692d0 0x7fe6bc19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c3e1e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6bc107760 0x7fe6bc197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c3e1e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc1982a0 con 0x7fe6bc068910 2024-03-24T10:46:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c3e1e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc198410 con 0x7fe6bc0692d0 2024-03-24T10:46:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c3e1e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc1985d0 con 0x7fe6bc107760 2024-03-24T10:46:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c1bba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc068910 0x7fe6bc19da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c1bba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc068910 0x7fe6bc19da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58858/0 (socket says 172.21.15.23:58858) 2024-03-24T10:46:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c1bba640 1 -- 172.21.15.23:0/2997022443 learned_addr learned my addr 172.21.15.23:0/2997022443 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c13b9640 1 --2- 172.21.15.23:0/2997022443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc0692d0 0x7fe6bc19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c23bb640 1 --2- 172.21.15.23:0/2997022443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6bc107760 0x7fe6bc197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c1bba640 1 -- 172.21.15.23:0/2997022443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6bc107760 msgr2=0x7fe6bc197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c1bba640 1 --2- 172.21.15.23:0/2997022443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6bc107760 0x7fe6bc197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c1bba640 1 -- 172.21.15.23:0/2997022443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc0692d0 msgr2=0x7fe6bc19df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:41.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c1bba640 1 --2- 172.21.15.23:0/2997022443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc0692d0 0x7fe6bc19df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c1bba640 1 -- 172.21.15.23:0/2997022443 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b8009560 con 0x7fe6bc068910 2024-03-24T10:46:41.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.954+0000 7fe6c1bba640 1 --2- 172.21.15.23:0/2997022443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc068910 0x7fe6bc19da50 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009230 tx=0x7fe6b8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:41.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.955+0000 7fe6b2ffd640 1 -- 172.21.15.23:0/2997022443 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b80042d0 con 0x7fe6bc068910 2024-03-24T10:46:41.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.955+0000 7fe6c3e1e640 1 -- 172.21.15.23:0/2997022443 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6bc198850 con 0x7fe6bc068910 2024-03-24T10:46:41.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.955+0000 7fe6b2ffd640 1 -- 172.21.15.23:0/2997022443 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b8004430 con 0x7fe6bc068910 2024-03-24T10:46:41.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.955+0000 7fe6b2ffd640 1 -- 172.21.15.23:0/2997022443 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8003c90 con 0x7fe6bc068910 2024-03-24T10:46:41.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.955+0000 7fe6c3e1e640 1 -- 172.21.15.23:0/2997022443 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6bc1a49d0 con 0x7fe6bc068910 2024-03-24T10:46:41.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6b2ffd640 1 -- 172.21.15.23:0/2997022443 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fe6b8005560 con 0x7fe6bc068910 2024-03-24T10:46:41.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c69fd740 1 Processor -- start 2024-03-24T10:46:41.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c69fd740 1 -- start start 2024-03-24T10:46:41.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c69fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc4560e1d0 0x55bc4560e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c69fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560eaf0 0x55bc45617190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c69fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc456176d0 0x55bc45619ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c69fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bc45532f30 con 0x55bc4560e1d0 2024-03-24T10:46:41.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c69fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bc4557c810 con 0x55bc456176d0 2024-03-24T10:46:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c69fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bc45579d40 con 0x55bc4560eaf0 2024-03-24T10:46:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c23bb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc4560e1d0 0x55bc4560e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c1bba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560eaf0 0x55bc45617190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c13b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc456176d0 0x55bc45619ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c23bb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc4560e1d0 0x55bc4560e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58862/0 (socket says 172.21.15.23:58862) 2024-03-24T10:46:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c23bb640 1 -- 172.21.15.23:0/2596599416 learned_addr learned my addr 172.21.15.23:0/2596599416 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6b2ffd640 1 --2- 172.21.15.23:0/2997022443 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe6a80a2230 0x7fe6a80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c23bb640 1 -- 172.21.15.23:0/2596599416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560eaf0 msgr2=0x55bc45617190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.958+0000 7fe6c23bb640 1 --2- 172.21.15.23:0/2596599416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560eaf0 0x55bc45617190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.959+0000 7fe6c23bb640 1 -- 172.21.15.23:0/2596599416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc456176d0 msgr2=0x55bc45619ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.959+0000 7fe6c23bb640 1 --2- 172.21.15.23:0/2596599416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc456176d0 0x55bc45619ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.959+0000 7fe6c23bb640 1 -- 172.21.15.23:0/2596599416 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc45536ea0 con 0x55bc4560e1d0 2024-03-24T10:46:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.959+0000 7fe6c13b9640 1 --2- 172.21.15.23:0/2596599416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc456176d0 0x55bc45619ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.959+0000 7fe6b2ffd640 1 -- 172.21.15.23:0/2997022443 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fe6b80effc0 con 0x7fe6bc068910 2024-03-24T10:46:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.959+0000 7fe6c23bb640 1 --2- 172.21.15.23:0/2997022443 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe6a80a2230 0x7fe6a80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.959+0000 7fe6c23bb640 1 --2- 172.21.15.23:0/2596599416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc4560e1d0 0x55bc4560e5b0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4011590 tx=0x7fe6b4011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3275c643da80d2b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.959+0000 7fe6a7fff640 1 -- 172.21.15.23:0/2596599416 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b400f840 con 0x55bc4560e1d0 2024-03-24T10:46:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.959+0000 7fe6a7fff640 1 -- 172.21.15.23:0/2596599416 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b400fe80 con 0x55bc4560e1d0 2024-03-24T10:46:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.959+0000 7fe6a7fff640 1 -- 172.21.15.23:0/2596599416 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b401bb70 con 0x55bc4560e1d0 2024-03-24T10:46:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.960+0000 7fe6c69fd740 1 -- 172.21.15.23:0/2596599416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc4560e1d0 msgr2=0x55bc4560e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.960+0000 7fe6c23bb640 1 --2- 172.21.15.23:0/2997022443 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe6a80a2230 0x7fe6a80a46f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4022000 tx=0x7fe6b4023000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.960+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/2596599416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc4560e1d0 0x55bc4560e5b0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4011590 tx=0x7fe6b4011a50 comp rx=0 tx=0).stop 2024-03-24T10:46:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.960+0000 7fe6c69fd740 1 -- 172.21.15.23:0/2596599416 shutdown_connections 2024-03-24T10:46:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.960+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/2596599416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc456176d0 0x55bc45619ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.960+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/2596599416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560eaf0 0x55bc45617190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.960+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/2596599416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc4560e1d0 0x55bc4560e5b0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.960+0000 7fe6c69fd740 1 -- 172.21.15.23:0/2596599416 >> 172.21.15.23:0/2596599416 conn(0x55bc45608900 msgr2=0x55bc45608ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.960+0000 7fe6c69fd740 1 -- 172.21.15.23:0/2596599416 shutdown_connections 2024-03-24T10:46:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.960+0000 7fe6c69fd740 1 -- 172.21.15.23:0/2596599416 wait complete. 2024-03-24T10:46:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.960+0000 7fe6c69fd740 1 -- start start 2024-03-24T10:46:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.960+0000 7fe6c69fd740 1 Processor -- start 2024-03-24T10:46:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c69fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc4560e1d0 0x55bc45614750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c69fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560eaf0 0x55bc45614c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c69fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc456176d0 0x55bc456151d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c69fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bc45575520 con 0x55bc4560e1d0 2024-03-24T10:46:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c69fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bc45248430 con 0x55bc4560eaf0 2024-03-24T10:46:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c69fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bc45204e00 con 0x55bc456176d0 2024-03-24T10:46:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c69fd740 10 client.0 _ll_register_callbacks cb 0x55bc45513d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c1bba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc4560e1d0 0x55bc45614750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c1bba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc4560e1d0 0x55bc45614750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58876/0 (socket says 172.21.15.23:58876) 2024-03-24T10:46:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c1bba640 1 -- 172.21.15.23:0/104964416 learned_addr learned my addr 172.21.15.23:0/104964416 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:41.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c13b9640 1 --2- 172.21.15.23:0/104964416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560eaf0 0x55bc45614c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c23bb640 1 --2- 172.21.15.23:0/104964416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc456176d0 0x55bc456151d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c1bba640 1 -- 172.21.15.23:0/104964416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc456176d0 msgr2=0x55bc456151d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:41.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c1bba640 1 --2- 172.21.15.23:0/104964416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc456176d0 0x55bc456151d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c1bba640 1 -- 172.21.15.23:0/104964416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560eaf0 msgr2=0x55bc45614c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:41.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c1bba640 1 --2- 172.21.15.23:0/104964416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560eaf0 0x55bc45614c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c1bba640 1 -- 172.21.15.23:0/104964416 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b4009cd0 con 0x55bc4560e1d0 2024-03-24T10:46:41.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c1bba640 1 --2- 172.21.15.23:0/104964416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc4560e1d0 0x55bc45614750 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80edd10 tx=0x7fe6b80f36c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a235c2806762d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:41.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:41.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6a57fa640 1 -- 172.21.15.23:0/104964416 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b80bc600 con 0x55bc4560e1d0 2024-03-24T10:46:41.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6a57fa640 1 -- 172.21.15.23:0/104964416 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fe6b80bd1c0 con 0x55bc4560e1d0 2024-03-24T10:46:41.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.961+0000 7fe6c69fd740 1 -- 172.21.15.23:0/104964416 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bc452b1720 con 0x55bc4560e1d0 2024-03-24T10:46:41.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.962+0000 7fe6a57fa640 1 -- 172.21.15.23:0/104964416 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe6b80bc420 con 0x55bc4560e1d0 2024-03-24T10:46:41.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.962+0000 7fe6c69fd740 10 client.17412 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:41.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.962+0000 7fe6c69fd740 1 -- 172.21.15.23:0/104964416 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bc45573ba0 con 0x55bc4560e1d0 2024-03-24T10:46:41.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.962+0000 7fe6a57fa640 1 -- 172.21.15.23:0/104964416 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bc452b1720 con 0x55bc4560e1d0 2024-03-24T10:46:41.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.962+0000 7fe6a57fa640 1 -- 172.21.15.23:0/104964416 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fe6b803c070 con 0x55bc4560e1d0 2024-03-24T10:46:41.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.962+0000 7fe6c69fd740 10 client.17412 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:41.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.962+0000 7fe6c69fd740 10 client.17412 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:41.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.962+0000 7fe6c69fd740 20 client.17412 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:41.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.962+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/104964416 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc4562bdd0 0x55bc4562e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.962+0000 7fe6c69fd740 4 client.17412 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:41.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.962+0000 7fe6c69fd740 1 -- 172.21.15.23:0/104964416 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc45300890 con 0x55bc4562bdd0 2024-03-24T10:46:41.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.962+0000 7fe6c13b9640 1 --2- 172.21.15.23:0/104964416 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc4562bdd0 0x55bc4562e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.963+0000 7fe6a57fa640 10 client.17412 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:41.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.963+0000 7fe6c13b9640 1 --2- 172.21.15.23:0/104964416 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc4562bdd0 0x55bc4562e1f0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5329e5d97abc1946 server_cookie=ae6b4ffdd4f6d986 in_seq=0 out_seq=0 2024-03-24T10:46:41.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6a57fa640 1 -- 172.21.15.23:0/104964416 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bc45300890 con 0x55bc4562bdd0 2024-03-24T10:46:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6a57fa640 10 client.17412 handle_command_reply: tid=0 2024-03-24T10:46:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6c69fd740 1 client.17412 shutdown 2024-03-24T10:46:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6c69fd740 1 -- 172.21.15.23:0/104964416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc4560e1d0 msgr2=0x55bc45614750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/104964416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc4560e1d0 0x55bc45614750 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80edd10 tx=0x7fe6b80f36c0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6c69fd740 1 -- 172.21.15.23:0/104964416 shutdown_connections 2024-03-24T10:46:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/104964416 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc4562bdd0 0x55bc4562e1f0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6a57fa640 0 client.17412 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/104964416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc456176d0 0x55bc456151d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/104964416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560eaf0 0x55bc45614c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/104964416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc4560e1d0 0x55bc45614750 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6c69fd740 1 -- 172.21.15.23:0/104964416 >> 172.21.15.23:0/104964416 conn(0x55bc45608900 msgr2=0x55bc45607b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6c69fd740 1 -- 172.21.15.23:0/104964416 shutdown_connections 2024-03-24T10:46:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6c69fd740 1 -- 172.21.15.23:0/104964416 wait complete. 2024-03-24T10:46:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.964+0000 7fe6c69fd740 20 client.17412 trim_cache size 0 max 16384 2024-03-24T10:46:41.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.983+0000 7fe6c69fd740 1 Processor -- start 2024-03-24T10:46:41.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.983+0000 7fe6c69fd740 1 -- start start 2024-03-24T10:46:41.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.983+0000 7fe6c69fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560e1d0 0x55bc45627370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.983+0000 7fe6c69fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560eaf0 0x55bc45614ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.983+0000 7fe6c69fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc456176d0 0x55bc45633ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.983+0000 7fe6c69fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bc454fa430 con 0x55bc456176d0 2024-03-24T10:46:41.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.983+0000 7fe6c69fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bc45609900 con 0x55bc4560eaf0 2024-03-24T10:46:41.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.983+0000 7fe6c69fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bc4560d970 con 0x55bc4560e1d0 2024-03-24T10:46:41.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.983+0000 7fe6c23bb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc456176d0 0x55bc45633ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.983+0000 7fe6c13b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560eaf0 0x55bc45614ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.983+0000 7fe6c1bba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560e1d0 0x55bc45627370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.983+0000 7fe6c23bb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc456176d0 0x55bc45633ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58882/0 (socket says 172.21.15.23:58882) 2024-03-24T10:46:41.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.983+0000 7fe6c23bb640 1 -- 172.21.15.23:0/1749426683 learned_addr learned my addr 172.21.15.23:0/1749426683 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:41.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.984+0000 7fe6c1bba640 1 -- 172.21.15.23:0/1749426683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560eaf0 msgr2=0x55bc45614ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:41.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.984+0000 7fe6c1bba640 1 --2- 172.21.15.23:0/1749426683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560eaf0 0x55bc45614ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.984+0000 7fe6c1bba640 1 -- 172.21.15.23:0/1749426683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc456176d0 msgr2=0x55bc45633ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:41.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.984+0000 7fe6c1bba640 1 --2- 172.21.15.23:0/1749426683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc456176d0 0x55bc45633ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.984+0000 7fe6c1bba640 1 -- 172.21.15.23:0/1749426683 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc4550fb20 con 0x55bc4560e1d0 2024-03-24T10:46:41.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.984+0000 7fe6c13b9640 1 --2- 172.21.15.23:0/1749426683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560eaf0 0x55bc45614ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:41.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.984+0000 7fe6c1bba640 1 --2- 172.21.15.23:0/1749426683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560e1d0 0x55bc45627370 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009430 tx=0x7fe6b8009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1eab57048b46bd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:41.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.984+0000 7fe6a77fe640 1 -- 172.21.15.23:0/1749426683 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b80bc600 con 0x55bc4560e1d0 2024-03-24T10:46:41.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.984+0000 7fe6a77fe640 1 -- 172.21.15.23:0/1749426683 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b80bc420 con 0x55bc4560e1d0 2024-03-24T10:46:41.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.984+0000 7fe6a77fe640 1 -- 172.21.15.23:0/1749426683 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b80baee0 con 0x55bc4560e1d0 2024-03-24T10:46:41.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.985+0000 7fe6c69fd740 1 -- 172.21.15.23:0/1749426683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560e1d0 msgr2=0x55bc45627370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:41.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.985+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/1749426683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560e1d0 0x55bc45627370 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009430 tx=0x7fe6b8009260 comp rx=0 tx=0).stop 2024-03-24T10:46:41.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.985+0000 7fe6c69fd740 1 -- 172.21.15.23:0/1749426683 shutdown_connections 2024-03-24T10:46:41.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.985+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/1749426683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc456176d0 0x55bc45633ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.985+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/1749426683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560eaf0 0x55bc45614ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.985+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/1749426683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560e1d0 0x55bc45627370 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.985+0000 7fe6c69fd740 1 -- 172.21.15.23:0/1749426683 >> 172.21.15.23:0/1749426683 conn(0x55bc45608900 msgr2=0x55bc45607b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.985+0000 7fe6c69fd740 1 -- 172.21.15.23:0/1749426683 shutdown_connections 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.985+0000 7fe6c69fd740 1 -- 172.21.15.23:0/1749426683 wait complete. 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.985+0000 7fe6c69fd740 1 -- start start 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.985+0000 7fe6c69fd740 1 Processor -- start 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c69fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560e1d0 0x55bc45611220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c69fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560eaf0 0x55bc45611760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c69fd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc456176d0 0x55bc45611ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c69fd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55bc456241f0 con 0x55bc456176d0 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c69fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bc45691110 con 0x55bc4560e1d0 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c69fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bc45691250 con 0x55bc4560eaf0 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c69fd740 10 client.0 _ll_register_callbacks cb 0x7fe6b80045a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:41.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c1bba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560e1d0 0x55bc45611220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c13b9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560eaf0 0x55bc45611760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c1bba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560e1d0 0x55bc45611220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:54972/0 (socket says 172.21.15.23:54972) 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c1bba640 1 -- 172.21.15.23:0/4147158853 learned_addr learned my addr 172.21.15.23:0/4147158853 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c23bb640 1 --2- 172.21.15.23:0/4147158853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc456176d0 0x55bc45611ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c13b9640 1 -- 172.21.15.23:0/4147158853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560e1d0 msgr2=0x55bc45611220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c13b9640 1 --2- 172.21.15.23:0/4147158853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560e1d0 0x55bc45611220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c13b9640 1 -- 172.21.15.23:0/4147158853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc456176d0 msgr2=0x55bc45611ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c13b9640 1 --2- 172.21.15.23:0/4147158853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc456176d0 0x55bc45611ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c13b9640 1 -- 172.21.15.23:0/4147158853 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6ac009b00 con 0x55bc4560eaf0 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c23bb640 1 --2- 172.21.15.23:0/4147158853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc456176d0 0x55bc45611ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6c13b9640 1 --2- 172.21.15.23:0/4147158853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560eaf0 0x55bc45611760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55bc4562ea50 tx=0x7fe6ac0208b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a031b9e9c09bbe4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.986+0000 7fe6a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6a57fa640 1 -- 172.21.15.23:0/4147158853 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac017040 con 0x55bc4560eaf0 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6a57fa640 1 -- 172.21.15.23:0/4147158853 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fe6ac003980 con 0x55bc4560eaf0 2024-03-24T10:46:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6c69fd740 1 -- 172.21.15.23:0/4147158853 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bc452747e0 con 0x55bc4560eaf0 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6a57fa640 1 -- 172.21.15.23:0/4147158853 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe6ac003b40 con 0x55bc4560eaf0 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6c69fd740 10 client.27277 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6c69fd740 1 -- 172.21.15.23:0/4147158853 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bc45573ba0 con 0x55bc4560eaf0 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6a57fa640 1 -- 172.21.15.23:0/4147158853 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bc452747e0 con 0x55bc4560eaf0 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6a57fa640 1 -- 172.21.15.23:0/4147158853 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fe6ac0049d0 con 0x55bc4560eaf0 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6c69fd740 10 client.27277 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6c69fd740 10 client.27277 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6c69fd740 20 client.27277 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/4147158853 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc4562bdd0 0x55bc45694b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6c69fd740 4 client.27277 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6c69fd740 1 -- 172.21.15.23:0/4147158853 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bc45220700 con 0x55bc4562bdd0 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.987+0000 7fe6c1bba640 1 --2- 172.21.15.23:0/4147158853 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc4562bdd0 0x55bc45694b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6a57fa640 10 client.27277 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6c1bba640 1 --2- 172.21.15.23:0/4147158853 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc4562bdd0 0x55bc45694b20 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=269b68492f4a2b00 server_cookie=a7f0eea3d5c55f37 in_seq=0 out_seq=0 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6a57fa640 1 -- 172.21.15.23:0/4147158853 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bc45220700 con 0x55bc4562bdd0 2024-03-24T10:46:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6a57fa640 10 client.27277 handle_command_reply: tid=0 2024-03-24T10:46:41.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6c69fd740 1 client.27277 shutdown 2024-03-24T10:46:41.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6c69fd740 1 -- 172.21.15.23:0/4147158853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560eaf0 msgr2=0x55bc45611760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:41.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/4147158853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560eaf0 0x55bc45611760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x55bc4562ea50 tx=0x7fe6ac0208b0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6c69fd740 1 -- 172.21.15.23:0/4147158853 shutdown_connections 2024-03-24T10:46:41.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/4147158853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bc456176d0 0x55bc45611ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/4147158853 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55bc4562bdd0 0x55bc45694b20 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/4147158853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bc4560eaf0 0x55bc45611760 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6c69fd740 1 --2- 172.21.15.23:0/4147158853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bc4560e1d0 0x55bc45611220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6c69fd740 1 -- 172.21.15.23:0/4147158853 >> 172.21.15.23:0/4147158853 conn(0x55bc45608900 msgr2=0x55bc45609be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:41.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6a57fa640 0 client.27277 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:41.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6c69fd740 1 -- 172.21.15.23:0/4147158853 shutdown_connections 2024-03-24T10:46:41.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6c69fd740 1 -- 172.21.15.23:0/4147158853 wait complete. 2024-03-24T10:46:41.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.988+0000 7fe6c69fd740 20 client.27277 trim_cache size 0 max 16384 2024-03-24T10:46:41.995 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:46:41.995 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:46:41.995 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:46:41.995 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:46:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.989+0000 7fe6c3e1e640 1 -- 172.21.15.23:0/2997022443 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe6a80a2230 msgr2=0x7fe6a80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.989+0000 7fe6c3e1e640 1 --2- 172.21.15.23:0/2997022443 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe6a80a2230 0x7fe6a80a46f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4022000 tx=0x7fe6b4023000 comp rx=0 tx=0).stop 2024-03-24T10:46:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.989+0000 7fe6c3e1e640 1 -- 172.21.15.23:0/2997022443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc068910 msgr2=0x7fe6bc19da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.989+0000 7fe6c3e1e640 1 --2- 172.21.15.23:0/2997022443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc068910 0x7fe6bc19da50 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009230 tx=0x7fe6b8004ab0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.989+0000 7fe6c3e1e640 1 -- 172.21.15.23:0/2997022443 shutdown_connections 2024-03-24T10:46:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.989+0000 7fe6c3e1e640 1 --2- 172.21.15.23:0/2997022443 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fe6a80a2230 0x7fe6a80a46f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.989+0000 7fe6c3e1e640 1 --2- 172.21.15.23:0/2997022443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6bc107760 0x7fe6bc197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.989+0000 7fe6c3e1e640 1 --2- 172.21.15.23:0/2997022443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6bc0692d0 0x7fe6bc19df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.989+0000 7fe6c3e1e640 1 --2- 172.21.15.23:0/2997022443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc068910 0x7fe6bc19da50 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.989+0000 7fe6c3e1e640 1 -- 172.21.15.23:0/2997022443 >> 172.21.15.23:0/2997022443 conn(0x7fe6bc077710 msgr2=0x7fe6bc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.989+0000 7fe6c3e1e640 1 -- 172.21.15.23:0/2997022443 shutdown_connections 2024-03-24T10:46:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:41.989+0000 7fe6c3e1e640 1 -- 172.21.15.23:0/2997022443 wait complete. 2024-03-24T10:46:42.000 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:46:42.000 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:46:42.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.107+0000 7fb29dd19640 1 -- 172.21.15.23:0/1735978380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb298100d40 msgr2=0x7fb29810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.107+0000 7fb29dd19640 1 --2- 172.21.15.23:0/1735978380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb298100d40 0x7fb29810d4e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fb290009ca0 tx=0x7fb29002c180 comp rx=0 tx=0).stop 2024-03-24T10:46:42.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.108+0000 7fb29dd19640 1 -- 172.21.15.23:0/1735978380 shutdown_connections 2024-03-24T10:46:42.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.108+0000 7fb29dd19640 1 --2- 172.21.15.23:0/1735978380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb298100d40 0x7fb29810d4e0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.108+0000 7fb29dd19640 1 --2- 172.21.15.23:0/1735978380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb298106fd0 0x7fb298100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.108+0000 7fb29dd19640 1 --2- 172.21.15.23:0/1735978380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2981066b0 0x7fb298106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.108+0000 7fb29dd19640 1 -- 172.21.15.23:0/1735978380 >> 172.21.15.23:0/1735978380 conn(0x7fb298077710 msgr2=0x7fb298077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.108+0000 7fb29dd19640 1 -- 172.21.15.23:0/1735978380 shutdown_connections 2024-03-24T10:46:42.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.108+0000 7fb29dd19640 1 -- 172.21.15.23:0/1735978380 wait complete. 2024-03-24T10:46:42.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.108+0000 7fb29dd19640 1 Processor -- start 2024-03-24T10:46:42.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.108+0000 7fb29dd19640 1 -- start start 2024-03-24T10:46:42.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb29dd19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb298100d40 0x7fb298197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb29dd19640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2981066b0 0x7fb2981982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb29dd19640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb298106fd0 0x7fb29819c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb29dd19640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb29819cc40 con 0x7fb298106fd0 2024-03-24T10:46:42.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb29dd19640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb29819cdb0 con 0x7fb298100d40 2024-03-24T10:46:42.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb29dd19640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb29819cf70 con 0x7fb2981066b0 2024-03-24T10:46:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb2977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb298100d40 0x7fb298197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb2977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb298100d40 0x7fb298197d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:54984/0 (socket says 172.21.15.23:54984) 2024-03-24T10:46:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb2977fe640 1 -- 172.21.15.23:0/3409041395 learned_addr learned my addr 172.21.15.23:0/3409041395 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb296ffd640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2981066b0 0x7fb2981982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb297fff640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb298106fd0 0x7fb29819c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb2977fe640 1 -- 172.21.15.23:0/3409041395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2981066b0 msgr2=0x7fb2981982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb2977fe640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2981066b0 0x7fb2981982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb2977fe640 1 -- 172.21.15.23:0/3409041395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb298106fd0 msgr2=0x7fb29819c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb2977fe640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb298106fd0 0x7fb29819c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb2977fe640 1 -- 172.21.15.23:0/3409041395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb290009900 con 0x7fb298100d40 2024-03-24T10:46:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb296ffd640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2981066b0 0x7fb2981982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.109+0000 7fb2977fe640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb298100d40 0x7fb298197d80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb28800e9c0 tx=0x7fb28800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.110+0000 7fb294ff9640 1 -- 172.21.15.23:0/3409041395 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2880096e0 con 0x7fb298100d40 2024-03-24T10:46:42.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.110+0000 7fb29dd19640 1 -- 172.21.15.23:0/3409041395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb29819d250 con 0x7fb298100d40 2024-03-24T10:46:42.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.110+0000 7fb294ff9640 1 -- 172.21.15.23:0/3409041395 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb28800cd80 con 0x7fb298100d40 2024-03-24T10:46:42.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.110+0000 7fb294ff9640 1 -- 172.21.15.23:0/3409041395 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb288010430 con 0x7fb298100d40 2024-03-24T10:46:42.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.110+0000 7fb29dd19640 1 -- 172.21.15.23:0/3409041395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2981a4ad0 con 0x7fb298100d40 2024-03-24T10:46:42.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.111+0000 7fb29dd19640 1 -- 172.21.15.23:0/3409041395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb298068500 con 0x7fb298100d40 2024-03-24T10:46:42.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.112+0000 7fb294ff9640 1 -- 172.21.15.23:0/3409041395 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fb288003680 con 0x7fb298100d40 2024-03-24T10:46:42.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.113+0000 7fb294ff9640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb2840a2230 0x7fb2840a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.113+0000 7fb296ffd640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb2840a2230 0x7fb2840a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.113+0000 7fb294ff9640 1 -- 172.21.15.23:0/3409041395 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fb288014070 con 0x7fb298100d40 2024-03-24T10:46:42.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.113+0000 7fb296ffd640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb2840a2230 0x7fb2840a46f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb298105460 tx=0x7fb280005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.119+0000 7fb294ff9640 1 -- 172.21.15.23:0/3409041395 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb288097130 con 0x7fb298100d40 2024-03-24T10:46:42.123 INFO:tasks.workunit.client.0.smithi023.stdout: 1 638992 11.10 MB/sec execute 179 sec latency 50.698 ms 2024-03-24T10:46:42.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.365+0000 7fb29dd19640 1 -- 172.21.15.23:0/3409041395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2981990d0 con 0x7fb298100d40 2024-03-24T10:46:42.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.366+0000 7fb294ff9640 1 -- 172.21.15.23:0/3409041395 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11869 (secure 0 0 0) 0x7fb28809a7f0 con 0x7fb298100d40 2024-03-24T10:46:42.366 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:46:42.366 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:45:17.709481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26933":{"gid":26933,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/2955742470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":2955742470},{"type":"v1","addr":"172.21.15.117:6837","nonce":2955742470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24539,24503,24505]},"id":1}]} 2024-03-24T10:46:42.366 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 29 2024-03-24T10:46:42.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.368+0000 7fb29dd19640 1 -- 172.21.15.23:0/3409041395 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb2840a2230 msgr2=0x7fb2840a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.368+0000 7fb29dd19640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb2840a2230 0x7fb2840a46f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb298105460 tx=0x7fb280005f90 comp rx=0 tx=0).stop 2024-03-24T10:46:42.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.368+0000 7fb29dd19640 1 -- 172.21.15.23:0/3409041395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb298100d40 msgr2=0x7fb298197d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.368+0000 7fb29dd19640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb298100d40 0x7fb298197d80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb28800e9c0 tx=0x7fb28800ee80 comp rx=0 tx=0).stop 2024-03-24T10:46:42.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.368+0000 7fb29dd19640 1 -- 172.21.15.23:0/3409041395 shutdown_connections 2024-03-24T10:46:42.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.368+0000 7fb29dd19640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb2840a2230 0x7fb2840a46f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.368+0000 7fb29dd19640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb298106fd0 0x7fb29819c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.368+0000 7fb29dd19640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2981066b0 0x7fb2981982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.368+0000 7fb29dd19640 1 --2- 172.21.15.23:0/3409041395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb298100d40 0x7fb298197d80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.368+0000 7fb29dd19640 1 -- 172.21.15.23:0/3409041395 >> 172.21.15.23:0/3409041395 conn(0x7fb298077710 msgr2=0x7fb2980ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.368+0000 7fb29dd19640 1 -- 172.21.15.23:0/3409041395 shutdown_connections 2024-03-24T10:46:42.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.368+0000 7fb29dd19640 1 -- 172.21.15.23:0/3409041395 wait complete. 2024-03-24T10:46:42.380 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:46:42.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.492+0000 7fcd262ae640 1 -- 172.21.15.23:0/4135586224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd20103240 msgr2=0x7fcd2010b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.492+0000 7fcd262ae640 1 --2- 172.21.15.23:0/4135586224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd20103240 0x7fcd2010b720 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fcd08009900 tx=0x7fcd0802bcd0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.492+0000 7fcd262ae640 1 -- 172.21.15.23:0/4135586224 shutdown_connections 2024-03-24T10:46:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.492+0000 7fcd262ae640 1 --2- 172.21.15.23:0/4135586224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd2010bcf0 0x7fcd2010e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.492+0000 7fcd262ae640 1 --2- 172.21.15.23:0/4135586224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd20103240 0x7fcd2010b720 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.492+0000 7fcd262ae640 1 --2- 172.21.15.23:0/4135586224 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd20102920 0x7fcd20102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.492+0000 7fcd262ae640 1 -- 172.21.15.23:0/4135586224 >> 172.21.15.23:0/4135586224 conn(0x7fcd20076990 msgr2=0x7fcd20076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.492+0000 7fcd262ae640 1 -- 172.21.15.23:0/4135586224 shutdown_connections 2024-03-24T10:46:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.492+0000 7fcd262ae640 1 -- 172.21.15.23:0/4135586224 wait complete. 2024-03-24T10:46:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.493+0000 7fcd262ae640 1 Processor -- start 2024-03-24T10:46:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.493+0000 7fcd262ae640 1 -- start start 2024-03-24T10:46:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.493+0000 7fcd262ae640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd20102920 0x7fcd2019a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.493+0000 7fcd262ae640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd20103240 0x7fcd2019a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.493+0000 7fcd262ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd2010bcf0 0x7fcd2019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.493+0000 7fcd262ae640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2019ef20 con 0x7fcd20102920 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.493+0000 7fcd262ae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2019f090 con 0x7fcd2010bcf0 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.493+0000 7fcd262ae640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2019f250 con 0x7fcd20103240 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.493+0000 7fcd1ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd20102920 0x7fcd2019a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.493+0000 7fcd1ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd20102920 0x7fcd2019a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58930/0 (socket says 172.21.15.23:58930) 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.493+0000 7fcd1ffff640 1 -- 172.21.15.23:0/4164405260 learned_addr learned my addr 172.21.15.23:0/4164405260 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.493+0000 7fcd2484b640 1 --2- 172.21.15.23:0/4164405260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd2010bcf0 0x7fcd2019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.493+0000 7fcd17fff640 1 --2- 172.21.15.23:0/4164405260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd20103240 0x7fcd2019a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.494+0000 7fcd1ffff640 1 -- 172.21.15.23:0/4164405260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd20103240 msgr2=0x7fcd2019a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.494+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/4164405260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd20103240 0x7fcd2019a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.494+0000 7fcd1ffff640 1 -- 172.21.15.23:0/4164405260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd2010bcf0 msgr2=0x7fcd2019e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.494+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/4164405260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd2010bcf0 0x7fcd2019e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.494+0000 7fcd1ffff640 1 -- 172.21.15.23:0/4164405260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd08009560 con 0x7fcd20102920 2024-03-24T10:46:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.494+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/4164405260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd20102920 0x7fcd2019a060 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fcd1000e9f0 tx=0x7fcd1000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.494+0000 7fcd1dffb640 1 -- 172.21.15.23:0/4164405260 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd10009720 con 0x7fcd20102920 2024-03-24T10:46:42.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.494+0000 7fcd262ae640 1 -- 172.21.15.23:0/4164405260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd2019f530 con 0x7fcd20102920 2024-03-24T10:46:42.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.494+0000 7fcd1dffb640 1 -- 172.21.15.23:0/4164405260 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd1000cd80 con 0x7fcd20102920 2024-03-24T10:46:42.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.494+0000 7fcd1dffb640 1 -- 172.21.15.23:0/4164405260 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd10010690 con 0x7fcd20102920 2024-03-24T10:46:42.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.494+0000 7fcd262ae640 1 -- 172.21.15.23:0/4164405260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd201a6db0 con 0x7fcd20102920 2024-03-24T10:46:42.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.496+0000 7fcd1dffb640 1 -- 172.21.15.23:0/4164405260 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fcd100046c0 con 0x7fcd20102920 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.496+0000 7fcd28f6e740 1 Processor -- start 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.496+0000 7fcd28f6e740 1 -- start start 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd28f6e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce403f0 0x55cfcce407d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd28f6e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce40d10 0x55cfcce493b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd28f6e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce498f0 0x55cfcce4bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd28f6e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55cfccdae840 con 0x55cfcce498f0 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd28f6e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cfccdabd70 con 0x55cfcce403f0 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd28f6e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cfccda7550 con 0x55cfcce40d10 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd1dffb640 1 --2- 172.21.15.23:0/4164405260 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcd0c0a2230 0x7fcd0c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd17fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce498f0 0x55cfcce4bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd2484b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce403f0 0x55cfcce407d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd17fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce498f0 0x55cfcce4bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58934/0 (socket says 172.21.15.23:58934) 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd17fff640 1 -- 172.21.15.23:0/2648085903 learned_addr learned my addr 172.21.15.23:0/2648085903 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd2484b640 1 --2- 172.21.15.23:0/4164405260 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcd0c0a2230 0x7fcd0c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/2648085903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce40d10 0x55cfcce493b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd1dffb640 1 -- 172.21.15.23:0/4164405260 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fcd10014070 con 0x7fcd20102920 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd17fff640 1 -- 172.21.15.23:0/2648085903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce40d10 msgr2=0x55cfcce493b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd17fff640 1 --2- 172.21.15.23:0/2648085903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce40d10 0x55cfcce493b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd17fff640 1 -- 172.21.15.23:0/2648085903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce403f0 msgr2=0x55cfcce407d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd17fff640 1 --2- 172.21.15.23:0/2648085903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce403f0 0x55cfcce407d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd17fff640 1 -- 172.21.15.23:0/2648085903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfccd3cde0 con 0x55cfcce498f0 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/2648085903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce40d10 0x55cfcce493b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd2484b640 1 --2- 172.21.15.23:0/4164405260 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcd0c0a2230 0x7fcd0c0a46f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009fd0 tx=0x7fcd18009e00 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd17fff640 1 --2- 172.21.15.23:0/2648085903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce498f0 0x55cfcce4bce0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fcd08009900 tx=0x7fcd0802c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2576d382a2cf47e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.497+0000 7fcd167fc640 1 -- 172.21.15.23:0/2648085903 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd08004740 con 0x55cfcce498f0 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.498+0000 7fcd167fc640 1 -- 172.21.15.23:0/2648085903 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd080048a0 con 0x55cfcce498f0 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.498+0000 7fcd167fc640 1 -- 172.21.15.23:0/2648085903 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd08005050 con 0x55cfcce498f0 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.498+0000 7fcd28f6e740 1 -- 172.21.15.23:0/2648085903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce498f0 msgr2=0x55cfcce4bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.498+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/2648085903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce498f0 0x55cfcce4bce0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fcd08009900 tx=0x7fcd0802c740 comp rx=0 tx=0).stop 2024-03-24T10:46:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.498+0000 7fcd28f6e740 1 -- 172.21.15.23:0/2648085903 shutdown_connections 2024-03-24T10:46:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.498+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/2648085903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce498f0 0x55cfcce4bce0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.498+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/2648085903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce40d10 0x55cfcce493b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.498+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/2648085903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce403f0 0x55cfcce407d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.498+0000 7fcd28f6e740 1 -- 172.21.15.23:0/2648085903 >> 172.21.15.23:0/2648085903 conn(0x55cfcce3ab20 msgr2=0x55cfcce3af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.498+0000 7fcd28f6e740 1 -- 172.21.15.23:0/2648085903 shutdown_connections 2024-03-24T10:46:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.498+0000 7fcd28f6e740 1 -- 172.21.15.23:0/2648085903 wait complete. 2024-03-24T10:46:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd28f6e740 1 -- start start 2024-03-24T10:46:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd28f6e740 1 Processor -- start 2024-03-24T10:46:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd28f6e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce403f0 0x55cfcce468b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd28f6e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce46df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd28f6e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce498f0 0x55cfcce47330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd28f6e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55cfcca7a430 con 0x55cfcce403f0 2024-03-24T10:46:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd28f6e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cfcca36e00 con 0x55cfcce40d10 2024-03-24T10:46:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd28f6e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cfccd2c590 con 0x55cfcce498f0 2024-03-24T10:46:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd28f6e740 10 client.0 _ll_register_callbacks cb 0x55cfccd9bfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd1ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce403f0 0x55cfcce468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce46df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce46df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:55022/0 (socket says 172.21.15.23:55022) 2024-03-24T10:46:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd17fff640 1 -- 172.21.15.23:0/4012556211 learned_addr learned my addr 172.21.15.23:0/4012556211 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/4012556211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce403f0 0x55cfcce468b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58938/0 (socket says 172.21.15.23:58938) 2024-03-24T10:46:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd2484b640 1 --2- 172.21.15.23:0/4012556211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce498f0 0x55cfcce47330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd17fff640 1 -- 172.21.15.23:0/4012556211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce498f0 msgr2=0x55cfcce47330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd17fff640 1 --2- 172.21.15.23:0/4012556211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce498f0 0x55cfcce47330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd17fff640 1 -- 172.21.15.23:0/4012556211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce403f0 msgr2=0x55cfcce468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.499+0000 7fcd17fff640 1 --2- 172.21.15.23:0/4012556211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce403f0 0x55cfcce468b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.500+0000 7fcd17fff640 1 -- 172.21.15.23:0/4012556211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd2010eda0 con 0x55cfcce40d10 2024-03-24T10:46:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.500+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/4012556211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce403f0 0x55cfcce468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.500+0000 7fcd17fff640 1 --2- 172.21.15.23:0/4012556211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce46df0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcd080045c0 tx=0x7fcd08004370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c253f5e3231b9470 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.500+0000 7fccff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:46:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.500+0000 7fccff7fe640 1 -- 172.21.15.23:0/4012556211 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd08004740 con 0x55cfcce40d10 2024-03-24T10:46:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.500+0000 7fccff7fe640 1 -- 172.21.15.23:0/4012556211 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fcd0802cc80 con 0x55cfcce40d10 2024-03-24T10:46:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.500+0000 7fcd28f6e740 1 -- 172.21.15.23:0/4012556211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cfccaa67e0 con 0x55cfcce40d10 2024-03-24T10:46:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.500+0000 7fccff7fe640 1 -- 172.21.15.23:0/4012556211 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcd08005200 con 0x55cfcce40d10 2024-03-24T10:46:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.500+0000 7fcd28f6e740 10 client.27323 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.500+0000 7fcd28f6e740 1 -- 172.21.15.23:0/4012556211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cfccda5bd0 con 0x55cfcce40d10 2024-03-24T10:46:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.500+0000 7fccff7fe640 1 -- 172.21.15.23:0/4012556211 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55cfccaa67e0 con 0x55cfcce40d10 2024-03-24T10:46:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.501+0000 7fccff7fe640 1 -- 172.21.15.23:0/4012556211 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fcd0803c050 con 0x55cfcce40d10 2024-03-24T10:46:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.501+0000 7fcd28f6e740 10 client.27323 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.501+0000 7fcd28f6e740 10 client.27323 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:46:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.501+0000 7fcd28f6e740 20 client.27323 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.501+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/4012556211 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cfcce593d0 0x55cfcce5b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.501+0000 7fcd28f6e740 4 client.27323 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.501+0000 7fcd28f6e740 1 -- 172.21.15.23:0/4012556211 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfccae3720 con 0x55cfcce593d0 2024-03-24T10:46:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.501+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/4012556211 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cfcce593d0 0x55cfcce5b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.501+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/4012556211 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cfcce593d0 0x55cfcce5b810 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2a7bad7d1187cc02 server_cookie=a3e44f58349bacbd in_seq=0 out_seq=0 2024-03-24T10:46:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.501+0000 7fccff7fe640 10 client.27323 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:46:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fccff7fe640 1 -- 172.21.15.23:0/4012556211 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cfccae3720 con 0x55cfcce593d0 2024-03-24T10:46:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fccff7fe640 10 client.27323 handle_command_reply: tid=0 2024-03-24T10:46:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fcd28f6e740 1 client.27323 shutdown 2024-03-24T10:46:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fcd28f6e740 1 -- 172.21.15.23:0/4012556211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 msgr2=0x55cfcce46df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/4012556211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce46df0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcd080045c0 tx=0x7fcd08004370 comp rx=0 tx=0).stop 2024-03-24T10:46:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fcd28f6e740 1 -- 172.21.15.23:0/4012556211 shutdown_connections 2024-03-24T10:46:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/4012556211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce498f0 0x55cfcce47330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/4012556211 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cfcce593d0 0x55cfcce5b810 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/4012556211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce46df0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/4012556211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce403f0 0x55cfcce468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fccff7fe640 0 client.27323 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:46:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fcd28f6e740 1 -- 172.21.15.23:0/4012556211 >> 172.21.15.23:0/4012556211 conn(0x55cfcce3ab20 msgr2=0x55cfcce3c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fcd28f6e740 1 -- 172.21.15.23:0/4012556211 shutdown_connections 2024-03-24T10:46:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fcd28f6e740 1 -- 172.21.15.23:0/4012556211 wait complete. 2024-03-24T10:46:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.505+0000 7fcd28f6e740 20 client.27323 trim_cache size 0 max 16384 2024-03-24T10:46:42.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd28f6e740 1 Processor -- start 2024-03-24T10:46:42.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd28f6e740 1 -- start start 2024-03-24T10:46:42.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd28f6e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce403f0 0x55cfcce48fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd28f6e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce55000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd28f6e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce498f0 0x55cfcce61120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd28f6e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55cfcce3bb20 con 0x55cfcce498f0 2024-03-24T10:46:42.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd28f6e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cfcce2be20 con 0x55cfcce40d10 2024-03-24T10:46:42.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd28f6e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cfcce3a950 con 0x55cfcce403f0 2024-03-24T10:46:42.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd2484b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce498f0 0x55cfcce61120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd1ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce403f0 0x55cfcce48fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce55000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd1ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce403f0 0x55cfcce48fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:47622/0 (socket says 172.21.15.23:47622) 2024-03-24T10:46:42.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd2484b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce498f0 0x55cfcce61120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58940/0 (socket says 172.21.15.23:58940) 2024-03-24T10:46:42.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd1ffff640 1 -- 172.21.15.23:0/219696869 learned_addr learned my addr 172.21.15.23:0/219696869 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.519+0000 7fcd17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce55000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:55038/0 (socket says 172.21.15.23:55038) 2024-03-24T10:46:42.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.520+0000 7fcd1ffff640 1 -- 172.21.15.23:0/219696869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 msgr2=0x55cfcce55000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.520+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/219696869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce55000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.520+0000 7fcd1ffff640 1 -- 172.21.15.23:0/219696869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce498f0 msgr2=0x55cfcce61120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.520+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/219696869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce498f0 0x55cfcce61120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.520+0000 7fcd1ffff640 1 -- 172.21.15.23:0/219696869 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfccd41c80 con 0x55cfcce403f0 2024-03-24T10:46:42.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.520+0000 7fcd17fff640 1 --2- 172.21.15.23:0/219696869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce55000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:42.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.520+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/219696869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce403f0 0x55cfcce48fe0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcd10097260 tx=0x7fcd10014f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=225cae295be9f965 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.520+0000 7fcd15ffb640 1 -- 172.21.15.23:0/219696869 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1009d040 con 0x55cfcce403f0 2024-03-24T10:46:42.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.520+0000 7fcd15ffb640 1 -- 172.21.15.23:0/219696869 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd100b9330 con 0x55cfcce403f0 2024-03-24T10:46:42.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.520+0000 7fcd15ffb640 1 -- 172.21.15.23:0/219696869 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd10097470 con 0x55cfcce403f0 2024-03-24T10:46:42.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.521+0000 7fcd28f6e740 1 -- 172.21.15.23:0/219696869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce403f0 msgr2=0x55cfcce48fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.521+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/219696869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce403f0 0x55cfcce48fe0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcd10097260 tx=0x7fcd10014f10 comp rx=0 tx=0).stop 2024-03-24T10:46:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.521+0000 7fcd28f6e740 1 -- 172.21.15.23:0/219696869 shutdown_connections 2024-03-24T10:46:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.521+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/219696869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce498f0 0x55cfcce61120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.521+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/219696869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce55000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.521+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/219696869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce403f0 0x55cfcce48fe0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.521+0000 7fcd28f6e740 1 -- 172.21.15.23:0/219696869 >> 172.21.15.23:0/219696869 conn(0x55cfcce3ab20 msgr2=0x55cfcce3c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.521+0000 7fcd28f6e740 1 -- 172.21.15.23:0/219696869 shutdown_connections 2024-03-24T10:46:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.521+0000 7fcd28f6e740 1 -- 172.21.15.23:0/219696869 wait complete. 2024-03-24T10:46:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.521+0000 7fcd28f6e740 1 -- start start 2024-03-24T10:46:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd28f6e740 1 Processor -- start 2024-03-24T10:46:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd28f6e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce403f0 0x55cfcce4d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd28f6e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce4dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd28f6e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce498f0 0x55cfcceb8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd28f6e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55cfcceb8c60 con 0x55cfcce403f0 2024-03-24T10:46:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd28f6e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cfcceb8dd0 con 0x55cfcce40d10 2024-03-24T10:46:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd28f6e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cfcceb8f90 con 0x55cfcce498f0 2024-03-24T10:46:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd28f6e740 10 client.0 _ll_register_callbacks cb 0x7fcd180105f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd1ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce403f0 0x55cfcce4d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd1ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce403f0 0x55cfcce4d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58950/0 (socket says 172.21.15.23:58950) 2024-03-24T10:46:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd1ffff640 1 -- 172.21.15.23:0/3798103610 learned_addr learned my addr 172.21.15.23:0/3798103610 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd2484b640 1 --2- 172.21.15.23:0/3798103610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce498f0 0x55cfcceb8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd17fff640 1 --2- 172.21.15.23:0/3798103610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce4dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd1ffff640 1 -- 172.21.15.23:0/3798103610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce498f0 msgr2=0x55cfcceb8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/3798103610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce498f0 0x55cfcceb8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd1ffff640 1 -- 172.21.15.23:0/3798103610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 msgr2=0x55cfcce4dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/3798103610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce4dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd1ffff640 1 -- 172.21.15.23:0/3798103610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfccda5bd0 con 0x55cfcce403f0 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fcd1ffff640 1 --2- 172.21.15.23:0/3798103610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce403f0 0x55cfcce4d7e0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fcd100ce390 tx=0x7fcd100ce3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3939f1fb31946333 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fccff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.522+0000 7fccff7fe640 1 -- 172.21.15.23:0/3798103610 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1009d040 con 0x55cfcce403f0 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.523+0000 7fccff7fe640 1 -- 172.21.15.23:0/3798103610 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7fcd100b7070 con 0x55cfcce403f0 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.523+0000 7fcd28f6e740 1 -- 172.21.15.23:0/3798103610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd080048a0 con 0x55cfcce403f0 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.523+0000 7fccff7fe640 1 -- 172.21.15.23:0/3798103610 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcd1009ef50 con 0x55cfcce403f0 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.523+0000 7fcd28f6e740 10 client.17442 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.523+0000 7fcd28f6e740 1 -- 172.21.15.23:0/3798103610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x7fcd2019f530 con 0x55cfcce403f0 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.523+0000 7fccff7fe640 1 -- 172.21.15.23:0/3798103610 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd080048a0 con 0x55cfcce403f0 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.523+0000 7fccff7fe640 1 -- 172.21.15.23:0/3798103610 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fcd100cd040 con 0x55cfcce403f0 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.523+0000 7fcd28f6e740 10 client.17442 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.523+0000 7fcd28f6e740 10 client.17442 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.523+0000 7fcd28f6e740 20 client.17442 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.523+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/3798103610 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cfccec0500 0x55cfccec28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.523+0000 7fcd28f6e740 4 client.17442 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.523+0000 7fcd28f6e740 1 -- 172.21.15.23:0/3798103610 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cfccb32890 con 0x55cfccec0500 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.523+0000 7fcd17fff640 1 --2- 172.21.15.23:0/3798103610 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cfccec0500 0x55cfccec28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.524+0000 7fcd17fff640 1 --2- 172.21.15.23:0/3798103610 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cfccec0500 0x55cfccec28f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e558900f1d550eb9 server_cookie=c8737e358afb3a11 in_seq=0 out_seq=0 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.524+0000 7fccff7fe640 10 client.17442 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.524+0000 7fccff7fe640 1 -- 172.21.15.23:0/3798103610 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cfccb32890 con 0x55cfccec0500 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.524+0000 7fccff7fe640 10 client.17442 handle_command_reply: tid=0 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.524+0000 7fcd28f6e740 1 client.17442 shutdown 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd28f6e740 1 -- 172.21.15.23:0/3798103610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce403f0 msgr2=0x55cfcce4d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/3798103610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce403f0 0x55cfcce4d7e0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fcd100ce390 tx=0x7fcd100ce3c0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd28f6e740 1 -- 172.21.15.23:0/3798103610 shutdown_connections 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/3798103610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cfcce498f0 0x55cfcceb8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/3798103610 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55cfccec0500 0x55cfccec28f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/3798103610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cfcce40d10 0x55cfcce4dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd28f6e740 1 --2- 172.21.15.23:0/3798103610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cfcce403f0 0x55cfcce4d7e0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fccff7fe640 0 client.17442 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:46:42.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd28f6e740 1 -- 172.21.15.23:0/3798103610 >> 172.21.15.23:0/3798103610 conn(0x55cfcce3ab20 msgr2=0x55cfcce46850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd28f6e740 1 -- 172.21.15.23:0/3798103610 shutdown_connections 2024-03-24T10:46:42.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd28f6e740 1 -- 172.21.15.23:0/3798103610 wait complete. 2024-03-24T10:46:42.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd28f6e740 20 client.17442 trim_cache size 0 max 16384 2024-03-24T10:46:42.530 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:46:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd262ae640 1 -- 172.21.15.23:0/4164405260 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcd0c0a2230 msgr2=0x7fcd0c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd262ae640 1 --2- 172.21.15.23:0/4164405260 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcd0c0a2230 0x7fcd0c0a46f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009fd0 tx=0x7fcd18009e00 comp rx=0 tx=0).stop 2024-03-24T10:46:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd262ae640 1 -- 172.21.15.23:0/4164405260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd20102920 msgr2=0x7fcd2019a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd262ae640 1 --2- 172.21.15.23:0/4164405260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd20102920 0x7fcd2019a060 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fcd1000e9f0 tx=0x7fcd1000edf0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd262ae640 1 -- 172.21.15.23:0/4164405260 shutdown_connections 2024-03-24T10:46:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd262ae640 1 --2- 172.21.15.23:0/4164405260 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcd0c0a2230 0x7fcd0c0a46f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd262ae640 1 --2- 172.21.15.23:0/4164405260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd2010bcf0 0x7fcd2019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd262ae640 1 --2- 172.21.15.23:0/4164405260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd20103240 0x7fcd2019a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd262ae640 1 --2- 172.21.15.23:0/4164405260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd20102920 0x7fcd2019a060 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd262ae640 1 -- 172.21.15.23:0/4164405260 >> 172.21.15.23:0/4164405260 conn(0x7fcd20076990 msgr2=0x7fcd201005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd262ae640 1 -- 172.21.15.23:0/4164405260 shutdown_connections 2024-03-24T10:46:42.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.525+0000 7fcd262ae640 1 -- 172.21.15.23:0/4164405260 wait complete. 2024-03-24T10:46:42.537 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:46:42.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.645+0000 7f023e7c9640 1 -- 172.21.15.23:0/2037547276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02381005a0 msgr2=0x7f0238100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.645+0000 7f023e7c9640 1 --2- 172.21.15.23:0/2037547276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02381005a0 0x7f0238100a20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f022c009890 tx=0x7f022c02bb70 comp rx=0 tx=0).stop 2024-03-24T10:46:42.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.645+0000 7f023e7c9640 1 -- 172.21.15.23:0/2037547276 shutdown_connections 2024-03-24T10:46:42.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.645+0000 7f023e7c9640 1 --2- 172.21.15.23:0/2037547276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0238100f60 0x7f023810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.645+0000 7f023e7c9640 1 --2- 172.21.15.23:0/2037547276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02381005a0 0x7f0238100a20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.645+0000 7f023e7c9640 1 --2- 172.21.15.23:0/2037547276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0238106660 0x7f0238106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.645+0000 7f023e7c9640 1 -- 172.21.15.23:0/2037547276 >> 172.21.15.23:0/2037547276 conn(0x7f0238077710 msgr2=0x7f0238077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.645+0000 7f023e7c9640 1 -- 172.21.15.23:0/2037547276 shutdown_connections 2024-03-24T10:46:42.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.645+0000 7f023e7c9640 1 -- 172.21.15.23:0/2037547276 wait complete. 2024-03-24T10:46:42.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.646+0000 7f023e7c9640 1 Processor -- start 2024-03-24T10:46:42.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.646+0000 7f023e7c9640 1 -- start start 2024-03-24T10:46:42.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.646+0000 7f023e7c9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02381005a0 0x7f0238198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.646+0000 7f023e7c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0238100f60 0x7f0238199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.646+0000 7f023e7c9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0238106660 0x7f02381a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.646+0000 7f023e7c9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f02381998e0 con 0x7f0238106660 2024-03-24T10:46:42.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.646+0000 7f023e7c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02381a40e0 con 0x7f0238100f60 2024-03-24T10:46:42.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.646+0000 7f023e7c9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f02381a4220 con 0x7f02381005a0 2024-03-24T10:46:42.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.646+0000 7f023cd66640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0238106660 0x7f02381a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.646+0000 7f02377fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0238100f60 0x7f0238199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.646+0000 7f0237fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02381005a0 0x7f0238198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.646+0000 7f02377fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0238100f60 0x7f0238199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:55052/0 (socket says 172.21.15.23:55052) 2024-03-24T10:46:42.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.647+0000 7f02377fe640 1 -- 172.21.15.23:0/3560385629 learned_addr learned my addr 172.21.15.23:0/3560385629 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.647+0000 7f023cd66640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0238106660 0x7f02381a3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58964/0 (socket says 172.21.15.23:58964) 2024-03-24T10:46:42.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.647+0000 7f023cd66640 1 -- 172.21.15.23:0/3560385629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02381005a0 msgr2=0x7f0238198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.647+0000 7f023cd66640 1 --2- 172.21.15.23:0/3560385629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02381005a0 0x7f0238198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.647+0000 7f023cd66640 1 -- 172.21.15.23:0/3560385629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0238100f60 msgr2=0x7f0238199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.647+0000 7f023cd66640 1 --2- 172.21.15.23:0/3560385629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0238100f60 0x7f0238199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.647+0000 7f023cd66640 1 -- 172.21.15.23:0/3560385629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0228009560 con 0x7f0238106660 2024-03-24T10:46:42.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.647+0000 7f02377fe640 1 --2- 172.21.15.23:0/3560385629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0238100f60 0x7f0238199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:42.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.647+0000 7f023cd66640 1 --2- 172.21.15.23:0/3560385629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0238106660 0x7f02381a3ba0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f023000ea80 tx=0x7f023000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.647+0000 7f02357fa640 1 -- 172.21.15.23:0/3560385629 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023000ce40 con 0x7f0238106660 2024-03-24T10:46:42.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.647+0000 7f023e7c9640 1 -- 172.21.15.23:0/3560385629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f022c009490 con 0x7f0238106660 2024-03-24T10:46:42.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.647+0000 7f02357fa640 1 -- 172.21.15.23:0/3560385629 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0230005020 con 0x7f0238106660 2024-03-24T10:46:42.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.647+0000 7f02357fa640 1 -- 172.21.15.23:0/3560385629 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0230010670 con 0x7f0238106660 2024-03-24T10:46:42.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.647+0000 7f023e7c9640 1 -- 172.21.15.23:0/3560385629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02381a4800 con 0x7f0238106660 2024-03-24T10:46:42.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.649+0000 7f02357fa640 1 -- 172.21.15.23:0/3560385629 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f0230005500 con 0x7f0238106660 2024-03-24T10:46:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02413ad740 1 Processor -- start 2024-03-24T10:46:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02413ad740 1 -- start start 2024-03-24T10:46:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02413ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad963f0 0x5629dad967d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02413ad740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad96d10 0x5629dad9f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02413ad740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad9f8f0 0x5629dada1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02413ad740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5629dad04840 con 0x5629dad96d10 2024-03-24T10:46:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02413ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5629dad01d70 con 0x5629dad963f0 2024-03-24T10:46:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02413ad740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5629dacfd550 con 0x5629dad9f8f0 2024-03-24T10:46:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02357fa640 1 --2- 172.21.15.23:0/3560385629 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f02200a2230 0x7f02200a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f0237fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad96d10 0x5629dad9f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02377fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad963f0 0x5629dad967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02377fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad963f0 0x5629dad967d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:55054/0 (socket says 172.21.15.23:55054) 2024-03-24T10:46:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02377fe640 1 -- 172.21.15.23:0/1590561708 learned_addr learned my addr 172.21.15.23:0/1590561708 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02377fe640 1 --2- 172.21.15.23:0/1590561708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad9f8f0 0x5629dada1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f023cd66640 1 --2- 172.21.15.23:0/3560385629 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f02200a2230 0x7f02200a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02357fa640 1 -- 172.21.15.23:0/3560385629 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f0230014070 con 0x7f0238106660 2024-03-24T10:46:42.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02377fe640 1 -- 172.21.15.23:0/1590561708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad9f8f0 msgr2=0x5629dada1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.650+0000 7f02377fe640 1 --2- 172.21.15.23:0/1590561708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad9f8f0 0x5629dada1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.651+0000 7f02377fe640 1 -- 172.21.15.23:0/1590561708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad96d10 msgr2=0x5629dad9f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.651+0000 7f02377fe640 1 --2- 172.21.15.23:0/1590561708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad96d10 0x5629dad9f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.651+0000 7f02377fe640 1 -- 172.21.15.23:0/1590561708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629dac92de0 con 0x5629dad963f0 2024-03-24T10:46:42.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.651+0000 7f02377fe640 1 --2- 172.21.15.23:0/1590561708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad963f0 0x5629dad967d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f022c005e00 tx=0x7f022c02e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=259e46525dbcf396 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.651+0000 7f023cd66640 1 --2- 172.21.15.23:0/3560385629 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f02200a2230 0x7f02200a46f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f023009ae20 tx=0x7f023006d0e0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.651+0000 7f021e7fc640 1 -- 172.21.15.23:0/1590561708 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c034640 con 0x5629dad963f0 2024-03-24T10:46:42.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.651+0000 7f021e7fc640 1 -- 172.21.15.23:0/1590561708 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f022c004510 con 0x5629dad963f0 2024-03-24T10:46:42.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.651+0000 7f021e7fc640 1 -- 172.21.15.23:0/1590561708 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c032860 con 0x5629dad963f0 2024-03-24T10:46:42.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.652+0000 7f02413ad740 1 -- 172.21.15.23:0/1590561708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad963f0 msgr2=0x5629dad967d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.652+0000 7f02413ad740 1 --2- 172.21.15.23:0/1590561708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad963f0 0x5629dad967d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f022c005e00 tx=0x7f022c02e040 comp rx=0 tx=0).stop 2024-03-24T10:46:42.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.652+0000 7f02413ad740 1 -- 172.21.15.23:0/1590561708 shutdown_connections 2024-03-24T10:46:42.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.652+0000 7f02413ad740 1 --2- 172.21.15.23:0/1590561708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad9f8f0 0x5629dada1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.652+0000 7f02413ad740 1 --2- 172.21.15.23:0/1590561708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad96d10 0x5629dad9f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.652+0000 7f02413ad740 1 --2- 172.21.15.23:0/1590561708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad963f0 0x5629dad967d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.652+0000 7f02413ad740 1 -- 172.21.15.23:0/1590561708 >> 172.21.15.23:0/1590561708 conn(0x5629dad90b20 msgr2=0x5629dad90f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.652+0000 7f02413ad740 1 -- 172.21.15.23:0/1590561708 shutdown_connections 2024-03-24T10:46:42.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.652+0000 7f02413ad740 1 -- 172.21.15.23:0/1590561708 wait complete. 2024-03-24T10:46:42.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.652+0000 7f02413ad740 1 -- start start 2024-03-24T10:46:42.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.652+0000 7f02413ad740 1 Processor -- start 2024-03-24T10:46:42.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.652+0000 7f02413ad740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad963f0 0x5629dad9c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.652+0000 7f02413ad740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad96d10 0x5629dad9cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.652+0000 7f02413ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad9f8f0 0x5629dad9d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02413ad740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5629da9d0430 con 0x5629dad963f0 2024-03-24T10:46:42.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02413ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5629da98ce00 con 0x5629dad9f8f0 2024-03-24T10:46:42.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02413ad740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5629dac82590 con 0x5629dad96d10 2024-03-24T10:46:42.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02413ad740 10 client.0 _ll_register_callbacks cb 0x5629dacf1fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:42.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02377fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad963f0 0x5629dad9c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02377fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad963f0 0x5629dad9c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58988/0 (socket says 172.21.15.23:58988) 2024-03-24T10:46:42.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02377fe640 1 -- 172.21.15.23:0/2766751669 learned_addr learned my addr 172.21.15.23:0/2766751669 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f0237fff640 1 --2- 172.21.15.23:0/2766751669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad96d10 0x5629dad9cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02377fe640 1 --2- 172.21.15.23:0/2766751669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad9f8f0 0x5629dad9d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02377fe640 1 -- 172.21.15.23:0/2766751669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad96d10 msgr2=0x5629dad9cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02377fe640 1 --2- 172.21.15.23:0/2766751669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad96d10 0x5629dad9cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02377fe640 1 -- 172.21.15.23:0/2766751669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad9f8f0 msgr2=0x5629dad9d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02377fe640 1 --2- 172.21.15.23:0/2766751669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad9f8f0 0x5629dad9d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02377fe640 1 -- 172.21.15.23:0/2766751669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f023810dd20 con 0x5629dad963f0 2024-03-24T10:46:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02377fe640 1 --2- 172.21.15.23:0/2766751669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad963f0 0x5629dad9c960 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f022c000c00 tx=0x7f022c005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6db841ee8101d33b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02177fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02177fe640 1 -- 172.21.15.23:0/2766751669 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c02c9d0 con 0x5629dad963f0 2024-03-24T10:46:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02413ad740 1 -- 172.21.15.23:0/2766751669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5629da9fc7e0 con 0x5629dad963f0 2024-03-24T10:46:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02177fe640 1 -- 172.21.15.23:0/2766751669 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f022c02cb30 con 0x5629dad963f0 2024-03-24T10:46:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02177fe640 1 -- 172.21.15.23:0/2766751669 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f022c032600 con 0x5629dad963f0 2024-03-24T10:46:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02413ad740 10 client.17454 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.653+0000 7f02413ad740 1 -- 172.21.15.23:0/2766751669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5629dacfbbd0 con 0x5629dad963f0 2024-03-24T10:46:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.654+0000 7f02177fe640 1 -- 172.21.15.23:0/2766751669 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5629da9fc7e0 con 0x5629dad963f0 2024-03-24T10:46:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.654+0000 7f02177fe640 1 -- 172.21.15.23:0/2766751669 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f022c04b020 con 0x5629dad963f0 2024-03-24T10:46:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.654+0000 7f02413ad740 10 client.17454 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.654+0000 7f02413ad740 10 client.17454 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:46:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.654+0000 7f02413ad740 20 client.17454 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:42.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.654+0000 7f02413ad740 1 --2- 172.21.15.23:0/2766751669 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5629dadaf520 0x5629dadb1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.654+0000 7f02413ad740 4 client.17454 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:42.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.654+0000 7f02413ad740 1 -- 172.21.15.23:0/2766751669 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629daa39720 con 0x5629dadaf520 2024-03-24T10:46:42.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.654+0000 7f0237fff640 1 --2- 172.21.15.23:0/2766751669 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5629dadaf520 0x5629dadb1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.654+0000 7f02177fe640 10 client.17454 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:46:42.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.654+0000 7f0237fff640 1 --2- 172.21.15.23:0/2766751669 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5629dadaf520 0x5629dadb1940 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=56d794176b1a34f8 server_cookie=578baa7cc69e9f88 in_seq=0 out_seq=0 2024-03-24T10:46:42.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02177fe640 1 -- 172.21.15.23:0/2766751669 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5629daa39720 con 0x5629dadaf520 2024-03-24T10:46:42.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02177fe640 10 client.17454 handle_command_reply: tid=0 2024-03-24T10:46:42.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02413ad740 1 client.17454 shutdown 2024-03-24T10:46:42.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02413ad740 1 -- 172.21.15.23:0/2766751669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad963f0 msgr2=0x5629dad9c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02413ad740 1 --2- 172.21.15.23:0/2766751669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad963f0 0x5629dad9c960 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f022c000c00 tx=0x7f022c005d20 comp rx=0 tx=0).stop 2024-03-24T10:46:42.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02413ad740 1 -- 172.21.15.23:0/2766751669 shutdown_connections 2024-03-24T10:46:42.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02413ad740 1 --2- 172.21.15.23:0/2766751669 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5629dadaf520 0x5629dadb1940 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02413ad740 1 --2- 172.21.15.23:0/2766751669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad9f8f0 0x5629dad9d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02413ad740 1 --2- 172.21.15.23:0/2766751669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad96d10 0x5629dad9cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02177fe640 0 client.17454 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:46:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02413ad740 1 --2- 172.21.15.23:0/2766751669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad963f0 0x5629dad9c960 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02413ad740 1 -- 172.21.15.23:0/2766751669 >> 172.21.15.23:0/2766751669 conn(0x5629dad90b20 msgr2=0x5629dad92870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02413ad740 1 -- 172.21.15.23:0/2766751669 shutdown_connections 2024-03-24T10:46:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02413ad740 1 -- 172.21.15.23:0/2766751669 wait complete. 2024-03-24T10:46:42.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.655+0000 7f02413ad740 20 client.17454 trim_cache size 0 max 16384 2024-03-24T10:46:42.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:42 smithi023 ceph-mon[32466]: pgmap v202: 97 pgs: 97 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 259 op/s 2024-03-24T10:46:42.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:42 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3409041395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:46:42.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02413ad740 1 Processor -- start 2024-03-24T10:46:42.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02413ad740 1 -- start start 2024-03-24T10:46:42.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02413ad740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad963f0 0x5629dad9f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02413ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad96d10 0x5629dadab120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02413ad740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad9f8f0 0x5629dad8fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02413ad740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5629dad9c750 con 0x5629dad9f8f0 2024-03-24T10:46:42.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02413ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5629dad8c3e0 con 0x5629dad96d10 2024-03-24T10:46:42.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02413ad740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5629dad8b0d0 con 0x5629dad963f0 2024-03-24T10:46:42.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02377fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad9f8f0 0x5629dad8fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02377fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad9f8f0 0x5629dad8fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59000/0 (socket says 172.21.15.23:59000) 2024-03-24T10:46:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02377fe640 1 -- 172.21.15.23:0/2574201040 learned_addr learned my addr 172.21.15.23:0/2574201040 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f0237fff640 1 --2- 172.21.15.23:0/2574201040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad96d10 0x5629dadab120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02377fe640 1 --2- 172.21.15.23:0/2574201040 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad963f0 0x5629dad9f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02377fe640 1 -- 172.21.15.23:0/2574201040 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad963f0 msgr2=0x5629dad9f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02377fe640 1 --2- 172.21.15.23:0/2574201040 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad963f0 0x5629dad9f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02377fe640 1 -- 172.21.15.23:0/2574201040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad96d10 msgr2=0x5629dadab120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02377fe640 1 --2- 172.21.15.23:0/2574201040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad96d10 0x5629dadab120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02377fe640 1 -- 172.21.15.23:0/2574201040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629dac97c80 con 0x5629dad9f8f0 2024-03-24T10:46:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f0237fff640 1 --2- 172.21.15.23:0/2574201040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad96d10 0x5629dadab120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:46:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.669+0000 7f02377fe640 1 --2- 172.21.15.23:0/2574201040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad9f8f0 0x5629dad8fb40 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f022c005cf0 tx=0x7f022c044040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc6ece07ab7bca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.670+0000 7f021dffb640 1 -- 172.21.15.23:0/2574201040 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c032340 con 0x5629dad9f8f0 2024-03-24T10:46:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.670+0000 7f021dffb640 1 -- 172.21.15.23:0/2574201040 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f022c02c270 con 0x5629dad9f8f0 2024-03-24T10:46:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.670+0000 7f021dffb640 1 -- 172.21.15.23:0/2574201040 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c034820 con 0x5629dad9f8f0 2024-03-24T10:46:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.670+0000 7f02413ad740 1 -- 172.21.15.23:0/2574201040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad9f8f0 msgr2=0x5629dad8fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.670+0000 7f02413ad740 1 --2- 172.21.15.23:0/2574201040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad9f8f0 0x5629dad8fb40 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f022c005cf0 tx=0x7f022c044040 comp rx=0 tx=0).stop 2024-03-24T10:46:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.670+0000 7f02413ad740 1 -- 172.21.15.23:0/2574201040 shutdown_connections 2024-03-24T10:46:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.670+0000 7f02413ad740 1 --2- 172.21.15.23:0/2574201040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad9f8f0 0x5629dad8fb40 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.670+0000 7f02413ad740 1 --2- 172.21.15.23:0/2574201040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad96d10 0x5629dadab120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.670+0000 7f02413ad740 1 --2- 172.21.15.23:0/2574201040 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad963f0 0x5629dad9f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.670+0000 7f02413ad740 1 -- 172.21.15.23:0/2574201040 >> 172.21.15.23:0/2574201040 conn(0x5629dad90b20 msgr2=0x5629dad8f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.670+0000 7f02413ad740 1 -- 172.21.15.23:0/2574201040 shutdown_connections 2024-03-24T10:46:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.670+0000 7f02413ad740 1 -- 172.21.15.23:0/2574201040 wait complete. 2024-03-24T10:46:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f02413ad740 1 -- start start 2024-03-24T10:46:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f02413ad740 1 Processor -- start 2024-03-24T10:46:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f02413ad740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad963f0 0x5629dada2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f02413ad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad96d10 0x5629dada3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f02413ad740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad9f8f0 0x5629dada3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f02413ad740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5629dada7fa0 con 0x5629dad9f8f0 2024-03-24T10:46:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f02413ad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5629dae14840 con 0x5629dad96d10 2024-03-24T10:46:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f02413ad740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5629dae149d0 con 0x5629dad963f0 2024-03-24T10:46:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f02413ad740 10 client.0 _ll_register_callbacks cb 0x7f02300093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f02377fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad9f8f0 0x5629dada3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f02377fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad9f8f0 0x5629dada3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59010/0 (socket says 172.21.15.23:59010) 2024-03-24T10:46:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f02377fe640 1 -- 172.21.15.23:0/1530681187 learned_addr learned my addr 172.21.15.23:0/1530681187 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f0237fff640 1 --2- 172.21.15.23:0/1530681187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad96d10 0x5629dada3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f02377fe640 1 --2- 172.21.15.23:0/1530681187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad963f0 0x5629dada2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f0237fff640 1 -- 172.21.15.23:0/1530681187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad963f0 msgr2=0x5629dada2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f0237fff640 1 --2- 172.21.15.23:0/1530681187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad963f0 0x5629dada2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f0237fff640 1 -- 172.21.15.23:0/1530681187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad9f8f0 msgr2=0x5629dada3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f0237fff640 1 --2- 172.21.15.23:0/1530681187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad9f8f0 0x5629dada3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.671+0000 7f0237fff640 1 -- 172.21.15.23:0/1530681187 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629dacfbbd0 con 0x5629dad96d10 2024-03-24T10:46:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f0237fff640 1 --2- 172.21.15.23:0/1530681187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad96d10 0x5629dada3510 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x5629dadb21b0 tx=0x7f0228013960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22639db2ea80e29 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02177fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:46:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02177fe640 1 -- 172.21.15.23:0/1530681187 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228017040 con 0x5629dad96d10 2024-03-24T10:46:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02413ad740 1 -- 172.21.15.23:0/1530681187 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5629daa88890 con 0x5629dad96d10 2024-03-24T10:46:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02177fe640 1 -- 172.21.15.23:0/1530681187 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f0228004220 con 0x5629dad96d10 2024-03-24T10:46:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02177fe640 1 -- 172.21.15.23:0/1530681187 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f02280043e0 con 0x5629dad96d10 2024-03-24T10:46:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02413ad740 10 client.27347 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02413ad740 1 -- 172.21.15.23:0/1530681187 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5629dacbd920 con 0x5629dad96d10 2024-03-24T10:46:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02177fe640 1 -- 172.21.15.23:0/1530681187 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5629daa88890 con 0x5629dad96d10 2024-03-24T10:46:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02177fe640 1 -- 172.21.15.23:0/1530681187 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f022801e040 con 0x5629dad96d10 2024-03-24T10:46:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02413ad740 10 client.27347 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02413ad740 10 client.27347 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:46:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02413ad740 20 client.27347 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02413ad740 1 --2- 172.21.15.23:0/1530681187 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5629dae162b0 0x5629dae186d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02413ad740 4 client.27347 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:46:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.672+0000 7f02413ad740 1 -- 172.21.15.23:0/1530681187 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5629da9a8700 con 0x5629dae162b0 2024-03-24T10:46:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02377fe640 1 --2- 172.21.15.23:0/1530681187 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5629dae162b0 0x5629dae186d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02177fe640 10 client.27347 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:46:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02377fe640 1 --2- 172.21.15.23:0/1530681187 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5629dae162b0 0x5629dae186d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5bb65468cfe78606 server_cookie=4ee0396a0a332e25 in_seq=0 out_seq=0 2024-03-24T10:46:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02177fe640 1 -- 172.21.15.23:0/1530681187 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5629da9a8700 con 0x5629dae162b0 2024-03-24T10:46:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02177fe640 10 client.27347 handle_command_reply: tid=0 2024-03-24T10:46:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02413ad740 1 client.27347 shutdown 2024-03-24T10:46:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02413ad740 1 -- 172.21.15.23:0/1530681187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad96d10 msgr2=0x5629dada3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02413ad740 1 --2- 172.21.15.23:0/1530681187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad96d10 0x5629dada3510 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x5629dadb21b0 tx=0x7f0228013960 comp rx=0 tx=0).stop 2024-03-24T10:46:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02413ad740 1 -- 172.21.15.23:0/1530681187 shutdown_connections 2024-03-24T10:46:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02413ad740 1 --2- 172.21.15.23:0/1530681187 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5629dae162b0 0x5629dae186d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02413ad740 1 --2- 172.21.15.23:0/1530681187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5629dad9f8f0 0x5629dada3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02413ad740 1 --2- 172.21.15.23:0/1530681187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629dad96d10 0x5629dada3510 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02413ad740 1 --2- 172.21.15.23:0/1530681187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5629dad963f0 0x5629dada2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02177fe640 0 client.27347 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:46:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02413ad740 1 -- 172.21.15.23:0/1530681187 >> 172.21.15.23:0/1530681187 conn(0x5629dad90b20 msgr2=0x5629dad9d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02413ad740 1 -- 172.21.15.23:0/1530681187 shutdown_connections 2024-03-24T10:46:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.673+0000 7f02413ad740 1 -- 172.21.15.23:0/1530681187 wait complete. 2024-03-24T10:46:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.674+0000 7f02413ad740 20 client.27347 trim_cache size 0 max 16384 2024-03-24T10:46:42.682 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:46:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.674+0000 7f023e7c9640 1 -- 172.21.15.23:0/3560385629 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f02200a2230 msgr2=0x7f02200a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.674+0000 7f023e7c9640 1 --2- 172.21.15.23:0/3560385629 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f02200a2230 0x7f02200a46f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f023009ae20 tx=0x7f023006d0e0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.674+0000 7f023e7c9640 1 -- 172.21.15.23:0/3560385629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0238106660 msgr2=0x7f02381a3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.674+0000 7f023e7c9640 1 --2- 172.21.15.23:0/3560385629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0238106660 0x7f02381a3ba0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f023000ea80 tx=0x7f023000ef40 comp rx=0 tx=0).stop 2024-03-24T10:46:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.674+0000 7f023e7c9640 1 -- 172.21.15.23:0/3560385629 shutdown_connections 2024-03-24T10:46:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.674+0000 7f023e7c9640 1 --2- 172.21.15.23:0/3560385629 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f02200a2230 0x7f02200a46f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.674+0000 7f023e7c9640 1 --2- 172.21.15.23:0/3560385629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0238106660 0x7f02381a3ba0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.674+0000 7f023e7c9640 1 --2- 172.21.15.23:0/3560385629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0238100f60 0x7f0238199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.674+0000 7f023e7c9640 1 --2- 172.21.15.23:0/3560385629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02381005a0 0x7f0238198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.674+0000 7f023e7c9640 1 -- 172.21.15.23:0/3560385629 >> 172.21.15.23:0/3560385629 conn(0x7f0238077710 msgr2=0x7f02380ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.674+0000 7f023e7c9640 1 -- 172.21.15.23:0/3560385629 shutdown_connections 2024-03-24T10:46:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.674+0000 7f023e7c9640 1 -- 172.21.15.23:0/3560385629 wait complete. 2024-03-24T10:46:42.685 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:46:42.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:42 smithi117 ceph-mon[36368]: pgmap v202: 97 pgs: 97 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 259 op/s 2024-03-24T10:46:42.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:42 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3409041395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:46:42.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:42 smithi116 ceph-mon[36248]: pgmap v202: 97 pgs: 97 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 259 op/s 2024-03-24T10:46:42.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:42 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3409041395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:46:42.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.797+0000 7f82a7120640 1 -- 172.21.15.23:0/923855975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82a0100630 msgr2=0x7f82a0100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.797+0000 7f82a7120640 1 --2- 172.21.15.23:0/923855975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82a0100630 0x7f82a0100ab0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8290009890 tx=0x7f829002bb70 comp rx=0 tx=0).stop 2024-03-24T10:46:42.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.797+0000 7f82a7120640 1 -- 172.21.15.23:0/923855975 shutdown_connections 2024-03-24T10:46:42.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.797+0000 7f82a7120640 1 --2- 172.21.15.23:0/923855975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a0100ff0 0x7f82a010d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.797+0000 7f82a7120640 1 --2- 172.21.15.23:0/923855975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82a0100630 0x7f82a0100ab0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.797+0000 7f82a7120640 1 --2- 172.21.15.23:0/923855975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82a0106640 0x7f82a0106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.797+0000 7f82a7120640 1 -- 172.21.15.23:0/923855975 >> 172.21.15.23:0/923855975 conn(0x7f82a0077770 msgr2=0x7f82a0077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.797+0000 7f82a7120640 1 -- 172.21.15.23:0/923855975 shutdown_connections 2024-03-24T10:46:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.797+0000 7f82a7120640 1 -- 172.21.15.23:0/923855975 wait complete. 2024-03-24T10:46:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.798+0000 7f82a7120640 1 Processor -- start 2024-03-24T10:46:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.798+0000 7f82a7120640 1 -- start start 2024-03-24T10:46:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.798+0000 7f82a7120640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82a0100630 0x7f82a019db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.798+0000 7f82a7120640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82a0100ff0 0x7f82a019e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.798+0000 7f82a7120640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a0106640 0x7f82a0195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f82a4ebc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82a0100630 0x7f82a019db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f8297fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82a0100ff0 0x7f82a019e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f82a7120640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f82a01965e0 con 0x7f82a0100ff0 2024-03-24T10:46:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f82a4ebc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82a0100630 0x7f82a019db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:47710/0 (socket says 172.21.15.23:47710) 2024-03-24T10:46:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f82a4ebc640 1 -- 172.21.15.23:0/1828010375 learned_addr learned my addr 172.21.15.23:0/1828010375 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f82a7120640 1 -- 172.21.15.23:0/1828010375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f82a0196750 con 0x7f82a0106640 2024-03-24T10:46:42.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f82a7120640 1 -- 172.21.15.23:0/1828010375 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f82a0196910 con 0x7f82a0100630 2024-03-24T10:46:42.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f82a56bd640 1 --2- 172.21.15.23:0/1828010375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a0106640 0x7f82a0195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f82a4ebc640 1 -- 172.21.15.23:0/1828010375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a0106640 msgr2=0x7f82a0195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f82a4ebc640 1 --2- 172.21.15.23:0/1828010375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a0106640 0x7f82a0195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f82a4ebc640 1 -- 172.21.15.23:0/1828010375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82a0100ff0 msgr2=0x7f82a019e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f82a4ebc640 1 --2- 172.21.15.23:0/1828010375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82a0100ff0 0x7f82a019e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f82a4ebc640 1 -- 172.21.15.23:0/1828010375 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829c009560 con 0x7f82a0100630 2024-03-24T10:46:42.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f8297fff640 1 --2- 172.21.15.23:0/1828010375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82a0100ff0 0x7f82a019e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:42.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f82a4ebc640 1 --2- 172.21.15.23:0/1828010375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82a0100630 0x7f82a019db50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f829c00e9c0 tx=0x7f829c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f8295ffb640 1 -- 172.21.15.23:0/1828010375 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829c00cde0 con 0x7f82a0100630 2024-03-24T10:46:42.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.799+0000 7f8295ffb640 1 -- 172.21.15.23:0/1828010375 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f829c0035f0 con 0x7f82a0100630 2024-03-24T10:46:42.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.800+0000 7f8295ffb640 1 -- 172.21.15.23:0/1828010375 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829c010670 con 0x7f82a0100630 2024-03-24T10:46:42.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.800+0000 7f82a7120640 1 -- 172.21.15.23:0/1828010375 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8290009490 con 0x7f82a0100630 2024-03-24T10:46:42.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.800+0000 7f82a7120640 1 -- 172.21.15.23:0/1828010375 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a01b37e0 con 0x7f82a0100630 2024-03-24T10:46:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.802+0000 7f8295ffb640 1 -- 172.21.15.23:0/1828010375 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f829c003ad0 con 0x7f82a0100630 2024-03-24T10:46:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.802+0000 7f82a9dd7740 1 Processor -- start 2024-03-24T10:46:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.802+0000 7f82a9dd7740 1 -- start start 2024-03-24T10:46:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.802+0000 7f8295ffb640 1 --2- 172.21.15.23:0/1828010375 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f82880a2230 0x7f82880a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f82a9dd7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0bc3f0 0x55f97b0bc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f82a9dd7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bcd10 0x55f97b0c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f82a9dd7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0c58f0 0x55f97b0c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f82a9dd7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f97b02a840 con 0x55f97b0bcd10 2024-03-24T10:46:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f82a9dd7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f97b027d70 con 0x55f97b0c58f0 2024-03-24T10:46:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f82a9dd7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f97b023550 con 0x55f97b0bc3f0 2024-03-24T10:46:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f82a56bd640 1 --2- 172.21.15.23:0/1828010375 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f82880a2230 0x7f82880a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f8295ffb640 1 -- 172.21.15.23:0/1828010375 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f829c014070 con 0x7f82a0100630 2024-03-24T10:46:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f8297fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bcd10 0x55f97b0c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f8297fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bcd10 0x55f97b0c53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59036/0 (socket says 172.21.15.23:59036) 2024-03-24T10:46:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f8297fff640 1 -- 172.21.15.23:0/948094739 learned_addr learned my addr 172.21.15.23:0/948094739 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f82a56bd640 1 --2- 172.21.15.23:0/948094739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0c58f0 0x55f97b0c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f82a4ebc640 1 --2- 172.21.15.23:0/948094739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0bc3f0 0x55f97b0bc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f8297fff640 1 -- 172.21.15.23:0/948094739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0bc3f0 msgr2=0x55f97b0bc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f8297fff640 1 --2- 172.21.15.23:0/948094739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0bc3f0 0x55f97b0bc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f8297fff640 1 -- 172.21.15.23:0/948094739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0c58f0 msgr2=0x55f97b0c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f8297fff640 1 --2- 172.21.15.23:0/948094739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0c58f0 0x55f97b0c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f8297fff640 1 -- 172.21.15.23:0/948094739 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f97afb8de0 con 0x55f97b0bcd10 2024-03-24T10:46:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f82a56bd640 1 --2- 172.21.15.23:0/1828010375 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f82880a2230 0x7f82880a46f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8298005fd0 tx=0x7f8298009e60 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f8297fff640 1 --2- 172.21.15.23:0/948094739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bcd10 0x55f97b0c53b0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f8290037040 tx=0x7f82900058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8fed7b0c20c650e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f8286ffd640 1 -- 172.21.15.23:0/948094739 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290004300 con 0x55f97b0bcd10 2024-03-24T10:46:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f8286ffd640 1 -- 172.21.15.23:0/948094739 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8290004460 con 0x55f97b0bcd10 2024-03-24T10:46:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.803+0000 7f8286ffd640 1 -- 172.21.15.23:0/948094739 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829002cbe0 con 0x55f97b0bcd10 2024-03-24T10:46:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.804+0000 7f82a9dd7740 1 -- 172.21.15.23:0/948094739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bcd10 msgr2=0x55f97b0c53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.804+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/948094739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bcd10 0x55f97b0c53b0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f8290037040 tx=0x7f82900058d0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.804+0000 7f82a9dd7740 1 -- 172.21.15.23:0/948094739 shutdown_connections 2024-03-24T10:46:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.804+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/948094739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0c58f0 0x55f97b0c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.804+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/948094739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bcd10 0x55f97b0c53b0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.804+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/948094739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0bc3f0 0x55f97b0bc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.804+0000 7f82a9dd7740 1 -- 172.21.15.23:0/948094739 >> 172.21.15.23:0/948094739 conn(0x55f97b0b6b20 msgr2=0x55f97b0b6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.804+0000 7f82a9dd7740 1 -- 172.21.15.23:0/948094739 shutdown_connections 2024-03-24T10:46:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.804+0000 7f82a9dd7740 1 -- 172.21.15.23:0/948094739 wait complete. 2024-03-24T10:46:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.805+0000 7f82a9dd7740 1 -- start start 2024-03-24T10:46:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.805+0000 7f82a9dd7740 1 Processor -- start 2024-03-24T10:46:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.805+0000 7f82a9dd7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bc3f0 0x55f97b0c2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.805+0000 7f82a9dd7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bcd10 0x55f97b0c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.805+0000 7f82a9dd7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0c58f0 0x55f97b0c3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.805+0000 7f82a9dd7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f97acf6430 con 0x55f97b0bcd10 2024-03-24T10:46:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.805+0000 7f82a9dd7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f97acb2e00 con 0x55f97b0bc3f0 2024-03-24T10:46:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.805+0000 7f82a9dd7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f97afa8590 con 0x55f97b0c58f0 2024-03-24T10:46:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.805+0000 7f82a9dd7740 10 client.0 _ll_register_callbacks cb 0x55f97b017fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f8297fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bcd10 0x55f97b0c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f8297fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bcd10 0x55f97b0c2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59040/0 (socket says 172.21.15.23:59040) 2024-03-24T10:46:42.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f8297fff640 1 -- 172.21.15.23:0/905921120 learned_addr learned my addr 172.21.15.23:0/905921120 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f82a4ebc640 1 --2- 172.21.15.23:0/905921120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bc3f0 0x55f97b0c2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f82a56bd640 1 --2- 172.21.15.23:0/905921120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0c58f0 0x55f97b0c3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f8297fff640 1 -- 172.21.15.23:0/905921120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0c58f0 msgr2=0x55f97b0c3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f8297fff640 1 --2- 172.21.15.23:0/905921120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0c58f0 0x55f97b0c3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f8297fff640 1 -- 172.21.15.23:0/905921120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bc3f0 msgr2=0x55f97b0c2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f8297fff640 1 --2- 172.21.15.23:0/905921120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bc3f0 0x55f97b0c2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f8297fff640 1 -- 172.21.15.23:0/905921120 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a010de00 con 0x55f97b0bcd10 2024-03-24T10:46:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f8297fff640 1 --2- 172.21.15.23:0/905921120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bcd10 0x55f97b0c2e50 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8290005e00 tx=0x7f829002c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0405fc683bb7f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f827ffff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f827ffff640 1 -- 172.21.15.23:0/905921120 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290034640 con 0x55f97b0bcd10 2024-03-24T10:46:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f827ffff640 1 -- 172.21.15.23:0/905921120 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f82900347a0 con 0x55f97b0bcd10 2024-03-24T10:46:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f82a9dd7740 1 -- 172.21.15.23:0/905921120 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f97ad227e0 con 0x55f97b0bcd10 2024-03-24T10:46:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f827ffff640 1 -- 172.21.15.23:0/905921120 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8290034960 con 0x55f97b0bcd10 2024-03-24T10:46:42.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f82a9dd7740 10 client.17472 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:42.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.806+0000 7f82a9dd7740 1 -- 172.21.15.23:0/905921120 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f97b021bd0 con 0x55f97b0bcd10 2024-03-24T10:46:42.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.807+0000 7f827ffff640 1 -- 172.21.15.23:0/905921120 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8290060080 con 0x55f97b0bcd10 2024-03-24T10:46:42.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.807+0000 7f827ffff640 1 -- 172.21.15.23:0/905921120 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f82900038d0 con 0x55f97b0bcd10 2024-03-24T10:46:42.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.807+0000 7f82a9dd7740 10 client.17472 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:42.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.807+0000 7f82a9dd7740 10 client.17472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:42.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.807+0000 7f82a9dd7740 20 client.17472 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:42.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.807+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/905921120 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f97b0d54d0 0x55f97b0d7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.807+0000 7f82a9dd7740 4 client.17472 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:42.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.807+0000 7f82a9dd7740 1 -- 172.21.15.23:0/905921120 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f97ad5f720 con 0x55f97b0d54d0 2024-03-24T10:46:42.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.807+0000 7f82a4ebc640 1 --2- 172.21.15.23:0/905921120 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f97b0d54d0 0x55f97b0d7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.807+0000 7f827ffff640 10 client.17472 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:42.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.807+0000 7f82a4ebc640 1 --2- 172.21.15.23:0/905921120 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f97b0d54d0 0x55f97b0d7900 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8114ad0a648872d6 server_cookie=487ba756ba60f373 in_seq=0 out_seq=0 2024-03-24T10:46:42.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.808+0000 7f827ffff640 1 -- 172.21.15.23:0/905921120 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f97ad5f720 con 0x55f97b0d54d0 2024-03-24T10:46:42.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.808+0000 7f827ffff640 10 client.17472 handle_command_reply: tid=0 2024-03-24T10:46:42.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.808+0000 7f82a9dd7740 1 client.17472 shutdown 2024-03-24T10:46:42.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.808+0000 7f82a9dd7740 1 -- 172.21.15.23:0/905921120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bcd10 msgr2=0x55f97b0c2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.808+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/905921120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bcd10 0x55f97b0c2e50 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8290005e00 tx=0x7f829002c740 comp rx=0 tx=0).stop 2024-03-24T10:46:42.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.808+0000 7f82a9dd7740 1 -- 172.21.15.23:0/905921120 shutdown_connections 2024-03-24T10:46:42.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.808+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/905921120 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f97b0d54d0 0x55f97b0d7900 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.808+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/905921120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0c58f0 0x55f97b0c3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.808+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/905921120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bcd10 0x55f97b0c2e50 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.808+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/905921120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bc3f0 0x55f97b0c2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.808+0000 7f827ffff640 0 client.17472 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:42.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.808+0000 7f82a9dd7740 1 -- 172.21.15.23:0/905921120 >> 172.21.15.23:0/905921120 conn(0x55f97b0b6b20 msgr2=0x55f97b0b8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.809+0000 7f82a9dd7740 1 -- 172.21.15.23:0/905921120 shutdown_connections 2024-03-24T10:46:42.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.809+0000 7f82a9dd7740 1 -- 172.21.15.23:0/905921120 wait complete. 2024-03-24T10:46:42.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.809+0000 7f82a9dd7740 20 client.17472 trim_cache size 0 max 16384 2024-03-24T10:46:42.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.823+0000 7f82a9dd7740 1 Processor -- start 2024-03-24T10:46:42.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.823+0000 7f82a9dd7740 1 -- start start 2024-03-24T10:46:42.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.823+0000 7f82a9dd7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bc3f0 0x55f97b0c4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.823+0000 7f82a9dd7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0bcd10 0x55f97b0d1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.823+0000 7f82a9dd7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0c58f0 0x55f97b0b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.823+0000 7f82a9dd7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f97b0b6650 con 0x55f97b0c58f0 2024-03-24T10:46:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.823+0000 7f82a9dd7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f97b0b23e0 con 0x55f97b0bc3f0 2024-03-24T10:46:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.823+0000 7f82a9dd7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f97b0b7b20 con 0x55f97b0bcd10 2024-03-24T10:46:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.824+0000 7f8297fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0bcd10 0x55f97b0d1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.824+0000 7f8297fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0bcd10 0x55f97b0d1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:47736/0 (socket says 172.21.15.23:47736) 2024-03-24T10:46:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.824+0000 7f8297fff640 1 -- 172.21.15.23:0/2336603669 learned_addr learned my addr 172.21.15.23:0/2336603669 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.824+0000 7f82a56bd640 1 --2- 172.21.15.23:0/2336603669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0c58f0 0x55f97b0b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.824+0000 7f82a4ebc640 1 --2- 172.21.15.23:0/2336603669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bc3f0 0x55f97b0c4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.824+0000 7f82a56bd640 1 -- 172.21.15.23:0/2336603669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0bcd10 msgr2=0x55f97b0d1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.824+0000 7f82a56bd640 1 --2- 172.21.15.23:0/2336603669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0bcd10 0x55f97b0d1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.824+0000 7f82a56bd640 1 -- 172.21.15.23:0/2336603669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bc3f0 msgr2=0x55f97b0c4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.824+0000 7f82a56bd640 1 --2- 172.21.15.23:0/2336603669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bc3f0 0x55f97b0c4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.824+0000 7f82a56bd640 1 -- 172.21.15.23:0/2336603669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f97afbdc80 con 0x55f97b0c58f0 2024-03-24T10:46:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.824+0000 7f82a56bd640 1 --2- 172.21.15.23:0/2336603669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0c58f0 0x55f97b0b5b40 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8298009270 tx=0x7f8298016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dac89b25e589260 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.825+0000 7f82867fc640 1 -- 172.21.15.23:0/2336603669 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82980129e0 con 0x55f97b0c58f0 2024-03-24T10:46:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.825+0000 7f82867fc640 1 -- 172.21.15.23:0/2336603669 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8298003e50 con 0x55f97b0c58f0 2024-03-24T10:46:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.825+0000 7f82867fc640 1 -- 172.21.15.23:0/2336603669 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8298004320 con 0x55f97b0c58f0 2024-03-24T10:46:42.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.826+0000 7f82a9dd7740 1 -- 172.21.15.23:0/2336603669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0c58f0 msgr2=0x55f97b0b5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.826+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/2336603669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0c58f0 0x55f97b0b5b40 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8298009270 tx=0x7f8298016040 comp rx=0 tx=0).stop 2024-03-24T10:46:42.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.826+0000 7f82a9dd7740 1 -- 172.21.15.23:0/2336603669 shutdown_connections 2024-03-24T10:46:42.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.826+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/2336603669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0c58f0 0x55f97b0b5b40 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.826+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/2336603669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0bcd10 0x55f97b0d1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.826+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/2336603669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bc3f0 0x55f97b0c4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.826+0000 7f82a9dd7740 1 -- 172.21.15.23:0/2336603669 >> 172.21.15.23:0/2336603669 conn(0x55f97b0b6b20 msgr2=0x55f97b0b5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.826+0000 7f82a9dd7740 1 -- 172.21.15.23:0/2336603669 shutdown_connections 2024-03-24T10:46:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.826+0000 7f82a9dd7740 1 -- 172.21.15.23:0/2336603669 wait complete. 2024-03-24T10:46:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.827+0000 7f82a9dd7740 1 -- start start 2024-03-24T10:46:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.827+0000 7f82a9dd7740 1 Processor -- start 2024-03-24T10:46:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.827+0000 7f82a9dd7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bc3f0 0x55f97b0cd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a9dd7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bcd10 0x55f97b0ca600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a9dd7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0c58f0 0x55f97b0cab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a4ebc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bc3f0 0x55f97b0cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a4ebc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bc3f0 0x55f97b0cd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59064/0 (socket says 172.21.15.23:59064) 2024-03-24T10:46:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a4ebc640 1 -- 172.21.15.23:0/721499528 learned_addr learned my addr 172.21.15.23:0/721499528 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a9dd7740 1 -- 172.21.15.23:0/721499528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f97b0cb080 con 0x55f97b0bc3f0 2024-03-24T10:46:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a9dd7740 1 -- 172.21.15.23:0/721499528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f97b0cb1f0 con 0x55f97b0bcd10 2024-03-24T10:46:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f8297fff640 1 --2- 172.21.15.23:0/721499528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bcd10 0x55f97b0ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a9dd7740 1 -- 172.21.15.23:0/721499528 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f97b0cb3b0 con 0x55f97b0c58f0 2024-03-24T10:46:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a9dd7740 10 client.0 _ll_register_callbacks cb 0x7f82980137a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a56bd640 1 --2- 172.21.15.23:0/721499528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0c58f0 0x55f97b0cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a4ebc640 1 -- 172.21.15.23:0/721499528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0c58f0 msgr2=0x55f97b0cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a4ebc640 1 --2- 172.21.15.23:0/721499528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0c58f0 0x55f97b0cab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a4ebc640 1 -- 172.21.15.23:0/721499528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bcd10 msgr2=0x55f97b0ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a4ebc640 1 --2- 172.21.15.23:0/721499528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bcd10 0x55f97b0ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f82a4ebc640 1 -- 172.21.15.23:0/721499528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f829801b040 con 0x55f97b0bc3f0 2024-03-24T10:46:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.828+0000 7f8297fff640 1 --2- 172.21.15.23:0/721499528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bcd10 0x55f97b0ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.829+0000 7f82a4ebc640 1 --2- 172.21.15.23:0/721499528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bc3f0 0x55f97b0cd820 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f829c0cc1c0 tx=0x7f829c09b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113077b245434462 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.829+0000 7f827ffff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.829+0000 7f827ffff640 1 -- 172.21.15.23:0/721499528 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829c0a3600 con 0x55f97b0bc3f0 2024-03-24T10:46:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.829+0000 7f827ffff640 1 -- 172.21.15.23:0/721499528 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f829c09d070 con 0x55f97b0bc3f0 2024-03-24T10:46:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.829+0000 7f82a9dd7740 1 -- 172.21.15.23:0/721499528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8290004460 con 0x55f97b0bc3f0 2024-03-24T10:46:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.829+0000 7f827ffff640 1 -- 172.21.15.23:0/721499528 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f829c098f00 con 0x55f97b0bc3f0 2024-03-24T10:46:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.829+0000 7f82a9dd7740 10 client.17478 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.829+0000 7f82a9dd7740 1 -- 172.21.15.23:0/721499528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f97afe3920 con 0x55f97b0bc3f0 2024-03-24T10:46:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.830+0000 7f827ffff640 1 -- 172.21.15.23:0/721499528 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8290004460 con 0x55f97b0bc3f0 2024-03-24T10:46:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.830+0000 7f827ffff640 1 -- 172.21.15.23:0/721499528 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f829c01f060 con 0x55f97b0bc3f0 2024-03-24T10:46:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.830+0000 7f82a9dd7740 10 client.17478 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.830+0000 7f82a9dd7740 10 client.17478 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.830+0000 7f82a9dd7740 20 client.17478 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.830+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/721499528 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f97b13cd00 0x55f97b13f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.830+0000 7f82a9dd7740 4 client.17478 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:46:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.830+0000 7f82a9dd7740 1 -- 172.21.15.23:0/721499528 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f97adae890 con 0x55f97b13cd00 2024-03-24T10:46:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.830+0000 7f8297fff640 1 --2- 172.21.15.23:0/721499528 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f97b13cd00 0x55f97b13f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f827ffff640 10 client.17478 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f8297fff640 1 --2- 172.21.15.23:0/721499528 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f97b13cd00 0x55f97b13f0f0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54fb75053d93439 server_cookie=e3f996919e35b68a in_seq=0 out_seq=0 2024-03-24T10:46:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f827ffff640 1 -- 172.21.15.23:0/721499528 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f97adae890 con 0x55f97b13cd00 2024-03-24T10:46:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f827ffff640 10 client.17478 handle_command_reply: tid=0 2024-03-24T10:46:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f82a9dd7740 1 client.17478 shutdown 2024-03-24T10:46:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f82a9dd7740 1 -- 172.21.15.23:0/721499528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bc3f0 msgr2=0x55f97b0cd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/721499528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bc3f0 0x55f97b0cd820 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f829c0cc1c0 tx=0x7f829c09b620 comp rx=0 tx=0).stop 2024-03-24T10:46:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f82a9dd7740 1 -- 172.21.15.23:0/721499528 shutdown_connections 2024-03-24T10:46:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/721499528 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55f97b13cd00 0x55f97b13f0f0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/721499528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f97b0c58f0 0x55f97b0cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f827ffff640 0 client.17478 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/721499528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f97b0bcd10 0x55f97b0ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f82a9dd7740 1 --2- 172.21.15.23:0/721499528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f97b0bc3f0 0x55f97b0cd820 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f82a9dd7740 1 -- 172.21.15.23:0/721499528 >> 172.21.15.23:0/721499528 conn(0x55f97b0b6b20 msgr2=0x55f97b0b5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.831+0000 7f82a9dd7740 1 -- 172.21.15.23:0/721499528 shutdown_connections 2024-03-24T10:46:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.832+0000 7f82a9dd7740 1 -- 172.21.15.23:0/721499528 wait complete. 2024-03-24T10:46:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.832+0000 7f82a9dd7740 20 client.17478 trim_cache size 0 max 16384 2024-03-24T10:46:42.836 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:46:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.832+0000 7f82a7120640 1 -- 172.21.15.23:0/1828010375 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f82880a2230 msgr2=0x7f82880a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.832+0000 7f82a7120640 1 --2- 172.21.15.23:0/1828010375 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f82880a2230 0x7f82880a46f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8298005fd0 tx=0x7f8298009e60 comp rx=0 tx=0).stop 2024-03-24T10:46:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.832+0000 7f82a7120640 1 -- 172.21.15.23:0/1828010375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82a0100630 msgr2=0x7f82a019db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.832+0000 7f82a7120640 1 --2- 172.21.15.23:0/1828010375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82a0100630 0x7f82a019db50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f829c00e9c0 tx=0x7f829c00ee80 comp rx=0 tx=0).stop 2024-03-24T10:46:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.832+0000 7f82a7120640 1 -- 172.21.15.23:0/1828010375 shutdown_connections 2024-03-24T10:46:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.832+0000 7f82a7120640 1 --2- 172.21.15.23:0/1828010375 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f82880a2230 0x7f82880a46f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.832+0000 7f82a7120640 1 --2- 172.21.15.23:0/1828010375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a0106640 0x7f82a0195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.832+0000 7f82a7120640 1 --2- 172.21.15.23:0/1828010375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82a0100ff0 0x7f82a019e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.832+0000 7f82a7120640 1 --2- 172.21.15.23:0/1828010375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82a0100630 0x7f82a019db50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.832+0000 7f82a7120640 1 -- 172.21.15.23:0/1828010375 >> 172.21.15.23:0/1828010375 conn(0x7f82a0077770 msgr2=0x7f82a010be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.833+0000 7f82a7120640 1 -- 172.21.15.23:0/1828010375 shutdown_connections 2024-03-24T10:46:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:42.833+0000 7f82a7120640 1 -- 172.21.15.23:0/1828010375 wait complete. 2024-03-24T10:46:43.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 640666 11.09 MB/sec execute 180 sec latency 19.874 ms 2024-03-24T10:46:43.845 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:46:43.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:46:43.846 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:46:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.960+0000 7f326e24e640 1 -- 172.21.15.23:0/657690056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32680688f0 msgr2=0x7f3268068d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.960+0000 7f326e24e640 1 --2- 172.21.15.23:0/657690056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32680688f0 0x7f3268068d70 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f325c009930 tx=0x7f325c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:46:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.960+0000 7f326e24e640 1 -- 172.21.15.23:0/657690056 shutdown_connections 2024-03-24T10:46:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.960+0000 7f326e24e640 1 --2- 172.21.15.23:0/657690056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32680692b0 0x7f326810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.960+0000 7f326e24e640 1 --2- 172.21.15.23:0/657690056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32680688f0 0x7f3268068d70 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.960+0000 7f326e24e640 1 --2- 172.21.15.23:0/657690056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3268107690 0x7f3268107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.960+0000 7f326e24e640 1 -- 172.21.15.23:0/657690056 >> 172.21.15.23:0/657690056 conn(0x7f3268077710 msgr2=0x7f3268077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:43.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.960+0000 7f326e24e640 1 -- 172.21.15.23:0/657690056 shutdown_connections 2024-03-24T10:46:43.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.961+0000 7f326e24e640 1 -- 172.21.15.23:0/657690056 wait complete. 2024-03-24T10:46:43.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.961+0000 7f326e24e640 1 Processor -- start 2024-03-24T10:46:43.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.961+0000 7f326e24e640 1 -- start start 2024-03-24T10:46:43.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f326e24e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32680688f0 0x7f3268199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f326e24e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32680692b0 0x7f326819a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f326e24e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3268107690 0x7f326819e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f326e24e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f326819edf0 con 0x7f32680688f0 2024-03-24T10:46:43.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f326e24e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f326819ef60 con 0x7f32680692b0 2024-03-24T10:46:43.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f326e24e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f326819f120 con 0x7f3268107690 2024-03-24T10:46:43.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f32677fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32680688f0 0x7f3268199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f3266ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32680692b0 0x7f326819a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f32677fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32680688f0 0x7f3268199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59086/0 (socket says 172.21.15.23:59086) 2024-03-24T10:46:43.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f32677fe640 1 -- 172.21.15.23:0/2484936668 learned_addr learned my addr 172.21.15.23:0/2484936668 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:43.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f3266ffd640 1 --2- 172.21.15.23:0/2484936668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32680692b0 0x7f326819a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:55148/0 (socket says 172.21.15.23:55148) 2024-03-24T10:46:43.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f3267fff640 1 --2- 172.21.15.23:0/2484936668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3268107690 0x7f326819e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f32677fe640 1 -- 172.21.15.23:0/2484936668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3268107690 msgr2=0x7f326819e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:43.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f32677fe640 1 --2- 172.21.15.23:0/2484936668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3268107690 0x7f326819e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f32677fe640 1 -- 172.21.15.23:0/2484936668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32680692b0 msgr2=0x7f326819a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:43.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f32677fe640 1 --2- 172.21.15.23:0/2484936668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32680692b0 0x7f326819a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.962+0000 7f32677fe640 1 -- 172.21.15.23:0/2484936668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f325c009560 con 0x7f32680688f0 2024-03-24T10:46:43.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.963+0000 7f32677fe640 1 --2- 172.21.15.23:0/2484936668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32680688f0 0x7f3268199f30 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f325800e9f0 tx=0x7f325800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:43.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.963+0000 7f3264ff9640 1 -- 172.21.15.23:0/2484936668 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258009700 con 0x7f32680688f0 2024-03-24T10:46:43.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.963+0000 7f326e24e640 1 -- 172.21.15.23:0/2484936668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f326819f400 con 0x7f32680688f0 2024-03-24T10:46:43.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.963+0000 7f3264ff9640 1 -- 172.21.15.23:0/2484936668 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f325800cd80 con 0x7f32680688f0 2024-03-24T10:46:43.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.963+0000 7f3264ff9640 1 -- 172.21.15.23:0/2484936668 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258018920 con 0x7f32680688f0 2024-03-24T10:46:43.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.963+0000 7f326e24e640 1 -- 172.21.15.23:0/2484936668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32681a6c80 con 0x7f32680688f0 2024-03-24T10:46:43.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.965+0000 7f3264ff9640 1 -- 172.21.15.23:0/2484936668 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f3258018a80 con 0x7f32680688f0 2024-03-24T10:46:43.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.965+0000 7f3270e2f740 1 Processor -- start 2024-03-24T10:46:43.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f3270e2f740 1 -- start start 2024-03-24T10:46:43.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f3270e2f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x560531616b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f3270e2f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 0x56053161f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f3270e2f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56053161fc60 0x560531622050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f3270e2f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x560531584530 con 0x560531616760 2024-03-24T10:46:43.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f3270e2f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560531581a60 con 0x560531617080 2024-03-24T10:46:43.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f3270e2f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56053157d240 con 0x56053161fc60 2024-03-24T10:46:43.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f3267fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x560531616b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f32677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 0x56053161f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f32677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 0x56053161f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:55160/0 (socket says 172.21.15.23:55160) 2024-03-24T10:46:43.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f32677fe640 1 -- 172.21.15.23:0/2091234753 learned_addr learned my addr 172.21.15.23:0/2091234753 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:43.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f3264ff9640 1 --2- 172.21.15.23:0/2484936668 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f32500a2230 0x7f32500a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f3266ffd640 1 --2- 172.21.15.23:0/2091234753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56053161fc60 0x560531622050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f3267fff640 1 --2- 172.21.15.23:0/2484936668 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f32500a2230 0x7f32500a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.966+0000 7f3264ff9640 1 -- 172.21.15.23:0/2484936668 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f3258014070 con 0x7f32680688f0 2024-03-24T10:46:43.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.967+0000 7f3267fff640 1 -- 172.21.15.23:0/2091234753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56053161fc60 msgr2=0x560531622050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:43.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.967+0000 7f3267fff640 1 --2- 172.21.15.23:0/2091234753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56053161fc60 0x560531622050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.967+0000 7f3267fff640 1 -- 172.21.15.23:0/2091234753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 msgr2=0x56053161f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:43.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.967+0000 7f3267fff640 1 --2- 172.21.15.23:0/2091234753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 0x56053161f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.967+0000 7f3267fff640 1 -- 172.21.15.23:0/2091234753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56053157b8c0 con 0x560531616760 2024-03-24T10:46:43.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.967+0000 7f3267fff640 1 --2- 172.21.15.23:0/2091234753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x560531616b40 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3260005fd0 tx=0x7f3260013ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d169eae3be4219 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:43.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.967+0000 7f324dffb640 1 -- 172.21.15.23:0/2091234753 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326000fe20 con 0x560531616760 2024-03-24T10:46:43.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.967+0000 7f324dffb640 1 -- 172.21.15.23:0/2091234753 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f326001fd30 con 0x560531616760 2024-03-24T10:46:43.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.967+0000 7f324dffb640 1 -- 172.21.15.23:0/2091234753 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260010a50 con 0x560531616760 2024-03-24T10:46:43.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.967+0000 7f3267fff640 1 --2- 172.21.15.23:0/2484936668 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f32500a2230 0x7f32500a46f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3260023000 tx=0x7f3260022000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:43.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.968+0000 7f3270e2f740 1 -- 172.21.15.23:0/2091234753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 msgr2=0x560531616b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:43.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.968+0000 7f3270e2f740 1 --2- 172.21.15.23:0/2091234753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x560531616b40 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3260005fd0 tx=0x7f3260013ef0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.968+0000 7f3270e2f740 1 -- 172.21.15.23:0/2091234753 shutdown_connections 2024-03-24T10:46:43.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.968+0000 7f3270e2f740 1 --2- 172.21.15.23:0/2091234753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56053161fc60 0x560531622050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.968+0000 7f3270e2f740 1 --2- 172.21.15.23:0/2091234753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 0x56053161f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.968+0000 7f3270e2f740 1 --2- 172.21.15.23:0/2091234753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x560531616b40 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.968+0000 7f3270e2f740 1 -- 172.21.15.23:0/2091234753 >> 172.21.15.23:0/2091234753 conn(0x560531610e90 msgr2=0x560531611270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:43.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.968+0000 7f3270e2f740 1 -- 172.21.15.23:0/2091234753 shutdown_connections 2024-03-24T10:46:43.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.968+0000 7f3270e2f740 1 -- 172.21.15.23:0/2091234753 wait complete. 2024-03-24T10:46:43.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.968+0000 7f3270e2f740 1 -- start start 2024-03-24T10:46:43.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.968+0000 7f3270e2f740 1 Processor -- start 2024-03-24T10:46:43.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f3270e2f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x56053161ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f3270e2f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560531617080 0x56053161d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f3270e2f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56053161fc60 0x56053161d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f3270e2f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x560531521f80 con 0x560531616760 2024-03-24T10:46:43.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f3270e2f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5605312504b0 con 0x56053161fc60 2024-03-24T10:46:43.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f3270e2f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560531502ad0 con 0x560531617080 2024-03-24T10:46:43.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f3270e2f740 10 client.0 _ll_register_callbacks cb 0x560531535b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:43.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f32677fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x56053161ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f32677fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x56053161ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59110/0 (socket says 172.21.15.23:59110) 2024-03-24T10:46:43.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f32677fe640 1 -- 172.21.15.23:0/1701631249 learned_addr learned my addr 172.21.15.23:0/1701631249 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:43.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f3266ffd640 1 --2- 172.21.15.23:0/1701631249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560531617080 0x56053161d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f3267fff640 1 --2- 172.21.15.23:0/1701631249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56053161fc60 0x56053161d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f32677fe640 1 -- 172.21.15.23:0/1701631249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560531617080 msgr2=0x56053161d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:43.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f32677fe640 1 --2- 172.21.15.23:0/1701631249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560531617080 0x56053161d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f32677fe640 1 -- 172.21.15.23:0/1701631249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56053161fc60 msgr2=0x56053161d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:43.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f32677fe640 1 --2- 172.21.15.23:0/1701631249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56053161fc60 0x56053161d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f32677fe640 1 -- 172.21.15.23:0/1701631249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3260013640 con 0x560531616760 2024-03-24T10:46:43.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f32677fe640 1 --2- 172.21.15.23:0/1701631249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x56053161ccc0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f325800e9c0 tx=0x7f3258011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223460a9b09b53e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:43.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f3246ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:43.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f3246ffd640 1 -- 172.21.15.23:0/1701631249 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325809a980 con 0x560531616760 2024-03-24T10:46:43.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f3246ffd640 1 -- 172.21.15.23:0/1701631249 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f325809ae40 con 0x560531616760 2024-03-24T10:46:43.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f3270e2f740 1 -- 172.21.15.23:0/1701631249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5605315a17a0 con 0x560531616760 2024-03-24T10:46:43.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.969+0000 7f3246ffd640 1 -- 172.21.15.23:0/1701631249 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f325809a7a0 con 0x560531616760 2024-03-24T10:46:43.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.970+0000 7f3270e2f740 10 client.17490 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:43.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.970+0000 7f3270e2f740 1 -- 172.21.15.23:0/1701631249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5605315336d0 con 0x560531616760 2024-03-24T10:46:43.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.970+0000 7f3246ffd640 1 -- 172.21.15.23:0/1701631249 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5605315a17a0 con 0x560531616760 2024-03-24T10:46:43.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.970+0000 7f3246ffd640 1 -- 172.21.15.23:0/1701631249 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f325809a290 con 0x560531616760 2024-03-24T10:46:43.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.970+0000 7f3270e2f740 10 client.17490 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:43.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.970+0000 7f3270e2f740 10 client.17490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:43.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.970+0000 7f3270e2f740 20 client.17490 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:43.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.970+0000 7f3270e2f740 1 --2- 172.21.15.23:0/1701631249 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56053162fcb0 0x5605316320c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.970+0000 7f3270e2f740 4 client.17490 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:43.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.970+0000 7f3270e2f740 1 -- 172.21.15.23:0/1701631249 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605315731f0 con 0x56053162fcb0 2024-03-24T10:46:43.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.970+0000 7f3266ffd640 1 --2- 172.21.15.23:0/1701631249 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56053162fcb0 0x5605316320c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.971+0000 7f3266ffd640 1 --2- 172.21.15.23:0/1701631249 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56053162fcb0 0x5605316320c0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d50245c1bc6485f server_cookie=a1f16d9bdae2f4c0 in_seq=0 out_seq=0 2024-03-24T10:46:43.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.971+0000 7f3246ffd640 10 client.17490 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:43.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.971+0000 7f3246ffd640 1 -- 172.21.15.23:0/1701631249 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5605315731f0 con 0x56053162fcb0 2024-03-24T10:46:43.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.971+0000 7f3246ffd640 10 client.17490 handle_command_reply: tid=0 2024-03-24T10:46:43.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.971+0000 7f3270e2f740 1 client.17490 shutdown 2024-03-24T10:46:43.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.972+0000 7f3270e2f740 1 -- 172.21.15.23:0/1701631249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 msgr2=0x56053161ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:43.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.972+0000 7f3270e2f740 1 --2- 172.21.15.23:0/1701631249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x56053161ccc0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f325800e9c0 tx=0x7f3258011000 comp rx=0 tx=0).stop 2024-03-24T10:46:43.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.972+0000 7f3270e2f740 1 -- 172.21.15.23:0/1701631249 shutdown_connections 2024-03-24T10:46:43.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.972+0000 7f3270e2f740 1 --2- 172.21.15.23:0/1701631249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56053161fc60 0x56053161d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.972+0000 7f3270e2f740 1 --2- 172.21.15.23:0/1701631249 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56053162fcb0 0x5605316320c0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.972+0000 7f3246ffd640 0 client.17490 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:43.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.972+0000 7f3270e2f740 1 --2- 172.21.15.23:0/1701631249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560531617080 0x56053161d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.972+0000 7f3270e2f740 1 --2- 172.21.15.23:0/1701631249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x56053161ccc0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.972+0000 7f3270e2f740 1 -- 172.21.15.23:0/1701631249 >> 172.21.15.23:0/1701631249 conn(0x560531610e90 msgr2=0x560531612be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:43.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.972+0000 7f3270e2f740 1 -- 172.21.15.23:0/1701631249 shutdown_connections 2024-03-24T10:46:43.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.972+0000 7f3270e2f740 1 -- 172.21.15.23:0/1701631249 wait complete. 2024-03-24T10:46:43.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.972+0000 7f3270e2f740 20 client.17490 trim_cache size 0 max 16384 2024-03-24T10:46:43.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.990+0000 7f3270e2f740 1 Processor -- start 2024-03-24T10:46:43.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.990+0000 7f3270e2f740 1 -- start start 2024-03-24T10:46:43.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.990+0000 7f3270e2f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x56053162b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.990+0000 7f3270e2f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 0x56053161d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.990+0000 7f3270e2f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56053161fc60 0x5605316379d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.990+0000 7f3270e2f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x560531616130 con 0x560531616760 2024-03-24T10:46:43.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f3270e2f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560531617b00 con 0x560531617080 2024-03-24T10:46:43.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f3270e2f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56053160aff0 con 0x56053161fc60 2024-03-24T10:46:43.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f32677fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x56053162b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f32677fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x56053162b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59118/0 (socket says 172.21.15.23:59118) 2024-03-24T10:46:43.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f32677fe640 1 -- 172.21.15.23:0/1340351669 learned_addr learned my addr 172.21.15.23:0/1340351669 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f3267fff640 1 --2- 172.21.15.23:0/1340351669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56053161fc60 0x5605316379d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f3266ffd640 1 --2- 172.21.15.23:0/1340351669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 0x56053161d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f3267fff640 1 -- 172.21.15.23:0/1340351669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 msgr2=0x56053161d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f3267fff640 1 --2- 172.21.15.23:0/1340351669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 0x56053161d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f3267fff640 1 -- 172.21.15.23:0/1340351669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 msgr2=0x56053162b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f3267fff640 1 --2- 172.21.15.23:0/1340351669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x56053162b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f3267fff640 1 -- 172.21.15.23:0/1340351669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560531517c60 con 0x56053161fc60 2024-03-24T10:46:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f32677fe640 1 --2- 172.21.15.23:0/1340351669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x56053162b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f3267fff640 1 --2- 172.21.15.23:0/1340351669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56053161fc60 0x5605316379d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f32600106c0 tx=0x7f32600106f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=330e8c3a33f73015 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f324d7fa640 1 -- 172.21.15.23:0/1340351669 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326002d070 con 0x56053161fc60 2024-03-24T10:46:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f324d7fa640 1 -- 172.21.15.23:0/1340351669 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3260010430 con 0x56053161fc60 2024-03-24T10:46:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.991+0000 7f324d7fa640 1 -- 172.21.15.23:0/1340351669 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260010760 con 0x56053161fc60 2024-03-24T10:46:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.992+0000 7f3270e2f740 1 -- 172.21.15.23:0/1340351669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56053161fc60 msgr2=0x5605316379d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.992+0000 7f3270e2f740 1 --2- 172.21.15.23:0/1340351669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56053161fc60 0x5605316379d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f32600106c0 tx=0x7f32600106f0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.992+0000 7f3270e2f740 1 -- 172.21.15.23:0/1340351669 shutdown_connections 2024-03-24T10:46:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.992+0000 7f3270e2f740 1 --2- 172.21.15.23:0/1340351669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56053161fc60 0x5605316379d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.992+0000 7f3270e2f740 1 --2- 172.21.15.23:0/1340351669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 0x56053161d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.992+0000 7f3270e2f740 1 --2- 172.21.15.23:0/1340351669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560531616760 0x56053162b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.992+0000 7f3270e2f740 1 -- 172.21.15.23:0/1340351669 >> 172.21.15.23:0/1340351669 conn(0x560531610e90 msgr2=0x560531612a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.992+0000 7f3270e2f740 1 -- 172.21.15.23:0/1340351669 shutdown_connections 2024-03-24T10:46:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.992+0000 7f3270e2f740 1 -- 172.21.15.23:0/1340351669 wait complete. 2024-03-24T10:46:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.992+0000 7f3270e2f740 1 -- start start 2024-03-24T10:46:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f3270e2f740 1 Processor -- start 2024-03-24T10:46:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f3270e2f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560531616760 0x560531627d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f3270e2f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 0x560531624af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f3270e2f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56053161fc60 0x560531625030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f3270e2f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x560531625700 con 0x56053161fc60 2024-03-24T10:46:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f3270e2f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560531625870 con 0x560531617080 2024-03-24T10:46:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f3270e2f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560531696010 con 0x560531616760 2024-03-24T10:46:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f3270e2f740 10 client.0 _ll_register_callbacks cb 0x7f3258010e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f3267fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56053161fc60 0x560531625030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f32677fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560531616760 0x560531627d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f32677fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560531616760 0x560531627d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:47792/0 (socket says 172.21.15.23:47792) 2024-03-24T10:46:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f32677fe640 1 -- 172.21.15.23:0/2133483451 learned_addr learned my addr 172.21.15.23:0/2133483451 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f3266ffd640 1 --2- 172.21.15.23:0/2133483451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 0x560531624af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f32677fe640 1 -- 172.21.15.23:0/2133483451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 msgr2=0x560531624af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f32677fe640 1 --2- 172.21.15.23:0/2133483451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 0x560531624af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f32677fe640 1 -- 172.21.15.23:0/2133483451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56053161fc60 msgr2=0x560531625030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f32677fe640 1 --2- 172.21.15.23:0/2133483451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56053161fc60 0x560531625030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f32677fe640 1 -- 172.21.15.23:0/2133483451 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605315336d0 con 0x560531616760 2024-03-24T10:46:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f3267fff640 1 --2- 172.21.15.23:0/2133483451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56053161fc60 0x560531625030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f32677fe640 1 --2- 172.21.15.23:0/2133483451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560531616760 0x560531627d10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f32580990f0 tx=0x7f32580cbbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b188471d8dcb6a52 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.993+0000 7f3246ffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:46:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3246ffd640 1 -- 172.21.15.23:0/2133483451 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325809a980 con 0x560531616760 2024-03-24T10:46:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3246ffd640 1 -- 172.21.15.23:0/2133483451 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f32580b92c0 con 0x560531616760 2024-03-24T10:46:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3270e2f740 1 -- 172.21.15.23:0/2133483451 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56053127c860 con 0x560531616760 2024-03-24T10:46:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3246ffd640 1 -- 172.21.15.23:0/2133483451 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f325809ae40 con 0x560531616760 2024-03-24T10:46:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3270e2f740 10 client.27355 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3270e2f740 1 -- 172.21.15.23:0/2133483451 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56053157b8c0 con 0x560531616760 2024-03-24T10:46:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3246ffd640 1 -- 172.21.15.23:0/2133483451 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56053127c860 con 0x560531616760 2024-03-24T10:46:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3246ffd640 1 -- 172.21.15.23:0/2133483451 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f32580b3be0 con 0x560531616760 2024-03-24T10:46:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3270e2f740 10 client.27355 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3270e2f740 10 client.27355 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3270e2f740 20 client.27355 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3270e2f740 1 --2- 172.21.15.23:0/2133483451 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x560531697420 0x560531699840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3270e2f740 4 client.27355 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:46:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3270e2f740 1 -- 172.21.15.23:0/2133483451 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5605312cd410 con 0x560531697420 2024-03-24T10:46:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.994+0000 7f3266ffd640 1 --2- 172.21.15.23:0/2133483451 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x560531697420 0x560531699840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.995+0000 7f3266ffd640 1 --2- 172.21.15.23:0/2133483451 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x560531697420 0x560531699840 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3b5869e565fd170 server_cookie=62af0a4298daac31 in_seq=0 out_seq=0 2024-03-24T10:46:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.995+0000 7f3246ffd640 10 client.27355 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:44.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.996+0000 7f3246ffd640 1 -- 172.21.15.23:0/2133483451 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5605312cd410 con 0x560531697420 2024-03-24T10:46:44.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.996+0000 7f3246ffd640 10 client.27355 handle_command_reply: tid=0 2024-03-24T10:46:44.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.996+0000 7f3270e2f740 1 client.27355 shutdown 2024-03-24T10:46:44.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.996+0000 7f3270e2f740 1 -- 172.21.15.23:0/2133483451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560531616760 msgr2=0x560531627d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:44.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.996+0000 7f3270e2f740 1 --2- 172.21.15.23:0/2133483451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560531616760 0x560531627d10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f32580990f0 tx=0x7f32580cbbe0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.996+0000 7f3270e2f740 1 -- 172.21.15.23:0/2133483451 shutdown_connections 2024-03-24T10:46:44.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.996+0000 7f3270e2f740 1 --2- 172.21.15.23:0/2133483451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56053161fc60 0x560531625030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.996+0000 7f3270e2f740 1 --2- 172.21.15.23:0/2133483451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560531617080 0x560531624af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.996+0000 7f3270e2f740 1 --2- 172.21.15.23:0/2133483451 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x560531697420 0x560531699840 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.996+0000 7f3270e2f740 1 --2- 172.21.15.23:0/2133483451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560531616760 0x560531627d10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.996+0000 7f3270e2f740 1 -- 172.21.15.23:0/2133483451 >> 172.21.15.23:0/2133483451 conn(0x560531610e90 msgr2=0x56053160f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:44.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.996+0000 7f3246ffd640 0 client.27355 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:44.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.996+0000 7f3270e2f740 1 -- 172.21.15.23:0/2133483451 shutdown_connections 2024-03-24T10:46:44.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.997+0000 7f3270e2f740 1 -- 172.21.15.23:0/2133483451 wait complete. 2024-03-24T10:46:44.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.997+0000 7f3270e2f740 20 client.27355 trim_cache size 0 max 16384 2024-03-24T10:46:44.004 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:46:44.004 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:46:44.004 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "ce07b8d7-f500-4bfe-9d9b-ed201b57fb2a", 2024-03-24T10:46:44.004 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:46:44.004 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:46:44.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.997+0000 7f326e24e640 1 -- 172.21.15.23:0/2484936668 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f32500a2230 msgr2=0x7f32500a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:44.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.997+0000 7f326e24e640 1 --2- 172.21.15.23:0/2484936668 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f32500a2230 0x7f32500a46f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3260023000 tx=0x7f3260022000 comp rx=0 tx=0).stop 2024-03-24T10:46:44.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.997+0000 7f326e24e640 1 -- 172.21.15.23:0/2484936668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32680688f0 msgr2=0x7f3268199f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:44.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.997+0000 7f326e24e640 1 --2- 172.21.15.23:0/2484936668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32680688f0 0x7f3268199f30 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f325800e9f0 tx=0x7f325800eeb0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.997+0000 7f326e24e640 1 -- 172.21.15.23:0/2484936668 shutdown_connections 2024-03-24T10:46:44.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.997+0000 7f326e24e640 1 --2- 172.21.15.23:0/2484936668 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f32500a2230 0x7f32500a46f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.997+0000 7f326e24e640 1 --2- 172.21.15.23:0/2484936668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3268107690 0x7f326819e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.997+0000 7f326e24e640 1 --2- 172.21.15.23:0/2484936668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32680692b0 0x7f326819a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.997+0000 7f326e24e640 1 --2- 172.21.15.23:0/2484936668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32680688f0 0x7f3268199f30 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.997+0000 7f326e24e640 1 -- 172.21.15.23:0/2484936668 >> 172.21.15.23:0/2484936668 conn(0x7f3268077710 msgr2=0x7f326810ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:44.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.997+0000 7f326e24e640 1 -- 172.21.15.23:0/2484936668 shutdown_connections 2024-03-24T10:46:44.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:43.997+0000 7f326e24e640 1 -- 172.21.15.23:0/2484936668 wait complete. 2024-03-24T10:46:44.008 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:46:44.008 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:46:44.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.119+0000 7f9e4eb19640 1 -- 172.21.15.23:0/684489028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e481076c0 msgr2=0x7f9e48107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:44.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.119+0000 7f9e4eb19640 1 --2- 172.21.15.23:0/684489028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e481076c0 0x7f9e48107aa0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f9e40009cd0 tx=0x7f9e4002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.119+0000 7f9e4eb19640 1 -- 172.21.15.23:0/684489028 shutdown_connections 2024-03-24T10:46:44.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.119+0000 7f9e4eb19640 1 --2- 172.21.15.23:0/684489028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e48102070 0x7f9e4810e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.119+0000 7f9e4eb19640 1 --2- 172.21.15.23:0/684489028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e481016b0 0x7f9e48101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.119+0000 7f9e4eb19640 1 --2- 172.21.15.23:0/684489028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e481076c0 0x7f9e48107aa0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.119+0000 7f9e4eb19640 1 -- 172.21.15.23:0/684489028 >> 172.21.15.23:0/684489028 conn(0x7f9e480fd140 msgr2=0x7f9e480ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:44.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.120+0000 7f9e4eb19640 1 -- 172.21.15.23:0/684489028 shutdown_connections 2024-03-24T10:46:44.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.120+0000 7f9e4eb19640 1 -- 172.21.15.23:0/684489028 wait complete. 2024-03-24T10:46:44.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.120+0000 7f9e4eb19640 1 Processor -- start 2024-03-24T10:46:44.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.120+0000 7f9e4eb19640 1 -- start start 2024-03-24T10:46:44.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4eb19640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e481016b0 0x7f9e48197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4eb19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e48102070 0x7f9e481982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4eb19640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e481076c0 0x7f9e4819c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4eb19640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4819cc00 con 0x7f9e481016b0 2024-03-24T10:46:44.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4eb19640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4819cd70 con 0x7f9e48102070 2024-03-24T10:46:44.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4eb19640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4819cf30 con 0x7f9e481076c0 2024-03-24T10:46:44.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4c8b5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e481016b0 0x7f9e48197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4c8b5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e481016b0 0x7f9e48197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59146/0 (socket says 172.21.15.23:59146) 2024-03-24T10:46:44.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4c8b5640 1 -- 172.21.15.23:0/2591637571 learned_addr learned my addr 172.21.15.23:0/2591637571 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:44.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e47fff640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e48102070 0x7f9e481982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4d0b6640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e481076c0 0x7f9e4819c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4c8b5640 1 -- 172.21.15.23:0/2591637571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e481076c0 msgr2=0x7f9e4819c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:44.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4c8b5640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e481076c0 0x7f9e4819c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4c8b5640 1 -- 172.21.15.23:0/2591637571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e48102070 msgr2=0x7f9e481982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:44.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4c8b5640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e48102070 0x7f9e481982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4c8b5640 1 -- 172.21.15.23:0/2591637571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e40009900 con 0x7f9e481016b0 2024-03-24T10:46:44.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4d0b6640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e481076c0 0x7f9e4819c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:44.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4c8b5640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e481016b0 0x7f9e48197d90 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f9e40004af0 tx=0x7f9e400325c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:44.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e45ffb640 1 -- 172.21.15.23:0/2591637571 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e40034e20 con 0x7f9e481016b0 2024-03-24T10:46:44.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e45ffb640 1 -- 172.21.15.23:0/2591637571 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e40003e50 con 0x7f9e481016b0 2024-03-24T10:46:44.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e4eb19640 1 -- 172.21.15.23:0/2591637571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e4819d1b0 con 0x7f9e481016b0 2024-03-24T10:46:44.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.121+0000 7f9e45ffb640 1 -- 172.21.15.23:0/2591637571 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e40004880 con 0x7f9e481016b0 2024-03-24T10:46:44.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.122+0000 7f9e4eb19640 1 -- 172.21.15.23:0/2591637571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e481a4a90 con 0x7f9e481016b0 2024-03-24T10:46:44.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 642940 11.10 MB/sec execute 181 sec latency 41.123 ms 2024-03-24T10:46:44.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.124+0000 7f9e45ffb640 1 -- 172.21.15.23:0/2591637571 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f9e4000a040 con 0x7f9e481016b0 2024-03-24T10:46:44.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.124+0000 7f9e517cb740 1 Processor -- start 2024-03-24T10:46:44.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.124+0000 7f9e517cb740 1 -- start start 2024-03-24T10:46:44.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.124+0000 7f9e45ffb640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f9e340a22b0 0x7f9e340a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.124+0000 7f9e517cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edab905b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.124+0000 7f9e517cb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edab99190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.124+0000 7f9e517cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 0x563edab9bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e517cb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563edaab4f30 con 0x563edab90af0 2024-03-24T10:46:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e517cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563edaafe810 con 0x563edab901d0 2024-03-24T10:46:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e517cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563edaafbd40 con 0x563edab996d0 2024-03-24T10:46:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e4d0b6640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f9e340a22b0 0x7f9e340a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e45ffb640 1 -- 172.21.15.23:0/2591637571 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f9e400f0a30 con 0x7f9e481016b0 2024-03-24T10:46:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e47fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edab99190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e47fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edab99190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59158/0 (socket says 172.21.15.23:59158) 2024-03-24T10:46:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e4c8b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edab905b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e47fff640 1 -- 172.21.15.23:0/3423594921 learned_addr learned my addr 172.21.15.23:0/3423594921 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e4d0b6640 1 --2- 172.21.15.23:0/3423594921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 0x563edab9bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e47fff640 1 -- 172.21.15.23:0/3423594921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 msgr2=0x563edab9bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e47fff640 1 --2- 172.21.15.23:0/3423594921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 0x563edab9bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e47fff640 1 -- 172.21.15.23:0/3423594921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 msgr2=0x563edab905b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:44.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e47fff640 1 --2- 172.21.15.23:0/3423594921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edab905b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e47fff640 1 -- 172.21.15.23:0/3423594921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563edaab8ea0 con 0x563edab90af0 2024-03-24T10:46:44.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e47fff640 1 --2- 172.21.15.23:0/3423594921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edab99190 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f9e30009c90 tx=0x7f9e30011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a87f1bb9c01afb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:44.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e4d0b6640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f9e340a22b0 0x7f9e340a4770 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9e38013820 tx=0x7f9e38009c20 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:44.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e3effd640 1 -- 172.21.15.23:0/3423594921 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e30019620 con 0x563edab90af0 2024-03-24T10:46:44.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e3effd640 1 -- 172.21.15.23:0/3423594921 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e30019c60 con 0x563edab90af0 2024-03-24T10:46:44.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.125+0000 7f9e3effd640 1 -- 172.21.15.23:0/3423594921 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e30018bb0 con 0x563edab90af0 2024-03-24T10:46:44.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.126+0000 7f9e517cb740 1 -- 172.21.15.23:0/3423594921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 msgr2=0x563edab99190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:44.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.126+0000 7f9e517cb740 1 --2- 172.21.15.23:0/3423594921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edab99190 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f9e30009c90 tx=0x7f9e30011810 comp rx=0 tx=0).stop 2024-03-24T10:46:44.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.126+0000 7f9e517cb740 1 -- 172.21.15.23:0/3423594921 shutdown_connections 2024-03-24T10:46:44.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.126+0000 7f9e517cb740 1 --2- 172.21.15.23:0/3423594921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 0x563edab9bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.126+0000 7f9e517cb740 1 --2- 172.21.15.23:0/3423594921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edab99190 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.126+0000 7f9e517cb740 1 --2- 172.21.15.23:0/3423594921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edab905b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.126+0000 7f9e517cb740 1 -- 172.21.15.23:0/3423594921 >> 172.21.15.23:0/3423594921 conn(0x563edab8a900 msgr2=0x563edab8ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:44.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.126+0000 7f9e517cb740 1 -- 172.21.15.23:0/3423594921 shutdown_connections 2024-03-24T10:46:44.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.126+0000 7f9e517cb740 1 -- 172.21.15.23:0/3423594921 wait complete. 2024-03-24T10:46:44.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.126+0000 7f9e517cb740 1 -- start start 2024-03-24T10:46:44.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.126+0000 7f9e517cb740 1 Processor -- start 2024-03-24T10:46:44.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e517cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edab96740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e517cb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edab96c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e517cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 0x563edab971c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e517cb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563edaaf7520 con 0x563edab90af0 2024-03-24T10:46:44.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e517cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563eda7ca430 con 0x563edab901d0 2024-03-24T10:46:44.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e517cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563eda786e00 con 0x563edab996d0 2024-03-24T10:46:44.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e517cb740 10 client.0 _ll_register_callbacks cb 0x563edaa95d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:44.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e47fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edab96c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e4c8b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edab96740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e4d0b6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 0x563edab971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e47fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edab96c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59168/0 (socket says 172.21.15.23:59168) 2024-03-24T10:46:44.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e47fff640 1 -- 172.21.15.23:0/2320717644 learned_addr learned my addr 172.21.15.23:0/2320717644 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:44.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e4c8b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edab96740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:55210/0 (socket says 172.21.15.23:55210) 2024-03-24T10:46:44.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e47fff640 1 -- 172.21.15.23:0/2320717644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 msgr2=0x563edab971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:44.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e47fff640 1 --2- 172.21.15.23:0/2320717644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 0x563edab971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e47fff640 1 -- 172.21.15.23:0/2320717644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 msgr2=0x563edab96740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e47fff640 1 --2- 172.21.15.23:0/2320717644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edab96740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e47fff640 1 -- 172.21.15.23:0/2320717644 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e300098f0 con 0x563edab90af0 2024-03-24T10:46:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e47fff640 1 --2- 172.21.15.23:0/2320717644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edab96c80 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f9e3001b040 tx=0x7f9e300183d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6e785c1f883a2f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.127+0000 7f9e27fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.128+0000 7f9e27fff640 1 -- 172.21.15.23:0/2320717644 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e30005240 con 0x563edab90af0 2024-03-24T10:46:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.128+0000 7f9e27fff640 1 -- 172.21.15.23:0/2320717644 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f9e30018480 con 0x563edab90af0 2024-03-24T10:46:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.128+0000 7f9e517cb740 1 -- 172.21.15.23:0/2320717644 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563eda7f67e0 con 0x563edab90af0 2024-03-24T10:46:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.128+0000 7f9e27fff640 1 -- 172.21.15.23:0/2320717644 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9e300053a0 con 0x563edab90af0 2024-03-24T10:46:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.128+0000 7f9e517cb740 10 client.17508 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.128+0000 7f9e517cb740 1 -- 172.21.15.23:0/2320717644 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563edaaf5ba0 con 0x563edab90af0 2024-03-24T10:46:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.128+0000 7f9e27fff640 1 -- 172.21.15.23:0/2320717644 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563eda7f67e0 con 0x563edab90af0 2024-03-24T10:46:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.128+0000 7f9e27fff640 1 -- 172.21.15.23:0/2320717644 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f9e30021040 con 0x563edab90af0 2024-03-24T10:46:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.128+0000 7f9e517cb740 10 client.17508 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.128+0000 7f9e517cb740 10 client.17508 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.128+0000 7f9e517cb740 20 client.17508 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.128+0000 7f9e517cb740 1 --2- 172.21.15.23:0/2320717644 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563edaba9b90 0x563edababff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.128+0000 7f9e517cb740 4 client.17508 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.128+0000 7f9e517cb740 1 -- 172.21.15.23:0/2320717644 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563eda833720 con 0x563edaba9b90 2024-03-24T10:46:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.129+0000 7f9e4c8b5640 1 --2- 172.21.15.23:0/2320717644 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563edaba9b90 0x563edababff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.129+0000 7f9e27fff640 10 client.17508 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.129+0000 7f9e4c8b5640 1 --2- 172.21.15.23:0/2320717644 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563edaba9b90 0x563edababff0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2916a8a278dae2d server_cookie=329f3b461d5d2ed in_seq=0 out_seq=0 2024-03-24T10:46:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.130+0000 7f9e27fff640 1 -- 172.21.15.23:0/2320717644 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563eda833720 con 0x563edaba9b90 2024-03-24T10:46:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.130+0000 7f9e27fff640 10 client.17508 handle_command_reply: tid=0 2024-03-24T10:46:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.130+0000 7f9e517cb740 1 client.17508 shutdown 2024-03-24T10:46:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.130+0000 7f9e517cb740 1 -- 172.21.15.23:0/2320717644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 msgr2=0x563edab96c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.130+0000 7f9e517cb740 1 --2- 172.21.15.23:0/2320717644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edab96c80 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f9e3001b040 tx=0x7f9e300183d0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.130+0000 7f9e517cb740 1 -- 172.21.15.23:0/2320717644 shutdown_connections 2024-03-24T10:46:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.130+0000 7f9e517cb740 1 --2- 172.21.15.23:0/2320717644 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563edaba9b90 0x563edababff0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.130+0000 7f9e517cb740 1 --2- 172.21.15.23:0/2320717644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 0x563edab971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.130+0000 7f9e517cb740 1 --2- 172.21.15.23:0/2320717644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edab96c80 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.130+0000 7f9e517cb740 1 --2- 172.21.15.23:0/2320717644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edab96740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.130+0000 7f9e27fff640 0 client.17508 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.130+0000 7f9e517cb740 1 -- 172.21.15.23:0/2320717644 >> 172.21.15.23:0/2320717644 conn(0x563edab8a900 msgr2=0x563edab8c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.130+0000 7f9e517cb740 1 -- 172.21.15.23:0/2320717644 shutdown_connections 2024-03-24T10:46:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.130+0000 7f9e517cb740 1 -- 172.21.15.23:0/2320717644 wait complete. 2024-03-24T10:46:44.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.131+0000 7f9e517cb740 20 client.17508 trim_cache size 0 max 16384 2024-03-24T10:46:44.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e517cb740 1 Processor -- start 2024-03-24T10:46:44.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e517cb740 1 -- start start 2024-03-24T10:46:44.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e517cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edab98e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e517cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab90af0 0x563edaba4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e517cb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab996d0 0x563edab89b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e517cb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563edaa7c430 con 0x563edab996d0 2024-03-24T10:46:44.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e517cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563edab8b900 con 0x563edab901d0 2024-03-24T10:46:44.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e517cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563edaba9890 con 0x563edab90af0 2024-03-24T10:46:44.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e4d0b6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab996d0 0x563edab89b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e4d0b6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab996d0 0x563edab89b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59184/0 (socket says 172.21.15.23:59184) 2024-03-24T10:46:44.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e4d0b6640 1 -- 172.21.15.23:0/1207455322 learned_addr learned my addr 172.21.15.23:0/1207455322 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:44.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e47fff640 1 --2- 172.21.15.23:0/1207455322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab90af0 0x563edaba4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e4c8b5640 1 --2- 172.21.15.23:0/1207455322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edab98e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e4d0b6640 1 -- 172.21.15.23:0/1207455322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab90af0 msgr2=0x563edaba4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:44.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e4d0b6640 1 --2- 172.21.15.23:0/1207455322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab90af0 0x563edaba4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e4d0b6640 1 -- 172.21.15.23:0/1207455322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 msgr2=0x563edab98e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:44.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e4d0b6640 1 --2- 172.21.15.23:0/1207455322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edab98e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e4d0b6640 1 -- 172.21.15.23:0/1207455322 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563edaa91b20 con 0x563edab996d0 2024-03-24T10:46:44.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e47fff640 1 --2- 172.21.15.23:0/1207455322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab90af0 0x563edaba4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:44.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e4c8b5640 1 --2- 172.21.15.23:0/1207455322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edab98e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:44.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.149+0000 7f9e4d0b6640 1 --2- 172.21.15.23:0/1207455322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab996d0 0x563edab89b60 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f9e380133a0 tx=0x7f9e38005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f3b384255133e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:44.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.150+0000 7f9e3e7fc640 1 -- 172.21.15.23:0/1207455322 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e38025070 con 0x563edab996d0 2024-03-24T10:46:44.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.150+0000 7f9e3e7fc640 1 -- 172.21.15.23:0/1207455322 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e38004510 con 0x563edab996d0 2024-03-24T10:46:44.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.150+0000 7f9e3e7fc640 1 -- 172.21.15.23:0/1207455322 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e380036e0 con 0x563edab996d0 2024-03-24T10:46:44.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.150+0000 7f9e517cb740 1 -- 172.21.15.23:0/1207455322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab996d0 msgr2=0x563edab89b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:44.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.150+0000 7f9e517cb740 1 --2- 172.21.15.23:0/1207455322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab996d0 0x563edab89b60 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f9e380133a0 tx=0x7f9e38005f90 comp rx=0 tx=0).stop 2024-03-24T10:46:44.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.150+0000 7f9e517cb740 1 -- 172.21.15.23:0/1207455322 shutdown_connections 2024-03-24T10:46:44.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.150+0000 7f9e517cb740 1 --2- 172.21.15.23:0/1207455322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab996d0 0x563edab89b60 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.150+0000 7f9e517cb740 1 --2- 172.21.15.23:0/1207455322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab90af0 0x563edaba4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.150+0000 7f9e517cb740 1 --2- 172.21.15.23:0/1207455322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edab98e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.150+0000 7f9e517cb740 1 -- 172.21.15.23:0/1207455322 >> 172.21.15.23:0/1207455322 conn(0x563edab8a900 msgr2=0x563edab8ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:44.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.151+0000 7f9e517cb740 1 -- 172.21.15.23:0/1207455322 shutdown_connections 2024-03-24T10:46:44.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.151+0000 7f9e517cb740 1 -- 172.21.15.23:0/1207455322 wait complete. 2024-03-24T10:46:44.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.151+0000 7f9e517cb740 1 -- start start 2024-03-24T10:46:44.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.151+0000 7f9e517cb740 1 Processor -- start 2024-03-24T10:46:44.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.151+0000 7f9e517cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edac0bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.151+0000 7f9e517cb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edaba1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.151+0000 7f9e517cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 0x563edac08e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.151+0000 7f9e517cb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563edaba1b50 con 0x563edab90af0 2024-03-24T10:46:44.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.151+0000 7f9e517cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563edaba1cc0 con 0x563edab901d0 2024-03-24T10:46:44.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.151+0000 7f9e517cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563edac09390 con 0x563edab996d0 2024-03-24T10:46:44.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e517cb740 10 client.0 _ll_register_callbacks cb 0x7f9e40005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:44.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e47fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edaba1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e47fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edaba1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59200/0 (socket says 172.21.15.23:59200) 2024-03-24T10:46:44.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e4d0b6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 0x563edac08e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e47fff640 1 -- 172.21.15.23:0/3946840831 learned_addr learned my addr 172.21.15.23:0/3946840831 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:44.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e4c8b5640 1 --2- 172.21.15.23:0/3946840831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edac0bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e4d0b6640 1 -- 172.21.15.23:0/3946840831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 msgr2=0x563edac0bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:44.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e4d0b6640 1 --2- 172.21.15.23:0/3946840831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edac0bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e4d0b6640 1 -- 172.21.15.23:0/3946840831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 msgr2=0x563edaba1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:44.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e4d0b6640 1 --2- 172.21.15.23:0/3946840831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edaba1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e4d0b6640 1 -- 172.21.15.23:0/3946840831 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e481a4a90 con 0x563edab996d0 2024-03-24T10:46:44.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e47fff640 1 --2- 172.21.15.23:0/3946840831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edaba1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:44.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e4d0b6640 1 --2- 172.21.15.23:0/3946840831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 0x563edac08e50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9e38005f60 tx=0x7f9e38005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c96ce914f4c47e2c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:44.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e27fff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:46:44.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e27fff640 1 -- 172.21.15.23:0/3946840831 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e38025040 con 0x563edab996d0 2024-03-24T10:46:44.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e27fff640 1 -- 172.21.15.23:0/3946840831 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f9e380048c0 con 0x563edab996d0 2024-03-24T10:46:44.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e517cb740 1 -- 172.21.15.23:0/3946840831 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e30019c60 con 0x563edab996d0 2024-03-24T10:46:44.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e27fff640 1 -- 172.21.15.23:0/3946840831 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9e38013ec0 con 0x563edab996d0 2024-03-24T10:46:44.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e517cb740 10 client.27373 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:44.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.152+0000 7f9e517cb740 1 -- 172.21.15.23:0/3946840831 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563edaaf5ba0 con 0x563edab996d0 2024-03-24T10:46:44.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.153+0000 7f9e27fff640 1 -- 172.21.15.23:0/3946840831 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e30019c60 con 0x563edab996d0 2024-03-24T10:46:44.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.153+0000 7f9e27fff640 1 -- 172.21.15.23:0/3946840831 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f9e3803b990 con 0x563edab996d0 2024-03-24T10:46:44.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.153+0000 7f9e517cb740 10 client.27373 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:44.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.153+0000 7f9e517cb740 10 client.27373 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:44.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.153+0000 7f9e517cb740 20 client.27373 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:44.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.153+0000 7f9e517cb740 1 --2- 172.21.15.23:0/3946840831 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563edac196e0 0x563edac1bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:44.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.153+0000 7f9e517cb740 4 client.27373 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:46:44.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.153+0000 7f9e517cb740 1 -- 172.21.15.23:0/3946840831 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563eda882890 con 0x563edac196e0 2024-03-24T10:46:44.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.153+0000 7f9e4c8b5640 1 --2- 172.21.15.23:0/3946840831 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563edac196e0 0x563edac1bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:44.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.153+0000 7f9e27fff640 10 client.27373 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:44.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.153+0000 7f9e4c8b5640 1 --2- 172.21.15.23:0/3946840831 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563edac196e0 0x563edac1bb00 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8fc9d0c5d6629c4 server_cookie=f1bfcd320775b090 in_seq=0 out_seq=0 2024-03-24T10:46:44.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e27fff640 1 -- 172.21.15.23:0/3946840831 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563eda882890 con 0x563edac196e0 2024-03-24T10:46:44.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e27fff640 10 client.27373 handle_command_reply: tid=0 2024-03-24T10:46:44.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e517cb740 1 client.27373 shutdown 2024-03-24T10:46:44.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e517cb740 1 -- 172.21.15.23:0/3946840831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 msgr2=0x563edac08e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:44.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e517cb740 1 --2- 172.21.15.23:0/3946840831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 0x563edac08e50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9e38005f60 tx=0x7f9e38005270 comp rx=0 tx=0).stop 2024-03-24T10:46:44.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e517cb740 1 -- 172.21.15.23:0/3946840831 shutdown_connections 2024-03-24T10:46:44.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e517cb740 1 --2- 172.21.15.23:0/3946840831 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563edac196e0 0x563edac1bb00 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e517cb740 1 --2- 172.21.15.23:0/3946840831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563edab996d0 0x563edac08e50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e27fff640 0 client.27373 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:44.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e517cb740 1 --2- 172.21.15.23:0/3946840831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563edab90af0 0x563edaba1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e517cb740 1 --2- 172.21.15.23:0/3946840831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563edab901d0 0x563edac0bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e517cb740 1 -- 172.21.15.23:0/3946840831 >> 172.21.15.23:0/3946840831 conn(0x563edab8a900 msgr2=0x563edab89b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:44.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e517cb740 1 -- 172.21.15.23:0/3946840831 shutdown_connections 2024-03-24T10:46:44.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e517cb740 1 -- 172.21.15.23:0/3946840831 wait complete. 2024-03-24T10:46:44.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e517cb740 20 client.27373 trim_cache size 0 max 16384 2024-03-24T10:46:44.162 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:46:44.162 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:46:44.162 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:46:44.162 INFO:teuthology.orchestra.run.smithi023.stdout: "ce07b8d7-f500-4bfe-9d9b-ed201b57fb2a": { 2024-03-24T10:46:44.163 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:46:44.163 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "ce07b8d7-f500-4bfe-9d9b-ed201b57fb2a", 2024-03-24T10:46:44.163 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:46:44.163 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:46:44.163 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:46:44.163 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:46:44.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e4eb19640 1 -- 172.21.15.23:0/2591637571 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f9e340a22b0 msgr2=0x7f9e340a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:44.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e4eb19640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f9e340a22b0 0x7f9e340a4770 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9e38013820 tx=0x7f9e38009c20 comp rx=0 tx=0).stop 2024-03-24T10:46:44.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e4eb19640 1 -- 172.21.15.23:0/2591637571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e481016b0 msgr2=0x7f9e48197d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:44.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.154+0000 7f9e4eb19640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e481016b0 0x7f9e48197d90 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f9e40004af0 tx=0x7f9e400325c0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.155+0000 7f9e4eb19640 1 -- 172.21.15.23:0/2591637571 shutdown_connections 2024-03-24T10:46:44.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.155+0000 7f9e4eb19640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f9e340a22b0 0x7f9e340a4770 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.155+0000 7f9e4eb19640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e481076c0 0x7f9e4819c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.155+0000 7f9e4eb19640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e48102070 0x7f9e481982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.155+0000 7f9e4eb19640 1 --2- 172.21.15.23:0/2591637571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e481016b0 0x7f9e48197d90 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:44.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.155+0000 7f9e4eb19640 1 -- 172.21.15.23:0/2591637571 >> 172.21.15.23:0/2591637571 conn(0x7f9e480fd140 msgr2=0x7f9e480fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:44.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.155+0000 7f9e4eb19640 1 -- 172.21.15.23:0/2591637571 shutdown_connections 2024-03-24T10:46:44.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:46:44.155+0000 7f9e4eb19640 1 -- 172.21.15.23:0/2591637571 wait complete. 2024-03-24T10:46:44.166 INFO:tasks.cephfs.filesystem:scrub status for tag:ce07b8d7-f500-4bfe-9d9b-ed201b57fb2a - {'path': '/', 'tag': 'ce07b8d7-f500-4bfe-9d9b-ed201b57fb2a', 'options': 'recursive,force'} 2024-03-24T10:46:44.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:44 smithi117 ceph-mon[36368]: pgmap v203: 97 pgs: 97 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 172 op/s 2024-03-24T10:46:44.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:44 smithi116 ceph-mon[36248]: pgmap v203: 97 pgs: 97 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 172 op/s 2024-03-24T10:46:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:44 smithi023 ceph-mon[32466]: pgmap v203: 97 pgs: 97 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 172 op/s 2024-03-24T10:46:45.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 644330 11.07 MB/sec execute 182 sec latency 48.698 ms 2024-03-24T10:46:45.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:45 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:46:45.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:45 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:45.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:45 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:46:45.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:45 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:46:45.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:45 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:45.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:45 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:45.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:45 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:46:45.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:45 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:45.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:45 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:46:45.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:45 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:46:45.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:45 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:45.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:45 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:45.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:45 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:46:45.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:45 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:45.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:45 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:46:45.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:45 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:46:45.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:45 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:45.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:45 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:46:46.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 646485 11.08 MB/sec execute 183 sec latency 100.720 ms 2024-03-24T10:46:46.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:46 smithi117 ceph-mon[36368]: pgmap v204: 97 pgs: 97 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 172 op/s 2024-03-24T10:46:46.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:46 smithi116 ceph-mon[36248]: pgmap v204: 97 pgs: 97 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 172 op/s 2024-03-24T10:46:46.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:46 smithi023 ceph-mon[32466]: pgmap v204: 97 pgs: 97 active+clean; 374 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 172 op/s 2024-03-24T10:46:47.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 647909 11.06 MB/sec execute 184 sec latency 32.991 ms 2024-03-24T10:46:48.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 650203 11.07 MB/sec execute 185 sec latency 17.595 ms 2024-03-24T10:46:48.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:48 smithi117 ceph-mon[36368]: pgmap v205: 97 pgs: 97 active+clean; 380 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 263 op/s 2024-03-24T10:46:48.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:48 smithi116 ceph-mon[36248]: pgmap v205: 97 pgs: 97 active+clean; 380 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 263 op/s 2024-03-24T10:46:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:48 smithi023 ceph-mon[32466]: pgmap v205: 97 pgs: 97 active+clean; 380 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 263 op/s 2024-03-24T10:46:49.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 651717 11.04 MB/sec execute 186 sec latency 19.325 ms 2024-03-24T10:46:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.937+0000 7f561c78a640 1 -- 172.21.15.23:0/2991192624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5614069730 msgr2=0x7f561410e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.937+0000 7f561c78a640 1 --2- 172.21.15.23:0/2991192624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5614069730 0x7f561410e5a0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f560c009ca0 tx=0x7f560c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.937+0000 7f561c78a640 1 -- 172.21.15.23:0/2991192624 shutdown_connections 2024-03-24T10:46:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.937+0000 7f561c78a640 1 --2- 172.21.15.23:0/2991192624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5614069730 0x7f561410e5a0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.937+0000 7f561c78a640 1 --2- 172.21.15.23:0/2991192624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5614068d90 0x7f56140691f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.937+0000 7f561c78a640 1 --2- 172.21.15.23:0/2991192624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5614107800 0x7f5614107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.937+0000 7f561c78a640 1 -- 172.21.15.23:0/2991192624 >> 172.21.15.23:0/2991192624 conn(0x7f5614077460 msgr2=0x7f5614077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.937+0000 7f561c78a640 1 -- 172.21.15.23:0/2991192624 shutdown_connections 2024-03-24T10:46:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.937+0000 7f561c78a640 1 -- 172.21.15.23:0/2991192624 wait complete. 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.937+0000 7f561c78a640 1 Processor -- start 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561c78a640 1 -- start start 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561c78a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5614068d90 0x7f561419d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561c78a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5614069730 0x7f561419ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561c78a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5614107800 0x7f5614197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561c78a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f5614198070 con 0x7f5614107800 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561c78a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56141981e0 con 0x7f5614069730 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561c78a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f56141983a0 con 0x7f5614068d90 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561ad27640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5614107800 0x7f5614197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561ad27640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5614107800 0x7f5614197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51096/0 (socket says 172.21.15.23:51096) 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561ad27640 1 -- 172.21.15.23:0/248441584 learned_addr learned my addr 172.21.15.23:0/248441584 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f5619d25640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5614069730 0x7f561419ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561a526640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5614068d90 0x7f561419d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561ad27640 1 -- 172.21.15.23:0/248441584 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5614068d90 msgr2=0x7f561419d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561ad27640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5614068d90 0x7f561419d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561ad27640 1 -- 172.21.15.23:0/248441584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5614069730 msgr2=0x7f561419ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561ad27640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5614069730 0x7f561419ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561ad27640 1 -- 172.21.15.23:0/248441584 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f560c009900 con 0x7f5614107800 2024-03-24T10:46:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561a526640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5614068d90 0x7f561419d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f5619d25640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5614069730 0x7f561419ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.938+0000 7f561ad27640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5614107800 0x7f5614197a20 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f560c02c700 tx=0x7f560c034a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.939+0000 7f560b7fe640 1 -- 172.21.15.23:0/248441584 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c0328b0 con 0x7f5614107800 2024-03-24T10:46:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.939+0000 7f560b7fe640 1 -- 172.21.15.23:0/248441584 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f560c003e50 con 0x7f5614107800 2024-03-24T10:46:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.939+0000 7f560b7fe640 1 -- 172.21.15.23:0/248441584 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c0047f0 con 0x7f5614107800 2024-03-24T10:46:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.939+0000 7f561c78a640 1 -- 172.21.15.23:0/248441584 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5614198620 con 0x7f5614107800 2024-03-24T10:46:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.939+0000 7f561c78a640 1 -- 172.21.15.23:0/248441584 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56141a4910 con 0x7f5614107800 2024-03-24T10:46:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.941+0000 7f560b7fe640 1 -- 172.21.15.23:0/248441584 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f560c00a040 con 0x7f5614107800 2024-03-24T10:46:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f560b7fe640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f56000a2350 0x7f56000a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f561f368740 1 Processor -- start 2024-03-24T10:46:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f560b7fe640 1 -- 172.21.15.23:0/248441584 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f560c0f0b30 con 0x7f5614107800 2024-03-24T10:46:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f561a526640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f56000a2350 0x7f56000a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f561f368740 1 -- start start 2024-03-24T10:46:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f561f368740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e7080 0x55b0786e7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f561f368740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786e79a0 0x55b0786f4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f561f368740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786f50e0 0x55b0786f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f561f368740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b0786a21b0 con 0x55b0786f50e0 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f561f368740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b078654f80 con 0x55b0786e79a0 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f561f368740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b0786524b0 con 0x55b0786e7080 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f5619d25640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786e79a0 0x55b0786f4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f561ad27640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786f50e0 0x55b0786f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f5619d25640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786e79a0 0x55b0786f4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:57026/0 (socket says 172.21.15.23:57026) 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f561a526640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f56000a2350 0x7f56000a4810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5614198b60 tx=0x7f5610005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f5619d25640 1 -- 172.21.15.23:0/2522111972 learned_addr learned my addr 172.21.15.23:0/2522111972 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f561a526640 1 --2- 172.21.15.23:0/2522111972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e7080 0x55b0786e7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f561ad27640 1 -- 172.21.15.23:0/2522111972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e7080 msgr2=0x55b0786e7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.942+0000 7f561ad27640 1 --2- 172.21.15.23:0/2522111972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e7080 0x55b0786e7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.943+0000 7f561ad27640 1 -- 172.21.15.23:0/2522111972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786e79a0 msgr2=0x55b0786f4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.943+0000 7f561ad27640 1 --2- 172.21.15.23:0/2522111972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786e79a0 0x55b0786f4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.943+0000 7f561ad27640 1 -- 172.21.15.23:0/2522111972 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0785e3810 con 0x55b0786f50e0 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.943+0000 7f5619d25640 1 --2- 172.21.15.23:0/2522111972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786e79a0 0x55b0786f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.943+0000 7f561ad27640 1 --2- 172.21.15.23:0/2522111972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786f50e0 0x55b0786f74d0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f560c0ee390 tx=0x7f560c0b9980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8bfa025365aea29 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.943+0000 7f5608ff9640 1 -- 172.21.15.23:0/2522111972 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c0bbe90 con 0x55b0786f50e0 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.943+0000 7f5608ff9640 1 -- 172.21.15.23:0/2522111972 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f560c0bbca0 con 0x55b0786f50e0 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.943+0000 7f5608ff9640 1 -- 172.21.15.23:0/2522111972 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c0c4890 con 0x55b0786f50e0 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 -- 172.21.15.23:0/2522111972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786f50e0 msgr2=0x55b0786f74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 --2- 172.21.15.23:0/2522111972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786f50e0 0x55b0786f74d0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f560c0ee390 tx=0x7f560c0b9980 comp rx=0 tx=0).stop 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 -- 172.21.15.23:0/2522111972 shutdown_connections 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 --2- 172.21.15.23:0/2522111972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786f50e0 0x55b0786f74d0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 --2- 172.21.15.23:0/2522111972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786e79a0 0x55b0786f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 --2- 172.21.15.23:0/2522111972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e7080 0x55b0786e7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 -- 172.21.15.23:0/2522111972 >> 172.21.15.23:0/2522111972 conn(0x55b0786e1f00 msgr2=0x55b0786e22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 -- 172.21.15.23:0/2522111972 shutdown_connections 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 -- 172.21.15.23:0/2522111972 wait complete. 2024-03-24T10:46:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 -- start start 2024-03-24T10:46:49.948 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 Processor -- start 2024-03-24T10:46:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e7080 0x55b0786f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e79a0 0x55b0786f2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786f50e0 0x55b0786f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b07864dc90 con 0x55b0786e79a0 2024-03-24T10:46:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b0785f2c50 con 0x55b0786f50e0 2024-03-24T10:46:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b078321250 con 0x55b0786e7080 2024-03-24T10:46:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f561f368740 10 client.0 _ll_register_callbacks cb 0x55b078641290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f5619d25640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e79a0 0x55b0786f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.944+0000 7f5619d25640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e79a0 0x55b0786f2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51116/0 (socket says 172.21.15.23:51116) 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f5619d25640 1 -- 172.21.15.23:0/3653006727 learned_addr learned my addr 172.21.15.23:0/3653006727 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f561a526640 1 --2- 172.21.15.23:0/3653006727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e7080 0x55b0786f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f561ad27640 1 --2- 172.21.15.23:0/3653006727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786f50e0 0x55b0786f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f5619d25640 1 -- 172.21.15.23:0/3653006727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e7080 msgr2=0x55b0786f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f5619d25640 1 --2- 172.21.15.23:0/3653006727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e7080 0x55b0786f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f5619d25640 1 -- 172.21.15.23:0/3653006727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786f50e0 msgr2=0x55b0786f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f5619d25640 1 --2- 172.21.15.23:0/3653006727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786f50e0 0x55b0786f2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f5619d25640 1 -- 172.21.15.23:0/3653006727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56141a4910 con 0x55b0786e79a0 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f5619d25640 1 --2- 172.21.15.23:0/3653006727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e79a0 0x55b0786f2620 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f5604012d20 tx=0x7f560400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4862b41b6eec1fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f55fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f55fdffb640 1 -- 172.21.15.23:0/3653006727 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5604017070 con 0x55b0786e79a0 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f55fdffb640 1 -- 172.21.15.23:0/3653006727 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f560400fe20 con 0x55b0786e79a0 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f561f368740 1 -- 172.21.15.23:0/3653006727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b078672790 con 0x55b0786e79a0 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f55fdffb640 1 -- 172.21.15.23:0/3653006727 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f56040122e0 con 0x55b0786e79a0 2024-03-24T10:46:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f561f368740 10 client.17526 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f561f368740 1 -- 172.21.15.23:0/3653006727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b07864c310 con 0x55b0786e79a0 2024-03-24T10:46:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.945+0000 7f55fdffb640 1 -- 172.21.15.23:0/3653006727 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5604028080 con 0x55b0786e79a0 2024-03-24T10:46:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.946+0000 7f55fdffb640 1 -- 172.21.15.23:0/3653006727 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f5604022030 con 0x55b0786e79a0 2024-03-24T10:46:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.946+0000 7f561f368740 10 client.17526 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.946+0000 7f561f368740 10 client.17526 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.946+0000 7f561f368740 20 client.17526 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.946+0000 7f561f368740 1 --2- 172.21.15.23:0/3653006727 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b078704ef0 0x55b078707310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.946+0000 7f561f368740 4 client.17526 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:46:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.946+0000 7f561f368740 1 -- 172.21.15.23:0/3653006727 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b07834d600 con 0x55b078704ef0 2024-03-24T10:46:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.946+0000 7f561a526640 1 --2- 172.21.15.23:0/3653006727 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b078704ef0 0x55b078707310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.946+0000 7f561a526640 1 --2- 172.21.15.23:0/3653006727 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b078704ef0 0x55b078707310 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80365e8862193a5e server_cookie=c2cd7e9af733421d in_seq=0 out_seq=0 2024-03-24T10:46:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.946+0000 7f55fdffb640 10 client.17526 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.947+0000 7f55fdffb640 1 -- 172.21.15.23:0/3653006727 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b07834d600 con 0x55b078704ef0 2024-03-24T10:46:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.947+0000 7f55fdffb640 10 client.17526 handle_command_reply: tid=0 2024-03-24T10:46:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.947+0000 7f561f368740 1 client.17526 shutdown 2024-03-24T10:46:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.947+0000 7f561f368740 1 -- 172.21.15.23:0/3653006727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e79a0 msgr2=0x55b0786f2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.947+0000 7f561f368740 1 --2- 172.21.15.23:0/3653006727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e79a0 0x55b0786f2620 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f5604012d20 tx=0x7f560400f790 comp rx=0 tx=0).stop 2024-03-24T10:46:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.947+0000 7f561f368740 1 -- 172.21.15.23:0/3653006727 shutdown_connections 2024-03-24T10:46:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.947+0000 7f561f368740 1 --2- 172.21.15.23:0/3653006727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786f50e0 0x55b0786f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.947+0000 7f561f368740 1 --2- 172.21.15.23:0/3653006727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e79a0 0x55b0786f2620 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.947+0000 7f561f368740 1 --2- 172.21.15.23:0/3653006727 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b078704ef0 0x55b078707310 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.948+0000 7f561f368740 1 --2- 172.21.15.23:0/3653006727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e7080 0x55b0786f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.948+0000 7f55fdffb640 0 client.17526 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.948+0000 7f561f368740 1 -- 172.21.15.23:0/3653006727 >> 172.21.15.23:0/3653006727 conn(0x55b0786e1f00 msgr2=0x55b0786e3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.948+0000 7f561f368740 1 -- 172.21.15.23:0/3653006727 shutdown_connections 2024-03-24T10:46:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.948+0000 7f561f368740 1 -- 172.21.15.23:0/3653006727 wait complete. 2024-03-24T10:46:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.948+0000 7f561f368740 20 client.17526 trim_cache size 0 max 16384 2024-03-24T10:46:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.963+0000 7f561f368740 1 Processor -- start 2024-03-24T10:46:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f561f368740 1 -- start start 2024-03-24T10:46:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f561f368740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786e7080 0x55b0787005b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f561f368740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e79a0 0x55b07870cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f561f368740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786f50e0 0x55b0786fcbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f561f368740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b0785d3660 con 0x55b0786e79a0 2024-03-24T10:46:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f561f368740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b0786e8590 con 0x55b0786e7080 2024-03-24T10:46:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f561f368740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b0787079a0 con 0x55b0786f50e0 2024-03-24T10:46:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f5619d25640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e79a0 0x55b07870cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f5619d25640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e79a0 0x55b07870cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51120/0 (socket says 172.21.15.23:51120) 2024-03-24T10:46:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f5619d25640 1 -- 172.21.15.23:0/911065957 learned_addr learned my addr 172.21.15.23:0/911065957 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f561a526640 1 --2- 172.21.15.23:0/911065957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786e7080 0x55b0787005b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f561ad27640 1 --2- 172.21.15.23:0/911065957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786f50e0 0x55b0786fcbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f5619d25640 1 -- 172.21.15.23:0/911065957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786f50e0 msgr2=0x55b0786fcbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f5619d25640 1 --2- 172.21.15.23:0/911065957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786f50e0 0x55b0786fcbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f5619d25640 1 -- 172.21.15.23:0/911065957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786e7080 msgr2=0x55b0787005b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f5619d25640 1 --2- 172.21.15.23:0/911065957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786e7080 0x55b0787005b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f5619d25640 1 -- 172.21.15.23:0/911065957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0785e8810 con 0x55b0786e79a0 2024-03-24T10:46:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f5619d25640 1 --2- 172.21.15.23:0/911065957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e79a0 0x55b07870cbe0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f560401c000 tx=0x7f5604012830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=435fdbd10732c91f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.964+0000 7f55fffff640 1 -- 172.21.15.23:0/911065957 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5604017070 con 0x55b0786e79a0 2024-03-24T10:46:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.965+0000 7f55fffff640 1 -- 172.21.15.23:0/911065957 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b078672790 con 0x55b0786e79a0 2024-03-24T10:46:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.965+0000 7f55fffff640 1 -- 172.21.15.23:0/911065957 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5604005780 con 0x55b0786e79a0 2024-03-24T10:46:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.965+0000 7f561f368740 1 -- 172.21.15.23:0/911065957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e79a0 msgr2=0x55b07870cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.965+0000 7f561f368740 1 --2- 172.21.15.23:0/911065957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e79a0 0x55b07870cbe0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f560401c000 tx=0x7f5604012830 comp rx=0 tx=0).stop 2024-03-24T10:46:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.965+0000 7f561f368740 1 -- 172.21.15.23:0/911065957 shutdown_connections 2024-03-24T10:46:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.965+0000 7f561f368740 1 --2- 172.21.15.23:0/911065957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786f50e0 0x55b0786fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.965+0000 7f561f368740 1 --2- 172.21.15.23:0/911065957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e79a0 0x55b07870cbe0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.965+0000 7f561f368740 1 --2- 172.21.15.23:0/911065957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786e7080 0x55b0787005b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.965+0000 7f561f368740 1 -- 172.21.15.23:0/911065957 >> 172.21.15.23:0/911065957 conn(0x55b0786e1f00 msgr2=0x55b0786f20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.965+0000 7f561f368740 1 -- 172.21.15.23:0/911065957 shutdown_connections 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.965+0000 7f561f368740 1 -- 172.21.15.23:0/911065957 wait complete. 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561f368740 1 -- start start 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561f368740 1 Processor -- start 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561f368740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e7080 0x55b0786fcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561f368740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e79a0 0x55b0786f90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561f368740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786f50e0 0x55b0786f9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561f368740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b0786f9bd0 con 0x55b0786e7080 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561f368740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b0786f9d40 con 0x55b0786f50e0 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561f368740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b0786f9f00 con 0x55b0786e79a0 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561f368740 10 client.0 _ll_register_callbacks cb 0x7f5604009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561a526640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e7080 0x55b0786fcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561a526640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e7080 0x55b0786fcd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51122/0 (socket says 172.21.15.23:51122) 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561a526640 1 -- 172.21.15.23:0/337245354 learned_addr learned my addr 172.21.15.23:0/337245354 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561ad27640 1 --2- 172.21.15.23:0/337245354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786f50e0 0x55b0786f9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f5619d25640 1 --2- 172.21.15.23:0/337245354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e79a0 0x55b0786f90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561a526640 1 -- 172.21.15.23:0/337245354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e79a0 msgr2=0x55b0786f90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561a526640 1 --2- 172.21.15.23:0/337245354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e79a0 0x55b0786f90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561a526640 1 -- 172.21.15.23:0/337245354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786f50e0 msgr2=0x55b0786f9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561a526640 1 --2- 172.21.15.23:0/337245354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786f50e0 0x55b0786f9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561a526640 1 -- 172.21.15.23:0/337245354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b07864c310 con 0x55b0786e7080 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f561a526640 1 --2- 172.21.15.23:0/337245354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e7080 0x55b0786fcd50 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x55b078707b80 tx=0x7f5610004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc18f911a6af0356 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.966+0000 7f55fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f55fdffb640 1 -- 172.21.15.23:0/337245354 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5610022040 con 0x55b0786e7080 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f55fdffb640 1 -- 172.21.15.23:0/337245354 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f561002c150 con 0x55b0786e7080 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f561f368740 1 -- 172.21.15.23:0/337245354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f560c0bbca0 con 0x55b0786e7080 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f55fdffb640 1 -- 172.21.15.23:0/337245354 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f561002c310 con 0x55b0786e7080 2024-03-24T10:46:49.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f561f368740 10 client.17532 fetch_fsmap learned FSMap version 29 2024-03-24T10:46:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f561f368740 1 -- 172.21.15.23:0/337245354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b07860e5a0 con 0x55b0786e7080 2024-03-24T10:46:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f55fdffb640 1 -- 172.21.15.23:0/337245354 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f560c0bbca0 con 0x55b0786e7080 2024-03-24T10:46:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f55fdffb640 1 -- 172.21.15.23:0/337245354 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f561000fbe0 con 0x55b0786e7080 2024-03-24T10:46:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f561f368740 10 client.17532 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:46:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f561f368740 10 client.17532 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:46:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f561f368740 20 client.17532 populate_metadata read hostname 'smithi023' 2024-03-24T10:46:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f561f368740 1 --2- 172.21.15.23:0/337245354 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b07876c900 0x55b07876ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:46:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f561f368740 4 client.17532 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:46:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f561f368740 1 -- 172.21.15.23:0/337245354 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b07839e1b0 con 0x55b07876c900 2024-03-24T10:46:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.967+0000 7f5619d25640 1 --2- 172.21.15.23:0/337245354 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b07876c900 0x55b07876ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:46:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.968+0000 7f55fdffb640 10 client.17532 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:49.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.968+0000 7f5619d25640 1 --2- 172.21.15.23:0/337245354 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b07876c900 0x55b07876ed20 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc7fe945315e437 server_cookie=88d57d6e4ce7e3b2 in_seq=0 out_seq=0 2024-03-24T10:46:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.978+0000 7f55fdffb640 1 -- 172.21.15.23:0/337245354 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b07839e1b0 con 0x55b07876c900 2024-03-24T10:46:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.978+0000 7f55fdffb640 10 client.17532 handle_command_reply: tid=0 2024-03-24T10:46:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.978+0000 7f561f368740 1 client.17532 shutdown 2024-03-24T10:46:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.978+0000 7f561f368740 1 -- 172.21.15.23:0/337245354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e7080 msgr2=0x55b0786fcd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.978+0000 7f561f368740 1 --2- 172.21.15.23:0/337245354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e7080 0x55b0786fcd50 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x55b078707b80 tx=0x7f5610004730 comp rx=0 tx=0).stop 2024-03-24T10:46:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.978+0000 7f561f368740 1 -- 172.21.15.23:0/337245354 shutdown_connections 2024-03-24T10:46:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.978+0000 7f561f368740 1 --2- 172.21.15.23:0/337245354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0786f50e0 0x55b0786f9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.978+0000 7f561f368740 1 --2- 172.21.15.23:0/337245354 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b07876c900 0x55b07876ed20 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.978+0000 7f561f368740 1 --2- 172.21.15.23:0/337245354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0786e79a0 0x55b0786f90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.978+0000 7f561f368740 1 --2- 172.21.15.23:0/337245354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b0786e7080 0x55b0786fcd50 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.978+0000 7f55fdffb640 0 client.17532 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:46:49.980 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.978+0000 7f561f368740 1 -- 172.21.15.23:0/337245354 >> 172.21.15.23:0/337245354 conn(0x55b0786e1f00 msgr2=0x55b0786f2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.978+0000 7f561f368740 1 -- 172.21.15.23:0/337245354 shutdown_connections 2024-03-24T10:46:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.978+0000 7f561f368740 1 -- 172.21.15.23:0/337245354 wait complete. 2024-03-24T10:46:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.979+0000 7f561f368740 20 client.17532 trim_cache size 0 max 16384 2024-03-24T10:46:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:46:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:46:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:46:49.981 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:46:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.979+0000 7f561c78a640 1 -- 172.21.15.23:0/248441584 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f56000a2350 msgr2=0x7f56000a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.979+0000 7f561c78a640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f56000a2350 0x7f56000a4810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5614198b60 tx=0x7f5610005f90 comp rx=0 tx=0).stop 2024-03-24T10:46:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.979+0000 7f561c78a640 1 -- 172.21.15.23:0/248441584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5614107800 msgr2=0x7f5614197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:46:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.979+0000 7f561c78a640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5614107800 0x7f5614197a20 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f560c02c700 tx=0x7f560c034a90 comp rx=0 tx=0).stop 2024-03-24T10:46:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.979+0000 7f561c78a640 1 -- 172.21.15.23:0/248441584 shutdown_connections 2024-03-24T10:46:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.979+0000 7f561c78a640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f56000a2350 0x7f56000a4810 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.979+0000 7f561c78a640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5614107800 0x7f5614197a20 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.979+0000 7f561c78a640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5614069730 0x7f561419ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.979+0000 7f561c78a640 1 --2- 172.21.15.23:0/248441584 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5614068d90 0x7f561419d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:46:49.982 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.979+0000 7f561c78a640 1 -- 172.21.15.23:0/248441584 >> 172.21.15.23:0/248441584 conn(0x7f5614077460 msgr2=0x7f56140fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:46:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.979+0000 7f561c78a640 1 -- 172.21.15.23:0/248441584 shutdown_connections 2024-03-24T10:46:49.983 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:46:49.979+0000 7f561c78a640 1 -- 172.21.15.23:0/248441584 wait complete. 2024-03-24T10:46:50.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 653895 11.05 MB/sec execute 187 sec latency 45.572 ms 2024-03-24T10:46:50.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:50 smithi117 ceph-mon[36368]: pgmap v206: 97 pgs: 97 active+clean; 380 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 179 op/s 2024-03-24T10:46:50.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:50 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:46:50.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:50 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:46:50.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:50 smithi116 ceph-mon[36248]: pgmap v206: 97 pgs: 97 active+clean; 380 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 179 op/s 2024-03-24T10:46:50.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:50 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:46:50.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:50 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:46:50.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:50 smithi023 ceph-mon[32466]: pgmap v206: 97 pgs: 97 active+clean; 380 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 179 op/s 2024-03-24T10:46:50.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:50 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:46:50.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:50 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:46:51.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 655396 11.03 MB/sec execute 188 sec latency 100.099 ms 2024-03-24T10:46:52.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 657550 11.04 MB/sec execute 189 sec latency 31.601 ms 2024-03-24T10:46:52.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:52 smithi117 ceph-mon[36368]: pgmap v207: 97 pgs: 97 active+clean; 387 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 267 op/s 2024-03-24T10:46:52.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:52 smithi116 ceph-mon[36248]: pgmap v207: 97 pgs: 97 active+clean; 387 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 267 op/s 2024-03-24T10:46:52.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:52 smithi023 ceph-mon[32466]: pgmap v207: 97 pgs: 97 active+clean; 387 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 267 op/s 2024-03-24T10:46:53.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 659438 11.02 MB/sec execute 190 sec latency 25.437 ms 2024-03-24T10:46:54.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 661337 11.03 MB/sec execute 191 sec latency 23.911 ms 2024-03-24T10:46:54.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:54 smithi117 ceph-mon[36368]: pgmap v208: 97 pgs: 97 active+clean; 387 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 177 op/s 2024-03-24T10:46:54.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:54 smithi116 ceph-mon[36248]: pgmap v208: 97 pgs: 97 active+clean; 387 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 177 op/s 2024-03-24T10:46:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:54 smithi023 ceph-mon[32466]: pgmap v208: 97 pgs: 97 active+clean; 387 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 177 op/s 2024-03-24T10:46:55.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 663431 11.03 MB/sec execute 192 sec latency 19.569 ms 2024-03-24T10:46:56.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 664957 11.01 MB/sec execute 193 sec latency 59.966 ms 2024-03-24T10:46:56.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:56 smithi117 ceph-mon[36368]: pgmap v209: 97 pgs: 97 active+clean; 387 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 177 op/s 2024-03-24T10:46:56.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:56 smithi116 ceph-mon[36248]: pgmap v209: 97 pgs: 97 active+clean; 387 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 177 op/s 2024-03-24T10:46:56.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:56 smithi023 ceph-mon[32466]: pgmap v209: 97 pgs: 97 active+clean; 387 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 177 op/s 2024-03-24T10:46:57.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 666949 11.01 MB/sec execute 194 sec latency 20.066 ms 2024-03-24T10:46:58.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 668764 11.00 MB/sec execute 195 sec latency 18.663 ms 2024-03-24T10:46:58.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:46:58 smithi117 ceph-mon[36368]: pgmap v210: 97 pgs: 97 active+clean; 395 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 264 op/s 2024-03-24T10:46:58.769 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:46:58 smithi116 ceph-mon[36248]: pgmap v210: 97 pgs: 97 active+clean; 395 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 264 op/s 2024-03-24T10:46:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:46:58 smithi023 ceph-mon[32466]: pgmap v210: 97 pgs: 97 active+clean; 395 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 264 op/s 2024-03-24T10:46:59.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 671118 11.03 MB/sec execute 196 sec latency 18.287 ms 2024-03-24T10:47:00.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 672589 11.00 MB/sec execute 197 sec latency 31.322 ms 2024-03-24T10:47:00.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:00 smithi117 ceph-mon[36368]: pgmap v211: 97 pgs: 97 active+clean; 395 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 174 op/s 2024-03-24T10:47:00.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:00 smithi116 ceph-mon[36248]: pgmap v211: 97 pgs: 97 active+clean; 395 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 174 op/s 2024-03-24T10:47:00.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:00 smithi023 ceph-mon[32466]: pgmap v211: 97 pgs: 97 active+clean; 395 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 174 op/s 2024-03-24T10:47:01.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 674452 10.99 MB/sec execute 198 sec latency 69.152 ms 2024-03-24T10:47:02.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 675448 10.97 MB/sec execute 199 sec latency 44.772 ms 2024-03-24T10:47:02.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:02 smithi117 ceph-mon[36368]: pgmap v212: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 290 op/s 2024-03-24T10:47:02.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:02 smithi116 ceph-mon[36248]: pgmap v212: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 290 op/s 2024-03-24T10:47:02.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:02 smithi023 ceph-mon[32466]: pgmap v212: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 290 op/s 2024-03-24T10:47:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.070+0000 7f2a88da1640 1 -- 172.21.15.23:0/1959680365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a841076c0 msgr2=0x7f2a84107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.070+0000 7f2a88da1640 1 --2- 172.21.15.23:0/1959680365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a841076c0 0x7f2a84107aa0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009930 tx=0x7f2a7402bc30 comp rx=0 tx=0).stop 2024-03-24T10:47:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.070+0000 7f2a88da1640 1 -- 172.21.15.23:0/1959680365 shutdown_connections 2024-03-24T10:47:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.070+0000 7f2a88da1640 1 --2- 172.21.15.23:0/1959680365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a84102070 0x7f2a8410e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.070+0000 7f2a88da1640 1 --2- 172.21.15.23:0/1959680365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a841016b0 0x7f2a84101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.070+0000 7f2a88da1640 1 --2- 172.21.15.23:0/1959680365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a841076c0 0x7f2a84107aa0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.070+0000 7f2a88da1640 1 -- 172.21.15.23:0/1959680365 >> 172.21.15.23:0/1959680365 conn(0x7f2a840fd1a0 msgr2=0x7f2a840ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:03.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.070+0000 7f2a88da1640 1 -- 172.21.15.23:0/1959680365 shutdown_connections 2024-03-24T10:47:03.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.070+0000 7f2a88da1640 1 -- 172.21.15.23:0/1959680365 wait complete. 2024-03-24T10:47:03.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a88da1640 1 Processor -- start 2024-03-24T10:47:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a88da1640 1 -- start start 2024-03-24T10:47:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a88da1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a841016b0 0x7f2a84199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a88da1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a84102070 0x7f2a8419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a88da1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a841076c0 0x7f2a8419e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a88da1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8419eda0 con 0x7f2a84102070 2024-03-24T10:47:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a88da1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8419ef10 con 0x7f2a841016b0 2024-03-24T10:47:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a88da1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8419f0d0 con 0x7f2a841076c0 2024-03-24T10:47:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a7bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a84102070 0x7f2a8419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a7bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a84102070 0x7f2a8419a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60698/0 (socket says 172.21.15.23:60698) 2024-03-24T10:47:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a8259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a841016b0 0x7f2a84199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a7bfff640 1 -- 172.21.15.23:0/2756180957 learned_addr learned my addr 172.21.15.23:0/2756180957 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a82d9d640 1 --2- 172.21.15.23:0/2756180957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a841076c0 0x7f2a8419e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a7bfff640 1 -- 172.21.15.23:0/2756180957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a841076c0 msgr2=0x7f2a8419e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a7bfff640 1 --2- 172.21.15.23:0/2756180957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a841076c0 0x7f2a8419e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a7bfff640 1 -- 172.21.15.23:0/2756180957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a841016b0 msgr2=0x7f2a84199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:03.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a7bfff640 1 --2- 172.21.15.23:0/2756180957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a841016b0 0x7f2a84199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a7bfff640 1 -- 172.21.15.23:0/2756180957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a74009560 con 0x7f2a84102070 2024-03-24T10:47:03.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.071+0000 7f2a7bfff640 1 --2- 172.21.15.23:0/2756180957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a84102070 0x7f2a8419a420 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00e9f0 tx=0x7f2a6c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:03.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.072+0000 7f2a7b7fe640 1 -- 172.21.15.23:0/2756180957 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c009700 con 0x7f2a84102070 2024-03-24T10:47:03.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.072+0000 7f2a7b7fe640 1 -- 172.21.15.23:0/2756180957 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a6c00cd80 con 0x7f2a84102070 2024-03-24T10:47:03.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.072+0000 7f2a88da1640 1 -- 172.21.15.23:0/2756180957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a8419f3b0 con 0x7f2a84102070 2024-03-24T10:47:03.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.072+0000 7f2a7b7fe640 1 -- 172.21.15.23:0/2756180957 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c018920 con 0x7f2a84102070 2024-03-24T10:47:03.074 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.072+0000 7f2a88da1640 1 -- 172.21.15.23:0/2756180957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a841a6c30 con 0x7f2a84102070 2024-03-24T10:47:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.074+0000 7f2a7b7fe640 1 -- 172.21.15.23:0/2756180957 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f2a6c018a80 con 0x7f2a84102070 2024-03-24T10:47:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.074+0000 7f2a8b985740 1 Processor -- start 2024-03-24T10:47:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a7b7fe640 1 --2- 172.21.15.23:0/2756180957 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2a700a22d0 0x7f2a700a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8b985740 1 -- start start 2024-03-24T10:47:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a7b7fe640 1 -- 172.21.15.23:0/2756180957 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f2a6c014070 con 0x7f2a84102070 2024-03-24T10:47:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a82d9d640 1 --2- 172.21.15.23:0/2756180957 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2a700a22d0 0x7f2a700a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8b985740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e892b5e0 0x5614e892b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8b985740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 0x5614e8934790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8b985740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e8934cd0 0x5614e89370c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8b985740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5614e88e61b0 con 0x5614e892b5e0 2024-03-24T10:47:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8b985740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5614e8898f80 con 0x5614e892bf00 2024-03-24T10:47:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8b985740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5614e88964b0 con 0x5614e8934cd0 2024-03-24T10:47:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e892b5e0 0x5614e892b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e892b5e0 0x5614e892b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60712/0 (socket says 172.21.15.23:60712) 2024-03-24T10:47:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8259c640 1 -- 172.21.15.23:0/1908110688 learned_addr learned my addr 172.21.15.23:0/1908110688 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a7bfff640 1 --2- 172.21.15.23:0/1908110688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 0x5614e8934790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a82d9d640 1 --2- 172.21.15.23:0/1908110688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e8934cd0 0x5614e89370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8259c640 1 -- 172.21.15.23:0/1908110688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e8934cd0 msgr2=0x5614e89370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8259c640 1 --2- 172.21.15.23:0/1908110688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e8934cd0 0x5614e89370c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8259c640 1 -- 172.21.15.23:0/1908110688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 msgr2=0x5614e8934790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8259c640 1 --2- 172.21.15.23:0/1908110688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 0x5614e8934790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8259c640 1 -- 172.21.15.23:0/1908110688 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614e8827810 con 0x5614e892b5e0 2024-03-24T10:47:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a8259c640 1 --2- 172.21.15.23:0/1908110688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e892b5e0 0x5614e892b9c0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f2a74036040 tx=0x7f2a7402c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a8667fd722e609 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:03.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.075+0000 7f2a82d9d640 1 --2- 172.21.15.23:0/2756180957 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2a700a22d0 0x7f2a700a4790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c005fd0 tx=0x7f2a7c013ae0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.076+0000 7f2a78ff9640 1 -- 172.21.15.23:0/1908110688 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a74004800 con 0x5614e892b5e0 2024-03-24T10:47:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.076+0000 7f2a78ff9640 1 -- 172.21.15.23:0/1908110688 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a74004960 con 0x5614e892b5e0 2024-03-24T10:47:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.076+0000 7f2a78ff9640 1 -- 172.21.15.23:0/1908110688 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a74004ff0 con 0x5614e892b5e0 2024-03-24T10:47:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.076+0000 7f2a8b985740 1 -- 172.21.15.23:0/1908110688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e892b5e0 msgr2=0x5614e892b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.076+0000 7f2a8b985740 1 --2- 172.21.15.23:0/1908110688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e892b5e0 0x5614e892b9c0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f2a74036040 tx=0x7f2a7402c5d0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.076+0000 7f2a8b985740 1 -- 172.21.15.23:0/1908110688 shutdown_connections 2024-03-24T10:47:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.076+0000 7f2a8b985740 1 --2- 172.21.15.23:0/1908110688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e8934cd0 0x5614e89370c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.076+0000 7f2a8b985740 1 --2- 172.21.15.23:0/1908110688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 0x5614e8934790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.076+0000 7f2a8b985740 1 --2- 172.21.15.23:0/1908110688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e892b5e0 0x5614e892b9c0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.076+0000 7f2a8b985740 1 -- 172.21.15.23:0/1908110688 >> 172.21.15.23:0/1908110688 conn(0x5614e8925f00 msgr2=0x5614e89262e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a8b985740 1 -- 172.21.15.23:0/1908110688 shutdown_connections 2024-03-24T10:47:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a8b985740 1 -- 172.21.15.23:0/1908110688 wait complete. 2024-03-24T10:47:03.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a8b985740 1 -- start start 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a8b985740 1 Processor -- start 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a8b985740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892b5e0 0x5614e8931ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a8b985740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 0x5614e89321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a8b985740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e8932720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a8b985740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5614e8891c90 con 0x5614e8934cd0 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a8b985740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5614e8836c50 con 0x5614e892bf00 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a8b985740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5614e8565250 con 0x5614e892b5e0 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a8b985740 10 client.0 _ll_register_callbacks cb 0x5614e8885290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a82d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e8932720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a82d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e8932720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60714/0 (socket says 172.21.15.23:60714) 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a82d9d640 1 -- 172.21.15.23:0/3508930955 learned_addr learned my addr 172.21.15.23:0/3508930955 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a8259c640 1 --2- 172.21.15.23:0/3508930955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892b5e0 0x5614e8931ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.077+0000 7f2a7bfff640 1 --2- 172.21.15.23:0/3508930955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 0x5614e89321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.078+0000 7f2a82d9d640 1 -- 172.21.15.23:0/3508930955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892b5e0 msgr2=0x5614e8931ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:03.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.078+0000 7f2a82d9d640 1 --2- 172.21.15.23:0/3508930955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892b5e0 0x5614e8931ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.078+0000 7f2a82d9d640 1 -- 172.21.15.23:0/3508930955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 msgr2=0x5614e89321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.078+0000 7f2a82d9d640 1 --2- 172.21.15.23:0/3508930955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 0x5614e89321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.078+0000 7f2a82d9d640 1 -- 172.21.15.23:0/3508930955 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a8410ee80 con 0x5614e8934cd0 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.078+0000 7f2a82d9d640 1 --2- 172.21.15.23:0/3508930955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e8932720 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c003d80 tx=0x7f2a7c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a73bb14fe2ffa835 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.078+0000 7f2a69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.078+0000 7f2a69ffb640 1 -- 172.21.15.23:0/3508930955 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c004310 con 0x5614e8934cd0 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.078+0000 7f2a69ffb640 1 -- 172.21.15.23:0/3508930955 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f2a7c004470 con 0x5614e8934cd0 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.078+0000 7f2a8b985740 1 -- 172.21.15.23:0/3508930955 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5614e88b6790 con 0x5614e8934cd0 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.078+0000 7f2a69ffb640 1 -- 172.21.15.23:0/3508930955 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2a7c004630 con 0x5614e8934cd0 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.078+0000 7f2a8b985740 10 client.17544 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.078+0000 7f2a8b985740 1 -- 172.21.15.23:0/3508930955 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5614e8890310 con 0x5614e8934cd0 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.078+0000 7f2a69ffb640 1 -- 172.21.15.23:0/3508930955 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5614e88b6790 con 0x5614e8934cd0 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.079+0000 7f2a69ffb640 1 -- 172.21.15.23:0/3508930955 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f2a7c020420 con 0x5614e8934cd0 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.079+0000 7f2a8b985740 10 client.17544 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.079+0000 7f2a8b985740 10 client.17544 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.079+0000 7f2a8b985740 20 client.17544 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.079+0000 7f2a8b985740 1 --2- 172.21.15.23:0/3508930955 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614e8944b10 0x5614e8946f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.079+0000 7f2a8b985740 4 client.17544 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.079+0000 7f2a8b985740 1 -- 172.21.15.23:0/3508930955 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614e8591600 con 0x5614e8944b10 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.079+0000 7f2a8259c640 1 --2- 172.21.15.23:0/3508930955 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614e8944b10 0x5614e8946f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.079+0000 7f2a8259c640 1 --2- 172.21.15.23:0/3508930955 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614e8944b10 0x5614e8946f30 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa8d912ede3ad7ba server_cookie=8ab5e6a9ad9fc352 in_seq=0 out_seq=0 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.079+0000 7f2a69ffb640 10 client.17544 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a69ffb640 1 -- 172.21.15.23:0/3508930955 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5614e8591600 con 0x5614e8944b10 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a69ffb640 10 client.17544 handle_command_reply: tid=0 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a8b985740 1 client.17544 shutdown 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a8b985740 1 -- 172.21.15.23:0/3508930955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 msgr2=0x5614e8932720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a8b985740 1 --2- 172.21.15.23:0/3508930955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e8932720 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c003d80 tx=0x7f2a7c004ab0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a8b985740 1 -- 172.21.15.23:0/3508930955 shutdown_connections 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a8b985740 1 --2- 172.21.15.23:0/3508930955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e8932720 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a8b985740 1 --2- 172.21.15.23:0/3508930955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 0x5614e89321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a8b985740 1 --2- 172.21.15.23:0/3508930955 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614e8944b10 0x5614e8946f30 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a8b985740 1 --2- 172.21.15.23:0/3508930955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892b5e0 0x5614e8931ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a69ffb640 0 client.17544 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a8b985740 1 -- 172.21.15.23:0/3508930955 >> 172.21.15.23:0/3508930955 conn(0x5614e8925f00 msgr2=0x5614e8927c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a8b985740 1 -- 172.21.15.23:0/3508930955 shutdown_connections 2024-03-24T10:47:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a8b985740 1 -- 172.21.15.23:0/3508930955 wait complete. 2024-03-24T10:47:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.080+0000 7f2a8b985740 20 client.17544 trim_cache size 0 max 16384 2024-03-24T10:47:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.095+0000 7f2a8b985740 1 Processor -- start 2024-03-24T10:47:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.095+0000 7f2a8b985740 1 -- start start 2024-03-24T10:47:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a8b985740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892b5e0 0x5614e8920d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a8b985740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892bf00 0x5614e8940340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a8b985740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e8932540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a8b985740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5614e8817660 con 0x5614e8934cd0 2024-03-24T10:47:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a8b985740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5614e8941590 con 0x5614e892b5e0 2024-03-24T10:47:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a8b985740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5614e8940880 con 0x5614e892bf00 2024-03-24T10:47:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a82d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e8932540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a82d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e8932540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60724/0 (socket says 172.21.15.23:60724) 2024-03-24T10:47:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a82d9d640 1 -- 172.21.15.23:0/1028259851 learned_addr learned my addr 172.21.15.23:0/1028259851 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a7bfff640 1 --2- 172.21.15.23:0/1028259851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892bf00 0x5614e8940340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a8259c640 1 --2- 172.21.15.23:0/1028259851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892b5e0 0x5614e8920d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a82d9d640 1 -- 172.21.15.23:0/1028259851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892bf00 msgr2=0x5614e8940340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a82d9d640 1 --2- 172.21.15.23:0/1028259851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892bf00 0x5614e8940340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a82d9d640 1 -- 172.21.15.23:0/1028259851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892b5e0 msgr2=0x5614e8920d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a82d9d640 1 --2- 172.21.15.23:0/1028259851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892b5e0 0x5614e8920d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a82d9d640 1 -- 172.21.15.23:0/1028259851 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614e882c810 con 0x5614e8934cd0 2024-03-24T10:47:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a7bfff640 1 --2- 172.21.15.23:0/1028259851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892bf00 0x5614e8940340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a82d9d640 1 --2- 172.21.15.23:0/1028259851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e8932540 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c020830 tx=0x7f2a7c009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4aeb95003593dd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a6bfff640 1 -- 172.21.15.23:0/1028259851 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c003970 con 0x5614e8934cd0 2024-03-24T10:47:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a6bfff640 1 -- 172.21.15.23:0/1028259851 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a7c013460 con 0x5614e8934cd0 2024-03-24T10:47:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.096+0000 7f2a6bfff640 1 -- 172.21.15.23:0/1028259851 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c005220 con 0x5614e8934cd0 2024-03-24T10:47:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.097+0000 7f2a8b985740 1 -- 172.21.15.23:0/1028259851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 msgr2=0x5614e8932540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.097+0000 7f2a8b985740 1 --2- 172.21.15.23:0/1028259851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e8932540 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c020830 tx=0x7f2a7c009630 comp rx=0 tx=0).stop 2024-03-24T10:47:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.097+0000 7f2a8b985740 1 -- 172.21.15.23:0/1028259851 shutdown_connections 2024-03-24T10:47:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.097+0000 7f2a8b985740 1 --2- 172.21.15.23:0/1028259851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e8932540 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.097+0000 7f2a8b985740 1 --2- 172.21.15.23:0/1028259851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892bf00 0x5614e8940340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.097+0000 7f2a8b985740 1 --2- 172.21.15.23:0/1028259851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892b5e0 0x5614e8920d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.097+0000 7f2a8b985740 1 -- 172.21.15.23:0/1028259851 >> 172.21.15.23:0/1028259851 conn(0x5614e8925f00 msgr2=0x5614e8924ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.097+0000 7f2a8b985740 1 -- 172.21.15.23:0/1028259851 shutdown_connections 2024-03-24T10:47:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.097+0000 7f2a8b985740 1 -- 172.21.15.23:0/1028259851 wait complete. 2024-03-24T10:47:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.097+0000 7f2a8b985740 1 -- start start 2024-03-24T10:47:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.097+0000 7f2a8b985740 1 Processor -- start 2024-03-24T10:47:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a8b985740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892b5e0 0x5614e8938960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a8b985740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 0x5614e8938ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a8b985740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e89a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a8b985740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5614e8939600 con 0x5614e8934cd0 2024-03-24T10:47:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a8b985740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5614e89aa3b0 con 0x5614e892bf00 2024-03-24T10:47:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a8b985740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5614e89aa570 con 0x5614e892b5e0 2024-03-24T10:47:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a8b985740 10 client.0 _ll_register_callbacks cb 0x7f2a7c012070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a82d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e89a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a8259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892b5e0 0x5614e8938960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a82d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e89a9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60740/0 (socket says 172.21.15.23:60740) 2024-03-24T10:47:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a82d9d640 1 -- 172.21.15.23:0/1082033726 learned_addr learned my addr 172.21.15.23:0/1082033726 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a7bfff640 1 --2- 172.21.15.23:0/1082033726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 0x5614e8938ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a8259c640 1 -- 172.21.15.23:0/1082033726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 msgr2=0x5614e8938ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a8259c640 1 --2- 172.21.15.23:0/1082033726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 0x5614e8938ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a8259c640 1 -- 172.21.15.23:0/1082033726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 msgr2=0x5614e89a9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a8259c640 1 --2- 172.21.15.23:0/1082033726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e89a9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a8259c640 1 -- 172.21.15.23:0/1082033726 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a841a6c30 con 0x5614e892b5e0 2024-03-24T10:47:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a82d9d640 1 --2- 172.21.15.23:0/1082033726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e89a9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a8259c640 1 --2- 172.21.15.23:0/1082033726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892b5e0 0x5614e8938960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2a7402c170 tx=0x7f2a74004400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f95ecfc219bc2601 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.098+0000 7f2a69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:47:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a69ffb640 1 -- 172.21.15.23:0/1082033726 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7404e040 con 0x5614e892b5e0 2024-03-24T10:47:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a8b985740 1 -- 172.21.15.23:0/1082033726 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a74004960 con 0x5614e892b5e0 2024-03-24T10:47:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a69ffb640 1 -- 172.21.15.23:0/1082033726 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8527+0+0 (secure 0 0 0) 0x7f2a74051a30 con 0x5614e892b5e0 2024-03-24T10:47:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a69ffb640 1 -- 172.21.15.23:0/1082033726 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2a74051c10 con 0x5614e892b5e0 2024-03-24T10:47:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a8b985740 10 client.27409 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a8b985740 1 -- 172.21.15.23:0/1082033726 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5614e8890310 con 0x5614e892b5e0 2024-03-24T10:47:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a69ffb640 1 -- 172.21.15.23:0/1082033726 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a74004960 con 0x5614e892b5e0 2024-03-24T10:47:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a69ffb640 1 -- 172.21.15.23:0/1082033726 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f2a74051050 con 0x5614e892b5e0 2024-03-24T10:47:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a8b985740 10 client.27409 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a8b985740 10 client.27409 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a8b985740 20 client.27409 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a8b985740 1 --2- 172.21.15.23:0/1082033726 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614e89abac0 0x5614e89adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a8b985740 4 client.27409 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:47:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a8b985740 1 -- 172.21.15.23:0/1082033726 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5614e85e21b0 con 0x5614e89abac0 2024-03-24T10:47:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.099+0000 7f2a7bfff640 1 --2- 172.21.15.23:0/1082033726 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614e89abac0 0x5614e89adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.100+0000 7f2a69ffb640 10 client.27409 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:03.100+0000 7f2a7bfff640 1 --2- 172.21.15.23:0/1082033726 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614e89abac0 0x5614e89adeb0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=712da73213c2a6ca server_cookie=39a73d053e923287 in_seq=0 out_seq=0 2024-03-24T10:47:03.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 677373 10.95 MB/sec execute 200 sec latency 19.821 ms 2024-03-24T10:47:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a69ffb640 1 -- 172.21.15.23:0/1082033726 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5614e85e21b0 con 0x5614e89abac0 2024-03-24T10:47:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a69ffb640 10 client.27409 handle_command_reply: tid=0 2024-03-24T10:47:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a8b985740 1 client.27409 shutdown 2024-03-24T10:47:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a8b985740 1 -- 172.21.15.23:0/1082033726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892b5e0 msgr2=0x5614e8938960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a8b985740 1 --2- 172.21.15.23:0/1082033726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892b5e0 0x5614e8938960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2a7402c170 tx=0x7f2a74004400 comp rx=0 tx=0).stop 2024-03-24T10:47:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a8b985740 1 -- 172.21.15.23:0/1082033726 shutdown_connections 2024-03-24T10:47:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a8b985740 1 --2- 172.21.15.23:0/1082033726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614e8934cd0 0x5614e89a9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a8b985740 1 --2- 172.21.15.23:0/1082033726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614e892bf00 0x5614e8938ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a8b985740 1 --2- 172.21.15.23:0/1082033726 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614e89abac0 0x5614e89adeb0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a8b985740 1 --2- 172.21.15.23:0/1082033726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614e892b5e0 0x5614e8938960 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a69ffb640 0 client.27409 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a8b985740 1 -- 172.21.15.23:0/1082033726 >> 172.21.15.23:0/1082033726 conn(0x5614e8925f00 msgr2=0x5614e8927270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a8b985740 1 -- 172.21.15.23:0/1082033726 shutdown_connections 2024-03-24T10:47:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a8b985740 1 -- 172.21.15.23:0/1082033726 wait complete. 2024-03-24T10:47:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.010+0000 7f2a8b985740 20 client.27409 trim_cache size 0 max 16384 2024-03-24T10:47:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:47:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:47:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:47:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:47:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.011+0000 7f2a88da1640 1 -- 172.21.15.23:0/2756180957 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2a700a22d0 msgr2=0x7f2a700a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.011+0000 7f2a88da1640 1 --2- 172.21.15.23:0/2756180957 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2a700a22d0 0x7f2a700a4790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c005fd0 tx=0x7f2a7c013ae0 comp rx=0 tx=0).stop 2024-03-24T10:47:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.011+0000 7f2a88da1640 1 -- 172.21.15.23:0/2756180957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a84102070 msgr2=0x7f2a8419a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.011+0000 7f2a88da1640 1 --2- 172.21.15.23:0/2756180957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a84102070 0x7f2a8419a420 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00e9f0 tx=0x7f2a6c00eeb0 comp rx=0 tx=0).stop 2024-03-24T10:47:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.011+0000 7f2a88da1640 1 -- 172.21.15.23:0/2756180957 shutdown_connections 2024-03-24T10:47:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.011+0000 7f2a88da1640 1 --2- 172.21.15.23:0/2756180957 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2a700a22d0 0x7f2a700a4790 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.011+0000 7f2a88da1640 1 --2- 172.21.15.23:0/2756180957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a841076c0 0x7f2a8419e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.011+0000 7f2a88da1640 1 --2- 172.21.15.23:0/2756180957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a84102070 0x7f2a8419a420 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.011+0000 7f2a88da1640 1 --2- 172.21.15.23:0/2756180957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a841016b0 0x7f2a84199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.011+0000 7f2a88da1640 1 -- 172.21.15.23:0/2756180957 >> 172.21.15.23:0/2756180957 conn(0x7f2a840fd1a0 msgr2=0x7f2a8410cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.011+0000 7f2a88da1640 1 -- 172.21.15.23:0/2756180957 shutdown_connections 2024-03-24T10:47:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:04.011+0000 7f2a88da1640 1 -- 172.21.15.23:0/2756180957 wait complete. 2024-03-24T10:47:04.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 679347 10.96 MB/sec execute 201 sec latency 16.579 ms 2024-03-24T10:47:04.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-mon[36368]: pgmap v213: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 202 op/s 2024-03-24T10:47:04.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:04 smithi116 ceph-mon[36248]: pgmap v213: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 202 op/s 2024-03-24T10:47:04.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:04 smithi023 ceph-mon[32466]: pgmap v213: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 202 op/s 2024-03-24T10:47:05.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 681667 10.97 MB/sec execute 202 sec latency 17.610 ms 2024-03-24T10:47:05.209 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: *** Caught signal (Segmentation fault) ** 2024-03-24T10:47:05.209 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: in thread 7f8207e26700 thread_name:md_log_replay 2024-03-24T10:47:05.209 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev) 2024-03-24T10:47:05.209 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f82191e7d20] 2024-03-24T10:47:05.209 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 2: /usr/bin/ceph-mds(+0x5487ef) [0x56404181e7ef] 2024-03-24T10:47:05.209 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x564041822233] 2024-03-24T10:47:05.209 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 4: (MDLog::_replay_thread()+0x617) [0x56404187e8f7] 2024-03-24T10:47:05.209 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 5: (MDLog::ReplayThread::entry()+0x11) [0x564041885041] 2024-03-24T10:47:05.209 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f82191dd1ca] 2024-03-24T10:47:05.209 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 7: clone() 2024-03-24T10:47:05.209 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 2024-03-24T10:47:04.853+0000 7f8207e26700 -1 *** Caught signal (Segmentation fault) ** 2024-03-24T10:47:05.210 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: in thread 7f8207e26700 thread_name:md_log_replay 2024-03-24T10:47:05.210 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 2024-03-24T10:47:05.210 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev) 2024-03-24T10:47:05.210 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f82191e7d20] 2024-03-24T10:47:05.210 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 2: /usr/bin/ceph-mds(+0x5487ef) [0x56404181e7ef] 2024-03-24T10:47:05.210 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x564041822233] 2024-03-24T10:47:05.210 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 4: (MDLog::_replay_thread()+0x617) [0x56404187e8f7] 2024-03-24T10:47:05.210 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 5: (MDLog::ReplayThread::entry()+0x11) [0x564041885041] 2024-03-24T10:47:05.210 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f82191dd1ca] 2024-03-24T10:47:05.210 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 7: clone() 2024-03-24T10:47:05.210 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T10:47:05.211 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 2024-03-24T10:47:05.211 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 0> 2024-03-24T10:47:04.853+0000 7f8207e26700 -1 *** Caught signal (Segmentation fault) ** 2024-03-24T10:47:05.211 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: in thread 7f8207e26700 thread_name:md_log_replay 2024-03-24T10:47:05.211 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 2024-03-24T10:47:05.211 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev) 2024-03-24T10:47:05.211 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f82191e7d20] 2024-03-24T10:47:05.211 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 2: /usr/bin/ceph-mds(+0x5487ef) [0x56404181e7ef] 2024-03-24T10:47:05.211 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x564041822233] 2024-03-24T10:47:05.211 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 4: (MDLog::_replay_thread()+0x617) [0x56404187e8f7] 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 5: (MDLog::ReplayThread::entry()+0x11) [0x564041885041] 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f82191dd1ca] 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 7: clone() 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 0> 2024-03-24T10:47:04.853+0000 7f8207e26700 -1 *** Caught signal (Segmentation fault) ** 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: in thread 7f8207e26700 thread_name:md_log_replay 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev) 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f82191e7d20] 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 2: /usr/bin/ceph-mds(+0x5487ef) [0x56404181e7ef] 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x564041822233] 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 4: (MDLog::_replay_thread()+0x617) [0x56404187e8f7] 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 5: (MDLog::ReplayThread::entry()+0x11) [0x564041885041] 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f82191dd1ca] 2024-03-24T10:47:05.212 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 7: clone() 2024-03-24T10:47:05.213 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T10:47:05.213 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71244]: 2024-03-24T10:47:05.213 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:04 smithi117 podman[71450]: 2024-03-24 10:47:04.903340756 +0000 UTC m=+0.023840454 container died e994541783aeff6baed12288e24342bf8fbb92290f562dbe3dd85a2d0100d7c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T10:47:05.767 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:05 smithi117 podman[71450]: 2024-03-24 10:47:05.517393885 +0000 UTC m=+0.637893583 container remove e994541783aeff6baed12288e24342bf8fbb92290f562dbe3dd85a2d0100d7c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T10:47:05.767 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:05 smithi117 systemd[1]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.c.service: Main process exited, code=exited, status=139/n/a 2024-03-24T10:47:05.767 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:05 smithi117 systemd[1]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.c.service: Failed with result 'exit-code'. 2024-03-24T10:47:06.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 682993 10.95 MB/sec execute 203 sec latency 80.166 ms 2024-03-24T10:47:06.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:06 smithi117 ceph-mon[36368]: pgmap v214: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 202 op/s 2024-03-24T10:47:06.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:06 smithi117 ceph-mon[36368]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T10:47:06.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:06 smithi116 ceph-mon[36248]: pgmap v214: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 202 op/s 2024-03-24T10:47:06.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:06 smithi116 ceph-mon[36248]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T10:47:06.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:06 smithi023 ceph-mon[32466]: pgmap v214: 97 pgs: 97 active+clean; 401 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 202 op/s 2024-03-24T10:47:06.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:06 smithi023 ceph-mon[32466]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T10:47:07.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 684880 10.95 MB/sec execute 204 sec latency 50.781 ms 2024-03-24T10:47:08.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 686800 10.95 MB/sec execute 205 sec latency 19.064 ms 2024-03-24T10:47:08.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:08 smithi117 ceph-mon[36368]: pgmap v216: 97 pgs: 97 active+clean; 409 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 249 op/s 2024-03-24T10:47:08.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:08 smithi116 ceph-mon[36248]: pgmap v216: 97 pgs: 97 active+clean; 409 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 249 op/s 2024-03-24T10:47:08.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:08 smithi023 ceph-mon[32466]: pgmap v216: 97 pgs: 97 active+clean; 409 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 249 op/s 2024-03-24T10:47:09.124 INFO:tasks.workunit.client.0.smithi023.stdout: 1 689083 10.96 MB/sec execute 206 sec latency 20.216 ms 2024-03-24T10:47:10.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 690597 10.94 MB/sec execute 207 sec latency 18.367 ms 2024-03-24T10:47:10.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:10 smithi117 ceph-mon[36368]: pgmap v217: 97 pgs: 97 active+clean; 409 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 249 op/s 2024-03-24T10:47:10.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:10 smithi116 ceph-mon[36248]: pgmap v217: 97 pgs: 97 active+clean; 409 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 249 op/s 2024-03-24T10:47:10.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:10 smithi023 ceph-mon[32466]: pgmap v217: 97 pgs: 97 active+clean; 409 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.5 MiB/s wr, 249 op/s 2024-03-24T10:47:11.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 692748 10.95 MB/sec execute 208 sec latency 49.951 ms 2024-03-24T10:47:12.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 694100 10.92 MB/sec execute 209 sec latency 99.872 ms 2024-03-24T10:47:12.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:12 smithi117 ceph-mon[36368]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T10:47:12.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:12 smithi117 ceph-mon[36368]: pgmap v219: 97 pgs: 97 active+clean; 427 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 249 op/s 2024-03-24T10:47:12.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:12 smithi116 ceph-mon[36248]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T10:47:12.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:12 smithi116 ceph-mon[36248]: pgmap v219: 97 pgs: 97 active+clean; 427 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 249 op/s 2024-03-24T10:47:12.667 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:12 smithi023 ceph-mon[32466]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T10:47:12.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:12 smithi023 ceph-mon[32466]: pgmap v219: 97 pgs: 97 active+clean; 427 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 249 op/s 2024-03-24T10:47:13.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 696361 10.94 MB/sec execute 210 sec latency 37.423 ms 2024-03-24T10:47:14.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 697744 10.91 MB/sec execute 211 sec latency 28.558 ms 2024-03-24T10:47:14.166 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:47:14.167 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:47:14.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.282+0000 7fbe6af63640 1 -- 172.21.15.23:0/2790605178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe64068910 msgr2=0x7fbe64068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.282+0000 7fbe6af63640 1 --2- 172.21.15.23:0/2790605178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe64068910 0x7fbe64068d90 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009cd0 tx=0x7fbe5402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.282+0000 7fbe6af63640 1 -- 172.21.15.23:0/2790605178 shutdown_connections 2024-03-24T10:47:14.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.282+0000 7fbe6af63640 1 --2- 172.21.15.23:0/2790605178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe640692d0 0x7fbe6410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.282+0000 7fbe6af63640 1 --2- 172.21.15.23:0/2790605178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe64068910 0x7fbe64068d90 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.282+0000 7fbe6af63640 1 --2- 172.21.15.23:0/2790605178 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe64107760 0x7fbe64107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.282+0000 7fbe6af63640 1 -- 172.21.15.23:0/2790605178 >> 172.21.15.23:0/2790605178 conn(0x7fbe64077710 msgr2=0x7fbe64077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.282+0000 7fbe6af63640 1 -- 172.21.15.23:0/2790605178 shutdown_connections 2024-03-24T10:47:14.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.282+0000 7fbe6af63640 1 -- 172.21.15.23:0/2790605178 wait complete. 2024-03-24T10:47:14.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.283+0000 7fbe6af63640 1 Processor -- start 2024-03-24T10:47:14.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.283+0000 7fbe6af63640 1 -- start start 2024-03-24T10:47:14.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.283+0000 7fbe6af63640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe64068910 0x7fbe64197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.283+0000 7fbe6af63640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe640692d0 0x7fbe64198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.283+0000 7fbe6af63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe64107760 0x7fbe6419c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.283+0000 7fbe6af63640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6419cbf0 con 0x7fbe64068910 2024-03-24T10:47:14.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.283+0000 7fbe6af63640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6419cd60 con 0x7fbe64107760 2024-03-24T10:47:14.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.283+0000 7fbe6af63640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6419cf20 con 0x7fbe640692d0 2024-03-24T10:47:14.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.283+0000 7fbe68cff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe64068910 0x7fbe64197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.283+0000 7fbe68cff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe64068910 0x7fbe64197d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45094/0 (socket says 172.21.15.23:45094) 2024-03-24T10:47:14.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.283+0000 7fbe68cff640 1 -- 172.21.15.23:0/757161495 learned_addr learned my addr 172.21.15.23:0/757161495 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:14.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.283+0000 7fbe63fff640 1 --2- 172.21.15.23:0/757161495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe640692d0 0x7fbe64198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.283+0000 7fbe69500640 1 --2- 172.21.15.23:0/757161495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe64107760 0x7fbe6419c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.284+0000 7fbe68cff640 1 -- 172.21.15.23:0/757161495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe640692d0 msgr2=0x7fbe64198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.284+0000 7fbe68cff640 1 --2- 172.21.15.23:0/757161495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe640692d0 0x7fbe64198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.284+0000 7fbe68cff640 1 -- 172.21.15.23:0/757161495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe64107760 msgr2=0x7fbe6419c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.284+0000 7fbe68cff640 1 --2- 172.21.15.23:0/757161495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe64107760 0x7fbe6419c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.284+0000 7fbe68cff640 1 -- 172.21.15.23:0/757161495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe54009900 con 0x7fbe64068910 2024-03-24T10:47:14.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.284+0000 7fbe68cff640 1 --2- 172.21.15.23:0/757161495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe64068910 0x7fbe64197d30 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00e9f0 tx=0x7fbe5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:14.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.284+0000 7fbe61ffb640 1 -- 172.21.15.23:0/757161495 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c009720 con 0x7fbe64068910 2024-03-24T10:47:14.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.284+0000 7fbe61ffb640 1 -- 172.21.15.23:0/757161495 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe5c00cd80 con 0x7fbe64068910 2024-03-24T10:47:14.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.284+0000 7fbe61ffb640 1 -- 172.21.15.23:0/757161495 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c010690 con 0x7fbe64068910 2024-03-24T10:47:14.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.284+0000 7fbe6af63640 1 -- 172.21.15.23:0/757161495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe6419d200 con 0x7fbe64068910 2024-03-24T10:47:14.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.284+0000 7fbe6af63640 1 -- 172.21.15.23:0/757161495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe641a4a80 con 0x7fbe64068910 2024-03-24T10:47:14.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.286+0000 7fbe61ffb640 1 -- 172.21.15.23:0/757161495 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fbe5c0046c0 con 0x7fbe64068910 2024-03-24T10:47:14.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe6dc17740 1 Processor -- start 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe61ffb640 1 --2- 172.21.15.23:0/757161495 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbe500a2230 0x7fbe500a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe6dc17740 1 -- start start 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe69500640 1 --2- 172.21.15.23:0/757161495 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbe500a2230 0x7fbe500a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe6dc17740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb51d0 0x557b43bb55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe6dc17740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bb5af0 0x557b43bbe190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe61ffb640 1 -- 172.21.15.23:0/757161495 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7fbe5c014070 con 0x7fbe64068910 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe6dc17740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bbe6d0 0x557b43bc0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe6dc17740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557b43ad9f30 con 0x557b43bb51d0 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe6dc17740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557b43b23810 con 0x557b43bb5af0 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe6dc17740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557b43b20d40 con 0x557b43bbe6d0 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe68cff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb51d0 0x557b43bb55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe68cff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb51d0 0x557b43bb55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45098/0 (socket says 172.21.15.23:45098) 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe68cff640 1 -- 172.21.15.23:0/4103428277 learned_addr learned my addr 172.21.15.23:0/4103428277 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe63fff640 1 --2- 172.21.15.23:0/4103428277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bb5af0 0x557b43bbe190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.287+0000 7fbe69500640 1 --2- 172.21.15.23:0/4103428277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bbe6d0 0x557b43bc0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.288+0000 7fbe68cff640 1 -- 172.21.15.23:0/4103428277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bbe6d0 msgr2=0x557b43bc0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.288+0000 7fbe68cff640 1 --2- 172.21.15.23:0/4103428277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bbe6d0 0x557b43bc0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.288+0000 7fbe68cff640 1 -- 172.21.15.23:0/4103428277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bb5af0 msgr2=0x557b43bbe190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.288+0000 7fbe68cff640 1 --2- 172.21.15.23:0/4103428277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bb5af0 0x557b43bbe190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.288+0000 7fbe68cff640 1 -- 172.21.15.23:0/4103428277 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b43addea0 con 0x557b43bb51d0 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.288+0000 7fbe69500640 1 --2- 172.21.15.23:0/4103428277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bbe6d0 0x557b43bc0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.288+0000 7fbe69500640 1 --2- 172.21.15.23:0/757161495 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbe500a2230 0x7fbe500a46f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c011440 tx=0x7fbe4c011840 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.288+0000 7fbe68cff640 1 --2- 172.21.15.23:0/4103428277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb51d0 0x557b43bb55b0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0977d0 tx=0x7fbe5c09b500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628b2651b0d882a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.288+0000 7fbe5affd640 1 -- 172.21.15.23:0/4103428277 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c09d030 con 0x557b43bb51d0 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.288+0000 7fbe5affd640 1 -- 172.21.15.23:0/4103428277 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe5c09ef50 con 0x557b43bb51d0 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.288+0000 7fbe5affd640 1 -- 172.21.15.23:0/4103428277 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c0a3370 con 0x557b43bb51d0 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 -- 172.21.15.23:0/4103428277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb51d0 msgr2=0x557b43bb55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/4103428277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb51d0 0x557b43bb55b0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0977d0 tx=0x7fbe5c09b500 comp rx=0 tx=0).stop 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 -- 172.21.15.23:0/4103428277 shutdown_connections 2024-03-24T10:47:14.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/4103428277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bbe6d0 0x557b43bc0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/4103428277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bb5af0 0x557b43bbe190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/4103428277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb51d0 0x557b43bb55b0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 -- 172.21.15.23:0/4103428277 >> 172.21.15.23:0/4103428277 conn(0x557b43baf900 msgr2=0x557b43bafce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 -- 172.21.15.23:0/4103428277 shutdown_connections 2024-03-24T10:47:14.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 -- 172.21.15.23:0/4103428277 wait complete. 2024-03-24T10:47:14.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 -- start start 2024-03-24T10:47:14.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 Processor -- start 2024-03-24T10:47:14.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bb51d0 0x557b43bbb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43bbbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bbe6d0 0x557b43bbc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557b43b1c520 con 0x557b43bb5af0 2024-03-24T10:47:14.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557b437ef430 con 0x557b43bb51d0 2024-03-24T10:47:14.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.289+0000 7fbe6dc17740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557b437abe00 con 0x557b43bbe6d0 2024-03-24T10:47:14.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe6dc17740 10 client.0 _ll_register_callbacks cb 0x557b43abad90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe63fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43bbbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe63fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43bbbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45108/0 (socket says 172.21.15.23:45108) 2024-03-24T10:47:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe63fff640 1 -- 172.21.15.23:0/2844071004 learned_addr learned my addr 172.21.15.23:0/2844071004 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe68cff640 1 --2- 172.21.15.23:0/2844071004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bb51d0 0x557b43bbb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe69500640 1 --2- 172.21.15.23:0/2844071004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bbe6d0 0x557b43bbc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe63fff640 1 -- 172.21.15.23:0/2844071004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bbe6d0 msgr2=0x557b43bbc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe63fff640 1 --2- 172.21.15.23:0/2844071004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bbe6d0 0x557b43bbc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe63fff640 1 -- 172.21.15.23:0/2844071004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bb51d0 msgr2=0x557b43bbb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe63fff640 1 --2- 172.21.15.23:0/2844071004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bb51d0 0x557b43bbb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe63fff640 1 -- 172.21.15.23:0/2844071004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe641a4a80 con 0x557b43bb5af0 2024-03-24T10:47:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe63fff640 1 --2- 172.21.15.23:0/2844071004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43bbbbd0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fbe54037040 tx=0x7fbe54034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6b898bd27bdb1b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe43fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe43fff640 1 -- 172.21.15.23:0/2844071004 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe540328a0 con 0x557b43bb5af0 2024-03-24T10:47:14.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe43fff640 1 -- 172.21.15.23:0/2844071004 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7fbe54032a00 con 0x557b43bb5af0 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe6dc17740 1 -- 172.21.15.23:0/2844071004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557b4381b7e0 con 0x557b43bb5af0 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe43fff640 1 -- 172.21.15.23:0/2844071004 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbe54032ea0 con 0x557b43bb5af0 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe6dc17740 10 client.17562 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.290+0000 7fbe6dc17740 1 -- 172.21.15.23:0/2844071004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557b43b1aba0 con 0x557b43bb5af0 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.291+0000 7fbe43fff640 1 -- 172.21.15.23:0/2844071004 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557b4381b7e0 con 0x557b43bb5af0 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.291+0000 7fbe43fff640 1 -- 172.21.15.23:0/2844071004 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fbe54004680 con 0x557b43bb5af0 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.291+0000 7fbe6dc17740 10 client.17562 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.291+0000 7fbe6dc17740 10 client.17562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.291+0000 7fbe6dc17740 20 client.17562 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.291+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/2844071004 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557b43bceb80 0x557b43bd0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.291+0000 7fbe6dc17740 4 client.17562 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.291+0000 7fbe6dc17740 1 -- 172.21.15.23:0/2844071004 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b43858720 con 0x557b43bceb80 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.291+0000 7fbe68cff640 1 --2- 172.21.15.23:0/2844071004 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557b43bceb80 0x557b43bd0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.291+0000 7fbe43fff640 10 client.17562 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:14.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.291+0000 7fbe68cff640 1 --2- 172.21.15.23:0/2844071004 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557b43bceb80 0x557b43bd0fa0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bca06f559f5bff89 server_cookie=b6f8c135aec9447c in_seq=0 out_seq=0 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.292+0000 7fbe43fff640 1 -- 172.21.15.23:0/2844071004 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557b43858720 con 0x557b43bceb80 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.292+0000 7fbe43fff640 10 client.17562 handle_command_reply: tid=0 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.292+0000 7fbe6dc17740 1 client.17562 shutdown 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.292+0000 7fbe6dc17740 1 -- 172.21.15.23:0/2844071004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 msgr2=0x557b43bbbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.292+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/2844071004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43bbbbd0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fbe54037040 tx=0x7fbe54034670 comp rx=0 tx=0).stop 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.292+0000 7fbe6dc17740 1 -- 172.21.15.23:0/2844071004 shutdown_connections 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.292+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/2844071004 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557b43bceb80 0x557b43bd0fa0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.293+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/2844071004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bbe6d0 0x557b43bbc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.293+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/2844071004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43bbbbd0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.293+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/2844071004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bb51d0 0x557b43bbb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.293+0000 7fbe43fff640 0 client.17562 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.293+0000 7fbe6dc17740 1 -- 172.21.15.23:0/2844071004 >> 172.21.15.23:0/2844071004 conn(0x557b43baf900 msgr2=0x557b43baeb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.293+0000 7fbe6dc17740 1 -- 172.21.15.23:0/2844071004 shutdown_connections 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.293+0000 7fbe6dc17740 1 -- 172.21.15.23:0/2844071004 wait complete. 2024-03-24T10:47:14.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.293+0000 7fbe6dc17740 20 client.17562 trim_cache size 0 max 16384 2024-03-24T10:47:14.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.310+0000 7fbe6dc17740 1 Processor -- start 2024-03-24T10:47:14.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.310+0000 7fbe6dc17740 1 -- start start 2024-03-24T10:47:14.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe6dc17740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bb51d0 0x557b43bbddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe6dc17740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43baeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe6dc17740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bbe6d0 0x557b43bbbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe6dc17740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557b43aa1430 con 0x557b43bb5af0 2024-03-24T10:47:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe6dc17740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557b43bc0ac0 con 0x557b43bbe6d0 2024-03-24T10:47:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe6dc17740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557b43bb4ba0 con 0x557b43bb51d0 2024-03-24T10:47:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe63fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43baeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe63fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43baeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45110/0 (socket says 172.21.15.23:45110) 2024-03-24T10:47:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe63fff640 1 -- 172.21.15.23:0/3890727101 learned_addr learned my addr 172.21.15.23:0/3890727101 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe68cff640 1 --2- 172.21.15.23:0/3890727101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bb51d0 0x557b43bbddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe69500640 1 --2- 172.21.15.23:0/3890727101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bbe6d0 0x557b43bbbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe63fff640 1 -- 172.21.15.23:0/3890727101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bb51d0 msgr2=0x557b43bbddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe63fff640 1 --2- 172.21.15.23:0/3890727101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bb51d0 0x557b43bbddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe63fff640 1 -- 172.21.15.23:0/3890727101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bbe6d0 msgr2=0x557b43bbbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe63fff640 1 --2- 172.21.15.23:0/3890727101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bbe6d0 0x557b43bbbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe63fff640 1 -- 172.21.15.23:0/3890727101 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b43ab6b20 con 0x557b43bb5af0 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe69500640 1 --2- 172.21.15.23:0/3890727101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bbe6d0 0x557b43bbbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe63fff640 1 --2- 172.21.15.23:0/3890727101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43baeb60 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fbe54002230 tx=0x7fbe54009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d4262906da2dc1f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe5a7fc640 1 -- 172.21.15.23:0/3890727101 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe540346e0 con 0x557b43bb5af0 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe5a7fc640 1 -- 172.21.15.23:0/3890727101 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe54048740 con 0x557b43bb5af0 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.311+0000 7fbe5a7fc640 1 -- 172.21.15.23:0/3890727101 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5403f6a0 con 0x557b43bb5af0 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.312+0000 7fbe6dc17740 1 -- 172.21.15.23:0/3890727101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 msgr2=0x557b43baeb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.312+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/3890727101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43baeb60 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fbe54002230 tx=0x7fbe54009360 comp rx=0 tx=0).stop 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.312+0000 7fbe6dc17740 1 -- 172.21.15.23:0/3890727101 shutdown_connections 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.312+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/3890727101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bbe6d0 0x557b43bbbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.312+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/3890727101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43baeb60 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.312+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/3890727101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bb51d0 0x557b43bbddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.312+0000 7fbe6dc17740 1 -- 172.21.15.23:0/3890727101 >> 172.21.15.23:0/3890727101 conn(0x557b43baf900 msgr2=0x557b43bb0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.312+0000 7fbe6dc17740 1 -- 172.21.15.23:0/3890727101 shutdown_connections 2024-03-24T10:47:14.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.312+0000 7fbe6dc17740 1 -- 172.21.15.23:0/3890727101 wait complete. 2024-03-24T10:47:14.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.312+0000 7fbe6dc17740 1 -- start start 2024-03-24T10:47:14.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.312+0000 7fbe6dc17740 1 Processor -- start 2024-03-24T10:47:14.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe6dc17740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bb51d0 0x557b43bc6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe6dc17740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43bc3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe6dc17740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bbe6d0 0x557b43bc3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe6dc17740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557b43bc6c60 con 0x557b43bb5af0 2024-03-24T10:47:14.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe6dc17740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557b43bc6dd0 con 0x557b43bbe6d0 2024-03-24T10:47:14.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe6dc17740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557b43bc3fe0 con 0x557b43bb51d0 2024-03-24T10:47:14.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe6dc17740 10 client.0 _ll_register_callbacks cb 0x7fbe5c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:14.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe63fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43bc3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe63fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43bc3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45126/0 (socket says 172.21.15.23:45126) 2024-03-24T10:47:14.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe63fff640 1 -- 172.21.15.23:0/981908171 learned_addr learned my addr 172.21.15.23:0/981908171 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:14.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe68cff640 1 --2- 172.21.15.23:0/981908171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bb51d0 0x557b43bc6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe69500640 1 --2- 172.21.15.23:0/981908171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bbe6d0 0x557b43bc3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe63fff640 1 -- 172.21.15.23:0/981908171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bb51d0 msgr2=0x557b43bc6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe63fff640 1 --2- 172.21.15.23:0/981908171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bb51d0 0x557b43bc6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe63fff640 1 -- 172.21.15.23:0/981908171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bbe6d0 msgr2=0x557b43bc3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe63fff640 1 --2- 172.21.15.23:0/981908171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bbe6d0 0x557b43bc3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe63fff640 1 -- 172.21.15.23:0/981908171 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b43b1aba0 con 0x557b43bb5af0 2024-03-24T10:47:14.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe63fff640 1 --2- 172.21.15.23:0/981908171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43bc3560 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fbe5402c630 tx=0x7fbe54005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b5fd15c71e76d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:14.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe43fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:14.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe43fff640 1 -- 172.21.15.23:0/981908171 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe540488b0 con 0x557b43bb5af0 2024-03-24T10:47:14.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe43fff640 1 -- 172.21.15.23:0/981908171 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7fbe5403f440 con 0x557b43bb5af0 2024-03-24T10:47:14.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.313+0000 7fbe6dc17740 1 -- 172.21.15.23:0/981908171 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe5c09ef50 con 0x557b43bb5af0 2024-03-24T10:47:14.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.314+0000 7fbe43fff640 1 -- 172.21.15.23:0/981908171 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbe54004b40 con 0x557b43bb5af0 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.314+0000 7fbe6dc17740 10 client.17568 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.314+0000 7fbe6dc17740 1 -- 172.21.15.23:0/981908171 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557b43adc700 con 0x557b43bb5af0 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.314+0000 7fbe43fff640 1 -- 172.21.15.23:0/981908171 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe54063020 con 0x557b43bb5af0 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.314+0000 7fbe43fff640 1 -- 172.21.15.23:0/981908171 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fbe540051b0 con 0x557b43bb5af0 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.314+0000 7fbe6dc17740 10 client.17568 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.314+0000 7fbe6dc17740 10 client.17568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.314+0000 7fbe6dc17740 20 client.17568 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.314+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/981908171 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557b43c35d10 0x557b43c38100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.314+0000 7fbe6dc17740 4 client.17568 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.314+0000 7fbe6dc17740 1 -- 172.21.15.23:0/981908171 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557b438a7890 con 0x557b43c35d10 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.314+0000 7fbe68cff640 1 --2- 172.21.15.23:0/981908171 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557b43c35d10 0x557b43c38100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.314+0000 7fbe43fff640 10 client.17568 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.314+0000 7fbe68cff640 1 --2- 172.21.15.23:0/981908171 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557b43c35d10 0x557b43c38100 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4013d554abe7f54 server_cookie=a565e1c9f200593d in_seq=0 out_seq=0 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe43fff640 1 -- 172.21.15.23:0/981908171 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557b438a7890 con 0x557b43c35d10 2024-03-24T10:47:14.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe43fff640 10 client.17568 handle_command_reply: tid=0 2024-03-24T10:47:14.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe6dc17740 1 client.17568 shutdown 2024-03-24T10:47:14.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe6dc17740 1 -- 172.21.15.23:0/981908171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 msgr2=0x557b43bc3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/981908171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43bc3560 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fbe5402c630 tx=0x7fbe54005fa0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe6dc17740 1 -- 172.21.15.23:0/981908171 shutdown_connections 2024-03-24T10:47:14.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/981908171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b43bbe6d0 0x557b43bc3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/981908171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b43bb5af0 0x557b43bc3560 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/981908171 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557b43c35d10 0x557b43c38100 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe6dc17740 1 --2- 172.21.15.23:0/981908171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b43bb51d0 0x557b43bc6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe43fff640 0 client.17568 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:14.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe6dc17740 1 -- 172.21.15.23:0/981908171 >> 172.21.15.23:0/981908171 conn(0x557b43baf900 msgr2=0x557b43bae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe6dc17740 1 -- 172.21.15.23:0/981908171 shutdown_connections 2024-03-24T10:47:14.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe6dc17740 1 -- 172.21.15.23:0/981908171 wait complete. 2024-03-24T10:47:14.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.315+0000 7fbe6dc17740 20 client.17568 trim_cache size 0 max 16384 2024-03-24T10:47:14.322 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:47:14.322 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:47:14.322 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:47:14.322 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:47:14.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.316+0000 7fbe6af63640 1 -- 172.21.15.23:0/757161495 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbe500a2230 msgr2=0x7fbe500a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.316+0000 7fbe6af63640 1 --2- 172.21.15.23:0/757161495 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbe500a2230 0x7fbe500a46f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c011440 tx=0x7fbe4c011840 comp rx=0 tx=0).stop 2024-03-24T10:47:14.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.316+0000 7fbe6af63640 1 -- 172.21.15.23:0/757161495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe64068910 msgr2=0x7fbe64197d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.316+0000 7fbe6af63640 1 --2- 172.21.15.23:0/757161495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe64068910 0x7fbe64197d30 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00e9f0 tx=0x7fbe5c00edf0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.316+0000 7fbe6af63640 1 -- 172.21.15.23:0/757161495 shutdown_connections 2024-03-24T10:47:14.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.316+0000 7fbe6af63640 1 --2- 172.21.15.23:0/757161495 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbe500a2230 0x7fbe500a46f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.316+0000 7fbe6af63640 1 --2- 172.21.15.23:0/757161495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe64107760 0x7fbe6419c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.316+0000 7fbe6af63640 1 --2- 172.21.15.23:0/757161495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe640692d0 0x7fbe64198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.316+0000 7fbe6af63640 1 --2- 172.21.15.23:0/757161495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe64068910 0x7fbe64197d30 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.316+0000 7fbe6af63640 1 -- 172.21.15.23:0/757161495 >> 172.21.15.23:0/757161495 conn(0x7fbe64077710 msgr2=0x7fbe640fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.316+0000 7fbe6af63640 1 -- 172.21.15.23:0/757161495 shutdown_connections 2024-03-24T10:47:14.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.316+0000 7fbe6af63640 1 -- 172.21.15.23:0/757161495 wait complete. 2024-03-24T10:47:14.326 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:47:14.327 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:14.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.433+0000 7f0b07a6f640 1 -- 172.21.15.23:0/3759978769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b001005c0 msgr2=0x7f0b00100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.433+0000 7f0b07a6f640 1 --2- 172.21.15.23:0/3759978769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b001005c0 0x7f0b00100a40 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0af0009930 tx=0x7f0af002bc30 comp rx=0 tx=0).stop 2024-03-24T10:47:14.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.433+0000 7f0b07a6f640 1 -- 172.21.15.23:0/3759978769 shutdown_connections 2024-03-24T10:47:14.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.433+0000 7f0b07a6f640 1 --2- 172.21.15.23:0/3759978769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b00100f80 0x7f0b0010d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.433+0000 7f0b07a6f640 1 --2- 172.21.15.23:0/3759978769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b001005c0 0x7f0b00100a40 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.433+0000 7f0b07a6f640 1 --2- 172.21.15.23:0/3759978769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b00106660 0x7f0b00106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.433+0000 7f0b07a6f640 1 -- 172.21.15.23:0/3759978769 >> 172.21.15.23:0/3759978769 conn(0x7f0b00076990 msgr2=0x7f0b00076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.433+0000 7f0b07a6f640 1 -- 172.21.15.23:0/3759978769 shutdown_connections 2024-03-24T10:47:14.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.434+0000 7f0b07a6f640 1 -- 172.21.15.23:0/3759978769 wait complete. 2024-03-24T10:47:14.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.434+0000 7f0b07a6f640 1 Processor -- start 2024-03-24T10:47:14.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.434+0000 7f0b07a6f640 1 -- start start 2024-03-24T10:47:14.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.434+0000 7f0b07a6f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b001005c0 0x7f0b0010de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.434+0000 7f0b07a6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b00100f80 0x7f0b0010e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.434+0000 7f0b07a6f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b00106660 0x7f0b0010e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.434+0000 7f0b07a6f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0b001a60b0 con 0x7f0b00106660 2024-03-24T10:47:14.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.434+0000 7f0b07a6f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0b001a6220 con 0x7f0b00100f80 2024-03-24T10:47:14.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.434+0000 7f0b07a6f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0b001a63e0 con 0x7f0b001005c0 2024-03-24T10:47:14.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.434+0000 7f0b0600c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b00106660 0x7f0b0010e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.434+0000 7f0b0600c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b00106660 0x7f0b0010e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45136/0 (socket says 172.21.15.23:45136) 2024-03-24T10:47:14.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.434+0000 7f0b0600c640 1 -- 172.21.15.23:0/278025147 learned_addr learned my addr 172.21.15.23:0/278025147 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:14.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.434+0000 7f0b0580b640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b001005c0 0x7f0b0010de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.434+0000 7f0b0500a640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b00100f80 0x7f0b0010e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.435+0000 7f0b0600c640 1 -- 172.21.15.23:0/278025147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b001005c0 msgr2=0x7f0b0010de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.435+0000 7f0b0600c640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b001005c0 0x7f0b0010de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.435+0000 7f0b0600c640 1 -- 172.21.15.23:0/278025147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b00100f80 msgr2=0x7f0b0010e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.435+0000 7f0b0600c640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b00100f80 0x7f0b0010e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.435+0000 7f0b0600c640 1 -- 172.21.15.23:0/278025147 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af0009560 con 0x7f0b00106660 2024-03-24T10:47:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.435+0000 7f0b0580b640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b001005c0 0x7f0b0010de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.435+0000 7f0b0600c640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b00106660 0x7f0b0010e8e0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0af800eb10 tx=0x7f0af800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.435+0000 7f0af6ffd640 1 -- 172.21.15.23:0/278025147 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af8004b20 con 0x7f0b00106660 2024-03-24T10:47:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.435+0000 7f0af6ffd640 1 -- 172.21.15.23:0/278025147 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0af8004c80 con 0x7f0b00106660 2024-03-24T10:47:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.435+0000 7f0b07a6f640 1 -- 172.21.15.23:0/278025147 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b001a66c0 con 0x7f0b00106660 2024-03-24T10:47:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.435+0000 7f0af6ffd640 1 -- 172.21.15.23:0/278025147 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af8010650 con 0x7f0b00106660 2024-03-24T10:47:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.435+0000 7f0b07a6f640 1 -- 172.21.15.23:0/278025147 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b001a6ba0 con 0x7f0b00106660 2024-03-24T10:47:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.436+0000 7f0b07a6f640 1 -- 172.21.15.23:0/278025147 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b00101d00 con 0x7f0b00106660 2024-03-24T10:47:14.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.437+0000 7f0af6ffd640 1 -- 172.21.15.23:0/278025147 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f0af8004df0 con 0x7f0b00106660 2024-03-24T10:47:14.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.438+0000 7f0af6ffd640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0aec0a2230 0x7f0aec0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.438+0000 7f0b0580b640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0aec0a2230 0x7f0aec0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.438+0000 7f0af6ffd640 1 -- 172.21.15.23:0/278025147 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f0af8014070 con 0x7f0b00106660 2024-03-24T10:47:14.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.438+0000 7f0b0580b640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0aec0a2230 0x7f0aec0a46f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0b00101700 tx=0x7f0afc005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:14.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.442+0000 7f0af6ffd640 1 -- 172.21.15.23:0/278025147 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0af8097490 con 0x7f0b00106660 2024-03-24T10:47:14.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.673+0000 7f0b07a6f640 1 -- 172.21.15.23:0/278025147 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b00068950 con 0x7f0b00106660 2024-03-24T10:47:14.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.673+0000 7f0af6ffd640 1 -- 172.21.15.23:0/278025147 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11869 (secure 0 0 0) 0x7f0af809ab50 con 0x7f0b00106660 2024-03-24T10:47:14.674 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:47:14.674 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:45:17.709481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26933":{"gid":26933,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/2955742470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":2955742470},{"type":"v1","addr":"172.21.15.117:6837","nonce":2955742470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24539,24503,24505]},"id":1}]} 2024-03-24T10:47:14.674 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 29 2024-03-24T10:47:14.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.675+0000 7f0b07a6f640 1 -- 172.21.15.23:0/278025147 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0aec0a2230 msgr2=0x7f0aec0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.676+0000 7f0b07a6f640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0aec0a2230 0x7f0aec0a46f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0b00101700 tx=0x7f0afc005f90 comp rx=0 tx=0).stop 2024-03-24T10:47:14.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.676+0000 7f0b07a6f640 1 -- 172.21.15.23:0/278025147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b00106660 msgr2=0x7f0b0010e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.676+0000 7f0b07a6f640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b00106660 0x7f0b0010e8e0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0af800eb10 tx=0x7f0af800c2d0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.676+0000 7f0b07a6f640 1 -- 172.21.15.23:0/278025147 shutdown_connections 2024-03-24T10:47:14.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.676+0000 7f0b07a6f640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0aec0a2230 0x7f0aec0a46f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.676+0000 7f0b07a6f640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b00106660 0x7f0b0010e8e0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.676+0000 7f0b07a6f640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b00100f80 0x7f0b0010e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.676+0000 7f0b07a6f640 1 --2- 172.21.15.23:0/278025147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b001005c0 0x7f0b0010de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.676+0000 7f0b07a6f640 1 -- 172.21.15.23:0/278025147 >> 172.21.15.23:0/278025147 conn(0x7f0b00076990 msgr2=0x7f0b000fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.676+0000 7f0b07a6f640 1 -- 172.21.15.23:0/278025147 shutdown_connections 2024-03-24T10:47:14.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.676+0000 7f0b07a6f640 1 -- 172.21.15.23:0/278025147 wait complete. 2024-03-24T10:47:14.687 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:47:14.711 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:14 smithi023 ceph-mon[32466]: pgmap v220: 97 pgs: 97 active+clean; 427 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 249 op/s 2024-03-24T10:47:14.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:14 smithi117 ceph-mon[36368]: pgmap v220: 97 pgs: 97 active+clean; 427 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 249 op/s 2024-03-24T10:47:14.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:14 smithi116 ceph-mon[36248]: pgmap v220: 97 pgs: 97 active+clean; 427 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 249 op/s 2024-03-24T10:47:14.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.800+0000 7f9753102640 1 -- 172.21.15.23:0/2976324044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f974c106660 msgr2=0x7f974c106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.800+0000 7f9753102640 1 --2- 172.21.15.23:0/2976324044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f974c106660 0x7f974c106a40 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f9740009900 tx=0x7f974002bc30 comp rx=0 tx=0).stop 2024-03-24T10:47:14.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.800+0000 7f9753102640 1 -- 172.21.15.23:0/2976324044 shutdown_connections 2024-03-24T10:47:14.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.800+0000 7f9753102640 1 --2- 172.21.15.23:0/2976324044 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f974c100f60 0x7f974c10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.800+0000 7f9753102640 1 --2- 172.21.15.23:0/2976324044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f974c1005a0 0x7f974c100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.800+0000 7f9753102640 1 --2- 172.21.15.23:0/2976324044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f974c106660 0x7f974c106a40 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.800+0000 7f9753102640 1 -- 172.21.15.23:0/2976324044 >> 172.21.15.23:0/2976324044 conn(0x7f974c077710 msgr2=0x7f974c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.800+0000 7f9753102640 1 -- 172.21.15.23:0/2976324044 shutdown_connections 2024-03-24T10:47:14.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.800+0000 7f9753102640 1 -- 172.21.15.23:0/2976324044 wait complete. 2024-03-24T10:47:14.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f9753102640 1 Processor -- start 2024-03-24T10:47:14.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f9753102640 1 -- start start 2024-03-24T10:47:14.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f9753102640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f974c1005a0 0x7f974c19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f9753102640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f974c100f60 0x7f974c19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f9753102640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f974c106660 0x7f974c197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f9753102640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f974c198180 con 0x7f974c106660 2024-03-24T10:47:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f9753102640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f974c1982f0 con 0x7f974c100f60 2024-03-24T10:47:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f9753102640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f974c1984b0 con 0x7f974c1005a0 2024-03-24T10:47:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f975169f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f974c106660 0x7f974c197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f975169f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f974c106660 0x7f974c197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45162/0 (socket says 172.21.15.23:45162) 2024-03-24T10:47:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f975169f640 1 -- 172.21.15.23:0/3628844324 learned_addr learned my addr 172.21.15.23:0/3628844324 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f974bfff640 1 --2- 172.21.15.23:0/3628844324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f974c100f60 0x7f974c19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f9750e9e640 1 --2- 172.21.15.23:0/3628844324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f974c1005a0 0x7f974c19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f975169f640 1 -- 172.21.15.23:0/3628844324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f974c1005a0 msgr2=0x7f974c19d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f975169f640 1 --2- 172.21.15.23:0/3628844324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f974c1005a0 0x7f974c19d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f975169f640 1 -- 172.21.15.23:0/3628844324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f974c100f60 msgr2=0x7f974c19de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f975169f640 1 --2- 172.21.15.23:0/3628844324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f974c100f60 0x7f974c19de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f975169f640 1 -- 172.21.15.23:0/3628844324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9740009560 con 0x7f974c106660 2024-03-24T10:47:14.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f975169f640 1 --2- 172.21.15.23:0/3628844324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f974c106660 0x7f974c197b30 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f974400eb10 tx=0x7f974400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:14.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.801+0000 7f9749ffb640 1 -- 172.21.15.23:0/3628844324 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9744004b20 con 0x7f974c106660 2024-03-24T10:47:14.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.802+0000 7f9749ffb640 1 -- 172.21.15.23:0/3628844324 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9744004c80 con 0x7f974c106660 2024-03-24T10:47:14.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.802+0000 7f9753102640 1 -- 172.21.15.23:0/3628844324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f974c198790 con 0x7f974c106660 2024-03-24T10:47:14.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.802+0000 7f9749ffb640 1 -- 172.21.15.23:0/3628844324 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9744010650 con 0x7f974c106660 2024-03-24T10:47:14.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.802+0000 7f9753102640 1 -- 172.21.15.23:0/3628844324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f974c1a49d0 con 0x7f974c106660 2024-03-24T10:47:14.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.804+0000 7f9749ffb640 1 -- 172.21.15.23:0/3628844324 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f9744005020 con 0x7f974c106660 2024-03-24T10:47:14.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.804+0000 7f9755dc4740 1 Processor -- start 2024-03-24T10:47:14.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.804+0000 7f9755dc4740 1 -- start start 2024-03-24T10:47:14.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.804+0000 7f9755dc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba03f0 0x55d4acba07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9755dc4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba0d10 0x55d4acba93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9755dc4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba98f0 0x55d4acbabce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9755dc4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d4acb0e840 con 0x55d4acba0d10 2024-03-24T10:47:14.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9755dc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d4acb0bd70 con 0x55d4acba03f0 2024-03-24T10:47:14.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9755dc4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d4acb07550 con 0x55d4acba98f0 2024-03-24T10:47:14.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9749ffb640 1 --2- 172.21.15.23:0/3628844324 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f973c0a2230 0x7f973c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9750e9e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba0d10 0x55d4acba93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9750e9e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba0d10 0x55d4acba93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45172/0 (socket says 172.21.15.23:45172) 2024-03-24T10:47:14.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9750e9e640 1 -- 172.21.15.23:0/3657852150 learned_addr learned my addr 172.21.15.23:0/3657852150 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:14.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f975169f640 1 --2- 172.21.15.23:0/3628844324 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f973c0a2230 0x7f973c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f974bfff640 1 --2- 172.21.15.23:0/3657852150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba03f0 0x55d4acba07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f974bfff640 1 --2- 172.21.15.23:0/3657852150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba98f0 0x55d4acbabce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9749ffb640 1 -- 172.21.15.23:0/3628844324 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f9744014070 con 0x7f974c106660 2024-03-24T10:47:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9750e9e640 1 -- 172.21.15.23:0/3657852150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba98f0 msgr2=0x55d4acbabce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9750e9e640 1 --2- 172.21.15.23:0/3657852150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba98f0 0x55d4acbabce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9750e9e640 1 -- 172.21.15.23:0/3657852150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba03f0 msgr2=0x55d4acba07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9750e9e640 1 --2- 172.21.15.23:0/3657852150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba03f0 0x55d4acba07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9750e9e640 1 -- 172.21.15.23:0/3657852150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4aca9cde0 con 0x55d4acba0d10 2024-03-24T10:47:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f974bfff640 1 --2- 172.21.15.23:0/3657852150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba03f0 0x55d4acba07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f975169f640 1 --2- 172.21.15.23:0/3628844324 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f973c0a2230 0x7f973c0a46f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f974409af60 tx=0x7f974406a170 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9750e9e640 1 --2- 172.21.15.23:0/3657852150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba0d10 0x55d4acba93b0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f9740009900 tx=0x7f974002c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548bd9c380dd5a55 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9736ffd640 1 -- 172.21.15.23:0/3657852150 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974002c910 con 0x55d4acba0d10 2024-03-24T10:47:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9736ffd640 1 -- 172.21.15.23:0/3657852150 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f974002ca70 con 0x55d4acba0d10 2024-03-24T10:47:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.805+0000 7f9736ffd640 1 -- 172.21.15.23:0/3657852150 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97400050e0 con 0x55d4acba0d10 2024-03-24T10:47:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.806+0000 7f9755dc4740 1 -- 172.21.15.23:0/3657852150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba0d10 msgr2=0x55d4acba93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.806+0000 7f9755dc4740 1 --2- 172.21.15.23:0/3657852150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba0d10 0x55d4acba93b0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f9740009900 tx=0x7f974002c6a0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.806+0000 7f9755dc4740 1 -- 172.21.15.23:0/3657852150 shutdown_connections 2024-03-24T10:47:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.806+0000 7f9755dc4740 1 --2- 172.21.15.23:0/3657852150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba98f0 0x55d4acbabce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.806+0000 7f9755dc4740 1 --2- 172.21.15.23:0/3657852150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba0d10 0x55d4acba93b0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.806+0000 7f9755dc4740 1 --2- 172.21.15.23:0/3657852150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba03f0 0x55d4acba07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.806+0000 7f9755dc4740 1 -- 172.21.15.23:0/3657852150 >> 172.21.15.23:0/3657852150 conn(0x55d4acb9ab20 msgr2=0x55d4acb9af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.806+0000 7f9755dc4740 1 -- 172.21.15.23:0/3657852150 shutdown_connections 2024-03-24T10:47:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.806+0000 7f9755dc4740 1 -- 172.21.15.23:0/3657852150 wait complete. 2024-03-24T10:47:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.807+0000 7f9755dc4740 1 -- start start 2024-03-24T10:47:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.807+0000 7f9755dc4740 1 Processor -- start 2024-03-24T10:47:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.807+0000 7f9755dc4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba03f0 0x55d4acba6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.807+0000 7f9755dc4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba0d10 0x55d4acba6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.807+0000 7f9755dc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba98f0 0x55d4acba73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.807+0000 7f9755dc4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d4ac7da430 con 0x55d4acba03f0 2024-03-24T10:47:14.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.807+0000 7f9755dc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d4ac796e00 con 0x55d4acba98f0 2024-03-24T10:47:14.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.807+0000 7f9755dc4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d4aca8c590 con 0x55d4acba0d10 2024-03-24T10:47:14.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.807+0000 7f9755dc4740 10 client.0 _ll_register_callbacks cb 0x55d4acafbfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:14.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.807+0000 7f974bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba03f0 0x55d4acba6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.807+0000 7f974bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba03f0 0x55d4acba6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45182/0 (socket says 172.21.15.23:45182) 2024-03-24T10:47:14.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.807+0000 7f974bfff640 1 -- 172.21.15.23:0/2337091332 learned_addr learned my addr 172.21.15.23:0/2337091332 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:14.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.807+0000 7f9750e9e640 1 --2- 172.21.15.23:0/2337091332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba0d10 0x55d4acba6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.807+0000 7f974bfff640 1 --2- 172.21.15.23:0/2337091332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba98f0 0x55d4acba73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f974bfff640 1 -- 172.21.15.23:0/2337091332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba0d10 msgr2=0x55d4acba6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f974bfff640 1 --2- 172.21.15.23:0/2337091332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba0d10 0x55d4acba6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f974bfff640 1 -- 172.21.15.23:0/2337091332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba98f0 msgr2=0x55d4acba73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f974bfff640 1 --2- 172.21.15.23:0/2337091332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba98f0 0x55d4acba73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f974bfff640 1 -- 172.21.15.23:0/2337091332 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f974c10dd20 con 0x55d4acba03f0 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f9750e9e640 1 --2- 172.21.15.23:0/2337091332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba0d10 0x55d4acba6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f974bfff640 1 --2- 172.21.15.23:0/2337091332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba03f0 0x55d4acba6950 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f97380037c0 tx=0x7f9738004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310ccbf1ba103deb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f972bfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f972bfff640 1 -- 172.21.15.23:0/2337091332 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9738005020 con 0x55d4acba03f0 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f9755dc4740 1 -- 172.21.15.23:0/2337091332 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d4ac8067e0 con 0x55d4acba03f0 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f972bfff640 1 -- 172.21.15.23:0/2337091332 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f9738005180 con 0x55d4acba03f0 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f972bfff640 1 -- 172.21.15.23:0/2337091332 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9738005340 con 0x55d4acba03f0 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f9755dc4740 10 client.17586 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f9755dc4740 1 -- 172.21.15.23:0/2337091332 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d4acb05bd0 con 0x55d4acba03f0 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f972bfff640 1 -- 172.21.15.23:0/2337091332 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d4ac8067e0 con 0x55d4acba03f0 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.808+0000 7f972bfff640 1 -- 172.21.15.23:0/2337091332 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f973800f7c0 con 0x55d4acba03f0 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.809+0000 7f9755dc4740 10 client.17586 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.809+0000 7f9755dc4740 10 client.17586 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:47:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.809+0000 7f9755dc4740 20 client.17586 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.809+0000 7f9755dc4740 1 --2- 172.21.15.23:0/2337091332 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55d4acbb99d0 0x55d4acbbbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.809+0000 7f9755dc4740 4 client.17586 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.809+0000 7f9755dc4740 1 -- 172.21.15.23:0/2337091332 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4ac843720 con 0x55d4acbb99d0 2024-03-24T10:47:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.809+0000 7f9750e9e640 1 --2- 172.21.15.23:0/2337091332 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55d4acbb99d0 0x55d4acbbbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.809+0000 7f972bfff640 10 client.17586 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:47:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.809+0000 7f9750e9e640 1 --2- 172.21.15.23:0/2337091332 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55d4acbb99d0 0x55d4acbbbdf0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6d7ac28ff70da62b server_cookie=c5fd100aba5f5e1c in_seq=0 out_seq=0 2024-03-24T10:47:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f972bfff640 1 -- 172.21.15.23:0/2337091332 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d4ac843720 con 0x55d4acbb99d0 2024-03-24T10:47:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f972bfff640 10 client.17586 handle_command_reply: tid=0 2024-03-24T10:47:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f9755dc4740 1 client.17586 shutdown 2024-03-24T10:47:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f9755dc4740 1 -- 172.21.15.23:0/2337091332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba03f0 msgr2=0x55d4acba6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f9755dc4740 1 --2- 172.21.15.23:0/2337091332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba03f0 0x55d4acba6950 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f97380037c0 tx=0x7f9738004730 comp rx=0 tx=0).stop 2024-03-24T10:47:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f9755dc4740 1 -- 172.21.15.23:0/2337091332 shutdown_connections 2024-03-24T10:47:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f9755dc4740 1 --2- 172.21.15.23:0/2337091332 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55d4acbb99d0 0x55d4acbbbdf0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f9755dc4740 1 --2- 172.21.15.23:0/2337091332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba98f0 0x55d4acba73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f9755dc4740 1 --2- 172.21.15.23:0/2337091332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba0d10 0x55d4acba6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f972bfff640 0 client.17586 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:47:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f9755dc4740 1 --2- 172.21.15.23:0/2337091332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba03f0 0x55d4acba6950 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f9755dc4740 1 -- 172.21.15.23:0/2337091332 >> 172.21.15.23:0/2337091332 conn(0x55d4acb9ab20 msgr2=0x55d4acb9c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f9755dc4740 1 -- 172.21.15.23:0/2337091332 shutdown_connections 2024-03-24T10:47:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f9755dc4740 1 -- 172.21.15.23:0/2337091332 wait complete. 2024-03-24T10:47:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.810+0000 7f9755dc4740 20 client.17586 trim_cache size 0 max 16384 2024-03-24T10:47:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.824+0000 7f9755dc4740 1 Processor -- start 2024-03-24T10:47:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.824+0000 7f9755dc4740 1 -- start start 2024-03-24T10:47:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.824+0000 7f9755dc4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba03f0 0x55d4acba9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.824+0000 7f9755dc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba0d10 0x55d4acbb5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.824+0000 7f9755dc4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba98f0 0x55d4acbc16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.824+0000 7f9755dc4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d4acbb5660 con 0x55d4acba03f0 2024-03-24T10:47:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.824+0000 7f9755dc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d4acb963e0 con 0x55d4acba0d10 2024-03-24T10:47:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.824+0000 7f9755dc4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d4acba0050 con 0x55d4acba98f0 2024-03-24T10:47:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.824+0000 7f974bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba03f0 0x55d4acba9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.824+0000 7f974bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba03f0 0x55d4acba9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45194/0 (socket says 172.21.15.23:45194) 2024-03-24T10:47:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.824+0000 7f974bfff640 1 -- 172.21.15.23:0/3014785136 learned_addr learned my addr 172.21.15.23:0/3014785136 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.824+0000 7f9750e9e640 1 --2- 172.21.15.23:0/3014785136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba0d10 0x55d4acbb5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.825+0000 7f974bfff640 1 --2- 172.21.15.23:0/3014785136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba98f0 0x55d4acbc16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.825+0000 7f974bfff640 1 -- 172.21.15.23:0/3014785136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba98f0 msgr2=0x55d4acbc16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.825+0000 7f974bfff640 1 --2- 172.21.15.23:0/3014785136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba98f0 0x55d4acbc16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.825+0000 7f974bfff640 1 -- 172.21.15.23:0/3014785136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba0d10 msgr2=0x55d4acbb5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.825+0000 7f974bfff640 1 --2- 172.21.15.23:0/3014785136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba0d10 0x55d4acbb5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.825+0000 7f974bfff640 1 -- 172.21.15.23:0/3014785136 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4acaa1c80 con 0x55d4acba03f0 2024-03-24T10:47:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.825+0000 7f974bfff640 1 --2- 172.21.15.23:0/3014785136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba03f0 0x55d4acba9080 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f97380059f0 tx=0x7f9738004310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e616bed4263d636e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.825+0000 7f97367fc640 1 -- 172.21.15.23:0/3014785136 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9738004560 con 0x55d4acba03f0 2024-03-24T10:47:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.825+0000 7f97367fc640 1 -- 172.21.15.23:0/3014785136 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97380035f0 con 0x55d4acba03f0 2024-03-24T10:47:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.825+0000 7f97367fc640 1 -- 172.21.15.23:0/3014785136 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9738005030 con 0x55d4acba03f0 2024-03-24T10:47:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.826+0000 7f9755dc4740 1 -- 172.21.15.23:0/3014785136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba03f0 msgr2=0x55d4acba9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.826+0000 7f9755dc4740 1 --2- 172.21.15.23:0/3014785136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba03f0 0x55d4acba9080 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f97380059f0 tx=0x7f9738004310 comp rx=0 tx=0).stop 2024-03-24T10:47:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.826+0000 7f9755dc4740 1 -- 172.21.15.23:0/3014785136 shutdown_connections 2024-03-24T10:47:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.826+0000 7f9755dc4740 1 --2- 172.21.15.23:0/3014785136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba98f0 0x55d4acbc16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.826+0000 7f9755dc4740 1 --2- 172.21.15.23:0/3014785136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba0d10 0x55d4acbb5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.826+0000 7f9755dc4740 1 --2- 172.21.15.23:0/3014785136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba03f0 0x55d4acba9080 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.826+0000 7f9755dc4740 1 -- 172.21.15.23:0/3014785136 >> 172.21.15.23:0/3014785136 conn(0x55d4acb9ab20 msgr2=0x55d4acb99430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.826+0000 7f9755dc4740 1 -- 172.21.15.23:0/3014785136 shutdown_connections 2024-03-24T10:47:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.826+0000 7f9755dc4740 1 -- 172.21.15.23:0/3014785136 wait complete. 2024-03-24T10:47:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.826+0000 7f9755dc4740 1 -- start start 2024-03-24T10:47:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.826+0000 7f9755dc4740 1 Processor -- start 2024-03-24T10:47:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9755dc4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba03f0 0x55d4acbad050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9755dc4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba0d10 0x55d4acbad590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9755dc4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba98f0 0x55d4acbadad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9755dc4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d4acc1ecf0 con 0x55d4acba98f0 2024-03-24T10:47:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9755dc4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d4acc1ee30 con 0x55d4acba03f0 2024-03-24T10:47:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9755dc4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d4acc1ef70 con 0x55d4acba0d10 2024-03-24T10:47:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9755dc4740 10 client.0 _ll_register_callbacks cb 0x7f9738009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f974bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba98f0 0x55d4acbadad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9750e9e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba0d10 0x55d4acbad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f974bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba98f0 0x55d4acbadad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45200/0 (socket says 172.21.15.23:45200) 2024-03-24T10:47:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f974bfff640 1 -- 172.21.15.23:0/2128659865 learned_addr learned my addr 172.21.15.23:0/2128659865 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f974bfff640 1 --2- 172.21.15.23:0/2128659865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba03f0 0x55d4acbad050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9750e9e640 1 -- 172.21.15.23:0/2128659865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba03f0 msgr2=0x55d4acbad050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9750e9e640 1 --2- 172.21.15.23:0/2128659865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba03f0 0x55d4acbad050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9750e9e640 1 -- 172.21.15.23:0/2128659865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba98f0 msgr2=0x55d4acbadad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9750e9e640 1 --2- 172.21.15.23:0/2128659865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba98f0 0x55d4acbadad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9750e9e640 1 -- 172.21.15.23:0/2128659865 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4acb05bd0 con 0x55d4acba0d10 2024-03-24T10:47:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f974bfff640 1 --2- 172.21.15.23:0/2128659865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba98f0 0x55d4acbadad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9750e9e640 1 --2- 172.21.15.23:0/2128659865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba0d10 0x55d4acbad590 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f974002c170 tx=0x7f974002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4027948ccca2ff0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f972bfff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:47:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f972bfff640 1 -- 172.21.15.23:0/2128659865 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974004e070 con 0x55d4acba0d10 2024-03-24T10:47:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f972bfff640 1 -- 172.21.15.23:0/2128659865 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f97400049d0 con 0x55d4acba0d10 2024-03-24T10:47:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.827+0000 7f9755dc4740 1 -- 172.21.15.23:0/2128659865 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f974002ca70 con 0x55d4acba0d10 2024-03-24T10:47:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.828+0000 7f972bfff640 1 -- 172.21.15.23:0/2128659865 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9740051a30 con 0x55d4acba0d10 2024-03-24T10:47:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.828+0000 7f9755dc4740 10 client.27451 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.828+0000 7f9755dc4740 1 -- 172.21.15.23:0/2128659865 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d4acac7920 con 0x55d4acba0d10 2024-03-24T10:47:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.828+0000 7f972bfff640 1 -- 172.21.15.23:0/2128659865 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f974005e080 con 0x55d4acba0d10 2024-03-24T10:47:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.828+0000 7f972bfff640 1 -- 172.21.15.23:0/2128659865 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f974003c050 con 0x55d4acba0d10 2024-03-24T10:47:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.828+0000 7f9755dc4740 10 client.27451 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.828+0000 7f9755dc4740 10 client.27451 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:47:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.828+0000 7f9755dc4740 20 client.27451 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.828+0000 7f9755dc4740 1 --2- 172.21.15.23:0/2128659865 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55d4acc203e0 0x55d4acc22800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.828+0000 7f9755dc4740 4 client.27451 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:47:14.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.828+0000 7f9755dc4740 1 -- 172.21.15.23:0/2128659865 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d4ac892890 con 0x55d4acc203e0 2024-03-24T10:47:14.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.828+0000 7f974bfff640 1 --2- 172.21.15.23:0/2128659865 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55d4acc203e0 0x55d4acc22800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.829+0000 7f974bfff640 1 --2- 172.21.15.23:0/2128659865 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55d4acc203e0 0x55d4acc22800 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3f40c8676f47a899 server_cookie=4b70139ea8711b4b in_seq=0 out_seq=0 2024-03-24T10:47:14.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.829+0000 7f972bfff640 10 client.27451 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:47:14.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f972bfff640 1 -- 172.21.15.23:0/2128659865 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d4ac892890 con 0x55d4acc203e0 2024-03-24T10:47:14.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f972bfff640 10 client.27451 handle_command_reply: tid=0 2024-03-24T10:47:14.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f9755dc4740 1 client.27451 shutdown 2024-03-24T10:47:14.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f9755dc4740 1 -- 172.21.15.23:0/2128659865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba0d10 msgr2=0x55d4acbad590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f9755dc4740 1 --2- 172.21.15.23:0/2128659865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba0d10 0x55d4acbad590 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f974002c170 tx=0x7f974002bc30 comp rx=0 tx=0).stop 2024-03-24T10:47:14.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f9755dc4740 1 -- 172.21.15.23:0/2128659865 shutdown_connections 2024-03-24T10:47:14.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f9755dc4740 1 --2- 172.21.15.23:0/2128659865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4acba98f0 0x55d4acbadad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f9755dc4740 1 --2- 172.21.15.23:0/2128659865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d4acba0d10 0x55d4acbad590 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f9755dc4740 1 --2- 172.21.15.23:0/2128659865 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55d4acc203e0 0x55d4acc22800 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f9755dc4740 1 --2- 172.21.15.23:0/2128659865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d4acba03f0 0x55d4acbad050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f972bfff640 0 client.27451 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:47:14.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f9755dc4740 1 -- 172.21.15.23:0/2128659865 >> 172.21.15.23:0/2128659865 conn(0x55d4acb9ab20 msgr2=0x55d4acba71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f9755dc4740 1 -- 172.21.15.23:0/2128659865 shutdown_connections 2024-03-24T10:47:14.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f9755dc4740 1 -- 172.21.15.23:0/2128659865 wait complete. 2024-03-24T10:47:14.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.832+0000 7f9755dc4740 20 client.27451 trim_cache size 0 max 16384 2024-03-24T10:47:14.837 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:47:14.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.833+0000 7f9753102640 1 -- 172.21.15.23:0/3628844324 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f973c0a2230 msgr2=0x7f973c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.833+0000 7f9753102640 1 --2- 172.21.15.23:0/3628844324 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f973c0a2230 0x7f973c0a46f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f974409af60 tx=0x7f974406a170 comp rx=0 tx=0).stop 2024-03-24T10:47:14.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.833+0000 7f9753102640 1 -- 172.21.15.23:0/3628844324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f974c106660 msgr2=0x7f974c197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.833+0000 7f9753102640 1 --2- 172.21.15.23:0/3628844324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f974c106660 0x7f974c197b30 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f974400eb10 tx=0x7f974400c2d0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.833+0000 7f9753102640 1 -- 172.21.15.23:0/3628844324 shutdown_connections 2024-03-24T10:47:14.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.833+0000 7f9753102640 1 --2- 172.21.15.23:0/3628844324 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f973c0a2230 0x7f973c0a46f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.833+0000 7f9753102640 1 --2- 172.21.15.23:0/3628844324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f974c106660 0x7f974c197b30 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.833+0000 7f9753102640 1 --2- 172.21.15.23:0/3628844324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f974c100f60 0x7f974c19de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.833+0000 7f9753102640 1 --2- 172.21.15.23:0/3628844324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f974c1005a0 0x7f974c19d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.833+0000 7f9753102640 1 -- 172.21.15.23:0/3628844324 >> 172.21.15.23:0/3628844324 conn(0x7f974c077710 msgr2=0x7f974c0ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.833+0000 7f9753102640 1 -- 172.21.15.23:0/3628844324 shutdown_connections 2024-03-24T10:47:14.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.833+0000 7f9753102640 1 -- 172.21.15.23:0/3628844324 wait complete. 2024-03-24T10:47:14.844 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:47:14.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.996+0000 7f8f60b4f640 1 -- 172.21.15.23:0/1420166291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f5c1005a0 msgr2=0x7f8f5c100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:14.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.996+0000 7f8f60b4f640 1 --2- 172.21.15.23:0/1420166291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f5c1005a0 0x7f8f5c100a20 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f8f44009900 tx=0x7f8f4402bc30 comp rx=0 tx=0).stop 2024-03-24T10:47:14.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.996+0000 7f8f60b4f640 1 -- 172.21.15.23:0/1420166291 shutdown_connections 2024-03-24T10:47:14.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.996+0000 7f8f60b4f640 1 --2- 172.21.15.23:0/1420166291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f5c100f60 0x7f8f5c10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.996+0000 7f8f60b4f640 1 --2- 172.21.15.23:0/1420166291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f5c1005a0 0x7f8f5c100a20 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.996+0000 7f8f60b4f640 1 --2- 172.21.15.23:0/1420166291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f5c106660 0x7f8f5c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:14.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.996+0000 7f8f60b4f640 1 -- 172.21.15.23:0/1420166291 >> 172.21.15.23:0/1420166291 conn(0x7f8f5c077710 msgr2=0x7f8f5c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:14.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.996+0000 7f8f60b4f640 1 -- 172.21.15.23:0/1420166291 shutdown_connections 2024-03-24T10:47:14.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.996+0000 7f8f60b4f640 1 -- 172.21.15.23:0/1420166291 wait complete. 2024-03-24T10:47:14.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.996+0000 7f8f60b4f640 1 Processor -- start 2024-03-24T10:47:14.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f60b4f640 1 -- start start 2024-03-24T10:47:14.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f60b4f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f5c1005a0 0x7f8f5c19d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f60b4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f5c100f60 0x7f8f5c19de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f60b4f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f5c106660 0x7f8f5c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:14.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f60b4f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c198120 con 0x7f8f5c106660 2024-03-24T10:47:14.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f60b4f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c198290 con 0x7f8f5c100f60 2024-03-24T10:47:14.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f60b4f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c198450 con 0x7f8f5c1005a0 2024-03-24T10:47:14.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f5ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f5c106660 0x7f8f5c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f5ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f5c106660 0x7f8f5c197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45218/0 (socket says 172.21.15.23:45218) 2024-03-24T10:47:14.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f5ad9d640 1 -- 172.21.15.23:0/270529243 learned_addr learned my addr 172.21.15.23:0/270529243 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:14.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f53fff640 1 --2- 172.21.15.23:0/270529243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f5c100f60 0x7f8f5c19de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:14.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f5a59c640 1 --2- 172.21.15.23:0/270529243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f5c1005a0 0x7f8f5c19d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f5ad9d640 1 -- 172.21.15.23:0/270529243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f5c1005a0 msgr2=0x7f8f5c19d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f5ad9d640 1 --2- 172.21.15.23:0/270529243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f5c1005a0 0x7f8f5c19d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f5ad9d640 1 -- 172.21.15.23:0/270529243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f5c100f60 msgr2=0x7f8f5c19de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f5ad9d640 1 --2- 172.21.15.23:0/270529243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f5c100f60 0x7f8f5c19de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f5ad9d640 1 -- 172.21.15.23:0/270529243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f44009560 con 0x7f8f5c106660 2024-03-24T10:47:15.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f5ad9d640 1 --2- 172.21.15.23:0/270529243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f5c106660 0x7f8f5c197ad0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400eb10 tx=0x7f8f5400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:15.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f537fe640 1 -- 172.21.15.23:0/270529243 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f54004b20 con 0x7f8f5c106660 2024-03-24T10:47:15.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f60b4f640 1 -- 172.21.15.23:0/270529243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f5c198730 con 0x7f8f5c106660 2024-03-24T10:47:15.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f537fe640 1 -- 172.21.15.23:0/270529243 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f54004c80 con 0x7f8f5c106660 2024-03-24T10:47:15.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.997+0000 7f8f537fe640 1 -- 172.21.15.23:0/270529243 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f54010670 con 0x7f8f5c106660 2024-03-24T10:47:15.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:14.998+0000 7f8f60b4f640 1 -- 172.21.15.23:0/270529243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f5c1a49c0 con 0x7f8f5c106660 2024-03-24T10:47:15.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.000+0000 7f8f537fe640 1 -- 172.21.15.23:0/270529243 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f8f54005020 con 0x7f8f5c106660 2024-03-24T10:47:15.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.000+0000 7f8f63807740 1 Processor -- start 2024-03-24T10:47:15.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f63807740 1 -- start start 2024-03-24T10:47:15.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f63807740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958c3f0 0x5643b958c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f63807740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b958cd10 0x5643b95953b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f63807740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b95958f0 0x5643b9597ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f63807740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5643b94fa840 con 0x5643b958cd10 2024-03-24T10:47:15.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f63807740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5643b94f7d70 con 0x5643b95958f0 2024-03-24T10:47:15.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f63807740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5643b94f3550 con 0x5643b958c3f0 2024-03-24T10:47:15.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f5a59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b958cd10 0x5643b95953b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f537fe640 1 --2- 172.21.15.23:0/270529243 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8f480a2230 0x7f8f480a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f5a59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b958cd10 0x5643b95953b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45224/0 (socket says 172.21.15.23:45224) 2024-03-24T10:47:15.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f5a59c640 1 -- 172.21.15.23:0/3714172532 learned_addr learned my addr 172.21.15.23:0/3714172532 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:15.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f53fff640 1 --2- 172.21.15.23:0/3714172532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958c3f0 0x5643b958c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f53fff640 1 --2- 172.21.15.23:0/3714172532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b95958f0 0x5643b9597ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f5ad9d640 1 --2- 172.21.15.23:0/270529243 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8f480a2230 0x7f8f480a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f537fe640 1 -- 172.21.15.23:0/270529243 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f8f54014070 con 0x7f8f5c106660 2024-03-24T10:47:15.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f5a59c640 1 -- 172.21.15.23:0/3714172532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958c3f0 msgr2=0x5643b958c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f5a59c640 1 --2- 172.21.15.23:0/3714172532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958c3f0 0x5643b958c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f5a59c640 1 -- 172.21.15.23:0/3714172532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b95958f0 msgr2=0x5643b9597ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f5a59c640 1 --2- 172.21.15.23:0/3714172532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b95958f0 0x5643b9597ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.001+0000 7f8f5a59c640 1 -- 172.21.15.23:0/3714172532 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643b9488de0 con 0x5643b958cd10 2024-03-24T10:47:15.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.002+0000 7f8f5a59c640 1 --2- 172.21.15.23:0/3714172532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b958cd10 0x5643b95953b0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009cf0 tx=0x7f8f4c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d328df38f40e1a90 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:15.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.002+0000 7f8f5ad9d640 1 --2- 172.21.15.23:0/270529243 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8f480a2230 0x7f8f480a46f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8f540cc340 tx=0x7f8f54000f40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:15.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.002+0000 7f8f50ff9640 1 -- 172.21.15.23:0/3714172532 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c009de0 con 0x5643b958cd10 2024-03-24T10:47:15.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.002+0000 7f8f50ff9640 1 -- 172.21.15.23:0/3714172532 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f4c0035f0 con 0x5643b958cd10 2024-03-24T10:47:15.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.002+0000 7f8f50ff9640 1 -- 172.21.15.23:0/3714172532 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c0196b0 con 0x5643b958cd10 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.003+0000 7f8f63807740 1 -- 172.21.15.23:0/3714172532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b958cd10 msgr2=0x5643b95953b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.003+0000 7f8f63807740 1 --2- 172.21.15.23:0/3714172532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b958cd10 0x5643b95953b0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009cf0 tx=0x7f8f4c011810 comp rx=0 tx=0).stop 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.003+0000 7f8f63807740 1 -- 172.21.15.23:0/3714172532 shutdown_connections 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.003+0000 7f8f63807740 1 --2- 172.21.15.23:0/3714172532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b95958f0 0x5643b9597ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.003+0000 7f8f63807740 1 --2- 172.21.15.23:0/3714172532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b958cd10 0x5643b95953b0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.003+0000 7f8f63807740 1 --2- 172.21.15.23:0/3714172532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958c3f0 0x5643b958c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.003+0000 7f8f63807740 1 -- 172.21.15.23:0/3714172532 >> 172.21.15.23:0/3714172532 conn(0x5643b9586b20 msgr2=0x5643b9586f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.003+0000 7f8f63807740 1 -- 172.21.15.23:0/3714172532 shutdown_connections 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.003+0000 7f8f63807740 1 -- 172.21.15.23:0/3714172532 wait complete. 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.003+0000 7f8f63807740 1 -- start start 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.003+0000 7f8f63807740 1 Processor -- start 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f63807740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958c3f0 0x5643b95929a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f63807740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958cd10 0x5643b9592ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f63807740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b95958f0 0x5643b9593420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f63807740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5643b91c6430 con 0x5643b95958f0 2024-03-24T10:47:15.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f63807740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5643b9182e00 con 0x5643b958c3f0 2024-03-24T10:47:15.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f63807740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5643b9478590 con 0x5643b958cd10 2024-03-24T10:47:15.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f53fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b95958f0 0x5643b9593420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f53fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b95958f0 0x5643b9593420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45240/0 (socket says 172.21.15.23:45240) 2024-03-24T10:47:15.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f53fff640 1 -- 172.21.15.23:0/1374641234 learned_addr learned my addr 172.21.15.23:0/1374641234 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:15.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f63807740 10 client.0 _ll_register_callbacks cb 0x5643b94e7fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:15.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f53fff640 1 --2- 172.21.15.23:0/1374641234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958c3f0 0x5643b95929a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f5a59c640 1 --2- 172.21.15.23:0/1374641234 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958cd10 0x5643b9592ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f53fff640 1 -- 172.21.15.23:0/1374641234 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958cd10 msgr2=0x5643b9592ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f53fff640 1 --2- 172.21.15.23:0/1374641234 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958cd10 0x5643b9592ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f53fff640 1 -- 172.21.15.23:0/1374641234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958c3f0 msgr2=0x5643b95929a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f53fff640 1 --2- 172.21.15.23:0/1374641234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958c3f0 0x5643b95929a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f53fff640 1 -- 172.21.15.23:0/1374641234 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f4c009950 con 0x5643b95958f0 2024-03-24T10:47:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f53fff640 1 --2- 172.21.15.23:0/1374641234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b95958f0 0x5643b9593420 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f8f4402cd80 tx=0x7f8f44009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a07793bd945927 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f41ffb640 1 -- 172.21.15.23:0/1374641234 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f44004480 con 0x5643b95958f0 2024-03-24T10:47:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f63807740 1 -- 172.21.15.23:0/1374641234 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5643b91f27e0 con 0x5643b95958f0 2024-03-24T10:47:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.004+0000 7f8f41ffb640 1 -- 172.21.15.23:0/1374641234 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f8f44004df0 con 0x5643b95958f0 2024-03-24T10:47:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.005+0000 7f8f41ffb640 1 -- 172.21.15.23:0/1374641234 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8f44004fb0 con 0x5643b95958f0 2024-03-24T10:47:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.005+0000 7f8f63807740 10 client.17604 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.005+0000 7f8f63807740 1 -- 172.21.15.23:0/1374641234 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5643b94f1bd0 con 0x5643b95958f0 2024-03-24T10:47:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.005+0000 7f8f41ffb640 1 -- 172.21.15.23:0/1374641234 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5643b91f27e0 con 0x5643b95958f0 2024-03-24T10:47:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.005+0000 7f8f41ffb640 1 -- 172.21.15.23:0/1374641234 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f8f44003b60 con 0x5643b95958f0 2024-03-24T10:47:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.005+0000 7f8f63807740 10 client.17604 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.005+0000 7f8f63807740 10 client.17604 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:47:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.005+0000 7f8f63807740 20 client.17604 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.005+0000 7f8f63807740 1 --2- 172.21.15.23:0/1374641234 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5643b95a5a20 0x5643b95a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.005+0000 7f8f63807740 4 client.17604 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:15.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.005+0000 7f8f63807740 1 -- 172.21.15.23:0/1374641234 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643b922f720 con 0x5643b95a5a20 2024-03-24T10:47:15.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.005+0000 7f8f5a59c640 1 --2- 172.21.15.23:0/1374641234 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5643b95a5a20 0x5643b95a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.005+0000 7f8f41ffb640 10 client.17604 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:47:15.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.005+0000 7f8f5a59c640 1 --2- 172.21.15.23:0/1374641234 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5643b95a5a20 0x5643b95a7e40 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=77b974188205dba2 server_cookie=4a7296b0283cf1d6 in_seq=0 out_seq=0 2024-03-24T10:47:15.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f41ffb640 1 -- 172.21.15.23:0/1374641234 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5643b922f720 con 0x5643b95a5a20 2024-03-24T10:47:15.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f41ffb640 10 client.17604 handle_command_reply: tid=0 2024-03-24T10:47:15.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f63807740 1 client.17604 shutdown 2024-03-24T10:47:15.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f63807740 1 -- 172.21.15.23:0/1374641234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b95958f0 msgr2=0x5643b9593420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f63807740 1 --2- 172.21.15.23:0/1374641234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b95958f0 0x5643b9593420 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f8f4402cd80 tx=0x7f8f44009260 comp rx=0 tx=0).stop 2024-03-24T10:47:15.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f63807740 1 -- 172.21.15.23:0/1374641234 shutdown_connections 2024-03-24T10:47:15.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f63807740 1 --2- 172.21.15.23:0/1374641234 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5643b95a5a20 0x5643b95a7e40 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f63807740 1 --2- 172.21.15.23:0/1374641234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b95958f0 0x5643b9593420 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f63807740 1 --2- 172.21.15.23:0/1374641234 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958cd10 0x5643b9592ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f41ffb640 0 client.17604 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:47:15.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f63807740 1 --2- 172.21.15.23:0/1374641234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958c3f0 0x5643b95929a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f63807740 1 -- 172.21.15.23:0/1374641234 >> 172.21.15.23:0/1374641234 conn(0x5643b9586b20 msgr2=0x5643b9588870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:15.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f63807740 1 -- 172.21.15.23:0/1374641234 shutdown_connections 2024-03-24T10:47:15.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f63807740 1 -- 172.21.15.23:0/1374641234 wait complete. 2024-03-24T10:47:15.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.006+0000 7f8f63807740 20 client.17604 trim_cache size 0 max 16384 2024-03-24T10:47:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f63807740 1 Processor -- start 2024-03-24T10:47:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f63807740 1 -- start start 2024-03-24T10:47:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f63807740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958c3f0 0x5643b95a1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f63807740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b958cd10 0x5643b9588660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f63807740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b95958f0 0x5643b95ad6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f63807740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5643b91f2810 con 0x5643b958cd10 2024-03-24T10:47:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f63807740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5643b95a42e0 con 0x5643b958c3f0 2024-03-24T10:47:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f63807740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5643b9587910 con 0x5643b95958f0 2024-03-24T10:47:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f5a59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b958cd10 0x5643b9588660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958c3f0 0x5643b95a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f5a59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b958cd10 0x5643b9588660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45246/0 (socket says 172.21.15.23:45246) 2024-03-24T10:47:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958c3f0 0x5643b95a1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:54854/0 (socket says 172.21.15.23:54854) 2024-03-24T10:47:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f5a59c640 1 -- 172.21.15.23:0/2116379786 learned_addr learned my addr 172.21.15.23:0/2116379786 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f53fff640 1 --2- 172.21.15.23:0/2116379786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b95958f0 0x5643b95ad6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f5a59c640 1 -- 172.21.15.23:0/2116379786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b95958f0 msgr2=0x5643b95ad6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f5a59c640 1 --2- 172.21.15.23:0/2116379786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b95958f0 0x5643b95ad6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f5a59c640 1 -- 172.21.15.23:0/2116379786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958c3f0 msgr2=0x5643b95a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f5a59c640 1 --2- 172.21.15.23:0/2116379786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958c3f0 0x5643b95a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.021+0000 7f8f5a59c640 1 -- 172.21.15.23:0/2116379786 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643b948dc80 con 0x5643b958cd10 2024-03-24T10:47:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.022+0000 7f8f5a59c640 1 --2- 172.21.15.23:0/2116379786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b958cd10 0x5643b9588660 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009cf0 tx=0x7f8f4c003dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c9aadd0aba8f2e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.022+0000 7f8f43fff640 1 -- 172.21.15.23:0/2116379786 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c0097a0 con 0x5643b958cd10 2024-03-24T10:47:15.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.022+0000 7f8f43fff640 1 -- 172.21.15.23:0/2116379786 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f4c029070 con 0x5643b958cd10 2024-03-24T10:47:15.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.022+0000 7f8f43fff640 1 -- 172.21.15.23:0/2116379786 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c025450 con 0x5643b958cd10 2024-03-24T10:47:15.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.022+0000 7f8f63807740 1 -- 172.21.15.23:0/2116379786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b958cd10 msgr2=0x5643b9588660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.022+0000 7f8f63807740 1 --2- 172.21.15.23:0/2116379786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b958cd10 0x5643b9588660 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009cf0 tx=0x7f8f4c003dd0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 -- 172.21.15.23:0/2116379786 shutdown_connections 2024-03-24T10:47:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 --2- 172.21.15.23:0/2116379786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b95958f0 0x5643b95ad6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 --2- 172.21.15.23:0/2116379786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b958cd10 0x5643b9588660 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 --2- 172.21.15.23:0/2116379786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958c3f0 0x5643b95a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 -- 172.21.15.23:0/2116379786 >> 172.21.15.23:0/2116379786 conn(0x5643b9586b20 msgr2=0x5643b9587f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 -- 172.21.15.23:0/2116379786 shutdown_connections 2024-03-24T10:47:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 -- 172.21.15.23:0/2116379786 wait complete. 2024-03-24T10:47:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 -- start start 2024-03-24T10:47:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 Processor -- start 2024-03-24T10:47:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958c3f0 0x5643b959db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958cd10 0x5643b9607020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b95958f0 0x5643b9607560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5643b9607c40 con 0x5643b95958f0 2024-03-24T10:47:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5643b9607db0 con 0x5643b958cd10 2024-03-24T10:47:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5643b96047a0 con 0x5643b958c3f0 2024-03-24T10:47:15.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f63807740 10 client.0 _ll_register_callbacks cb 0x7f8f44004ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:15.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f5a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958cd10 0x5643b9607020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f53fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b95958f0 0x5643b9607560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f5a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958cd10 0x5643b9607020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:54862/0 (socket says 172.21.15.23:54862) 2024-03-24T10:47:15.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f5a59c640 1 -- 172.21.15.23:0/2237115922 learned_addr learned my addr 172.21.15.23:0/2237115922 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:15.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f53fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b95958f0 0x5643b9607560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45254/0 (socket says 172.21.15.23:45254) 2024-03-24T10:47:15.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.023+0000 7f8f53fff640 1 --2- 172.21.15.23:0/2237115922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958c3f0 0x5643b959db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.024+0000 7f8f5a59c640 1 -- 172.21.15.23:0/2237115922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958c3f0 msgr2=0x5643b959db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.024+0000 7f8f5a59c640 1 --2- 172.21.15.23:0/2237115922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958c3f0 0x5643b959db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.024+0000 7f8f5a59c640 1 -- 172.21.15.23:0/2237115922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b95958f0 msgr2=0x5643b9607560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.024+0000 7f8f5a59c640 1 --2- 172.21.15.23:0/2237115922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b95958f0 0x5643b9607560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.024+0000 7f8f5a59c640 1 -- 172.21.15.23:0/2237115922 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643b9488de0 con 0x5643b958cd10 2024-03-24T10:47:15.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.024+0000 7f8f53fff640 1 --2- 172.21.15.23:0/2237115922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b95958f0 0x5643b9607560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:15.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.024+0000 7f8f5a59c640 1 --2- 172.21.15.23:0/2237115922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958cd10 0x5643b9607020 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c005fd0 tx=0x7f8f4c005170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=290db4eae18b3a57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:15.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.024+0000 7f8f41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:47:15.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.024+0000 7f8f41ffb640 1 -- 172.21.15.23:0/2237115922 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c005400 con 0x5643b958cd10 2024-03-24T10:47:15.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.024+0000 7f8f63807740 1 -- 172.21.15.23:0/2237115922 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f4c0035f0 con 0x5643b958cd10 2024-03-24T10:47:15.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.024+0000 7f8f41ffb640 1 -- 172.21.15.23:0/2237115922 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f8f4c005560 con 0x5643b958cd10 2024-03-24T10:47:15.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.024+0000 7f8f41ffb640 1 -- 172.21.15.23:0/2237115922 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8f4c003760 con 0x5643b958cd10 2024-03-24T10:47:15.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.024+0000 7f8f63807740 10 client.27497 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:15.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.024+0000 7f8f63807740 1 -- 172.21.15.23:0/2237115922 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5643b94b3920 con 0x5643b958cd10 2024-03-24T10:47:15.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.025+0000 7f8f41ffb640 1 -- 172.21.15.23:0/2237115922 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f4c0035f0 con 0x5643b958cd10 2024-03-24T10:47:15.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.025+0000 7f8f41ffb640 1 -- 172.21.15.23:0/2237115922 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f8f4c025a50 con 0x5643b958cd10 2024-03-24T10:47:15.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.025+0000 7f8f63807740 10 client.27497 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:15.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.025+0000 7f8f63807740 10 client.27497 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:47:15.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.025+0000 7f8f63807740 20 client.27497 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.025+0000 7f8f63807740 1 --2- 172.21.15.23:0/2237115922 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5643b9614ea0 0x5643b96172c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.025+0000 7f8f63807740 4 client.27497 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:47:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.025+0000 7f8f63807740 1 -- 172.21.15.23:0/2237115922 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5643b927e890 con 0x5643b9614ea0 2024-03-24T10:47:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.025+0000 7f8f53fff640 1 --2- 172.21.15.23:0/2237115922 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5643b9614ea0 0x5643b96172c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.025+0000 7f8f41ffb640 10 client.27497 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:47:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.025+0000 7f8f53fff640 1 --2- 172.21.15.23:0/2237115922 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5643b9614ea0 0x5643b96172c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6aa88f3f16085adc server_cookie=c9e01a3d08382c32 in_seq=0 out_seq=0 2024-03-24T10:47:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.025+0000 7f8f41ffb640 1 -- 172.21.15.23:0/2237115922 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5643b927e890 con 0x5643b9614ea0 2024-03-24T10:47:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.025+0000 7f8f41ffb640 10 client.27497 handle_command_reply: tid=0 2024-03-24T10:47:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.025+0000 7f8f63807740 1 client.27497 shutdown 2024-03-24T10:47:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f63807740 1 -- 172.21.15.23:0/2237115922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958cd10 msgr2=0x5643b9607020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f63807740 1 --2- 172.21.15.23:0/2237115922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958cd10 0x5643b9607020 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c005fd0 tx=0x7f8f4c005170 comp rx=0 tx=0).stop 2024-03-24T10:47:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f63807740 1 -- 172.21.15.23:0/2237115922 shutdown_connections 2024-03-24T10:47:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f63807740 1 --2- 172.21.15.23:0/2237115922 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x5643b9614ea0 0x5643b96172c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f63807740 1 --2- 172.21.15.23:0/2237115922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5643b95958f0 0x5643b9607560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f63807740 1 --2- 172.21.15.23:0/2237115922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5643b958cd10 0x5643b9607020 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f63807740 1 --2- 172.21.15.23:0/2237115922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5643b958c3f0 0x5643b959db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f41ffb640 0 client.27497 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:47:15.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f63807740 1 -- 172.21.15.23:0/2237115922 >> 172.21.15.23:0/2237115922 conn(0x5643b9586b20 msgr2=0x5643b95887e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:15.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f63807740 1 -- 172.21.15.23:0/2237115922 shutdown_connections 2024-03-24T10:47:15.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f63807740 1 -- 172.21.15.23:0/2237115922 wait complete. 2024-03-24T10:47:15.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f63807740 20 client.27497 trim_cache size 0 max 16384 2024-03-24T10:47:15.033 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:47:15.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f60b4f640 1 -- 172.21.15.23:0/270529243 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8f480a2230 msgr2=0x7f8f480a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f60b4f640 1 --2- 172.21.15.23:0/270529243 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8f480a2230 0x7f8f480a46f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8f540cc340 tx=0x7f8f54000f40 comp rx=0 tx=0).stop 2024-03-24T10:47:15.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f60b4f640 1 -- 172.21.15.23:0/270529243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f5c106660 msgr2=0x7f8f5c197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f60b4f640 1 --2- 172.21.15.23:0/270529243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f5c106660 0x7f8f5c197ad0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400eb10 tx=0x7f8f5400c2d0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f60b4f640 1 -- 172.21.15.23:0/270529243 shutdown_connections 2024-03-24T10:47:15.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f60b4f640 1 --2- 172.21.15.23:0/270529243 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8f480a2230 0x7f8f480a46f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f60b4f640 1 --2- 172.21.15.23:0/270529243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f5c106660 0x7f8f5c197ad0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f60b4f640 1 --2- 172.21.15.23:0/270529243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f5c100f60 0x7f8f5c19de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f60b4f640 1 --2- 172.21.15.23:0/270529243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f5c1005a0 0x7f8f5c19d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f60b4f640 1 -- 172.21.15.23:0/270529243 >> 172.21.15.23:0/270529243 conn(0x7f8f5c077710 msgr2=0x7f8f5c068af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:15.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f60b4f640 1 -- 172.21.15.23:0/270529243 shutdown_connections 2024-03-24T10:47:15.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.026+0000 7f8f60b4f640 1 -- 172.21.15.23:0/270529243 wait complete. 2024-03-24T10:47:15.037 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:47:15.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 700020 10.92 MB/sec execute 212 sec latency 18.982 ms 2024-03-24T10:47:15.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.146+0000 7f01c303b640 1 -- 172.21.15.23:0/1025308339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01bc1005a0 msgr2=0x7f01bc100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.146+0000 7f01c303b640 1 --2- 172.21.15.23:0/1025308339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01bc1005a0 0x7f01bc100a20 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f01a8009900 tx=0x7f01a802bc30 comp rx=0 tx=0).stop 2024-03-24T10:47:15.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 -- 172.21.15.23:0/1025308339 shutdown_connections 2024-03-24T10:47:15.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 --2- 172.21.15.23:0/1025308339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01bc100f60 0x7f01bc10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 --2- 172.21.15.23:0/1025308339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01bc1005a0 0x7f01bc100a20 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 --2- 172.21.15.23:0/1025308339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01bc106660 0x7f01bc106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 -- 172.21.15.23:0/1025308339 >> 172.21.15.23:0/1025308339 conn(0x7f01bc077710 msgr2=0x7f01bc077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:15.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 -- 172.21.15.23:0/1025308339 shutdown_connections 2024-03-24T10:47:15.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 -- 172.21.15.23:0/1025308339 wait complete. 2024-03-24T10:47:15.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 Processor -- start 2024-03-24T10:47:15.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 -- start start 2024-03-24T10:47:15.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01bc1005a0 0x7f01bc19d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01bc100f60 0x7f01bc19de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01bc106660 0x7f01bc197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f01bc198120 con 0x7f01bc100f60 2024-03-24T10:47:15.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f01bc198290 con 0x7f01bc106660 2024-03-24T10:47:15.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.147+0000 7f01c303b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f01bc198450 con 0x7f01bc1005a0 2024-03-24T10:47:15.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01bbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01bc100f60 0x7f01bc19de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01bbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01bc100f60 0x7f01bc19de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45280/0 (socket says 172.21.15.23:45280) 2024-03-24T10:47:15.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01bbfff640 1 -- 172.21.15.23:0/2804006811 learned_addr learned my addr 172.21.15.23:0/2804006811 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:15.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01c0dd7640 1 --2- 172.21.15.23:0/2804006811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01bc1005a0 0x7f01bc19d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01c15d8640 1 --2- 172.21.15.23:0/2804006811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01bc106660 0x7f01bc197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01bbfff640 1 -- 172.21.15.23:0/2804006811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01bc1005a0 msgr2=0x7f01bc19d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01bbfff640 1 --2- 172.21.15.23:0/2804006811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01bc1005a0 0x7f01bc19d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01bbfff640 1 -- 172.21.15.23:0/2804006811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01bc106660 msgr2=0x7f01bc197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01bbfff640 1 --2- 172.21.15.23:0/2804006811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01bc106660 0x7f01bc197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01bbfff640 1 -- 172.21.15.23:0/2804006811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a8009560 con 0x7f01bc100f60 2024-03-24T10:47:15.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01bbfff640 1 --2- 172.21.15.23:0/2804006811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01bc100f60 0x7f01bc19de60 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f01a8005940 tx=0x7f01a802c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:15.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01b9ffb640 1 -- 172.21.15.23:0/2804006811 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a80042d0 con 0x7f01bc100f60 2024-03-24T10:47:15.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01b9ffb640 1 -- 172.21.15.23:0/2804006811 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01a8004430 con 0x7f01bc100f60 2024-03-24T10:47:15.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01c303b640 1 -- 172.21.15.23:0/2804006811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01bc1986d0 con 0x7f01bc100f60 2024-03-24T10:47:15.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01b9ffb640 1 -- 172.21.15.23:0/2804006811 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a8003c50 con 0x7f01bc100f60 2024-03-24T10:47:15.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.148+0000 7f01c303b640 1 -- 172.21.15.23:0/2804006811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01bc1a49c0 con 0x7f01bc100f60 2024-03-24T10:47:15.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01b9ffb640 1 -- 172.21.15.23:0/2804006811 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f01a80054f0 con 0x7f01bc100f60 2024-03-24T10:47:15.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01c5c16740 1 Processor -- start 2024-03-24T10:47:15.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01c5c16740 1 -- start start 2024-03-24T10:47:15.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01c5c16740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42e63f0 0x559ce42e67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01c5c16740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e6d10 0x559ce42ef3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01c5c16740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42ef8f0 0x559ce42f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01c5c16740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x559ce4254840 con 0x559ce42e63f0 2024-03-24T10:47:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01c5c16740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559ce4251d70 con 0x559ce42ef8f0 2024-03-24T10:47:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01c5c16740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559ce424d550 con 0x559ce42e6d10 2024-03-24T10:47:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01c15d8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42e63f0 0x559ce42e67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01c15d8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42e63f0 0x559ce42e67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45296/0 (socket says 172.21.15.23:45296) 2024-03-24T10:47:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01c15d8640 1 -- 172.21.15.23:0/2403641029 learned_addr learned my addr 172.21.15.23:0/2403641029 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01c0dd7640 1 --2- 172.21.15.23:0/2403641029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e6d10 0x559ce42ef3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01bbfff640 1 --2- 172.21.15.23:0/2403641029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42ef8f0 0x559ce42f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.151+0000 7f01b9ffb640 1 --2- 172.21.15.23:0/2804006811 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f01ac0a22b0 0x7f01ac0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.152+0000 7f01c15d8640 1 -- 172.21.15.23:0/2403641029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e6d10 msgr2=0x559ce42ef3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.152+0000 7f01c15d8640 1 --2- 172.21.15.23:0/2403641029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e6d10 0x559ce42ef3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.152+0000 7f01c15d8640 1 -- 172.21.15.23:0/2403641029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42ef8f0 msgr2=0x559ce42f1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.152+0000 7f01c15d8640 1 --2- 172.21.15.23:0/2403641029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42ef8f0 0x559ce42f1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.152+0000 7f01c15d8640 1 -- 172.21.15.23:0/2403641029 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ce41e2de0 con 0x559ce42e63f0 2024-03-24T10:47:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.152+0000 7f01c0dd7640 1 --2- 172.21.15.23:0/2403641029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e6d10 0x559ce42ef3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.152+0000 7f01b9ffb640 1 -- 172.21.15.23:0/2804006811 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f01a80effd0 con 0x7f01bc100f60 2024-03-24T10:47:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.152+0000 7f01c15d8640 1 --2- 172.21.15.23:0/2804006811 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f01ac0a22b0 0x7f01ac0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.152+0000 7f01c15d8640 1 --2- 172.21.15.23:0/2403641029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42e63f0 0x559ce42e67d0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f01b4011590 tx=0x7f01b4011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbfc0e2f92221c04 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.152+0000 7f01a6ffd640 1 -- 172.21.15.23:0/2403641029 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b400f840 con 0x559ce42e63f0 2024-03-24T10:47:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.152+0000 7f01a6ffd640 1 -- 172.21.15.23:0/2403641029 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01b400fe80 con 0x559ce42e63f0 2024-03-24T10:47:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.152+0000 7f01a6ffd640 1 -- 172.21.15.23:0/2403641029 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b401bb40 con 0x559ce42e63f0 2024-03-24T10:47:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.153+0000 7f01c5c16740 1 -- 172.21.15.23:0/2403641029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42e63f0 msgr2=0x559ce42e67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.153+0000 7f01c15d8640 1 --2- 172.21.15.23:0/2804006811 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f01ac0a22b0 0x7f01ac0a4770 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f01b4022000 tx=0x7f01b4023000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.153+0000 7f01c5c16740 1 --2- 172.21.15.23:0/2403641029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42e63f0 0x559ce42e67d0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f01b4011590 tx=0x7f01b4011a50 comp rx=0 tx=0).stop 2024-03-24T10:47:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.153+0000 7f01c5c16740 1 -- 172.21.15.23:0/2403641029 shutdown_connections 2024-03-24T10:47:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.153+0000 7f01c5c16740 1 --2- 172.21.15.23:0/2403641029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42ef8f0 0x559ce42f1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.153+0000 7f01c5c16740 1 --2- 172.21.15.23:0/2403641029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e6d10 0x559ce42ef3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.153+0000 7f01c5c16740 1 --2- 172.21.15.23:0/2403641029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42e63f0 0x559ce42e67d0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.153+0000 7f01c5c16740 1 -- 172.21.15.23:0/2403641029 >> 172.21.15.23:0/2403641029 conn(0x559ce42e0b20 msgr2=0x559ce42e0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.153+0000 7f01c5c16740 1 -- 172.21.15.23:0/2403641029 shutdown_connections 2024-03-24T10:47:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.153+0000 7f01c5c16740 1 -- 172.21.15.23:0/2403641029 wait complete. 2024-03-24T10:47:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.153+0000 7f01c5c16740 1 -- start start 2024-03-24T10:47:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.153+0000 7f01c5c16740 1 Processor -- start 2024-03-24T10:47:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c5c16740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e63f0 0x559ce42ec9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c5c16740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42e6d10 0x559ce42ecef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c5c16740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42ef8f0 0x559ce42ed430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c5c16740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x559ce3f20430 con 0x559ce42ef8f0 2024-03-24T10:47:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c5c16740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559ce3edce00 con 0x559ce42e6d10 2024-03-24T10:47:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c5c16740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559ce41d2590 con 0x559ce42e63f0 2024-03-24T10:47:15.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c5c16740 10 client.0 _ll_register_callbacks cb 0x559ce4241fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:15.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c15d8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42ef8f0 0x559ce42ed430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42e6d10 0x559ce42ecef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c0dd7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e63f0 0x559ce42ec9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42e6d10 0x559ce42ecef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:54902/0 (socket says 172.21.15.23:54902) 2024-03-24T10:47:15.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c15d8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42ef8f0 0x559ce42ed430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45304/0 (socket says 172.21.15.23:45304) 2024-03-24T10:47:15.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01bbfff640 1 -- 172.21.15.23:0/1471754836 learned_addr learned my addr 172.21.15.23:0/1471754836 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:15.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c0dd7640 1 -- 172.21.15.23:0/1471754836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42e6d10 msgr2=0x559ce42ecef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c0dd7640 1 --2- 172.21.15.23:0/1471754836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42e6d10 0x559ce42ecef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c0dd7640 1 -- 172.21.15.23:0/1471754836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42ef8f0 msgr2=0x559ce42ed430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c0dd7640 1 --2- 172.21.15.23:0/1471754836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42ef8f0 0x559ce42ed430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c0dd7640 1 -- 172.21.15.23:0/1471754836 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01b4009cd0 con 0x559ce42e63f0 2024-03-24T10:47:15.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c15d8640 1 --2- 172.21.15.23:0/1471754836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42ef8f0 0x559ce42ed430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:15.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f01c0dd7640 1 --2- 172.21.15.23:0/1471754836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e63f0 0x559ce42ec9b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f01b00126d0 tx=0x7f01b0012b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f62dadd2092a835b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:15.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.154+0000 7f019bfff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:47:15.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f019bfff640 1 -- 172.21.15.23:0/1471754836 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b0017070 con 0x559ce42e63f0 2024-03-24T10:47:15.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f01c5c16740 1 -- 172.21.15.23:0/1471754836 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559ce3f89720 con 0x559ce42e63f0 2024-03-24T10:47:15.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f019bfff640 1 -- 172.21.15.23:0/1471754836 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f01b0004030 con 0x559ce42e63f0 2024-03-24T10:47:15.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f019bfff640 1 -- 172.21.15.23:0/1471754836 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f01b00041f0 con 0x559ce42e63f0 2024-03-24T10:47:15.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f01c5c16740 10 client.27481 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:15.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f01c5c16740 1 -- 172.21.15.23:0/1471754836 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559ce424bbd0 con 0x559ce42e63f0 2024-03-24T10:47:15.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f019bfff640 1 -- 172.21.15.23:0/1471754836 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559ce3f89720 con 0x559ce42e63f0 2024-03-24T10:47:15.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f019bfff640 1 -- 172.21.15.23:0/1471754836 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f01b0003a00 con 0x559ce42e63f0 2024-03-24T10:47:15.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f01c5c16740 10 client.27481 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:15.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f01c5c16740 10 client.27481 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:15.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f01c5c16740 20 client.27481 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:15.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f01c5c16740 1 --2- 172.21.15.23:0/1471754836 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559ce42ffb00 0x559ce4301f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f01c5c16740 4 client.27481 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f01c5c16740 1 -- 172.21.15.23:0/1471754836 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ce3fd8890 con 0x559ce42ffb00 2024-03-24T10:47:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.155+0000 7f01bbfff640 1 --2- 172.21.15.23:0/1471754836 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559ce42ffb00 0x559ce4301f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.156+0000 7f01bbfff640 1 --2- 172.21.15.23:0/1471754836 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559ce42ffb00 0x559ce4301f20 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=914f7bddad5e871f server_cookie=e4e8ffad6635fe2 in_seq=0 out_seq=0 2024-03-24T10:47:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.156+0000 7f019bfff640 10 client.27481 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f019bfff640 1 -- 172.21.15.23:0/1471754836 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559ce3fd8890 con 0x559ce42ffb00 2024-03-24T10:47:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f019bfff640 10 client.27481 handle_command_reply: tid=0 2024-03-24T10:47:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f01c5c16740 1 client.27481 shutdown 2024-03-24T10:47:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f01c5c16740 1 -- 172.21.15.23:0/1471754836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e63f0 msgr2=0x559ce42ec9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f01c5c16740 1 --2- 172.21.15.23:0/1471754836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e63f0 0x559ce42ec9b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f01b00126d0 tx=0x7f01b0012b90 comp rx=0 tx=0).stop 2024-03-24T10:47:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f01c5c16740 1 -- 172.21.15.23:0/1471754836 shutdown_connections 2024-03-24T10:47:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f01c5c16740 1 --2- 172.21.15.23:0/1471754836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42ef8f0 0x559ce42ed430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f01c5c16740 1 --2- 172.21.15.23:0/1471754836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42e6d10 0x559ce42ecef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f01c5c16740 1 --2- 172.21.15.23:0/1471754836 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559ce42ffb00 0x559ce4301f20 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f01c5c16740 1 --2- 172.21.15.23:0/1471754836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e63f0 0x559ce42ec9b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f019bfff640 0 client.27481 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:15.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f01c5c16740 1 -- 172.21.15.23:0/1471754836 >> 172.21.15.23:0/1471754836 conn(0x559ce42e0b20 msgr2=0x559ce42e2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:15.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f01c5c16740 1 -- 172.21.15.23:0/1471754836 shutdown_connections 2024-03-24T10:47:15.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f01c5c16740 1 -- 172.21.15.23:0/1471754836 wait complete. 2024-03-24T10:47:15.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.157+0000 7f01c5c16740 20 client.27481 trim_cache size 0 max 16384 2024-03-24T10:47:15.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01c5c16740 1 Processor -- start 2024-03-24T10:47:15.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01c5c16740 1 -- start start 2024-03-24T10:47:15.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01c5c16740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e63f0 0x559ce42fb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01c5c16740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42e6d10 0x559ce42ed250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01c5c16740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42ef8f0 0x559ce4307830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01c5c16740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x559ce42fe2c0 con 0x559ce42e6d10 2024-03-24T10:47:15.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01c5c16740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559ce42e2e10 con 0x559ce42ef8f0 2024-03-24T10:47:15.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01c5c16740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559ce42fb6c0 con 0x559ce42e63f0 2024-03-24T10:47:15.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01bbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42e6d10 0x559ce42ed250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01bbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42e6d10 0x559ce42ed250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45308/0 (socket says 172.21.15.23:45308) 2024-03-24T10:47:15.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01bbfff640 1 -- 172.21.15.23:0/2853608572 learned_addr learned my addr 172.21.15.23:0/2853608572 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:15.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01c15d8640 1 --2- 172.21.15.23:0/2853608572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42ef8f0 0x559ce4307830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01c0dd7640 1 --2- 172.21.15.23:0/2853608572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e63f0 0x559ce42fb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01bbfff640 1 -- 172.21.15.23:0/2853608572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e63f0 msgr2=0x559ce42fb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01bbfff640 1 --2- 172.21.15.23:0/2853608572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e63f0 0x559ce42fb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01bbfff640 1 -- 172.21.15.23:0/2853608572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42ef8f0 msgr2=0x559ce4307830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01bbfff640 1 --2- 172.21.15.23:0/2853608572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42ef8f0 0x559ce4307830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.171+0000 7f01bbfff640 1 -- 172.21.15.23:0/2853608572 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ce41e7c80 con 0x559ce42e6d10 2024-03-24T10:47:15.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.172+0000 7f01bbfff640 1 --2- 172.21.15.23:0/2853608572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42e6d10 0x559ce42ed250 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f01a8006010 tx=0x7f01a80a7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1223cff591f754e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:15.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.172+0000 7f01a67fc640 1 -- 172.21.15.23:0/2853608572 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a80bc5e0 con 0x559ce42e6d10 2024-03-24T10:47:15.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.172+0000 7f01a67fc640 1 -- 172.21.15.23:0/2853608572 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01a80bc400 con 0x559ce42e6d10 2024-03-24T10:47:15.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.172+0000 7f01a67fc640 1 -- 172.21.15.23:0/2853608572 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a8003e40 con 0x559ce42e6d10 2024-03-24T10:47:15.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.172+0000 7f01c5c16740 1 -- 172.21.15.23:0/2853608572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42e6d10 msgr2=0x559ce42ed250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.172+0000 7f01c5c16740 1 --2- 172.21.15.23:0/2853608572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42e6d10 0x559ce42ed250 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f01a8006010 tx=0x7f01a80a7040 comp rx=0 tx=0).stop 2024-03-24T10:47:15.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.172+0000 7f01c5c16740 1 -- 172.21.15.23:0/2853608572 shutdown_connections 2024-03-24T10:47:15.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.172+0000 7f01c5c16740 1 --2- 172.21.15.23:0/2853608572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42ef8f0 0x559ce4307830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.172+0000 7f01c5c16740 1 --2- 172.21.15.23:0/2853608572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42e6d10 0x559ce42ed250 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.172+0000 7f01c5c16740 1 --2- 172.21.15.23:0/2853608572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e63f0 0x559ce42fb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.172+0000 7f01c5c16740 1 -- 172.21.15.23:0/2853608572 >> 172.21.15.23:0/2853608572 conn(0x559ce42e0b20 msgr2=0x559ce42dfcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:15.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c5c16740 1 -- 172.21.15.23:0/2853608572 shutdown_connections 2024-03-24T10:47:15.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c5c16740 1 -- 172.21.15.23:0/2853608572 wait complete. 2024-03-24T10:47:15.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c5c16740 1 -- start start 2024-03-24T10:47:15.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c5c16740 1 Processor -- start 2024-03-24T10:47:15.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c5c16740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42e63f0 0x559ce42f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c5c16740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e6d10 0x559ce42f3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c5c16740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42ef8f0 0x559ce42f3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c5c16740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x559ce4364e60 con 0x559ce42ef8f0 2024-03-24T10:47:15.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c5c16740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559ce4364fd0 con 0x559ce42e63f0 2024-03-24T10:47:15.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c5c16740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559ce4365190 con 0x559ce42e6d10 2024-03-24T10:47:15.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c5c16740 10 client.0 _ll_register_callbacks cb 0x7f01b0009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:15.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c15d8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42ef8f0 0x559ce42f3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c0dd7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42e63f0 0x559ce42f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01bbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e6d10 0x559ce42f3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c15d8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42ef8f0 0x559ce42f3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45318/0 (socket says 172.21.15.23:45318) 2024-03-24T10:47:15.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c0dd7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42e63f0 0x559ce42f31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:54916/0 (socket says 172.21.15.23:54916) 2024-03-24T10:47:15.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.173+0000 7f01c15d8640 1 -- 172.21.15.23:0/780657050 learned_addr learned my addr 172.21.15.23:0/780657050 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:15.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f01c15d8640 1 -- 172.21.15.23:0/780657050 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e6d10 msgr2=0x559ce42f3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f01c15d8640 1 --2- 172.21.15.23:0/780657050 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e6d10 0x559ce42f3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f01c15d8640 1 -- 172.21.15.23:0/780657050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42e63f0 msgr2=0x559ce42f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:15.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f01c15d8640 1 --2- 172.21.15.23:0/780657050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42e63f0 0x559ce42f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f01c15d8640 1 -- 172.21.15.23:0/780657050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01bc1a49c0 con 0x559ce42ef8f0 2024-03-24T10:47:15.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f01c0dd7640 1 --2- 172.21.15.23:0/780657050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42e63f0 0x559ce42f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:15.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f01bbfff640 1 --2- 172.21.15.23:0/780657050 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e6d10 0x559ce42f3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:15.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f01c15d8640 1 --2- 172.21.15.23:0/780657050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42ef8f0 0x559ce42f3c50 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f01b4011f50 tx=0x7f01b400fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea2d8c9d003c21f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:15.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f019bfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:15.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f019bfff640 1 -- 172.21.15.23:0/780657050 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b402d040 con 0x559ce42ef8f0 2024-03-24T10:47:15.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f019bfff640 1 -- 172.21.15.23:0/780657050 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f01b401b430 con 0x559ce42ef8f0 2024-03-24T10:47:15.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f01c5c16740 1 -- 172.21.15.23:0/780657050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559ce3f4c7e0 con 0x559ce42ef8f0 2024-03-24T10:47:15.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f019bfff640 1 -- 172.21.15.23:0/780657050 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f01b4003eb0 con 0x559ce42ef8f0 2024-03-24T10:47:15.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f01c5c16740 10 client.17628 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:15.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.174+0000 7f01c5c16740 1 -- 172.21.15.23:0/780657050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559ce420d920 con 0x559ce42ef8f0 2024-03-24T10:47:15.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.175+0000 7f019bfff640 1 -- 172.21.15.23:0/780657050 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559ce3f4c7e0 con 0x559ce42ef8f0 2024-03-24T10:47:15.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.175+0000 7f019bfff640 1 -- 172.21.15.23:0/780657050 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f01b4024020 con 0x559ce42ef8f0 2024-03-24T10:47:15.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.175+0000 7f01c5c16740 10 client.17628 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:15.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.175+0000 7f01c5c16740 10 client.17628 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:15.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.175+0000 7f01c5c16740 20 client.17628 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:15.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.175+0000 7f01c5c16740 1 --2- 172.21.15.23:0/780657050 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559ce43669b0 0x559ce4368dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:15.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.175+0000 7f01c5c16740 4 client.17628 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:47:15.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.175+0000 7f01c5c16740 1 -- 172.21.15.23:0/780657050 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559ce3ef8700 con 0x559ce43669b0 2024-03-24T10:47:15.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.175+0000 7f01c0dd7640 1 --2- 172.21.15.23:0/780657050 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559ce43669b0 0x559ce4368dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:15.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.175+0000 7f01c0dd7640 1 --2- 172.21.15.23:0/780657050 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559ce43669b0 0x559ce4368dd0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0a7f64b63994b41 server_cookie=b0ecf9ca685596b5 in_seq=0 out_seq=0 2024-03-24T10:47:15.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.175+0000 7f019bfff640 10 client.17628 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:15.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f019bfff640 1 -- 172.21.15.23:0/780657050 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559ce3ef8700 con 0x559ce43669b0 2024-03-24T10:47:15.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f019bfff640 10 client.17628 handle_command_reply: tid=0 2024-03-24T10:47:15.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c5c16740 1 client.17628 shutdown 2024-03-24T10:47:15.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c5c16740 1 -- 172.21.15.23:0/780657050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42ef8f0 msgr2=0x559ce42f3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c5c16740 1 --2- 172.21.15.23:0/780657050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42ef8f0 0x559ce42f3c50 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f01b4011f50 tx=0x7f01b400fc70 comp rx=0 tx=0).stop 2024-03-24T10:47:15.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c5c16740 1 -- 172.21.15.23:0/780657050 shutdown_connections 2024-03-24T10:47:15.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c5c16740 1 --2- 172.21.15.23:0/780657050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559ce42ef8f0 0x559ce42f3c50 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c5c16740 1 --2- 172.21.15.23:0/780657050 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x559ce43669b0 0x559ce4368dd0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c5c16740 1 --2- 172.21.15.23:0/780657050 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ce42e6d10 0x559ce42f3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c5c16740 1 --2- 172.21.15.23:0/780657050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ce42e63f0 0x559ce42f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c5c16740 1 -- 172.21.15.23:0/780657050 >> 172.21.15.23:0/780657050 conn(0x559ce42e0b20 msgr2=0x559ce42df430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:15.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f019bfff640 0 client.17628 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:15.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c5c16740 1 -- 172.21.15.23:0/780657050 shutdown_connections 2024-03-24T10:47:15.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c5c16740 1 -- 172.21.15.23:0/780657050 wait complete. 2024-03-24T10:47:15.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c5c16740 20 client.17628 trim_cache size 0 max 16384 2024-03-24T10:47:15.183 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:47:15.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c303b640 1 -- 172.21.15.23:0/2804006811 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f01ac0a22b0 msgr2=0x7f01ac0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c303b640 1 --2- 172.21.15.23:0/2804006811 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f01ac0a22b0 0x7f01ac0a4770 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f01b4022000 tx=0x7f01b4023000 comp rx=0 tx=0).stop 2024-03-24T10:47:15.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c303b640 1 -- 172.21.15.23:0/2804006811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01bc100f60 msgr2=0x7f01bc19de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:15.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c303b640 1 --2- 172.21.15.23:0/2804006811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01bc100f60 0x7f01bc19de60 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f01a8005940 tx=0x7f01a802c670 comp rx=0 tx=0).stop 2024-03-24T10:47:15.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.176+0000 7f01c303b640 1 -- 172.21.15.23:0/2804006811 shutdown_connections 2024-03-24T10:47:15.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.177+0000 7f01c303b640 1 --2- 172.21.15.23:0/2804006811 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f01ac0a22b0 0x7f01ac0a4770 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.177+0000 7f01c303b640 1 --2- 172.21.15.23:0/2804006811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01bc106660 0x7f01bc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.177+0000 7f01c303b640 1 --2- 172.21.15.23:0/2804006811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01bc100f60 0x7f01bc19de60 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.177+0000 7f01c303b640 1 --2- 172.21.15.23:0/2804006811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01bc1005a0 0x7f01bc19d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:15.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.177+0000 7f01c303b640 1 -- 172.21.15.23:0/2804006811 >> 172.21.15.23:0/2804006811 conn(0x7f01bc077710 msgr2=0x7f01bc068af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:15.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.177+0000 7f01c303b640 1 -- 172.21.15.23:0/2804006811 shutdown_connections 2024-03-24T10:47:15.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:15.177+0000 7f01c303b640 1 -- 172.21.15.23:0/2804006811 wait complete. 2024-03-24T10:47:15.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:15 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/278025147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:47:15.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:15 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/278025147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:47:15.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:15 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/278025147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:47:16.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 701675 10.90 MB/sec execute 213 sec latency 30.033 ms 2024-03-24T10:47:16.188 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:47:16.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:47:16.188 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:47:16.266 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:16 smithi117 systemd[1]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.c.service: Scheduled restart job, restart counter is at 2. 2024-03-24T10:47:16.267 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:16 smithi117 systemd[1]: Stopped Ceph mds.c for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:47:16.267 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:16 smithi117 systemd[1]: Starting Ceph mds.c for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:47:16.267 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:16 smithi117 podman[71547]: 2024-03-24 10:47:16.142003349 +0000 UTC m=+0.019315430 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:47:16.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.304+0000 7f737006f640 1 -- 172.21.15.23:0/2819139965 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73681032a0 msgr2=0x7f736810b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.304+0000 7f737006f640 1 --2- 172.21.15.23:0/2819139965 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73681032a0 0x7f736810b780 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f7364009900 tx=0x7f736402bcd0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.304+0000 7f737006f640 1 -- 172.21.15.23:0/2819139965 shutdown_connections 2024-03-24T10:47:16.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.304+0000 7f737006f640 1 --2- 172.21.15.23:0/2819139965 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f736810bd50 0x7f736810e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.304+0000 7f737006f640 1 --2- 172.21.15.23:0/2819139965 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73681032a0 0x7f736810b780 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.304+0000 7f737006f640 1 --2- 172.21.15.23:0/2819139965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7368102980 0x7f7368102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.304+0000 7f737006f640 1 -- 172.21.15.23:0/2819139965 >> 172.21.15.23:0/2819139965 conn(0x7f7368077770 msgr2=0x7f7368077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:16.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.304+0000 7f737006f640 1 -- 172.21.15.23:0/2819139965 shutdown_connections 2024-03-24T10:47:16.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.304+0000 7f737006f640 1 -- 172.21.15.23:0/2819139965 wait complete. 2024-03-24T10:47:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.304+0000 7f737006f640 1 Processor -- start 2024-03-24T10:47:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.304+0000 7f737006f640 1 -- start start 2024-03-24T10:47:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f737006f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7368102980 0x7f7368197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f737006f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73681032a0 0x7f73681983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f737006f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f736810bd50 0x7f736819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f737006f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f736819ccf0 con 0x7f7368102980 2024-03-24T10:47:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f737006f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f736819ce60 con 0x7f73681032a0 2024-03-24T10:47:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f737006f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f736819d020 con 0x7f736810bd50 2024-03-24T10:47:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f736de0b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7368102980 0x7f7368197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f736de0b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7368102980 0x7f7368197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45340/0 (socket says 172.21.15.23:45340) 2024-03-24T10:47:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f736de0b640 1 -- 172.21.15.23:0/2695223310 learned_addr learned my addr 172.21.15.23:0/2695223310 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f736d60a640 1 --2- 172.21.15.23:0/2695223310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73681032a0 0x7f73681983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f736e60c640 1 --2- 172.21.15.23:0/2695223310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f736810bd50 0x7f736819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f736de0b640 1 -- 172.21.15.23:0/2695223310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f736810bd50 msgr2=0x7f736819c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f736de0b640 1 --2- 172.21.15.23:0/2695223310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f736810bd50 0x7f736819c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f736de0b640 1 -- 172.21.15.23:0/2695223310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73681032a0 msgr2=0x7f73681983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f736de0b640 1 --2- 172.21.15.23:0/2695223310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73681032a0 0x7f73681983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f736de0b640 1 -- 172.21.15.23:0/2695223310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7364009560 con 0x7f7368102980 2024-03-24T10:47:16.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f736de0b640 1 --2- 172.21.15.23:0/2695223310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7368102980 0x7f7368197e80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f735c00e9f0 tx=0x7f735c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:16.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f735affd640 1 -- 172.21.15.23:0/2695223310 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c009700 con 0x7f7368102980 2024-03-24T10:47:16.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f735affd640 1 -- 172.21.15.23:0/2695223310 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f735c00cd80 con 0x7f7368102980 2024-03-24T10:47:16.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f737006f640 1 -- 172.21.15.23:0/2695223310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f736819d300 con 0x7f7368102980 2024-03-24T10:47:16.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f735affd640 1 -- 172.21.15.23:0/2695223310 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c018920 con 0x7f7368102980 2024-03-24T10:47:16.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.305+0000 7f737006f640 1 -- 172.21.15.23:0/2695223310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73681a4b80 con 0x7f7368102980 2024-03-24T10:47:16.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.307+0000 7f735affd640 1 -- 172.21.15.23:0/2695223310 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f735c018a80 con 0x7f7368102980 2024-03-24T10:47:16.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.308+0000 7f735affd640 1 --2- 172.21.15.23:0/2695223310 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f73540a2230 0x7f73540a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.308+0000 7f7372c53740 1 Processor -- start 2024-03-24T10:47:16.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.308+0000 7f735affd640 1 -- 172.21.15.23:0/2695223310 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f735c014070 con 0x7f7368102980 2024-03-24T10:47:16.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.308+0000 7f736d60a640 1 --2- 172.21.15.23:0/2695223310 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f73540a2230 0x7f73540a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.308+0000 7f7372c53740 1 -- start start 2024-03-24T10:47:16.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.308+0000 7f7372c53740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbeb760 0x55638dbebb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.308+0000 7f7372c53740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbec080 0x55638dbf4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.308+0000 7f7372c53740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 0x55638dbf7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f7372c53740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55638db59530 con 0x55638dbec080 2024-03-24T10:47:16.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f7372c53740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55638db56a60 con 0x55638dbeb760 2024-03-24T10:47:16.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f7372c53740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55638db52240 con 0x55638dbf4c60 2024-03-24T10:47:16.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f736d60a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbec080 0x55638dbf4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f736d60a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbec080 0x55638dbf4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45346/0 (socket says 172.21.15.23:45346) 2024-03-24T10:47:16.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f736d60a640 1 -- 172.21.15.23:0/3410544751 learned_addr learned my addr 172.21.15.23:0/3410544751 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:16.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f736de0b640 1 --2- 172.21.15.23:0/3410544751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbeb760 0x55638dbebb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f736d60a640 1 --2- 172.21.15.23:0/2695223310 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f73540a2230 0x7f73540a46f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7364006010 tx=0x7f7364005e40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:16.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f736e60c640 1 --2- 172.21.15.23:0/3410544751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 0x55638dbf7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f736d60a640 1 -- 172.21.15.23:0/3410544751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 msgr2=0x55638dbf7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f736d60a640 1 --2- 172.21.15.23:0/3410544751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 0x55638dbf7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f736d60a640 1 -- 172.21.15.23:0/3410544751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbeb760 msgr2=0x55638dbebb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f736d60a640 1 --2- 172.21.15.23:0/3410544751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbeb760 0x55638dbebb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f736d60a640 1 -- 172.21.15.23:0/3410544751 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55638db508c0 con 0x55638dbec080 2024-03-24T10:47:16.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f736d60a640 1 --2- 172.21.15.23:0/3410544751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbec080 0x55638dbf4720 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f7364035000 tx=0x7f736403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457b8d394efc4b58 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f7353fff640 1 -- 172.21.15.23:0/3410544751 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736402cb00 con 0x55638dbec080 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f7353fff640 1 -- 172.21.15.23:0/3410544751 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f736402cc60 con 0x55638dbec080 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.309+0000 7f7353fff640 1 -- 172.21.15.23:0/3410544751 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364003850 con 0x55638dbec080 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.310+0000 7f7372c53740 1 -- 172.21.15.23:0/3410544751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbec080 msgr2=0x55638dbf4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.310+0000 7f7372c53740 1 --2- 172.21.15.23:0/3410544751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbec080 0x55638dbf4720 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f7364035000 tx=0x7f736403f000 comp rx=0 tx=0).stop 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.310+0000 7f7372c53740 1 -- 172.21.15.23:0/3410544751 shutdown_connections 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.310+0000 7f7372c53740 1 --2- 172.21.15.23:0/3410544751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 0x55638dbf7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.310+0000 7f7372c53740 1 --2- 172.21.15.23:0/3410544751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbec080 0x55638dbf4720 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.310+0000 7f7372c53740 1 --2- 172.21.15.23:0/3410544751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbeb760 0x55638dbebb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.310+0000 7f7372c53740 1 -- 172.21.15.23:0/3410544751 >> 172.21.15.23:0/3410544751 conn(0x55638dbe5e90 msgr2=0x55638dbe6270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.310+0000 7f7372c53740 1 -- 172.21.15.23:0/3410544751 shutdown_connections 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.310+0000 7f7372c53740 1 -- 172.21.15.23:0/3410544751 wait complete. 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.310+0000 7f7372c53740 1 -- start start 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.310+0000 7f7372c53740 1 Processor -- start 2024-03-24T10:47:16.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f7372c53740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbeb760 0x55638dbf1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f7372c53740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbec080 0x55638dbf2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f7372c53740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 0x55638dbf26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f7372c53740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55638daf6f80 con 0x55638dbeb760 2024-03-24T10:47:16.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f7372c53740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55638d8254b0 con 0x55638dbec080 2024-03-24T10:47:16.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f7372c53740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55638dad7ad0 con 0x55638dbf4c60 2024-03-24T10:47:16.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f736de0b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbeb760 0x55638dbf1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f7372c53740 10 client.0 _ll_register_callbacks cb 0x55638db0ab70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:16.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f736de0b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbeb760 0x55638dbf1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45362/0 (socket says 172.21.15.23:45362) 2024-03-24T10:47:16.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f736de0b640 1 -- 172.21.15.23:0/1433741066 learned_addr learned my addr 172.21.15.23:0/1433741066 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:16.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f736d60a640 1 --2- 172.21.15.23:0/1433741066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbec080 0x55638dbf2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f736e60c640 1 --2- 172.21.15.23:0/1433741066 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 0x55638dbf26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f736de0b640 1 -- 172.21.15.23:0/1433741066 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 msgr2=0x55638dbf26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f736de0b640 1 --2- 172.21.15.23:0/1433741066 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 0x55638dbf26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f736de0b640 1 -- 172.21.15.23:0/1433741066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbec080 msgr2=0x55638dbf2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f736de0b640 1 --2- 172.21.15.23:0/1433741066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbec080 0x55638dbf2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f736de0b640 1 -- 172.21.15.23:0/1433741066 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f736810ee40 con 0x55638dbeb760 2024-03-24T10:47:16.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f736de0b640 1 --2- 172.21.15.23:0/1433741066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbeb760 0x55638dbf1c40 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f735c09abe0 tx=0x7f735c09b020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59bffcb7afd297d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:16.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f73517fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:16.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.311+0000 7f73517fa640 1 -- 172.21.15.23:0/1433741066 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c010c40 con 0x55638dbeb760 2024-03-24T10:47:16.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.312+0000 7f73517fa640 1 -- 172.21.15.23:0/1433741066 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f735c097550 con 0x55638dbeb760 2024-03-24T10:47:16.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.312+0000 7f7372c53740 1 -- 172.21.15.23:0/1433741066 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55638db767a0 con 0x55638dbeb760 2024-03-24T10:47:16.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.312+0000 7f73517fa640 1 -- 172.21.15.23:0/1433741066 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f735c097710 con 0x55638dbeb760 2024-03-24T10:47:16.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.312+0000 7f7372c53740 10 client.17640 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:16.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.312+0000 7f7372c53740 1 -- 172.21.15.23:0/1433741066 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55638db086d0 con 0x55638dbeb760 2024-03-24T10:47:16.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.312+0000 7f73517fa640 1 -- 172.21.15.23:0/1433741066 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55638db767a0 con 0x55638dbeb760 2024-03-24T10:47:16.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.312+0000 7f73517fa640 1 -- 172.21.15.23:0/1433741066 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f735c09a430 con 0x55638dbeb760 2024-03-24T10:47:16.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.312+0000 7f7372c53740 10 client.17640 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:16.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.312+0000 7f7372c53740 10 client.17640 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:16.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.312+0000 7f7372c53740 20 client.17640 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:16.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.312+0000 7f7372c53740 1 --2- 172.21.15.23:0/1433741066 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55638dc04bc0 0x55638dc06fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.312+0000 7f7372c53740 4 client.17640 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:16.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.312+0000 7f7372c53740 1 -- 172.21.15.23:0/1433741066 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55638db481f0 con 0x55638dc04bc0 2024-03-24T10:47:16.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.312+0000 7f736d60a640 1 --2- 172.21.15.23:0/1433741066 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55638dc04bc0 0x55638dc06fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.313+0000 7f736d60a640 1 --2- 172.21.15.23:0/1433741066 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55638dc04bc0 0x55638dc06fb0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f7f8ae25ef17587 server_cookie=79039b1a536b2989 in_seq=0 out_seq=0 2024-03-24T10:47:16.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.313+0000 7f73517fa640 10 client.17640 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:16.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f73517fa640 1 -- 172.21.15.23:0/1433741066 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55638db481f0 con 0x55638dc04bc0 2024-03-24T10:47:16.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f73517fa640 10 client.17640 handle_command_reply: tid=0 2024-03-24T10:47:16.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f7372c53740 1 client.17640 shutdown 2024-03-24T10:47:16.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f7372c53740 1 -- 172.21.15.23:0/1433741066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbeb760 msgr2=0x55638dbf1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f7372c53740 1 --2- 172.21.15.23:0/1433741066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbeb760 0x55638dbf1c40 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f735c09abe0 tx=0x7f735c09b020 comp rx=0 tx=0).stop 2024-03-24T10:47:16.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f7372c53740 1 -- 172.21.15.23:0/1433741066 shutdown_connections 2024-03-24T10:47:16.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f7372c53740 1 --2- 172.21.15.23:0/1433741066 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55638dc04bc0 0x55638dc06fb0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f7372c53740 1 --2- 172.21.15.23:0/1433741066 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 0x55638dbf26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f7372c53740 1 --2- 172.21.15.23:0/1433741066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbec080 0x55638dbf2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f7372c53740 1 --2- 172.21.15.23:0/1433741066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbeb760 0x55638dbf1c40 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f73517fa640 0 client.17640 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:16.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f7372c53740 1 -- 172.21.15.23:0/1433741066 >> 172.21.15.23:0/1433741066 conn(0x55638dbe5e90 msgr2=0x55638dbe7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:16.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f7372c53740 1 -- 172.21.15.23:0/1433741066 shutdown_connections 2024-03-24T10:47:16.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f7372c53740 1 -- 172.21.15.23:0/1433741066 wait complete. 2024-03-24T10:47:16.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.314+0000 7f7372c53740 20 client.17640 trim_cache size 0 max 16384 2024-03-24T10:47:16.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.332+0000 7f7372c53740 1 Processor -- start 2024-03-24T10:47:16.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.332+0000 7f7372c53740 1 -- start start 2024-03-24T10:47:16.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.332+0000 7f7372c53740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbeb760 0x55638dc003d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.332+0000 7f7372c53740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbec080 0x55638dbf24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.332+0000 7f7372c53740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbf4c60 0x55638dc0c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.332+0000 7f7372c53740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55638dc00940 con 0x55638dbf4c60 2024-03-24T10:47:16.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.332+0000 7f7372c53740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55638dbeb3f0 con 0x55638dbec080 2024-03-24T10:47:16.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.332+0000 7f7372c53740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55638dbed8d0 con 0x55638dbeb760 2024-03-24T10:47:16.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f736e60c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbf4c60 0x55638dc0c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f736e60c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbf4c60 0x55638dc0c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45376/0 (socket says 172.21.15.23:45376) 2024-03-24T10:47:16.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f736e60c640 1 -- 172.21.15.23:0/1305605111 learned_addr learned my addr 172.21.15.23:0/1305605111 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:16.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f736de0b640 1 --2- 172.21.15.23:0/1305605111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbeb760 0x55638dc003d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f736d60a640 1 --2- 172.21.15.23:0/1305605111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbec080 0x55638dbf24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f736e60c640 1 -- 172.21.15.23:0/1305605111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbeb760 msgr2=0x55638dc003d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f736e60c640 1 --2- 172.21.15.23:0/1305605111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbeb760 0x55638dc003d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f736e60c640 1 -- 172.21.15.23:0/1305605111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbec080 msgr2=0x55638dbf24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f736e60c640 1 --2- 172.21.15.23:0/1305605111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbec080 0x55638dbf24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f736e60c640 1 -- 172.21.15.23:0/1305605111 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55638daecc60 con 0x55638dbf4c60 2024-03-24T10:47:16.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f736de0b640 1 --2- 172.21.15.23:0/1305605111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbeb760 0x55638dc003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:16.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f736e60c640 1 --2- 172.21.15.23:0/1305605111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbf4c60 0x55638dc0c890 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f736000a480 tx=0x7f7360009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5515f7bbd6202e38 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:16.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f73537fe640 1 -- 172.21.15.23:0/1305605111 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7360010040 con 0x55638dbf4c60 2024-03-24T10:47:16.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f73537fe640 1 -- 172.21.15.23:0/1305605111 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7360017d10 con 0x55638dbf4c60 2024-03-24T10:47:16.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.333+0000 7f73537fe640 1 -- 172.21.15.23:0/1305605111 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736001fe70 con 0x55638dbf4c60 2024-03-24T10:47:16.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.334+0000 7f7372c53740 1 -- 172.21.15.23:0/1305605111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbf4c60 msgr2=0x55638dc0c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.334+0000 7f7372c53740 1 --2- 172.21.15.23:0/1305605111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbf4c60 0x55638dc0c890 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f736000a480 tx=0x7f7360009650 comp rx=0 tx=0).stop 2024-03-24T10:47:16.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.334+0000 7f7372c53740 1 -- 172.21.15.23:0/1305605111 shutdown_connections 2024-03-24T10:47:16.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.334+0000 7f7372c53740 1 --2- 172.21.15.23:0/1305605111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbf4c60 0x55638dc0c890 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.334+0000 7f7372c53740 1 --2- 172.21.15.23:0/1305605111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbec080 0x55638dbf24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.334+0000 7f7372c53740 1 --2- 172.21.15.23:0/1305605111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbeb760 0x55638dc003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.334+0000 7f7372c53740 1 -- 172.21.15.23:0/1305605111 >> 172.21.15.23:0/1305605111 conn(0x55638dbe5e90 msgr2=0x55638dbe4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:16.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.334+0000 7f7372c53740 1 -- 172.21.15.23:0/1305605111 shutdown_connections 2024-03-24T10:47:16.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.334+0000 7f7372c53740 1 -- 172.21.15.23:0/1305605111 wait complete. 2024-03-24T10:47:16.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.334+0000 7f7372c53740 1 -- start start 2024-03-24T10:47:16.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.334+0000 7f7372c53740 1 Processor -- start 2024-03-24T10:47:16.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f7372c53740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbeb760 0x55638dbfccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f7372c53740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbec080 0x55638dbf99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f7372c53740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 0x55638dbf9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f7372c53740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55638dbfa5e0 con 0x55638dbec080 2024-03-24T10:47:16.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f7372c53740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55638dbfa750 con 0x55638dbeb760 2024-03-24T10:47:16.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f7372c53740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55638dc6aed0 con 0x55638dbf4c60 2024-03-24T10:47:16.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f7372c53740 10 client.0 _ll_register_callbacks cb 0x7f735c09a840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:16.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f736d60a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbec080 0x55638dbf99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f736d60a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbec080 0x55638dbf99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45388/0 (socket says 172.21.15.23:45388) 2024-03-24T10:47:16.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f736de0b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbeb760 0x55638dbfccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f736d60a640 1 -- 172.21.15.23:0/299452597 learned_addr learned my addr 172.21.15.23:0/299452597 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:16.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f736e60c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 0x55638dbf9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f736d60a640 1 -- 172.21.15.23:0/299452597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 msgr2=0x55638dbf9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f736d60a640 1 --2- 172.21.15.23:0/299452597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 0x55638dbf9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f736d60a640 1 -- 172.21.15.23:0/299452597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbeb760 msgr2=0x55638dbfccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f736d60a640 1 --2- 172.21.15.23:0/299452597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbeb760 0x55638dbfccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f736d60a640 1 -- 172.21.15.23:0/299452597 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55638db086d0 con 0x55638dbec080 2024-03-24T10:47:16.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f736de0b640 1 --2- 172.21.15.23:0/299452597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbeb760 0x55638dbfccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:16.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f736e60c640 1 --2- 172.21.15.23:0/299452597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 0x55638dbf9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:16.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f736d60a640 1 --2- 172.21.15.23:0/299452597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbec080 0x55638dbf99d0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7364035000 tx=0x7f73640053c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef4d29afa8af5ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:16.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f73517fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:16.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f73517fa640 1 -- 172.21.15.23:0/299452597 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736405eaa0 con 0x55638dbec080 2024-03-24T10:47:16.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f73517fa640 1 -- 172.21.15.23:0/299452597 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f736405b070 con 0x55638dbec080 2024-03-24T10:47:16.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f7372c53740 1 -- 172.21.15.23:0/299452597 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f736402cc60 con 0x55638dbec080 2024-03-24T10:47:16.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.335+0000 7f73517fa640 1 -- 172.21.15.23:0/299452597 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f736405ec00 con 0x55638dbec080 2024-03-24T10:47:16.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.336+0000 7f7372c53740 10 client.17646 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:16.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.336+0000 7f7372c53740 1 -- 172.21.15.23:0/299452597 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x7f7360009d20 con 0x55638dbec080 2024-03-24T10:47:16.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.336+0000 7f73517fa640 1 -- 172.21.15.23:0/299452597 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f736402cc60 con 0x55638dbec080 2024-03-24T10:47:16.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.336+0000 7f73517fa640 1 -- 172.21.15.23:0/299452597 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f7364067070 con 0x55638dbec080 2024-03-24T10:47:16.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.336+0000 7f7372c53740 10 client.17646 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:16.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.336+0000 7f7372c53740 10 client.17646 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:16.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.336+0000 7f7372c53740 20 client.17646 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:16.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.336+0000 7f7372c53740 1 --2- 172.21.15.23:0/299452597 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55638dc6c1f0 0x55638dc6e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.336+0000 7f7372c53740 4 client.17646 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:47:16.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.336+0000 7f7372c53740 1 -- 172.21.15.23:0/299452597 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55638d851860 con 0x55638dc6c1f0 2024-03-24T10:47:16.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.336+0000 7f736de0b640 1 --2- 172.21.15.23:0/299452597 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55638dc6c1f0 0x55638dc6e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.336+0000 7f73517fa640 10 client.17646 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.336+0000 7f736de0b640 1 --2- 172.21.15.23:0/299452597 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55638dc6c1f0 0x55638dc6e610 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a2acc15f6cfa372 server_cookie=1d71527c7ff7ed57 in_seq=0 out_seq=0 2024-03-24T10:47:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f73517fa640 1 -- 172.21.15.23:0/299452597 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55638d851860 con 0x55638dc6c1f0 2024-03-24T10:47:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f73517fa640 10 client.17646 handle_command_reply: tid=0 2024-03-24T10:47:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f7372c53740 1 client.17646 shutdown 2024-03-24T10:47:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f7372c53740 1 -- 172.21.15.23:0/299452597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbec080 msgr2=0x55638dbf99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f7372c53740 1 --2- 172.21.15.23:0/299452597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbec080 0x55638dbf99d0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7364035000 tx=0x7f73640053c0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f7372c53740 1 -- 172.21.15.23:0/299452597 shutdown_connections 2024-03-24T10:47:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f7372c53740 1 --2- 172.21.15.23:0/299452597 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55638dc6c1f0 0x55638dc6e610 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f7372c53740 1 --2- 172.21.15.23:0/299452597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55638dbf4c60 0x55638dbf9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f7372c53740 1 --2- 172.21.15.23:0/299452597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55638dbec080 0x55638dbf99d0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f7372c53740 1 --2- 172.21.15.23:0/299452597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55638dbeb760 0x55638dbfccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f73517fa640 0 client.17646 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f7372c53740 1 -- 172.21.15.23:0/299452597 >> 172.21.15.23:0/299452597 conn(0x55638dbe5e90 msgr2=0x55638dbe7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:16.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f7372c53740 1 -- 172.21.15.23:0/299452597 shutdown_connections 2024-03-24T10:47:16.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f7372c53740 1 -- 172.21.15.23:0/299452597 wait complete. 2024-03-24T10:47:16.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f7372c53740 20 client.17646 trim_cache size 0 max 16384 2024-03-24T10:47:16.346 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:47:16.346 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:47:16.346 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "561a5177-9074-4cb5-9610-3e6bcf099ad6", 2024-03-24T10:47:16.346 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:47:16.346 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:47:16.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f737006f640 1 -- 172.21.15.23:0/2695223310 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f73540a2230 msgr2=0x7f73540a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f737006f640 1 --2- 172.21.15.23:0/2695223310 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f73540a2230 0x7f73540a46f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7364006010 tx=0x7f7364005e40 comp rx=0 tx=0).stop 2024-03-24T10:47:16.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f737006f640 1 -- 172.21.15.23:0/2695223310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7368102980 msgr2=0x7f7368197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.338+0000 7f737006f640 1 --2- 172.21.15.23:0/2695223310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7368102980 0x7f7368197e80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f735c00e9f0 tx=0x7f735c00eeb0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.339+0000 7f737006f640 1 -- 172.21.15.23:0/2695223310 shutdown_connections 2024-03-24T10:47:16.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.339+0000 7f737006f640 1 --2- 172.21.15.23:0/2695223310 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f73540a2230 0x7f73540a46f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.339+0000 7f737006f640 1 --2- 172.21.15.23:0/2695223310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f736810bd50 0x7f736819c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.339+0000 7f737006f640 1 --2- 172.21.15.23:0/2695223310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73681032a0 0x7f73681983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.339+0000 7f737006f640 1 --2- 172.21.15.23:0/2695223310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7368102980 0x7f7368197e80 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.339+0000 7f737006f640 1 -- 172.21.15.23:0/2695223310 >> 172.21.15.23:0/2695223310 conn(0x7f7368077770 msgr2=0x7f73680fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:16.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.339+0000 7f737006f640 1 -- 172.21.15.23:0/2695223310 shutdown_connections 2024-03-24T10:47:16.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.339+0000 7f737006f640 1 -- 172.21.15.23:0/2695223310 wait complete. 2024-03-24T10:47:16.350 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:47:16.350 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:47:16.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.458+0000 7f09c49b5640 1 -- 172.21.15.23:0/3112341339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c0068910 msgr2=0x7f09c0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.458+0000 7f09c49b5640 1 --2- 172.21.15.23:0/3112341339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c0068910 0x7f09c0068d90 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009930 tx=0x7f09b402bc30 comp rx=0 tx=0).stop 2024-03-24T10:47:16.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.458+0000 7f09c49b5640 1 -- 172.21.15.23:0/3112341339 shutdown_connections 2024-03-24T10:47:16.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.458+0000 7f09c49b5640 1 --2- 172.21.15.23:0/3112341339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c00692d0 0x7f09c010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.458+0000 7f09c49b5640 1 --2- 172.21.15.23:0/3112341339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c0068910 0x7f09c0068d90 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.458+0000 7f09c49b5640 1 --2- 172.21.15.23:0/3112341339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09c0107760 0x7f09c0107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.458+0000 7f09c49b5640 1 -- 172.21.15.23:0/3112341339 >> 172.21.15.23:0/3112341339 conn(0x7f09c0077710 msgr2=0x7f09c0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:16.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.458+0000 7f09c49b5640 1 -- 172.21.15.23:0/3112341339 shutdown_connections 2024-03-24T10:47:16.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.458+0000 7f09c49b5640 1 -- 172.21.15.23:0/3112341339 wait complete. 2024-03-24T10:47:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09c49b5640 1 Processor -- start 2024-03-24T10:47:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09c49b5640 1 -- start start 2024-03-24T10:47:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09c49b5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09c0068910 0x7f09c019da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09c49b5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c00692d0 0x7f09c019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09c49b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c0107760 0x7f09c0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09c49b5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f09c01982c0 con 0x7f09c00692d0 2024-03-24T10:47:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09c49b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f09c0198430 con 0x7f09c0107760 2024-03-24T10:47:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09c49b5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f09c01985f0 con 0x7f09c0068910 2024-03-24T10:47:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09bdd9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c00692d0 0x7f09c019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09bdd9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c00692d0 0x7f09c019dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45406/0 (socket says 172.21.15.23:45406) 2024-03-24T10:47:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09bdd9b640 1 -- 172.21.15.23:0/2081341516 learned_addr learned my addr 172.21.15.23:0/2081341516 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09be59c640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09c0068910 0x7f09c019da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09bed9d640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c0107760 0x7f09c0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09bdd9b640 1 -- 172.21.15.23:0/2081341516 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09c0068910 msgr2=0x7f09c019da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09bdd9b640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09c0068910 0x7f09c019da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09bdd9b640 1 -- 172.21.15.23:0/2081341516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c0107760 msgr2=0x7f09c0197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09bdd9b640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c0107760 0x7f09c0197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09bdd9b640 1 -- 172.21.15.23:0/2081341516 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b4009560 con 0x7f09c00692d0 2024-03-24T10:47:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09be59c640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09c0068910 0x7f09c019da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.459+0000 7f09bdd9b640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c00692d0 0x7f09c019dfb0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009230 tx=0x7f09b4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.460+0000 7f09af7fe640 1 -- 172.21.15.23:0/2081341516 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b40042d0 con 0x7f09c00692d0 2024-03-24T10:47:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.460+0000 7f09c49b5640 1 -- 172.21.15.23:0/2081341516 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c0198870 con 0x7f09c00692d0 2024-03-24T10:47:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.460+0000 7f09af7fe640 1 -- 172.21.15.23:0/2081341516 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09b4004430 con 0x7f09c00692d0 2024-03-24T10:47:16.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.460+0000 7f09af7fe640 1 -- 172.21.15.23:0/2081341516 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4003c90 con 0x7f09c00692d0 2024-03-24T10:47:16.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.460+0000 7f09c49b5640 1 -- 172.21.15.23:0/2081341516 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c01a49f0 con 0x7f09c00692d0 2024-03-24T10:47:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.462+0000 7f09af7fe640 1 -- 172.21.15.23:0/2081341516 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f09b4005560 con 0x7f09c00692d0 2024-03-24T10:47:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.462+0000 7f09c7598740 1 Processor -- start 2024-03-24T10:47:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.462+0000 7f09c7598740 1 -- start start 2024-03-24T10:47:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09c7598740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557ee1d0 0x55a8557ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09c7598740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557eeaf0 0x55a8557f7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09af7fe640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f09a80a2230 0x7f09a80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09c7598740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f76d0 0x55a8557f9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09c7598740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a855712f30 con 0x55a8557f76d0 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09c7598740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a85575c810 con 0x55a8557ee1d0 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09c7598740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a855759d40 con 0x55a8557eeaf0 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09bed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557ee1d0 0x55a8557ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09be59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557eeaf0 0x55a8557f7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09be59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557eeaf0 0x55a8557f7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:38880/0 (socket says 172.21.15.23:38880) 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09bed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557ee1d0 0x55a8557ee5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:55006/0 (socket says 172.21.15.23:55006) 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09be59c640 1 -- 172.21.15.23:0/2145080623 learned_addr learned my addr 172.21.15.23:0/2145080623 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09af7fe640 1 -- 172.21.15.23:0/2081341516 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f09b40efff0 con 0x7f09c00692d0 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09bdd9b640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f09a80a2230 0x7f09a80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09bed9d640 1 --2- 172.21.15.23:0/2145080623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f76d0 0x55a8557f9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09be59c640 1 -- 172.21.15.23:0/2145080623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557ee1d0 msgr2=0x55a8557ee5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09be59c640 1 --2- 172.21.15.23:0/2145080623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557ee1d0 0x55a8557ee5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09be59c640 1 -- 172.21.15.23:0/2145080623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f76d0 msgr2=0x55a8557f9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09be59c640 1 --2- 172.21.15.23:0/2145080623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f76d0 0x55a8557f9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09be59c640 1 -- 172.21.15.23:0/2145080623 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a855716ea0 con 0x55a8557eeaf0 2024-03-24T10:47:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.463+0000 7f09bed9d640 1 --2- 172.21.15.23:0/2145080623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f76d0 0x55a8557f9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:47:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.464+0000 7f09be59c640 1 --2- 172.21.15.23:0/2145080623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557eeaf0 0x55a8557f7190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f09b8005f40 tx=0x7f09b8011d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ed043d59aea8f88 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.464+0000 7f09bdd9b640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f09a80a2230 0x7f09a80a46f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f09b40ed8d0 tx=0x7f09b4003760 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.468+0000 7f09acff9640 1 -- 172.21.15.23:0/2145080623 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b8011330 con 0x55a8557eeaf0 2024-03-24T10:47:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.468+0000 7f09acff9640 1 -- 172.21.15.23:0/2145080623 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09b800fe80 con 0x55a8557eeaf0 2024-03-24T10:47:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.468+0000 7f09acff9640 1 -- 172.21.15.23:0/2145080623 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b80193e0 con 0x55a8557eeaf0 2024-03-24T10:47:16.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.468+0000 7f09c7598740 1 -- 172.21.15.23:0/2145080623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557eeaf0 msgr2=0x55a8557f7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.468+0000 7f09c7598740 1 --2- 172.21.15.23:0/2145080623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557eeaf0 0x55a8557f7190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f09b8005f40 tx=0x7f09b8011d30 comp rx=0 tx=0).stop 2024-03-24T10:47:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 -- 172.21.15.23:0/2145080623 shutdown_connections 2024-03-24T10:47:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 --2- 172.21.15.23:0/2145080623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f76d0 0x55a8557f9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 --2- 172.21.15.23:0/2145080623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557eeaf0 0x55a8557f7190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 --2- 172.21.15.23:0/2145080623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557ee1d0 0x55a8557ee5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 -- 172.21.15.23:0/2145080623 >> 172.21.15.23:0/2145080623 conn(0x55a8557e8900 msgr2=0x55a8557e8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 -- 172.21.15.23:0/2145080623 shutdown_connections 2024-03-24T10:47:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 -- 172.21.15.23:0/2145080623 wait complete. 2024-03-24T10:47:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 -- start start 2024-03-24T10:47:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 Processor -- start 2024-03-24T10:47:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557ee1d0 0x55a8557f4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f76d0 0x55a8557f4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f5280 0x55a855803630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a855755520 con 0x55a8557f5280 2024-03-24T10:47:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a855428430 con 0x55a8557f76d0 2024-03-24T10:47:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a8553e4e00 con 0x55a8557ee1d0 2024-03-24T10:47:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09c7598740 10 client.0 _ll_register_callbacks cb 0x55a8556f3d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09bed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f5280 0x55a855803630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09bed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f5280 0x55a855803630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45420/0 (socket says 172.21.15.23:45420) 2024-03-24T10:47:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09bed9d640 1 -- 172.21.15.23:0/4170752064 learned_addr learned my addr 172.21.15.23:0/4170752064 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09be59c640 1 --2- 172.21.15.23:0/4170752064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f76d0 0x55a8557f4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.469+0000 7f09bed9d640 1 --2- 172.21.15.23:0/4170752064 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557ee1d0 0x55a8557f4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.470+0000 7f09bed9d640 1 -- 172.21.15.23:0/4170752064 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557ee1d0 msgr2=0x55a8557f4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.470+0000 7f09bed9d640 1 --2- 172.21.15.23:0/4170752064 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557ee1d0 0x55a8557f4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.470+0000 7f09bed9d640 1 -- 172.21.15.23:0/4170752064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f76d0 msgr2=0x55a8557f4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.470+0000 7f09bed9d640 1 --2- 172.21.15.23:0/4170752064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f76d0 0x55a8557f4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.470+0000 7f09bed9d640 1 -- 172.21.15.23:0/4170752064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09b8011500 con 0x55a8557f5280 2024-03-24T10:47:16.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.470+0000 7f09bed9d640 1 --2- 172.21.15.23:0/4170752064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f5280 0x55a855803630 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f09b0011e40 tx=0x7f09b0009aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c95668b36ab72fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:16.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.470+0000 7f09a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:16.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.470+0000 7f09a5ffb640 1 -- 172.21.15.23:0/4170752064 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b0016040 con 0x55a8557f5280 2024-03-24T10:47:16.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.470+0000 7f09a5ffb640 1 -- 172.21.15.23:0/4170752064 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f09b0011030 con 0x55a8557f5280 2024-03-24T10:47:16.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.470+0000 7f09c7598740 1 -- 172.21.15.23:0/4170752064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a8554547e0 con 0x55a8557f5280 2024-03-24T10:47:16.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.470+0000 7f09a5ffb640 1 -- 172.21.15.23:0/4170752064 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f09b000fed0 con 0x55a8557f5280 2024-03-24T10:47:16.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.470+0000 7f09c7598740 10 client.17658 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:16.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.470+0000 7f09c7598740 1 -- 172.21.15.23:0/4170752064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a855753ba0 con 0x55a8557f5280 2024-03-24T10:47:16.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.471+0000 7f09a5ffb640 1 -- 172.21.15.23:0/4170752064 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09b002e020 con 0x55a8557f5280 2024-03-24T10:47:16.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.471+0000 7f09a5ffb640 1 -- 172.21.15.23:0/4170752064 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f09b0010430 con 0x55a8557f5280 2024-03-24T10:47:16.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.471+0000 7f09c7598740 10 client.17658 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:16.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.471+0000 7f09c7598740 10 client.17658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:16.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.471+0000 7f09c7598740 20 client.17658 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:16.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.471+0000 7f09c7598740 1 --2- 172.21.15.23:0/4170752064 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a855808120 0x55a85580a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.471+0000 7f09c7598740 4 client.17658 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:16.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.471+0000 7f09c7598740 1 -- 172.21.15.23:0/4170752064 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a855491720 con 0x55a855808120 2024-03-24T10:47:16.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.471+0000 7f09be59c640 1 --2- 172.21.15.23:0/4170752064 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a855808120 0x55a85580a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.471+0000 7f09a5ffb640 10 client.17658 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:16.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.471+0000 7f09be59c640 1 --2- 172.21.15.23:0/4170752064 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a855808120 0x55a85580a540 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2ae0b231604d52d server_cookie=c6152592945a2f85 in_seq=0 out_seq=0 2024-03-24T10:47:16.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.472+0000 7f09a5ffb640 1 -- 172.21.15.23:0/4170752064 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a855491720 con 0x55a855808120 2024-03-24T10:47:16.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.472+0000 7f09a5ffb640 10 client.17658 handle_command_reply: tid=0 2024-03-24T10:47:16.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.472+0000 7f09c7598740 1 client.17658 shutdown 2024-03-24T10:47:16.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.472+0000 7f09c7598740 1 -- 172.21.15.23:0/4170752064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f5280 msgr2=0x55a855803630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.472+0000 7f09c7598740 1 --2- 172.21.15.23:0/4170752064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f5280 0x55a855803630 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f09b0011e40 tx=0x7f09b0009aa0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.472+0000 7f09c7598740 1 -- 172.21.15.23:0/4170752064 shutdown_connections 2024-03-24T10:47:16.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.472+0000 7f09c7598740 1 --2- 172.21.15.23:0/4170752064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f5280 0x55a855803630 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.472+0000 7f09c7598740 1 --2- 172.21.15.23:0/4170752064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f76d0 0x55a8557f4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.472+0000 7f09c7598740 1 --2- 172.21.15.23:0/4170752064 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a855808120 0x55a85580a540 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.472+0000 7f09c7598740 1 --2- 172.21.15.23:0/4170752064 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557ee1d0 0x55a8557f4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.472+0000 7f09a5ffb640 0 client.17658 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:16.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.472+0000 7f09c7598740 1 -- 172.21.15.23:0/4170752064 >> 172.21.15.23:0/4170752064 conn(0x55a8557e8900 msgr2=0x55a8557e7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:16.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.472+0000 7f09c7598740 1 -- 172.21.15.23:0/4170752064 shutdown_connections 2024-03-24T10:47:16.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.472+0000 7f09c7598740 1 -- 172.21.15.23:0/4170752064 wait complete. 2024-03-24T10:47:16.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.473+0000 7f09c7598740 20 client.17658 trim_cache size 0 max 16384 2024-03-24T10:47:16.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09c7598740 1 Processor -- start 2024-03-24T10:47:16.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09c7598740 1 -- start start 2024-03-24T10:47:16.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09c7598740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557ee1d0 0x55a8557e79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09c7598740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557f5280 0x55a85580fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09c7598740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f76d0 0x55a8557ff580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09c7598740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a8556da430 con 0x55a8557ee1d0 2024-03-24T10:47:16.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09c7598740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a855804b70 con 0x55a8557f76d0 2024-03-24T10:47:16.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09c7598740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a8557ee950 con 0x55a8557f5280 2024-03-24T10:47:16.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09bed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557ee1d0 0x55a8557e79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09bed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557ee1d0 0x55a8557e79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45436/0 (socket says 172.21.15.23:45436) 2024-03-24T10:47:16.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09bed9d640 1 -- 172.21.15.23:0/1150719101 learned_addr learned my addr 172.21.15.23:0/1150719101 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:16.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09be59c640 1 --2- 172.21.15.23:0/1150719101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557f5280 0x55a85580fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09bed9d640 1 --2- 172.21.15.23:0/1150719101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f76d0 0x55a8557ff580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09bed9d640 1 -- 172.21.15.23:0/1150719101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557f5280 msgr2=0x55a85580fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09bed9d640 1 --2- 172.21.15.23:0/1150719101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557f5280 0x55a85580fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09bed9d640 1 -- 172.21.15.23:0/1150719101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f76d0 msgr2=0x55a8557ff580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.491+0000 7f09bed9d640 1 --2- 172.21.15.23:0/1150719101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f76d0 0x55a8557ff580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.492+0000 7f09bed9d640 1 -- 172.21.15.23:0/1150719101 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8556efb20 con 0x55a8557ee1d0 2024-03-24T10:47:16.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.492+0000 7f09bed9d640 1 --2- 172.21.15.23:0/1150719101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557ee1d0 0x55a8557e79e0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f09b00092e0 tx=0x7f09b0009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bb0f6c9cf7174eb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:16.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.492+0000 7f09a7fff640 1 -- 172.21.15.23:0/1150719101 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b0016070 con 0x55a8557ee1d0 2024-03-24T10:47:16.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.492+0000 7f09a7fff640 1 -- 172.21.15.23:0/1150719101 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a8554547e0 con 0x55a8557ee1d0 2024-03-24T10:47:16.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.492+0000 7f09a7fff640 1 -- 172.21.15.23:0/1150719101 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b0003850 con 0x55a8557ee1d0 2024-03-24T10:47:16.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 -- 172.21.15.23:0/1150719101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557ee1d0 msgr2=0x55a8557e79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 --2- 172.21.15.23:0/1150719101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557ee1d0 0x55a8557e79e0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f09b00092e0 tx=0x7f09b0009f60 comp rx=0 tx=0).stop 2024-03-24T10:47:16.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 -- 172.21.15.23:0/1150719101 shutdown_connections 2024-03-24T10:47:16.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 --2- 172.21.15.23:0/1150719101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f76d0 0x55a8557ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 --2- 172.21.15.23:0/1150719101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557f5280 0x55a85580fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 --2- 172.21.15.23:0/1150719101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557ee1d0 0x55a8557e79e0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 -- 172.21.15.23:0/1150719101 >> 172.21.15.23:0/1150719101 conn(0x55a8557e8900 msgr2=0x55a8557e72d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:16.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 -- 172.21.15.23:0/1150719101 shutdown_connections 2024-03-24T10:47:16.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 -- 172.21.15.23:0/1150719101 wait complete. 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 -- start start 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 Processor -- start 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557ee1d0 0x55a8557f19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f5280 0x55a8557f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f76d0 0x55a85586d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a8557f2570 con 0x55a8557f76d0 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a85586d9c0 con 0x55a8557f5280 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a85586db00 con 0x55a8557ee1d0 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09c7598740 10 client.0 _ll_register_callbacks cb 0x7f09b0009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09bed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f76d0 0x55a85586d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09be59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f5280 0x55a8557f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09bed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f76d0 0x55a85586d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45440/0 (socket says 172.21.15.23:45440) 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09be59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f5280 0x55a8557f1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:55026/0 (socket says 172.21.15.23:55026) 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09bed9d640 1 -- 172.21.15.23:0/524447285 learned_addr learned my addr 172.21.15.23:0/524447285 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:16.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.493+0000 7f09bed9d640 1 --2- 172.21.15.23:0/524447285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557ee1d0 0x55a8557f19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.494+0000 7f09be59c640 1 -- 172.21.15.23:0/524447285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557ee1d0 msgr2=0x55a8557f19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.494+0000 7f09be59c640 1 --2- 172.21.15.23:0/524447285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557ee1d0 0x55a8557f19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.494+0000 7f09be59c640 1 -- 172.21.15.23:0/524447285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f76d0 msgr2=0x55a85586d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:16.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.494+0000 7f09be59c640 1 --2- 172.21.15.23:0/524447285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f76d0 0x55a85586d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.494+0000 7f09be59c640 1 -- 172.21.15.23:0/524447285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a855753ba0 con 0x55a8557f5280 2024-03-24T10:47:16.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.494+0000 7f09bed9d640 1 --2- 172.21.15.23:0/524447285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f76d0 0x55a85586d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:16.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.494+0000 7f09be59c640 1 --2- 172.21.15.23:0/524447285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f5280 0x55a8557f1f10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55a8557e9800 tx=0x7f09b8009f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2620162cf0a01a6a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:16.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.494+0000 7f09a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:47:16.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.494+0000 7f09a5ffb640 1 -- 172.21.15.23:0/524447285 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b8011330 con 0x55a8557f5280 2024-03-24T10:47:16.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.494+0000 7f09c7598740 1 -- 172.21.15.23:0/524447285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09b800fe80 con 0x55a8557f5280 2024-03-24T10:47:16.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.494+0000 7f09a5ffb640 1 -- 172.21.15.23:0/524447285 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7f09b8010bd0 con 0x55a8557f5280 2024-03-24T10:47:16.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.494+0000 7f09a5ffb640 1 -- 172.21.15.23:0/524447285 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f09b8010ea0 con 0x55a8557f5280 2024-03-24T10:47:16.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.494+0000 7f09c7598740 10 client.27551 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:16.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.494+0000 7f09c7598740 1 -- 172.21.15.23:0/524447285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a855715700 con 0x55a8557f5280 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.495+0000 7f09a5ffb640 1 -- 172.21.15.23:0/524447285 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09b8037020 con 0x55a8557f5280 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.495+0000 7f09a5ffb640 1 -- 172.21.15.23:0/524447285 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7f09b802b460 con 0x55a8557f5280 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.495+0000 7f09c7598740 10 client.27551 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.495+0000 7f09c7598740 10 client.27551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.495+0000 7f09c7598740 20 client.27551 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.495+0000 7f09c7598740 1 --2- 172.21.15.23:0/524447285 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a85586f2b0 0x55a8558716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.495+0000 7f09c7598740 4 client.27551 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.495+0000 7f09c7598740 1 -- 172.21.15.23:0/524447285 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a8554e0890 con 0x55a85586f2b0 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.495+0000 7f09bed9d640 1 --2- 172.21.15.23:0/524447285 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a85586f2b0 0x55a8558716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.495+0000 7f09bed9d640 1 --2- 172.21.15.23:0/524447285 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a85586f2b0 0x55a8558716d0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79dd29ece6642ac1 server_cookie=2227415575c807e6 in_seq=0 out_seq=0 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.495+0000 7f09a5ffb640 10 client.27551 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.495+0000 7f09a5ffb640 1 -- 172.21.15.23:0/524447285 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a8554e0890 con 0x55a85586f2b0 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.495+0000 7f09a5ffb640 10 client.27551 handle_command_reply: tid=0 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.495+0000 7f09c7598740 1 client.27551 shutdown 2024-03-24T10:47:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c7598740 1 -- 172.21.15.23:0/524447285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f5280 msgr2=0x55a8557f1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c7598740 1 --2- 172.21.15.23:0/524447285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f5280 0x55a8557f1f10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55a8557e9800 tx=0x7f09b8009f00 comp rx=0 tx=0).stop 2024-03-24T10:47:16.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c7598740 1 -- 172.21.15.23:0/524447285 shutdown_connections 2024-03-24T10:47:16.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c7598740 1 --2- 172.21.15.23:0/524447285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8557f76d0 0x55a85586d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c7598740 1 --2- 172.21.15.23:0/524447285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a8557f5280 0x55a8557f1f10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c7598740 1 --2- 172.21.15.23:0/524447285 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a85586f2b0 0x55a8558716d0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c7598740 1 --2- 172.21.15.23:0/524447285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8557ee1d0 0x55a8557f19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c7598740 1 -- 172.21.15.23:0/524447285 >> 172.21.15.23:0/524447285 conn(0x55a8557e8900 msgr2=0x55a8557ea5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:16.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09a5ffb640 0 client.27551 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:16.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c7598740 1 -- 172.21.15.23:0/524447285 shutdown_connections 2024-03-24T10:47:16.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c7598740 1 -- 172.21.15.23:0/524447285 wait complete. 2024-03-24T10:47:16.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c7598740 20 client.27551 trim_cache size 0 max 16384 2024-03-24T10:47:16.503 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:47:16.503 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:47:16.503 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:47:16.503 INFO:teuthology.orchestra.run.smithi023.stdout: "561a5177-9074-4cb5-9610-3e6bcf099ad6": { 2024-03-24T10:47:16.503 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:47:16.503 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "561a5177-9074-4cb5-9610-3e6bcf099ad6", 2024-03-24T10:47:16.503 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:47:16.503 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:47:16.503 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:47:16.503 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:47:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c49b5640 1 -- 172.21.15.23:0/2081341516 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f09a80a2230 msgr2=0x7f09a80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c49b5640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f09a80a2230 0x7f09a80a46f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f09b40ed8d0 tx=0x7f09b4003760 comp rx=0 tx=0).stop 2024-03-24T10:47:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c49b5640 1 -- 172.21.15.23:0/2081341516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c00692d0 msgr2=0x7f09c019dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c49b5640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c00692d0 0x7f09c019dfb0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009230 tx=0x7f09b4004ab0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c49b5640 1 -- 172.21.15.23:0/2081341516 shutdown_connections 2024-03-24T10:47:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c49b5640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f09a80a2230 0x7f09a80a46f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c49b5640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09c0107760 0x7f09c0197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c49b5640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c00692d0 0x7f09c019dfb0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c49b5640 1 --2- 172.21.15.23:0/2081341516 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09c0068910 0x7f09c019da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.496+0000 7f09c49b5640 1 -- 172.21.15.23:0/2081341516 >> 172.21.15.23:0/2081341516 conn(0x7f09c0077710 msgr2=0x7f09c00fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.497+0000 7f09c49b5640 1 -- 172.21.15.23:0/2081341516 shutdown_connections 2024-03-24T10:47:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:16.497+0000 7f09c49b5640 1 -- 172.21.15.23:0/2081341516 wait complete. 2024-03-24T10:47:16.507 INFO:tasks.cephfs.filesystem:scrub status for tag:561a5177-9074-4cb5-9610-3e6bcf099ad6 - {'path': '/', 'tag': '561a5177-9074-4cb5-9610-3e6bcf099ad6', 'options': 'recursive,force'} 2024-03-24T10:47:16.586 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:16 smithi117 ceph-mon[36368]: pgmap v221: 97 pgs: 97 active+clean; 427 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 201 op/s 2024-03-24T10:47:16.586 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:16 smithi117 podman[71547]: 2024-03-24 10:47:16.267513513 +0000 UTC m=+0.144825577 container create 48528e1915dac0eed881f9ac4c2ead61f4c323d23c458a92affd0de6d41ad4ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-24T10:47:16.586 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:16 smithi117 podman[71547]: 2024-03-24 10:47:16.486774246 +0000 UTC m=+0.364086325 container init 48528e1915dac0eed881f9ac4c2ead61f4c323d23c458a92affd0de6d41ad4ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T10:47:16.586 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:16 smithi117 podman[71547]: 2024-03-24 10:47:16.490166582 +0000 UTC m=+0.367478651 container start 48528e1915dac0eed881f9ac4c2ead61f4c323d23c458a92affd0de6d41ad4ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:47:16.586 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:16 smithi117 ceph-mds[71565]: -- 172.21.15.117:0/1144708059 <== mon.1 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e81bf961a0 con 0x55e81bf6c800 2024-03-24T10:47:16.587 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:16 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71561]: starting mds.c at 2024-03-24T10:47:16.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:16 smithi116 ceph-mon[36248]: pgmap v221: 97 pgs: 97 active+clean; 427 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 201 op/s 2024-03-24T10:47:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:16 smithi023 ceph-mon[32466]: pgmap v221: 97 pgs: 97 active+clean; 427 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 201 op/s 2024-03-24T10:47:17.002 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:16 smithi117 bash[71547]: 48528e1915dac0eed881f9ac4c2ead61f4c323d23c458a92affd0de6d41ad4ce 2024-03-24T10:47:17.003 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:47:16 smithi117 systemd[1]: Started Ceph mds.c for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:47:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.104+0000 7fb4618b1640 1 -- 172.21.15.23:0/4156465717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb45c0fff90 msgr2=0x7fb45c100370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.104+0000 7fb4618b1640 1 --2- 172.21.15.23:0/4156465717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb45c0fff90 0x7fb45c100370 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fb44c009900 tx=0x7fb44c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:47:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.104+0000 7fb4618b1640 1 -- 172.21.15.23:0/4156465717 shutdown_connections 2024-03-24T10:47:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.104+0000 7fb4618b1640 1 --2- 172.21.15.23:0/4156465717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45c105060 0x7fb45c10cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.104+0000 7fb4618b1640 1 --2- 172.21.15.23:0/4156465717 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb45c100940 0x7fb45c104980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.104+0000 7fb4618b1640 1 --2- 172.21.15.23:0/4156465717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb45c0fff90 0x7fb45c100370 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.104+0000 7fb4618b1640 1 -- 172.21.15.23:0/4156465717 >> 172.21.15.23:0/4156465717 conn(0x7fb45c076ba0 msgr2=0x7fb45c0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.104+0000 7fb4618b1640 1 -- 172.21.15.23:0/4156465717 shutdown_connections 2024-03-24T10:47:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.104+0000 7fb4618b1640 1 -- 172.21.15.23:0/4156465717 wait complete. 2024-03-24T10:47:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.104+0000 7fb4618b1640 1 Processor -- start 2024-03-24T10:47:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.104+0000 7fb4618b1640 1 -- start start 2024-03-24T10:47:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb4618b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45c0fff90 0x7fb45c10e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb4618b1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb45c105060 0x7fb45c10e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb4618b1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb45c10edd0 0x7fb45c1a5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb4618b1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c1a6450 con 0x7fb45c105060 2024-03-24T10:47:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb4618b1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c1a65c0 con 0x7fb45c0fff90 2024-03-24T10:47:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb4618b1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c1a6780 con 0x7fb45c10edd0 2024-03-24T10:47:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb453fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb45c105060 0x7fb45c10e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb453fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb45c105060 0x7fb45c10e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45448/0 (socket says 172.21.15.23:45448) 2024-03-24T10:47:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb453fff640 1 -- 172.21.15.23:0/2569914387 learned_addr learned my addr 172.21.15.23:0/2569914387 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb45affd640 1 --2- 172.21.15.23:0/2569914387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45c0fff90 0x7fb45c10e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb45b7fe640 1 --2- 172.21.15.23:0/2569914387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb45c10edd0 0x7fb45c1a5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb453fff640 1 -- 172.21.15.23:0/2569914387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb45c10edd0 msgr2=0x7fb45c1a5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb453fff640 1 --2- 172.21.15.23:0/2569914387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb45c10edd0 0x7fb45c1a5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb453fff640 1 -- 172.21.15.23:0/2569914387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45c0fff90 msgr2=0x7fb45c10e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb453fff640 1 --2- 172.21.15.23:0/2569914387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45c0fff90 0x7fb45c10e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb453fff640 1 -- 172.21.15.23:0/2569914387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb44c009560 con 0x7fb45c105060 2024-03-24T10:47:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb453fff640 1 --2- 172.21.15.23:0/2569914387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb45c105060 0x7fb45c10e890 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb44400e9f0 tx=0x7fb44400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb4608af640 1 -- 172.21.15.23:0/2569914387 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4440096c0 con 0x7fb45c105060 2024-03-24T10:47:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb4608af640 1 -- 172.21.15.23:0/2569914387 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb444005500 con 0x7fb45c105060 2024-03-24T10:47:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb4618b1640 1 -- 172.21.15.23:0/2569914387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb45c1a6a60 con 0x7fb45c105060 2024-03-24T10:47:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb4608af640 1 -- 172.21.15.23:0/2569914387 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb444010430 con 0x7fb45c105060 2024-03-24T10:47:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.105+0000 7fb4618b1640 1 -- 172.21.15.23:0/2569914387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb45c1a6f70 con 0x7fb45c105060 2024-03-24T10:47:17.117 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.108+0000 7fb4608af640 1 -- 172.21.15.23:0/2569914387 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fb444005020 con 0x7fb45c105060 2024-03-24T10:47:17.117 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.108+0000 7fb464568740 1 Processor -- start 2024-03-24T10:47:17.117 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb464568740 1 -- start start 2024-03-24T10:47:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb4608af640 1 --2- 172.21.15.23:0/2569914387 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb4480a2380 0x7fb4480a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb45b7fe640 1 --2- 172.21.15.23:0/2569914387 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb4480a2380 0x7fb4480a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb464568740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab689c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb464568740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab71790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb464568740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab71cd0 0x56432ab740c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb464568740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56432ab231b0 con 0x56432ab685e0 2024-03-24T10:47:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb464568740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56432aad5f80 con 0x56432ab71cd0 2024-03-24T10:47:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb4608af640 1 -- 172.21.15.23:0/2569914387 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7fb444014070 con 0x7fb45c105060 2024-03-24T10:47:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb464568740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56432aad34b0 con 0x56432ab68f00 2024-03-24T10:47:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb45affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb45affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab689c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45456/0 (socket says 172.21.15.23:45456) 2024-03-24T10:47:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb45affd640 1 -- 172.21.15.23:0/3551736222 learned_addr learned my addr 172.21.15.23:0/3551736222 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb453fff640 1 --2- 172.21.15.23:0/3551736222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab71790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb45b7fe640 1 --2- 172.21.15.23:0/3551736222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab71cd0 0x56432ab740c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb453fff640 1 -- 172.21.15.23:0/3551736222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab71cd0 msgr2=0x56432ab740c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb453fff640 1 --2- 172.21.15.23:0/3551736222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab71cd0 0x56432ab740c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb453fff640 1 -- 172.21.15.23:0/3551736222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 msgr2=0x56432ab689c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb453fff640 1 --2- 172.21.15.23:0/3551736222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab689c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb453fff640 1 -- 172.21.15.23:0/3551736222 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56432aa64810 con 0x56432ab68f00 2024-03-24T10:47:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb45affd640 1 --2- 172.21.15.23:0/3551736222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab689c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.109+0000 7fb45b7fe640 1 --2- 172.21.15.23:0/2569914387 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb4480a2380 0x7fb4480a4840 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb454005fd0 tx=0x7fb454011b40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.110+0000 7fb453fff640 1 --2- 172.21.15.23:0/3551736222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab71790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb4440cbaa0 tx=0x7fb444005730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=788587ac2b08413c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.110+0000 7fb451ffb640 1 -- 172.21.15.23:0/3551736222 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44409ee90 con 0x56432ab68f00 2024-03-24T10:47:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.110+0000 7fb451ffb640 1 -- 172.21.15.23:0/3551736222 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4440a34e0 con 0x56432ab68f00 2024-03-24T10:47:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.110+0000 7fb451ffb640 1 -- 172.21.15.23:0/3551736222 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4440973b0 con 0x56432ab68f00 2024-03-24T10:47:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.110+0000 7fb464568740 1 -- 172.21.15.23:0/3551736222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 msgr2=0x56432ab71790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.110+0000 7fb464568740 1 --2- 172.21.15.23:0/3551736222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab71790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb4440cbaa0 tx=0x7fb444005730 comp rx=0 tx=0).stop 2024-03-24T10:47:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.110+0000 7fb464568740 1 -- 172.21.15.23:0/3551736222 shutdown_connections 2024-03-24T10:47:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.110+0000 7fb464568740 1 --2- 172.21.15.23:0/3551736222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab71cd0 0x56432ab740c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.110+0000 7fb464568740 1 --2- 172.21.15.23:0/3551736222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab71790 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.110+0000 7fb464568740 1 --2- 172.21.15.23:0/3551736222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab689c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.110+0000 7fb464568740 1 -- 172.21.15.23:0/3551736222 >> 172.21.15.23:0/3551736222 conn(0x56432ab62f00 msgr2=0x56432ab632e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.110+0000 7fb464568740 1 -- 172.21.15.23:0/3551736222 shutdown_connections 2024-03-24T10:47:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.111+0000 7fb464568740 1 -- 172.21.15.23:0/3551736222 wait complete. 2024-03-24T10:47:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.111+0000 7fb464568740 1 -- start start 2024-03-24T10:47:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.111+0000 7fb464568740 1 Processor -- start 2024-03-24T10:47:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.111+0000 7fb464568740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab6ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.111+0000 7fb464568740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab68f00 0x56432ab6f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.111+0000 7fb464568740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab71cd0 0x56432ab6f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.111+0000 7fb464568740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56432aacec90 con 0x56432ab685e0 2024-03-24T10:47:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.111+0000 7fb464568740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56432aa73c50 con 0x56432ab68f00 2024-03-24T10:47:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.111+0000 7fb464568740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56432a7a2250 con 0x56432ab71cd0 2024-03-24T10:47:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb464568740 10 client.0 _ll_register_callbacks cb 0x56432aac2290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb45affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab6ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb45affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab6ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45472/0 (socket says 172.21.15.23:45472) 2024-03-24T10:47:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb45affd640 1 -- 172.21.15.23:0/3644175491 learned_addr learned my addr 172.21.15.23:0/3644175491 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb45b7fe640 1 --2- 172.21.15.23:0/3644175491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab71cd0 0x56432ab6f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb453fff640 1 --2- 172.21.15.23:0/3644175491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab68f00 0x56432ab6f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb45affd640 1 -- 172.21.15.23:0/3644175491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab71cd0 msgr2=0x56432ab6f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb45affd640 1 --2- 172.21.15.23:0/3644175491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab71cd0 0x56432ab6f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb45affd640 1 -- 172.21.15.23:0/3644175491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab68f00 msgr2=0x56432ab6f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb45affd640 1 --2- 172.21.15.23:0/3644175491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab68f00 0x56432ab6f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb45affd640 1 -- 172.21.15.23:0/3644175491 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb45c1a6f70 con 0x56432ab685e0 2024-03-24T10:47:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb45affd640 1 --2- 172.21.15.23:0/3644175491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab6ed60 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fb44c036040 tx=0x7fb44c02ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40361fa941d9ea50 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb43affd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb43affd640 1 -- 172.21.15.23:0/3644175491 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c004040 con 0x56432ab685e0 2024-03-24T10:47:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb464568740 1 -- 172.21.15.23:0/3644175491 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56432aaf3790 con 0x56432ab685e0 2024-03-24T10:47:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb43affd640 1 -- 172.21.15.23:0/3644175491 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7fb44c0041a0 con 0x56432ab685e0 2024-03-24T10:47:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.112+0000 7fb43affd640 1 -- 172.21.15.23:0/3644175491 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb44c004360 con 0x56432ab685e0 2024-03-24T10:47:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.113+0000 7fb464568740 10 client.17682 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.113+0000 7fb464568740 1 -- 172.21.15.23:0/3644175491 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56432aacd310 con 0x56432ab685e0 2024-03-24T10:47:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.113+0000 7fb43affd640 1 -- 172.21.15.23:0/3644175491 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56432aaf3790 con 0x56432ab685e0 2024-03-24T10:47:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.113+0000 7fb43affd640 1 -- 172.21.15.23:0/3644175491 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fb44c03c040 con 0x56432ab685e0 2024-03-24T10:47:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.113+0000 7fb464568740 10 client.17682 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.113+0000 7fb464568740 10 client.17682 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.113+0000 7fb464568740 20 client.17682 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.113+0000 7fb464568740 1 --2- 172.21.15.23:0/3644175491 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56432ab81be0 0x56432ab83ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.113+0000 7fb464568740 4 client.17682 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.113+0000 7fb464568740 1 -- 172.21.15.23:0/3644175491 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56432a7ce600 con 0x56432ab81be0 2024-03-24T10:47:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.113+0000 7fb453fff640 1 --2- 172.21.15.23:0/3644175491 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56432ab81be0 0x56432ab83ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.114+0000 7fb43affd640 10 client.17682 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.114+0000 7fb453fff640 1 --2- 172.21.15.23:0/3644175491 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56432ab81be0 0x56432ab83ff0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a5ab0a3e69d54de server_cookie=61749d21d81f3c64 in_seq=0 out_seq=0 2024-03-24T10:47:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb43affd640 1 -- 172.21.15.23:0/3644175491 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56432a7ce600 con 0x56432ab81be0 2024-03-24T10:47:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb43affd640 10 client.17682 handle_command_reply: tid=0 2024-03-24T10:47:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb464568740 1 client.17682 shutdown 2024-03-24T10:47:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb464568740 1 -- 172.21.15.23:0/3644175491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 msgr2=0x56432ab6ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb464568740 1 --2- 172.21.15.23:0/3644175491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab6ed60 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fb44c036040 tx=0x7fb44c02ca10 comp rx=0 tx=0).stop 2024-03-24T10:47:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb464568740 1 -- 172.21.15.23:0/3644175491 shutdown_connections 2024-03-24T10:47:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb464568740 1 --2- 172.21.15.23:0/3644175491 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56432ab81be0 0x56432ab83ff0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb464568740 1 --2- 172.21.15.23:0/3644175491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab71cd0 0x56432ab6f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb464568740 1 --2- 172.21.15.23:0/3644175491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab68f00 0x56432ab6f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb43affd640 0 client.17682 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb464568740 1 --2- 172.21.15.23:0/3644175491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab6ed60 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb464568740 1 -- 172.21.15.23:0/3644175491 >> 172.21.15.23:0/3644175491 conn(0x56432ab62f00 msgr2=0x56432ab64c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb464568740 1 -- 172.21.15.23:0/3644175491 shutdown_connections 2024-03-24T10:47:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb464568740 1 -- 172.21.15.23:0/3644175491 wait complete. 2024-03-24T10:47:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.115+0000 7fb464568740 20 client.17682 trim_cache size 0 max 16384 2024-03-24T10:47:17.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 703510 10.91 MB/sec execute 214 sec latency 133.938 ms 2024-03-24T10:47:17.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.130+0000 7fb464568740 1 Processor -- start 2024-03-24T10:47:17.132 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.130+0000 7fb464568740 1 -- start start 2024-03-24T10:47:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.130+0000 7fb464568740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab7d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.130+0000 7fb464568740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab6f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.130+0000 7fb464568740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab71cd0 0x56432ab89900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.130+0000 7fb464568740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56432aa54660 con 0x56432ab685e0 2024-03-24T10:47:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.130+0000 7fb464568740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56432ab67ef0 con 0x56432ab71cd0 2024-03-24T10:47:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.130+0000 7fb464568740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56432ab5e6c0 con 0x56432ab68f00 2024-03-24T10:47:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.130+0000 7fb45affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab7d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.130+0000 7fb45affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab7d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45486/0 (socket says 172.21.15.23:45486) 2024-03-24T10:47:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.130+0000 7fb45affd640 1 -- 172.21.15.23:0/3603946959 learned_addr learned my addr 172.21.15.23:0/3603946959 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.131+0000 7fb453fff640 1 --2- 172.21.15.23:0/3603946959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab6f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.131+0000 7fb45b7fe640 1 --2- 172.21.15.23:0/3603946959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab71cd0 0x56432ab89900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.131+0000 7fb45affd640 1 -- 172.21.15.23:0/3603946959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 msgr2=0x56432ab6f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:17.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.131+0000 7fb45affd640 1 --2- 172.21.15.23:0/3603946959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab6f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.131+0000 7fb45affd640 1 -- 172.21.15.23:0/3603946959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab71cd0 msgr2=0x56432ab89900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.131+0000 7fb45affd640 1 --2- 172.21.15.23:0/3603946959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab71cd0 0x56432ab89900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.131+0000 7fb45affd640 1 -- 172.21.15.23:0/3603946959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56432aa69810 con 0x56432ab685e0 2024-03-24T10:47:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.131+0000 7fb453fff640 1 --2- 172.21.15.23:0/3603946959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab6f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.131+0000 7fb45affd640 1 --2- 172.21.15.23:0/3603946959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab7d3a0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fb44c004ab0 tx=0x7fb44c02c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c151a2c927e93fe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.131+0000 7fb4517fa640 1 -- 172.21.15.23:0/3603946959 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c0035f0 con 0x56432ab685e0 2024-03-24T10:47:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.131+0000 7fb4517fa640 1 -- 172.21.15.23:0/3603946959 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb44c0043b0 con 0x56432ab685e0 2024-03-24T10:47:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.131+0000 7fb4517fa640 1 -- 172.21.15.23:0/3603946959 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c005420 con 0x56432ab685e0 2024-03-24T10:47:17.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.132+0000 7fb464568740 1 -- 172.21.15.23:0/3603946959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 msgr2=0x56432ab7d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:17.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.132+0000 7fb464568740 1 --2- 172.21.15.23:0/3603946959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab7d3a0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fb44c004ab0 tx=0x7fb44c02c740 comp rx=0 tx=0).stop 2024-03-24T10:47:17.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.132+0000 7fb464568740 1 -- 172.21.15.23:0/3603946959 shutdown_connections 2024-03-24T10:47:17.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.132+0000 7fb464568740 1 --2- 172.21.15.23:0/3603946959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab71cd0 0x56432ab89900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.132+0000 7fb464568740 1 --2- 172.21.15.23:0/3603946959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab6f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.132+0000 7fb464568740 1 --2- 172.21.15.23:0/3603946959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab685e0 0x56432ab7d3a0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.132+0000 7fb464568740 1 -- 172.21.15.23:0/3603946959 >> 172.21.15.23:0/3603946959 conn(0x56432ab62f00 msgr2=0x56432ab6d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:17.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.132+0000 7fb464568740 1 -- 172.21.15.23:0/3603946959 shutdown_connections 2024-03-24T10:47:17.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.132+0000 7fb464568740 1 -- 172.21.15.23:0/3603946959 wait complete. 2024-03-24T10:47:17.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.132+0000 7fb464568740 1 -- start start 2024-03-24T10:47:17.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb464568740 1 Processor -- start 2024-03-24T10:47:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb464568740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab685e0 0x56432ab79bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb464568740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab6b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb464568740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab71cd0 0x56432ab6bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb464568740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56432ab6c1e0 con 0x56432ab71cd0 2024-03-24T10:47:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb464568740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56432ab6c350 con 0x56432ab685e0 2024-03-24T10:47:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb464568740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56432abe6f30 con 0x56432ab68f00 2024-03-24T10:47:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb464568740 10 client.0 _ll_register_callbacks cb 0x7fb44c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb453fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab6b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb45b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab71cd0 0x56432ab6bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb45affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab685e0 0x56432ab79bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb45b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab71cd0 0x56432ab6bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45488/0 (socket says 172.21.15.23:45488) 2024-03-24T10:47:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb45b7fe640 1 -- 172.21.15.23:0/2280072174 learned_addr learned my addr 172.21.15.23:0/2280072174 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb45affd640 1 -- 172.21.15.23:0/2280072174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 msgr2=0x56432ab6b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb45affd640 1 --2- 172.21.15.23:0/2280072174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab6b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb45affd640 1 -- 172.21.15.23:0/2280072174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab71cd0 msgr2=0x56432ab6bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb45affd640 1 --2- 172.21.15.23:0/2280072174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab71cd0 0x56432ab6bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb45affd640 1 -- 172.21.15.23:0/2280072174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4440b7040 con 0x56432ab685e0 2024-03-24T10:47:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb453fff640 1 --2- 172.21.15.23:0/2280072174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab6b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.133+0000 7fb45b7fe640 1 --2- 172.21.15.23:0/2280072174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab71cd0 0x56432ab6bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.134+0000 7fb45affd640 1 --2- 172.21.15.23:0/2280072174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab685e0 0x56432ab79bc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb44c004a80 tx=0x7fb44c004340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f92d2cba250698b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.134+0000 7fb43affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:47:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.134+0000 7fb43affd640 1 -- 172.21.15.23:0/2280072174 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c004520 con 0x56432ab685e0 2024-03-24T10:47:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.134+0000 7fb43affd640 1 -- 172.21.15.23:0/2280072174 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8571+0+0 (secure 0 0 0) 0x7fb44c000e40 con 0x56432ab685e0 2024-03-24T10:47:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.134+0000 7fb464568740 1 -- 172.21.15.23:0/2280072174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4440a34e0 con 0x56432ab685e0 2024-03-24T10:47:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.134+0000 7fb43affd640 1 -- 172.21.15.23:0/2280072174 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb44c02c9f0 con 0x56432ab685e0 2024-03-24T10:47:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.134+0000 7fb464568740 10 client.27575 fetch_fsmap learned FSMap version 29 2024-03-24T10:47:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.134+0000 7fb464568740 1 -- 172.21.15.23:0/2280072174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56432aacd310 con 0x56432ab685e0 2024-03-24T10:47:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.134+0000 7fb43affd640 1 -- 172.21.15.23:0/2280072174 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb44c070080 con 0x56432ab685e0 2024-03-24T10:47:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.134+0000 7fb43affd640 1 -- 172.21.15.23:0/2280072174 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8338+0+0 (secure 0 0 0) 0x7fb44c03c040 con 0x56432ab685e0 2024-03-24T10:47:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.134+0000 7fb464568740 10 client.27575 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T10:47:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.135+0000 7fb464568740 10 client.27575 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.135+0000 7fb464568740 20 client.27575 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.135+0000 7fb464568740 1 --2- 172.21.15.23:0/2280072174 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56432abe81c0 0x56432abea5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.135+0000 7fb464568740 4 client.27575 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:47:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.135+0000 7fb464568740 1 -- 172.21.15.23:0/2280072174 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56432a81f1b0 con 0x56432abe81c0 2024-03-24T10:47:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.135+0000 7fb453fff640 1 --2- 172.21.15.23:0/2280072174 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56432abe81c0 0x56432abea5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.135+0000 7fb453fff640 1 --2- 172.21.15.23:0/2280072174 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56432abe81c0 0x56432abea5b0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae25464369bd99d5 server_cookie=53ccb1af781987c9 in_seq=0 out_seq=0 2024-03-24T10:47:17.141 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.135+0000 7fb43affd640 10 client.27575 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.147+0000 7fb43affd640 1 -- 172.21.15.23:0/2280072174 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56432a81f1b0 con 0x56432abe81c0 2024-03-24T10:47:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.147+0000 7fb43affd640 10 client.27575 handle_command_reply: tid=0 2024-03-24T10:47:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.147+0000 7fb464568740 1 client.27575 shutdown 2024-03-24T10:47:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.147+0000 7fb464568740 1 -- 172.21.15.23:0/2280072174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab685e0 msgr2=0x56432ab79bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb464568740 1 --2- 172.21.15.23:0/2280072174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab685e0 0x56432ab79bc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb44c004a80 tx=0x7fb44c004340 comp rx=0 tx=0).stop 2024-03-24T10:47:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb464568740 1 -- 172.21.15.23:0/2280072174 shutdown_connections 2024-03-24T10:47:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb464568740 1 --2- 172.21.15.23:0/2280072174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56432ab71cd0 0x56432ab6bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb464568740 1 --2- 172.21.15.23:0/2280072174 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56432abe81c0 0x56432abea5b0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb464568740 1 --2- 172.21.15.23:0/2280072174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56432ab68f00 0x56432ab6b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb464568740 1 --2- 172.21.15.23:0/2280072174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56432ab685e0 0x56432ab79bc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb43affd640 0 client.27575 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb464568740 1 -- 172.21.15.23:0/2280072174 >> 172.21.15.23:0/2280072174 conn(0x56432ab62f00 msgr2=0x56432ab64350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb464568740 1 -- 172.21.15.23:0/2280072174 shutdown_connections 2024-03-24T10:47:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb464568740 1 -- 172.21.15.23:0/2280072174 wait complete. 2024-03-24T10:47:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb464568740 20 client.27575 trim_cache size 0 max 16384 2024-03-24T10:47:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:47:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:47:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:47:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:47:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb4618b1640 1 -- 172.21.15.23:0/2569914387 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb4480a2380 msgr2=0x7fb4480a4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb4618b1640 1 --2- 172.21.15.23:0/2569914387 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb4480a2380 0x7fb4480a4840 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb454005fd0 tx=0x7fb454011b40 comp rx=0 tx=0).stop 2024-03-24T10:47:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb4618b1640 1 -- 172.21.15.23:0/2569914387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb45c105060 msgr2=0x7fb45c10e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb4618b1640 1 --2- 172.21.15.23:0/2569914387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb45c105060 0x7fb45c10e890 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb44400e9f0 tx=0x7fb44400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb4618b1640 1 -- 172.21.15.23:0/2569914387 shutdown_connections 2024-03-24T10:47:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb4618b1640 1 --2- 172.21.15.23:0/2569914387 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb4480a2380 0x7fb4480a4840 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb4618b1640 1 --2- 172.21.15.23:0/2569914387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb45c10edd0 0x7fb45c1a5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb4618b1640 1 --2- 172.21.15.23:0/2569914387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb45c105060 0x7fb45c10e890 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb4618b1640 1 --2- 172.21.15.23:0/2569914387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb45c0fff90 0x7fb45c10e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb4618b1640 1 -- 172.21.15.23:0/2569914387 >> 172.21.15.23:0/2569914387 conn(0x7fb45c076ba0 msgr2=0x7fb45c1038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb4618b1640 1 -- 172.21.15.23:0/2569914387 shutdown_connections 2024-03-24T10:47:17.151 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:17.148+0000 7fb4618b1640 1 -- 172.21.15.23:0/2569914387 wait complete. 2024-03-24T10:47:17.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:17 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:47:17.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:17 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:17.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:17 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:47:17.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:17 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:47:17.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:17 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:17.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:17 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:17.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:17 smithi117 ceph-mon[36368]: daemon mds.c restarted 2024-03-24T10:47:17.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:17 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:47:17.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:17 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:47:17.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:17 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:47:17.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:17 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:47:17.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:17 smithi117 ceph-mon[36368]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T10:47:17.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:17 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:47:17.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:17 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:17.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:17 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:47:17.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:17 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:47:17.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:17 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:17.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:17 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:17.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:17 smithi116 ceph-mon[36248]: daemon mds.c restarted 2024-03-24T10:47:17.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:17 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:47:17.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:17 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:47:17.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:17 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:47:17.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:17 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:47:17.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:17 smithi116 ceph-mon[36248]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T10:47:17.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:17 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:47:17.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:17 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:17.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:17 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:47:17.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:17 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:47:17.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:17 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:17.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:17 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:17.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:17 smithi023 ceph-mon[32466]: daemon mds.c restarted 2024-03-24T10:47:17.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:17 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:47:17.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:17 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:47:17.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:17 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:47:17.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:17 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:47:17.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:17 smithi023 ceph-mon[32466]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T10:47:18.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 704789 10.88 MB/sec execute 215 sec latency 26.349 ms 2024-03-24T10:47:18.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:18 smithi117 ceph-mon[36368]: pgmap v223: 97 pgs: 97 active+clean; 407 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.7 MiB/s wr, 250 op/s 2024-03-24T10:47:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:18 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.117:6836/2017557046,v1:172.21.15.117:6837/2017557046] up:boot 2024-03-24T10:47:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:18 smithi117 ceph-mon[36368]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T10:47:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:18 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:47:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:18 smithi117 ceph-mon[36368]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:47:18.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:18 smithi116 ceph-mon[36248]: pgmap v223: 97 pgs: 97 active+clean; 407 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.7 MiB/s wr, 250 op/s 2024-03-24T10:47:18.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:18 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.117:6836/2017557046,v1:172.21.15.117:6837/2017557046] up:boot 2024-03-24T10:47:18.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:18 smithi116 ceph-mon[36248]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T10:47:18.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:18 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:47:18.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:18 smithi116 ceph-mon[36248]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:47:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:18 smithi023 ceph-mon[32466]: pgmap v223: 97 pgs: 97 active+clean; 407 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.7 MiB/s wr, 250 op/s 2024-03-24T10:47:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:18 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.117:6836/2017557046,v1:172.21.15.117:6837/2017557046] up:boot 2024-03-24T10:47:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:18 smithi023 ceph-mon[32466]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T10:47:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:18 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:47:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:18 smithi023 ceph-mon[32466]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-24T10:47:19.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 707044 10.90 MB/sec execute 216 sec latency 27.645 ms 2024-03-24T10:47:20.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 708588 10.87 MB/sec execute 217 sec latency 20.093 ms 2024-03-24T10:47:20.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:20 smithi117 ceph-mon[36368]: pgmap v224: 97 pgs: 97 active+clean; 407 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.6 MiB/s wr, 243 op/s 2024-03-24T10:47:20.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:20 smithi116 ceph-mon[36248]: pgmap v224: 97 pgs: 97 active+clean; 407 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.6 MiB/s wr, 243 op/s 2024-03-24T10:47:20.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:20 smithi023 ceph-mon[32466]: pgmap v224: 97 pgs: 97 active+clean; 407 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.6 MiB/s wr, 243 op/s 2024-03-24T10:47:21.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 710787 10.89 MB/sec execute 218 sec latency 15.273 ms 2024-03-24T10:47:22.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 712050 10.86 MB/sec execute 219 sec latency 113.610 ms 2024-03-24T10:47:22.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:22 smithi117 ceph-mon[36368]: pgmap v225: 97 pgs: 97 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 215 op/s 2024-03-24T10:47:22.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:22 smithi116 ceph-mon[36248]: pgmap v225: 97 pgs: 97 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 215 op/s 2024-03-24T10:47:22.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:22 smithi023 ceph-mon[32466]: pgmap v225: 97 pgs: 97 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 215 op/s 2024-03-24T10:47:23.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 714210 10.87 MB/sec execute 220 sec latency 35.729 ms 2024-03-24T10:47:24.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 715847 10.85 MB/sec execute 221 sec latency 32.410 ms 2024-03-24T10:47:24.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:24 smithi117 ceph-mon[36368]: pgmap v226: 97 pgs: 97 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 215 op/s 2024-03-24T10:47:24.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:24 smithi116 ceph-mon[36248]: pgmap v226: 97 pgs: 97 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 215 op/s 2024-03-24T10:47:24.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:24 smithi023 ceph-mon[32466]: pgmap v226: 97 pgs: 97 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 215 op/s 2024-03-24T10:47:25.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 717874 10.86 MB/sec execute 222 sec latency 15.378 ms 2024-03-24T10:47:25.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:25 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:47:25.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:25 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:47:25.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:25 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:47:25.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:25 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:47:25.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:25 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:47:25.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:25 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:47:26.125 INFO:tasks.workunit.client.0.smithi023.stdout: 1 719781 10.85 MB/sec execute 223 sec latency 25.331 ms 2024-03-24T10:47:26.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:26 smithi117 ceph-mon[36368]: pgmap v227: 97 pgs: 97 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 215 op/s 2024-03-24T10:47:26.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:26 smithi116 ceph-mon[36248]: pgmap v227: 97 pgs: 97 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 215 op/s 2024-03-24T10:47:26.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:26 smithi023 ceph-mon[32466]: pgmap v227: 97 pgs: 97 active+clean; 412 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 215 op/s 2024-03-24T10:47:27.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 721468 10.85 MB/sec execute 224 sec latency 108.781 ms 2024-03-24T10:47:28.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 723286 10.83 MB/sec execute 225 sec latency 19.309 ms 2024-03-24T10:47:28.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:28 smithi117 ceph-mon[36368]: pgmap v228: 97 pgs: 97 active+clean; 415 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 192 op/s 2024-03-24T10:47:28.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:28 smithi116 ceph-mon[36248]: pgmap v228: 97 pgs: 97 active+clean; 415 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 192 op/s 2024-03-24T10:47:28.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:28 smithi023 ceph-mon[32466]: pgmap v228: 97 pgs: 97 active+clean; 415 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 192 op/s 2024-03-24T10:47:29.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 725190 10.84 MB/sec execute 226 sec latency 20.012 ms 2024-03-24T10:47:30.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 727164 10.84 MB/sec execute 227 sec latency 21.870 ms 2024-03-24T10:47:30.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.242+0000 7f4d98847640 1 -- 172.21.15.23:0/3270194850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d94100860 msgr2=0x7f4d94100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:30.244 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.242+0000 7f4d98847640 1 --2- 172.21.15.23:0/3270194850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d94100860 0x7f4d94100cc0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f4d88009900 tx=0x7f4d8802bc30 comp rx=0 tx=0).stop 2024-03-24T10:47:30.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.242+0000 7f4d98847640 1 -- 172.21.15.23:0/3270194850 shutdown_connections 2024-03-24T10:47:30.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.242+0000 7f4d98847640 1 --2- 172.21.15.23:0/3270194850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d94101200 0x7f4d9410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.242+0000 7f4d98847640 1 --2- 172.21.15.23:0/3270194850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d94100860 0x7f4d94100cc0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.242+0000 7f4d98847640 1 --2- 172.21.15.23:0/3270194850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d94105590 0x7f4d94105970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.242+0000 7f4d98847640 1 -- 172.21.15.23:0/3270194850 >> 172.21.15.23:0/3270194850 conn(0x7f4d940773e0 msgr2=0x7f4d940777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:30.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.242+0000 7f4d98847640 1 -- 172.21.15.23:0/3270194850 shutdown_connections 2024-03-24T10:47:30.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.242+0000 7f4d98847640 1 -- 172.21.15.23:0/3270194850 wait complete. 2024-03-24T10:47:30.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.242+0000 7f4d98847640 1 Processor -- start 2024-03-24T10:47:30.245 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d98847640 1 -- start start 2024-03-24T10:47:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d98847640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d94100860 0x7f4d94197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d98847640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d94101200 0x7f4d941982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d98847640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d94105590 0x7f4d9419c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d98847640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9419cc50 con 0x7f4d94105590 2024-03-24T10:47:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d98847640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9419cdc0 con 0x7f4d94101200 2024-03-24T10:47:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d98847640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9419cf80 con 0x7f4d94100860 2024-03-24T10:47:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d92d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d94105590 0x7f4d9419c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d9259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d94100860 0x7f4d94197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d9259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d94100860 0x7f4d94197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:40458/0 (socket says 172.21.15.23:40458) 2024-03-24T10:47:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d91d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d94101200 0x7f4d941982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d9259c640 1 -- 172.21.15.23:0/91256359 learned_addr learned my addr 172.21.15.23:0/91256359 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:30.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d92d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d94105590 0x7f4d9419c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59638/0 (socket says 172.21.15.23:59638) 2024-03-24T10:47:30.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d9259c640 1 -- 172.21.15.23:0/91256359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d94101200 msgr2=0x7f4d941982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:30.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d9259c640 1 --2- 172.21.15.23:0/91256359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d94101200 0x7f4d941982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d9259c640 1 -- 172.21.15.23:0/91256359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d94105590 msgr2=0x7f4d9419c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:30.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d9259c640 1 --2- 172.21.15.23:0/91256359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d94105590 0x7f4d9419c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d9259c640 1 -- 172.21.15.23:0/91256359 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d88009560 con 0x7f4d94100860 2024-03-24T10:47:30.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d92d9d640 1 --2- 172.21.15.23:0/91256359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d94105590 0x7f4d9419c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:30.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.243+0000 7f4d9259c640 1 --2- 172.21.15.23:0/91256359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d94100860 0x7f4d94197d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4d8400e9f0 tx=0x7f4d8400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:30.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.244+0000 7f4d837fe640 1 -- 172.21.15.23:0/91256359 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84009700 con 0x7f4d94100860 2024-03-24T10:47:30.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.244+0000 7f4d837fe640 1 -- 172.21.15.23:0/91256359 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d8400cd80 con 0x7f4d94100860 2024-03-24T10:47:30.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.244+0000 7f4d837fe640 1 -- 172.21.15.23:0/91256359 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84018920 con 0x7f4d94100860 2024-03-24T10:47:30.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.244+0000 7f4d98847640 1 -- 172.21.15.23:0/91256359 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d9419d260 con 0x7f4d94100860 2024-03-24T10:47:30.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.244+0000 7f4d98847640 1 -- 172.21.15.23:0/91256359 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d941a4ae0 con 0x7f4d94100860 2024-03-24T10:47:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d837fe640 1 -- 172.21.15.23:0/91256359 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f4d84018a80 con 0x7f4d94100860 2024-03-24T10:47:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d9b423740 1 Processor -- start 2024-03-24T10:47:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d9b423740 1 -- start start 2024-03-24T10:47:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d9b423740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb24080 0x555ceeb24460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d9b423740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb249a0 0x555ceeb31ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d9b423740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb320e0 0x555ceeb344d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d9b423740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555ceeadf1b0 con 0x555ceeb320e0 2024-03-24T10:47:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d9b423740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555ceea91f80 con 0x555ceeb249a0 2024-03-24T10:47:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d9b423740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555ceea8f4b0 con 0x555ceeb24080 2024-03-24T10:47:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d92d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb24080 0x555ceeb24460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d91d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb320e0 0x555ceeb344d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d92d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb24080 0x555ceeb24460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:40460/0 (socket says 172.21.15.23:40460) 2024-03-24T10:47:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d92d9d640 1 -- 172.21.15.23:0/976765620 learned_addr learned my addr 172.21.15.23:0/976765620 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d91d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb320e0 0x555ceeb344d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59654/0 (socket says 172.21.15.23:59654) 2024-03-24T10:47:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.247+0000 7f4d9259c640 1 --2- 172.21.15.23:0/976765620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb249a0 0x555ceeb31ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.248+0000 7f4d837fe640 1 --2- 172.21.15.23:0/91256359 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f4d7c0a2350 0x7f4d7c0a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.248+0000 7f4d91d9b640 1 -- 172.21.15.23:0/976765620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb24080 msgr2=0x555ceeb24460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.248+0000 7f4d91d9b640 1 --2- 172.21.15.23:0/976765620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb24080 0x555ceeb24460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.248+0000 7f4d91d9b640 1 -- 172.21.15.23:0/976765620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb249a0 msgr2=0x555ceeb31ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.248+0000 7f4d91d9b640 1 --2- 172.21.15.23:0/976765620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb249a0 0x555ceeb31ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.248+0000 7f4d91d9b640 1 -- 172.21.15.23:0/976765620 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ceea20810 con 0x555ceeb320e0 2024-03-24T10:47:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.248+0000 7f4d92d9d640 1 --2- 172.21.15.23:0/976765620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb24080 0x555ceeb24460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.248+0000 7f4d92d9d640 1 --2- 172.21.15.23:0/91256359 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f4d7c0a2350 0x7f4d7c0a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.248+0000 7f4d837fe640 1 -- 172.21.15.23:0/91256359 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f4d84014070 con 0x7f4d94100860 2024-03-24T10:47:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.248+0000 7f4d91d9b640 1 --2- 172.21.15.23:0/976765620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb320e0 0x555ceeb344d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4d88000c00 tx=0x7f4d8802c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a414db801e1bbe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.248+0000 7f4d80ff9640 1 -- 172.21.15.23:0/976765620 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d880047c0 con 0x555ceeb320e0 2024-03-24T10:47:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.248+0000 7f4d80ff9640 1 -- 172.21.15.23:0/976765620 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d88004920 con 0x555ceeb320e0 2024-03-24T10:47:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.248+0000 7f4d80ff9640 1 -- 172.21.15.23:0/976765620 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d880050e0 con 0x555ceeb320e0 2024-03-24T10:47:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.249+0000 7f4d9b423740 1 -- 172.21.15.23:0/976765620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb320e0 msgr2=0x555ceeb344d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.249+0000 7f4d92d9d640 1 --2- 172.21.15.23:0/91256359 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f4d7c0a2350 0x7f4d7c0a4810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c009fd0 tx=0x7f4d8c004a40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.249+0000 7f4d9b423740 1 --2- 172.21.15.23:0/976765620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb320e0 0x555ceeb344d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4d88000c00 tx=0x7f4d8802c6a0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.249+0000 7f4d9b423740 1 -- 172.21.15.23:0/976765620 shutdown_connections 2024-03-24T10:47:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.249+0000 7f4d9b423740 1 --2- 172.21.15.23:0/976765620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb320e0 0x555ceeb344d0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.249+0000 7f4d9b423740 1 --2- 172.21.15.23:0/976765620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb249a0 0x555ceeb31ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.249+0000 7f4d9b423740 1 --2- 172.21.15.23:0/976765620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb24080 0x555ceeb24460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.249+0000 7f4d9b423740 1 -- 172.21.15.23:0/976765620 >> 172.21.15.23:0/976765620 conn(0x555ceeb1ef00 msgr2=0x555ceeb1f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.249+0000 7f4d9b423740 1 -- 172.21.15.23:0/976765620 shutdown_connections 2024-03-24T10:47:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.249+0000 7f4d9b423740 1 -- 172.21.15.23:0/976765620 wait complete. 2024-03-24T10:47:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9b423740 1 -- start start 2024-03-24T10:47:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9b423740 1 Processor -- start 2024-03-24T10:47:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9b423740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb24080 0x555ceeb2f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9b423740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb249a0 0x555ceeb2f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9b423740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb320e0 0x555ceeb2fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9b423740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555ceea8ac90 con 0x555ceeb24080 2024-03-24T10:47:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9b423740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555ceea2fc50 con 0x555ceeb320e0 2024-03-24T10:47:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9b423740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555cee75e250 con 0x555ceeb249a0 2024-03-24T10:47:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9b423740 10 client.0 _ll_register_callbacks cb 0x555ceea7e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb24080 0x555ceeb2f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb24080 0x555ceeb2f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59666/0 (socket says 172.21.15.23:59666) 2024-03-24T10:47:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9259c640 1 -- 172.21.15.23:0/1021127315 learned_addr learned my addr 172.21.15.23:0/1021127315 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d91d9b640 1 --2- 172.21.15.23:0/1021127315 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb249a0 0x555ceeb2f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d92d9d640 1 --2- 172.21.15.23:0/1021127315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb320e0 0x555ceeb2fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9259c640 1 -- 172.21.15.23:0/1021127315 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb249a0 msgr2=0x555ceeb2f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9259c640 1 --2- 172.21.15.23:0/1021127315 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb249a0 0x555ceeb2f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9259c640 1 -- 172.21.15.23:0/1021127315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb320e0 msgr2=0x555ceeb2fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9259c640 1 --2- 172.21.15.23:0/1021127315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb320e0 0x555ceeb2fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d9259c640 1 -- 172.21.15.23:0/1021127315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d9410ddc0 con 0x555ceeb24080 2024-03-24T10:47:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.250+0000 7f4d91d9b640 1 --2- 172.21.15.23:0/1021127315 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb249a0 0x555ceeb2f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d9259c640 1 --2- 172.21.15.23:0/1021127315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb24080 0x555ceeb2f130 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4d840cc050 tx=0x7f4d8409ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c48fa204518b6a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d79ffb640 1 -- 172.21.15.23:0/1021127315 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8409ee40 con 0x555ceeb24080 2024-03-24T10:47:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d79ffb640 1 -- 172.21.15.23:0/1021127315 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f4d840a3490 con 0x555ceeb24080 2024-03-24T10:47:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d9b423740 1 -- 172.21.15.23:0/1021127315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555ceeaaf790 con 0x555ceeb24080 2024-03-24T10:47:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d79ffb640 1 -- 172.21.15.23:0/1021127315 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4d8409ada0 con 0x555ceeb24080 2024-03-24T10:47:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d9b423740 10 client.17700 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d9b423740 1 -- 172.21.15.23:0/1021127315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555ceea89310 con 0x555ceeb24080 2024-03-24T10:47:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d79ffb640 1 -- 172.21.15.23:0/1021127315 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555ceeaaf790 con 0x555ceeb24080 2024-03-24T10:47:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d79ffb640 1 -- 172.21.15.23:0/1021127315 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f4d84096c80 con 0x555ceeb24080 2024-03-24T10:47:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d9b423740 10 client.17700 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d9b423740 10 client.17700 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d9b423740 20 client.17700 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d9b423740 1 --2- 172.21.15.23:0/1021127315 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555ceeb420d0 0x555ceeb444e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d9b423740 4 client.17700 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d9b423740 1 -- 172.21.15.23:0/1021127315 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555cee78a600 con 0x555ceeb420d0 2024-03-24T10:47:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.251+0000 7f4d91d9b640 1 --2- 172.21.15.23:0/1021127315 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555ceeb420d0 0x555ceeb444e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.252+0000 7f4d91d9b640 1 --2- 172.21.15.23:0/1021127315 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555ceeb420d0 0x555ceeb444e0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce66515c85393e71 server_cookie=e51a26831b47e801 in_seq=0 out_seq=0 2024-03-24T10:47:30.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.252+0000 7f4d79ffb640 10 client.17700 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:30.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.252+0000 7f4d79ffb640 1 -- 172.21.15.23:0/1021127315 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555cee78a600 con 0x555ceeb420d0 2024-03-24T10:47:30.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.252+0000 7f4d79ffb640 10 client.17700 handle_command_reply: tid=0 2024-03-24T10:47:30.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.253+0000 7f4d9b423740 1 client.17700 shutdown 2024-03-24T10:47:30.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.253+0000 7f4d9b423740 1 -- 172.21.15.23:0/1021127315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb24080 msgr2=0x555ceeb2f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:30.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.253+0000 7f4d9b423740 1 --2- 172.21.15.23:0/1021127315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb24080 0x555ceeb2f130 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4d840cc050 tx=0x7f4d8409ac90 comp rx=0 tx=0).stop 2024-03-24T10:47:30.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.253+0000 7f4d9b423740 1 -- 172.21.15.23:0/1021127315 shutdown_connections 2024-03-24T10:47:30.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.253+0000 7f4d9b423740 1 --2- 172.21.15.23:0/1021127315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb320e0 0x555ceeb2fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.253+0000 7f4d9b423740 1 --2- 172.21.15.23:0/1021127315 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555ceeb420d0 0x555ceeb444e0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.253+0000 7f4d9b423740 1 --2- 172.21.15.23:0/1021127315 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb249a0 0x555ceeb2f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.253+0000 7f4d9b423740 1 --2- 172.21.15.23:0/1021127315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb24080 0x555ceeb2f130 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.253+0000 7f4d79ffb640 0 client.17700 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:30.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.253+0000 7f4d9b423740 1 -- 172.21.15.23:0/1021127315 >> 172.21.15.23:0/1021127315 conn(0x555ceeb1ef00 msgr2=0x555ceeb20c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:30.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.253+0000 7f4d9b423740 1 -- 172.21.15.23:0/1021127315 shutdown_connections 2024-03-24T10:47:30.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.253+0000 7f4d9b423740 1 -- 172.21.15.23:0/1021127315 wait complete. 2024-03-24T10:47:30.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.253+0000 7f4d9b423740 20 client.17700 trim_cache size 0 max 16384 2024-03-24T10:47:30.274 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.269+0000 7f4d9b423740 1 Processor -- start 2024-03-24T10:47:30.274 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.269+0000 7f4d9b423740 1 -- start start 2024-03-24T10:47:30.274 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.269+0000 7f4d9b423740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb24080 0x555ceeb3d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.274 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.269+0000 7f4d9b423740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb249a0 0x555ceeb2f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.274 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.269+0000 7f4d9b423740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb320e0 0x555ceeb1dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.269+0000 7f4d9b423740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555ceea10660 con 0x555ceeb24080 2024-03-24T10:47:30.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.269+0000 7f4d9b423740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555cee78a600 con 0x555ceeb249a0 2024-03-24T10:47:30.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.269+0000 7f4d9b423740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555ceeb1d3f0 con 0x555ceeb320e0 2024-03-24T10:47:30.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.270+0000 7f4d9259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb24080 0x555ceeb3d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.270+0000 7f4d9259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb24080 0x555ceeb3d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59682/0 (socket says 172.21.15.23:59682) 2024-03-24T10:47:30.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.270+0000 7f4d9259c640 1 -- 172.21.15.23:0/2307895469 learned_addr learned my addr 172.21.15.23:0/2307895469 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:30.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.270+0000 7f4d92d9d640 1 --2- 172.21.15.23:0/2307895469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb320e0 0x555ceeb1dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.270+0000 7f4d91d9b640 1 --2- 172.21.15.23:0/2307895469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb249a0 0x555ceeb2f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.270+0000 7f4d9259c640 1 -- 172.21.15.23:0/2307895469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb320e0 msgr2=0x555ceeb1dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:30.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.270+0000 7f4d9259c640 1 --2- 172.21.15.23:0/2307895469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb320e0 0x555ceeb1dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.270+0000 7f4d9259c640 1 -- 172.21.15.23:0/2307895469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb249a0 msgr2=0x555ceeb2f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:30.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.270+0000 7f4d9259c640 1 --2- 172.21.15.23:0/2307895469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb249a0 0x555ceeb2f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.276 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.270+0000 7f4d9259c640 1 -- 172.21.15.23:0/2307895469 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ceea25810 con 0x555ceeb24080 2024-03-24T10:47:30.276 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.270+0000 7f4d9259c640 1 --2- 172.21.15.23:0/2307895469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb24080 0x555ceeb3d7c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4d8409ad30 tx=0x7f4d840d1840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e1d60a01d0c121 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:30.276 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.270+0000 7f4d7bfff640 1 -- 172.21.15.23:0/2307895469 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d840a3490 con 0x555ceeb24080 2024-03-24T10:47:30.276 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.270+0000 7f4d7bfff640 1 -- 172.21.15.23:0/2307895469 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d8409ee40 con 0x555ceeb24080 2024-03-24T10:47:30.276 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.270+0000 7f4d7bfff640 1 -- 172.21.15.23:0/2307895469 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84096d10 con 0x555ceeb24080 2024-03-24T10:47:30.276 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.271+0000 7f4d9b423740 1 -- 172.21.15.23:0/2307895469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb24080 msgr2=0x555ceeb3d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:30.276 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.271+0000 7f4d9b423740 1 --2- 172.21.15.23:0/2307895469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb24080 0x555ceeb3d7c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4d8409ad30 tx=0x7f4d840d1840 comp rx=0 tx=0).stop 2024-03-24T10:47:30.276 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.271+0000 7f4d9b423740 1 -- 172.21.15.23:0/2307895469 shutdown_connections 2024-03-24T10:47:30.276 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.271+0000 7f4d9b423740 1 --2- 172.21.15.23:0/2307895469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb320e0 0x555ceeb1dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.276 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.271+0000 7f4d9b423740 1 --2- 172.21.15.23:0/2307895469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb249a0 0x555ceeb2f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.276 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.271+0000 7f4d9b423740 1 --2- 172.21.15.23:0/2307895469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb24080 0x555ceeb3d7c0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.276 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.271+0000 7f4d9b423740 1 -- 172.21.15.23:0/2307895469 >> 172.21.15.23:0/2307895469 conn(0x555ceeb1ef00 msgr2=0x555ceeb1d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:30.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.271+0000 7f4d9b423740 1 -- 172.21.15.23:0/2307895469 shutdown_connections 2024-03-24T10:47:30.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.271+0000 7f4d9b423740 1 -- 172.21.15.23:0/2307895469 wait complete. 2024-03-24T10:47:30.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.271+0000 7f4d9b423740 1 -- start start 2024-03-24T10:47:30.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.271+0000 7f4d9b423740 1 Processor -- start 2024-03-24T10:47:30.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d9b423740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb24080 0x555ceeb3a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d9b423740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb249a0 0x555ceeb2b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d9b423740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb320e0 0x555ceeb2be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d9b423740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x555ceeb2c550 con 0x555ceeb249a0 2024-03-24T10:47:30.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d9b423740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555ceeb2c6c0 con 0x555ceeb320e0 2024-03-24T10:47:30.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d9b423740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555ceeba73e0 con 0x555ceeb24080 2024-03-24T10:47:30.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d9b423740 10 client.0 _ll_register_callbacks cb 0x7f4d84010e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:30.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d91d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb249a0 0x555ceeb2b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d9259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb24080 0x555ceeb3a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d91d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb249a0 0x555ceeb2b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59692/0 (socket says 172.21.15.23:59692) 2024-03-24T10:47:30.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d91d9b640 1 -- 172.21.15.23:0/4155362994 learned_addr learned my addr 172.21.15.23:0/4155362994 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:30.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d92d9d640 1 --2- 172.21.15.23:0/4155362994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb320e0 0x555ceeb2be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d91d9b640 1 -- 172.21.15.23:0/4155362994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb24080 msgr2=0x555ceeb3a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:30.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d91d9b640 1 --2- 172.21.15.23:0/4155362994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb24080 0x555ceeb3a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d91d9b640 1 -- 172.21.15.23:0/4155362994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb320e0 msgr2=0x555ceeb2be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:30.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d91d9b640 1 --2- 172.21.15.23:0/4155362994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb320e0 0x555ceeb2be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d91d9b640 1 -- 172.21.15.23:0/4155362994 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ceea89310 con 0x555ceeb249a0 2024-03-24T10:47:30.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d91d9b640 1 --2- 172.21.15.23:0/4155362994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb249a0 0x555ceeb2b940 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f4d880045e0 tx=0x7f4d88004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=841ad0ac688aa8c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d79ffb640 1 -- 172.21.15.23:0/4155362994 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d88058070 con 0x555ceeb249a0 2024-03-24T10:47:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d79ffb640 1 -- 172.21.15.23:0/4155362994 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f4d8802cbf0 con 0x555ceeb249a0 2024-03-24T10:47:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d9b423740 1 -- 172.21.15.23:0/4155362994 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d88004920 con 0x555ceeb249a0 2024-03-24T10:47:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.272+0000 7f4d79ffb640 1 -- 172.21.15.23:0/4155362994 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4d8804e040 con 0x555ceeb249a0 2024-03-24T10:47:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.273+0000 7f4d9b423740 10 client.17706 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.273+0000 7f4d9b423740 1 -- 172.21.15.23:0/4155362994 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555ceea4b5a0 con 0x555ceeb249a0 2024-03-24T10:47:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.273+0000 7f4d79ffb640 1 -- 172.21.15.23:0/4155362994 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d88004920 con 0x555ceeb249a0 2024-03-24T10:47:30.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.273+0000 7f4d79ffb640 1 -- 172.21.15.23:0/4155362994 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f4d8805b020 con 0x555ceeb249a0 2024-03-24T10:47:30.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.273+0000 7f4d9b423740 10 client.17706 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:30.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.273+0000 7f4d9b423740 10 client.17706 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:30.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.273+0000 7f4d9b423740 20 client.17706 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:30.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.273+0000 7f4d9b423740 1 --2- 172.21.15.23:0/4155362994 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555ceeba8820 0x555ceebaac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.273+0000 7f4d9b423740 4 client.17706 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:47:30.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.273+0000 7f4d9b423740 1 -- 172.21.15.23:0/4155362994 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555cee7db1b0 con 0x555ceeba8820 2024-03-24T10:47:30.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.273+0000 7f4d9259c640 1 --2- 172.21.15.23:0/4155362994 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555ceeba8820 0x555ceebaac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.273+0000 7f4d9259c640 1 --2- 172.21.15.23:0/4155362994 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555ceeba8820 0x555ceebaac40 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ed75a2f9d67032a server_cookie=a50285bbfd2c92cb in_seq=0 out_seq=0 2024-03-24T10:47:30.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.273+0000 7f4d79ffb640 10 client.17706 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.284+0000 7f4d79ffb640 1 -- 172.21.15.23:0/4155362994 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555cee7db1b0 con 0x555ceeba8820 2024-03-24T10:47:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.284+0000 7f4d79ffb640 10 client.17706 handle_command_reply: tid=0 2024-03-24T10:47:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.284+0000 7f4d9b423740 1 client.17706 shutdown 2024-03-24T10:47:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.284+0000 7f4d9b423740 1 -- 172.21.15.23:0/4155362994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb249a0 msgr2=0x555ceeb2b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.284+0000 7f4d9b423740 1 --2- 172.21.15.23:0/4155362994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb249a0 0x555ceeb2b940 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f4d880045e0 tx=0x7f4d88004570 comp rx=0 tx=0).stop 2024-03-24T10:47:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.284+0000 7f4d9b423740 1 -- 172.21.15.23:0/4155362994 shutdown_connections 2024-03-24T10:47:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.284+0000 7f4d9b423740 1 --2- 172.21.15.23:0/4155362994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ceeb320e0 0x555ceeb2be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.284+0000 7f4d9b423740 1 --2- 172.21.15.23:0/4155362994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555ceeb249a0 0x555ceeb2b940 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.284+0000 7f4d9b423740 1 --2- 172.21.15.23:0/4155362994 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x555ceeba8820 0x555ceebaac40 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.284+0000 7f4d9b423740 1 --2- 172.21.15.23:0/4155362994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555ceeb24080 0x555ceeb3a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.284+0000 7f4d9b423740 1 -- 172.21.15.23:0/4155362994 >> 172.21.15.23:0/4155362994 conn(0x555ceeb1ef00 msgr2=0x555ceeb1dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.284+0000 7f4d79ffb640 0 client.17706 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.284+0000 7f4d9b423740 1 -- 172.21.15.23:0/4155362994 shutdown_connections 2024-03-24T10:47:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.284+0000 7f4d9b423740 1 -- 172.21.15.23:0/4155362994 wait complete. 2024-03-24T10:47:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.285+0000 7f4d9b423740 20 client.17706 trim_cache size 0 max 16384 2024-03-24T10:47:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:47:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:47:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:47:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:47:30.287 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.285+0000 7f4d98847640 1 -- 172.21.15.23:0/91256359 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f4d7c0a2350 msgr2=0x7f4d7c0a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:30.287 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.285+0000 7f4d98847640 1 --2- 172.21.15.23:0/91256359 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f4d7c0a2350 0x7f4d7c0a4810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c009fd0 tx=0x7f4d8c004a40 comp rx=0 tx=0).stop 2024-03-24T10:47:30.287 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.285+0000 7f4d98847640 1 -- 172.21.15.23:0/91256359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d94100860 msgr2=0x7f4d94197d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:30.287 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.285+0000 7f4d98847640 1 --2- 172.21.15.23:0/91256359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d94100860 0x7f4d94197d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4d8400e9f0 tx=0x7f4d8400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.285+0000 7f4d98847640 1 -- 172.21.15.23:0/91256359 shutdown_connections 2024-03-24T10:47:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.285+0000 7f4d98847640 1 --2- 172.21.15.23:0/91256359 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f4d7c0a2350 0x7f4d7c0a4810 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.285+0000 7f4d98847640 1 --2- 172.21.15.23:0/91256359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d94105590 0x7f4d9419c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.285+0000 7f4d98847640 1 --2- 172.21.15.23:0/91256359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d94101200 0x7f4d941982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.285+0000 7f4d98847640 1 --2- 172.21.15.23:0/91256359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d94100860 0x7f4d94197d90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.285+0000 7f4d98847640 1 -- 172.21.15.23:0/91256359 >> 172.21.15.23:0/91256359 conn(0x7f4d940773e0 msgr2=0x7f4d9406a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.285+0000 7f4d98847640 1 -- 172.21.15.23:0/91256359 shutdown_connections 2024-03-24T10:47:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:30.285+0000 7f4d98847640 1 -- 172.21.15.23:0/91256359 wait complete. 2024-03-24T10:47:30.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:30 smithi117 ceph-mon[36368]: pgmap v229: 97 pgs: 97 active+clean; 415 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 160 op/s 2024-03-24T10:47:30.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:30 smithi116 ceph-mon[36248]: pgmap v229: 97 pgs: 97 active+clean; 415 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 160 op/s 2024-03-24T10:47:30.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:30 smithi023 ceph-mon[32466]: pgmap v229: 97 pgs: 97 active+clean; 415 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 160 op/s 2024-03-24T10:47:31.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 728996 10.84 MB/sec execute 228 sec latency 20.776 ms 2024-03-24T10:47:32.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 730959 10.84 MB/sec execute 229 sec latency 84.738 ms 2024-03-24T10:47:32.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:32 smithi117 ceph-mon[36368]: pgmap v230: 97 pgs: 97 active+clean; 423 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 247 op/s 2024-03-24T10:47:32.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:32 smithi116 ceph-mon[36248]: pgmap v230: 97 pgs: 97 active+clean; 423 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 247 op/s 2024-03-24T10:47:32.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:32 smithi023 ceph-mon[32466]: pgmap v230: 97 pgs: 97 active+clean; 423 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 247 op/s 2024-03-24T10:47:33.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 732659 10.83 MB/sec execute 230 sec latency 25.488 ms 2024-03-24T10:47:34.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 734844 10.83 MB/sec execute 231 sec latency 33.166 ms 2024-03-24T10:47:34.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:34 smithi117 ceph-mon[36368]: pgmap v231: 97 pgs: 97 active+clean; 423 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 159 op/s 2024-03-24T10:47:34.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:34 smithi116 ceph-mon[36248]: pgmap v231: 97 pgs: 97 active+clean; 423 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 159 op/s 2024-03-24T10:47:34.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:34 smithi023 ceph-mon[32466]: pgmap v231: 97 pgs: 97 active+clean; 423 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 159 op/s 2024-03-24T10:47:35.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 736525 10.83 MB/sec execute 232 sec latency 17.311 ms 2024-03-24T10:47:36.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 738794 10.84 MB/sec execute 233 sec latency 22.846 ms 2024-03-24T10:47:36.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:36 smithi117 ceph-mon[36368]: pgmap v232: 97 pgs: 97 active+clean; 423 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 159 op/s 2024-03-24T10:47:36.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:36 smithi116 ceph-mon[36248]: pgmap v232: 97 pgs: 97 active+clean; 423 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 159 op/s 2024-03-24T10:47:36.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:36 smithi023 ceph-mon[32466]: pgmap v232: 97 pgs: 97 active+clean; 423 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 159 op/s 2024-03-24T10:47:37.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 740099 10.82 MB/sec execute 234 sec latency 99.192 ms 2024-03-24T10:47:38.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 742344 10.83 MB/sec execute 235 sec latency 51.149 ms 2024-03-24T10:47:38.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:38 smithi117 ceph-mon[36368]: pgmap v233: 97 pgs: 97 active+clean; 430 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 246 op/s 2024-03-24T10:47:38.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:38 smithi116 ceph-mon[36248]: pgmap v233: 97 pgs: 97 active+clean; 430 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 246 op/s 2024-03-24T10:47:38.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:38 smithi023 ceph-mon[32466]: pgmap v233: 97 pgs: 97 active+clean; 430 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 246 op/s 2024-03-24T10:47:39.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 743599 10.81 MB/sec execute 236 sec latency 26.731 ms 2024-03-24T10:47:40.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 745626 10.81 MB/sec execute 237 sec latency 28.138 ms 2024-03-24T10:47:40.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:40 smithi117 ceph-mon[36368]: pgmap v234: 97 pgs: 97 active+clean; 430 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 173 op/s 2024-03-24T10:47:40.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:40 smithi116 ceph-mon[36248]: pgmap v234: 97 pgs: 97 active+clean; 430 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 173 op/s 2024-03-24T10:47:40.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:40 smithi023 ceph-mon[32466]: pgmap v234: 97 pgs: 97 active+clean; 430 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 173 op/s 2024-03-24T10:47:41.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 747129 10.80 MB/sec execute 238 sec latency 19.293 ms 2024-03-24T10:47:42.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 749315 10.80 MB/sec execute 239 sec latency 50.616 ms 2024-03-24T10:47:42.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:42 smithi117 ceph-mon[36368]: pgmap v235: 97 pgs: 97 active+clean; 438 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 256 op/s 2024-03-24T10:47:42.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:42 smithi116 ceph-mon[36248]: pgmap v235: 97 pgs: 97 active+clean; 438 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 256 op/s 2024-03-24T10:47:42.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:42 smithi023 ceph-mon[32466]: pgmap v235: 97 pgs: 97 active+clean; 438 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 256 op/s 2024-03-24T10:47:43.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 750707 10.79 MB/sec execute 240 sec latency 19.081 ms 2024-03-24T10:47:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.374+0000 7f395fc33640 1 -- 172.21.15.23:0/3909152647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3958101220 msgr2=0x7f395810d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.374+0000 7f395fc33640 1 --2- 172.21.15.23:0/3909152647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3958101220 0x7f395810d440 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f3950009ca0 tx=0x7f395002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.374+0000 7f395fc33640 1 -- 172.21.15.23:0/3909152647 shutdown_connections 2024-03-24T10:47:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.374+0000 7f395fc33640 1 --2- 172.21.15.23:0/3909152647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3958101220 0x7f395810d440 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.374+0000 7f395fc33640 1 --2- 172.21.15.23:0/3909152647 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3958100860 0x7f3958100ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.374+0000 7f395fc33640 1 --2- 172.21.15.23:0/3909152647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39581055b0 0x7f3958105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.374+0000 7f395fc33640 1 -- 172.21.15.23:0/3909152647 >> 172.21.15.23:0/3909152647 conn(0x7f3958077460 msgr2=0x7f3958077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.374+0000 7f395fc33640 1 -- 172.21.15.23:0/3909152647 shutdown_connections 2024-03-24T10:47:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.374+0000 7f395fc33640 1 -- 172.21.15.23:0/3909152647 wait complete. 2024-03-24T10:47:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.374+0000 7f395fc33640 1 Processor -- start 2024-03-24T10:47:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.374+0000 7f395fc33640 1 -- start start 2024-03-24T10:47:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395fc33640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3958100860 0x7f3958199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395fc33640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3958101220 0x7f395819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395fc33640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39581055b0 0x7f395819e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395fc33640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f395819edc0 con 0x7f3958101220 2024-03-24T10:47:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395fc33640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f395819ef30 con 0x7f3958100860 2024-03-24T10:47:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395fc33640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f395819f0f0 con 0x7f39581055b0 2024-03-24T10:47:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395d1ce640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3958101220 0x7f395819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395d1ce640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3958101220 0x7f395819a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36384/0 (socket says 172.21.15.23:36384) 2024-03-24T10:47:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395d1ce640 1 -- 172.21.15.23:0/2317534353 learned_addr learned my addr 172.21.15.23:0/2317534353 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395d9cf640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3958100860 0x7f3958199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395e1d0640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39581055b0 0x7f395819e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395d1ce640 1 -- 172.21.15.23:0/2317534353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39581055b0 msgr2=0x7f395819e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395d1ce640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39581055b0 0x7f395819e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395d1ce640 1 -- 172.21.15.23:0/2317534353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3958100860 msgr2=0x7f3958199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395d1ce640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3958100860 0x7f3958199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395d1ce640 1 -- 172.21.15.23:0/2317534353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3950009900 con 0x7f3958101220 2024-03-24T10:47:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395d9cf640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3958100860 0x7f3958199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395d1ce640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3958101220 0x7f395819a440 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f395400e9f0 tx=0x7f395400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f394effd640 1 -- 172.21.15.23:0/2317534353 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3954009720 con 0x7f3958101220 2024-03-24T10:47:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395fc33640 1 -- 172.21.15.23:0/2317534353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f395819f3d0 con 0x7f3958101220 2024-03-24T10:47:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f394effd640 1 -- 172.21.15.23:0/2317534353 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f395400cd80 con 0x7f3958101220 2024-03-24T10:47:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f394effd640 1 -- 172.21.15.23:0/2317534353 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3954010690 con 0x7f3958101220 2024-03-24T10:47:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.375+0000 7f395fc33640 1 -- 172.21.15.23:0/2317534353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39581a6c50 con 0x7f3958101220 2024-03-24T10:47:43.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f394effd640 1 -- 172.21.15.23:0/2317534353 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f39540046c0 con 0x7f3958101220 2024-03-24T10:47:43.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f396280c740 1 Processor -- start 2024-03-24T10:47:43.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f396280c740 1 -- start start 2024-03-24T10:47:43.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f396280c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5575837825e0 0x5575837829c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.386 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f396280c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 0x55758378b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f396280c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55758378bcd0 0x55758378e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f396280c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55758373d1b0 con 0x55758378bcd0 2024-03-24T10:47:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f396280c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5575836eff80 con 0x557583782f00 2024-03-24T10:47:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f396280c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5575836ed4b0 con 0x5575837825e0 2024-03-24T10:47:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f395d1ce640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55758378bcd0 0x55758378e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f395d9cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 0x55758378b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f395e1d0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5575837825e0 0x5575837829c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f395d1ce640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55758378bcd0 0x55758378e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36398/0 (socket says 172.21.15.23:36398) 2024-03-24T10:47:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f395d1ce640 1 -- 172.21.15.23:0/3484323325 learned_addr learned my addr 172.21.15.23:0/3484323325 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.378+0000 7f395d9cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 0x55758378b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:52740/0 (socket says 172.21.15.23:52740) 2024-03-24T10:47:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.379+0000 7f394effd640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f39440a22d0 0x7f39440a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.379+0000 7f395e1d0640 1 -- 172.21.15.23:0/3484323325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 msgr2=0x55758378b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.379+0000 7f395e1d0640 1 --2- 172.21.15.23:0/3484323325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 0x55758378b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.379+0000 7f395e1d0640 1 -- 172.21.15.23:0/3484323325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55758378bcd0 msgr2=0x55758378e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.379+0000 7f395e1d0640 1 --2- 172.21.15.23:0/3484323325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55758378bcd0 0x55758378e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.379+0000 7f395e1d0640 1 -- 172.21.15.23:0/3484323325 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55758367e810 con 0x5575837825e0 2024-03-24T10:47:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.379+0000 7f394effd640 1 -- 172.21.15.23:0/2317534353 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f3954014070 con 0x7f3958101220 2024-03-24T10:47:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.379+0000 7f395e1d0640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f39440a22d0 0x7f39440a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.379+0000 7f395e1d0640 1 --2- 172.21.15.23:0/3484323325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5575837825e0 0x5575837829c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f395002c700 tx=0x7f3950032de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6f947acf3c00e7a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.379+0000 7f3943fff640 1 -- 172.21.15.23:0/3484323325 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39500326a0 con 0x5575837825e0 2024-03-24T10:47:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.379+0000 7f3943fff640 1 -- 172.21.15.23:0/3484323325 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3950003e50 con 0x5575837825e0 2024-03-24T10:47:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.379+0000 7f3943fff640 1 -- 172.21.15.23:0/3484323325 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39500044e0 con 0x5575837825e0 2024-03-24T10:47:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.380+0000 7f396280c740 1 -- 172.21.15.23:0/3484323325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5575837825e0 msgr2=0x5575837829c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.380+0000 7f395e1d0640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f39440a22d0 0x7f39440a4790 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3950005cc0 tx=0x7f395004a000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.380+0000 7f396280c740 1 --2- 172.21.15.23:0/3484323325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5575837825e0 0x5575837829c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f395002c700 tx=0x7f3950032de0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.380+0000 7f396280c740 1 -- 172.21.15.23:0/3484323325 shutdown_connections 2024-03-24T10:47:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.380+0000 7f396280c740 1 --2- 172.21.15.23:0/3484323325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55758378bcd0 0x55758378e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.380+0000 7f396280c740 1 --2- 172.21.15.23:0/3484323325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 0x55758378b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.380+0000 7f396280c740 1 --2- 172.21.15.23:0/3484323325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5575837825e0 0x5575837829c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.380+0000 7f396280c740 1 -- 172.21.15.23:0/3484323325 >> 172.21.15.23:0/3484323325 conn(0x55758377cf00 msgr2=0x55758377d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.380+0000 7f396280c740 1 -- 172.21.15.23:0/3484323325 shutdown_connections 2024-03-24T10:47:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.380+0000 7f396280c740 1 -- 172.21.15.23:0/3484323325 wait complete. 2024-03-24T10:47:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.380+0000 7f396280c740 1 -- start start 2024-03-24T10:47:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f396280c740 1 Processor -- start 2024-03-24T10:47:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f396280c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575837825e0 0x557583788d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f396280c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 0x557583789280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f396280c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55758378bcd0 0x5575837897c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f396280c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5575836e8c90 con 0x5575837825e0 2024-03-24T10:47:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f396280c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55758368dc50 con 0x557583782f00 2024-03-24T10:47:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f396280c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5575833bc250 con 0x55758378bcd0 2024-03-24T10:47:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f396280c740 10 client.0 _ll_register_callbacks cb 0x5575836dc290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f395d9cf640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575837825e0 0x557583788d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f395d9cf640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575837825e0 0x557583788d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36412/0 (socket says 172.21.15.23:36412) 2024-03-24T10:47:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f395d9cf640 1 -- 172.21.15.23:0/181811429 learned_addr learned my addr 172.21.15.23:0/181811429 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f395d1ce640 1 --2- 172.21.15.23:0/181811429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 0x557583789280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f395e1d0640 1 --2- 172.21.15.23:0/181811429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55758378bcd0 0x5575837897c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f395d9cf640 1 -- 172.21.15.23:0/181811429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55758378bcd0 msgr2=0x5575837897c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f395d9cf640 1 --2- 172.21.15.23:0/181811429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55758378bcd0 0x5575837897c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f395d9cf640 1 -- 172.21.15.23:0/181811429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 msgr2=0x557583789280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f395d9cf640 1 --2- 172.21.15.23:0/181811429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 0x557583789280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.381+0000 7f395d9cf640 1 -- 172.21.15.23:0/181811429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f395810ddf0 con 0x5575837825e0 2024-03-24T10:47:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f395d9cf640 1 --2- 172.21.15.23:0/181811429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575837825e0 0x557583788d40 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f3948012d20 tx=0x7f394800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2063c1b368f53b43 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f39417fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f39417fa640 1 -- 172.21.15.23:0/181811429 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3948017070 con 0x5575837825e0 2024-03-24T10:47:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f39417fa640 1 -- 172.21.15.23:0/181811429 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f3948012110 con 0x5575837825e0 2024-03-24T10:47:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f396280c740 1 -- 172.21.15.23:0/181811429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55758370d790 con 0x5575837825e0 2024-03-24T10:47:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f39417fa640 1 -- 172.21.15.23:0/181811429 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f39480122d0 con 0x5575837825e0 2024-03-24T10:47:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f396280c740 10 client.17718 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f396280c740 1 -- 172.21.15.23:0/181811429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5575836e7310 con 0x5575837825e0 2024-03-24T10:47:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f39417fa640 1 -- 172.21.15.23:0/181811429 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55758370d790 con 0x5575837825e0 2024-03-24T10:47:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f39417fa640 1 -- 172.21.15.23:0/181811429 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f3948010430 con 0x5575837825e0 2024-03-24T10:47:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f396280c740 10 client.17718 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f396280c740 10 client.17718 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f396280c740 20 client.17718 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:43.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f396280c740 1 --2- 172.21.15.23:0/181811429 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55758379bcd0 0x55758379e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f396280c740 4 client.17718 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.382+0000 7f396280c740 1 -- 172.21.15.23:0/181811429 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575833e8600 con 0x55758379bcd0 2024-03-24T10:47:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.383+0000 7f395d1ce640 1 --2- 172.21.15.23:0/181811429 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55758379bcd0 0x55758379e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.383+0000 7f395d1ce640 1 --2- 172.21.15.23:0/181811429 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55758379bcd0 0x55758379e110 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed0b453e49b48334 server_cookie=69ef0c7bf27ec4c0 in_seq=0 out_seq=0 2024-03-24T10:47:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.383+0000 7f39417fa640 10 client.17718 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f39417fa640 1 -- 172.21.15.23:0/181811429 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5575833e8600 con 0x55758379bcd0 2024-03-24T10:47:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f39417fa640 10 client.17718 handle_command_reply: tid=0 2024-03-24T10:47:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f396280c740 1 client.17718 shutdown 2024-03-24T10:47:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f396280c740 1 -- 172.21.15.23:0/181811429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575837825e0 msgr2=0x557583788d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f396280c740 1 --2- 172.21.15.23:0/181811429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575837825e0 0x557583788d40 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f3948012d20 tx=0x7f394800f790 comp rx=0 tx=0).stop 2024-03-24T10:47:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f396280c740 1 -- 172.21.15.23:0/181811429 shutdown_connections 2024-03-24T10:47:43.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f396280c740 1 --2- 172.21.15.23:0/181811429 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55758379bcd0 0x55758379e110 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f396280c740 1 --2- 172.21.15.23:0/181811429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55758378bcd0 0x5575837897c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f396280c740 1 --2- 172.21.15.23:0/181811429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 0x557583789280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f396280c740 1 --2- 172.21.15.23:0/181811429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575837825e0 0x557583788d40 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f39417fa640 0 client.17718 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:43.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f396280c740 1 -- 172.21.15.23:0/181811429 >> 172.21.15.23:0/181811429 conn(0x55758377cf00 msgr2=0x55758377ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:43.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f396280c740 1 -- 172.21.15.23:0/181811429 shutdown_connections 2024-03-24T10:47:43.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f396280c740 1 -- 172.21.15.23:0/181811429 wait complete. 2024-03-24T10:47:43.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.384+0000 7f396280c740 20 client.17718 trim_cache size 0 max 16384 2024-03-24T10:47:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.399+0000 7f396280c740 1 Processor -- start 2024-03-24T10:47:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.399+0000 7f396280c740 1 -- start start 2024-03-24T10:47:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f396280c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575837825e0 0x5575837973a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f396280c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557583782f00 0x5575837895e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f396280c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55758378bcd0 0x55758377bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f396280c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55758366e660 con 0x557583782f00 2024-03-24T10:47:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f396280c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557583783b80 con 0x5575837825e0 2024-03-24T10:47:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f396280c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557583777140 con 0x55758378bcd0 2024-03-24T10:47:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f395d1ce640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557583782f00 0x5575837895e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f395d1ce640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557583782f00 0x5575837895e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36420/0 (socket says 172.21.15.23:36420) 2024-03-24T10:47:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f395d1ce640 1 -- 172.21.15.23:0/1787202876 learned_addr learned my addr 172.21.15.23:0/1787202876 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f395d9cf640 1 --2- 172.21.15.23:0/1787202876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575837825e0 0x5575837973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f395e1d0640 1 --2- 172.21.15.23:0/1787202876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55758378bcd0 0x55758377bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f395d1ce640 1 -- 172.21.15.23:0/1787202876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55758378bcd0 msgr2=0x55758377bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f395d1ce640 1 --2- 172.21.15.23:0/1787202876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55758378bcd0 0x55758377bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f395d1ce640 1 -- 172.21.15.23:0/1787202876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575837825e0 msgr2=0x5575837973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f395d1ce640 1 --2- 172.21.15.23:0/1787202876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575837825e0 0x5575837973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f395d1ce640 1 -- 172.21.15.23:0/1787202876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557583683810 con 0x557583782f00 2024-03-24T10:47:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f395d1ce640 1 --2- 172.21.15.23:0/1787202876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557583782f00 0x5575837895e0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f395400c6a0 tx=0x7f39540971d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf45c08e7bb56614 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f39437fe640 1 -- 172.21.15.23:0/1787202876 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395409d030 con 0x557583782f00 2024-03-24T10:47:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f39437fe640 1 -- 172.21.15.23:0/1787202876 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f395409a940 con 0x557583782f00 2024-03-24T10:47:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.400+0000 7f39437fe640 1 -- 172.21.15.23:0/1787202876 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39540cbde0 con 0x557583782f00 2024-03-24T10:47:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.401+0000 7f396280c740 1 -- 172.21.15.23:0/1787202876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557583782f00 msgr2=0x5575837895e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:43.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.401+0000 7f396280c740 1 --2- 172.21.15.23:0/1787202876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557583782f00 0x5575837895e0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f395400c6a0 tx=0x7f39540971d0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.401+0000 7f396280c740 1 -- 172.21.15.23:0/1787202876 shutdown_connections 2024-03-24T10:47:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.401+0000 7f396280c740 1 --2- 172.21.15.23:0/1787202876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55758378bcd0 0x55758377bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.401+0000 7f396280c740 1 --2- 172.21.15.23:0/1787202876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557583782f00 0x5575837895e0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.401+0000 7f396280c740 1 --2- 172.21.15.23:0/1787202876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5575837825e0 0x5575837973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.401+0000 7f396280c740 1 -- 172.21.15.23:0/1787202876 >> 172.21.15.23:0/1787202876 conn(0x55758377cf00 msgr2=0x55758377b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.401+0000 7f396280c740 1 -- 172.21.15.23:0/1787202876 shutdown_connections 2024-03-24T10:47:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.401+0000 7f396280c740 1 -- 172.21.15.23:0/1787202876 wait complete. 2024-03-24T10:47:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.401+0000 7f396280c740 1 -- start start 2024-03-24T10:47:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.401+0000 7f396280c740 1 Processor -- start 2024-03-24T10:47:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f396280c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5575837825e0 0x55758378fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f396280c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 0x55758378ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f396280c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55758378bcd0 0x557583801050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f396280c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x557583801590 con 0x55758378bcd0 2024-03-24T10:47:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f396280c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5575838016d0 con 0x557583782f00 2024-03-24T10:47:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f396280c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557583801810 con 0x5575837825e0 2024-03-24T10:47:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f396280c740 10 client.0 _ll_register_callbacks cb 0x7f3948009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f395e1d0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55758378bcd0 0x557583801050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f395d9cf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5575837825e0 0x55758378fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f395d9cf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5575837825e0 0x55758378fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:50552/0 (socket says 172.21.15.23:50552) 2024-03-24T10:47:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f395e1d0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55758378bcd0 0x557583801050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36426/0 (socket says 172.21.15.23:36426) 2024-03-24T10:47:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f395d9cf640 1 -- 172.21.15.23:0/793270818 learned_addr learned my addr 172.21.15.23:0/793270818 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f395d1ce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 0x55758378ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f395e1d0640 1 -- 172.21.15.23:0/793270818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5575837825e0 msgr2=0x55758378fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f395e1d0640 1 --2- 172.21.15.23:0/793270818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5575837825e0 0x55758378fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f395e1d0640 1 -- 172.21.15.23:0/793270818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 msgr2=0x55758378ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f395e1d0640 1 --2- 172.21.15.23:0/793270818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 0x55758378ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f395e1d0640 1 -- 172.21.15.23:0/793270818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39581a6c50 con 0x55758378bcd0 2024-03-24T10:47:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f395d1ce640 1 --2- 172.21.15.23:0/793270818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 0x55758378ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f395e1d0640 1 --2- 172.21.15.23:0/793270818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55758378bcd0 0x557583801050 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f3950037040 tx=0x7f395002c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d50e3207d9d0145 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f39417fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.402+0000 7f39417fa640 1 -- 172.21.15.23:0/793270818 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3950032480 con 0x55758378bcd0 2024-03-24T10:47:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.403+0000 7f39417fa640 1 -- 172.21.15.23:0/793270818 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f3950004240 con 0x55758378bcd0 2024-03-24T10:47:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.403+0000 7f396280c740 1 -- 172.21.15.23:0/793270818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3950003e50 con 0x55758378bcd0 2024-03-24T10:47:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.403+0000 7f39417fa640 1 -- 172.21.15.23:0/793270818 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3950034e60 con 0x55758378bcd0 2024-03-24T10:47:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.403+0000 7f396280c740 10 client.17724 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.403+0000 7f396280c740 1 -- 172.21.15.23:0/793270818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5575836a95a0 con 0x55758378bcd0 2024-03-24T10:47:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.403+0000 7f39417fa640 1 -- 172.21.15.23:0/793270818 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f395006b080 con 0x55758378bcd0 2024-03-24T10:47:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.403+0000 7f39417fa640 1 -- 172.21.15.23:0/793270818 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f395004a990 con 0x55758378bcd0 2024-03-24T10:47:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.403+0000 7f396280c740 10 client.17724 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.403+0000 7f396280c740 10 client.17724 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:43.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.403+0000 7f396280c740 20 client.17724 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.403+0000 7f396280c740 1 --2- 172.21.15.23:0/793270818 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557583802cc0 0x5575838050b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.403+0000 7f396280c740 4 client.17724 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:47:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.403+0000 7f396280c740 1 -- 172.21.15.23:0/793270818 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575834391b0 con 0x557583802cc0 2024-03-24T10:47:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.403+0000 7f395d9cf640 1 --2- 172.21.15.23:0/793270818 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557583802cc0 0x5575838050b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.404+0000 7f39417fa640 10 client.17724 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:43.409 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.404+0000 7f395d9cf640 1 --2- 172.21.15.23:0/793270818 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557583802cc0 0x5575838050b0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc21cac26c899603 server_cookie=167f6169afabbf5f in_seq=0 out_seq=0 2024-03-24T10:47:43.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.415+0000 7f39417fa640 1 -- 172.21.15.23:0/793270818 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5575834391b0 con 0x557583802cc0 2024-03-24T10:47:43.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.415+0000 7f39417fa640 10 client.17724 handle_command_reply: tid=0 2024-03-24T10:47:43.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.415+0000 7f396280c740 1 client.17724 shutdown 2024-03-24T10:47:43.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.415+0000 7f396280c740 1 -- 172.21.15.23:0/793270818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55758378bcd0 msgr2=0x557583801050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:43.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.415+0000 7f396280c740 1 --2- 172.21.15.23:0/793270818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55758378bcd0 0x557583801050 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f3950037040 tx=0x7f395002c770 comp rx=0 tx=0).stop 2024-03-24T10:47:43.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f396280c740 1 -- 172.21.15.23:0/793270818 shutdown_connections 2024-03-24T10:47:43.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f396280c740 1 --2- 172.21.15.23:0/793270818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55758378bcd0 0x557583801050 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f396280c740 1 --2- 172.21.15.23:0/793270818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557583782f00 0x55758378ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f396280c740 1 --2- 172.21.15.23:0/793270818 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x557583802cc0 0x5575838050b0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f396280c740 1 --2- 172.21.15.23:0/793270818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5575837825e0 0x55758378fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f39417fa640 0 client.17724 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f396280c740 1 -- 172.21.15.23:0/793270818 >> 172.21.15.23:0/793270818 conn(0x55758377cf00 msgr2=0x55758377bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f396280c740 1 -- 172.21.15.23:0/793270818 shutdown_connections 2024-03-24T10:47:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f396280c740 1 -- 172.21.15.23:0/793270818 wait complete. 2024-03-24T10:47:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f396280c740 20 client.17724 trim_cache size 0 max 16384 2024-03-24T10:47:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:47:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:47:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:47:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:47:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f395fc33640 1 -- 172.21.15.23:0/2317534353 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f39440a22d0 msgr2=0x7f39440a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f395fc33640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f39440a22d0 0x7f39440a4790 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3950005cc0 tx=0x7f395004a000 comp rx=0 tx=0).stop 2024-03-24T10:47:43.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f395fc33640 1 -- 172.21.15.23:0/2317534353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3958101220 msgr2=0x7f395819a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f395fc33640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3958101220 0x7f395819a440 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f395400e9f0 tx=0x7f395400edf0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f395fc33640 1 -- 172.21.15.23:0/2317534353 shutdown_connections 2024-03-24T10:47:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f395fc33640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f39440a22d0 0x7f39440a4790 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f395fc33640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39581055b0 0x7f395819e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f395fc33640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3958101220 0x7f395819a440 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.416+0000 7f395fc33640 1 --2- 172.21.15.23:0/2317534353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3958100860 0x7f3958199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.417+0000 7f395fc33640 1 -- 172.21.15.23:0/2317534353 >> 172.21.15.23:0/2317534353 conn(0x7f3958077460 msgr2=0x7f395810ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.417+0000 7f395fc33640 1 -- 172.21.15.23:0/2317534353 shutdown_connections 2024-03-24T10:47:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:43.417+0000 7f395fc33640 1 -- 172.21.15.23:0/2317534353 wait complete. 2024-03-24T10:47:44.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 752923 10.79 MB/sec execute 241 sec latency 19.400 ms 2024-03-24T10:47:44.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:44 smithi117 ceph-mon[36368]: pgmap v236: 97 pgs: 97 active+clean; 438 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 170 op/s 2024-03-24T10:47:44.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:44 smithi116 ceph-mon[36248]: pgmap v236: 97 pgs: 97 active+clean; 438 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 170 op/s 2024-03-24T10:47:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:44 smithi023 ceph-mon[32466]: pgmap v236: 97 pgs: 97 active+clean; 438 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 170 op/s 2024-03-24T10:47:45.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 754298 10.78 MB/sec execute 242 sec latency 23.975 ms 2024-03-24T10:47:46.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 756562 10.79 MB/sec execute 243 sec latency 29.358 ms 2024-03-24T10:47:46.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:47:46.508 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:47:46.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 -- 172.21.15.23:0/2918335339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a38106660 msgr2=0x7f1a38106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 --2- 172.21.15.23:0/2918335339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a38106660 0x7f1a38106a40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009930 tx=0x7f1a3402bc20 comp rx=0 tx=0).stop 2024-03-24T10:47:46.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 -- 172.21.15.23:0/2918335339 shutdown_connections 2024-03-24T10:47:46.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 --2- 172.21.15.23:0/2918335339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a38100f60 0x7f1a3810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 --2- 172.21.15.23:0/2918335339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a381005a0 0x7f1a38100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 --2- 172.21.15.23:0/2918335339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a38106660 0x7f1a38106a40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 -- 172.21.15.23:0/2918335339 >> 172.21.15.23:0/2918335339 conn(0x7f1a38077710 msgr2=0x7f1a38077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:46.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 -- 172.21.15.23:0/2918335339 shutdown_connections 2024-03-24T10:47:46.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 -- 172.21.15.23:0/2918335339 wait complete. 2024-03-24T10:47:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 Processor -- start 2024-03-24T10:47:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 -- start start 2024-03-24T10:47:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a381005a0 0x7f1a3819da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a38100f60 0x7f1a3819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a38106660 0x7f1a38197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f1a38198310 con 0x7f1a381005a0 2024-03-24T10:47:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a38198480 con 0x7f1a38106660 2024-03-24T10:47:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.619+0000 7f1a3fb63640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1a38198640 con 0x7f1a38100f60 2024-03-24T10:47:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a3d8ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a381005a0 0x7f1a3819da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a3d8ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a381005a0 0x7f1a3819da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36434/0 (socket says 172.21.15.23:36434) 2024-03-24T10:47:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/3359984248 learned_addr learned my addr 172.21.15.23:0/3359984248 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a3d0fe640 1 --2- 172.21.15.23:0/3359984248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a38100f60 0x7f1a3819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a3e100640 1 --2- 172.21.15.23:0/3359984248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a38106660 0x7f1a38197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/3359984248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a38100f60 msgr2=0x7f1a3819dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a3d8ff640 1 --2- 172.21.15.23:0/3359984248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a38100f60 0x7f1a3819dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/3359984248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a38106660 msgr2=0x7f1a38197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a3d8ff640 1 --2- 172.21.15.23:0/3359984248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a38106660 0x7f1a38197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/3359984248 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a30009840 con 0x7f1a381005a0 2024-03-24T10:47:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a3d8ff640 1 --2- 172.21.15.23:0/3359984248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a381005a0 0x7f1a3819da70 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009230 tx=0x7f1a3402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a2effd640 1 -- 172.21.15.23:0/3359984248 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a34004350 con 0x7f1a381005a0 2024-03-24T10:47:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a3fb63640 1 -- 172.21.15.23:0/3359984248 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a34009560 con 0x7f1a381005a0 2024-03-24T10:47:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a2effd640 1 -- 172.21.15.23:0/3359984248 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a340044b0 con 0x7f1a381005a0 2024-03-24T10:47:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a2effd640 1 -- 172.21.15.23:0/3359984248 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a34003bf0 con 0x7f1a381005a0 2024-03-24T10:47:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.620+0000 7f1a3fb63640 1 -- 172.21.15.23:0/3359984248 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a381a49f0 con 0x7f1a381005a0 2024-03-24T10:47:46.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.622+0000 7f1a2effd640 1 -- 172.21.15.23:0/3359984248 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f1a34003d50 con 0x7f1a381005a0 2024-03-24T10:47:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.623+0000 7f1a42816740 1 Processor -- start 2024-03-24T10:47:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.623+0000 7f1a42816740 1 -- start start 2024-03-24T10:47:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.623+0000 7f1a2effd640 1 --2- 172.21.15.23:0/3359984248 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1a240a2230 0x7f1a240a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.623+0000 7f1a42816740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21c1d0 0x55e66f21c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.623+0000 7f1a42816740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21caf0 0x55e66f225190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.623+0000 7f1a42816740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f2256d0 0x55e66f227ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.623+0000 7f1a42816740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e66f140f30 con 0x55e66f2256d0 2024-03-24T10:47:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.623+0000 7f1a42816740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e66f18a810 con 0x55e66f21caf0 2024-03-24T10:47:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.623+0000 7f1a42816740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e66f187d40 con 0x55e66f21c1d0 2024-03-24T10:47:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.623+0000 7f1a3e100640 1 --2- 172.21.15.23:0/3359984248 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1a240a2230 0x7f1a240a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.623+0000 7f1a2effd640 1 -- 172.21.15.23:0/3359984248 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f1a340eff20 con 0x7f1a381005a0 2024-03-24T10:47:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a3d8ff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21c1d0 0x55e66f21c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a3d0fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21caf0 0x55e66f225190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a3e100640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f2256d0 0x55e66f227ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a3d0fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21caf0 0x55e66f225190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:52774/0 (socket says 172.21.15.23:52774) 2024-03-24T10:47:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a3d0fe640 1 -- 172.21.15.23:0/4012137663 learned_addr learned my addr 172.21.15.23:0/4012137663 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/4012137663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21caf0 msgr2=0x55e66f225190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a3d8ff640 1 --2- 172.21.15.23:0/4012137663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21caf0 0x55e66f225190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/4012137663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f2256d0 msgr2=0x55e66f227ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a3d8ff640 1 --2- 172.21.15.23:0/4012137663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f2256d0 0x55e66f227ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/4012137663 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e66f144ea0 con 0x55e66f21c1d0 2024-03-24T10:47:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a3e100640 1 --2- 172.21.15.23:0/3359984248 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1a240a2230 0x7f1a240a46f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1a30005fd0 tx=0x7f1a300114f0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a3d8ff640 1 --2- 172.21.15.23:0/4012137663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21c1d0 0x55e66f21c5b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1a340ed780 tx=0x7f1a34005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeaf6f7044bf46bf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a23fff640 1 -- 172.21.15.23:0/4012137663 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3409a040 con 0x55e66f21c1d0 2024-03-24T10:47:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a23fff640 1 -- 172.21.15.23:0/4012137663 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a34099e60 con 0x55e66f21c1d0 2024-03-24T10:47:46.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.624+0000 7f1a23fff640 1 -- 172.21.15.23:0/4012137663 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a340c5230 con 0x55e66f21c1d0 2024-03-24T10:47:46.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.625+0000 7f1a42816740 1 -- 172.21.15.23:0/4012137663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21c1d0 msgr2=0x55e66f21c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.625+0000 7f1a42816740 1 --2- 172.21.15.23:0/4012137663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21c1d0 0x55e66f21c5b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1a340ed780 tx=0x7f1a34005fd0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.625+0000 7f1a42816740 1 -- 172.21.15.23:0/4012137663 shutdown_connections 2024-03-24T10:47:46.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.625+0000 7f1a42816740 1 --2- 172.21.15.23:0/4012137663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f2256d0 0x55e66f227ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.625+0000 7f1a42816740 1 --2- 172.21.15.23:0/4012137663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21caf0 0x55e66f225190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.625+0000 7f1a42816740 1 --2- 172.21.15.23:0/4012137663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21c1d0 0x55e66f21c5b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.625+0000 7f1a42816740 1 -- 172.21.15.23:0/4012137663 >> 172.21.15.23:0/4012137663 conn(0x55e66f216900 msgr2=0x55e66f216ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:46.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.625+0000 7f1a42816740 1 -- 172.21.15.23:0/4012137663 shutdown_connections 2024-03-24T10:47:46.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.625+0000 7f1a42816740 1 -- 172.21.15.23:0/4012137663 wait complete. 2024-03-24T10:47:46.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.625+0000 7f1a42816740 1 -- start start 2024-03-24T10:47:46.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a42816740 1 Processor -- start 2024-03-24T10:47:46.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a42816740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f21c1d0 0x55e66f2226a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a42816740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21caf0 0x55e66f222be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a42816740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f2256d0 0x55e66f223120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a42816740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e66f183520 con 0x55e66f21c1d0 2024-03-24T10:47:46.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a42816740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e66ee56430 con 0x55e66f21caf0 2024-03-24T10:47:46.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a42816740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e66ee12e00 con 0x55e66f2256d0 2024-03-24T10:47:46.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a42816740 10 client.0 _ll_register_callbacks cb 0x55e66f121d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:46.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a3d8ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f21c1d0 0x55e66f2226a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a3d8ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f21c1d0 0x55e66f2226a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36450/0 (socket says 172.21.15.23:36450) 2024-03-24T10:47:46.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/2690164340 learned_addr learned my addr 172.21.15.23:0/2690164340 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:46.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a3d0fe640 1 --2- 172.21.15.23:0/2690164340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21caf0 0x55e66f222be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a3e100640 1 --2- 172.21.15.23:0/2690164340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f2256d0 0x55e66f223120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/2690164340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f2256d0 msgr2=0x55e66f223120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:46.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a3d8ff640 1 --2- 172.21.15.23:0/2690164340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f2256d0 0x55e66f223120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/2690164340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21caf0 msgr2=0x55e66f222be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a3d8ff640 1 --2- 172.21.15.23:0/2690164340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21caf0 0x55e66f222be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.626+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/2690164340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a381a49f0 con 0x55e66f21c1d0 2024-03-24T10:47:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a3d8ff640 1 --2- 172.21.15.23:0/2690164340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f21c1d0 0x55e66f2226a0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f1a340ed7e0 tx=0x7f1a340baf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e561b7654f401b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a217fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a217fa640 1 -- 172.21.15.23:0/2690164340 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3409a080 con 0x55e66f21c1d0 2024-03-24T10:47:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a217fa640 1 -- 172.21.15.23:0/2690164340 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f1a340dc040 con 0x55e66f21c1d0 2024-03-24T10:47:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a42816740 1 -- 172.21.15.23:0/2690164340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e66ee827e0 con 0x55e66f21c1d0 2024-03-24T10:47:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a217fa640 1 -- 172.21.15.23:0/2690164340 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1a340bd120 con 0x55e66f21c1d0 2024-03-24T10:47:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a42816740 10 client.17736 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a42816740 1 -- 172.21.15.23:0/2690164340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e66f181ba0 con 0x55e66f21c1d0 2024-03-24T10:47:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a217fa640 1 -- 172.21.15.23:0/2690164340 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e66ee827e0 con 0x55e66f21c1d0 2024-03-24T10:47:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a217fa640 1 -- 172.21.15.23:0/2690164340 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f1a3403c070 con 0x55e66f21c1d0 2024-03-24T10:47:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a42816740 10 client.17736 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a42816740 10 client.17736 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a42816740 20 client.17736 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a42816740 1 --2- 172.21.15.23:0/2690164340 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e66f235400 0x55e66f237840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a42816740 4 client.17736 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.627+0000 7f1a42816740 1 -- 172.21.15.23:0/2690164340 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e66eebf720 con 0x55e66f235400 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.628+0000 7f1a3d0fe640 1 --2- 172.21.15.23:0/2690164340 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e66f235400 0x55e66f237840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.628+0000 7f1a217fa640 10 client.17736 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.628+0000 7f1a3d0fe640 1 --2- 172.21.15.23:0/2690164340 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e66f235400 0x55e66f237840 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=402ba678e2cf10c0 server_cookie=8b8e72d0bbf03c76 in_seq=0 out_seq=0 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a217fa640 1 -- 172.21.15.23:0/2690164340 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e66eebf720 con 0x55e66f235400 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a217fa640 10 client.17736 handle_command_reply: tid=0 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a42816740 1 client.17736 shutdown 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a42816740 1 -- 172.21.15.23:0/2690164340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f21c1d0 msgr2=0x55e66f2226a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a42816740 1 --2- 172.21.15.23:0/2690164340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f21c1d0 0x55e66f2226a0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f1a340ed7e0 tx=0x7f1a340baf30 comp rx=0 tx=0).stop 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a42816740 1 -- 172.21.15.23:0/2690164340 shutdown_connections 2024-03-24T10:47:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a42816740 1 --2- 172.21.15.23:0/2690164340 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e66f235400 0x55e66f237840 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a42816740 1 --2- 172.21.15.23:0/2690164340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f2256d0 0x55e66f223120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a42816740 1 --2- 172.21.15.23:0/2690164340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21caf0 0x55e66f222be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a217fa640 0 client.17736 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:46.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a42816740 1 --2- 172.21.15.23:0/2690164340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f21c1d0 0x55e66f2226a0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a42816740 1 -- 172.21.15.23:0/2690164340 >> 172.21.15.23:0/2690164340 conn(0x55e66f216900 msgr2=0x55e66f215b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:46.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a42816740 1 -- 172.21.15.23:0/2690164340 shutdown_connections 2024-03-24T10:47:46.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a42816740 1 -- 172.21.15.23:0/2690164340 wait complete. 2024-03-24T10:47:46.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.629+0000 7f1a42816740 20 client.17736 trim_cache size 0 max 16384 2024-03-24T10:47:46.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a42816740 1 Processor -- start 2024-03-24T10:47:46.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a42816740 1 -- start start 2024-03-24T10:47:46.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a42816740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21c1d0 0x55e66f224dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a42816740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21caf0 0x55e66f230d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a42816740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f2256d0 0x55e66f2159e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a42816740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e66f108430 con 0x55e66f2256d0 2024-03-24T10:47:46.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a42816740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e66f223670 con 0x55e66f21c1d0 2024-03-24T10:47:46.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a42816740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e66f216770 con 0x55e66f21caf0 2024-03-24T10:47:46.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a3d8ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21c1d0 0x55e66f224dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a3e100640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f2256d0 0x55e66f2159e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a3d8ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21c1d0 0x55e66f224dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:52780/0 (socket says 172.21.15.23:52780) 2024-03-24T10:47:46.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/3423199877 learned_addr learned my addr 172.21.15.23:0/3423199877 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:46.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a3d0fe640 1 --2- 172.21.15.23:0/3423199877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21caf0 0x55e66f230d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a3e100640 1 -- 172.21.15.23:0/3423199877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21caf0 msgr2=0x55e66f230d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:46.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a3e100640 1 --2- 172.21.15.23:0/3423199877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21caf0 0x55e66f230d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a3e100640 1 -- 172.21.15.23:0/3423199877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21c1d0 msgr2=0x55e66f224dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:46.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a3e100640 1 --2- 172.21.15.23:0/3423199877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21c1d0 0x55e66f224dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.649+0000 7f1a3e100640 1 -- 172.21.15.23:0/3423199877 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e66f11db20 con 0x55e66f2256d0 2024-03-24T10:47:46.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.650+0000 7f1a3e100640 1 --2- 172.21.15.23:0/3423199877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f2256d0 0x55e66f2159e0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x55e66f217a90 tx=0x7f1a30021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65ffec5b725049 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:46.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.650+0000 7f1a237fe640 1 -- 172.21.15.23:0/3423199877 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a30016050 con 0x55e66f2256d0 2024-03-24T10:47:46.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.650+0000 7f1a237fe640 1 -- 172.21.15.23:0/3423199877 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a30011df0 con 0x55e66f2256d0 2024-03-24T10:47:46.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.650+0000 7f1a237fe640 1 -- 172.21.15.23:0/3423199877 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3001fb60 con 0x55e66f2256d0 2024-03-24T10:47:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.650+0000 7f1a42816740 1 -- 172.21.15.23:0/3423199877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f2256d0 msgr2=0x55e66f2159e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.650+0000 7f1a42816740 1 --2- 172.21.15.23:0/3423199877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f2256d0 0x55e66f2159e0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x55e66f217a90 tx=0x7f1a30021000 comp rx=0 tx=0).stop 2024-03-24T10:47:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.650+0000 7f1a42816740 1 -- 172.21.15.23:0/3423199877 shutdown_connections 2024-03-24T10:47:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.650+0000 7f1a42816740 1 --2- 172.21.15.23:0/3423199877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f2256d0 0x55e66f2159e0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.650+0000 7f1a42816740 1 --2- 172.21.15.23:0/3423199877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21caf0 0x55e66f230d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.650+0000 7f1a42816740 1 --2- 172.21.15.23:0/3423199877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f21c1d0 0x55e66f224dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.651+0000 7f1a42816740 1 -- 172.21.15.23:0/3423199877 >> 172.21.15.23:0/3423199877 conn(0x55e66f216900 msgr2=0x55e66f2152d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.651+0000 7f1a42816740 1 -- 172.21.15.23:0/3423199877 shutdown_connections 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.651+0000 7f1a42816740 1 -- 172.21.15.23:0/3423199877 wait complete. 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.651+0000 7f1a42816740 1 -- start start 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.651+0000 7f1a42816740 1 Processor -- start 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.651+0000 7f1a42816740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f21c1d0 0x55e66f22d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.651+0000 7f1a42816740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21caf0 0x55e66f2288f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.651+0000 7f1a42816740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f2256d0 0x55e66f228e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.651+0000 7f1a42816740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e66f229500 con 0x55e66f21c1d0 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.651+0000 7f1a42816740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e66f229670 con 0x55e66f2256d0 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.651+0000 7f1a42816740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e66f29a780 con 0x55e66f21caf0 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.651+0000 7f1a42816740 10 client.0 _ll_register_callbacks cb 0x7f1a30011940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.651+0000 7f1a3d8ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f21c1d0 0x55e66f22d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.651+0000 7f1a3d8ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f21c1d0 0x55e66f22d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36472/0 (socket says 172.21.15.23:36472) 2024-03-24T10:47:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/2687135779 learned_addr learned my addr 172.21.15.23:0/2687135779 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a3d0fe640 1 --2- 172.21.15.23:0/2687135779 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21caf0 0x55e66f2288f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a3e100640 1 --2- 172.21.15.23:0/2687135779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f2256d0 0x55e66f228e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/2687135779 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21caf0 msgr2=0x55e66f2288f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a3d8ff640 1 --2- 172.21.15.23:0/2687135779 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21caf0 0x55e66f2288f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/2687135779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f2256d0 msgr2=0x55e66f228e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a3d8ff640 1 --2- 172.21.15.23:0/2687135779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f2256d0 0x55e66f228e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a3d8ff640 1 -- 172.21.15.23:0/2687135779 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a30019040 con 0x55e66f21c1d0 2024-03-24T10:47:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a3d8ff640 1 --2- 172.21.15.23:0/2687135779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f21c1d0 0x55e66f22d580 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f1a3402c770 tx=0x7f1a3409d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a5aa7f5224ed6d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a217fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a217fa640 1 -- 172.21.15.23:0/2687135779 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a340056f0 con 0x55e66f21c1d0 2024-03-24T10:47:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a217fa640 1 -- 172.21.15.23:0/2687135779 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f1a340dc040 con 0x55e66f21c1d0 2024-03-24T10:47:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a42816740 1 -- 172.21.15.23:0/2687135779 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a34099e60 con 0x55e66f21c1d0 2024-03-24T10:47:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a217fa640 1 -- 172.21.15.23:0/2687135779 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1a340bd120 con 0x55e66f21c1d0 2024-03-24T10:47:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a42816740 10 client.17742 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a42816740 1 -- 172.21.15.23:0/2687135779 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e66f143700 con 0x55e66f21c1d0 2024-03-24T10:47:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.652+0000 7f1a217fa640 1 -- 172.21.15.23:0/2687135779 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a34099e60 con 0x55e66f21c1d0 2024-03-24T10:47:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.653+0000 7f1a217fa640 1 -- 172.21.15.23:0/2687135779 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f1a3403c050 con 0x55e66f21c1d0 2024-03-24T10:47:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.653+0000 7f1a42816740 10 client.17742 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.653+0000 7f1a42816740 10 client.17742 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.653+0000 7f1a42816740 20 client.17742 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.653+0000 7f1a42816740 1 --2- 172.21.15.23:0/2687135779 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e66f29b910 0x55e66f29dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.653+0000 7f1a42816740 4 client.17742 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:47:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.653+0000 7f1a42816740 1 -- 172.21.15.23:0/2687135779 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e66ef0e890 con 0x55e66f29b910 2024-03-24T10:47:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.653+0000 7f1a3d0fe640 1 --2- 172.21.15.23:0/2687135779 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e66f29b910 0x55e66f29dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.653+0000 7f1a3d0fe640 1 --2- 172.21.15.23:0/2687135779 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e66f29b910 0x55e66f29dd00 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7dbcdf27ff05f90 server_cookie=759bd440bcacad9 in_seq=0 out_seq=0 2024-03-24T10:47:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.653+0000 7f1a217fa640 10 client.17742 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.653+0000 7f1a217fa640 1 -- 172.21.15.23:0/2687135779 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e66ef0e890 con 0x55e66f29b910 2024-03-24T10:47:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.653+0000 7f1a217fa640 10 client.17742 handle_command_reply: tid=0 2024-03-24T10:47:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.653+0000 7f1a42816740 1 client.17742 shutdown 2024-03-24T10:47:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a42816740 1 -- 172.21.15.23:0/2687135779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f21c1d0 msgr2=0x55e66f22d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a42816740 1 --2- 172.21.15.23:0/2687135779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f21c1d0 0x55e66f22d580 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f1a3402c770 tx=0x7f1a3409d320 comp rx=0 tx=0).stop 2024-03-24T10:47:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a42816740 1 -- 172.21.15.23:0/2687135779 shutdown_connections 2024-03-24T10:47:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a42816740 1 --2- 172.21.15.23:0/2687135779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e66f2256d0 0x55e66f228e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a42816740 1 --2- 172.21.15.23:0/2687135779 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e66f29b910 0x55e66f29dd00 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a42816740 1 --2- 172.21.15.23:0/2687135779 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e66f21caf0 0x55e66f2288f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a217fa640 0 client.17742 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a42816740 1 --2- 172.21.15.23:0/2687135779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e66f21c1d0 0x55e66f22d580 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a42816740 1 -- 172.21.15.23:0/2687135779 >> 172.21.15.23:0/2687135779 conn(0x55e66f216900 msgr2=0x55e66f215b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a42816740 1 -- 172.21.15.23:0/2687135779 shutdown_connections 2024-03-24T10:47:46.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a42816740 1 -- 172.21.15.23:0/2687135779 wait complete. 2024-03-24T10:47:46.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a42816740 20 client.17742 trim_cache size 0 max 16384 2024-03-24T10:47:46.661 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:47:46.661 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:47:46.661 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:47:46.661 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:47:46.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a3fb63640 1 -- 172.21.15.23:0/3359984248 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1a240a2230 msgr2=0x7f1a240a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a3fb63640 1 --2- 172.21.15.23:0/3359984248 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1a240a2230 0x7f1a240a46f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1a30005fd0 tx=0x7f1a300114f0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a3fb63640 1 -- 172.21.15.23:0/3359984248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a381005a0 msgr2=0x7f1a3819da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a3fb63640 1 --2- 172.21.15.23:0/3359984248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a381005a0 0x7f1a3819da70 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009230 tx=0x7f1a3402c640 comp rx=0 tx=0).stop 2024-03-24T10:47:46.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a3fb63640 1 -- 172.21.15.23:0/3359984248 shutdown_connections 2024-03-24T10:47:46.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a3fb63640 1 --2- 172.21.15.23:0/3359984248 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f1a240a2230 0x7f1a240a46f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a3fb63640 1 --2- 172.21.15.23:0/3359984248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a38106660 0x7f1a38197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a3fb63640 1 --2- 172.21.15.23:0/3359984248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a38100f60 0x7f1a3819dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a3fb63640 1 --2- 172.21.15.23:0/3359984248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a381005a0 0x7f1a3819da70 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a3fb63640 1 -- 172.21.15.23:0/3359984248 >> 172.21.15.23:0/3359984248 conn(0x7f1a38077710 msgr2=0x7f1a380ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:46.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a3fb63640 1 -- 172.21.15.23:0/3359984248 shutdown_connections 2024-03-24T10:47:46.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.654+0000 7f1a3fb63640 1 -- 172.21.15.23:0/3359984248 wait complete. 2024-03-24T10:47:46.665 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:47:46.665 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:46.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:46 smithi116 ceph-mon[36248]: pgmap v237: 97 pgs: 97 active+clean; 438 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 170 op/s 2024-03-24T10:47:46.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.773+0000 7fc3c8743640 1 -- 172.21.15.23:0/1410469990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3c0102980 msgr2=0x7fc3c0102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.773+0000 7fc3c8743640 1 --2- 172.21.15.23:0/1410469990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3c0102980 0x7fc3c0102d60 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc009900 tx=0x7fc3bc02bc30 comp rx=0 tx=0).stop 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 -- 172.21.15.23:0/1410469990 shutdown_connections 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 --2- 172.21.15.23:0/1410469990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3c010bd50 0x7fc3c010e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 --2- 172.21.15.23:0/1410469990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3c01032a0 0x7fc3c010b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 --2- 172.21.15.23:0/1410469990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3c0102980 0x7fc3c0102d60 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 -- 172.21.15.23:0/1410469990 >> 172.21.15.23:0/1410469990 conn(0x7fc3c0077710 msgr2=0x7fc3c0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 -- 172.21.15.23:0/1410469990 shutdown_connections 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 -- 172.21.15.23:0/1410469990 wait complete. 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 Processor -- start 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 -- start start 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3c0102980 0x7fc3c0197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3c01032a0 0x7fc3c01982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3c010bd50 0x7fc3c019c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c019cc10 con 0x7fc3c010bd50 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c019cd80 con 0x7fc3c01032a0 2024-03-24T10:47:46.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.774+0000 7fc3c8743640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c019cf40 con 0x7fc3c0102980 2024-03-24T10:47:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3c6ce0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3c010bd50 0x7fc3c019c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3c6ce0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3c010bd50 0x7fc3c019c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36496/0 (socket says 172.21.15.23:36496) 2024-03-24T10:47:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3c5cde640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3c01032a0 0x7fc3c01982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3c6ce0640 1 -- 172.21.15.23:0/1497649884 learned_addr learned my addr 172.21.15.23:0/1497649884 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3c64df640 1 --2- 172.21.15.23:0/1497649884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3c0102980 0x7fc3c0197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3c6ce0640 1 -- 172.21.15.23:0/1497649884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3c0102980 msgr2=0x7fc3c0197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3c6ce0640 1 --2- 172.21.15.23:0/1497649884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3c0102980 0x7fc3c0197da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3c6ce0640 1 -- 172.21.15.23:0/1497649884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3c01032a0 msgr2=0x7fc3c01982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3c6ce0640 1 --2- 172.21.15.23:0/1497649884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3c01032a0 0x7fc3c01982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3c6ce0640 1 -- 172.21.15.23:0/1497649884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3bc009560 con 0x7fc3c010bd50 2024-03-24T10:47:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3c6ce0640 1 --2- 172.21.15.23:0/1497649884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3c010bd50 0x7fc3c019c610 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009fd0 tx=0x7fc3b800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3b77fe640 1 -- 172.21.15.23:0/1497649884 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b800bea0 con 0x7fc3c010bd50 2024-03-24T10:47:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3b77fe640 1 -- 172.21.15.23:0/1497649884 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3b8009940 con 0x7fc3c010bd50 2024-03-24T10:47:46.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3b77fe640 1 -- 172.21.15.23:0/1497649884 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b80106c0 con 0x7fc3c010bd50 2024-03-24T10:47:46.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.775+0000 7fc3c8743640 1 -- 172.21.15.23:0/1497649884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c019d220 con 0x7fc3c010bd50 2024-03-24T10:47:46.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.776+0000 7fc3c8743640 1 -- 172.21.15.23:0/1497649884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3c01a48c0 con 0x7fc3c010bd50 2024-03-24T10:47:46.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.777+0000 7fc3c8743640 1 -- 172.21.15.23:0/1497649884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3c0068500 con 0x7fc3c010bd50 2024-03-24T10:47:46.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.778+0000 7fc3b77fe640 1 -- 172.21.15.23:0/1497649884 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fc3b8004240 con 0x7fc3c010bd50 2024-03-24T10:47:46.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.779+0000 7fc3b77fe640 1 --2- 172.21.15.23:0/1497649884 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc3ac0a22b0 0x7fc3ac0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.779+0000 7fc3c64df640 1 --2- 172.21.15.23:0/1497649884 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc3ac0a22b0 0x7fc3ac0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.779+0000 7fc3b77fe640 1 -- 172.21.15.23:0/1497649884 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7fc3b80cdcb0 con 0x7fc3c010bd50 2024-03-24T10:47:46.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.779+0000 7fc3c64df640 1 --2- 172.21.15.23:0/1497649884 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc3ac0a22b0 0x7fc3ac0a4770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc005ab0 tx=0x7fc3bc005a20 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:46.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:46.783+0000 7fc3b77fe640 1 -- 172.21.15.23:0/1497649884 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc3b80969f0 con 0x7fc3c010bd50 2024-03-24T10:47:46.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:46 smithi023 ceph-mon[32466]: pgmap v237: 97 pgs: 97 active+clean; 438 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 170 op/s 2024-03-24T10:47:47.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:46 smithi117 ceph-mon[36368]: pgmap v237: 97 pgs: 97 active+clean; 438 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 170 op/s 2024-03-24T10:47:47.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.030+0000 7fc3c8743640 1 -- 172.21.15.23:0/1497649884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3c0111950 con 0x7fc3c010bd50 2024-03-24T10:47:47.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.031+0000 7fc3b77fe640 1 -- 172.21.15.23:0/1497649884 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11869 (secure 0 0 0) 0x7fc3b809a0b0 con 0x7fc3c010bd50 2024-03-24T10:47:47.031 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:47:47.031 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:47:17.474711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27526":{"gid":27526,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/2017557046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":2017557046},{"type":"v1","addr":"172.21.15.117:6837","nonce":2017557046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:47:47.032 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 31 2024-03-24T10:47:47.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.033+0000 7fc3c8743640 1 -- 172.21.15.23:0/1497649884 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc3ac0a22b0 msgr2=0x7fc3ac0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.033+0000 7fc3c8743640 1 --2- 172.21.15.23:0/1497649884 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc3ac0a22b0 0x7fc3ac0a4770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc005ab0 tx=0x7fc3bc005a20 comp rx=0 tx=0).stop 2024-03-24T10:47:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.033+0000 7fc3c8743640 1 -- 172.21.15.23:0/1497649884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3c010bd50 msgr2=0x7fc3c019c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.033+0000 7fc3c8743640 1 --2- 172.21.15.23:0/1497649884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3c010bd50 0x7fc3c019c610 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009fd0 tx=0x7fc3b800b260 comp rx=0 tx=0).stop 2024-03-24T10:47:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.033+0000 7fc3c8743640 1 -- 172.21.15.23:0/1497649884 shutdown_connections 2024-03-24T10:47:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.033+0000 7fc3c8743640 1 --2- 172.21.15.23:0/1497649884 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc3ac0a22b0 0x7fc3ac0a4770 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.033+0000 7fc3c8743640 1 --2- 172.21.15.23:0/1497649884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3c010bd50 0x7fc3c019c610 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.033+0000 7fc3c8743640 1 --2- 172.21.15.23:0/1497649884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3c01032a0 0x7fc3c01982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.033+0000 7fc3c8743640 1 --2- 172.21.15.23:0/1497649884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3c0102980 0x7fc3c0197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.033+0000 7fc3c8743640 1 -- 172.21.15.23:0/1497649884 >> 172.21.15.23:0/1497649884 conn(0x7fc3c0077710 msgr2=0x7fc3c00fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.033+0000 7fc3c8743640 1 -- 172.21.15.23:0/1497649884 shutdown_connections 2024-03-24T10:47:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.033+0000 7fc3c8743640 1 -- 172.21.15.23:0/1497649884 wait complete. 2024-03-24T10:47:47.044 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:47:47.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 757876 10.77 MB/sec execute 244 sec latency 118.110 ms 2024-03-24T10:47:47.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.149+0000 7fae7c6bb640 1 -- 172.21.15.23:0/2948931948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae740692d0 msgr2=0x7fae7410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.149+0000 7fae7c6bb640 1 --2- 172.21.15.23:0/2948931948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae740692d0 0x7fae7410e570 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fae70009cd0 tx=0x7fae7002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 -- 172.21.15.23:0/2948931948 shutdown_connections 2024-03-24T10:47:47.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 --2- 172.21.15.23:0/2948931948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae740692d0 0x7fae7410e570 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 --2- 172.21.15.23:0/2948931948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae74068910 0x7fae74068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 --2- 172.21.15.23:0/2948931948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae74107760 0x7fae74107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 -- 172.21.15.23:0/2948931948 >> 172.21.15.23:0/2948931948 conn(0x7fae74077710 msgr2=0x7fae74077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 -- 172.21.15.23:0/2948931948 shutdown_connections 2024-03-24T10:47:47.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 -- 172.21.15.23:0/2948931948 wait complete. 2024-03-24T10:47:47.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 Processor -- start 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 -- start start 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae74068910 0x7fae7419d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae740692d0 0x7fae7419de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae74107760 0x7fae74197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fae74198120 con 0x7fae740692d0 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fae74198290 con 0x7fae74107760 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae7c6bb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fae74198450 con 0x7fae74068910 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae79c56640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae740692d0 0x7fae7419de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.150+0000 7fae79c56640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae740692d0 0x7fae7419de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36504/0 (socket says 172.21.15.23:36504) 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.151+0000 7fae79c56640 1 -- 172.21.15.23:0/4133421311 learned_addr learned my addr 172.21.15.23:0/4133421311 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.151+0000 7fae7a457640 1 --2- 172.21.15.23:0/4133421311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae74068910 0x7fae7419d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.151+0000 7fae7ac58640 1 --2- 172.21.15.23:0/4133421311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae74107760 0x7fae74197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.151+0000 7fae79c56640 1 -- 172.21.15.23:0/4133421311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae74068910 msgr2=0x7fae7419d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.151+0000 7fae79c56640 1 --2- 172.21.15.23:0/4133421311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae74068910 0x7fae7419d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.151+0000 7fae79c56640 1 -- 172.21.15.23:0/4133421311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae74107760 msgr2=0x7fae74197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.151+0000 7fae79c56640 1 --2- 172.21.15.23:0/4133421311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae74107760 0x7fae74197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.151+0000 7fae79c56640 1 -- 172.21.15.23:0/4133421311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae70009900 con 0x7fae740692d0 2024-03-24T10:47:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.151+0000 7fae79c56640 1 --2- 172.21.15.23:0/4133421311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae740692d0 0x7fae7419de60 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fae6400e9f0 tx=0x7fae6400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.151+0000 7fae6b7fe640 1 -- 172.21.15.23:0/4133421311 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae64009720 con 0x7fae740692d0 2024-03-24T10:47:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.151+0000 7fae6b7fe640 1 -- 172.21.15.23:0/4133421311 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae6400cd80 con 0x7fae740692d0 2024-03-24T10:47:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.151+0000 7fae7c6bb640 1 -- 172.21.15.23:0/4133421311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae74198730 con 0x7fae740692d0 2024-03-24T10:47:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.151+0000 7fae6b7fe640 1 -- 172.21.15.23:0/4133421311 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae64010690 con 0x7fae740692d0 2024-03-24T10:47:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.151+0000 7fae7c6bb640 1 -- 172.21.15.23:0/4133421311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae741a49c0 con 0x7fae740692d0 2024-03-24T10:47:47.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae7f370740 1 Processor -- start 2024-03-24T10:47:47.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae7f370740 1 -- start start 2024-03-24T10:47:47.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae6b7fe640 1 -- 172.21.15.23:0/4133421311 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fae640046c0 con 0x7fae740692d0 2024-03-24T10:47:47.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae7f370740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd7d3f0 0x5587ebd7d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae7f370740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd863b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae7f370740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd868f0 0x5587ebd88ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae7f370740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5587ebceb840 con 0x5587ebd868f0 2024-03-24T10:47:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae7f370740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5587ebce8d70 con 0x5587ebd7d3f0 2024-03-24T10:47:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae7f370740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5587ebce4550 con 0x5587ebd7dd10 2024-03-24T10:47:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae7a457640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd863b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae79c56640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd868f0 0x5587ebd88ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae7ac58640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd7d3f0 0x5587ebd7d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae7a457640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd863b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:50660/0 (socket says 172.21.15.23:50660) 2024-03-24T10:47:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae7ac58640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd7d3f0 0x5587ebd7d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:52830/0 (socket says 172.21.15.23:52830) 2024-03-24T10:47:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae79c56640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd868f0 0x5587ebd88ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36512/0 (socket says 172.21.15.23:36512) 2024-03-24T10:47:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.154+0000 7fae7a457640 1 -- 172.21.15.23:0/1611284353 learned_addr learned my addr 172.21.15.23:0/1611284353 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.155+0000 7fae7a457640 1 -- 172.21.15.23:0/1611284353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd7d3f0 msgr2=0x5587ebd7d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.155+0000 7fae7a457640 1 --2- 172.21.15.23:0/1611284353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd7d3f0 0x5587ebd7d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.155+0000 7fae7a457640 1 -- 172.21.15.23:0/1611284353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd868f0 msgr2=0x5587ebd88ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.155+0000 7fae7a457640 1 --2- 172.21.15.23:0/1611284353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd868f0 0x5587ebd88ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.155+0000 7fae7a457640 1 -- 172.21.15.23:0/1611284353 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587ebc79de0 con 0x5587ebd7dd10 2024-03-24T10:47:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.155+0000 7fae6b7fe640 1 --2- 172.21.15.23:0/4133421311 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fae600a22b0 0x7fae600a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.155+0000 7fae7ac58640 1 --2- 172.21.15.23:0/1611284353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd7d3f0 0x5587ebd7d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.155+0000 7fae7ac58640 1 --2- 172.21.15.23:0/4133421311 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fae600a22b0 0x7fae600a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.155+0000 7fae6b7fe640 1 -- 172.21.15.23:0/4133421311 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7fae64014070 con 0x7fae740692d0 2024-03-24T10:47:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.155+0000 7fae7a457640 1 --2- 172.21.15.23:0/1611284353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd863b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009cf0 tx=0x7fae6c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=431ca7e19032371f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.155+0000 7fae68ff9640 1 -- 172.21.15.23:0/1611284353 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c009ea0 con 0x5587ebd7dd10 2024-03-24T10:47:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.155+0000 7fae68ff9640 1 -- 172.21.15.23:0/1611284353 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae6c004230 con 0x5587ebd7dd10 2024-03-24T10:47:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.155+0000 7fae68ff9640 1 -- 172.21.15.23:0/1611284353 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c019880 con 0x5587ebd7dd10 2024-03-24T10:47:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.156+0000 7fae7ac58640 1 --2- 172.21.15.23:0/4133421311 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fae600a22b0 0x7fae600a4770 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fae7002c630 tx=0x7fae7002e040 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.156+0000 7fae7f370740 1 -- 172.21.15.23:0/1611284353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 msgr2=0x5587ebd863b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.156+0000 7fae7f370740 1 --2- 172.21.15.23:0/1611284353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd863b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009cf0 tx=0x7fae6c011810 comp rx=0 tx=0).stop 2024-03-24T10:47:47.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.156+0000 7fae7f370740 1 -- 172.21.15.23:0/1611284353 shutdown_connections 2024-03-24T10:47:47.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.156+0000 7fae7f370740 1 --2- 172.21.15.23:0/1611284353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd868f0 0x5587ebd88ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.156+0000 7fae7f370740 1 --2- 172.21.15.23:0/1611284353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd863b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.156+0000 7fae7f370740 1 --2- 172.21.15.23:0/1611284353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd7d3f0 0x5587ebd7d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.156+0000 7fae7f370740 1 -- 172.21.15.23:0/1611284353 >> 172.21.15.23:0/1611284353 conn(0x5587ebd77b20 msgr2=0x5587ebd77f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.156+0000 7fae7f370740 1 -- 172.21.15.23:0/1611284353 shutdown_connections 2024-03-24T10:47:47.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.156+0000 7fae7f370740 1 -- 172.21.15.23:0/1611284353 wait complete. 2024-03-24T10:47:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae7f370740 1 -- start start 2024-03-24T10:47:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae7f370740 1 Processor -- start 2024-03-24T10:47:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae7f370740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd7d3f0 0x5587ebd838d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae7f370740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd83e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae7f370740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd868f0 0x5587ebd84350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae7f370740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5587eb9b7430 con 0x5587ebd7d3f0 2024-03-24T10:47:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae7f370740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5587eb973e00 con 0x5587ebd868f0 2024-03-24T10:47:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae7f370740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5587ebc69590 con 0x5587ebd7dd10 2024-03-24T10:47:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae7f370740 10 client.0 _ll_register_callbacks cb 0x5587ebcd8fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae7a457640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd7d3f0 0x5587ebd838d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae79c56640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd83e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae7ac58640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd868f0 0x5587ebd84350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae7a457640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd7d3f0 0x5587ebd838d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36528/0 (socket says 172.21.15.23:36528) 2024-03-24T10:47:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae7a457640 1 -- 172.21.15.23:0/1972150295 learned_addr learned my addr 172.21.15.23:0/1972150295 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.157+0000 7fae79c56640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd83e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:50668/0 (socket says 172.21.15.23:50668) 2024-03-24T10:47:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.158+0000 7fae7a457640 1 -- 172.21.15.23:0/1972150295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 msgr2=0x5587ebd83e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.158+0000 7fae7a457640 1 --2- 172.21.15.23:0/1972150295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd83e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.158+0000 7fae7a457640 1 -- 172.21.15.23:0/1972150295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd868f0 msgr2=0x5587ebd84350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.158+0000 7fae7a457640 1 --2- 172.21.15.23:0/1972150295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd868f0 0x5587ebd84350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.158+0000 7fae7a457640 1 -- 172.21.15.23:0/1972150295 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae6c009950 con 0x5587ebd7d3f0 2024-03-24T10:47:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.158+0000 7fae7a457640 1 --2- 172.21.15.23:0/1972150295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd7d3f0 0x5587ebd838d0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fae6c003d40 tx=0x7fae6c005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3561c181f4482fa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.158+0000 7fae5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.158+0000 7fae5dffb640 1 -- 172.21.15.23:0/1972150295 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c009ea0 con 0x5587ebd7d3f0 2024-03-24T10:47:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.158+0000 7fae5dffb640 1 -- 172.21.15.23:0/1972150295 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7fae6c005040 con 0x5587ebd7d3f0 2024-03-24T10:47:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.158+0000 7fae7f370740 1 -- 172.21.15.23:0/1972150295 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5587eb9e37e0 con 0x5587ebd7d3f0 2024-03-24T10:47:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.158+0000 7fae5dffb640 1 -- 172.21.15.23:0/1972150295 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fae6c005200 con 0x5587ebd7d3f0 2024-03-24T10:47:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.158+0000 7fae7f370740 10 client.17760 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.158+0000 7fae7f370740 1 -- 172.21.15.23:0/1972150295 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5587ebce2bd0 con 0x5587ebd7d3f0 2024-03-24T10:47:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.158+0000 7fae5dffb640 1 -- 172.21.15.23:0/1972150295 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5587eb9e37e0 con 0x5587ebd7d3f0 2024-03-24T10:47:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.159+0000 7fae5dffb640 1 -- 172.21.15.23:0/1972150295 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fae6c021040 con 0x5587ebd7d3f0 2024-03-24T10:47:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.159+0000 7fae7f370740 10 client.17760 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.159+0000 7fae7f370740 10 client.17760 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:47:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.159+0000 7fae7f370740 20 client.17760 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.159+0000 7fae7f370740 1 --2- 172.21.15.23:0/1972150295 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5587ebd96910 0x5587ebd98d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.159+0000 7fae7f370740 4 client.17760 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.159+0000 7fae7f370740 1 -- 172.21.15.23:0/1972150295 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587eba20720 con 0x5587ebd96910 2024-03-24T10:47:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.159+0000 7fae79c56640 1 --2- 172.21.15.23:0/1972150295 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5587ebd96910 0x5587ebd98d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.159+0000 7fae5dffb640 10 client.17760 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:47:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.159+0000 7fae79c56640 1 --2- 172.21.15.23:0/1972150295 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5587ebd96910 0x5587ebd98d20 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=89a184606f6ffe1d server_cookie=afec47da4314564e in_seq=0 out_seq=0 2024-03-24T10:47:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.160+0000 7fae5dffb640 1 -- 172.21.15.23:0/1972150295 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5587eba20720 con 0x5587ebd96910 2024-03-24T10:47:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.160+0000 7fae5dffb640 10 client.17760 handle_command_reply: tid=0 2024-03-24T10:47:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.160+0000 7fae7f370740 1 client.17760 shutdown 2024-03-24T10:47:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.161+0000 7fae7f370740 1 -- 172.21.15.23:0/1972150295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd7d3f0 msgr2=0x5587ebd838d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.161+0000 7fae7f370740 1 --2- 172.21.15.23:0/1972150295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd7d3f0 0x5587ebd838d0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fae6c003d40 tx=0x7fae6c005d30 comp rx=0 tx=0).stop 2024-03-24T10:47:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.161+0000 7fae7f370740 1 -- 172.21.15.23:0/1972150295 shutdown_connections 2024-03-24T10:47:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.161+0000 7fae7f370740 1 --2- 172.21.15.23:0/1972150295 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5587ebd96910 0x5587ebd98d20 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.161+0000 7fae7f370740 1 --2- 172.21.15.23:0/1972150295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd868f0 0x5587ebd84350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.161+0000 7fae7f370740 1 --2- 172.21.15.23:0/1972150295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd83e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.161+0000 7fae5dffb640 0 client.17760 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:47:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.161+0000 7fae7f370740 1 --2- 172.21.15.23:0/1972150295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd7d3f0 0x5587ebd838d0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.161+0000 7fae7f370740 1 -- 172.21.15.23:0/1972150295 >> 172.21.15.23:0/1972150295 conn(0x5587ebd77b20 msgr2=0x5587ebd79870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.161+0000 7fae7f370740 1 -- 172.21.15.23:0/1972150295 shutdown_connections 2024-03-24T10:47:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.161+0000 7fae7f370740 1 -- 172.21.15.23:0/1972150295 wait complete. 2024-03-24T10:47:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.161+0000 7fae7f370740 20 client.17760 trim_cache size 0 max 16384 2024-03-24T10:47:47.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.174+0000 7fae7f370740 1 Processor -- start 2024-03-24T10:47:47.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.174+0000 7fae7f370740 1 -- start start 2024-03-24T10:47:47.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.174+0000 7fae7f370740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd7d3f0 0x5587ebd86000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7f370740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd92180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7f370740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd868f0 0x5587ebd9e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7f370740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5587ebd78b20 con 0x5587ebd7d3f0 2024-03-24T10:47:47.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7f370740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5587ebd88ce0 con 0x5587ebd868f0 2024-03-24T10:47:47.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7f370740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5587ebd720d0 con 0x5587ebd7dd10 2024-03-24T10:47:47.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7a457640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd7d3f0 0x5587ebd86000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7a457640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd7d3f0 0x5587ebd86000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36536/0 (socket says 172.21.15.23:36536) 2024-03-24T10:47:47.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7a457640 1 -- 172.21.15.23:0/3649377682 learned_addr learned my addr 172.21.15.23:0/3649377682 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7ac58640 1 --2- 172.21.15.23:0/3649377682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd868f0 0x5587ebd9e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae79c56640 1 --2- 172.21.15.23:0/3649377682 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd92180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7a457640 1 -- 172.21.15.23:0/3649377682 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 msgr2=0x5587ebd92180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7a457640 1 --2- 172.21.15.23:0/3649377682 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd92180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7a457640 1 -- 172.21.15.23:0/3649377682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd868f0 msgr2=0x5587ebd9e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7a457640 1 --2- 172.21.15.23:0/3649377682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd868f0 0x5587ebd9e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7a457640 1 -- 172.21.15.23:0/3649377682 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587ebc7ec80 con 0x5587ebd7d3f0 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae7a457640 1 --2- 172.21.15.23:0/3649377682 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd7d3f0 0x5587ebd86000 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fae6c000c00 tx=0x7fae6c005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbeafa78d01bc40c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae5ffff640 1 -- 172.21.15.23:0/3649377682 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c019a30 con 0x5587ebd7d3f0 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae5ffff640 1 -- 172.21.15.23:0/3649377682 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae6c0097a0 con 0x5587ebd7d3f0 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.175+0000 7fae5ffff640 1 -- 172.21.15.23:0/3649377682 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c0045d0 con 0x5587ebd7d3f0 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.176+0000 7fae7f370740 1 -- 172.21.15.23:0/3649377682 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd7d3f0 msgr2=0x5587ebd86000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.176+0000 7fae7f370740 1 --2- 172.21.15.23:0/3649377682 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd7d3f0 0x5587ebd86000 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fae6c000c00 tx=0x7fae6c005e30 comp rx=0 tx=0).stop 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.176+0000 7fae7f370740 1 -- 172.21.15.23:0/3649377682 shutdown_connections 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.176+0000 7fae7f370740 1 --2- 172.21.15.23:0/3649377682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd868f0 0x5587ebd9e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.176+0000 7fae7f370740 1 --2- 172.21.15.23:0/3649377682 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7dd10 0x5587ebd92180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.176+0000 7fae7f370740 1 --2- 172.21.15.23:0/3649377682 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd7d3f0 0x5587ebd86000 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.176+0000 7fae7f370740 1 -- 172.21.15.23:0/3649377682 >> 172.21.15.23:0/3649377682 conn(0x5587ebd77b20 msgr2=0x5587ebd77f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.176+0000 7fae7f370740 1 -- 172.21.15.23:0/3649377682 shutdown_connections 2024-03-24T10:47:47.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.176+0000 7fae7f370740 1 -- 172.21.15.23:0/3649377682 wait complete. 2024-03-24T10:47:47.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.176+0000 7fae7f370740 1 -- start start 2024-03-24T10:47:47.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.176+0000 7fae7f370740 1 Processor -- start 2024-03-24T10:47:47.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7f370740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7d3f0 0x5587ebd8ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7f370740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd7dd10 0x5587ebd802f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7f370740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd868f0 0x5587ebd80a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7f370740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5587ebd81130 con 0x5587ebd868f0 2024-03-24T10:47:47.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7f370740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5587ebdfbc20 con 0x5587ebd7dd10 2024-03-24T10:47:47.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7f370740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5587ebdfbd60 con 0x5587ebd7d3f0 2024-03-24T10:47:47.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7f370740 10 client.0 _ll_register_callbacks cb 0x7fae640d1bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:47.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7ac58640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd868f0 0x5587ebd80a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7ac58640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd868f0 0x5587ebd80a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36548/0 (socket says 172.21.15.23:36548) 2024-03-24T10:47:47.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7ac58640 1 -- 172.21.15.23:0/4047778365 learned_addr learned my addr 172.21.15.23:0/4047778365 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7a457640 1 --2- 172.21.15.23:0/4047778365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7d3f0 0x5587ebd8ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae79c56640 1 --2- 172.21.15.23:0/4047778365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd7dd10 0x5587ebd802f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7ac58640 1 -- 172.21.15.23:0/4047778365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7d3f0 msgr2=0x5587ebd8ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7ac58640 1 --2- 172.21.15.23:0/4047778365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7d3f0 0x5587ebd8ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7ac58640 1 -- 172.21.15.23:0/4047778365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd7dd10 msgr2=0x5587ebd802f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7ac58640 1 --2- 172.21.15.23:0/4047778365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd7dd10 0x5587ebd802f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7ac58640 1 -- 172.21.15.23:0/4047778365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587ebce2bd0 con 0x5587ebd868f0 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7ac58640 1 --2- 172.21.15.23:0/4047778365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd868f0 0x5587ebd80a30 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fae7002e010 tx=0x7fae7002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6769ea569789b2d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae5dffb640 1 -- 172.21.15.23:0/4047778365 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae70003b70 con 0x5587ebd868f0 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae5dffb640 1 -- 172.21.15.23:0/4047778365 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7fae70003cd0 con 0x5587ebd868f0 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae7f370740 1 -- 172.21.15.23:0/4047778365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae6c004230 con 0x5587ebd868f0 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.177+0000 7fae5dffb640 1 -- 172.21.15.23:0/4047778365 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fae70003e90 con 0x5587ebd868f0 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.178+0000 7fae7f370740 10 client.17766 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:47.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.178+0000 7fae7f370740 1 -- 172.21.15.23:0/4047778365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5587ebca4920 con 0x5587ebd868f0 2024-03-24T10:47:47.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.178+0000 7fae5dffb640 1 -- 172.21.15.23:0/4047778365 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae70064020 con 0x5587ebd868f0 2024-03-24T10:47:47.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.178+0000 7fae5dffb640 1 -- 172.21.15.23:0/4047778365 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fae70004400 con 0x5587ebd868f0 2024-03-24T10:47:47.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.178+0000 7fae7f370740 10 client.17766 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:47.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.178+0000 7fae7f370740 10 client.17766 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:47:47.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.178+0000 7fae7f370740 20 client.17766 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:47.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.178+0000 7fae7f370740 1 --2- 172.21.15.23:0/4047778365 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5587ebdfd4f0 0x5587ebdff950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.178+0000 7fae7f370740 4 client.17766 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:47:47.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.178+0000 7fae7f370740 1 -- 172.21.15.23:0/4047778365 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5587eba6f890 con 0x5587ebdfd4f0 2024-03-24T10:47:47.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.178+0000 7fae7a457640 1 --2- 172.21.15.23:0/4047778365 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5587ebdfd4f0 0x5587ebdff950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.178+0000 7fae7a457640 1 --2- 172.21.15.23:0/4047778365 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5587ebdfd4f0 0x5587ebdff950 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3ffe2c21900eecc5 server_cookie=65d945a51f734169 in_seq=0 out_seq=0 2024-03-24T10:47:47.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.178+0000 7fae5dffb640 10 client.17766 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:47:47.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.180+0000 7fae5dffb640 1 -- 172.21.15.23:0/4047778365 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5587eba6f890 con 0x5587ebdfd4f0 2024-03-24T10:47:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.180+0000 7fae5dffb640 10 client.17766 handle_command_reply: tid=0 2024-03-24T10:47:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.180+0000 7fae7f370740 1 client.17766 shutdown 2024-03-24T10:47:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.180+0000 7fae7f370740 1 -- 172.21.15.23:0/4047778365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd868f0 msgr2=0x5587ebd80a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.180+0000 7fae7f370740 1 --2- 172.21.15.23:0/4047778365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd868f0 0x5587ebd80a30 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fae7002e010 tx=0x7fae7002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.180+0000 7fae7f370740 1 -- 172.21.15.23:0/4047778365 shutdown_connections 2024-03-24T10:47:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.180+0000 7fae7f370740 1 --2- 172.21.15.23:0/4047778365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587ebd868f0 0x5587ebd80a30 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.180+0000 7fae7f370740 1 --2- 172.21.15.23:0/4047778365 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5587ebdfd4f0 0x5587ebdff950 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.180+0000 7fae7f370740 1 --2- 172.21.15.23:0/4047778365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587ebd7dd10 0x5587ebd802f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.180+0000 7fae7f370740 1 --2- 172.21.15.23:0/4047778365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587ebd7d3f0 0x5587ebd8ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.180+0000 7fae5dffb640 0 client.17766 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:47:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.180+0000 7fae7f370740 1 -- 172.21.15.23:0/4047778365 >> 172.21.15.23:0/4047778365 conn(0x5587ebd77b20 msgr2=0x5587ebd83870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7f370740 1 -- 172.21.15.23:0/4047778365 shutdown_connections 2024-03-24T10:47:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7f370740 1 -- 172.21.15.23:0/4047778365 wait complete. 2024-03-24T10:47:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7f370740 20 client.17766 trim_cache size 0 max 16384 2024-03-24T10:47:47.186 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:47:47.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7c6bb640 1 -- 172.21.15.23:0/4133421311 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fae600a22b0 msgr2=0x7fae600a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7c6bb640 1 --2- 172.21.15.23:0/4133421311 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fae600a22b0 0x7fae600a4770 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fae7002c630 tx=0x7fae7002e040 comp rx=0 tx=0).stop 2024-03-24T10:47:47.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7c6bb640 1 -- 172.21.15.23:0/4133421311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae740692d0 msgr2=0x7fae7419de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7c6bb640 1 --2- 172.21.15.23:0/4133421311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae740692d0 0x7fae7419de60 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fae6400e9f0 tx=0x7fae6400edf0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7c6bb640 1 -- 172.21.15.23:0/4133421311 shutdown_connections 2024-03-24T10:47:47.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7c6bb640 1 --2- 172.21.15.23:0/4133421311 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fae600a22b0 0x7fae600a4770 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7c6bb640 1 --2- 172.21.15.23:0/4133421311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae74107760 0x7fae74197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7c6bb640 1 --2- 172.21.15.23:0/4133421311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae740692d0 0x7fae7419de60 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7c6bb640 1 --2- 172.21.15.23:0/4133421311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae74068910 0x7fae7419d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7c6bb640 1 -- 172.21.15.23:0/4133421311 >> 172.21.15.23:0/4133421311 conn(0x7fae74077710 msgr2=0x7fae740fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7c6bb640 1 -- 172.21.15.23:0/4133421311 shutdown_connections 2024-03-24T10:47:47.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.181+0000 7fae7c6bb640 1 -- 172.21.15.23:0/4133421311 wait complete. 2024-03-24T10:47:47.193 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:47:47.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.301+0000 7f06f48a4640 1 -- 172.21.15.23:0/2620954744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f01005a0 msgr2=0x7f06f0100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.301+0000 7f06f48a4640 1 --2- 172.21.15.23:0/2620954744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f01005a0 0x7f06f0100a20 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f06d8009900 tx=0x7f06d802bc30 comp rx=0 tx=0).stop 2024-03-24T10:47:47.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.301+0000 7f06f48a4640 1 -- 172.21.15.23:0/2620954744 shutdown_connections 2024-03-24T10:47:47.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.301+0000 7f06f48a4640 1 --2- 172.21.15.23:0/2620954744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06f0100f60 0x7f06f010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.301+0000 7f06f48a4640 1 --2- 172.21.15.23:0/2620954744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f01005a0 0x7f06f0100a20 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.301+0000 7f06f48a4640 1 --2- 172.21.15.23:0/2620954744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06f0106660 0x7f06f0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.301+0000 7f06f48a4640 1 -- 172.21.15.23:0/2620954744 >> 172.21.15.23:0/2620954744 conn(0x7f06f0077710 msgr2=0x7f06f0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.301+0000 7f06f48a4640 1 -- 172.21.15.23:0/2620954744 shutdown_connections 2024-03-24T10:47:47.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.301+0000 7f06f48a4640 1 -- 172.21.15.23:0/2620954744 wait complete. 2024-03-24T10:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06f48a4640 1 Processor -- start 2024-03-24T10:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06f48a4640 1 -- start start 2024-03-24T10:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06f48a4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f01005a0 0x7f06f019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06f48a4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06f0100f60 0x7f06f019cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06f48a4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06f0106660 0x7f06f0196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06f48a4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f06f0197050 con 0x7f06f01005a0 2024-03-24T10:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06f48a4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f06f01971c0 con 0x7f06f0106660 2024-03-24T10:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06f48a4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f06f0197380 con 0x7f06f0100f60 2024-03-24T10:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06ee59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f01005a0 0x7f06f019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06ee59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f01005a0 0x7f06f019c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36554/0 (socket says 172.21.15.23:36554) 2024-03-24T10:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06ee59c640 1 -- 172.21.15.23:0/1137148332 learned_addr learned my addr 172.21.15.23:0/1137148332 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06eed9d640 1 --2- 172.21.15.23:0/1137148332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06f0106660 0x7f06f0196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06e7fff640 1 --2- 172.21.15.23:0/1137148332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06f0100f60 0x7f06f019cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06ee59c640 1 -- 172.21.15.23:0/1137148332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06f0100f60 msgr2=0x7f06f019cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06ee59c640 1 --2- 172.21.15.23:0/1137148332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06f0100f60 0x7f06f019cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06ee59c640 1 -- 172.21.15.23:0/1137148332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06f0106660 msgr2=0x7f06f0196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06ee59c640 1 --2- 172.21.15.23:0/1137148332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06f0106660 0x7f06f0196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06ee59c640 1 -- 172.21.15.23:0/1137148332 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e8009840 con 0x7f06f01005a0 2024-03-24T10:47:47.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.302+0000 7f06ee59c640 1 --2- 172.21.15.23:0/1137148332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f01005a0 0x7f06f019c850 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f06e000e9f0 tx=0x7f06e000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.303+0000 7f06e77fe640 1 -- 172.21.15.23:0/1137148332 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e0009700 con 0x7f06f01005a0 2024-03-24T10:47:47.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.303+0000 7f06e77fe640 1 -- 172.21.15.23:0/1137148332 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06e000ccc0 con 0x7f06f01005a0 2024-03-24T10:47:47.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.303+0000 7f06f48a4640 1 -- 172.21.15.23:0/1137148332 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d8009560 con 0x7f06f01005a0 2024-03-24T10:47:47.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.303+0000 7f06e77fe640 1 -- 172.21.15.23:0/1137148332 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e0018820 con 0x7f06f01005a0 2024-03-24T10:47:47.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.303+0000 7f06f48a4640 1 -- 172.21.15.23:0/1137148332 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06f01a3710 con 0x7f06f01005a0 2024-03-24T10:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.305+0000 7f06e77fe640 1 -- 172.21.15.23:0/1137148332 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f06e0010430 con 0x7f06f01005a0 2024-03-24T10:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.305+0000 7f06f7567740 1 Processor -- start 2024-03-24T10:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06f7567740 1 -- start start 2024-03-24T10:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06f7567740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df921737d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06f7567740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df92173d10 0x55df9217c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06f7567740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df9217c8f0 0x55df9217ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06f7567740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55df920e1840 con 0x55df921733f0 2024-03-24T10:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06f7567740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55df920ded70 con 0x55df9217c8f0 2024-03-24T10:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06f7567740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55df920da550 con 0x55df92173d10 2024-03-24T10:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06e77fe640 1 --2- 172.21.15.23:0/1137148332 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06dc0a22b0 0x7f06dc0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06ee59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df92173d10 0x55df9217c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06eed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df921737d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06e7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df9217c8f0 0x55df9217ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06eed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df921737d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36570/0 (socket says 172.21.15.23:36570) 2024-03-24T10:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06eed9d640 1 -- 172.21.15.23:0/673463892 learned_addr learned my addr 172.21.15.23:0/673463892 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06eed9d640 1 --2- 172.21.15.23:0/1137148332 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06dc0a22b0 0x7f06dc0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06e77fe640 1 -- 172.21.15.23:0/1137148332 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f06e0014070 con 0x7f06f01005a0 2024-03-24T10:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06e7fff640 1 -- 172.21.15.23:0/673463892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df92173d10 msgr2=0x55df9217c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06e7fff640 1 --2- 172.21.15.23:0/673463892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df92173d10 0x55df9217c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06e7fff640 1 -- 172.21.15.23:0/673463892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 msgr2=0x55df921737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06e7fff640 1 --2- 172.21.15.23:0/673463892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df921737d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.306+0000 7f06e7fff640 1 -- 172.21.15.23:0/673463892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df9206fde0 con 0x55df9217c8f0 2024-03-24T10:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.307+0000 7f06eed9d640 1 --2- 172.21.15.23:0/1137148332 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06dc0a22b0 0x7f06dc0a4770 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f06e8009fd0 tx=0x7f06e8009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.307+0000 7f06e7fff640 1 --2- 172.21.15.23:0/673463892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df9217c8f0 0x55df9217ece0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f06d8000c00 tx=0x7f06d802c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2ed755dbbb9216 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.307+0000 7f06e4ff9640 1 -- 172.21.15.23:0/673463892 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d802c910 con 0x55df9217c8f0 2024-03-24T10:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.307+0000 7f06e4ff9640 1 -- 172.21.15.23:0/673463892 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06d802ca70 con 0x55df9217c8f0 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.307+0000 7f06e4ff9640 1 -- 172.21.15.23:0/673463892 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d80050e0 con 0x55df9217c8f0 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.308+0000 7f06f7567740 1 -- 172.21.15.23:0/673463892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df9217c8f0 msgr2=0x55df9217ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.308+0000 7f06f7567740 1 --2- 172.21.15.23:0/673463892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df9217c8f0 0x55df9217ece0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f06d8000c00 tx=0x7f06d802c6a0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.308+0000 7f06f7567740 1 -- 172.21.15.23:0/673463892 shutdown_connections 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.308+0000 7f06f7567740 1 --2- 172.21.15.23:0/673463892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df9217c8f0 0x55df9217ece0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.308+0000 7f06f7567740 1 --2- 172.21.15.23:0/673463892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df92173d10 0x55df9217c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.308+0000 7f06f7567740 1 --2- 172.21.15.23:0/673463892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df921737d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.308+0000 7f06f7567740 1 -- 172.21.15.23:0/673463892 >> 172.21.15.23:0/673463892 conn(0x55df9216db20 msgr2=0x55df9216df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.308+0000 7f06f7567740 1 -- 172.21.15.23:0/673463892 shutdown_connections 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.308+0000 7f06f7567740 1 -- 172.21.15.23:0/673463892 wait complete. 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.308+0000 7f06f7567740 1 -- start start 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.308+0000 7f06f7567740 1 Processor -- start 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06f7567740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df92179930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06f7567740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 0x55df92179e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06f7567740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 0x55df9217a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06f7567740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55df91dad430 con 0x55df921733f0 2024-03-24T10:47:47.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06f7567740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55df91d69e00 con 0x55df92173d10 2024-03-24T10:47:47.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06f7567740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55df9205f590 con 0x55df9217c8f0 2024-03-24T10:47:47.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06f7567740 10 client.0 _ll_register_callbacks cb 0x55df920cefc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:47.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06ee59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df92179930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06ee59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df92179930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36574/0 (socket says 172.21.15.23:36574) 2024-03-24T10:47:47.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06ee59c640 1 -- 172.21.15.23:0/1219231975 learned_addr learned my addr 172.21.15.23:0/1219231975 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06eed9d640 1 --2- 172.21.15.23:0/1219231975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 0x55df9217a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06e7fff640 1 --2- 172.21.15.23:0/1219231975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 0x55df92179e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06ee59c640 1 -- 172.21.15.23:0/1219231975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 msgr2=0x55df9217a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06ee59c640 1 --2- 172.21.15.23:0/1219231975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 0x55df9217a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06ee59c640 1 -- 172.21.15.23:0/1219231975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 msgr2=0x55df92179e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06ee59c640 1 --2- 172.21.15.23:0/1219231975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 0x55df92179e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06ee59c640 1 -- 172.21.15.23:0/1219231975 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06f010dd20 con 0x55df921733f0 2024-03-24T10:47:47.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06ee59c640 1 --2- 172.21.15.23:0/1219231975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df92179930 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f06e00cbc10 tx=0x7f06e0010a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659b515dd98a0dfe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:47.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06d5ffb640 1 -- 172.21.15.23:0/1219231975 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e00cc030 con 0x55df921733f0 2024-03-24T10:47:47.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06d5ffb640 1 -- 172.21.15.23:0/1219231975 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f06e009efa0 con 0x55df921733f0 2024-03-24T10:47:47.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.309+0000 7f06f7567740 1 -- 172.21.15.23:0/1219231975 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55df91dd97e0 con 0x55df921733f0 2024-03-24T10:47:47.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.310+0000 7f06d5ffb640 1 -- 172.21.15.23:0/1219231975 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f06e00a35f0 con 0x55df921733f0 2024-03-24T10:47:47.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.310+0000 7f06f7567740 10 client.17778 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:47.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.310+0000 7f06f7567740 1 -- 172.21.15.23:0/1219231975 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55df920d8bd0 con 0x55df921733f0 2024-03-24T10:47:47.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.310+0000 7f06d5ffb640 1 -- 172.21.15.23:0/1219231975 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55df91dd97e0 con 0x55df921733f0 2024-03-24T10:47:47.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.310+0000 7f06d5ffb640 1 -- 172.21.15.23:0/1219231975 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f06e00b3c10 con 0x55df921733f0 2024-03-24T10:47:47.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.310+0000 7f06f7567740 10 client.17778 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:47.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.310+0000 7f06f7567740 10 client.17778 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:47:47.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.310+0000 7f06f7567740 20 client.17778 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:47.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.310+0000 7f06f7567740 1 --2- 172.21.15.23:0/1219231975 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55df9218ca10 0x55df9218ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.310+0000 7f06f7567740 4 client.17778 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:47.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.310+0000 7f06f7567740 1 -- 172.21.15.23:0/1219231975 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df91e16720 con 0x55df9218ca10 2024-03-24T10:47:47.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.310+0000 7f06e7fff640 1 --2- 172.21.15.23:0/1219231975 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55df9218ca10 0x55df9218ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.310+0000 7f06d5ffb640 10 client.17778 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:47:47.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.310+0000 7f06e7fff640 1 --2- 172.21.15.23:0/1219231975 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55df9218ca10 0x55df9218ee70 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c551f3cdf9763e70 server_cookie=30c1faabdb592f08 in_seq=0 out_seq=0 2024-03-24T10:47:47.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.311+0000 7f06d5ffb640 1 -- 172.21.15.23:0/1219231975 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55df91e16720 con 0x55df9218ca10 2024-03-24T10:47:47.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.311+0000 7f06d5ffb640 10 client.17778 handle_command_reply: tid=0 2024-03-24T10:47:47.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.311+0000 7f06f7567740 1 client.17778 shutdown 2024-03-24T10:47:47.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.312+0000 7f06f7567740 1 -- 172.21.15.23:0/1219231975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 msgr2=0x55df92179930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.312+0000 7f06f7567740 1 --2- 172.21.15.23:0/1219231975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df92179930 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f06e00cbc10 tx=0x7f06e0010a90 comp rx=0 tx=0).stop 2024-03-24T10:47:47.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.312+0000 7f06f7567740 1 -- 172.21.15.23:0/1219231975 shutdown_connections 2024-03-24T10:47:47.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.312+0000 7f06f7567740 1 --2- 172.21.15.23:0/1219231975 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55df9218ca10 0x55df9218ee70 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.312+0000 7f06f7567740 1 --2- 172.21.15.23:0/1219231975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 0x55df9217a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.312+0000 7f06f7567740 1 --2- 172.21.15.23:0/1219231975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 0x55df92179e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.312+0000 7f06d5ffb640 0 client.17778 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:47:47.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.312+0000 7f06f7567740 1 --2- 172.21.15.23:0/1219231975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df92179930 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.312+0000 7f06f7567740 1 -- 172.21.15.23:0/1219231975 >> 172.21.15.23:0/1219231975 conn(0x55df9216db20 msgr2=0x55df9216f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.312+0000 7f06f7567740 1 -- 172.21.15.23:0/1219231975 shutdown_connections 2024-03-24T10:47:47.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.312+0000 7f06f7567740 1 -- 172.21.15.23:0/1219231975 wait complete. 2024-03-24T10:47:47.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.312+0000 7f06f7567740 20 client.17778 trim_cache size 0 max 16384 2024-03-24T10:47:47.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06f7567740 1 Processor -- start 2024-03-24T10:47:47.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06f7567740 1 -- start start 2024-03-24T10:47:47.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06f7567740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df9217c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06f7567740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 0x55df9216ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06f7567740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 0x55df92194780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06f7567740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55df92172e30 con 0x55df921733f0 2024-03-24T10:47:47.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06f7567740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55df921727f0 con 0x55df92173d10 2024-03-24T10:47:47.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06f7567740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55df92173050 con 0x55df9217c8f0 2024-03-24T10:47:47.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06ee59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df9217c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06ee59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df9217c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36576/0 (socket says 172.21.15.23:36576) 2024-03-24T10:47:47.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06ee59c640 1 -- 172.21.15.23:0/3221040366 learned_addr learned my addr 172.21.15.23:0/3221040366 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06e7fff640 1 --2- 172.21.15.23:0/3221040366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 0x55df9216ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06eed9d640 1 --2- 172.21.15.23:0/3221040366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 0x55df92194780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06ee59c640 1 -- 172.21.15.23:0/3221040366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 msgr2=0x55df92194780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06ee59c640 1 --2- 172.21.15.23:0/3221040366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 0x55df92194780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06ee59c640 1 -- 172.21.15.23:0/3221040366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 msgr2=0x55df9216ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06ee59c640 1 --2- 172.21.15.23:0/3221040366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 0x55df9216ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06ee59c640 1 -- 172.21.15.23:0/3221040366 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df9216d3c0 con 0x55df921733f0 2024-03-24T10:47:47.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.326+0000 7f06ee59c640 1 --2- 172.21.15.23:0/3221040366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df9217c060 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f06e0097820 tx=0x7f06e00970f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e322bdb96f243d98 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.327+0000 7f06d7fff640 1 -- 172.21.15.23:0/3221040366 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e00aeea0 con 0x55df921733f0 2024-03-24T10:47:47.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.327+0000 7f06d7fff640 1 -- 172.21.15.23:0/3221040366 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06e009a7b0 con 0x55df921733f0 2024-03-24T10:47:47.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.327+0000 7f06d7fff640 1 -- 172.21.15.23:0/3221040366 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e00a35f0 con 0x55df921733f0 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.327+0000 7f06f7567740 1 -- 172.21.15.23:0/3221040366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 msgr2=0x55df9217c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.327+0000 7f06f7567740 1 --2- 172.21.15.23:0/3221040366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df9217c060 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f06e0097820 tx=0x7f06e00970f0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.327+0000 7f06f7567740 1 -- 172.21.15.23:0/3221040366 shutdown_connections 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.327+0000 7f06f7567740 1 --2- 172.21.15.23:0/3221040366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 0x55df92194780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.327+0000 7f06f7567740 1 --2- 172.21.15.23:0/3221040366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 0x55df9216ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.327+0000 7f06f7567740 1 --2- 172.21.15.23:0/3221040366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df9217c060 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.327+0000 7f06f7567740 1 -- 172.21.15.23:0/3221040366 >> 172.21.15.23:0/3221040366 conn(0x55df9216db20 msgr2=0x55df9216ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.327+0000 7f06f7567740 1 -- 172.21.15.23:0/3221040366 shutdown_connections 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.327+0000 7f06f7567740 1 -- 172.21.15.23:0/3221040366 wait complete. 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06f7567740 1 -- start start 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06f7567740 1 Processor -- start 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06f7567740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df921808a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06f7567740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 0x55df92180de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06f7567740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 0x55df921ebd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06f7567740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55df921ec2c0 con 0x55df921733f0 2024-03-24T10:47:47.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06f7567740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55df921ec400 con 0x55df92173d10 2024-03-24T10:47:47.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06f7567740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55df921ec5c0 con 0x55df9217c8f0 2024-03-24T10:47:47.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06f7567740 10 client.0 _ll_register_callbacks cb 0x7f06e009aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:47.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06ee59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df921808a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06ee59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df921808a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36584/0 (socket says 172.21.15.23:36584) 2024-03-24T10:47:47.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06ee59c640 1 -- 172.21.15.23:0/2621128177 learned_addr learned my addr 172.21.15.23:0/2621128177 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06e7fff640 1 --2- 172.21.15.23:0/2621128177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 0x55df92180de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06eed9d640 1 --2- 172.21.15.23:0/2621128177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 0x55df921ebd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06ee59c640 1 -- 172.21.15.23:0/2621128177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 msgr2=0x55df921ebd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06ee59c640 1 --2- 172.21.15.23:0/2621128177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 0x55df921ebd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06ee59c640 1 -- 172.21.15.23:0/2621128177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 msgr2=0x55df92180de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06ee59c640 1 --2- 172.21.15.23:0/2621128177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 0x55df92180de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06ee59c640 1 -- 172.21.15.23:0/2621128177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df920d8bd0 con 0x55df921733f0 2024-03-24T10:47:47.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06eed9d640 1 --2- 172.21.15.23:0/2621128177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 0x55df921ebd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:47:47.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06ee59c640 1 --2- 172.21.15.23:0/2621128177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df921808a0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f06e00a30c0 tx=0x7f06e001ae40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42eb392eb6429e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.328+0000 7f06d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:47.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06d5ffb640 1 -- 172.21.15.23:0/2621128177 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e00aeea0 con 0x55df921733f0 2024-03-24T10:47:47.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06d5ffb640 1 -- 172.21.15.23:0/2621128177 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f06e009b550 con 0x55df921733f0 2024-03-24T10:47:47.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06f7567740 1 -- 172.21.15.23:0/2621128177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06d802ca70 con 0x55df921733f0 2024-03-24T10:47:47.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06d5ffb640 1 -- 172.21.15.23:0/2621128177 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f06e00a35f0 con 0x55df921733f0 2024-03-24T10:47:47.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06f7567740 10 client.17784 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:47.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06f7567740 1 -- 172.21.15.23:0/2621128177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55df92074c80 con 0x55df921733f0 2024-03-24T10:47:47.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06d5ffb640 1 -- 172.21.15.23:0/2621128177 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06e00a7020 con 0x55df921733f0 2024-03-24T10:47:47.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06d5ffb640 1 -- 172.21.15.23:0/2621128177 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f06e0099d10 con 0x55df921733f0 2024-03-24T10:47:47.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06f7567740 10 client.17784 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:47.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06f7567740 10 client.17784 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06f7567740 20 client.17784 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06f7567740 1 --2- 172.21.15.23:0/2621128177 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55df921f3a00 0x55df921f5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06f7567740 4 client.17784 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06f7567740 1 -- 172.21.15.23:0/2621128177 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55df91e65890 con 0x55df921f3a00 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.329+0000 7f06e7fff640 1 --2- 172.21.15.23:0/2621128177 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55df921f3a00 0x55df921f5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06e7fff640 1 --2- 172.21.15.23:0/2621128177 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55df921f3a00 0x55df921f5df0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c9897f759dfb57ae server_cookie=b5c9f42964549828 in_seq=0 out_seq=0 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06d5ffb640 10 client.17784 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06d5ffb640 1 -- 172.21.15.23:0/2621128177 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55df91e65890 con 0x55df921f3a00 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06d5ffb640 10 client.17784 handle_command_reply: tid=0 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06f7567740 1 client.17784 shutdown 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06f7567740 1 -- 172.21.15.23:0/2621128177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 msgr2=0x55df921808a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06f7567740 1 --2- 172.21.15.23:0/2621128177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df921808a0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f06e00a30c0 tx=0x7f06e001ae40 comp rx=0 tx=0).stop 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06f7567740 1 -- 172.21.15.23:0/2621128177 shutdown_connections 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06f7567740 1 --2- 172.21.15.23:0/2621128177 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55df921f3a00 0x55df921f5df0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06f7567740 1 --2- 172.21.15.23:0/2621128177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df9217c8f0 0x55df921ebd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06d5ffb640 0 client.17784 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:47:47.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06f7567740 1 --2- 172.21.15.23:0/2621128177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55df92173d10 0x55df92180de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06f7567740 1 --2- 172.21.15.23:0/2621128177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df921733f0 0x55df921808a0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06f7567740 1 -- 172.21.15.23:0/2621128177 >> 172.21.15.23:0/2621128177 conn(0x55df9216db20 msgr2=0x55df9216cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06f7567740 1 -- 172.21.15.23:0/2621128177 shutdown_connections 2024-03-24T10:47:47.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06f7567740 1 -- 172.21.15.23:0/2621128177 wait complete. 2024-03-24T10:47:47.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.330+0000 7f06f7567740 20 client.17784 trim_cache size 0 max 16384 2024-03-24T10:47:47.337 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:47:47.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.331+0000 7f06f48a4640 1 -- 172.21.15.23:0/1137148332 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06dc0a22b0 msgr2=0x7f06dc0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.331+0000 7f06f48a4640 1 --2- 172.21.15.23:0/1137148332 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06dc0a22b0 0x7f06dc0a4770 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f06e8009fd0 tx=0x7f06e8009500 comp rx=0 tx=0).stop 2024-03-24T10:47:47.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.331+0000 7f06f48a4640 1 -- 172.21.15.23:0/1137148332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f01005a0 msgr2=0x7f06f019c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.331+0000 7f06f48a4640 1 --2- 172.21.15.23:0/1137148332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f01005a0 0x7f06f019c850 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f06e000e9f0 tx=0x7f06e000edf0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.331+0000 7f06f48a4640 1 -- 172.21.15.23:0/1137148332 shutdown_connections 2024-03-24T10:47:47.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.331+0000 7f06f48a4640 1 --2- 172.21.15.23:0/1137148332 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06dc0a22b0 0x7f06dc0a4770 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.331+0000 7f06f48a4640 1 --2- 172.21.15.23:0/1137148332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06f0106660 0x7f06f0196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.331+0000 7f06f48a4640 1 --2- 172.21.15.23:0/1137148332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06f0100f60 0x7f06f019cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.331+0000 7f06f48a4640 1 --2- 172.21.15.23:0/1137148332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f01005a0 0x7f06f019c850 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.331+0000 7f06f48a4640 1 -- 172.21.15.23:0/1137148332 >> 172.21.15.23:0/1137148332 conn(0x7f06f0077710 msgr2=0x7f06f00ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.331+0000 7f06f48a4640 1 -- 172.21.15.23:0/1137148332 shutdown_connections 2024-03-24T10:47:47.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.331+0000 7f06f48a4640 1 -- 172.21.15.23:0/1137148332 wait complete. 2024-03-24T10:47:47.342 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:47:47.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.468+0000 7f80b3bc7640 1 -- 172.21.15.23:0/2676462652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80ac100f80 msgr2=0x7f80ac10d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.468+0000 7f80b3bc7640 1 --2- 172.21.15.23:0/2676462652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80ac100f80 0x7f80ac10d490 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f80a8009cd0 tx=0x7f80a802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.468+0000 7f80b3bc7640 1 -- 172.21.15.23:0/2676462652 shutdown_connections 2024-03-24T10:47:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.468+0000 7f80b3bc7640 1 --2- 172.21.15.23:0/2676462652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80ac100f80 0x7f80ac10d490 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.468+0000 7f80b3bc7640 1 --2- 172.21.15.23:0/2676462652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80ac1005c0 0x7f80ac100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.468+0000 7f80b3bc7640 1 --2- 172.21.15.23:0/2676462652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80ac106660 0x7f80ac106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.468+0000 7f80b3bc7640 1 -- 172.21.15.23:0/2676462652 >> 172.21.15.23:0/2676462652 conn(0x7f80ac076990 msgr2=0x7f80ac076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.468+0000 7f80b3bc7640 1 -- 172.21.15.23:0/2676462652 shutdown_connections 2024-03-24T10:47:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.468+0000 7f80b3bc7640 1 -- 172.21.15.23:0/2676462652 wait complete. 2024-03-24T10:47:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.469+0000 7f80b3bc7640 1 Processor -- start 2024-03-24T10:47:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.469+0000 7f80b3bc7640 1 -- start start 2024-03-24T10:47:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.469+0000 7f80b3bc7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80ac1005c0 0x7f80ac102340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.469+0000 7f80b3bc7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80ac100f80 0x7f80ac102880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.469+0000 7f80b3bc7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80ac106660 0x7f80ac102dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.469+0000 7f80b3bc7640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f80ac1a6090 con 0x7f80ac106660 2024-03-24T10:47:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.469+0000 7f80b3bc7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f80ac1a6200 con 0x7f80ac1005c0 2024-03-24T10:47:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.469+0000 7f80b3bc7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f80ac1a63c0 con 0x7f80ac100f80 2024-03-24T10:47:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.469+0000 7f80b2164640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80ac106660 0x7f80ac102dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.469+0000 7f80b1162640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80ac100f80 0x7f80ac102880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.469+0000 7f80b2164640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80ac106660 0x7f80ac102dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36588/0 (socket says 172.21.15.23:36588) 2024-03-24T10:47:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.469+0000 7f80b1963640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80ac1005c0 0x7f80ac102340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.469+0000 7f80b2164640 1 -- 172.21.15.23:0/4190556376 learned_addr learned my addr 172.21.15.23:0/4190556376 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.470+0000 7f80b2164640 1 -- 172.21.15.23:0/4190556376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80ac100f80 msgr2=0x7f80ac102880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.470+0000 7f80b2164640 1 --2- 172.21.15.23:0/4190556376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80ac100f80 0x7f80ac102880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.470+0000 7f80b2164640 1 -- 172.21.15.23:0/4190556376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80ac1005c0 msgr2=0x7f80ac102340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.470+0000 7f80b2164640 1 --2- 172.21.15.23:0/4190556376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80ac1005c0 0x7f80ac102340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.470+0000 7f80b2164640 1 -- 172.21.15.23:0/4190556376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a8009900 con 0x7f80ac106660 2024-03-24T10:47:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.470+0000 7f80b1963640 1 --2- 172.21.15.23:0/4190556376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80ac1005c0 0x7f80ac102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:47.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.470+0000 7f80b1162640 1 --2- 172.21.15.23:0/4190556376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80ac100f80 0x7f80ac102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:47.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.470+0000 7f80b2164640 1 --2- 172.21.15.23:0/4190556376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80ac106660 0x7f80ac102dc0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f80a8004af0 tx=0x7f80a8032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.470+0000 7f80a2ffd640 1 -- 172.21.15.23:0/4190556376 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a80328b0 con 0x7f80ac106660 2024-03-24T10:47:47.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.470+0000 7f80a2ffd640 1 -- 172.21.15.23:0/4190556376 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80a8003e50 con 0x7f80ac106660 2024-03-24T10:47:47.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.470+0000 7f80b3bc7640 1 -- 172.21.15.23:0/4190556376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80ac1a6640 con 0x7f80ac106660 2024-03-24T10:47:47.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.470+0000 7f80a2ffd640 1 -- 172.21.15.23:0/4190556376 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a8004740 con 0x7f80ac106660 2024-03-24T10:47:47.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.470+0000 7f80b3bc7640 1 -- 172.21.15.23:0/4190556376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80ac1a6af0 con 0x7f80ac106660 2024-03-24T10:47:47.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.472+0000 7f80a2ffd640 1 -- 172.21.15.23:0/4190556376 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f80a800a040 con 0x7f80ac106660 2024-03-24T10:47:47.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80b679e740 1 Processor -- start 2024-03-24T10:47:47.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80b679e740 1 -- start start 2024-03-24T10:47:47.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80a2ffd640 1 --2- 172.21.15.23:0/4190556376 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f80980a2230 0x7f80980a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80b679e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e3d3f0 0x5587f0e3d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80b679e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3dd10 0x5587f0e463b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80b679e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e468f0 0x5587f0e48ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80b679e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5587f0dab840 con 0x5587f0e468f0 2024-03-24T10:47:47.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80b679e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5587f0da8d70 con 0x5587f0e3dd10 2024-03-24T10:47:47.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80b679e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5587f0da4550 con 0x5587f0e3d3f0 2024-03-24T10:47:47.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80b1162640 1 --2- 172.21.15.23:0/4190556376 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f80980a2230 0x7f80980a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80a2ffd640 1 -- 172.21.15.23:0/4190556376 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f80a80f07a0 con 0x7f80ac106660 2024-03-24T10:47:47.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80b2164640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e468f0 0x5587f0e48ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80b2164640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e468f0 0x5587f0e48ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36592/0 (socket says 172.21.15.23:36592) 2024-03-24T10:47:47.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80b1963640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e3d3f0 0x5587f0e3d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80b2164640 1 -- 172.21.15.23:0/1314135709 learned_addr learned my addr 172.21.15.23:0/1314135709 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.473+0000 7f80b1162640 1 --2- 172.21.15.23:0/1314135709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3dd10 0x5587f0e463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.474+0000 7f80b1963640 1 -- 172.21.15.23:0/1314135709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3dd10 msgr2=0x5587f0e463b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.474+0000 7f80b1963640 1 --2- 172.21.15.23:0/1314135709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3dd10 0x5587f0e463b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.474+0000 7f80b1963640 1 -- 172.21.15.23:0/1314135709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e468f0 msgr2=0x5587f0e48ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.474+0000 7f80b1963640 1 --2- 172.21.15.23:0/1314135709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e468f0 0x5587f0e48ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.474+0000 7f80b1963640 1 -- 172.21.15.23:0/1314135709 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587f0d39de0 con 0x5587f0e3d3f0 2024-03-24T10:47:47.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.474+0000 7f80b1963640 1 --2- 172.21.15.23:0/1314135709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e3d3f0 0x5587f0e3d7d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f80a4009f20 tx=0x7f80a40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a33b74869bd33701 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.474+0000 7f80b1162640 1 --2- 172.21.15.23:0/4190556376 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f80980a2230 0x7f80980a46f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f809c009710 tx=0x7f809c009f50 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.474+0000 7f8097fff640 1 -- 172.21.15.23:0/1314135709 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a40040f0 con 0x5587f0e3d3f0 2024-03-24T10:47:47.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.474+0000 7f8097fff640 1 -- 172.21.15.23:0/1314135709 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80a4005500 con 0x5587f0e3d3f0 2024-03-24T10:47:47.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.474+0000 7f8097fff640 1 -- 172.21.15.23:0/1314135709 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a4004db0 con 0x5587f0e3d3f0 2024-03-24T10:47:47.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.475+0000 7f80b679e740 1 -- 172.21.15.23:0/1314135709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e3d3f0 msgr2=0x5587f0e3d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.475+0000 7f80b679e740 1 --2- 172.21.15.23:0/1314135709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e3d3f0 0x5587f0e3d7d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f80a4009f20 tx=0x7f80a40118d0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.475+0000 7f80b679e740 1 -- 172.21.15.23:0/1314135709 shutdown_connections 2024-03-24T10:47:47.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.475+0000 7f80b679e740 1 --2- 172.21.15.23:0/1314135709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e468f0 0x5587f0e48ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.475+0000 7f80b679e740 1 --2- 172.21.15.23:0/1314135709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3dd10 0x5587f0e463b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.475+0000 7f80b679e740 1 --2- 172.21.15.23:0/1314135709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e3d3f0 0x5587f0e3d7d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.475+0000 7f80b679e740 1 -- 172.21.15.23:0/1314135709 >> 172.21.15.23:0/1314135709 conn(0x5587f0e37b20 msgr2=0x5587f0e37f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.475+0000 7f80b679e740 1 -- 172.21.15.23:0/1314135709 shutdown_connections 2024-03-24T10:47:47.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.475+0000 7f80b679e740 1 -- 172.21.15.23:0/1314135709 wait complete. 2024-03-24T10:47:47.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.475+0000 7f80b679e740 1 -- start start 2024-03-24T10:47:47.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.475+0000 7f80b679e740 1 Processor -- start 2024-03-24T10:47:47.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b679e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3d3f0 0x5587f0e43930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b679e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e43e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b679e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e468f0 0x5587f0e443b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b679e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5587f0a77430 con 0x5587f0e3dd10 2024-03-24T10:47:47.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b679e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5587f0a33e00 con 0x5587f0e3d3f0 2024-03-24T10:47:47.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b679e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5587f0d29590 con 0x5587f0e468f0 2024-03-24T10:47:47.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b679e740 10 client.0 _ll_register_callbacks cb 0x5587f0d98fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:47.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b1162640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e43e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b1162640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e43e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36598/0 (socket says 172.21.15.23:36598) 2024-03-24T10:47:47.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b1162640 1 -- 172.21.15.23:0/2020653157 learned_addr learned my addr 172.21.15.23:0/2020653157 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b1963640 1 --2- 172.21.15.23:0/2020653157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3d3f0 0x5587f0e43930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b2164640 1 --2- 172.21.15.23:0/2020653157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e468f0 0x5587f0e443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b1162640 1 -- 172.21.15.23:0/2020653157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e468f0 msgr2=0x5587f0e443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b1162640 1 --2- 172.21.15.23:0/2020653157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e468f0 0x5587f0e443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b1162640 1 -- 172.21.15.23:0/2020653157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3d3f0 msgr2=0x5587f0e43930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b1162640 1 --2- 172.21.15.23:0/2020653157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3d3f0 0x5587f0e43930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b1162640 1 -- 172.21.15.23:0/2020653157 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80a4009c00 con 0x5587f0e3dd10 2024-03-24T10:47:47.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b1162640 1 --2- 172.21.15.23:0/2020653157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e43e70 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f809c004530 tx=0x7f809c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f3ea065d446a9b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80957fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:47.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80957fa640 1 -- 172.21.15.23:0/2020653157 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c016040 con 0x5587f0e3dd10 2024-03-24T10:47:47.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80957fa640 1 -- 172.21.15.23:0/2020653157 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f809c0116a0 con 0x5587f0e3dd10 2024-03-24T10:47:47.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b679e740 1 -- 172.21.15.23:0/2020653157 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5587f0aa37e0 con 0x5587f0e3dd10 2024-03-24T10:47:47.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80957fa640 1 -- 172.21.15.23:0/2020653157 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f809c031460 con 0x5587f0e3dd10 2024-03-24T10:47:47.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b679e740 10 client.17796 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:47.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.476+0000 7f80b679e740 1 -- 172.21.15.23:0/2020653157 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5587f0da2bd0 con 0x5587f0e3dd10 2024-03-24T10:47:47.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.477+0000 7f80957fa640 1 -- 172.21.15.23:0/2020653157 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5587f0aa37e0 con 0x5587f0e3dd10 2024-03-24T10:47:47.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.477+0000 7f80957fa640 1 -- 172.21.15.23:0/2020653157 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f809c026030 con 0x5587f0e3dd10 2024-03-24T10:47:47.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.477+0000 7f80b679e740 10 client.17796 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:47.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.477+0000 7f80b679e740 10 client.17796 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:47.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.477+0000 7f80b679e740 20 client.17796 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:47.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.477+0000 7f80b679e740 1 --2- 172.21.15.23:0/2020653157 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5587f0e56af0 0x5587f0e58f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.477+0000 7f80b679e740 4 client.17796 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:47.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.477+0000 7f80b679e740 1 -- 172.21.15.23:0/2020653157 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587f0ae0720 con 0x5587f0e56af0 2024-03-24T10:47:47.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.477+0000 7f80b1963640 1 --2- 172.21.15.23:0/2020653157 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5587f0e56af0 0x5587f0e58f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.477+0000 7f80957fa640 10 client.17796 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:47.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.477+0000 7f80b1963640 1 --2- 172.21.15.23:0/2020653157 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5587f0e56af0 0x5587f0e58f10 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cadbc78d1ac1db3 server_cookie=7c0cf3ce7c8683d in_seq=0 out_seq=0 2024-03-24T10:47:47.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.478+0000 7f80957fa640 1 -- 172.21.15.23:0/2020653157 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5587f0ae0720 con 0x5587f0e56af0 2024-03-24T10:47:47.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.478+0000 7f80957fa640 10 client.17796 handle_command_reply: tid=0 2024-03-24T10:47:47.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.478+0000 7f80b679e740 1 client.17796 shutdown 2024-03-24T10:47:47.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.479+0000 7f80b679e740 1 -- 172.21.15.23:0/2020653157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 msgr2=0x5587f0e43e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.479+0000 7f80b679e740 1 --2- 172.21.15.23:0/2020653157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e43e70 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f809c004530 tx=0x7f809c01d000 comp rx=0 tx=0).stop 2024-03-24T10:47:47.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.479+0000 7f80b679e740 1 -- 172.21.15.23:0/2020653157 shutdown_connections 2024-03-24T10:47:47.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.479+0000 7f80b679e740 1 --2- 172.21.15.23:0/2020653157 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5587f0e56af0 0x5587f0e58f10 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.479+0000 7f80b679e740 1 --2- 172.21.15.23:0/2020653157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e468f0 0x5587f0e443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.479+0000 7f80b679e740 1 --2- 172.21.15.23:0/2020653157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e43e70 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.479+0000 7f80957fa640 0 client.17796 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:47.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.479+0000 7f80b679e740 1 --2- 172.21.15.23:0/2020653157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3d3f0 0x5587f0e43930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.479+0000 7f80b679e740 1 -- 172.21.15.23:0/2020653157 >> 172.21.15.23:0/2020653157 conn(0x5587f0e37b20 msgr2=0x5587f0e39870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.479+0000 7f80b679e740 1 -- 172.21.15.23:0/2020653157 shutdown_connections 2024-03-24T10:47:47.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.479+0000 7f80b679e740 1 -- 172.21.15.23:0/2020653157 wait complete. 2024-03-24T10:47:47.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.479+0000 7f80b679e740 20 client.17796 trim_cache size 0 max 16384 2024-03-24T10:47:47.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.493+0000 7f80b679e740 1 Processor -- start 2024-03-24T10:47:47.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.493+0000 7f80b679e740 1 -- start start 2024-03-24T10:47:47.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.493+0000 7f80b679e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e3d3f0 0x5587f0e46060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.493+0000 7f80b679e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e36cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.493+0000 7f80b679e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e468f0 0x5587f0e441d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.493+0000 7f80b679e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5587f0e53490 con 0x5587f0e3dd10 2024-03-24T10:47:47.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.493+0000 7f80b679e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5587f0e3d050 con 0x5587f0e468f0 2024-03-24T10:47:47.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.493+0000 7f80b679e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5587f0e38b20 con 0x5587f0e3d3f0 2024-03-24T10:47:47.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.493+0000 7f80b1162640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e36cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.493+0000 7f80b1162640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e36cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36612/0 (socket says 172.21.15.23:36612) 2024-03-24T10:47:47.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.493+0000 7f80b1162640 1 -- 172.21.15.23:0/490024459 learned_addr learned my addr 172.21.15.23:0/490024459 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.493+0000 7f80b1963640 1 --2- 172.21.15.23:0/490024459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e3d3f0 0x5587f0e46060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.493+0000 7f80b2164640 1 --2- 172.21.15.23:0/490024459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e468f0 0x5587f0e441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.494+0000 7f80b1162640 1 -- 172.21.15.23:0/490024459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e3d3f0 msgr2=0x5587f0e46060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.494+0000 7f80b1162640 1 --2- 172.21.15.23:0/490024459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e3d3f0 0x5587f0e46060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.494+0000 7f80b1162640 1 -- 172.21.15.23:0/490024459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e468f0 msgr2=0x5587f0e441d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.494+0000 7f80b1162640 1 --2- 172.21.15.23:0/490024459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e468f0 0x5587f0e441d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.494+0000 7f80b1162640 1 -- 172.21.15.23:0/490024459 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587f0e373c0 con 0x5587f0e3dd10 2024-03-24T10:47:47.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.494+0000 7f80b2164640 1 --2- 172.21.15.23:0/490024459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e468f0 0x5587f0e441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:47:47.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.494+0000 7f80b1162640 1 --2- 172.21.15.23:0/490024459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e36cc0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f809c00fc40 tx=0x7f809c011f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eccfffeb6419419f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.494+0000 7f80977fe640 1 -- 172.21.15.23:0/490024459 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c016070 con 0x5587f0e3dd10 2024-03-24T10:47:47.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.494+0000 7f80977fe640 1 -- 172.21.15.23:0/490024459 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f809c003e50 con 0x5587f0e3dd10 2024-03-24T10:47:47.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.494+0000 7f80977fe640 1 -- 172.21.15.23:0/490024459 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c028b50 con 0x5587f0e3dd10 2024-03-24T10:47:47.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.495+0000 7f80b679e740 1 -- 172.21.15.23:0/490024459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 msgr2=0x5587f0e36cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.495+0000 7f80b679e740 1 --2- 172.21.15.23:0/490024459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e36cc0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f809c00fc40 tx=0x7f809c011f20 comp rx=0 tx=0).stop 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.495+0000 7f80b679e740 1 -- 172.21.15.23:0/490024459 shutdown_connections 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.495+0000 7f80b679e740 1 --2- 172.21.15.23:0/490024459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e468f0 0x5587f0e441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.495+0000 7f80b679e740 1 --2- 172.21.15.23:0/490024459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e36cc0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.495+0000 7f80b679e740 1 --2- 172.21.15.23:0/490024459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e3d3f0 0x5587f0e46060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.495+0000 7f80b679e740 1 -- 172.21.15.23:0/490024459 >> 172.21.15.23:0/490024459 conn(0x5587f0e37b20 msgr2=0x5587f0e38d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.495+0000 7f80b679e740 1 -- 172.21.15.23:0/490024459 shutdown_connections 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.495+0000 7f80b679e740 1 -- 172.21.15.23:0/490024459 wait complete. 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.495+0000 7f80b679e740 1 -- start start 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.495+0000 7f80b679e740 1 Processor -- start 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b679e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3d3f0 0x5587f0e4a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b679e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e4a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b679e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e468f0 0x5587f0e4ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b679e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5587f0ebbf30 con 0x5587f0e3dd10 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b679e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5587f0ebc0a0 con 0x5587f0e3d3f0 2024-03-24T10:47:47.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b679e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5587f0ebc260 con 0x5587f0e468f0 2024-03-24T10:47:47.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b679e740 10 client.0 _ll_register_callbacks cb 0x7f809c011560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:47.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b1162640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e4a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b2164640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e468f0 0x5587f0e4ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b1963640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3d3f0 0x5587f0e4a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b1162640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e4a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36618/0 (socket says 172.21.15.23:36618) 2024-03-24T10:47:47.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b1162640 1 -- 172.21.15.23:0/1439004377 learned_addr learned my addr 172.21.15.23:0/1439004377 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:47.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b1162640 1 -- 172.21.15.23:0/1439004377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e468f0 msgr2=0x5587f0e4ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b1162640 1 --2- 172.21.15.23:0/1439004377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e468f0 0x5587f0e4ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b1162640 1 -- 172.21.15.23:0/1439004377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3d3f0 msgr2=0x5587f0e4a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:47.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b1162640 1 --2- 172.21.15.23:0/1439004377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3d3f0 0x5587f0e4a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b1162640 1 -- 172.21.15.23:0/1439004377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587f0da2bd0 con 0x5587f0e3dd10 2024-03-24T10:47:47.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b2164640 1 --2- 172.21.15.23:0/1439004377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e468f0 0x5587f0e4ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80b1162640 1 --2- 172.21.15.23:0/1439004377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e4a800 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f809c011610 tx=0x7f809c004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f35cc1f8d3e065 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.496+0000 7f80957fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80957fa640 1 -- 172.21.15.23:0/1439004377 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c016040 con 0x5587f0e3dd10 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80957fa640 1 -- 172.21.15.23:0/1439004377 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f809c028400 con 0x5587f0e3dd10 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80b679e740 1 -- 172.21.15.23:0/1439004377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80a4005500 con 0x5587f0e3dd10 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80957fa640 1 -- 172.21.15.23:0/1439004377 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f809c0116a0 con 0x5587f0e3dd10 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80b679e740 10 client.17802 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80b679e740 1 -- 172.21.15.23:0/1439004377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5587f0d3ec80 con 0x5587f0e3dd10 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80957fa640 1 -- 172.21.15.23:0/1439004377 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f809c050020 con 0x5587f0e3dd10 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80957fa640 1 -- 172.21.15.23:0/1439004377 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f809c041070 con 0x5587f0e3dd10 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80b679e740 10 client.17802 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80b679e740 10 client.17802 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80b679e740 20 client.17802 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80b679e740 1 --2- 172.21.15.23:0/1439004377 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5587f0ebd620 0x5587f0ebfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:47.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80b679e740 4 client.17802 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80b679e740 1 -- 172.21.15.23:0/1439004377 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5587f0b2f890 con 0x5587f0ebd620 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.497+0000 7f80b1963640 1 --2- 172.21.15.23:0/1439004377 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5587f0ebd620 0x5587f0ebfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80b1963640 1 --2- 172.21.15.23:0/1439004377 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5587f0ebd620 0x5587f0ebfa10 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc7028941d332b08 server_cookie=587f38ad2ec88c07 in_seq=0 out_seq=0 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80957fa640 10 client.17802 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80957fa640 1 -- 172.21.15.23:0/1439004377 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5587f0b2f890 con 0x5587f0ebd620 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80957fa640 10 client.17802 handle_command_reply: tid=0 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80b679e740 1 client.17802 shutdown 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80b679e740 1 -- 172.21.15.23:0/1439004377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 msgr2=0x5587f0e4a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80b679e740 1 --2- 172.21.15.23:0/1439004377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e4a800 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f809c011610 tx=0x7f809c004690 comp rx=0 tx=0).stop 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80b679e740 1 -- 172.21.15.23:0/1439004377 shutdown_connections 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80b679e740 1 --2- 172.21.15.23:0/1439004377 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5587f0ebd620 0x5587f0ebfa10 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80b679e740 1 --2- 172.21.15.23:0/1439004377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5587f0e468f0 0x5587f0e4ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80b679e740 1 --2- 172.21.15.23:0/1439004377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5587f0e3dd10 0x5587f0e4a800 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80957fa640 0 client.17802 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80b679e740 1 --2- 172.21.15.23:0/1439004377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5587f0e3d3f0 0x5587f0e4a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80b679e740 1 -- 172.21.15.23:0/1439004377 >> 172.21.15.23:0/1439004377 conn(0x5587f0e37b20 msgr2=0x5587f0e36b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80b679e740 1 -- 172.21.15.23:0/1439004377 shutdown_connections 2024-03-24T10:47:47.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80b679e740 1 -- 172.21.15.23:0/1439004377 wait complete. 2024-03-24T10:47:47.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.498+0000 7f80b679e740 20 client.17802 trim_cache size 0 max 16384 2024-03-24T10:47:47.505 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:47:47.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.499+0000 7f80b3bc7640 1 -- 172.21.15.23:0/4190556376 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f80980a2230 msgr2=0x7f80980a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.499+0000 7f80b3bc7640 1 --2- 172.21.15.23:0/4190556376 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f80980a2230 0x7f80980a46f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f809c009710 tx=0x7f809c009f50 comp rx=0 tx=0).stop 2024-03-24T10:47:47.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.499+0000 7f80b3bc7640 1 -- 172.21.15.23:0/4190556376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80ac106660 msgr2=0x7f80ac102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.499+0000 7f80b3bc7640 1 --2- 172.21.15.23:0/4190556376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80ac106660 0x7f80ac102dc0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f80a8004af0 tx=0x7f80a8032570 comp rx=0 tx=0).stop 2024-03-24T10:47:47.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.499+0000 7f80b3bc7640 1 -- 172.21.15.23:0/4190556376 shutdown_connections 2024-03-24T10:47:47.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.499+0000 7f80b3bc7640 1 --2- 172.21.15.23:0/4190556376 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f80980a2230 0x7f80980a46f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.499+0000 7f80b3bc7640 1 --2- 172.21.15.23:0/4190556376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80ac106660 0x7f80ac102dc0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.499+0000 7f80b3bc7640 1 --2- 172.21.15.23:0/4190556376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80ac100f80 0x7f80ac102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.499+0000 7f80b3bc7640 1 --2- 172.21.15.23:0/4190556376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80ac1005c0 0x7f80ac102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.499+0000 7f80b3bc7640 1 -- 172.21.15.23:0/4190556376 >> 172.21.15.23:0/4190556376 conn(0x7f80ac076990 msgr2=0x7f80ac069fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.499+0000 7f80b3bc7640 1 -- 172.21.15.23:0/4190556376 shutdown_connections 2024-03-24T10:47:47.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:47.499+0000 7f80b3bc7640 1 -- 172.21.15.23:0/4190556376 wait complete. 2024-03-24T10:47:47.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:47 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1497649884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:47:47.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:47 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1497649884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:47:48.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:47 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1497649884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:47:48.126 INFO:tasks.workunit.client.0.smithi023.stdout: 1 760166 10.78 MB/sec execute 245 sec latency 42.948 ms 2024-03-24T10:47:48.511 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:47:48.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:47:48.512 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:47:48.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.628+0000 7faa69eb5640 1 -- 172.21.15.23:0/658845903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa64069ec0 msgr2=0x7faa6406a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.628+0000 7faa69eb5640 1 --2- 172.21.15.23:0/658845903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa64069ec0 0x7faa6406a2a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7faa54009900 tx=0x7faa5402bc30 comp rx=0 tx=0).stop 2024-03-24T10:47:48.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.628+0000 7faa69eb5640 1 -- 172.21.15.23:0/658845903 shutdown_connections 2024-03-24T10:47:48.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.628+0000 7faa69eb5640 1 --2- 172.21.15.23:0/658845903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa64106b10 0x7faa6410e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.628+0000 7faa69eb5640 1 --2- 172.21.15.23:0/658845903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa64106150 0x7faa641065d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.628+0000 7faa69eb5640 1 --2- 172.21.15.23:0/658845903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa64069ec0 0x7faa6406a2a0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.628+0000 7faa69eb5640 1 -- 172.21.15.23:0/658845903 >> 172.21.15.23:0/658845903 conn(0x7faa640776a0 msgr2=0x7faa64077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:48.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.628+0000 7faa69eb5640 1 -- 172.21.15.23:0/658845903 shutdown_connections 2024-03-24T10:47:48.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.628+0000 7faa69eb5640 1 -- 172.21.15.23:0/658845903 wait complete. 2024-03-24T10:47:48.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.628+0000 7faa69eb5640 1 Processor -- start 2024-03-24T10:47:48.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa69eb5640 1 -- start start 2024-03-24T10:47:48.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa69eb5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa64069ec0 0x7faa64100640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa69eb5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa64106150 0x7faa64100b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa69eb5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa64106b10 0x7faa641010c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa69eb5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7faa641a6090 con 0x7faa64106b10 2024-03-24T10:47:48.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa69eb5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faa641a6200 con 0x7faa64106150 2024-03-24T10:47:48.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa69eb5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7faa641a63c0 con 0x7faa64069ec0 2024-03-24T10:47:48.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa63fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa64106b10 0x7faa641010c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa63fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa64106b10 0x7faa641010c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52886/0 (socket says 172.21.15.23:52886) 2024-03-24T10:47:48.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa63fff640 1 -- 172.21.15.23:0/854542885 learned_addr learned my addr 172.21.15.23:0/854542885 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:48.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa637fe640 1 --2- 172.21.15.23:0/854542885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa64069ec0 0x7faa64100640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa62ffd640 1 --2- 172.21.15.23:0/854542885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa64106150 0x7faa64100b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa63fff640 1 -- 172.21.15.23:0/854542885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa64069ec0 msgr2=0x7faa64100640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa63fff640 1 --2- 172.21.15.23:0/854542885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa64069ec0 0x7faa64100640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa63fff640 1 -- 172.21.15.23:0/854542885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa64106150 msgr2=0x7faa64100b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa63fff640 1 --2- 172.21.15.23:0/854542885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa64106150 0x7faa64100b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa63fff640 1 -- 172.21.15.23:0/854542885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa54009560 con 0x7faa64106b10 2024-03-24T10:47:48.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa63fff640 1 --2- 172.21.15.23:0/854542885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa64106b10 0x7faa641010c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7faa5c00eb10 tx=0x7faa5c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:48.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.629+0000 7faa60ff9640 1 -- 172.21.15.23:0/854542885 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c004b20 con 0x7faa64106b10 2024-03-24T10:47:48.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.630+0000 7faa69eb5640 1 -- 172.21.15.23:0/854542885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa641a66a0 con 0x7faa64106b10 2024-03-24T10:47:48.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.630+0000 7faa60ff9640 1 -- 172.21.15.23:0/854542885 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa5c004c80 con 0x7faa64106b10 2024-03-24T10:47:48.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.630+0000 7faa60ff9640 1 -- 172.21.15.23:0/854542885 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c010650 con 0x7faa64106b10 2024-03-24T10:47:48.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.630+0000 7faa69eb5640 1 -- 172.21.15.23:0/854542885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa641a6bb0 con 0x7faa64106b10 2024-03-24T10:47:48.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.632+0000 7faa6cb72740 1 Processor -- start 2024-03-24T10:47:48.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa60ff9640 1 -- 172.21.15.23:0/854542885 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7faa5c010810 con 0x7faa64106b10 2024-03-24T10:47:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa6cb72740 1 -- start start 2024-03-24T10:47:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa6cb72740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930c760 0x5578d930cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa6cb72740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930d080 0x5578d9315720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa6cb72740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d9318050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa6cb72740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5578d927a530 con 0x5578d9315c60 2024-03-24T10:47:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa6cb72740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5578d9277a60 con 0x5578d930c760 2024-03-24T10:47:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa6cb72740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5578d9273240 con 0x5578d930d080 2024-03-24T10:47:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa637fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930d080 0x5578d9315720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa62ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930c760 0x5578d930cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa637fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930d080 0x5578d9315720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:44192/0 (socket says 172.21.15.23:44192) 2024-03-24T10:47:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa637fe640 1 -- 172.21.15.23:0/1753604088 learned_addr learned my addr 172.21.15.23:0/1753604088 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa60ff9640 1 --2- 172.21.15.23:0/854542885 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7faa4c0a2230 0x7faa4c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa62ffd640 1 --2- 172.21.15.23:0/1753604088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d9318050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa63fff640 1 --2- 172.21.15.23:0/854542885 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7faa4c0a2230 0x7faa4c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa60ff9640 1 -- 172.21.15.23:0/854542885 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7faa5c014070 con 0x7faa64106b10 2024-03-24T10:47:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa637fe640 1 -- 172.21.15.23:0/1753604088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930c760 msgr2=0x5578d930cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa637fe640 1 --2- 172.21.15.23:0/1753604088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930c760 0x5578d930cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa637fe640 1 -- 172.21.15.23:0/1753604088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 msgr2=0x5578d9318050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa637fe640 1 --2- 172.21.15.23:0/1753604088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d9318050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa637fe640 1 -- 172.21.15.23:0/1753604088 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578d92718c0 con 0x5578d930d080 2024-03-24T10:47:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.633+0000 7faa62ffd640 1 --2- 172.21.15.23:0/1753604088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d9318050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:47:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.634+0000 7faa637fe640 1 --2- 172.21.15.23:0/1753604088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930d080 0x5578d9315720 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faa54009900 tx=0x7faa5402c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c37c7eea36f1c6a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.634+0000 7faa63fff640 1 --2- 172.21.15.23:0/854542885 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7faa4c0a2230 0x7faa4c0a46f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faa5c09acd0 tx=0x7faa5c013b60 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.634+0000 7faa49ffb640 1 -- 172.21.15.23:0/1753604088 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa540047c0 con 0x5578d930d080 2024-03-24T10:47:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.634+0000 7faa49ffb640 1 -- 172.21.15.23:0/1753604088 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa54004920 con 0x5578d930d080 2024-03-24T10:47:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.634+0000 7faa49ffb640 1 -- 172.21.15.23:0/1753604088 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa54005130 con 0x5578d930d080 2024-03-24T10:47:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.635+0000 7faa6cb72740 1 -- 172.21.15.23:0/1753604088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930d080 msgr2=0x5578d9315720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.635+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1753604088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930d080 0x5578d9315720 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faa54009900 tx=0x7faa5402c6a0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.635+0000 7faa6cb72740 1 -- 172.21.15.23:0/1753604088 shutdown_connections 2024-03-24T10:47:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.635+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1753604088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d9318050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.635+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1753604088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930d080 0x5578d9315720 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.635+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1753604088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930c760 0x5578d930cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.635+0000 7faa6cb72740 1 -- 172.21.15.23:0/1753604088 >> 172.21.15.23:0/1753604088 conn(0x5578d9306e90 msgr2=0x5578d9307270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.635+0000 7faa6cb72740 1 -- 172.21.15.23:0/1753604088 shutdown_connections 2024-03-24T10:47:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.635+0000 7faa6cb72740 1 -- 172.21.15.23:0/1753604088 wait complete. 2024-03-24T10:47:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.635+0000 7faa6cb72740 1 -- start start 2024-03-24T10:47:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.635+0000 7faa6cb72740 1 Processor -- start 2024-03-24T10:47:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa6cb72740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930c760 0x5578d9312c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa6cb72740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930d080 0x5578d93131b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa6cb72740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d93136f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa6cb72740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5578d9217f80 con 0x5578d9315c60 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa6cb72740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5578d8f464b0 con 0x5578d930c760 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa6cb72740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5578d91f8ad0 con 0x5578d930d080 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa6cb72740 10 client.0 _ll_register_callbacks cb 0x5578d922bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa62ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d93136f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa637fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930d080 0x5578d93131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa62ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d93136f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52900/0 (socket says 172.21.15.23:52900) 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa62ffd640 1 -- 172.21.15.23:0/1393612598 learned_addr learned my addr 172.21.15.23:0/1393612598 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa637fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930d080 0x5578d93131b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:44198/0 (socket says 172.21.15.23:44198) 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa62ffd640 1 --2- 172.21.15.23:0/1393612598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930c760 0x5578d9312c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa637fe640 1 -- 172.21.15.23:0/1393612598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930c760 msgr2=0x5578d9312c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa637fe640 1 --2- 172.21.15.23:0/1393612598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930c760 0x5578d9312c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa637fe640 1 -- 172.21.15.23:0/1393612598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 msgr2=0x5578d93136f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa637fe640 1 --2- 172.21.15.23:0/1393612598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d93136f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa637fe640 1 -- 172.21.15.23:0/1393612598 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa6410ede0 con 0x5578d930d080 2024-03-24T10:47:48.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa637fe640 1 --2- 172.21.15.23:0/1393612598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930d080 0x5578d93131b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7faa54009900 tx=0x7faa5402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b07cc963df232521 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.636+0000 7faa42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:47:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa42ffd640 1 -- 172.21.15.23:0/1393612598 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa540047c0 con 0x5578d930d080 2024-03-24T10:47:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa42ffd640 1 -- 172.21.15.23:0/1393612598 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7faa5404d070 con 0x5578d930d080 2024-03-24T10:47:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa6cb72740 1 -- 172.21.15.23:0/1393612598 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5578d92977a0 con 0x5578d930d080 2024-03-24T10:47:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa42ffd640 1 -- 172.21.15.23:0/1393612598 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faa54004390 con 0x5578d930d080 2024-03-24T10:47:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa6cb72740 10 client.27670 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa6cb72740 1 -- 172.21.15.23:0/1393612598 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5578d92296d0 con 0x5578d930d080 2024-03-24T10:47:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa42ffd640 1 -- 172.21.15.23:0/1393612598 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5578d92977a0 con 0x5578d930d080 2024-03-24T10:47:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa42ffd640 1 -- 172.21.15.23:0/1393612598 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7faa5403c050 con 0x5578d930d080 2024-03-24T10:47:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa6cb72740 10 client.27670 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa6cb72740 10 client.27670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa6cb72740 20 client.27670 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1393612598 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5578d93259b0 0x5578d9327dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa6cb72740 4 client.27670 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa6cb72740 1 -- 172.21.15.23:0/1393612598 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578d92691f0 con 0x5578d93259b0 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.637+0000 7faa62ffd640 1 --2- 172.21.15.23:0/1393612598 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5578d93259b0 0x5578d9327dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.638+0000 7faa62ffd640 1 --2- 172.21.15.23:0/1393612598 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5578d93259b0 0x5578d9327dd0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac7318e8bf06eac9 server_cookie=163fdeab1d6836e7 in_seq=0 out_seq=0 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.638+0000 7faa42ffd640 10 client.27670 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa42ffd640 1 -- 172.21.15.23:0/1393612598 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5578d92691f0 con 0x5578d93259b0 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa42ffd640 10 client.27670 handle_command_reply: tid=0 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa6cb72740 1 client.27670 shutdown 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa6cb72740 1 -- 172.21.15.23:0/1393612598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930d080 msgr2=0x5578d93131b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1393612598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930d080 0x5578d93131b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7faa54009900 tx=0x7faa5402bc30 comp rx=0 tx=0).stop 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa6cb72740 1 -- 172.21.15.23:0/1393612598 shutdown_connections 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1393612598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d93136f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1393612598 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5578d93259b0 0x5578d9327dd0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1393612598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930d080 0x5578d93131b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1393612598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930c760 0x5578d9312c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa42ffd640 0 client.27670 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:48.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa6cb72740 1 -- 172.21.15.23:0/1393612598 >> 172.21.15.23:0/1393612598 conn(0x5578d9306e90 msgr2=0x5578d9308be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:48.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa6cb72740 1 -- 172.21.15.23:0/1393612598 shutdown_connections 2024-03-24T10:47:48.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa6cb72740 1 -- 172.21.15.23:0/1393612598 wait complete. 2024-03-24T10:47:48.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.639+0000 7faa6cb72740 20 client.27670 trim_cache size 0 max 16384 2024-03-24T10:47:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.658+0000 7faa6cb72740 1 Processor -- start 2024-03-24T10:47:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.658+0000 7faa6cb72740 1 -- start start 2024-03-24T10:47:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.658+0000 7faa6cb72740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930c760 0x5578d9321490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa6cb72740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d930d080 0x5578d9313510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa6cb72740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d9315c60 0x5578d932d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa6cb72740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5578d93185b0 con 0x5578d930d080 2024-03-24T10:47:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa6cb72740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5578d930e890 con 0x5578d9315c60 2024-03-24T10:47:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa6cb72740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5578d9307e90 con 0x5578d930c760 2024-03-24T10:47:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa637fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d930d080 0x5578d9313510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa637fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d930d080 0x5578d9313510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52916/0 (socket says 172.21.15.23:52916) 2024-03-24T10:47:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa637fe640 1 -- 172.21.15.23:0/1423035464 learned_addr learned my addr 172.21.15.23:0/1423035464 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa62ffd640 1 --2- 172.21.15.23:0/1423035464 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930c760 0x5578d9321490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa62ffd640 1 --2- 172.21.15.23:0/1423035464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d9315c60 0x5578d932d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa637fe640 1 -- 172.21.15.23:0/1423035464 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930c760 msgr2=0x5578d9321490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa637fe640 1 --2- 172.21.15.23:0/1423035464 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930c760 0x5578d9321490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa637fe640 1 -- 172.21.15.23:0/1423035464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d9315c60 msgr2=0x5578d932d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa637fe640 1 --2- 172.21.15.23:0/1423035464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d9315c60 0x5578d932d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa637fe640 1 -- 172.21.15.23:0/1423035464 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578d920dc60 con 0x5578d930d080 2024-03-24T10:47:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa62ffd640 1 --2- 172.21.15.23:0/1423035464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d9315c60 0x5578d932d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:47:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa637fe640 1 --2- 172.21.15.23:0/1423035464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d930d080 0x5578d9313510 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7faa540058c0 tx=0x7faa54004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a414a8cb805ba2d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa497fa640 1 -- 172.21.15.23:0/1423035464 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa54043070 con 0x5578d930d080 2024-03-24T10:47:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa497fa640 1 -- 172.21.15.23:0/1423035464 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa54004a90 con 0x5578d930d080 2024-03-24T10:47:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.659+0000 7faa497fa640 1 -- 172.21.15.23:0/1423035464 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa54005670 con 0x5578d930d080 2024-03-24T10:47:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.660+0000 7faa6cb72740 1 -- 172.21.15.23:0/1423035464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d930d080 msgr2=0x5578d9313510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.660+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1423035464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d930d080 0x5578d9313510 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7faa540058c0 tx=0x7faa54004df0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.660+0000 7faa6cb72740 1 -- 172.21.15.23:0/1423035464 shutdown_connections 2024-03-24T10:47:48.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.660+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1423035464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d9315c60 0x5578d932d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.660+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1423035464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d930d080 0x5578d9313510 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.660+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1423035464 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930c760 0x5578d9321490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.660+0000 7faa6cb72740 1 -- 172.21.15.23:0/1423035464 >> 172.21.15.23:0/1423035464 conn(0x5578d9306e90 msgr2=0x5578d9305ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:48.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.660+0000 7faa6cb72740 1 -- 172.21.15.23:0/1423035464 shutdown_connections 2024-03-24T10:47:48.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.660+0000 7faa6cb72740 1 -- 172.21.15.23:0/1423035464 wait complete. 2024-03-24T10:47:48.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.661+0000 7faa6cb72740 1 -- start start 2024-03-24T10:47:48.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.661+0000 7faa6cb72740 1 Processor -- start 2024-03-24T10:47:48.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.661+0000 7faa6cb72740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930c760 0x5578d930f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.661+0000 7faa6cb72740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930d080 0x5578d930fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.661+0000 7faa6cb72740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d9310400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.661+0000 7faa6cb72740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5578d938ad40 con 0x5578d9315c60 2024-03-24T10:47:48.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.661+0000 7faa6cb72740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5578d938aeb0 con 0x5578d930d080 2024-03-24T10:47:48.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.661+0000 7faa6cb72740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5578d938b070 con 0x5578d930c760 2024-03-24T10:47:48.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.661+0000 7faa6cb72740 10 client.0 _ll_register_callbacks cb 0x7faa5402cb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:48.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.661+0000 7faa62ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d9310400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.661+0000 7faa637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930d080 0x5578d930fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.661+0000 7faa62ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d9310400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52926/0 (socket says 172.21.15.23:52926) 2024-03-24T10:47:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.661+0000 7faa62ffd640 1 -- 172.21.15.23:0/1486496987 learned_addr learned my addr 172.21.15.23:0/1486496987 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.661+0000 7faa62ffd640 1 --2- 172.21.15.23:0/1486496987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930c760 0x5578d930f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.662+0000 7faa637fe640 1 -- 172.21.15.23:0/1486496987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930c760 msgr2=0x5578d930f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.662+0000 7faa637fe640 1 --2- 172.21.15.23:0/1486496987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930c760 0x5578d930f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.662+0000 7faa637fe640 1 -- 172.21.15.23:0/1486496987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 msgr2=0x5578d9310400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.662+0000 7faa637fe640 1 --2- 172.21.15.23:0/1486496987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d9310400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.662+0000 7faa637fe640 1 -- 172.21.15.23:0/1486496987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578d92296d0 con 0x5578d930d080 2024-03-24T10:47:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.662+0000 7faa62ffd640 1 --2- 172.21.15.23:0/1486496987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d9310400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.662+0000 7faa637fe640 1 --2- 172.21.15.23:0/1486496987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930d080 0x5578d930fec0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faa54009900 tx=0x7faa5402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92aee37064f2b206 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.662+0000 7faa42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:47:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.662+0000 7faa42ffd640 1 -- 172.21.15.23:0/1486496987 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa54043070 con 0x5578d930d080 2024-03-24T10:47:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.662+0000 7faa42ffd640 1 -- 172.21.15.23:0/1486496987 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7faa5404d040 con 0x5578d930d080 2024-03-24T10:47:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.662+0000 7faa6cb72740 1 -- 172.21.15.23:0/1486496987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa54004920 con 0x5578d930d080 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.662+0000 7faa42ffd640 1 -- 172.21.15.23:0/1486496987 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faa5403fc60 con 0x5578d930d080 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.662+0000 7faa6cb72740 10 client.27707 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.662+0000 7faa6cb72740 1 -- 172.21.15.23:0/1486496987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5578d9233970 con 0x5578d930d080 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.663+0000 7faa42ffd640 1 -- 172.21.15.23:0/1486496987 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa5406f080 con 0x5578d930d080 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.663+0000 7faa42ffd640 1 -- 172.21.15.23:0/1486496987 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7faa5403c040 con 0x5578d930d080 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.663+0000 7faa6cb72740 10 client.27707 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.663+0000 7faa6cb72740 10 client.27707 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.663+0000 7faa6cb72740 20 client.27707 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.663+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1486496987 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5578d938c490 0x5578d938e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.663+0000 7faa6cb72740 4 client.27707 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.663+0000 7faa6cb72740 1 -- 172.21.15.23:0/1486496987 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5578d8f72860 con 0x5578d938c490 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.663+0000 7faa62ffd640 1 --2- 172.21.15.23:0/1486496987 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5578d938c490 0x5578d938e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.663+0000 7faa42ffd640 10 client.27707 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.663+0000 7faa62ffd640 1 --2- 172.21.15.23:0/1486496987 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5578d938c490 0x5578d938e880 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a12f6e24dd153693 server_cookie=13274dbba623bfff in_seq=0 out_seq=0 2024-03-24T10:47:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.664+0000 7faa42ffd640 1 -- 172.21.15.23:0/1486496987 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5578d8f72860 con 0x5578d938c490 2024-03-24T10:47:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.664+0000 7faa42ffd640 10 client.27707 handle_command_reply: tid=0 2024-03-24T10:47:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.664+0000 7faa6cb72740 1 client.27707 shutdown 2024-03-24T10:47:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa6cb72740 1 -- 172.21.15.23:0/1486496987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930d080 msgr2=0x5578d930fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1486496987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930d080 0x5578d930fec0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faa54009900 tx=0x7faa5402bc30 comp rx=0 tx=0).stop 2024-03-24T10:47:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa6cb72740 1 -- 172.21.15.23:0/1486496987 shutdown_connections 2024-03-24T10:47:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1486496987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578d9315c60 0x5578d9310400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1486496987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5578d930d080 0x5578d930fec0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1486496987 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5578d938c490 0x5578d938e880 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa6cb72740 1 --2- 172.21.15.23:0/1486496987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5578d930c760 0x5578d930f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa42ffd640 0 client.27707 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa6cb72740 1 -- 172.21.15.23:0/1486496987 >> 172.21.15.23:0/1486496987 conn(0x5578d9306e90 msgr2=0x5578d9307e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa6cb72740 1 -- 172.21.15.23:0/1486496987 shutdown_connections 2024-03-24T10:47:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa6cb72740 1 -- 172.21.15.23:0/1486496987 wait complete. 2024-03-24T10:47:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa6cb72740 20 client.27707 trim_cache size 0 max 16384 2024-03-24T10:47:48.671 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:47:48.671 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:47:48.671 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "a855b1ad-41b5-44bd-b6d6-3573326e39c8", 2024-03-24T10:47:48.671 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:47:48.671 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:47:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa69eb5640 1 -- 172.21.15.23:0/854542885 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7faa4c0a2230 msgr2=0x7faa4c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa69eb5640 1 --2- 172.21.15.23:0/854542885 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7faa4c0a2230 0x7faa4c0a46f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faa5c09acd0 tx=0x7faa5c013b60 comp rx=0 tx=0).stop 2024-03-24T10:47:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa69eb5640 1 -- 172.21.15.23:0/854542885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa64106b10 msgr2=0x7faa641010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa69eb5640 1 --2- 172.21.15.23:0/854542885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa64106b10 0x7faa641010c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7faa5c00eb10 tx=0x7faa5c00c2d0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa69eb5640 1 -- 172.21.15.23:0/854542885 shutdown_connections 2024-03-24T10:47:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa69eb5640 1 --2- 172.21.15.23:0/854542885 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7faa4c0a2230 0x7faa4c0a46f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa69eb5640 1 --2- 172.21.15.23:0/854542885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa64106b10 0x7faa641010c0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa69eb5640 1 --2- 172.21.15.23:0/854542885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa64106150 0x7faa64100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa69eb5640 1 --2- 172.21.15.23:0/854542885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa64069ec0 0x7faa64100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.665+0000 7faa69eb5640 1 -- 172.21.15.23:0/854542885 >> 172.21.15.23:0/854542885 conn(0x7faa640776a0 msgr2=0x7faa640fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.666+0000 7faa69eb5640 1 -- 172.21.15.23:0/854542885 shutdown_connections 2024-03-24T10:47:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.666+0000 7faa69eb5640 1 -- 172.21.15.23:0/854542885 wait complete. 2024-03-24T10:47:48.677 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:47:48.677 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:47:48.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:48 smithi116 ceph-mon[36248]: pgmap v238: 97 pgs: 97 active+clean; 445 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 256 op/s 2024-03-24T10:47:48.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.788+0000 7fbcad27c640 1 -- 172.21.15.23:0/3051789055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca8102040 msgr2=0x7fbca810e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.788+0000 7fbcad27c640 1 --2- 172.21.15.23:0/3051789055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca8102040 0x7fbca810e550 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fbca0009cd0 tx=0x7fbca002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.788+0000 7fbcad27c640 1 -- 172.21.15.23:0/3051789055 shutdown_connections 2024-03-24T10:47:48.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.788+0000 7fbcad27c640 1 --2- 172.21.15.23:0/3051789055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca8102040 0x7fbca810e550 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.788+0000 7fbcad27c640 1 --2- 172.21.15.23:0/3051789055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbca8101680 0x7fbca8101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.788+0000 7fbcad27c640 1 --2- 172.21.15.23:0/3051789055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbca8107690 0x7fbca8107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.788+0000 7fbcad27c640 1 -- 172.21.15.23:0/3051789055 >> 172.21.15.23:0/3051789055 conn(0x7fbca80fd170 msgr2=0x7fbca80ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:48.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.788+0000 7fbcad27c640 1 -- 172.21.15.23:0/3051789055 shutdown_connections 2024-03-24T10:47:48.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.788+0000 7fbcad27c640 1 -- 172.21.15.23:0/3051789055 wait complete. 2024-03-24T10:47:48.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.789+0000 7fbcad27c640 1 Processor -- start 2024-03-24T10:47:48.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.789+0000 7fbcad27c640 1 -- start start 2024-03-24T10:47:48.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.789+0000 7fbcad27c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbca8101680 0x7fbca819da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.789+0000 7fbcad27c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbca8102040 0x7fbca819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.789+0000 7fbcad27c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca8107690 0x7fbca8197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.789+0000 7fbcad27c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fbca8198310 con 0x7fbca8107690 2024-03-24T10:47:48.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.789+0000 7fbcad27c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbca8198480 con 0x7fbca8101680 2024-03-24T10:47:48.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.789+0000 7fbcad27c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbca8198640 con 0x7fbca8102040 2024-03-24T10:47:48.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.789+0000 7fbca77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca8107690 0x7fbca8197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.789+0000 7fbca77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca8107690 0x7fbca8197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52940/0 (socket says 172.21.15.23:52940) 2024-03-24T10:47:48.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.789+0000 7fbca77fe640 1 -- 172.21.15.23:0/11888461 learned_addr learned my addr 172.21.15.23:0/11888461 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:48.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.789+0000 7fbca6ffd640 1 --2- 172.21.15.23:0/11888461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbca8101680 0x7fbca819da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.789+0000 7fbc9ffff640 1 --2- 172.21.15.23:0/11888461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbca8102040 0x7fbca819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.790+0000 7fbca77fe640 1 -- 172.21.15.23:0/11888461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbca8102040 msgr2=0x7fbca819dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.790+0000 7fbca77fe640 1 --2- 172.21.15.23:0/11888461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbca8102040 0x7fbca819dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.790+0000 7fbca77fe640 1 -- 172.21.15.23:0/11888461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbca8101680 msgr2=0x7fbca819da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.790+0000 7fbca77fe640 1 --2- 172.21.15.23:0/11888461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbca8101680 0x7fbca819da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.790+0000 7fbca77fe640 1 -- 172.21.15.23:0/11888461 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc98009490 con 0x7fbca8107690 2024-03-24T10:47:48.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.790+0000 7fbca77fe640 1 --2- 172.21.15.23:0/11888461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca8107690 0x7fbca8197b80 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fbca0004af0 tx=0x7fbca0032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:48.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.790+0000 7fbca4ff9640 1 -- 172.21.15.23:0/11888461 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca00328b0 con 0x7fbca8107690 2024-03-24T10:47:48.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.790+0000 7fbca4ff9640 1 -- 172.21.15.23:0/11888461 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbca0003e50 con 0x7fbca8107690 2024-03-24T10:47:48.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.790+0000 7fbcad27c640 1 -- 172.21.15.23:0/11888461 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbca0009900 con 0x7fbca8107690 2024-03-24T10:47:48.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.790+0000 7fbca4ff9640 1 -- 172.21.15.23:0/11888461 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca0004740 con 0x7fbca8107690 2024-03-24T10:47:48.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.790+0000 7fbcad27c640 1 -- 172.21.15.23:0/11888461 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbca81a4810 con 0x7fbca8107690 2024-03-24T10:47:48.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.792+0000 7fbca4ff9640 1 -- 172.21.15.23:0/11888461 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fbca000a040 con 0x7fbca8107690 2024-03-24T10:47:48.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbcafe57740 1 Processor -- start 2024-03-24T10:47:48.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbcafe57740 1 -- start start 2024-03-24T10:47:48.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbcafe57740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737471d0 0x55af737475b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbcafe57740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af73747af0 0x55af73750190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbcafe57740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af737506d0 0x55af73752ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbcafe57740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55af7366bf30 con 0x55af737506d0 2024-03-24T10:47:48.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbcafe57740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55af736b5810 con 0x55af73747af0 2024-03-24T10:47:48.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbcafe57740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55af736b2d40 con 0x55af737471d0 2024-03-24T10:47:48.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbca4ff9640 1 --2- 172.21.15.23:0/11888461 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbc940a2230 0x7fbc940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbc9ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737471d0 0x55af737475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbc9ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737471d0 0x55af737475b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:44234/0 (socket says 172.21.15.23:44234) 2024-03-24T10:47:48.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbc9ffff640 1 -- 172.21.15.23:0/2691631600 learned_addr learned my addr 172.21.15.23:0/2691631600 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:48.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbc9ffff640 1 --2- 172.21.15.23:0/2691631600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af737506d0 0x55af73752ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbca77fe640 1 --2- 172.21.15.23:0/11888461 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbc940a2230 0x7fbc940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbca6ffd640 1 --2- 172.21.15.23:0/2691631600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af73747af0 0x55af73750190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.793+0000 7fbca4ff9640 1 -- 172.21.15.23:0/11888461 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7fbca00f0780 con 0x7fbca8107690 2024-03-24T10:47:48.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.794+0000 7fbc9ffff640 1 -- 172.21.15.23:0/2691631600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af73747af0 msgr2=0x55af73750190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.794+0000 7fbc9ffff640 1 --2- 172.21.15.23:0/2691631600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af73747af0 0x55af73750190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.794+0000 7fbc9ffff640 1 -- 172.21.15.23:0/2691631600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af737506d0 msgr2=0x55af73752ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.794+0000 7fbc9ffff640 1 --2- 172.21.15.23:0/2691631600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af737506d0 0x55af73752ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.794+0000 7fbc9ffff640 1 -- 172.21.15.23:0/2691631600 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af7366fea0 con 0x55af737471d0 2024-03-24T10:47:48.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.794+0000 7fbc9ffff640 1 --2- 172.21.15.23:0/2691631600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737471d0 0x55af737475b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbc900058c0 tx=0x7fbc90005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=349199d733d07804 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:48.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.794+0000 7fbca77fe640 1 --2- 172.21.15.23:0/11888461 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbc940a2230 0x7fbc940a46f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbca00ee090 tx=0x7fbca00b2710 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:48.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.794+0000 7fbc9d7fa640 1 -- 172.21.15.23:0/2691631600 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc90004030 con 0x55af737471d0 2024-03-24T10:47:48.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.794+0000 7fbc9d7fa640 1 -- 172.21.15.23:0/2691631600 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc90005020 con 0x55af737471d0 2024-03-24T10:47:48.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.794+0000 7fbc9d7fa640 1 -- 172.21.15.23:0/2691631600 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc90004d40 con 0x55af737471d0 2024-03-24T10:47:48.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.795+0000 7fbcafe57740 1 -- 172.21.15.23:0/2691631600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737471d0 msgr2=0x55af737475b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.795+0000 7fbcafe57740 1 --2- 172.21.15.23:0/2691631600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737471d0 0x55af737475b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbc900058c0 tx=0x7fbc90005850 comp rx=0 tx=0).stop 2024-03-24T10:47:48.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.795+0000 7fbcafe57740 1 -- 172.21.15.23:0/2691631600 shutdown_connections 2024-03-24T10:47:48.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.795+0000 7fbcafe57740 1 --2- 172.21.15.23:0/2691631600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af737506d0 0x55af73752ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.795+0000 7fbcafe57740 1 --2- 172.21.15.23:0/2691631600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af73747af0 0x55af73750190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.795+0000 7fbcafe57740 1 --2- 172.21.15.23:0/2691631600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737471d0 0x55af737475b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.795+0000 7fbcafe57740 1 -- 172.21.15.23:0/2691631600 >> 172.21.15.23:0/2691631600 conn(0x55af73741900 msgr2=0x55af73741ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:48.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.795+0000 7fbcafe57740 1 -- 172.21.15.23:0/2691631600 shutdown_connections 2024-03-24T10:47:48.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.795+0000 7fbcafe57740 1 -- 172.21.15.23:0/2691631600 wait complete. 2024-03-24T10:47:48.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.795+0000 7fbcafe57740 1 -- start start 2024-03-24T10:47:48.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.795+0000 7fbcafe57740 1 Processor -- start 2024-03-24T10:47:48.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbcafe57740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737471d0 0x55af7374d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbcafe57740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af73747af0 0x55af7374dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbcafe57740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737506d0 0x55af7374e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbcafe57740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55af736ae520 con 0x55af73747af0 2024-03-24T10:47:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbcafe57740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55af73381430 con 0x55af737471d0 2024-03-24T10:47:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbcafe57740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55af7333de00 con 0x55af737506d0 2024-03-24T10:47:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbcafe57740 10 client.0 _ll_register_callbacks cb 0x55af7364cd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbca6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af73747af0 0x55af7374dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbca6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af73747af0 0x55af7374dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52950/0 (socket says 172.21.15.23:52950) 2024-03-24T10:47:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbca6ffd640 1 -- 172.21.15.23:0/3897040437 learned_addr learned my addr 172.21.15.23:0/3897040437 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbc9ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737471d0 0x55af7374d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbc9ffff640 1 --2- 172.21.15.23:0/3897040437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737506d0 0x55af7374e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbca6ffd640 1 -- 172.21.15.23:0/3897040437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737506d0 msgr2=0x55af7374e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbca6ffd640 1 --2- 172.21.15.23:0/3897040437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737506d0 0x55af7374e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbca6ffd640 1 -- 172.21.15.23:0/3897040437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737471d0 msgr2=0x55af7374d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbca6ffd640 1 --2- 172.21.15.23:0/3897040437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737471d0 0x55af7374d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbca6ffd640 1 -- 172.21.15.23:0/3897040437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc90019040 con 0x55af73747af0 2024-03-24T10:47:48.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbca6ffd640 1 --2- 172.21.15.23:0/3897040437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af73747af0 0x55af7374dcd0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fbc98012810 tx=0x7fbc98012cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e400a0803a66596 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:48.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbc8e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:48.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbc8e7fc640 1 -- 172.21.15.23:0/3897040437 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc98017070 con 0x55af73747af0 2024-03-24T10:47:48.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbc8e7fc640 1 -- 172.21.15.23:0/3897040437 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7fbc98004050 con 0x55af73747af0 2024-03-24T10:47:48.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.796+0000 7fbcafe57740 1 -- 172.21.15.23:0/3897040437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55af733ad7e0 con 0x55af73747af0 2024-03-24T10:47:48.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.797+0000 7fbc8e7fc640 1 -- 172.21.15.23:0/3897040437 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbc98004210 con 0x55af73747af0 2024-03-24T10:47:48.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.797+0000 7fbcafe57740 10 client.17832 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:48.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.797+0000 7fbcafe57740 1 -- 172.21.15.23:0/3897040437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55af736acba0 con 0x55af73747af0 2024-03-24T10:47:48.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.797+0000 7fbc8e7fc640 1 -- 172.21.15.23:0/3897040437 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55af733ad7e0 con 0x55af73747af0 2024-03-24T10:47:48.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.797+0000 7fbc8e7fc640 1 -- 172.21.15.23:0/3897040437 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fbc980039c0 con 0x55af73747af0 2024-03-24T10:47:48.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.797+0000 7fbcafe57740 10 client.17832 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:48.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.797+0000 7fbcafe57740 10 client.17832 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:48.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.797+0000 7fbcafe57740 20 client.17832 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:48.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.797+0000 7fbcafe57740 1 --2- 172.21.15.23:0/3897040437 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55af73760bf0 0x55af73763030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.797+0000 7fbcafe57740 4 client.17832 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:48.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.797+0000 7fbcafe57740 1 -- 172.21.15.23:0/3897040437 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af733ea720 con 0x55af73760bf0 2024-03-24T10:47:48.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.797+0000 7fbc9ffff640 1 --2- 172.21.15.23:0/3897040437 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55af73760bf0 0x55af73763030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.798+0000 7fbc8e7fc640 10 client.17832 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:48.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.798+0000 7fbc9ffff640 1 --2- 172.21.15.23:0/3897040437 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55af73760bf0 0x55af73763030 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2491365960c9e988 server_cookie=c4682ab113f8441e in_seq=0 out_seq=0 2024-03-24T10:47:48.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.798+0000 7fbc8e7fc640 1 -- 172.21.15.23:0/3897040437 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55af733ea720 con 0x55af73760bf0 2024-03-24T10:47:48.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.798+0000 7fbc8e7fc640 10 client.17832 handle_command_reply: tid=0 2024-03-24T10:47:48.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.798+0000 7fbcafe57740 1 client.17832 shutdown 2024-03-24T10:47:48.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.799+0000 7fbcafe57740 1 -- 172.21.15.23:0/3897040437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af73747af0 msgr2=0x55af7374dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.799+0000 7fbcafe57740 1 --2- 172.21.15.23:0/3897040437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af73747af0 0x55af7374dcd0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fbc98012810 tx=0x7fbc98012cd0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.799+0000 7fbcafe57740 1 -- 172.21.15.23:0/3897040437 shutdown_connections 2024-03-24T10:47:48.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.799+0000 7fbcafe57740 1 --2- 172.21.15.23:0/3897040437 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55af73760bf0 0x55af73763030 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.799+0000 7fbcafe57740 1 --2- 172.21.15.23:0/3897040437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737506d0 0x55af7374e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.799+0000 7fbcafe57740 1 --2- 172.21.15.23:0/3897040437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af73747af0 0x55af7374dcd0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.799+0000 7fbc8e7fc640 0 client.17832 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:48.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.799+0000 7fbcafe57740 1 --2- 172.21.15.23:0/3897040437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737471d0 0x55af7374d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.799+0000 7fbcafe57740 1 -- 172.21.15.23:0/3897040437 >> 172.21.15.23:0/3897040437 conn(0x55af73741900 msgr2=0x55af73740b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:48.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.799+0000 7fbcafe57740 1 -- 172.21.15.23:0/3897040437 shutdown_connections 2024-03-24T10:47:48.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.799+0000 7fbcafe57740 1 -- 172.21.15.23:0/3897040437 wait complete. 2024-03-24T10:47:48.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.799+0000 7fbcafe57740 20 client.17832 trim_cache size 0 max 16384 2024-03-24T10:47:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.817+0000 7fbcafe57740 1 Processor -- start 2024-03-24T10:47:48.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.817+0000 7fbcafe57740 1 -- start start 2024-03-24T10:47:48.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.817+0000 7fbcafe57740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af737471d0 0x55af7375bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.817+0000 7fbcafe57740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af73747af0 0x55af7374e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.817+0000 7fbcafe57740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737506d0 0x55af73768940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.817+0000 7fbcafe57740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55af73633430 con 0x55af737471d0 2024-03-24T10:47:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.817+0000 7fbcafe57740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55af73746970 con 0x55af737506d0 2024-03-24T10:47:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.817+0000 7fbcafe57740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55af73741430 con 0x55af73747af0 2024-03-24T10:47:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.817+0000 7fbc9ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af737471d0 0x55af7375bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.817+0000 7fbc9ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af737471d0 0x55af7375bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52962/0 (socket says 172.21.15.23:52962) 2024-03-24T10:47:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.817+0000 7fbc9ffff640 1 -- 172.21.15.23:0/4240489989 learned_addr learned my addr 172.21.15.23:0/4240489989 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.817+0000 7fbca6ffd640 1 --2- 172.21.15.23:0/4240489989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af73747af0 0x55af7374e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.817+0000 7fbc9ffff640 1 --2- 172.21.15.23:0/4240489989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737506d0 0x55af73768940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.818+0000 7fbca6ffd640 1 -- 172.21.15.23:0/4240489989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737506d0 msgr2=0x55af73768940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.818+0000 7fbca6ffd640 1 --2- 172.21.15.23:0/4240489989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737506d0 0x55af73768940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.818+0000 7fbca6ffd640 1 -- 172.21.15.23:0/4240489989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af737471d0 msgr2=0x55af7375bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.818+0000 7fbca6ffd640 1 --2- 172.21.15.23:0/4240489989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af737471d0 0x55af7375bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.818+0000 7fbca6ffd640 1 -- 172.21.15.23:0/4240489989 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af73648b20 con 0x55af73747af0 2024-03-24T10:47:48.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.818+0000 7fbc9ffff640 1 --2- 172.21.15.23:0/4240489989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af737471d0 0x55af7375bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:47:48.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.818+0000 7fbca6ffd640 1 --2- 172.21.15.23:0/4240489989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af73747af0 0x55af7374e030 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbc98003df0 tx=0x7fbc98009400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3736a14aea890cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:48.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.818+0000 7fbc9cff9640 1 -- 172.21.15.23:0/4240489989 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc98017040 con 0x55af73747af0 2024-03-24T10:47:48.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.818+0000 7fbc9cff9640 1 -- 172.21.15.23:0/4240489989 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc980055c0 con 0x55af73747af0 2024-03-24T10:47:48.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.818+0000 7fbc9cff9640 1 -- 172.21.15.23:0/4240489989 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9800fb30 con 0x55af73747af0 2024-03-24T10:47:48.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.819+0000 7fbcafe57740 1 -- 172.21.15.23:0/4240489989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af73747af0 msgr2=0x55af7374e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.819+0000 7fbcafe57740 1 --2- 172.21.15.23:0/4240489989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af73747af0 0x55af7374e030 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbc98003df0 tx=0x7fbc98009400 comp rx=0 tx=0).stop 2024-03-24T10:47:48.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.819+0000 7fbcafe57740 1 -- 172.21.15.23:0/4240489989 shutdown_connections 2024-03-24T10:47:48.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.819+0000 7fbcafe57740 1 --2- 172.21.15.23:0/4240489989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737506d0 0x55af73768940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.819+0000 7fbcafe57740 1 --2- 172.21.15.23:0/4240489989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af73747af0 0x55af7374e030 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.819+0000 7fbcafe57740 1 --2- 172.21.15.23:0/4240489989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af737471d0 0x55af7375bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.819+0000 7fbcafe57740 1 -- 172.21.15.23:0/4240489989 >> 172.21.15.23:0/4240489989 conn(0x55af73741900 msgr2=0x55af73740b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:48.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.819+0000 7fbcafe57740 1 -- 172.21.15.23:0/4240489989 shutdown_connections 2024-03-24T10:47:48.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.819+0000 7fbcafe57740 1 -- 172.21.15.23:0/4240489989 wait complete. 2024-03-24T10:47:48.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.819+0000 7fbcafe57740 1 -- start start 2024-03-24T10:47:48.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.819+0000 7fbcafe57740 1 Processor -- start 2024-03-24T10:47:48.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.819+0000 7fbcafe57740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737471d0 0x55af7374a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.819+0000 7fbcafe57740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af73747af0 0x55af7374aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbcafe57740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737506d0 0x55af737c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbcafe57740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55af7374b5c0 con 0x55af73747af0 2024-03-24T10:47:48.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbcafe57740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55af737c64b0 con 0x55af737506d0 2024-03-24T10:47:48.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbcafe57740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55af737c65f0 con 0x55af737471d0 2024-03-24T10:47:48.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbcafe57740 10 client.0 _ll_register_callbacks cb 0x7fbc98009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:48.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbca6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af73747af0 0x55af7374aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbca6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af73747af0 0x55af7374aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52972/0 (socket says 172.21.15.23:52972) 2024-03-24T10:47:48.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbca6ffd640 1 -- 172.21.15.23:0/2440275741 learned_addr learned my addr 172.21.15.23:0/2440275741 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:48.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbc9ffff640 1 --2- 172.21.15.23:0/2440275741 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737471d0 0x55af7374a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbc9ffff640 1 --2- 172.21.15.23:0/2440275741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737506d0 0x55af737c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbca6ffd640 1 -- 172.21.15.23:0/2440275741 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737471d0 msgr2=0x55af7374a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbca6ffd640 1 --2- 172.21.15.23:0/2440275741 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737471d0 0x55af7374a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbca6ffd640 1 -- 172.21.15.23:0/2440275741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737506d0 msgr2=0x55af737c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbca6ffd640 1 --2- 172.21.15.23:0/2440275741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737506d0 0x55af737c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbca6ffd640 1 -- 172.21.15.23:0/2440275741 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af7366fea0 con 0x55af73747af0 2024-03-24T10:47:48.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbca6ffd640 1 --2- 172.21.15.23:0/2440275741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af73747af0 0x55af7374aef0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009ee0 tx=0x7fbc980038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6a2cee89b2d028 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:48.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbc8e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:48.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbc8e7fc640 1 -- 172.21.15.23:0/2440275741 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc98005460 con 0x55af73747af0 2024-03-24T10:47:48.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbc8e7fc640 1 -- 172.21.15.23:0/2440275741 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7fbc9800f790 con 0x55af73747af0 2024-03-24T10:47:48.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbcafe57740 1 -- 172.21.15.23:0/2440275741 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc90005020 con 0x55af73747af0 2024-03-24T10:47:48.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbc8e7fc640 1 -- 172.21.15.23:0/2440275741 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbc9800f950 con 0x55af73747af0 2024-03-24T10:47:48.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.820+0000 7fbcafe57740 10 client.17838 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:48.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.821+0000 7fbcafe57740 1 -- 172.21.15.23:0/2440275741 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55af736acba0 con 0x55af73747af0 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.821+0000 7fbc8e7fc640 1 -- 172.21.15.23:0/2440275741 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc90005020 con 0x55af73747af0 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.821+0000 7fbc8e7fc640 1 -- 172.21.15.23:0/2440275741 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fbc98010b80 con 0x55af73747af0 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.821+0000 7fbcafe57740 10 client.17838 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.821+0000 7fbcafe57740 10 client.17838 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.821+0000 7fbcafe57740 20 client.17838 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.821+0000 7fbcafe57740 1 --2- 172.21.15.23:0/2440275741 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55af737c7a30 0x55af737c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.821+0000 7fbcafe57740 4 client.17838 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.821+0000 7fbcafe57740 1 -- 172.21.15.23:0/2440275741 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55af73439890 con 0x55af737c7a30 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.821+0000 7fbc9ffff640 1 --2- 172.21.15.23:0/2440275741 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55af737c7a30 0x55af737c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.821+0000 7fbc8e7fc640 10 client.17838 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.821+0000 7fbc9ffff640 1 --2- 172.21.15.23:0/2440275741 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55af737c7a30 0x55af737c9e50 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=878a81cd0e77c2d5 server_cookie=d580726ef1b34c93 in_seq=0 out_seq=0 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbc8e7fc640 1 -- 172.21.15.23:0/2440275741 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55af73439890 con 0x55af737c7a30 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbc8e7fc640 10 client.17838 handle_command_reply: tid=0 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcafe57740 1 client.17838 shutdown 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcafe57740 1 -- 172.21.15.23:0/2440275741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af73747af0 msgr2=0x55af7374aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcafe57740 1 --2- 172.21.15.23:0/2440275741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af73747af0 0x55af7374aef0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009ee0 tx=0x7fbc980038a0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcafe57740 1 -- 172.21.15.23:0/2440275741 shutdown_connections 2024-03-24T10:47:48.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcafe57740 1 --2- 172.21.15.23:0/2440275741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af737506d0 0x55af737c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcafe57740 1 --2- 172.21.15.23:0/2440275741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af73747af0 0x55af7374aef0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcafe57740 1 --2- 172.21.15.23:0/2440275741 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55af737c7a30 0x55af737c9e50 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcafe57740 1 --2- 172.21.15.23:0/2440275741 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af737471d0 0x55af7374a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbc8e7fc640 0 client.17838 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:48.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcafe57740 1 -- 172.21.15.23:0/2440275741 >> 172.21.15.23:0/2440275741 conn(0x55af73741900 msgr2=0x55af73742c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:48.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcafe57740 1 -- 172.21.15.23:0/2440275741 shutdown_connections 2024-03-24T10:47:48.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcafe57740 1 -- 172.21.15.23:0/2440275741 wait complete. 2024-03-24T10:47:48.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcafe57740 20 client.17838 trim_cache size 0 max 16384 2024-03-24T10:47:48.828 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:47:48.828 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:47:48.829 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:47:48.829 INFO:teuthology.orchestra.run.smithi023.stdout: "a855b1ad-41b5-44bd-b6d6-3573326e39c8": { 2024-03-24T10:47:48.829 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:47:48.829 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "a855b1ad-41b5-44bd-b6d6-3573326e39c8", 2024-03-24T10:47:48.829 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:47:48.829 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:47:48.829 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:47:48.829 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:47:48.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcad27c640 1 -- 172.21.15.23:0/11888461 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbc940a2230 msgr2=0x7fbc940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcad27c640 1 --2- 172.21.15.23:0/11888461 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbc940a2230 0x7fbc940a46f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbca00ee090 tx=0x7fbca00b2710 comp rx=0 tx=0).stop 2024-03-24T10:47:48.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcad27c640 1 -- 172.21.15.23:0/11888461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca8107690 msgr2=0x7fbca8197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcad27c640 1 --2- 172.21.15.23:0/11888461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca8107690 0x7fbca8197b80 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fbca0004af0 tx=0x7fbca0032570 comp rx=0 tx=0).stop 2024-03-24T10:47:48.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcad27c640 1 -- 172.21.15.23:0/11888461 shutdown_connections 2024-03-24T10:47:48.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcad27c640 1 --2- 172.21.15.23:0/11888461 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fbc940a2230 0x7fbc940a46f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcad27c640 1 --2- 172.21.15.23:0/11888461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca8107690 0x7fbca8197b80 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcad27c640 1 --2- 172.21.15.23:0/11888461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbca8102040 0x7fbca819dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcad27c640 1 --2- 172.21.15.23:0/11888461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbca8101680 0x7fbca819da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.822+0000 7fbcad27c640 1 -- 172.21.15.23:0/11888461 >> 172.21.15.23:0/11888461 conn(0x7fbca80fd170 msgr2=0x7fbca80fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:48.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.823+0000 7fbcad27c640 1 -- 172.21.15.23:0/11888461 shutdown_connections 2024-03-24T10:47:48.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:47:48.823+0000 7fbcad27c640 1 -- 172.21.15.23:0/11888461 wait complete. 2024-03-24T10:47:48.832 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:48 smithi023 ceph-mon[32466]: pgmap v238: 97 pgs: 97 active+clean; 445 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 256 op/s 2024-03-24T10:47:48.833 INFO:tasks.cephfs.filesystem:scrub status for tag:a855b1ad-41b5-44bd-b6d6-3573326e39c8 - {'path': '/', 'tag': 'a855b1ad-41b5-44bd-b6d6-3573326e39c8', 'options': 'recursive,force'} 2024-03-24T10:47:49.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:48 smithi117 ceph-mon[36368]: pgmap v238: 97 pgs: 97 active+clean; 445 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 256 op/s 2024-03-24T10:47:49.127 INFO:tasks.workunit.client.0.smithi023.stdout: 1 761525 10.76 MB/sec execute 246 sec latency 28.592 ms 2024-03-24T10:47:49.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:49 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:47:49.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:49 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:49.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:49 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:47:49.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:49 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:47:49.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:49 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:49.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:49 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:49.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:49 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:47:49.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:49 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:49.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:49 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:47:49.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:49 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:47:49.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:49 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:49.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:49 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:50.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:49 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:47:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:49 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:49 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:47:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:49 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:47:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:49 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:50.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:49 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:47:50.127 INFO:tasks.workunit.client.0.smithi023.stdout: 1 763720 10.77 MB/sec execute 247 sec latency 29.509 ms 2024-03-24T10:47:50.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:50 smithi116 ceph-mon[36248]: pgmap v239: 97 pgs: 97 active+clean; 445 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 169 op/s 2024-03-24T10:47:50.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:50 smithi023 ceph-mon[32466]: pgmap v239: 97 pgs: 97 active+clean; 445 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 169 op/s 2024-03-24T10:47:51.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:50 smithi117 ceph-mon[36368]: pgmap v239: 97 pgs: 97 active+clean; 445 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 169 op/s 2024-03-24T10:47:51.127 INFO:tasks.workunit.client.0.smithi023.stdout: 1 765058 10.75 MB/sec execute 248 sec latency 29.687 ms 2024-03-24T10:47:52.127 INFO:tasks.workunit.client.0.smithi023.stdout: 1 767170 10.76 MB/sec execute 249 sec latency 109.485 ms 2024-03-24T10:47:52.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:52 smithi023 ceph-mon[32466]: pgmap v240: 97 pgs: 97 active+clean; 453 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 258 op/s 2024-03-24T10:47:53.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:52 smithi117 ceph-mon[36368]: pgmap v240: 97 pgs: 97 active+clean; 453 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 258 op/s 2024-03-24T10:47:53.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:52 smithi116 ceph-mon[36248]: pgmap v240: 97 pgs: 97 active+clean; 453 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 258 op/s 2024-03-24T10:47:53.127 INFO:tasks.workunit.client.0.smithi023.stdout: 1 768566 10.74 MB/sec execute 250 sec latency 35.977 ms 2024-03-24T10:47:54.127 INFO:tasks.workunit.client.0.smithi023.stdout: 1 770882 10.75 MB/sec execute 251 sec latency 17.615 ms 2024-03-24T10:47:54.819 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:54 smithi116 ceph-mon[36248]: pgmap v241: 97 pgs: 97 active+clean; 453 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 175 op/s 2024-03-24T10:47:54.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:54 smithi023 ceph-mon[32466]: pgmap v241: 97 pgs: 97 active+clean; 453 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 175 op/s 2024-03-24T10:47:55.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:54 smithi117 ceph-mon[36368]: pgmap v241: 97 pgs: 97 active+clean; 453 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 175 op/s 2024-03-24T10:47:55.127 INFO:tasks.workunit.client.0.smithi023.stdout: 1 772414 10.73 MB/sec execute 252 sec latency 22.868 ms 2024-03-24T10:47:55.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:55 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:47:55.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:55 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:47:56.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:55 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:47:56.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:55 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:47:56.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:55 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:47:56.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:55 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:47:56.127 INFO:tasks.workunit.client.0.smithi023.stdout: 1 774612 10.74 MB/sec execute 253 sec latency 16.353 ms 2024-03-24T10:47:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.512+0000 7f8d4a5a0640 1 -- 172.21.15.23:0/4230603751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d44102740 msgr2=0x7f8d44102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.512+0000 7f8d4a5a0640 1 --2- 172.21.15.23:0/4230603751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d44102740 0x7f8d44102b20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8d40009860 tx=0x7f8d4002bc20 comp rx=0 tx=0).stop 2024-03-24T10:47:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.512+0000 7f8d4a5a0640 1 -- 172.21.15.23:0/4230603751 shutdown_connections 2024-03-24T10:47:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.512+0000 7f8d4a5a0640 1 --2- 172.21.15.23:0/4230603751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d4410bb10 0x7f8d4410df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.512+0000 7f8d4a5a0640 1 --2- 172.21.15.23:0/4230603751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d44103060 0x7f8d4410b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.512+0000 7f8d4a5a0640 1 --2- 172.21.15.23:0/4230603751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d44102740 0x7f8d44102b20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.512+0000 7f8d4a5a0640 1 -- 172.21.15.23:0/4230603751 >> 172.21.15.23:0/4230603751 conn(0x7f8d440fbf60 msgr2=0x7f8d440fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.512+0000 7f8d4a5a0640 1 -- 172.21.15.23:0/4230603751 shutdown_connections 2024-03-24T10:47:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.512+0000 7f8d4a5a0640 1 -- 172.21.15.23:0/4230603751 wait complete. 2024-03-24T10:47:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.513+0000 7f8d4a5a0640 1 Processor -- start 2024-03-24T10:47:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.513+0000 7f8d4a5a0640 1 -- start start 2024-03-24T10:47:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.513+0000 7f8d4a5a0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d44102740 0x7f8d4419d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.513+0000 7f8d4a5a0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d44103060 0x7f8d4419dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.513+0000 7f8d4a5a0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d4410bb10 0x7f8d44197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.513+0000 7f8d4a5a0640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f8d44198070 con 0x7f8d44102740 2024-03-24T10:47:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.513+0000 7f8d4a5a0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8d441981e0 con 0x7f8d4410bb10 2024-03-24T10:47:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.513+0000 7f8d4a5a0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8d441983a0 con 0x7f8d44103060 2024-03-24T10:47:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.513+0000 7f8d4959e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d44102740 0x7f8d4419d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.513+0000 7f8d4959e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d44102740 0x7f8d4419d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52990/0 (socket says 172.21.15.23:52990) 2024-03-24T10:47:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.513+0000 7f8d4959e640 1 -- 172.21.15.23:0/2714818707 learned_addr learned my addr 172.21.15.23:0/2714818707 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.513+0000 7f8d48d9d640 1 --2- 172.21.15.23:0/2714818707 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d44103060 0x7f8d4419dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.513+0000 7f8d49d9f640 1 --2- 172.21.15.23:0/2714818707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d4410bb10 0x7f8d44197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.514+0000 7f8d4959e640 1 -- 172.21.15.23:0/2714818707 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d44103060 msgr2=0x7f8d4419dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.514+0000 7f8d4959e640 1 --2- 172.21.15.23:0/2714818707 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d44103060 0x7f8d4419dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.514+0000 7f8d4959e640 1 -- 172.21.15.23:0/2714818707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d4410bb10 msgr2=0x7f8d44197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.514+0000 7f8d4959e640 1 --2- 172.21.15.23:0/2714818707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d4410bb10 0x7f8d44197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.514+0000 7f8d4959e640 1 -- 172.21.15.23:0/2714818707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d40009490 con 0x7f8d44102740 2024-03-24T10:47:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.514+0000 7f8d4959e640 1 --2- 172.21.15.23:0/2714818707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d44102740 0x7f8d4419d820 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f8d40009980 tx=0x7f8d40034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.514+0000 7f8d3a7fc640 1 -- 172.21.15.23:0/2714818707 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4002c7c0 con 0x7f8d44102740 2024-03-24T10:47:56.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.514+0000 7f8d4a5a0640 1 -- 172.21.15.23:0/2714818707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d44198620 con 0x7f8d44102740 2024-03-24T10:47:56.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.514+0000 7f8d3a7fc640 1 -- 172.21.15.23:0/2714818707 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d40034a70 con 0x7f8d44102740 2024-03-24T10:47:56.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.514+0000 7f8d4a5a0640 1 -- 172.21.15.23:0/2714818707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d441a48c0 con 0x7f8d44102740 2024-03-24T10:47:56.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.514+0000 7f8d3a7fc640 1 -- 172.21.15.23:0/2714818707 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4003d3f0 con 0x7f8d44102740 2024-03-24T10:47:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.516+0000 7f8d3a7fc640 1 -- 172.21.15.23:0/2714818707 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f8d4003b040 con 0x7f8d44102740 2024-03-24T10:47:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.517+0000 7f8d4e3de740 1 Processor -- start 2024-03-24T10:47:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.517+0000 7f8d4e3de740 1 -- start start 2024-03-24T10:47:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.517+0000 7f8d3a7fc640 1 --2- 172.21.15.23:0/2714818707 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8d300a2220 0x7f8d300a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.517+0000 7f8d4e3de740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308de290 0x55d7308de670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d4e3de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308debb0 0x55d7308ebdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d4e3de740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308ec2f0 0x55d7308ee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d4e3de740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d7308991b0 con 0x55d7308ec2f0 2024-03-24T10:47:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d4e3de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d73084bf80 con 0x55d7308debb0 2024-03-24T10:47:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d4e3de740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d7308494b0 con 0x55d7308de290 2024-03-24T10:47:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d3a7fc640 1 -- 172.21.15.23:0/2714818707 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f8d400ef8c0 con 0x7f8d44102740 2024-03-24T10:47:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d49d9f640 1 --2- 172.21.15.23:0/2714818707 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8d300a2220 0x7f8d300a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d4959e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308de290 0x55d7308de670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d48d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308debb0 0x55d7308ebdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d4959e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308de290 0x55d7308de670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:44282/0 (socket says 172.21.15.23:44282) 2024-03-24T10:47:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d48d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308debb0 0x55d7308ebdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:36272/0 (socket says 172.21.15.23:36272) 2024-03-24T10:47:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d4959e640 1 -- 172.21.15.23:0/1229215108 learned_addr learned my addr 172.21.15.23:0/1229215108 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d49d9f640 1 --2- 172.21.15.23:0/1229215108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308ec2f0 0x55d7308ee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d4959e640 1 -- 172.21.15.23:0/1229215108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308debb0 msgr2=0x55d7308ebdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d4959e640 1 --2- 172.21.15.23:0/1229215108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308debb0 0x55d7308ebdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d4959e640 1 -- 172.21.15.23:0/1229215108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308ec2f0 msgr2=0x55d7308ee6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d4959e640 1 --2- 172.21.15.23:0/1229215108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308ec2f0 0x55d7308ee6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d4959e640 1 -- 172.21.15.23:0/1229215108 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7307da810 con 0x55d7308de290 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d49d9f640 1 --2- 172.21.15.23:0/2714818707 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8d300a2220 0x7f8d300a46e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c005fd0 tx=0x7f8d3c009e90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d4959e640 1 --2- 172.21.15.23:0/1229215108 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308de290 0x55d7308de670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8d400bc0c0 tx=0x7f8d4002bc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19076cb288c51e6c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d2f7fe640 1 -- 172.21.15.23:0/1229215108 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d400432c0 con 0x55d7308de290 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d2f7fe640 1 -- 172.21.15.23:0/1229215108 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d400312a0 con 0x55d7308de290 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.518+0000 7f8d2f7fe640 1 -- 172.21.15.23:0/1229215108 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d400c4bf0 con 0x55d7308de290 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.519+0000 7f8d4e3de740 1 -- 172.21.15.23:0/1229215108 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308de290 msgr2=0x55d7308de670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.519+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/1229215108 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308de290 0x55d7308de670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8d400bc0c0 tx=0x7f8d4002bc20 comp rx=0 tx=0).stop 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.519+0000 7f8d4e3de740 1 -- 172.21.15.23:0/1229215108 shutdown_connections 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.519+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/1229215108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308ec2f0 0x55d7308ee6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.519+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/1229215108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308debb0 0x55d7308ebdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.519+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/1229215108 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308de290 0x55d7308de670 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.519+0000 7f8d4e3de740 1 -- 172.21.15.23:0/1229215108 >> 172.21.15.23:0/1229215108 conn(0x55d7308d9110 msgr2=0x55d7308d94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.519+0000 7f8d4e3de740 1 -- 172.21.15.23:0/1229215108 shutdown_connections 2024-03-24T10:47:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.519+0000 7f8d4e3de740 1 -- 172.21.15.23:0/1229215108 wait complete. 2024-03-24T10:47:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4e3de740 1 -- start start 2024-03-24T10:47:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4e3de740 1 Processor -- start 2024-03-24T10:47:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4e3de740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308de290 0x55d7308e9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4e3de740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308debb0 0x55d7308e9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4e3de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308ec2f0 0x55d7308e9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4e3de740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d730844c90 con 0x55d7308de290 2024-03-24T10:47:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4e3de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d7307e9c50 con 0x55d7308ec2f0 2024-03-24T10:47:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4e3de740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d730518250 con 0x55d7308debb0 2024-03-24T10:47:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4e3de740 10 client.0 _ll_register_callbacks cb 0x55d730838290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4959e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308de290 0x55d7308e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4959e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308de290 0x55d7308e9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53000/0 (socket says 172.21.15.23:53000) 2024-03-24T10:47:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4959e640 1 -- 172.21.15.23:0/4089825448 learned_addr learned my addr 172.21.15.23:0/4089825448 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d48d9d640 1 --2- 172.21.15.23:0/4089825448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308debb0 0x55d7308e9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d49d9f640 1 --2- 172.21.15.23:0/4089825448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308ec2f0 0x55d7308e9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4959e640 1 -- 172.21.15.23:0/4089825448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308debb0 msgr2=0x55d7308e9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4959e640 1 --2- 172.21.15.23:0/4089825448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308debb0 0x55d7308e9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4959e640 1 -- 172.21.15.23:0/4089825448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308ec2f0 msgr2=0x55d7308e9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4959e640 1 --2- 172.21.15.23:0/4089825448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308ec2f0 0x55d7308e9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.520+0000 7f8d4959e640 1 -- 172.21.15.23:0/4089825448 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d441a48c0 con 0x55d7308de290 2024-03-24T10:47:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d4959e640 1 --2- 172.21.15.23:0/4089825448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308de290 0x55d7308e9310 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f8d4002cdd0 tx=0x7f8d40005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60a50b9864d9dc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d2cff9640 1 -- 172.21.15.23:0/4089825448 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d400432c0 con 0x55d7308de290 2024-03-24T10:47:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d2cff9640 1 -- 172.21.15.23:0/4089825448 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f8d400bc3c0 con 0x55d7308de290 2024-03-24T10:47:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d4e3de740 1 -- 172.21.15.23:0/4089825448 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d730869790 con 0x55d7308de290 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d2cff9640 1 -- 172.21.15.23:0/4089825448 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8d400bcac0 con 0x55d7308de290 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d4e3de740 10 client.17850 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d4e3de740 1 -- 172.21.15.23:0/4089825448 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d730843310 con 0x55d7308de290 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d2cff9640 1 -- 172.21.15.23:0/4089825448 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d730869790 con 0x55d7308de290 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d2cff9640 1 -- 172.21.15.23:0/4089825448 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f8d40003710 con 0x55d7308de290 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d4e3de740 10 client.17850 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d4e3de740 10 client.17850 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d4e3de740 20 client.17850 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/4089825448 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d7308fbae0 0x55d7308fded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d4e3de740 4 client.17850 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d4e3de740 1 -- 172.21.15.23:0/4089825448 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d730544600 con 0x55d7308fbae0 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.521+0000 7f8d48d9d640 1 --2- 172.21.15.23:0/4089825448 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d7308fbae0 0x55d7308fded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.522+0000 7f8d48d9d640 1 --2- 172.21.15.23:0/4089825448 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d7308fbae0 0x55d7308fded0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39dcc00fb0187347 server_cookie=b905a818e0aa2d34 in_seq=0 out_seq=0 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.522+0000 7f8d2cff9640 10 client.17850 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.522+0000 7f8d2cff9640 1 -- 172.21.15.23:0/4089825448 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d730544600 con 0x55d7308fbae0 2024-03-24T10:47:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.522+0000 7f8d2cff9640 10 client.17850 handle_command_reply: tid=0 2024-03-24T10:47:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.523+0000 7f8d4e3de740 1 client.17850 shutdown 2024-03-24T10:47:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.523+0000 7f8d4e3de740 1 -- 172.21.15.23:0/4089825448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308de290 msgr2=0x55d7308e9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.523+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/4089825448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308de290 0x55d7308e9310 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f8d4002cdd0 tx=0x7f8d40005b20 comp rx=0 tx=0).stop 2024-03-24T10:47:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.523+0000 7f8d4e3de740 1 -- 172.21.15.23:0/4089825448 shutdown_connections 2024-03-24T10:47:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.523+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/4089825448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308ec2f0 0x55d7308e9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.523+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/4089825448 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d7308fbae0 0x55d7308fded0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.523+0000 7f8d2cff9640 0 client.17850 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.523+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/4089825448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308debb0 0x55d7308e9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.523+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/4089825448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308de290 0x55d7308e9310 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.523+0000 7f8d4e3de740 1 -- 172.21.15.23:0/4089825448 >> 172.21.15.23:0/4089825448 conn(0x55d7308d9110 msgr2=0x55d7308dae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.523+0000 7f8d4e3de740 1 -- 172.21.15.23:0/4089825448 shutdown_connections 2024-03-24T10:47:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.523+0000 7f8d4e3de740 1 -- 172.21.15.23:0/4089825448 wait complete. 2024-03-24T10:47:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.523+0000 7f8d4e3de740 20 client.17850 trim_cache size 0 max 16384 2024-03-24T10:47:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.538+0000 7f8d4e3de740 1 Processor -- start 2024-03-24T10:47:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.538+0000 7f8d4e3de740 1 -- start start 2024-03-24T10:47:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.538+0000 7f8d4e3de740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308de290 0x55d7308eba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.538+0000 7f8d4e3de740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308debb0 0x55d7308f76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.538+0000 7f8d4e3de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308ec2f0 0x55d730903790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.538+0000 7f8d4e3de740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d7307ca660 con 0x55d7308debb0 2024-03-24T10:47:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.538+0000 7f8d4e3de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d7308f7c60 con 0x55d7308ec2f0 2024-03-24T10:47:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.538+0000 7f8d4e3de740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d7308f8980 con 0x55d7308de290 2024-03-24T10:47:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.539+0000 7f8d48d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308debb0 0x55d7308f76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.539+0000 7f8d48d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308debb0 0x55d7308f76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53012/0 (socket says 172.21.15.23:53012) 2024-03-24T10:47:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.539+0000 7f8d48d9d640 1 -- 172.21.15.23:0/2197748263 learned_addr learned my addr 172.21.15.23:0/2197748263 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.539+0000 7f8d4959e640 1 --2- 172.21.15.23:0/2197748263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308de290 0x55d7308eba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.539+0000 7f8d49d9f640 1 --2- 172.21.15.23:0/2197748263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308ec2f0 0x55d730903790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.539+0000 7f8d48d9d640 1 -- 172.21.15.23:0/2197748263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308de290 msgr2=0x55d7308eba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.539+0000 7f8d48d9d640 1 --2- 172.21.15.23:0/2197748263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308de290 0x55d7308eba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.539+0000 7f8d48d9d640 1 -- 172.21.15.23:0/2197748263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308ec2f0 msgr2=0x55d730903790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.539+0000 7f8d48d9d640 1 --2- 172.21.15.23:0/2197748263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308ec2f0 0x55d730903790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.539+0000 7f8d48d9d640 1 -- 172.21.15.23:0/2197748263 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7307df810 con 0x55d7308debb0 2024-03-24T10:47:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.539+0000 7f8d48d9d640 1 --2- 172.21.15.23:0/2197748263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308debb0 0x55d7308f76f0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f8d4410cee0 tx=0x7f8d34009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5915ea88775b06ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.539+0000 7f8d2effd640 1 -- 172.21.15.23:0/2197748263 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d34017070 con 0x55d7308debb0 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.539+0000 7f8d2effd640 1 -- 172.21.15.23:0/2197748263 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d34018040 con 0x55d7308debb0 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.539+0000 7f8d2effd640 1 -- 172.21.15.23:0/2197748263 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d340038e0 con 0x55d7308debb0 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.540+0000 7f8d4e3de740 1 -- 172.21.15.23:0/2197748263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308debb0 msgr2=0x55d7308f76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.540+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/2197748263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308debb0 0x55d7308f76f0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f8d4410cee0 tx=0x7f8d34009260 comp rx=0 tx=0).stop 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.540+0000 7f8d4e3de740 1 -- 172.21.15.23:0/2197748263 shutdown_connections 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.540+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/2197748263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308ec2f0 0x55d730903790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.540+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/2197748263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308debb0 0x55d7308f76f0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.540+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/2197748263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308de290 0x55d7308eba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.540+0000 7f8d4e3de740 1 -- 172.21.15.23:0/2197748263 >> 172.21.15.23:0/2197748263 conn(0x55d7308d9110 msgr2=0x55d7308d94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.540+0000 7f8d4e3de740 1 -- 172.21.15.23:0/2197748263 shutdown_connections 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.540+0000 7f8d4e3de740 1 -- 172.21.15.23:0/2197748263 wait complete. 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.540+0000 7f8d4e3de740 1 -- start start 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.540+0000 7f8d4e3de740 1 Processor -- start 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4e3de740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308de290 0x55d7308e65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4e3de740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308debb0 0x55d7308e6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4e3de740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308ec2f0 0x55d730960dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4e3de740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55d7308e7230 con 0x55d7308de290 2024-03-24T10:47:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4e3de740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d730961300 con 0x55d7308debb0 2024-03-24T10:47:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4e3de740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d730961470 con 0x55d7308ec2f0 2024-03-24T10:47:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4e3de740 10 client.0 _ll_register_callbacks cb 0x7f8d3c011720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:47:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4959e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308de290 0x55d7308e65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4959e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308de290 0x55d7308e65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53016/0 (socket says 172.21.15.23:53016) 2024-03-24T10:47:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4959e640 1 -- 172.21.15.23:0/72176487 learned_addr learned my addr 172.21.15.23:0/72176487 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:47:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d48d9d640 1 --2- 172.21.15.23:0/72176487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308debb0 0x55d7308e6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d49d9f640 1 --2- 172.21.15.23:0/72176487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308ec2f0 0x55d730960dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4959e640 1 -- 172.21.15.23:0/72176487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308ec2f0 msgr2=0x55d730960dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4959e640 1 --2- 172.21.15.23:0/72176487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308ec2f0 0x55d730960dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4959e640 1 -- 172.21.15.23:0/72176487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308debb0 msgr2=0x55d7308e6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4959e640 1 --2- 172.21.15.23:0/72176487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308debb0 0x55d7308e6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4959e640 1 -- 172.21.15.23:0/72176487 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d34009bc0 con 0x55d7308de290 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d4959e640 1 --2- 172.21.15.23:0/72176487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308de290 0x55d7308e65d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f8d400ba910 tx=0x7f8d4002c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e0ca4c66821341 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d2cff9640 1 -- 172.21.15.23:0/72176487 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d40031410 con 0x55d7308de290 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.541+0000 7f8d2cff9640 1 -- 172.21.15.23:0/72176487 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8527+0+0 (secure 0 0 0) 0x7f8d400c0510 con 0x55d7308de290 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.542+0000 7f8d4e3de740 1 -- 172.21.15.23:0/72176487 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d400312a0 con 0x55d7308de290 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.542+0000 7f8d2cff9640 1 -- 172.21.15.23:0/72176487 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8d400c46d0 con 0x55d7308de290 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.542+0000 7f8d4e3de740 10 client.17856 fetch_fsmap learned FSMap version 31 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.542+0000 7f8d4e3de740 1 -- 172.21.15.23:0/72176487 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d7308055a0 con 0x55d7308de290 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.542+0000 7f8d2cff9640 1 -- 172.21.15.23:0/72176487 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d400312a0 con 0x55d7308de290 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.542+0000 7f8d2cff9640 1 -- 172.21.15.23:0/72176487 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f8d40003680 con 0x55d7308de290 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.542+0000 7f8d4e3de740 10 client.17856 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.542+0000 7f8d4e3de740 10 client.17856 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T10:47:56.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.542+0000 7f8d4e3de740 20 client.17856 populate_metadata read hostname 'smithi023' 2024-03-24T10:47:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.542+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/72176487 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d7308fbae0 0x55d730964800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.542+0000 7f8d4e3de740 4 client.17856 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T10:47:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.542+0000 7f8d4e3de740 1 -- 172.21.15.23:0/72176487 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d7305951b0 con 0x55d7308fbae0 2024-03-24T10:47:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.542+0000 7f8d48d9d640 1 --2- 172.21.15.23:0/72176487 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d7308fbae0 0x55d730964800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.543+0000 7f8d2cff9640 10 client.17856 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:56.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.543+0000 7f8d48d9d640 1 --2- 172.21.15.23:0/72176487 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d7308fbae0 0x55d730964800 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b99bdcc6691430c server_cookie=fb47ba93d7397eab in_seq=0 out_seq=0 2024-03-24T10:47:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.553+0000 7f8d2cff9640 1 -- 172.21.15.23:0/72176487 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d7305951b0 con 0x55d7308fbae0 2024-03-24T10:47:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.553+0000 7f8d2cff9640 10 client.17856 handle_command_reply: tid=0 2024-03-24T10:47:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.553+0000 7f8d4e3de740 1 client.17856 shutdown 2024-03-24T10:47:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.554+0000 7f8d4e3de740 1 -- 172.21.15.23:0/72176487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308de290 msgr2=0x55d7308e65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.554+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/72176487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308de290 0x55d7308e65d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f8d400ba910 tx=0x7f8d4002c440 comp rx=0 tx=0).stop 2024-03-24T10:47:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.554+0000 7f8d4e3de740 1 -- 172.21.15.23:0/72176487 shutdown_connections 2024-03-24T10:47:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.554+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/72176487 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55d7308fbae0 0x55d730964800 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.554+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/72176487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7308ec2f0 0x55d730960dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.554+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/72176487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7308debb0 0x55d7308e6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.554+0000 7f8d4e3de740 1 --2- 172.21.15.23:0/72176487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7308de290 0x55d7308e65d0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.554+0000 7f8d4e3de740 1 -- 172.21.15.23:0/72176487 >> 172.21.15.23:0/72176487 conn(0x55d7308d9110 msgr2=0x55d7308e92b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.554+0000 7f8d2cff9640 0 client.17856 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:47:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.554+0000 7f8d4e3de740 1 -- 172.21.15.23:0/72176487 shutdown_connections 2024-03-24T10:47:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.554+0000 7f8d4e3de740 1 -- 172.21.15.23:0/72176487 wait complete. 2024-03-24T10:47:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.554+0000 7f8d4e3de740 20 client.17856 trim_cache size 0 max 16384 2024-03-24T10:47:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-24T10:47:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-24T10:47:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-24T10:47:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-24T10:47:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.555+0000 7f8d4a5a0640 1 -- 172.21.15.23:0/2714818707 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8d300a2220 msgr2=0x7f8d300a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.555+0000 7f8d4a5a0640 1 --2- 172.21.15.23:0/2714818707 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8d300a2220 0x7f8d300a46e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c005fd0 tx=0x7f8d3c009e90 comp rx=0 tx=0).stop 2024-03-24T10:47:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.555+0000 7f8d4a5a0640 1 -- 172.21.15.23:0/2714818707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d44102740 msgr2=0x7f8d4419d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.555+0000 7f8d4a5a0640 1 --2- 172.21.15.23:0/2714818707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d44102740 0x7f8d4419d820 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f8d40009980 tx=0x7f8d40034670 comp rx=0 tx=0).stop 2024-03-24T10:47:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.555+0000 7f8d4a5a0640 1 -- 172.21.15.23:0/2714818707 shutdown_connections 2024-03-24T10:47:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.555+0000 7f8d4a5a0640 1 --2- 172.21.15.23:0/2714818707 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8d300a2220 0x7f8d300a46e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.555+0000 7f8d4a5a0640 1 --2- 172.21.15.23:0/2714818707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d4410bb10 0x7f8d44197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.555+0000 7f8d4a5a0640 1 --2- 172.21.15.23:0/2714818707 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d44103060 0x7f8d4419dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.555+0000 7f8d4a5a0640 1 --2- 172.21.15.23:0/2714818707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d44102740 0x7f8d4419d820 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.555+0000 7f8d4a5a0640 1 -- 172.21.15.23:0/2714818707 >> 172.21.15.23:0/2714818707 conn(0x7f8d440fbf60 msgr2=0x7f8d440fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.555+0000 7f8d4a5a0640 1 -- 172.21.15.23:0/2714818707 shutdown_connections 2024-03-24T10:47:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-24T10:47:56.555+0000 7f8d4a5a0640 1 -- 172.21.15.23:0/2714818707 wait complete. 2024-03-24T10:47:56.904 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:56 smithi117 ceph-mon[36368]: pgmap v242: 97 pgs: 97 active+clean; 453 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 175 op/s 2024-03-24T10:47:56.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:56 smithi023 ceph-mon[32466]: pgmap v242: 97 pgs: 97 active+clean; 453 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 175 op/s 2024-03-24T10:47:57.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:56 smithi116 ceph-mon[36248]: pgmap v242: 97 pgs: 97 active+clean; 453 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 175 op/s 2024-03-24T10:47:57.127 INFO:tasks.workunit.client.0.smithi023.stdout: 1 776395 10.73 MB/sec execute 254 sec latency 21.738 ms 2024-03-24T10:47:58.127 INFO:tasks.workunit.client.0.smithi023.stdout: 1 778155 10.73 MB/sec execute 255 sec latency 53.063 ms 2024-03-24T10:47:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:47:58 smithi023 ceph-mon[32466]: pgmap v243: 97 pgs: 97 active+clean; 461 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 258 op/s 2024-03-24T10:47:59.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:47:58 smithi117 ceph-mon[36368]: pgmap v243: 97 pgs: 97 active+clean; 461 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 258 op/s 2024-03-24T10:47:59.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:47:58 smithi116 ceph-mon[36248]: pgmap v243: 97 pgs: 97 active+clean; 461 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 258 op/s 2024-03-24T10:47:59.127 INFO:tasks.workunit.client.0.smithi023.stdout: 1 780031 10.72 MB/sec execute 256 sec latency 20.761 ms 2024-03-24T10:48:00.127 INFO:tasks.workunit.client.0.smithi023.stdout: 1 781961 10.72 MB/sec execute 257 sec latency 16.419 ms 2024-03-24T10:48:00.825 INFO:tasks.workunit.client.0.smithi023.stdout:[783189] write failed on handle 13788 (Resource temporarily unavailable) 2024-03-24T10:48:00.825 INFO:tasks.workunit.client.0.smithi023.stdout:Child failed with status 1 2024-03-24T10:48:00.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:48:00.826 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-03-24T10:48:00.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-24T10:48:00.850 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:00 smithi023 ceph-mon[32466]: pgmap v244: 97 pgs: 97 active+clean; 461 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 171 op/s 2024-03-24T10:48:01.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:00 smithi117 ceph-mon[36368]: pgmap v244: 97 pgs: 97 active+clean; 461 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 171 op/s 2024-03-24T10:48:01.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:00 smithi116 ceph-mon[36248]: pgmap v244: 97 pgs: 97 active+clean; 461 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 171 op/s 2024-03-24T10:48:01.112 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-24T10:48:01.349 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=887abaa611ed40c8b7bd91097dff5e36 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-24T10:48:01.352 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-24T10:48:01.362 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T10:48:01.376 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-24T10:48:01.385 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-24T10:48:01.385 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T10:48:01.386 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T10:48:01.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-24T10:48:01.443 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-24T10:48:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:02 smithi023 ceph-mon[32466]: pgmap v245: 97 pgs: 97 active+clean; 468 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 272 op/s 2024-03-24T10:48:03.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:02 smithi117 ceph-mon[36368]: pgmap v245: 97 pgs: 97 active+clean; 468 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 272 op/s 2024-03-24T10:48:03.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:02 smithi116 ceph-mon[36248]: pgmap v245: 97 pgs: 97 active+clean; 468 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 272 op/s 2024-03-24T10:48:04.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:04 smithi023 ceph-mon[32466]: pgmap v246: 97 pgs: 97 active+clean; 468 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 183 op/s 2024-03-24T10:48:05.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:04 smithi117 ceph-mon[36368]: pgmap v246: 97 pgs: 97 active+clean; 468 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 183 op/s 2024-03-24T10:48:05.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:04 smithi116 ceph-mon[36248]: pgmap v246: 97 pgs: 97 active+clean; 468 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 183 op/s 2024-03-24T10:48:06.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:06 smithi023 ceph-mon[32466]: pgmap v247: 97 pgs: 97 active+clean; 468 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 183 op/s 2024-03-24T10:48:06.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:06 smithi023 ceph-mon[32466]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T10:48:07.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:06 smithi117 ceph-mon[36368]: pgmap v247: 97 pgs: 97 active+clean; 468 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 183 op/s 2024-03-24T10:48:07.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:06 smithi117 ceph-mon[36368]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T10:48:07.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:06 smithi116 ceph-mon[36248]: pgmap v247: 97 pgs: 97 active+clean; 468 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 183 op/s 2024-03-24T10:48:07.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:06 smithi116 ceph-mon[36248]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T10:48:08.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:08 smithi023 ceph-mon[32466]: pgmap v249: 97 pgs: 97 active+clean; 464 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-03-24T10:48:09.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:08 smithi117 ceph-mon[36368]: pgmap v249: 97 pgs: 97 active+clean; 464 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-03-24T10:48:09.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:08 smithi116 ceph-mon[36248]: pgmap v249: 97 pgs: 97 active+clean; 464 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-03-24T10:48:10.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:10 smithi023 ceph-mon[32466]: pgmap v250: 97 pgs: 97 active+clean; 464 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-03-24T10:48:11.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:10 smithi117 ceph-mon[36368]: pgmap v250: 97 pgs: 97 active+clean; 464 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-03-24T10:48:11.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:10 smithi116 ceph-mon[36248]: pgmap v250: 97 pgs: 97 active+clean; 464 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 156 op/s 2024-03-24T10:48:12.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:12 smithi117 ceph-mon[36368]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T10:48:12.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:12 smithi117 ceph-mon[36368]: pgmap v252: 97 pgs: 97 active+clean; 463 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-03-24T10:48:12.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:12 smithi116 ceph-mon[36248]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T10:48:12.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:12 smithi116 ceph-mon[36248]: pgmap v252: 97 pgs: 97 active+clean; 463 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-03-24T10:48:12.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:12 smithi023 ceph-mon[32466]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T10:48:12.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:12 smithi023 ceph-mon[32466]: pgmap v252: 97 pgs: 97 active+clean; 463 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-03-24T10:48:14.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:14 smithi117 ceph-mon[36368]: pgmap v253: 97 pgs: 97 active+clean; 463 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-03-24T10:48:14.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:14 smithi116 ceph-mon[36248]: pgmap v253: 97 pgs: 97 active+clean; 463 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-03-24T10:48:14.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:14 smithi023 ceph-mon[32466]: pgmap v253: 97 pgs: 97 active+clean; 463 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 2.0 MiB/s wr, 84 op/s 2024-03-24T10:48:16.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:16 smithi117 ceph-mon[36368]: pgmap v254: 97 pgs: 97 active+clean; 463 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 1.6 MiB/s wr, 68 op/s 2024-03-24T10:48:16.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:16 smithi116 ceph-mon[36248]: pgmap v254: 97 pgs: 97 active+clean; 463 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 1.6 MiB/s wr, 68 op/s 2024-03-24T10:48:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:16 smithi023 ceph-mon[32466]: pgmap v254: 97 pgs: 97 active+clean; 463 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 1.6 MiB/s wr, 68 op/s 2024-03-24T10:48:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:18 smithi117 ceph-mon[36368]: pgmap v255: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.6 MiB/s wr, 61 op/s 2024-03-24T10:48:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:18 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:48:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:18 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:48:18.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:18 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:48:18.754 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:18 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:48:18.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:18 smithi116 ceph-mon[36248]: pgmap v255: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.6 MiB/s wr, 61 op/s 2024-03-24T10:48:18.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:18 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:48:18.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:18 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:48:18.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:18 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:48:18.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:18 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:48:18.833 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:48:18.834 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:48:18.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:18 smithi023 ceph-mon[32466]: pgmap v255: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.6 MiB/s wr, 61 op/s 2024-03-24T10:48:18.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:18 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:48:18.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:18 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:48:18.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:18 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:48:18.858 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:18 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:48:18.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.942+0000 7f20444fa640 1 -- 172.21.15.23:0/3973997337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f203c0692b0 msgr2=0x7f203c10e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:18.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.942+0000 7f20444fa640 1 --2- 172.21.15.23:0/3973997337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f203c0692b0 0x7f203c10e550 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f202c009cd0 tx=0x7f202c02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.942+0000 7f20444fa640 1 -- 172.21.15.23:0/3973997337 shutdown_connections 2024-03-24T10:48:18.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.942+0000 7f20444fa640 1 --2- 172.21.15.23:0/3973997337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f203c0692b0 0x7f203c10e550 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.942+0000 7f20444fa640 1 --2- 172.21.15.23:0/3973997337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f203c0688f0 0x7f203c068d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.942+0000 7f20444fa640 1 --2- 172.21.15.23:0/3973997337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f203c107690 0x7f203c107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.942+0000 7f20444fa640 1 -- 172.21.15.23:0/3973997337 >> 172.21.15.23:0/3973997337 conn(0x7f203c077710 msgr2=0x7f203c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:18.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.942+0000 7f20444fa640 1 -- 172.21.15.23:0/3973997337 shutdown_connections 2024-03-24T10:48:18.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.942+0000 7f20444fa640 1 -- 172.21.15.23:0/3973997337 wait complete. 2024-03-24T10:48:18.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.942+0000 7f20444fa640 1 Processor -- start 2024-03-24T10:48:18.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.942+0000 7f20444fa640 1 -- start start 2024-03-24T10:48:18.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.942+0000 7f20444fa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f203c0688f0 0x7f203c197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.942+0000 7f20444fa640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f203c0692b0 0x7f203c198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f20444fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f203c107690 0x7f203c19c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f20444fa640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f203c19cbe0 con 0x7f203c0692b0 2024-03-24T10:48:18.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f20444fa640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f203c19cd50 con 0x7f203c107690 2024-03-24T10:48:18.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f20444fa640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f203c19cf10 con 0x7f203c0688f0 2024-03-24T10:48:18.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f2041a95640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f203c0692b0 0x7f203c198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f2041a95640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f203c0692b0 0x7f203c198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40178/0 (socket says 172.21.15.23:40178) 2024-03-24T10:48:18.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f2041a95640 1 -- 172.21.15.23:0/2488558119 learned_addr learned my addr 172.21.15.23:0/2488558119 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:18.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f2042a97640 1 --2- 172.21.15.23:0/2488558119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f203c107690 0x7f203c19c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f2042296640 1 --2- 172.21.15.23:0/2488558119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f203c0688f0 0x7f203c197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f2041a95640 1 -- 172.21.15.23:0/2488558119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f203c0688f0 msgr2=0x7f203c197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:18.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f2041a95640 1 --2- 172.21.15.23:0/2488558119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f203c0688f0 0x7f203c197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f2041a95640 1 -- 172.21.15.23:0/2488558119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f203c107690 msgr2=0x7f203c19c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:18.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f2041a95640 1 --2- 172.21.15.23:0/2488558119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f203c107690 0x7f203c19c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f2041a95640 1 -- 172.21.15.23:0/2488558119 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2030009490 con 0x7f203c0692b0 2024-03-24T10:48:18.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f2041a95640 1 --2- 172.21.15.23:0/2488558119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f203c0692b0 0x7f203c198260 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f203000ea60 tx=0x7f203000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:18.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f203b7fe640 1 -- 172.21.15.23:0/2488558119 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203000cde0 con 0x7f203c0692b0 2024-03-24T10:48:18.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f20444fa640 1 -- 172.21.15.23:0/2488558119 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202c009900 con 0x7f203c0692b0 2024-03-24T10:48:18.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f203b7fe640 1 -- 172.21.15.23:0/2488558119 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20300035f0 con 0x7f203c0692b0 2024-03-24T10:48:18.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f203b7fe640 1 -- 172.21.15.23:0/2488558119 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2030010690 con 0x7f203c0692b0 2024-03-24T10:48:18.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.943+0000 7f20444fa640 1 -- 172.21.15.23:0/2488558119 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f203c1a4890 con 0x7f203c0692b0 2024-03-24T10:48:18.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.946+0000 7f203b7fe640 1 -- 172.21.15.23:0/2488558119 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f2030003af0 con 0x7f203c0692b0 2024-03-24T10:48:18.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.946+0000 7f20471b0740 1 Processor -- start 2024-03-24T10:48:18.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.946+0000 7f20471b0740 1 -- start start 2024-03-24T10:48:18.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.946+0000 7f203b7fe640 1 --2- 172.21.15.23:0/2488558119 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f20240a2230 0x7f20240a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.946+0000 7f20471b0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee757435b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f20471b0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 0x55ee7574c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f20471b0740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee7574eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f20471b0740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ee75667f30 con 0x55ee7574c6d0 2024-03-24T10:48:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f20471b0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ee756b1810 con 0x55ee757431d0 2024-03-24T10:48:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f20471b0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ee756aed40 con 0x55ee75743af0 2024-03-24T10:48:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2042a97640 1 --2- 172.21.15.23:0/2488558119 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f20240a2230 0x7f20240a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f203b7fe640 1 -- 172.21.15.23:0/2488558119 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f2030014070 con 0x7f203c0692b0 2024-03-24T10:48:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2042296640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee757435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2042296640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee757435b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:50562/0 (socket says 172.21.15.23:50562) 2024-03-24T10:48:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2042296640 1 -- 172.21.15.23:0/3218131786 learned_addr learned my addr 172.21.15.23:0/3218131786 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2042a97640 1 --2- 172.21.15.23:0/3218131786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee7574eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2041a95640 1 --2- 172.21.15.23:0/3218131786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 0x55ee7574c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2042296640 1 -- 172.21.15.23:0/3218131786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 msgr2=0x55ee7574c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2042296640 1 --2- 172.21.15.23:0/3218131786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 0x55ee7574c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2042296640 1 -- 172.21.15.23:0/3218131786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 msgr2=0x55ee7574eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2042296640 1 --2- 172.21.15.23:0/3218131786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee7574eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2042296640 1 -- 172.21.15.23:0/3218131786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee7566bea0 con 0x55ee757431d0 2024-03-24T10:48:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2042a97640 1 --2- 172.21.15.23:0/2488558119 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f20240a2230 0x7f20240a46f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f202c005cc0 tx=0x7f202c032f20 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2042296640 1 --2- 172.21.15.23:0/3218131786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee757435b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2034009ca0 tx=0x7f2034011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5223a5097bc2ec19 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2038bfe640 1 -- 172.21.15.23:0/3218131786 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2034019630 con 0x55ee757431d0 2024-03-24T10:48:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2038bfe640 1 -- 172.21.15.23:0/3218131786 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2034019c70 con 0x55ee757431d0 2024-03-24T10:48:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.947+0000 7f2038bfe640 1 -- 172.21.15.23:0/3218131786 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203400eb70 con 0x55ee757431d0 2024-03-24T10:48:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.948+0000 7f20471b0740 1 -- 172.21.15.23:0/3218131786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 msgr2=0x55ee757435b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.948+0000 7f20471b0740 1 --2- 172.21.15.23:0/3218131786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee757435b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2034009ca0 tx=0x7f2034011810 comp rx=0 tx=0).stop 2024-03-24T10:48:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.948+0000 7f20471b0740 1 -- 172.21.15.23:0/3218131786 shutdown_connections 2024-03-24T10:48:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.948+0000 7f20471b0740 1 --2- 172.21.15.23:0/3218131786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee7574eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.948+0000 7f20471b0740 1 --2- 172.21.15.23:0/3218131786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 0x55ee7574c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.948+0000 7f20471b0740 1 --2- 172.21.15.23:0/3218131786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee757435b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.948+0000 7f20471b0740 1 -- 172.21.15.23:0/3218131786 >> 172.21.15.23:0/3218131786 conn(0x55ee7573d900 msgr2=0x55ee7573dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.948+0000 7f20471b0740 1 -- 172.21.15.23:0/3218131786 shutdown_connections 2024-03-24T10:48:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.948+0000 7f20471b0740 1 -- 172.21.15.23:0/3218131786 wait complete. 2024-03-24T10:48:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f20471b0740 1 -- start start 2024-03-24T10:48:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f20471b0740 1 Processor -- start 2024-03-24T10:48:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f20471b0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee75749710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f20471b0740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee75743af0 0x55ee75749c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f20471b0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee7574c6d0 0x55ee7574a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f20471b0740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ee756aa520 con 0x55ee75743af0 2024-03-24T10:48:18.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f20471b0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ee7537d430 con 0x55ee757431d0 2024-03-24T10:48:18.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f20471b0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ee75339e00 con 0x55ee7574c6d0 2024-03-24T10:48:18.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f20471b0740 10 client.0 _ll_register_callbacks cb 0x55ee75648d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:18.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f2041a95640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee75743af0 0x55ee75749c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f2041a95640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee75743af0 0x55ee75749c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40200/0 (socket says 172.21.15.23:40200) 2024-03-24T10:48:18.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f2041a95640 1 -- 172.21.15.23:0/4280734322 learned_addr learned my addr 172.21.15.23:0/4280734322 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:18.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f2042296640 1 --2- 172.21.15.23:0/4280734322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee75749710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f2042a97640 1 --2- 172.21.15.23:0/4280734322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee7574c6d0 0x55ee7574a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f2041a95640 1 -- 172.21.15.23:0/4280734322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee7574c6d0 msgr2=0x55ee7574a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:18.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f2041a95640 1 --2- 172.21.15.23:0/4280734322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee7574c6d0 0x55ee7574a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f2041a95640 1 -- 172.21.15.23:0/4280734322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 msgr2=0x55ee75749710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:18.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f2041a95640 1 --2- 172.21.15.23:0/4280734322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee75749710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f2041a95640 1 -- 172.21.15.23:0/4280734322 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20340098d0 con 0x55ee75743af0 2024-03-24T10:48:18.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f2042a97640 1 --2- 172.21.15.23:0/4280734322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee7574c6d0 0x55ee7574a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:48:18.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f2041a95640 1 --2- 172.21.15.23:0/4280734322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee75743af0 0x55ee75749c50 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f203009abf0 tx=0x7f203009b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d285f02b64b766f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:18.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.949+0000 7f2021ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:18.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f2021ffb640 1 -- 172.21.15.23:0/4280734322 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203009aef0 con 0x55ee75743af0 2024-03-24T10:48:18.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f2021ffb640 1 -- 172.21.15.23:0/4280734322 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f203009f040 con 0x55ee75743af0 2024-03-24T10:48:18.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f20471b0740 1 -- 172.21.15.23:0/4280734322 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ee753a97e0 con 0x55ee75743af0 2024-03-24T10:48:18.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f2021ffb640 1 -- 172.21.15.23:0/4280734322 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f20300a3690 con 0x55ee75743af0 2024-03-24T10:48:18.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f20471b0740 10 client.17868 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:18.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f20471b0740 1 -- 172.21.15.23:0/4280734322 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ee756a8ba0 con 0x55ee75743af0 2024-03-24T10:48:18.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f2021ffb640 1 -- 172.21.15.23:0/4280734322 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ee753a97e0 con 0x55ee75743af0 2024-03-24T10:48:18.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f2021ffb640 1 -- 172.21.15.23:0/4280734322 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f20300a3200 con 0x55ee75743af0 2024-03-24T10:48:18.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f20471b0740 10 client.17868 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:18.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f20471b0740 10 client.17868 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:18.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f20471b0740 20 client.17868 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:18.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f20471b0740 1 --2- 172.21.15.23:0/4280734322 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ee7575cbb0 0x55ee7575efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f20471b0740 4 client.17868 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:48:18.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f20471b0740 1 -- 172.21.15.23:0/4280734322 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee753e6720 con 0x55ee7575cbb0 2024-03-24T10:48:18.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.950+0000 7f2042296640 1 --2- 172.21.15.23:0/4280734322 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ee7575cbb0 0x55ee7575efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.951+0000 7f2021ffb640 10 client.17868 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:18.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.951+0000 7f2042296640 1 --2- 172.21.15.23:0/4280734322 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ee7575cbb0 0x55ee7575efd0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38a5ee25c8d50664 server_cookie=22c52df07381be56 in_seq=0 out_seq=0 2024-03-24T10:48:18.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.951+0000 7f2021ffb640 1 -- 172.21.15.23:0/4280734322 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ee753e6720 con 0x55ee7575cbb0 2024-03-24T10:48:18.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.951+0000 7f2021ffb640 10 client.17868 handle_command_reply: tid=0 2024-03-24T10:48:18.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.951+0000 7f20471b0740 1 client.17868 shutdown 2024-03-24T10:48:18.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.951+0000 7f20471b0740 1 -- 172.21.15.23:0/4280734322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee75743af0 msgr2=0x55ee75749c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:18.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.951+0000 7f20471b0740 1 --2- 172.21.15.23:0/4280734322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee75743af0 0x55ee75749c50 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f203009abf0 tx=0x7f203009b730 comp rx=0 tx=0).stop 2024-03-24T10:48:18.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.951+0000 7f20471b0740 1 -- 172.21.15.23:0/4280734322 shutdown_connections 2024-03-24T10:48:18.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.951+0000 7f20471b0740 1 --2- 172.21.15.23:0/4280734322 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ee7575cbb0 0x55ee7575efd0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.951+0000 7f20471b0740 1 --2- 172.21.15.23:0/4280734322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee7574c6d0 0x55ee7574a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.951+0000 7f20471b0740 1 --2- 172.21.15.23:0/4280734322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee75743af0 0x55ee75749c50 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.951+0000 7f2021ffb640 0 client.17868 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:18.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.951+0000 7f20471b0740 1 --2- 172.21.15.23:0/4280734322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee75749710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.951+0000 7f20471b0740 1 -- 172.21.15.23:0/4280734322 >> 172.21.15.23:0/4280734322 conn(0x55ee7573d900 msgr2=0x55ee7573f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:18.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.952+0000 7f20471b0740 1 -- 172.21.15.23:0/4280734322 shutdown_connections 2024-03-24T10:48:18.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.952+0000 7f20471b0740 1 -- 172.21.15.23:0/4280734322 wait complete. 2024-03-24T10:48:18.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.952+0000 7f20471b0740 20 client.17868 trim_cache size 0 max 16384 2024-03-24T10:48:18.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.969+0000 7f20471b0740 1 Processor -- start 2024-03-24T10:48:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.969+0000 7f20471b0740 1 -- start start 2024-03-24T10:48:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.969+0000 7f20471b0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee7574be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.969+0000 7f20471b0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 0x55ee7573cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.969+0000 7f20471b0740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee75749fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.969+0000 7f20471b0740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ee7562f430 con 0x55ee7574c6d0 2024-03-24T10:48:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.969+0000 7f20471b0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ee7573e4d0 con 0x55ee757431d0 2024-03-24T10:48:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.969+0000 7f20471b0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ee75742e30 con 0x55ee75743af0 2024-03-24T10:48:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.969+0000 7f2042a97640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee75749fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.969+0000 7f2042a97640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee75749fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40210/0 (socket says 172.21.15.23:40210) 2024-03-24T10:48:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.969+0000 7f2042a97640 1 -- 172.21.15.23:0/680558743 learned_addr learned my addr 172.21.15.23:0/680558743 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.969+0000 7f2042296640 1 --2- 172.21.15.23:0/680558743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee7574be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.969+0000 7f2041a95640 1 --2- 172.21.15.23:0/680558743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 0x55ee7573cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.969+0000 7f2042a97640 1 -- 172.21.15.23:0/680558743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 msgr2=0x55ee7573cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:18.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f2042a97640 1 --2- 172.21.15.23:0/680558743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 0x55ee7573cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f2042a97640 1 -- 172.21.15.23:0/680558743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 msgr2=0x55ee7574be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:18.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f2042a97640 1 --2- 172.21.15.23:0/680558743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee7574be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f2042a97640 1 -- 172.21.15.23:0/680558743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee75644b20 con 0x55ee7574c6d0 2024-03-24T10:48:18.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f2042a97640 1 --2- 172.21.15.23:0/680558743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee75749fb0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f202c009cd0 tx=0x7f202c04b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4921cf17726d3b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:18.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f2023fff640 1 -- 172.21.15.23:0/680558743 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c048070 con 0x55ee7574c6d0 2024-03-24T10:48:18.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f2023fff640 1 -- 172.21.15.23:0/680558743 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f202c004d40 con 0x55ee7574c6d0 2024-03-24T10:48:18.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f2023fff640 1 -- 172.21.15.23:0/680558743 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c003c80 con 0x55ee7574c6d0 2024-03-24T10:48:18.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f20471b0740 1 -- 172.21.15.23:0/680558743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 msgr2=0x55ee75749fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:18.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f20471b0740 1 --2- 172.21.15.23:0/680558743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee75749fb0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f202c009cd0 tx=0x7f202c04b000 comp rx=0 tx=0).stop 2024-03-24T10:48:18.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f20471b0740 1 -- 172.21.15.23:0/680558743 shutdown_connections 2024-03-24T10:48:18.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f20471b0740 1 --2- 172.21.15.23:0/680558743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee75749fb0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f20471b0740 1 --2- 172.21.15.23:0/680558743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 0x55ee7573cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f20471b0740 1 --2- 172.21.15.23:0/680558743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee7574be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.970+0000 7f20471b0740 1 -- 172.21.15.23:0/680558743 >> 172.21.15.23:0/680558743 conn(0x55ee7573d900 msgr2=0x55ee7573ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:18.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f20471b0740 1 -- 172.21.15.23:0/680558743 shutdown_connections 2024-03-24T10:48:18.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f20471b0740 1 -- 172.21.15.23:0/680558743 wait complete. 2024-03-24T10:48:18.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f20471b0740 1 -- start start 2024-03-24T10:48:18.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f20471b0740 1 Processor -- start 2024-03-24T10:48:18.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f20471b0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee757504b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f20471b0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 0x55ee757509f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f20471b0740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee757c1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f20471b0740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ee757510f0 con 0x55ee7574c6d0 2024-03-24T10:48:18.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f20471b0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ee757c2410 con 0x55ee757431d0 2024-03-24T10:48:18.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f20471b0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ee757c2550 con 0x55ee75743af0 2024-03-24T10:48:18.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f20471b0740 10 client.0 _ll_register_callbacks cb 0x7f203009b5f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:18.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f2042a97640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee757c1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f2042a97640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee757c1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40220/0 (socket says 172.21.15.23:40220) 2024-03-24T10:48:18.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f2042a97640 1 -- 172.21.15.23:0/3219368340 learned_addr learned my addr 172.21.15.23:0/3219368340 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:18.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f2041a95640 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 0x55ee757509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f2042296640 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee757504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f2042a97640 1 -- 172.21.15.23:0/3219368340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 msgr2=0x55ee757509f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:18.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f2042a97640 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 0x55ee757509f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f2042a97640 1 -- 172.21.15.23:0/3219368340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 msgr2=0x55ee757504b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f2042a97640 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee757504b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f2042a97640 1 -- 172.21.15.23:0/3219368340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f203c10ee50 con 0x55ee7574c6d0 2024-03-24T10:48:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f2041a95640 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 0x55ee757509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.971+0000 7f2042296640 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee757504b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f2042a97640 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee757c1ed0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f202c02c630 tx=0x7f202c034440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8535c7b186bde1fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f2021ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f2021ffb640 1 -- 172.21.15.23:0/3219368340 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c048070 con 0x55ee7574c6d0 2024-03-24T10:48:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f2021ffb640 1 -- 172.21.15.23:0/3219368340 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f202c032920 con 0x55ee7574c6d0 2024-03-24T10:48:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f20471b0740 1 -- 172.21.15.23:0/3219368340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2034019c70 con 0x55ee7574c6d0 2024-03-24T10:48:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f2021ffb640 1 -- 172.21.15.23:0/3219368340 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f202c003970 con 0x55ee7574c6d0 2024-03-24T10:48:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f20471b0740 10 client.17874 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f20471b0740 1 -- 172.21.15.23:0/3219368340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ee7566a700 con 0x55ee7574c6d0 2024-03-24T10:48:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f2021ffb640 1 -- 172.21.15.23:0/3219368340 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f202c065020 con 0x55ee7574c6d0 2024-03-24T10:48:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f2021ffb640 1 -- 172.21.15.23:0/3219368340 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f202c059460 con 0x55ee7574c6d0 2024-03-24T10:48:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f20471b0740 10 client.17874 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f20471b0740 10 client.17874 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f20471b0740 20 client.17874 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f20471b0740 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ee757c3a50 0x55ee757c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f20471b0740 4 client.17874 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:48:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f20471b0740 1 -- 172.21.15.23:0/3219368340 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ee75435890 con 0x55ee757c3a50 2024-03-24T10:48:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.972+0000 7f2042296640 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ee757c3a50 0x55ee757c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f2021ffb640 10 client.17874 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f2042296640 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ee757c3a50 0x55ee757c5e70 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c00145f427330e4 server_cookie=858ea4951976269c in_seq=0 out_seq=0 2024-03-24T10:48:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f2021ffb640 1 -- 172.21.15.23:0/3219368340 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ee75435890 con 0x55ee757c3a50 2024-03-24T10:48:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f2021ffb640 10 client.17874 handle_command_reply: tid=0 2024-03-24T10:48:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f20471b0740 1 client.17874 shutdown 2024-03-24T10:48:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f20471b0740 1 -- 172.21.15.23:0/3219368340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 msgr2=0x55ee757c1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f20471b0740 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee757c1ed0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f202c02c630 tx=0x7f202c034440 comp rx=0 tx=0).stop 2024-03-24T10:48:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f20471b0740 1 -- 172.21.15.23:0/3219368340 shutdown_connections 2024-03-24T10:48:18.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f20471b0740 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ee7574c6d0 0x55ee757c1ed0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f20471b0740 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ee757c3a50 0x55ee757c5e70 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f20471b0740 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ee75743af0 0x55ee757509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f2021ffb640 0 client.17874 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:18.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f20471b0740 1 --2- 172.21.15.23:0/3219368340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ee757431d0 0x55ee757504b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f20471b0740 1 -- 172.21.15.23:0/3219368340 >> 172.21.15.23:0/3219368340 conn(0x55ee7573d900 msgr2=0x55ee7573f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:18.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f20471b0740 1 -- 172.21.15.23:0/3219368340 shutdown_connections 2024-03-24T10:48:18.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f20471b0740 1 -- 172.21.15.23:0/3219368340 wait complete. 2024-03-24T10:48:18.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.973+0000 7f20471b0740 20 client.17874 trim_cache size 0 max 16384 2024-03-24T10:48:18.987 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:48:18.987 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:48:18.988 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:48:18.988 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:48:18.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.974+0000 7f20444fa640 1 -- 172.21.15.23:0/2488558119 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f20240a2230 msgr2=0x7f20240a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:18.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.974+0000 7f20444fa640 1 --2- 172.21.15.23:0/2488558119 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f20240a2230 0x7f20240a46f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f202c005cc0 tx=0x7f202c032f20 comp rx=0 tx=0).stop 2024-03-24T10:48:18.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.974+0000 7f20444fa640 1 -- 172.21.15.23:0/2488558119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f203c0692b0 msgr2=0x7f203c198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:18.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.974+0000 7f20444fa640 1 --2- 172.21.15.23:0/2488558119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f203c0692b0 0x7f203c198260 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f203000ea60 tx=0x7f203000ef20 comp rx=0 tx=0).stop 2024-03-24T10:48:18.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.974+0000 7f20444fa640 1 -- 172.21.15.23:0/2488558119 shutdown_connections 2024-03-24T10:48:18.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.974+0000 7f20444fa640 1 --2- 172.21.15.23:0/2488558119 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f20240a2230 0x7f20240a46f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.974+0000 7f20444fa640 1 --2- 172.21.15.23:0/2488558119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f203c107690 0x7f203c19c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.974+0000 7f20444fa640 1 --2- 172.21.15.23:0/2488558119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f203c0692b0 0x7f203c198260 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.974+0000 7f20444fa640 1 --2- 172.21.15.23:0/2488558119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f203c0688f0 0x7f203c197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.974+0000 7f20444fa640 1 -- 172.21.15.23:0/2488558119 >> 172.21.15.23:0/2488558119 conn(0x7f203c077710 msgr2=0x7f203c0fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:18.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.974+0000 7f20444fa640 1 -- 172.21.15.23:0/2488558119 shutdown_connections 2024-03-24T10:48:18.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:18.974+0000 7f20444fa640 1 -- 172.21.15.23:0/2488558119 wait complete. 2024-03-24T10:48:18.991 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:48:18.991 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:19.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.133+0000 7fb9184f1640 1 -- 172.21.15.23:0/677677618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb91006a7a0 msgr2=0x7fb91010e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.133+0000 7fb9184f1640 1 --2- 172.21.15.23:0/677677618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb91006a7a0 0x7fb91010e560 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fb90c009930 tx=0x7fb90c02bcd0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.133+0000 7fb9184f1640 1 -- 172.21.15.23:0/677677618 shutdown_connections 2024-03-24T10:48:19.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.133+0000 7fb9184f1640 1 --2- 172.21.15.23:0/677677618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb91006a7a0 0x7fb91010e560 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.133+0000 7fb9184f1640 1 --2- 172.21.15.23:0/677677618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb910069e00 0x7fb91006a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.133+0000 7fb9184f1640 1 --2- 172.21.15.23:0/677677618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb910107710 0x7fb910107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.133+0000 7fb9184f1640 1 -- 172.21.15.23:0/677677618 >> 172.21.15.23:0/677677618 conn(0x7fb9100776b0 msgr2=0x7fb910077ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.133+0000 7fb9184f1640 1 -- 172.21.15.23:0/677677618 shutdown_connections 2024-03-24T10:48:19.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.133+0000 7fb9184f1640 1 -- 172.21.15.23:0/677677618 wait complete. 2024-03-24T10:48:19.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.133+0000 7fb9184f1640 1 Processor -- start 2024-03-24T10:48:19.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.133+0000 7fb9184f1640 1 -- start start 2024-03-24T10:48:19.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb9184f1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb910069e00 0x7fb910100640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb9184f1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb91006a7a0 0x7fb910100b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb9184f1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb910107710 0x7fb9101010c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb9184f1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fb9101a60a0 con 0x7fb910069e00 2024-03-24T10:48:19.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb9184f1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb9101a6210 con 0x7fb910107710 2024-03-24T10:48:19.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb9184f1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb9101a63d0 con 0x7fb91006a7a0 2024-03-24T10:48:19.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb91628d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb910069e00 0x7fb910100640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb91628d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb910069e00 0x7fb910100640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40238/0 (socket says 172.21.15.23:40238) 2024-03-24T10:48:19.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb91628d640 1 -- 172.21.15.23:0/4164353046 learned_addr learned my addr 172.21.15.23:0/4164353046 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb915a8c640 1 --2- 172.21.15.23:0/4164353046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb91006a7a0 0x7fb910100b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb916a8e640 1 --2- 172.21.15.23:0/4164353046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb910107710 0x7fb9101010c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb91628d640 1 -- 172.21.15.23:0/4164353046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb91006a7a0 msgr2=0x7fb910100b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb91628d640 1 --2- 172.21.15.23:0/4164353046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb91006a7a0 0x7fb910100b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb91628d640 1 -- 172.21.15.23:0/4164353046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb910107710 msgr2=0x7fb9101010c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb91628d640 1 --2- 172.21.15.23:0/4164353046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb910107710 0x7fb9101010c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb91628d640 1 -- 172.21.15.23:0/4164353046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90c009560 con 0x7fb910069e00 2024-03-24T10:48:19.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb91628d640 1 --2- 172.21.15.23:0/4164353046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb910069e00 0x7fb910100640 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fb90800db80 tx=0x7fb90800b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb9077fe640 1 -- 172.21.15.23:0/4164353046 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb908003b70 con 0x7fb910069e00 2024-03-24T10:48:19.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb9077fe640 1 -- 172.21.15.23:0/4164353046 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb90800de80 con 0x7fb910069e00 2024-03-24T10:48:19.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb9184f1640 1 -- 172.21.15.23:0/4164353046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9101a66b0 con 0x7fb910069e00 2024-03-24T10:48:19.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb9077fe640 1 -- 172.21.15.23:0/4164353046 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9080106c0 con 0x7fb910069e00 2024-03-24T10:48:19.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.134+0000 7fb9184f1640 1 -- 172.21.15.23:0/4164353046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9101a6bc0 con 0x7fb910069e00 2024-03-24T10:48:19.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.135+0000 7fb9184f1640 1 -- 172.21.15.23:0/4164353046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb910102e30 con 0x7fb910069e00 2024-03-24T10:48:19.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.136+0000 7fb9077fe640 1 -- 172.21.15.23:0/4164353046 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fb908004020 con 0x7fb910069e00 2024-03-24T10:48:19.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.137+0000 7fb9077fe640 1 --2- 172.21.15.23:0/4164353046 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb8fc0a2260 0x7fb8fc0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.137+0000 7fb915a8c640 1 --2- 172.21.15.23:0/4164353046 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb8fc0a2260 0x7fb8fc0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.137+0000 7fb9077fe640 1 -- 172.21.15.23:0/4164353046 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7fb9080cdfd0 con 0x7fb910069e00 2024-03-24T10:48:19.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.137+0000 7fb915a8c640 1 --2- 172.21.15.23:0/4164353046 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb8fc0a2260 0x7fb8fc0a4720 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb910102080 tx=0x7fb900005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.141+0000 7fb9077fe640 1 -- 172.21.15.23:0/4164353046 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb908096c30 con 0x7fb910069e00 2024-03-24T10:48:19.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.365+0000 7fb9184f1640 1 -- 172.21.15.23:0/4164353046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb910109c50 con 0x7fb910069e00 2024-03-24T10:48:19.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.366+0000 7fb9077fe640 1 -- 172.21.15.23:0/4164353046 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11869 (secure 0 0 0) 0x7fb90800be60 con 0x7fb910069e00 2024-03-24T10:48:19.366 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:48:19.367 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:47:17.474711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27526":{"gid":27526,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/2017557046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":2017557046},{"type":"v1","addr":"172.21.15.117:6837","nonce":2017557046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:48:19.367 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 31 2024-03-24T10:48:19.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.368+0000 7fb9184f1640 1 -- 172.21.15.23:0/4164353046 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb8fc0a2260 msgr2=0x7fb8fc0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.368+0000 7fb9184f1640 1 --2- 172.21.15.23:0/4164353046 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb8fc0a2260 0x7fb8fc0a4720 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb910102080 tx=0x7fb900005f90 comp rx=0 tx=0).stop 2024-03-24T10:48:19.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.368+0000 7fb9184f1640 1 -- 172.21.15.23:0/4164353046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb910069e00 msgr2=0x7fb910100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.368+0000 7fb9184f1640 1 --2- 172.21.15.23:0/4164353046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb910069e00 0x7fb910100640 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fb90800db80 tx=0x7fb90800b630 comp rx=0 tx=0).stop 2024-03-24T10:48:19.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.368+0000 7fb9184f1640 1 -- 172.21.15.23:0/4164353046 shutdown_connections 2024-03-24T10:48:19.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.368+0000 7fb9184f1640 1 --2- 172.21.15.23:0/4164353046 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fb8fc0a2260 0x7fb8fc0a4720 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.368+0000 7fb9184f1640 1 --2- 172.21.15.23:0/4164353046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb910107710 0x7fb9101010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.368+0000 7fb9184f1640 1 --2- 172.21.15.23:0/4164353046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb91006a7a0 0x7fb910100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.368+0000 7fb9184f1640 1 --2- 172.21.15.23:0/4164353046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb910069e00 0x7fb910100640 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.368+0000 7fb9184f1640 1 -- 172.21.15.23:0/4164353046 >> 172.21.15.23:0/4164353046 conn(0x7fb9100776b0 msgr2=0x7fb9100ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.368+0000 7fb9184f1640 1 -- 172.21.15.23:0/4164353046 shutdown_connections 2024-03-24T10:48:19.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.368+0000 7fb9184f1640 1 -- 172.21.15.23:0/4164353046 wait complete. 2024-03-24T10:48:19.381 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:48:19.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.498+0000 7f0448b56640 1 -- 172.21.15.23:0/128722240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0444106660 msgr2=0x7f0444106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.498+0000 7f0448b56640 1 --2- 172.21.15.23:0/128722240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0444106660 0x7f0444106a40 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f0434009900 tx=0x7f043402bc30 comp rx=0 tx=0).stop 2024-03-24T10:48:19.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.499+0000 7f0448b56640 1 -- 172.21.15.23:0/128722240 shutdown_connections 2024-03-24T10:48:19.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.499+0000 7f0448b56640 1 --2- 172.21.15.23:0/128722240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0444100f60 0x7f044410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.499+0000 7f0448b56640 1 --2- 172.21.15.23:0/128722240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04441005a0 0x7f0444100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.499+0000 7f0448b56640 1 --2- 172.21.15.23:0/128722240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0444106660 0x7f0444106a40 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.499+0000 7f0448b56640 1 -- 172.21.15.23:0/128722240 >> 172.21.15.23:0/128722240 conn(0x7f0444077710 msgr2=0x7f0444077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.499+0000 7f0448b56640 1 -- 172.21.15.23:0/128722240 shutdown_connections 2024-03-24T10:48:19.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.499+0000 7f0448b56640 1 -- 172.21.15.23:0/128722240 wait complete. 2024-03-24T10:48:19.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.499+0000 7f0448b56640 1 Processor -- start 2024-03-24T10:48:19.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.499+0000 7f0448b56640 1 -- start start 2024-03-24T10:48:19.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0448b56640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04441005a0 0x7f044419d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0448b56640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0444100f60 0x7f044419de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0448b56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0444106660 0x7f0444197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0448b56640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0444198130 con 0x7f0444100f60 2024-03-24T10:48:19.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0448b56640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f04441982a0 con 0x7f0444106660 2024-03-24T10:48:19.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0448b56640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0444198460 con 0x7f04441005a0 2024-03-24T10:48:19.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0441d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0444100f60 0x7f044419de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0441d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0444100f60 0x7f044419de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40262/0 (socket says 172.21.15.23:40262) 2024-03-24T10:48:19.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0441d9b640 1 -- 172.21.15.23:0/108215746 learned_addr learned my addr 172.21.15.23:0/108215746 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f044259c640 1 --2- 172.21.15.23:0/108215746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04441005a0 0x7f044419d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0442d9d640 1 --2- 172.21.15.23:0/108215746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0444106660 0x7f0444197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0441d9b640 1 -- 172.21.15.23:0/108215746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04441005a0 msgr2=0x7f044419d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0441d9b640 1 --2- 172.21.15.23:0/108215746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04441005a0 0x7f044419d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0441d9b640 1 -- 172.21.15.23:0/108215746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0444106660 msgr2=0x7f0444197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0441d9b640 1 --2- 172.21.15.23:0/108215746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0444106660 0x7f0444197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0441d9b640 1 -- 172.21.15.23:0/108215746 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0434009560 con 0x7f0444100f60 2024-03-24T10:48:19.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0441d9b640 1 --2- 172.21.15.23:0/108215746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0444100f60 0x7f044419de70 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f043800e9f0 tx=0x7f043800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f04337fe640 1 -- 172.21.15.23:0/108215746 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0438009720 con 0x7f0444100f60 2024-03-24T10:48:19.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f04337fe640 1 -- 172.21.15.23:0/108215746 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f043800cd80 con 0x7f0444100f60 2024-03-24T10:48:19.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0448b56640 1 -- 172.21.15.23:0/108215746 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0444198740 con 0x7f0444100f60 2024-03-24T10:48:19.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f04337fe640 1 -- 172.21.15.23:0/108215746 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0438010690 con 0x7f0444100f60 2024-03-24T10:48:19.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.500+0000 7f0448b56640 1 -- 172.21.15.23:0/108215746 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04441a49d0 con 0x7f0444100f60 2024-03-24T10:48:19.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.503+0000 7f04337fe640 1 -- 172.21.15.23:0/108215746 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f04380046c0 con 0x7f0444100f60 2024-03-24T10:48:19.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.503+0000 7f044b811740 1 Processor -- start 2024-03-24T10:48:19.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.503+0000 7f044b811740 1 -- start start 2024-03-24T10:48:19.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.503+0000 7f04337fe640 1 --2- 172.21.15.23:0/108215746 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f042c0a22b0 0x7f042c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.503+0000 7f044b811740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e03f0 0x5561ef5e07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.503+0000 7f044b811740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 0x5561ef5e93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.503+0000 7f044b811740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 0x5561ef5ebce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.503+0000 7f044b811740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5561ef54e840 con 0x5561ef5e98f0 2024-03-24T10:48:19.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.503+0000 7f044b811740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5561ef54bd70 con 0x5561ef5e03f0 2024-03-24T10:48:19.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.503+0000 7f044b811740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5561ef547550 con 0x5561ef5e0d10 2024-03-24T10:48:19.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f0442d9d640 1 --2- 172.21.15.23:0/108215746 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f042c0a22b0 0x7f042c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f04337fe640 1 -- 172.21.15.23:0/108215746 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f0438014070 con 0x7f0444100f60 2024-03-24T10:48:19.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f044259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e03f0 0x5561ef5e07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f0441d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 0x5561ef5e93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f044259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e03f0 0x5561ef5e07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:50632/0 (socket says 172.21.15.23:50632) 2024-03-24T10:48:19.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f044259c640 1 -- 172.21.15.23:0/851669314 learned_addr learned my addr 172.21.15.23:0/851669314 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f0442d9d640 1 --2- 172.21.15.23:0/851669314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 0x5561ef5ebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f044259c640 1 -- 172.21.15.23:0/851669314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 msgr2=0x5561ef5e93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f044259c640 1 --2- 172.21.15.23:0/851669314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 0x5561ef5e93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f044259c640 1 -- 172.21.15.23:0/851669314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 msgr2=0x5561ef5ebce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f044259c640 1 --2- 172.21.15.23:0/851669314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 0x5561ef5ebce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f044259c640 1 -- 172.21.15.23:0/851669314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561ef4dcde0 con 0x5561ef5e03f0 2024-03-24T10:48:19.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f0442d9d640 1 --2- 172.21.15.23:0/108215746 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f042c0a22b0 0x7f042c0a4770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f043c005fd0 tx=0x7f043c009f10 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f044259c640 1 --2- 172.21.15.23:0/851669314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e03f0 0x5561ef5e07d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0434000c00 tx=0x7f043402c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a95edbea8013c33 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f0430ff9640 1 -- 172.21.15.23:0/851669314 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043402dea0 con 0x5561ef5e03f0 2024-03-24T10:48:19.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f0430ff9640 1 -- 172.21.15.23:0/851669314 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0434004800 con 0x5561ef5e03f0 2024-03-24T10:48:19.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.504+0000 7f0430ff9640 1 -- 172.21.15.23:0/851669314 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0434005050 con 0x5561ef5e03f0 2024-03-24T10:48:19.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.505+0000 7f044b811740 1 -- 172.21.15.23:0/851669314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e03f0 msgr2=0x5561ef5e07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.505+0000 7f044b811740 1 --2- 172.21.15.23:0/851669314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e03f0 0x5561ef5e07d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0434000c00 tx=0x7f043402c670 comp rx=0 tx=0).stop 2024-03-24T10:48:19.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.505+0000 7f044b811740 1 -- 172.21.15.23:0/851669314 shutdown_connections 2024-03-24T10:48:19.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.505+0000 7f044b811740 1 --2- 172.21.15.23:0/851669314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 0x5561ef5ebce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.505+0000 7f044b811740 1 --2- 172.21.15.23:0/851669314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 0x5561ef5e93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.505+0000 7f044b811740 1 --2- 172.21.15.23:0/851669314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e03f0 0x5561ef5e07d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.505+0000 7f044b811740 1 -- 172.21.15.23:0/851669314 >> 172.21.15.23:0/851669314 conn(0x5561ef5dab20 msgr2=0x5561ef5daf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.505+0000 7f044b811740 1 -- 172.21.15.23:0/851669314 shutdown_connections 2024-03-24T10:48:19.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.505+0000 7f044b811740 1 -- 172.21.15.23:0/851669314 wait complete. 2024-03-24T10:48:19.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.505+0000 7f044b811740 1 -- start start 2024-03-24T10:48:19.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044b811740 1 Processor -- start 2024-03-24T10:48:19.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044b811740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e03f0 0x5561ef5e6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044b811740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e0d10 0x5561ef5e6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044b811740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 0x5561ef5e7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044b811740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5561ef21a430 con 0x5561ef5e98f0 2024-03-24T10:48:19.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044b811740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5561ef1d6e00 con 0x5561ef5e0d10 2024-03-24T10:48:19.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044b811740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5561ef4cc590 con 0x5561ef5e03f0 2024-03-24T10:48:19.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044b811740 10 client.0 _ll_register_callbacks cb 0x5561ef53bfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:19.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e03f0 0x5561ef5e6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e03f0 0x5561ef5e6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:58946/0 (socket says 172.21.15.23:58946) 2024-03-24T10:48:19.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f0442d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 0x5561ef5e7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044259c640 1 -- 172.21.15.23:0/3729141828 learned_addr learned my addr 172.21.15.23:0/3729141828 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f0441d9b640 1 --2- 172.21.15.23:0/3729141828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e0d10 0x5561ef5e6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044259c640 1 -- 172.21.15.23:0/3729141828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e0d10 msgr2=0x5561ef5e6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044259c640 1 --2- 172.21.15.23:0/3729141828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e0d10 0x5561ef5e6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044259c640 1 -- 172.21.15.23:0/3729141828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 msgr2=0x5561ef5e7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044259c640 1 --2- 172.21.15.23:0/3729141828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 0x5561ef5e7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044259c640 1 -- 172.21.15.23:0/3729141828 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f044410dd20 con 0x5561ef5e03f0 2024-03-24T10:48:19.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f044259c640 1 --2- 172.21.15.23:0/3729141828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e03f0 0x5561ef5e6980 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0434005ab0 tx=0x7f043402ca40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48f8e2b9f3f80937 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.506+0000 7f0429ffb640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:48:19.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f0429ffb640 1 -- 172.21.15.23:0/3729141828 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0434004490 con 0x5561ef5e03f0 2024-03-24T10:48:19.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f0429ffb640 1 -- 172.21.15.23:0/3729141828 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f0434004df0 con 0x5561ef5e03f0 2024-03-24T10:48:19.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f044b811740 1 -- 172.21.15.23:0/3729141828 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5561ef2467e0 con 0x5561ef5e03f0 2024-03-24T10:48:19.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f0429ffb640 1 -- 172.21.15.23:0/3729141828 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0434004fb0 con 0x5561ef5e03f0 2024-03-24T10:48:19.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f044b811740 10 client.27748 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:19.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f044b811740 1 -- 172.21.15.23:0/3729141828 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5561ef545bd0 con 0x5561ef5e03f0 2024-03-24T10:48:19.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f0429ffb640 1 -- 172.21.15.23:0/3729141828 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5561ef2467e0 con 0x5561ef5e03f0 2024-03-24T10:48:19.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f0429ffb640 1 -- 172.21.15.23:0/3729141828 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f043403c050 con 0x5561ef5e03f0 2024-03-24T10:48:19.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f044b811740 10 client.27748 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:19.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f044b811740 10 client.27748 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:48:19.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f044b811740 20 client.27748 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:19.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f044b811740 1 --2- 172.21.15.23:0/3729141828 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5561ef5f96b0 0x5561ef5fbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f044b811740 4 client.27748 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:48:19.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f044b811740 1 -- 172.21.15.23:0/3729141828 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561ef283720 con 0x5561ef5f96b0 2024-03-24T10:48:19.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.507+0000 7f0441d9b640 1 --2- 172.21.15.23:0/3729141828 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5561ef5f96b0 0x5561ef5fbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f0429ffb640 10 client.27748 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:48:19.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f0441d9b640 1 --2- 172.21.15.23:0/3729141828 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5561ef5f96b0 0x5561ef5fbaf0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=78bca179ed29df5b server_cookie=b7aa2f5e2d71d667 in_seq=0 out_seq=0 2024-03-24T10:48:19.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f0429ffb640 1 -- 172.21.15.23:0/3729141828 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5561ef283720 con 0x5561ef5f96b0 2024-03-24T10:48:19.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f0429ffb640 10 client.27748 handle_command_reply: tid=0 2024-03-24T10:48:19.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f044b811740 1 client.27748 shutdown 2024-03-24T10:48:19.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f044b811740 1 -- 172.21.15.23:0/3729141828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e03f0 msgr2=0x5561ef5e6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f044b811740 1 --2- 172.21.15.23:0/3729141828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e03f0 0x5561ef5e6980 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0434005ab0 tx=0x7f043402ca40 comp rx=0 tx=0).stop 2024-03-24T10:48:19.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f044b811740 1 -- 172.21.15.23:0/3729141828 shutdown_connections 2024-03-24T10:48:19.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f044b811740 1 --2- 172.21.15.23:0/3729141828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 0x5561ef5e7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f044b811740 1 --2- 172.21.15.23:0/3729141828 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5561ef5f96b0 0x5561ef5fbaf0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f044b811740 1 --2- 172.21.15.23:0/3729141828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e0d10 0x5561ef5e6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f044b811740 1 --2- 172.21.15.23:0/3729141828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e03f0 0x5561ef5e6980 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f0429ffb640 0 client.27748 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:48:19.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f044b811740 1 -- 172.21.15.23:0/3729141828 >> 172.21.15.23:0/3729141828 conn(0x5561ef5dab20 msgr2=0x5561ef5dc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f044b811740 1 -- 172.21.15.23:0/3729141828 shutdown_connections 2024-03-24T10:48:19.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.508+0000 7f044b811740 1 -- 172.21.15.23:0/3729141828 wait complete. 2024-03-24T10:48:19.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.509+0000 7f044b811740 20 client.27748 trim_cache size 0 max 16384 2024-03-24T10:48:19.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f044b811740 1 Processor -- start 2024-03-24T10:48:19.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f044b811740 1 -- start start 2024-03-24T10:48:19.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f044b811740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e03f0 0x5561ef5f5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f044b811740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 0x5561ef5d9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f044b811740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 0x5561ef6013c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f044b811740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5561ef5e0050 con 0x5561ef5e98f0 2024-03-24T10:48:19.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f044b811740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5561ef5e6770 con 0x5561ef5e03f0 2024-03-24T10:48:19.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f044b811740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5561ef5da650 con 0x5561ef5e0d10 2024-03-24T10:48:19.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f0442d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 0x5561ef6013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f0442d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 0x5561ef6013c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40284/0 (socket says 172.21.15.23:40284) 2024-03-24T10:48:19.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f0442d9d640 1 -- 172.21.15.23:0/4033920634 learned_addr learned my addr 172.21.15.23:0/4033920634 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f044259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e03f0 0x5561ef5f5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f0441d9b640 1 --2- 172.21.15.23:0/4033920634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 0x5561ef5d9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f0442d9d640 1 -- 172.21.15.23:0/4033920634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 msgr2=0x5561ef5d9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f0442d9d640 1 --2- 172.21.15.23:0/4033920634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 0x5561ef5d9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f0442d9d640 1 -- 172.21.15.23:0/4033920634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e03f0 msgr2=0x5561ef5f5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f0442d9d640 1 --2- 172.21.15.23:0/4033920634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e03f0 0x5561ef5f5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f0442d9d640 1 -- 172.21.15.23:0/4033920634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561ef4e1c80 con 0x5561ef5e98f0 2024-03-24T10:48:19.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f044259c640 1 --2- 172.21.15.23:0/4033920634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e03f0 0x5561ef5f5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:48:19.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f0442d9d640 1 --2- 172.21.15.23:0/4033920634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 0x5561ef6013c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f043c005c00 tx=0x7f043c021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d360ef9bb4e8b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.522+0000 7f042bfff640 1 -- 172.21.15.23:0/4033920634 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c016040 con 0x5561ef5e98f0 2024-03-24T10:48:19.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.523+0000 7f042bfff640 1 -- 172.21.15.23:0/4033920634 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f043c01ed90 con 0x5561ef5e98f0 2024-03-24T10:48:19.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.523+0000 7f042bfff640 1 -- 172.21.15.23:0/4033920634 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c0203f0 con 0x5561ef5e98f0 2024-03-24T10:48:19.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.523+0000 7f044b811740 1 -- 172.21.15.23:0/4033920634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 msgr2=0x5561ef6013c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.523+0000 7f044b811740 1 --2- 172.21.15.23:0/4033920634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 0x5561ef6013c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f043c005c00 tx=0x7f043c021000 comp rx=0 tx=0).stop 2024-03-24T10:48:19.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.523+0000 7f044b811740 1 -- 172.21.15.23:0/4033920634 shutdown_connections 2024-03-24T10:48:19.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.523+0000 7f044b811740 1 --2- 172.21.15.23:0/4033920634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e98f0 0x5561ef6013c0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.523+0000 7f044b811740 1 --2- 172.21.15.23:0/4033920634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 0x5561ef5d9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.523+0000 7f044b811740 1 --2- 172.21.15.23:0/4033920634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e03f0 0x5561ef5f5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.523+0000 7f044b811740 1 -- 172.21.15.23:0/4033920634 >> 172.21.15.23:0/4033920634 conn(0x5561ef5dab20 msgr2=0x5561ef5d9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.523+0000 7f044b811740 1 -- 172.21.15.23:0/4033920634 shutdown_connections 2024-03-24T10:48:19.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.523+0000 7f044b811740 1 -- 172.21.15.23:0/4033920634 wait complete. 2024-03-24T10:48:19.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.523+0000 7f044b811740 1 -- start start 2024-03-24T10:48:19.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.523+0000 7f044b811740 1 Processor -- start 2024-03-24T10:48:19.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044b811740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e03f0 0x5561ef5f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044b811740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 0x5561ef5ecd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044b811740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e98f0 0x5561ef5ed270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044b811740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5561ef5ed950 con 0x5561ef5e03f0 2024-03-24T10:48:19.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044b811740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5561ef5edac0 con 0x5561ef5e98f0 2024-03-24T10:48:19.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044b811740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5561ef65e9f0 con 0x5561ef5e0d10 2024-03-24T10:48:19.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044b811740 10 client.0 _ll_register_callbacks cb 0x7f043c0117a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:19.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e03f0 0x5561ef5f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e03f0 0x5561ef5f19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40300/0 (socket says 172.21.15.23:40300) 2024-03-24T10:48:19.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044259c640 1 -- 172.21.15.23:0/2782062591 learned_addr learned my addr 172.21.15.23:0/2782062591 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f0441d9b640 1 --2- 172.21.15.23:0/2782062591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 0x5561ef5ecd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f0442d9d640 1 --2- 172.21.15.23:0/2782062591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e98f0 0x5561ef5ed270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044259c640 1 -- 172.21.15.23:0/2782062591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 msgr2=0x5561ef5ecd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044259c640 1 --2- 172.21.15.23:0/2782062591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 0x5561ef5ecd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044259c640 1 -- 172.21.15.23:0/2782062591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e98f0 msgr2=0x5561ef5ed270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044259c640 1 --2- 172.21.15.23:0/2782062591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e98f0 0x5561ef5ed270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044259c640 1 -- 172.21.15.23:0/2782062591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f043c019040 con 0x5561ef5e03f0 2024-03-24T10:48:19.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f0442d9d640 1 --2- 172.21.15.23:0/2782062591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e98f0 0x5561ef5ed270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:48:19.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044259c640 1 --2- 172.21.15.23:0/2782062591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e03f0 0x5561ef5f19c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f0434004380 tx=0x7f043402cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf0c06a7e0e4b5b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f0429ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:19.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f0429ffb640 1 -- 172.21.15.23:0/2782062591 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0434004420 con 0x5561ef5e03f0 2024-03-24T10:48:19.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f0429ffb640 1 -- 172.21.15.23:0/2782062591 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f0434000e40 con 0x5561ef5e03f0 2024-03-24T10:48:19.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044b811740 1 -- 172.21.15.23:0/2782062591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0434004800 con 0x5561ef5e03f0 2024-03-24T10:48:19.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f0429ffb640 1 -- 172.21.15.23:0/2782062591 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f04340550b0 con 0x5561ef5e03f0 2024-03-24T10:48:19.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044b811740 10 client.17898 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:19.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.524+0000 7f044b811740 1 -- 172.21.15.23:0/2782062591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5561ef507920 con 0x5561ef5e03f0 2024-03-24T10:48:19.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.525+0000 7f0429ffb640 1 -- 172.21.15.23:0/2782062591 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0434004800 con 0x5561ef5e03f0 2024-03-24T10:48:19.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.525+0000 7f0429ffb640 1 -- 172.21.15.23:0/2782062591 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f043403c040 con 0x5561ef5e03f0 2024-03-24T10:48:19.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.525+0000 7f044b811740 10 client.17898 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:19.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.525+0000 7f044b811740 10 client.17898 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:48:19.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.525+0000 7f044b811740 20 client.17898 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:19.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.525+0000 7f044b811740 1 --2- 172.21.15.23:0/2782062591 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5561ef65fd10 0x5561ef662100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.525+0000 7f044b811740 4 client.17898 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:48:19.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.525+0000 7f044b811740 1 -- 172.21.15.23:0/2782062591 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5561ef2d2890 con 0x5561ef65fd10 2024-03-24T10:48:19.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.525+0000 7f0441d9b640 1 --2- 172.21.15.23:0/2782062591 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5561ef65fd10 0x5561ef662100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.525+0000 7f0429ffb640 10 client.17898 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:48:19.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.525+0000 7f0441d9b640 1 --2- 172.21.15.23:0/2782062591 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5561ef65fd10 0x5561ef662100 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=32a3f11687c2eb3f server_cookie=61490959fee76e0d in_seq=0 out_seq=0 2024-03-24T10:48:19.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.525+0000 7f0429ffb640 1 -- 172.21.15.23:0/2782062591 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5561ef2d2890 con 0x5561ef65fd10 2024-03-24T10:48:19.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.525+0000 7f0429ffb640 10 client.17898 handle_command_reply: tid=0 2024-03-24T10:48:19.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.525+0000 7f044b811740 1 client.17898 shutdown 2024-03-24T10:48:19.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f044b811740 1 -- 172.21.15.23:0/2782062591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e03f0 msgr2=0x5561ef5f19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f044b811740 1 --2- 172.21.15.23:0/2782062591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e03f0 0x5561ef5f19c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f0434004380 tx=0x7f043402cb40 comp rx=0 tx=0).stop 2024-03-24T10:48:19.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f044b811740 1 -- 172.21.15.23:0/2782062591 shutdown_connections 2024-03-24T10:48:19.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f044b811740 1 --2- 172.21.15.23:0/2782062591 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5561ef65fd10 0x5561ef662100 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f044b811740 1 --2- 172.21.15.23:0/2782062591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561ef5e98f0 0x5561ef5ed270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f0429ffb640 0 client.17898 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:48:19.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f044b811740 1 --2- 172.21.15.23:0/2782062591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5561ef5e0d10 0x5561ef5ecd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f044b811740 1 --2- 172.21.15.23:0/2782062591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561ef5e03f0 0x5561ef5f19c0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f044b811740 1 -- 172.21.15.23:0/2782062591 >> 172.21.15.23:0/2782062591 conn(0x5561ef5dab20 msgr2=0x5561ef5e7190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f044b811740 1 -- 172.21.15.23:0/2782062591 shutdown_connections 2024-03-24T10:48:19.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f044b811740 1 -- 172.21.15.23:0/2782062591 wait complete. 2024-03-24T10:48:19.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f044b811740 20 client.17898 trim_cache size 0 max 16384 2024-03-24T10:48:19.537 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:48:19.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f0448b56640 1 -- 172.21.15.23:0/108215746 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f042c0a22b0 msgr2=0x7f042c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f0448b56640 1 --2- 172.21.15.23:0/108215746 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f042c0a22b0 0x7f042c0a4770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f043c005fd0 tx=0x7f043c009f10 comp rx=0 tx=0).stop 2024-03-24T10:48:19.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f0448b56640 1 -- 172.21.15.23:0/108215746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0444100f60 msgr2=0x7f044419de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f0448b56640 1 --2- 172.21.15.23:0/108215746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0444100f60 0x7f044419de70 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f043800e9f0 tx=0x7f043800edf0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f0448b56640 1 -- 172.21.15.23:0/108215746 shutdown_connections 2024-03-24T10:48:19.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f0448b56640 1 --2- 172.21.15.23:0/108215746 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f042c0a22b0 0x7f042c0a4770 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f0448b56640 1 --2- 172.21.15.23:0/108215746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0444106660 0x7f0444197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f0448b56640 1 --2- 172.21.15.23:0/108215746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0444100f60 0x7f044419de70 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f0448b56640 1 --2- 172.21.15.23:0/108215746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04441005a0 0x7f044419d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f0448b56640 1 -- 172.21.15.23:0/108215746 >> 172.21.15.23:0/108215746 conn(0x7f0444077710 msgr2=0x7f04440ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f0448b56640 1 -- 172.21.15.23:0/108215746 shutdown_connections 2024-03-24T10:48:19.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.526+0000 7f0448b56640 1 -- 172.21.15.23:0/108215746 wait complete. 2024-03-24T10:48:19.540 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:48:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.645+0000 7f0a7e457640 1 -- 172.21.15.23:0/2543013528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a78107910 msgr2=0x7f0a78107cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.645+0000 7f0a7e457640 1 --2- 172.21.15.23:0/2543013528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a78107910 0x7f0a78107cf0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009900 tx=0x7f0a6802bc30 comp rx=0 tx=0).stop 2024-03-24T10:48:19.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.645+0000 7f0a7e457640 1 -- 172.21.15.23:0/2543013528 shutdown_connections 2024-03-24T10:48:19.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.645+0000 7f0a7e457640 1 --2- 172.21.15.23:0/2543013528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a78100990 0x7f0a7810e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.645+0000 7f0a7e457640 1 --2- 172.21.15.23:0/2543013528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a780fffd0 0x7f0a78100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.645+0000 7f0a7e457640 1 --2- 172.21.15.23:0/2543013528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a78107910 0x7f0a78107cf0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.645+0000 7f0a7e457640 1 -- 172.21.15.23:0/2543013528 >> 172.21.15.23:0/2543013528 conn(0x7f0a78075950 msgr2=0x7f0a78075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.645+0000 7f0a7e457640 1 -- 172.21.15.23:0/2543013528 shutdown_connections 2024-03-24T10:48:19.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.645+0000 7f0a7e457640 1 -- 172.21.15.23:0/2543013528 wait complete. 2024-03-24T10:48:19.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.646+0000 7f0a7e457640 1 Processor -- start 2024-03-24T10:48:19.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.646+0000 7f0a7e457640 1 -- start start 2024-03-24T10:48:19.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.646+0000 7f0a7e457640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a780fffd0 0x7f0a78199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.646+0000 7f0a7e457640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a78100990 0x7f0a7819a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.646+0000 7f0a7e457640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a78107910 0x7f0a7819e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.646+0000 7f0a7e457640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7819ee00 con 0x7f0a78100990 2024-03-24T10:48:19.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.646+0000 7f0a7e457640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7819ef70 con 0x7f0a780fffd0 2024-03-24T10:48:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.646+0000 7f0a7e457640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7819f130 con 0x7f0a78107910 2024-03-24T10:48:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.646+0000 7f0a777fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a78100990 0x7f0a7819a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.646+0000 7f0a777fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a78100990 0x7f0a7819a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40316/0 (socket says 172.21.15.23:40316) 2024-03-24T10:48:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.646+0000 7f0a777fe640 1 -- 172.21.15.23:0/1320090265 learned_addr learned my addr 172.21.15.23:0/1320090265 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.646+0000 7f0a77fff640 1 --2- 172.21.15.23:0/1320090265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a780fffd0 0x7f0a78199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.646+0000 7f0a7c9f4640 1 --2- 172.21.15.23:0/1320090265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a78107910 0x7f0a7819e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.647+0000 7f0a777fe640 1 -- 172.21.15.23:0/1320090265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a78107910 msgr2=0x7f0a7819e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.647+0000 7f0a777fe640 1 --2- 172.21.15.23:0/1320090265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a78107910 0x7f0a7819e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.647+0000 7f0a777fe640 1 -- 172.21.15.23:0/1320090265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a780fffd0 msgr2=0x7f0a78199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.647+0000 7f0a777fe640 1 --2- 172.21.15.23:0/1320090265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a780fffd0 0x7f0a78199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.647+0000 7f0a777fe640 1 -- 172.21.15.23:0/1320090265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a68009560 con 0x7f0a78100990 2024-03-24T10:48:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.647+0000 7f0a777fe640 1 --2- 172.21.15.23:0/1320090265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a78100990 0x7f0a7819a480 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00e9f0 tx=0x7f0a6c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.647+0000 7f0a757fa640 1 -- 172.21.15.23:0/1320090265 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c009700 con 0x7f0a78100990 2024-03-24T10:48:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.647+0000 7f0a757fa640 1 -- 172.21.15.23:0/1320090265 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a6c00cd80 con 0x7f0a78100990 2024-03-24T10:48:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.647+0000 7f0a7e457640 1 -- 172.21.15.23:0/1320090265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a7819f410 con 0x7f0a78100990 2024-03-24T10:48:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.647+0000 7f0a757fa640 1 -- 172.21.15.23:0/1320090265 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c018920 con 0x7f0a78100990 2024-03-24T10:48:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.647+0000 7f0a7e457640 1 -- 172.21.15.23:0/1320090265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a781a6c90 con 0x7f0a78100990 2024-03-24T10:48:19.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.649+0000 7f0a757fa640 1 -- 172.21.15.23:0/1320090265 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f0a6c018a80 con 0x7f0a78100990 2024-03-24T10:48:19.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.649+0000 7f0a81031740 1 Processor -- start 2024-03-24T10:48:19.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.649+0000 7f0a81031740 1 -- start start 2024-03-24T10:48:19.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.649+0000 7f0a81031740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257edf97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a81031740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257edf9d10 0x56257ee023b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a81031740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257ee028f0 0x56257ee04ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a757fa640 1 --2- 172.21.15.23:0/1320090265 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a600a2230 0x7f0a600a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a81031740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56257ed67840 con 0x56257ee028f0 2024-03-24T10:48:19.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a81031740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56257ed64d70 con 0x56257edf93f0 2024-03-24T10:48:19.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a81031740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56257ed60550 con 0x56257edf9d10 2024-03-24T10:48:19.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a7c9f4640 1 --2- 172.21.15.23:0/1320090265 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a600a2230 0x7f0a600a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a777fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257ee028f0 0x56257ee04ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a7c9f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257edf97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a77fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257edf9d10 0x56257ee023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a7c9f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257edf97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:50696/0 (socket says 172.21.15.23:50696) 2024-03-24T10:48:19.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a7c9f4640 1 -- 172.21.15.23:0/2381629047 learned_addr learned my addr 172.21.15.23:0/2381629047 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a757fa640 1 -- 172.21.15.23:0/1320090265 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f0a6c014070 con 0x7f0a78100990 2024-03-24T10:48:19.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a7c9f4640 1 -- 172.21.15.23:0/2381629047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257edf9d10 msgr2=0x56257ee023b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a7c9f4640 1 --2- 172.21.15.23:0/2381629047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257edf9d10 0x56257ee023b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a7c9f4640 1 -- 172.21.15.23:0/2381629047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257ee028f0 msgr2=0x56257ee04ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a7c9f4640 1 --2- 172.21.15.23:0/2381629047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257ee028f0 0x56257ee04ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a7c9f4640 1 -- 172.21.15.23:0/2381629047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56257ecf5de0 con 0x56257edf93f0 2024-03-24T10:48:19.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a77fff640 1 --2- 172.21.15.23:0/2381629047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257edf9d10 0x56257ee023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:19.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a7c9f4640 1 --2- 172.21.15.23:0/1320090265 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a600a2230 0x7f0a600a46f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0a7000fc90 tx=0x7f0a70018000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a7c9f4640 1 --2- 172.21.15.23:0/2381629047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257edf97d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0a700049d0 tx=0x7f0a7000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8da5bc93c16a59f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.650+0000 7f0a5e7fc640 1 -- 172.21.15.23:0/2381629047 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a70010730 con 0x56257edf93f0 2024-03-24T10:48:19.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.651+0000 7f0a5e7fc640 1 -- 172.21.15.23:0/2381629047 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a70009e30 con 0x56257edf93f0 2024-03-24T10:48:19.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.651+0000 7f0a5e7fc640 1 -- 172.21.15.23:0/2381629047 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a700253a0 con 0x56257edf93f0 2024-03-24T10:48:19.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.651+0000 7f0a81031740 1 -- 172.21.15.23:0/2381629047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 msgr2=0x56257edf97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.651+0000 7f0a81031740 1 --2- 172.21.15.23:0/2381629047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257edf97d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0a700049d0 tx=0x7f0a7000f790 comp rx=0 tx=0).stop 2024-03-24T10:48:19.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.651+0000 7f0a81031740 1 -- 172.21.15.23:0/2381629047 shutdown_connections 2024-03-24T10:48:19.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.651+0000 7f0a81031740 1 --2- 172.21.15.23:0/2381629047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257ee028f0 0x56257ee04ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.651+0000 7f0a81031740 1 --2- 172.21.15.23:0/2381629047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257edf9d10 0x56257ee023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.651+0000 7f0a81031740 1 --2- 172.21.15.23:0/2381629047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257edf97d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.651+0000 7f0a81031740 1 -- 172.21.15.23:0/2381629047 >> 172.21.15.23:0/2381629047 conn(0x56257edf3b20 msgr2=0x56257edf3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.651+0000 7f0a81031740 1 -- 172.21.15.23:0/2381629047 shutdown_connections 2024-03-24T10:48:19.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.651+0000 7f0a81031740 1 -- 172.21.15.23:0/2381629047 wait complete. 2024-03-24T10:48:19.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a81031740 1 -- start start 2024-03-24T10:48:19.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a81031740 1 Processor -- start 2024-03-24T10:48:19.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a81031740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257edff9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a81031740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257edf9d10 0x56257edffee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a81031740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257ee028f0 0x56257ee00420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a81031740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56257ea33430 con 0x56257edf9d10 2024-03-24T10:48:19.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a81031740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56257e9efe00 con 0x56257edf93f0 2024-03-24T10:48:19.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a81031740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56257ece5590 con 0x56257ee028f0 2024-03-24T10:48:19.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a81031740 10 client.0 _ll_register_callbacks cb 0x56257ed54fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:19.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a777fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257edf9d10 0x56257edffee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a777fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257edf9d10 0x56257edffee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40334/0 (socket says 172.21.15.23:40334) 2024-03-24T10:48:19.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a777fe640 1 -- 172.21.15.23:0/2305119913 learned_addr learned my addr 172.21.15.23:0/2305119913 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a77fff640 1 --2- 172.21.15.23:0/2305119913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257edff9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a7c9f4640 1 --2- 172.21.15.23:0/2305119913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257ee028f0 0x56257ee00420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a777fe640 1 -- 172.21.15.23:0/2305119913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257ee028f0 msgr2=0x56257ee00420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a777fe640 1 --2- 172.21.15.23:0/2305119913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257ee028f0 0x56257ee00420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a777fe640 1 -- 172.21.15.23:0/2305119913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 msgr2=0x56257edff9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a777fe640 1 --2- 172.21.15.23:0/2305119913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257edff9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a777fe640 1 -- 172.21.15.23:0/2305119913 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a70010a50 con 0x56257edf9d10 2024-03-24T10:48:19.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.652+0000 7f0a7c9f4640 1 --2- 172.21.15.23:0/2305119913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257ee028f0 0x56257ee00420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:48:19.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a777fe640 1 --2- 172.21.15.23:0/2305119913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257edf9d10 0x56257edffee0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c097300 tx=0x7f0a6c066500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802bc50b727de531 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a577fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:19.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a577fe640 1 -- 172.21.15.23:0/2305119913 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c09aea0 con 0x56257edf9d10 2024-03-24T10:48:19.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a577fe640 1 -- 172.21.15.23:0/2305119913 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f0a6c09ef40 con 0x56257edf9d10 2024-03-24T10:48:19.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a81031740 1 -- 172.21.15.23:0/2305119913 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56257ea5f7e0 con 0x56257edf9d10 2024-03-24T10:48:19.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a577fe640 1 -- 172.21.15.23:0/2305119913 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0a6c0a3590 con 0x56257edf9d10 2024-03-24T10:48:19.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a81031740 10 client.17910 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:19.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a81031740 1 -- 172.21.15.23:0/2305119913 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56257ed5ebd0 con 0x56257edf9d10 2024-03-24T10:48:19.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a577fe640 1 -- 172.21.15.23:0/2305119913 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56257ea5f7e0 con 0x56257edf9d10 2024-03-24T10:48:19.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a577fe640 1 -- 172.21.15.23:0/2305119913 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f0a6c09a2f0 con 0x56257edf9d10 2024-03-24T10:48:19.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a81031740 10 client.17910 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:19.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a81031740 10 client.17910 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:48:19.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a81031740 20 client.17910 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:19.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a81031740 1 --2- 172.21.15.23:0/2305119913 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56257ee12a80 0x56257ee14e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a81031740 4 client.17910 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:48:19.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.653+0000 7f0a81031740 1 -- 172.21.15.23:0/2305119913 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56257ea9c720 con 0x56257ee12a80 2024-03-24T10:48:19.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.654+0000 7f0a77fff640 1 --2- 172.21.15.23:0/2305119913 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56257ee12a80 0x56257ee14e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.654+0000 7f0a77fff640 1 --2- 172.21.15.23:0/2305119913 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56257ee12a80 0x56257ee14e70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eb4d40248c2afb8c server_cookie=ee215fe3e9bb2a4e in_seq=0 out_seq=0 2024-03-24T10:48:19.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.654+0000 7f0a577fe640 10 client.17910 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:48:19.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.654+0000 7f0a577fe640 1 -- 172.21.15.23:0/2305119913 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56257ea9c720 con 0x56257ee12a80 2024-03-24T10:48:19.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.654+0000 7f0a577fe640 10 client.17910 handle_command_reply: tid=0 2024-03-24T10:48:19.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.655+0000 7f0a81031740 1 client.17910 shutdown 2024-03-24T10:48:19.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.655+0000 7f0a81031740 1 -- 172.21.15.23:0/2305119913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257edf9d10 msgr2=0x56257edffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.655+0000 7f0a81031740 1 --2- 172.21.15.23:0/2305119913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257edf9d10 0x56257edffee0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c097300 tx=0x7f0a6c066500 comp rx=0 tx=0).stop 2024-03-24T10:48:19.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.655+0000 7f0a81031740 1 -- 172.21.15.23:0/2305119913 shutdown_connections 2024-03-24T10:48:19.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.655+0000 7f0a81031740 1 --2- 172.21.15.23:0/2305119913 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56257ee12a80 0x56257ee14e70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.655+0000 7f0a81031740 1 --2- 172.21.15.23:0/2305119913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257ee028f0 0x56257ee00420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.655+0000 7f0a81031740 1 --2- 172.21.15.23:0/2305119913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257edf9d10 0x56257edffee0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.655+0000 7f0a577fe640 0 client.17910 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:48:19.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.655+0000 7f0a81031740 1 --2- 172.21.15.23:0/2305119913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257edff9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.655+0000 7f0a81031740 1 -- 172.21.15.23:0/2305119913 >> 172.21.15.23:0/2305119913 conn(0x56257edf3b20 msgr2=0x56257edf5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.655+0000 7f0a81031740 1 -- 172.21.15.23:0/2305119913 shutdown_connections 2024-03-24T10:48:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.655+0000 7f0a81031740 1 -- 172.21.15.23:0/2305119913 wait complete. 2024-03-24T10:48:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.655+0000 7f0a81031740 20 client.17910 trim_cache size 0 max 16384 2024-03-24T10:48:19.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:19 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4164353046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:48:19.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.668+0000 7f0a81031740 1 Processor -- start 2024-03-24T10:48:19.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.668+0000 7f0a81031740 1 -- start start 2024-03-24T10:48:19.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a81031740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257ee0e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a81031740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257edf9d10 0x56257edf5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a81031740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257ee028f0 0x56257ee00240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a81031740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56257edfb5c0 con 0x56257ee028f0 2024-03-24T10:48:19.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a81031740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56257edf8dc0 con 0x56257edf93f0 2024-03-24T10:48:19.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a81031740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56257edf84e0 con 0x56257edf9d10 2024-03-24T10:48:19.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a7c9f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257ee028f0 0x56257ee00240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a7c9f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257ee028f0 0x56257ee00240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40348/0 (socket says 172.21.15.23:40348) 2024-03-24T10:48:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a7c9f4640 1 -- 172.21.15.23:0/3208444812 learned_addr learned my addr 172.21.15.23:0/3208444812 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a77fff640 1 --2- 172.21.15.23:0/3208444812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257ee0e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a777fe640 1 --2- 172.21.15.23:0/3208444812 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257edf9d10 0x56257edf5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a7c9f4640 1 -- 172.21.15.23:0/3208444812 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257edf9d10 msgr2=0x56257edf5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a7c9f4640 1 --2- 172.21.15.23:0/3208444812 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257edf9d10 0x56257edf5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a7c9f4640 1 -- 172.21.15.23:0/3208444812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 msgr2=0x56257ee0e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a7c9f4640 1 --2- 172.21.15.23:0/3208444812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257ee0e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a7c9f4640 1 -- 172.21.15.23:0/3208444812 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56257ecfac80 con 0x56257ee028f0 2024-03-24T10:48:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a7c9f4640 1 --2- 172.21.15.23:0/3208444812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257ee028f0 0x56257ee00240 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f0a70000f80 tx=0x7f0a70005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa1c312f9988d3e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a5dffb640 1 -- 172.21.15.23:0/3208444812 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a700253a0 con 0x56257ee028f0 2024-03-24T10:48:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a5dffb640 1 -- 172.21.15.23:0/3208444812 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a7000ed10 con 0x56257ee028f0 2024-03-24T10:48:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.669+0000 7f0a5dffb640 1 -- 172.21.15.23:0/3208444812 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a70025a30 con 0x56257ee028f0 2024-03-24T10:48:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.670+0000 7f0a81031740 1 -- 172.21.15.23:0/3208444812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257ee028f0 msgr2=0x56257ee00240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.670+0000 7f0a81031740 1 --2- 172.21.15.23:0/3208444812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257ee028f0 0x56257ee00240 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f0a70000f80 tx=0x7f0a70005c30 comp rx=0 tx=0).stop 2024-03-24T10:48:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.670+0000 7f0a81031740 1 -- 172.21.15.23:0/3208444812 shutdown_connections 2024-03-24T10:48:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.670+0000 7f0a81031740 1 --2- 172.21.15.23:0/3208444812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257ee028f0 0x56257ee00240 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.670+0000 7f0a81031740 1 --2- 172.21.15.23:0/3208444812 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257edf9d10 0x56257edf5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.670+0000 7f0a81031740 1 --2- 172.21.15.23:0/3208444812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257ee0e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.670+0000 7f0a81031740 1 -- 172.21.15.23:0/3208444812 >> 172.21.15.23:0/3208444812 conn(0x56257edf3b20 msgr2=0x56257edf4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.670+0000 7f0a81031740 1 -- 172.21.15.23:0/3208444812 shutdown_connections 2024-03-24T10:48:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.670+0000 7f0a81031740 1 -- 172.21.15.23:0/3208444812 wait complete. 2024-03-24T10:48:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.670+0000 7f0a81031740 1 -- start start 2024-03-24T10:48:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.670+0000 7f0a81031740 1 Processor -- start 2024-03-24T10:48:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a81031740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257edfc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a81031740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257edf9d10 0x56257edfc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a81031740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257ee028f0 0x56257edfce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a81031740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56257ee0afc0 con 0x56257edf9d10 2024-03-24T10:48:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a81031740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56257ee77d70 con 0x56257edf93f0 2024-03-24T10:48:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a81031740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56257ee77eb0 con 0x56257ee028f0 2024-03-24T10:48:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a81031740 10 client.0 _ll_register_callbacks cb 0x7f0a6c010e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a777fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257edf9d10 0x56257edfc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a777fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257edf9d10 0x56257edfc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40364/0 (socket says 172.21.15.23:40364) 2024-03-24T10:48:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a777fe640 1 -- 172.21.15.23:0/3397669134 learned_addr learned my addr 172.21.15.23:0/3397669134 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a77fff640 1 --2- 172.21.15.23:0/3397669134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257edfc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a7c9f4640 1 --2- 172.21.15.23:0/3397669134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257ee028f0 0x56257edfce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a777fe640 1 -- 172.21.15.23:0/3397669134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257ee028f0 msgr2=0x56257edfce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a777fe640 1 --2- 172.21.15.23:0/3397669134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257ee028f0 0x56257edfce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a777fe640 1 -- 172.21.15.23:0/3397669134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 msgr2=0x56257edfc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a777fe640 1 --2- 172.21.15.23:0/3397669134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257edfc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a777fe640 1 -- 172.21.15.23:0/3397669134 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56257ecf5de0 con 0x56257edf9d10 2024-03-24T10:48:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a777fe640 1 --2- 172.21.15.23:0/3397669134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257edf9d10 0x56257edfc950 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0cc150 tx=0x7f0a6c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde3fe560f119f12 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a577fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a577fe640 1 -- 172.21.15.23:0/3397669134 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c065070 con 0x56257edf9d10 2024-03-24T10:48:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a577fe640 1 -- 172.21.15.23:0/3397669134 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f0a6c0aee40 con 0x56257edf9d10 2024-03-24T10:48:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.671+0000 7f0a81031740 1 -- 172.21.15.23:0/3397669134 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a70009e30 con 0x56257edf9d10 2024-03-24T10:48:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.672+0000 7f0a577fe640 1 -- 172.21.15.23:0/3397669134 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0a6c09ef40 con 0x56257edf9d10 2024-03-24T10:48:19.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.672+0000 7f0a81031740 10 client.17916 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:19.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.672+0000 7f0a81031740 1 -- 172.21.15.23:0/3397669134 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56257ed20920 con 0x56257edf9d10 2024-03-24T10:48:19.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.672+0000 7f0a577fe640 1 -- 172.21.15.23:0/3397669134 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a70009e30 con 0x56257edf9d10 2024-03-24T10:48:19.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.672+0000 7f0a577fe640 1 -- 172.21.15.23:0/3397669134 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f0a6c099e80 con 0x56257edf9d10 2024-03-24T10:48:19.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.672+0000 7f0a81031740 10 client.17916 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:19.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.672+0000 7f0a81031740 10 client.17916 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:48:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.672+0000 7f0a81031740 20 client.17916 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.672+0000 7f0a81031740 1 --2- 172.21.15.23:0/3397669134 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56257ee79220 0x56257ee7b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.672+0000 7f0a81031740 4 client.17916 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:48:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.672+0000 7f0a81031740 1 -- 172.21.15.23:0/3397669134 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56257eaeb890 con 0x56257ee79220 2024-03-24T10:48:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.672+0000 7f0a77fff640 1 --2- 172.21.15.23:0/3397669134 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56257ee79220 0x56257ee7b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.672+0000 7f0a577fe640 10 client.17916 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:48:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.672+0000 7f0a77fff640 1 --2- 172.21.15.23:0/3397669134 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56257ee79220 0x56257ee7b610 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=67b473a1d156c3cd server_cookie=9a946c20a23e8ea5 in_seq=0 out_seq=0 2024-03-24T10:48:19.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a577fe640 1 -- 172.21.15.23:0/3397669134 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56257eaeb890 con 0x56257ee79220 2024-03-24T10:48:19.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a577fe640 10 client.17916 handle_command_reply: tid=0 2024-03-24T10:48:19.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a81031740 1 client.17916 shutdown 2024-03-24T10:48:19.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a81031740 1 -- 172.21.15.23:0/3397669134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257edf9d10 msgr2=0x56257edfc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a81031740 1 --2- 172.21.15.23:0/3397669134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257edf9d10 0x56257edfc950 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0cc150 tx=0x7f0a6c01f000 comp rx=0 tx=0).stop 2024-03-24T10:48:19.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a81031740 1 -- 172.21.15.23:0/3397669134 shutdown_connections 2024-03-24T10:48:19.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a81031740 1 --2- 172.21.15.23:0/3397669134 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56257ee79220 0x56257ee7b610 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a81031740 1 --2- 172.21.15.23:0/3397669134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56257ee028f0 0x56257edfce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a81031740 1 --2- 172.21.15.23:0/3397669134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56257edf9d10 0x56257edfc950 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a81031740 1 --2- 172.21.15.23:0/3397669134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56257edf93f0 0x56257edfc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a577fe640 0 client.17916 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:48:19.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a81031740 1 -- 172.21.15.23:0/3397669134 >> 172.21.15.23:0/3397669134 conn(0x56257edf3b20 msgr2=0x56257edf57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a81031740 1 -- 172.21.15.23:0/3397669134 shutdown_connections 2024-03-24T10:48:19.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a81031740 1 -- 172.21.15.23:0/3397669134 wait complete. 2024-03-24T10:48:19.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a81031740 20 client.17916 trim_cache size 0 max 16384 2024-03-24T10:48:19.687 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:48:19.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a7e457640 1 -- 172.21.15.23:0/1320090265 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a600a2230 msgr2=0x7f0a600a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a7e457640 1 --2- 172.21.15.23:0/1320090265 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a600a2230 0x7f0a600a46f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0a7000fc90 tx=0x7f0a70018000 comp rx=0 tx=0).stop 2024-03-24T10:48:19.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a7e457640 1 -- 172.21.15.23:0/1320090265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a78100990 msgr2=0x7f0a7819a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.673+0000 7f0a7e457640 1 --2- 172.21.15.23:0/1320090265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a78100990 0x7f0a7819a480 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00e9f0 tx=0x7f0a6c00eeb0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.674+0000 7f0a7e457640 1 -- 172.21.15.23:0/1320090265 shutdown_connections 2024-03-24T10:48:19.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.674+0000 7f0a7e457640 1 --2- 172.21.15.23:0/1320090265 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a600a2230 0x7f0a600a46f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.674+0000 7f0a7e457640 1 --2- 172.21.15.23:0/1320090265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a78107910 0x7f0a7819e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.674+0000 7f0a7e457640 1 --2- 172.21.15.23:0/1320090265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a78100990 0x7f0a7819a480 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.674+0000 7f0a7e457640 1 --2- 172.21.15.23:0/1320090265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a780fffd0 0x7f0a78199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.674+0000 7f0a7e457640 1 -- 172.21.15.23:0/1320090265 >> 172.21.15.23:0/1320090265 conn(0x7f0a78075950 msgr2=0x7f0a780fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.674+0000 7f0a7e457640 1 -- 172.21.15.23:0/1320090265 shutdown_connections 2024-03-24T10:48:19.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.674+0000 7f0a7e457640 1 -- 172.21.15.23:0/1320090265 wait complete. 2024-03-24T10:48:19.691 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:48:19.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:19 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4164353046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:48:19.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:19 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4164353046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:48:19.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.796+0000 7efca6488640 1 -- 172.21.15.23:0/3708175686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca01005a0 msgr2=0x7efca0100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.796+0000 7efca6488640 1 --2- 172.21.15.23:0/3708175686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca01005a0 0x7efca0100a20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efc94009890 tx=0x7efc9402bb70 comp rx=0 tx=0).stop 2024-03-24T10:48:19.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.796+0000 7efca6488640 1 -- 172.21.15.23:0/3708175686 shutdown_connections 2024-03-24T10:48:19.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.796+0000 7efca6488640 1 --2- 172.21.15.23:0/3708175686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca0100f60 0x7efca010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.796+0000 7efca6488640 1 --2- 172.21.15.23:0/3708175686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca01005a0 0x7efca0100a20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.796+0000 7efca6488640 1 --2- 172.21.15.23:0/3708175686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efca0106660 0x7efca0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.796+0000 7efca6488640 1 -- 172.21.15.23:0/3708175686 >> 172.21.15.23:0/3708175686 conn(0x7efca0077710 msgr2=0x7efca0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.796+0000 7efca6488640 1 -- 172.21.15.23:0/3708175686 shutdown_connections 2024-03-24T10:48:19.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.796+0000 7efca6488640 1 -- 172.21.15.23:0/3708175686 wait complete. 2024-03-24T10:48:19.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.796+0000 7efca6488640 1 Processor -- start 2024-03-24T10:48:19.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.796+0000 7efca6488640 1 -- start start 2024-03-24T10:48:19.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efca6488640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca01005a0 0x7efca0102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efca6488640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efca0100f60 0x7efca0102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efca6488640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca0106660 0x7efca0102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efca6488640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7efca01a6100 con 0x7efca01005a0 2024-03-24T10:48:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efca6488640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efca01a6270 con 0x7efca0106660 2024-03-24T10:48:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efca6488640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efca01a6430 con 0x7efca0100f60 2024-03-24T10:48:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efc9ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca01005a0 0x7efca0102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efc9f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efca0100f60 0x7efca0102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efc9ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca01005a0 0x7efca0102210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40382/0 (socket says 172.21.15.23:40382) 2024-03-24T10:48:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efc9ffff640 1 -- 172.21.15.23:0/2047404636 learned_addr learned my addr 172.21.15.23:0/2047404636 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efca4a25640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca0106660 0x7efca0102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efc9ffff640 1 -- 172.21.15.23:0/2047404636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efca0100f60 msgr2=0x7efca0102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efc9ffff640 1 --2- 172.21.15.23:0/2047404636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efca0100f60 0x7efca0102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efc9ffff640 1 -- 172.21.15.23:0/2047404636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca0106660 msgr2=0x7efca0102c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efc9ffff640 1 --2- 172.21.15.23:0/2047404636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca0106660 0x7efca0102c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efc9ffff640 1 -- 172.21.15.23:0/2047404636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc98009900 con 0x7efca01005a0 2024-03-24T10:48:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efc9f7fe640 1 --2- 172.21.15.23:0/2047404636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efca0100f60 0x7efca0102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efca4a25640 1 --2- 172.21.15.23:0/2047404636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca0106660 0x7efca0102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:19.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efc9ffff640 1 --2- 172.21.15.23:0/2047404636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca01005a0 0x7efca0102210 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7efc9000e9f0 tx=0x7efc9000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efc9d7fa640 1 -- 172.21.15.23:0/2047404636 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90009700 con 0x7efca01005a0 2024-03-24T10:48:19.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efca6488640 1 -- 172.21.15.23:0/2047404636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc94009490 con 0x7efca01005a0 2024-03-24T10:48:19.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.797+0000 7efc9d7fa640 1 -- 172.21.15.23:0/2047404636 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc9000cd80 con 0x7efca01005a0 2024-03-24T10:48:19.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.798+0000 7efc9d7fa640 1 -- 172.21.15.23:0/2047404636 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90018920 con 0x7efca01005a0 2024-03-24T10:48:19.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.798+0000 7efca6488640 1 -- 172.21.15.23:0/2047404636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca01a6ab0 con 0x7efca01005a0 2024-03-24T10:48:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.800+0000 7efc9d7fa640 1 -- 172.21.15.23:0/2047404636 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7efc90018a80 con 0x7efca01005a0 2024-03-24T10:48:19.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.800+0000 7efca906d740 1 Processor -- start 2024-03-24T10:48:19.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.800+0000 7efca906d740 1 -- start start 2024-03-24T10:48:19.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.800+0000 7efca906d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed030643f0 0x55ed030647d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.800+0000 7efca906d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed03064d10 0x55ed0306d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.800+0000 7efca906d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed0306d8f0 0x55ed0306fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.800+0000 7efca906d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ed02fd2840 con 0x55ed03064d10 2024-03-24T10:48:19.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.800+0000 7efca906d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ed02fcfd70 con 0x55ed030643f0 2024-03-24T10:48:19.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.800+0000 7efca906d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ed02fcb550 con 0x55ed0306d8f0 2024-03-24T10:48:19.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.800+0000 7efc9d7fa640 1 --2- 172.21.15.23:0/2047404636 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7efc880a2230 0x7efc880a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.800+0000 7efc9ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed03064d10 0x55ed0306d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efca4a25640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed030643f0 0x55ed030647d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efca4a25640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed030643f0 0x55ed030647d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:50748/0 (socket says 172.21.15.23:50748) 2024-03-24T10:48:19.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efc9ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed03064d10 0x55ed0306d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40392/0 (socket says 172.21.15.23:40392) 2024-03-24T10:48:19.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efca4a25640 1 -- 172.21.15.23:0/1172618858 learned_addr learned my addr 172.21.15.23:0/1172618858 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efc9f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed0306d8f0 0x55ed0306fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efca4a25640 1 --2- 172.21.15.23:0/2047404636 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7efc880a2230 0x7efc880a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efc9d7fa640 1 -- 172.21.15.23:0/2047404636 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7efc90014070 con 0x7efca01005a0 2024-03-24T10:48:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efca4a25640 1 -- 172.21.15.23:0/1172618858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed0306d8f0 msgr2=0x55ed0306fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efca4a25640 1 --2- 172.21.15.23:0/1172618858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed0306d8f0 0x55ed0306fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efca4a25640 1 -- 172.21.15.23:0/1172618858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed03064d10 msgr2=0x55ed0306d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efca4a25640 1 --2- 172.21.15.23:0/1172618858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed03064d10 0x55ed0306d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efca4a25640 1 -- 172.21.15.23:0/1172618858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed02f60de0 con 0x55ed030643f0 2024-03-24T10:48:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efca4a25640 1 --2- 172.21.15.23:0/1172618858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed030643f0 0x55ed030647d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efc9800c830 tx=0x7efc9800ccf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df2cfc0807fca95f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efca4a25640 1 --2- 172.21.15.23:0/2047404636 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7efc880a2230 0x7efc880a46f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efc9800b730 tx=0x7efc98011000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efc867fc640 1 -- 172.21.15.23:0/1172618858 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc98013070 con 0x55ed030643f0 2024-03-24T10:48:19.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efc867fc640 1 -- 172.21.15.23:0/1172618858 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc98014040 con 0x55ed030643f0 2024-03-24T10:48:19.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.801+0000 7efc867fc640 1 -- 172.21.15.23:0/1172618858 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9801ccb0 con 0x55ed030643f0 2024-03-24T10:48:19.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.802+0000 7efca906d740 1 -- 172.21.15.23:0/1172618858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed030643f0 msgr2=0x55ed030647d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.802+0000 7efca906d740 1 --2- 172.21.15.23:0/1172618858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed030643f0 0x55ed030647d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efc9800c830 tx=0x7efc9800ccf0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.802+0000 7efca906d740 1 -- 172.21.15.23:0/1172618858 shutdown_connections 2024-03-24T10:48:19.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.802+0000 7efca906d740 1 --2- 172.21.15.23:0/1172618858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed0306d8f0 0x55ed0306fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.802+0000 7efca906d740 1 --2- 172.21.15.23:0/1172618858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed03064d10 0x55ed0306d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.802+0000 7efca906d740 1 --2- 172.21.15.23:0/1172618858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed030643f0 0x55ed030647d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.802+0000 7efca906d740 1 -- 172.21.15.23:0/1172618858 >> 172.21.15.23:0/1172618858 conn(0x55ed0305eb20 msgr2=0x55ed0305ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.802+0000 7efca906d740 1 -- 172.21.15.23:0/1172618858 shutdown_connections 2024-03-24T10:48:19.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.802+0000 7efca906d740 1 -- 172.21.15.23:0/1172618858 wait complete. 2024-03-24T10:48:19.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.802+0000 7efca906d740 1 -- start start 2024-03-24T10:48:19.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca906d740 1 Processor -- start 2024-03-24T10:48:19.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca906d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed030643f0 0x55ed0306a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca906d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed03064d10 0x55ed0306af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca906d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed0306d8f0 0x55ed0306b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca906d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ed02c9e430 con 0x55ed0306d8f0 2024-03-24T10:48:19.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca906d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ed02c5ae00 con 0x55ed03064d10 2024-03-24T10:48:19.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca906d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ed02f50590 con 0x55ed030643f0 2024-03-24T10:48:19.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca906d740 10 client.0 _ll_register_callbacks cb 0x55ed02fbffc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:19.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca4a25640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed0306d8f0 0x55ed0306b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efc9ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed030643f0 0x55ed0306a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca4a25640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed0306d8f0 0x55ed0306b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40408/0 (socket says 172.21.15.23:40408) 2024-03-24T10:48:19.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca4a25640 1 -- 172.21.15.23:0/3172756743 learned_addr learned my addr 172.21.15.23:0/3172756743 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efc9f7fe640 1 --2- 172.21.15.23:0/3172756743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed03064d10 0x55ed0306af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca4a25640 1 -- 172.21.15.23:0/3172756743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed030643f0 msgr2=0x55ed0306a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca4a25640 1 --2- 172.21.15.23:0/3172756743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed030643f0 0x55ed0306a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca4a25640 1 -- 172.21.15.23:0/3172756743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed03064d10 msgr2=0x55ed0306af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca4a25640 1 --2- 172.21.15.23:0/3172756743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed03064d10 0x55ed0306af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca4a25640 1 -- 172.21.15.23:0/3172756743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc9800c410 con 0x55ed0306d8f0 2024-03-24T10:48:19.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efca4a25640 1 --2- 172.21.15.23:0/3172756743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed0306d8f0 0x55ed0306b460 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7efc98006010 tx=0x7efc98003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf7343e9f32190c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.803+0000 7efc7f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:19.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efc7f7fe640 1 -- 172.21.15.23:0/3172756743 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc98013040 con 0x55ed0306d8f0 2024-03-24T10:48:19.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efca906d740 1 -- 172.21.15.23:0/3172756743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ed02cca7e0 con 0x55ed0306d8f0 2024-03-24T10:48:19.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efc7f7fe640 1 -- 172.21.15.23:0/3172756743 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7efc9801ce80 con 0x55ed0306d8f0 2024-03-24T10:48:19.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efc7f7fe640 1 -- 172.21.15.23:0/3172756743 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efc9801c780 con 0x55ed0306d8f0 2024-03-24T10:48:19.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efca906d740 10 client.17928 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:19.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efca906d740 1 -- 172.21.15.23:0/3172756743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ed02fc9bd0 con 0x55ed0306d8f0 2024-03-24T10:48:19.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efc7f7fe640 1 -- 172.21.15.23:0/3172756743 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc9803b020 con 0x55ed0306d8f0 2024-03-24T10:48:19.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efc7f7fe640 1 -- 172.21.15.23:0/3172756743 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7efc9802c070 con 0x55ed0306d8f0 2024-03-24T10:48:19.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efca906d740 10 client.17928 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:19.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efca906d740 10 client.17928 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:19.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efca906d740 20 client.17928 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:19.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efca906d740 1 --2- 172.21.15.23:0/3172756743 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ed0307dab0 0x55ed0307fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efca906d740 4 client.17928 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:48:19.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efca906d740 1 -- 172.21.15.23:0/3172756743 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed02d07720 con 0x55ed0307dab0 2024-03-24T10:48:19.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.804+0000 7efc9ffff640 1 --2- 172.21.15.23:0/3172756743 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ed0307dab0 0x55ed0307fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.805+0000 7efc7f7fe640 10 client.17928 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:19.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.805+0000 7efc9ffff640 1 --2- 172.21.15.23:0/3172756743 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ed0307dab0 0x55ed0307fea0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e757164e6faddaf server_cookie=1470c0769b149c5f in_seq=0 out_seq=0 2024-03-24T10:48:19.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.805+0000 7efc7f7fe640 1 -- 172.21.15.23:0/3172756743 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ed02d07720 con 0x55ed0307dab0 2024-03-24T10:48:19.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.805+0000 7efc7f7fe640 10 client.17928 handle_command_reply: tid=0 2024-03-24T10:48:19.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.805+0000 7efca906d740 1 client.17928 shutdown 2024-03-24T10:48:19.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.806+0000 7efca906d740 1 -- 172.21.15.23:0/3172756743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed0306d8f0 msgr2=0x55ed0306b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.806+0000 7efca906d740 1 --2- 172.21.15.23:0/3172756743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed0306d8f0 0x55ed0306b460 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7efc98006010 tx=0x7efc98003c20 comp rx=0 tx=0).stop 2024-03-24T10:48:19.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.806+0000 7efca906d740 1 -- 172.21.15.23:0/3172756743 shutdown_connections 2024-03-24T10:48:19.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.806+0000 7efca906d740 1 --2- 172.21.15.23:0/3172756743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed0306d8f0 0x55ed0306b460 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.806+0000 7efca906d740 1 --2- 172.21.15.23:0/3172756743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed03064d10 0x55ed0306af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.806+0000 7efca906d740 1 --2- 172.21.15.23:0/3172756743 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ed0307dab0 0x55ed0307fea0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.806+0000 7efca906d740 1 --2- 172.21.15.23:0/3172756743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed030643f0 0x55ed0306a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.806+0000 7efc7f7fe640 0 client.17928 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:19.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.806+0000 7efca906d740 1 -- 172.21.15.23:0/3172756743 >> 172.21.15.23:0/3172756743 conn(0x55ed0305eb20 msgr2=0x55ed03060870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.806+0000 7efca906d740 1 -- 172.21.15.23:0/3172756743 shutdown_connections 2024-03-24T10:48:19.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.806+0000 7efca906d740 1 -- 172.21.15.23:0/3172756743 wait complete. 2024-03-24T10:48:19.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.806+0000 7efca906d740 20 client.17928 trim_cache size 0 max 16384 2024-03-24T10:48:19.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efca906d740 1 Processor -- start 2024-03-24T10:48:19.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efca906d740 1 -- start start 2024-03-24T10:48:19.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efca906d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed030643f0 0x55ed0306d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efca906d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed03064d10 0x55ed03079120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efca906d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed0306d8f0 0x55ed03085770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efca906d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ed03063e30 con 0x55ed03064d10 2024-03-24T10:48:19.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efca906d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ed03064b70 con 0x55ed0306d8f0 2024-03-24T10:48:19.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efca906d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ed0305fd50 con 0x55ed030643f0 2024-03-24T10:48:19.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efc9f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed03064d10 0x55ed03079120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efc9f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed03064d10 0x55ed03079120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40418/0 (socket says 172.21.15.23:40418) 2024-03-24T10:48:19.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efc9f7fe640 1 -- 172.21.15.23:0/1518054916 learned_addr learned my addr 172.21.15.23:0/1518054916 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efc9ffff640 1 --2- 172.21.15.23:0/1518054916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed030643f0 0x55ed0306d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efca4a25640 1 --2- 172.21.15.23:0/1518054916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed0306d8f0 0x55ed03085770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efc9f7fe640 1 -- 172.21.15.23:0/1518054916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed030643f0 msgr2=0x55ed0306d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efc9f7fe640 1 --2- 172.21.15.23:0/1518054916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed030643f0 0x55ed0306d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efc9f7fe640 1 -- 172.21.15.23:0/1518054916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed0306d8f0 msgr2=0x55ed03085770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efc9f7fe640 1 --2- 172.21.15.23:0/1518054916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed0306d8f0 0x55ed03085770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.819+0000 7efc9f7fe640 1 -- 172.21.15.23:0/1518054916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed02f65c80 con 0x55ed03064d10 2024-03-24T10:48:19.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.820+0000 7efc9f7fe640 1 --2- 172.21.15.23:0/1518054916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed03064d10 0x55ed03079120 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7efc9402c0b0 tx=0x7efc9402e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9581dfb2ad42f1a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.820+0000 7efc85ffb640 1 -- 172.21.15.23:0/1518054916 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc94004180 con 0x55ed03064d10 2024-03-24T10:48:19.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.820+0000 7efc85ffb640 1 -- 172.21.15.23:0/1518054916 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc94034920 con 0x55ed03064d10 2024-03-24T10:48:19.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.820+0000 7efc85ffb640 1 -- 172.21.15.23:0/1518054916 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc94032600 con 0x55ed03064d10 2024-03-24T10:48:19.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.820+0000 7efca906d740 1 -- 172.21.15.23:0/1518054916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed03064d10 msgr2=0x55ed03079120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.820+0000 7efca906d740 1 --2- 172.21.15.23:0/1518054916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed03064d10 0x55ed03079120 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7efc9402c0b0 tx=0x7efc9402e040 comp rx=0 tx=0).stop 2024-03-24T10:48:19.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.820+0000 7efca906d740 1 -- 172.21.15.23:0/1518054916 shutdown_connections 2024-03-24T10:48:19.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.820+0000 7efca906d740 1 --2- 172.21.15.23:0/1518054916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed0306d8f0 0x55ed03085770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.820+0000 7efca906d740 1 --2- 172.21.15.23:0/1518054916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed03064d10 0x55ed03079120 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.820+0000 7efca906d740 1 --2- 172.21.15.23:0/1518054916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed030643f0 0x55ed0306d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.820+0000 7efca906d740 1 -- 172.21.15.23:0/1518054916 >> 172.21.15.23:0/1518054916 conn(0x55ed0305eb20 msgr2=0x55ed030607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.820+0000 7efca906d740 1 -- 172.21.15.23:0/1518054916 shutdown_connections 2024-03-24T10:48:19.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.820+0000 7efca906d740 1 -- 172.21.15.23:0/1518054916 wait complete. 2024-03-24T10:48:19.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca906d740 1 -- start start 2024-03-24T10:48:19.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca906d740 1 Processor -- start 2024-03-24T10:48:19.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca906d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed030643f0 0x55ed03075750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca906d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed03064d10 0x55ed03072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca906d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed0306d8f0 0x55ed03072c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca906d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ed03075e20 con 0x55ed0306d8f0 2024-03-24T10:48:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca906d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ed03075f90 con 0x55ed030643f0 2024-03-24T10:48:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca906d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ed030731f0 con 0x55ed03064d10 2024-03-24T10:48:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca906d740 10 client.0 _ll_register_callbacks cb 0x7efc98028070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca4a25640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed0306d8f0 0x55ed03072c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca4a25640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed0306d8f0 0x55ed03072c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40424/0 (socket says 172.21.15.23:40424) 2024-03-24T10:48:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca4a25640 1 -- 172.21.15.23:0/3963769583 learned_addr learned my addr 172.21.15.23:0/3963769583 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efc9f7fe640 1 --2- 172.21.15.23:0/3963769583 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed03064d10 0x55ed03072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efc9ffff640 1 --2- 172.21.15.23:0/3963769583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed030643f0 0x55ed03075750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca4a25640 1 -- 172.21.15.23:0/3963769583 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed03064d10 msgr2=0x55ed03072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca4a25640 1 --2- 172.21.15.23:0/3963769583 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed03064d10 0x55ed03072720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca4a25640 1 -- 172.21.15.23:0/3963769583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed030643f0 msgr2=0x55ed03075750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca4a25640 1 --2- 172.21.15.23:0/3963769583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed030643f0 0x55ed03075750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.821+0000 7efca4a25640 1 -- 172.21.15.23:0/3963769583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca010dd20 con 0x55ed0306d8f0 2024-03-24T10:48:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efca4a25640 1 --2- 172.21.15.23:0/3963769583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed0306d8f0 0x55ed03072c60 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7efc980258a0 tx=0x7efc9801cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a78a9378578ff68 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efc7f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efc7f7fe640 1 -- 172.21.15.23:0/3963769583 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc98013070 con 0x55ed0306d8f0 2024-03-24T10:48:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efc7f7fe640 1 -- 172.21.15.23:0/3963769583 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7efc9803c080 con 0x55ed0306d8f0 2024-03-24T10:48:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efca906d740 1 -- 172.21.15.23:0/3963769583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc98014040 con 0x55ed0306d8f0 2024-03-24T10:48:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efc7f7fe640 1 -- 172.21.15.23:0/3963769583 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efc98003c90 con 0x55ed0306d8f0 2024-03-24T10:48:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efca906d740 10 client.17934 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efca906d740 1 -- 172.21.15.23:0/3963769583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ed02f8b920 con 0x55ed0306d8f0 2024-03-24T10:48:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efc7f7fe640 1 -- 172.21.15.23:0/3963769583 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc98014040 con 0x55ed0306d8f0 2024-03-24T10:48:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efc7f7fe640 1 -- 172.21.15.23:0/3963769583 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7efc9802c040 con 0x55ed0306d8f0 2024-03-24T10:48:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efca906d740 10 client.17934 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efca906d740 10 client.17934 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efca906d740 20 client.17934 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efca906d740 1 --2- 172.21.15.23:0/3963769583 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ed030e4d70 0x55ed030e7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efca906d740 4 client.17934 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:48:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.822+0000 7efca906d740 1 -- 172.21.15.23:0/3963769583 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ed02d56890 con 0x55ed030e4d70 2024-03-24T10:48:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efc9ffff640 1 --2- 172.21.15.23:0/3963769583 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ed030e4d70 0x55ed030e7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efc9ffff640 1 --2- 172.21.15.23:0/3963769583 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ed030e4d70 0x55ed030e7160 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf6b0e353d3a9f91 server_cookie=9965fe010878582e in_seq=0 out_seq=0 2024-03-24T10:48:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efc7f7fe640 10 client.17934 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efc7f7fe640 1 -- 172.21.15.23:0/3963769583 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ed02d56890 con 0x55ed030e4d70 2024-03-24T10:48:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efc7f7fe640 10 client.17934 handle_command_reply: tid=0 2024-03-24T10:48:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efca906d740 1 client.17934 shutdown 2024-03-24T10:48:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efca906d740 1 -- 172.21.15.23:0/3963769583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed0306d8f0 msgr2=0x55ed03072c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efca906d740 1 --2- 172.21.15.23:0/3963769583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed0306d8f0 0x55ed03072c60 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7efc980258a0 tx=0x7efc9801cd60 comp rx=0 tx=0).stop 2024-03-24T10:48:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efca906d740 1 -- 172.21.15.23:0/3963769583 shutdown_connections 2024-03-24T10:48:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efca906d740 1 --2- 172.21.15.23:0/3963769583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed0306d8f0 0x55ed03072c60 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efca906d740 1 --2- 172.21.15.23:0/3963769583 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55ed030e4d70 0x55ed030e7160 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efca906d740 1 --2- 172.21.15.23:0/3963769583 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed03064d10 0x55ed03072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efca906d740 1 --2- 172.21.15.23:0/3963769583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ed030643f0 0x55ed03075750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efc7f7fe640 0 client.17934 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efca906d740 1 -- 172.21.15.23:0/3963769583 >> 172.21.15.23:0/3963769583 conn(0x55ed0305eb20 msgr2=0x55ed0305d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efca906d740 1 -- 172.21.15.23:0/3963769583 shutdown_connections 2024-03-24T10:48:19.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efca906d740 1 -- 172.21.15.23:0/3963769583 wait complete. 2024-03-24T10:48:19.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.823+0000 7efca906d740 20 client.17934 trim_cache size 0 max 16384 2024-03-24T10:48:19.836 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:48:19.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.824+0000 7efca6488640 1 -- 172.21.15.23:0/2047404636 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7efc880a2230 msgr2=0x7efc880a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.824+0000 7efca6488640 1 --2- 172.21.15.23:0/2047404636 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7efc880a2230 0x7efc880a46f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efc9800b730 tx=0x7efc98011000 comp rx=0 tx=0).stop 2024-03-24T10:48:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.824+0000 7efca6488640 1 -- 172.21.15.23:0/2047404636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca01005a0 msgr2=0x7efca0102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.824+0000 7efca6488640 1 --2- 172.21.15.23:0/2047404636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca01005a0 0x7efca0102210 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7efc9000e9f0 tx=0x7efc9000eeb0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.824+0000 7efca6488640 1 -- 172.21.15.23:0/2047404636 shutdown_connections 2024-03-24T10:48:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.824+0000 7efca6488640 1 --2- 172.21.15.23:0/2047404636 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7efc880a2230 0x7efc880a46f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.824+0000 7efca6488640 1 --2- 172.21.15.23:0/2047404636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca0106660 0x7efca0102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.824+0000 7efca6488640 1 --2- 172.21.15.23:0/2047404636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efca0100f60 0x7efca0102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.824+0000 7efca6488640 1 --2- 172.21.15.23:0/2047404636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca01005a0 0x7efca0102210 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.824+0000 7efca6488640 1 -- 172.21.15.23:0/2047404636 >> 172.21.15.23:0/2047404636 conn(0x7efca0077710 msgr2=0x7efca0068af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.824+0000 7efca6488640 1 -- 172.21.15.23:0/2047404636 shutdown_connections 2024-03-24T10:48:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:19.824+0000 7efca6488640 1 -- 172.21.15.23:0/2047404636 wait complete. 2024-03-24T10:48:20.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:20 smithi117 ceph-mon[36368]: pgmap v256: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.6 MiB/s wr, 61 op/s 2024-03-24T10:48:20.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:20 smithi116 ceph-mon[36248]: pgmap v256: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.6 MiB/s wr, 61 op/s 2024-03-24T10:48:20.840 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:48:20.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:48:20.840 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:48:20.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:20 smithi023 ceph-mon[32466]: pgmap v256: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.6 MiB/s wr, 61 op/s 2024-03-24T10:48:20.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.959+0000 7fa6bbb02640 1 -- 172.21.15.23:0/541188976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b41005a0 msgr2=0x7fa6b4100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:20.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.959+0000 7fa6bbb02640 1 --2- 172.21.15.23:0/541188976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b41005a0 0x7fa6b4100a20 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009900 tx=0x7fa6a402bc30 comp rx=0 tx=0).stop 2024-03-24T10:48:20.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.959+0000 7fa6bbb02640 1 -- 172.21.15.23:0/541188976 shutdown_connections 2024-03-24T10:48:20.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.959+0000 7fa6bbb02640 1 --2- 172.21.15.23:0/541188976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6b4100f60 0x7fa6b410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.959+0000 7fa6bbb02640 1 --2- 172.21.15.23:0/541188976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b41005a0 0x7fa6b4100a20 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.959+0000 7fa6bbb02640 1 --2- 172.21.15.23:0/541188976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b4106660 0x7fa6b4106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.959+0000 7fa6bbb02640 1 -- 172.21.15.23:0/541188976 >> 172.21.15.23:0/541188976 conn(0x7fa6b4077710 msgr2=0x7fa6b4077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:20.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.959+0000 7fa6bbb02640 1 -- 172.21.15.23:0/541188976 shutdown_connections 2024-03-24T10:48:20.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.959+0000 7fa6bbb02640 1 -- 172.21.15.23:0/541188976 wait complete. 2024-03-24T10:48:20.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.959+0000 7fa6bbb02640 1 Processor -- start 2024-03-24T10:48:20.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.959+0000 7fa6bbb02640 1 -- start start 2024-03-24T10:48:20.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6bbb02640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6b41005a0 0x7fa6b4198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6bbb02640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4100f60 0x7fa6b41990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6bbb02640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b4106660 0x7fa6b41a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6bbb02640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b4199970 con 0x7fa6b4100f60 2024-03-24T10:48:20.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6bbb02640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b41a4170 con 0x7fa6b41005a0 2024-03-24T10:48:20.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6bbb02640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b41a42b0 con 0x7fa6b4106660 2024-03-24T10:48:20.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6b909d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4100f60 0x7fa6b41990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6b909d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4100f60 0x7fa6b41990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40456/0 (socket says 172.21.15.23:40456) 2024-03-24T10:48:20.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6b909d640 1 -- 172.21.15.23:0/1671670435 learned_addr learned my addr 172.21.15.23:0/1671670435 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:20.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6ba09f640 1 --2- 172.21.15.23:0/1671670435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b4106660 0x7fa6b41a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6b989e640 1 --2- 172.21.15.23:0/1671670435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6b41005a0 0x7fa6b4198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6b909d640 1 -- 172.21.15.23:0/1671670435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b4106660 msgr2=0x7fa6b41a3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:20.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6b909d640 1 --2- 172.21.15.23:0/1671670435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b4106660 0x7fa6b41a3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6b909d640 1 -- 172.21.15.23:0/1671670435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6b41005a0 msgr2=0x7fa6b4198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:20.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6b909d640 1 --2- 172.21.15.23:0/1671670435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6b41005a0 0x7fa6b4198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6b909d640 1 -- 172.21.15.23:0/1671670435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a4009560 con 0x7fa6b4100f60 2024-03-24T10:48:20.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6b909d640 1 --2- 172.21.15.23:0/1671670435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4100f60 0x7fa6b41990f0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fa6a402c240 tx=0x7fa6a402c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:20.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.960+0000 7fa6aaffd640 1 -- 172.21.15.23:0/1671670435 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a402dea0 con 0x7fa6b4100f60 2024-03-24T10:48:20.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.961+0000 7fa6aaffd640 1 -- 172.21.15.23:0/1671670435 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6a4004410 con 0x7fa6b4100f60 2024-03-24T10:48:20.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.961+0000 7fa6bbb02640 1 -- 172.21.15.23:0/1671670435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6b41a44b0 con 0x7fa6b4100f60 2024-03-24T10:48:20.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.961+0000 7fa6aaffd640 1 -- 172.21.15.23:0/1671670435 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4003bb0 con 0x7fa6b4100f60 2024-03-24T10:48:20.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.961+0000 7fa6bbb02640 1 -- 172.21.15.23:0/1671670435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6b41a4960 con 0x7fa6b4100f60 2024-03-24T10:48:20.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6aaffd640 1 -- 172.21.15.23:0/1671670435 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fa6a4003d10 con 0x7fa6b4100f60 2024-03-24T10:48:20.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6be7bf740 1 Processor -- start 2024-03-24T10:48:20.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6be7bf740 1 -- start start 2024-03-24T10:48:20.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6be7bf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8a490 0x5614ecd8a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6be7bf740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8adb0 0x5614ecd97b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6be7bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd98070 0x5614ecd9a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6be7bf740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5614eccf8530 con 0x5614ecd8adb0 2024-03-24T10:48:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6be7bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5614eccf5a60 con 0x5614ecd98070 2024-03-24T10:48:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6be7bf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5614eccf1240 con 0x5614ecd8a490 2024-03-24T10:48:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6b989e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8adb0 0x5614ecd97b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6ba09f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8a490 0x5614ecd8a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6b989e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8adb0 0x5614ecd97b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40462/0 (socket says 172.21.15.23:40462) 2024-03-24T10:48:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6b989e640 1 -- 172.21.15.23:0/276629457 learned_addr learned my addr 172.21.15.23:0/276629457 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6b909d640 1 --2- 172.21.15.23:0/276629457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd98070 0x5614ecd9a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.964+0000 7fa6aaffd640 1 --2- 172.21.15.23:0/1671670435 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa6a00a2230 0x7fa6a00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.965+0000 7fa6ba09f640 1 --2- 172.21.15.23:0/1671670435 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa6a00a2230 0x7fa6a00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.965+0000 7fa6b989e640 1 -- 172.21.15.23:0/276629457 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8a490 msgr2=0x5614ecd8a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:20.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.965+0000 7fa6b989e640 1 --2- 172.21.15.23:0/276629457 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8a490 0x5614ecd8a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.965+0000 7fa6b989e640 1 -- 172.21.15.23:0/276629457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd98070 msgr2=0x5614ecd9a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:20.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.965+0000 7fa6b989e640 1 --2- 172.21.15.23:0/276629457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd98070 0x5614ecd9a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.965+0000 7fa6b989e640 1 -- 172.21.15.23:0/276629457 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614eccef8c0 con 0x5614ecd8adb0 2024-03-24T10:48:20.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.965+0000 7fa6aaffd640 1 -- 172.21.15.23:0/1671670435 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7fa6a40f0010 con 0x7fa6b4100f60 2024-03-24T10:48:20.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.965+0000 7fa6b909d640 1 --2- 172.21.15.23:0/276629457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd98070 0x5614ecd9a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:20.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.965+0000 7fa6b989e640 1 --2- 172.21.15.23:0/276629457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8adb0 0x5614ecd97b30 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009c80 tx=0x7fa6b0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76dfae2ed2cc28c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:20.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.965+0000 7fa6ba09f640 1 --2- 172.21.15.23:0/1671670435 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa6a00a2230 0x7fa6a00a46f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac005c40 tx=0x7fa6ac004ab0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:20.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.965+0000 7fa69ffff640 1 -- 172.21.15.23:0/276629457 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0019630 con 0x5614ecd8adb0 2024-03-24T10:48:20.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.965+0000 7fa69ffff640 1 -- 172.21.15.23:0/276629457 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6b0019c70 con 0x5614ecd8adb0 2024-03-24T10:48:20.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.965+0000 7fa69ffff640 1 -- 172.21.15.23:0/276629457 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b000eb70 con 0x5614ecd8adb0 2024-03-24T10:48:20.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.966+0000 7fa6be7bf740 1 -- 172.21.15.23:0/276629457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8adb0 msgr2=0x5614ecd97b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:20.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.966+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/276629457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8adb0 0x5614ecd97b30 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009c80 tx=0x7fa6b0011810 comp rx=0 tx=0).stop 2024-03-24T10:48:20.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.966+0000 7fa6be7bf740 1 -- 172.21.15.23:0/276629457 shutdown_connections 2024-03-24T10:48:20.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.966+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/276629457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd98070 0x5614ecd9a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.966+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/276629457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8adb0 0x5614ecd97b30 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.966+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/276629457 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8a490 0x5614ecd8a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.966+0000 7fa6be7bf740 1 -- 172.21.15.23:0/276629457 >> 172.21.15.23:0/276629457 conn(0x5614ecd84e90 msgr2=0x5614ecd85270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:20.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.966+0000 7fa6be7bf740 1 -- 172.21.15.23:0/276629457 shutdown_connections 2024-03-24T10:48:20.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.966+0000 7fa6be7bf740 1 -- 172.21.15.23:0/276629457 wait complete. 2024-03-24T10:48:20.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.966+0000 7fa6be7bf740 1 -- start start 2024-03-24T10:48:20.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.966+0000 7fa6be7bf740 1 Processor -- start 2024-03-24T10:48:20.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6be7bf740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8a490 0x5614ecd95110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6be7bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd8adb0 0x5614ecd95650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6be7bf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd98070 0x5614ecd95b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6be7bf740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5614ecc95f80 con 0x5614ecd8a490 2024-03-24T10:48:20.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6be7bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5614ec9c44b0 con 0x5614ecd8adb0 2024-03-24T10:48:20.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6be7bf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5614ecc76ad0 con 0x5614ecd98070 2024-03-24T10:48:20.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6be7bf740 10 client.0 _ll_register_callbacks cb 0x5614ecca9b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:20.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6b989e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8a490 0x5614ecd95110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6b989e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8a490 0x5614ecd95110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40474/0 (socket says 172.21.15.23:40474) 2024-03-24T10:48:20.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6b989e640 1 -- 172.21.15.23:0/2544068466 learned_addr learned my addr 172.21.15.23:0/2544068466 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:20.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6ba09f640 1 --2- 172.21.15.23:0/2544068466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd98070 0x5614ecd95b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6b909d640 1 --2- 172.21.15.23:0/2544068466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd8adb0 0x5614ecd95650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6b989e640 1 -- 172.21.15.23:0/2544068466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd98070 msgr2=0x5614ecd95b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:20.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6b989e640 1 --2- 172.21.15.23:0/2544068466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd98070 0x5614ecd95b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6b989e640 1 -- 172.21.15.23:0/2544068466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd8adb0 msgr2=0x5614ecd95650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:20.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6b989e640 1 --2- 172.21.15.23:0/2544068466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd8adb0 0x5614ecd95650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6b989e640 1 -- 172.21.15.23:0/2544068466 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6b00098b0 con 0x5614ecd8a490 2024-03-24T10:48:20.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.967+0000 7fa6ba09f640 1 --2- 172.21.15.23:0/2544068466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd98070 0x5614ecd95b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:20.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa6b989e640 1 --2- 172.21.15.23:0/2544068466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8a490 0x5614ecd95110 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0000c00 tx=0x7fa6b000e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d126f9b567383d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:20.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa69d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:20.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa69d7fa640 1 -- 172.21.15.23:0/2544068466 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b000eee0 con 0x5614ecd8a490 2024-03-24T10:48:20.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa69d7fa640 1 -- 172.21.15.23:0/2544068466 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7fa6b0005040 con 0x5614ecd8a490 2024-03-24T10:48:20.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa6be7bf740 1 -- 172.21.15.23:0/2544068466 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5614ecd157a0 con 0x5614ecd8a490 2024-03-24T10:48:20.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa69d7fa640 1 -- 172.21.15.23:0/2544068466 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa6b0005200 con 0x5614ecd8a490 2024-03-24T10:48:20.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa6be7bf740 10 client.17946 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:20.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa6be7bf740 1 -- 172.21.15.23:0/2544068466 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5614ecca76d0 con 0x5614ecd8a490 2024-03-24T10:48:20.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa69d7fa640 1 -- 172.21.15.23:0/2544068466 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5614ecd157a0 con 0x5614ecd8a490 2024-03-24T10:48:20.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa69d7fa640 1 -- 172.21.15.23:0/2544068466 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fa6b0029020 con 0x5614ecd8a490 2024-03-24T10:48:20.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa6be7bf740 10 client.17946 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:20.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa6be7bf740 10 client.17946 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:20.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa6be7bf740 20 client.17946 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:20.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/2544068466 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614ecda8100 0x5614ecdaa510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa6be7bf740 4 client.17946 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:48:20.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.968+0000 7fa6be7bf740 1 -- 172.21.15.23:0/2544068466 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614ecce71f0 con 0x5614ecda8100 2024-03-24T10:48:20.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.969+0000 7fa6b909d640 1 --2- 172.21.15.23:0/2544068466 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614ecda8100 0x5614ecdaa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.969+0000 7fa6b909d640 1 --2- 172.21.15.23:0/2544068466 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614ecda8100 0x5614ecdaa510 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70e927a2f2af9a2e server_cookie=70f679af4785b759 in_seq=0 out_seq=0 2024-03-24T10:48:20.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.969+0000 7fa69d7fa640 10 client.17946 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:20.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.969+0000 7fa69d7fa640 1 -- 172.21.15.23:0/2544068466 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5614ecce71f0 con 0x5614ecda8100 2024-03-24T10:48:20.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.969+0000 7fa69d7fa640 10 client.17946 handle_command_reply: tid=0 2024-03-24T10:48:20.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.970+0000 7fa6be7bf740 1 client.17946 shutdown 2024-03-24T10:48:20.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.970+0000 7fa6be7bf740 1 -- 172.21.15.23:0/2544068466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8a490 msgr2=0x5614ecd95110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:20.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.970+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/2544068466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8a490 0x5614ecd95110 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0000c00 tx=0x7fa6b000e8b0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.970+0000 7fa6be7bf740 1 -- 172.21.15.23:0/2544068466 shutdown_connections 2024-03-24T10:48:20.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.970+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/2544068466 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614ecda8100 0x5614ecdaa510 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.970+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/2544068466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd98070 0x5614ecd95b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.970+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/2544068466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd8adb0 0x5614ecd95650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.970+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/2544068466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8a490 0x5614ecd95110 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.970+0000 7fa69d7fa640 0 client.17946 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:20.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.970+0000 7fa6be7bf740 1 -- 172.21.15.23:0/2544068466 >> 172.21.15.23:0/2544068466 conn(0x5614ecd84e90 msgr2=0x5614ecd86be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:20.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.970+0000 7fa6be7bf740 1 -- 172.21.15.23:0/2544068466 shutdown_connections 2024-03-24T10:48:20.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.970+0000 7fa6be7bf740 1 -- 172.21.15.23:0/2544068466 wait complete. 2024-03-24T10:48:20.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.970+0000 7fa6be7bf740 20 client.17946 trim_cache size 0 max 16384 2024-03-24T10:48:20.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.987+0000 7fa6be7bf740 1 Processor -- start 2024-03-24T10:48:20.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6be7bf740 1 -- start start 2024-03-24T10:48:20.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6be7bf740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8a490 0x5614ecd97840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6be7bf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8adb0 0x5614ecda3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6be7bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd98070 0x5614ecdafe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6be7bf740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5614ecd89850 con 0x5614ecd8a490 2024-03-24T10:48:20.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6be7bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5614ecda3e50 con 0x5614ecd98070 2024-03-24T10:48:20.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6be7bf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5614ecdaa720 con 0x5614ecd8adb0 2024-03-24T10:48:20.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6b989e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8a490 0x5614ecd97840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6b989e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8a490 0x5614ecd97840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40480/0 (socket says 172.21.15.23:40480) 2024-03-24T10:48:20.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6b989e640 1 -- 172.21.15.23:0/3969556481 learned_addr learned my addr 172.21.15.23:0/3969556481 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:20.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6b909d640 1 --2- 172.21.15.23:0/3969556481 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8adb0 0x5614ecda3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6ba09f640 1 --2- 172.21.15.23:0/3969556481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd98070 0x5614ecdafe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6b989e640 1 -- 172.21.15.23:0/3969556481 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8adb0 msgr2=0x5614ecda3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:20.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6b989e640 1 --2- 172.21.15.23:0/3969556481 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8adb0 0x5614ecda3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6b989e640 1 -- 172.21.15.23:0/3969556481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd98070 msgr2=0x5614ecdafe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:20.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6b989e640 1 --2- 172.21.15.23:0/3969556481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd98070 0x5614ecdafe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6b989e640 1 -- 172.21.15.23:0/3969556481 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614ecc8bc60 con 0x5614ecd8a490 2024-03-24T10:48:20.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6ba09f640 1 --2- 172.21.15.23:0/3969556481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd98070 0x5614ecdafe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:48:20.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa6b989e640 1 --2- 172.21.15.23:0/3969556481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8a490 0x5614ecd97840 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fa6b001c040 tx=0x7fa6b0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b967926f999c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:20.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa69f7fe640 1 -- 172.21.15.23:0/3969556481 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b000eee0 con 0x5614ecd8a490 2024-03-24T10:48:20.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa69f7fe640 1 -- 172.21.15.23:0/3969556481 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6b000e920 con 0x5614ecd8a490 2024-03-24T10:48:20.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.988+0000 7fa69f7fe640 1 -- 172.21.15.23:0/3969556481 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0005230 con 0x5614ecd8a490 2024-03-24T10:48:20.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.989+0000 7fa6be7bf740 1 -- 172.21.15.23:0/3969556481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8a490 msgr2=0x5614ecd97840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:20.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.989+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/3969556481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8a490 0x5614ecd97840 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fa6b001c040 tx=0x7fa6b0011810 comp rx=0 tx=0).stop 2024-03-24T10:48:20.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.989+0000 7fa6be7bf740 1 -- 172.21.15.23:0/3969556481 shutdown_connections 2024-03-24T10:48:20.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.989+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/3969556481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd98070 0x5614ecdafe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.989+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/3969556481 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8adb0 0x5614ecda3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.989+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/3969556481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd8a490 0x5614ecd97840 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.989+0000 7fa6be7bf740 1 -- 172.21.15.23:0/3969556481 >> 172.21.15.23:0/3969556481 conn(0x5614ecd84e90 msgr2=0x5614ecd7fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:20.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.989+0000 7fa6be7bf740 1 -- 172.21.15.23:0/3969556481 shutdown_connections 2024-03-24T10:48:20.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.989+0000 7fa6be7bf740 1 -- 172.21.15.23:0/3969556481 wait complete. 2024-03-24T10:48:20.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.989+0000 7fa6be7bf740 1 -- start start 2024-03-24T10:48:20.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.989+0000 7fa6be7bf740 1 Processor -- start 2024-03-24T10:48:20.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6be7bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd8a490 0x5614ecd9bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6be7bf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8adb0 0x5614ecd9c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6be7bf740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd98070 0x5614ece0d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:20.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6be7bf740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5614ecd9ca20 con 0x5614ecd98070 2024-03-24T10:48:20.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6be7bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5614ece0d990 con 0x5614ecd8a490 2024-03-24T10:48:20.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6be7bf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5614ece0dad0 con 0x5614ecd8adb0 2024-03-24T10:48:20.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6be7bf740 10 client.0 _ll_register_callbacks cb 0x7fa6ac009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:20.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6ba09f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd98070 0x5614ece0d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6b989e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd8a490 0x5614ecd9bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6ba09f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd98070 0x5614ece0d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40488/0 (socket says 172.21.15.23:40488) 2024-03-24T10:48:20.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6ba09f640 1 -- 172.21.15.23:0/1664226827 learned_addr learned my addr 172.21.15.23:0/1664226827 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:20.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6b909d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8adb0 0x5614ecd9c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:20.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6ba09f640 1 -- 172.21.15.23:0/1664226827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8adb0 msgr2=0x5614ecd9c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:20.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6ba09f640 1 --2- 172.21.15.23:0/1664226827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8adb0 0x5614ecd9c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6ba09f640 1 -- 172.21.15.23:0/1664226827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd8a490 msgr2=0x5614ecd9bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:20.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6ba09f640 1 --2- 172.21.15.23:0/1664226827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd8a490 0x5614ecd9bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:20.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6ba09f640 1 -- 172.21.15.23:0/1664226827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614ecca76d0 con 0x5614ecd98070 2024-03-24T10:48:20.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa6ba09f640 1 --2- 172.21.15.23:0/1664226827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd98070 0x5614ece0d450 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac003ea0 tx=0x7fa6ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449bcf82903b1032 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:20.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.990+0000 7fa69d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:20.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa69d7fa640 1 -- 172.21.15.23:0/1664226827 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac0047d0 con 0x5614ecd98070 2024-03-24T10:48:20.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa69d7fa640 1 -- 172.21.15.23:0/1664226827 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7fa6ac004b20 con 0x5614ecd98070 2024-03-24T10:48:20.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa6be7bf740 1 -- 172.21.15.23:0/1664226827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6b0019c70 con 0x5614ecd98070 2024-03-24T10:48:20.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa69d7fa640 1 -- 172.21.15.23:0/1664226827 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa6ac004ce0 con 0x5614ecd98070 2024-03-24T10:48:20.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa6be7bf740 10 client.17952 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:20.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa6be7bf740 1 -- 172.21.15.23:0/1664226827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5614eccb1970 con 0x5614ecd98070 2024-03-24T10:48:20.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa69d7fa640 1 -- 172.21.15.23:0/1664226827 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6ac037080 con 0x5614ecd98070 2024-03-24T10:48:20.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa69d7fa640 1 -- 172.21.15.23:0/1664226827 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fa6ac022450 con 0x5614ecd98070 2024-03-24T10:48:20.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa6be7bf740 10 client.17952 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:20.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa6be7bf740 10 client.17952 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:21.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa6be7bf740 20 client.17952 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:21.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/1664226827 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614ece0f390 0x5614ece117b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa6be7bf740 4 client.17952 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:48:21.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa6be7bf740 1 -- 172.21.15.23:0/1664226827 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5614ec9f0860 con 0x5614ece0f390 2024-03-24T10:48:21.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.991+0000 7fa6b989e640 1 --2- 172.21.15.23:0/1664226827 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614ece0f390 0x5614ece117b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.992+0000 7fa6b989e640 1 --2- 172.21.15.23:0/1664226827 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614ece0f390 0x5614ece117b0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3a85333113deb37 server_cookie=6e49b4119874baee in_seq=0 out_seq=0 2024-03-24T10:48:21.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.992+0000 7fa69d7fa640 10 client.17952 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:21.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa69d7fa640 1 -- 172.21.15.23:0/1664226827 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5614ec9f0860 con 0x5614ece0f390 2024-03-24T10:48:21.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa69d7fa640 10 client.17952 handle_command_reply: tid=0 2024-03-24T10:48:21.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa6be7bf740 1 client.17952 shutdown 2024-03-24T10:48:21.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa6be7bf740 1 -- 172.21.15.23:0/1664226827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd98070 msgr2=0x5614ece0d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/1664226827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd98070 0x5614ece0d450 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac003ea0 tx=0x7fa6ac004a40 comp rx=0 tx=0).stop 2024-03-24T10:48:21.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa6be7bf740 1 -- 172.21.15.23:0/1664226827 shutdown_connections 2024-03-24T10:48:21.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/1664226827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614ecd98070 0x5614ece0d450 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/1664226827 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5614ece0f390 0x5614ece117b0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/1664226827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614ecd8adb0 0x5614ecd9c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa6be7bf740 1 --2- 172.21.15.23:0/1664226827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5614ecd8a490 0x5614ecd9bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa6be7bf740 1 -- 172.21.15.23:0/1664226827 >> 172.21.15.23:0/1664226827 conn(0x5614ecd84e90 msgr2=0x5614ecd83660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:21.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa69d7fa640 0 client.17952 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:21.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa6be7bf740 1 -- 172.21.15.23:0/1664226827 shutdown_connections 2024-03-24T10:48:21.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa6be7bf740 1 -- 172.21.15.23:0/1664226827 wait complete. 2024-03-24T10:48:21.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.993+0000 7fa6be7bf740 20 client.17952 trim_cache size 0 max 16384 2024-03-24T10:48:21.002 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:48:21.002 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:48:21.003 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "83ee85a5-ebda-4f3a-b419-4eda76944c5b", 2024-03-24T10:48:21.003 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:48:21.003 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:48:21.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.994+0000 7fa6bbb02640 1 -- 172.21.15.23:0/1671670435 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa6a00a2230 msgr2=0x7fa6a00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.994+0000 7fa6bbb02640 1 --2- 172.21.15.23:0/1671670435 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa6a00a2230 0x7fa6a00a46f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac005c40 tx=0x7fa6ac004ab0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.994+0000 7fa6bbb02640 1 -- 172.21.15.23:0/1671670435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4100f60 msgr2=0x7fa6b41990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.994+0000 7fa6bbb02640 1 --2- 172.21.15.23:0/1671670435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4100f60 0x7fa6b41990f0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fa6a402c240 tx=0x7fa6a402c6b0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.994+0000 7fa6bbb02640 1 -- 172.21.15.23:0/1671670435 shutdown_connections 2024-03-24T10:48:21.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.994+0000 7fa6bbb02640 1 --2- 172.21.15.23:0/1671670435 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa6a00a2230 0x7fa6a00a46f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.994+0000 7fa6bbb02640 1 --2- 172.21.15.23:0/1671670435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b4106660 0x7fa6b41a3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.994+0000 7fa6bbb02640 1 --2- 172.21.15.23:0/1671670435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6b4100f60 0x7fa6b41990f0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.994+0000 7fa6bbb02640 1 --2- 172.21.15.23:0/1671670435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6b41005a0 0x7fa6b4198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.994+0000 7fa6bbb02640 1 -- 172.21.15.23:0/1671670435 >> 172.21.15.23:0/1671670435 conn(0x7fa6b4077710 msgr2=0x7fa6b40ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:21.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.994+0000 7fa6bbb02640 1 -- 172.21.15.23:0/1671670435 shutdown_connections 2024-03-24T10:48:21.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:20.994+0000 7fa6bbb02640 1 -- 172.21.15.23:0/1671670435 wait complete. 2024-03-24T10:48:21.006 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:48:21.006 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:48:21.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.151+0000 7f3e3a6d4640 1 -- 172.21.15.23:0/4278994364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e340688f0 msgr2=0x7f3e34068d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.151+0000 7f3e3a6d4640 1 --2- 172.21.15.23:0/4278994364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e340688f0 0x7f3e34068d70 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009930 tx=0x7f3e2802bc30 comp rx=0 tx=0).stop 2024-03-24T10:48:21.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.151+0000 7f3e3a6d4640 1 -- 172.21.15.23:0/4278994364 shutdown_connections 2024-03-24T10:48:21.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.151+0000 7f3e3a6d4640 1 --2- 172.21.15.23:0/4278994364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e340692b0 0x7f3e3410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.151+0000 7f3e3a6d4640 1 --2- 172.21.15.23:0/4278994364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e340688f0 0x7f3e34068d70 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.151+0000 7f3e3a6d4640 1 --2- 172.21.15.23:0/4278994364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e34107690 0x7f3e34107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.151+0000 7f3e3a6d4640 1 -- 172.21.15.23:0/4278994364 >> 172.21.15.23:0/4278994364 conn(0x7f3e34077710 msgr2=0x7f3e34077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:21.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.151+0000 7f3e3a6d4640 1 -- 172.21.15.23:0/4278994364 shutdown_connections 2024-03-24T10:48:21.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.151+0000 7f3e3a6d4640 1 -- 172.21.15.23:0/4278994364 wait complete. 2024-03-24T10:48:21.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.152+0000 7f3e3a6d4640 1 Processor -- start 2024-03-24T10:48:21.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.152+0000 7f3e3a6d4640 1 -- start start 2024-03-24T10:48:21.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.152+0000 7f3e3a6d4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e340688f0 0x7f3e341005d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.152+0000 7f3e3a6d4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e340692b0 0x7f3e34100b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.152+0000 7f3e3a6d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e34107690 0x7f3e34101050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.152+0000 7f3e3a6d4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341a6170 con 0x7f3e340692b0 2024-03-24T10:48:21.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.152+0000 7f3e3a6d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341a62e0 con 0x7f3e34107690 2024-03-24T10:48:21.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.152+0000 7f3e3a6d4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341a64a0 con 0x7f3e340688f0 2024-03-24T10:48:21.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e340692b0 0x7f3e34100b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e340692b0 0x7f3e34100b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40518/0 (socket says 172.21.15.23:40518) 2024-03-24T10:48:21.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e337fe640 1 -- 172.21.15.23:0/3847676596 learned_addr learned my addr 172.21.15.23:0/3847676596 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:21.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e33fff640 1 --2- 172.21.15.23:0/3847676596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e340688f0 0x7f3e341005d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e38c71640 1 --2- 172.21.15.23:0/3847676596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e34107690 0x7f3e34101050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e337fe640 1 -- 172.21.15.23:0/3847676596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e340688f0 msgr2=0x7f3e341005d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e337fe640 1 --2- 172.21.15.23:0/3847676596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e340688f0 0x7f3e341005d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e337fe640 1 -- 172.21.15.23:0/3847676596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e34107690 msgr2=0x7f3e34101050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e337fe640 1 --2- 172.21.15.23:0/3847676596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e34107690 0x7f3e34101050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e337fe640 1 -- 172.21.15.23:0/3847676596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e28009560 con 0x7f3e340692b0 2024-03-24T10:48:21.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e337fe640 1 --2- 172.21.15.23:0/3847676596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e340692b0 0x7f3e34100b10 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009230 tx=0x7f3e2802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:21.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e317fa640 1 -- 172.21.15.23:0/3847676596 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e28004350 con 0x7f3e340692b0 2024-03-24T10:48:21.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e3a6d4640 1 -- 172.21.15.23:0/3847676596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e341a6720 con 0x7f3e340692b0 2024-03-24T10:48:21.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e317fa640 1 -- 172.21.15.23:0/3847676596 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e280044b0 con 0x7f3e340692b0 2024-03-24T10:48:21.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e317fa640 1 -- 172.21.15.23:0/3847676596 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e28003d00 con 0x7f3e340692b0 2024-03-24T10:48:21.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.153+0000 7f3e3a6d4640 1 -- 172.21.15.23:0/3847676596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e341a6c30 con 0x7f3e340692b0 2024-03-24T10:48:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e317fa640 1 -- 172.21.15.23:0/3847676596 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f3e28004df0 con 0x7f3e340692b0 2024-03-24T10:48:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e3d38e740 1 Processor -- start 2024-03-24T10:48:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e3d38e740 1 -- start start 2024-03-24T10:48:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e3d38e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5618646331d0 0x5618646335b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e3d38e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561864633af0 0x56186463c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e3d38e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x56186463eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e3d38e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x561864557f30 con 0x561864633af0 2024-03-24T10:48:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e3d38e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5618645a1810 con 0x56186463c6d0 2024-03-24T10:48:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e3d38e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56186459ed40 con 0x5618646331d0 2024-03-24T10:48:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e33fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561864633af0 0x56186463c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e33fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561864633af0 0x56186463c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40534/0 (socket says 172.21.15.23:40534) 2024-03-24T10:48:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e33fff640 1 -- 172.21.15.23:0/2091665719 learned_addr learned my addr 172.21.15.23:0/2091665719 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e317fa640 1 --2- 172.21.15.23:0/3847676596 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3e1c0a2230 0x7f3e1c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e38c71640 1 --2- 172.21.15.23:0/2091665719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5618646331d0 0x5618646335b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e337fe640 1 --2- 172.21.15.23:0/2091665719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x56186463eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e38c71640 1 --2- 172.21.15.23:0/3847676596 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3e1c0a2230 0x7f3e1c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e33fff640 1 -- 172.21.15.23:0/2091665719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5618646331d0 msgr2=0x5618646335b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:21.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e33fff640 1 --2- 172.21.15.23:0/2091665719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5618646331d0 0x5618646335b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e33fff640 1 -- 172.21.15.23:0/2091665719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 msgr2=0x56186463eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:21.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e33fff640 1 --2- 172.21.15.23:0/2091665719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x56186463eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.156+0000 7f3e33fff640 1 -- 172.21.15.23:0/2091665719 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56186455bea0 con 0x561864633af0 2024-03-24T10:48:21.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.157+0000 7f3e317fa640 1 -- 172.21.15.23:0/3847676596 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f3e280efff0 con 0x7f3e340692b0 2024-03-24T10:48:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.157+0000 7f3e33fff640 1 --2- 172.21.15.23:0/2091665719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561864633af0 0x56186463c190 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f3e24009c70 tx=0x7f3e24011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3364af3760ef97e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.157+0000 7f3e1a7fc640 1 -- 172.21.15.23:0/2091665719 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24004230 con 0x561864633af0 2024-03-24T10:48:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.157+0000 7f3e38c71640 1 --2- 172.21.15.23:0/3847676596 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3e1c0a2230 0x7f3e1c0a46f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c005c40 tx=0x7f3e2c004ab0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.157+0000 7f3e1a7fc640 1 -- 172.21.15.23:0/2091665719 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e24004390 con 0x561864633af0 2024-03-24T10:48:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.157+0000 7f3e1a7fc640 1 -- 172.21.15.23:0/2091665719 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24019760 con 0x561864633af0 2024-03-24T10:48:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2091665719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561864633af0 msgr2=0x56186463c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2091665719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561864633af0 0x56186463c190 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f3e24009c70 tx=0x7f3e24011810 comp rx=0 tx=0).stop 2024-03-24T10:48:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2091665719 shutdown_connections 2024-03-24T10:48:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2091665719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x56186463eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2091665719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561864633af0 0x56186463c190 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2091665719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5618646331d0 0x5618646335b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2091665719 >> 172.21.15.23:0/2091665719 conn(0x56186462d900 msgr2=0x56186462dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:21.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2091665719 shutdown_connections 2024-03-24T10:48:21.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2091665719 wait complete. 2024-03-24T10:48:21.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 -- start start 2024-03-24T10:48:21.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 Processor -- start 2024-03-24T10:48:21.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5618646331d0 0x561864639710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561864633af0 0x561864639c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x56186463a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56186459a520 con 0x561864633af0 2024-03-24T10:48:21.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56186426d430 con 0x56186463c6d0 2024-03-24T10:48:21.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561864229e00 con 0x5618646331d0 2024-03-24T10:48:21.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.158+0000 7f3e3d38e740 10 client.0 _ll_register_callbacks cb 0x561864538d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:21.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561864633af0 0x561864639c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561864633af0 0x561864639c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40548/0 (socket says 172.21.15.23:40548) 2024-03-24T10:48:21.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e337fe640 1 -- 172.21.15.23:0/2090033336 learned_addr learned my addr 172.21.15.23:0/2090033336 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:21.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e33fff640 1 --2- 172.21.15.23:0/2090033336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5618646331d0 0x561864639710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e38c71640 1 --2- 172.21.15.23:0/2090033336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x56186463a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e337fe640 1 -- 172.21.15.23:0/2090033336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5618646331d0 msgr2=0x561864639710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:21.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e337fe640 1 --2- 172.21.15.23:0/2090033336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5618646331d0 0x561864639710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e337fe640 1 -- 172.21.15.23:0/2090033336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 msgr2=0x56186463a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:21.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e337fe640 1 --2- 172.21.15.23:0/2090033336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x56186463a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e337fe640 1 -- 172.21.15.23:0/2090033336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e240098d0 con 0x561864633af0 2024-03-24T10:48:21.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e337fe640 1 --2- 172.21.15.23:0/2090033336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561864633af0 0x561864639c50 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f3e280b8ec0 tx=0x7f3e28005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6856cc22138e1b19 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:21.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e137fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:21.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e137fe640 1 -- 172.21.15.23:0/2090033336 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e280035f0 con 0x561864633af0 2024-03-24T10:48:21.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e137fe640 1 -- 172.21.15.23:0/2090033336 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f3e280bd160 con 0x561864633af0 2024-03-24T10:48:21.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2090033336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5618642997e0 con 0x561864633af0 2024-03-24T10:48:21.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e137fe640 1 -- 172.21.15.23:0/2090033336 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3e280bca60 con 0x561864633af0 2024-03-24T10:48:21.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e3d38e740 10 client.17964 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:21.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2090033336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561864598ba0 con 0x561864633af0 2024-03-24T10:48:21.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.159+0000 7f3e137fe640 1 -- 172.21.15.23:0/2090033336 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e280c9080 con 0x561864633af0 2024-03-24T10:48:21.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.160+0000 7f3e137fe640 1 -- 172.21.15.23:0/2090033336 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f3e2803c070 con 0x561864633af0 2024-03-24T10:48:21.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.160+0000 7f3e3d38e740 10 client.17964 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:21.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.160+0000 7f3e3d38e740 10 client.17964 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:21.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.160+0000 7f3e3d38e740 20 client.17964 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:21.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.160+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2090033336 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56186464c9e0 0x56186464ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.160+0000 7f3e3d38e740 4 client.17964 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:48:21.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.160+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2090033336 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618642d6720 con 0x56186464c9e0 2024-03-24T10:48:21.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.160+0000 7f3e33fff640 1 --2- 172.21.15.23:0/2090033336 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56186464c9e0 0x56186464ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.160+0000 7f3e137fe640 10 client.17964 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:21.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.160+0000 7f3e33fff640 1 --2- 172.21.15.23:0/2090033336 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56186464c9e0 0x56186464ee00 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5caf23fe6a38e2a server_cookie=8f189e4c762145f0 in_seq=0 out_seq=0 2024-03-24T10:48:21.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e137fe640 1 -- 172.21.15.23:0/2090033336 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5618642d6720 con 0x56186464c9e0 2024-03-24T10:48:21.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e137fe640 10 client.17964 handle_command_reply: tid=0 2024-03-24T10:48:21.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e3d38e740 1 client.17964 shutdown 2024-03-24T10:48:21.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2090033336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561864633af0 msgr2=0x561864639c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2090033336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561864633af0 0x561864639c50 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f3e280b8ec0 tx=0x7f3e28005c90 comp rx=0 tx=0).stop 2024-03-24T10:48:21.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2090033336 shutdown_connections 2024-03-24T10:48:21.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2090033336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x56186463a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2090033336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561864633af0 0x561864639c50 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2090033336 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x56186464c9e0 0x56186464ee00 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2090033336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5618646331d0 0x561864639710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e137fe640 0 client.17964 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:21.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2090033336 >> 172.21.15.23:0/2090033336 conn(0x56186462d900 msgr2=0x56186462f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:21.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2090033336 shutdown_connections 2024-03-24T10:48:21.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2090033336 wait complete. 2024-03-24T10:48:21.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.161+0000 7f3e3d38e740 20 client.17964 trim_cache size 0 max 16384 2024-03-24T10:48:21.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e3d38e740 1 Processor -- start 2024-03-24T10:48:21.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e3d38e740 1 -- start start 2024-03-24T10:48:21.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e3d38e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5618646331d0 0x561864647f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e3d38e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561864633af0 0x561864639fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e3d38e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x561864654710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e3d38e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56186451f430 con 0x5618646331d0 2024-03-24T10:48:21.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e3d38e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56186462e900 con 0x56186463c6d0 2024-03-24T10:48:21.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e3d38e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561864635300 con 0x561864633af0 2024-03-24T10:48:21.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e33fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5618646331d0 0x561864647f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e33fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5618646331d0 0x561864647f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40552/0 (socket says 172.21.15.23:40552) 2024-03-24T10:48:21.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e33fff640 1 -- 172.21.15.23:0/3678657111 learned_addr learned my addr 172.21.15.23:0/3678657111 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:21.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e38c71640 1 --2- 172.21.15.23:0/3678657111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x561864654710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e337fe640 1 --2- 172.21.15.23:0/3678657111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561864633af0 0x561864639fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e33fff640 1 -- 172.21.15.23:0/3678657111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561864633af0 msgr2=0x561864639fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:21.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e33fff640 1 --2- 172.21.15.23:0/3678657111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561864633af0 0x561864639fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e33fff640 1 -- 172.21.15.23:0/3678657111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 msgr2=0x561864654710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:21.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e33fff640 1 --2- 172.21.15.23:0/3678657111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x561864654710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e33fff640 1 -- 172.21.15.23:0/3678657111 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561864534b20 con 0x5618646331d0 2024-03-24T10:48:21.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e38c71640 1 --2- 172.21.15.23:0/3678657111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x561864654710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:48:21.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e337fe640 1 --2- 172.21.15.23:0/3678657111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561864633af0 0x561864639fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:48:21.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e33fff640 1 --2- 172.21.15.23:0/3678657111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5618646331d0 0x561864647f60 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3e24009c70 tx=0x7f3e24005610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1e698290c99be6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:21.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e19ffb640 1 -- 172.21.15.23:0/3678657111 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24009e20 con 0x5618646331d0 2024-03-24T10:48:21.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e19ffb640 1 -- 172.21.15.23:0/3678657111 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e24003ad0 con 0x5618646331d0 2024-03-24T10:48:21.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.179+0000 7f3e19ffb640 1 -- 172.21.15.23:0/3678657111 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2401a6a0 con 0x5618646331d0 2024-03-24T10:48:21.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.180+0000 7f3e3d38e740 1 -- 172.21.15.23:0/3678657111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5618646331d0 msgr2=0x561864647f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.180+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/3678657111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5618646331d0 0x561864647f60 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3e24009c70 tx=0x7f3e24005610 comp rx=0 tx=0).stop 2024-03-24T10:48:21.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.180+0000 7f3e3d38e740 1 -- 172.21.15.23:0/3678657111 shutdown_connections 2024-03-24T10:48:21.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.180+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/3678657111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x561864654710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.180+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/3678657111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561864633af0 0x561864639fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.180+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/3678657111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5618646331d0 0x561864647f60 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.180+0000 7f3e3d38e740 1 -- 172.21.15.23:0/3678657111 >> 172.21.15.23:0/3678657111 conn(0x56186462d900 msgr2=0x56186462cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:21.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.180+0000 7f3e3d38e740 1 -- 172.21.15.23:0/3678657111 shutdown_connections 2024-03-24T10:48:21.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.180+0000 7f3e3d38e740 1 -- 172.21.15.23:0/3678657111 wait complete. 2024-03-24T10:48:21.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.180+0000 7f3e3d38e740 1 -- start start 2024-03-24T10:48:21.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e3d38e740 1 Processor -- start 2024-03-24T10:48:21.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e3d38e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5618646331d0 0x561864644460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e3d38e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561864633af0 0x5618646449a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e3d38e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x5618646409e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e3d38e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5618646410b0 con 0x5618646331d0 2024-03-24T10:48:21.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e3d38e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561864641220 con 0x56186463c6d0 2024-03-24T10:48:21.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e3d38e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5618646413e0 con 0x561864633af0 2024-03-24T10:48:21.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e3d38e740 10 client.0 _ll_register_callbacks cb 0x7f3e2804afb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:21.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e33fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5618646331d0 0x561864644460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e33fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5618646331d0 0x561864644460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40560/0 (socket says 172.21.15.23:40560) 2024-03-24T10:48:21.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e33fff640 1 -- 172.21.15.23:0/2867726351 learned_addr learned my addr 172.21.15.23:0/2867726351 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:21.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e337fe640 1 --2- 172.21.15.23:0/2867726351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561864633af0 0x5618646449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e38c71640 1 --2- 172.21.15.23:0/2867726351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x5618646409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e33fff640 1 -- 172.21.15.23:0/2867726351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561864633af0 msgr2=0x5618646449a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:21.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e33fff640 1 --2- 172.21.15.23:0/2867726351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561864633af0 0x5618646449a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e33fff640 1 -- 172.21.15.23:0/2867726351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 msgr2=0x5618646409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:21.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e33fff640 1 --2- 172.21.15.23:0/2867726351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x5618646409e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e33fff640 1 -- 172.21.15.23:0/2867726351 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e2401e040 con 0x5618646331d0 2024-03-24T10:48:21.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e33fff640 1 --2- 172.21.15.23:0/2867726351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5618646331d0 0x561864644460 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f3e24000c00 tx=0x7f3e24009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e562ff19772b874d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:21.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.181+0000 7f3e137fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:21.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e137fe640 1 -- 172.21.15.23:0/2867726351 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24009310 con 0x5618646331d0 2024-03-24T10:48:21.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e137fe640 1 -- 172.21.15.23:0/2867726351 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f3e24009e20 con 0x5618646331d0 2024-03-24T10:48:21.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2867726351 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e24004390 con 0x5618646331d0 2024-03-24T10:48:21.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e137fe640 1 -- 172.21.15.23:0/2867726351 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3e24003d80 con 0x5618646331d0 2024-03-24T10:48:21.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e3d38e740 10 client.17970 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:21.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2867726351 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56186455a700 con 0x5618646331d0 2024-03-24T10:48:21.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e137fe640 1 -- 172.21.15.23:0/2867726351 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e24044020 con 0x5618646331d0 2024-03-24T10:48:21.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e137fe640 1 -- 172.21.15.23:0/2867726351 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f3e24030b20 con 0x5618646331d0 2024-03-24T10:48:21.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e3d38e740 10 client.17970 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:21.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e3d38e740 10 client.17970 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:21.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e3d38e740 20 client.17970 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:21.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2867726351 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5618646b3e20 0x5618646b6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e3d38e740 4 client.17970 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:48:21.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2867726351 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561864325890 con 0x5618646b3e20 2024-03-24T10:48:21.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.182+0000 7f3e337fe640 1 --2- 172.21.15.23:0/2867726351 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5618646b3e20 0x5618646b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e137fe640 10 client.17970 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:21.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e337fe640 1 --2- 172.21.15.23:0/2867726351 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5618646b3e20 0x5618646b6240 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5a2493e8c69c14 server_cookie=dd383d4739a0c093 in_seq=0 out_seq=0 2024-03-24T10:48:21.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e137fe640 1 -- 172.21.15.23:0/2867726351 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561864325890 con 0x5618646b3e20 2024-03-24T10:48:21.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e137fe640 10 client.17970 handle_command_reply: tid=0 2024-03-24T10:48:21.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e3d38e740 1 client.17970 shutdown 2024-03-24T10:48:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2867726351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5618646331d0 msgr2=0x561864644460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2867726351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5618646331d0 0x561864644460 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f3e24000c00 tx=0x7f3e24009830 comp rx=0 tx=0).stop 2024-03-24T10:48:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2867726351 shutdown_connections 2024-03-24T10:48:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2867726351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56186463c6d0 0x5618646409e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2867726351 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5618646b3e20 0x5618646b6240 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2867726351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561864633af0 0x5618646449a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e3d38e740 1 --2- 172.21.15.23:0/2867726351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5618646331d0 0x561864644460 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e137fe640 0 client.17970 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:21.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2867726351 >> 172.21.15.23:0/2867726351 conn(0x56186462d900 msgr2=0x56186462ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:21.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2867726351 shutdown_connections 2024-03-24T10:48:21.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e3d38e740 1 -- 172.21.15.23:0/2867726351 wait complete. 2024-03-24T10:48:21.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.183+0000 7f3e3d38e740 20 client.17970 trim_cache size 0 max 16384 2024-03-24T10:48:21.194 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:48:21.194 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:48:21.195 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:48:21.195 INFO:teuthology.orchestra.run.smithi023.stdout: "83ee85a5-ebda-4f3a-b419-4eda76944c5b": { 2024-03-24T10:48:21.195 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:48:21.195 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "83ee85a5-ebda-4f3a-b419-4eda76944c5b", 2024-03-24T10:48:21.195 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:48:21.195 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:48:21.195 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:48:21.195 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:48:21.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.184+0000 7f3e3a6d4640 1 -- 172.21.15.23:0/3847676596 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3e1c0a2230 msgr2=0x7f3e1c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.184+0000 7f3e3a6d4640 1 --2- 172.21.15.23:0/3847676596 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3e1c0a2230 0x7f3e1c0a46f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c005c40 tx=0x7f3e2c004ab0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.184+0000 7f3e3a6d4640 1 -- 172.21.15.23:0/3847676596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e340692b0 msgr2=0x7f3e34100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.184+0000 7f3e3a6d4640 1 --2- 172.21.15.23:0/3847676596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e340692b0 0x7f3e34100b10 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009230 tx=0x7f3e2802c650 comp rx=0 tx=0).stop 2024-03-24T10:48:21.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.184+0000 7f3e3a6d4640 1 -- 172.21.15.23:0/3847676596 shutdown_connections 2024-03-24T10:48:21.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.184+0000 7f3e3a6d4640 1 --2- 172.21.15.23:0/3847676596 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f3e1c0a2230 0x7f3e1c0a46f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.184+0000 7f3e3a6d4640 1 --2- 172.21.15.23:0/3847676596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e34107690 0x7f3e34101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.184+0000 7f3e3a6d4640 1 --2- 172.21.15.23:0/3847676596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e340692b0 0x7f3e34100b10 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.184+0000 7f3e3a6d4640 1 --2- 172.21.15.23:0/3847676596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e340688f0 0x7f3e341005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.184+0000 7f3e3a6d4640 1 -- 172.21.15.23:0/3847676596 >> 172.21.15.23:0/3847676596 conn(0x7f3e34077710 msgr2=0x7f3e340fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:21.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.184+0000 7f3e3a6d4640 1 -- 172.21.15.23:0/3847676596 shutdown_connections 2024-03-24T10:48:21.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:21.184+0000 7f3e3a6d4640 1 -- 172.21.15.23:0/3847676596 wait complete. 2024-03-24T10:48:21.199 INFO:tasks.cephfs.filesystem:scrub status for tag:83ee85a5-ebda-4f3a-b419-4eda76944c5b - {'path': '/', 'tag': '83ee85a5-ebda-4f3a-b419-4eda76944c5b', 'options': 'recursive,force'} 2024-03-24T10:48:21.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:21 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:48:21.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:21 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:48:21.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:21 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:48:22.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:22 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:48:22.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:22 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:22.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:22 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:48:22.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:22 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:22.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:22 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:22.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:22 smithi117 ceph-mon[36368]: pgmap v257: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 2.3 MiB/s wr, 94 op/s 2024-03-24T10:48:22.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:22 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:48:22.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:22 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:22.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:22 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:48:22.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:22 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:22.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:22 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:22.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:22 smithi116 ceph-mon[36248]: pgmap v257: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 2.3 MiB/s wr, 94 op/s 2024-03-24T10:48:22.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:22 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:48:22.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:22 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:22.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:22 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:48:22.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:22 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:22.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:22 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:22.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:22 smithi023 ceph-mon[32466]: pgmap v257: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 2.3 MiB/s wr, 94 op/s 2024-03-24T10:48:24.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:24 smithi117 ceph-mon[36368]: pgmap v258: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-03-24T10:48:24.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:24 smithi116 ceph-mon[36248]: pgmap v258: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-03-24T10:48:24.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:24 smithi023 ceph-mon[32466]: pgmap v258: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-03-24T10:48:26.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:26 smithi116 ceph-mon[36248]: pgmap v259: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-03-24T10:48:26.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:26 smithi023 ceph-mon[32466]: pgmap v259: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-03-24T10:48:27.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:26 smithi117 ceph-mon[36368]: pgmap v259: 97 pgs: 97 active+clean; 462 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-03-24T10:48:28.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:28 smithi116 ceph-mon[36248]: pgmap v260: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 2.1 MiB/s wr, 78 op/s 2024-03-24T10:48:28.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:28 smithi023 ceph-mon[32466]: pgmap v260: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 2.1 MiB/s wr, 78 op/s 2024-03-24T10:48:29.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:28 smithi117 ceph-mon[36368]: pgmap v260: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 2.1 MiB/s wr, 78 op/s 2024-03-24T10:48:30.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:30 smithi116 ceph-mon[36248]: pgmap v261: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 1.4 MiB/s wr, 53 op/s 2024-03-24T10:48:30.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:30 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:48:30.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:30 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:48:30.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:30 smithi023 ceph-mon[32466]: pgmap v261: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 1.4 MiB/s wr, 53 op/s 2024-03-24T10:48:30.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:30 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:48:30.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:30 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:48:31.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:30 smithi117 ceph-mon[36368]: pgmap v261: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 1.4 MiB/s wr, 53 op/s 2024-03-24T10:48:31.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:30 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:48:31.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:30 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:48:32.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:32 smithi023 ceph-mon[32466]: pgmap v262: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 2.1 MiB/s wr, 82 op/s 2024-03-24T10:48:33.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:32 smithi117 ceph-mon[36368]: pgmap v262: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 2.1 MiB/s wr, 82 op/s 2024-03-24T10:48:33.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:32 smithi116 ceph-mon[36248]: pgmap v262: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 2.1 MiB/s wr, 82 op/s 2024-03-24T10:48:34.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:34 smithi023 ceph-mon[32466]: pgmap v263: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 1.4 MiB/s wr, 53 op/s 2024-03-24T10:48:35.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:34 smithi117 ceph-mon[36368]: pgmap v263: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 1.4 MiB/s wr, 53 op/s 2024-03-24T10:48:35.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:34 smithi116 ceph-mon[36248]: pgmap v263: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 1.4 MiB/s wr, 53 op/s 2024-03-24T10:48:36.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:36 smithi023 ceph-mon[32466]: pgmap v264: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 1.4 MiB/s wr, 53 op/s 2024-03-24T10:48:37.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:36 smithi117 ceph-mon[36368]: pgmap v264: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 1.4 MiB/s wr, 53 op/s 2024-03-24T10:48:37.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:36 smithi116 ceph-mon[36248]: pgmap v264: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 1.4 MiB/s wr, 53 op/s 2024-03-24T10:48:38.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:38 smithi023 ceph-mon[32466]: pgmap v265: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 1.8 MiB/s wr, 70 op/s 2024-03-24T10:48:39.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:38 smithi117 ceph-mon[36368]: pgmap v265: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 1.8 MiB/s wr, 70 op/s 2024-03-24T10:48:39.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:38 smithi116 ceph-mon[36248]: pgmap v265: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 1.8 MiB/s wr, 70 op/s 2024-03-24T10:48:40.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:40 smithi023 ceph-mon[32466]: pgmap v266: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.1 MiB/s wr, 46 op/s 2024-03-24T10:48:41.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:40 smithi117 ceph-mon[36368]: pgmap v266: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.1 MiB/s wr, 46 op/s 2024-03-24T10:48:41.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:40 smithi116 ceph-mon[36248]: pgmap v266: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.1 MiB/s wr, 46 op/s 2024-03-24T10:48:42.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:42 smithi117 ceph-mon[36368]: pgmap v267: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.1 MiB/s wr, 49 op/s 2024-03-24T10:48:42.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:42 smithi116 ceph-mon[36248]: pgmap v267: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.1 MiB/s wr, 49 op/s 2024-03-24T10:48:42.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:42 smithi023 ceph-mon[32466]: pgmap v267: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.1 MiB/s wr, 49 op/s 2024-03-24T10:48:44.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:44 smithi117 ceph-mon[36368]: pgmap v268: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 323 KiB/s wr, 19 op/s 2024-03-24T10:48:44.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:44 smithi116 ceph-mon[36248]: pgmap v268: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 323 KiB/s wr, 19 op/s 2024-03-24T10:48:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:44 smithi023 ceph-mon[32466]: pgmap v268: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 323 KiB/s wr, 19 op/s 2024-03-24T10:48:46.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:46 smithi117 ceph-mon[36368]: pgmap v269: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 323 KiB/s wr, 19 op/s 2024-03-24T10:48:46.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:46 smithi116 ceph-mon[36248]: pgmap v269: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 323 KiB/s wr, 19 op/s 2024-03-24T10:48:46.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:46 smithi023 ceph-mon[32466]: pgmap v269: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 323 KiB/s wr, 19 op/s 2024-03-24T10:48:48.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:48 smithi117 ceph-mon[36368]: pgmap v270: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 332 KiB/s wr, 22 op/s 2024-03-24T10:48:48.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:48 smithi116 ceph-mon[36248]: pgmap v270: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 332 KiB/s wr, 22 op/s 2024-03-24T10:48:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:48 smithi023 ceph-mon[32466]: pgmap v270: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 332 KiB/s wr, 22 op/s 2024-03-24T10:48:50.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:50 smithi117 ceph-mon[36368]: pgmap v271: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 6 op/s 2024-03-24T10:48:50.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:50 smithi116 ceph-mon[36248]: pgmap v271: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 6 op/s 2024-03-24T10:48:50.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:50 smithi023 ceph-mon[32466]: pgmap v271: 97 pgs: 97 active+clean; 420 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 6 op/s 2024-03-24T10:48:51.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:48:51.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:48:51.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.315+0000 7fd20e46f640 1 -- 172.21.15.23:0/3790322990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2081005a0 msgr2=0x7fd208100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.315+0000 7fd20e46f640 1 --2- 172.21.15.23:0/3790322990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2081005a0 0x7fd208100a20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009930 tx=0x7fd1fc02bc20 comp rx=0 tx=0).stop 2024-03-24T10:48:51.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.315+0000 7fd20e46f640 1 -- 172.21.15.23:0/3790322990 shutdown_connections 2024-03-24T10:48:51.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.315+0000 7fd20e46f640 1 --2- 172.21.15.23:0/3790322990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd208100f60 0x7fd20810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.315+0000 7fd20e46f640 1 --2- 172.21.15.23:0/3790322990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2081005a0 0x7fd208100a20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.315+0000 7fd20e46f640 1 --2- 172.21.15.23:0/3790322990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd208106660 0x7fd208106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.315+0000 7fd20e46f640 1 -- 172.21.15.23:0/3790322990 >> 172.21.15.23:0/3790322990 conn(0x7fd208077710 msgr2=0x7fd208077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.315+0000 7fd20e46f640 1 -- 172.21.15.23:0/3790322990 shutdown_connections 2024-03-24T10:48:51.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.315+0000 7fd20e46f640 1 -- 172.21.15.23:0/3790322990 wait complete. 2024-03-24T10:48:51.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.315+0000 7fd20e46f640 1 Processor -- start 2024-03-24T10:48:51.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd20e46f640 1 -- start start 2024-03-24T10:48:51.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd20e46f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2081005a0 0x7fd208198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd20e46f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd208100f60 0x7fd2081990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd20e46f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd208106660 0x7fd2081a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd20e46f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd2081998f0 con 0x7fd208100f60 2024-03-24T10:48:51.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd20e46f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd208199a60 con 0x7fd208106660 2024-03-24T10:48:51.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd20e46f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd2081a41e0 con 0x7fd2081005a0 2024-03-24T10:48:51.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd2077fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd208100f60 0x7fd2081990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd2077fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd208100f60 0x7fd2081990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46152/0 (socket says 172.21.15.23:46152) 2024-03-24T10:48:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd2077fe640 1 -- 172.21.15.23:0/3850340367 learned_addr learned my addr 172.21.15.23:0/3850340367 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd20ca0c640 1 --2- 172.21.15.23:0/3850340367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd208106660 0x7fd2081a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd207fff640 1 --2- 172.21.15.23:0/3850340367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2081005a0 0x7fd208198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd2077fe640 1 -- 172.21.15.23:0/3850340367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2081005a0 msgr2=0x7fd208198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd2077fe640 1 --2- 172.21.15.23:0/3850340367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2081005a0 0x7fd208198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd2077fe640 1 -- 172.21.15.23:0/3850340367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd208106660 msgr2=0x7fd2081a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd2077fe640 1 --2- 172.21.15.23:0/3850340367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd208106660 0x7fd2081a3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.316+0000 7fd2077fe640 1 -- 172.21.15.23:0/3850340367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd200009840 con 0x7fd208100f60 2024-03-24T10:48:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.317+0000 7fd2077fe640 1 --2- 172.21.15.23:0/3850340367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd208100f60 0x7fd2081990e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009230 tx=0x7fd1fc02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.317+0000 7fd2057fa640 1 -- 172.21.15.23:0/3850340367 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc004350 con 0x7fd208100f60 2024-03-24T10:48:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.317+0000 7fd20e46f640 1 -- 172.21.15.23:0/3850340367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1fc009560 con 0x7fd208100f60 2024-03-24T10:48:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.317+0000 7fd2057fa640 1 -- 172.21.15.23:0/3850340367 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1fc0044b0 con 0x7fd208100f60 2024-03-24T10:48:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.317+0000 7fd2057fa640 1 -- 172.21.15.23:0/3850340367 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc003bf0 con 0x7fd208100f60 2024-03-24T10:48:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.317+0000 7fd20e46f640 1 -- 172.21.15.23:0/3850340367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2081a4830 con 0x7fd208100f60 2024-03-24T10:48:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.319+0000 7fd2057fa640 1 -- 172.21.15.23:0/3850340367 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fd1fc003d50 con 0x7fd208100f60 2024-03-24T10:48:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.319+0000 7fd21104e740 1 Processor -- start 2024-03-24T10:48:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.319+0000 7fd21104e740 1 -- start start 2024-03-24T10:48:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.319+0000 7fd2057fa640 1 --2- 172.21.15.23:0/3850340367 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd1f00a2230 0x7fd1f00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd21104e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c71d0 0x55e1869c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd21104e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c7af0 0x55e1869d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd21104e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869d06d0 0x55e1869d2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd21104e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e1868ebf30 con 0x55e1869c7af0 2024-03-24T10:48:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd21104e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e186935810 con 0x55e1869c71d0 2024-03-24T10:48:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd20ca0c640 1 --2- 172.21.15.23:0/3850340367 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd1f00a2230 0x7fd1f00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd21104e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e186932d40 con 0x55e1869d06d0 2024-03-24T10:48:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd2057fa640 1 -- 172.21.15.23:0/3850340367 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7fd1fc0effc0 con 0x7fd208100f60 2024-03-24T10:48:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd2077fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c7af0 0x55e1869d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd2077fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c7af0 0x55e1869d0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46166/0 (socket says 172.21.15.23:46166) 2024-03-24T10:48:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd207fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c71d0 0x55e1869c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd2077fe640 1 -- 172.21.15.23:0/981759363 learned_addr learned my addr 172.21.15.23:0/981759363 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd20ca0c640 1 --2- 172.21.15.23:0/981759363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869d06d0 0x55e1869d2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd207fff640 1 -- 172.21.15.23:0/981759363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869d06d0 msgr2=0x55e1869d2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd207fff640 1 --2- 172.21.15.23:0/981759363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869d06d0 0x55e1869d2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd207fff640 1 -- 172.21.15.23:0/981759363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c7af0 msgr2=0x55e1869d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd207fff640 1 --2- 172.21.15.23:0/981759363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c7af0 0x55e1869d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd207fff640 1 -- 172.21.15.23:0/981759363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1868efea0 con 0x55e1869c71d0 2024-03-24T10:48:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd2077fe640 1 --2- 172.21.15.23:0/981759363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c7af0 0x55e1869d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd20ca0c640 1 --2- 172.21.15.23:0/3850340367 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd1f00a2230 0x7fd1f00a46f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd200005fd0 tx=0x7fd200013d10 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd207fff640 1 --2- 172.21.15.23:0/981759363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c71d0 0x55e1869c75b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009c20 tx=0x7fd1f8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=349d35eba809d1d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.320+0000 7fd1ee7fc640 1 -- 172.21.15.23:0/981759363 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8004050 con 0x55e1869c71d0 2024-03-24T10:48:51.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.321+0000 7fd1ee7fc640 1 -- 172.21.15.23:0/981759363 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1f8005500 con 0x55e1869c71d0 2024-03-24T10:48:51.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.321+0000 7fd1ee7fc640 1 -- 172.21.15.23:0/981759363 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8004d00 con 0x55e1869c71d0 2024-03-24T10:48:51.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.321+0000 7fd21104e740 1 -- 172.21.15.23:0/981759363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c71d0 msgr2=0x55e1869c75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.321+0000 7fd21104e740 1 --2- 172.21.15.23:0/981759363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c71d0 0x55e1869c75b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009c20 tx=0x7fd1f8011810 comp rx=0 tx=0).stop 2024-03-24T10:48:51.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.321+0000 7fd21104e740 1 -- 172.21.15.23:0/981759363 shutdown_connections 2024-03-24T10:48:51.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.321+0000 7fd21104e740 1 --2- 172.21.15.23:0/981759363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869d06d0 0x55e1869d2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.321+0000 7fd21104e740 1 --2- 172.21.15.23:0/981759363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c7af0 0x55e1869d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.321+0000 7fd21104e740 1 --2- 172.21.15.23:0/981759363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c71d0 0x55e1869c75b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.321+0000 7fd21104e740 1 -- 172.21.15.23:0/981759363 >> 172.21.15.23:0/981759363 conn(0x55e1869c1900 msgr2=0x55e1869c1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.321+0000 7fd21104e740 1 -- 172.21.15.23:0/981759363 shutdown_connections 2024-03-24T10:48:51.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.321+0000 7fd21104e740 1 -- 172.21.15.23:0/981759363 wait complete. 2024-03-24T10:48:51.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd21104e740 1 -- start start 2024-03-24T10:48:51.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd21104e740 1 Processor -- start 2024-03-24T10:48:51.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd21104e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c71d0 0x55e1869cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd21104e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869c7af0 0x55e1869cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd21104e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869d06d0 0x55e1869ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd21104e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e18692e520 con 0x55e1869c71d0 2024-03-24T10:48:51.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd21104e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e186601430 con 0x55e1869d06d0 2024-03-24T10:48:51.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd21104e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e1865bde00 con 0x55e1869c7af0 2024-03-24T10:48:51.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd21104e740 10 client.0 _ll_register_callbacks cb 0x55e1868ccd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:51.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd207fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c71d0 0x55e1869cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd207fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c71d0 0x55e1869cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46180/0 (socket says 172.21.15.23:46180) 2024-03-24T10:48:51.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd207fff640 1 -- 172.21.15.23:0/2666213406 learned_addr learned my addr 172.21.15.23:0/2666213406 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd20ca0c640 1 --2- 172.21.15.23:0/2666213406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869d06d0 0x55e1869ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd2077fe640 1 --2- 172.21.15.23:0/2666213406 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869c7af0 0x55e1869cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd207fff640 1 -- 172.21.15.23:0/2666213406 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869c7af0 msgr2=0x55e1869cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.322+0000 7fd207fff640 1 --2- 172.21.15.23:0/2666213406 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869c7af0 0x55e1869cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.323+0000 7fd207fff640 1 -- 172.21.15.23:0/2666213406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869d06d0 msgr2=0x55e1869ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.323+0000 7fd207fff640 1 --2- 172.21.15.23:0/2666213406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869d06d0 0x55e1869ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.323+0000 7fd207fff640 1 -- 172.21.15.23:0/2666213406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f8009850 con 0x55e1869c71d0 2024-03-24T10:48:51.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.323+0000 7fd20ca0c640 1 --2- 172.21.15.23:0/2666213406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869d06d0 0x55e1869ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:51.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.323+0000 7fd207fff640 1 --2- 172.21.15.23:0/2666213406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c71d0 0x55e1869cd690 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8004920 tx=0x7fd1f8004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7cdbad74816c076 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.323+0000 7fd1e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:51.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.323+0000 7fd1e77fe640 1 -- 172.21.15.23:0/2666213406 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8003a00 con 0x55e1869c71d0 2024-03-24T10:48:51.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.323+0000 7fd1e77fe640 1 -- 172.21.15.23:0/2666213406 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7fd1f8003b60 con 0x55e1869c71d0 2024-03-24T10:48:51.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.323+0000 7fd21104e740 1 -- 172.21.15.23:0/2666213406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e18662d7e0 con 0x55e1869c71d0 2024-03-24T10:48:51.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.323+0000 7fd1e77fe640 1 -- 172.21.15.23:0/2666213406 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd1f8003d20 con 0x55e1869c71d0 2024-03-24T10:48:51.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.323+0000 7fd21104e740 10 client.17982 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:51.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.323+0000 7fd21104e740 1 -- 172.21.15.23:0/2666213406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e18692cba0 con 0x55e1869c71d0 2024-03-24T10:48:51.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.323+0000 7fd1e77fe640 1 -- 172.21.15.23:0/2666213406 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e18662d7e0 con 0x55e1869c71d0 2024-03-24T10:48:51.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.323+0000 7fd1e77fe640 1 -- 172.21.15.23:0/2666213406 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fd1f80245b0 con 0x55e1869c71d0 2024-03-24T10:48:51.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.324+0000 7fd21104e740 10 client.17982 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:51.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.324+0000 7fd21104e740 10 client.17982 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:51.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.324+0000 7fd21104e740 20 client.17982 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:51.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.324+0000 7fd21104e740 1 --2- 172.21.15.23:0/2666213406 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e1869e0b40 0x55e1869e2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.324+0000 7fd21104e740 4 client.17982 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:48:51.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.324+0000 7fd21104e740 1 -- 172.21.15.23:0/2666213406 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e18666a720 con 0x55e1869e0b40 2024-03-24T10:48:51.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.324+0000 7fd2077fe640 1 --2- 172.21.15.23:0/2666213406 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e1869e0b40 0x55e1869e2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.324+0000 7fd2077fe640 1 --2- 172.21.15.23:0/2666213406 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e1869e0b40 0x55e1869e2f50 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32f7027321a845d1 server_cookie=b6b27af0248321f7 in_seq=0 out_seq=0 2024-03-24T10:48:51.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.324+0000 7fd1e77fe640 10 client.17982 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:51.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd1e77fe640 1 -- 172.21.15.23:0/2666213406 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e18666a720 con 0x55e1869e0b40 2024-03-24T10:48:51.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd1e77fe640 10 client.17982 handle_command_reply: tid=0 2024-03-24T10:48:51.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd21104e740 1 client.17982 shutdown 2024-03-24T10:48:51.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd21104e740 1 -- 172.21.15.23:0/2666213406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c71d0 msgr2=0x55e1869cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd21104e740 1 --2- 172.21.15.23:0/2666213406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c71d0 0x55e1869cd690 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8004920 tx=0x7fd1f8004280 comp rx=0 tx=0).stop 2024-03-24T10:48:51.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd21104e740 1 -- 172.21.15.23:0/2666213406 shutdown_connections 2024-03-24T10:48:51.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd21104e740 1 --2- 172.21.15.23:0/2666213406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869d06d0 0x55e1869ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd21104e740 1 --2- 172.21.15.23:0/2666213406 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e1869e0b40 0x55e1869e2f50 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd21104e740 1 --2- 172.21.15.23:0/2666213406 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869c7af0 0x55e1869cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd21104e740 1 --2- 172.21.15.23:0/2666213406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c71d0 0x55e1869cd690 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd1e77fe640 0 client.17982 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:51.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd21104e740 1 -- 172.21.15.23:0/2666213406 >> 172.21.15.23:0/2666213406 conn(0x55e1869c1900 msgr2=0x55e1869c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd21104e740 1 -- 172.21.15.23:0/2666213406 shutdown_connections 2024-03-24T10:48:51.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd21104e740 1 -- 172.21.15.23:0/2666213406 wait complete. 2024-03-24T10:48:51.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.325+0000 7fd21104e740 20 client.17982 trim_cache size 0 max 16384 2024-03-24T10:48:51.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd21104e740 1 Processor -- start 2024-03-24T10:48:51.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd21104e740 1 -- start start 2024-03-24T10:48:51.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd21104e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c71d0 0x55e1869cfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd21104e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c7af0 0x55e1869dbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd21104e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869d06d0 0x55e1869c09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd21104e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e1868b3430 con 0x55e1869c71d0 2024-03-24T10:48:51.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd21104e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e1869d3090 con 0x55e1869c7af0 2024-03-24T10:48:51.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd21104e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e1869dd510 con 0x55e1869d06d0 2024-03-24T10:48:51.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd207fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c71d0 0x55e1869cfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd207fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c71d0 0x55e1869cfdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46188/0 (socket says 172.21.15.23:46188) 2024-03-24T10:48:51.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd207fff640 1 -- 172.21.15.23:0/220916862 learned_addr learned my addr 172.21.15.23:0/220916862 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd2077fe640 1 --2- 172.21.15.23:0/220916862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c7af0 0x55e1869dbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd20ca0c640 1 --2- 172.21.15.23:0/220916862 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869d06d0 0x55e1869c09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd207fff640 1 -- 172.21.15.23:0/220916862 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869d06d0 msgr2=0x55e1869c09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd207fff640 1 --2- 172.21.15.23:0/220916862 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869d06d0 0x55e1869c09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd207fff640 1 -- 172.21.15.23:0/220916862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c7af0 msgr2=0x55e1869dbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd207fff640 1 --2- 172.21.15.23:0/220916862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c7af0 0x55e1869dbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd207fff640 1 -- 172.21.15.23:0/220916862 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1868c8b20 con 0x55e1869c71d0 2024-03-24T10:48:51.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd20ca0c640 1 --2- 172.21.15.23:0/220916862 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869d06d0 0x55e1869c09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:48:51.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.343+0000 7fd207fff640 1 --2- 172.21.15.23:0/220916862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c71d0 0x55e1869cfdc0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8030040 tx=0x7fd1f8003fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b8766af7ec47fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.344+0000 7fd1edffb640 1 -- 172.21.15.23:0/220916862 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8005670 con 0x55e1869c71d0 2024-03-24T10:48:51.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.344+0000 7fd1edffb640 1 -- 172.21.15.23:0/220916862 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1f80042f0 con 0x55e1869c71d0 2024-03-24T10:48:51.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.344+0000 7fd1edffb640 1 -- 172.21.15.23:0/220916862 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8004950 con 0x55e1869c71d0 2024-03-24T10:48:51.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.344+0000 7fd21104e740 1 -- 172.21.15.23:0/220916862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c71d0 msgr2=0x55e1869cfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.344+0000 7fd21104e740 1 --2- 172.21.15.23:0/220916862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c71d0 0x55e1869cfdc0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8030040 tx=0x7fd1f8003fe0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.344+0000 7fd21104e740 1 -- 172.21.15.23:0/220916862 shutdown_connections 2024-03-24T10:48:51.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.344+0000 7fd21104e740 1 --2- 172.21.15.23:0/220916862 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869d06d0 0x55e1869c09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.344+0000 7fd21104e740 1 --2- 172.21.15.23:0/220916862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c7af0 0x55e1869dbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.344+0000 7fd21104e740 1 --2- 172.21.15.23:0/220916862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869c71d0 0x55e1869cfdc0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.344+0000 7fd21104e740 1 -- 172.21.15.23:0/220916862 >> 172.21.15.23:0/220916862 conn(0x55e1869c1900 msgr2=0x55e1869c02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.344+0000 7fd21104e740 1 -- 172.21.15.23:0/220916862 shutdown_connections 2024-03-24T10:48:51.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.344+0000 7fd21104e740 1 -- 172.21.15.23:0/220916862 wait complete. 2024-03-24T10:48:51.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.345+0000 7fd21104e740 1 -- start start 2024-03-24T10:48:51.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.345+0000 7fd21104e740 1 Processor -- start 2024-03-24T10:48:51.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.345+0000 7fd21104e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869c71d0 0x55e1869ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.345+0000 7fd21104e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c7af0 0x55e1869ca870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.345+0000 7fd21104e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869d06d0 0x55e1869cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.345+0000 7fd21104e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e186a45e90 con 0x55e1869d06d0 2024-03-24T10:48:51.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.345+0000 7fd21104e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e186a45fd0 con 0x55e1869c7af0 2024-03-24T10:48:51.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.345+0000 7fd21104e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e186a46190 con 0x55e1869c71d0 2024-03-24T10:48:51.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.345+0000 7fd21104e740 10 client.0 _ll_register_callbacks cb 0x7fd1fc005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:51.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.345+0000 7fd20ca0c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869d06d0 0x55e1869cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.345+0000 7fd2077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c7af0 0x55e1869ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.345+0000 7fd20ca0c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869d06d0 0x55e1869cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46192/0 (socket says 172.21.15.23:46192) 2024-03-24T10:48:51.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.345+0000 7fd20ca0c640 1 -- 172.21.15.23:0/1909924273 learned_addr learned my addr 172.21.15.23:0/1909924273 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.345+0000 7fd207fff640 1 --2- 172.21.15.23:0/1909924273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869c71d0 0x55e1869ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd2077fe640 1 -- 172.21.15.23:0/1909924273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869c71d0 msgr2=0x55e1869ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd2077fe640 1 --2- 172.21.15.23:0/1909924273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869c71d0 0x55e1869ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd2077fe640 1 -- 172.21.15.23:0/1909924273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869d06d0 msgr2=0x55e1869cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd2077fe640 1 --2- 172.21.15.23:0/1909924273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869d06d0 0x55e1869cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd2077fe640 1 -- 172.21.15.23:0/1909924273 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e18692cba0 con 0x55e1869c7af0 2024-03-24T10:48:51.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd2077fe640 1 --2- 172.21.15.23:0/1909924273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c7af0 0x55e1869ca870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009900 tx=0x7fd1fc0036f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ed3b9c1414217b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd1e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:48:51.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd1e77fe640 1 -- 172.21.15.23:0/1909924273 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0c5230 con 0x55e1869c7af0 2024-03-24T10:48:51.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd21104e740 1 -- 172.21.15.23:0/1909924273 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1f8005500 con 0x55e1869c7af0 2024-03-24T10:48:51.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd1e77fe640 1 -- 172.21.15.23:0/1909924273 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7fd1fc0b9100 con 0x55e1869c7af0 2024-03-24T10:48:51.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd1e77fe640 1 -- 172.21.15.23:0/1909924273 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd1fc0b92c0 con 0x55e1869c7af0 2024-03-24T10:48:51.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd21104e740 10 client.27875 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:51.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd21104e740 1 -- 172.21.15.23:0/1909924273 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e1868ee700 con 0x55e1869c7af0 2024-03-24T10:48:51.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd1e77fe640 1 -- 172.21.15.23:0/1909924273 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1fc086080 con 0x55e1869c7af0 2024-03-24T10:48:51.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd1e77fe640 1 -- 172.21.15.23:0/1909924273 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fd1fc03c070 con 0x55e1869c7af0 2024-03-24T10:48:51.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd21104e740 10 client.27875 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:51.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd21104e740 10 client.27875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:51.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd21104e740 20 client.27875 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:51.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd21104e740 1 --2- 172.21.15.23:0/1909924273 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e186a47960 0x55e186a49d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd21104e740 4 client.27875 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:48:51.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.346+0000 7fd21104e740 1 -- 172.21.15.23:0/1909924273 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e1866b9890 con 0x55e186a47960 2024-03-24T10:48:51.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd207fff640 1 --2- 172.21.15.23:0/1909924273 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e186a47960 0x55e186a49d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd1e77fe640 10 client.27875 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:51.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd207fff640 1 --2- 172.21.15.23:0/1909924273 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e186a47960 0x55e186a49d50 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e575173b3ecaac7 server_cookie=765e2ee9e93d0f4d in_seq=0 out_seq=0 2024-03-24T10:48:51.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd1e77fe640 1 -- 172.21.15.23:0/1909924273 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e1866b9890 con 0x55e186a47960 2024-03-24T10:48:51.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd1e77fe640 10 client.27875 handle_command_reply: tid=0 2024-03-24T10:48:51.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd21104e740 1 client.27875 shutdown 2024-03-24T10:48:51.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd21104e740 1 -- 172.21.15.23:0/1909924273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c7af0 msgr2=0x55e1869ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd21104e740 1 --2- 172.21.15.23:0/1909924273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c7af0 0x55e1869ca870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009900 tx=0x7fd1fc0036f0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd21104e740 1 -- 172.21.15.23:0/1909924273 shutdown_connections 2024-03-24T10:48:51.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd21104e740 1 --2- 172.21.15.23:0/1909924273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e1869d06d0 0x55e1869cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd21104e740 1 --2- 172.21.15.23:0/1909924273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e1869c7af0 0x55e1869ca870 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd21104e740 1 --2- 172.21.15.23:0/1909924273 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e186a47960 0x55e186a49d50 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd21104e740 1 --2- 172.21.15.23:0/1909924273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e1869c71d0 0x55e1869ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd21104e740 1 -- 172.21.15.23:0/1909924273 >> 172.21.15.23:0/1909924273 conn(0x55e1869c1900 msgr2=0x55e1869cdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd1e77fe640 0 client.27875 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:51.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.347+0000 7fd21104e740 1 -- 172.21.15.23:0/1909924273 shutdown_connections 2024-03-24T10:48:51.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.348+0000 7fd21104e740 1 -- 172.21.15.23:0/1909924273 wait complete. 2024-03-24T10:48:51.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.348+0000 7fd21104e740 20 client.27875 trim_cache size 0 max 16384 2024-03-24T10:48:51.358 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:48:51.358 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:48:51.358 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:48:51.358 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:48:51.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.348+0000 7fd20e46f640 1 -- 172.21.15.23:0/3850340367 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd1f00a2230 msgr2=0x7fd1f00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.348+0000 7fd20e46f640 1 --2- 172.21.15.23:0/3850340367 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd1f00a2230 0x7fd1f00a46f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd200005fd0 tx=0x7fd200013d10 comp rx=0 tx=0).stop 2024-03-24T10:48:51.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.348+0000 7fd20e46f640 1 -- 172.21.15.23:0/3850340367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd208100f60 msgr2=0x7fd2081990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.348+0000 7fd20e46f640 1 --2- 172.21.15.23:0/3850340367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd208100f60 0x7fd2081990e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009230 tx=0x7fd1fc02c640 comp rx=0 tx=0).stop 2024-03-24T10:48:51.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.348+0000 7fd20e46f640 1 -- 172.21.15.23:0/3850340367 shutdown_connections 2024-03-24T10:48:51.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.348+0000 7fd20e46f640 1 --2- 172.21.15.23:0/3850340367 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd1f00a2230 0x7fd1f00a46f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.348+0000 7fd20e46f640 1 --2- 172.21.15.23:0/3850340367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd208106660 0x7fd2081a3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.348+0000 7fd20e46f640 1 --2- 172.21.15.23:0/3850340367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd208100f60 0x7fd2081990e0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.348+0000 7fd20e46f640 1 --2- 172.21.15.23:0/3850340367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2081005a0 0x7fd208198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.348+0000 7fd20e46f640 1 -- 172.21.15.23:0/3850340367 >> 172.21.15.23:0/3850340367 conn(0x7fd208077710 msgr2=0x7fd2080ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.348+0000 7fd20e46f640 1 -- 172.21.15.23:0/3850340367 shutdown_connections 2024-03-24T10:48:51.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.348+0000 7fd20e46f640 1 -- 172.21.15.23:0/3850340367 wait complete. 2024-03-24T10:48:51.361 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:48:51.362 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:51.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.465+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4237363518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c901066a0 msgr2=0x7f2c90106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.465+0000 7f2c95e6b640 1 --2- 172.21.15.23:0/4237363518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c901066a0 0x7f2c90106a80 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f2c80009900 tx=0x7f2c8002bcd0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.465+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4237363518 shutdown_connections 2024-03-24T10:48:51.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.465+0000 7f2c95e6b640 1 --2- 172.21.15.23:0/4237363518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c901016e0 0x7f2c9010d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.465+0000 7f2c95e6b640 1 --2- 172.21.15.23:0/4237363518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c90100d20 0x7f2c901011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.465+0000 7f2c95e6b640 1 --2- 172.21.15.23:0/4237363518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c901066a0 0x7f2c90106a80 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.465+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4237363518 >> 172.21.15.23:0/4237363518 conn(0x7f2c90076ba0 msgr2=0x7f2c900fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.465+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4237363518 shutdown_connections 2024-03-24T10:48:51.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.465+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4237363518 wait complete. 2024-03-24T10:48:51.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c95e6b640 1 Processor -- start 2024-03-24T10:48:51.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c95e6b640 1 -- start start 2024-03-24T10:48:51.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c95e6b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90100d20 0x7f2c90198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c95e6b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c901016e0 0x7f2c901990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c95e6b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c901066a0 0x7f2c901a3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c95e6b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f2c90199900 con 0x7f2c901016e0 2024-03-24T10:48:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c95e6b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c901a4120 con 0x7f2c90100d20 2024-03-24T10:48:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c95e6b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2c901a4260 con 0x7f2c901066a0 2024-03-24T10:48:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c87fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c901016e0 0x7f2c901990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c87fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c901016e0 0x7f2c901990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46212/0 (socket says 172.21.15.23:46212) 2024-03-24T10:48:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c87fff640 1 -- 172.21.15.23:0/4218110867 learned_addr learned my addr 172.21.15.23:0/4218110867 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c8f7fe640 1 --2- 172.21.15.23:0/4218110867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90100d20 0x7f2c90198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c8ffff640 1 --2- 172.21.15.23:0/4218110867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c901066a0 0x7f2c901a3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c87fff640 1 -- 172.21.15.23:0/4218110867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c901066a0 msgr2=0x7f2c901a3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c87fff640 1 --2- 172.21.15.23:0/4218110867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c901066a0 0x7f2c901a3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c87fff640 1 -- 172.21.15.23:0/4218110867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90100d20 msgr2=0x7f2c90198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c87fff640 1 --2- 172.21.15.23:0/4218110867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90100d20 0x7f2c90198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c87fff640 1 -- 172.21.15.23:0/4218110867 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c80009560 con 0x7f2c901016e0 2024-03-24T10:48:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.466+0000 7f2c87fff640 1 --2- 172.21.15.23:0/4218110867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c901016e0 0x7f2c901990a0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800e9f0 tx=0x7f2c7800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.467+0000 7f2c8d7fa640 1 -- 172.21.15.23:0/4218110867 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c78009720 con 0x7f2c901016e0 2024-03-24T10:48:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.467+0000 7f2c8d7fa640 1 -- 172.21.15.23:0/4218110867 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c7800cd80 con 0x7f2c901016e0 2024-03-24T10:48:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.467+0000 7f2c8d7fa640 1 -- 172.21.15.23:0/4218110867 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c78010690 con 0x7f2c901016e0 2024-03-24T10:48:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.467+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4218110867 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c901a44c0 con 0x7f2c901016e0 2024-03-24T10:48:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.467+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4218110867 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c901a4980 con 0x7f2c901016e0 2024-03-24T10:48:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.468+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4218110867 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c90101d30 con 0x7f2c901016e0 2024-03-24T10:48:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.469+0000 7f2c8d7fa640 1 -- 172.21.15.23:0/4218110867 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f2c780046c0 con 0x7f2c901016e0 2024-03-24T10:48:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.469+0000 7f2c8d7fa640 1 --2- 172.21.15.23:0/4218110867 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2c7c0a22b0 0x7f2c7c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.469+0000 7f2c8f7fe640 1 --2- 172.21.15.23:0/4218110867 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2c7c0a22b0 0x7f2c7c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.470+0000 7f2c8d7fa640 1 -- 172.21.15.23:0/4218110867 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f2c78014070 con 0x7f2c901016e0 2024-03-24T10:48:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.470+0000 7f2c8f7fe640 1 --2- 172.21.15.23:0/4218110867 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2c7c0a22b0 0x7f2c7c0a4770 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2c80006010 tx=0x7f2c80005e40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.473+0000 7f2c8d7fa640 1 -- 172.21.15.23:0/4218110867 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2c780972d0 con 0x7f2c901016e0 2024-03-24T10:48:51.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.695+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4218110867 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c90068950 con 0x7f2c901016e0 2024-03-24T10:48:51.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.695+0000 7f2c8d7fa640 1 -- 172.21.15.23:0/4218110867 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11869 (secure 0 0 0) 0x7f2c7809a990 con 0x7f2c901016e0 2024-03-24T10:48:51.696 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:48:51.696 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:47:17.474711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27526":{"gid":27526,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/2017557046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":2017557046},{"type":"v1","addr":"172.21.15.117:6837","nonce":2017557046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:48:51.696 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 31 2024-03-24T10:48:51.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.697+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4218110867 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2c7c0a22b0 msgr2=0x7f2c7c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.697+0000 7f2c95e6b640 1 --2- 172.21.15.23:0/4218110867 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2c7c0a22b0 0x7f2c7c0a4770 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2c80006010 tx=0x7f2c80005e40 comp rx=0 tx=0).stop 2024-03-24T10:48:51.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.697+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4218110867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c901016e0 msgr2=0x7f2c901990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.697+0000 7f2c95e6b640 1 --2- 172.21.15.23:0/4218110867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c901016e0 0x7f2c901990a0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800e9f0 tx=0x7f2c7800edf0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.697+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4218110867 shutdown_connections 2024-03-24T10:48:51.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.697+0000 7f2c95e6b640 1 --2- 172.21.15.23:0/4218110867 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f2c7c0a22b0 0x7f2c7c0a4770 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.697+0000 7f2c95e6b640 1 --2- 172.21.15.23:0/4218110867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c901066a0 0x7f2c901a3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.697+0000 7f2c95e6b640 1 --2- 172.21.15.23:0/4218110867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c901016e0 0x7f2c901990a0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.697+0000 7f2c95e6b640 1 --2- 172.21.15.23:0/4218110867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90100d20 0x7f2c90198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.697+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4218110867 >> 172.21.15.23:0/4218110867 conn(0x7f2c90076ba0 msgr2=0x7f2c900fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.697+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4218110867 shutdown_connections 2024-03-24T10:48:51.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.697+0000 7f2c95e6b640 1 -- 172.21.15.23:0/4218110867 wait complete. 2024-03-24T10:48:51.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:48:51.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.813+0000 7f938adac640 1 -- 172.21.15.23:0/888590535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9384100630 msgr2=0x7f9384100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.813+0000 7f938adac640 1 --2- 172.21.15.23:0/888590535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9384100630 0x7f9384100ab0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f9378009930 tx=0x7f937802bc30 comp rx=0 tx=0).stop 2024-03-24T10:48:51.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.813+0000 7f938adac640 1 -- 172.21.15.23:0/888590535 shutdown_connections 2024-03-24T10:48:51.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.813+0000 7f938adac640 1 --2- 172.21.15.23:0/888590535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9384100ff0 0x7f938410d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.813+0000 7f938adac640 1 --2- 172.21.15.23:0/888590535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9384100630 0x7f9384100ab0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.813+0000 7f938adac640 1 --2- 172.21.15.23:0/888590535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9384106640 0x7f9384106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.814+0000 7f938adac640 1 -- 172.21.15.23:0/888590535 >> 172.21.15.23:0/888590535 conn(0x7f9384077770 msgr2=0x7f9384077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.814+0000 7f938adac640 1 -- 172.21.15.23:0/888590535 shutdown_connections 2024-03-24T10:48:51.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.814+0000 7f938adac640 1 -- 172.21.15.23:0/888590535 wait complete. 2024-03-24T10:48:51.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.814+0000 7f938adac640 1 Processor -- start 2024-03-24T10:48:51.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.814+0000 7f938adac640 1 -- start start 2024-03-24T10:48:51.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.814+0000 7f938adac640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9384100630 0x7f9384197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.814+0000 7f938adac640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9384100ff0 0x7f93841983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.814+0000 7f938adac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9384106640 0x7f938419c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.814+0000 7f938adac640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f938419cd60 con 0x7f9384100630 2024-03-24T10:48:51.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.814+0000 7f938adac640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f938419ced0 con 0x7f9384106640 2024-03-24T10:48:51.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.814+0000 7f938adac640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f938419d090 con 0x7f9384100ff0 2024-03-24T10:48:51.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f9388b48640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9384100630 0x7f9384197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f9388b48640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9384100630 0x7f9384197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46242/0 (socket says 172.21.15.23:46242) 2024-03-24T10:48:51.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f9388b48640 1 -- 172.21.15.23:0/2119397366 learned_addr learned my addr 172.21.15.23:0/2119397366 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f9383fff640 1 --2- 172.21.15.23:0/2119397366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9384100ff0 0x7f93841983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f9389349640 1 --2- 172.21.15.23:0/2119397366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9384106640 0x7f938419c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f9388b48640 1 -- 172.21.15.23:0/2119397366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9384100ff0 msgr2=0x7f93841983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f9388b48640 1 --2- 172.21.15.23:0/2119397366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9384100ff0 0x7f93841983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f9388b48640 1 -- 172.21.15.23:0/2119397366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9384106640 msgr2=0x7f938419c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f9388b48640 1 --2- 172.21.15.23:0/2119397366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9384106640 0x7f938419c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f9388b48640 1 -- 172.21.15.23:0/2119397366 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9378009560 con 0x7f9384100630 2024-03-24T10:48:51.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f9388b48640 1 --2- 172.21.15.23:0/2119397366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9384100630 0x7f9384197ea0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f937c00ebc0 tx=0x7f937c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f9381ffb640 1 -- 172.21.15.23:0/2119397366 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c0099b0 con 0x7f9384100630 2024-03-24T10:48:51.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f9381ffb640 1 -- 172.21.15.23:0/2119397366 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f937c00eec0 con 0x7f9384100630 2024-03-24T10:48:51.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f938adac640 1 -- 172.21.15.23:0/2119397366 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f938419d370 con 0x7f9384100630 2024-03-24T10:48:51.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f9381ffb640 1 -- 172.21.15.23:0/2119397366 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c018b60 con 0x7f9384100630 2024-03-24T10:48:51.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.815+0000 7f938adac640 1 -- 172.21.15.23:0/2119397366 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93841a4bf0 con 0x7f9384100630 2024-03-24T10:48:51.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.817+0000 7f9381ffb640 1 -- 172.21.15.23:0/2119397366 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f937c018cc0 con 0x7f9384100630 2024-03-24T10:48:51.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f938d988740 1 Processor -- start 2024-03-24T10:48:51.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f938d988740 1 -- start start 2024-03-24T10:48:51.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f938d988740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c63f0 0x5585e55c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f938d988740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c6d10 0x5585e55cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f938d988740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55cf8f0 0x5585e55d1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f938d988740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5585e5534840 con 0x5585e55c63f0 2024-03-24T10:48:51.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f938d988740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5585e5531d70 con 0x5585e55c6d10 2024-03-24T10:48:51.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f938d988740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5585e552d550 con 0x5585e55cf8f0 2024-03-24T10:48:51.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f9381ffb640 1 --2- 172.21.15.23:0/2119397366 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f936c0a2230 0x7f936c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f9389349640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c63f0 0x5585e55c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f9389349640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c63f0 0x5585e55c67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46246/0 (socket says 172.21.15.23:46246) 2024-03-24T10:48:51.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f9389349640 1 -- 172.21.15.23:0/752509076 learned_addr learned my addr 172.21.15.23:0/752509076 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f9388b48640 1 --2- 172.21.15.23:0/752509076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c6d10 0x5585e55cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f9389349640 1 --2- 172.21.15.23:0/2119397366 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f936c0a2230 0x7f936c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f9383fff640 1 --2- 172.21.15.23:0/752509076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55cf8f0 0x5585e55d1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f9381ffb640 1 -- 172.21.15.23:0/2119397366 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f937c014070 con 0x7f9384100630 2024-03-24T10:48:51.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f9389349640 1 -- 172.21.15.23:0/752509076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55cf8f0 msgr2=0x5585e55d1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f9389349640 1 --2- 172.21.15.23:0/752509076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55cf8f0 0x5585e55d1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f9389349640 1 -- 172.21.15.23:0/752509076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c6d10 msgr2=0x5585e55cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f9389349640 1 --2- 172.21.15.23:0/752509076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c6d10 0x5585e55cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.818+0000 7f9389349640 1 -- 172.21.15.23:0/752509076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585e54c2de0 con 0x5585e55c63f0 2024-03-24T10:48:51.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.819+0000 7f9389349640 1 --2- 172.21.15.23:0/752509076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c63f0 0x5585e55c67d0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9374011ad0 tx=0x7f9374011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ac98e48024de4d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.819+0000 7f9389349640 1 --2- 172.21.15.23:0/2119397366 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f936c0a2230 0x7f936c0a46f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f937400f850 tx=0x7f937401e000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.819+0000 7f936affd640 1 -- 172.21.15.23:0/752509076 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9374011050 con 0x5585e55c63f0 2024-03-24T10:48:51.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.819+0000 7f936affd640 1 -- 172.21.15.23:0/752509076 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f937401ddc0 con 0x5585e55c63f0 2024-03-24T10:48:51.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.819+0000 7f936affd640 1 -- 172.21.15.23:0/752509076 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9374026ab0 con 0x5585e55c63f0 2024-03-24T10:48:51.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 -- 172.21.15.23:0/752509076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c63f0 msgr2=0x5585e55c67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 --2- 172.21.15.23:0/752509076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c63f0 0x5585e55c67d0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9374011ad0 tx=0x7f9374011f90 comp rx=0 tx=0).stop 2024-03-24T10:48:51.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 -- 172.21.15.23:0/752509076 shutdown_connections 2024-03-24T10:48:51.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 --2- 172.21.15.23:0/752509076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55cf8f0 0x5585e55d1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 --2- 172.21.15.23:0/752509076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c6d10 0x5585e55cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 --2- 172.21.15.23:0/752509076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c63f0 0x5585e55c67d0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 -- 172.21.15.23:0/752509076 >> 172.21.15.23:0/752509076 conn(0x5585e55c0b20 msgr2=0x5585e55c0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 -- 172.21.15.23:0/752509076 shutdown_connections 2024-03-24T10:48:51.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 -- 172.21.15.23:0/752509076 wait complete. 2024-03-24T10:48:51.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 -- start start 2024-03-24T10:48:51.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 Processor -- start 2024-03-24T10:48:51.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c63f0 0x5585e55cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c6d10 0x5585e55cce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55cf8f0 0x5585e55cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5585e5200430 con 0x5585e55c6d10 2024-03-24T10:48:51.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5585e51bce00 con 0x5585e55c63f0 2024-03-24T10:48:51.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5585e54b2590 con 0x5585e55cf8f0 2024-03-24T10:48:51.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.820+0000 7f938d988740 10 client.0 _ll_register_callbacks cb 0x5585e5521fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:51.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9383fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c6d10 0x5585e55cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9383fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c6d10 0x5585e55cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46256/0 (socket says 172.21.15.23:46256) 2024-03-24T10:48:51.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9383fff640 1 -- 172.21.15.23:0/2678324935 learned_addr learned my addr 172.21.15.23:0/2678324935 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9388b48640 1 --2- 172.21.15.23:0/2678324935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c63f0 0x5585e55cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9389349640 1 --2- 172.21.15.23:0/2678324935 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55cf8f0 0x5585e55cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9383fff640 1 -- 172.21.15.23:0/2678324935 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55cf8f0 msgr2=0x5585e55cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9383fff640 1 --2- 172.21.15.23:0/2678324935 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55cf8f0 0x5585e55cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9383fff640 1 -- 172.21.15.23:0/2678324935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c63f0 msgr2=0x5585e55cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9383fff640 1 --2- 172.21.15.23:0/2678324935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c63f0 0x5585e55cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9383fff640 1 -- 172.21.15.23:0/2678324935 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9374019040 con 0x5585e55c6d10 2024-03-24T10:48:51.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9388b48640 1 --2- 172.21.15.23:0/2678324935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c63f0 0x5585e55cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:51.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9383fff640 1 --2- 172.21.15.23:0/2678324935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c6d10 0x5585e55cce90 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f9378006010 tx=0x7f937802ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc221f3fd81707f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9363fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:51.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9363fff640 1 -- 172.21.15.23:0/2678324935 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937802cc00 con 0x5585e55c6d10 2024-03-24T10:48:51.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9363fff640 1 -- 172.21.15.23:0/2678324935 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f9378046070 con 0x5585e55c6d10 2024-03-24T10:48:51.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f938d988740 1 -- 172.21.15.23:0/2678324935 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5585e522c7e0 con 0x5585e55c6d10 2024-03-24T10:48:51.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f9363fff640 1 -- 172.21.15.23:0/2678324935 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9378004250 con 0x5585e55c6d10 2024-03-24T10:48:51.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f938d988740 10 client.18006 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:51.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.821+0000 7f938d988740 1 -- 172.21.15.23:0/2678324935 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5585e552bbd0 con 0x5585e55c6d10 2024-03-24T10:48:51.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.822+0000 7f9363fff640 1 -- 172.21.15.23:0/2678324935 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5585e522c7e0 con 0x5585e55c6d10 2024-03-24T10:48:51.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.822+0000 7f9363fff640 1 -- 172.21.15.23:0/2678324935 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f937803c040 con 0x5585e55c6d10 2024-03-24T10:48:51.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.822+0000 7f938d988740 10 client.18006 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:51.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.822+0000 7f938d988740 10 client.18006 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:48:51.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.822+0000 7f938d988740 20 client.18006 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:51.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.822+0000 7f938d988740 1 --2- 172.21.15.23:0/2678324935 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5585e55df970 0x5585e55e1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.822+0000 7f938d988740 4 client.18006 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:48:51.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.822+0000 7f938d988740 1 -- 172.21.15.23:0/2678324935 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585e5269720 con 0x5585e55df970 2024-03-24T10:48:51.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.822+0000 7f9388b48640 1 --2- 172.21.15.23:0/2678324935 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5585e55df970 0x5585e55e1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.822+0000 7f9363fff640 10 client.18006 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:48:51.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.822+0000 7f9388b48640 1 --2- 172.21.15.23:0/2678324935 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5585e55df970 0x5585e55e1db0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b3897dd7413f4c1 server_cookie=33d4f637758e4b02 in_seq=0 out_seq=0 2024-03-24T10:48:51.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f9363fff640 1 -- 172.21.15.23:0/2678324935 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5585e5269720 con 0x5585e55df970 2024-03-24T10:48:51.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f9363fff640 10 client.18006 handle_command_reply: tid=0 2024-03-24T10:48:51.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f938d988740 1 client.18006 shutdown 2024-03-24T10:48:51.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f938d988740 1 -- 172.21.15.23:0/2678324935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c6d10 msgr2=0x5585e55cce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f938d988740 1 --2- 172.21.15.23:0/2678324935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c6d10 0x5585e55cce90 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f9378006010 tx=0x7f937802ca10 comp rx=0 tx=0).stop 2024-03-24T10:48:51.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f938d988740 1 -- 172.21.15.23:0/2678324935 shutdown_connections 2024-03-24T10:48:51.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f938d988740 1 --2- 172.21.15.23:0/2678324935 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55cf8f0 0x5585e55cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f938d988740 1 --2- 172.21.15.23:0/2678324935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c6d10 0x5585e55cce90 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f938d988740 1 --2- 172.21.15.23:0/2678324935 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5585e55df970 0x5585e55e1db0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f938d988740 1 --2- 172.21.15.23:0/2678324935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c63f0 0x5585e55cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f9363fff640 0 client.18006 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:48:51.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f938d988740 1 -- 172.21.15.23:0/2678324935 >> 172.21.15.23:0/2678324935 conn(0x5585e55c0b20 msgr2=0x5585e55c2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f938d988740 1 -- 172.21.15.23:0/2678324935 shutdown_connections 2024-03-24T10:48:51.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f938d988740 1 -- 172.21.15.23:0/2678324935 wait complete. 2024-03-24T10:48:51.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.823+0000 7f938d988740 20 client.18006 trim_cache size 0 max 16384 2024-03-24T10:48:51.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.836+0000 7f938d988740 1 Processor -- start 2024-03-24T10:48:51.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f938d988740 1 -- start start 2024-03-24T10:48:51.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f938d988740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55c63f0 0x5585e55db120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f938d988740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c6d10 0x5585e55ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f938d988740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55cf8f0 0x5585e55e76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f938d988740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5585e55de1f0 con 0x5585e55c6d10 2024-03-24T10:48:51.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f938d988740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5585e55cdde0 con 0x5585e55cf8f0 2024-03-24T10:48:51.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f938d988740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5585e55cf3b0 con 0x5585e55c63f0 2024-03-24T10:48:51.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f9383fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c6d10 0x5585e55ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f9383fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c6d10 0x5585e55ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46272/0 (socket says 172.21.15.23:46272) 2024-03-24T10:48:51.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f9383fff640 1 -- 172.21.15.23:0/1975759029 learned_addr learned my addr 172.21.15.23:0/1975759029 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f9389349640 1 --2- 172.21.15.23:0/1975759029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55cf8f0 0x5585e55e76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f9388b48640 1 --2- 172.21.15.23:0/1975759029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55c63f0 0x5585e55db120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f9383fff640 1 -- 172.21.15.23:0/1975759029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55c63f0 msgr2=0x5585e55db120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f9383fff640 1 --2- 172.21.15.23:0/1975759029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55c63f0 0x5585e55db120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f9383fff640 1 -- 172.21.15.23:0/1975759029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55cf8f0 msgr2=0x5585e55e76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f9383fff640 1 --2- 172.21.15.23:0/1975759029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55cf8f0 0x5585e55e76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f9383fff640 1 -- 172.21.15.23:0/1975759029 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585e54c7c80 con 0x5585e55c6d10 2024-03-24T10:48:51.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f9388b48640 1 --2- 172.21.15.23:0/1975759029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55c63f0 0x5585e55db120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:48:51.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f9383fff640 1 --2- 172.21.15.23:0/1975759029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c6d10 0x5585e55ce780 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f937802c310 tx=0x7f9378004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11196b28f365e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f936a7fc640 1 -- 172.21.15.23:0/1975759029 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9378005d50 con 0x5585e55c6d10 2024-03-24T10:48:51.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f936a7fc640 1 -- 172.21.15.23:0/1975759029 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9378044070 con 0x5585e55c6d10 2024-03-24T10:48:51.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.837+0000 7f936a7fc640 1 -- 172.21.15.23:0/1975759029 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93780044b0 con 0x5585e55c6d10 2024-03-24T10:48:51.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.838+0000 7f938d988740 1 -- 172.21.15.23:0/1975759029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c6d10 msgr2=0x5585e55ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.838+0000 7f938d988740 1 --2- 172.21.15.23:0/1975759029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c6d10 0x5585e55ce780 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f937802c310 tx=0x7f9378004a60 comp rx=0 tx=0).stop 2024-03-24T10:48:51.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.838+0000 7f938d988740 1 -- 172.21.15.23:0/1975759029 shutdown_connections 2024-03-24T10:48:51.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.838+0000 7f938d988740 1 --2- 172.21.15.23:0/1975759029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55cf8f0 0x5585e55e76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.838+0000 7f938d988740 1 --2- 172.21.15.23:0/1975759029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55c6d10 0x5585e55ce780 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.838+0000 7f938d988740 1 --2- 172.21.15.23:0/1975759029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55c63f0 0x5585e55db120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.838+0000 7f938d988740 1 -- 172.21.15.23:0/1975759029 >> 172.21.15.23:0/1975759029 conn(0x5585e55c0b20 msgr2=0x5585e55ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.838+0000 7f938d988740 1 -- 172.21.15.23:0/1975759029 shutdown_connections 2024-03-24T10:48:51.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.838+0000 7f938d988740 1 -- 172.21.15.23:0/1975759029 wait complete. 2024-03-24T10:48:51.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.838+0000 7f938d988740 1 -- start start 2024-03-24T10:48:51.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.838+0000 7f938d988740 1 Processor -- start 2024-03-24T10:48:51.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f938d988740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55c63f0 0x5585e55c9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f938d988740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c6d10 0x5585e55c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f938d988740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55cf8f0 0x5585e5644cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f938d988740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5585e55ca5a0 con 0x5585e55cf8f0 2024-03-24T10:48:51.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f938d988740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5585e5645230 con 0x5585e55c6d10 2024-03-24T10:48:51.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f938d988740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5585e5645370 con 0x5585e55c63f0 2024-03-24T10:48:51.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f938d988740 10 client.0 _ll_register_callbacks cb 0x7f9378009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:51.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f9389349640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55cf8f0 0x5585e5644cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f9389349640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55cf8f0 0x5585e5644cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46278/0 (socket says 172.21.15.23:46278) 2024-03-24T10:48:51.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f9389349640 1 -- 172.21.15.23:0/378129693 learned_addr learned my addr 172.21.15.23:0/378129693 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f9383fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c6d10 0x5585e55c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f9388b48640 1 --2- 172.21.15.23:0/378129693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55c63f0 0x5585e55c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f9389349640 1 -- 172.21.15.23:0/378129693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55c63f0 msgr2=0x5585e55c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f9389349640 1 --2- 172.21.15.23:0/378129693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55c63f0 0x5585e55c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f9389349640 1 -- 172.21.15.23:0/378129693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c6d10 msgr2=0x5585e55c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f9389349640 1 --2- 172.21.15.23:0/378129693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c6d10 0x5585e55c9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f9389349640 1 -- 172.21.15.23:0/378129693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585e552bbd0 con 0x5585e55cf8f0 2024-03-24T10:48:51.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f9389349640 1 --2- 172.21.15.23:0/378129693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55cf8f0 0x5585e5644cf0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f93740096f0 tx=0x7f9374005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4adaf4a7b27cb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f9363fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:51.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f9363fff640 1 -- 172.21.15.23:0/378129693 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937402a040 con 0x5585e55cf8f0 2024-03-24T10:48:51.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f9363fff640 1 -- 172.21.15.23:0/378129693 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f9374026450 con 0x5585e55cf8f0 2024-03-24T10:48:51.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.839+0000 7f938d988740 1 -- 172.21.15.23:0/378129693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f937401ddc0 con 0x5585e55cf8f0 2024-03-24T10:48:51.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.840+0000 7f9363fff640 1 -- 172.21.15.23:0/378129693 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9374009b50 con 0x5585e55cf8f0 2024-03-24T10:48:51.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.840+0000 7f938d988740 10 client.18012 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:51.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.840+0000 7f938d988740 1 -- 172.21.15.23:0/378129693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5585e54ed920 con 0x5585e55cf8f0 2024-03-24T10:48:51.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.840+0000 7f9363fff640 1 -- 172.21.15.23:0/378129693 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f937401ddc0 con 0x5585e55cf8f0 2024-03-24T10:48:51.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.840+0000 7f9363fff640 1 -- 172.21.15.23:0/378129693 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f9374026d20 con 0x5585e55cf8f0 2024-03-24T10:48:51.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.840+0000 7f938d988740 10 client.18012 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:51.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.840+0000 7f938d988740 10 client.18012 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:48:51.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.840+0000 7f938d988740 20 client.18012 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:51.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.840+0000 7f938d988740 1 --2- 172.21.15.23:0/378129693 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5585e5646ac0 0x5585e5648ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.840+0000 7f938d988740 4 client.18012 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:48:51.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.840+0000 7f938d988740 1 -- 172.21.15.23:0/378129693 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5585e52b8890 con 0x5585e5646ac0 2024-03-24T10:48:51.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.840+0000 7f9388b48640 1 --2- 172.21.15.23:0/378129693 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5585e5646ac0 0x5585e5648ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.840+0000 7f9388b48640 1 --2- 172.21.15.23:0/378129693 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5585e5646ac0 0x5585e5648ee0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dc45861d3fbd4f76 server_cookie=de8044fe7465a1c9 in_seq=0 out_seq=0 2024-03-24T10:48:51.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.840+0000 7f9363fff640 10 client.18012 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:48:51.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f9363fff640 1 -- 172.21.15.23:0/378129693 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5585e52b8890 con 0x5585e5646ac0 2024-03-24T10:48:51.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f9363fff640 10 client.18012 handle_command_reply: tid=0 2024-03-24T10:48:51.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938d988740 1 client.18012 shutdown 2024-03-24T10:48:51.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938d988740 1 -- 172.21.15.23:0/378129693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55cf8f0 msgr2=0x5585e5644cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938d988740 1 --2- 172.21.15.23:0/378129693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55cf8f0 0x5585e5644cf0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f93740096f0 tx=0x7f9374005f10 comp rx=0 tx=0).stop 2024-03-24T10:48:51.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938d988740 1 -- 172.21.15.23:0/378129693 shutdown_connections 2024-03-24T10:48:51.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938d988740 1 --2- 172.21.15.23:0/378129693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585e55cf8f0 0x5585e5644cf0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938d988740 1 --2- 172.21.15.23:0/378129693 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5585e5646ac0 0x5585e5648ee0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938d988740 1 --2- 172.21.15.23:0/378129693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585e55c6d10 0x5585e55c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938d988740 1 --2- 172.21.15.23:0/378129693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585e55c63f0 0x5585e55c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938d988740 1 -- 172.21.15.23:0/378129693 >> 172.21.15.23:0/378129693 conn(0x5585e55c0b20 msgr2=0x5585e55bfcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f9363fff640 0 client.18012 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:48:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938d988740 1 -- 172.21.15.23:0/378129693 shutdown_connections 2024-03-24T10:48:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938d988740 1 -- 172.21.15.23:0/378129693 wait complete. 2024-03-24T10:48:51.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938d988740 20 client.18012 trim_cache size 0 max 16384 2024-03-24T10:48:51.853 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:48:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938adac640 1 -- 172.21.15.23:0/2119397366 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f936c0a2230 msgr2=0x7f936c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938adac640 1 --2- 172.21.15.23:0/2119397366 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f936c0a2230 0x7f936c0a46f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f937400f850 tx=0x7f937401e000 comp rx=0 tx=0).stop 2024-03-24T10:48:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938adac640 1 -- 172.21.15.23:0/2119397366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9384100630 msgr2=0x7f9384197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.841+0000 7f938adac640 1 --2- 172.21.15.23:0/2119397366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9384100630 0x7f9384197ea0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f937c00ebc0 tx=0x7f937c00c6a0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.842+0000 7f938adac640 1 -- 172.21.15.23:0/2119397366 shutdown_connections 2024-03-24T10:48:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.842+0000 7f938adac640 1 --2- 172.21.15.23:0/2119397366 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f936c0a2230 0x7f936c0a46f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.842+0000 7f938adac640 1 --2- 172.21.15.23:0/2119397366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9384106640 0x7f938419c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.842+0000 7f938adac640 1 --2- 172.21.15.23:0/2119397366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9384100ff0 0x7f93841983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.842+0000 7f938adac640 1 --2- 172.21.15.23:0/2119397366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9384100630 0x7f9384197ea0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.842+0000 7f938adac640 1 -- 172.21.15.23:0/2119397366 >> 172.21.15.23:0/2119397366 conn(0x7f9384077770 msgr2=0x7f938406a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.842+0000 7f938adac640 1 -- 172.21.15.23:0/2119397366 shutdown_connections 2024-03-24T10:48:51.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.842+0000 7f938adac640 1 -- 172.21.15.23:0/2119397366 wait complete. 2024-03-24T10:48:51.856 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:48:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.960+0000 7f0d99023640 1 -- 172.21.15.23:0/3527788567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d94107690 msgr2=0x7f0d94107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.960+0000 7f0d99023640 1 --2- 172.21.15.23:0/3527788567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d94107690 0x7f0d94107a70 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c009930 tx=0x7f0d8c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:48:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.960+0000 7f0d99023640 1 -- 172.21.15.23:0/3527788567 shutdown_connections 2024-03-24T10:48:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.960+0000 7f0d99023640 1 --2- 172.21.15.23:0/3527788567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d940692b0 0x7f0d9410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.960+0000 7f0d99023640 1 --2- 172.21.15.23:0/3527788567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d940688f0 0x7f0d94068d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.960+0000 7f0d99023640 1 --2- 172.21.15.23:0/3527788567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d94107690 0x7f0d94107a70 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.960+0000 7f0d99023640 1 -- 172.21.15.23:0/3527788567 >> 172.21.15.23:0/3527788567 conn(0x7f0d94077710 msgr2=0x7f0d94077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.960+0000 7f0d99023640 1 -- 172.21.15.23:0/3527788567 shutdown_connections 2024-03-24T10:48:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.960+0000 7f0d99023640 1 -- 172.21.15.23:0/3527788567 wait complete. 2024-03-24T10:48:51.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.961+0000 7f0d99023640 1 Processor -- start 2024-03-24T10:48:51.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.961+0000 7f0d99023640 1 -- start start 2024-03-24T10:48:51.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.961+0000 7f0d99023640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d940688f0 0x7f0d941005d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.961+0000 7f0d99023640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d940692b0 0x7f0d94100b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.961+0000 7f0d99023640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d94107690 0x7f0d94101050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.961+0000 7f0d99023640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0d941a60f0 con 0x7f0d940692b0 2024-03-24T10:48:51.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d99023640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0d941a6260 con 0x7f0d94107690 2024-03-24T10:48:51.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d99023640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0d941a6420 con 0x7f0d940688f0 2024-03-24T10:48:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d9259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d940692b0 0x7f0d94100b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d9259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d940692b0 0x7f0d94100b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46286/0 (socket says 172.21.15.23:46286) 2024-03-24T10:48:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d9259c640 1 -- 172.21.15.23:0/580588599 learned_addr learned my addr 172.21.15.23:0/580588599 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d92d9d640 1 --2- 172.21.15.23:0/580588599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d940688f0 0x7f0d941005d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d9359e640 1 --2- 172.21.15.23:0/580588599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d94107690 0x7f0d94101050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d9259c640 1 -- 172.21.15.23:0/580588599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d940688f0 msgr2=0x7f0d941005d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d9259c640 1 --2- 172.21.15.23:0/580588599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d940688f0 0x7f0d941005d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d9259c640 1 -- 172.21.15.23:0/580588599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d94107690 msgr2=0x7f0d94101050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d9259c640 1 --2- 172.21.15.23:0/580588599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d94107690 0x7f0d94101050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d9259c640 1 -- 172.21.15.23:0/580588599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d88009490 con 0x7f0d940692b0 2024-03-24T10:48:51.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d9259c640 1 --2- 172.21.15.23:0/580588599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d940692b0 0x7f0d94100b10 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800ea60 tx=0x7f0d8800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d7bfff640 1 -- 172.21.15.23:0/580588599 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8800cd20 con 0x7f0d940692b0 2024-03-24T10:48:51.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d99023640 1 -- 172.21.15.23:0/580588599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d84009840 con 0x7f0d940692b0 2024-03-24T10:48:51.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d7bfff640 1 -- 172.21.15.23:0/580588599 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d8800ce80 con 0x7f0d940692b0 2024-03-24T10:48:51.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d7bfff640 1 -- 172.21.15.23:0/580588599 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d88010690 con 0x7f0d940692b0 2024-03-24T10:48:51.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.962+0000 7f0d99023640 1 -- 172.21.15.23:0/580588599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d8c009560 con 0x7f0d940692b0 2024-03-24T10:48:51.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.965+0000 7f0d9bc03740 1 Processor -- start 2024-03-24T10:48:51.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.965+0000 7f0d9bc03740 1 -- start start 2024-03-24T10:48:51.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.965+0000 7f0d7bfff640 1 -- 172.21.15.23:0/580588599 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f0d88003ee0 con 0x7f0d940692b0 2024-03-24T10:48:51.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.965+0000 7f0d9bc03740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2073f0 0x55f3ca2077d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.965+0000 7f0d9bc03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 0x55f3ca2103b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.965+0000 7f0d9bc03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2108f0 0x55f3ca212ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.965+0000 7f0d9bc03740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f3ca175840 con 0x55f3ca2073f0 2024-03-24T10:48:51.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.965+0000 7f0d9bc03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3ca172d70 con 0x55f3ca207d10 2024-03-24T10:48:51.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.965+0000 7f0d9bc03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f3ca16e550 con 0x55f3ca2108f0 2024-03-24T10:48:51.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d9359e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2073f0 0x55f3ca2077d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d9359e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2073f0 0x55f3ca2077d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46302/0 (socket says 172.21.15.23:46302) 2024-03-24T10:48:51.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d9359e640 1 -- 172.21.15.23:0/3728871903 learned_addr learned my addr 172.21.15.23:0/3728871903 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d92d9d640 1 --2- 172.21.15.23:0/3728871903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 0x55f3ca2103b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d9259c640 1 --2- 172.21.15.23:0/3728871903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2108f0 0x55f3ca212ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d9359e640 1 -- 172.21.15.23:0/3728871903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2108f0 msgr2=0x55f3ca212ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d9359e640 1 --2- 172.21.15.23:0/3728871903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2108f0 0x55f3ca212ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d9359e640 1 -- 172.21.15.23:0/3728871903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 msgr2=0x55f3ca2103b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d9359e640 1 --2- 172.21.15.23:0/3728871903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 0x55f3ca2103b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d9359e640 1 -- 172.21.15.23:0/3728871903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3ca103de0 con 0x55f3ca2073f0 2024-03-24T10:48:51.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d9259c640 1 --2- 172.21.15.23:0/3728871903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2108f0 0x55f3ca212ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:48:51.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d7bfff640 1 --2- 172.21.15.23:0/580588599 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0d7c0a2260 0x7f0d7c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d9359e640 1 --2- 172.21.15.23:0/3728871903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2073f0 0x55f3ca2077d0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f0d84011570 tx=0x7f0d84011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd9f8767729d758 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d92d9d640 1 --2- 172.21.15.23:0/580588599 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0d7c0a2260 0x7f0d7c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d7bfff640 1 -- 172.21.15.23:0/580588599 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f0d88014070 con 0x7f0d940692b0 2024-03-24T10:48:51.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d797fa640 1 -- 172.21.15.23:0/3728871903 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d84004d20 con 0x55f3ca2073f0 2024-03-24T10:48:51.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d797fa640 1 -- 172.21.15.23:0/3728871903 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d84004e80 con 0x55f3ca2073f0 2024-03-24T10:48:51.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.966+0000 7f0d797fa640 1 -- 172.21.15.23:0/3728871903 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8400f8a0 con 0x55f3ca2073f0 2024-03-24T10:48:51.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.967+0000 7f0d9bc03740 1 -- 172.21.15.23:0/3728871903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2073f0 msgr2=0x55f3ca2077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.967+0000 7f0d92d9d640 1 --2- 172.21.15.23:0/580588599 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0d7c0a2260 0x7f0d7c0a4720 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c005ab0 tx=0x7f0d8c005fd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.967+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/3728871903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2073f0 0x55f3ca2077d0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f0d84011570 tx=0x7f0d84011a30 comp rx=0 tx=0).stop 2024-03-24T10:48:51.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.967+0000 7f0d9bc03740 1 -- 172.21.15.23:0/3728871903 shutdown_connections 2024-03-24T10:48:51.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.967+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/3728871903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2108f0 0x55f3ca212ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.967+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/3728871903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 0x55f3ca2103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.967+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/3728871903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2073f0 0x55f3ca2077d0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.967+0000 7f0d9bc03740 1 -- 172.21.15.23:0/3728871903 >> 172.21.15.23:0/3728871903 conn(0x55f3ca201b20 msgr2=0x55f3ca201f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.967+0000 7f0d9bc03740 1 -- 172.21.15.23:0/3728871903 shutdown_connections 2024-03-24T10:48:51.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.967+0000 7f0d9bc03740 1 -- 172.21.15.23:0/3728871903 wait complete. 2024-03-24T10:48:51.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.967+0000 7f0d9bc03740 1 -- start start 2024-03-24T10:48:51.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.967+0000 7f0d9bc03740 1 Processor -- start 2024-03-24T10:48:51.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d9bc03740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2073f0 0x55f3ca20d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d9bc03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 0x55f3ca20df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d9bc03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2108f0 0x55f3ca20e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d9bc03740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f3c9e41430 con 0x55f3ca2073f0 2024-03-24T10:48:51.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d9bc03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3c9dfde00 con 0x55f3ca207d10 2024-03-24T10:48:51.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d9bc03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f3ca0f3590 con 0x55f3ca2108f0 2024-03-24T10:48:51.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d9bc03740 10 client.0 _ll_register_callbacks cb 0x55f3ca162fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:51.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d92d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2073f0 0x55f3ca20d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d92d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2073f0 0x55f3ca20d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46308/0 (socket says 172.21.15.23:46308) 2024-03-24T10:48:51.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d92d9d640 1 -- 172.21.15.23:0/3437340780 learned_addr learned my addr 172.21.15.23:0/3437340780 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d9259c640 1 --2- 172.21.15.23:0/3437340780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 0x55f3ca20df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d9359e640 1 --2- 172.21.15.23:0/3437340780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2108f0 0x55f3ca20e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d92d9d640 1 -- 172.21.15.23:0/3437340780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2108f0 msgr2=0x55f3ca20e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d92d9d640 1 --2- 172.21.15.23:0/3437340780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2108f0 0x55f3ca20e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d92d9d640 1 -- 172.21.15.23:0/3437340780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 msgr2=0x55f3ca20df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d92d9d640 1 --2- 172.21.15.23:0/3437340780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 0x55f3ca20df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d92d9d640 1 -- 172.21.15.23:0/3437340780 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d84009da0 con 0x55f3ca2073f0 2024-03-24T10:48:51.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d92d9d640 1 --2- 172.21.15.23:0/3437340780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2073f0 0x55f3ca20d9e0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c035000 tx=0x7f0d8c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b1766150e020793 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d76ff5640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:51.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d76ff5640 1 -- 172.21.15.23:0/3437340780 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c005000 con 0x55f3ca2073f0 2024-03-24T10:48:51.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d76ff5640 1 -- 172.21.15.23:0/3437340780 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f0d8c005160 con 0x55f3ca2073f0 2024-03-24T10:48:51.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.968+0000 7f0d9bc03740 1 -- 172.21.15.23:0/3437340780 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f3c9eaa720 con 0x55f3ca2073f0 2024-03-24T10:48:51.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.969+0000 7f0d76ff5640 1 -- 172.21.15.23:0/3437340780 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0d8c005320 con 0x55f3ca2073f0 2024-03-24T10:48:51.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.969+0000 7f0d9bc03740 10 client.18024 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:51.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.969+0000 7f0d9bc03740 1 -- 172.21.15.23:0/3437340780 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f3ca16cbd0 con 0x55f3ca2073f0 2024-03-24T10:48:51.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.969+0000 7f0d76ff5640 1 -- 172.21.15.23:0/3437340780 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f3c9eaa720 con 0x55f3ca2073f0 2024-03-24T10:48:51.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.969+0000 7f0d76ff5640 1 -- 172.21.15.23:0/3437340780 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f0d8c03c040 con 0x55f3ca2073f0 2024-03-24T10:48:51.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.969+0000 7f0d9bc03740 10 client.18024 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:51.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.969+0000 7f0d9bc03740 10 client.18024 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:48:51.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.969+0000 7f0d9bc03740 20 client.18024 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:51.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.969+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/3437340780 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f3ca220ad0 0x55f3ca222ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.969+0000 7f0d9bc03740 4 client.18024 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:48:51.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.969+0000 7f0d9bc03740 1 -- 172.21.15.23:0/3437340780 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3c9ef9890 con 0x55f3ca220ad0 2024-03-24T10:48:51.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.969+0000 7f0d9259c640 1 --2- 172.21.15.23:0/3437340780 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f3ca220ad0 0x55f3ca222ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.970+0000 7f0d9259c640 1 --2- 172.21.15.23:0/3437340780 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f3ca220ad0 0x55f3ca222ef0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e35db7b5ae1cb652 server_cookie=f35cd2f558cc673b in_seq=0 out_seq=0 2024-03-24T10:48:51.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.970+0000 7f0d76ff5640 10 client.18024 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:48:51.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d76ff5640 1 -- 172.21.15.23:0/3437340780 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f3c9ef9890 con 0x55f3ca220ad0 2024-03-24T10:48:51.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d76ff5640 10 client.18024 handle_command_reply: tid=0 2024-03-24T10:48:51.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d9bc03740 1 client.18024 shutdown 2024-03-24T10:48:51.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d9bc03740 1 -- 172.21.15.23:0/3437340780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2073f0 msgr2=0x55f3ca20d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/3437340780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2073f0 0x55f3ca20d9e0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c035000 tx=0x7f0d8c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:48:51.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d9bc03740 1 -- 172.21.15.23:0/3437340780 shutdown_connections 2024-03-24T10:48:51.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/3437340780 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f3ca220ad0 0x55f3ca222ef0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/3437340780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2108f0 0x55f3ca20e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/3437340780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 0x55f3ca20df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d76ff5640 0 client.18024 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:48:51.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/3437340780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2073f0 0x55f3ca20d9e0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d9bc03740 1 -- 172.21.15.23:0/3437340780 >> 172.21.15.23:0/3437340780 conn(0x55f3ca201b20 msgr2=0x55f3ca203870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d9bc03740 1 -- 172.21.15.23:0/3437340780 shutdown_connections 2024-03-24T10:48:51.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d9bc03740 1 -- 172.21.15.23:0/3437340780 wait complete. 2024-03-24T10:48:51.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.971+0000 7f0d9bc03740 20 client.18024 trim_cache size 0 max 16384 2024-03-24T10:48:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.984+0000 7f0d9bc03740 1 Processor -- start 2024-03-24T10:48:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.984+0000 7f0d9bc03740 1 -- start start 2024-03-24T10:48:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.984+0000 7f0d9bc03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca2073f0 0x55f3ca21c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.984+0000 7f0d9bc03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca207d10 0x55f3ca20e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.984+0000 7f0d9bc03740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2108f0 0x55f3ca203660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.984+0000 7f0d9bc03740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f3ca202b20 con 0x55f3ca2108f0 2024-03-24T10:48:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.984+0000 7f0d9bc03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3ca213050 con 0x55f3ca2073f0 2024-03-24T10:48:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.984+0000 7f0d9bc03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f3ca21c6c0 con 0x55f3ca207d10 2024-03-24T10:48:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.984+0000 7f0d9359e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2108f0 0x55f3ca203660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.984+0000 7f0d9359e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2108f0 0x55f3ca203660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46316/0 (socket says 172.21.15.23:46316) 2024-03-24T10:48:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.984+0000 7f0d9359e640 1 -- 172.21.15.23:0/642550432 learned_addr learned my addr 172.21.15.23:0/642550432 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.984+0000 7f0d92d9d640 1 --2- 172.21.15.23:0/642550432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca2073f0 0x55f3ca21c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.984+0000 7f0d9259c640 1 --2- 172.21.15.23:0/642550432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca207d10 0x55f3ca20e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.985+0000 7f0d9359e640 1 -- 172.21.15.23:0/642550432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca207d10 msgr2=0x55f3ca20e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.985+0000 7f0d9359e640 1 --2- 172.21.15.23:0/642550432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca207d10 0x55f3ca20e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.985+0000 7f0d9359e640 1 -- 172.21.15.23:0/642550432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca2073f0 msgr2=0x55f3ca21c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.985+0000 7f0d9359e640 1 --2- 172.21.15.23:0/642550432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca2073f0 0x55f3ca21c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.985+0000 7f0d9359e640 1 -- 172.21.15.23:0/642550432 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3ca108c80 con 0x55f3ca2108f0 2024-03-24T10:48:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.985+0000 7f0d9359e640 1 --2- 172.21.15.23:0/642550432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2108f0 0x55f3ca203660 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f0d84011540 tx=0x7f0d84003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413f4aed3fa43908 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.985+0000 7f0d78ff9640 1 -- 172.21.15.23:0/642550432 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d84003da0 con 0x55f3ca2108f0 2024-03-24T10:48:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.985+0000 7f0d78ff9640 1 -- 172.21.15.23:0/642550432 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d84028070 con 0x55f3ca2108f0 2024-03-24T10:48:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.985+0000 7f0d78ff9640 1 -- 172.21.15.23:0/642550432 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d84005640 con 0x55f3ca2108f0 2024-03-24T10:48:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 -- 172.21.15.23:0/642550432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2108f0 msgr2=0x55f3ca203660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/642550432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2108f0 0x55f3ca203660 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f0d84011540 tx=0x7f0d84003970 comp rx=0 tx=0).stop 2024-03-24T10:48:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 -- 172.21.15.23:0/642550432 shutdown_connections 2024-03-24T10:48:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/642550432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2108f0 0x55f3ca203660 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/642550432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca207d10 0x55f3ca20e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/642550432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca2073f0 0x55f3ca21c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 -- 172.21.15.23:0/642550432 >> 172.21.15.23:0/642550432 conn(0x55f3ca201b20 msgr2=0x55f3ca202f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 -- 172.21.15.23:0/642550432 shutdown_connections 2024-03-24T10:48:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 -- 172.21.15.23:0/642550432 wait complete. 2024-03-24T10:48:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 -- start start 2024-03-24T10:48:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 Processor -- start 2024-03-24T10:48:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2073f0 0x55f3ca2187b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 0x55f3ca215780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2108f0 0x55f3ca215cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55f3ca218cf0 con 0x55f3ca2108f0 2024-03-24T10:48:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3ca218e30 con 0x55f3ca207d10 2024-03-24T10:48:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f3ca218ff0 con 0x55f3ca2073f0 2024-03-24T10:48:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9bc03740 10 client.0 _ll_register_callbacks cb 0x7f0d8c044050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9359e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2108f0 0x55f3ca215cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9359e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2108f0 0x55f3ca215cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46322/0 (socket says 172.21.15.23:46322) 2024-03-24T10:48:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9359e640 1 -- 172.21.15.23:0/1426543014 learned_addr learned my addr 172.21.15.23:0/1426543014 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d9259c640 1 --2- 172.21.15.23:0/1426543014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 0x55f3ca215780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.986+0000 7f0d92d9d640 1 --2- 172.21.15.23:0/1426543014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2073f0 0x55f3ca2187b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d9359e640 1 -- 172.21.15.23:0/1426543014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2073f0 msgr2=0x55f3ca2187b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d9359e640 1 --2- 172.21.15.23:0/1426543014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2073f0 0x55f3ca2187b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d9359e640 1 -- 172.21.15.23:0/1426543014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 msgr2=0x55f3ca215780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d9359e640 1 --2- 172.21.15.23:0/1426543014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 0x55f3ca215780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d9359e640 1 -- 172.21.15.23:0/1426543014 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3ca103de0 con 0x55f3ca2108f0 2024-03-24T10:48:51.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d9259c640 1 --2- 172.21.15.23:0/1426543014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 0x55f3ca215780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:51.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d9359e640 1 --2- 172.21.15.23:0/1426543014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2108f0 0x55f3ca215cc0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f0d84011f30 tx=0x7f0d840095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532567972cd39288 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d76ff5640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:51.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d76ff5640 1 -- 172.21.15.23:0/1426543014 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d840058d0 con 0x55f3ca2108f0 2024-03-24T10:48:51.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d76ff5640 1 -- 172.21.15.23:0/1426543014 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f0d8402a070 con 0x55f3ca2108f0 2024-03-24T10:48:51.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d9bc03740 1 -- 172.21.15.23:0/1426543014 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f3c9e6d7e0 con 0x55f3ca2108f0 2024-03-24T10:48:51.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d76ff5640 1 -- 172.21.15.23:0/1426543014 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0d8400f640 con 0x55f3ca2108f0 2024-03-24T10:48:51.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d9bc03740 10 client.18030 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:51.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d9bc03740 1 -- 172.21.15.23:0/1426543014 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f3ca12e920 con 0x55f3ca2108f0 2024-03-24T10:48:51.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d76ff5640 1 -- 172.21.15.23:0/1426543014 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f3c9e6d7e0 con 0x55f3ca2108f0 2024-03-24T10:48:51.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.987+0000 7f0d76ff5640 1 -- 172.21.15.23:0/1426543014 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f0d84021040 con 0x55f3ca2108f0 2024-03-24T10:48:51.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 10 client.18030 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:51.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 10 client.18030 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:48:51.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 20 client.18030 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:51.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/1426543014 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f3ca287ad0 0x55f3ca289ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 4 client.18030 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:48:51.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 1 -- 172.21.15.23:0/1426543014 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3c9e19700 con 0x55f3ca287ad0 2024-03-24T10:48:51.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d92d9d640 1 --2- 172.21.15.23:0/1426543014 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f3ca287ad0 0x55f3ca289ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d76ff5640 10 client.18030 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:48:51.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d92d9d640 1 --2- 172.21.15.23:0/1426543014 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f3ca287ad0 0x55f3ca289ec0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ef3c41c57d16db1f server_cookie=50bf5054bd6a10ff in_seq=0 out_seq=0 2024-03-24T10:48:51.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d76ff5640 1 -- 172.21.15.23:0/1426543014 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3c9e19700 con 0x55f3ca287ad0 2024-03-24T10:48:51.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d76ff5640 10 client.18030 handle_command_reply: tid=0 2024-03-24T10:48:51.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 1 client.18030 shutdown 2024-03-24T10:48:52.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 1 -- 172.21.15.23:0/1426543014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2108f0 msgr2=0x55f3ca215cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:52.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/1426543014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2108f0 0x55f3ca215cc0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f0d84011f30 tx=0x7f0d840095a0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 1 -- 172.21.15.23:0/1426543014 shutdown_connections 2024-03-24T10:48:52.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/1426543014 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55f3ca287ad0 0x55f3ca289ec0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/1426543014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f3ca2108f0 0x55f3ca215cc0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/1426543014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3ca207d10 0x55f3ca215780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 1 --2- 172.21.15.23:0/1426543014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3ca2073f0 0x55f3ca2187b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d76ff5640 0 client.18030 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:48:52.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 1 -- 172.21.15.23:0/1426543014 >> 172.21.15.23:0/1426543014 conn(0x55f3ca201b20 msgr2=0x55f3ca2037e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:52.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.988+0000 7f0d9bc03740 1 -- 172.21.15.23:0/1426543014 shutdown_connections 2024-03-24T10:48:52.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.989+0000 7f0d9bc03740 1 -- 172.21.15.23:0/1426543014 wait complete. 2024-03-24T10:48:52.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.989+0000 7f0d9bc03740 20 client.18030 trim_cache size 0 max 16384 2024-03-24T10:48:52.003 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:48:52.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.989+0000 7f0d99023640 1 -- 172.21.15.23:0/580588599 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0d7c0a2260 msgr2=0x7f0d7c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:52.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.989+0000 7f0d99023640 1 --2- 172.21.15.23:0/580588599 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0d7c0a2260 0x7f0d7c0a4720 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c005ab0 tx=0x7f0d8c005fd0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.989+0000 7f0d99023640 1 -- 172.21.15.23:0/580588599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d940692b0 msgr2=0x7f0d94100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:52.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.989+0000 7f0d99023640 1 --2- 172.21.15.23:0/580588599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d940692b0 0x7f0d94100b10 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800ea60 tx=0x7f0d8800ee60 comp rx=0 tx=0).stop 2024-03-24T10:48:52.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.989+0000 7f0d99023640 1 -- 172.21.15.23:0/580588599 shutdown_connections 2024-03-24T10:48:52.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.989+0000 7f0d99023640 1 --2- 172.21.15.23:0/580588599 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0d7c0a2260 0x7f0d7c0a4720 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.989+0000 7f0d99023640 1 --2- 172.21.15.23:0/580588599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d94107690 0x7f0d94101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.989+0000 7f0d99023640 1 --2- 172.21.15.23:0/580588599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d940692b0 0x7f0d94100b10 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.989+0000 7f0d99023640 1 --2- 172.21.15.23:0/580588599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d940688f0 0x7f0d941005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.989+0000 7f0d99023640 1 -- 172.21.15.23:0/580588599 >> 172.21.15.23:0/580588599 conn(0x7f0d94077710 msgr2=0x7f0d940fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:52.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.989+0000 7f0d99023640 1 -- 172.21.15.23:0/580588599 shutdown_connections 2024-03-24T10:48:52.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:51.989+0000 7f0d99023640 1 -- 172.21.15.23:0/580588599 wait complete. 2024-03-24T10:48:52.006 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:48:52.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.112+0000 7f71c6042640 1 -- 172.21.15.23:0/1059692557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71c0101ae0 msgr2=0x7f71c010d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:52.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.112+0000 7f71c6042640 1 --2- 172.21.15.23:0/1059692557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71c0101ae0 0x7f71c010d490 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f71b8009cd0 tx=0x7f71b802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.112+0000 7f71c6042640 1 -- 172.21.15.23:0/1059692557 shutdown_connections 2024-03-24T10:48:52.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.112+0000 7f71c6042640 1 --2- 172.21.15.23:0/1059692557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71c0101ae0 0x7f71c010d490 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.112+0000 7f71c6042640 1 --2- 172.21.15.23:0/1059692557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71c0101120 0x7f71c01015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.112+0000 7f71c6042640 1 --2- 172.21.15.23:0/1059692557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c01055b0 0x7f71c0105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.112+0000 7f71c6042640 1 -- 172.21.15.23:0/1059692557 >> 172.21.15.23:0/1059692557 conn(0x7f71c0075870 msgr2=0x7f71c0075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.112+0000 7f71c6042640 1 -- 172.21.15.23:0/1059692557 shutdown_connections 2024-03-24T10:48:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.112+0000 7f71c6042640 1 -- 172.21.15.23:0/1059692557 wait complete. 2024-03-24T10:48:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71c6042640 1 Processor -- start 2024-03-24T10:48:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71c6042640 1 -- start start 2024-03-24T10:48:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71c6042640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0101120 0x7f71c019d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71c6042640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71c0101ae0 0x7f71c019ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71c6042640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71c01055b0 0x7f71c0197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71c6042640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0198090 con 0x7f71c0101ae0 2024-03-24T10:48:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71c6042640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0198200 con 0x7f71c0101120 2024-03-24T10:48:52.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71c6042640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01983c0 con 0x7f71c01055b0 2024-03-24T10:48:52.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71c0101ae0 0x7f71c019ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71c0101ae0 0x7f71c019ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46340/0 (socket says 172.21.15.23:46340) 2024-03-24T10:48:52.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71beffd640 1 -- 172.21.15.23:0/1343430939 learned_addr learned my addr 172.21.15.23:0/1343430939 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:52.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71bf7fe640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0101120 0x7f71c019d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71bffff640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71c01055b0 0x7f71c0197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71beffd640 1 -- 172.21.15.23:0/1343430939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71c01055b0 msgr2=0x7f71c0197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:52.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71beffd640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71c01055b0 0x7f71c0197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71beffd640 1 -- 172.21.15.23:0/1343430939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0101120 msgr2=0x7f71c019d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:52.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71beffd640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0101120 0x7f71c019d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71beffd640 1 -- 172.21.15.23:0/1343430939 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b0009490 con 0x7f71c0101ae0 2024-03-24T10:48:52.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.113+0000 7f71bf7fe640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0101120 0x7f71c019d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:52.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.114+0000 7f71beffd640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71c0101ae0 0x7f71c019ddd0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f71b400e9f0 tx=0x7f71b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:52.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.114+0000 7f71bcff9640 1 -- 172.21.15.23:0/1343430939 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b4009700 con 0x7f71c0101ae0 2024-03-24T10:48:52.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.114+0000 7f71bcff9640 1 -- 172.21.15.23:0/1343430939 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b400cd80 con 0x7f71c0101ae0 2024-03-24T10:48:52.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.114+0000 7f71bcff9640 1 -- 172.21.15.23:0/1343430939 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b4018920 con 0x7f71c0101ae0 2024-03-24T10:48:52.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.114+0000 7f71c6042640 1 -- 172.21.15.23:0/1343430939 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71b8009900 con 0x7f71c0101ae0 2024-03-24T10:48:52.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.114+0000 7f71c6042640 1 -- 172.21.15.23:0/1343430939 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71c01a4750 con 0x7f71c0101ae0 2024-03-24T10:48:52.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.116+0000 7f71bcff9640 1 -- 172.21.15.23:0/1343430939 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f71b4018a80 con 0x7f71c0101ae0 2024-03-24T10:48:52.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.116+0000 7f71c8c1c740 1 Processor -- start 2024-03-24T10:48:52.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.116+0000 7f71c8c1c740 1 -- start start 2024-03-24T10:48:52.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71c8c1c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b2210a0 0x55702b221480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71c8c1c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71c8c1c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b22ed00 0x55702b2310f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71c8c1c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55702b18f840 con 0x55702b2219c0 2024-03-24T10:48:52.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71c8c1c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55702b18cd70 con 0x55702b2210a0 2024-03-24T10:48:52.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71c8c1c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55702b188550 con 0x55702b22ed00 2024-03-24T10:48:52.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bcff9640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f71a80a2230 0x7f71a80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b2210a0 0x55702b221480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46342/0 (socket says 172.21.15.23:46342) 2024-03-24T10:48:52.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bf7fe640 1 -- 172.21.15.23:0/2796389263 learned_addr learned my addr 172.21.15.23:0/2796389263 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:52.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71beffd640 1 --2- 172.21.15.23:0/2796389263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b22ed00 0x55702b2310f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bffff640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f71a80a2230 0x7f71a80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bcff9640 1 -- 172.21.15.23:0/1343430939 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f71b4014070 con 0x7f71c0101ae0 2024-03-24T10:48:52.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bf7fe640 1 -- 172.21.15.23:0/2796389263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b22ed00 msgr2=0x55702b2310f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:52.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bf7fe640 1 --2- 172.21.15.23:0/2796389263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b22ed00 0x55702b2310f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bf7fe640 1 -- 172.21.15.23:0/2796389263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b2210a0 msgr2=0x55702b221480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:52.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bf7fe640 1 --2- 172.21.15.23:0/2796389263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b2210a0 0x55702b221480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bf7fe640 1 -- 172.21.15.23:0/2796389263 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55702b11dde0 con 0x55702b2219c0 2024-03-24T10:48:52.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bffff640 1 --2- 172.21.15.23:0/2796389263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b2210a0 0x55702b221480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:52.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bf7fe640 1 --2- 172.21.15.23:0/2796389263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22e7c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f71b0011680 tx=0x7f71b0011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1174ccb1d0829503 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:52.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71bffff640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f71a80a2230 0x7f71a80a46f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f71b8005cc0 tx=0x7f71b8005c50 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:52.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.117+0000 7f71a5ffb640 1 -- 172.21.15.23:0/2796389263 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b000fc00 con 0x55702b2219c0 2024-03-24T10:48:52.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.118+0000 7f71a5ffb640 1 -- 172.21.15.23:0/2796389263 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b000fd60 con 0x55702b2219c0 2024-03-24T10:48:52.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.118+0000 7f71a5ffb640 1 -- 172.21.15.23:0/2796389263 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0004d90 con 0x55702b2219c0 2024-03-24T10:48:52.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.118+0000 7f71c8c1c740 1 -- 172.21.15.23:0/2796389263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 msgr2=0x55702b22e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:52.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.118+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/2796389263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22e7c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f71b0011680 tx=0x7f71b0011b40 comp rx=0 tx=0).stop 2024-03-24T10:48:52.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.118+0000 7f71c8c1c740 1 -- 172.21.15.23:0/2796389263 shutdown_connections 2024-03-24T10:48:52.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.118+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/2796389263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b22ed00 0x55702b2310f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.118+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/2796389263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22e7c0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.118+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/2796389263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b2210a0 0x55702b221480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.118+0000 7f71c8c1c740 1 -- 172.21.15.23:0/2796389263 >> 172.21.15.23:0/2796389263 conn(0x55702b21bb20 msgr2=0x55702b21bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:52.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.118+0000 7f71c8c1c740 1 -- 172.21.15.23:0/2796389263 shutdown_connections 2024-03-24T10:48:52.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.118+0000 7f71c8c1c740 1 -- 172.21.15.23:0/2796389263 wait complete. 2024-03-24T10:48:52.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71c8c1c740 1 -- start start 2024-03-24T10:48:52.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71c8c1c740 1 Processor -- start 2024-03-24T10:48:52.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71c8c1c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b2210a0 0x55702b22bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71c8c1c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71c8c1c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b22ed00 0x55702b22c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71c8c1c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55702ae5b430 con 0x55702b2219c0 2024-03-24T10:48:52.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71c8c1c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55702ae17e00 con 0x55702b22ed00 2024-03-24T10:48:52.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71c8c1c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55702b10d590 con 0x55702b2210a0 2024-03-24T10:48:52.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71c8c1c740 10 client.0 _ll_register_callbacks cb 0x55702b17cfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:52.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46356/0 (socket says 172.21.15.23:46356) 2024-03-24T10:48:52.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71beffd640 1 -- 172.21.15.23:0/63190180 learned_addr learned my addr 172.21.15.23:0/63190180 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:52.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71bffff640 1 --2- 172.21.15.23:0/63190180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b22ed00 0x55702b22c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71bf7fe640 1 --2- 172.21.15.23:0/63190180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b2210a0 0x55702b22bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71beffd640 1 -- 172.21.15.23:0/63190180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b2210a0 msgr2=0x55702b22bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:52.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71beffd640 1 --2- 172.21.15.23:0/63190180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b2210a0 0x55702b22bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71beffd640 1 -- 172.21.15.23:0/63190180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b22ed00 msgr2=0x55702b22c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71beffd640 1 --2- 172.21.15.23:0/63190180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b22ed00 0x55702b22c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71beffd640 1 -- 172.21.15.23:0/63190180 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71b0009e10 con 0x55702b2219c0 2024-03-24T10:48:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.119+0000 7f71bf7fe640 1 --2- 172.21.15.23:0/63190180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b2210a0 0x55702b22bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:48:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f71beffd640 1 --2- 172.21.15.23:0/63190180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22c220 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f71b4097890 tx=0x7f71b4090350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365dcf2fb8242ef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f719effd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f719effd640 1 -- 172.21.15.23:0/63190180 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b409ef60 con 0x55702b2219c0 2024-03-24T10:48:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f719effd640 1 -- 172.21.15.23:0/63190180 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f71b40a35b0 con 0x55702b2219c0 2024-03-24T10:48:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f71c8c1c740 1 -- 172.21.15.23:0/63190180 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55702ae877e0 con 0x55702b2219c0 2024-03-24T10:48:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f719effd640 1 -- 172.21.15.23:0/63190180 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f71b4097530 con 0x55702b2219c0 2024-03-24T10:48:52.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f71c8c1c740 10 client.18042 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:52.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f71c8c1c740 1 -- 172.21.15.23:0/63190180 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55702b186bd0 con 0x55702b2219c0 2024-03-24T10:48:52.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f719effd640 1 -- 172.21.15.23:0/63190180 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55702ae877e0 con 0x55702b2219c0 2024-03-24T10:48:52.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f719effd640 1 -- 172.21.15.23:0/63190180 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f71b409a4f0 con 0x55702b2219c0 2024-03-24T10:48:52.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f71c8c1c740 10 client.18042 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:52.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f71c8c1c740 10 client.18042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:52.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f71c8c1c740 20 client.18042 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:52.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/63190180 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55702b23ee50 0x55702b2412d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f71c8c1c740 4 client.18042 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:48:52.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f71c8c1c740 1 -- 172.21.15.23:0/63190180 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55702aec4720 con 0x55702b23ee50 2024-03-24T10:48:52.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.120+0000 7f71bf7fe640 1 --2- 172.21.15.23:0/63190180 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55702b23ee50 0x55702b2412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.121+0000 7f719effd640 10 client.18042 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:52.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.121+0000 7f71bf7fe640 1 --2- 172.21.15.23:0/63190180 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55702b23ee50 0x55702b2412d0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d653975cbaa1c438 server_cookie=71dcaa2554dc1c61 in_seq=0 out_seq=0 2024-03-24T10:48:52.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f719effd640 1 -- 172.21.15.23:0/63190180 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55702aec4720 con 0x55702b23ee50 2024-03-24T10:48:52.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f719effd640 10 client.18042 handle_command_reply: tid=0 2024-03-24T10:48:52.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f71c8c1c740 1 client.18042 shutdown 2024-03-24T10:48:52.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f71c8c1c740 1 -- 172.21.15.23:0/63190180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 msgr2=0x55702b22c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/63190180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22c220 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f71b4097890 tx=0x7f71b4090350 comp rx=0 tx=0).stop 2024-03-24T10:48:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f71c8c1c740 1 -- 172.21.15.23:0/63190180 shutdown_connections 2024-03-24T10:48:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/63190180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b22ed00 0x55702b22c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/63190180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22c220 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/63190180 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55702b23ee50 0x55702b2412d0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/63190180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b2210a0 0x55702b22bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f71c8c1c740 1 -- 172.21.15.23:0/63190180 >> 172.21.15.23:0/63190180 conn(0x55702b21bb20 msgr2=0x55702b21d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f719effd640 0 client.18042 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f71c8c1c740 1 -- 172.21.15.23:0/63190180 shutdown_connections 2024-03-24T10:48:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f71c8c1c740 1 -- 172.21.15.23:0/63190180 wait complete. 2024-03-24T10:48:52.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.122+0000 7f71c8c1c740 20 client.18042 trim_cache size 0 max 16384 2024-03-24T10:48:52.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.135+0000 7f71c8c1c740 1 Processor -- start 2024-03-24T10:48:52.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.135+0000 7f71c8c1c740 1 -- start start 2024-03-24T10:48:52.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.135+0000 7f71c8c1c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b2210a0 0x55702b22e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.135+0000 7f71c8c1c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.135+0000 7f71c8c1c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b22ed00 0x55702b246be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.135+0000 7f71c8c1c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55702b20ce20 con 0x55702b2219c0 2024-03-24T10:48:52.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.135+0000 7f71c8c1c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55702b220710 con 0x55702b22ed00 2024-03-24T10:48:52.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.135+0000 7f71c8c1c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55702b2173e0 con 0x55702b2210a0 2024-03-24T10:48:52.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.135+0000 7f71beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.135+0000 7f71beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46368/0 (socket says 172.21.15.23:46368) 2024-03-24T10:48:52.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.136+0000 7f71beffd640 1 -- 172.21.15.23:0/3881043051 learned_addr learned my addr 172.21.15.23:0/3881043051 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:52.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.136+0000 7f71bf7fe640 1 --2- 172.21.15.23:0/3881043051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b2210a0 0x55702b22e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.136+0000 7f71bffff640 1 --2- 172.21.15.23:0/3881043051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b22ed00 0x55702b246be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.136+0000 7f71beffd640 1 -- 172.21.15.23:0/3881043051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b2210a0 msgr2=0x55702b22e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:52.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.136+0000 7f71beffd640 1 --2- 172.21.15.23:0/3881043051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b2210a0 0x55702b22e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.136+0000 7f71beffd640 1 -- 172.21.15.23:0/3881043051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b22ed00 msgr2=0x55702b246be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:52.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.136+0000 7f71beffd640 1 --2- 172.21.15.23:0/3881043051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b22ed00 0x55702b246be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.136+0000 7f71beffd640 1 -- 172.21.15.23:0/3881043051 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55702b122c80 con 0x55702b2219c0 2024-03-24T10:48:52.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.136+0000 7f71bffff640 1 --2- 172.21.15.23:0/3881043051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b22ed00 0x55702b246be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:48:52.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.136+0000 7f71beffd640 1 --2- 172.21.15.23:0/3881043051 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22c580 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f71b409b080 tx=0x7f71b4010f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af199ae137aaa8d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:52.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.136+0000 7f71a57fa640 1 -- 172.21.15.23:0/3881043051 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b409a040 con 0x55702b2219c0 2024-03-24T10:48:52.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.136+0000 7f71a57fa640 1 -- 172.21.15.23:0/3881043051 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b409d070 con 0x55702b2219c0 2024-03-24T10:48:52.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.136+0000 7f71a57fa640 1 -- 172.21.15.23:0/3881043051 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b40a3380 con 0x55702b2219c0 2024-03-24T10:48:52.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 -- 172.21.15.23:0/3881043051 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 msgr2=0x55702b22c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:52.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/3881043051 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22c580 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f71b409b080 tx=0x7f71b4010f50 comp rx=0 tx=0).stop 2024-03-24T10:48:52.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 -- 172.21.15.23:0/3881043051 shutdown_connections 2024-03-24T10:48:52.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/3881043051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b22ed00 0x55702b246be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/3881043051 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2219c0 0x55702b22c580 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/3881043051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b2210a0 0x55702b22e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 -- 172.21.15.23:0/3881043051 >> 172.21.15.23:0/3881043051 conn(0x55702b21bb20 msgr2=0x55702b21cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:52.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 -- 172.21.15.23:0/3881043051 shutdown_connections 2024-03-24T10:48:52.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 -- 172.21.15.23:0/3881043051 wait complete. 2024-03-24T10:48:52.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 -- start start 2024-03-24T10:48:52.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 Processor -- start 2024-03-24T10:48:52.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2210a0 0x55702b232970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b2219c0 0x55702b232eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b22ed00 0x55702b2a4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55702b2337b0 con 0x55702b2210a0 2024-03-24T10:48:52.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55702b2a4750 con 0x55702b2219c0 2024-03-24T10:48:52.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55702b2a4890 con 0x55702b22ed00 2024-03-24T10:48:52.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71c8c1c740 10 client.0 _ll_register_callbacks cb 0x7f71b400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:52.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.137+0000 7f71bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2210a0 0x55702b232970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f71bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2210a0 0x55702b232970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46374/0 (socket says 172.21.15.23:46374) 2024-03-24T10:48:52.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f71bf7fe640 1 -- 172.21.15.23:0/637434743 learned_addr learned my addr 172.21.15.23:0/637434743 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:52.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f71beffd640 1 --2- 172.21.15.23:0/637434743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b2219c0 0x55702b232eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f71bffff640 1 --2- 172.21.15.23:0/637434743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b22ed00 0x55702b2a4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f71bf7fe640 1 -- 172.21.15.23:0/637434743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b22ed00 msgr2=0x55702b2a4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:52.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f71bf7fe640 1 --2- 172.21.15.23:0/637434743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b22ed00 0x55702b2a4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f71bf7fe640 1 -- 172.21.15.23:0/637434743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b2219c0 msgr2=0x55702b232eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:52.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f71bf7fe640 1 --2- 172.21.15.23:0/637434743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b2219c0 0x55702b232eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f71bf7fe640 1 -- 172.21.15.23:0/637434743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55702b186bd0 con 0x55702b2210a0 2024-03-24T10:48:52.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f71bffff640 1 --2- 172.21.15.23:0/637434743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b22ed00 0x55702b2a4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:48:52.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f71bf7fe640 1 --2- 172.21.15.23:0/637434743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2210a0 0x55702b232970 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f71b0005fd0 tx=0x7f71b000f940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b0ebebfdddfc0e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:52.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f719effd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:52.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f719effd640 1 -- 172.21.15.23:0/637434743 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b002a040 con 0x55702b2210a0 2024-03-24T10:48:52.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f719effd640 1 -- 172.21.15.23:0/637434743 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f71b0004250 con 0x55702b2210a0 2024-03-24T10:48:52.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f71c8c1c740 1 -- 172.21.15.23:0/637434743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71b000fd60 con 0x55702b2210a0 2024-03-24T10:48:52.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f719effd640 1 -- 172.21.15.23:0/637434743 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f71b0005a70 con 0x55702b2210a0 2024-03-24T10:48:52.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f71c8c1c740 10 client.18048 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:52.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f71c8c1c740 1 -- 172.21.15.23:0/637434743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55702b148920 con 0x55702b2210a0 2024-03-24T10:48:52.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.138+0000 7f719effd640 1 -- 172.21.15.23:0/637434743 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b000fd60 con 0x55702b2210a0 2024-03-24T10:48:52.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f719effd640 1 -- 172.21.15.23:0/637434743 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f71b0035070 con 0x55702b2210a0 2024-03-24T10:48:52.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f71c8c1c740 10 client.18048 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:52.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f71c8c1c740 10 client.18048 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:52.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f71c8c1c740 20 client.18048 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:52.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/637434743 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55702b2a5f70 0x55702b2a8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:52.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f71c8c1c740 4 client.18048 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:48:52.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f71c8c1c740 1 -- 172.21.15.23:0/637434743 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55702af13890 con 0x55702b2a5f70 2024-03-24T10:48:52.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f71beffd640 1 --2- 172.21.15.23:0/637434743 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55702b2a5f70 0x55702b2a8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:52.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f719effd640 10 client.18048 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:52.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f71beffd640 1 --2- 172.21.15.23:0/637434743 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55702b2a5f70 0x55702b2a8360 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d9cff4386030cb1 server_cookie=7e0274682e48e935 in_seq=0 out_seq=0 2024-03-24T10:48:52.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f719effd640 1 -- 172.21.15.23:0/637434743 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55702af13890 con 0x55702b2a5f70 2024-03-24T10:48:52.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f719effd640 10 client.18048 handle_command_reply: tid=0 2024-03-24T10:48:52.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f71c8c1c740 1 client.18048 shutdown 2024-03-24T10:48:52.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f71c8c1c740 1 -- 172.21.15.23:0/637434743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2210a0 msgr2=0x55702b232970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:52.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.139+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/637434743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2210a0 0x55702b232970 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f71b0005fd0 tx=0x7f71b000f940 comp rx=0 tx=0).stop 2024-03-24T10:48:52.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c8c1c740 1 -- 172.21.15.23:0/637434743 shutdown_connections 2024-03-24T10:48:52.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/637434743 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55702b2a5f70 0x55702b2a8360 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/637434743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55702b22ed00 0x55702b2a4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f719effd640 0 client.18048 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:52.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/637434743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55702b2219c0 0x55702b232eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c8c1c740 1 --2- 172.21.15.23:0/637434743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55702b2210a0 0x55702b232970 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c8c1c740 1 -- 172.21.15.23:0/637434743 >> 172.21.15.23:0/637434743 conn(0x55702b21bb20 msgr2=0x55702b21cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:52.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c8c1c740 1 -- 172.21.15.23:0/637434743 shutdown_connections 2024-03-24T10:48:52.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c8c1c740 1 -- 172.21.15.23:0/637434743 wait complete. 2024-03-24T10:48:52.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c8c1c740 20 client.18048 trim_cache size 0 max 16384 2024-03-24T10:48:52.152 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:48:52.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c6042640 1 -- 172.21.15.23:0/1343430939 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f71a80a2230 msgr2=0x7f71a80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:52.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c6042640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f71a80a2230 0x7f71a80a46f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f71b8005cc0 tx=0x7f71b8005c50 comp rx=0 tx=0).stop 2024-03-24T10:48:52.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c6042640 1 -- 172.21.15.23:0/1343430939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71c0101ae0 msgr2=0x7f71c019ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:52.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c6042640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71c0101ae0 0x7f71c019ddd0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f71b400e9f0 tx=0x7f71b400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c6042640 1 -- 172.21.15.23:0/1343430939 shutdown_connections 2024-03-24T10:48:52.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c6042640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f71a80a2230 0x7f71a80a46f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c6042640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71c01055b0 0x7f71c0197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c6042640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71c0101ae0 0x7f71c019ddd0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c6042640 1 --2- 172.21.15.23:0/1343430939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0101120 0x7f71c019d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:52.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c6042640 1 -- 172.21.15.23:0/1343430939 >> 172.21.15.23:0/1343430939 conn(0x7f71c0075870 msgr2=0x7f71c00fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:52.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c6042640 1 -- 172.21.15.23:0/1343430939 shutdown_connections 2024-03-24T10:48:52.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:52.140+0000 7f71c6042640 1 -- 172.21.15.23:0/1343430939 wait complete. 2024-03-24T10:48:52.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:52 smithi117 ceph-mon[36368]: pgmap v272: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 9 op/s 2024-03-24T10:48:52.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:52 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4218110867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:48:52.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:52 smithi116 ceph-mon[36248]: pgmap v272: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 9 op/s 2024-03-24T10:48:52.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:52 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4218110867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:48:52.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:52 smithi023 ceph-mon[32466]: pgmap v272: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 9 op/s 2024-03-24T10:48:52.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:52 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4218110867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:48:53.155 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:48:53.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:48:53.156 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:48:53.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.275+0000 7f6f325c7640 1 -- 172.21.15.23:0/2166958013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2c100c30 msgr2=0x7f6f2c106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.275+0000 7f6f325c7640 1 --2- 172.21.15.23:0/2166958013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2c100c30 0x7f6f2c106360 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009930 tx=0x7f6f2002bc30 comp rx=0 tx=0).stop 2024-03-24T10:48:53.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.275+0000 7f6f325c7640 1 -- 172.21.15.23:0/2166958013 shutdown_connections 2024-03-24T10:48:53.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.275+0000 7f6f325c7640 1 --2- 172.21.15.23:0/2166958013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c106a40 0x7f6f2c10e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.275+0000 7f6f325c7640 1 --2- 172.21.15.23:0/2166958013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2c100c30 0x7f6f2c106360 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.275+0000 7f6f325c7640 1 --2- 172.21.15.23:0/2166958013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f2c100280 0x7f6f2c100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.275+0000 7f6f325c7640 1 -- 172.21.15.23:0/2166958013 >> 172.21.15.23:0/2166958013 conn(0x7f6f2c075870 msgr2=0x7f6f2c075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.275+0000 7f6f325c7640 1 -- 172.21.15.23:0/2166958013 shutdown_connections 2024-03-24T10:48:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.275+0000 7f6f325c7640 1 -- 172.21.15.23:0/2166958013 wait complete. 2024-03-24T10:48:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.276+0000 7f6f325c7640 1 Processor -- start 2024-03-24T10:48:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.276+0000 7f6f325c7640 1 -- start start 2024-03-24T10:48:53.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.276+0000 7f6f325c7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2c100280 0x7f6f2c197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.276+0000 7f6f325c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c106a40 0x7f6f2c198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.276+0000 7f6f325c7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f2c19c6c0 0x7f6f2c19cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.276+0000 7f6f325c7640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c19d0b0 con 0x7f6f2c100280 2024-03-24T10:48:53.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.276+0000 7f6f325c7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c19d220 con 0x7f6f2c106a40 2024-03-24T10:48:53.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.276+0000 7f6f325c7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c19d390 con 0x7f6f2c19c6c0 2024-03-24T10:48:53.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.276+0000 7f6f2bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2c100280 0x7f6f2c197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.276+0000 7f6f2bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2c100280 0x7f6f2c197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46400/0 (socket says 172.21.15.23:46400) 2024-03-24T10:48:53.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.276+0000 7f6f2bfff640 1 -- 172.21.15.23:0/1536378850 learned_addr learned my addr 172.21.15.23:0/1536378850 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:53.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.276+0000 7f6f2b7fe640 1 --2- 172.21.15.23:0/1536378850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c106a40 0x7f6f2c198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.276+0000 7f6f30b64640 1 --2- 172.21.15.23:0/1536378850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f2c19c6c0 0x7f6f2c19cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.277+0000 7f6f2bfff640 1 -- 172.21.15.23:0/1536378850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f2c19c6c0 msgr2=0x7f6f2c19cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.277+0000 7f6f2bfff640 1 --2- 172.21.15.23:0/1536378850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f2c19c6c0 0x7f6f2c19cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.277+0000 7f6f2bfff640 1 -- 172.21.15.23:0/1536378850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c106a40 msgr2=0x7f6f2c198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.277+0000 7f6f2bfff640 1 --2- 172.21.15.23:0/1536378850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c106a40 0x7f6f2c198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.277+0000 7f6f2bfff640 1 -- 172.21.15.23:0/1536378850 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f20009560 con 0x7f6f2c100280 2024-03-24T10:48:53.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.277+0000 7f6f2bfff640 1 --2- 172.21.15.23:0/1536378850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2c100280 0x7f6f2c197e00 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00e9f0 tx=0x7f6f1c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.277+0000 7f6f297fa640 1 -- 172.21.15.23:0/1536378850 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c009700 con 0x7f6f2c100280 2024-03-24T10:48:53.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.277+0000 7f6f325c7640 1 -- 172.21.15.23:0/1536378850 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f2c1a4970 con 0x7f6f2c100280 2024-03-24T10:48:53.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.277+0000 7f6f297fa640 1 -- 172.21.15.23:0/1536378850 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f1c00cd80 con 0x7f6f2c100280 2024-03-24T10:48:53.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.277+0000 7f6f297fa640 1 -- 172.21.15.23:0/1536378850 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c018920 con 0x7f6f2c100280 2024-03-24T10:48:53.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.277+0000 7f6f325c7640 1 -- 172.21.15.23:0/1536378850 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f2c1a4e40 con 0x7f6f2c100280 2024-03-24T10:48:53.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.279+0000 7f6f297fa640 1 -- 172.21.15.23:0/1536378850 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f6f1c018a80 con 0x7f6f2c100280 2024-03-24T10:48:53.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.280+0000 7f6f351a6740 1 Processor -- start 2024-03-24T10:48:53.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.280+0000 7f6f351a6740 1 -- start start 2024-03-24T10:48:53.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.280+0000 7f6f351a6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d30b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.280+0000 7f6f351a6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d31080 0x5617f1d39720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.280+0000 7f6f297fa640 1 --2- 172.21.15.23:0/1536378850 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6f140a2230 0x7f6f140a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.280+0000 7f6f351a6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d39c60 0x5617f1d3c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.280+0000 7f6f351a6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5617f1c9e530 con 0x5617f1d31080 2024-03-24T10:48:53.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.280+0000 7f6f351a6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617f1c9ba60 con 0x5617f1d30760 2024-03-24T10:48:53.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.280+0000 7f6f351a6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617f1c97240 con 0x5617f1d39c60 2024-03-24T10:48:53.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.280+0000 7f6f2bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d31080 0x5617f1d39720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.280+0000 7f6f30b64640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d30b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.280+0000 7f6f2b7fe640 1 --2- 172.21.15.23:0/1536378850 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6f140a2230 0x7f6f140a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.280+0000 7f6f30b64640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d30b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:50038/0 (socket says 172.21.15.23:50038) 2024-03-24T10:48:53.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.281+0000 7f6f30b64640 1 -- 172.21.15.23:0/2487498276 learned_addr learned my addr 172.21.15.23:0/2487498276 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:53.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.281+0000 7f6f30b64640 1 --2- 172.21.15.23:0/2487498276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d39c60 0x5617f1d3c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.281+0000 7f6f297fa640 1 -- 172.21.15.23:0/1536378850 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f6f1c014070 con 0x7f6f2c100280 2024-03-24T10:48:53.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.281+0000 7f6f30b64640 1 -- 172.21.15.23:0/2487498276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d39c60 msgr2=0x5617f1d3c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.281+0000 7f6f30b64640 1 --2- 172.21.15.23:0/2487498276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d39c60 0x5617f1d3c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.281+0000 7f6f30b64640 1 -- 172.21.15.23:0/2487498276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d31080 msgr2=0x5617f1d39720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.281+0000 7f6f30b64640 1 --2- 172.21.15.23:0/2487498276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d31080 0x5617f1d39720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.281+0000 7f6f30b64640 1 -- 172.21.15.23:0/2487498276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617f1c958c0 con 0x5617f1d30760 2024-03-24T10:48:53.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.281+0000 7f6f2b7fe640 1 --2- 172.21.15.23:0/1536378850 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6f140a2230 0x7f6f140a46f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6f20000c00 tx=0x7f6f20005fd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.281+0000 7f6f30b64640 1 --2- 172.21.15.23:0/2487498276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d30b40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6f24005f40 tx=0x7f6f24005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7d39fa0cddf05d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.281+0000 7f6f127fc640 1 -- 172.21.15.23:0/2487498276 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f24004b20 con 0x5617f1d30760 2024-03-24T10:48:53.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.281+0000 7f6f127fc640 1 -- 172.21.15.23:0/2487498276 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f2400fd60 con 0x5617f1d30760 2024-03-24T10:48:53.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.281+0000 7f6f127fc640 1 -- 172.21.15.23:0/2487498276 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f24005550 con 0x5617f1d30760 2024-03-24T10:48:53.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.282+0000 7f6f351a6740 1 -- 172.21.15.23:0/2487498276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 msgr2=0x5617f1d30b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.282+0000 7f6f351a6740 1 --2- 172.21.15.23:0/2487498276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d30b40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6f24005f40 tx=0x7f6f24005bd0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.282+0000 7f6f351a6740 1 -- 172.21.15.23:0/2487498276 shutdown_connections 2024-03-24T10:48:53.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.282+0000 7f6f351a6740 1 --2- 172.21.15.23:0/2487498276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d39c60 0x5617f1d3c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.282+0000 7f6f351a6740 1 --2- 172.21.15.23:0/2487498276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d31080 0x5617f1d39720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.282+0000 7f6f351a6740 1 --2- 172.21.15.23:0/2487498276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d30b40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.282+0000 7f6f351a6740 1 -- 172.21.15.23:0/2487498276 >> 172.21.15.23:0/2487498276 conn(0x5617f1d2ae90 msgr2=0x5617f1d2b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:53.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.282+0000 7f6f351a6740 1 -- 172.21.15.23:0/2487498276 shutdown_connections 2024-03-24T10:48:53.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.282+0000 7f6f351a6740 1 -- 172.21.15.23:0/2487498276 wait complete. 2024-03-24T10:48:53.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.282+0000 7f6f351a6740 1 -- start start 2024-03-24T10:48:53.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f351a6740 1 Processor -- start 2024-03-24T10:48:53.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f351a6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d30760 0x5617f1d36c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f351a6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d31080 0x5617f1d37180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f351a6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d39c60 0x5617f1d376c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f351a6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5617f1c3bf80 con 0x5617f1d39c60 2024-03-24T10:48:53.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f351a6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617f196a4b0 con 0x5617f1d31080 2024-03-24T10:48:53.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f351a6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617f1c1cad0 con 0x5617f1d30760 2024-03-24T10:48:53.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f351a6740 10 client.0 _ll_register_callbacks cb 0x5617f1c4fb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:53.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f2bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d30760 0x5617f1d36c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f2b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d31080 0x5617f1d37180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f30b64640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d39c60 0x5617f1d376c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f2bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d30760 0x5617f1d36c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39464/0 (socket says 172.21.15.23:39464) 2024-03-24T10:48:53.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f2bfff640 1 -- 172.21.15.23:0/2591764806 learned_addr learned my addr 172.21.15.23:0/2591764806 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:53.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f2b7fe640 1 -- 172.21.15.23:0/2591764806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d30760 msgr2=0x5617f1d36c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f2b7fe640 1 --2- 172.21.15.23:0/2591764806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d30760 0x5617f1d36c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f2b7fe640 1 -- 172.21.15.23:0/2591764806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d39c60 msgr2=0x5617f1d376c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f2b7fe640 1 --2- 172.21.15.23:0/2591764806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d39c60 0x5617f1d376c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f2b7fe640 1 -- 172.21.15.23:0/2591764806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f24019040 con 0x5617f1d31080 2024-03-24T10:48:53.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f2b7fe640 1 --2- 172.21.15.23:0/2591764806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d31080 0x5617f1d37180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6f2002c170 tx=0x7f6f20009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee97c11d9fb3b1e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.283+0000 7f6f0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:48:53.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.284+0000 7f6f0b7fe640 1 -- 172.21.15.23:0/2591764806 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f20004310 con 0x5617f1d31080 2024-03-24T10:48:53.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.284+0000 7f6f351a6740 1 -- 172.21.15.23:0/2591764806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5617f1cbb7a0 con 0x5617f1d31080 2024-03-24T10:48:53.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.284+0000 7f6f0b7fe640 1 -- 172.21.15.23:0/2591764806 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f6f20047050 con 0x5617f1d31080 2024-03-24T10:48:53.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.284+0000 7f6f0b7fe640 1 -- 172.21.15.23:0/2591764806 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6f20004470 con 0x5617f1d31080 2024-03-24T10:48:53.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.284+0000 7f6f351a6740 10 client.27947 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:53.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.284+0000 7f6f351a6740 1 -- 172.21.15.23:0/2591764806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5617f1c4d6d0 con 0x5617f1d31080 2024-03-24T10:48:53.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.284+0000 7f6f0b7fe640 1 -- 172.21.15.23:0/2591764806 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5617f1cbb7a0 con 0x5617f1d31080 2024-03-24T10:48:53.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.284+0000 7f6f0b7fe640 1 -- 172.21.15.23:0/2591764806 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f6f200038f0 con 0x5617f1d31080 2024-03-24T10:48:53.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.284+0000 7f6f351a6740 10 client.27947 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:53.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.284+0000 7f6f351a6740 10 client.27947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:53.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.284+0000 7f6f351a6740 20 client.27947 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:53.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.284+0000 7f6f351a6740 1 --2- 172.21.15.23:0/2591764806 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617f1d49d40 0x5617f1d4c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.284+0000 7f6f351a6740 4 client.27947 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:48:53.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.284+0000 7f6f351a6740 1 -- 172.21.15.23:0/2591764806 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617f1c8d1f0 con 0x5617f1d49d40 2024-03-24T10:48:53.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.285+0000 7f6f2bfff640 1 --2- 172.21.15.23:0/2591764806 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617f1d49d40 0x5617f1d4c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.285+0000 7f6f0b7fe640 10 client.27947 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:53.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.285+0000 7f6f2bfff640 1 --2- 172.21.15.23:0/2591764806 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617f1d49d40 0x5617f1d4c160 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ba0066abf8cb50c server_cookie=2e003badea9bfcc8 in_seq=0 out_seq=0 2024-03-24T10:48:53.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.285+0000 7f6f0b7fe640 1 -- 172.21.15.23:0/2591764806 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5617f1c8d1f0 con 0x5617f1d49d40 2024-03-24T10:48:53.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.285+0000 7f6f0b7fe640 10 client.27947 handle_command_reply: tid=0 2024-03-24T10:48:53.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.286+0000 7f6f351a6740 1 client.27947 shutdown 2024-03-24T10:48:53.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.286+0000 7f6f351a6740 1 -- 172.21.15.23:0/2591764806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d31080 msgr2=0x5617f1d37180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.286+0000 7f6f351a6740 1 --2- 172.21.15.23:0/2591764806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d31080 0x5617f1d37180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6f2002c170 tx=0x7f6f20009260 comp rx=0 tx=0).stop 2024-03-24T10:48:53.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.286+0000 7f6f351a6740 1 -- 172.21.15.23:0/2591764806 shutdown_connections 2024-03-24T10:48:53.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.286+0000 7f6f351a6740 1 --2- 172.21.15.23:0/2591764806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d39c60 0x5617f1d376c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.286+0000 7f6f351a6740 1 --2- 172.21.15.23:0/2591764806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d31080 0x5617f1d37180 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.286+0000 7f6f351a6740 1 --2- 172.21.15.23:0/2591764806 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617f1d49d40 0x5617f1d4c160 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.286+0000 7f6f351a6740 1 --2- 172.21.15.23:0/2591764806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d30760 0x5617f1d36c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.286+0000 7f6f351a6740 1 -- 172.21.15.23:0/2591764806 >> 172.21.15.23:0/2591764806 conn(0x5617f1d2ae90 msgr2=0x5617f1d2cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:53.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.286+0000 7f6f0b7fe640 0 client.27947 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:53.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.286+0000 7f6f351a6740 1 -- 172.21.15.23:0/2591764806 shutdown_connections 2024-03-24T10:48:53.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.286+0000 7f6f351a6740 1 -- 172.21.15.23:0/2591764806 wait complete. 2024-03-24T10:48:53.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.286+0000 7f6f351a6740 20 client.27947 trim_cache size 0 max 16384 2024-03-24T10:48:53.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f351a6740 1 Processor -- start 2024-03-24T10:48:53.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f351a6740 1 -- start start 2024-03-24T10:48:53.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f351a6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d45520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f351a6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d31080 0x5617f1d374e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f351a6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d39c60 0x5617f1d51a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f351a6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5617f1d26620 con 0x5617f1d31080 2024-03-24T10:48:53.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f351a6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617f1d2c0c0 con 0x5617f1d30760 2024-03-24T10:48:53.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f351a6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617f1d351d0 con 0x5617f1d39c60 2024-03-24T10:48:53.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f2b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d31080 0x5617f1d374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f2b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d31080 0x5617f1d374e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46426/0 (socket says 172.21.15.23:46426) 2024-03-24T10:48:53.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f2b7fe640 1 -- 172.21.15.23:0/2152080404 learned_addr learned my addr 172.21.15.23:0/2152080404 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:53.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f2bfff640 1 --2- 172.21.15.23:0/2152080404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d45520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f30b64640 1 --2- 172.21.15.23:0/2152080404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d39c60 0x5617f1d51a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f2b7fe640 1 -- 172.21.15.23:0/2152080404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d39c60 msgr2=0x5617f1d51a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f2b7fe640 1 --2- 172.21.15.23:0/2152080404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d39c60 0x5617f1d51a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f2b7fe640 1 -- 172.21.15.23:0/2152080404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 msgr2=0x5617f1d45520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f2b7fe640 1 --2- 172.21.15.23:0/2152080404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d45520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f2b7fe640 1 -- 172.21.15.23:0/2152080404 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617f1c31c60 con 0x5617f1d31080 2024-03-24T10:48:53.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f2b7fe640 1 --2- 172.21.15.23:0/2152080404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d31080 0x5617f1d374e0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f6f20003ec0 tx=0x7f6f200097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f016aa7cffcf078a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.304+0000 7f6f11ffb640 1 -- 172.21.15.23:0/2152080404 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f20004510 con 0x5617f1d31080 2024-03-24T10:48:53.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.305+0000 7f6f11ffb640 1 -- 172.21.15.23:0/2152080404 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f20003ad0 con 0x5617f1d31080 2024-03-24T10:48:53.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.305+0000 7f6f11ffb640 1 -- 172.21.15.23:0/2152080404 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f20034440 con 0x5617f1d31080 2024-03-24T10:48:53.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.305+0000 7f6f351a6740 1 -- 172.21.15.23:0/2152080404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d31080 msgr2=0x5617f1d374e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.305+0000 7f6f351a6740 1 --2- 172.21.15.23:0/2152080404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d31080 0x5617f1d374e0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f6f20003ec0 tx=0x7f6f200097a0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.305+0000 7f6f351a6740 1 -- 172.21.15.23:0/2152080404 shutdown_connections 2024-03-24T10:48:53.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.305+0000 7f6f351a6740 1 --2- 172.21.15.23:0/2152080404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d39c60 0x5617f1d51a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.305+0000 7f6f351a6740 1 --2- 172.21.15.23:0/2152080404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d31080 0x5617f1d374e0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.305+0000 7f6f351a6740 1 --2- 172.21.15.23:0/2152080404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d45520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.305+0000 7f6f351a6740 1 -- 172.21.15.23:0/2152080404 >> 172.21.15.23:0/2152080404 conn(0x5617f1d2ae90 msgr2=0x5617f1d2cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:53.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.305+0000 7f6f351a6740 1 -- 172.21.15.23:0/2152080404 shutdown_connections 2024-03-24T10:48:53.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.305+0000 7f6f351a6740 1 -- 172.21.15.23:0/2152080404 wait complete. 2024-03-24T10:48:53.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.305+0000 7f6f351a6740 1 -- start start 2024-03-24T10:48:53.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f351a6740 1 Processor -- start 2024-03-24T10:48:53.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f351a6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d41e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f351a6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d31080 0x5617f1d3eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f351a6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d39c60 0x5617f1d3f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f351a6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5617f1d3f7a0 con 0x5617f1d39c60 2024-03-24T10:48:53.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f351a6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617f1d3f910 con 0x5617f1d30760 2024-03-24T10:48:53.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f351a6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617f1db0090 con 0x5617f1d31080 2024-03-24T10:48:53.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f351a6740 10 client.0 _ll_register_callbacks cb 0x7f6f2002c7e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:53.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f30b64640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d39c60 0x5617f1d3f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f2bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d41e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f30b64640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d39c60 0x5617f1d3f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46432/0 (socket says 172.21.15.23:46432) 2024-03-24T10:48:53.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f30b64640 1 -- 172.21.15.23:0/1165711317 learned_addr learned my addr 172.21.15.23:0/1165711317 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:53.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f2b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d31080 0x5617f1d3eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f30b64640 1 -- 172.21.15.23:0/1165711317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d31080 msgr2=0x5617f1d3eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f30b64640 1 --2- 172.21.15.23:0/1165711317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d31080 0x5617f1d3eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f30b64640 1 -- 172.21.15.23:0/1165711317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 msgr2=0x5617f1d41e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f30b64640 1 --2- 172.21.15.23:0/1165711317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d41e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f30b64640 1 -- 172.21.15.23:0/1165711317 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617f1c4d6d0 con 0x5617f1d39c60 2024-03-24T10:48:53.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f2bfff640 1 --2- 172.21.15.23:0/1165711317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d41e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:53.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f30b64640 1 --2- 172.21.15.23:0/1165711317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d39c60 0x5617f1d3f0a0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f6f24005f10 tx=0x7f6f24009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2bee30b595f485d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.306+0000 7f6f0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:53.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f0b7fe640 1 -- 172.21.15.23:0/1165711317 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2400fed0 con 0x5617f1d39c60 2024-03-24T10:48:53.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f0b7fe640 1 -- 172.21.15.23:0/1165711317 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f6f24003b80 con 0x5617f1d39c60 2024-03-24T10:48:53.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f351a6740 1 -- 172.21.15.23:0/1165711317 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f2400fd60 con 0x5617f1d39c60 2024-03-24T10:48:53.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f0b7fe640 1 -- 172.21.15.23:0/1165711317 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6f24003d40 con 0x5617f1d39c60 2024-03-24T10:48:53.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f351a6740 10 client.18066 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:53.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f351a6740 1 -- 172.21.15.23:0/1165711317 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5617f1c57970 con 0x5617f1d39c60 2024-03-24T10:48:53.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f0b7fe640 1 -- 172.21.15.23:0/1165711317 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f2400fd60 con 0x5617f1d39c60 2024-03-24T10:48:53.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f0b7fe640 1 -- 172.21.15.23:0/1165711317 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f6f2401f470 con 0x5617f1d39c60 2024-03-24T10:48:53.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f351a6740 10 client.18066 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:53.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f351a6740 10 client.18066 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:53.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f351a6740 20 client.18066 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:53.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f351a6740 1 --2- 172.21.15.23:0/1165711317 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617f1db17f0 0x5617f1db3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f351a6740 4 client.18066 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:48:53.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f351a6740 1 -- 172.21.15.23:0/1165711317 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5617f1996860 con 0x5617f1db17f0 2024-03-24T10:48:53.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.307+0000 7f6f2bfff640 1 --2- 172.21.15.23:0/1165711317 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617f1db17f0 0x5617f1db3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.308+0000 7f6f0b7fe640 10 client.18066 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:53.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.308+0000 7f6f2bfff640 1 --2- 172.21.15.23:0/1165711317 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617f1db17f0 0x5617f1db3c10 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bd509b9cf8fbe70 server_cookie=e5614d4d18b27cae in_seq=0 out_seq=0 2024-03-24T10:48:53.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f0b7fe640 1 -- 172.21.15.23:0/1165711317 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5617f1996860 con 0x5617f1db17f0 2024-03-24T10:48:53.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f0b7fe640 10 client.18066 handle_command_reply: tid=0 2024-03-24T10:48:53.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f351a6740 1 client.18066 shutdown 2024-03-24T10:48:53.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f351a6740 1 -- 172.21.15.23:0/1165711317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d39c60 msgr2=0x5617f1d3f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f351a6740 1 --2- 172.21.15.23:0/1165711317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d39c60 0x5617f1d3f0a0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f6f24005f10 tx=0x7f6f24009e20 comp rx=0 tx=0).stop 2024-03-24T10:48:53.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f351a6740 1 -- 172.21.15.23:0/1165711317 shutdown_connections 2024-03-24T10:48:53.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f351a6740 1 --2- 172.21.15.23:0/1165711317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5617f1d39c60 0x5617f1d3f0a0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f351a6740 1 --2- 172.21.15.23:0/1165711317 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5617f1db17f0 0x5617f1db3c10 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f351a6740 1 --2- 172.21.15.23:0/1165711317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617f1d31080 0x5617f1d3eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f351a6740 1 --2- 172.21.15.23:0/1165711317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617f1d30760 0x5617f1d41e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f351a6740 1 -- 172.21.15.23:0/1165711317 >> 172.21.15.23:0/1165711317 conn(0x5617f1d2ae90 msgr2=0x5617f1d29660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f0b7fe640 0 client.18066 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f351a6740 1 -- 172.21.15.23:0/1165711317 shutdown_connections 2024-03-24T10:48:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f351a6740 1 -- 172.21.15.23:0/1165711317 wait complete. 2024-03-24T10:48:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.309+0000 7f6f351a6740 20 client.18066 trim_cache size 0 max 16384 2024-03-24T10:48:53.320 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:48:53.320 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:48:53.320 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "6c4e339c-b52f-4cc5-bde7-17d23d54769d", 2024-03-24T10:48:53.320 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:48:53.320 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:48:53.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.310+0000 7f6f325c7640 1 -- 172.21.15.23:0/1536378850 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6f140a2230 msgr2=0x7f6f140a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.310+0000 7f6f325c7640 1 --2- 172.21.15.23:0/1536378850 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6f140a2230 0x7f6f140a46f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6f20000c00 tx=0x7f6f20005fd0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.310+0000 7f6f325c7640 1 -- 172.21.15.23:0/1536378850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2c100280 msgr2=0x7f6f2c197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.310+0000 7f6f325c7640 1 --2- 172.21.15.23:0/1536378850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2c100280 0x7f6f2c197e00 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c00e9f0 tx=0x7f6f1c00eeb0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.310+0000 7f6f325c7640 1 -- 172.21.15.23:0/1536378850 shutdown_connections 2024-03-24T10:48:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.310+0000 7f6f325c7640 1 --2- 172.21.15.23:0/1536378850 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f6f140a2230 0x7f6f140a46f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.310+0000 7f6f325c7640 1 --2- 172.21.15.23:0/1536378850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f2c19c6c0 0x7f6f2c19cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.310+0000 7f6f325c7640 1 --2- 172.21.15.23:0/1536378850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f2c106a40 0x7f6f2c198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.310+0000 7f6f325c7640 1 --2- 172.21.15.23:0/1536378850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f2c100280 0x7f6f2c197e00 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.310+0000 7f6f325c7640 1 -- 172.21.15.23:0/1536378850 >> 172.21.15.23:0/1536378850 conn(0x7f6f2c075870 msgr2=0x7f6f2c0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.310+0000 7f6f325c7640 1 -- 172.21.15.23:0/1536378850 shutdown_connections 2024-03-24T10:48:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.310+0000 7f6f325c7640 1 -- 172.21.15.23:0/1536378850 wait complete. 2024-03-24T10:48:53.323 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:48:53.323 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:48:53.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.427+0000 7f34d9f7a640 1 -- 172.21.15.23:0/693626730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34d4106660 msgr2=0x7f34d4106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.428+0000 7f34d9f7a640 1 --2- 172.21.15.23:0/693626730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34d4106660 0x7f34d4106a40 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009ca0 tx=0x7f34c402c0f0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.428+0000 7f34d9f7a640 1 -- 172.21.15.23:0/693626730 shutdown_connections 2024-03-24T10:48:53.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.428+0000 7f34d9f7a640 1 --2- 172.21.15.23:0/693626730 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34d4100f60 0x7f34d410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.428+0000 7f34d9f7a640 1 --2- 172.21.15.23:0/693626730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34d41005a0 0x7f34d4100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.428+0000 7f34d9f7a640 1 --2- 172.21.15.23:0/693626730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34d4106660 0x7f34d4106a40 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.428+0000 7f34d9f7a640 1 -- 172.21.15.23:0/693626730 >> 172.21.15.23:0/693626730 conn(0x7f34d4077710 msgr2=0x7f34d4077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:53.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.428+0000 7f34d9f7a640 1 -- 172.21.15.23:0/693626730 shutdown_connections 2024-03-24T10:48:53.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.428+0000 7f34d9f7a640 1 -- 172.21.15.23:0/693626730 wait complete. 2024-03-24T10:48:53.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.428+0000 7f34d9f7a640 1 Processor -- start 2024-03-24T10:48:53.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.428+0000 7f34d9f7a640 1 -- start start 2024-03-24T10:48:53.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d9f7a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34d41005a0 0x7f34d419d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d9f7a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34d4100f60 0x7f34d419de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d9f7a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34d4106660 0x7f34d4197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d9f7a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f34d4198190 con 0x7f34d4100f60 2024-03-24T10:48:53.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d9f7a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f34d4198300 con 0x7f34d41005a0 2024-03-24T10:48:53.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d9f7a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f34d41984c0 con 0x7f34d4106660 2024-03-24T10:48:53.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34d4100f60 0x7f34d419de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34d41005a0 0x7f34d419d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34d4100f60 0x7f34d419de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46446/0 (socket says 172.21.15.23:46446) 2024-03-24T10:48:53.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d2ffd640 1 -- 172.21.15.23:0/534045038 learned_addr learned my addr 172.21.15.23:0/534045038 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:53.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d3fff640 1 --2- 172.21.15.23:0/534045038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34d4106660 0x7f34d4197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d37fe640 1 -- 172.21.15.23:0/534045038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34d4106660 msgr2=0x7f34d4197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d37fe640 1 --2- 172.21.15.23:0/534045038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34d4106660 0x7f34d4197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d37fe640 1 -- 172.21.15.23:0/534045038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34d4100f60 msgr2=0x7f34d419de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d37fe640 1 --2- 172.21.15.23:0/534045038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34d4100f60 0x7f34d419de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d37fe640 1 -- 172.21.15.23:0/534045038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c4009900 con 0x7f34d41005a0 2024-03-24T10:48:53.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d2ffd640 1 --2- 172.21.15.23:0/534045038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34d4100f60 0x7f34d419de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:53.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d37fe640 1 --2- 172.21.15.23:0/534045038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34d41005a0 0x7f34d419d940 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f34c4006010 tx=0x7f34c40325f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d0ff9640 1 -- 172.21.15.23:0/534045038 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c40326a0 con 0x7f34d41005a0 2024-03-24T10:48:53.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.429+0000 7f34d9f7a640 1 -- 172.21.15.23:0/534045038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d4198740 con 0x7f34d41005a0 2024-03-24T10:48:53.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.430+0000 7f34d0ff9640 1 -- 172.21.15.23:0/534045038 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34c4003e50 con 0x7f34d41005a0 2024-03-24T10:48:53.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.430+0000 7f34d0ff9640 1 -- 172.21.15.23:0/534045038 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c40047c0 con 0x7f34d41005a0 2024-03-24T10:48:53.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.430+0000 7f34d9f7a640 1 -- 172.21.15.23:0/534045038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d41a49e0 con 0x7f34d41005a0 2024-03-24T10:48:53.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.432+0000 7f34d0ff9640 1 -- 172.21.15.23:0/534045038 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f34c403b040 con 0x7f34d41005a0 2024-03-24T10:48:53.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.432+0000 7f34dcc36740 1 Processor -- start 2024-03-24T10:48:53.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.432+0000 7f34dcc36740 1 -- start start 2024-03-24T10:48:53.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.432+0000 7f34dcc36740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b41d0 0x55b1d68b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.432+0000 7f34dcc36740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b4af0 0x55b1d68bd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.432+0000 7f34dcc36740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68bd6d0 0x55b1d68bfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.432+0000 7f34dcc36740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b1d67d8f30 con 0x55b1d68bd6d0 2024-03-24T10:48:53.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.432+0000 7f34dcc36740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b1d6822810 con 0x55b1d68b4af0 2024-03-24T10:48:53.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.432+0000 7f34dcc36740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b1d681fd40 con 0x55b1d68b41d0 2024-03-24T10:48:53.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.432+0000 7f34d37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b4af0 0x55b1d68bd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.432+0000 7f34d2ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68bd6d0 0x55b1d68bfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.432+0000 7f34d37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b4af0 0x55b1d68bd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:50092/0 (socket says 172.21.15.23:50092) 2024-03-24T10:48:53.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.432+0000 7f34d3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b41d0 0x55b1d68b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.432+0000 7f34d37fe640 1 -- 172.21.15.23:0/464525825 learned_addr learned my addr 172.21.15.23:0/464525825 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:53.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.433+0000 7f34d0ff9640 1 --2- 172.21.15.23:0/534045038 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34c00a2230 0x7f34c00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.433+0000 7f34d2ffd640 1 -- 172.21.15.23:0/464525825 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b41d0 msgr2=0x55b1d68b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.433+0000 7f34d2ffd640 1 --2- 172.21.15.23:0/464525825 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b41d0 0x55b1d68b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.433+0000 7f34d2ffd640 1 -- 172.21.15.23:0/464525825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b4af0 msgr2=0x55b1d68bd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.433+0000 7f34d2ffd640 1 --2- 172.21.15.23:0/464525825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b4af0 0x55b1d68bd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.433+0000 7f34d2ffd640 1 -- 172.21.15.23:0/464525825 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1d67dcea0 con 0x55b1d68bd6d0 2024-03-24T10:48:53.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.433+0000 7f34d37fe640 1 --2- 172.21.15.23:0/464525825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b4af0 0x55b1d68bd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:53.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.433+0000 7f34d3fff640 1 --2- 172.21.15.23:0/534045038 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34c00a2230 0x7f34c00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.433+0000 7f34d0ff9640 1 -- 172.21.15.23:0/534045038 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f34c40f0860 con 0x7f34d41005a0 2024-03-24T10:48:53.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.433+0000 7f34d2ffd640 1 --2- 172.21.15.23:0/464525825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68bd6d0 0x55b1d68bfac0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f34d41056b0 tx=0x7f34bc011950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88b7e6c1a06894b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.433+0000 7f34c9ffb640 1 -- 172.21.15.23:0/464525825 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc004030 con 0x55b1d68bd6d0 2024-03-24T10:48:53.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.433+0000 7f34d3fff640 1 --2- 172.21.15.23:0/534045038 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34c00a2230 0x7f34c00a46f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f34cc009d80 tx=0x7f34cc0046c0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.433+0000 7f34c9ffb640 1 -- 172.21.15.23:0/464525825 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34bc0035f0 con 0x55b1d68bd6d0 2024-03-24T10:48:53.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.433+0000 7f34c9ffb640 1 -- 172.21.15.23:0/464525825 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc00f870 con 0x55b1d68bd6d0 2024-03-24T10:48:53.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.434+0000 7f34dcc36740 1 -- 172.21.15.23:0/464525825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68bd6d0 msgr2=0x55b1d68bfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.434+0000 7f34dcc36740 1 --2- 172.21.15.23:0/464525825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68bd6d0 0x55b1d68bfac0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f34d41056b0 tx=0x7f34bc011950 comp rx=0 tx=0).stop 2024-03-24T10:48:53.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.434+0000 7f34dcc36740 1 -- 172.21.15.23:0/464525825 shutdown_connections 2024-03-24T10:48:53.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.434+0000 7f34dcc36740 1 --2- 172.21.15.23:0/464525825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68bd6d0 0x55b1d68bfac0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.434+0000 7f34dcc36740 1 --2- 172.21.15.23:0/464525825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b4af0 0x55b1d68bd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.434+0000 7f34dcc36740 1 --2- 172.21.15.23:0/464525825 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b41d0 0x55b1d68b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.434+0000 7f34dcc36740 1 -- 172.21.15.23:0/464525825 >> 172.21.15.23:0/464525825 conn(0x55b1d68ae900 msgr2=0x55b1d68aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:53.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.434+0000 7f34dcc36740 1 -- 172.21.15.23:0/464525825 shutdown_connections 2024-03-24T10:48:53.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.434+0000 7f34dcc36740 1 -- 172.21.15.23:0/464525825 wait complete. 2024-03-24T10:48:53.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.434+0000 7f34dcc36740 1 -- start start 2024-03-24T10:48:53.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.434+0000 7f34dcc36740 1 Processor -- start 2024-03-24T10:48:53.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34dcc36740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68b41d0 0x55b1d68ba690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34dcc36740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b4af0 0x55b1d68babd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34dcc36740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68bd6d0 0x55b1d68bb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34dcc36740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b1d681b520 con 0x55b1d68b41d0 2024-03-24T10:48:53.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34dcc36740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b1d64ee430 con 0x55b1d68bd6d0 2024-03-24T10:48:53.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34dcc36740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b1d64aae00 con 0x55b1d68b4af0 2024-03-24T10:48:53.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34dcc36740 10 client.0 _ll_register_callbacks cb 0x55b1d67b9d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:53.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34d37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68b41d0 0x55b1d68ba690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34d2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b4af0 0x55b1d68babd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68bd6d0 0x55b1d68bb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34d2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b4af0 0x55b1d68babd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39526/0 (socket says 172.21.15.23:39526) 2024-03-24T10:48:53.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34d37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68b41d0 0x55b1d68ba690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46454/0 (socket says 172.21.15.23:46454) 2024-03-24T10:48:53.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68bd6d0 0x55b1d68bb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:50094/0 (socket says 172.21.15.23:50094) 2024-03-24T10:48:53.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34d2ffd640 1 -- 172.21.15.23:0/4008473053 learned_addr learned my addr 172.21.15.23:0/4008473053 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:53.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34d2ffd640 1 -- 172.21.15.23:0/4008473053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68bd6d0 msgr2=0x55b1d68bb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34d2ffd640 1 --2- 172.21.15.23:0/4008473053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68bd6d0 0x55b1d68bb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34d2ffd640 1 -- 172.21.15.23:0/4008473053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68b41d0 msgr2=0x55b1d68ba690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34d2ffd640 1 --2- 172.21.15.23:0/4008473053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68b41d0 0x55b1d68ba690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34d2ffd640 1 -- 172.21.15.23:0/4008473053 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34bc009cd0 con 0x55b1d68b4af0 2024-03-24T10:48:53.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34d2ffd640 1 --2- 172.21.15.23:0/4008473053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b4af0 0x55b1d68babd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009620 tx=0x7f34bc0042b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=922db1c3232e8cda server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:48:53.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34b2ffd640 1 -- 172.21.15.23:0/4008473053 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc003cf0 con 0x55b1d68b4af0 2024-03-24T10:48:53.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34b2ffd640 1 -- 172.21.15.23:0/4008473053 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f34bc0044a0 con 0x55b1d68b4af0 2024-03-24T10:48:53.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.435+0000 7f34dcc36740 1 -- 172.21.15.23:0/4008473053 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b1d651a7e0 con 0x55b1d68b4af0 2024-03-24T10:48:53.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.436+0000 7f34b2ffd640 1 -- 172.21.15.23:0/4008473053 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f34bc005260 con 0x55b1d68b4af0 2024-03-24T10:48:53.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.436+0000 7f34dcc36740 10 client.27934 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:53.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.436+0000 7f34dcc36740 1 -- 172.21.15.23:0/4008473053 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b1d6819ba0 con 0x55b1d68b4af0 2024-03-24T10:48:53.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.436+0000 7f34b2ffd640 1 -- 172.21.15.23:0/4008473053 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b1d651a7e0 con 0x55b1d68b4af0 2024-03-24T10:48:53.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.436+0000 7f34b2ffd640 1 -- 172.21.15.23:0/4008473053 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f34bc00f770 con 0x55b1d68b4af0 2024-03-24T10:48:53.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.436+0000 7f34dcc36740 10 client.27934 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:53.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.436+0000 7f34dcc36740 10 client.27934 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:53.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.436+0000 7f34dcc36740 20 client.27934 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:53.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.436+0000 7f34dcc36740 1 --2- 172.21.15.23:0/4008473053 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b1d68cd970 0x55b1d68cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.436+0000 7f34dcc36740 4 client.27934 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:48:53.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.436+0000 7f34dcc36740 1 -- 172.21.15.23:0/4008473053 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1d6557720 con 0x55b1d68cd970 2024-03-24T10:48:53.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.436+0000 7f34d37fe640 1 --2- 172.21.15.23:0/4008473053 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b1d68cd970 0x55b1d68cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.437+0000 7f34b2ffd640 10 client.27934 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:53.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.437+0000 7f34d37fe640 1 --2- 172.21.15.23:0/4008473053 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b1d68cd970 0x55b1d68cfdd0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca209e399e6888f1 server_cookie=b2b999c4a358c4d1 in_seq=0 out_seq=0 2024-03-24T10:48:53.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.437+0000 7f34b2ffd640 1 -- 172.21.15.23:0/4008473053 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b1d6557720 con 0x55b1d68cd970 2024-03-24T10:48:53.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.437+0000 7f34b2ffd640 10 client.27934 handle_command_reply: tid=0 2024-03-24T10:48:53.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.437+0000 7f34dcc36740 1 client.27934 shutdown 2024-03-24T10:48:53.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.437+0000 7f34dcc36740 1 -- 172.21.15.23:0/4008473053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b4af0 msgr2=0x55b1d68babd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.437+0000 7f34dcc36740 1 --2- 172.21.15.23:0/4008473053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b4af0 0x55b1d68babd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009620 tx=0x7f34bc0042b0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.437+0000 7f34dcc36740 1 -- 172.21.15.23:0/4008473053 shutdown_connections 2024-03-24T10:48:53.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.437+0000 7f34dcc36740 1 --2- 172.21.15.23:0/4008473053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68bd6d0 0x55b1d68bb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.437+0000 7f34dcc36740 1 --2- 172.21.15.23:0/4008473053 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b1d68cd970 0x55b1d68cfdd0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.437+0000 7f34dcc36740 1 --2- 172.21.15.23:0/4008473053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b4af0 0x55b1d68babd0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.437+0000 7f34dcc36740 1 --2- 172.21.15.23:0/4008473053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68b41d0 0x55b1d68ba690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.437+0000 7f34b2ffd640 0 client.27934 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:53.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.437+0000 7f34dcc36740 1 -- 172.21.15.23:0/4008473053 >> 172.21.15.23:0/4008473053 conn(0x55b1d68ae900 msgr2=0x55b1d68adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:53.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.438+0000 7f34dcc36740 1 -- 172.21.15.23:0/4008473053 shutdown_connections 2024-03-24T10:48:53.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.438+0000 7f34dcc36740 1 -- 172.21.15.23:0/4008473053 wait complete. 2024-03-24T10:48:53.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.438+0000 7f34dcc36740 20 client.27934 trim_cache size 0 max 16384 2024-03-24T10:48:53.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34dcc36740 1 Processor -- start 2024-03-24T10:48:53.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34dcc36740 1 -- start start 2024-03-24T10:48:53.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34dcc36740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b41d0 0x55b1d68bcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34dcc36740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b4af0 0x55b1d68c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34dcc36740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68bd6d0 0x55b1d68ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34dcc36740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b1d67a0430 con 0x55b1d68bd6d0 2024-03-24T10:48:53.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34dcc36740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b1d68aa270 con 0x55b1d68b41d0 2024-03-24T10:48:53.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34dcc36740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b1d68b3550 con 0x55b1d68b4af0 2024-03-24T10:48:53.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34d3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68bd6d0 0x55b1d68ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34d3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68bd6d0 0x55b1d68ad9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46462/0 (socket says 172.21.15.23:46462) 2024-03-24T10:48:53.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34d3fff640 1 -- 172.21.15.23:0/1502706440 learned_addr learned my addr 172.21.15.23:0/1502706440 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:53.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34d37fe640 1 --2- 172.21.15.23:0/1502706440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b41d0 0x55b1d68bcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34d2ffd640 1 --2- 172.21.15.23:0/1502706440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b4af0 0x55b1d68c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34d3fff640 1 -- 172.21.15.23:0/1502706440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b4af0 msgr2=0x55b1d68c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34d3fff640 1 --2- 172.21.15.23:0/1502706440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b4af0 0x55b1d68c8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34d3fff640 1 -- 172.21.15.23:0/1502706440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b41d0 msgr2=0x55b1d68bcdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.455+0000 7f34d3fff640 1 --2- 172.21.15.23:0/1502706440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b41d0 0x55b1d68bcdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34d3fff640 1 -- 172.21.15.23:0/1502706440 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1d67b5b20 con 0x55b1d68bd6d0 2024-03-24T10:48:53.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34d37fe640 1 --2- 172.21.15.23:0/1502706440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b41d0 0x55b1d68bcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:53.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34d3fff640 1 --2- 172.21.15.23:0/1502706440 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68bd6d0 0x55b1d68ad9e0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f34cc004330 tx=0x7f34cc013070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba8d3b88324cdb81 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34c97fa640 1 -- 172.21.15.23:0/1502706440 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34cc024070 con 0x55b1d68bd6d0 2024-03-24T10:48:53.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34c97fa640 1 -- 172.21.15.23:0/1502706440 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34cc025040 con 0x55b1d68bd6d0 2024-03-24T10:48:53.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34c97fa640 1 -- 172.21.15.23:0/1502706440 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34cc01d930 con 0x55b1d68bd6d0 2024-03-24T10:48:53.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34dcc36740 1 -- 172.21.15.23:0/1502706440 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68bd6d0 msgr2=0x55b1d68ad9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34dcc36740 1 --2- 172.21.15.23:0/1502706440 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68bd6d0 0x55b1d68ad9e0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f34cc004330 tx=0x7f34cc013070 comp rx=0 tx=0).stop 2024-03-24T10:48:53.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34dcc36740 1 -- 172.21.15.23:0/1502706440 shutdown_connections 2024-03-24T10:48:53.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34dcc36740 1 --2- 172.21.15.23:0/1502706440 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68bd6d0 0x55b1d68ad9e0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34dcc36740 1 --2- 172.21.15.23:0/1502706440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68b4af0 0x55b1d68c8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34dcc36740 1 --2- 172.21.15.23:0/1502706440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b41d0 0x55b1d68bcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34dcc36740 1 -- 172.21.15.23:0/1502706440 >> 172.21.15.23:0/1502706440 conn(0x55b1d68ae900 msgr2=0x55b1d68ad2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:53.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34dcc36740 1 -- 172.21.15.23:0/1502706440 shutdown_connections 2024-03-24T10:48:53.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.456+0000 7f34dcc36740 1 -- 172.21.15.23:0/1502706440 wait complete. 2024-03-24T10:48:53.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.457+0000 7f34dcc36740 1 -- start start 2024-03-24T10:48:53.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.457+0000 7f34dcc36740 1 Processor -- start 2024-03-24T10:48:53.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.457+0000 7f34dcc36740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68b41d0 0x55b1d68b7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.457+0000 7f34dcc36740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b4af0 0x55b1d68b7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.457+0000 7f34dcc36740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68bd6d0 0x55b1d6932d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.457+0000 7f34dcc36740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b1d68b8280 con 0x55b1d68b41d0 2024-03-24T10:48:53.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.457+0000 7f34dcc36740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b1d68b83f0 con 0x55b1d68b4af0 2024-03-24T10:48:53.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.457+0000 7f34dcc36740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b1d6933250 con 0x55b1d68bd6d0 2024-03-24T10:48:53.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.457+0000 7f34dcc36740 10 client.0 _ll_register_callbacks cb 0x7f34bc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:48:53.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.457+0000 7f34d37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68b41d0 0x55b1d68b7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.457+0000 7f34d37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68b41d0 0x55b1d68b7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46470/0 (socket says 172.21.15.23:46470) 2024-03-24T10:48:53.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.457+0000 7f34d37fe640 1 -- 172.21.15.23:0/487735271 learned_addr learned my addr 172.21.15.23:0/487735271 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:48:53.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.457+0000 7f34d2ffd640 1 --2- 172.21.15.23:0/487735271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b4af0 0x55b1d68b7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.457+0000 7f34d3fff640 1 --2- 172.21.15.23:0/487735271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68bd6d0 0x55b1d6932d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34d37fe640 1 -- 172.21.15.23:0/487735271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68bd6d0 msgr2=0x55b1d6932d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34d37fe640 1 --2- 172.21.15.23:0/487735271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68bd6d0 0x55b1d6932d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34d37fe640 1 -- 172.21.15.23:0/487735271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b4af0 msgr2=0x55b1d68b7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34d37fe640 1 --2- 172.21.15.23:0/487735271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b4af0 0x55b1d68b7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34d37fe640 1 -- 172.21.15.23:0/487735271 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d4198740 con 0x55b1d68b41d0 2024-03-24T10:48:53.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34d37fe640 1 --2- 172.21.15.23:0/487735271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68b41d0 0x55b1d68b7570 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f34c40bd350 tx=0x7f34c40f2d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa16b30cc867fea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:48:53.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34b2ffd640 1 -- 172.21.15.23:0/487735271 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c40bdac0 con 0x55b1d68b41d0 2024-03-24T10:48:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34b2ffd640 1 -- 172.21.15.23:0/487735271 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8571+0+0 (secure 0 0 0) 0x7f34c40bd3c0 con 0x55b1d68b41d0 2024-03-24T10:48:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34dcc36740 1 -- 172.21.15.23:0/487735271 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34bc0035f0 con 0x55b1d68b41d0 2024-03-24T10:48:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34b2ffd640 1 -- 172.21.15.23:0/487735271 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f34c40c1510 con 0x55b1d68b41d0 2024-03-24T10:48:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34dcc36740 10 client.18084 fetch_fsmap learned FSMap version 31 2024-03-24T10:48:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34dcc36740 1 -- 172.21.15.23:0/487735271 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x7f34cc004030 con 0x55b1d68b41d0 2024-03-24T10:48:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34b2ffd640 1 -- 172.21.15.23:0/487735271 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34bc0035f0 con 0x55b1d68b41d0 2024-03-24T10:48:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34b2ffd640 1 -- 172.21.15.23:0/487735271 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f34c40f8040 con 0x55b1d68b41d0 2024-03-24T10:48:53.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34dcc36740 10 client.18084 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:48:53.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34dcc36740 10 client.18084 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:48:53.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34dcc36740 20 client.18084 populate_metadata read hostname 'smithi023' 2024-03-24T10:48:53.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34dcc36740 1 --2- 172.21.15.23:0/487735271 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b1d69347a0 0x55b1d6936b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34dcc36740 4 client.18084 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:48:53.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.458+0000 7f34dcc36740 1 -- 172.21.15.23:0/487735271 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b1d65a6890 con 0x55b1d69347a0 2024-03-24T10:48:53.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34d2ffd640 1 --2- 172.21.15.23:0/487735271 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b1d69347a0 0x55b1d6936b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34b2ffd640 10 client.18084 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34d2ffd640 1 --2- 172.21.15.23:0/487735271 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b1d69347a0 0x55b1d6936b90 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa4db6037e9d2364 server_cookie=bc39ee0f8d938d3e in_seq=0 out_seq=0 2024-03-24T10:48:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34b2ffd640 1 -- 172.21.15.23:0/487735271 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b1d65a6890 con 0x55b1d69347a0 2024-03-24T10:48:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34b2ffd640 10 client.18084 handle_command_reply: tid=0 2024-03-24T10:48:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34dcc36740 1 client.18084 shutdown 2024-03-24T10:48:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34dcc36740 1 -- 172.21.15.23:0/487735271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68b41d0 msgr2=0x55b1d68b7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34dcc36740 1 --2- 172.21.15.23:0/487735271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68b41d0 0x55b1d68b7570 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f34c40bd350 tx=0x7f34c40f2d30 comp rx=0 tx=0).stop 2024-03-24T10:48:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34dcc36740 1 -- 172.21.15.23:0/487735271 shutdown_connections 2024-03-24T10:48:53.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34dcc36740 1 --2- 172.21.15.23:0/487735271 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b1d69347a0 0x55b1d6936b90 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34dcc36740 1 --2- 172.21.15.23:0/487735271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b1d68bd6d0 0x55b1d6932d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34dcc36740 1 --2- 172.21.15.23:0/487735271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1d68b4af0 0x55b1d68b7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34b2ffd640 0 client.18084 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:48:53.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34dcc36740 1 --2- 172.21.15.23:0/487735271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1d68b41d0 0x55b1d68b7570 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34dcc36740 1 -- 172.21.15.23:0/487735271 >> 172.21.15.23:0/487735271 conn(0x55b1d68ae900 msgr2=0x55b1d68adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:53.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34dcc36740 1 -- 172.21.15.23:0/487735271 shutdown_connections 2024-03-24T10:48:53.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34dcc36740 1 -- 172.21.15.23:0/487735271 wait complete. 2024-03-24T10:48:53.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.459+0000 7f34dcc36740 20 client.18084 trim_cache size 0 max 16384 2024-03-24T10:48:53.476 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:48:53.476 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:48:53.476 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:48:53.476 INFO:teuthology.orchestra.run.smithi023.stdout: "6c4e339c-b52f-4cc5-bde7-17d23d54769d": { 2024-03-24T10:48:53.476 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:48:53.477 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "6c4e339c-b52f-4cc5-bde7-17d23d54769d", 2024-03-24T10:48:53.477 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:48:53.477 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:48:53.477 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:48:53.477 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:48:53.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.460+0000 7f34d9f7a640 1 -- 172.21.15.23:0/534045038 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34c00a2230 msgr2=0x7f34c00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.460+0000 7f34d9f7a640 1 --2- 172.21.15.23:0/534045038 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34c00a2230 0x7f34c00a46f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f34cc009d80 tx=0x7f34cc0046c0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.460+0000 7f34d9f7a640 1 -- 172.21.15.23:0/534045038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34d41005a0 msgr2=0x7f34d419d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.460+0000 7f34d9f7a640 1 --2- 172.21.15.23:0/534045038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34d41005a0 0x7f34d419d940 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f34c4006010 tx=0x7f34c40325f0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.460+0000 7f34d9f7a640 1 -- 172.21.15.23:0/534045038 shutdown_connections 2024-03-24T10:48:53.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.460+0000 7f34d9f7a640 1 --2- 172.21.15.23:0/534045038 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f34c00a2230 0x7f34c00a46f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.460+0000 7f34d9f7a640 1 --2- 172.21.15.23:0/534045038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34d4106660 0x7f34d4197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.460+0000 7f34d9f7a640 1 --2- 172.21.15.23:0/534045038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34d4100f60 0x7f34d419de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.460+0000 7f34d9f7a640 1 --2- 172.21.15.23:0/534045038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34d41005a0 0x7f34d419d940 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.460+0000 7f34d9f7a640 1 -- 172.21.15.23:0/534045038 >> 172.21.15.23:0/534045038 conn(0x7f34d4077710 msgr2=0x7f34d40ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:53.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.460+0000 7f34d9f7a640 1 -- 172.21.15.23:0/534045038 shutdown_connections 2024-03-24T10:48:53.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:48:53.460+0000 7f34d9f7a640 1 -- 172.21.15.23:0/534045038 wait complete. 2024-03-24T10:48:53.480 INFO:tasks.cephfs.filesystem:scrub status for tag:6c4e339c-b52f-4cc5-bde7-17d23d54769d - {'path': '/', 'tag': '6c4e339c-b52f-4cc5-bde7-17d23d54769d', 'options': 'recursive,force'} 2024-03-24T10:48:54.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:54 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:48:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:54 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:54 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:48:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:54 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:48:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:54 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:54 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:54.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:54 smithi116 ceph-mon[36248]: pgmap v273: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 5 op/s 2024-03-24T10:48:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:54 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:48:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:54 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:54 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:48:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:54 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:48:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:54 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:54 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:54.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:54 smithi023 ceph-mon[32466]: pgmap v273: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 5 op/s 2024-03-24T10:48:55.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:54 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:48:55.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:54 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:55.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:54 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:48:55.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:54 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:48:55.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:54 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:55.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:54 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:48:55.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:54 smithi117 ceph-mon[36368]: pgmap v273: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 5 op/s 2024-03-24T10:48:56.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:56 smithi116 ceph-mon[36248]: pgmap v274: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 5 op/s 2024-03-24T10:48:56.849 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:56 smithi117 ceph-mon[36368]: pgmap v274: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 5 op/s 2024-03-24T10:48:56.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:56 smithi023 ceph-mon[32466]: pgmap v274: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 5 op/s 2024-03-24T10:48:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:48:58 smithi023 ceph-mon[32466]: pgmap v275: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-24T10:48:59.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:48:58 smithi117 ceph-mon[36368]: pgmap v275: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-24T10:48:59.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:48:58 smithi116 ceph-mon[36248]: pgmap v275: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 KiB/s wr, 10 op/s 2024-03-24T10:49:00.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:00 smithi023 ceph-mon[32466]: pgmap v276: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 KiB/s wr, 7 op/s 2024-03-24T10:49:00.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:00 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:49:00.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:00 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:49:01.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:00 smithi117 ceph-mon[36368]: pgmap v276: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 KiB/s wr, 7 op/s 2024-03-24T10:49:01.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:00 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:49:01.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:00 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:49:01.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:00 smithi116 ceph-mon[36248]: pgmap v276: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 KiB/s wr, 7 op/s 2024-03-24T10:49:01.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:00 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:49:01.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:00 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:49:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:02 smithi023 ceph-mon[32466]: pgmap v277: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 51 KiB/s wr, 27 op/s 2024-03-24T10:49:03.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:02 smithi117 ceph-mon[36368]: pgmap v277: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 51 KiB/s wr, 27 op/s 2024-03-24T10:49:03.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:02 smithi116 ceph-mon[36248]: pgmap v277: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 51 KiB/s wr, 27 op/s 2024-03-24T10:49:04.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:04 smithi023 ceph-mon[32466]: pgmap v278: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 KiB/s wr, 24 op/s 2024-03-24T10:49:05.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:04 smithi117 ceph-mon[36368]: pgmap v278: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 KiB/s wr, 24 op/s 2024-03-24T10:49:05.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:04 smithi116 ceph-mon[36248]: pgmap v278: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 KiB/s wr, 24 op/s 2024-03-24T10:49:06.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:06 smithi023 ceph-mon[32466]: pgmap v279: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 KiB/s wr, 24 op/s 2024-03-24T10:49:07.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:06 smithi117 ceph-mon[36368]: pgmap v279: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 KiB/s wr, 24 op/s 2024-03-24T10:49:07.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:06 smithi116 ceph-mon[36248]: pgmap v279: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 KiB/s wr, 24 op/s 2024-03-24T10:49:08.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:08 smithi023 ceph-mon[32466]: pgmap v280: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 65 KiB/s wr, 27 op/s 2024-03-24T10:49:09.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:08 smithi117 ceph-mon[36368]: pgmap v280: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 65 KiB/s wr, 27 op/s 2024-03-24T10:49:09.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:08 smithi116 ceph-mon[36248]: pgmap v280: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 65 KiB/s wr, 27 op/s 2024-03-24T10:49:10.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:10 smithi023 ceph-mon[32466]: pgmap v281: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 62 KiB/s wr, 22 op/s 2024-03-24T10:49:11.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:10 smithi117 ceph-mon[36368]: pgmap v281: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 62 KiB/s wr, 22 op/s 2024-03-24T10:49:11.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:10 smithi116 ceph-mon[36248]: pgmap v281: 97 pgs: 97 active+clean; 421 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 62 KiB/s wr, 22 op/s 2024-03-24T10:49:11.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:11 smithi023 ceph-mon[32466]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T10:49:12.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:11 smithi117 ceph-mon[36368]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T10:49:12.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:11 smithi116 ceph-mon[36248]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T10:49:12.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:12 smithi023 ceph-mon[32466]: pgmap v283: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 7 op/s 2024-03-24T10:49:13.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:12 smithi117 ceph-mon[36368]: pgmap v283: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 7 op/s 2024-03-24T10:49:13.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:12 smithi116 ceph-mon[36248]: pgmap v283: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 7 op/s 2024-03-24T10:49:14.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:14 smithi023 ceph-mon[32466]: pgmap v284: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 7 op/s 2024-03-24T10:49:15.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:14 smithi117 ceph-mon[36368]: pgmap v284: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 7 op/s 2024-03-24T10:49:15.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:14 smithi116 ceph-mon[36248]: pgmap v284: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 7 op/s 2024-03-24T10:49:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:16 smithi023 ceph-mon[32466]: pgmap v285: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 7 op/s 2024-03-24T10:49:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:16 smithi023 ceph-mon[32466]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T10:49:17.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:16 smithi117 ceph-mon[36368]: pgmap v285: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 7 op/s 2024-03-24T10:49:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:16 smithi117 ceph-mon[36368]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T10:49:17.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:16 smithi116 ceph-mon[36248]: pgmap v285: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 7 op/s 2024-03-24T10:49:17.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:16 smithi116 ceph-mon[36248]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T10:49:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:18 smithi023 ceph-mon[32466]: pgmap v287: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 8 op/s 2024-03-24T10:49:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:18 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:49:19.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:18 smithi117 ceph-mon[36368]: pgmap v287: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 8 op/s 2024-03-24T10:49:19.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:18 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:49:19.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:18 smithi116 ceph-mon[36248]: pgmap v287: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 8 op/s 2024-03-24T10:49:19.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:18 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:49:19.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:19 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:49:19.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:19 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:49:19.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:19 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:49:20.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:19 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:49:20.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:19 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:49:20.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:19 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:49:20.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:19 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:49:20.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:19 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:49:20.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:19 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:49:21.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:21 smithi023 ceph-mon[32466]: pgmap v288: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 6.8 KiB/s wr, 8 op/s 2024-03-24T10:49:21.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:21 smithi117 ceph-mon[36368]: pgmap v288: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 6.8 KiB/s wr, 8 op/s 2024-03-24T10:49:21.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:21 smithi116 ceph-mon[36248]: pgmap v288: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 6.8 KiB/s wr, 8 op/s 2024-03-24T10:49:23.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:23 smithi023 ceph-mon[32466]: pgmap v289: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-03-24T10:49:23.482 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:49:23.482 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:49:23.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:23 smithi117 ceph-mon[36368]: pgmap v289: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-03-24T10:49:23.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:23 smithi116 ceph-mon[36248]: pgmap v289: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-03-24T10:49:23.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.598+0000 7fda93ec0640 1 -- 172.21.15.23:0/1432250978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda8c100630 msgr2=0x7fda8c100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.598+0000 7fda93ec0640 1 --2- 172.21.15.23:0/1432250978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda8c100630 0x7fda8c100ab0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009930 tx=0x7fda7c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:49:23.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.598+0000 7fda93ec0640 1 -- 172.21.15.23:0/1432250978 shutdown_connections 2024-03-24T10:49:23.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.598+0000 7fda93ec0640 1 --2- 172.21.15.23:0/1432250978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda8c100ff0 0x7fda8c10d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.598+0000 7fda93ec0640 1 --2- 172.21.15.23:0/1432250978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda8c100630 0x7fda8c100ab0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.598+0000 7fda93ec0640 1 --2- 172.21.15.23:0/1432250978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda8c106640 0x7fda8c106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.598+0000 7fda93ec0640 1 -- 172.21.15.23:0/1432250978 >> 172.21.15.23:0/1432250978 conn(0x7fda8c077770 msgr2=0x7fda8c077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:23.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.598+0000 7fda93ec0640 1 -- 172.21.15.23:0/1432250978 shutdown_connections 2024-03-24T10:49:23.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.599+0000 7fda93ec0640 1 -- 172.21.15.23:0/1432250978 wait complete. 2024-03-24T10:49:23.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.599+0000 7fda93ec0640 1 Processor -- start 2024-03-24T10:49:23.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.599+0000 7fda93ec0640 1 -- start start 2024-03-24T10:49:23.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.599+0000 7fda93ec0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda8c100630 0x7fda8c197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.599+0000 7fda93ec0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda8c100ff0 0x7fda8c198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.599+0000 7fda93ec0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda8c106640 0x7fda8c19c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.599+0000 7fda93ec0640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c19cca0 con 0x7fda8c100630 2024-03-24T10:49:23.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.599+0000 7fda93ec0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c19ce10 con 0x7fda8c106640 2024-03-24T10:49:23.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.599+0000 7fda93ec0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c19cfd0 con 0x7fda8c100ff0 2024-03-24T10:49:23.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.599+0000 7fda91c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda8c100630 0x7fda8c197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.599+0000 7fda9145b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda8c100ff0 0x7fda8c198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.599+0000 7fda91c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda8c100630 0x7fda8c197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45908/0 (socket says 172.21.15.23:45908) 2024-03-24T10:49:23.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.599+0000 7fda91c5c640 1 -- 172.21.15.23:0/873175864 learned_addr learned my addr 172.21.15.23:0/873175864 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:23.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.599+0000 7fda9245d640 1 --2- 172.21.15.23:0/873175864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda8c106640 0x7fda8c19c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.600+0000 7fda91c5c640 1 -- 172.21.15.23:0/873175864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda8c100ff0 msgr2=0x7fda8c198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.600+0000 7fda91c5c640 1 --2- 172.21.15.23:0/873175864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda8c100ff0 0x7fda8c198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.600+0000 7fda91c5c640 1 -- 172.21.15.23:0/873175864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda8c106640 msgr2=0x7fda8c19c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.600+0000 7fda91c5c640 1 --2- 172.21.15.23:0/873175864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda8c106640 0x7fda8c19c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.600+0000 7fda91c5c640 1 -- 172.21.15.23:0/873175864 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda7c009560 con 0x7fda8c100630 2024-03-24T10:49:23.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.600+0000 7fda9145b640 1 --2- 172.21.15.23:0/873175864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda8c100ff0 0x7fda8c198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:23.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.600+0000 7fda91c5c640 1 --2- 172.21.15.23:0/873175864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda8c100630 0x7fda8c197de0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fda8800e9f0 tx=0x7fda8800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:23.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.600+0000 7fda82ffd640 1 -- 172.21.15.23:0/873175864 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88009700 con 0x7fda8c100630 2024-03-24T10:49:23.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.600+0000 7fda82ffd640 1 -- 172.21.15.23:0/873175864 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda8800cd80 con 0x7fda8c100630 2024-03-24T10:49:23.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.600+0000 7fda93ec0640 1 -- 172.21.15.23:0/873175864 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda8c19d2b0 con 0x7fda8c100630 2024-03-24T10:49:23.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.600+0000 7fda82ffd640 1 -- 172.21.15.23:0/873175864 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88018920 con 0x7fda8c100630 2024-03-24T10:49:23.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.600+0000 7fda93ec0640 1 -- 172.21.15.23:0/873175864 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda8c1a4b30 con 0x7fda8c100630 2024-03-24T10:49:23.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.602+0000 7fda82ffd640 1 -- 172.21.15.23:0/873175864 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fda88018a80 con 0x7fda8c100630 2024-03-24T10:49:23.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda96b78740 1 Processor -- start 2024-03-24T10:49:23.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda96b78740 1 -- start start 2024-03-24T10:49:23.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda82ffd640 1 --2- 172.21.15.23:0/873175864 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fda780a2230 0x7fda780a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda96b78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de21d0 0x564892de25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda96b78740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de2af0 0x564892deb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda96b78740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 0x564892dedac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda96b78740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x564892d06f30 con 0x564892de2af0 2024-03-24T10:49:23.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda96b78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564892d50810 con 0x564892de21d0 2024-03-24T10:49:23.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda96b78740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564892d4dd40 con 0x564892deb6d0 2024-03-24T10:49:23.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda9245d640 1 --2- 172.21.15.23:0/873175864 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fda780a2230 0x7fda780a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda82ffd640 1 -- 172.21.15.23:0/873175864 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fda88014070 con 0x7fda8c100630 2024-03-24T10:49:23.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda9145b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de2af0 0x564892deb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda9145b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de2af0 0x564892deb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45912/0 (socket says 172.21.15.23:45912) 2024-03-24T10:49:23.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda9145b640 1 -- 172.21.15.23:0/3785636554 learned_addr learned my addr 172.21.15.23:0/3785636554 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:23.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda9245d640 1 --2- 172.21.15.23:0/3785636554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 0x564892dedac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.603+0000 7fda91c5c640 1 --2- 172.21.15.23:0/3785636554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de21d0 0x564892de25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.604+0000 7fda91c5c640 1 -- 172.21.15.23:0/3785636554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 msgr2=0x564892dedac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:23.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.604+0000 7fda91c5c640 1 --2- 172.21.15.23:0/3785636554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 0x564892dedac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.604+0000 7fda91c5c640 1 -- 172.21.15.23:0/3785636554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de2af0 msgr2=0x564892deb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:23.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.604+0000 7fda91c5c640 1 --2- 172.21.15.23:0/3785636554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de2af0 0x564892deb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.604+0000 7fda91c5c640 1 -- 172.21.15.23:0/3785636554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564892d0aea0 con 0x564892de21d0 2024-03-24T10:49:23.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.604+0000 7fda9145b640 1 --2- 172.21.15.23:0/3785636554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de2af0 0x564892deb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:23.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.604+0000 7fda9245d640 1 --2- 172.21.15.23:0/873175864 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fda780a2230 0x7fda780a46f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fda84005fd0 tx=0x7fda84009e00 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:23.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.604+0000 7fda91c5c640 1 --2- 172.21.15.23:0/3785636554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de21d0 0x564892de25b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fda880cbad0 tx=0x7fda8809aa20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a0780c15d10a5f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:23.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.604+0000 7fda77fff640 1 -- 172.21.15.23:0/3785636554 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8809eec0 con 0x564892de21d0 2024-03-24T10:49:23.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.604+0000 7fda77fff640 1 -- 172.21.15.23:0/3785636554 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda880a3510 con 0x564892de21d0 2024-03-24T10:49:23.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.604+0000 7fda77fff640 1 -- 172.21.15.23:0/3785636554 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88097490 con 0x564892de21d0 2024-03-24T10:49:23.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.605+0000 7fda96b78740 1 -- 172.21.15.23:0/3785636554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de21d0 msgr2=0x564892de25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.605+0000 7fda96b78740 1 --2- 172.21.15.23:0/3785636554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de21d0 0x564892de25b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fda880cbad0 tx=0x7fda8809aa20 comp rx=0 tx=0).stop 2024-03-24T10:49:23.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.605+0000 7fda96b78740 1 -- 172.21.15.23:0/3785636554 shutdown_connections 2024-03-24T10:49:23.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.605+0000 7fda96b78740 1 --2- 172.21.15.23:0/3785636554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 0x564892dedac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.605+0000 7fda96b78740 1 --2- 172.21.15.23:0/3785636554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de2af0 0x564892deb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.605+0000 7fda96b78740 1 --2- 172.21.15.23:0/3785636554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de21d0 0x564892de25b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.605+0000 7fda96b78740 1 -- 172.21.15.23:0/3785636554 >> 172.21.15.23:0/3785636554 conn(0x564892ddc900 msgr2=0x564892ddcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:23.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.605+0000 7fda96b78740 1 -- 172.21.15.23:0/3785636554 shutdown_connections 2024-03-24T10:49:23.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.605+0000 7fda96b78740 1 -- 172.21.15.23:0/3785636554 wait complete. 2024-03-24T10:49:23.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.605+0000 7fda96b78740 1 -- start start 2024-03-24T10:49:23.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.605+0000 7fda96b78740 1 Processor -- start 2024-03-24T10:49:23.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda96b78740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892de21d0 0x564892de86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda96b78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de2af0 0x564892de8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda96b78740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892deb6d0 0x564892de9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda96b78740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x564892d49520 con 0x564892deb6d0 2024-03-24T10:49:23.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda96b78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564892a1c430 con 0x564892de2af0 2024-03-24T10:49:23.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda96b78740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5648929d8e00 con 0x564892de21d0 2024-03-24T10:49:23.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda96b78740 10 client.0 _ll_register_callbacks cb 0x564892ce7d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:23.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda9245d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892deb6d0 0x564892de9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda91c5c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892de21d0 0x564892de86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda9145b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de2af0 0x564892de8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda9245d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892deb6d0 0x564892de9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45916/0 (socket says 172.21.15.23:45916) 2024-03-24T10:49:23.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda9245d640 1 -- 172.21.15.23:0/3360454078 learned_addr learned my addr 172.21.15.23:0/3360454078 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:23.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda9245d640 1 -- 172.21.15.23:0/3360454078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892de21d0 msgr2=0x564892de86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:23.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda9245d640 1 --2- 172.21.15.23:0/3360454078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892de21d0 0x564892de86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda9245d640 1 -- 172.21.15.23:0/3360454078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de2af0 msgr2=0x564892de8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:23.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda9245d640 1 --2- 172.21.15.23:0/3360454078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de2af0 0x564892de8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda9245d640 1 -- 172.21.15.23:0/3360454078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda8c1a4b30 con 0x564892deb6d0 2024-03-24T10:49:23.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda91c5c640 1 --2- 172.21.15.23:0/3360454078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892de21d0 0x564892de86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:23.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda9245d640 1 --2- 172.21.15.23:0/3360454078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892deb6d0 0x564892de9150 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fda84003bf0 tx=0x7fda8401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516ca66aa291f467 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:23.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.606+0000 7fda757fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:23.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda757fa640 1 -- 172.21.15.23:0/3360454078 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda84004670 con 0x564892deb6d0 2024-03-24T10:49:23.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda757fa640 1 -- 172.21.15.23:0/3360454078 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fda84025030 con 0x564892deb6d0 2024-03-24T10:49:23.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda96b78740 1 -- 172.21.15.23:0/3360454078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564892a487e0 con 0x564892deb6d0 2024-03-24T10:49:23.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda757fa640 1 -- 172.21.15.23:0/3360454078 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fda840047d0 con 0x564892deb6d0 2024-03-24T10:49:23.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda96b78740 10 client.18096 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:23.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda96b78740 1 -- 172.21.15.23:0/3360454078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564892d47ba0 con 0x564892deb6d0 2024-03-24T10:49:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda757fa640 1 -- 172.21.15.23:0/3360454078 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564892a487e0 con 0x564892deb6d0 2024-03-24T10:49:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda757fa640 1 -- 172.21.15.23:0/3360454078 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fda84021530 con 0x564892deb6d0 2024-03-24T10:49:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda96b78740 10 client.18096 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda96b78740 10 client.18096 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda96b78740 20 client.18096 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda96b78740 1 --2- 172.21.15.23:0/3360454078 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564892dfbae0 0x564892dfded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda96b78740 4 client.18096 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:49:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda96b78740 1 -- 172.21.15.23:0/3360454078 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564892a85720 con 0x564892dfbae0 2024-03-24T10:49:23.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.607+0000 7fda91c5c640 1 --2- 172.21.15.23:0/3360454078 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564892dfbae0 0x564892dfded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.608+0000 7fda757fa640 10 client.18096 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:23.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.608+0000 7fda91c5c640 1 --2- 172.21.15.23:0/3360454078 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564892dfbae0 0x564892dfded0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dc2586c8d6db6b0 server_cookie=c739f4103ea81978 in_seq=0 out_seq=0 2024-03-24T10:49:23.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda757fa640 1 -- 172.21.15.23:0/3360454078 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564892a85720 con 0x564892dfbae0 2024-03-24T10:49:23.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda757fa640 10 client.18096 handle_command_reply: tid=0 2024-03-24T10:49:23.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda96b78740 1 client.18096 shutdown 2024-03-24T10:49:23.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda96b78740 1 -- 172.21.15.23:0/3360454078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892deb6d0 msgr2=0x564892de9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda96b78740 1 --2- 172.21.15.23:0/3360454078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892deb6d0 0x564892de9150 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fda84003bf0 tx=0x7fda8401d000 comp rx=0 tx=0).stop 2024-03-24T10:49:23.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda96b78740 1 -- 172.21.15.23:0/3360454078 shutdown_connections 2024-03-24T10:49:23.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda96b78740 1 --2- 172.21.15.23:0/3360454078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892deb6d0 0x564892de9150 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda96b78740 1 --2- 172.21.15.23:0/3360454078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de2af0 0x564892de8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda96b78740 1 --2- 172.21.15.23:0/3360454078 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564892dfbae0 0x564892dfded0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda96b78740 1 --2- 172.21.15.23:0/3360454078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892de21d0 0x564892de86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda96b78740 1 -- 172.21.15.23:0/3360454078 >> 172.21.15.23:0/3360454078 conn(0x564892ddc900 msgr2=0x564892ddbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda757fa640 0 client.18096 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda96b78740 1 -- 172.21.15.23:0/3360454078 shutdown_connections 2024-03-24T10:49:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda96b78740 1 -- 172.21.15.23:0/3360454078 wait complete. 2024-03-24T10:49:23.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.609+0000 7fda96b78740 20 client.18096 trim_cache size 0 max 16384 2024-03-24T10:49:23.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda96b78740 1 Processor -- start 2024-03-24T10:49:23.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda96b78740 1 -- start start 2024-03-24T10:49:23.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda96b78740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de21d0 0x564892df6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda96b78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de2af0 0x564892e037c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda96b78740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 0x564892de8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda96b78740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x564892cce430 con 0x564892de21d0 2024-03-24T10:49:23.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda96b78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564892de4300 con 0x564892de2af0 2024-03-24T10:49:23.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda96b78740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564892de3570 con 0x564892deb6d0 2024-03-24T10:49:23.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda91c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de21d0 0x564892df6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda91c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de21d0 0x564892df6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45926/0 (socket says 172.21.15.23:45926) 2024-03-24T10:49:23.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda91c5c640 1 -- 172.21.15.23:0/696601448 learned_addr learned my addr 172.21.15.23:0/696601448 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:23.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda9145b640 1 --2- 172.21.15.23:0/696601448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de2af0 0x564892e037c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda9245d640 1 --2- 172.21.15.23:0/696601448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 0x564892de8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda91c5c640 1 -- 172.21.15.23:0/696601448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 msgr2=0x564892de8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:23.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda91c5c640 1 --2- 172.21.15.23:0/696601448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 0x564892de8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda91c5c640 1 -- 172.21.15.23:0/696601448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de2af0 msgr2=0x564892e037c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:23.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda91c5c640 1 --2- 172.21.15.23:0/696601448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de2af0 0x564892e037c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda91c5c640 1 -- 172.21.15.23:0/696601448 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564892ce3b20 con 0x564892de21d0 2024-03-24T10:49:23.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda9145b640 1 --2- 172.21.15.23:0/696601448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de2af0 0x564892e037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:49:23.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda91c5c640 1 --2- 172.21.15.23:0/696601448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de21d0 0x564892df6f60 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fda8809adb0 tx=0x7fda880056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92376fd92244c140 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:23.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda777fe640 1 -- 172.21.15.23:0/696601448 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8809ef40 con 0x564892de21d0 2024-03-24T10:49:23.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda777fe640 1 -- 172.21.15.23:0/696601448 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda880b7070 con 0x564892de21d0 2024-03-24T10:49:23.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.627+0000 7fda777fe640 1 -- 172.21.15.23:0/696601448 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88082d10 con 0x564892de21d0 2024-03-24T10:49:23.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.628+0000 7fda96b78740 1 -- 172.21.15.23:0/696601448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de21d0 msgr2=0x564892df6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.628+0000 7fda96b78740 1 --2- 172.21.15.23:0/696601448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de21d0 0x564892df6f60 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fda8809adb0 tx=0x7fda880056d0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.628+0000 7fda96b78740 1 -- 172.21.15.23:0/696601448 shutdown_connections 2024-03-24T10:49:23.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.628+0000 7fda96b78740 1 --2- 172.21.15.23:0/696601448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 0x564892de8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.628+0000 7fda96b78740 1 --2- 172.21.15.23:0/696601448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de2af0 0x564892e037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.628+0000 7fda96b78740 1 --2- 172.21.15.23:0/696601448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de21d0 0x564892df6f60 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.628+0000 7fda96b78740 1 -- 172.21.15.23:0/696601448 >> 172.21.15.23:0/696601448 conn(0x564892ddc900 msgr2=0x564892ddbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:23.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.628+0000 7fda96b78740 1 -- 172.21.15.23:0/696601448 shutdown_connections 2024-03-24T10:49:23.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.628+0000 7fda96b78740 1 -- 172.21.15.23:0/696601448 wait complete. 2024-03-24T10:49:23.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.628+0000 7fda96b78740 1 -- start start 2024-03-24T10:49:23.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.628+0000 7fda96b78740 1 Processor -- start 2024-03-24T10:49:23.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda96b78740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de21d0 0x564892df3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda96b78740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de2af0 0x564892deed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda96b78740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 0x564892def440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda96b78740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x564892defb10 con 0x564892de2af0 2024-03-24T10:49:23.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda96b78740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564892e60df0 con 0x564892de21d0 2024-03-24T10:49:23.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda96b78740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564892e60f30 con 0x564892deb6d0 2024-03-24T10:49:23.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda96b78740 10 client.0 _ll_register_callbacks cb 0x7fda840136c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:23.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda9145b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de2af0 0x564892deed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda9145b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de2af0 0x564892deed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45928/0 (socket says 172.21.15.23:45928) 2024-03-24T10:49:23.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda9145b640 1 -- 172.21.15.23:0/3821533222 learned_addr learned my addr 172.21.15.23:0/3821533222 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:23.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda91c5c640 1 --2- 172.21.15.23:0/3821533222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de21d0 0x564892df3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda9245d640 1 --2- 172.21.15.23:0/3821533222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 0x564892def440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda91c5c640 1 -- 172.21.15.23:0/3821533222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 msgr2=0x564892def440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:23.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda91c5c640 1 --2- 172.21.15.23:0/3821533222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 0x564892def440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda91c5c640 1 -- 172.21.15.23:0/3821533222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de2af0 msgr2=0x564892deed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda91c5c640 1 --2- 172.21.15.23:0/3821533222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de2af0 0x564892deed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda91c5c640 1 -- 172.21.15.23:0/3821533222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564892d0aea0 con 0x564892de21d0 2024-03-24T10:49:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.629+0000 7fda9145b640 1 --2- 172.21.15.23:0/3821533222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de2af0 0x564892deed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda91c5c640 1 --2- 172.21.15.23:0/3821533222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de21d0 0x564892df3990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fda8809ab20 tx=0x7fda88010d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdc01e39c903ce4e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda757fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:49:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda757fa640 1 -- 172.21.15.23:0/3821533222 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda880970c0 con 0x564892de21d0 2024-03-24T10:49:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda96b78740 1 -- 172.21.15.23:0/3821533222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda880a3510 con 0x564892de21d0 2024-03-24T10:49:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda757fa640 1 -- 172.21.15.23:0/3821533222 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fda88099120 con 0x564892de21d0 2024-03-24T10:49:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda757fa640 1 -- 172.21.15.23:0/3821533222 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fda88082d10 con 0x564892de21d0 2024-03-24T10:49:23.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda96b78740 10 client.27989 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:23.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda96b78740 1 -- 172.21.15.23:0/3821533222 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564892d09700 con 0x564892de21d0 2024-03-24T10:49:23.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda757fa640 1 -- 172.21.15.23:0/3821533222 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda880a3510 con 0x564892de21d0 2024-03-24T10:49:23.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda757fa640 1 -- 172.21.15.23:0/3821533222 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fda88096b70 con 0x564892de21d0 2024-03-24T10:49:23.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda96b78740 10 client.27989 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:23.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda96b78740 10 client.27989 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:23.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda96b78740 20 client.27989 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:23.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda96b78740 1 --2- 172.21.15.23:0/3821533222 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564892e62220 0x564892e64640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda96b78740 4 client.27989 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:49:23.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda96b78740 1 -- 172.21.15.23:0/3821533222 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564892ad4890 con 0x564892e62220 2024-03-24T10:49:23.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.630+0000 7fda9145b640 1 --2- 172.21.15.23:0/3821533222 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564892e62220 0x564892e64640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda757fa640 10 client.27989 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:23.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda9145b640 1 --2- 172.21.15.23:0/3821533222 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564892e62220 0x564892e64640 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b6b5fa67efc8977 server_cookie=81aa5abc27aebee4 in_seq=0 out_seq=0 2024-03-24T10:49:23.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda757fa640 1 -- 172.21.15.23:0/3821533222 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564892ad4890 con 0x564892e62220 2024-03-24T10:49:23.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda757fa640 10 client.27989 handle_command_reply: tid=0 2024-03-24T10:49:23.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda96b78740 1 client.27989 shutdown 2024-03-24T10:49:23.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda96b78740 1 -- 172.21.15.23:0/3821533222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de21d0 msgr2=0x564892df3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda96b78740 1 --2- 172.21.15.23:0/3821533222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de21d0 0x564892df3990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fda8809ab20 tx=0x7fda88010d20 comp rx=0 tx=0).stop 2024-03-24T10:49:23.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda96b78740 1 -- 172.21.15.23:0/3821533222 shutdown_connections 2024-03-24T10:49:23.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda96b78740 1 --2- 172.21.15.23:0/3821533222 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x564892e62220 0x564892e64640 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda96b78740 1 --2- 172.21.15.23:0/3821533222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564892deb6d0 0x564892def440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda96b78740 1 --2- 172.21.15.23:0/3821533222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564892de2af0 0x564892deed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda96b78740 1 --2- 172.21.15.23:0/3821533222 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564892de21d0 0x564892df3990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda757fa640 0 client.27989 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:23.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda96b78740 1 -- 172.21.15.23:0/3821533222 >> 172.21.15.23:0/3821533222 conn(0x564892ddc900 msgr2=0x564892dddb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:23.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda96b78740 1 -- 172.21.15.23:0/3821533222 shutdown_connections 2024-03-24T10:49:23.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda96b78740 1 -- 172.21.15.23:0/3821533222 wait complete. 2024-03-24T10:49:23.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.631+0000 7fda96b78740 20 client.27989 trim_cache size 0 max 16384 2024-03-24T10:49:23.642 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:49:23.642 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:49:23.642 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:49:23.642 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:49:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.632+0000 7fda93ec0640 1 -- 172.21.15.23:0/873175864 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fda780a2230 msgr2=0x7fda780a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.632+0000 7fda93ec0640 1 --2- 172.21.15.23:0/873175864 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fda780a2230 0x7fda780a46f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fda84005fd0 tx=0x7fda84009e00 comp rx=0 tx=0).stop 2024-03-24T10:49:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.632+0000 7fda93ec0640 1 -- 172.21.15.23:0/873175864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda8c100630 msgr2=0x7fda8c197de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.632+0000 7fda93ec0640 1 --2- 172.21.15.23:0/873175864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda8c100630 0x7fda8c197de0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fda8800e9f0 tx=0x7fda8800eeb0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.632+0000 7fda93ec0640 1 -- 172.21.15.23:0/873175864 shutdown_connections 2024-03-24T10:49:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.632+0000 7fda93ec0640 1 --2- 172.21.15.23:0/873175864 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fda780a2230 0x7fda780a46f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.632+0000 7fda93ec0640 1 --2- 172.21.15.23:0/873175864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda8c106640 0x7fda8c19c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.632+0000 7fda93ec0640 1 --2- 172.21.15.23:0/873175864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda8c100ff0 0x7fda8c198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.632+0000 7fda93ec0640 1 --2- 172.21.15.23:0/873175864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda8c100630 0x7fda8c197de0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.632+0000 7fda93ec0640 1 -- 172.21.15.23:0/873175864 >> 172.21.15.23:0/873175864 conn(0x7fda8c077770 msgr2=0x7fda8c069fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:23.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.632+0000 7fda93ec0640 1 -- 172.21.15.23:0/873175864 shutdown_connections 2024-03-24T10:49:23.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.632+0000 7fda93ec0640 1 -- 172.21.15.23:0/873175864 wait complete. 2024-03-24T10:49:23.645 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:49:23.646 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:23.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.748+0000 7fd8454c3640 1 -- 172.21.15.23:0/2724183957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8401076d0 msgr2=0x7fd840107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.748+0000 7fd8454c3640 1 --2- 172.21.15.23:0/2724183957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8401076d0 0x7fd840107ab0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd838009890 tx=0x7fd83802bb70 comp rx=0 tx=0).stop 2024-03-24T10:49:23.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.748+0000 7fd8454c3640 1 -- 172.21.15.23:0/2724183957 shutdown_connections 2024-03-24T10:49:23.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.748+0000 7fd8454c3640 1 --2- 172.21.15.23:0/2724183957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd840100590 0x7fd84010e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.748+0000 7fd8454c3640 1 --2- 172.21.15.23:0/2724183957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8400ffc10 0x7fd840100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.748+0000 7fd8454c3640 1 --2- 172.21.15.23:0/2724183957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8401076d0 0x7fd840107ab0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.748+0000 7fd8454c3640 1 -- 172.21.15.23:0/2724183957 >> 172.21.15.23:0/2724183957 conn(0x7fd840076ba0 msgr2=0x7fd8400fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:23.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.748+0000 7fd8454c3640 1 -- 172.21.15.23:0/2724183957 shutdown_connections 2024-03-24T10:49:23.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.748+0000 7fd8454c3640 1 -- 172.21.15.23:0/2724183957 wait complete. 2024-03-24T10:49:23.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.748+0000 7fd8454c3640 1 Processor -- start 2024-03-24T10:49:23.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd8454c3640 1 -- start start 2024-03-24T10:49:23.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd8454c3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8400ffc10 0x7fd840197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd8454c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd840100590 0x7fd840198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd8454c3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8401076d0 0x7fd84019c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd8454c3640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd84019cbf0 con 0x7fd8401076d0 2024-03-24T10:49:23.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd8454c3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd84019cd60 con 0x7fd840100590 2024-03-24T10:49:23.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd8454c3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd84019cf20 con 0x7fd8400ffc10 2024-03-24T10:49:23.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd83effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8400ffc10 0x7fd840197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd83f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8401076d0 0x7fd84019c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd83effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8400ffc10 0x7fd840197d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:54496/0 (socket says 172.21.15.23:54496) 2024-03-24T10:49:23.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd83f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8401076d0 0x7fd84019c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45954/0 (socket says 172.21.15.23:45954) 2024-03-24T10:49:23.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd83effd640 1 -- 172.21.15.23:0/4228504907 learned_addr learned my addr 172.21.15.23:0/4228504907 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:23.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd83e7fc640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd840100590 0x7fd840198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd83f7fe640 1 -- 172.21.15.23:0/4228504907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8400ffc10 msgr2=0x7fd840197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd83f7fe640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8400ffc10 0x7fd840197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd83f7fe640 1 -- 172.21.15.23:0/4228504907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd840100590 msgr2=0x7fd840198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd83f7fe640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd840100590 0x7fd840198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd83f7fe640 1 -- 172.21.15.23:0/4228504907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd838009490 con 0x7fd8401076d0 2024-03-24T10:49:23.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd83effd640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8400ffc10 0x7fd840197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:23.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.749+0000 7fd83e7fc640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd840100590 0x7fd840198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:23.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.750+0000 7fd83f7fe640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8401076d0 0x7fd84019c5f0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fd830009fd0 tx=0x7fd83000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:23.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.750+0000 7fd827fff640 1 -- 172.21.15.23:0/4228504907 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83000cd60 con 0x7fd8401076d0 2024-03-24T10:49:23.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.750+0000 7fd8454c3640 1 -- 172.21.15.23:0/4228504907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd84019d200 con 0x7fd8401076d0 2024-03-24T10:49:23.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.750+0000 7fd827fff640 1 -- 172.21.15.23:0/4228504907 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd830005020 con 0x7fd8401076d0 2024-03-24T10:49:23.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.750+0000 7fd827fff640 1 -- 172.21.15.23:0/4228504907 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd830010690 con 0x7fd8401076d0 2024-03-24T10:49:23.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.750+0000 7fd8454c3640 1 -- 172.21.15.23:0/4228504907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8401a4a80 con 0x7fd8401076d0 2024-03-24T10:49:23.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.751+0000 7fd8454c3640 1 -- 172.21.15.23:0/4228504907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd840068950 con 0x7fd8401076d0 2024-03-24T10:49:23.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.752+0000 7fd827fff640 1 -- 172.21.15.23:0/4228504907 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fd830005190 con 0x7fd8401076d0 2024-03-24T10:49:23.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.753+0000 7fd827fff640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd8280a2230 0x7fd8280a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.753+0000 7fd83effd640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd8280a2230 0x7fd8280a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.753+0000 7fd827fff640 1 -- 172.21.15.23:0/4228504907 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fd830014070 con 0x7fd8401076d0 2024-03-24T10:49:23.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.753+0000 7fd83effd640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd8280a2230 0x7fd8280a46f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd83802c180 tx=0x7fd83802e040 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:23.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.757+0000 7fd827fff640 1 -- 172.21.15.23:0/4228504907 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd83009aef0 con 0x7fd8401076d0 2024-03-24T10:49:23.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.977+0000 7fd8454c3640 1 -- 172.21.15.23:0/4228504907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd840199080 con 0x7fd8401076d0 2024-03-24T10:49:23.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.977+0000 7fd827fff640 1 -- 172.21.15.23:0/4228504907 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11869 (secure 0 0 0) 0x7fd8300b2e70 con 0x7fd8401076d0 2024-03-24T10:49:23.980 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:49:23.980 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:47:17.474711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27526":{"gid":27526,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/2017557046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":2017557046},{"type":"v1","addr":"172.21.15.117:6837","nonce":2017557046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:49:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 31 2024-03-24T10:49:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.979+0000 7fd8454c3640 1 -- 172.21.15.23:0/4228504907 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd8280a2230 msgr2=0x7fd8280a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.979+0000 7fd8454c3640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd8280a2230 0x7fd8280a46f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd83802c180 tx=0x7fd83802e040 comp rx=0 tx=0).stop 2024-03-24T10:49:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.979+0000 7fd8454c3640 1 -- 172.21.15.23:0/4228504907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8401076d0 msgr2=0x7fd84019c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.979+0000 7fd8454c3640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8401076d0 0x7fd84019c5f0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fd830009fd0 tx=0x7fd83000ee50 comp rx=0 tx=0).stop 2024-03-24T10:49:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.979+0000 7fd8454c3640 1 -- 172.21.15.23:0/4228504907 shutdown_connections 2024-03-24T10:49:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.979+0000 7fd8454c3640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd8280a2230 0x7fd8280a46f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.979+0000 7fd8454c3640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8401076d0 0x7fd84019c5f0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.979+0000 7fd8454c3640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd840100590 0x7fd840198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.979+0000 7fd8454c3640 1 --2- 172.21.15.23:0/4228504907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8400ffc10 0x7fd840197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.980+0000 7fd8454c3640 1 -- 172.21.15.23:0/4228504907 >> 172.21.15.23:0/4228504907 conn(0x7fd840076ba0 msgr2=0x7fd8400fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:23.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.980+0000 7fd8454c3640 1 -- 172.21.15.23:0/4228504907 shutdown_connections 2024-03-24T10:49:23.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:23.980+0000 7fd8454c3640 1 -- 172.21.15.23:0/4228504907 wait complete. 2024-03-24T10:49:23.992 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:49:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.096+0000 7f246e7d0640 1 -- 172.21.15.23:0/218076102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2468100ff0 msgr2=0x7f246810d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.096+0000 7f246e7d0640 1 --2- 172.21.15.23:0/218076102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2468100ff0 0x7f246810d500 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f2460009cd0 tx=0x7f246002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.096+0000 7f246e7d0640 1 -- 172.21.15.23:0/218076102 shutdown_connections 2024-03-24T10:49:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.096+0000 7f246e7d0640 1 --2- 172.21.15.23:0/218076102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2468100ff0 0x7f246810d500 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.096+0000 7f246e7d0640 1 --2- 172.21.15.23:0/218076102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2468100630 0x7f2468100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.096+0000 7f246e7d0640 1 --2- 172.21.15.23:0/218076102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2468106640 0x7f2468106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.096+0000 7f246e7d0640 1 -- 172.21.15.23:0/218076102 >> 172.21.15.23:0/218076102 conn(0x7f2468077770 msgr2=0x7f2468077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.096+0000 7f246e7d0640 1 -- 172.21.15.23:0/218076102 shutdown_connections 2024-03-24T10:49:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.096+0000 7f246e7d0640 1 -- 172.21.15.23:0/218076102 wait complete. 2024-03-24T10:49:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.096+0000 7f246e7d0640 1 Processor -- start 2024-03-24T10:49:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f246e7d0640 1 -- start start 2024-03-24T10:49:24.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f246e7d0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2468100630 0x7f2468197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f246e7d0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2468100ff0 0x7f24681982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f246e7d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2468106640 0x7f246819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f246e7d0640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f246819cc50 con 0x7f2468100630 2024-03-24T10:49:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f246e7d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f246819cdc0 con 0x7f2468106640 2024-03-24T10:49:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f246e7d0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f246819cf80 con 0x7f2468100ff0 2024-03-24T10:49:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f2467fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2468100630 0x7f2468197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f2467fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2468100630 0x7f2468197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45978/0 (socket says 172.21.15.23:45978) 2024-03-24T10:49:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f2467fff640 1 -- 172.21.15.23:0/3824915335 learned_addr learned my addr 172.21.15.23:0/3824915335 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f246cd6d640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2468106640 0x7f246819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f24677fe640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2468100ff0 0x7f24681982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f2467fff640 1 -- 172.21.15.23:0/3824915335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2468100ff0 msgr2=0x7f24681982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f2467fff640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2468100ff0 0x7f24681982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f2467fff640 1 -- 172.21.15.23:0/3824915335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2468106640 msgr2=0x7f246819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f2467fff640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2468106640 0x7f246819c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f2467fff640 1 -- 172.21.15.23:0/3824915335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2460009900 con 0x7f2468100630 2024-03-24T10:49:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.097+0000 7f246cd6d640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2468106640 0x7f246819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.098+0000 7f2467fff640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2468100630 0x7f2468197d90 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f245800e9f0 tx=0x7f245800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.098+0000 7f24657fa640 1 -- 172.21.15.23:0/3824915335 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2458009700 con 0x7f2468100630 2024-03-24T10:49:24.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.098+0000 7f24657fa640 1 -- 172.21.15.23:0/3824915335 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f245800cd80 con 0x7f2468100630 2024-03-24T10:49:24.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.098+0000 7f246e7d0640 1 -- 172.21.15.23:0/3824915335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f246819d260 con 0x7f2468100630 2024-03-24T10:49:24.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.098+0000 7f24657fa640 1 -- 172.21.15.23:0/3824915335 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2458018920 con 0x7f2468100630 2024-03-24T10:49:24.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.098+0000 7f246e7d0640 1 -- 172.21.15.23:0/3824915335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24681a4ae0 con 0x7f2468100630 2024-03-24T10:49:24.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.100+0000 7f24657fa640 1 -- 172.21.15.23:0/3824915335 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f2458018a80 con 0x7f2468100630 2024-03-24T10:49:24.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f24713b5740 1 Processor -- start 2024-03-24T10:49:24.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f24713b5740 1 -- start start 2024-03-24T10:49:24.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f24713b5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad0a0 0x5595ff8ad480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f24713b5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad9c0 0x5595ff8ba7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f24713b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8bd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f24713b5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5595ff81b840 con 0x5595ff8ad9c0 2024-03-24T10:49:24.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f24713b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5595ff818d70 con 0x5595ff8bad00 2024-03-24T10:49:24.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f24713b5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5595ff814550 con 0x5595ff8ad0a0 2024-03-24T10:49:24.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f2467fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad9c0 0x5595ff8ba7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f2467fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad9c0 0x5595ff8ba7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45992/0 (socket says 172.21.15.23:45992) 2024-03-24T10:49:24.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f2467fff640 1 -- 172.21.15.23:0/3915378338 learned_addr learned my addr 172.21.15.23:0/3915378338 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f246cd6d640 1 --2- 172.21.15.23:0/3915378338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad0a0 0x5595ff8ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f24677fe640 1 --2- 172.21.15.23:0/3915378338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8bd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f24657fa640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24500a2230 0x7f24500a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f246cd6d640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24500a2230 0x7f24500a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f24657fa640 1 -- 172.21.15.23:0/3824915335 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f2458014070 con 0x7f2468100630 2024-03-24T10:49:24.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f2467fff640 1 -- 172.21.15.23:0/3915378338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad0a0 msgr2=0x5595ff8ad480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f2467fff640 1 --2- 172.21.15.23:0/3915378338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad0a0 0x5595ff8ad480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f2467fff640 1 -- 172.21.15.23:0/3915378338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 msgr2=0x5595ff8bd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f2467fff640 1 --2- 172.21.15.23:0/3915378338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8bd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.101+0000 7f2467fff640 1 -- 172.21.15.23:0/3915378338 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595ff7a9de0 con 0x5595ff8ad9c0 2024-03-24T10:49:24.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.102+0000 7f24677fe640 1 --2- 172.21.15.23:0/3915378338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8bd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:24.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.102+0000 7f246cd6d640 1 --2- 172.21.15.23:0/3915378338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad0a0 0x5595ff8ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:24.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.102+0000 7f2467fff640 1 --2- 172.21.15.23:0/3915378338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad9c0 0x5595ff8ba7c0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f245809ab20 tx=0x7f2458010f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a623c99a56911b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.102+0000 7f244e7fc640 1 -- 172.21.15.23:0/3915378338 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24580d1810 con 0x5595ff8ad9c0 2024-03-24T10:49:24.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.102+0000 7f246cd6d640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24500a2230 0x7f24500a46f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2460005cc0 tx=0x7f246002c0f0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.102+0000 7f244e7fc640 1 -- 172.21.15.23:0/3915378338 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f245805c0d0 con 0x5595ff8ad9c0 2024-03-24T10:49:24.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.102+0000 7f244e7fc640 1 -- 172.21.15.23:0/3915378338 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24580cbd50 con 0x5595ff8ad9c0 2024-03-24T10:49:24.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.102+0000 7f24713b5740 1 -- 172.21.15.23:0/3915378338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad9c0 msgr2=0x5595ff8ba7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.102+0000 7f24713b5740 1 --2- 172.21.15.23:0/3915378338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad9c0 0x5595ff8ba7c0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f245809ab20 tx=0x7f2458010f50 comp rx=0 tx=0).stop 2024-03-24T10:49:24.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 -- 172.21.15.23:0/3915378338 shutdown_connections 2024-03-24T10:49:24.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 --2- 172.21.15.23:0/3915378338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8bd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 --2- 172.21.15.23:0/3915378338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad9c0 0x5595ff8ba7c0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 --2- 172.21.15.23:0/3915378338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad0a0 0x5595ff8ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 -- 172.21.15.23:0/3915378338 >> 172.21.15.23:0/3915378338 conn(0x5595ff8a7b20 msgr2=0x5595ff8a7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 -- 172.21.15.23:0/3915378338 shutdown_connections 2024-03-24T10:49:24.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 -- 172.21.15.23:0/3915378338 wait complete. 2024-03-24T10:49:24.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 -- start start 2024-03-24T10:49:24.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 Processor -- start 2024-03-24T10:49:24.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad0a0 0x5595ff8b7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad9c0 0x5595ff8b8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5595ff4e7430 con 0x5595ff8ad0a0 2024-03-24T10:49:24.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5595ff4a3e00 con 0x5595ff8bad00 2024-03-24T10:49:24.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5595ff799590 con 0x5595ff8ad9c0 2024-03-24T10:49:24.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f24713b5740 10 client.0 _ll_register_callbacks cb 0x5595ff808fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:24.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f2467fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad0a0 0x5595ff8b7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f2467fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad0a0 0x5595ff8b7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45996/0 (socket says 172.21.15.23:45996) 2024-03-24T10:49:24.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.103+0000 7f2467fff640 1 -- 172.21.15.23:0/1192070609 learned_addr learned my addr 172.21.15.23:0/1192070609 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f24677fe640 1 --2- 172.21.15.23:0/1192070609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad9c0 0x5595ff8b8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f246cd6d640 1 --2- 172.21.15.23:0/1192070609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8b87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f2467fff640 1 -- 172.21.15.23:0/1192070609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad9c0 msgr2=0x5595ff8b8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f2467fff640 1 --2- 172.21.15.23:0/1192070609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad9c0 0x5595ff8b8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f2467fff640 1 -- 172.21.15.23:0/1192070609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 msgr2=0x5595ff8b87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f2467fff640 1 --2- 172.21.15.23:0/1192070609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8b87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f2467fff640 1 -- 172.21.15.23:0/1192070609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24681a4ae0 con 0x5595ff8ad0a0 2024-03-24T10:49:24.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f246cd6d640 1 --2- 172.21.15.23:0/1192070609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:49:24.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f2467fff640 1 --2- 172.21.15.23:0/1192070609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad0a0 0x5595ff8b7d40 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f24580cbb80 tx=0x7f24580a35f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c5f4bde576ae40 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f24477fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:24.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f24477fe640 1 -- 172.21.15.23:0/1192070609 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245809eec0 con 0x5595ff8ad0a0 2024-03-24T10:49:24.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f24477fe640 1 -- 172.21.15.23:0/1192070609 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f2458097490 con 0x5595ff8ad0a0 2024-03-24T10:49:24.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f24713b5740 1 -- 172.21.15.23:0/1192070609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5595ff5137e0 con 0x5595ff8ad0a0 2024-03-24T10:49:24.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f24477fe640 1 -- 172.21.15.23:0/1192070609 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2458097650 con 0x5595ff8ad0a0 2024-03-24T10:49:24.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f24713b5740 10 client.18120 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:24.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f24713b5740 1 -- 172.21.15.23:0/1192070609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5595ff812bd0 con 0x5595ff8ad0a0 2024-03-24T10:49:24.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.104+0000 7f24477fe640 1 -- 172.21.15.23:0/1192070609 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5595ff5137e0 con 0x5595ff8ad0a0 2024-03-24T10:49:24.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.105+0000 7f24477fe640 1 -- 172.21.15.23:0/1192070609 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f24580605d0 con 0x5595ff8ad0a0 2024-03-24T10:49:24.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.105+0000 7f24713b5740 10 client.18120 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:24.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.105+0000 7f24713b5740 10 client.18120 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:49:24.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.105+0000 7f24713b5740 20 client.18120 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:24.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.105+0000 7f24713b5740 1 --2- 172.21.15.23:0/1192070609 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5595ff8caae0 0x5595ff8ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.105+0000 7f24713b5740 4 client.18120 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:49:24.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.105+0000 7f24713b5740 1 -- 172.21.15.23:0/1192070609 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595ff550720 con 0x5595ff8caae0 2024-03-24T10:49:24.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.105+0000 7f24677fe640 1 --2- 172.21.15.23:0/1192070609 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5595ff8caae0 0x5595ff8ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.105+0000 7f24477fe640 10 client.18120 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:49:24.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.105+0000 7f24677fe640 1 --2- 172.21.15.23:0/1192070609 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5595ff8caae0 0x5595ff8ccf00 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=657161c057cd31cc server_cookie=16b8d592a6b84e74 in_seq=0 out_seq=0 2024-03-24T10:49:24.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.106+0000 7f24477fe640 1 -- 172.21.15.23:0/1192070609 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5595ff550720 con 0x5595ff8caae0 2024-03-24T10:49:24.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.106+0000 7f24477fe640 10 client.18120 handle_command_reply: tid=0 2024-03-24T10:49:24.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.106+0000 7f24713b5740 1 client.18120 shutdown 2024-03-24T10:49:24.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.106+0000 7f24713b5740 1 -- 172.21.15.23:0/1192070609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad0a0 msgr2=0x5595ff8b7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.106+0000 7f24713b5740 1 --2- 172.21.15.23:0/1192070609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad0a0 0x5595ff8b7d40 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f24580cbb80 tx=0x7f24580a35f0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.106+0000 7f24713b5740 1 -- 172.21.15.23:0/1192070609 shutdown_connections 2024-03-24T10:49:24.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.106+0000 7f24713b5740 1 --2- 172.21.15.23:0/1192070609 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5595ff8caae0 0x5595ff8ccf00 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.106+0000 7f24713b5740 1 --2- 172.21.15.23:0/1192070609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.106+0000 7f24713b5740 1 --2- 172.21.15.23:0/1192070609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad9c0 0x5595ff8b8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.106+0000 7f24713b5740 1 --2- 172.21.15.23:0/1192070609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad0a0 0x5595ff8b7d40 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.106+0000 7f24477fe640 0 client.18120 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:49:24.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.106+0000 7f24713b5740 1 -- 172.21.15.23:0/1192070609 >> 172.21.15.23:0/1192070609 conn(0x5595ff8a7b20 msgr2=0x5595ff8a9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.107+0000 7f24713b5740 1 -- 172.21.15.23:0/1192070609 shutdown_connections 2024-03-24T10:49:24.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.107+0000 7f24713b5740 1 -- 172.21.15.23:0/1192070609 wait complete. 2024-03-24T10:49:24.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.107+0000 7f24713b5740 20 client.18120 trim_cache size 0 max 16384 2024-03-24T10:49:24.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.120+0000 7f24713b5740 1 Processor -- start 2024-03-24T10:49:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.120+0000 7f24713b5740 1 -- start start 2024-03-24T10:49:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.120+0000 7f24713b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8ad0a0 0x5595ff8ba470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f24713b5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad9c0 0x5595ff8c6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f24713b5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8bad00 0x5595ff8d27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f24713b5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5595ff8c72c0 con 0x5595ff8bad00 2024-03-24T10:49:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f24713b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5595ff8c6b60 con 0x5595ff8ad0a0 2024-03-24T10:49:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f24713b5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5595ff8b7b30 con 0x5595ff8ad9c0 2024-03-24T10:49:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f246cd6d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8bad00 0x5595ff8d27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f24677fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad9c0 0x5595ff8c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f2467fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8ad0a0 0x5595ff8ba470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f24677fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad9c0 0x5595ff8c6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:54542/0 (socket says 172.21.15.23:54542) 2024-03-24T10:49:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f24677fe640 1 -- 172.21.15.23:0/2799110131 learned_addr learned my addr 172.21.15.23:0/2799110131 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f24677fe640 1 -- 172.21.15.23:0/2799110131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8ad0a0 msgr2=0x5595ff8ba470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f24677fe640 1 --2- 172.21.15.23:0/2799110131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8ad0a0 0x5595ff8ba470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f24677fe640 1 -- 172.21.15.23:0/2799110131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8bad00 msgr2=0x5595ff8d27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f24677fe640 1 --2- 172.21.15.23:0/2799110131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8bad00 0x5595ff8d27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f24677fe640 1 -- 172.21.15.23:0/2799110131 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595ff7aec80 con 0x5595ff8ad9c0 2024-03-24T10:49:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f246cd6d640 1 --2- 172.21.15.23:0/2799110131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8bad00 0x5595ff8d27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:24.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.121+0000 7f24677fe640 1 --2- 172.21.15.23:0/2799110131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad9c0 0x5595ff8c6620 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f245c00be10 tx=0x7f245c00b410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4993d039e883a8b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.123+0000 7f244dffb640 1 -- 172.21.15.23:0/2799110131 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c016040 con 0x5595ff8ad9c0 2024-03-24T10:49:24.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.123+0000 7f244dffb640 1 -- 172.21.15.23:0/2799110131 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f245c017040 con 0x5595ff8ad9c0 2024-03-24T10:49:24.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.123+0000 7f244dffb640 1 -- 172.21.15.23:0/2799110131 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c0054c0 con 0x5595ff8ad9c0 2024-03-24T10:49:24.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.123+0000 7f24713b5740 1 -- 172.21.15.23:0/2799110131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad9c0 msgr2=0x5595ff8c6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.123+0000 7f24713b5740 1 --2- 172.21.15.23:0/2799110131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad9c0 0x5595ff8c6620 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f245c00be10 tx=0x7f245c00b410 comp rx=0 tx=0).stop 2024-03-24T10:49:24.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.123+0000 7f24713b5740 1 -- 172.21.15.23:0/2799110131 shutdown_connections 2024-03-24T10:49:24.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.123+0000 7f24713b5740 1 --2- 172.21.15.23:0/2799110131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8bad00 0x5595ff8d27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.123+0000 7f24713b5740 1 --2- 172.21.15.23:0/2799110131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad9c0 0x5595ff8c6620 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.123+0000 7f24713b5740 1 --2- 172.21.15.23:0/2799110131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8ad0a0 0x5595ff8ba470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.123+0000 7f24713b5740 1 -- 172.21.15.23:0/2799110131 >> 172.21.15.23:0/2799110131 conn(0x5595ff8a7b20 msgr2=0x5595ff8a9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f24713b5740 1 -- 172.21.15.23:0/2799110131 shutdown_connections 2024-03-24T10:49:24.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f24713b5740 1 -- 172.21.15.23:0/2799110131 wait complete. 2024-03-24T10:49:24.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f24713b5740 1 -- start start 2024-03-24T10:49:24.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f24713b5740 1 Processor -- start 2024-03-24T10:49:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f24713b5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad0a0 0x5595ff8c3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f24713b5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad9c0 0x5595ff8bf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f24713b5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f24713b5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5595ff8c0230 con 0x5595ff8ad9c0 2024-03-24T10:49:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f24713b5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5595ff930e10 con 0x5595ff8bad00 2024-03-24T10:49:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f24713b5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5595ff930f50 con 0x5595ff8ad0a0 2024-03-24T10:49:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f24713b5740 10 client.0 _ll_register_callbacks cb 0x7f245c00b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f24677fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad9c0 0x5595ff8bf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f246cd6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f24677fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad9c0 0x5595ff8bf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46016/0 (socket says 172.21.15.23:46016) 2024-03-24T10:49:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f24677fe640 1 -- 172.21.15.23:0/127897331 learned_addr learned my addr 172.21.15.23:0/127897331 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.124+0000 7f2467fff640 1 --2- 172.21.15.23:0/127897331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad0a0 0x5595ff8c3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f24677fe640 1 -- 172.21.15.23:0/127897331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad0a0 msgr2=0x5595ff8c3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f24677fe640 1 --2- 172.21.15.23:0/127897331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad0a0 0x5595ff8c3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f24677fe640 1 -- 172.21.15.23:0/127897331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 msgr2=0x5595ff8bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f24677fe640 1 --2- 172.21.15.23:0/127897331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8bfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f24677fe640 1 -- 172.21.15.23:0/127897331 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2460030070 con 0x5595ff8ad9c0 2024-03-24T10:49:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f246cd6d640 1 --2- 172.21.15.23:0/127897331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8bfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f24677fe640 1 --2- 172.21.15.23:0/127897331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad9c0 0x5595ff8bf3f0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f245c000c00 tx=0x7f245c0041c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7575f7a0d1d2b845 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f24477fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f24477fe640 1 -- 172.21.15.23:0/127897331 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c02d070 con 0x5595ff8ad9c0 2024-03-24T10:49:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f24477fe640 1 -- 172.21.15.23:0/127897331 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f245c0043b0 con 0x5595ff8ad9c0 2024-03-24T10:49:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f24713b5740 1 -- 172.21.15.23:0/127897331 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f245805c0d0 con 0x5595ff8ad9c0 2024-03-24T10:49:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f24477fe640 1 -- 172.21.15.23:0/127897331 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f245c005160 con 0x5595ff8ad9c0 2024-03-24T10:49:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f24713b5740 10 client.18126 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f24713b5740 1 -- 172.21.15.23:0/127897331 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x7f245c00bb40 con 0x5595ff8ad9c0 2024-03-24T10:49:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.125+0000 7f24477fe640 1 -- 172.21.15.23:0/127897331 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f245805c0d0 con 0x5595ff8ad9c0 2024-03-24T10:49:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.126+0000 7f24477fe640 1 -- 172.21.15.23:0/127897331 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f245c030030 con 0x5595ff8ad9c0 2024-03-24T10:49:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.126+0000 7f24713b5740 10 client.18126 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.126+0000 7f24713b5740 10 client.18126 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:49:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.126+0000 7f24713b5740 20 client.18126 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.126+0000 7f24713b5740 1 --2- 172.21.15.23:0/127897331 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5595ff932740 0x5595ff934b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.126+0000 7f24713b5740 4 client.18126 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:49:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.126+0000 7f24713b5740 1 -- 172.21.15.23:0/127897331 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5595ff59f890 con 0x5595ff932740 2024-03-24T10:49:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.126+0000 7f2467fff640 1 --2- 172.21.15.23:0/127897331 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5595ff932740 0x5595ff934b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.126+0000 7f24477fe640 10 client.18126 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:49:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.126+0000 7f2467fff640 1 --2- 172.21.15.23:0/127897331 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5595ff932740 0x5595ff934b60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=921cd28e7c162a1c server_cookie=3fc7df24b6f7346b in_seq=0 out_seq=0 2024-03-24T10:49:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.126+0000 7f24477fe640 1 -- 172.21.15.23:0/127897331 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5595ff59f890 con 0x5595ff932740 2024-03-24T10:49:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.126+0000 7f24477fe640 10 client.18126 handle_command_reply: tid=0 2024-03-24T10:49:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.126+0000 7f24713b5740 1 client.18126 shutdown 2024-03-24T10:49:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f24713b5740 1 -- 172.21.15.23:0/127897331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad9c0 msgr2=0x5595ff8bf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f24713b5740 1 --2- 172.21.15.23:0/127897331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad9c0 0x5595ff8bf3f0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f245c000c00 tx=0x7f245c0041c0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f24713b5740 1 -- 172.21.15.23:0/127897331 shutdown_connections 2024-03-24T10:49:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f24713b5740 1 --2- 172.21.15.23:0/127897331 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x5595ff932740 0x5595ff934b60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f24713b5740 1 --2- 172.21.15.23:0/127897331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5595ff8bad00 0x5595ff8bfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f24713b5740 1 --2- 172.21.15.23:0/127897331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ff8ad9c0 0x5595ff8bf3f0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f24713b5740 1 --2- 172.21.15.23:0/127897331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5595ff8ad0a0 0x5595ff8c3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f24477fe640 0 client.18126 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:49:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f24713b5740 1 -- 172.21.15.23:0/127897331 >> 172.21.15.23:0/127897331 conn(0x5595ff8a7b20 msgr2=0x5595ff8a8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f24713b5740 1 -- 172.21.15.23:0/127897331 shutdown_connections 2024-03-24T10:49:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f24713b5740 1 -- 172.21.15.23:0/127897331 wait complete. 2024-03-24T10:49:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f24713b5740 20 client.18126 trim_cache size 0 max 16384 2024-03-24T10:49:24.137 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:49:24.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f246e7d0640 1 -- 172.21.15.23:0/3824915335 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24500a2230 msgr2=0x7f24500a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f246e7d0640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24500a2230 0x7f24500a46f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2460005cc0 tx=0x7f246002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f246e7d0640 1 -- 172.21.15.23:0/3824915335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2468100630 msgr2=0x7f2468197d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f246e7d0640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2468100630 0x7f2468197d90 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f245800e9f0 tx=0x7f245800eeb0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f246e7d0640 1 -- 172.21.15.23:0/3824915335 shutdown_connections 2024-03-24T10:49:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f246e7d0640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f24500a2230 0x7f24500a46f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f246e7d0640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2468106640 0x7f246819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f246e7d0640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2468100ff0 0x7f24681982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f246e7d0640 1 --2- 172.21.15.23:0/3824915335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2468100630 0x7f2468197d90 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f246e7d0640 1 -- 172.21.15.23:0/3824915335 >> 172.21.15.23:0/3824915335 conn(0x7f2468077770 msgr2=0x7f246806a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f246e7d0640 1 -- 172.21.15.23:0/3824915335 shutdown_connections 2024-03-24T10:49:24.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.127+0000 7f246e7d0640 1 -- 172.21.15.23:0/3824915335 wait complete. 2024-03-24T10:49:24.139 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:49:24.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.243+0000 7fdadf46f640 1 -- 172.21.15.23:0/942300585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdad8068910 msgr2=0x7fdad8068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.243+0000 7fdadf46f640 1 --2- 172.21.15.23:0/942300585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdad8068910 0x7fdad8068d90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009890 tx=0x7fdac802bb70 comp rx=0 tx=0).stop 2024-03-24T10:49:24.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.243+0000 7fdadf46f640 1 -- 172.21.15.23:0/942300585 shutdown_connections 2024-03-24T10:49:24.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.243+0000 7fdadf46f640 1 --2- 172.21.15.23:0/942300585 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdad80692d0 0x7fdad810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.243+0000 7fdadf46f640 1 --2- 172.21.15.23:0/942300585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdad8068910 0x7fdad8068d90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.243+0000 7fdadf46f640 1 --2- 172.21.15.23:0/942300585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad8107760 0x7fdad8107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.243+0000 7fdadf46f640 1 -- 172.21.15.23:0/942300585 >> 172.21.15.23:0/942300585 conn(0x7fdad8077710 msgr2=0x7fdad8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.243+0000 7fdadf46f640 1 -- 172.21.15.23:0/942300585 shutdown_connections 2024-03-24T10:49:24.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.243+0000 7fdadf46f640 1 -- 172.21.15.23:0/942300585 wait complete. 2024-03-24T10:49:24.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.243+0000 7fdadf46f640 1 Processor -- start 2024-03-24T10:49:24.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.243+0000 7fdadf46f640 1 -- start start 2024-03-24T10:49:24.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadf46f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdad8068910 0x7fdad819da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadf46f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad80692d0 0x7fdad819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadf46f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdad8107760 0x7fdad8197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadf46f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fdad81982c0 con 0x7fdad8068910 2024-03-24T10:49:24.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadf46f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdad8198430 con 0x7fdad80692d0 2024-03-24T10:49:24.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadf46f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdad81985f0 con 0x7fdad8107760 2024-03-24T10:49:24.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadd20b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdad8068910 0x7fdad819da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadd20b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdad8068910 0x7fdad819da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46034/0 (socket says 172.21.15.23:46034) 2024-03-24T10:49:24.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadd20b640 1 -- 172.21.15.23:0/3387526767 learned_addr learned my addr 172.21.15.23:0/3387526767 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadca0a640 1 --2- 172.21.15.23:0/3387526767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad80692d0 0x7fdad819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadda0c640 1 --2- 172.21.15.23:0/3387526767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdad8107760 0x7fdad8197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadd20b640 1 -- 172.21.15.23:0/3387526767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdad8107760 msgr2=0x7fdad8197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadd20b640 1 --2- 172.21.15.23:0/3387526767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdad8107760 0x7fdad8197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadd20b640 1 -- 172.21.15.23:0/3387526767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad80692d0 msgr2=0x7fdad819dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadd20b640 1 --2- 172.21.15.23:0/3387526767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad80692d0 0x7fdad819dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadd20b640 1 -- 172.21.15.23:0/3387526767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad0009900 con 0x7fdad8068910 2024-03-24T10:49:24.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadd20b640 1 --2- 172.21.15.23:0/3387526767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdad8068910 0x7fdad819da70 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fdad400ea60 tx=0x7fdad400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdace7fc640 1 -- 172.21.15.23:0/3387526767 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad400cde0 con 0x7fdad8068910 2024-03-24T10:49:24.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadf46f640 1 -- 172.21.15.23:0/3387526767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad4009490 con 0x7fdad8068910 2024-03-24T10:49:24.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdace7fc640 1 -- 172.21.15.23:0/3387526767 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdad40035f0 con 0x7fdad8068910 2024-03-24T10:49:24.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdace7fc640 1 -- 172.21.15.23:0/3387526767 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad4010690 con 0x7fdad8068910 2024-03-24T10:49:24.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.244+0000 7fdadf46f640 1 -- 172.21.15.23:0/3387526767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac8009490 con 0x7fdad8068910 2024-03-24T10:49:24.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.247+0000 7fdae2052740 1 Processor -- start 2024-03-24T10:49:24.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.247+0000 7fdae2052740 1 -- start start 2024-03-24T10:49:24.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdace7fc640 1 -- 172.21.15.23:0/3387526767 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fdad4003af0 con 0x7fdad8068910 2024-03-24T10:49:24.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdae2052740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5e3f0 0x560fe2e5e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdae2052740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5ed10 0x560fe2e673b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdae2052740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 0x560fe2e69ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdae2052740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x560fe2dcc840 con 0x560fe2e5e3f0 2024-03-24T10:49:24.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdae2052740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560fe2dc9d70 con 0x560fe2e5ed10 2024-03-24T10:49:24.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdae2052740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560fe2dc5550 con 0x560fe2e678f0 2024-03-24T10:49:24.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdadda0c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5e3f0 0x560fe2e5e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdadda0c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5e3f0 0x560fe2e5e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46042/0 (socket says 172.21.15.23:46042) 2024-03-24T10:49:24.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdadda0c640 1 -- 172.21.15.23:0/3793738721 learned_addr learned my addr 172.21.15.23:0/3793738721 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdadd20b640 1 --2- 172.21.15.23:0/3793738721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5ed10 0x560fe2e673b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdadca0a640 1 --2- 172.21.15.23:0/3793738721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 0x560fe2e69ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdace7fc640 1 --2- 172.21.15.23:0/3387526767 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdac40a2230 0x7fdac40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdadda0c640 1 -- 172.21.15.23:0/3793738721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 msgr2=0x560fe2e69ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdadda0c640 1 --2- 172.21.15.23:0/3793738721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 0x560fe2e69ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdadda0c640 1 -- 172.21.15.23:0/3793738721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5ed10 msgr2=0x560fe2e673b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdadda0c640 1 --2- 172.21.15.23:0/3793738721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5ed10 0x560fe2e673b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdadda0c640 1 -- 172.21.15.23:0/3793738721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fe2d5ade0 con 0x560fe2e5e3f0 2024-03-24T10:49:24.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdace7fc640 1 -- 172.21.15.23:0/3387526767 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fdad4014070 con 0x7fdad8068910 2024-03-24T10:49:24.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdadda0c640 1 --2- 172.21.15.23:0/3387526767 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdac40a2230 0x7fdac40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.248+0000 7fdadda0c640 1 --2- 172.21.15.23:0/3793738721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5e3f0 0x560fe2e5e7d0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fdad000e5e0 tx=0x7fdad00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f81b93cc549e2f3f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.249+0000 7fdac37fe640 1 -- 172.21.15.23:0/3793738721 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0013040 con 0x560fe2e5e3f0 2024-03-24T10:49:24.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.249+0000 7fdac37fe640 1 -- 172.21.15.23:0/3793738721 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdad001e070 con 0x560fe2e5e3f0 2024-03-24T10:49:24.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.249+0000 7fdac37fe640 1 -- 172.21.15.23:0/3793738721 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0024ea0 con 0x560fe2e5e3f0 2024-03-24T10:49:24.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.249+0000 7fdae2052740 1 -- 172.21.15.23:0/3793738721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5e3f0 msgr2=0x560fe2e5e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.249+0000 7fdadda0c640 1 --2- 172.21.15.23:0/3387526767 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdac40a2230 0x7fdac40a46f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdad0021000 tx=0x7fdad0024000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.249+0000 7fdae2052740 1 --2- 172.21.15.23:0/3793738721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5e3f0 0x560fe2e5e7d0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fdad000e5e0 tx=0x7fdad00095c0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.249+0000 7fdae2052740 1 -- 172.21.15.23:0/3793738721 shutdown_connections 2024-03-24T10:49:24.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.249+0000 7fdae2052740 1 --2- 172.21.15.23:0/3793738721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 0x560fe2e69ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.249+0000 7fdae2052740 1 --2- 172.21.15.23:0/3793738721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5ed10 0x560fe2e673b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.249+0000 7fdae2052740 1 --2- 172.21.15.23:0/3793738721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5e3f0 0x560fe2e5e7d0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.249+0000 7fdae2052740 1 -- 172.21.15.23:0/3793738721 >> 172.21.15.23:0/3793738721 conn(0x560fe2e58b20 msgr2=0x560fe2e58f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdae2052740 1 -- 172.21.15.23:0/3793738721 shutdown_connections 2024-03-24T10:49:24.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdae2052740 1 -- 172.21.15.23:0/3793738721 wait complete. 2024-03-24T10:49:24.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdae2052740 1 -- start start 2024-03-24T10:49:24.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdae2052740 1 Processor -- start 2024-03-24T10:49:24.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdae2052740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5e3f0 0x560fe2e648d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdae2052740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5ed10 0x560fe2e64e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdae2052740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 0x560fe2e65350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdae2052740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x560fe2a98430 con 0x560fe2e5e3f0 2024-03-24T10:49:24.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdae2052740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560fe2a54e00 con 0x560fe2e5ed10 2024-03-24T10:49:24.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdae2052740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560fe2d4a590 con 0x560fe2e678f0 2024-03-24T10:49:24.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdae2052740 10 client.0 _ll_register_callbacks cb 0x560fe2db9fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:24.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdadd20b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5e3f0 0x560fe2e648d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdadd20b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5e3f0 0x560fe2e648d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46048/0 (socket says 172.21.15.23:46048) 2024-03-24T10:49:24.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdadd20b640 1 -- 172.21.15.23:0/3302216537 learned_addr learned my addr 172.21.15.23:0/3302216537 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdadca0a640 1 --2- 172.21.15.23:0/3302216537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5ed10 0x560fe2e64e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.250+0000 7fdadda0c640 1 --2- 172.21.15.23:0/3302216537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 0x560fe2e65350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.251+0000 7fdadd20b640 1 -- 172.21.15.23:0/3302216537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 msgr2=0x560fe2e65350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.251+0000 7fdadd20b640 1 --2- 172.21.15.23:0/3302216537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 0x560fe2e65350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.251+0000 7fdadd20b640 1 -- 172.21.15.23:0/3302216537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5ed10 msgr2=0x560fe2e64e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.251+0000 7fdadd20b640 1 --2- 172.21.15.23:0/3302216537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5ed10 0x560fe2e64e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.251+0000 7fdadd20b640 1 -- 172.21.15.23:0/3302216537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad0009e30 con 0x560fe2e5e3f0 2024-03-24T10:49:24.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.251+0000 7fdadd20b640 1 --2- 172.21.15.23:0/3302216537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5e3f0 0x560fe2e648d0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fdad401f000 tx=0x7fdad4097250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9088896ba99b17 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.251+0000 7fdac0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:24.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.251+0000 7fdac0ff9640 1 -- 172.21.15.23:0/3302216537 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad409b5f0 con 0x560fe2e5e3f0 2024-03-24T10:49:24.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.251+0000 7fdac0ff9640 1 -- 172.21.15.23:0/3302216537 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fdad409aef0 con 0x560fe2e5e3f0 2024-03-24T10:49:24.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.251+0000 7fdae2052740 1 -- 172.21.15.23:0/3302216537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560fe2ac47e0 con 0x560fe2e5e3f0 2024-03-24T10:49:24.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.251+0000 7fdac0ff9640 1 -- 172.21.15.23:0/3302216537 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdad409f040 con 0x560fe2e5e3f0 2024-03-24T10:49:24.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.251+0000 7fdae2052740 10 client.18138 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:24.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.251+0000 7fdae2052740 1 -- 172.21.15.23:0/3302216537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560fe2dc3bd0 con 0x560fe2e5e3f0 2024-03-24T10:49:24.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.251+0000 7fdac0ff9640 1 -- 172.21.15.23:0/3302216537 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560fe2ac47e0 con 0x560fe2e5e3f0 2024-03-24T10:49:24.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.252+0000 7fdac0ff9640 1 -- 172.21.15.23:0/3302216537 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fdad409a3b0 con 0x560fe2e5e3f0 2024-03-24T10:49:24.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.252+0000 7fdae2052740 10 client.18138 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:24.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.252+0000 7fdae2052740 10 client.18138 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:49:24.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.252+0000 7fdae2052740 20 client.18138 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:24.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.252+0000 7fdae2052740 1 --2- 172.21.15.23:0/3302216537 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x560fe2e77a20 0x560fe2e79e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.252+0000 7fdae2052740 4 client.18138 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:49:24.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.252+0000 7fdae2052740 1 -- 172.21.15.23:0/3302216537 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fe2b01720 con 0x560fe2e77a20 2024-03-24T10:49:24.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.252+0000 7fdadca0a640 1 --2- 172.21.15.23:0/3302216537 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x560fe2e77a20 0x560fe2e79e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.252+0000 7fdadca0a640 1 --2- 172.21.15.23:0/3302216537 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x560fe2e77a20 0x560fe2e79e30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=465e8e403a212d47 server_cookie=ab40fcd7c22e19a5 in_seq=0 out_seq=0 2024-03-24T10:49:24.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.252+0000 7fdac0ff9640 10 client.18138 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:49:24.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdac0ff9640 1 -- 172.21.15.23:0/3302216537 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560fe2b01720 con 0x560fe2e77a20 2024-03-24T10:49:24.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdac0ff9640 10 client.18138 handle_command_reply: tid=0 2024-03-24T10:49:24.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdae2052740 1 client.18138 shutdown 2024-03-24T10:49:24.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdae2052740 1 -- 172.21.15.23:0/3302216537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5e3f0 msgr2=0x560fe2e648d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdae2052740 1 --2- 172.21.15.23:0/3302216537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5e3f0 0x560fe2e648d0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fdad401f000 tx=0x7fdad4097250 comp rx=0 tx=0).stop 2024-03-24T10:49:24.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdae2052740 1 -- 172.21.15.23:0/3302216537 shutdown_connections 2024-03-24T10:49:24.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdae2052740 1 --2- 172.21.15.23:0/3302216537 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x560fe2e77a20 0x560fe2e79e30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdae2052740 1 --2- 172.21.15.23:0/3302216537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 0x560fe2e65350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdae2052740 1 --2- 172.21.15.23:0/3302216537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5ed10 0x560fe2e64e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdac0ff9640 0 client.18138 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:49:24.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdae2052740 1 --2- 172.21.15.23:0/3302216537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5e3f0 0x560fe2e648d0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdae2052740 1 -- 172.21.15.23:0/3302216537 >> 172.21.15.23:0/3302216537 conn(0x560fe2e58b20 msgr2=0x560fe2e5a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdae2052740 1 -- 172.21.15.23:0/3302216537 shutdown_connections 2024-03-24T10:49:24.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdae2052740 1 -- 172.21.15.23:0/3302216537 wait complete. 2024-03-24T10:49:24.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.253+0000 7fdae2052740 20 client.18138 trim_cache size 0 max 16384 2024-03-24T10:49:24.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.266+0000 7fdae2052740 1 Processor -- start 2024-03-24T10:49:24.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.266+0000 7fdae2052740 1 -- start start 2024-03-24T10:49:24.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdae2052740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5e3f0 0x560fe2e67000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdae2052740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5ed10 0x560fe2e65170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdae2052740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 0x560fe2e7f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdae2052740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x560fe2e49e20 con 0x560fe2e5ed10 2024-03-24T10:49:24.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdae2052740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560fe2e543e0 con 0x560fe2e5e3f0 2024-03-24T10:49:24.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdae2052740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560fe2e62e20 con 0x560fe2e678f0 2024-03-24T10:49:24.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdadca0a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5ed10 0x560fe2e65170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdadca0a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5ed10 0x560fe2e65170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46060/0 (socket says 172.21.15.23:46060) 2024-03-24T10:49:24.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdadca0a640 1 -- 172.21.15.23:0/1855468671 learned_addr learned my addr 172.21.15.23:0/1855468671 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdadd20b640 1 --2- 172.21.15.23:0/1855468671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5e3f0 0x560fe2e67000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdadda0c640 1 --2- 172.21.15.23:0/1855468671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 0x560fe2e7f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdadca0a640 1 -- 172.21.15.23:0/1855468671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 msgr2=0x560fe2e7f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdadca0a640 1 --2- 172.21.15.23:0/1855468671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 0x560fe2e7f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdadca0a640 1 -- 172.21.15.23:0/1855468671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5e3f0 msgr2=0x560fe2e67000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdadca0a640 1 --2- 172.21.15.23:0/1855468671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5e3f0 0x560fe2e67000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdadca0a640 1 -- 172.21.15.23:0/1855468671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fe2d5fc80 con 0x560fe2e5ed10 2024-03-24T10:49:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdadd20b640 1 --2- 172.21.15.23:0/1855468671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5e3f0 0x560fe2e67000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdadca0a640 1 --2- 172.21.15.23:0/1855468671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5ed10 0x560fe2e65170 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009890 tx=0x7fdac802bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a0e9b3501d7dd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdac2ffd640 1 -- 172.21.15.23:0/1855468671 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac8045040 con 0x560fe2e5ed10 2024-03-24T10:49:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdac2ffd640 1 -- 172.21.15.23:0/1855468671 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdac804c070 con 0x560fe2e5ed10 2024-03-24T10:49:24.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.267+0000 7fdac2ffd640 1 -- 172.21.15.23:0/1855468671 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac80347c0 con 0x560fe2e5ed10 2024-03-24T10:49:24.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.268+0000 7fdae2052740 1 -- 172.21.15.23:0/1855468671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5ed10 msgr2=0x560fe2e65170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.268+0000 7fdae2052740 1 --2- 172.21.15.23:0/1855468671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5ed10 0x560fe2e65170 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009890 tx=0x7fdac802bb70 comp rx=0 tx=0).stop 2024-03-24T10:49:24.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.268+0000 7fdae2052740 1 -- 172.21.15.23:0/1855468671 shutdown_connections 2024-03-24T10:49:24.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.268+0000 7fdae2052740 1 --2- 172.21.15.23:0/1855468671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e678f0 0x560fe2e7f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.268+0000 7fdae2052740 1 --2- 172.21.15.23:0/1855468671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e5ed10 0x560fe2e65170 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.268+0000 7fdae2052740 1 --2- 172.21.15.23:0/1855468671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5e3f0 0x560fe2e67000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.268+0000 7fdae2052740 1 -- 172.21.15.23:0/1855468671 >> 172.21.15.23:0/1855468671 conn(0x560fe2e58b20 msgr2=0x560fe2e57430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.268+0000 7fdae2052740 1 -- 172.21.15.23:0/1855468671 shutdown_connections 2024-03-24T10:49:24.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.268+0000 7fdae2052740 1 -- 172.21.15.23:0/1855468671 wait complete. 2024-03-24T10:49:24.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.268+0000 7fdae2052740 1 -- start start 2024-03-24T10:49:24.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.268+0000 7fdae2052740 1 Processor -- start 2024-03-24T10:49:24.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdae2052740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5e3f0 0x560fe2e6f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdae2052740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e5ed10 0x560fe2e6c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdae2052740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e678f0 0x560fe2e6ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdae2052740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x560fe2e6d390 con 0x560fe2e678f0 2024-03-24T10:49:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdae2052740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560fe2e6d500 con 0x560fe2e5e3f0 2024-03-24T10:49:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdae2052740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560fe2eddd10 con 0x560fe2e5ed10 2024-03-24T10:49:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdae2052740 10 client.0 _ll_register_callbacks cb 0x7fdad40d1b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdadda0c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e678f0 0x560fe2e6ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdadda0c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e678f0 0x560fe2e6ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46076/0 (socket says 172.21.15.23:46076) 2024-03-24T10:49:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdadd20b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5e3f0 0x560fe2e6f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdadda0c640 1 -- 172.21.15.23:0/2910264251 learned_addr learned my addr 172.21.15.23:0/2910264251 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdadca0a640 1 --2- 172.21.15.23:0/2910264251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e5ed10 0x560fe2e6c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdadda0c640 1 -- 172.21.15.23:0/2910264251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e5ed10 msgr2=0x560fe2e6c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdadda0c640 1 --2- 172.21.15.23:0/2910264251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e5ed10 0x560fe2e6c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdadda0c640 1 -- 172.21.15.23:0/2910264251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5e3f0 msgr2=0x560fe2e6f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdadda0c640 1 --2- 172.21.15.23:0/2910264251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5e3f0 0x560fe2e6f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdadda0c640 1 -- 172.21.15.23:0/2910264251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fe2dc3bd0 con 0x560fe2e678f0 2024-03-24T10:49:24.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdadda0c640 1 --2- 172.21.15.23:0/2910264251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e678f0 0x560fe2e6ccc0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fdad000e5e0 tx=0x7fdad000d870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf9037a2cc3c4d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdac0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:24.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.269+0000 7fdac0ff9640 1 -- 172.21.15.23:0/2910264251 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0024ea0 con 0x560fe2e678f0 2024-03-24T10:49:24.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.270+0000 7fdac0ff9640 1 -- 172.21.15.23:0/2910264251 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fdad0023390 con 0x560fe2e678f0 2024-03-24T10:49:24.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.270+0000 7fdae2052740 1 -- 172.21.15.23:0/2910264251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdad001e070 con 0x560fe2e678f0 2024-03-24T10:49:24.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.270+0000 7fdac0ff9640 1 -- 172.21.15.23:0/2910264251 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdad000db90 con 0x560fe2e678f0 2024-03-24T10:49:24.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.270+0000 7fdae2052740 10 client.18144 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:24.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.270+0000 7fdae2052740 1 -- 172.21.15.23:0/2910264251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x7fdad810ee70 con 0x560fe2e678f0 2024-03-24T10:49:24.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.270+0000 7fdac0ff9640 1 -- 172.21.15.23:0/2910264251 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdad001e070 con 0x560fe2e678f0 2024-03-24T10:49:24.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.270+0000 7fdac0ff9640 1 -- 172.21.15.23:0/2910264251 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fdad0023b80 con 0x560fe2e678f0 2024-03-24T10:49:24.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.270+0000 7fdae2052740 10 client.18144 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:24.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.270+0000 7fdae2052740 10 client.18144 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:49:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.270+0000 7fdae2052740 20 client.18144 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.270+0000 7fdae2052740 1 --2- 172.21.15.23:0/2910264251 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x560fe2edf330 0x560fe2ee1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.270+0000 7fdae2052740 4 client.18144 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:49:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.270+0000 7fdae2052740 1 -- 172.21.15.23:0/2910264251 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560fe2b50890 con 0x560fe2edf330 2024-03-24T10:49:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.270+0000 7fdadd20b640 1 --2- 172.21.15.23:0/2910264251 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x560fe2edf330 0x560fe2ee1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdac0ff9640 10 client.18144 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:49:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdadd20b640 1 --2- 172.21.15.23:0/2910264251 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x560fe2edf330 0x560fe2ee1750 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=84b158a31ba15bc5 server_cookie=a648cbdb515eb60f in_seq=0 out_seq=0 2024-03-24T10:49:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdac0ff9640 1 -- 172.21.15.23:0/2910264251 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560fe2b50890 con 0x560fe2edf330 2024-03-24T10:49:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdac0ff9640 10 client.18144 handle_command_reply: tid=0 2024-03-24T10:49:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdae2052740 1 client.18144 shutdown 2024-03-24T10:49:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdae2052740 1 -- 172.21.15.23:0/2910264251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e678f0 msgr2=0x560fe2e6ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdae2052740 1 --2- 172.21.15.23:0/2910264251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e678f0 0x560fe2e6ccc0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fdad000e5e0 tx=0x7fdad000d870 comp rx=0 tx=0).stop 2024-03-24T10:49:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdae2052740 1 -- 172.21.15.23:0/2910264251 shutdown_connections 2024-03-24T10:49:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdae2052740 1 --2- 172.21.15.23:0/2910264251 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x560fe2edf330 0x560fe2ee1750 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdae2052740 1 --2- 172.21.15.23:0/2910264251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560fe2e678f0 0x560fe2e6ccc0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdae2052740 1 --2- 172.21.15.23:0/2910264251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560fe2e5ed10 0x560fe2e6c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdac0ff9640 0 client.18144 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:49:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdae2052740 1 --2- 172.21.15.23:0/2910264251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560fe2e5e3f0 0x560fe2e6f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdae2052740 1 -- 172.21.15.23:0/2910264251 >> 172.21.15.23:0/2910264251 conn(0x560fe2e58b20 msgr2=0x560fe2e57cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdae2052740 1 -- 172.21.15.23:0/2910264251 shutdown_connections 2024-03-24T10:49:24.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdae2052740 1 -- 172.21.15.23:0/2910264251 wait complete. 2024-03-24T10:49:24.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.271+0000 7fdae2052740 20 client.18144 trim_cache size 0 max 16384 2024-03-24T10:49:24.285 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:49:24.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.272+0000 7fdadf46f640 1 -- 172.21.15.23:0/3387526767 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdac40a2230 msgr2=0x7fdac40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.272+0000 7fdadf46f640 1 --2- 172.21.15.23:0/3387526767 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdac40a2230 0x7fdac40a46f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdad0021000 tx=0x7fdad0024000 comp rx=0 tx=0).stop 2024-03-24T10:49:24.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.272+0000 7fdadf46f640 1 -- 172.21.15.23:0/3387526767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdad8068910 msgr2=0x7fdad819da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.272+0000 7fdadf46f640 1 --2- 172.21.15.23:0/3387526767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdad8068910 0x7fdad819da70 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fdad400ea60 tx=0x7fdad400ef20 comp rx=0 tx=0).stop 2024-03-24T10:49:24.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.272+0000 7fdadf46f640 1 -- 172.21.15.23:0/3387526767 shutdown_connections 2024-03-24T10:49:24.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.272+0000 7fdadf46f640 1 --2- 172.21.15.23:0/3387526767 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdac40a2230 0x7fdac40a46f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.272+0000 7fdadf46f640 1 --2- 172.21.15.23:0/3387526767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdad8107760 0x7fdad8197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.272+0000 7fdadf46f640 1 --2- 172.21.15.23:0/3387526767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdad80692d0 0x7fdad819dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.272+0000 7fdadf46f640 1 --2- 172.21.15.23:0/3387526767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdad8068910 0x7fdad819da70 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.272+0000 7fdadf46f640 1 -- 172.21.15.23:0/3387526767 >> 172.21.15.23:0/3387526767 conn(0x7fdad8077710 msgr2=0x7fdad80fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.272+0000 7fdadf46f640 1 -- 172.21.15.23:0/3387526767 shutdown_connections 2024-03-24T10:49:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.272+0000 7fdadf46f640 1 -- 172.21.15.23:0/3387526767 wait complete. 2024-03-24T10:49:24.288 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:24 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4228504907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:49:24.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:49:24.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.393+0000 7f843ac17640 1 -- 172.21.15.23:0/4204765209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8434106660 msgr2=0x7f8434106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.393+0000 7f843ac17640 1 --2- 172.21.15.23:0/4204765209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8434106660 0x7f8434106a40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f842c009890 tx=0x7f842c02bb70 comp rx=0 tx=0).stop 2024-03-24T10:49:24.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.393+0000 7f843ac17640 1 -- 172.21.15.23:0/4204765209 shutdown_connections 2024-03-24T10:49:24.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.393+0000 7f843ac17640 1 --2- 172.21.15.23:0/4204765209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8434100f80 0x7f843410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.393+0000 7f843ac17640 1 --2- 172.21.15.23:0/4204765209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84341005c0 0x7f8434100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.393+0000 7f843ac17640 1 --2- 172.21.15.23:0/4204765209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8434106660 0x7f8434106a40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.393+0000 7f843ac17640 1 -- 172.21.15.23:0/4204765209 >> 172.21.15.23:0/4204765209 conn(0x7f8434076990 msgr2=0x7f8434076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.393+0000 7f843ac17640 1 -- 172.21.15.23:0/4204765209 shutdown_connections 2024-03-24T10:49:24.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.393+0000 7f843ac17640 1 -- 172.21.15.23:0/4204765209 wait complete. 2024-03-24T10:49:24.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.393+0000 7f843ac17640 1 Processor -- start 2024-03-24T10:49:24.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.393+0000 7f843ac17640 1 -- start start 2024-03-24T10:49:24.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f843ac17640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84341005c0 0x7f843419d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f843ac17640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8434100f80 0x7f843419de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f843ac17640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8434106660 0x7f8434197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f843ac17640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f8434198150 con 0x7f8434100f80 2024-03-24T10:49:24.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f843ac17640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f84341982c0 con 0x7f84341005c0 2024-03-24T10:49:24.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f843ac17640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8434198480 con 0x7f8434106660 2024-03-24T10:49:24.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f8433fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8434100f80 0x7f843419de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f8433fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8434100f80 0x7f843419de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46100/0 (socket says 172.21.15.23:46100) 2024-03-24T10:49:24.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f8433fff640 1 -- 172.21.15.23:0/2505751339 learned_addr learned my addr 172.21.15.23:0/2505751339 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f84389b3640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84341005c0 0x7f843419d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f84391b4640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8434106660 0x7f8434197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f8433fff640 1 -- 172.21.15.23:0/2505751339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8434106660 msgr2=0x7f8434197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f8433fff640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8434106660 0x7f8434197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f8433fff640 1 -- 172.21.15.23:0/2505751339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84341005c0 msgr2=0x7f843419d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f8433fff640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84341005c0 0x7f843419d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f8433fff640 1 -- 172.21.15.23:0/2505751339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f842c009490 con 0x7f8434100f80 2024-03-24T10:49:24.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f84389b3640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84341005c0 0x7f843419d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:24.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f8433fff640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8434100f80 0x7f843419de90 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f842400efb0 tx=0x7f842400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f8431ffb640 1 -- 172.21.15.23:0/2505751339 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842400b570 con 0x7f8434100f80 2024-03-24T10:49:24.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f843ac17640 1 -- 172.21.15.23:0/2505751339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8434198760 con 0x7f8434100f80 2024-03-24T10:49:24.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f8431ffb640 1 -- 172.21.15.23:0/2505751339 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f842400bbb0 con 0x7f8434100f80 2024-03-24T10:49:24.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.394+0000 7f8431ffb640 1 -- 172.21.15.23:0/2505751339 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424016d20 con 0x7f8434100f80 2024-03-24T10:49:24.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.395+0000 7f843ac17640 1 -- 172.21.15.23:0/2505751339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84341a49f0 con 0x7f8434100f80 2024-03-24T10:49:24.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.397+0000 7f8431ffb640 1 -- 172.21.15.23:0/2505751339 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f842400a040 con 0x7f8434100f80 2024-03-24T10:49:24.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.397+0000 7f843d7f1740 1 Processor -- start 2024-03-24T10:49:24.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.397+0000 7f843d7f1740 1 -- start start 2024-03-24T10:49:24.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.397+0000 7f8431ffb640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f84200a2230 0x7f84200a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.397+0000 7f843d7f1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c19f93f0 0x5588c19f97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.397+0000 7f843d7f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f9d10 0x5588c1a023b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.397+0000 7f843d7f1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c1a028f0 0x5588c1a04ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.397+0000 7f843d7f1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5588c1967840 con 0x5588c1a028f0 2024-03-24T10:49:24.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.397+0000 7f843d7f1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5588c1964d70 con 0x5588c19f9d10 2024-03-24T10:49:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.397+0000 7f843d7f1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5588c1960550 con 0x5588c19f93f0 2024-03-24T10:49:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.397+0000 7f84391b4640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f84200a2230 0x7f84200a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f8433fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f9d10 0x5588c1a023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f84389b3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c19f93f0 0x5588c19f97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f84389b3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c19f93f0 0x5588c19f97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:54604/0 (socket says 172.21.15.23:54604) 2024-03-24T10:49:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f8431ffb640 1 -- 172.21.15.23:0/2505751339 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f84240ce0d0 con 0x7f8434100f80 2024-03-24T10:49:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f84391b4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c1a028f0 0x5588c1a04ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f8433fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f9d10 0x5588c1a023b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:53328/0 (socket says 172.21.15.23:53328) 2024-03-24T10:49:24.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f84389b3640 1 -- 172.21.15.23:0/1722171063 learned_addr learned my addr 172.21.15.23:0/1722171063 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f8433fff640 1 -- 172.21.15.23:0/1722171063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c19f93f0 msgr2=0x5588c19f97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f8433fff640 1 --2- 172.21.15.23:0/1722171063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c19f93f0 0x5588c19f97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f8433fff640 1 -- 172.21.15.23:0/1722171063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c1a028f0 msgr2=0x5588c1a04ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f8433fff640 1 --2- 172.21.15.23:0/1722171063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c1a028f0 0x5588c1a04ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f8433fff640 1 -- 172.21.15.23:0/1722171063 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588c18f5de0 con 0x5588c19f9d10 2024-03-24T10:49:24.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f84391b4640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f84200a2230 0x7f84200a46f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f841c009fd0 tx=0x7f841c009b80 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f8433fff640 1 --2- 172.21.15.23:0/1722171063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f9d10 0x5588c1a023b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f84240cb7f0 tx=0x7f84240117e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93a09fc1de30d70d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f842affd640 1 -- 172.21.15.23:0/1722171063 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84240ad2e0 con 0x5588c19f9d10 2024-03-24T10:49:24.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f842affd640 1 -- 172.21.15.23:0/1722171063 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84240ad100 con 0x5588c19f9d10 2024-03-24T10:49:24.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.398+0000 7f842affd640 1 -- 172.21.15.23:0/1722171063 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84240aee90 con 0x5588c19f9d10 2024-03-24T10:49:24.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.399+0000 7f843d7f1740 1 -- 172.21.15.23:0/1722171063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f9d10 msgr2=0x5588c1a023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.399+0000 7f843d7f1740 1 --2- 172.21.15.23:0/1722171063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f9d10 0x5588c1a023b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f84240cb7f0 tx=0x7f84240117e0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.399+0000 7f843d7f1740 1 -- 172.21.15.23:0/1722171063 shutdown_connections 2024-03-24T10:49:24.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.399+0000 7f843d7f1740 1 --2- 172.21.15.23:0/1722171063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c1a028f0 0x5588c1a04ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.399+0000 7f843d7f1740 1 --2- 172.21.15.23:0/1722171063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f9d10 0x5588c1a023b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.399+0000 7f843d7f1740 1 --2- 172.21.15.23:0/1722171063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c19f93f0 0x5588c19f97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.399+0000 7f843d7f1740 1 -- 172.21.15.23:0/1722171063 >> 172.21.15.23:0/1722171063 conn(0x5588c19f3b20 msgr2=0x5588c19f3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.399+0000 7f843d7f1740 1 -- 172.21.15.23:0/1722171063 shutdown_connections 2024-03-24T10:49:24.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.399+0000 7f843d7f1740 1 -- 172.21.15.23:0/1722171063 wait complete. 2024-03-24T10:49:24.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.399+0000 7f843d7f1740 1 -- start start 2024-03-24T10:49:24.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.399+0000 7f843d7f1740 1 Processor -- start 2024-03-24T10:49:24.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f843d7f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f93f0 0x5588c19ff9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f843d7f1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c19f9d10 0x5588c19fff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f843d7f1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c1a028f0 0x5588c1a00470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f843d7f1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5588c1633430 con 0x5588c19f9d10 2024-03-24T10:49:24.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f843d7f1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5588c15efe00 con 0x5588c19f93f0 2024-03-24T10:49:24.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f843d7f1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5588c18e5590 con 0x5588c1a028f0 2024-03-24T10:49:24.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f843d7f1740 10 client.0 _ll_register_callbacks cb 0x5588c1954fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:24.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f8433fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c19f9d10 0x5588c19fff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f84389b3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f93f0 0x5588c19ff9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f8433fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c19f9d10 0x5588c19fff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46114/0 (socket says 172.21.15.23:46114) 2024-03-24T10:49:24.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f8433fff640 1 -- 172.21.15.23:0/19025354 learned_addr learned my addr 172.21.15.23:0/19025354 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f84391b4640 1 --2- 172.21.15.23:0/19025354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c1a028f0 0x5588c1a00470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f84389b3640 1 -- 172.21.15.23:0/19025354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c1a028f0 msgr2=0x5588c1a00470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f84389b3640 1 --2- 172.21.15.23:0/19025354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c1a028f0 0x5588c1a00470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f84389b3640 1 -- 172.21.15.23:0/19025354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c19f9d10 msgr2=0x5588c19fff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f84389b3640 1 --2- 172.21.15.23:0/19025354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c19f9d10 0x5588c19fff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f84389b3640 1 -- 172.21.15.23:0/19025354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84341a49f0 con 0x5588c19f93f0 2024-03-24T10:49:24.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f8433fff640 1 --2- 172.21.15.23:0/19025354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c19f9d10 0x5588c19fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:24.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f84389b3640 1 --2- 172.21.15.23:0/19025354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f93f0 0x5588c19ff9f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f842c000c00 tx=0x7f842c0045c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8add8a9901f8b536 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f8413fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:49:24.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f8413fff640 1 -- 172.21.15.23:0/19025354 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c02c510 con 0x5588c19f93f0 2024-03-24T10:49:24.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f843d7f1740 1 -- 172.21.15.23:0/19025354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5588c165f7e0 con 0x5588c19f93f0 2024-03-24T10:49:24.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.400+0000 7f8413fff640 1 -- 172.21.15.23:0/19025354 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f842c045070 con 0x5588c19f93f0 2024-03-24T10:49:24.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.401+0000 7f8413fff640 1 -- 172.21.15.23:0/19025354 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f842c02c670 con 0x5588c19f93f0 2024-03-24T10:49:24.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.401+0000 7f843d7f1740 10 client.28043 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:24.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.401+0000 7f843d7f1740 1 -- 172.21.15.23:0/19025354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5588c195ebd0 con 0x5588c19f93f0 2024-03-24T10:49:24.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.401+0000 7f8413fff640 1 -- 172.21.15.23:0/19025354 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5588c165f7e0 con 0x5588c19f93f0 2024-03-24T10:49:24.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.401+0000 7f8413fff640 1 -- 172.21.15.23:0/19025354 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f842c003d00 con 0x5588c19f93f0 2024-03-24T10:49:24.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.401+0000 7f843d7f1740 10 client.28043 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:24.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.401+0000 7f843d7f1740 10 client.28043 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:24.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.401+0000 7f843d7f1740 20 client.28043 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:24.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.401+0000 7f843d7f1740 1 --2- 172.21.15.23:0/19025354 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5588c1a12ad0 0x5588c1a14ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.401+0000 7f843d7f1740 4 client.28043 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:49:24.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.401+0000 7f843d7f1740 1 -- 172.21.15.23:0/19025354 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588c169c720 con 0x5588c1a12ad0 2024-03-24T10:49:24.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.401+0000 7f8433fff640 1 --2- 172.21.15.23:0/19025354 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5588c1a12ad0 0x5588c1a14ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.402+0000 7f8433fff640 1 --2- 172.21.15.23:0/19025354 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5588c1a12ad0 0x5588c1a14ee0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=967f0c1683eb67eb server_cookie=6f3deb06d72c3f60 in_seq=0 out_seq=0 2024-03-24T10:49:24.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.402+0000 7f8413fff640 10 client.28043 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:24.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.402+0000 7f8413fff640 1 -- 172.21.15.23:0/19025354 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5588c169c720 con 0x5588c1a12ad0 2024-03-24T10:49:24.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.402+0000 7f8413fff640 10 client.28043 handle_command_reply: tid=0 2024-03-24T10:49:24.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.403+0000 7f843d7f1740 1 client.28043 shutdown 2024-03-24T10:49:24.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.403+0000 7f843d7f1740 1 -- 172.21.15.23:0/19025354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f93f0 msgr2=0x5588c19ff9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.403+0000 7f843d7f1740 1 --2- 172.21.15.23:0/19025354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f93f0 0x5588c19ff9f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f842c000c00 tx=0x7f842c0045c0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.403+0000 7f843d7f1740 1 -- 172.21.15.23:0/19025354 shutdown_connections 2024-03-24T10:49:24.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.403+0000 7f843d7f1740 1 --2- 172.21.15.23:0/19025354 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5588c1a12ad0 0x5588c1a14ee0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.403+0000 7f843d7f1740 1 --2- 172.21.15.23:0/19025354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c1a028f0 0x5588c1a00470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.403+0000 7f8413fff640 0 client.28043 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:24.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.403+0000 7f843d7f1740 1 --2- 172.21.15.23:0/19025354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c19f9d10 0x5588c19fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.403+0000 7f843d7f1740 1 --2- 172.21.15.23:0/19025354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f93f0 0x5588c19ff9f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.403+0000 7f843d7f1740 1 -- 172.21.15.23:0/19025354 >> 172.21.15.23:0/19025354 conn(0x5588c19f3b20 msgr2=0x5588c19f5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.403+0000 7f843d7f1740 1 -- 172.21.15.23:0/19025354 shutdown_connections 2024-03-24T10:49:24.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.403+0000 7f843d7f1740 1 -- 172.21.15.23:0/19025354 wait complete. 2024-03-24T10:49:24.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.403+0000 7f843d7f1740 20 client.28043 trim_cache size 0 max 16384 2024-03-24T10:49:24.423 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:24 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4228504907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:49:24.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.416+0000 7f843d7f1740 1 Processor -- start 2024-03-24T10:49:24.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.416+0000 7f843d7f1740 1 -- start start 2024-03-24T10:49:24.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.416+0000 7f843d7f1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c19f93f0 0x5588c1a0e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.416+0000 7f843d7f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f9d10 0x5588c19f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.416+0000 7f843d7f1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c1a028f0 0x5588c1a00290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.416+0000 7f843d7f1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5588c19ef250 con 0x5588c19f93f0 2024-03-24T10:49:24.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.416+0000 7f843d7f1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5588c19f9050 con 0x5588c19f9d10 2024-03-24T10:49:24.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.416+0000 7f843d7f1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5588c19f8b90 con 0x5588c1a028f0 2024-03-24T10:49:24.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.416+0000 7f84389b3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c19f93f0 0x5588c1a0e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.416+0000 7f84389b3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c19f93f0 0x5588c1a0e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46120/0 (socket says 172.21.15.23:46120) 2024-03-24T10:49:24.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.416+0000 7f84389b3640 1 -- 172.21.15.23:0/344821565 learned_addr learned my addr 172.21.15.23:0/344821565 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.417+0000 7f84391b4640 1 --2- 172.21.15.23:0/344821565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c1a028f0 0x5588c1a00290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.417+0000 7f8433fff640 1 --2- 172.21.15.23:0/344821565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f9d10 0x5588c19f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.417+0000 7f84389b3640 1 -- 172.21.15.23:0/344821565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c1a028f0 msgr2=0x5588c1a00290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.417+0000 7f84389b3640 1 --2- 172.21.15.23:0/344821565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c1a028f0 0x5588c1a00290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.417+0000 7f84389b3640 1 -- 172.21.15.23:0/344821565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f9d10 msgr2=0x5588c19f2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.417+0000 7f84389b3640 1 --2- 172.21.15.23:0/344821565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f9d10 0x5588c19f2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.417+0000 7f84389b3640 1 -- 172.21.15.23:0/344821565 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588c18fac80 con 0x5588c19f93f0 2024-03-24T10:49:24.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.417+0000 7f84389b3640 1 --2- 172.21.15.23:0/344821565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c19f93f0 0x5588c1a0e120 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f842c02e010 tx=0x7f842c0327c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21307552640b850b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.417+0000 7f842a7fc640 1 -- 172.21.15.23:0/344821565 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c034440 con 0x5588c19f93f0 2024-03-24T10:49:24.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.417+0000 7f842a7fc640 1 -- 172.21.15.23:0/344821565 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f842c043070 con 0x5588c19f93f0 2024-03-24T10:49:24.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.417+0000 7f842a7fc640 1 -- 172.21.15.23:0/344821565 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c032c20 con 0x5588c19f93f0 2024-03-24T10:49:24.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 -- 172.21.15.23:0/344821565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c19f93f0 msgr2=0x5588c1a0e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 --2- 172.21.15.23:0/344821565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c19f93f0 0x5588c1a0e120 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f842c02e010 tx=0x7f842c0327c0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 -- 172.21.15.23:0/344821565 shutdown_connections 2024-03-24T10:49:24.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 --2- 172.21.15.23:0/344821565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c1a028f0 0x5588c1a00290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 --2- 172.21.15.23:0/344821565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f9d10 0x5588c19f2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 --2- 172.21.15.23:0/344821565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c19f93f0 0x5588c1a0e120 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 -- 172.21.15.23:0/344821565 >> 172.21.15.23:0/344821565 conn(0x5588c19f3b20 msgr2=0x5588c19f3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 -- 172.21.15.23:0/344821565 shutdown_connections 2024-03-24T10:49:24.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 -- 172.21.15.23:0/344821565 wait complete. 2024-03-24T10:49:24.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 -- start start 2024-03-24T10:49:24.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 Processor -- start 2024-03-24T10:49:24.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f93f0 0x5588c1a06150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c19f9d10 0x5588c1a06690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c1a028f0 0x5588c1a06bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5588c1a77e20 con 0x5588c1a028f0 2024-03-24T10:49:24.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.418+0000 7f843d7f1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5588c1a77f60 con 0x5588c19f93f0 2024-03-24T10:49:24.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f843d7f1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5588c1a78120 con 0x5588c19f9d10 2024-03-24T10:49:24.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f843d7f1740 10 client.0 _ll_register_callbacks cb 0x7f841c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:24.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f84391b4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c1a028f0 0x5588c1a06bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f84391b4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c1a028f0 0x5588c1a06bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46132/0 (socket says 172.21.15.23:46132) 2024-03-24T10:49:24.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f84391b4640 1 -- 172.21.15.23:0/4221985658 learned_addr learned my addr 172.21.15.23:0/4221985658 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:24.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f84389b3640 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f93f0 0x5588c1a06150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f8433fff640 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c19f9d10 0x5588c1a06690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f84391b4640 1 -- 172.21.15.23:0/4221985658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c19f9d10 msgr2=0x5588c1a06690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f84391b4640 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c19f9d10 0x5588c1a06690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f84391b4640 1 -- 172.21.15.23:0/4221985658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f93f0 msgr2=0x5588c1a06150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:24.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f84391b4640 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f93f0 0x5588c1a06150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f84391b4640 1 -- 172.21.15.23:0/4221985658 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588c195ebd0 con 0x5588c1a028f0 2024-03-24T10:49:24.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f8433fff640 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c19f9d10 0x5588c1a06690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:24.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f84389b3640 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f93f0 0x5588c1a06150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:24.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f84391b4640 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c1a028f0 0x5588c1a06bd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f841c009710 tx=0x7f841c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3915acf497c5da07 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:24.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f8413fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:24.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f8413fff640 1 -- 172.21.15.23:0/4221985658 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841c016040 con 0x5588c1a028f0 2024-03-24T10:49:24.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f843d7f1740 1 -- 172.21.15.23:0/4221985658 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84240ad100 con 0x5588c1a028f0 2024-03-24T10:49:24.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f8413fff640 1 -- 172.21.15.23:0/4221985658 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f841c00fe20 con 0x5588c1a028f0 2024-03-24T10:49:24.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f8413fff640 1 -- 172.21.15.23:0/4221985658 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f841c004500 con 0x5588c1a028f0 2024-03-24T10:49:24.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.419+0000 7f843d7f1740 10 client.18162 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:24.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.420+0000 7f843d7f1740 1 -- 172.21.15.23:0/4221985658 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5588c1920920 con 0x5588c1a028f0 2024-03-24T10:49:24.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.420+0000 7f8413fff640 1 -- 172.21.15.23:0/4221985658 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84240ad100 con 0x5588c1a028f0 2024-03-24T10:49:24.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.420+0000 7f8413fff640 1 -- 172.21.15.23:0/4221985658 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f841c0047a0 con 0x5588c1a028f0 2024-03-24T10:49:24.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.420+0000 7f843d7f1740 10 client.18162 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:24.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.420+0000 7f843d7f1740 10 client.18162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:24.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.420+0000 7f843d7f1740 20 client.18162 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:24.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.420+0000 7f843d7f1740 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5588c1a79a40 0x5588c1a7be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:24.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.420+0000 7f843d7f1740 4 client.18162 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:49:24.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.420+0000 7f843d7f1740 1 -- 172.21.15.23:0/4221985658 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5588c16eb890 con 0x5588c1a79a40 2024-03-24T10:49:24.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.420+0000 7f84389b3640 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5588c1a79a40 0x5588c1a7be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:24.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.420+0000 7f84389b3640 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5588c1a79a40 0x5588c1a7be60 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cfab77f8c609eea server_cookie=8d3f2e0ea8d7181e in_seq=0 out_seq=0 2024-03-24T10:49:24.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.420+0000 7f8413fff640 10 client.18162 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:24.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f8413fff640 1 -- 172.21.15.23:0/4221985658 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5588c16eb890 con 0x5588c1a79a40 2024-03-24T10:49:24.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f8413fff640 10 client.18162 handle_command_reply: tid=0 2024-03-24T10:49:24.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843d7f1740 1 client.18162 shutdown 2024-03-24T10:49:24.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843d7f1740 1 -- 172.21.15.23:0/4221985658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c1a028f0 msgr2=0x5588c1a06bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843d7f1740 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c1a028f0 0x5588c1a06bd0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f841c009710 tx=0x7f841c005f90 comp rx=0 tx=0).stop 2024-03-24T10:49:24.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843d7f1740 1 -- 172.21.15.23:0/4221985658 shutdown_connections 2024-03-24T10:49:24.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843d7f1740 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5588c1a028f0 0x5588c1a06bd0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843d7f1740 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5588c1a79a40 0x5588c1a7be60 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843d7f1740 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5588c19f9d10 0x5588c1a06690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843d7f1740 1 --2- 172.21.15.23:0/4221985658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588c19f93f0 0x5588c1a06150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f8413fff640 0 client.18162 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:24.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843d7f1740 1 -- 172.21.15.23:0/4221985658 >> 172.21.15.23:0/4221985658 conn(0x5588c19f3b20 msgr2=0x5588c19f2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843d7f1740 1 -- 172.21.15.23:0/4221985658 shutdown_connections 2024-03-24T10:49:24.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843d7f1740 1 -- 172.21.15.23:0/4221985658 wait complete. 2024-03-24T10:49:24.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843d7f1740 20 client.18162 trim_cache size 0 max 16384 2024-03-24T10:49:24.435 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:49:24.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843ac17640 1 -- 172.21.15.23:0/2505751339 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f84200a2230 msgr2=0x7f84200a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843ac17640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f84200a2230 0x7f84200a46f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f841c009fd0 tx=0x7f841c009b80 comp rx=0 tx=0).stop 2024-03-24T10:49:24.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843ac17640 1 -- 172.21.15.23:0/2505751339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8434100f80 msgr2=0x7f843419de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.421+0000 7f843ac17640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8434100f80 0x7f843419de90 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f842400efb0 tx=0x7f842400c800 comp rx=0 tx=0).stop 2024-03-24T10:49:24.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.422+0000 7f843ac17640 1 -- 172.21.15.23:0/2505751339 shutdown_connections 2024-03-24T10:49:24.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.422+0000 7f843ac17640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f84200a2230 0x7f84200a46f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.422+0000 7f843ac17640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8434106660 0x7f8434197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.422+0000 7f843ac17640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8434100f80 0x7f843419de90 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.422+0000 7f843ac17640 1 --2- 172.21.15.23:0/2505751339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84341005c0 0x7f843419d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.422+0000 7f843ac17640 1 -- 172.21.15.23:0/2505751339 >> 172.21.15.23:0/2505751339 conn(0x7f8434076990 msgr2=0x7f84340fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.422+0000 7f843ac17640 1 -- 172.21.15.23:0/2505751339 shutdown_connections 2024-03-24T10:49:24.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:24.422+0000 7f843ac17640 1 -- 172.21.15.23:0/2505751339 wait complete. 2024-03-24T10:49:24.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:24 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4228504907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:49:25.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:25 smithi023 ceph-mon[32466]: pgmap v290: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-03-24T10:49:25.438 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:49:25.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:49:25.439 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:49:25.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:25 smithi117 ceph-mon[36368]: pgmap v290: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-03-24T10:49:25.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:25 smithi116 ceph-mon[36248]: pgmap v290: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-03-24T10:49:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.557+0000 7fe009a0b640 1 -- 172.21.15.23:0/4177324203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe004102070 msgr2=0x7fe00410e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.557+0000 7fe009a0b640 1 --2- 172.21.15.23:0/4177324203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe004102070 0x7fe00410e580 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fdffc009cd0 tx=0x7fdffc02c0f0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.557+0000 7fe009a0b640 1 -- 172.21.15.23:0/4177324203 shutdown_connections 2024-03-24T10:49:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.557+0000 7fe009a0b640 1 --2- 172.21.15.23:0/4177324203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe004102070 0x7fe00410e580 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.557+0000 7fe009a0b640 1 --2- 172.21.15.23:0/4177324203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0041016b0 0x7fe004101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.557+0000 7fe009a0b640 1 --2- 172.21.15.23:0/4177324203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0041076c0 0x7fe004107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.557+0000 7fe009a0b640 1 -- 172.21.15.23:0/4177324203 >> 172.21.15.23:0/4177324203 conn(0x7fe0040fd140 msgr2=0x7fe0040ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.557+0000 7fe009a0b640 1 -- 172.21.15.23:0/4177324203 shutdown_connections 2024-03-24T10:49:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.557+0000 7fe009a0b640 1 -- 172.21.15.23:0/4177324203 wait complete. 2024-03-24T10:49:25.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe009a0b640 1 Processor -- start 2024-03-24T10:49:25.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe009a0b640 1 -- start start 2024-03-24T10:49:25.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe009a0b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0041016b0 0x7fe004197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe009a0b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe004102070 0x7fe004198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe009a0b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0041076c0 0x7fe00419c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe009a0b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fe00419cbf0 con 0x7fe0041076c0 2024-03-24T10:49:25.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe009a0b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe00419cd60 con 0x7fe0041016b0 2024-03-24T10:49:25.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe009a0b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe00419cf20 con 0x7fe004102070 2024-03-24T10:49:25.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe0037fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0041076c0 0x7fe00419c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe002ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0041016b0 0x7fe004197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe0037fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0041076c0 0x7fe00419c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46138/0 (socket says 172.21.15.23:46138) 2024-03-24T10:49:25.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe0037fe640 1 -- 172.21.15.23:0/19900401 learned_addr learned my addr 172.21.15.23:0/19900401 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:25.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe002ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0041016b0 0x7fe004197d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:53378/0 (socket says 172.21.15.23:53378) 2024-03-24T10:49:25.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe0027fc640 1 --2- 172.21.15.23:0/19900401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe004102070 0x7fe004198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe0037fe640 1 -- 172.21.15.23:0/19900401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe004102070 msgr2=0x7fe004198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe0037fe640 1 --2- 172.21.15.23:0/19900401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe004102070 0x7fe004198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe0037fe640 1 -- 172.21.15.23:0/19900401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0041016b0 msgr2=0x7fe004197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe0037fe640 1 --2- 172.21.15.23:0/19900401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0041016b0 0x7fe004197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.558+0000 7fe0037fe640 1 -- 172.21.15.23:0/19900401 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdffc009900 con 0x7fe0041076c0 2024-03-24T10:49:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.559+0000 7fe002ffd640 1 --2- 172.21.15.23:0/19900401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0041016b0 0x7fe004197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.559+0000 7fe0037fe640 1 --2- 172.21.15.23:0/19900401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0041076c0 0x7fe00419c5f0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fdffc004af0 tx=0x7fdffc032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.559+0000 7fe008a09640 1 -- 172.21.15.23:0/19900401 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdffc0328b0 con 0x7fe0041076c0 2024-03-24T10:49:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.559+0000 7fe009a0b640 1 -- 172.21.15.23:0/19900401 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe00419d1a0 con 0x7fe0041076c0 2024-03-24T10:49:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.559+0000 7fe008a09640 1 -- 172.21.15.23:0/19900401 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdffc003e50 con 0x7fe0041076c0 2024-03-24T10:49:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.559+0000 7fe008a09640 1 -- 172.21.15.23:0/19900401 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdffc004740 con 0x7fe0041076c0 2024-03-24T10:49:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.559+0000 7fe009a0b640 1 -- 172.21.15.23:0/19900401 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe00419d470 con 0x7fe0041076c0 2024-03-24T10:49:25.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.561+0000 7fe008a09640 1 -- 172.21.15.23:0/19900401 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fdffc00a040 con 0x7fe0041076c0 2024-03-24T10:49:25.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe008a09640 1 --2- 172.21.15.23:0/19900401 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdff00a2230 0x7fdff00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe00c5e0740 1 Processor -- start 2024-03-24T10:49:25.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe00c5e0740 1 -- start start 2024-03-24T10:49:25.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe002ffd640 1 --2- 172.21.15.23:0/19900401 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdff00a2230 0x7fdff00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe008a09640 1 -- 172.21.15.23:0/19900401 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fdffc0f0b90 con 0x7fe0041076c0 2024-03-24T10:49:25.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe00c5e0740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77c760 0x55adaa77cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe00c5e0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa77d080 0x55adaa785720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe00c5e0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa785c60 0x55adaa788050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe00c5e0740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55adaa6ea530 con 0x55adaa77c760 2024-03-24T10:49:25.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe00c5e0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55adaa6e7a60 con 0x55adaa77d080 2024-03-24T10:49:25.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe00c5e0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55adaa6e3240 con 0x55adaa785c60 2024-03-24T10:49:25.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe002ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77c760 0x55adaa77cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe0027fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa77d080 0x55adaa785720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe002ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77c760 0x55adaa77cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46140/0 (socket says 172.21.15.23:46140) 2024-03-24T10:49:25.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe002ffd640 1 -- 172.21.15.23:0/4095433497 learned_addr learned my addr 172.21.15.23:0/4095433497 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:25.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe0037fe640 1 --2- 172.21.15.23:0/4095433497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa785c60 0x55adaa788050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe002ffd640 1 --2- 172.21.15.23:0/19900401 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdff00a2230 0x7fdff00a46f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009f90 tx=0x7fdfec011810 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:25.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe0027fc640 1 -- 172.21.15.23:0/4095433497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa785c60 msgr2=0x55adaa788050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe0027fc640 1 --2- 172.21.15.23:0/4095433497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa785c60 0x55adaa788050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe0027fc640 1 -- 172.21.15.23:0/4095433497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77c760 msgr2=0x55adaa77cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe0027fc640 1 --2- 172.21.15.23:0/4095433497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77c760 0x55adaa77cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe0027fc640 1 -- 172.21.15.23:0/4095433497 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adaa6e18c0 con 0x55adaa77d080 2024-03-24T10:49:25.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.562+0000 7fe002ffd640 1 --2- 172.21.15.23:0/4095433497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77c760 0x55adaa77cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:25.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.563+0000 7fe0027fc640 1 --2- 172.21.15.23:0/4095433497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa77d080 0x55adaa785720 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009bf0 tx=0x7fdff4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62fd15de1f5a99f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:25.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.563+0000 7fdff9ffb640 1 -- 172.21.15.23:0/4095433497 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4004050 con 0x55adaa77d080 2024-03-24T10:49:25.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.563+0000 7fdff9ffb640 1 -- 172.21.15.23:0/4095433497 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdff4005500 con 0x55adaa77d080 2024-03-24T10:49:25.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.563+0000 7fdff9ffb640 1 -- 172.21.15.23:0/4095433497 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4004cd0 con 0x55adaa77d080 2024-03-24T10:49:25.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.563+0000 7fe00c5e0740 1 -- 172.21.15.23:0/4095433497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa77d080 msgr2=0x55adaa785720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.563+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/4095433497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa77d080 0x55adaa785720 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009bf0 tx=0x7fdff4011810 comp rx=0 tx=0).stop 2024-03-24T10:49:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.563+0000 7fe00c5e0740 1 -- 172.21.15.23:0/4095433497 shutdown_connections 2024-03-24T10:49:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.563+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/4095433497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa785c60 0x55adaa788050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.563+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/4095433497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa77d080 0x55adaa785720 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.563+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/4095433497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77c760 0x55adaa77cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.563+0000 7fe00c5e0740 1 -- 172.21.15.23:0/4095433497 >> 172.21.15.23:0/4095433497 conn(0x55adaa776e90 msgr2=0x55adaa777270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.564+0000 7fe00c5e0740 1 -- 172.21.15.23:0/4095433497 shutdown_connections 2024-03-24T10:49:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.564+0000 7fe00c5e0740 1 -- 172.21.15.23:0/4095433497 wait complete. 2024-03-24T10:49:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.564+0000 7fe00c5e0740 1 -- start start 2024-03-24T10:49:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.564+0000 7fe00c5e0740 1 Processor -- start 2024-03-24T10:49:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.564+0000 7fe00c5e0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 0x55adaa782c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.564+0000 7fe00c5e0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa77d080 0x55adaa783180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.564+0000 7fe00c5e0740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa785c60 0x55adaa7836c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.564+0000 7fe00c5e0740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55adaa687f80 con 0x55adaa785c60 2024-03-24T10:49:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.564+0000 7fe00c5e0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55adaa3b64b0 con 0x55adaa77d080 2024-03-24T10:49:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.564+0000 7fe00c5e0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55adaa668ad0 con 0x55adaa77c760 2024-03-24T10:49:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.564+0000 7fe00c5e0740 10 client.0 _ll_register_callbacks cb 0x55adaa69bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.564+0000 7fe0037fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa785c60 0x55adaa7836c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.564+0000 7fe0037fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa785c60 0x55adaa7836c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46154/0 (socket says 172.21.15.23:46154) 2024-03-24T10:49:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.564+0000 7fe0037fe640 1 -- 172.21.15.23:0/4260981376 learned_addr learned my addr 172.21.15.23:0/4260981376 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fe002ffd640 1 --2- 172.21.15.23:0/4260981376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 0x55adaa782c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fe0027fc640 1 --2- 172.21.15.23:0/4260981376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa77d080 0x55adaa783180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fe0037fe640 1 -- 172.21.15.23:0/4260981376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 msgr2=0x55adaa782c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fe0037fe640 1 --2- 172.21.15.23:0/4260981376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 0x55adaa782c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fe0037fe640 1 -- 172.21.15.23:0/4260981376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa77d080 msgr2=0x55adaa783180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fe0037fe640 1 --2- 172.21.15.23:0/4260981376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa77d080 0x55adaa783180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fe0037fe640 1 -- 172.21.15.23:0/4260981376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdff4009850 con 0x55adaa785c60 2024-03-24T10:49:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fe0037fe640 1 --2- 172.21.15.23:0/4260981376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa785c60 0x55adaa7836c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0bd300 tx=0x7fdffc0b99d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57fbbaa33f6f417 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fdfe2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fdfe2ffd640 1 -- 172.21.15.23:0/4260981376 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdffc0bdd00 con 0x55adaa785c60 2024-03-24T10:49:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fdfe2ffd640 1 -- 172.21.15.23:0/4260981376 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fdffc0bd600 con 0x55adaa785c60 2024-03-24T10:49:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fe00c5e0740 1 -- 172.21.15.23:0/4260981376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55adaa7077a0 con 0x55adaa785c60 2024-03-24T10:49:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fdfe2ffd640 1 -- 172.21.15.23:0/4260981376 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdffc0c1750 con 0x55adaa785c60 2024-03-24T10:49:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fe00c5e0740 10 client.18174 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fe00c5e0740 1 -- 172.21.15.23:0/4260981376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55adaa6996d0 con 0x55adaa785c60 2024-03-24T10:49:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.565+0000 7fdfe2ffd640 1 -- 172.21.15.23:0/4260981376 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55adaa7077a0 con 0x55adaa785c60 2024-03-24T10:49:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.566+0000 7fdfe2ffd640 1 -- 172.21.15.23:0/4260981376 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fdffc0b9c70 con 0x55adaa785c60 2024-03-24T10:49:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.566+0000 7fe00c5e0740 10 client.18174 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.566+0000 7fe00c5e0740 10 client.18174 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.566+0000 7fe00c5e0740 20 client.18174 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.566+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/4260981376 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55adaa795af0 0x55adaa797f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.566+0000 7fe00c5e0740 4 client.18174 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:49:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.566+0000 7fe00c5e0740 1 -- 172.21.15.23:0/4260981376 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adaa6d91f0 con 0x55adaa795af0 2024-03-24T10:49:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.566+0000 7fe002ffd640 1 --2- 172.21.15.23:0/4260981376 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55adaa795af0 0x55adaa797f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.566+0000 7fe002ffd640 1 --2- 172.21.15.23:0/4260981376 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55adaa795af0 0x55adaa797f10 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd14edbb9db75f9c server_cookie=8b6b04d0106a7d9 in_seq=0 out_seq=0 2024-03-24T10:49:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.566+0000 7fdfe2ffd640 10 client.18174 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fdfe2ffd640 1 -- 172.21.15.23:0/4260981376 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55adaa6d91f0 con 0x55adaa795af0 2024-03-24T10:49:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fdfe2ffd640 10 client.18174 handle_command_reply: tid=0 2024-03-24T10:49:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fe00c5e0740 1 client.18174 shutdown 2024-03-24T10:49:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fe00c5e0740 1 -- 172.21.15.23:0/4260981376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa785c60 msgr2=0x55adaa7836c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/4260981376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa785c60 0x55adaa7836c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0bd300 tx=0x7fdffc0b99d0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fe00c5e0740 1 -- 172.21.15.23:0/4260981376 shutdown_connections 2024-03-24T10:49:25.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/4260981376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa785c60 0x55adaa7836c0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/4260981376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa77d080 0x55adaa783180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/4260981376 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55adaa795af0 0x55adaa797f10 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/4260981376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 0x55adaa782c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fdfe2ffd640 0 client.18174 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:25.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fe00c5e0740 1 -- 172.21.15.23:0/4260981376 >> 172.21.15.23:0/4260981376 conn(0x55adaa776e90 msgr2=0x55adaa778be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:25.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fe00c5e0740 1 -- 172.21.15.23:0/4260981376 shutdown_connections 2024-03-24T10:49:25.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fe00c5e0740 1 -- 172.21.15.23:0/4260981376 wait complete. 2024-03-24T10:49:25.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.567+0000 7fe00c5e0740 20 client.18174 trim_cache size 0 max 16384 2024-03-24T10:49:25.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe00c5e0740 1 Processor -- start 2024-03-24T10:49:25.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe00c5e0740 1 -- start start 2024-03-24T10:49:25.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe00c5e0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 0x55adaa785370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe00c5e0740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77d080 0x55adaa791520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe00c5e0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa785c60 0x55adaa775d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe00c5e0740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55adaa7985c0 con 0x55adaa77d080 2024-03-24T10:49:25.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe00c5e0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55adaa772650 con 0x55adaa785c60 2024-03-24T10:49:25.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe00c5e0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55adaa782a30 con 0x55adaa77c760 2024-03-24T10:49:25.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe0027fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77d080 0x55adaa791520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe0027fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77d080 0x55adaa791520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46166/0 (socket says 172.21.15.23:46166) 2024-03-24T10:49:25.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe0027fc640 1 -- 172.21.15.23:0/885304762 learned_addr learned my addr 172.21.15.23:0/885304762 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:25.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe002ffd640 1 --2- 172.21.15.23:0/885304762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 0x55adaa785370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe0037fe640 1 --2- 172.21.15.23:0/885304762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa785c60 0x55adaa775d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe0027fc640 1 -- 172.21.15.23:0/885304762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 msgr2=0x55adaa785370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe0027fc640 1 --2- 172.21.15.23:0/885304762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 0x55adaa785370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe0027fc640 1 -- 172.21.15.23:0/885304762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa785c60 msgr2=0x55adaa775d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe0027fc640 1 --2- 172.21.15.23:0/885304762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa785c60 0x55adaa775d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe0027fc640 1 -- 172.21.15.23:0/885304762 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adaa67dc60 con 0x55adaa77d080 2024-03-24T10:49:25.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.585+0000 7fe002ffd640 1 --2- 172.21.15.23:0/885304762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 0x55adaa785370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:25.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.586+0000 7fe0027fc640 1 --2- 172.21.15.23:0/885304762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77d080 0x55adaa791520 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fdff4014040 tx=0x7fdff4009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c1d0192b85ab7e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:25.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.586+0000 7fdff97fa640 1 -- 172.21.15.23:0/885304762 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4009ee0 con 0x55adaa77d080 2024-03-24T10:49:25.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.586+0000 7fdff97fa640 1 -- 172.21.15.23:0/885304762 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdff4004620 con 0x55adaa77d080 2024-03-24T10:49:25.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.586+0000 7fdff97fa640 1 -- 172.21.15.23:0/885304762 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff40049e0 con 0x55adaa77d080 2024-03-24T10:49:25.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.586+0000 7fe00c5e0740 1 -- 172.21.15.23:0/885304762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77d080 msgr2=0x55adaa791520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.586+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/885304762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77d080 0x55adaa791520 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fdff4014040 tx=0x7fdff4009380 comp rx=0 tx=0).stop 2024-03-24T10:49:25.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.586+0000 7fe00c5e0740 1 -- 172.21.15.23:0/885304762 shutdown_connections 2024-03-24T10:49:25.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.586+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/885304762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa785c60 0x55adaa775d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.586+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/885304762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77d080 0x55adaa791520 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.586+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/885304762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 0x55adaa785370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.586+0000 7fe00c5e0740 1 -- 172.21.15.23:0/885304762 >> 172.21.15.23:0/885304762 conn(0x55adaa776e90 msgr2=0x55adaa775660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:25.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.586+0000 7fe00c5e0740 1 -- 172.21.15.23:0/885304762 shutdown_connections 2024-03-24T10:49:25.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.586+0000 7fe00c5e0740 1 -- 172.21.15.23:0/885304762 wait complete. 2024-03-24T10:49:25.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.587+0000 7fe00c5e0740 1 -- start start 2024-03-24T10:49:25.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.587+0000 7fe00c5e0740 1 Processor -- start 2024-03-24T10:49:25.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.587+0000 7fe00c5e0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 0x55adaa78dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.587+0000 7fe00c5e0740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77d080 0x55adaa78ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.587+0000 7fe00c5e0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa785c60 0x55adaa78b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.587+0000 7fe00c5e0740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55adaa78b5a0 con 0x55adaa77d080 2024-03-24T10:49:25.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.587+0000 7fe00c5e0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55adaa78b710 con 0x55adaa785c60 2024-03-24T10:49:25.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.587+0000 7fe00c5e0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55adaa78b8d0 con 0x55adaa77c760 2024-03-24T10:49:25.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.587+0000 7fe00c5e0740 10 client.0 _ll_register_callbacks cb 0x7fdffc0b9900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:25.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.587+0000 7fe0027fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77d080 0x55adaa78ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.587+0000 7fe0027fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77d080 0x55adaa78ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46182/0 (socket says 172.21.15.23:46182) 2024-03-24T10:49:25.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.587+0000 7fe0027fc640 1 -- 172.21.15.23:0/3579979268 learned_addr learned my addr 172.21.15.23:0/3579979268 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:25.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.587+0000 7fe002ffd640 1 --2- 172.21.15.23:0/3579979268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 0x55adaa78dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.587+0000 7fe0037fe640 1 --2- 172.21.15.23:0/3579979268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa785c60 0x55adaa78b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fe0027fc640 1 -- 172.21.15.23:0/3579979268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 msgr2=0x55adaa78dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fe0027fc640 1 --2- 172.21.15.23:0/3579979268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 0x55adaa78dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fe0027fc640 1 -- 172.21.15.23:0/3579979268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa785c60 msgr2=0x55adaa78b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fe0027fc640 1 --2- 172.21.15.23:0/3579979268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa785c60 0x55adaa78b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fe0027fc640 1 -- 172.21.15.23:0/3579979268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfec0044b0 con 0x55adaa77d080 2024-03-24T10:49:25.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fe0037fe640 1 --2- 172.21.15.23:0/3579979268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa785c60 0x55adaa78b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:25.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fe0027fc640 1 --2- 172.21.15.23:0/3579979268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77d080 0x55adaa78ab20 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fdff401b040 tx=0x7fdff4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c70e76e7f11d6dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:25.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fdfe2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:25.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fdfe2ffd640 1 -- 172.21.15.23:0/3579979268 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4004010 con 0x55adaa77d080 2024-03-24T10:49:25.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fdfe2ffd640 1 -- 172.21.15.23:0/3579979268 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fdff4027070 con 0x55adaa77d080 2024-03-24T10:49:25.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fe00c5e0740 1 -- 172.21.15.23:0/3579979268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdff4005500 con 0x55adaa77d080 2024-03-24T10:49:25.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fdfe2ffd640 1 -- 172.21.15.23:0/3579979268 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdff4004170 con 0x55adaa77d080 2024-03-24T10:49:25.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fe00c5e0740 10 client.18180 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:25.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fe00c5e0740 1 -- 172.21.15.23:0/3579979268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55adaa6e18c0 con 0x55adaa77d080 2024-03-24T10:49:25.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fdfe2ffd640 1 -- 172.21.15.23:0/3579979268 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdff4005500 con 0x55adaa77d080 2024-03-24T10:49:25.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fdfe2ffd640 1 -- 172.21.15.23:0/3579979268 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fdff4021040 con 0x55adaa77d080 2024-03-24T10:49:25.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fe00c5e0740 10 client.18180 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:25.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fe00c5e0740 10 client.18180 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:25.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.588+0000 7fe00c5e0740 20 client.18180 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:25.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.589+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/3579979268 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55adaa7fd0a0 0x55adaa7ff490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.589+0000 7fe00c5e0740 4 client.18180 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:49:25.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.589+0000 7fe00c5e0740 1 -- 172.21.15.23:0/3579979268 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55adaa3e2860 con 0x55adaa7fd0a0 2024-03-24T10:49:25.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.589+0000 7fe002ffd640 1 --2- 172.21.15.23:0/3579979268 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55adaa7fd0a0 0x55adaa7ff490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.589+0000 7fdfe2ffd640 10 client.18180 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:25.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.589+0000 7fe002ffd640 1 --2- 172.21.15.23:0/3579979268 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55adaa7fd0a0 0x55adaa7ff490 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5f6ccbdc41fe54c server_cookie=7f4b88dc72e0300e in_seq=0 out_seq=0 2024-03-24T10:49:25.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.590+0000 7fdfe2ffd640 1 -- 172.21.15.23:0/3579979268 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55adaa3e2860 con 0x55adaa7fd0a0 2024-03-24T10:49:25.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.590+0000 7fdfe2ffd640 10 client.18180 handle_command_reply: tid=0 2024-03-24T10:49:25.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.590+0000 7fe00c5e0740 1 client.18180 shutdown 2024-03-24T10:49:25.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.590+0000 7fe00c5e0740 1 -- 172.21.15.23:0/3579979268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77d080 msgr2=0x55adaa78ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.590+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/3579979268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77d080 0x55adaa78ab20 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fdff401b040 tx=0x7fdff4011810 comp rx=0 tx=0).stop 2024-03-24T10:49:25.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.590+0000 7fe00c5e0740 1 -- 172.21.15.23:0/3579979268 shutdown_connections 2024-03-24T10:49:25.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.590+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/3579979268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55adaa785c60 0x55adaa78b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.590+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/3579979268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55adaa77d080 0x55adaa78ab20 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.590+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/3579979268 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55adaa7fd0a0 0x55adaa7ff490 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.590+0000 7fe00c5e0740 1 --2- 172.21.15.23:0/3579979268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55adaa77c760 0x55adaa78dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.590+0000 7fdfe2ffd640 0 client.18180 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:25.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.590+0000 7fe00c5e0740 1 -- 172.21.15.23:0/3579979268 >> 172.21.15.23:0/3579979268 conn(0x55adaa776e90 msgr2=0x55adaa783450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:25.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.590+0000 7fe00c5e0740 1 -- 172.21.15.23:0/3579979268 shutdown_connections 2024-03-24T10:49:25.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.591+0000 7fe00c5e0740 1 -- 172.21.15.23:0/3579979268 wait complete. 2024-03-24T10:49:25.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.591+0000 7fe00c5e0740 20 client.18180 trim_cache size 0 max 16384 2024-03-24T10:49:25.607 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:49:25.607 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:49:25.607 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "e2d86ebb-fc4b-4bfa-8a57-2197aa002187", 2024-03-24T10:49:25.607 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:49:25.608 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:49:25.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.591+0000 7fe009a0b640 1 -- 172.21.15.23:0/19900401 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdff00a2230 msgr2=0x7fdff00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.591+0000 7fe009a0b640 1 --2- 172.21.15.23:0/19900401 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdff00a2230 0x7fdff00a46f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009f90 tx=0x7fdfec011810 comp rx=0 tx=0).stop 2024-03-24T10:49:25.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.591+0000 7fe009a0b640 1 -- 172.21.15.23:0/19900401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0041076c0 msgr2=0x7fe00419c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.591+0000 7fe009a0b640 1 --2- 172.21.15.23:0/19900401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0041076c0 0x7fe00419c5f0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fdffc004af0 tx=0x7fdffc032570 comp rx=0 tx=0).stop 2024-03-24T10:49:25.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.591+0000 7fe009a0b640 1 -- 172.21.15.23:0/19900401 shutdown_connections 2024-03-24T10:49:25.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.591+0000 7fe009a0b640 1 --2- 172.21.15.23:0/19900401 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fdff00a2230 0x7fdff00a46f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.591+0000 7fe009a0b640 1 --2- 172.21.15.23:0/19900401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0041076c0 0x7fe00419c5f0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.591+0000 7fe009a0b640 1 --2- 172.21.15.23:0/19900401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe004102070 0x7fe004198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.591+0000 7fe009a0b640 1 --2- 172.21.15.23:0/19900401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0041016b0 0x7fe004197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.591+0000 7fe009a0b640 1 -- 172.21.15.23:0/19900401 >> 172.21.15.23:0/19900401 conn(0x7fe0040fd140 msgr2=0x7fe0040fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:25.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.591+0000 7fe009a0b640 1 -- 172.21.15.23:0/19900401 shutdown_connections 2024-03-24T10:49:25.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.591+0000 7fe009a0b640 1 -- 172.21.15.23:0/19900401 wait complete. 2024-03-24T10:49:25.611 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:49:25.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:49:25.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.715+0000 7fea94925640 1 -- 172.21.15.23:0/1624932998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea901032a0 msgr2=0x7fea9010b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.715+0000 7fea94925640 1 --2- 172.21.15.23:0/1624932998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea901032a0 0x7fea9010b780 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fea78009900 tx=0x7fea7802bcd0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.715+0000 7fea94925640 1 -- 172.21.15.23:0/1624932998 shutdown_connections 2024-03-24T10:49:25.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.715+0000 7fea94925640 1 --2- 172.21.15.23:0/1624932998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea9010bd50 0x7fea9010e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.715+0000 7fea94925640 1 --2- 172.21.15.23:0/1624932998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea901032a0 0x7fea9010b780 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.715+0000 7fea94925640 1 --2- 172.21.15.23:0/1624932998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea90102980 0x7fea90102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.715+0000 7fea94925640 1 -- 172.21.15.23:0/1624932998 >> 172.21.15.23:0/1624932998 conn(0x7fea90077770 msgr2=0x7fea90077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:25.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.715+0000 7fea94925640 1 -- 172.21.15.23:0/1624932998 shutdown_connections 2024-03-24T10:49:25.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.715+0000 7fea94925640 1 -- 172.21.15.23:0/1624932998 wait complete. 2024-03-24T10:49:25.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.716+0000 7fea94925640 1 Processor -- start 2024-03-24T10:49:25.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.716+0000 7fea94925640 1 -- start start 2024-03-24T10:49:25.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.716+0000 7fea94925640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea90102980 0x7fea90197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.716+0000 7fea94925640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea901032a0 0x7fea901982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.716+0000 7fea94925640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea9010bd50 0x7fea9019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.716+0000 7fea94925640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fea9019cc40 con 0x7fea90102980 2024-03-24T10:49:25.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.716+0000 7fea94925640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fea9019cdb0 con 0x7fea901032a0 2024-03-24T10:49:25.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.716+0000 7fea94925640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fea9019cf70 con 0x7fea9010bd50 2024-03-24T10:49:25.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.716+0000 7fea8e59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea90102980 0x7fea90197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.716+0000 7fea8e59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea90102980 0x7fea90197d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46192/0 (socket says 172.21.15.23:46192) 2024-03-24T10:49:25.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.716+0000 7fea8e59c640 1 -- 172.21.15.23:0/1452261791 learned_addr learned my addr 172.21.15.23:0/1452261791 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:25.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.716+0000 7fea87fff640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea901032a0 0x7fea901982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.716+0000 7fea8ed9d640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea9010bd50 0x7fea9019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.717+0000 7fea8e59c640 1 -- 172.21.15.23:0/1452261791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea9010bd50 msgr2=0x7fea9019c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.717+0000 7fea8e59c640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea9010bd50 0x7fea9019c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.717+0000 7fea8e59c640 1 -- 172.21.15.23:0/1452261791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea901032a0 msgr2=0x7fea901982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.717+0000 7fea8e59c640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea901032a0 0x7fea901982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.717+0000 7fea8e59c640 1 -- 172.21.15.23:0/1452261791 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea78009560 con 0x7fea90102980 2024-03-24T10:49:25.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.717+0000 7fea87fff640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea901032a0 0x7fea901982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:25.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.717+0000 7fea8e59c640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea90102980 0x7fea90197d80 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fea8000e9f0 tx=0x7fea8000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:25.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.717+0000 7fea877fe640 1 -- 172.21.15.23:0/1452261791 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80009720 con 0x7fea90102980 2024-03-24T10:49:25.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.717+0000 7fea94925640 1 -- 172.21.15.23:0/1452261791 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea9019d250 con 0x7fea90102980 2024-03-24T10:49:25.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.717+0000 7fea877fe640 1 -- 172.21.15.23:0/1452261791 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea8000cd80 con 0x7fea90102980 2024-03-24T10:49:25.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.717+0000 7fea877fe640 1 -- 172.21.15.23:0/1452261791 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80010690 con 0x7fea90102980 2024-03-24T10:49:25.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.717+0000 7fea94925640 1 -- 172.21.15.23:0/1452261791 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea901a4ad0 con 0x7fea90102980 2024-03-24T10:49:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.719+0000 7fea877fe640 1 -- 172.21.15.23:0/1452261791 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fea800046c0 con 0x7fea90102980 2024-03-24T10:49:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.719+0000 7fea975e6740 1 Processor -- start 2024-03-24T10:49:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea975e6740 1 -- start start 2024-03-24T10:49:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea975e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5e1d0 0x55e485c5e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea975e6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 0x55e485c67190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea975e6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c676d0 0x55e485c69ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea975e6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e485b82f30 con 0x55e485c676d0 2024-03-24T10:49:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea975e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e485bcc810 con 0x55e485c5e1d0 2024-03-24T10:49:25.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea975e6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e485bc9d40 con 0x55e485c5eaf0 2024-03-24T10:49:25.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea877fe640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fea7c0a2230 0x7fea7c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea87fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c676d0 0x55e485c69ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea8ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5e1d0 0x55e485c5e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea8e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 0x55e485c67190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea87fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c676d0 0x55e485c69ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46200/0 (socket says 172.21.15.23:46200) 2024-03-24T10:49:25.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea87fff640 1 -- 172.21.15.23:0/260885494 learned_addr learned my addr 172.21.15.23:0/260885494 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:25.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea8ed9d640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fea7c0a2230 0x7fea7c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea877fe640 1 -- 172.21.15.23:0/1452261791 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fea80014070 con 0x7fea90102980 2024-03-24T10:49:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea87fff640 1 -- 172.21.15.23:0/260885494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 msgr2=0x55e485c67190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea87fff640 1 --2- 172.21.15.23:0/260885494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 0x55e485c67190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea87fff640 1 -- 172.21.15.23:0/260885494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5e1d0 msgr2=0x55e485c5e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea87fff640 1 --2- 172.21.15.23:0/260885494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5e1d0 0x55e485c5e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea87fff640 1 -- 172.21.15.23:0/260885494 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e485b86ea0 con 0x55e485c676d0 2024-03-24T10:49:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea8ed9d640 1 --2- 172.21.15.23:0/260885494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5e1d0 0x55e485c5e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea87fff640 1 --2- 172.21.15.23:0/260885494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c676d0 0x55e485c69ac0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fea78009900 tx=0x7fea7802c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03ecd30f529d314 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.720+0000 7fea8ed9d640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fea7c0a2230 0x7fea7c0a46f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fea88005fd0 tx=0x7fea88009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.721+0000 7fea84ff9640 1 -- 172.21.15.23:0/260885494 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7802c880 con 0x55e485c676d0 2024-03-24T10:49:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.721+0000 7fea84ff9640 1 -- 172.21.15.23:0/260885494 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea7802c9e0 con 0x55e485c676d0 2024-03-24T10:49:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.721+0000 7fea84ff9640 1 -- 172.21.15.23:0/260885494 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea780054d0 con 0x55e485c676d0 2024-03-24T10:49:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.721+0000 7fea975e6740 1 -- 172.21.15.23:0/260885494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c676d0 msgr2=0x55e485c69ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.721+0000 7fea975e6740 1 --2- 172.21.15.23:0/260885494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c676d0 0x55e485c69ac0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fea78009900 tx=0x7fea7802c720 comp rx=0 tx=0).stop 2024-03-24T10:49:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.721+0000 7fea975e6740 1 -- 172.21.15.23:0/260885494 shutdown_connections 2024-03-24T10:49:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.721+0000 7fea975e6740 1 --2- 172.21.15.23:0/260885494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c676d0 0x55e485c69ac0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.721+0000 7fea975e6740 1 --2- 172.21.15.23:0/260885494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 0x55e485c67190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.721+0000 7fea975e6740 1 --2- 172.21.15.23:0/260885494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5e1d0 0x55e485c5e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.721+0000 7fea975e6740 1 -- 172.21.15.23:0/260885494 >> 172.21.15.23:0/260885494 conn(0x55e485c58900 msgr2=0x55e485c58ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.721+0000 7fea975e6740 1 -- 172.21.15.23:0/260885494 shutdown_connections 2024-03-24T10:49:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.721+0000 7fea975e6740 1 -- 172.21.15.23:0/260885494 wait complete. 2024-03-24T10:49:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea975e6740 1 -- start start 2024-03-24T10:49:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea975e6740 1 Processor -- start 2024-03-24T10:49:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea975e6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c5e1d0 0x55e485c646e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea975e6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 0x55e485c64c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea975e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c676d0 0x55e485c65160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea975e6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e485bc5520 con 0x55e485c5e1d0 2024-03-24T10:49:25.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea975e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e485898430 con 0x55e485c676d0 2024-03-24T10:49:25.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea975e6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e485854e00 con 0x55e485c5eaf0 2024-03-24T10:49:25.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea975e6740 10 client.0 _ll_register_callbacks cb 0x55e485b63d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:25.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea8e59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c5e1d0 0x55e485c646e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea8e59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c5e1d0 0x55e485c646e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46216/0 (socket says 172.21.15.23:46216) 2024-03-24T10:49:25.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea8e59c640 1 -- 172.21.15.23:0/4282673146 learned_addr learned my addr 172.21.15.23:0/4282673146 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:25.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea87fff640 1 --2- 172.21.15.23:0/4282673146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 0x55e485c64c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea8ed9d640 1 --2- 172.21.15.23:0/4282673146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c676d0 0x55e485c65160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea8e59c640 1 -- 172.21.15.23:0/4282673146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 msgr2=0x55e485c64c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea8e59c640 1 --2- 172.21.15.23:0/4282673146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 0x55e485c64c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea8e59c640 1 -- 172.21.15.23:0/4282673146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c676d0 msgr2=0x55e485c65160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea8e59c640 1 --2- 172.21.15.23:0/4282673146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c676d0 0x55e485c65160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.722+0000 7fea8e59c640 1 -- 172.21.15.23:0/4282673146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea9010ee40 con 0x55e485c5e1d0 2024-03-24T10:49:25.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea8e59c640 1 --2- 172.21.15.23:0/4282673146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c5e1d0 0x55e485c646e0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fea80097260 tx=0x7fea80066460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd68db10286c37df server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:25.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:25.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea75ffb640 1 -- 172.21.15.23:0/4282673146 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8009d030 con 0x55e485c5e1d0 2024-03-24T10:49:25.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea75ffb640 1 -- 172.21.15.23:0/4282673146 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fea8009e070 con 0x55e485c5e1d0 2024-03-24T10:49:25.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea975e6740 1 -- 172.21.15.23:0/4282673146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e4858c47e0 con 0x55e485c5e1d0 2024-03-24T10:49:25.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea75ffb640 1 -- 172.21.15.23:0/4282673146 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fea8009ae00 con 0x55e485c5e1d0 2024-03-24T10:49:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea975e6740 10 client.18192 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea975e6740 1 -- 172.21.15.23:0/4282673146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e485bc3ba0 con 0x55e485c5e1d0 2024-03-24T10:49:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea75ffb640 1 -- 172.21.15.23:0/4282673146 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e4858c47e0 con 0x55e485c5e1d0 2024-03-24T10:49:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea75ffb640 1 -- 172.21.15.23:0/4282673146 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fea800cd070 con 0x55e485c5e1d0 2024-03-24T10:49:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea975e6740 10 client.18192 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea975e6740 10 client.18192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea975e6740 20 client.18192 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea975e6740 1 --2- 172.21.15.23:0/4282673146 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e485c779e0 0x55e485c79dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea975e6740 4 client.18192 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:49:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea975e6740 1 -- 172.21.15.23:0/4282673146 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e485901720 con 0x55e485c779e0 2024-03-24T10:49:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.723+0000 7fea87fff640 1 --2- 172.21.15.23:0/4282673146 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e485c779e0 0x55e485c79dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.724+0000 7fea87fff640 1 --2- 172.21.15.23:0/4282673146 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e485c779e0 0x55e485c79dd0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5345e151d48173b8 server_cookie=be95298c6b63b8cb in_seq=0 out_seq=0 2024-03-24T10:49:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.724+0000 7fea75ffb640 10 client.18192 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.724+0000 7fea75ffb640 1 -- 172.21.15.23:0/4282673146 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e485901720 con 0x55e485c779e0 2024-03-24T10:49:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.724+0000 7fea75ffb640 10 client.18192 handle_command_reply: tid=0 2024-03-24T10:49:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.724+0000 7fea975e6740 1 client.18192 shutdown 2024-03-24T10:49:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.725+0000 7fea975e6740 1 -- 172.21.15.23:0/4282673146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c5e1d0 msgr2=0x55e485c646e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.725+0000 7fea975e6740 1 --2- 172.21.15.23:0/4282673146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c5e1d0 0x55e485c646e0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fea80097260 tx=0x7fea80066460 comp rx=0 tx=0).stop 2024-03-24T10:49:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.725+0000 7fea975e6740 1 -- 172.21.15.23:0/4282673146 shutdown_connections 2024-03-24T10:49:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.725+0000 7fea975e6740 1 --2- 172.21.15.23:0/4282673146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c676d0 0x55e485c65160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.725+0000 7fea975e6740 1 --2- 172.21.15.23:0/4282673146 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e485c779e0 0x55e485c79dd0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.725+0000 7fea975e6740 1 --2- 172.21.15.23:0/4282673146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 0x55e485c64c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.725+0000 7fea975e6740 1 --2- 172.21.15.23:0/4282673146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c5e1d0 0x55e485c646e0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.725+0000 7fea75ffb640 0 client.18192 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:25.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.725+0000 7fea975e6740 1 -- 172.21.15.23:0/4282673146 >> 172.21.15.23:0/4282673146 conn(0x55e485c58900 msgr2=0x55e485c57b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:25.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.725+0000 7fea975e6740 1 -- 172.21.15.23:0/4282673146 shutdown_connections 2024-03-24T10:49:25.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.725+0000 7fea975e6740 1 -- 172.21.15.23:0/4282673146 wait complete. 2024-03-24T10:49:25.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.725+0000 7fea975e6740 20 client.18192 trim_cache size 0 max 16384 2024-03-24T10:49:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.742+0000 7fea975e6740 1 Processor -- start 2024-03-24T10:49:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.742+0000 7fea975e6740 1 -- start start 2024-03-24T10:49:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.742+0000 7fea975e6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c5e1d0 0x55e485c66e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.742+0000 7fea975e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5eaf0 0x55e485c64f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.742+0000 7fea975e6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c676d0 0x55e485c7f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.742+0000 7fea975e6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e485b4a430 con 0x55e485c5e1d0 2024-03-24T10:49:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.742+0000 7fea975e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e485c59900 con 0x55e485c5eaf0 2024-03-24T10:49:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.742+0000 7fea975e6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e485c5dc10 con 0x55e485c676d0 2024-03-24T10:49:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.742+0000 7fea8e59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c5e1d0 0x55e485c66e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.742+0000 7fea8e59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c5e1d0 0x55e485c66e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46232/0 (socket says 172.21.15.23:46232) 2024-03-24T10:49:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.742+0000 7fea8e59c640 1 -- 172.21.15.23:0/2731776821 learned_addr learned my addr 172.21.15.23:0/2731776821 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.742+0000 7fea8ed9d640 1 --2- 172.21.15.23:0/2731776821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c676d0 0x55e485c7f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.742+0000 7fea87fff640 1 --2- 172.21.15.23:0/2731776821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5eaf0 0x55e485c64f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea8e59c640 1 -- 172.21.15.23:0/2731776821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c676d0 msgr2=0x55e485c7f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea8e59c640 1 --2- 172.21.15.23:0/2731776821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c676d0 0x55e485c7f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea8e59c640 1 -- 172.21.15.23:0/2731776821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5eaf0 msgr2=0x55e485c64f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea8e59c640 1 --2- 172.21.15.23:0/2731776821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5eaf0 0x55e485c64f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea8e59c640 1 -- 172.21.15.23:0/2731776821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e485b5fb20 con 0x55e485c5e1d0 2024-03-24T10:49:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea87fff640 1 --2- 172.21.15.23:0/2731776821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5eaf0 0x55e485c64f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:49:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea8e59c640 1 --2- 172.21.15.23:0/2731776821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c5e1d0 0x55e485c66e10 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fea80004ae0 tx=0x7fea80005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=674742e651d6baaa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea77fff640 1 -- 172.21.15.23:0/2731776821 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8009a570 con 0x55e485c5e1d0 2024-03-24T10:49:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea77fff640 1 -- 172.21.15.23:0/2731776821 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea800a3380 con 0x55e485c5e1d0 2024-03-24T10:49:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea77fff640 1 -- 172.21.15.23:0/2731776821 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80099f20 con 0x55e485c5e1d0 2024-03-24T10:49:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea975e6740 1 -- 172.21.15.23:0/2731776821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c5e1d0 msgr2=0x55e485c66e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea975e6740 1 --2- 172.21.15.23:0/2731776821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c5e1d0 0x55e485c66e10 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fea80004ae0 tx=0x7fea80005e30 comp rx=0 tx=0).stop 2024-03-24T10:49:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea975e6740 1 -- 172.21.15.23:0/2731776821 shutdown_connections 2024-03-24T10:49:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea975e6740 1 --2- 172.21.15.23:0/2731776821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c676d0 0x55e485c7f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea975e6740 1 --2- 172.21.15.23:0/2731776821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5eaf0 0x55e485c64f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea975e6740 1 --2- 172.21.15.23:0/2731776821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c5e1d0 0x55e485c66e10 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea975e6740 1 -- 172.21.15.23:0/2731776821 >> 172.21.15.23:0/2731776821 conn(0x55e485c58900 msgr2=0x55e485c59b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:25.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.743+0000 7fea975e6740 1 -- 172.21.15.23:0/2731776821 shutdown_connections 2024-03-24T10:49:25.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea975e6740 1 -- 172.21.15.23:0/2731776821 wait complete. 2024-03-24T10:49:25.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea975e6740 1 -- start start 2024-03-24T10:49:25.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea975e6740 1 Processor -- start 2024-03-24T10:49:25.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea975e6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5e1d0 0x55e485c6b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea975e6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 0x55e485c6b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea975e6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c676d0 0x55e485cdccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea975e6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e485c6bf40 con 0x55e485c676d0 2024-03-24T10:49:25.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea975e6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e485cdd200 con 0x55e485c5e1d0 2024-03-24T10:49:25.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea975e6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e485cdd340 con 0x55e485c5eaf0 2024-03-24T10:49:25.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea975e6740 10 client.0 _ll_register_callbacks cb 0x7fea8000cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:25.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea8ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c676d0 0x55e485cdccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea8e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5e1d0 0x55e485c6b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea87fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 0x55e485c6b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea8ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c676d0 0x55e485cdccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46234/0 (socket says 172.21.15.23:46234) 2024-03-24T10:49:25.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea8e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5e1d0 0x55e485c6b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:53456/0 (socket says 172.21.15.23:53456) 2024-03-24T10:49:25.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea8ed9d640 1 -- 172.21.15.23:0/3714744371 learned_addr learned my addr 172.21.15.23:0/3714744371 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:25.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea8ed9d640 1 -- 172.21.15.23:0/3714744371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 msgr2=0x55e485c6b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea8ed9d640 1 --2- 172.21.15.23:0/3714744371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 0x55e485c6b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea8ed9d640 1 -- 172.21.15.23:0/3714744371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5e1d0 msgr2=0x55e485c6b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea8ed9d640 1 --2- 172.21.15.23:0/3714744371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5e1d0 0x55e485c6b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.744+0000 7fea8ed9d640 1 -- 172.21.15.23:0/3714744371 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e485bc3ba0 con 0x55e485c676d0 2024-03-24T10:49:25.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea8ed9d640 1 --2- 172.21.15.23:0/3714744371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c676d0 0x55e485cdccc0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fea8801f9f0 tx=0x7fea88004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98718f932d4e630d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:25.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:25.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea75ffb640 1 -- 172.21.15.23:0/3714744371 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8802e070 con 0x55e485c676d0 2024-03-24T10:49:25.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea75ffb640 1 -- 172.21.15.23:0/3714744371 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fea88004b20 con 0x55e485c676d0 2024-03-24T10:49:25.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea975e6740 1 -- 172.21.15.23:0/3714744371 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea7802c9e0 con 0x55e485c676d0 2024-03-24T10:49:25.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea75ffb640 1 -- 172.21.15.23:0/3714744371 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fea88025040 con 0x55e485c676d0 2024-03-24T10:49:25.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea975e6740 10 client.18198 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:25.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea975e6740 1 -- 172.21.15.23:0/3714744371 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e485b85700 con 0x55e485c676d0 2024-03-24T10:49:25.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea75ffb640 1 -- 172.21.15.23:0/3714744371 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea7802c9e0 con 0x55e485c676d0 2024-03-24T10:49:25.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea75ffb640 1 -- 172.21.15.23:0/3714744371 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fea88031020 con 0x55e485c676d0 2024-03-24T10:49:25.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea975e6740 10 client.18198 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:25.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea975e6740 10 client.18198 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:25.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea975e6740 20 client.18198 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:25.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea975e6740 1 --2- 172.21.15.23:0/3714744371 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e485cdebf0 0x55e485ce1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea975e6740 4 client.18198 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:49:25.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea975e6740 1 -- 172.21.15.23:0/3714744371 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e485950890 con 0x55e485cdebf0 2024-03-24T10:49:25.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.745+0000 7fea8e59c640 1 --2- 172.21.15.23:0/3714744371 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e485cdebf0 0x55e485ce1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea8e59c640 1 --2- 172.21.15.23:0/3714744371 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e485cdebf0 0x55e485ce1010 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4b6611d906df82d server_cookie=7aaa2b438955d214 in_seq=0 out_seq=0 2024-03-24T10:49:25.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea75ffb640 10 client.18198 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:25.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea75ffb640 1 -- 172.21.15.23:0/3714744371 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e485950890 con 0x55e485cdebf0 2024-03-24T10:49:25.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea75ffb640 10 client.18198 handle_command_reply: tid=0 2024-03-24T10:49:25.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea975e6740 1 client.18198 shutdown 2024-03-24T10:49:25.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea975e6740 1 -- 172.21.15.23:0/3714744371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c676d0 msgr2=0x55e485cdccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea975e6740 1 --2- 172.21.15.23:0/3714744371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c676d0 0x55e485cdccc0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fea8801f9f0 tx=0x7fea88004ab0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea975e6740 1 -- 172.21.15.23:0/3714744371 shutdown_connections 2024-03-24T10:49:25.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea975e6740 1 --2- 172.21.15.23:0/3714744371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e485c676d0 0x55e485cdccc0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea975e6740 1 --2- 172.21.15.23:0/3714744371 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55e485cdebf0 0x55e485ce1010 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea975e6740 1 --2- 172.21.15.23:0/3714744371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e485c5eaf0 0x55e485c6b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea75ffb640 0 client.18198 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:25.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea975e6740 1 --2- 172.21.15.23:0/3714744371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e485c5e1d0 0x55e485c6b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea975e6740 1 -- 172.21.15.23:0/3714744371 >> 172.21.15.23:0/3714744371 conn(0x55e485c58900 msgr2=0x55e485c5a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:25.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea975e6740 1 -- 172.21.15.23:0/3714744371 shutdown_connections 2024-03-24T10:49:25.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea975e6740 1 -- 172.21.15.23:0/3714744371 wait complete. 2024-03-24T10:49:25.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.746+0000 7fea975e6740 20 client.18198 trim_cache size 0 max 16384 2024-03-24T10:49:25.762 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:49:25.762 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:49:25.762 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:49:25.762 INFO:teuthology.orchestra.run.smithi023.stdout: "e2d86ebb-fc4b-4bfa-8a57-2197aa002187": { 2024-03-24T10:49:25.762 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:49:25.762 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "e2d86ebb-fc4b-4bfa-8a57-2197aa002187", 2024-03-24T10:49:25.763 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:49:25.763 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:49:25.763 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:49:25.763 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:49:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.747+0000 7fea94925640 1 -- 172.21.15.23:0/1452261791 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fea7c0a2230 msgr2=0x7fea7c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.747+0000 7fea94925640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fea7c0a2230 0x7fea7c0a46f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fea88005fd0 tx=0x7fea88009500 comp rx=0 tx=0).stop 2024-03-24T10:49:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.747+0000 7fea94925640 1 -- 172.21.15.23:0/1452261791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea90102980 msgr2=0x7fea90197d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.747+0000 7fea94925640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea90102980 0x7fea90197d80 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fea8000e9f0 tx=0x7fea8000edf0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.747+0000 7fea94925640 1 -- 172.21.15.23:0/1452261791 shutdown_connections 2024-03-24T10:49:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.747+0000 7fea94925640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fea7c0a2230 0x7fea7c0a46f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.747+0000 7fea94925640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea9010bd50 0x7fea9019c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.747+0000 7fea94925640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea901032a0 0x7fea901982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.747+0000 7fea94925640 1 --2- 172.21.15.23:0/1452261791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea90102980 0x7fea90197d80 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.747+0000 7fea94925640 1 -- 172.21.15.23:0/1452261791 >> 172.21.15.23:0/1452261791 conn(0x7fea90077770 msgr2=0x7fea900fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:25.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.747+0000 7fea94925640 1 -- 172.21.15.23:0/1452261791 shutdown_connections 2024-03-24T10:49:25.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:25.747+0000 7fea94925640 1 -- 172.21.15.23:0/1452261791 wait complete. 2024-03-24T10:49:25.766 INFO:tasks.cephfs.filesystem:scrub status for tag:e2d86ebb-fc4b-4bfa-8a57-2197aa002187 - {'path': '/', 'tag': 'e2d86ebb-fc4b-4bfa-8a57-2197aa002187', 'options': 'recursive,force'} 2024-03-24T10:49:26.404 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:26 smithi117 ceph-mon[36368]: pgmap v291: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-03-24T10:49:26.404 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:26 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:49:26.404 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:26 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:26.404 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:26 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:49:26.404 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:26 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:49:26.404 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:26 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:26.405 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:26 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:26.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:26 smithi023 ceph-mon[32466]: pgmap v291: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-03-24T10:49:26.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:26 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:49:26.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:26 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:26.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:26 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:49:26.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:26 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:49:26.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:26 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:26.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:26 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:26.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:26 smithi116 ceph-mon[36248]: pgmap v291: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-03-24T10:49:26.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:26 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:49:26.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:26 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:26.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:26 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:49:26.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:26 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:49:26.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:26 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:26.518 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:26 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:28.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:28 smithi117 ceph-mon[36368]: pgmap v292: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-24T10:49:28.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:28 smithi116 ceph-mon[36248]: pgmap v292: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-24T10:49:28.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:28 smithi023 ceph-mon[32466]: pgmap v292: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-24T10:49:30.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:30 smithi117 ceph-mon[36368]: pgmap v293: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-03-24T10:49:30.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:30 smithi116 ceph-mon[36248]: pgmap v293: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-03-24T10:49:30.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:30 smithi023 ceph-mon[32466]: pgmap v293: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-03-24T10:49:32.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:32 smithi117 ceph-mon[36368]: pgmap v294: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-24T10:49:32.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:32 smithi116 ceph-mon[36248]: pgmap v294: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-24T10:49:32.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:32 smithi023 ceph-mon[32466]: pgmap v294: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-24T10:49:34.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:34 smithi116 ceph-mon[36248]: pgmap v295: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-24T10:49:34.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:34 smithi023 ceph-mon[32466]: pgmap v295: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-24T10:49:35.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:34 smithi117 ceph-mon[36368]: pgmap v295: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-24T10:49:35.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:35 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:49:35.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:35 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:49:35.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:35 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:49:35.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:35 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:49:36.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:35 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:49:36.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:35 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:49:36.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:36 smithi116 ceph-mon[36248]: pgmap v296: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-24T10:49:36.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:36 smithi023 ceph-mon[32466]: pgmap v296: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-24T10:49:37.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:36 smithi117 ceph-mon[36368]: pgmap v296: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-24T10:49:38.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:38 smithi116 ceph-mon[36248]: pgmap v297: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-24T10:49:38.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:38 smithi023 ceph-mon[32466]: pgmap v297: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-24T10:49:39.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:38 smithi117 ceph-mon[36368]: pgmap v297: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-24T10:49:40.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:40 smithi023 ceph-mon[32466]: pgmap v298: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:41.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:40 smithi117 ceph-mon[36368]: pgmap v298: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:41.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:40 smithi116 ceph-mon[36248]: pgmap v298: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:42.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:42 smithi023 ceph-mon[32466]: pgmap v299: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:49:43.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:42 smithi117 ceph-mon[36368]: pgmap v299: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:49:43.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:42 smithi116 ceph-mon[36248]: pgmap v299: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:49:44.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:44 smithi023 ceph-mon[32466]: pgmap v300: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:45.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:44 smithi117 ceph-mon[36368]: pgmap v300: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:45.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:44 smithi116 ceph-mon[36248]: pgmap v300: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:46.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:46 smithi023 ceph-mon[32466]: pgmap v301: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:47.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:46 smithi117 ceph-mon[36368]: pgmap v301: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:47.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:46 smithi116 ceph-mon[36248]: pgmap v301: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:48.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:48 smithi023 ceph-mon[32466]: pgmap v302: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:49:49.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:48 smithi117 ceph-mon[36368]: pgmap v302: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:49:49.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:48 smithi116 ceph-mon[36248]: pgmap v302: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:49:50.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:50 smithi023 ceph-mon[32466]: pgmap v303: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:51.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:50 smithi117 ceph-mon[36368]: pgmap v303: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:51.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:50 smithi116 ceph-mon[36248]: pgmap v303: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:52.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:52 smithi117 ceph-mon[36368]: pgmap v304: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:49:52.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:52 smithi116 ceph-mon[36248]: pgmap v304: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:49:52.668 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:52 smithi023 ceph-mon[32466]: pgmap v304: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:49:54.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:54 smithi117 ceph-mon[36368]: pgmap v305: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:54.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:54 smithi116 ceph-mon[36248]: pgmap v305: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:54.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:54 smithi023 ceph-mon[32466]: pgmap v305: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:55.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:49:55.768 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:49:55.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.886+0000 7f957f4d4640 1 -- 172.21.15.23:0/2029832556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9578105080 msgr2=0x7f957810cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:55.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.886+0000 7f957f4d4640 1 --2- 172.21.15.23:0/2029832556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9578105080 0x7f957810cc50 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f9570009cd0 tx=0x7f957002c0f0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.886+0000 7f957f4d4640 1 -- 172.21.15.23:0/2029832556 shutdown_connections 2024-03-24T10:49:55.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.886+0000 7f957f4d4640 1 --2- 172.21.15.23:0/2029832556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9578105080 0x7f957810cc50 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.886+0000 7f957f4d4640 1 --2- 172.21.15.23:0/2029832556 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9578100960 0x7f95781049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.886+0000 7f957f4d4640 1 --2- 172.21.15.23:0/2029832556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95780fffb0 0x7f9578100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.886+0000 7f957f4d4640 1 -- 172.21.15.23:0/2029832556 >> 172.21.15.23:0/2029832556 conn(0x7f9578076bd0 msgr2=0x7f95780fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:55.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.887+0000 7f957f4d4640 1 -- 172.21.15.23:0/2029832556 shutdown_connections 2024-03-24T10:49:55.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.887+0000 7f957f4d4640 1 -- 172.21.15.23:0/2029832556 wait complete. 2024-03-24T10:49:55.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.887+0000 7f957f4d4640 1 Processor -- start 2024-03-24T10:49:55.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.887+0000 7f957f4d4640 1 -- start start 2024-03-24T10:49:55.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.887+0000 7f957f4d4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95780fffb0 0x7f9578197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.887+0000 7f957f4d4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9578105080 0x7f9578198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.887+0000 7f957f4d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f957819c610 0x7f957819cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.887+0000 7f957f4d4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f957819d000 con 0x7f9578105080 2024-03-24T10:49:55.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.887+0000 7f957f4d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f957819d170 con 0x7f957819c610 2024-03-24T10:49:55.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.887+0000 7f957f4d4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f957819d2e0 con 0x7f95780fffb0 2024-03-24T10:49:55.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.887+0000 7f957ca6f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9578105080 0x7f9578198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f957ca6f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9578105080 0x7f9578198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34682/0 (socket says 172.21.15.23:34682) 2024-03-24T10:49:55.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f957ca6f640 1 -- 172.21.15.23:0/2076911297 learned_addr learned my addr 172.21.15.23:0/2076911297 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:55.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f957da71640 1 --2- 172.21.15.23:0/2076911297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f957819c610 0x7f957819cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f957d270640 1 --2- 172.21.15.23:0/2076911297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95780fffb0 0x7f9578197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f957ca6f640 1 -- 172.21.15.23:0/2076911297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95780fffb0 msgr2=0x7f9578197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:55.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f957ca6f640 1 --2- 172.21.15.23:0/2076911297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95780fffb0 0x7f9578197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f957ca6f640 1 -- 172.21.15.23:0/2076911297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f957819c610 msgr2=0x7f957819cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:55.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f957ca6f640 1 --2- 172.21.15.23:0/2076911297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f957819c610 0x7f957819cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f957ca6f640 1 -- 172.21.15.23:0/2076911297 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9570009900 con 0x7f9578105080 2024-03-24T10:49:55.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f957ca6f640 1 --2- 172.21.15.23:0/2076911297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9578105080 0x7f9578198290 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f956800e9c0 tx=0x7f956800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:55.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f956e7fc640 1 -- 172.21.15.23:0/2076911297 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9568009700 con 0x7f9578105080 2024-03-24T10:49:55.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f957f4d4640 1 -- 172.21.15.23:0/2076911297 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95781a48c0 con 0x7f9578105080 2024-03-24T10:49:55.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f956e7fc640 1 -- 172.21.15.23:0/2076911297 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f956800cd80 con 0x7f9578105080 2024-03-24T10:49:55.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f956e7fc640 1 -- 172.21.15.23:0/2076911297 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9568010430 con 0x7f9578105080 2024-03-24T10:49:55.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.888+0000 7f957f4d4640 1 -- 172.21.15.23:0/2076911297 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95781a4dc0 con 0x7f9578105080 2024-03-24T10:49:55.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.890+0000 7f956e7fc640 1 -- 172.21.15.23:0/2076911297 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f9568005000 con 0x7f9578105080 2024-03-24T10:49:55.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f9582190740 1 Processor -- start 2024-03-24T10:49:55.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f9582190740 1 -- start start 2024-03-24T10:49:55.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f956e7fc640 1 --2- 172.21.15.23:0/2076911297 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f95640a2230 0x7f95640a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f9582190740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf682a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f9582190740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf687e0 0x55a7bdf755a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f957da71640 1 --2- 172.21.15.23:0/2076911297 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f95640a2230 0x7f95640a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f9582190740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf77ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f956e7fc640 1 -- 172.21.15.23:0/2076911297 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f9568014070 con 0x7f9578105080 2024-03-24T10:49:55.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f9582190740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a7bde8cf30 con 0x55a7bdf75ae0 2024-03-24T10:49:55.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f9582190740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a7bded6810 con 0x55a7bdf67ec0 2024-03-24T10:49:55.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f9582190740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a7bded3d40 con 0x55a7bdf687e0 2024-03-24T10:49:55.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f957da71640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf77ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f957d270640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf682a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f957da71640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf77ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34686/0 (socket says 172.21.15.23:34686) 2024-03-24T10:49:55.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f957d270640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf682a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:55552/0 (socket says 172.21.15.23:55552) 2024-03-24T10:49:55.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f957ca6f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf687e0 0x55a7bdf755a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.891+0000 7f957da71640 1 -- 172.21.15.23:0/351674379 learned_addr learned my addr 172.21.15.23:0/351674379 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:55.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.892+0000 7f957d270640 1 -- 172.21.15.23:0/351674379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf687e0 msgr2=0x55a7bdf755a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:55.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.892+0000 7f957d270640 1 --2- 172.21.15.23:0/351674379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf687e0 0x55a7bdf755a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.892+0000 7f957d270640 1 -- 172.21.15.23:0/351674379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf75ae0 msgr2=0x55a7bdf77ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:55.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.892+0000 7f957d270640 1 --2- 172.21.15.23:0/351674379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf77ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.892+0000 7f957d270640 1 -- 172.21.15.23:0/351674379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7bde90ea0 con 0x55a7bdf67ec0 2024-03-24T10:49:55.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.892+0000 7f957ca6f640 1 --2- 172.21.15.23:0/351674379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf687e0 0x55a7bdf755a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:55.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.892+0000 7f957da71640 1 --2- 172.21.15.23:0/2076911297 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f95640a2230 0x7f95640a46f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9570004af0 tx=0x7f9570032f00 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:55.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.892+0000 7f957d270640 1 --2- 172.21.15.23:0/351674379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf682a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f957400ac90 tx=0x7f9574005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=929b78bd6fbc4134 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:55.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.892+0000 7f95637fe640 1 -- 172.21.15.23:0/351674379 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9574013040 con 0x55a7bdf67ec0 2024-03-24T10:49:55.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.892+0000 7f95637fe640 1 -- 172.21.15.23:0/351674379 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f957400ae40 con 0x55a7bdf67ec0 2024-03-24T10:49:55.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.892+0000 7f95637fe640 1 -- 172.21.15.23:0/351674379 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9574018690 con 0x55a7bdf67ec0 2024-03-24T10:49:55.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.893+0000 7f9582190740 1 -- 172.21.15.23:0/351674379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf67ec0 msgr2=0x55a7bdf682a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:55.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.893+0000 7f9582190740 1 --2- 172.21.15.23:0/351674379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf682a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f957400ac90 tx=0x7f9574005f90 comp rx=0 tx=0).stop 2024-03-24T10:49:55.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.893+0000 7f9582190740 1 -- 172.21.15.23:0/351674379 shutdown_connections 2024-03-24T10:49:55.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.893+0000 7f9582190740 1 --2- 172.21.15.23:0/351674379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf77ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.893+0000 7f9582190740 1 --2- 172.21.15.23:0/351674379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf687e0 0x55a7bdf755a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.893+0000 7f9582190740 1 --2- 172.21.15.23:0/351674379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf682a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.893+0000 7f9582190740 1 -- 172.21.15.23:0/351674379 >> 172.21.15.23:0/351674379 conn(0x55a7bdf62900 msgr2=0x55a7bdf62ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:55.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.893+0000 7f9582190740 1 -- 172.21.15.23:0/351674379 shutdown_connections 2024-03-24T10:49:55.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.893+0000 7f9582190740 1 -- 172.21.15.23:0/351674379 wait complete. 2024-03-24T10:49:55.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.893+0000 7f9582190740 1 -- start start 2024-03-24T10:49:55.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.893+0000 7f9582190740 1 Processor -- start 2024-03-24T10:49:55.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f9582190740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf72a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f9582190740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf687e0 0x55a7bdf72f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f9582190740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf73490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f9582190740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a7bdecf520 con 0x55a7bdf67ec0 2024-03-24T10:49:55.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f9582190740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a7bdba2430 con 0x55a7bdf687e0 2024-03-24T10:49:55.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f9582190740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a7bdb5ee00 con 0x55a7bdf75ae0 2024-03-24T10:49:55.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f9582190740 10 client.0 _ll_register_callbacks cb 0x55a7bde6dd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:55.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f957d270640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf72a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f957d270640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf72a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34696/0 (socket says 172.21.15.23:34696) 2024-03-24T10:49:55.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f957d270640 1 -- 172.21.15.23:0/1954757828 learned_addr learned my addr 172.21.15.23:0/1954757828 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:55.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f957ca6f640 1 --2- 172.21.15.23:0/1954757828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf687e0 0x55a7bdf72f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f957da71640 1 --2- 172.21.15.23:0/1954757828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf73490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f957d270640 1 -- 172.21.15.23:0/1954757828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf75ae0 msgr2=0x55a7bdf73490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:55.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f957d270640 1 --2- 172.21.15.23:0/1954757828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf73490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f957d270640 1 -- 172.21.15.23:0/1954757828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf687e0 msgr2=0x55a7bdf72f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:55.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f957d270640 1 --2- 172.21.15.23:0/1954757828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf687e0 0x55a7bdf72f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f957d270640 1 -- 172.21.15.23:0/1954757828 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f957400a8c0 con 0x55a7bdf67ec0 2024-03-24T10:49:55.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f957ca6f640 1 --2- 172.21.15.23:0/1954757828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf687e0 0x55a7bdf72f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:55.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f957d270640 1 --2- 172.21.15.23:0/1954757828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf72a10 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f9574003cb0 tx=0x7f9574005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eea8d1035fcfc1b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:55.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f9560ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:55.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f9560ff9640 1 -- 172.21.15.23:0/1954757828 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95740037f0 con 0x55a7bdf67ec0 2024-03-24T10:49:55.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.894+0000 7f9560ff9640 1 -- 172.21.15.23:0/1954757828 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f9574003950 con 0x55a7bdf67ec0 2024-03-24T10:49:55.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.895+0000 7f9582190740 1 -- 172.21.15.23:0/1954757828 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a7bdbce7e0 con 0x55a7bdf67ec0 2024-03-24T10:49:55.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.895+0000 7f9560ff9640 1 -- 172.21.15.23:0/1954757828 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f95740043a0 con 0x55a7bdf67ec0 2024-03-24T10:49:55.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.895+0000 7f9582190740 10 client.18210 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:55.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.895+0000 7f9582190740 1 -- 172.21.15.23:0/1954757828 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a7bdecdba0 con 0x55a7bdf67ec0 2024-03-24T10:49:55.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.895+0000 7f9560ff9640 1 -- 172.21.15.23:0/1954757828 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a7bdbce7e0 con 0x55a7bdf67ec0 2024-03-24T10:49:55.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.895+0000 7f9560ff9640 1 -- 172.21.15.23:0/1954757828 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f957402b060 con 0x55a7bdf67ec0 2024-03-24T10:49:55.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.895+0000 7f9582190740 10 client.18210 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:55.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.895+0000 7f9582190740 10 client.18210 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:55.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.895+0000 7f9582190740 20 client.18210 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:55.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.895+0000 7f9582190740 1 --2- 172.21.15.23:0/1954757828 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a7bdf85eb0 0x55a7bdf882e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.895+0000 7f9582190740 4 client.18210 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:49:55.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.895+0000 7f9582190740 1 -- 172.21.15.23:0/1954757828 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7bdc0b720 con 0x55a7bdf85eb0 2024-03-24T10:49:55.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.895+0000 7f957ca6f640 1 --2- 172.21.15.23:0/1954757828 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a7bdf85eb0 0x55a7bdf882e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.896+0000 7f9560ff9640 10 client.18210 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:55.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.896+0000 7f957ca6f640 1 --2- 172.21.15.23:0/1954757828 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a7bdf85eb0 0x55a7bdf882e0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aac12ce7b5d42dc2 server_cookie=9b0b5bb4a1388be7 in_seq=0 out_seq=0 2024-03-24T10:49:55.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9560ff9640 1 -- 172.21.15.23:0/1954757828 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a7bdc0b720 con 0x55a7bdf85eb0 2024-03-24T10:49:55.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9560ff9640 10 client.18210 handle_command_reply: tid=0 2024-03-24T10:49:55.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9582190740 1 client.18210 shutdown 2024-03-24T10:49:55.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9582190740 1 -- 172.21.15.23:0/1954757828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 msgr2=0x55a7bdf72a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9582190740 1 --2- 172.21.15.23:0/1954757828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf72a10 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f9574003cb0 tx=0x7f9574005db0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9582190740 1 -- 172.21.15.23:0/1954757828 shutdown_connections 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9582190740 1 --2- 172.21.15.23:0/1954757828 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a7bdf85eb0 0x55a7bdf882e0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9582190740 1 --2- 172.21.15.23:0/1954757828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf73490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9582190740 1 --2- 172.21.15.23:0/1954757828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf687e0 0x55a7bdf72f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9582190740 1 --2- 172.21.15.23:0/1954757828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf72a10 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9560ff9640 0 client.18210 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9582190740 1 -- 172.21.15.23:0/1954757828 >> 172.21.15.23:0/1954757828 conn(0x55a7bdf62900 msgr2=0x55a7bdf64650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9582190740 1 -- 172.21.15.23:0/1954757828 shutdown_connections 2024-03-24T10:49:55.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9582190740 1 -- 172.21.15.23:0/1954757828 wait complete. 2024-03-24T10:49:55.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.897+0000 7f9582190740 20 client.18210 trim_cache size 0 max 16384 2024-03-24T10:49:55.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f9582190740 1 Processor -- start 2024-03-24T10:49:55.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f9582190740 1 -- start start 2024-03-24T10:49:55.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f9582190740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf612d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f9582190740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf687e0 0x55a7bdf722f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f9582190740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf813b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f9582190740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a7bde54430 con 0x55a7bdf67ec0 2024-03-24T10:49:55.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f9582190740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a7bdf72860 con 0x55a7bdf75ae0 2024-03-24T10:49:55.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f9582190740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a7bdf6e5d0 con 0x55a7bdf687e0 2024-03-24T10:49:55.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f957d270640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf612d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f957d270640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf612d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34708/0 (socket says 172.21.15.23:34708) 2024-03-24T10:49:55.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f957d270640 1 -- 172.21.15.23:0/273900376 learned_addr learned my addr 172.21.15.23:0/273900376 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:55.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f957da71640 1 --2- 172.21.15.23:0/273900376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf813b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f957ca6f640 1 --2- 172.21.15.23:0/273900376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf687e0 0x55a7bdf722f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f957da71640 1 -- 172.21.15.23:0/273900376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf687e0 msgr2=0x55a7bdf722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:55.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f957da71640 1 --2- 172.21.15.23:0/273900376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf687e0 0x55a7bdf722f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f957da71640 1 -- 172.21.15.23:0/273900376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 msgr2=0x55a7bdf612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:55.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f957da71640 1 --2- 172.21.15.23:0/273900376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf612d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f957da71640 1 -- 172.21.15.23:0/273900376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7bde69b20 con 0x55a7bdf75ae0 2024-03-24T10:49:55.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.915+0000 7f957d270640 1 --2- 172.21.15.23:0/273900376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf612d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:55.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.916+0000 7f957da71640 1 --2- 172.21.15.23:0/273900376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf813b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9570006010 tx=0x7f9570005c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef93219d8ac1a87 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:55.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.916+0000 7f9562ffd640 1 -- 172.21.15.23:0/273900376 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9570048070 con 0x55a7bdf75ae0 2024-03-24T10:49:55.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.916+0000 7f9562ffd640 1 -- 172.21.15.23:0/273900376 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9570034d70 con 0x55a7bdf75ae0 2024-03-24T10:49:55.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.916+0000 7f9562ffd640 1 -- 172.21.15.23:0/273900376 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9570004270 con 0x55a7bdf75ae0 2024-03-24T10:49:55.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.917+0000 7f9582190740 1 -- 172.21.15.23:0/273900376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf75ae0 msgr2=0x55a7bdf813b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:55.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.917+0000 7f9582190740 1 --2- 172.21.15.23:0/273900376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf813b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9570006010 tx=0x7f9570005c50 comp rx=0 tx=0).stop 2024-03-24T10:49:55.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.917+0000 7f9582190740 1 -- 172.21.15.23:0/273900376 shutdown_connections 2024-03-24T10:49:55.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.917+0000 7f9582190740 1 --2- 172.21.15.23:0/273900376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf813b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.917+0000 7f9582190740 1 --2- 172.21.15.23:0/273900376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf687e0 0x55a7bdf722f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.917+0000 7f9582190740 1 --2- 172.21.15.23:0/273900376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf612d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.917+0000 7f9582190740 1 -- 172.21.15.23:0/273900376 >> 172.21.15.23:0/273900376 conn(0x55a7bdf62900 msgr2=0x55a7bdf73260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:55.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.917+0000 7f9582190740 1 -- 172.21.15.23:0/273900376 shutdown_connections 2024-03-24T10:49:55.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.917+0000 7f9582190740 1 -- 172.21.15.23:0/273900376 wait complete. 2024-03-24T10:49:55.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.917+0000 7f9582190740 1 -- start start 2024-03-24T10:49:55.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.917+0000 7f9582190740 1 Processor -- start 2024-03-24T10:49:55.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.917+0000 7f9582190740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf793d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.917+0000 7f9582190740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf687e0 0x55a7bdf79910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.917+0000 7f9582190740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf79e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f9582190740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55a7bdfeb1c0 con 0x55a7bdf67ec0 2024-03-24T10:49:55.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f9582190740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a7bdfeb300 con 0x55a7bdf687e0 2024-03-24T10:49:55.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f9582190740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a7bdfeb4c0 con 0x55a7bdf75ae0 2024-03-24T10:49:55.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f9582190740 10 client.0 _ll_register_callbacks cb 0x7f9574005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:55.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f957d270640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf793d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f957d270640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf793d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34710/0 (socket says 172.21.15.23:34710) 2024-03-24T10:49:55.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f957d270640 1 -- 172.21.15.23:0/169209127 learned_addr learned my addr 172.21.15.23:0/169209127 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:55.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f957ca6f640 1 --2- 172.21.15.23:0/169209127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf687e0 0x55a7bdf79910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f957da71640 1 --2- 172.21.15.23:0/169209127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf79e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f957d270640 1 -- 172.21.15.23:0/169209127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf75ae0 msgr2=0x55a7bdf79e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:55.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f957d270640 1 --2- 172.21.15.23:0/169209127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf79e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f957d270640 1 -- 172.21.15.23:0/169209127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf687e0 msgr2=0x55a7bdf79910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:55.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f957d270640 1 --2- 172.21.15.23:0/169209127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf687e0 0x55a7bdf79910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f957d270640 1 -- 172.21.15.23:0/169209127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7bdecdba0 con 0x55a7bdf67ec0 2024-03-24T10:49:55.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f957d270640 1 --2- 172.21.15.23:0/169209127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf793d0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f9574003820 tx=0x7f9574003850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa2c157bbffbeaa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:55.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f9560ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:55.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f9560ff9640 1 -- 172.21.15.23:0/169209127 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95740249f0 con 0x55a7bdf67ec0 2024-03-24T10:49:55.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f9560ff9640 1 -- 172.21.15.23:0/169209127 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f957400aae0 con 0x55a7bdf67ec0 2024-03-24T10:49:55.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f9582190740 1 -- 172.21.15.23:0/169209127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a7bdf61930 con 0x55a7bdf67ec0 2024-03-24T10:49:55.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f9560ff9640 1 -- 172.21.15.23:0/169209127 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9574024630 con 0x55a7bdf67ec0 2024-03-24T10:49:55.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f9582190740 10 client.18216 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:55.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.918+0000 7f9582190740 1 -- 172.21.15.23:0/169209127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x7f957810d460 con 0x55a7bdf67ec0 2024-03-24T10:49:55.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.919+0000 7f9560ff9640 1 -- 172.21.15.23:0/169209127 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a7bdf61930 con 0x55a7bdf67ec0 2024-03-24T10:49:55.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.919+0000 7f9560ff9640 1 -- 172.21.15.23:0/169209127 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f957402b040 con 0x55a7bdf67ec0 2024-03-24T10:49:55.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.919+0000 7f9582190740 10 client.18216 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:55.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.919+0000 7f9582190740 10 client.18216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:55.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.919+0000 7f9582190740 20 client.18216 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:55.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.919+0000 7f9582190740 1 --2- 172.21.15.23:0/169209127 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a7bdfec980 0x55a7bdfeeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.919+0000 7f9582190740 4 client.18216 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:49:55.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.919+0000 7f9582190740 1 -- 172.21.15.23:0/169209127 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f957400ae40 con 0x55a7bdfec980 2024-03-24T10:49:55.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.919+0000 7f957ca6f640 1 --2- 172.21.15.23:0/169209127 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a7bdfec980 0x55a7bdfeeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.919+0000 7f957ca6f640 1 --2- 172.21.15.23:0/169209127 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a7bdfec980 0x55a7bdfeeda0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=193ec215e975b7e8 server_cookie=444b35c8ba44721b in_seq=0 out_seq=0 2024-03-24T10:49:55.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.919+0000 7f9560ff9640 10 client.18216 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:55.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.919+0000 7f9560ff9640 1 -- 172.21.15.23:0/169209127 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f957400ae40 con 0x55a7bdfec980 2024-03-24T10:49:55.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.919+0000 7f9560ff9640 10 client.18216 handle_command_reply: tid=0 2024-03-24T10:49:55.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f9582190740 1 client.18216 shutdown 2024-03-24T10:49:55.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f9582190740 1 -- 172.21.15.23:0/169209127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 msgr2=0x55a7bdf793d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:55.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f9582190740 1 --2- 172.21.15.23:0/169209127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf793d0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f9574003820 tx=0x7f9574003850 comp rx=0 tx=0).stop 2024-03-24T10:49:55.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f9582190740 1 -- 172.21.15.23:0/169209127 shutdown_connections 2024-03-24T10:49:55.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f9582190740 1 --2- 172.21.15.23:0/169209127 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55a7bdfec980 0x55a7bdfeeda0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f9582190740 1 --2- 172.21.15.23:0/169209127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a7bdf75ae0 0x55a7bdf79e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f9560ff9640 0 client.18216 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:55.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f9582190740 1 --2- 172.21.15.23:0/169209127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a7bdf687e0 0x55a7bdf79910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f9582190740 1 --2- 172.21.15.23:0/169209127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a7bdf67ec0 0x55a7bdf793d0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f9582190740 1 -- 172.21.15.23:0/169209127 >> 172.21.15.23:0/169209127 conn(0x55a7bdf62900 msgr2=0x55a7bdf63900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:55.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f9582190740 1 -- 172.21.15.23:0/169209127 shutdown_connections 2024-03-24T10:49:55.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f9582190740 1 -- 172.21.15.23:0/169209127 wait complete. 2024-03-24T10:49:55.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f9582190740 20 client.18216 trim_cache size 0 max 16384 2024-03-24T10:49:55.932 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:49:55.933 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:49:55.933 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:49:55.933 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:49:55.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f957f4d4640 1 -- 172.21.15.23:0/2076911297 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f95640a2230 msgr2=0x7f95640a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:55.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f957f4d4640 1 --2- 172.21.15.23:0/2076911297 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f95640a2230 0x7f95640a46f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9570004af0 tx=0x7f9570032f00 comp rx=0 tx=0).stop 2024-03-24T10:49:55.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f957f4d4640 1 -- 172.21.15.23:0/2076911297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9578105080 msgr2=0x7f9578198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:55.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f957f4d4640 1 --2- 172.21.15.23:0/2076911297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9578105080 0x7f9578198290 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f956800e9c0 tx=0x7f956800ee80 comp rx=0 tx=0).stop 2024-03-24T10:49:55.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f957f4d4640 1 -- 172.21.15.23:0/2076911297 shutdown_connections 2024-03-24T10:49:55.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f957f4d4640 1 --2- 172.21.15.23:0/2076911297 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f95640a2230 0x7f95640a46f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f957f4d4640 1 --2- 172.21.15.23:0/2076911297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f957819c610 0x7f957819cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f957f4d4640 1 --2- 172.21.15.23:0/2076911297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9578105080 0x7f9578198290 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f957f4d4640 1 --2- 172.21.15.23:0/2076911297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95780fffb0 0x7f9578197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.920+0000 7f957f4d4640 1 -- 172.21.15.23:0/2076911297 >> 172.21.15.23:0/2076911297 conn(0x7f9578076bd0 msgr2=0x7f9578103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:55.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.921+0000 7f957f4d4640 1 -- 172.21.15.23:0/2076911297 shutdown_connections 2024-03-24T10:49:55.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:55.921+0000 7f957f4d4640 1 -- 172.21.15.23:0/2076911297 wait complete. 2024-03-24T10:49:55.936 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:49:55.936 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:56.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.040+0000 7f06c742e640 1 -- 172.21.15.23:0/1530987946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c01066b0 msgr2=0x7f06c0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.040+0000 7f06c742e640 1 --2- 172.21.15.23:0/1530987946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c01066b0 0x7f06c0106a90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009860 tx=0x7f06b402bc00 comp rx=0 tx=0).stop 2024-03-24T10:49:56.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.040+0000 7f06c742e640 1 -- 172.21.15.23:0/1530987946 shutdown_connections 2024-03-24T10:49:56.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.040+0000 7f06c742e640 1 --2- 172.21.15.23:0/1530987946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06c0100d40 0x7f06c010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.040+0000 7f06c742e640 1 --2- 172.21.15.23:0/1530987946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06c0106fd0 0x7f06c0100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.040+0000 7f06c742e640 1 --2- 172.21.15.23:0/1530987946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c01066b0 0x7f06c0106a90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.040+0000 7f06c742e640 1 -- 172.21.15.23:0/1530987946 >> 172.21.15.23:0/1530987946 conn(0x7f06c0077710 msgr2=0x7f06c0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.040+0000 7f06c742e640 1 -- 172.21.15.23:0/1530987946 shutdown_connections 2024-03-24T10:49:56.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.040+0000 7f06c742e640 1 -- 172.21.15.23:0/1530987946 wait complete. 2024-03-24T10:49:56.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.040+0000 7f06c742e640 1 Processor -- start 2024-03-24T10:49:56.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.040+0000 7f06c742e640 1 -- start start 2024-03-24T10:49:56.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c742e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06c0100d40 0x7f06c019c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c742e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06c01066b0 0x7f06c019ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c742e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c0106fd0 0x7f06c0196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c742e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f06c0197110 con 0x7f06c01066b0 2024-03-24T10:49:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c742e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f06c0197280 con 0x7f06c0106fd0 2024-03-24T10:49:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c742e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f06c0197440 con 0x7f06c0100d40 2024-03-24T10:49:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c49c9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06c01066b0 0x7f06c019ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c49c9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06c01066b0 0x7f06c019ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34716/0 (socket says 172.21.15.23:34716) 2024-03-24T10:49:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c49c9640 1 -- 172.21.15.23:0/4037188297 learned_addr learned my addr 172.21.15.23:0/4037188297 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c59cb640 1 --2- 172.21.15.23:0/4037188297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c0106fd0 0x7f06c0196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c51ca640 1 --2- 172.21.15.23:0/4037188297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06c0100d40 0x7f06c019c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c49c9640 1 -- 172.21.15.23:0/4037188297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06c0100d40 msgr2=0x7f06c019c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c49c9640 1 --2- 172.21.15.23:0/4037188297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06c0100d40 0x7f06c019c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c49c9640 1 -- 172.21.15.23:0/4037188297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c0106fd0 msgr2=0x7f06c0196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c49c9640 1 --2- 172.21.15.23:0/4037188297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c0106fd0 0x7f06c0196ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c49c9640 1 -- 172.21.15.23:0/4037188297 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b8009900 con 0x7f06c01066b0 2024-03-24T10:49:56.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c49c9640 1 --2- 172.21.15.23:0/4037188297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06c01066b0 0x7f06c019ce50 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00e9c0 tx=0x7f06bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06b27fc640 1 -- 172.21.15.23:0/4037188297 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc0096c0 con 0x7f06c01066b0 2024-03-24T10:49:56.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c742e640 1 -- 172.21.15.23:0/4037188297 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b4009490 con 0x7f06c01066b0 2024-03-24T10:49:56.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06b27fc640 1 -- 172.21.15.23:0/4037188297 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06bc005500 con 0x7f06c01066b0 2024-03-24T10:49:56.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06b27fc640 1 -- 172.21.15.23:0/4037188297 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc010430 con 0x7f06c01066b0 2024-03-24T10:49:56.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.041+0000 7f06c742e640 1 -- 172.21.15.23:0/4037188297 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c01a37d0 con 0x7f06c01066b0 2024-03-24T10:49:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.042+0000 7f06c742e640 1 -- 172.21.15.23:0/4037188297 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06c0068500 con 0x7f06c01066b0 2024-03-24T10:49:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.043+0000 7f06b27fc640 1 -- 172.21.15.23:0/4037188297 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f06bc005020 con 0x7f06c01066b0 2024-03-24T10:49:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.044+0000 7f06b27fc640 1 --2- 172.21.15.23:0/4037188297 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06ac0a2230 0x7f06ac0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.044+0000 7f06c51ca640 1 --2- 172.21.15.23:0/4037188297 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06ac0a2230 0x7f06ac0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.044+0000 7f06b27fc640 1 -- 172.21.15.23:0/4037188297 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f06bc014070 con 0x7f06c01066b0 2024-03-24T10:49:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.044+0000 7f06c51ca640 1 --2- 172.21.15.23:0/4037188297 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06ac0a2230 0x7f06ac0a46f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f06c0197c00 tx=0x7f06b40058d0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.048+0000 7f06b27fc640 1 -- 172.21.15.23:0/4037188297 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f06bc0970a0 con 0x7f06c01066b0 2024-03-24T10:49:56.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.271+0000 7f06c742e640 1 -- 172.21.15.23:0/4037188297 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06c0108b80 con 0x7f06c01066b0 2024-03-24T10:49:56.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.272+0000 7f06b27fc640 1 -- 172.21.15.23:0/4037188297 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11869 (secure 0 0 0) 0x7f06bc09a760 con 0x7f06c01066b0 2024-03-24T10:49:56.272 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:49:56.273 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:47:17.474711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27526":{"gid":27526,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/2017557046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":2017557046},{"type":"v1","addr":"172.21.15.117:6837","nonce":2017557046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:49:56.273 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 31 2024-03-24T10:49:56.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.274+0000 7f06c742e640 1 -- 172.21.15.23:0/4037188297 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06ac0a2230 msgr2=0x7f06ac0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.274+0000 7f06c742e640 1 --2- 172.21.15.23:0/4037188297 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06ac0a2230 0x7f06ac0a46f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f06c0197c00 tx=0x7f06b40058d0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.274+0000 7f06c742e640 1 -- 172.21.15.23:0/4037188297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06c01066b0 msgr2=0x7f06c019ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.274+0000 7f06c742e640 1 --2- 172.21.15.23:0/4037188297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06c01066b0 0x7f06c019ce50 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00e9c0 tx=0x7f06bc00ee80 comp rx=0 tx=0).stop 2024-03-24T10:49:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.274+0000 7f06c742e640 1 -- 172.21.15.23:0/4037188297 shutdown_connections 2024-03-24T10:49:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.274+0000 7f06c742e640 1 --2- 172.21.15.23:0/4037188297 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f06ac0a2230 0x7f06ac0a46f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.274+0000 7f06c742e640 1 --2- 172.21.15.23:0/4037188297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06c0106fd0 0x7f06c0196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.274+0000 7f06c742e640 1 --2- 172.21.15.23:0/4037188297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06c01066b0 0x7f06c019ce50 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.274+0000 7f06c742e640 1 --2- 172.21.15.23:0/4037188297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06c0100d40 0x7f06c019c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.274+0000 7f06c742e640 1 -- 172.21.15.23:0/4037188297 >> 172.21.15.23:0/4037188297 conn(0x7f06c0077710 msgr2=0x7f06c00ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.274+0000 7f06c742e640 1 -- 172.21.15.23:0/4037188297 shutdown_connections 2024-03-24T10:49:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.274+0000 7f06c742e640 1 -- 172.21.15.23:0/4037188297 wait complete. 2024-03-24T10:49:56.286 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:49:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.391+0000 7f0a6362e640 1 -- 172.21.15.23:0/1391003433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5c106640 msgr2=0x7f0a5c106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.391+0000 7f0a6362e640 1 --2- 172.21.15.23:0/1391003433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5c106640 0x7f0a5c106a20 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f0a58009930 tx=0x7f0a5802bc30 comp rx=0 tx=0).stop 2024-03-24T10:49:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.391+0000 7f0a6362e640 1 -- 172.21.15.23:0/1391003433 shutdown_connections 2024-03-24T10:49:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.391+0000 7f0a6362e640 1 --2- 172.21.15.23:0/1391003433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a5c100ff0 0x7f0a5c10d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.391+0000 7f0a6362e640 1 --2- 172.21.15.23:0/1391003433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a5c100630 0x7f0a5c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.391+0000 7f0a6362e640 1 --2- 172.21.15.23:0/1391003433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5c106640 0x7f0a5c106a20 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.391+0000 7f0a6362e640 1 -- 172.21.15.23:0/1391003433 >> 172.21.15.23:0/1391003433 conn(0x7f0a5c077770 msgr2=0x7f0a5c077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.391+0000 7f0a6362e640 1 -- 172.21.15.23:0/1391003433 shutdown_connections 2024-03-24T10:49:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.391+0000 7f0a6362e640 1 -- 172.21.15.23:0/1391003433 wait complete. 2024-03-24T10:49:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a6362e640 1 Processor -- start 2024-03-24T10:49:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a6362e640 1 -- start start 2024-03-24T10:49:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a6362e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5c100630 0x7f0a5c198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a6362e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a5c100ff0 0x7f0a5c199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a6362e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a5c106640 0x7f0a5c1a3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a6362e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5c1999d0 con 0x7f0a5c100630 2024-03-24T10:49:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a6362e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5c1a41d0 con 0x7f0a5c106640 2024-03-24T10:49:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a6362e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5c1a4310 con 0x7f0a5c100ff0 2024-03-24T10:49:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a613ca640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5c100630 0x7f0a5c198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a613ca640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5c100630 0x7f0a5c198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34728/0 (socket says 172.21.15.23:34728) 2024-03-24T10:49:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a613ca640 1 -- 172.21.15.23:0/3982946033 learned_addr learned my addr 172.21.15.23:0/3982946033 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a61bcb640 1 --2- 172.21.15.23:0/3982946033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a5c106640 0x7f0a5c1a3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a60bc9640 1 --2- 172.21.15.23:0/3982946033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a5c100ff0 0x7f0a5c199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a613ca640 1 -- 172.21.15.23:0/3982946033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a5c100ff0 msgr2=0x7f0a5c199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a613ca640 1 --2- 172.21.15.23:0/3982946033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a5c100ff0 0x7f0a5c199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a613ca640 1 -- 172.21.15.23:0/3982946033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a5c106640 msgr2=0x7f0a5c1a3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a613ca640 1 --2- 172.21.15.23:0/3982946033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a5c106640 0x7f0a5c1a3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.392+0000 7f0a613ca640 1 -- 172.21.15.23:0/3982946033 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a58009560 con 0x7f0a5c100630 2024-03-24T10:49:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.393+0000 7f0a613ca640 1 --2- 172.21.15.23:0/3982946033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5c100630 0x7f0a5c198c10 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f0a58009430 tx=0x7f0a5802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.393+0000 7f0a527fc640 1 -- 172.21.15.23:0/3982946033 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a58004350 con 0x7f0a5c100630 2024-03-24T10:49:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.393+0000 7f0a527fc640 1 -- 172.21.15.23:0/3982946033 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a580044b0 con 0x7f0a5c100630 2024-03-24T10:49:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.393+0000 7f0a6362e640 1 -- 172.21.15.23:0/3982946033 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a5c1a4510 con 0x7f0a5c100630 2024-03-24T10:49:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.393+0000 7f0a527fc640 1 -- 172.21.15.23:0/3982946033 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a58003d30 con 0x7f0a5c100630 2024-03-24T10:49:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.393+0000 7f0a6362e640 1 -- 172.21.15.23:0/3982946033 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a5c1a49c0 con 0x7f0a5c100630 2024-03-24T10:49:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.395+0000 7f0a527fc640 1 -- 172.21.15.23:0/3982946033 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f0a58004df0 con 0x7f0a5c100630 2024-03-24T10:49:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.395+0000 7f0a66215740 1 Processor -- start 2024-03-24T10:49:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.395+0000 7f0a66215740 1 -- start start 2024-03-24T10:49:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.395+0000 7f0a66215740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 0x55e6b72707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.395+0000 7f0a66215740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b72793b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.395+0000 7f0a66215740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b727bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.395+0000 7f0a66215740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e6b71de840 con 0x55e6b7270d10 2024-03-24T10:49:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.395+0000 7f0a66215740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e6b71dbd70 con 0x55e6b72798f0 2024-03-24T10:49:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.395+0000 7f0a66215740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e6b71d7550 con 0x55e6b72703f0 2024-03-24T10:49:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.395+0000 7f0a527fc640 1 --2- 172.21.15.23:0/3982946033 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a480a2230 0x7f0a480a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a613ca640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b72793b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a61bcb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 0x55e6b72707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a60bc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b727bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a61bcb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 0x55e6b72707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:55662/0 (socket says 172.21.15.23:55662) 2024-03-24T10:49:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a613ca640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b72793b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34734/0 (socket says 172.21.15.23:34734) 2024-03-24T10:49:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a61bcb640 1 -- 172.21.15.23:0/388492953 learned_addr learned my addr 172.21.15.23:0/388492953 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a61bcb640 1 --2- 172.21.15.23:0/3982946033 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a480a2230 0x7f0a480a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a527fc640 1 -- 172.21.15.23:0/3982946033 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f0a580f0050 con 0x7f0a5c100630 2024-03-24T10:49:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a61bcb640 1 -- 172.21.15.23:0/388492953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 msgr2=0x55e6b727bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a61bcb640 1 --2- 172.21.15.23:0/388492953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b727bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a61bcb640 1 -- 172.21.15.23:0/388492953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 msgr2=0x55e6b72793b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a61bcb640 1 --2- 172.21.15.23:0/388492953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b72793b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a61bcb640 1 -- 172.21.15.23:0/388492953 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6b716cde0 con 0x55e6b72703f0 2024-03-24T10:49:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a60bc9640 1 --2- 172.21.15.23:0/388492953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b727bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a613ca640 1 --2- 172.21.15.23:0/388492953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b72793b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a61bcb640 1 --2- 172.21.15.23:0/388492953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 0x55e6b72707d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0a54005f40 tx=0x7f0a5400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f669c067d80aa4f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a61bcb640 1 --2- 172.21.15.23:0/3982946033 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a480a2230 0x7f0a480a46f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400fc90 tx=0x7f0a54018000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a477fe640 1 -- 172.21.15.23:0/388492953 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a54011e70 con 0x55e6b72703f0 2024-03-24T10:49:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a477fe640 1 -- 172.21.15.23:0/388492953 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a54009dd0 con 0x55e6b72703f0 2024-03-24T10:49:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.396+0000 7f0a477fe640 1 -- 172.21.15.23:0/388492953 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a54004250 con 0x55e6b72703f0 2024-03-24T10:49:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.397+0000 7f0a66215740 1 -- 172.21.15.23:0/388492953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 msgr2=0x55e6b72707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.397+0000 7f0a66215740 1 --2- 172.21.15.23:0/388492953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 0x55e6b72707d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0a54005f40 tx=0x7f0a5400f790 comp rx=0 tx=0).stop 2024-03-24T10:49:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.397+0000 7f0a66215740 1 -- 172.21.15.23:0/388492953 shutdown_connections 2024-03-24T10:49:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.397+0000 7f0a66215740 1 --2- 172.21.15.23:0/388492953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b727bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.397+0000 7f0a66215740 1 --2- 172.21.15.23:0/388492953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b72793b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.397+0000 7f0a66215740 1 --2- 172.21.15.23:0/388492953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 0x55e6b72707d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.397+0000 7f0a66215740 1 -- 172.21.15.23:0/388492953 >> 172.21.15.23:0/388492953 conn(0x55e6b726ab20 msgr2=0x55e6b726af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.397+0000 7f0a66215740 1 -- 172.21.15.23:0/388492953 shutdown_connections 2024-03-24T10:49:56.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.397+0000 7f0a66215740 1 -- 172.21.15.23:0/388492953 wait complete. 2024-03-24T10:49:56.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.397+0000 7f0a66215740 1 -- start start 2024-03-24T10:49:56.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a66215740 1 Processor -- start 2024-03-24T10:49:56.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a66215740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 0x55e6b7276a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a66215740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b7276f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a66215740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b72774b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a66215740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e6b6eaa430 con 0x55e6b7270d10 2024-03-24T10:49:56.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a66215740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e6b6e66e00 con 0x55e6b72798f0 2024-03-24T10:49:56.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a66215740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e6b715c590 con 0x55e6b72703f0 2024-03-24T10:49:56.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a60bc9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b7276f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a66215740 10 client.0 _ll_register_callbacks cb 0x55e6b71cbfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:56.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a60bc9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b7276f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34746/0 (socket says 172.21.15.23:34746) 2024-03-24T10:49:56.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a60bc9640 1 -- 172.21.15.23:0/737854744 learned_addr learned my addr 172.21.15.23:0/737854744 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a61bcb640 1 --2- 172.21.15.23:0/737854744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b72774b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a613ca640 1 --2- 172.21.15.23:0/737854744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 0x55e6b7276a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a60bc9640 1 -- 172.21.15.23:0/737854744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 msgr2=0x55e6b7276a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a60bc9640 1 --2- 172.21.15.23:0/737854744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 0x55e6b7276a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a60bc9640 1 -- 172.21.15.23:0/737854744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 msgr2=0x55e6b72774b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a60bc9640 1 --2- 172.21.15.23:0/737854744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b72774b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.398+0000 7f0a60bc9640 1 -- 172.21.15.23:0/737854744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a54019040 con 0x55e6b7270d10 2024-03-24T10:49:56.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a60bc9640 1 --2- 172.21.15.23:0/737854744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b7276f70 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c0126e0 tx=0x7f0a4c012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8583e50716905a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:56.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a44ff9640 1 -- 172.21.15.23:0/737854744 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c017070 con 0x55e6b7270d10 2024-03-24T10:49:56.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a44ff9640 1 -- 172.21.15.23:0/737854744 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f0a4c004050 con 0x55e6b7270d10 2024-03-24T10:49:56.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a66215740 1 -- 172.21.15.23:0/737854744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e6b6ed67e0 con 0x55e6b7270d10 2024-03-24T10:49:56.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a44ff9640 1 -- 172.21.15.23:0/737854744 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0a4c004210 con 0x55e6b7270d10 2024-03-24T10:49:56.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a66215740 10 client.18234 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:56.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a66215740 1 -- 172.21.15.23:0/737854744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e6b71d5bd0 con 0x55e6b7270d10 2024-03-24T10:49:56.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a44ff9640 1 -- 172.21.15.23:0/737854744 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e6b6ed67e0 con 0x55e6b7270d10 2024-03-24T10:49:56.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a44ff9640 1 -- 172.21.15.23:0/737854744 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f0a4c003a60 con 0x55e6b7270d10 2024-03-24T10:49:56.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a66215740 10 client.18234 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:56.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a66215740 10 client.18234 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:49:56.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a66215740 20 client.18234 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:56.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a66215740 1 --2- 172.21.15.23:0/737854744 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55e6b7289b30 0x55e6b728bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a66215740 4 client.18234 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:49:56.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.399+0000 7f0a66215740 1 -- 172.21.15.23:0/737854744 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6b6f13720 con 0x55e6b7289b30 2024-03-24T10:49:56.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.400+0000 7f0a613ca640 1 --2- 172.21.15.23:0/737854744 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55e6b7289b30 0x55e6b728bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.400+0000 7f0a613ca640 1 --2- 172.21.15.23:0/737854744 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55e6b7289b30 0x55e6b728bf50 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a3a888268c65c3b1 server_cookie=eb51c73ede9e645 in_seq=0 out_seq=0 2024-03-24T10:49:56.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.400+0000 7f0a44ff9640 10 client.18234 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:49:56.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.400+0000 7f0a44ff9640 1 -- 172.21.15.23:0/737854744 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e6b6f13720 con 0x55e6b7289b30 2024-03-24T10:49:56.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.401+0000 7f0a44ff9640 10 client.18234 handle_command_reply: tid=0 2024-03-24T10:49:56.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.401+0000 7f0a66215740 1 client.18234 shutdown 2024-03-24T10:49:56.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.401+0000 7f0a66215740 1 -- 172.21.15.23:0/737854744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 msgr2=0x55e6b7276f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.401+0000 7f0a66215740 1 --2- 172.21.15.23:0/737854744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b7276f70 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c0126e0 tx=0x7f0a4c012ba0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.401+0000 7f0a66215740 1 -- 172.21.15.23:0/737854744 shutdown_connections 2024-03-24T10:49:56.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.401+0000 7f0a66215740 1 --2- 172.21.15.23:0/737854744 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55e6b7289b30 0x55e6b728bf50 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.401+0000 7f0a66215740 1 --2- 172.21.15.23:0/737854744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b72774b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.401+0000 7f0a66215740 1 --2- 172.21.15.23:0/737854744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b7276f70 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.401+0000 7f0a66215740 1 --2- 172.21.15.23:0/737854744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 0x55e6b7276a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.401+0000 7f0a44ff9640 0 client.18234 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:49:56.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.401+0000 7f0a66215740 1 -- 172.21.15.23:0/737854744 >> 172.21.15.23:0/737854744 conn(0x55e6b726ab20 msgr2=0x55e6b726c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.401+0000 7f0a66215740 1 -- 172.21.15.23:0/737854744 shutdown_connections 2024-03-24T10:49:56.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.401+0000 7f0a66215740 1 -- 172.21.15.23:0/737854744 wait complete. 2024-03-24T10:49:56.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.401+0000 7f0a66215740 20 client.18234 trim_cache size 0 max 16384 2024-03-24T10:49:56.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.414+0000 7f0a66215740 1 Processor -- start 2024-03-24T10:49:56.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.414+0000 7f0a66215740 1 -- start start 2024-03-24T10:49:56.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.414+0000 7f0a66215740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b72703f0 0x55e6b7285120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a66215740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b7270d10 0x55e6b72772d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a66215740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b7291820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a66215740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e6b727c0b0 con 0x55e6b72703f0 2024-03-24T10:49:56.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a66215740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e6b726b910 con 0x55e6b72798f0 2024-03-24T10:49:56.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a66215740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e6b72882e0 con 0x55e6b7270d10 2024-03-24T10:49:56.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a613ca640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b72703f0 0x55e6b7285120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a613ca640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b72703f0 0x55e6b7285120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34752/0 (socket says 172.21.15.23:34752) 2024-03-24T10:49:56.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a613ca640 1 -- 172.21.15.23:0/3903370298 learned_addr learned my addr 172.21.15.23:0/3903370298 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a61bcb640 1 --2- 172.21.15.23:0/3903370298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b7291820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a60bc9640 1 --2- 172.21.15.23:0/3903370298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b7270d10 0x55e6b72772d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a613ca640 1 -- 172.21.15.23:0/3903370298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b7270d10 msgr2=0x55e6b72772d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a613ca640 1 --2- 172.21.15.23:0/3903370298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b7270d10 0x55e6b72772d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a613ca640 1 -- 172.21.15.23:0/3903370298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 msgr2=0x55e6b7291820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a613ca640 1 --2- 172.21.15.23:0/3903370298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b7291820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a613ca640 1 -- 172.21.15.23:0/3903370298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6b7171c80 con 0x55e6b72703f0 2024-03-24T10:49:56.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a613ca640 1 --2- 172.21.15.23:0/3903370298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b72703f0 0x55e6b7285120 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f0a580edd70 tx=0x7f0a5808eae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a73339d98b72275 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a46ffd640 1 -- 172.21.15.23:0/3903370298 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a58089070 con 0x55e6b72703f0 2024-03-24T10:49:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a46ffd640 1 -- 172.21.15.23:0/3903370298 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a580bd220 con 0x55e6b72703f0 2024-03-24T10:49:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.415+0000 7f0a46ffd640 1 -- 172.21.15.23:0/3903370298 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a580bac50 con 0x55e6b72703f0 2024-03-24T10:49:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.416+0000 7f0a66215740 1 -- 172.21.15.23:0/3903370298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b72703f0 msgr2=0x55e6b7285120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.416+0000 7f0a66215740 1 --2- 172.21.15.23:0/3903370298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b72703f0 0x55e6b7285120 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f0a580edd70 tx=0x7f0a5808eae0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.416+0000 7f0a66215740 1 -- 172.21.15.23:0/3903370298 shutdown_connections 2024-03-24T10:49:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.416+0000 7f0a66215740 1 --2- 172.21.15.23:0/3903370298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b7291820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.416+0000 7f0a66215740 1 --2- 172.21.15.23:0/3903370298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b7270d10 0x55e6b72772d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.416+0000 7f0a66215740 1 --2- 172.21.15.23:0/3903370298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b72703f0 0x55e6b7285120 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.416+0000 7f0a66215740 1 -- 172.21.15.23:0/3903370298 >> 172.21.15.23:0/3903370298 conn(0x55e6b726ab20 msgr2=0x55e6b726c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.416+0000 7f0a66215740 1 -- 172.21.15.23:0/3903370298 shutdown_connections 2024-03-24T10:49:56.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.416+0000 7f0a66215740 1 -- 172.21.15.23:0/3903370298 wait complete. 2024-03-24T10:49:56.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.416+0000 7f0a66215740 1 -- start start 2024-03-24T10:49:56.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a66215740 1 Processor -- start 2024-03-24T10:49:56.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a66215740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 0x55e6b727d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a66215740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b727d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a66215740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b727dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a66215740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55e6b72eee50 con 0x55e6b7270d10 2024-03-24T10:49:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a66215740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e6b72eef90 con 0x55e6b72798f0 2024-03-24T10:49:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a66215740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e6b72ef0d0 con 0x55e6b72703f0 2024-03-24T10:49:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a66215740 10 client.0 _ll_register_callbacks cb 0x7f0a4c009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a60bc9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b727d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a60bc9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b727d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34754/0 (socket says 172.21.15.23:34754) 2024-03-24T10:49:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a60bc9640 1 -- 172.21.15.23:0/334153219 learned_addr learned my addr 172.21.15.23:0/334153219 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a61bcb640 1 --2- 172.21.15.23:0/334153219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b727dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a613ca640 1 --2- 172.21.15.23:0/334153219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 0x55e6b727d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a60bc9640 1 -- 172.21.15.23:0/334153219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 msgr2=0x55e6b727d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a60bc9640 1 --2- 172.21.15.23:0/334153219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 0x55e6b727d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a60bc9640 1 -- 172.21.15.23:0/334153219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 msgr2=0x55e6b727dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a60bc9640 1 --2- 172.21.15.23:0/334153219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b727dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.417+0000 7f0a60bc9640 1 -- 172.21.15.23:0/334153219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a5c1a4510 con 0x55e6b7270d10 2024-03-24T10:49:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a60bc9640 1 --2- 172.21.15.23:0/334153219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b727d5a0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x55e6b7278000 tx=0x7f0a4c0096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a09b2235cf801e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:56.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a44ff9640 1 -- 172.21.15.23:0/334153219 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c017070 con 0x55e6b7270d10 2024-03-24T10:49:56.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a44ff9640 1 -- 172.21.15.23:0/334153219 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f0a4c005270 con 0x55e6b7270d10 2024-03-24T10:49:56.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a66215740 1 -- 172.21.15.23:0/334153219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a54009dd0 con 0x55e6b7270d10 2024-03-24T10:49:56.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a44ff9640 1 -- 172.21.15.23:0/334153219 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0a4c00fe90 con 0x55e6b7270d10 2024-03-24T10:49:56.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a66215740 10 client.18240 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:56.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a66215740 1 -- 172.21.15.23:0/334153219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e6b7197920 con 0x55e6b7270d10 2024-03-24T10:49:56.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a44ff9640 1 -- 172.21.15.23:0/334153219 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a4c039080 con 0x55e6b7270d10 2024-03-24T10:49:56.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a44ff9640 1 -- 172.21.15.23:0/334153219 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f0a4c0105d0 con 0x55e6b7270d10 2024-03-24T10:49:56.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a66215740 10 client.18240 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:56.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a66215740 10 client.18240 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:49:56.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a66215740 20 client.18240 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:56.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a66215740 1 --2- 172.21.15.23:0/334153219 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55e6b72f08b0 0x55e6b72f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a66215740 4 client.18240 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:49:56.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a66215740 1 -- 172.21.15.23:0/334153219 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e6b6f62890 con 0x55e6b72f08b0 2024-03-24T10:49:56.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.418+0000 7f0a613ca640 1 --2- 172.21.15.23:0/334153219 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55e6b72f08b0 0x55e6b72f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a613ca640 1 --2- 172.21.15.23:0/334153219 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55e6b72f08b0 0x55e6b72f2cd0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5576eab80e7d5f73 server_cookie=db72282ad9910f04 in_seq=0 out_seq=0 2024-03-24T10:49:56.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a44ff9640 10 client.18240 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:49:56.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a44ff9640 1 -- 172.21.15.23:0/334153219 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e6b6f62890 con 0x55e6b72f08b0 2024-03-24T10:49:56.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a44ff9640 10 client.18240 handle_command_reply: tid=0 2024-03-24T10:49:56.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a66215740 1 client.18240 shutdown 2024-03-24T10:49:56.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a66215740 1 -- 172.21.15.23:0/334153219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 msgr2=0x55e6b727d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a66215740 1 --2- 172.21.15.23:0/334153219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b727d5a0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x55e6b7278000 tx=0x7f0a4c0096a0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a66215740 1 -- 172.21.15.23:0/334153219 shutdown_connections 2024-03-24T10:49:56.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a66215740 1 --2- 172.21.15.23:0/334153219 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x55e6b72f08b0 0x55e6b72f2cd0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a66215740 1 --2- 172.21.15.23:0/334153219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e6b72798f0 0x55e6b727dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a66215740 1 --2- 172.21.15.23:0/334153219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6b7270d10 0x55e6b727d5a0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a44ff9640 0 client.18240 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:49:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a66215740 1 --2- 172.21.15.23:0/334153219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e6b72703f0 0x55e6b727d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a66215740 1 -- 172.21.15.23:0/334153219 >> 172.21.15.23:0/334153219 conn(0x55e6b726ab20 msgr2=0x55e6b7269430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a66215740 1 -- 172.21.15.23:0/334153219 shutdown_connections 2024-03-24T10:49:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a66215740 1 -- 172.21.15.23:0/334153219 wait complete. 2024-03-24T10:49:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.419+0000 7f0a66215740 20 client.18240 trim_cache size 0 max 16384 2024-03-24T10:49:56.435 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:49:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.420+0000 7f0a6362e640 1 -- 172.21.15.23:0/3982946033 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a480a2230 msgr2=0x7f0a480a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.420+0000 7f0a6362e640 1 --2- 172.21.15.23:0/3982946033 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a480a2230 0x7f0a480a46f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400fc90 tx=0x7f0a54018000 comp rx=0 tx=0).stop 2024-03-24T10:49:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.420+0000 7f0a6362e640 1 -- 172.21.15.23:0/3982946033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5c100630 msgr2=0x7f0a5c198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.420+0000 7f0a6362e640 1 --2- 172.21.15.23:0/3982946033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5c100630 0x7f0a5c198c10 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f0a58009430 tx=0x7f0a5802c650 comp rx=0 tx=0).stop 2024-03-24T10:49:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.420+0000 7f0a6362e640 1 -- 172.21.15.23:0/3982946033 shutdown_connections 2024-03-24T10:49:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.420+0000 7f0a6362e640 1 --2- 172.21.15.23:0/3982946033 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0a480a2230 0x7f0a480a46f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.420+0000 7f0a6362e640 1 --2- 172.21.15.23:0/3982946033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a5c106640 0x7f0a5c1a3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.420+0000 7f0a6362e640 1 --2- 172.21.15.23:0/3982946033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a5c100ff0 0x7f0a5c199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.420+0000 7f0a6362e640 1 --2- 172.21.15.23:0/3982946033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a5c100630 0x7f0a5c198c10 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.420+0000 7f0a6362e640 1 -- 172.21.15.23:0/3982946033 >> 172.21.15.23:0/3982946033 conn(0x7f0a5c077770 msgr2=0x7f0a5c103260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.420+0000 7f0a6362e640 1 -- 172.21.15.23:0/3982946033 shutdown_connections 2024-03-24T10:49:56.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.420+0000 7f0a6362e640 1 -- 172.21.15.23:0/3982946033 wait complete. 2024-03-24T10:49:56.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:49:56.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.543+0000 7f2054b3e640 1 -- 172.21.15.23:0/3666154216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20501016a0 msgr2=0x7f2050101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.543+0000 7f2054b3e640 1 --2- 172.21.15.23:0/3666154216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20501016a0 0x7f2050101b20 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f2038009930 tx=0x7f203802bc30 comp rx=0 tx=0).stop 2024-03-24T10:49:56.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.543+0000 7f2054b3e640 1 -- 172.21.15.23:0/3666154216 shutdown_connections 2024-03-24T10:49:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.543+0000 7f2054b3e640 1 --2- 172.21.15.23:0/3666154216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2050102060 0x7f205010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.543+0000 7f2054b3e640 1 --2- 172.21.15.23:0/3666154216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20501016a0 0x7f2050101b20 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.543+0000 7f2054b3e640 1 --2- 172.21.15.23:0/3666154216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20501076b0 0x7f2050107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.543+0000 7f2054b3e640 1 -- 172.21.15.23:0/3666154216 >> 172.21.15.23:0/3666154216 conn(0x7f2050078b10 msgr2=0x7f2050078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.543+0000 7f2054b3e640 1 -- 172.21.15.23:0/3666154216 shutdown_connections 2024-03-24T10:49:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.543+0000 7f2054b3e640 1 -- 172.21.15.23:0/3666154216 wait complete. 2024-03-24T10:49:56.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f2054b3e640 1 Processor -- start 2024-03-24T10:49:56.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f2054b3e640 1 -- start start 2024-03-24T10:49:56.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f2054b3e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20501016a0 0x7f2050197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f2054b3e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2050102060 0x7f2050198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f2054b3e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20501076b0 0x7f205019c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f2054b3e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f205019cc00 con 0x7f20501016a0 2024-03-24T10:49:56.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f2054b3e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f205019cd70 con 0x7f2050102060 2024-03-24T10:49:56.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f2054b3e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f205019cf30 con 0x7f20501076b0 2024-03-24T10:49:56.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f204e59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20501016a0 0x7f2050197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f204e59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20501016a0 0x7f2050197d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34772/0 (socket says 172.21.15.23:34772) 2024-03-24T10:49:56.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f204e59c640 1 -- 172.21.15.23:0/2332486792 learned_addr learned my addr 172.21.15.23:0/2332486792 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f204dd9b640 1 --2- 172.21.15.23:0/2332486792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2050102060 0x7f2050198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f204ed9d640 1 --2- 172.21.15.23:0/2332486792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20501076b0 0x7f205019c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f204e59c640 1 -- 172.21.15.23:0/2332486792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20501076b0 msgr2=0x7f205019c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f204e59c640 1 --2- 172.21.15.23:0/2332486792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20501076b0 0x7f205019c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f204e59c640 1 -- 172.21.15.23:0/2332486792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2050102060 msgr2=0x7f2050198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f204e59c640 1 --2- 172.21.15.23:0/2332486792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2050102060 0x7f2050198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f204e59c640 1 -- 172.21.15.23:0/2332486792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2038009560 con 0x7f20501016a0 2024-03-24T10:49:56.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f204e59c640 1 --2- 172.21.15.23:0/2332486792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20501016a0 0x7f2050197d40 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f204000e9f0 tx=0x7f204000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f20477fe640 1 -- 172.21.15.23:0/2332486792 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2040009720 con 0x7f20501016a0 2024-03-24T10:49:56.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.544+0000 7f2054b3e640 1 -- 172.21.15.23:0/2332486792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f205019d210 con 0x7f20501016a0 2024-03-24T10:49:56.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.545+0000 7f20477fe640 1 -- 172.21.15.23:0/2332486792 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f204000cd80 con 0x7f20501016a0 2024-03-24T10:49:56.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.545+0000 7f20477fe640 1 -- 172.21.15.23:0/2332486792 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2040010690 con 0x7f20501016a0 2024-03-24T10:49:56.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.545+0000 7f2054b3e640 1 -- 172.21.15.23:0/2332486792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20501a4a90 con 0x7f20501016a0 2024-03-24T10:49:56.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.547+0000 7f20477fe640 1 -- 172.21.15.23:0/2332486792 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f20400046c0 con 0x7f20501016a0 2024-03-24T10:49:56.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.547+0000 7f2057801740 1 Processor -- start 2024-03-24T10:49:56.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f2057801740 1 -- start start 2024-03-24T10:49:56.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f2057801740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc93f0 0x55ff04fc97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f2057801740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc9d10 0x55ff04fd23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f2057801740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f2057801740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ff04f37840 con 0x55ff04fd28f0 2024-03-24T10:49:56.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f2057801740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ff04f34d70 con 0x55ff04fc93f0 2024-03-24T10:49:56.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f2057801740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ff04f30550 con 0x55ff04fc9d10 2024-03-24T10:49:56.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f204dd9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f204dd9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34788/0 (socket says 172.21.15.23:34788) 2024-03-24T10:49:56.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f204dd9b640 1 -- 172.21.15.23:0/3662472133 learned_addr learned my addr 172.21.15.23:0/3662472133 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f20477fe640 1 --2- 172.21.15.23:0/2332486792 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f203c0a2230 0x7f203c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f204e59c640 1 --2- 172.21.15.23:0/3662472133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc9d10 0x55ff04fd23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f204ed9d640 1 --2- 172.21.15.23:0/3662472133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc93f0 0x55ff04fc97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f204ed9d640 1 --2- 172.21.15.23:0/2332486792 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f203c0a2230 0x7f203c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f204dd9b640 1 -- 172.21.15.23:0/3662472133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc9d10 msgr2=0x55ff04fd23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f204dd9b640 1 --2- 172.21.15.23:0/3662472133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc9d10 0x55ff04fd23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f204dd9b640 1 -- 172.21.15.23:0/3662472133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc93f0 msgr2=0x55ff04fc97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f204dd9b640 1 --2- 172.21.15.23:0/3662472133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc93f0 0x55ff04fc97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f204dd9b640 1 -- 172.21.15.23:0/3662472133 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff04ec5de0 con 0x55ff04fd28f0 2024-03-24T10:49:56.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.548+0000 7f20477fe640 1 -- 172.21.15.23:0/2332486792 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f2040014070 con 0x7f20501016a0 2024-03-24T10:49:56.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.549+0000 7f204dd9b640 1 --2- 172.21.15.23:0/3662472133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd4ce0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f2038036040 tx=0x7f203802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2686c9c91c38d9eb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.549+0000 7f204ed9d640 1 --2- 172.21.15.23:0/2332486792 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f203c0a2230 0x7f203c0a46f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2048005c40 tx=0x7f2048004ab0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.549+0000 7f2044ff9640 1 -- 172.21.15.23:0/3662472133 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038004800 con 0x55ff04fd28f0 2024-03-24T10:49:56.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.549+0000 7f2044ff9640 1 -- 172.21.15.23:0/3662472133 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2038004960 con 0x55ff04fd28f0 2024-03-24T10:49:56.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.549+0000 7f2044ff9640 1 -- 172.21.15.23:0/3662472133 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20380050e0 con 0x55ff04fd28f0 2024-03-24T10:49:56.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.549+0000 7f2057801740 1 -- 172.21.15.23:0/3662472133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 msgr2=0x55ff04fd4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.549+0000 7f2057801740 1 --2- 172.21.15.23:0/3662472133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd4ce0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f2038036040 tx=0x7f203802c5d0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.549+0000 7f2057801740 1 -- 172.21.15.23:0/3662472133 shutdown_connections 2024-03-24T10:49:56.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.549+0000 7f2057801740 1 --2- 172.21.15.23:0/3662472133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd4ce0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.549+0000 7f2057801740 1 --2- 172.21.15.23:0/3662472133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc9d10 0x55ff04fd23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.549+0000 7f2057801740 1 --2- 172.21.15.23:0/3662472133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc93f0 0x55ff04fc97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.549+0000 7f2057801740 1 -- 172.21.15.23:0/3662472133 >> 172.21.15.23:0/3662472133 conn(0x55ff04fc3b20 msgr2=0x55ff04fc3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f2057801740 1 -- 172.21.15.23:0/3662472133 shutdown_connections 2024-03-24T10:49:56.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f2057801740 1 -- 172.21.15.23:0/3662472133 wait complete. 2024-03-24T10:49:56.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f2057801740 1 -- start start 2024-03-24T10:49:56.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f2057801740 1 Processor -- start 2024-03-24T10:49:56.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f2057801740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 0x55ff04fcf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f2057801740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 0x55ff04fcfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f2057801740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f2057801740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ff04c03430 con 0x55ff04fd28f0 2024-03-24T10:49:56.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f2057801740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ff04bbfe00 con 0x55ff04fc9d10 2024-03-24T10:49:56.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f2057801740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ff04eb5590 con 0x55ff04fc93f0 2024-03-24T10:49:56.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f2057801740 10 client.0 _ll_register_callbacks cb 0x55ff04f24fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:56.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f204ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f204ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34800/0 (socket says 172.21.15.23:34800) 2024-03-24T10:49:56.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f204ed9d640 1 -- 172.21.15.23:0/712814963 learned_addr learned my addr 172.21.15.23:0/712814963 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f204e59c640 1 --2- 172.21.15.23:0/712814963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 0x55ff04fcf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.550+0000 7f204dd9b640 1 --2- 172.21.15.23:0/712814963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 0x55ff04fcfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f204ed9d640 1 -- 172.21.15.23:0/712814963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 msgr2=0x55ff04fcf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f204ed9d640 1 --2- 172.21.15.23:0/712814963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 0x55ff04fcf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f204ed9d640 1 -- 172.21.15.23:0/712814963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 msgr2=0x55ff04fcfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f204ed9d640 1 --2- 172.21.15.23:0/712814963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 0x55ff04fcfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f204ed9d640 1 -- 172.21.15.23:0/712814963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f205010ee70 con 0x55ff04fd28f0 2024-03-24T10:49:56.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f204e59c640 1 --2- 172.21.15.23:0/712814963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 0x55ff04fcf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:56.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f204ed9d640 1 --2- 172.21.15.23:0/712814963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd0360 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f2048003dc0 tx=0x7f2048004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a0aececfe9efd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f2035ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:56.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f2035ffb640 1 -- 172.21.15.23:0/712814963 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20480045d0 con 0x55ff04fd28f0 2024-03-24T10:49:56.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f2035ffb640 1 -- 172.21.15.23:0/712814963 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f2048004730 con 0x55ff04fd28f0 2024-03-24T10:49:56.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f2057801740 1 -- 172.21.15.23:0/712814963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ff04c2f7e0 con 0x55ff04fd28f0 2024-03-24T10:49:56.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f2035ffb640 1 -- 172.21.15.23:0/712814963 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2048021470 con 0x55ff04fd28f0 2024-03-24T10:49:56.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f2057801740 10 client.18252 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:56.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f2057801740 1 -- 172.21.15.23:0/712814963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ff04f2ebd0 con 0x55ff04fd28f0 2024-03-24T10:49:56.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f2035ffb640 1 -- 172.21.15.23:0/712814963 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ff04c2f7e0 con 0x55ff04fd28f0 2024-03-24T10:49:56.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f2035ffb640 1 -- 172.21.15.23:0/712814963 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f2048032620 con 0x55ff04fd28f0 2024-03-24T10:49:56.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f2057801740 10 client.18252 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:56.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f2057801740 10 client.18252 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:49:56.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.551+0000 7f2057801740 20 client.18252 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:56.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.552+0000 7f2057801740 1 --2- 172.21.15.23:0/712814963 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ff04fe29d0 0x55ff04fe4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.552+0000 7f2057801740 4 client.18252 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:49:56.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.552+0000 7f2057801740 1 -- 172.21.15.23:0/712814963 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff04c6c720 con 0x55ff04fe29d0 2024-03-24T10:49:56.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.552+0000 7f204e59c640 1 --2- 172.21.15.23:0/712814963 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ff04fe29d0 0x55ff04fe4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.552+0000 7f204e59c640 1 --2- 172.21.15.23:0/712814963 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ff04fe29d0 0x55ff04fe4e10 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7549317bdb7c8265 server_cookie=d4fa32c9985335b2 in_seq=0 out_seq=0 2024-03-24T10:49:56.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.552+0000 7f2035ffb640 10 client.18252 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:49:56.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2035ffb640 1 -- 172.21.15.23:0/712814963 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ff04c6c720 con 0x55ff04fe29d0 2024-03-24T10:49:56.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2035ffb640 10 client.18252 handle_command_reply: tid=0 2024-03-24T10:49:56.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2057801740 1 client.18252 shutdown 2024-03-24T10:49:56.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2057801740 1 -- 172.21.15.23:0/712814963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 msgr2=0x55ff04fd0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2057801740 1 --2- 172.21.15.23:0/712814963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd0360 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f2048003dc0 tx=0x7f2048004a40 comp rx=0 tx=0).stop 2024-03-24T10:49:56.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2057801740 1 -- 172.21.15.23:0/712814963 shutdown_connections 2024-03-24T10:49:56.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2057801740 1 --2- 172.21.15.23:0/712814963 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ff04fe29d0 0x55ff04fe4e10 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2057801740 1 --2- 172.21.15.23:0/712814963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd0360 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2057801740 1 --2- 172.21.15.23:0/712814963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 0x55ff04fcfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2035ffb640 0 client.18252 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:49:56.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2057801740 1 --2- 172.21.15.23:0/712814963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 0x55ff04fcf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2057801740 1 -- 172.21.15.23:0/712814963 >> 172.21.15.23:0/712814963 conn(0x55ff04fc3b20 msgr2=0x55ff04fc5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2057801740 1 -- 172.21.15.23:0/712814963 shutdown_connections 2024-03-24T10:49:56.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2057801740 1 -- 172.21.15.23:0/712814963 wait complete. 2024-03-24T10:49:56.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.553+0000 7f2057801740 20 client.18252 trim_cache size 0 max 16384 2024-03-24T10:49:56.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.566+0000 7f2057801740 1 Processor -- start 2024-03-24T10:49:56.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.566+0000 7f2057801740 1 -- start start 2024-03-24T10:49:56.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.566+0000 7f2057801740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 0x55ff04fd2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.566+0000 7f2057801740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 0x55ff04fd0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.566+0000 7f2057801740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fea720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.566+0000 7f2057801740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ff04fdf3f0 con 0x55ff04fd28f0 2024-03-24T10:49:56.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.566+0000 7f2057801740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ff04fc8e30 con 0x55ff04fc9d10 2024-03-24T10:49:56.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.566+0000 7f2057801740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ff04fc3950 con 0x55ff04fc93f0 2024-03-24T10:49:56.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.566+0000 7f204ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fea720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.566+0000 7f204ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fea720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34816/0 (socket says 172.21.15.23:34816) 2024-03-24T10:49:56.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.566+0000 7f204ed9d640 1 -- 172.21.15.23:0/2683117085 learned_addr learned my addr 172.21.15.23:0/2683117085 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f204e59c640 1 --2- 172.21.15.23:0/2683117085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 0x55ff04fd2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f204dd9b640 1 --2- 172.21.15.23:0/2683117085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 0x55ff04fd0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f204ed9d640 1 -- 172.21.15.23:0/2683117085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 msgr2=0x55ff04fd2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f204ed9d640 1 --2- 172.21.15.23:0/2683117085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 0x55ff04fd2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f204ed9d640 1 -- 172.21.15.23:0/2683117085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 msgr2=0x55ff04fd0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f204ed9d640 1 --2- 172.21.15.23:0/2683117085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 0x55ff04fd0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f204ed9d640 1 -- 172.21.15.23:0/2683117085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff04ecac80 con 0x55ff04fd28f0 2024-03-24T10:49:56.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f204dd9b640 1 --2- 172.21.15.23:0/2683117085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 0x55ff04fd0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:49:56.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f204e59c640 1 --2- 172.21.15.23:0/2683117085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 0x55ff04fd2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:49:56.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f204ed9d640 1 --2- 172.21.15.23:0/2683117085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fea720 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f2048013470 tx=0x7f2048005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8923dd80fee228a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f2037fff640 1 -- 172.21.15.23:0/2683117085 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2048013800 con 0x55ff04fd28f0 2024-03-24T10:49:56.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f2037fff640 1 -- 172.21.15.23:0/2683117085 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2048003eb0 con 0x55ff04fd28f0 2024-03-24T10:49:56.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f2037fff640 1 -- 172.21.15.23:0/2683117085 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2048004680 con 0x55ff04fd28f0 2024-03-24T10:49:56.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f2057801740 1 -- 172.21.15.23:0/2683117085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 msgr2=0x55ff04fea720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.567+0000 7f2057801740 1 --2- 172.21.15.23:0/2683117085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fea720 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f2048013470 tx=0x7f2048005f10 comp rx=0 tx=0).stop 2024-03-24T10:49:56.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 -- 172.21.15.23:0/2683117085 shutdown_connections 2024-03-24T10:49:56.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 --2- 172.21.15.23:0/2683117085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fea720 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 --2- 172.21.15.23:0/2683117085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 0x55ff04fd0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 --2- 172.21.15.23:0/2683117085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 0x55ff04fd2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 -- 172.21.15.23:0/2683117085 >> 172.21.15.23:0/2683117085 conn(0x55ff04fc3b20 msgr2=0x55ff04fc4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 -- 172.21.15.23:0/2683117085 shutdown_connections 2024-03-24T10:49:56.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 -- 172.21.15.23:0/2683117085 wait complete. 2024-03-24T10:49:56.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 -- start start 2024-03-24T10:49:56.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 Processor -- start 2024-03-24T10:49:56.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 0x55ff04fda750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 0x55ff04fd7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55ff04fdae20 con 0x55ff04fd28f0 2024-03-24T10:49:56.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ff04fdaf90 con 0x55ff04fc9d10 2024-03-24T10:49:56.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ff04fd81a0 con 0x55ff04fc93f0 2024-03-24T10:49:56.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f2057801740 10 client.0 _ll_register_callbacks cb 0x7f2048003a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:56.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f204ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f204ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34824/0 (socket says 172.21.15.23:34824) 2024-03-24T10:49:56.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f204ed9d640 1 -- 172.21.15.23:0/2120023153 learned_addr learned my addr 172.21.15.23:0/2120023153 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f204dd9b640 1 --2- 172.21.15.23:0/2120023153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 0x55ff04fd7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.568+0000 7f204e59c640 1 --2- 172.21.15.23:0/2120023153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 0x55ff04fda750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f204ed9d640 1 -- 172.21.15.23:0/2120023153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 msgr2=0x55ff04fda750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f204ed9d640 1 --2- 172.21.15.23:0/2120023153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 0x55ff04fda750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f204ed9d640 1 -- 172.21.15.23:0/2120023153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 msgr2=0x55ff04fd7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f204ed9d640 1 --2- 172.21.15.23:0/2120023153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 0x55ff04fd7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f204ed9d640 1 -- 172.21.15.23:0/2120023153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff04f2ebd0 con 0x55ff04fd28f0 2024-03-24T10:49:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f204ed9d640 1 --2- 172.21.15.23:0/2120023153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd7c60 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f204800f820 tx=0x7f2048032690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22feb062f1e20b2a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2035ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2035ffb640 1 -- 172.21.15.23:0/2120023153 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2048032e40 con 0x55ff04fd28f0 2024-03-24T10:49:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2035ffb640 1 -- 172.21.15.23:0/2120023153 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f2048037070 con 0x55ff04fd28f0 2024-03-24T10:49:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2057801740 1 -- 172.21.15.23:0/2120023153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2038004960 con 0x55ff04fd28f0 2024-03-24T10:49:56.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2035ffb640 1 -- 172.21.15.23:0/2120023153 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2048004b20 con 0x55ff04fd28f0 2024-03-24T10:49:56.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2057801740 10 client.18258 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:56.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2057801740 1 -- 172.21.15.23:0/2120023153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ff04ef0920 con 0x55ff04fd28f0 2024-03-24T10:49:56.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2035ffb640 1 -- 172.21.15.23:0/2120023153 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2038004960 con 0x55ff04fd28f0 2024-03-24T10:49:56.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2035ffb640 1 -- 172.21.15.23:0/2120023153 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f2048021c90 con 0x55ff04fd28f0 2024-03-24T10:49:56.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2057801740 10 client.18258 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:56.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2057801740 10 client.18258 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:49:56.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2057801740 20 client.18258 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:56.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2057801740 1 --2- 172.21.15.23:0/2120023153 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ff05049c20 0x55ff0504c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2057801740 4 client.18258 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:49:56.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.569+0000 7f2057801740 1 -- 172.21.15.23:0/2120023153 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ff04cbb890 con 0x55ff05049c20 2024-03-24T10:49:56.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f204e59c640 1 --2- 172.21.15.23:0/2120023153 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ff05049c20 0x55ff0504c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2035ffb640 10 client.18258 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:49:56.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f204e59c640 1 --2- 172.21.15.23:0/2120023153 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ff05049c20 0x55ff0504c040 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=624195ebf6af8c4 server_cookie=c4c2926df12ff81d in_seq=0 out_seq=0 2024-03-24T10:49:56.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2035ffb640 1 -- 172.21.15.23:0/2120023153 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ff04cbb890 con 0x55ff05049c20 2024-03-24T10:49:56.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2035ffb640 10 client.18258 handle_command_reply: tid=0 2024-03-24T10:49:56.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2057801740 1 client.18258 shutdown 2024-03-24T10:49:56.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2057801740 1 -- 172.21.15.23:0/2120023153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 msgr2=0x55ff04fd7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2057801740 1 --2- 172.21.15.23:0/2120023153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd7c60 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f204800f820 tx=0x7f2048032690 comp rx=0 tx=0).stop 2024-03-24T10:49:56.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2057801740 1 -- 172.21.15.23:0/2120023153 shutdown_connections 2024-03-24T10:49:56.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2057801740 1 --2- 172.21.15.23:0/2120023153 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x55ff05049c20 0x55ff0504c040 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2057801740 1 --2- 172.21.15.23:0/2120023153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ff04fd28f0 0x55ff04fd7c60 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2035ffb640 0 client.18258 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:49:56.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2057801740 1 --2- 172.21.15.23:0/2120023153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ff04fc9d10 0x55ff04fd7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2057801740 1 --2- 172.21.15.23:0/2120023153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ff04fc93f0 0x55ff04fda750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2057801740 1 -- 172.21.15.23:0/2120023153 >> 172.21.15.23:0/2120023153 conn(0x55ff04fc3b20 msgr2=0x55ff04fc5570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2057801740 1 -- 172.21.15.23:0/2120023153 shutdown_connections 2024-03-24T10:49:56.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2057801740 1 -- 172.21.15.23:0/2120023153 wait complete. 2024-03-24T10:49:56.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.570+0000 7f2057801740 20 client.18258 trim_cache size 0 max 16384 2024-03-24T10:49:56.582 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:49:56.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.571+0000 7f2054b3e640 1 -- 172.21.15.23:0/2332486792 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f203c0a2230 msgr2=0x7f203c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.571+0000 7f2054b3e640 1 --2- 172.21.15.23:0/2332486792 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f203c0a2230 0x7f203c0a46f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2048005c40 tx=0x7f2048004ab0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.571+0000 7f2054b3e640 1 -- 172.21.15.23:0/2332486792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20501016a0 msgr2=0x7f2050197d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.571+0000 7f2054b3e640 1 --2- 172.21.15.23:0/2332486792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20501016a0 0x7f2050197d40 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f204000e9f0 tx=0x7f204000edf0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.571+0000 7f2054b3e640 1 -- 172.21.15.23:0/2332486792 shutdown_connections 2024-03-24T10:49:56.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.571+0000 7f2054b3e640 1 --2- 172.21.15.23:0/2332486792 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f203c0a2230 0x7f203c0a46f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.571+0000 7f2054b3e640 1 --2- 172.21.15.23:0/2332486792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20501076b0 0x7f205019c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.571+0000 7f2054b3e640 1 --2- 172.21.15.23:0/2332486792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2050102060 0x7f2050198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.571+0000 7f2054b3e640 1 --2- 172.21.15.23:0/2332486792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20501016a0 0x7f2050197d40 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.571+0000 7f2054b3e640 1 -- 172.21.15.23:0/2332486792 >> 172.21.15.23:0/2332486792 conn(0x7f2050078b10 msgr2=0x7f20501040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.571+0000 7f2054b3e640 1 -- 172.21.15.23:0/2332486792 shutdown_connections 2024-03-24T10:49:56.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.571+0000 7f2054b3e640 1 -- 172.21.15.23:0/2332486792 wait complete. 2024-03-24T10:49:56.585 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:56 smithi023 ceph-mon[32466]: pgmap v306: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:56.585 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:56 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4037188297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:49:56.586 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:49:56.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.690+0000 7fcd6b0d4640 1 -- 172.21.15.23:0/3814943719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd641076b0 msgr2=0x7fcd64107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.690+0000 7fcd6b0d4640 1 --2- 172.21.15.23:0/3814943719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd641076b0 0x7fcd64107a90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c009c10 tx=0x7fcd5c02bef0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.690+0000 7fcd6b0d4640 1 -- 172.21.15.23:0/3814943719 shutdown_connections 2024-03-24T10:49:56.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.690+0000 7fcd6b0d4640 1 --2- 172.21.15.23:0/3814943719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd64102060 0x7fcd6410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.690+0000 7fcd6b0d4640 1 --2- 172.21.15.23:0/3814943719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd641016a0 0x7fcd64101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.690+0000 7fcd6b0d4640 1 --2- 172.21.15.23:0/3814943719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd641076b0 0x7fcd64107a90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.690+0000 7fcd6b0d4640 1 -- 172.21.15.23:0/3814943719 >> 172.21.15.23:0/3814943719 conn(0x7fcd64078b10 msgr2=0x7fcd64078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.690+0000 7fcd6b0d4640 1 -- 172.21.15.23:0/3814943719 shutdown_connections 2024-03-24T10:49:56.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.690+0000 7fcd6b0d4640 1 -- 172.21.15.23:0/3814943719 wait complete. 2024-03-24T10:49:56.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.690+0000 7fcd6b0d4640 1 Processor -- start 2024-03-24T10:49:56.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.690+0000 7fcd6b0d4640 1 -- start start 2024-03-24T10:49:56.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd6b0d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd641016a0 0x7fcd6419da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd6b0d4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd64102060 0x7fcd6419dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd6b0d4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd641076b0 0x7fcd64197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd6b0d4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fcd64198310 con 0x7fcd641076b0 2024-03-24T10:49:56.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd6b0d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcd64198480 con 0x7fcd641016a0 2024-03-24T10:49:56.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd6b0d4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcd64198640 con 0x7fcd64102060 2024-03-24T10:49:56.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd69671640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd641076b0 0x7fcd64197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd69671640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd641076b0 0x7fcd64197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34840/0 (socket says 172.21.15.23:34840) 2024-03-24T10:49:56.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd69671640 1 -- 172.21.15.23:0/1849473367 learned_addr learned my addr 172.21.15.23:0/1849473367 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd68e70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd641016a0 0x7fcd6419da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd63fff640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd64102060 0x7fcd6419dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd69671640 1 -- 172.21.15.23:0/1849473367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd64102060 msgr2=0x7fcd6419dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd69671640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd64102060 0x7fcd6419dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd69671640 1 -- 172.21.15.23:0/1849473367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd641016a0 msgr2=0x7fcd6419da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd69671640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd641016a0 0x7fcd6419da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd69671640 1 -- 172.21.15.23:0/1849473367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5c009840 con 0x7fcd641076b0 2024-03-24T10:49:56.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd63fff640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd64102060 0x7fcd6419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:56.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd68e70640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd641016a0 0x7fcd6419da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd69671640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd641076b0 0x7fcd64197b80 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fcd5400e970 tx=0x7fcd5400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd61ffb640 1 -- 172.21.15.23:0/1849473367 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd54009720 con 0x7fcd641076b0 2024-03-24T10:49:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd6b0d4640 1 -- 172.21.15.23:0/1849473367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd64198920 con 0x7fcd641076b0 2024-03-24T10:49:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.691+0000 7fcd61ffb640 1 -- 172.21.15.23:0/1849473367 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd54005500 con 0x7fcd641076b0 2024-03-24T10:49:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.692+0000 7fcd61ffb640 1 -- 172.21.15.23:0/1849473367 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd54010430 con 0x7fcd641076b0 2024-03-24T10:49:56.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.692+0000 7fcd6b0d4640 1 -- 172.21.15.23:0/1849473367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd641a49f0 con 0x7fcd641076b0 2024-03-24T10:49:56.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd61ffb640 1 -- 172.21.15.23:0/1849473367 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fcd54004f30 con 0x7fcd641076b0 2024-03-24T10:49:56.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd6dd90740 1 Processor -- start 2024-03-24T10:49:56.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd6dd90740 1 -- start start 2024-03-24T10:49:56.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd6dd90740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 0x55da0fcc47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd6dd90740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fccd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd6dd90740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fccfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd6dd90740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55da0fc32840 con 0x55da0fccd8f0 2024-03-24T10:49:56.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd6dd90740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55da0fc2fd70 con 0x55da0fcc4d10 2024-03-24T10:49:56.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd6dd90740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55da0fc2b550 con 0x55da0fcc43f0 2024-03-24T10:49:56.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd61ffb640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcd500a2230 0x7fcd500a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd63fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fccfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd68e70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fccd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd63fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fccfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34850/0 (socket says 172.21.15.23:34850) 2024-03-24T10:49:56.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd63fff640 1 -- 172.21.15.23:0/957357469 learned_addr learned my addr 172.21.15.23:0/957357469 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd69671640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcd500a2230 0x7fcd500a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.694+0000 7fcd63fff640 1 --2- 172.21.15.23:0/957357469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 0x55da0fcc47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.695+0000 7fcd61ffb640 1 -- 172.21.15.23:0/1849473367 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fcd54014070 con 0x7fcd641076b0 2024-03-24T10:49:56.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.695+0000 7fcd68e70640 1 -- 172.21.15.23:0/957357469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 msgr2=0x55da0fcc47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.695+0000 7fcd68e70640 1 --2- 172.21.15.23:0/957357469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 0x55da0fcc47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.695+0000 7fcd68e70640 1 -- 172.21.15.23:0/957357469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 msgr2=0x55da0fccfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.695+0000 7fcd68e70640 1 --2- 172.21.15.23:0/957357469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fccfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.695+0000 7fcd68e70640 1 -- 172.21.15.23:0/957357469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da0fbc0de0 con 0x55da0fcc4d10 2024-03-24T10:49:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.695+0000 7fcd63fff640 1 --2- 172.21.15.23:0/957357469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fccfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.695+0000 7fcd69671640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcd500a2230 0x7fcd500a46f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcd5409acb0 tx=0x7fcd5406dd90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.695+0000 7fcd68e70640 1 --2- 172.21.15.23:0/957357469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fccd3b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c009be0 tx=0x7fcd5c033e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11a00a4f7d89fd36 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.695+0000 7fcd5affd640 1 -- 172.21.15.23:0/957357469 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c02ca90 con 0x55da0fcc4d10 2024-03-24T10:49:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.695+0000 7fcd5affd640 1 -- 172.21.15.23:0/957357469 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd5c02cde0 con 0x55da0fcc4d10 2024-03-24T10:49:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.695+0000 7fcd5affd640 1 -- 172.21.15.23:0/957357469 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c004240 con 0x55da0fcc4d10 2024-03-24T10:49:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.696+0000 7fcd6dd90740 1 -- 172.21.15.23:0/957357469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 msgr2=0x55da0fccd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.696+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/957357469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fccd3b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c009be0 tx=0x7fcd5c033e80 comp rx=0 tx=0).stop 2024-03-24T10:49:56.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.696+0000 7fcd6dd90740 1 -- 172.21.15.23:0/957357469 shutdown_connections 2024-03-24T10:49:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.696+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/957357469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fccfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.696+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/957357469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fccd3b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.696+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/957357469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 0x55da0fcc47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.696+0000 7fcd6dd90740 1 -- 172.21.15.23:0/957357469 >> 172.21.15.23:0/957357469 conn(0x55da0fcbeb20 msgr2=0x55da0fcbef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.696+0000 7fcd6dd90740 1 -- 172.21.15.23:0/957357469 shutdown_connections 2024-03-24T10:49:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.696+0000 7fcd6dd90740 1 -- 172.21.15.23:0/957357469 wait complete. 2024-03-24T10:49:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.696+0000 7fcd6dd90740 1 -- start start 2024-03-24T10:49:56.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.696+0000 7fcd6dd90740 1 Processor -- start 2024-03-24T10:49:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd6dd90740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 0x55da0fcca990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd6dd90740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fccaed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd6dd90740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fccb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd6dd90740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55da0f8fe430 con 0x55da0fccd8f0 2024-03-24T10:49:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd6dd90740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55da0f8bae00 con 0x55da0fcc4d10 2024-03-24T10:49:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd6dd90740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55da0fbb0590 con 0x55da0fcc43f0 2024-03-24T10:49:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd6dd90740 10 client.0 _ll_register_callbacks cb 0x55da0fc1ffc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:56.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd63fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 0x55da0fcca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd68e70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fccaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd63fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 0x55da0fcca990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:55768/0 (socket says 172.21.15.23:55768) 2024-03-24T10:49:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd63fff640 1 -- 172.21.15.23:0/2188540617 learned_addr learned my addr 172.21.15.23:0/2188540617 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd68e70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fccaed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:55746/0 (socket says 172.21.15.23:55746) 2024-03-24T10:49:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd63fff640 1 --2- 172.21.15.23:0/2188540617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fccb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd63fff640 1 -- 172.21.15.23:0/2188540617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 msgr2=0x55da0fccaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd63fff640 1 --2- 172.21.15.23:0/2188540617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fccaed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd63fff640 1 -- 172.21.15.23:0/2188540617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 msgr2=0x55da0fccb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd63fff640 1 --2- 172.21.15.23:0/2188540617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fccb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd63fff640 1 -- 172.21.15.23:0/2188540617 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd6410ee70 con 0x55da0fcc43f0 2024-03-24T10:49:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd63fff640 1 --2- 172.21.15.23:0/2188540617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 0x55da0fcca990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c013df0 tx=0x7fcd4c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e4e98629ab42d5d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd43fff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:49:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd43fff640 1 -- 172.21.15.23:0/2188540617 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c016070 con 0x55da0fcc43f0 2024-03-24T10:49:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd6dd90740 1 -- 172.21.15.23:0/2188540617 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55da0f92a7e0 con 0x55da0fcc43f0 2024-03-24T10:49:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.697+0000 7fcd43fff640 1 -- 172.21.15.23:0/2188540617 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fcd4c004460 con 0x55da0fcc43f0 2024-03-24T10:49:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.698+0000 7fcd43fff640 1 -- 172.21.15.23:0/2188540617 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcd4c00fee0 con 0x55da0fcc43f0 2024-03-24T10:49:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.698+0000 7fcd6dd90740 10 client.28126 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.698+0000 7fcd6dd90740 1 -- 172.21.15.23:0/2188540617 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55da0fc29bd0 con 0x55da0fcc43f0 2024-03-24T10:49:56.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.698+0000 7fcd43fff640 1 -- 172.21.15.23:0/2188540617 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd4c02d020 con 0x55da0fcc43f0 2024-03-24T10:49:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.698+0000 7fcd43fff640 1 -- 172.21.15.23:0/2188540617 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fcd4c0130d0 con 0x55da0fcc43f0 2024-03-24T10:49:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.698+0000 7fcd6dd90740 10 client.28126 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.698+0000 7fcd6dd90740 10 client.28126 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.698+0000 7fcd6dd90740 20 client.28126 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.698+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/2188540617 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55da0fcdd9c0 0x55da0fcdfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.698+0000 7fcd6dd90740 4 client.28126 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:49:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.698+0000 7fcd6dd90740 1 -- 172.21.15.23:0/2188540617 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da0f967720 con 0x55da0fcdd9c0 2024-03-24T10:49:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.698+0000 7fcd68e70640 1 --2- 172.21.15.23:0/2188540617 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55da0fcdd9c0 0x55da0fcdfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.698+0000 7fcd43fff640 10 client.28126 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:56.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.698+0000 7fcd68e70640 1 --2- 172.21.15.23:0/2188540617 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55da0fcdd9c0 0x55da0fcdfde0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5277883f81473c0b server_cookie=2bca0d9b0d5ef005 in_seq=0 out_seq=0 2024-03-24T10:49:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd43fff640 1 -- 172.21.15.23:0/2188540617 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55da0f967720 con 0x55da0fcdd9c0 2024-03-24T10:49:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd43fff640 10 client.28126 handle_command_reply: tid=0 2024-03-24T10:49:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd6dd90740 1 client.28126 shutdown 2024-03-24T10:49:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd6dd90740 1 -- 172.21.15.23:0/2188540617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 msgr2=0x55da0fcca990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/2188540617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 0x55da0fcca990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c013df0 tx=0x7fcd4c004730 comp rx=0 tx=0).stop 2024-03-24T10:49:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd6dd90740 1 -- 172.21.15.23:0/2188540617 shutdown_connections 2024-03-24T10:49:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/2188540617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fccb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/2188540617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fccaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/2188540617 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55da0fcdd9c0 0x55da0fcdfde0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/2188540617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 0x55da0fcca990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd6dd90740 1 -- 172.21.15.23:0/2188540617 >> 172.21.15.23:0/2188540617 conn(0x55da0fcbeb20 msgr2=0x55da0fcc0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd43fff640 0 client.28126 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:56.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd6dd90740 1 -- 172.21.15.23:0/2188540617 shutdown_connections 2024-03-24T10:49:56.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd6dd90740 1 -- 172.21.15.23:0/2188540617 wait complete. 2024-03-24T10:49:56.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.699+0000 7fcd6dd90740 20 client.28126 trim_cache size 0 max 16384 2024-03-24T10:49:56.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd6dd90740 1 Processor -- start 2024-03-24T10:49:56.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd6dd90740 1 -- start start 2024-03-24T10:49:56.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd6dd90740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 0x55da0fcd9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd6dd90740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fce56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd6dd90740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fcd5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd6dd90740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55da0fcbf8f0 con 0x55da0fccd8f0 2024-03-24T10:49:56.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd6dd90740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55da0fcbf6f0 con 0x55da0fcc4d10 2024-03-24T10:49:56.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd6dd90740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55da0fcafe20 con 0x55da0fcc43f0 2024-03-24T10:49:56.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd68e70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fce56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd63fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fcd5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd68e70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fce56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:55758/0 (socket says 172.21.15.23:55758) 2024-03-24T10:49:56.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd68e70640 1 -- 172.21.15.23:0/3149772904 learned_addr learned my addr 172.21.15.23:0/3149772904 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd63fff640 1 --2- 172.21.15.23:0/3149772904 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 0x55da0fcd9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd63fff640 1 -- 172.21.15.23:0/3149772904 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 msgr2=0x55da0fcd9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd63fff640 1 --2- 172.21.15.23:0/3149772904 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 0x55da0fcd9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd63fff640 1 -- 172.21.15.23:0/3149772904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 msgr2=0x55da0fce56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd63fff640 1 --2- 172.21.15.23:0/3149772904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fce56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd63fff640 1 -- 172.21.15.23:0/3149772904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da0fbc5c80 con 0x55da0fccd8f0 2024-03-24T10:49:56.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd68e70640 1 --2- 172.21.15.23:0/3149772904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fce56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:56.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.713+0000 7fcd63fff640 1 --2- 172.21.15.23:0/3149772904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fcd5740 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c015010 tx=0x7fcd4c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d59cdc2104071e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.714+0000 7fcd5a7fc640 1 -- 172.21.15.23:0/3149772904 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c016040 con 0x55da0fccd8f0 2024-03-24T10:49:56.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.714+0000 7fcd5a7fc640 1 -- 172.21.15.23:0/3149772904 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd4c02d070 con 0x55da0fccd8f0 2024-03-24T10:49:56.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.714+0000 7fcd5a7fc640 1 -- 172.21.15.23:0/3149772904 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c004260 con 0x55da0fccd8f0 2024-03-24T10:49:56.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.714+0000 7fcd6dd90740 1 -- 172.21.15.23:0/3149772904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 msgr2=0x55da0fcd5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.714+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/3149772904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fcd5740 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c015010 tx=0x7fcd4c009260 comp rx=0 tx=0).stop 2024-03-24T10:49:56.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.714+0000 7fcd6dd90740 1 -- 172.21.15.23:0/3149772904 shutdown_connections 2024-03-24T10:49:56.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.714+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/3149772904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fccd8f0 0x55da0fcd5740 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.714+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/3149772904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc4d10 0x55da0fce56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.714+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/3149772904 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fcc43f0 0x55da0fcd9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.714+0000 7fcd6dd90740 1 -- 172.21.15.23:0/3149772904 >> 172.21.15.23:0/3149772904 conn(0x55da0fcbeb20 msgr2=0x55da0fcca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.714+0000 7fcd6dd90740 1 -- 172.21.15.23:0/3149772904 shutdown_connections 2024-03-24T10:49:56.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.714+0000 7fcd6dd90740 1 -- 172.21.15.23:0/3149772904 wait complete. 2024-03-24T10:49:56.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.715+0000 7fcd6dd90740 1 -- start start 2024-03-24T10:49:56.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.715+0000 7fcd6dd90740 1 Processor -- start 2024-03-24T10:49:56.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.715+0000 7fcd6dd90740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc43f0 0x55da0fcd5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.715+0000 7fcd6dd90740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fcc4d10 0x55da0fcd1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.715+0000 7fcd6dd90740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fccd8f0 0x55da0fcd2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.715+0000 7fcd6dd90740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55da0fcd2680 con 0x55da0fcc4d10 2024-03-24T10:49:56.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.715+0000 7fcd6dd90740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55da0fcd27f0 con 0x55da0fcc43f0 2024-03-24T10:49:56.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.715+0000 7fcd6dd90740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55da0fcd29b0 con 0x55da0fccd8f0 2024-03-24T10:49:56.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.715+0000 7fcd6dd90740 10 client.0 _ll_register_callbacks cb 0x7fcd4c0139f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.715+0000 7fcd68e70640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fcc4d10 0x55da0fcd1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.715+0000 7fcd68e70640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fcc4d10 0x55da0fcd1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34880/0 (socket says 172.21.15.23:34880) 2024-03-24T10:49:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.715+0000 7fcd68e70640 1 -- 172.21.15.23:0/1457858387 learned_addr learned my addr 172.21.15.23:0/1457858387 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.715+0000 7fcd63fff640 1 --2- 172.21.15.23:0/1457858387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc43f0 0x55da0fcd5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.715+0000 7fcd63fff640 1 --2- 172.21.15.23:0/1457858387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fccd8f0 0x55da0fcd2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd68e70640 1 -- 172.21.15.23:0/1457858387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fccd8f0 msgr2=0x55da0fcd2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd68e70640 1 --2- 172.21.15.23:0/1457858387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fccd8f0 0x55da0fcd2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd68e70640 1 -- 172.21.15.23:0/1457858387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc43f0 msgr2=0x55da0fcd5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:56.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd68e70640 1 --2- 172.21.15.23:0/1457858387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc43f0 0x55da0fcd5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd68e70640 1 -- 172.21.15.23:0/1457858387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da0fbc0de0 con 0x55da0fcc4d10 2024-03-24T10:49:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd63fff640 1 --2- 172.21.15.23:0/1457858387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fccd8f0 0x55da0fcd2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:49:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd68e70640 1 --2- 172.21.15.23:0/1457858387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fcc4d10 0x55da0fcd1c00 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c02bf30 tx=0x7fcd5c005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e445926d960e1870 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd43fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd43fff640 1 -- 172.21.15.23:0/1457858387 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c003c70 con 0x55da0fcc4d10 2024-03-24T10:49:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd43fff640 1 -- 172.21.15.23:0/1457858387 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fcd5c0333e0 con 0x55da0fcc4d10 2024-03-24T10:49:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd6dd90740 1 -- 172.21.15.23:0/1457858387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd5c02cde0 con 0x55da0fcc4d10 2024-03-24T10:49:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd43fff640 1 -- 172.21.15.23:0/1457858387 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcd5c02c9b0 con 0x55da0fcc4d10 2024-03-24T10:49:56.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd6dd90740 10 client.18276 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd6dd90740 1 -- 172.21.15.23:0/1457858387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55da0fc29bd0 con 0x55da0fcc4d10 2024-03-24T10:49:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd43fff640 1 -- 172.21.15.23:0/1457858387 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd5c063020 con 0x55da0fcc4d10 2024-03-24T10:49:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd43fff640 1 -- 172.21.15.23:0/1457858387 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fcd5c003f20 con 0x55da0fcc4d10 2024-03-24T10:49:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.716+0000 7fcd6dd90740 10 client.18276 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 10 client.18276 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 20 client.18276 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/1457858387 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55da0fd45120 0x55da0fd47540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 4 client.18276 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:49:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 1 -- 172.21.15.23:0/1457858387 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55da0f9b6890 con 0x55da0fd45120 2024-03-24T10:49:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd63fff640 1 --2- 172.21.15.23:0/1457858387 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55da0fd45120 0x55da0fd47540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd43fff640 10 client.18276 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd63fff640 1 --2- 172.21.15.23:0/1457858387 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55da0fd45120 0x55da0fd47540 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17e0e1137a2e6448 server_cookie=fb04b96afa432d8c in_seq=0 out_seq=0 2024-03-24T10:49:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd43fff640 1 -- 172.21.15.23:0/1457858387 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55da0f9b6890 con 0x55da0fd45120 2024-03-24T10:49:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd43fff640 10 client.18276 handle_command_reply: tid=0 2024-03-24T10:49:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 1 client.18276 shutdown 2024-03-24T10:49:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 1 -- 172.21.15.23:0/1457858387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fcc4d10 msgr2=0x55da0fcd1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/1457858387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fcc4d10 0x55da0fcd1c00 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c02bf30 tx=0x7fcd5c005c80 comp rx=0 tx=0).stop 2024-03-24T10:49:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 1 -- 172.21.15.23:0/1457858387 shutdown_connections 2024-03-24T10:49:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/1457858387 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55da0fd45120 0x55da0fd47540 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/1457858387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da0fccd8f0 0x55da0fcd2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/1457858387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da0fcc4d10 0x55da0fcd1c00 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd43fff640 0 client.18276 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 1 --2- 172.21.15.23:0/1457858387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0fcc43f0 0x55da0fcd5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 1 -- 172.21.15.23:0/1457858387 >> 172.21.15.23:0/1457858387 conn(0x55da0fcbeb20 msgr2=0x55da0fcc0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.717+0000 7fcd6dd90740 1 -- 172.21.15.23:0/1457858387 shutdown_connections 2024-03-24T10:49:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.718+0000 7fcd6dd90740 1 -- 172.21.15.23:0/1457858387 wait complete. 2024-03-24T10:49:56.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.718+0000 7fcd6dd90740 20 client.18276 trim_cache size 0 max 16384 2024-03-24T10:49:56.729 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:49:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.718+0000 7fcd6b0d4640 1 -- 172.21.15.23:0/1849473367 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcd500a2230 msgr2=0x7fcd500a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.718+0000 7fcd6b0d4640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcd500a2230 0x7fcd500a46f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcd5409acb0 tx=0x7fcd5406dd90 comp rx=0 tx=0).stop 2024-03-24T10:49:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.718+0000 7fcd6b0d4640 1 -- 172.21.15.23:0/1849473367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd641076b0 msgr2=0x7fcd64197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.718+0000 7fcd6b0d4640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd641076b0 0x7fcd64197b80 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fcd5400e970 tx=0x7fcd5400ed70 comp rx=0 tx=0).stop 2024-03-24T10:49:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.718+0000 7fcd6b0d4640 1 -- 172.21.15.23:0/1849473367 shutdown_connections 2024-03-24T10:49:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.718+0000 7fcd6b0d4640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcd500a2230 0x7fcd500a46f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.718+0000 7fcd6b0d4640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd641076b0 0x7fcd64197b80 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.718+0000 7fcd6b0d4640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd64102060 0x7fcd6419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.718+0000 7fcd6b0d4640 1 --2- 172.21.15.23:0/1849473367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd641016a0 0x7fcd6419da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.718+0000 7fcd6b0d4640 1 -- 172.21.15.23:0/1849473367 >> 172.21.15.23:0/1849473367 conn(0x7fcd64078b10 msgr2=0x7fcd64104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.718+0000 7fcd6b0d4640 1 -- 172.21.15.23:0/1849473367 shutdown_connections 2024-03-24T10:49:56.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:56.718+0000 7fcd6b0d4640 1 -- 172.21.15.23:0/1849473367 wait complete. 2024-03-24T10:49:56.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:56 smithi116 ceph-mon[36248]: pgmap v306: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:56.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:56 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4037188297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:49:57.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:56 smithi117 ceph-mon[36368]: pgmap v306: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:49:57.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:56 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4037188297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:49:57.732 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T10:49:57.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:49:57.733 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T10:49:57.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.852+0000 7f44dc75e640 1 -- 172.21.15.23:0/3577483880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44d4106660 msgr2=0x7f44d4106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:57.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.852+0000 7f44dc75e640 1 --2- 172.21.15.23:0/3577483880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44d4106660 0x7f44d4106a40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009890 tx=0x7f44d002bb70 comp rx=0 tx=0).stop 2024-03-24T10:49:57.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.852+0000 7f44dc75e640 1 -- 172.21.15.23:0/3577483880 shutdown_connections 2024-03-24T10:49:57.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.852+0000 7f44dc75e640 1 --2- 172.21.15.23:0/3577483880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44d4100f60 0x7f44d410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.852+0000 7f44dc75e640 1 --2- 172.21.15.23:0/3577483880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44d41005a0 0x7f44d4100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.852+0000 7f44dc75e640 1 --2- 172.21.15.23:0/3577483880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44d4106660 0x7f44d4106a40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.852+0000 7f44dc75e640 1 -- 172.21.15.23:0/3577483880 >> 172.21.15.23:0/3577483880 conn(0x7f44d4077710 msgr2=0x7f44d4077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:57.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.852+0000 7f44dc75e640 1 -- 172.21.15.23:0/3577483880 shutdown_connections 2024-03-24T10:49:57.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.852+0000 7f44dc75e640 1 -- 172.21.15.23:0/3577483880 wait complete. 2024-03-24T10:49:57.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44dc75e640 1 Processor -- start 2024-03-24T10:49:57.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44dc75e640 1 -- start start 2024-03-24T10:49:57.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44dc75e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44d41005a0 0x7f44d419da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44dc75e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44d4100f60 0x7f44d419dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44dc75e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44d4106660 0x7f44d4197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44dc75e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f44d41982c0 con 0x7f44d4100f60 2024-03-24T10:49:57.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44dc75e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f44d4198430 con 0x7f44d41005a0 2024-03-24T10:49:57.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44dc75e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f44d41985f0 con 0x7f44d4106660 2024-03-24T10:49:57.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44d9cf9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44d4100f60 0x7f44d419dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44da4fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44d41005a0 0x7f44d419da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44da4fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44d41005a0 0x7f44d419da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:50930/0 (socket says 172.21.15.23:50930) 2024-03-24T10:49:57.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44da4fa640 1 -- 172.21.15.23:0/1658966118 learned_addr learned my addr 172.21.15.23:0/1658966118 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:57.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44dacfb640 1 --2- 172.21.15.23:0/1658966118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44d4106660 0x7f44d4197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44d9cf9640 1 -- 172.21.15.23:0/1658966118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44d4106660 msgr2=0x7f44d4197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:57.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/1658966118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44d4106660 0x7f44d4197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44d9cf9640 1 -- 172.21.15.23:0/1658966118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44d41005a0 msgr2=0x7f44d419da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:57.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/1658966118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44d41005a0 0x7f44d419da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.853+0000 7f44d9cf9640 1 -- 172.21.15.23:0/1658966118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d0009490 con 0x7f44d4100f60 2024-03-24T10:49:57.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.854+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/1658966118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44d4100f60 0x7f44d419dfb0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f44c800e9f0 tx=0x7f44c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:57.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.854+0000 7f44c77fe640 1 -- 172.21.15.23:0/1658966118 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c800ccc0 con 0x7f44d4100f60 2024-03-24T10:49:57.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.854+0000 7f44c77fe640 1 -- 172.21.15.23:0/1658966118 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44c800ce20 con 0x7f44d4100f60 2024-03-24T10:49:57.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.854+0000 7f44dc75e640 1 -- 172.21.15.23:0/1658966118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d41988d0 con 0x7f44d4100f60 2024-03-24T10:49:57.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.854+0000 7f44c77fe640 1 -- 172.21.15.23:0/1658966118 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8004a00 con 0x7f44d4100f60 2024-03-24T10:49:57.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.854+0000 7f44dc75e640 1 -- 172.21.15.23:0/1658966118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d41a49f0 con 0x7f44d4100f60 2024-03-24T10:49:57.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.856+0000 7f44c77fe640 1 -- 172.21.15.23:0/1658966118 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f44c80050b0 con 0x7f44d4100f60 2024-03-24T10:49:57.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.856+0000 7f44df41c740 1 Processor -- start 2024-03-24T10:49:57.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44df41c740 1 -- start start 2024-03-24T10:49:57.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44df41c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904a760 0x56195904ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44df41c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904b080 0x561959053720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44df41c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x561959056050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44df41c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x561958fb8530 con 0x56195904a760 2024-03-24T10:49:57.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44df41c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561958fb5a60 con 0x561959053c60 2024-03-24T10:49:57.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44c77fe640 1 --2- 172.21.15.23:0/1658966118 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f44c00a2230 0x7f44c00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44df41c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561958fb1240 con 0x56195904b080 2024-03-24T10:49:57.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44dacfb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904a760 0x56195904ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44dacfb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904a760 0x56195904ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40184/0 (socket says 172.21.15.23:40184) 2024-03-24T10:49:57.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44dacfb640 1 -- 172.21.15.23:0/2498887253 learned_addr learned my addr 172.21.15.23:0/2498887253 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:57.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44da4fa640 1 --2- 172.21.15.23:0/2498887253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904b080 0x561959053720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/2498887253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x561959056050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44dacfb640 1 --2- 172.21.15.23:0/1658966118 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f44c00a2230 0x7f44c00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44c77fe640 1 -- 172.21.15.23:0/1658966118 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f44c8014070 con 0x7f44d4100f60 2024-03-24T10:49:57.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.857+0000 7f44dacfb640 1 -- 172.21.15.23:0/2498887253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904b080 msgr2=0x561959053720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:57.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.858+0000 7f44dacfb640 1 --2- 172.21.15.23:0/2498887253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904b080 0x561959053720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.858+0000 7f44dacfb640 1 -- 172.21.15.23:0/2498887253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 msgr2=0x561959056050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:57.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.858+0000 7f44dacfb640 1 --2- 172.21.15.23:0/2498887253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x561959056050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.858+0000 7f44dacfb640 1 -- 172.21.15.23:0/2498887253 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561958faf8c0 con 0x56195904a760 2024-03-24T10:49:57.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.858+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/2498887253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x561959056050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:57.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.858+0000 7f44dacfb640 1 --2- 172.21.15.23:0/2498887253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904a760 0x56195904ab40 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00c420 tx=0x7f44cc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a762f6a806f8ddb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:57.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.858+0000 7f44dacfb640 1 --2- 172.21.15.23:0/1658966118 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f44c00a2230 0x7f44c00a46f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00cde0 tx=0x7f44cc011000 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:57.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.858+0000 7f44c4ff9640 1 -- 172.21.15.23:0/2498887253 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc013070 con 0x56195904a760 2024-03-24T10:49:57.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.858+0000 7f44c4ff9640 1 -- 172.21.15.23:0/2498887253 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44cc014040 con 0x56195904a760 2024-03-24T10:49:57.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.858+0000 7f44c4ff9640 1 -- 172.21.15.23:0/2498887253 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc004290 con 0x56195904a760 2024-03-24T10:49:57.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.859+0000 7f44df41c740 1 -- 172.21.15.23:0/2498887253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904a760 msgr2=0x56195904ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:57.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.859+0000 7f44df41c740 1 --2- 172.21.15.23:0/2498887253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904a760 0x56195904ab40 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00c420 tx=0x7f44cc00c8e0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.859+0000 7f44df41c740 1 -- 172.21.15.23:0/2498887253 shutdown_connections 2024-03-24T10:49:57.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.859+0000 7f44df41c740 1 --2- 172.21.15.23:0/2498887253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x561959056050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.859+0000 7f44df41c740 1 --2- 172.21.15.23:0/2498887253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904b080 0x561959053720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.859+0000 7f44df41c740 1 --2- 172.21.15.23:0/2498887253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904a760 0x56195904ab40 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.859+0000 7f44df41c740 1 -- 172.21.15.23:0/2498887253 >> 172.21.15.23:0/2498887253 conn(0x561959044e90 msgr2=0x561959045270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:57.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.859+0000 7f44df41c740 1 -- 172.21.15.23:0/2498887253 shutdown_connections 2024-03-24T10:49:57.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.859+0000 7f44df41c740 1 -- 172.21.15.23:0/2498887253 wait complete. 2024-03-24T10:49:57.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.859+0000 7f44df41c740 1 -- start start 2024-03-24T10:49:57.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.859+0000 7f44df41c740 1 Processor -- start 2024-03-24T10:49:57.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44df41c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904a760 0x561959050c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44df41c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904b080 0x561959051170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44df41c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x5619590516b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44df41c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x561958f55f80 con 0x56195904a760 2024-03-24T10:49:57.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44df41c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561958c844b0 con 0x561959053c60 2024-03-24T10:49:57.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44df41c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561958f36ad0 con 0x56195904b080 2024-03-24T10:49:57.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44df41c740 10 client.0 _ll_register_callbacks cb 0x561958f69b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:57.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44da4fa640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904a760 0x561959050c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44da4fa640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904a760 0x561959050c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40192/0 (socket says 172.21.15.23:40192) 2024-03-24T10:49:57.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44da4fa640 1 -- 172.21.15.23:0/1127117030 learned_addr learned my addr 172.21.15.23:0/1127117030 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:57.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44dacfb640 1 --2- 172.21.15.23:0/1127117030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x5619590516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/1127117030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904b080 0x561959051170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44da4fa640 1 -- 172.21.15.23:0/1127117030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904b080 msgr2=0x561959051170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:57.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44da4fa640 1 --2- 172.21.15.23:0/1127117030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904b080 0x561959051170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44da4fa640 1 -- 172.21.15.23:0/1127117030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 msgr2=0x5619590516b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:57.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44da4fa640 1 --2- 172.21.15.23:0/1127117030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x5619590516b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44da4fa640 1 -- 172.21.15.23:0/1127117030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44cc00ec00 con 0x56195904a760 2024-03-24T10:49:57.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/1127117030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904b080 0x561959051170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:49:57.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44da4fa640 1 --2- 172.21.15.23:0/1127117030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904a760 0x561959050c30 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f44d0004270 tx=0x7f44d00042a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3852707ff766c212 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:57.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:57.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44bdffb640 1 -- 172.21.15.23:0/1127117030 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d0034850 con 0x56195904a760 2024-03-24T10:49:57.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44bdffb640 1 -- 172.21.15.23:0/1127117030 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f44d00044c0 con 0x56195904a760 2024-03-24T10:49:57.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44df41c740 1 -- 172.21.15.23:0/1127117030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561958fd57a0 con 0x56195904a760 2024-03-24T10:49:57.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44bdffb640 1 -- 172.21.15.23:0/1127117030 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f44d00349b0 con 0x56195904a760 2024-03-24T10:49:57.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.860+0000 7f44df41c740 10 client.18288 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:57.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.861+0000 7f44df41c740 1 -- 172.21.15.23:0/1127117030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561958f676d0 con 0x56195904a760 2024-03-24T10:49:57.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.861+0000 7f44bdffb640 1 -- 172.21.15.23:0/1127117030 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561958fd57a0 con 0x56195904a760 2024-03-24T10:49:57.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.861+0000 7f44bdffb640 1 -- 172.21.15.23:0/1127117030 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f44d0045020 con 0x56195904a760 2024-03-24T10:49:57.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.861+0000 7f44df41c740 10 client.18288 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:57.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.861+0000 7f44df41c740 10 client.18288 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:57.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.861+0000 7f44df41c740 20 client.18288 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:57.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.861+0000 7f44df41c740 1 --2- 172.21.15.23:0/1127117030 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561959063b40 0x561959065f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.861+0000 7f44df41c740 4 client.18288 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:49:57.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.861+0000 7f44df41c740 1 -- 172.21.15.23:0/1127117030 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561958fa71f0 con 0x561959063b40 2024-03-24T10:49:57.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.861+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/1127117030 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561959063b40 0x561959065f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.861+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/1127117030 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561959063b40 0x561959065f50 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d53588448637dc7d server_cookie=70b005282586b165 in_seq=0 out_seq=0 2024-03-24T10:49:57.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.861+0000 7f44bdffb640 10 client.18288 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:57.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44bdffb640 1 -- 172.21.15.23:0/1127117030 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561958fa71f0 con 0x561959063b40 2024-03-24T10:49:57.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44bdffb640 10 client.18288 handle_command_reply: tid=0 2024-03-24T10:49:57.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44df41c740 1 client.18288 shutdown 2024-03-24T10:49:57.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44df41c740 1 -- 172.21.15.23:0/1127117030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904a760 msgr2=0x561959050c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:57.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44df41c740 1 --2- 172.21.15.23:0/1127117030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904a760 0x561959050c30 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f44d0004270 tx=0x7f44d00042a0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44df41c740 1 -- 172.21.15.23:0/1127117030 shutdown_connections 2024-03-24T10:49:57.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44df41c740 1 --2- 172.21.15.23:0/1127117030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x5619590516b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44df41c740 1 --2- 172.21.15.23:0/1127117030 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x561959063b40 0x561959065f50 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44df41c740 1 --2- 172.21.15.23:0/1127117030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904b080 0x561959051170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44df41c740 1 --2- 172.21.15.23:0/1127117030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904a760 0x561959050c30 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44bdffb640 0 client.18288 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:57.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44df41c740 1 -- 172.21.15.23:0/1127117030 >> 172.21.15.23:0/1127117030 conn(0x561959044e90 msgr2=0x561959046be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:57.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44df41c740 1 -- 172.21.15.23:0/1127117030 shutdown_connections 2024-03-24T10:49:57.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44df41c740 1 -- 172.21.15.23:0/1127117030 wait complete. 2024-03-24T10:49:57.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.862+0000 7f44df41c740 20 client.18288 trim_cache size 0 max 16384 2024-03-24T10:49:57.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.880+0000 7f44df41c740 1 Processor -- start 2024-03-24T10:49:57.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.880+0000 7f44df41c740 1 -- start start 2024-03-24T10:49:57.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.880+0000 7f44df41c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904a760 0x56195905f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.880+0000 7f44df41c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904b080 0x5619590514d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.880+0000 7f44df41c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x56195906b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.880+0000 7f44df41c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x561959040620 con 0x56195904b080 2024-03-24T10:49:57.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.880+0000 7f44df41c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561959045e90 con 0x561959053c60 2024-03-24T10:49:57.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.880+0000 7f44df41c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5619590471a0 con 0x56195904a760 2024-03-24T10:49:57.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.880+0000 7f44d9cf9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904b080 0x5619590514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.880+0000 7f44d9cf9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904b080 0x5619590514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40208/0 (socket says 172.21.15.23:40208) 2024-03-24T10:49:57.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.880+0000 7f44d9cf9640 1 -- 172.21.15.23:0/3982653893 learned_addr learned my addr 172.21.15.23:0/3982653893 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:57.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.880+0000 7f44da4fa640 1 --2- 172.21.15.23:0/3982653893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904a760 0x56195905f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.880+0000 7f44dacfb640 1 --2- 172.21.15.23:0/3982653893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x56195906b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44d9cf9640 1 -- 172.21.15.23:0/3982653893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904a760 msgr2=0x56195905f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/3982653893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904a760 0x56195905f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44d9cf9640 1 -- 172.21.15.23:0/3982653893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 msgr2=0x56195906b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/3982653893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x56195906b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44d9cf9640 1 -- 172.21.15.23:0/3982653893 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561958f4bc60 con 0x56195904b080 2024-03-24T10:49:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/3982653893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904b080 0x5619590514d0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f44c8096e60 tx=0x7f44c800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1864c8691fb045ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:57.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44bffff640 1 -- 172.21.15.23:0/3982653893 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8085040 con 0x56195904b080 2024-03-24T10:49:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44bffff640 1 -- 172.21.15.23:0/3982653893 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44c809aa00 con 0x56195904b080 2024-03-24T10:49:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44bffff640 1 -- 172.21.15.23:0/3982653893 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c80cbb70 con 0x56195904b080 2024-03-24T10:49:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44df41c740 1 -- 172.21.15.23:0/3982653893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904b080 msgr2=0x5619590514d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44df41c740 1 --2- 172.21.15.23:0/3982653893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904b080 0x5619590514d0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f44c8096e60 tx=0x7f44c800cf90 comp rx=0 tx=0).stop 2024-03-24T10:49:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44df41c740 1 -- 172.21.15.23:0/3982653893 shutdown_connections 2024-03-24T10:49:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44df41c740 1 --2- 172.21.15.23:0/3982653893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x56195906b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44df41c740 1 --2- 172.21.15.23:0/3982653893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904b080 0x5619590514d0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44df41c740 1 --2- 172.21.15.23:0/3982653893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904a760 0x56195905f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.881+0000 7f44df41c740 1 -- 172.21.15.23:0/3982653893 >> 172.21.15.23:0/3982653893 conn(0x561959044e90 msgr2=0x561959043ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44df41c740 1 -- 172.21.15.23:0/3982653893 shutdown_connections 2024-03-24T10:49:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44df41c740 1 -- 172.21.15.23:0/3982653893 wait complete. 2024-03-24T10:49:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44df41c740 1 -- start start 2024-03-24T10:49:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44df41c740 1 Processor -- start 2024-03-24T10:49:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44df41c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904a760 0x56195904e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44df41c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904b080 0x56195904e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44df41c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x5619590c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44df41c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5619590c93d0 con 0x56195904b080 2024-03-24T10:49:57.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44df41c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5619590c9510 con 0x561959053c60 2024-03-24T10:49:57.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44df41c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5619590c96d0 con 0x56195904a760 2024-03-24T10:49:57.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44df41c740 10 client.0 _ll_register_callbacks cb 0x7f44d0032340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:57.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44d9cf9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904b080 0x56195904e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44d9cf9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904b080 0x56195904e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40212/0 (socket says 172.21.15.23:40212) 2024-03-24T10:49:57.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44d9cf9640 1 -- 172.21.15.23:0/2027878524 learned_addr learned my addr 172.21.15.23:0/2027878524 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:57.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44da4fa640 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904a760 0x56195904e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44dacfb640 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x5619590c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44d9cf9640 1 -- 172.21.15.23:0/2027878524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904a760 msgr2=0x56195904e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:57.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904a760 0x56195904e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44d9cf9640 1 -- 172.21.15.23:0/2027878524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 msgr2=0x5619590c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:57.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x5619590c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44d9cf9640 1 -- 172.21.15.23:0/2027878524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d41988d0 con 0x56195904b080 2024-03-24T10:49:57.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44da4fa640 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904a760 0x56195904e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:57.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.882+0000 7f44dacfb640 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x5619590c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:57.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44d9cf9640 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904b080 0x56195904e570 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f44c80cb810 tx=0x7f44c8005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a26e42615fe1385 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:57.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:57.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44bdffb640 1 -- 172.21.15.23:0/2027878524 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8096ca0 con 0x56195904b080 2024-03-24T10:49:57.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44bdffb640 1 -- 172.21.15.23:0/2027878524 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7f44c80cbb50 con 0x56195904b080 2024-03-24T10:49:57.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44df41c740 1 -- 172.21.15.23:0/2027878524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44cc014040 con 0x56195904b080 2024-03-24T10:49:57.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44bdffb640 1 -- 172.21.15.23:0/2027878524 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f44c8098fc0 con 0x56195904b080 2024-03-24T10:49:57.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44df41c740 10 client.18294 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:57.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44df41c740 1 -- 172.21.15.23:0/2027878524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561958f71970 con 0x56195904b080 2024-03-24T10:49:57.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44bdffb640 1 -- 172.21.15.23:0/2027878524 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44cc014040 con 0x56195904b080 2024-03-24T10:49:57.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44bdffb640 1 -- 172.21.15.23:0/2027878524 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f44c80a2df0 con 0x56195904b080 2024-03-24T10:49:57.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44df41c740 10 client.18294 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:57.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44df41c740 10 client.18294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:57.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44df41c740 20 client.18294 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:57.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44df41c740 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5619590cab70 0x5619590ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:57.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44df41c740 4 client.18294 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T10:49:57.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.883+0000 7f44df41c740 1 -- 172.21.15.23:0/2027878524 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561958cb0860 con 0x5619590cab70 2024-03-24T10:49:57.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.884+0000 7f44da4fa640 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5619590cab70 0x5619590ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:57.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.884+0000 7f44da4fa640 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5619590cab70 0x5619590ccf90 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6f085a055eb3f45 server_cookie=158393bb0ca1bcc6 in_seq=0 out_seq=0 2024-03-24T10:49:57.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.884+0000 7f44bdffb640 10 client.18294 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:57.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44bdffb640 1 -- 172.21.15.23:0/2027878524 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561958cb0860 con 0x5619590cab70 2024-03-24T10:49:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44bdffb640 10 client.18294 handle_command_reply: tid=0 2024-03-24T10:49:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44df41c740 1 client.18294 shutdown 2024-03-24T10:49:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44df41c740 1 -- 172.21.15.23:0/2027878524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904b080 msgr2=0x56195904e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44df41c740 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904b080 0x56195904e570 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f44c80cb810 tx=0x7f44c8005c50 comp rx=0 tx=0).stop 2024-03-24T10:49:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44df41c740 1 -- 172.21.15.23:0/2027878524 shutdown_connections 2024-03-24T10:49:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44df41c740 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561959053c60 0x5619590c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44df41c740 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56195904b080 0x56195904e570 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44df41c740 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5619590cab70 0x5619590ccf90 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44df41c740 1 --2- 172.21.15.23:0/2027878524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56195904a760 0x56195904e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44bdffb640 0 client.18294 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:57.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44df41c740 1 -- 172.21.15.23:0/2027878524 >> 172.21.15.23:0/2027878524 conn(0x561959044e90 msgr2=0x561959043660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:57.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44df41c740 1 -- 172.21.15.23:0/2027878524 shutdown_connections 2024-03-24T10:49:57.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44df41c740 1 -- 172.21.15.23:0/2027878524 wait complete. 2024-03-24T10:49:57.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44df41c740 20 client.18294 trim_cache size 0 max 16384 2024-03-24T10:49:57.895 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:49:57.896 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-24T10:49:57.896 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "f249665f-afe2-49ce-84e7-a897ad3ef2fe", 2024-03-24T10:49:57.896 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-24T10:49:57.896 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:49:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44dc75e640 1 -- 172.21.15.23:0/1658966118 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f44c00a2230 msgr2=0x7f44c00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.885+0000 7f44dc75e640 1 --2- 172.21.15.23:0/1658966118 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f44c00a2230 0x7f44c00a46f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00cde0 tx=0x7f44cc011000 comp rx=0 tx=0).stop 2024-03-24T10:49:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.886+0000 7f44dc75e640 1 -- 172.21.15.23:0/1658966118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44d4100f60 msgr2=0x7f44d419dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:57.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.886+0000 7f44dc75e640 1 --2- 172.21.15.23:0/1658966118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44d4100f60 0x7f44d419dfb0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f44c800e9f0 tx=0x7f44c800eeb0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.886+0000 7f44dc75e640 1 -- 172.21.15.23:0/1658966118 shutdown_connections 2024-03-24T10:49:57.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.886+0000 7f44dc75e640 1 --2- 172.21.15.23:0/1658966118 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f44c00a2230 0x7f44c00a46f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.886+0000 7f44dc75e640 1 --2- 172.21.15.23:0/1658966118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44d4106660 0x7f44d4197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.886+0000 7f44dc75e640 1 --2- 172.21.15.23:0/1658966118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44d4100f60 0x7f44d419dfb0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.886+0000 7f44dc75e640 1 --2- 172.21.15.23:0/1658966118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44d41005a0 0x7f44d419da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:57.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.886+0000 7f44dc75e640 1 -- 172.21.15.23:0/1658966118 >> 172.21.15.23:0/1658966118 conn(0x7f44d4077710 msgr2=0x7f44d40ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:57.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.886+0000 7f44dc75e640 1 -- 172.21.15.23:0/1658966118 shutdown_connections 2024-03-24T10:49:57.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:57.886+0000 7f44dc75e640 1 -- 172.21.15.23:0/1658966118 wait complete. 2024-03-24T10:49:57.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:49:57.899 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:49:58.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.004+0000 7feaecde2640 1 -- 172.21.15.23:0/3392489618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8106660 msgr2=0x7feae8106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.004+0000 7feaecde2640 1 --2- 172.21.15.23:0/3392489618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8106660 0x7feae8106a40 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fead8009900 tx=0x7fead802bc30 comp rx=0 tx=0).stop 2024-03-24T10:49:58.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.004+0000 7feaecde2640 1 -- 172.21.15.23:0/3392489618 shutdown_connections 2024-03-24T10:49:58.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.004+0000 7feaecde2640 1 --2- 172.21.15.23:0/3392489618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae8100f60 0x7feae810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.004+0000 7feaecde2640 1 --2- 172.21.15.23:0/3392489618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feae81005a0 0x7feae8100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.004+0000 7feaecde2640 1 --2- 172.21.15.23:0/3392489618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8106660 0x7feae8106a40 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.004+0000 7feaecde2640 1 -- 172.21.15.23:0/3392489618 >> 172.21.15.23:0/3392489618 conn(0x7feae8077710 msgr2=0x7feae8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.004+0000 7feaecde2640 1 -- 172.21.15.23:0/3392489618 shutdown_connections 2024-03-24T10:49:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.004+0000 7feaecde2640 1 -- 172.21.15.23:0/3392489618 wait complete. 2024-03-24T10:49:58.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.004+0000 7feaecde2640 1 Processor -- start 2024-03-24T10:49:58.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.004+0000 7feaecde2640 1 -- start start 2024-03-24T10:49:58.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feaecde2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae81005a0 0x7feae819d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feaecde2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feae8100f60 0x7feae819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feaecde2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8106660 0x7feae8197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feaecde2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7feae8198120 con 0x7feae8106660 2024-03-24T10:49:58.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feaecde2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feae8198290 con 0x7feae81005a0 2024-03-24T10:49:58.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feaecde2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feae8198450 con 0x7feae8100f60 2024-03-24T10:49:58.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feae6d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8106660 0x7feae8197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feae659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae81005a0 0x7feae819d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feae6d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8106660 0x7feae8197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40232/0 (socket says 172.21.15.23:40232) 2024-03-24T10:49:58.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feae6d9d640 1 -- 172.21.15.23:0/1112644276 learned_addr learned my addr 172.21.15.23:0/1112644276 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:58.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feae5d9b640 1 --2- 172.21.15.23:0/1112644276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feae8100f60 0x7feae819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feae6d9d640 1 -- 172.21.15.23:0/1112644276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feae8100f60 msgr2=0x7feae819de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feae6d9d640 1 --2- 172.21.15.23:0/1112644276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feae8100f60 0x7feae819de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feae6d9d640 1 -- 172.21.15.23:0/1112644276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae81005a0 msgr2=0x7feae819d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feae6d9d640 1 --2- 172.21.15.23:0/1112644276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae81005a0 0x7feae819d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feae6d9d640 1 -- 172.21.15.23:0/1112644276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead0009490 con 0x7feae8106660 2024-03-24T10:49:58.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feae6d9d640 1 --2- 172.21.15.23:0/1112644276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8106660 0x7feae8197ad0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7feae000eb10 tx=0x7feae000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:58.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feadf7fe640 1 -- 172.21.15.23:0/1112644276 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae0004b20 con 0x7feae8106660 2024-03-24T10:49:58.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feaecde2640 1 -- 172.21.15.23:0/1112644276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead8009560 con 0x7feae8106660 2024-03-24T10:49:58.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feadf7fe640 1 -- 172.21.15.23:0/1112644276 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feae0004c80 con 0x7feae8106660 2024-03-24T10:49:58.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.005+0000 7feadf7fe640 1 -- 172.21.15.23:0/1112644276 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae0010650 con 0x7feae8106660 2024-03-24T10:49:58.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.006+0000 7feaecde2640 1 -- 172.21.15.23:0/1112644276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae81a47e0 con 0x7feae8106660 2024-03-24T10:49:58.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.008+0000 7feadf7fe640 1 -- 172.21.15.23:0/1112644276 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7feae0005020 con 0x7feae8106660 2024-03-24T10:49:58.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.008+0000 7feaef9bf740 1 Processor -- start 2024-03-24T10:49:58.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.008+0000 7feaef9bf740 1 -- start start 2024-03-24T10:49:58.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.008+0000 7feaef9bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56374282d1d0 0x56374282d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.008+0000 7feaef9bf740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282daf0 0x563742836190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.008+0000 7feaef9bf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5637428366d0 0x563742838ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.008+0000 7feaef9bf740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563742751f30 con 0x56374282daf0 2024-03-24T10:49:58.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.008+0000 7feaef9bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56374279b810 con 0x56374282d1d0 2024-03-24T10:49:58.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.008+0000 7feaef9bf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563742798d40 con 0x5637428366d0 2024-03-24T10:49:58.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.008+0000 7feadf7fe640 1 --2- 172.21.15.23:0/1112644276 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fead40a2230 0x7fead40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.008+0000 7feae659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282daf0 0x563742836190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.008+0000 7feae5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56374282d1d0 0x56374282d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feae659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282daf0 0x563742836190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40246/0 (socket says 172.21.15.23:40246) 2024-03-24T10:49:58.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feae659c640 1 -- 172.21.15.23:0/3219866198 learned_addr learned my addr 172.21.15.23:0/3219866198 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:58.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feae5d9b640 1 --2- 172.21.15.23:0/3219866198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5637428366d0 0x563742838ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feae6d9d640 1 --2- 172.21.15.23:0/1112644276 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fead40a2230 0x7fead40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feadf7fe640 1 -- 172.21.15.23:0/1112644276 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7feae0014070 con 0x7feae8106660 2024-03-24T10:49:58.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feae659c640 1 -- 172.21.15.23:0/3219866198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5637428366d0 msgr2=0x563742838ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:58.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feae659c640 1 --2- 172.21.15.23:0/3219866198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5637428366d0 0x563742838ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feae659c640 1 -- 172.21.15.23:0/3219866198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56374282d1d0 msgr2=0x56374282d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:58.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feae659c640 1 --2- 172.21.15.23:0/3219866198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56374282d1d0 0x56374282d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feae659c640 1 -- 172.21.15.23:0/3219866198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563742755ea0 con 0x56374282daf0 2024-03-24T10:49:58.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feae5d9b640 1 --2- 172.21.15.23:0/3219866198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56374282d1d0 0x56374282d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:58.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feae659c640 1 --2- 172.21.15.23:0/3219866198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282daf0 0x563742836190 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fead8009900 tx=0x7fead802c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67d5f0d47e3d8b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:58.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feae6d9d640 1 --2- 172.21.15.23:0/1112644276 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fead40a2230 0x7fead40a46f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7feae00cc390 tx=0x7feae000cf70 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:58.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feadcff9640 1 -- 172.21.15.23:0/3219866198 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead802c910 con 0x56374282daf0 2024-03-24T10:49:58.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feadcff9640 1 -- 172.21.15.23:0/3219866198 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fead802ca70 con 0x56374282daf0 2024-03-24T10:49:58.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.009+0000 7feadcff9640 1 -- 172.21.15.23:0/3219866198 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead8005050 con 0x56374282daf0 2024-03-24T10:49:58.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.010+0000 7feaef9bf740 1 -- 172.21.15.23:0/3219866198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282daf0 msgr2=0x563742836190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.010+0000 7feaef9bf740 1 --2- 172.21.15.23:0/3219866198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282daf0 0x563742836190 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fead8009900 tx=0x7fead802c6a0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.010+0000 7feaef9bf740 1 -- 172.21.15.23:0/3219866198 shutdown_connections 2024-03-24T10:49:58.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.010+0000 7feaef9bf740 1 --2- 172.21.15.23:0/3219866198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5637428366d0 0x563742838ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.010+0000 7feaef9bf740 1 --2- 172.21.15.23:0/3219866198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282daf0 0x563742836190 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.010+0000 7feaef9bf740 1 --2- 172.21.15.23:0/3219866198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56374282d1d0 0x56374282d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.010+0000 7feaef9bf740 1 -- 172.21.15.23:0/3219866198 >> 172.21.15.23:0/3219866198 conn(0x563742827900 msgr2=0x563742827ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:58.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.010+0000 7feaef9bf740 1 -- 172.21.15.23:0/3219866198 shutdown_connections 2024-03-24T10:49:58.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.010+0000 7feaef9bf740 1 -- 172.21.15.23:0/3219866198 wait complete. 2024-03-24T10:49:58.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.010+0000 7feaef9bf740 1 -- start start 2024-03-24T10:49:58.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.010+0000 7feaef9bf740 1 Processor -- start 2024-03-24T10:49:58.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feaef9bf740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282d1d0 0x563742833770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feaef9bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56374282daf0 0x563742833cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feaef9bf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5637428366d0 0x5637428341f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feaef9bf740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563742794520 con 0x56374282d1d0 2024-03-24T10:49:58.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feaef9bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563742467430 con 0x56374282daf0 2024-03-24T10:49:58.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feaef9bf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563742423e00 con 0x5637428366d0 2024-03-24T10:49:58.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feaef9bf740 10 client.0 _ll_register_callbacks cb 0x563742732d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:58.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feae5d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282d1d0 0x563742833770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feae5d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282d1d0 0x563742833770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40250/0 (socket says 172.21.15.23:40250) 2024-03-24T10:49:58.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feae5d9b640 1 -- 172.21.15.23:0/2976867919 learned_addr learned my addr 172.21.15.23:0/2976867919 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:58.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feae659c640 1 --2- 172.21.15.23:0/2976867919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56374282daf0 0x563742833cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feae5d9b640 1 --2- 172.21.15.23:0/2976867919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5637428366d0 0x5637428341f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feae5d9b640 1 -- 172.21.15.23:0/2976867919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5637428366d0 msgr2=0x5637428341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:58.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feae5d9b640 1 --2- 172.21.15.23:0/2976867919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5637428366d0 0x5637428341f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feae5d9b640 1 -- 172.21.15.23:0/2976867919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56374282daf0 msgr2=0x563742833cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:58.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feae5d9b640 1 --2- 172.21.15.23:0/2976867919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56374282daf0 0x563742833cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feae5d9b640 1 -- 172.21.15.23:0/2976867919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae8068870 con 0x56374282d1d0 2024-03-24T10:49:58.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feae5d9b640 1 --2- 172.21.15.23:0/2976867919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282d1d0 0x563742833770 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fead0004650 tx=0x7fead0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d792847b4a28ebd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:58.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feacdffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:58.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feacdffb640 1 -- 172.21.15.23:0/2976867919 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0003d00 con 0x56374282d1d0 2024-03-24T10:49:58.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feacdffb640 1 -- 172.21.15.23:0/2976867919 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fead00130d0 con 0x56374282d1d0 2024-03-24T10:49:58.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feaef9bf740 1 -- 172.21.15.23:0/2976867919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5637424937e0 con 0x56374282d1d0 2024-03-24T10:49:58.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feacdffb640 1 -- 172.21.15.23:0/2976867919 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fead0003e60 con 0x56374282d1d0 2024-03-24T10:49:58.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feaef9bf740 10 client.18306 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:58.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.011+0000 7feaef9bf740 1 -- 172.21.15.23:0/2976867919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563742792ba0 con 0x56374282d1d0 2024-03-24T10:49:58.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.012+0000 7feacdffb640 1 -- 172.21.15.23:0/2976867919 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5637424937e0 con 0x56374282d1d0 2024-03-24T10:49:58.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.012+0000 7feacdffb640 1 -- 172.21.15.23:0/2976867919 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fead0020470 con 0x56374282d1d0 2024-03-24T10:49:58.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.012+0000 7feaef9bf740 10 client.18306 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:58.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.012+0000 7feaef9bf740 10 client.18306 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:58.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.012+0000 7feaef9bf740 20 client.18306 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:58.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.012+0000 7feaef9bf740 1 --2- 172.21.15.23:0/2976867919 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563742846c20 0x563742849040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.012+0000 7feaef9bf740 4 client.18306 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:49:58.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.012+0000 7feaef9bf740 1 -- 172.21.15.23:0/2976867919 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637424d0720 con 0x563742846c20 2024-03-24T10:49:58.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.012+0000 7feae659c640 1 --2- 172.21.15.23:0/2976867919 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563742846c20 0x563742849040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.012+0000 7feacdffb640 10 client.18306 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:58.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.012+0000 7feae659c640 1 --2- 172.21.15.23:0/2976867919 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563742846c20 0x563742849040 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49f711074f22d37d server_cookie=1f1441e5d9942004 in_seq=0 out_seq=0 2024-03-24T10:49:58.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feacdffb640 1 -- 172.21.15.23:0/2976867919 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5637424d0720 con 0x563742846c20 2024-03-24T10:49:58.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feacdffb640 10 client.18306 handle_command_reply: tid=0 2024-03-24T10:49:58.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feaef9bf740 1 client.18306 shutdown 2024-03-24T10:49:58.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feaef9bf740 1 -- 172.21.15.23:0/2976867919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282d1d0 msgr2=0x563742833770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feaef9bf740 1 --2- 172.21.15.23:0/2976867919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282d1d0 0x563742833770 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fead0004650 tx=0x7fead0004730 comp rx=0 tx=0).stop 2024-03-24T10:49:58.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feaef9bf740 1 -- 172.21.15.23:0/2976867919 shutdown_connections 2024-03-24T10:49:58.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feaef9bf740 1 --2- 172.21.15.23:0/2976867919 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x563742846c20 0x563742849040 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feaef9bf740 1 --2- 172.21.15.23:0/2976867919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5637428366d0 0x5637428341f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feaef9bf740 1 --2- 172.21.15.23:0/2976867919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56374282daf0 0x563742833cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feaef9bf740 1 --2- 172.21.15.23:0/2976867919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282d1d0 0x563742833770 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feaef9bf740 1 -- 172.21.15.23:0/2976867919 >> 172.21.15.23:0/2976867919 conn(0x563742827900 msgr2=0x563742829650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:58.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feacdffb640 0 client.18306 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:58.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feaef9bf740 1 -- 172.21.15.23:0/2976867919 shutdown_connections 2024-03-24T10:49:58.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feaef9bf740 1 -- 172.21.15.23:0/2976867919 wait complete. 2024-03-24T10:49:58.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.013+0000 7feaef9bf740 20 client.18306 trim_cache size 0 max 16384 2024-03-24T10:49:58.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feaef9bf740 1 Processor -- start 2024-03-24T10:49:58.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feaef9bf740 1 -- start start 2024-03-24T10:49:58.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feaef9bf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56374282d1d0 0x563742835ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feaef9bf740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282daf0 0x563742841f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feaef9bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5637428366d0 0x56374284e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feaef9bf740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563742719430 con 0x56374282daf0 2024-03-24T10:49:58.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feaef9bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56374282c970 con 0x5637428366d0 2024-03-24T10:49:58.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feaef9bf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56374282ce30 con 0x56374282d1d0 2024-03-24T10:49:58.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feae659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282daf0 0x563742841f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feae659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282daf0 0x563742841f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40266/0 (socket says 172.21.15.23:40266) 2024-03-24T10:49:58.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feae659c640 1 -- 172.21.15.23:0/1576195541 learned_addr learned my addr 172.21.15.23:0/1576195541 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:58.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feae5d9b640 1 --2- 172.21.15.23:0/1576195541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5637428366d0 0x56374284e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feae5d9b640 1 --2- 172.21.15.23:0/1576195541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56374282d1d0 0x563742835ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feae659c640 1 -- 172.21.15.23:0/1576195541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56374282d1d0 msgr2=0x563742835ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:58.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feae659c640 1 --2- 172.21.15.23:0/1576195541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56374282d1d0 0x563742835ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feae659c640 1 -- 172.21.15.23:0/1576195541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5637428366d0 msgr2=0x56374284e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:58.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feae659c640 1 --2- 172.21.15.23:0/1576195541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5637428366d0 0x56374284e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feae659c640 1 -- 172.21.15.23:0/1576195541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56374272eb20 con 0x56374282daf0 2024-03-24T10:49:58.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.031+0000 7feae659c640 1 --2- 172.21.15.23:0/1576195541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282daf0 0x563742841f00 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fead802cd00 tx=0x7fead8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff912a9728ffa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:58.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.032+0000 7feacffff640 1 -- 172.21.15.23:0/1576195541 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead80044a0 con 0x56374282daf0 2024-03-24T10:49:58.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.032+0000 7feacffff640 1 -- 172.21.15.23:0/1576195541 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fead80046c0 con 0x56374282daf0 2024-03-24T10:49:58.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.032+0000 7feacffff640 1 -- 172.21.15.23:0/1576195541 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead80050e0 con 0x56374282daf0 2024-03-24T10:49:58.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.032+0000 7feaef9bf740 1 -- 172.21.15.23:0/1576195541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282daf0 msgr2=0x563742841f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.032+0000 7feaef9bf740 1 --2- 172.21.15.23:0/1576195541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282daf0 0x563742841f00 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fead802cd00 tx=0x7fead8005fd0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.032+0000 7feaef9bf740 1 -- 172.21.15.23:0/1576195541 shutdown_connections 2024-03-24T10:49:58.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.032+0000 7feaef9bf740 1 --2- 172.21.15.23:0/1576195541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5637428366d0 0x56374284e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.032+0000 7feaef9bf740 1 --2- 172.21.15.23:0/1576195541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282daf0 0x563742841f00 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.032+0000 7feaef9bf740 1 --2- 172.21.15.23:0/1576195541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56374282d1d0 0x563742835ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.032+0000 7feaef9bf740 1 -- 172.21.15.23:0/1576195541 >> 172.21.15.23:0/1576195541 conn(0x563742827900 msgr2=0x56374282ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:58.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.032+0000 7feaef9bf740 1 -- 172.21.15.23:0/1576195541 shutdown_connections 2024-03-24T10:49:58.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.032+0000 7feaef9bf740 1 -- 172.21.15.23:0/1576195541 wait complete. 2024-03-24T10:49:58.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.032+0000 7feaef9bf740 1 -- start start 2024-03-24T10:49:58.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feaef9bf740 1 Processor -- start 2024-03-24T10:49:58.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feaef9bf740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282d1d0 0x5637428a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feaef9bf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56374282daf0 0x5637428a5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feaef9bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5637428366d0 0x563742839840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feaef9bf740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x563742839e00 con 0x56374282d1d0 2024-03-24T10:49:58.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feaef9bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563742839f70 con 0x5637428366d0 2024-03-24T10:49:58.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feaef9bf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56374283a130 con 0x56374282daf0 2024-03-24T10:49:58.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feaef9bf740 10 client.0 _ll_register_callbacks cb 0x7fead0013740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:49:58.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feae5d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282d1d0 0x5637428a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feae5d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282d1d0 0x5637428a52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40270/0 (socket says 172.21.15.23:40270) 2024-03-24T10:49:58.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feae5d9b640 1 -- 172.21.15.23:0/2289320164 learned_addr learned my addr 172.21.15.23:0/2289320164 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:49:58.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feae659c640 1 --2- 172.21.15.23:0/2289320164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56374282daf0 0x5637428a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feae5d9b640 1 --2- 172.21.15.23:0/2289320164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5637428366d0 0x563742839840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feae5d9b640 1 -- 172.21.15.23:0/2289320164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56374282daf0 msgr2=0x5637428a5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:58.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feae5d9b640 1 --2- 172.21.15.23:0/2289320164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56374282daf0 0x5637428a5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feae5d9b640 1 -- 172.21.15.23:0/2289320164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5637428366d0 msgr2=0x563742839840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:58.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feae5d9b640 1 --2- 172.21.15.23:0/2289320164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5637428366d0 0x563742839840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feae5d9b640 1 -- 172.21.15.23:0/2289320164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563742755ea0 con 0x56374282d1d0 2024-03-24T10:49:58.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feae5d9b640 1 --2- 172.21.15.23:0/2289320164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282d1d0 0x5637428a52c0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fead0020880 tx=0x7fead0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29eae488d0d89740 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:58.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feacdffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:49:58.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feacdffb640 1 -- 172.21.15.23:0/2289320164 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead00130d0 con 0x56374282d1d0 2024-03-24T10:49:58.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feacdffb640 1 -- 172.21.15.23:0/2289320164 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8571+0+0 (secure 0 0 0) 0x7fead0000e40 con 0x56374282d1d0 2024-03-24T10:49:58.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.033+0000 7feaef9bf740 1 -- 172.21.15.23:0/2289320164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fead802ca70 con 0x56374282d1d0 2024-03-24T10:49:58.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.034+0000 7feacdffb640 1 -- 172.21.15.23:0/2289320164 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fead0013230 con 0x56374282d1d0 2024-03-24T10:49:58.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.034+0000 7feaef9bf740 10 client.18312 fetch_fsmap learned FSMap version 31 2024-03-24T10:49:58.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.034+0000 7feaef9bf740 1 -- 172.21.15.23:0/2289320164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563742754700 con 0x56374282d1d0 2024-03-24T10:49:58.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.034+0000 7feacdffb640 1 -- 172.21.15.23:0/2289320164 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fead802ca70 con 0x56374282d1d0 2024-03-24T10:49:58.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.034+0000 7feacdffb640 1 -- 172.21.15.23:0/2289320164 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fead0029460 con 0x56374282d1d0 2024-03-24T10:49:58.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.034+0000 7feaef9bf740 10 client.18312 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:49:58.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.034+0000 7feaef9bf740 10 client.18312 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:49:58.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.034+0000 7feaef9bf740 20 client.18312 populate_metadata read hostname 'smithi023' 2024-03-24T10:49:58.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.034+0000 7feaef9bf740 1 --2- 172.21.15.23:0/2289320164 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5637428ad110 0x5637428af500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.034+0000 7feaef9bf740 4 client.18312 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:49:58.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.034+0000 7feaef9bf740 1 -- 172.21.15.23:0/2289320164 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56374251f890 con 0x5637428ad110 2024-03-24T10:49:58.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.034+0000 7feae659c640 1 --2- 172.21.15.23:0/2289320164 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5637428ad110 0x5637428af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.034+0000 7feae659c640 1 --2- 172.21.15.23:0/2289320164 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5637428ad110 0x5637428af500 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be919106934d51b1 server_cookie=97b404b8f8766fa1 in_seq=0 out_seq=0 2024-03-24T10:49:58.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.034+0000 7feacdffb640 10 client.18312 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:58.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feacdffb640 1 -- 172.21.15.23:0/2289320164 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56374251f890 con 0x5637428ad110 2024-03-24T10:49:58.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feacdffb640 10 client.18312 handle_command_reply: tid=0 2024-03-24T10:49:58.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaef9bf740 1 client.18312 shutdown 2024-03-24T10:49:58.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaef9bf740 1 -- 172.21.15.23:0/2289320164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282d1d0 msgr2=0x5637428a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaef9bf740 1 --2- 172.21.15.23:0/2289320164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282d1d0 0x5637428a52c0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fead0020880 tx=0x7fead0004730 comp rx=0 tx=0).stop 2024-03-24T10:49:58.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaef9bf740 1 -- 172.21.15.23:0/2289320164 shutdown_connections 2024-03-24T10:49:58.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaef9bf740 1 --2- 172.21.15.23:0/2289320164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5637428366d0 0x563742839840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaef9bf740 1 --2- 172.21.15.23:0/2289320164 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5637428ad110 0x5637428af500 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaef9bf740 1 --2- 172.21.15.23:0/2289320164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56374282daf0 0x5637428a5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaef9bf740 1 --2- 172.21.15.23:0/2289320164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374282d1d0 0x5637428a52c0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feacdffb640 0 client.18312 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:49:58.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaef9bf740 1 -- 172.21.15.23:0/2289320164 >> 172.21.15.23:0/2289320164 conn(0x563742827900 msgr2=0x5637428262d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:58.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaef9bf740 1 -- 172.21.15.23:0/2289320164 shutdown_connections 2024-03-24T10:49:58.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaef9bf740 1 -- 172.21.15.23:0/2289320164 wait complete. 2024-03-24T10:49:58.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaef9bf740 20 client.18312 trim_cache size 0 max 16384 2024-03-24T10:49:58.046 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:49:58.046 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T10:49:58.046 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-24T10:49:58.046 INFO:teuthology.orchestra.run.smithi023.stdout: "f249665f-afe2-49ce-84e7-a897ad3ef2fe": { 2024-03-24T10:49:58.046 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-24T10:49:58.046 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "f249665f-afe2-49ce-84e7-a897ad3ef2fe", 2024-03-24T10:49:58.046 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-24T10:49:58.047 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:49:58.047 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:49:58.047 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:49:58.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaecde2640 1 -- 172.21.15.23:0/1112644276 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fead40a2230 msgr2=0x7fead40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaecde2640 1 --2- 172.21.15.23:0/1112644276 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fead40a2230 0x7fead40a46f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7feae00cc390 tx=0x7feae000cf70 comp rx=0 tx=0).stop 2024-03-24T10:49:58.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaecde2640 1 -- 172.21.15.23:0/1112644276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8106660 msgr2=0x7feae8197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaecde2640 1 --2- 172.21.15.23:0/1112644276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8106660 0x7feae8197ad0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7feae000eb10 tx=0x7feae000c2d0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaecde2640 1 -- 172.21.15.23:0/1112644276 shutdown_connections 2024-03-24T10:49:58.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaecde2640 1 --2- 172.21.15.23:0/1112644276 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fead40a2230 0x7fead40a46f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaecde2640 1 --2- 172.21.15.23:0/1112644276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8106660 0x7feae8197ad0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaecde2640 1 --2- 172.21.15.23:0/1112644276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feae8100f60 0x7feae819de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaecde2640 1 --2- 172.21.15.23:0/1112644276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feae81005a0 0x7feae819d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaecde2640 1 -- 172.21.15.23:0/1112644276 >> 172.21.15.23:0/1112644276 conn(0x7feae8077710 msgr2=0x7feae80ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:58.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaecde2640 1 -- 172.21.15.23:0/1112644276 shutdown_connections 2024-03-24T10:49:58.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:49:58.035+0000 7feaecde2640 1 -- 172.21.15.23:0/1112644276 wait complete. 2024-03-24T10:49:58.052 INFO:tasks.cephfs.filesystem:scrub status for tag:f249665f-afe2-49ce-84e7-a897ad3ef2fe - {'path': '/', 'tag': 'f249665f-afe2-49ce-84e7-a897ad3ef2fe', 'options': 'recursive,force'} 2024-03-24T10:49:58.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:58 smithi116 ceph-mon[36248]: pgmap v307: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:49:58.767 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:58 smithi116 ceph-mon[36248]: scrub queued for path: / 2024-03-24T10:49:58.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:58 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:58.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:58 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:49:58.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:58 smithi116 ceph-mon[36248]: scrub summary: active paths [/] 2024-03-24T10:49:58.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:58 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:58.768 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:49:58 smithi116 ceph-mon[36248]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:58 smithi023 ceph-mon[32466]: pgmap v307: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:49:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:58 smithi023 ceph-mon[32466]: scrub queued for path: / 2024-03-24T10:49:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:58 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:58.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:58 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:49:58.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:58 smithi023 ceph-mon[32466]: scrub summary: active paths [/] 2024-03-24T10:49:58.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:58 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:58.919 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:49:58 smithi023 ceph-mon[32466]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:59.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:58 smithi117 ceph-mon[36368]: pgmap v307: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:49:59.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:58 smithi117 ceph-mon[36368]: scrub queued for path: / 2024-03-24T10:49:59.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:58 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:59.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:58 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:49:59.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:58 smithi117 ceph-mon[36368]: scrub summary: active paths [/] 2024-03-24T10:49:59.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:58 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:49:59.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:49:58 smithi117 ceph-mon[36368]: scrub summary: idle+waiting paths [/] 2024-03-24T10:50:00.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:00 smithi023 ceph-mon[32466]: pgmap v308: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:00.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:00 smithi023 ceph-mon[32466]: overall HEALTH_OK 2024-03-24T10:50:01.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:00 smithi117 ceph-mon[36368]: pgmap v308: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:01.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:00 smithi117 ceph-mon[36368]: overall HEALTH_OK 2024-03-24T10:50:01.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:00 smithi116 ceph-mon[36248]: pgmap v308: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:01.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:00 smithi116 ceph-mon[36248]: overall HEALTH_OK 2024-03-24T10:50:02.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:02 smithi023 ceph-mon[32466]: pgmap v309: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T10:50:03.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:02 smithi117 ceph-mon[36368]: pgmap v309: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T10:50:03.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:02 smithi116 ceph-mon[36248]: pgmap v309: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T10:50:04.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:04 smithi023 ceph-mon[32466]: pgmap v310: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:50:05.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:04 smithi117 ceph-mon[36368]: pgmap v310: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:50:05.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:04 smithi116 ceph-mon[36248]: pgmap v310: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:50:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:05 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:50:05.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:05 smithi023 ceph-mon[32466]: scrub summary: idle 2024-03-24T10:50:06.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:05 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:50:06.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:05 smithi117 ceph-mon[36368]: scrub summary: idle 2024-03-24T10:50:06.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:05 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:50:06.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:05 smithi116 ceph-mon[36248]: scrub summary: idle 2024-03-24T10:50:06.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:06 smithi023 ceph-mon[32466]: pgmap v311: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:50:07.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:06 smithi117 ceph-mon[36368]: pgmap v311: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:50:07.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:06 smithi116 ceph-mon[36248]: pgmap v311: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:50:08.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:08 smithi023 ceph-mon[32466]: pgmap v312: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T10:50:09.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:08 smithi117 ceph-mon[36368]: pgmap v312: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T10:50:09.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:08 smithi116 ceph-mon[36248]: pgmap v312: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T10:50:10.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:10 smithi023 ceph-mon[32466]: pgmap v313: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:50:11.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:10 smithi117 ceph-mon[36368]: pgmap v313: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:50:11.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:10 smithi116 ceph-mon[36248]: pgmap v313: 97 pgs: 97 active+clean; 417 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:50:11.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:11 smithi023 ceph-mon[32466]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T10:50:12.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:11 smithi117 ceph-mon[36368]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T10:50:12.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:11 smithi116 ceph-mon[36248]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T10:50:12.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:12 smithi023 ceph-mon[32466]: pgmap v315: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T10:50:13.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:12 smithi117 ceph-mon[36368]: pgmap v315: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T10:50:13.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:12 smithi116 ceph-mon[36248]: pgmap v315: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T10:50:13.514 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-24T10:50:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.630+0000 7fcfdeeda640 1 -- 172.21.15.23:0/3249266743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfd81076f0 msgr2=0x7fcfd8107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.630+0000 7fcfdeeda640 1 --2- 172.21.15.23:0/3249266743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfd81076f0 0x7fcfd8107ad0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009930 tx=0x7fcfc802bc30 comp rx=0 tx=0).stop 2024-03-24T10:50:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.630+0000 7fcfdeeda640 1 -- 172.21.15.23:0/3249266743 shutdown_connections 2024-03-24T10:50:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.630+0000 7fcfdeeda640 1 --2- 172.21.15.23:0/3249266743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcfd81005d0 0x7fcfd810e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.630+0000 7fcfdeeda640 1 --2- 172.21.15.23:0/3249266743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd80ffc10 0x7fcfd8100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.630+0000 7fcfdeeda640 1 --2- 172.21.15.23:0/3249266743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfd81076f0 0x7fcfd8107ad0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.630+0000 7fcfdeeda640 1 -- 172.21.15.23:0/3249266743 >> 172.21.15.23:0/3249266743 conn(0x7fcfd8076bd0 msgr2=0x7fcfd80fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.630+0000 7fcfdeeda640 1 -- 172.21.15.23:0/3249266743 shutdown_connections 2024-03-24T10:50:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.630+0000 7fcfdeeda640 1 -- 172.21.15.23:0/3249266743 wait complete. 2024-03-24T10:50:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.631+0000 7fcfdeeda640 1 Processor -- start 2024-03-24T10:50:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.631+0000 7fcfdeeda640 1 -- start start 2024-03-24T10:50:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.631+0000 7fcfdeeda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd80ffc10 0x7fcfd819d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.631+0000 7fcfdeeda640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcfd81005d0 0x7fcfd819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.631+0000 7fcfdeeda640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfd81076f0 0x7fcfd8197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.631+0000 7fcfdeeda640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8198130 con 0x7fcfd81076f0 2024-03-24T10:50:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.631+0000 7fcfdeeda640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81982a0 con 0x7fcfd80ffc10 2024-03-24T10:50:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.631+0000 7fcfdeeda640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8198460 con 0x7fcfd81005d0 2024-03-24T10:50:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.631+0000 7fcfdd477640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfd81076f0 0x7fcfd8197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.631+0000 7fcfdcc76640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd80ffc10 0x7fcfd819d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.631+0000 7fcfdd477640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfd81076f0 0x7fcfd8197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41548/0 (socket says 172.21.15.23:41548) 2024-03-24T10:50:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.631+0000 7fcfdd477640 1 -- 172.21.15.23:0/4241935755 learned_addr learned my addr 172.21.15.23:0/4241935755 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.631+0000 7fcfd7fff640 1 --2- 172.21.15.23:0/4241935755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcfd81005d0 0x7fcfd819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.632+0000 7fcfdd477640 1 -- 172.21.15.23:0/4241935755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcfd81005d0 msgr2=0x7fcfd819de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.632+0000 7fcfdd477640 1 --2- 172.21.15.23:0/4241935755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcfd81005d0 0x7fcfd819de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.632+0000 7fcfdd477640 1 -- 172.21.15.23:0/4241935755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd80ffc10 msgr2=0x7fcfd819d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.632+0000 7fcfdd477640 1 --2- 172.21.15.23:0/4241935755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd80ffc10 0x7fcfd819d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.632+0000 7fcfdd477640 1 -- 172.21.15.23:0/4241935755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfcc009490 con 0x7fcfd81076f0 2024-03-24T10:50:13.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.632+0000 7fcfdcc76640 1 --2- 172.21.15.23:0/4241935755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd80ffc10 0x7fcfd819d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:50:13.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.632+0000 7fcfdd477640 1 --2- 172.21.15.23:0/4241935755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfd81076f0 0x7fcfd8197ae0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000eb10 tx=0x7fcfd000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:13.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.632+0000 7fcfd5ffb640 1 -- 172.21.15.23:0/4241935755 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0004b20 con 0x7fcfd81076f0 2024-03-24T10:50:13.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.632+0000 7fcfdeeda640 1 -- 172.21.15.23:0/4241935755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfc8009560 con 0x7fcfd81076f0 2024-03-24T10:50:13.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.632+0000 7fcfd5ffb640 1 -- 172.21.15.23:0/4241935755 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfd0004c80 con 0x7fcfd81076f0 2024-03-24T10:50:13.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.632+0000 7fcfd5ffb640 1 -- 172.21.15.23:0/4241935755 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0010670 con 0x7fcfd81076f0 2024-03-24T10:50:13.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.632+0000 7fcfdeeda640 1 -- 172.21.15.23:0/4241935755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd8198930 con 0x7fcfd81076f0 2024-03-24T10:50:13.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.633+0000 7fcfdeeda640 1 -- 172.21.15.23:0/4241935755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd804f470 con 0x7fcfd81076f0 2024-03-24T10:50:13.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.634+0000 7fcfd5ffb640 1 -- 172.21.15.23:0/4241935755 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fcfd0005020 con 0x7fcfd81076f0 2024-03-24T10:50:13.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.635+0000 7fcfd5ffb640 1 --2- 172.21.15.23:0/4241935755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcfc00a22d0 0x7fcfc00a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:13.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.635+0000 7fcfd5ffb640 1 -- 172.21.15.23:0/4241935755 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8615+0+0 (secure 0 0 0) 0x7fcfd0014070 con 0x7fcfd81076f0 2024-03-24T10:50:13.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.635+0000 7fcfdcc76640 1 --2- 172.21.15.23:0/4241935755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcfc00a22d0 0x7fcfc00a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:13.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.635+0000 7fcfdcc76640 1 --2- 172.21.15.23:0/4241935755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcfc00a22d0 0x7fcfc00a4790 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8005ab0 tx=0x7fcfc8005fd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:13.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.639+0000 7fcfd5ffb640 1 -- 172.21.15.23:0/4241935755 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcfd0097210 con 0x7fcfd81076f0 2024-03-24T10:50:13.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.844+0000 7fcfdeeda640 1 -- 172.21.15.23:0/4241935755 --> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fcfd8111840 con 0x7fcfc00a22d0 2024-03-24T10:50:13.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.852+0000 7fcfd5ffb640 1 -- 172.21.15.23:0/4241935755 <== mgr.15288 v2:172.21.15.23:6800/581236369 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fcfd8111840 con 0x7fcfc00a22d0 2024-03-24T10:50:13.852 INFO:teuthology.orchestra.run.smithi023.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-24T00:00:00", "created": "2024-03-24T10:40:12", "first": "2024-03-24T10:41:00", "last": "2024-03-24T10:50:00", "last_pruned": "2024-03-24T10:50:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-24T10:50:13.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.854+0000 7fcfdeeda640 1 -- 172.21.15.23:0/4241935755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcfc00a22d0 msgr2=0x7fcfc00a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:13.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.854+0000 7fcfdeeda640 1 --2- 172.21.15.23:0/4241935755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcfc00a22d0 0x7fcfc00a4790 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8005ab0 tx=0x7fcfc8005fd0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.854+0000 7fcfdeeda640 1 -- 172.21.15.23:0/4241935755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfd81076f0 msgr2=0x7fcfd8197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:13.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.854+0000 7fcfdeeda640 1 --2- 172.21.15.23:0/4241935755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfd81076f0 0x7fcfd8197ae0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000eb10 tx=0x7fcfd000c2d0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.854+0000 7fcfdeeda640 1 -- 172.21.15.23:0/4241935755 shutdown_connections 2024-03-24T10:50:13.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.854+0000 7fcfdeeda640 1 --2- 172.21.15.23:0/4241935755 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcfc00a22d0 0x7fcfc00a4790 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.854+0000 7fcfdeeda640 1 --2- 172.21.15.23:0/4241935755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfd81076f0 0x7fcfd8197ae0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.854+0000 7fcfdeeda640 1 --2- 172.21.15.23:0/4241935755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcfd81005d0 0x7fcfd819de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.854+0000 7fcfdeeda640 1 --2- 172.21.15.23:0/4241935755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd80ffc10 0x7fcfd819d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.855+0000 7fcfdeeda640 1 -- 172.21.15.23:0/4241935755 >> 172.21.15.23:0/4241935755 conn(0x7fcfd8076bd0 msgr2=0x7fcfd80fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:13.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.855+0000 7fcfdeeda640 1 -- 172.21.15.23:0/4241935755 shutdown_connections 2024-03-24T10:50:13.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.855+0000 7fcfdeeda640 1 -- 172.21.15.23:0/4241935755 wait complete. 2024-03-24T10:50:13.866 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-24T10:50:13.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.971+0000 7fed715a1640 1 -- 172.21.15.23:0/654098246 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed6c100860 msgr2=0x7fed6c100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:13.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.971+0000 7fed715a1640 1 --2- 172.21.15.23:0/654098246 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed6c100860 0x7fed6c100cc0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fed54009900 tx=0x7fed5402bc30 comp rx=0 tx=0).stop 2024-03-24T10:50:13.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.971+0000 7fed715a1640 1 -- 172.21.15.23:0/654098246 shutdown_connections 2024-03-24T10:50:13.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.971+0000 7fed715a1640 1 --2- 172.21.15.23:0/654098246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed6c101200 0x7fed6c10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.971+0000 7fed715a1640 1 --2- 172.21.15.23:0/654098246 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed6c100860 0x7fed6c100cc0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.971+0000 7fed715a1640 1 --2- 172.21.15.23:0/654098246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed6c105590 0x7fed6c105970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.971+0000 7fed715a1640 1 -- 172.21.15.23:0/654098246 >> 172.21.15.23:0/654098246 conn(0x7fed6c0773e0 msgr2=0x7fed6c0777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:13.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.971+0000 7fed715a1640 1 -- 172.21.15.23:0/654098246 shutdown_connections 2024-03-24T10:50:13.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.971+0000 7fed715a1640 1 -- 172.21.15.23:0/654098246 wait complete. 2024-03-24T10:50:13.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed715a1640 1 Processor -- start 2024-03-24T10:50:13.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed715a1640 1 -- start start 2024-03-24T10:50:13.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed715a1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed6c100860 0x7fed6c19da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:13.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed715a1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed6c101200 0x7fed6c19dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:13.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed715a1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed6c105590 0x7fed6c197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:13.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed715a1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c1982c0 con 0x7fed6c100860 2024-03-24T10:50:13.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed715a1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c198430 con 0x7fed6c101200 2024-03-24T10:50:13.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed715a1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c1985f0 con 0x7fed6c105590 2024-03-24T10:50:13.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed6affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed6c100860 0x7fed6c19da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed6affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed6c100860 0x7fed6c19da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41568/0 (socket says 172.21.15.23:41568) 2024-03-24T10:50:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed6affd640 1 -- 172.21.15.23:0/869356056 learned_addr learned my addr 172.21.15.23:0/869356056 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed6b7fe640 1 --2- 172.21.15.23:0/869356056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed6c105590 0x7fed6c197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed63fff640 1 --2- 172.21.15.23:0/869356056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed6c101200 0x7fed6c19dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed6affd640 1 -- 172.21.15.23:0/869356056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed6c105590 msgr2=0x7fed6c197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed6affd640 1 --2- 172.21.15.23:0/869356056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed6c105590 0x7fed6c197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed6affd640 1 -- 172.21.15.23:0/869356056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed6c101200 msgr2=0x7fed6c19dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed6affd640 1 --2- 172.21.15.23:0/869356056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed6c101200 0x7fed6c19dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:13.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed6affd640 1 -- 172.21.15.23:0/869356056 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed54009560 con 0x7fed6c100860 2024-03-24T10:50:13.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.972+0000 7fed6affd640 1 --2- 172.21.15.23:0/869356056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed6c100860 0x7fed6c19da70 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00e9f0 tx=0x7fed5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:13.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.973+0000 7fed68ff9640 1 -- 172.21.15.23:0/869356056 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5c009720 con 0x7fed6c100860 2024-03-24T10:50:13.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.973+0000 7fed68ff9640 1 -- 172.21.15.23:0/869356056 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed5c00cd80 con 0x7fed6c100860 2024-03-24T10:50:13.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.973+0000 7fed68ff9640 1 -- 172.21.15.23:0/869356056 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5c010690 con 0x7fed6c100860 2024-03-24T10:50:13.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.973+0000 7fed715a1640 1 -- 172.21.15.23:0/869356056 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed6c1988d0 con 0x7fed6c100860 2024-03-24T10:50:13.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.973+0000 7fed715a1640 1 -- 172.21.15.23:0/869356056 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed6c1a49f0 con 0x7fed6c100860 2024-03-24T10:50:13.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.974+0000 7fed715a1640 1 -- 172.21.15.23:0/869356056 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed6c101d20 con 0x7fed6c100860 2024-03-24T10:50:13.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.975+0000 7fed68ff9640 1 -- 172.21.15.23:0/869356056 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fed5c0046c0 con 0x7fed6c100860 2024-03-24T10:50:13.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.976+0000 7fed68ff9640 1 --2- 172.21.15.23:0/869356056 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fed580a2350 0x7fed580a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:13.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.976+0000 7fed63fff640 1 --2- 172.21.15.23:0/869356056 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fed580a2350 0x7fed580a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:13.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.976+0000 7fed68ff9640 1 -- 172.21.15.23:0/869356056 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8615+0+0 (secure 0 0 0) 0x7fed5c014070 con 0x7fed6c100860 2024-03-24T10:50:13.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.976+0000 7fed63fff640 1 --2- 172.21.15.23:0/869356056 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fed580a2350 0x7fed580a4810 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fed54006010 tx=0x7fed54005e40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:13.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:13.979+0000 7fed68ff9640 1 -- 172.21.15.23:0/869356056 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fed5c0971c0 con 0x7fed6c100860 2024-03-24T10:50:14.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:14.187+0000 7fed715a1640 1 -- 172.21.15.23:0/869356056 --> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fed6c109b80 con 0x7fed580a2350 2024-03-24T10:50:14.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:14.194+0000 7fed68ff9640 1 -- 172.21.15.23:0/869356056 <== mgr.15288 v2:172.21.15.23:6800/581236369 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fed6c109b80 con 0x7fed580a2350 2024-03-24T10:50:14.195 INFO:teuthology.orchestra.run.smithi023.stdout:/ 1m 6m3h 2024-03-24T10:50:14.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:14.196+0000 7fed715a1640 1 -- 172.21.15.23:0/869356056 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fed580a2350 msgr2=0x7fed580a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:14.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:14.196+0000 7fed715a1640 1 --2- 172.21.15.23:0/869356056 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fed580a2350 0x7fed580a4810 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fed54006010 tx=0x7fed54005e40 comp rx=0 tx=0).stop 2024-03-24T10:50:14.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:14.196+0000 7fed715a1640 1 -- 172.21.15.23:0/869356056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed6c100860 msgr2=0x7fed6c19da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:14.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:14.196+0000 7fed715a1640 1 --2- 172.21.15.23:0/869356056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed6c100860 0x7fed6c19da70 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00e9f0 tx=0x7fed5c00edf0 comp rx=0 tx=0).stop 2024-03-24T10:50:14.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:14.196+0000 7fed715a1640 1 -- 172.21.15.23:0/869356056 shutdown_connections 2024-03-24T10:50:14.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:14.196+0000 7fed715a1640 1 --2- 172.21.15.23:0/869356056 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fed580a2350 0x7fed580a4810 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:14.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:14.196+0000 7fed715a1640 1 --2- 172.21.15.23:0/869356056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed6c105590 0x7fed6c197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:14.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:14.196+0000 7fed715a1640 1 --2- 172.21.15.23:0/869356056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed6c101200 0x7fed6c19dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:14.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:14.196+0000 7fed715a1640 1 --2- 172.21.15.23:0/869356056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed6c100860 0x7fed6c19da70 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:14.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:14.196+0000 7fed715a1640 1 -- 172.21.15.23:0/869356056 >> 172.21.15.23:0/869356056 conn(0x7fed6c0773e0 msgr2=0x7fed6c06a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:14.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:14.197+0000 7fed715a1640 1 -- 172.21.15.23:0/869356056 shutdown_connections 2024-03-24T10:50:14.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:14.197+0000 7fed715a1640 1 -- 172.21.15.23:0/869356056 wait complete. 2024-03-24T10:50:14.207 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-24T10:50:14.218 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-24T10:50:14.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:14 smithi023 ceph-mon[32466]: pgmap v316: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T10:50:14.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:14 smithi023 ceph-mon[32466]: from='client.18318 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:50:15.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:14 smithi117 ceph-mon[36368]: pgmap v316: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T10:50:15.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:14 smithi117 ceph-mon[36368]: from='client.18318 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:50:15.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:14 smithi116 ceph-mon[36248]: pgmap v316: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T10:50:15.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:14 smithi116 ceph-mon[36248]: from='client.18318 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:50:15.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:15 smithi023 ceph-mon[32466]: from='client.18324 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:50:16.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:15 smithi117 ceph-mon[36368]: from='client.18324 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:50:16.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:15 smithi116 ceph-mon[36248]: from='client.18324 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T10:50:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:16 smithi023 ceph-mon[32466]: pgmap v317: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T10:50:16.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:16 smithi023 ceph-mon[32466]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T10:50:17.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:16 smithi117 ceph-mon[36368]: pgmap v317: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T10:50:17.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:16 smithi117 ceph-mon[36368]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T10:50:17.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:16 smithi116 ceph-mon[36248]: pgmap v317: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-24T10:50:17.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:16 smithi116 ceph-mon[36248]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T10:50:18.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:18 smithi023 ceph-mon[32466]: pgmap v319: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T10:50:19.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:18 smithi117 ceph-mon[36368]: pgmap v319: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T10:50:19.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:18 smithi116 ceph-mon[36248]: pgmap v319: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T10:50:19.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:19 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:50:20.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:19 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:50:20.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:19 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T10:50:20.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:20 smithi023 ceph-mon[32466]: pgmap v320: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 10 op/s 2024-03-24T10:50:21.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:20 smithi117 ceph-mon[36368]: pgmap v320: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 10 op/s 2024-03-24T10:50:21.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:20 smithi116 ceph-mon[36248]: pgmap v320: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 10 op/s 2024-03-24T10:50:22.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:21 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:21 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:21 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:21 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:21 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:21 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:21 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:21 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:21 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:21 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:21 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:21 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:21 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:21 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:21 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:21 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:21 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:21 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:21 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:21 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:21 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:21 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:21 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:21 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:21 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:21 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:21 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:21 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:21 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:21 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:21 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:21 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:21 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:21 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:22.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:21 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:21 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:21 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:21 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:21 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:21 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:21 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:22.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:21 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:23.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:22 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi117 to 174.1M 2024-03-24T10:50:23.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:22 smithi023 ceph-mon[32466]: Unable to set osd_memory_target on smithi117 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T10:50:23.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:22 smithi023 ceph-mon[32466]: pgmap v321: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-24T10:50:23.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:22 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-24T10:50:23.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:22 smithi023 ceph-mon[32466]: Unable to set osd_memory_target on smithi023 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-03-24T10:50:23.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:22 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:23.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:22 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:23.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:22 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:23.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:22 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:23.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:22 smithi023 ceph-mon[32466]: Adjusting osd_memory_target on smithi116 to 174.1M 2024-03-24T10:50:23.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:22 smithi023 ceph-mon[32466]: Unable to set osd_memory_target on smithi116 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-24T10:50:23.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:22 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:50:23.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:22 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:50:23.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:22 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:23.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:22 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi117 to 174.1M 2024-03-24T10:50:23.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:22 smithi117 ceph-mon[36368]: Unable to set osd_memory_target on smithi117 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T10:50:23.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:22 smithi117 ceph-mon[36368]: pgmap v321: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-24T10:50:23.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:22 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-24T10:50:23.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:22 smithi117 ceph-mon[36368]: Unable to set osd_memory_target on smithi023 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-03-24T10:50:23.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:22 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:23.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:22 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:23.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:22 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:23.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:22 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:23.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:22 smithi117 ceph-mon[36368]: Adjusting osd_memory_target on smithi116 to 174.1M 2024-03-24T10:50:23.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:22 smithi117 ceph-mon[36368]: Unable to set osd_memory_target on smithi116 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-24T10:50:23.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:22 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:50:23.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:22 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:50:23.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:22 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:23.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:22 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi117 to 174.1M 2024-03-24T10:50:23.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:22 smithi116 ceph-mon[36248]: Unable to set osd_memory_target on smithi117 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T10:50:23.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:22 smithi116 ceph-mon[36248]: pgmap v321: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-24T10:50:23.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:22 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-24T10:50:23.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:22 smithi116 ceph-mon[36248]: Unable to set osd_memory_target on smithi023 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-03-24T10:50:23.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:22 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:23.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:22 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:23.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:22 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:23.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:22 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T10:50:23.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:22 smithi116 ceph-mon[36248]: Adjusting osd_memory_target on smithi116 to 174.1M 2024-03-24T10:50:23.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:22 smithi116 ceph-mon[36248]: Unable to set osd_memory_target on smithi116 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-24T10:50:23.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:22 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T10:50:23.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:22 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T10:50:23.269 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:22 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' 2024-03-24T10:50:25.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:24 smithi023 ceph-mon[32466]: pgmap v322: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-24T10:50:25.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:24 smithi117 ceph-mon[36368]: pgmap v322: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-24T10:50:25.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:24 smithi116 ceph-mon[36248]: pgmap v322: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-24T10:50:27.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:26 smithi023 ceph-mon[32466]: pgmap v323: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-24T10:50:27.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:26 smithi117 ceph-mon[36368]: pgmap v323: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-24T10:50:27.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:26 smithi116 ceph-mon[36248]: pgmap v323: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-24T10:50:28.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T10:50:28.053 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T10:50:28.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.173+0000 7fc2523ef640 1 -- 172.21.15.23:0/1383735861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc24c1005a0 msgr2=0x7fc24c100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.173+0000 7fc2523ef640 1 --2- 172.21.15.23:0/1383735861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc24c1005a0 0x7fc24c100a20 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fc240009900 tx=0x7fc24002bc30 comp rx=0 tx=0).stop 2024-03-24T10:50:28.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.173+0000 7fc2523ef640 1 -- 172.21.15.23:0/1383735861 shutdown_connections 2024-03-24T10:50:28.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.173+0000 7fc2523ef640 1 --2- 172.21.15.23:0/1383735861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc24c100f60 0x7fc24c10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.173+0000 7fc2523ef640 1 --2- 172.21.15.23:0/1383735861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc24c1005a0 0x7fc24c100a20 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.173+0000 7fc2523ef640 1 --2- 172.21.15.23:0/1383735861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c106660 0x7fc24c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.173+0000 7fc2523ef640 1 -- 172.21.15.23:0/1383735861 >> 172.21.15.23:0/1383735861 conn(0x7fc24c077710 msgr2=0x7fc24c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.173+0000 7fc2523ef640 1 -- 172.21.15.23:0/1383735861 shutdown_connections 2024-03-24T10:50:28.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.173+0000 7fc2523ef640 1 -- 172.21.15.23:0/1383735861 wait complete. 2024-03-24T10:50:28.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.173+0000 7fc2523ef640 1 Processor -- start 2024-03-24T10:50:28.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.173+0000 7fc2523ef640 1 -- start start 2024-03-24T10:50:28.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc2523ef640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc24c1005a0 0x7fc24c196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc2523ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c100f60 0x7fc24c196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc2523ef640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc24c106660 0x7fc24c197400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc2523ef640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1a60c0 con 0x7fc24c1005a0 2024-03-24T10:50:28.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc2523ef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1a6230 con 0x7fc24c100f60 2024-03-24T10:50:28.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc2523ef640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1a63f0 con 0x7fc24c106660 2024-03-24T10:50:28.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc24bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc24c1005a0 0x7fc24c196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc24bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc24c1005a0 0x7fc24c196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41104/0 (socket says 172.21.15.23:41104) 2024-03-24T10:50:28.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc24bfff640 1 -- 172.21.15.23:0/1133170194 learned_addr learned my addr 172.21.15.23:0/1133170194 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc24b7fe640 1 --2- 172.21.15.23:0/1133170194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c100f60 0x7fc24c196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc25098c640 1 --2- 172.21.15.23:0/1133170194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc24c106660 0x7fc24c197400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc24bfff640 1 -- 172.21.15.23:0/1133170194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc24c106660 msgr2=0x7fc24c197400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc24bfff640 1 --2- 172.21.15.23:0/1133170194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc24c106660 0x7fc24c197400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc24bfff640 1 -- 172.21.15.23:0/1133170194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c100f60 msgr2=0x7fc24c196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc24bfff640 1 --2- 172.21.15.23:0/1133170194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c100f60 0x7fc24c196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.174+0000 7fc24bfff640 1 -- 172.21.15.23:0/1133170194 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc240009560 con 0x7fc24c1005a0 2024-03-24T10:50:28.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.175+0000 7fc24bfff640 1 --2- 172.21.15.23:0/1133170194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc24c1005a0 0x7fc24c196980 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fc23c00e9f0 tx=0x7fc23c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.175+0000 7fc2497fa640 1 -- 172.21.15.23:0/1133170194 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c009700 con 0x7fc24c1005a0 2024-03-24T10:50:28.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.175+0000 7fc2523ef640 1 -- 172.21.15.23:0/1133170194 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24c1a66d0 con 0x7fc24c1005a0 2024-03-24T10:50:28.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.175+0000 7fc2497fa640 1 -- 172.21.15.23:0/1133170194 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc23c00cd80 con 0x7fc24c1005a0 2024-03-24T10:50:28.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.175+0000 7fc2497fa640 1 -- 172.21.15.23:0/1133170194 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c018920 con 0x7fc24c1005a0 2024-03-24T10:50:28.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.175+0000 7fc2523ef640 1 -- 172.21.15.23:0/1133170194 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24c1a6bb0 con 0x7fc24c1005a0 2024-03-24T10:50:28.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.177+0000 7fc2497fa640 1 -- 172.21.15.23:0/1133170194 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fc23c018a80 con 0x7fc24c1005a0 2024-03-24T10:50:28.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.177+0000 7fc254fc5740 1 Processor -- start 2024-03-24T10:50:28.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.177+0000 7fc254fc5740 1 -- start start 2024-03-24T10:50:28.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc2497fa640 1 --2- 172.21.15.23:0/1133170194 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc2340a2230 0x7fc2340a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc254fc5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585984441d0 0x5585984445b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc254fc5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558598444af0 0x55859844d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc254fc5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55859844d6d0 0x55859844fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc254fc5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558598368f30 con 0x55859844d6d0 2024-03-24T10:50:28.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc254fc5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5585983b2810 con 0x5585984441d0 2024-03-24T10:50:28.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc254fc5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5585983afd40 con 0x558598444af0 2024-03-24T10:50:28.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc25098c640 1 --2- 172.21.15.23:0/1133170194 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc2340a2230 0x7fc2340a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc24bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585984441d0 0x5585984445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc25098c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55859844d6d0 0x55859844fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc24bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585984441d0 0x5585984445b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:34540/0 (socket says 172.21.15.23:34540) 2024-03-24T10:50:28.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc24b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558598444af0 0x55859844d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc25098c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55859844d6d0 0x55859844fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41106/0 (socket says 172.21.15.23:41106) 2024-03-24T10:50:28.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc2497fa640 1 -- 172.21.15.23:0/1133170194 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fc23c014070 con 0x7fc24c1005a0 2024-03-24T10:50:28.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc24bfff640 1 -- 172.21.15.23:0/2184019748 learned_addr learned my addr 172.21.15.23:0/2184019748 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc24bfff640 1 -- 172.21.15.23:0/2184019748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558598444af0 msgr2=0x55859844d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc24bfff640 1 --2- 172.21.15.23:0/2184019748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558598444af0 0x55859844d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc24bfff640 1 -- 172.21.15.23:0/2184019748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55859844d6d0 msgr2=0x55859844fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc24bfff640 1 --2- 172.21.15.23:0/2184019748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55859844d6d0 0x55859844fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc24bfff640 1 -- 172.21.15.23:0/2184019748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55859836cea0 con 0x5585984441d0 2024-03-24T10:50:28.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc25098c640 1 --2- 172.21.15.23:0/1133170194 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc2340a2230 0x7fc2340a46f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc24c069950 tx=0x7fc244009f10 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.178+0000 7fc24bfff640 1 --2- 172.21.15.23:0/2184019748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585984441d0 0x5585984445b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc23c097800 tx=0x7fc23c097200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad72d745e75dcb57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.179+0000 7fc2327fc640 1 -- 172.21.15.23:0/2184019748 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c09ae30 con 0x5585984441d0 2024-03-24T10:50:28.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.179+0000 7fc2327fc640 1 -- 172.21.15.23:0/2184019748 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc23c09eed0 con 0x5585984441d0 2024-03-24T10:50:28.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.179+0000 7fc2327fc640 1 -- 172.21.15.23:0/2184019748 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c0750e0 con 0x5585984441d0 2024-03-24T10:50:28.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.179+0000 7fc254fc5740 1 -- 172.21.15.23:0/2184019748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585984441d0 msgr2=0x5585984445b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.179+0000 7fc254fc5740 1 --2- 172.21.15.23:0/2184019748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585984441d0 0x5585984445b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc23c097800 tx=0x7fc23c097200 comp rx=0 tx=0).stop 2024-03-24T10:50:28.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.179+0000 7fc254fc5740 1 -- 172.21.15.23:0/2184019748 shutdown_connections 2024-03-24T10:50:28.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.179+0000 7fc254fc5740 1 --2- 172.21.15.23:0/2184019748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55859844d6d0 0x55859844fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.179+0000 7fc254fc5740 1 --2- 172.21.15.23:0/2184019748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558598444af0 0x55859844d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.179+0000 7fc254fc5740 1 --2- 172.21.15.23:0/2184019748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5585984441d0 0x5585984445b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.179+0000 7fc254fc5740 1 -- 172.21.15.23:0/2184019748 >> 172.21.15.23:0/2184019748 conn(0x55859843e900 msgr2=0x55859843ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.179+0000 7fc254fc5740 1 -- 172.21.15.23:0/2184019748 shutdown_connections 2024-03-24T10:50:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.179+0000 7fc254fc5740 1 -- 172.21.15.23:0/2184019748 wait complete. 2024-03-24T10:50:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.180+0000 7fc254fc5740 1 -- start start 2024-03-24T10:50:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.180+0000 7fc254fc5740 1 Processor -- start 2024-03-24T10:50:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.180+0000 7fc254fc5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585984441d0 0x55859844a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.180+0000 7fc254fc5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558598444af0 0x55859844ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.180+0000 7fc254fc5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55859844d6d0 0x55859844b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.180+0000 7fc254fc5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5585983ab520 con 0x558598444af0 2024-03-24T10:50:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.180+0000 7fc254fc5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55859807e430 con 0x55859844d6d0 2024-03-24T10:50:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.180+0000 7fc254fc5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55859803ae00 con 0x5585984441d0 2024-03-24T10:50:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.180+0000 7fc254fc5740 10 client.0 _ll_register_callbacks cb 0x558598349d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:50:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.180+0000 7fc24b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558598444af0 0x55859844ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.180+0000 7fc24b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558598444af0 0x55859844ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41120/0 (socket says 172.21.15.23:41120) 2024-03-24T10:50:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.180+0000 7fc24b7fe640 1 -- 172.21.15.23:0/216980274 learned_addr learned my addr 172.21.15.23:0/216980274 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.180+0000 7fc24bfff640 1 --2- 172.21.15.23:0/216980274 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585984441d0 0x55859844a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.180+0000 7fc25098c640 1 --2- 172.21.15.23:0/216980274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55859844d6d0 0x55859844b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc24bfff640 1 -- 172.21.15.23:0/216980274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55859844d6d0 msgr2=0x55859844b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc24bfff640 1 --2- 172.21.15.23:0/216980274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55859844d6d0 0x55859844b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc24bfff640 1 -- 172.21.15.23:0/216980274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558598444af0 msgr2=0x55859844ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc24bfff640 1 --2- 172.21.15.23:0/216980274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558598444af0 0x55859844ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc24bfff640 1 -- 172.21.15.23:0/216980274 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24c1a6bb0 con 0x5585984441d0 2024-03-24T10:50:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc24b7fe640 1 --2- 172.21.15.23:0/216980274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558598444af0 0x55859844ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:50:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc24bfff640 1 --2- 172.21.15.23:0/216980274 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585984441d0 0x55859844a750 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x558598450110 tx=0x7fc23c00c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=874bb11ad7851e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc22b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T10:50:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc22b7fe640 1 -- 172.21.15.23:0/216980274 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c0a3520 con 0x5585984441d0 2024-03-24T10:50:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc22b7fe640 1 -- 172.21.15.23:0/216980274 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fc23c09ae30 con 0x5585984441d0 2024-03-24T10:50:28.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc254fc5740 1 -- 172.21.15.23:0/216980274 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5585980aa7e0 con 0x5585984441d0 2024-03-24T10:50:28.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc22b7fe640 1 -- 172.21.15.23:0/216980274 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc23c075660 con 0x5585984441d0 2024-03-24T10:50:28.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc254fc5740 10 client.28192 fetch_fsmap learned FSMap version 31 2024-03-24T10:50:28.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc254fc5740 1 -- 172.21.15.23:0/216980274 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5585983a9ba0 con 0x5585984441d0 2024-03-24T10:50:28.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc22b7fe640 1 -- 172.21.15.23:0/216980274 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5585980aa7e0 con 0x5585984441d0 2024-03-24T10:50:28.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.181+0000 7fc22b7fe640 1 -- 172.21.15.23:0/216980274 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fc23c0b3ab0 con 0x5585984441d0 2024-03-24T10:50:28.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.182+0000 7fc254fc5740 10 client.28192 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:50:28.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.182+0000 7fc254fc5740 10 client.28192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:50:28.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.182+0000 7fc254fc5740 20 client.28192 populate_metadata read hostname 'smithi023' 2024-03-24T10:50:28.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.182+0000 7fc254fc5740 1 --2- 172.21.15.23:0/216980274 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55859845d8c0 0x55859845fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.182+0000 7fc254fc5740 4 client.28192 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:50:28.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.182+0000 7fc254fc5740 1 -- 172.21.15.23:0/216980274 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585980e7720 con 0x55859845d8c0 2024-03-24T10:50:28.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.182+0000 7fc24b7fe640 1 --2- 172.21.15.23:0/216980274 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55859845d8c0 0x55859845fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.182+0000 7fc24b7fe640 1 --2- 172.21.15.23:0/216980274 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55859845d8c0 0x55859845fcd0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b4d8be3f3027657 server_cookie=d026dd8a0e9d0de4 in_seq=0 out_seq=0 2024-03-24T10:50:28.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.182+0000 7fc22b7fe640 10 client.28192 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:50:28.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc22b7fe640 1 -- 172.21.15.23:0/216980274 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5585980e7720 con 0x55859845d8c0 2024-03-24T10:50:28.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc22b7fe640 10 client.28192 handle_command_reply: tid=0 2024-03-24T10:50:28.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc254fc5740 1 client.28192 shutdown 2024-03-24T10:50:28.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc254fc5740 1 -- 172.21.15.23:0/216980274 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585984441d0 msgr2=0x55859844a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc254fc5740 1 --2- 172.21.15.23:0/216980274 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585984441d0 0x55859844a750 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x558598450110 tx=0x7fc23c00c7d0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc254fc5740 1 -- 172.21.15.23:0/216980274 shutdown_connections 2024-03-24T10:50:28.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc254fc5740 1 --2- 172.21.15.23:0/216980274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55859844d6d0 0x55859844b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc254fc5740 1 --2- 172.21.15.23:0/216980274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558598444af0 0x55859844ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc254fc5740 1 --2- 172.21.15.23:0/216980274 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55859845d8c0 0x55859845fcd0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc254fc5740 1 --2- 172.21.15.23:0/216980274 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585984441d0 0x55859844a750 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc22b7fe640 0 client.28192 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:50:28.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc254fc5740 1 -- 172.21.15.23:0/216980274 >> 172.21.15.23:0/216980274 conn(0x55859843e900 msgr2=0x55859843db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc254fc5740 1 -- 172.21.15.23:0/216980274 shutdown_connections 2024-03-24T10:50:28.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc254fc5740 1 -- 172.21.15.23:0/216980274 wait complete. 2024-03-24T10:50:28.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.183+0000 7fc254fc5740 20 client.28192 trim_cache size 0 max 16384 2024-03-24T10:50:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.200+0000 7fc254fc5740 1 Processor -- start 2024-03-24T10:50:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.200+0000 7fc254fc5740 1 -- start start 2024-03-24T10:50:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc254fc5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585984441d0 0x55859844ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc254fc5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558598444af0 0x558598449690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc254fc5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55859844d6d0 0x558598458f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc254fc5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x558598330430 con 0x5585984441d0 2024-03-24T10:50:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc254fc5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5585984463a0 con 0x558598444af0 2024-03-24T10:50:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc254fc5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55859845a5d0 con 0x55859844d6d0 2024-03-24T10:50:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc24bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585984441d0 0x55859844ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc24bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585984441d0 0x55859844ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41134/0 (socket says 172.21.15.23:41134) 2024-03-24T10:50:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc24b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558598444af0 0x558598449690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc24bfff640 1 -- 172.21.15.23:0/622090307 learned_addr learned my addr 172.21.15.23:0/622090307 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc25098c640 1 --2- 172.21.15.23:0/622090307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55859844d6d0 0x558598458f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc24b7fe640 1 -- 172.21.15.23:0/622090307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55859844d6d0 msgr2=0x558598458f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc24b7fe640 1 --2- 172.21.15.23:0/622090307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55859844d6d0 0x558598458f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc24b7fe640 1 -- 172.21.15.23:0/622090307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585984441d0 msgr2=0x55859844ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc24b7fe640 1 --2- 172.21.15.23:0/622090307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585984441d0 0x55859844ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc24b7fe640 1 -- 172.21.15.23:0/622090307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558598345b20 con 0x558598444af0 2024-03-24T10:50:28.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc24bfff640 1 --2- 172.21.15.23:0/622090307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585984441d0 0x55859844ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:50:28.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc24b7fe640 1 --2- 172.21.15.23:0/622090307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558598444af0 0x558598449690 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc240000c00 tx=0x7fc240005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b854ab829c66de7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc231ffb640 1 -- 172.21.15.23:0/622090307 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc240045040 con 0x558598444af0 2024-03-24T10:50:28.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc231ffb640 1 -- 172.21.15.23:0/622090307 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc240047040 con 0x558598444af0 2024-03-24T10:50:28.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.201+0000 7fc231ffb640 1 -- 172.21.15.23:0/622090307 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24002dea0 con 0x558598444af0 2024-03-24T10:50:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.202+0000 7fc254fc5740 1 -- 172.21.15.23:0/622090307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558598444af0 msgr2=0x558598449690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.202+0000 7fc254fc5740 1 --2- 172.21.15.23:0/622090307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558598444af0 0x558598449690 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc240000c00 tx=0x7fc240005a20 comp rx=0 tx=0).stop 2024-03-24T10:50:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.202+0000 7fc254fc5740 1 -- 172.21.15.23:0/622090307 shutdown_connections 2024-03-24T10:50:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.202+0000 7fc254fc5740 1 --2- 172.21.15.23:0/622090307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55859844d6d0 0x558598458f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.202+0000 7fc254fc5740 1 --2- 172.21.15.23:0/622090307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558598444af0 0x558598449690 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.202+0000 7fc254fc5740 1 --2- 172.21.15.23:0/622090307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5585984441d0 0x55859844ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.202+0000 7fc254fc5740 1 -- 172.21.15.23:0/622090307 >> 172.21.15.23:0/622090307 conn(0x55859843e900 msgr2=0x55859843db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.202+0000 7fc254fc5740 1 -- 172.21.15.23:0/622090307 shutdown_connections 2024-03-24T10:50:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.202+0000 7fc254fc5740 1 -- 172.21.15.23:0/622090307 wait complete. 2024-03-24T10:50:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.202+0000 7fc254fc5740 1 -- start start 2024-03-24T10:50:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc254fc5740 1 Processor -- start 2024-03-24T10:50:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc254fc5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585984441d0 0x5585984474b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc254fc5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558598444af0 0x5585984479f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc254fc5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55859844d6d0 0x558598447f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc254fc5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x5585984c2c30 con 0x558598444af0 2024-03-24T10:50:28.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc254fc5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5585984c2da0 con 0x55859844d6d0 2024-03-24T10:50:28.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc254fc5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5585984c2f60 con 0x5585984441d0 2024-03-24T10:50:28.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc254fc5740 10 client.0 _ll_register_callbacks cb 0x7fc2440117a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:50:28.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc24b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558598444af0 0x5585984479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc24b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558598444af0 0x5585984479f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41140/0 (socket says 172.21.15.23:41140) 2024-03-24T10:50:28.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc24b7fe640 1 -- 172.21.15.23:0/2264938624 learned_addr learned my addr 172.21.15.23:0/2264938624 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc25098c640 1 --2- 172.21.15.23:0/2264938624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55859844d6d0 0x558598447f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc24bfff640 1 --2- 172.21.15.23:0/2264938624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585984441d0 0x5585984474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc24b7fe640 1 -- 172.21.15.23:0/2264938624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585984441d0 msgr2=0x5585984474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc24b7fe640 1 --2- 172.21.15.23:0/2264938624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585984441d0 0x5585984474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc24b7fe640 1 -- 172.21.15.23:0/2264938624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55859844d6d0 msgr2=0x558598447f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc24b7fe640 1 --2- 172.21.15.23:0/2264938624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55859844d6d0 0x558598447f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.203+0000 7fc24b7fe640 1 -- 172.21.15.23:0/2264938624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585983a9ba0 con 0x558598444af0 2024-03-24T10:50:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc24b7fe640 1 --2- 172.21.15.23:0/2264938624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558598444af0 0x5585984479f0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fc240006010 tx=0x7fc24002c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743bf74748e0d0a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc22b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:50:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc22b7fe640 1 -- 172.21.15.23:0/2264938624 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc240045050 con 0x558598444af0 2024-03-24T10:50:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc22b7fe640 1 -- 172.21.15.23:0/2264938624 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fc24002cbf0 con 0x558598444af0 2024-03-24T10:50:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc254fc5740 1 -- 172.21.15.23:0/2264938624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc23c09eed0 con 0x558598444af0 2024-03-24T10:50:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc22b7fe640 1 -- 172.21.15.23:0/2264938624 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc24002dea0 con 0x558598444af0 2024-03-24T10:50:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc254fc5740 10 client.18342 fetch_fsmap learned FSMap version 31 2024-03-24T10:50:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc254fc5740 1 -- 172.21.15.23:0/2264938624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x7fc24c10dd20 con 0x558598444af0 2024-03-24T10:50:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc22b7fe640 1 -- 172.21.15.23:0/2264938624 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc23c09eed0 con 0x558598444af0 2024-03-24T10:50:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc22b7fe640 1 -- 172.21.15.23:0/2264938624 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fc24003c040 con 0x558598444af0 2024-03-24T10:50:28.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc254fc5740 10 client.18342 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:50:28.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc254fc5740 10 client.18342 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:50:28.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc254fc5740 20 client.18342 populate_metadata read hostname 'smithi023' 2024-03-24T10:50:28.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc254fc5740 1 --2- 172.21.15.23:0/2264938624 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5585984c4510 0x5585984c6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc254fc5740 4 client.18342 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T10:50:28.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc254fc5740 1 -- 172.21.15.23:0/2264938624 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558598136890 con 0x5585984c4510 2024-03-24T10:50:28.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.204+0000 7fc24bfff640 1 --2- 172.21.15.23:0/2264938624 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5585984c4510 0x5585984c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc24bfff640 1 --2- 172.21.15.23:0/2264938624 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5585984c4510 0x5585984c6930 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8b9c21a1408c7e6 server_cookie=2c4eb9081771450f in_seq=0 out_seq=0 2024-03-24T10:50:28.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc22b7fe640 10 client.18342 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:50:28.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc22b7fe640 1 -- 172.21.15.23:0/2264938624 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558598136890 con 0x5585984c4510 2024-03-24T10:50:28.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc22b7fe640 10 client.18342 handle_command_reply: tid=0 2024-03-24T10:50:28.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc254fc5740 1 client.18342 shutdown 2024-03-24T10:50:28.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc254fc5740 1 -- 172.21.15.23:0/2264938624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558598444af0 msgr2=0x5585984479f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc254fc5740 1 --2- 172.21.15.23:0/2264938624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558598444af0 0x5585984479f0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fc240006010 tx=0x7fc24002c740 comp rx=0 tx=0).stop 2024-03-24T10:50:28.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc254fc5740 1 -- 172.21.15.23:0/2264938624 shutdown_connections 2024-03-24T10:50:28.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc254fc5740 1 --2- 172.21.15.23:0/2264938624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55859844d6d0 0x558598447f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc254fc5740 1 --2- 172.21.15.23:0/2264938624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558598444af0 0x5585984479f0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc254fc5740 1 --2- 172.21.15.23:0/2264938624 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x5585984c4510 0x5585984c6930 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc254fc5740 1 --2- 172.21.15.23:0/2264938624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585984441d0 0x5585984474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc254fc5740 1 -- 172.21.15.23:0/2264938624 >> 172.21.15.23:0/2264938624 conn(0x55859843e900 msgr2=0x55859844a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc22b7fe640 0 client.18342 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:50:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc254fc5740 1 -- 172.21.15.23:0/2264938624 shutdown_connections 2024-03-24T10:50:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc254fc5740 1 -- 172.21.15.23:0/2264938624 wait complete. 2024-03-24T10:50:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.205+0000 7fc254fc5740 20 client.18342 trim_cache size 0 max 16384 2024-03-24T10:50:28.215 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:50:28.216 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-24T10:50:28.216 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-24T10:50:28.216 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:50:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.206+0000 7fc2523ef640 1 -- 172.21.15.23:0/1133170194 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc2340a2230 msgr2=0x7fc2340a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.206+0000 7fc2523ef640 1 --2- 172.21.15.23:0/1133170194 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc2340a2230 0x7fc2340a46f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc24c069950 tx=0x7fc244009f10 comp rx=0 tx=0).stop 2024-03-24T10:50:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.206+0000 7fc2523ef640 1 -- 172.21.15.23:0/1133170194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc24c1005a0 msgr2=0x7fc24c196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.206+0000 7fc2523ef640 1 --2- 172.21.15.23:0/1133170194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc24c1005a0 0x7fc24c196980 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fc23c00e9f0 tx=0x7fc23c00eeb0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.206+0000 7fc2523ef640 1 -- 172.21.15.23:0/1133170194 shutdown_connections 2024-03-24T10:50:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.206+0000 7fc2523ef640 1 --2- 172.21.15.23:0/1133170194 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc2340a2230 0x7fc2340a46f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.206+0000 7fc2523ef640 1 --2- 172.21.15.23:0/1133170194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc24c106660 0x7fc24c197400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.206+0000 7fc2523ef640 1 --2- 172.21.15.23:0/1133170194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc24c100f60 0x7fc24c196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.206+0000 7fc2523ef640 1 --2- 172.21.15.23:0/1133170194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc24c1005a0 0x7fc24c196980 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.206+0000 7fc2523ef640 1 -- 172.21.15.23:0/1133170194 >> 172.21.15.23:0/1133170194 conn(0x7fc24c077710 msgr2=0x7fc24c0ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.206+0000 7fc2523ef640 1 -- 172.21.15.23:0/1133170194 shutdown_connections 2024-03-24T10:50:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.206+0000 7fc2523ef640 1 -- 172.21.15.23:0/1133170194 wait complete. 2024-03-24T10:50:28.218 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T10:50:28.219 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.321+0000 7f88f681c640 1 -- 172.21.15.23:0/3584628785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0102980 msgr2=0x7f88f0102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.321+0000 7f88f681c640 1 --2- 172.21.15.23:0/3584628785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0102980 0x7f88f0102d60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009860 tx=0x7f88e402bb70 comp rx=0 tx=0).stop 2024-03-24T10:50:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.321+0000 7f88f681c640 1 -- 172.21.15.23:0/3584628785 shutdown_connections 2024-03-24T10:50:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.321+0000 7f88f681c640 1 --2- 172.21.15.23:0/3584628785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f010bd50 0x7f88f010e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.321+0000 7f88f681c640 1 --2- 172.21.15.23:0/3584628785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f01032a0 0x7f88f010b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.321+0000 7f88f681c640 1 --2- 172.21.15.23:0/3584628785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0102980 0x7f88f0102d60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.322+0000 7f88f681c640 1 -- 172.21.15.23:0/3584628785 >> 172.21.15.23:0/3584628785 conn(0x7f88f0077710 msgr2=0x7f88f0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.322+0000 7f88f681c640 1 -- 172.21.15.23:0/3584628785 shutdown_connections 2024-03-24T10:50:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.322+0000 7f88f681c640 1 -- 172.21.15.23:0/3584628785 wait complete. 2024-03-24T10:50:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.322+0000 7f88f681c640 1 Processor -- start 2024-03-24T10:50:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.322+0000 7f88f681c640 1 -- start start 2024-03-24T10:50:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.322+0000 7f88f681c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f0102980 0x7f88f0197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.322+0000 7f88f681c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f01032a0 0x7f88f01982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.322+0000 7f88f681c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f010bd50 0x7f88f019c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.322+0000 7f88f681c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f88f019cc30 con 0x7f88f010bd50 2024-03-24T10:50:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.322+0000 7f88f681c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f88f019cda0 con 0x7f88f0102980 2024-03-24T10:50:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.322+0000 7f88f681c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88f019cf60 con 0x7f88f01032a0 2024-03-24T10:50:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88f4db9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f010bd50 0x7f88f019c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f0102980 0x7f88f0197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88f4db9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f010bd50 0x7f88f019c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41164/0 (socket says 172.21.15.23:41164) 2024-03-24T10:50:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88f4db9640 1 -- 172.21.15.23:0/3233860325 learned_addr learned my addr 172.21.15.23:0/3233860325 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88ef7fe640 1 --2- 172.21.15.23:0/3233860325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f01032a0 0x7f88f01982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88f4db9640 1 -- 172.21.15.23:0/3233860325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f01032a0 msgr2=0x7f88f01982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88f4db9640 1 --2- 172.21.15.23:0/3233860325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f01032a0 0x7f88f01982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88f4db9640 1 -- 172.21.15.23:0/3233860325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f0102980 msgr2=0x7f88f0197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88f4db9640 1 --2- 172.21.15.23:0/3233860325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f0102980 0x7f88f0197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88f4db9640 1 -- 172.21.15.23:0/3233860325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e0009560 con 0x7f88f010bd50 2024-03-24T10:50:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88f4db9640 1 --2- 172.21.15.23:0/3233860325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f010bd50 0x7f88f019c630 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f88e800eb10 tx=0x7f88e800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88ed7fa640 1 -- 172.21.15.23:0/3233860325 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8004b20 con 0x7f88f010bd50 2024-03-24T10:50:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88f681c640 1 -- 172.21.15.23:0/3233860325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88e4009490 con 0x7f88f010bd50 2024-03-24T10:50:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88ed7fa640 1 -- 172.21.15.23:0/3233860325 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e8004c80 con 0x7f88f010bd50 2024-03-24T10:50:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88ed7fa640 1 -- 172.21.15.23:0/3233860325 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8010650 con 0x7f88f010bd50 2024-03-24T10:50:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.323+0000 7f88f681c640 1 -- 172.21.15.23:0/3233860325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f01a48e0 con 0x7f88f010bd50 2024-03-24T10:50:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.324+0000 7f88f681c640 1 -- 172.21.15.23:0/3233860325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f0068500 con 0x7f88f010bd50 2024-03-24T10:50:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.325+0000 7f88ed7fa640 1 -- 172.21.15.23:0/3233860325 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f88e8005020 con 0x7f88f010bd50 2024-03-24T10:50:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.326+0000 7f88ed7fa640 1 --2- 172.21.15.23:0/3233860325 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f88d80a2230 0x7f88d80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.326+0000 7f88effff640 1 --2- 172.21.15.23:0/3233860325 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f88d80a2230 0x7f88d80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.326+0000 7f88ed7fa640 1 -- 172.21.15.23:0/3233860325 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f88e8014070 con 0x7f88f010bd50 2024-03-24T10:50:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.326+0000 7f88effff640 1 --2- 172.21.15.23:0/3233860325 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f88d80a2230 0x7f88d80a46f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f88e402c180 tx=0x7f88e40058d0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.330+0000 7f88ed7fa640 1 -- 172.21.15.23:0/3233860325 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f88e80972b0 con 0x7f88f010bd50 2024-03-24T10:50:28.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.552+0000 7f88f681c640 1 -- 172.21.15.23:0/3233860325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88f01990c0 con 0x7f88f010bd50 2024-03-24T10:50:28.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.553+0000 7f88ed7fa640 1 -- 172.21.15.23:0/3233860325 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11869 (secure 0 0 0) 0x7f88e809a970 con 0x7f88f010bd50 2024-03-24T10:50:28.553 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:50:28.554 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:47:17.474711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27526":{"gid":27526,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/2017557046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":2017557046},{"type":"v1","addr":"172.21.15.117:6837","nonce":2017557046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:50:28.554 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 31 2024-03-24T10:50:28.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.555+0000 7f88f681c640 1 -- 172.21.15.23:0/3233860325 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f88d80a2230 msgr2=0x7f88d80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.555+0000 7f88f681c640 1 --2- 172.21.15.23:0/3233860325 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f88d80a2230 0x7f88d80a46f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f88e402c180 tx=0x7f88e40058d0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.555+0000 7f88f681c640 1 -- 172.21.15.23:0/3233860325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f010bd50 msgr2=0x7f88f019c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.555+0000 7f88f681c640 1 --2- 172.21.15.23:0/3233860325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f010bd50 0x7f88f019c630 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f88e800eb10 tx=0x7f88e800c2d0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.555+0000 7f88f681c640 1 -- 172.21.15.23:0/3233860325 shutdown_connections 2024-03-24T10:50:28.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.555+0000 7f88f681c640 1 --2- 172.21.15.23:0/3233860325 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f88d80a2230 0x7f88d80a46f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.555+0000 7f88f681c640 1 --2- 172.21.15.23:0/3233860325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f010bd50 0x7f88f019c630 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.555+0000 7f88f681c640 1 --2- 172.21.15.23:0/3233860325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f01032a0 0x7f88f01982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.555+0000 7f88f681c640 1 --2- 172.21.15.23:0/3233860325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f0102980 0x7f88f0197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.555+0000 7f88f681c640 1 -- 172.21.15.23:0/3233860325 >> 172.21.15.23:0/3233860325 conn(0x7f88f0077710 msgr2=0x7f88f00fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.556+0000 7f88f681c640 1 -- 172.21.15.23:0/3233860325 shutdown_connections 2024-03-24T10:50:28.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.556+0000 7f88f681c640 1 -- 172.21.15.23:0/3233860325 wait complete. 2024-03-24T10:50:28.567 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T10:50:28.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.671+0000 7feab2719640 1 -- 172.21.15.23:0/2713485616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feaac106660 msgr2=0x7feaac106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.671+0000 7feab2719640 1 --2- 172.21.15.23:0/2713485616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feaac106660 0x7feaac106a40 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fea94009900 tx=0x7fea9402bc30 comp rx=0 tx=0).stop 2024-03-24T10:50:28.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.671+0000 7feab2719640 1 -- 172.21.15.23:0/2713485616 shutdown_connections 2024-03-24T10:50:28.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.671+0000 7feab2719640 1 --2- 172.21.15.23:0/2713485616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaac100f60 0x7feaac10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.671+0000 7feab2719640 1 --2- 172.21.15.23:0/2713485616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaac1005a0 0x7feaac100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.671+0000 7feab2719640 1 --2- 172.21.15.23:0/2713485616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feaac106660 0x7feaac106a40 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.671+0000 7feab2719640 1 -- 172.21.15.23:0/2713485616 >> 172.21.15.23:0/2713485616 conn(0x7feaac077710 msgr2=0x7feaac077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.671+0000 7feab2719640 1 -- 172.21.15.23:0/2713485616 shutdown_connections 2024-03-24T10:50:28.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.671+0000 7feab2719640 1 -- 172.21.15.23:0/2713485616 wait complete. 2024-03-24T10:50:28.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.672+0000 7feab2719640 1 Processor -- start 2024-03-24T10:50:28.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.672+0000 7feab2719640 1 -- start start 2024-03-24T10:50:28.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.672+0000 7feab2719640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaac1005a0 0x7feaac19d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.672+0000 7feab2719640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feaac100f60 0x7feaac19de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.672+0000 7feab2719640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaac106660 0x7feaac197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.672+0000 7feab2719640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7feaac198140 con 0x7feaac100f60 2024-03-24T10:50:28.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.672+0000 7feab2719640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feaac1982b0 con 0x7feaac1005a0 2024-03-24T10:50:28.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.672+0000 7feab2719640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feaac198470 con 0x7feaac106660 2024-03-24T10:50:28.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.672+0000 7feaab7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feaac100f60 0x7feaac19de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.672+0000 7feaab7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feaac100f60 0x7feaac19de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41184/0 (socket says 172.21.15.23:41184) 2024-03-24T10:50:28.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.672+0000 7feaab7fe640 1 -- 172.21.15.23:0/1798753060 learned_addr learned my addr 172.21.15.23:0/1798753060 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.672+0000 7feaabfff640 1 --2- 172.21.15.23:0/1798753060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaac1005a0 0x7feaac19d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.672+0000 7feab0cb6640 1 --2- 172.21.15.23:0/1798753060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaac106660 0x7feaac197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.673+0000 7feaab7fe640 1 -- 172.21.15.23:0/1798753060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaac106660 msgr2=0x7feaac197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.673+0000 7feaab7fe640 1 --2- 172.21.15.23:0/1798753060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaac106660 0x7feaac197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.673+0000 7feaab7fe640 1 -- 172.21.15.23:0/1798753060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaac1005a0 msgr2=0x7feaac19d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.673+0000 7feaab7fe640 1 --2- 172.21.15.23:0/1798753060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaac1005a0 0x7feaac19d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.673+0000 7feaab7fe640 1 -- 172.21.15.23:0/1798753060 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9c009490 con 0x7feaac100f60 2024-03-24T10:50:28.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.673+0000 7feaab7fe640 1 --2- 172.21.15.23:0/1798753060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feaac100f60 0x7feaac19de80 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00ea60 tx=0x7fea9c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.673+0000 7feaa97fa640 1 -- 172.21.15.23:0/1798753060 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c00cd20 con 0x7feaac100f60 2024-03-24T10:50:28.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.673+0000 7feaa97fa640 1 -- 172.21.15.23:0/1798753060 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea9c00ce80 con 0x7feaac100f60 2024-03-24T10:50:28.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.673+0000 7feab2719640 1 -- 172.21.15.23:0/1798753060 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaa4009840 con 0x7feaac100f60 2024-03-24T10:50:28.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.673+0000 7feaa97fa640 1 -- 172.21.15.23:0/1798753060 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c010690 con 0x7feaac100f60 2024-03-24T10:50:28.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.673+0000 7feab2719640 1 -- 172.21.15.23:0/1798753060 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea94009560 con 0x7feaac100f60 2024-03-24T10:50:28.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.675+0000 7feaa97fa640 1 -- 172.21.15.23:0/1798753060 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fea9c003ee0 con 0x7feaac100f60 2024-03-24T10:50:28.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.675+0000 7feab53cb740 1 Processor -- start 2024-03-24T10:50:28.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.675+0000 7feab53cb740 1 -- start start 2024-03-24T10:50:28.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feab53cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a363f0 0x564d19a367d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feab53cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a36d10 0x564d19a3f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feab53cb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a3f8f0 0x564d19a41ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feab53cb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x564d199a4840 con 0x564d19a3f8f0 2024-03-24T10:50:28.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feab53cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564d199a1d70 con 0x564d19a36d10 2024-03-24T10:50:28.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feab53cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564d1999d550 con 0x564d19a363f0 2024-03-24T10:50:28.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feaa97fa640 1 --2- 172.21.15.23:0/1798753060 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fea980a2230 0x7fea980a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feaab7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a3f8f0 0x564d19a41ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feaabfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a36d10 0x564d19a3f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feab0cb6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a363f0 0x564d19a367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feaabfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a36d10 0x564d19a3f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:34618/0 (socket says 172.21.15.23:34618) 2024-03-24T10:50:28.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feaab7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a3f8f0 0x564d19a41ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41200/0 (socket says 172.21.15.23:41200) 2024-03-24T10:50:28.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feaabfff640 1 -- 172.21.15.23:0/4098888407 learned_addr learned my addr 172.21.15.23:0/4098888407 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feab0cb6640 1 --2- 172.21.15.23:0/1798753060 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fea980a2230 0x7fea980a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feaa97fa640 1 -- 172.21.15.23:0/1798753060 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fea9c014070 con 0x7feaac100f60 2024-03-24T10:50:28.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feaabfff640 1 -- 172.21.15.23:0/4098888407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a363f0 msgr2=0x564d19a367d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feaabfff640 1 --2- 172.21.15.23:0/4098888407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a363f0 0x564d19a367d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feaabfff640 1 -- 172.21.15.23:0/4098888407 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a3f8f0 msgr2=0x564d19a41ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feaabfff640 1 --2- 172.21.15.23:0/4098888407 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a3f8f0 0x564d19a41ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feaabfff640 1 -- 172.21.15.23:0/4098888407 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d19932de0 con 0x564d19a36d10 2024-03-24T10:50:28.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feab0cb6640 1 --2- 172.21.15.23:0/4098888407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a363f0 0x564d19a367d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:50:28.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feab0cb6640 1 --2- 172.21.15.23:0/1798753060 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fea980a2230 0x7fea980a46f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7feaa4009b80 tx=0x7feaa4009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feaabfff640 1 --2- 172.21.15.23:0/4098888407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a36d10 0x564d19a3f3b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fea94009900 tx=0x7fea9402c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9911a5f4c708c060 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.676+0000 7feaa27fc640 1 -- 172.21.15.23:0/4098888407 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9402c910 con 0x564d19a36d10 2024-03-24T10:50:28.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.677+0000 7feaa27fc640 1 -- 172.21.15.23:0/4098888407 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea9402ca70 con 0x564d19a36d10 2024-03-24T10:50:28.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.677+0000 7feaa27fc640 1 -- 172.21.15.23:0/4098888407 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea940050e0 con 0x564d19a36d10 2024-03-24T10:50:28.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.677+0000 7feab53cb740 1 -- 172.21.15.23:0/4098888407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a36d10 msgr2=0x564d19a3f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.677+0000 7feab53cb740 1 --2- 172.21.15.23:0/4098888407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a36d10 0x564d19a3f3b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fea94009900 tx=0x7fea9402c6a0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.677+0000 7feab53cb740 1 -- 172.21.15.23:0/4098888407 shutdown_connections 2024-03-24T10:50:28.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.677+0000 7feab53cb740 1 --2- 172.21.15.23:0/4098888407 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a3f8f0 0x564d19a41ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.677+0000 7feab53cb740 1 --2- 172.21.15.23:0/4098888407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a36d10 0x564d19a3f3b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.677+0000 7feab53cb740 1 --2- 172.21.15.23:0/4098888407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a363f0 0x564d19a367d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.677+0000 7feab53cb740 1 -- 172.21.15.23:0/4098888407 >> 172.21.15.23:0/4098888407 conn(0x564d19a30b20 msgr2=0x564d19a30f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.677+0000 7feab53cb740 1 -- 172.21.15.23:0/4098888407 shutdown_connections 2024-03-24T10:50:28.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.677+0000 7feab53cb740 1 -- 172.21.15.23:0/4098888407 wait complete. 2024-03-24T10:50:28.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.677+0000 7feab53cb740 1 -- start start 2024-03-24T10:50:28.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feab53cb740 1 Processor -- start 2024-03-24T10:50:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feab53cb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a3c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feab53cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 0x564d19a3ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feab53cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 0x564d19a3d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feab53cb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x564d19670430 con 0x564d19a363f0 2024-03-24T10:50:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feab53cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564d1962ce00 con 0x564d19a3f8f0 2024-03-24T10:50:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feab53cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564d19922590 con 0x564d19a36d10 2024-03-24T10:50:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feab53cb740 10 client.0 _ll_register_callbacks cb 0x564d19991fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:50:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feaabfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a3c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feaabfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a3c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41214/0 (socket says 172.21.15.23:41214) 2024-03-24T10:50:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feaabfff640 1 -- 172.21.15.23:0/2364514676 learned_addr learned my addr 172.21.15.23:0/2364514676 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feab0cb6640 1 --2- 172.21.15.23:0/2364514676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 0x564d19a3d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feaab7fe640 1 --2- 172.21.15.23:0/2364514676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 0x564d19a3ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feaabfff640 1 -- 172.21.15.23:0/2364514676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 msgr2=0x564d19a3ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feaabfff640 1 --2- 172.21.15.23:0/2364514676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 0x564d19a3ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feaabfff640 1 -- 172.21.15.23:0/2364514676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 msgr2=0x564d19a3d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feaabfff640 1 --2- 172.21.15.23:0/2364514676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 0x564d19a3d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.678+0000 7feaabfff640 1 -- 172.21.15.23:0/2364514676 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaac068870 con 0x564d19a363f0 2024-03-24T10:50:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7feaabfff640 1 --2- 172.21.15.23:0/2364514676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a3c910 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fea9402cc60 tx=0x7fea940044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07655e0d02a5c5c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7fea8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:50:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7fea8b7fe640 1 -- 172.21.15.23:0/2364514676 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea94005270 con 0x564d19a363f0 2024-03-24T10:50:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7fea8b7fe640 1 -- 172.21.15.23:0/2364514676 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fea940053d0 con 0x564d19a363f0 2024-03-24T10:50:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7feab53cb740 1 -- 172.21.15.23:0/2364514676 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564d1969c7e0 con 0x564d19a363f0 2024-03-24T10:50:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7fea8b7fe640 1 -- 172.21.15.23:0/2364514676 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fea940055d0 con 0x564d19a363f0 2024-03-24T10:50:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7feab53cb740 10 client.18360 fetch_fsmap learned FSMap version 31 2024-03-24T10:50:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7feab53cb740 1 -- 172.21.15.23:0/2364514676 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564d1999bbd0 con 0x564d19a363f0 2024-03-24T10:50:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7fea8b7fe640 1 -- 172.21.15.23:0/2364514676 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea94057020 con 0x564d19a363f0 2024-03-24T10:50:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7fea8b7fe640 1 -- 172.21.15.23:0/2364514676 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fea9403c050 con 0x564d19a363f0 2024-03-24T10:50:28.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7feab53cb740 10 client.18360 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:50:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7feab53cb740 10 client.18360 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:50:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7feab53cb740 20 client.18360 populate_metadata read hostname 'smithi023' 2024-03-24T10:50:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7feab53cb740 1 --2- 172.21.15.23:0/2364514676 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x564d19a4f690 0x564d19a51aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7feab53cb740 4 client.18360 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:50:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.679+0000 7feab53cb740 1 -- 172.21.15.23:0/2364514676 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d196d9720 con 0x564d19a4f690 2024-03-24T10:50:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.680+0000 7feaab7fe640 1 --2- 172.21.15.23:0/2364514676 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x564d19a4f690 0x564d19a51aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.680+0000 7fea8b7fe640 10 client.18360 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:50:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.680+0000 7feaab7fe640 1 --2- 172.21.15.23:0/2364514676 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x564d19a4f690 0x564d19a51aa0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a3ce0603c6a43cb9 server_cookie=d6e4d45936d29b4a in_seq=0 out_seq=0 2024-03-24T10:50:28.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7fea8b7fe640 1 -- 172.21.15.23:0/2364514676 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564d196d9720 con 0x564d19a4f690 2024-03-24T10:50:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7fea8b7fe640 10 client.18360 handle_command_reply: tid=0 2024-03-24T10:50:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7feab53cb740 1 client.18360 shutdown 2024-03-24T10:50:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7feab53cb740 1 -- 172.21.15.23:0/2364514676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 msgr2=0x564d19a3c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7feab53cb740 1 --2- 172.21.15.23:0/2364514676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a3c910 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fea9402cc60 tx=0x7fea940044d0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7feab53cb740 1 -- 172.21.15.23:0/2364514676 shutdown_connections 2024-03-24T10:50:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7feab53cb740 1 --2- 172.21.15.23:0/2364514676 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x564d19a4f690 0x564d19a51aa0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7feab53cb740 1 --2- 172.21.15.23:0/2364514676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 0x564d19a3d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7feab53cb740 1 --2- 172.21.15.23:0/2364514676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 0x564d19a3ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7fea8b7fe640 0 client.18360 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:50:28.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7feab53cb740 1 --2- 172.21.15.23:0/2364514676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a3c910 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7feab53cb740 1 -- 172.21.15.23:0/2364514676 >> 172.21.15.23:0/2364514676 conn(0x564d19a30b20 msgr2=0x564d19a32870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7feab53cb740 1 -- 172.21.15.23:0/2364514676 shutdown_connections 2024-03-24T10:50:28.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7feab53cb740 1 -- 172.21.15.23:0/2364514676 wait complete. 2024-03-24T10:50:28.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.681+0000 7feab53cb740 20 client.18360 trim_cache size 0 max 16384 2024-03-24T10:50:28.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.694+0000 7feab53cb740 1 Processor -- start 2024-03-24T10:50:28.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.694+0000 7feab53cb740 1 -- start start 2024-03-24T10:50:28.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.694+0000 7feab53cb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a3f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.694+0000 7feab53cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 0x564d19a4b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.694+0000 7feab53cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 0x564d19a32680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.694+0000 7feab53cb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x564d19a31b20 con 0x564d19a363f0 2024-03-24T10:50:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.694+0000 7feab53cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564d19a21e20 con 0x564d19a3f8f0 2024-03-24T10:50:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.694+0000 7feab53cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564d19a2b0d0 con 0x564d19a36d10 2024-03-24T10:50:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.694+0000 7feaabfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a3f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.694+0000 7feaabfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a3f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41220/0 (socket says 172.21.15.23:41220) 2024-03-24T10:50:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.694+0000 7feaabfff640 1 -- 172.21.15.23:0/3060975968 learned_addr learned my addr 172.21.15.23:0/3060975968 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.695+0000 7feab0cb6640 1 --2- 172.21.15.23:0/3060975968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 0x564d19a32680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.695+0000 7feaab7fe640 1 --2- 172.21.15.23:0/3060975968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 0x564d19a4b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.695+0000 7feaabfff640 1 -- 172.21.15.23:0/3060975968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 msgr2=0x564d19a4b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.695+0000 7feaabfff640 1 --2- 172.21.15.23:0/3060975968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 0x564d19a4b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.695+0000 7feaabfff640 1 -- 172.21.15.23:0/3060975968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 msgr2=0x564d19a32680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.695+0000 7feaabfff640 1 --2- 172.21.15.23:0/3060975968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 0x564d19a32680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.695+0000 7feaabfff640 1 -- 172.21.15.23:0/3060975968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d19937c80 con 0x564d19a363f0 2024-03-24T10:50:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.695+0000 7feaabfff640 1 --2- 172.21.15.23:0/3060975968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a3f040 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fea9402bc70 tx=0x7fea9402c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa2044e5b0d3d6a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.695+0000 7feaa1ffb640 1 -- 172.21.15.23:0/3060975968 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea94034690 con 0x564d19a363f0 2024-03-24T10:50:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.695+0000 7feaa1ffb640 1 -- 172.21.15.23:0/3060975968 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea94057040 con 0x564d19a363f0 2024-03-24T10:50:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.695+0000 7feaa1ffb640 1 -- 172.21.15.23:0/3060975968 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9403f440 con 0x564d19a363f0 2024-03-24T10:50:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 -- 172.21.15.23:0/3060975968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 msgr2=0x564d19a3f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 --2- 172.21.15.23:0/3060975968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a3f040 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fea9402bc70 tx=0x7fea9402c3b0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 -- 172.21.15.23:0/3060975968 shutdown_connections 2024-03-24T10:50:28.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 --2- 172.21.15.23:0/3060975968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 0x564d19a32680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 --2- 172.21.15.23:0/3060975968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 0x564d19a4b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 --2- 172.21.15.23:0/3060975968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a3f040 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 -- 172.21.15.23:0/3060975968 >> 172.21.15.23:0/3060975968 conn(0x564d19a30b20 msgr2=0x564d19a31f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 -- 172.21.15.23:0/3060975968 shutdown_connections 2024-03-24T10:50:28.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 -- 172.21.15.23:0/3060975968 wait complete. 2024-03-24T10:50:28.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 -- start start 2024-03-24T10:50:28.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 Processor -- start 2024-03-24T10:50:28.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a39670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 0x564d19a39bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 0x564d19a3a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x564d19ab4a20 con 0x564d19a363f0 2024-03-24T10:50:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564d19ab4b90 con 0x564d19a3f8f0 2024-03-24T10:50:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564d19ab4d50 con 0x564d19a36d10 2024-03-24T10:50:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.696+0000 7feab53cb740 10 client.0 _ll_register_callbacks cb 0x7fea9c003ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:50:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feaabfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a39670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feaabfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a39670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41230/0 (socket says 172.21.15.23:41230) 2024-03-24T10:50:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feaabfff640 1 -- 172.21.15.23:0/2716733792 learned_addr learned my addr 172.21.15.23:0/2716733792 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feab0cb6640 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 0x564d19a3a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feaab7fe640 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 0x564d19a39bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feaabfff640 1 -- 172.21.15.23:0/2716733792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 msgr2=0x564d19a39bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feaabfff640 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 0x564d19a39bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feaabfff640 1 -- 172.21.15.23:0/2716733792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 msgr2=0x564d19a3a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feaabfff640 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 0x564d19a3a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feaabfff640 1 -- 172.21.15.23:0/2716733792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d1999bbd0 con 0x564d19a363f0 2024-03-24T10:50:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feaab7fe640 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 0x564d19a39bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:50:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feab0cb6640 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 0x564d19a3a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:50:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feaabfff640 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a39670 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fea940344a0 tx=0x7fea94034520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71bc68e44a257fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7fea8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:50:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7fea8b7fe640 1 -- 172.21.15.23:0/2716733792 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea94005480 con 0x564d19a363f0 2024-03-24T10:50:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feab53cb740 1 -- 172.21.15.23:0/2716733792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea9402ca70 con 0x564d19a363f0 2024-03-24T10:50:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7fea8b7fe640 1 -- 172.21.15.23:0/2716733792 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fea94000e40 con 0x564d19a363f0 2024-03-24T10:50:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7fea8b7fe640 1 -- 172.21.15.23:0/2716733792 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fea940055e0 con 0x564d19a363f0 2024-03-24T10:50:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feab53cb740 10 client.18366 fetch_fsmap learned FSMap version 31 2024-03-24T10:50:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7feab53cb740 1 -- 172.21.15.23:0/2716733792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564d1995d920 con 0x564d19a363f0 2024-03-24T10:50:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.697+0000 7fea8b7fe640 1 -- 172.21.15.23:0/2716733792 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea9402ca70 con 0x564d19a363f0 2024-03-24T10:50:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.698+0000 7fea8b7fe640 1 -- 172.21.15.23:0/2716733792 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7fea9403c040 con 0x564d19a363f0 2024-03-24T10:50:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.698+0000 7feab53cb740 10 client.18366 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:50:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.698+0000 7feab53cb740 10 client.18366 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-24T10:50:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.698+0000 7feab53cb740 20 client.18366 populate_metadata read hostname 'smithi023' 2024-03-24T10:50:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.698+0000 7feab53cb740 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x564d19ab6160 0x564d19ab8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.698+0000 7feab53cb740 4 client.18366 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:50:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.698+0000 7feab53cb740 1 -- 172.21.15.23:0/2716733792 --> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564d19728890 con 0x564d19ab6160 2024-03-24T10:50:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.698+0000 7feaab7fe640 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x564d19ab6160 0x564d19ab8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.698+0000 7fea8b7fe640 10 client.18366 ms_handle_connect on v2:172.21.15.116:6836/3576031724 2024-03-24T10:50:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.698+0000 7feaab7fe640 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x564d19ab6160 0x564d19ab8580 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5d89465c48b112c8 server_cookie=c894feb3b96ad89f in_seq=0 out_seq=0 2024-03-24T10:50:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.698+0000 7fea8b7fe640 1 -- 172.21.15.23:0/2716733792 <== mds.2 v2:172.21.15.116:6836/3576031724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564d19728890 con 0x564d19ab6160 2024-03-24T10:50:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.698+0000 7fea8b7fe640 10 client.18366 handle_command_reply: tid=0 2024-03-24T10:50:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.698+0000 7feab53cb740 1 client.18366 shutdown 2024-03-24T10:50:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab53cb740 1 -- 172.21.15.23:0/2716733792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 msgr2=0x564d19a39670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab53cb740 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a39670 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fea940344a0 tx=0x7fea94034520 comp rx=0 tx=0).stop 2024-03-24T10:50:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab53cb740 1 -- 172.21.15.23:0/2716733792 shutdown_connections 2024-03-24T10:50:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab53cb740 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.116:6836/3576031724,v1:172.21.15.116:6837/3576031724] conn(0x564d19ab6160 0x564d19ab8580 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab53cb740 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564d19a3f8f0 0x564d19a3a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab53cb740 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564d19a36d10 0x564d19a39bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7fea8b7fe640 0 client.18366 ms_handle_reset on v2:172.21.15.116:6836/3576031724 2024-03-24T10:50:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab53cb740 1 --2- 172.21.15.23:0/2716733792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d19a363f0 0x564d19a39670 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab53cb740 1 -- 172.21.15.23:0/2716733792 >> 172.21.15.23:0/2716733792 conn(0x564d19a30b20 msgr2=0x564d19a3d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab53cb740 1 -- 172.21.15.23:0/2716733792 shutdown_connections 2024-03-24T10:50:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab53cb740 1 -- 172.21.15.23:0/2716733792 wait complete. 2024-03-24T10:50:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab53cb740 20 client.18366 trim_cache size 0 max 16384 2024-03-24T10:50:28.708 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:50:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab2719640 1 -- 172.21.15.23:0/1798753060 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fea980a2230 msgr2=0x7fea980a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab2719640 1 --2- 172.21.15.23:0/1798753060 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fea980a2230 0x7fea980a46f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7feaa4009b80 tx=0x7feaa4009500 comp rx=0 tx=0).stop 2024-03-24T10:50:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab2719640 1 -- 172.21.15.23:0/1798753060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feaac100f60 msgr2=0x7feaac19de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab2719640 1 --2- 172.21.15.23:0/1798753060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feaac100f60 0x7feaac19de80 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00ea60 tx=0x7fea9c00ee60 comp rx=0 tx=0).stop 2024-03-24T10:50:28.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab2719640 1 -- 172.21.15.23:0/1798753060 shutdown_connections 2024-03-24T10:50:28.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab2719640 1 --2- 172.21.15.23:0/1798753060 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fea980a2230 0x7fea980a46f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab2719640 1 --2- 172.21.15.23:0/1798753060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaac106660 0x7feaac197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab2719640 1 --2- 172.21.15.23:0/1798753060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feaac100f60 0x7feaac19de80 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab2719640 1 --2- 172.21.15.23:0/1798753060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaac1005a0 0x7feaac19d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab2719640 1 -- 172.21.15.23:0/1798753060 >> 172.21.15.23:0/1798753060 conn(0x7feaac077710 msgr2=0x7feaac0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab2719640 1 -- 172.21.15.23:0/1798753060 shutdown_connections 2024-03-24T10:50:28.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.699+0000 7feab2719640 1 -- 172.21.15.23:0/1798753060 wait complete. 2024-03-24T10:50:28.711 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T10:50:28.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.816+0000 7f177f9a5640 1 -- 172.21.15.23:0/793069394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17781016a0 msgr2=0x7f1778101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.816+0000 7f177f9a5640 1 --2- 172.21.15.23:0/793069394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17781016a0 0x7f1778101b20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1774009860 tx=0x7f177402bb70 comp rx=0 tx=0).stop 2024-03-24T10:50:28.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.816+0000 7f177f9a5640 1 -- 172.21.15.23:0/793069394 shutdown_connections 2024-03-24T10:50:28.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.816+0000 7f177f9a5640 1 --2- 172.21.15.23:0/793069394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1778102060 0x7f177810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.816+0000 7f177f9a5640 1 --2- 172.21.15.23:0/793069394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17781016a0 0x7f1778101b20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.816+0000 7f177f9a5640 1 --2- 172.21.15.23:0/793069394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17781076b0 0x7f1778107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.816+0000 7f177f9a5640 1 -- 172.21.15.23:0/793069394 >> 172.21.15.23:0/793069394 conn(0x7f1778078b10 msgr2=0x7f1778078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.816+0000 7f177f9a5640 1 -- 172.21.15.23:0/793069394 shutdown_connections 2024-03-24T10:50:28.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.816+0000 7f177f9a5640 1 -- 172.21.15.23:0/793069394 wait complete. 2024-03-24T10:50:28.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177f9a5640 1 Processor -- start 2024-03-24T10:50:28.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177f9a5640 1 -- start start 2024-03-24T10:50:28.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177f9a5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17781016a0 0x7f177819da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177f9a5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1778102060 0x7f177819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177f9a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17781076b0 0x7f1778197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177f9a5640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f17781982b0 con 0x7f17781016a0 2024-03-24T10:50:28.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177f9a5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1778198420 con 0x7f17781076b0 2024-03-24T10:50:28.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177f9a5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f17781985e0 con 0x7f1778102060 2024-03-24T10:50:28.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177d741640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17781016a0 0x7f177819da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177d741640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17781016a0 0x7f177819da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41240/0 (socket says 172.21.15.23:41240) 2024-03-24T10:50:28.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177d741640 1 -- 172.21.15.23:0/329682912 learned_addr learned my addr 172.21.15.23:0/329682912 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177cf40640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1778102060 0x7f177819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177df42640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17781076b0 0x7f1778197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177d741640 1 -- 172.21.15.23:0/329682912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1778102060 msgr2=0x7f177819dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177d741640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1778102060 0x7f177819dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177d741640 1 -- 172.21.15.23:0/329682912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17781076b0 msgr2=0x7f1778197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177d741640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17781076b0 0x7f1778197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177d741640 1 -- 172.21.15.23:0/329682912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f176c009560 con 0x7f17781016a0 2024-03-24T10:50:28.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177cf40640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1778102060 0x7f177819dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:50:28.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.817+0000 7f177d741640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17781016a0 0x7f177819da60 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f176c00e9c0 tx=0x7f176c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.818+0000 7f176a7fc640 1 -- 172.21.15.23:0/329682912 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c00cde0 con 0x7f17781016a0 2024-03-24T10:50:28.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.818+0000 7f177f9a5640 1 -- 172.21.15.23:0/329682912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1774009490 con 0x7f17781016a0 2024-03-24T10:50:28.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.818+0000 7f176a7fc640 1 -- 172.21.15.23:0/329682912 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f176c0035f0 con 0x7f17781016a0 2024-03-24T10:50:28.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.818+0000 7f176a7fc640 1 -- 172.21.15.23:0/329682912 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c010650 con 0x7f17781016a0 2024-03-24T10:50:28.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.818+0000 7f177f9a5640 1 -- 172.21.15.23:0/329682912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17781a49e0 con 0x7f17781016a0 2024-03-24T10:50:28.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.820+0000 7f176a7fc640 1 -- 172.21.15.23:0/329682912 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f176c003ad0 con 0x7f17781016a0 2024-03-24T10:50:28.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.820+0000 7f1782583740 1 Processor -- start 2024-03-24T10:50:28.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.820+0000 7f1782583740 1 -- start start 2024-03-24T10:50:28.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.820+0000 7f176a7fc640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f17640a2230 0x7f17640a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f1782583740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1bdd0 0x56216cc1c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f1782583740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 0x56216cc297c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f176a7fc640 1 -- 172.21.15.23:0/329682912 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f176c014070 con 0x7f17781016a0 2024-03-24T10:50:28.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f1782583740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 0x56216cc2c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f1782583740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56216cb8a840 con 0x56216cc1bdd0 2024-03-24T10:50:28.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f1782583740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56216cb87d70 con 0x56216cc1c780 2024-03-24T10:50:28.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f1782583740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56216cb83550 con 0x56216cc29d00 2024-03-24T10:50:28.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f177df42640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f17640a2230 0x7f17640a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f177d741640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1bdd0 0x56216cc1c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f177cf40640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 0x56216cc297c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f177d741640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1bdd0 0x56216cc1c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41252/0 (socket says 172.21.15.23:41252) 2024-03-24T10:50:28.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f177d741640 1 -- 172.21.15.23:0/26380723 learned_addr learned my addr 172.21.15.23:0/26380723 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f177df42640 1 --2- 172.21.15.23:0/26380723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 0x56216cc2c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f177cf40640 1 -- 172.21.15.23:0/26380723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 msgr2=0x56216cc2c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f177cf40640 1 --2- 172.21.15.23:0/26380723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 0x56216cc2c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f177cf40640 1 -- 172.21.15.23:0/26380723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1bdd0 msgr2=0x56216cc1c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f177cf40640 1 --2- 172.21.15.23:0/26380723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1bdd0 0x56216cc1c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f177cf40640 1 -- 172.21.15.23:0/26380723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56216cb18de0 con 0x56216cc1c780 2024-03-24T10:50:28.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f177d741640 1 --2- 172.21.15.23:0/26380723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1bdd0 0x56216cc1c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:50:28.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f177df42640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f17640a2230 0x7f17640a46f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1770005fd0 tx=0x7f1770009b80 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f177cf40640 1 --2- 172.21.15.23:0/26380723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 0x56216cc297c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1774037040 tx=0x7f17740056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76fe9629ac70a872 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.821+0000 7f17637fe640 1 -- 172.21.15.23:0/26380723 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1774004300 con 0x56216cc1c780 2024-03-24T10:50:28.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.822+0000 7f17637fe640 1 -- 172.21.15.23:0/26380723 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1774004460 con 0x56216cc1c780 2024-03-24T10:50:28.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.822+0000 7f17637fe640 1 -- 172.21.15.23:0/26380723 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177402cbf0 con 0x56216cc1c780 2024-03-24T10:50:28.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.822+0000 7f1782583740 1 -- 172.21.15.23:0/26380723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 msgr2=0x56216cc297c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.822+0000 7f1782583740 1 --2- 172.21.15.23:0/26380723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 0x56216cc297c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1774037040 tx=0x7f17740056d0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.822+0000 7f1782583740 1 -- 172.21.15.23:0/26380723 shutdown_connections 2024-03-24T10:50:28.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.822+0000 7f1782583740 1 --2- 172.21.15.23:0/26380723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 0x56216cc2c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.822+0000 7f1782583740 1 --2- 172.21.15.23:0/26380723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 0x56216cc297c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.822+0000 7f1782583740 1 --2- 172.21.15.23:0/26380723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1bdd0 0x56216cc1c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.822+0000 7f1782583740 1 -- 172.21.15.23:0/26380723 >> 172.21.15.23:0/26380723 conn(0x56216cc16b20 msgr2=0x56216cc16f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.822+0000 7f1782583740 1 -- 172.21.15.23:0/26380723 shutdown_connections 2024-03-24T10:50:28.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.822+0000 7f1782583740 1 -- 172.21.15.23:0/26380723 wait complete. 2024-03-24T10:50:28.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f1782583740 1 -- start start 2024-03-24T10:50:28.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f1782583740 1 Processor -- start 2024-03-24T10:50:28.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f1782583740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1bdd0 0x56216cc26c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f1782583740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1c780 0x56216cc271c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f1782583740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 0x56216cc27700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f1782583740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56216c856430 con 0x56216cc1c780 2024-03-24T10:50:28.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f1782583740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56216c812e00 con 0x56216cc1bdd0 2024-03-24T10:50:28.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f1782583740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56216cb08590 con 0x56216cc29d00 2024-03-24T10:50:28.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f1782583740 10 client.0 _ll_register_callbacks cb 0x56216cb77fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:50:28.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f177cf40640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1c780 0x56216cc271c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f177cf40640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1c780 0x56216cc271c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41268/0 (socket says 172.21.15.23:41268) 2024-03-24T10:50:28.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f177cf40640 1 -- 172.21.15.23:0/2557529080 learned_addr learned my addr 172.21.15.23:0/2557529080 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f177d741640 1 --2- 172.21.15.23:0/2557529080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1bdd0 0x56216cc26c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f177df42640 1 --2- 172.21.15.23:0/2557529080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 0x56216cc27700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f177cf40640 1 -- 172.21.15.23:0/2557529080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 msgr2=0x56216cc27700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f177cf40640 1 --2- 172.21.15.23:0/2557529080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 0x56216cc27700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f177cf40640 1 -- 172.21.15.23:0/2557529080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1bdd0 msgr2=0x56216cc26c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f177cf40640 1 --2- 172.21.15.23:0/2557529080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1bdd0 0x56216cc26c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.823+0000 7f177cf40640 1 -- 172.21.15.23:0/2557529080 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f177810ee70 con 0x56216cc1c780 2024-03-24T10:50:28.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f177cf40640 1 --2- 172.21.15.23:0/2557529080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1c780 0x56216cc271c0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f1774000c00 tx=0x7f177402cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d770a4b6eb00963 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1760ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:50:28.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1760ff9640 1 -- 172.21.15.23:0/2557529080 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1774034640 con 0x56216cc1c780 2024-03-24T10:50:28.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1760ff9640 1 -- 172.21.15.23:0/2557529080 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f17740347a0 con 0x56216cc1c780 2024-03-24T10:50:28.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1782583740 1 -- 172.21.15.23:0/2557529080 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56216c8827e0 con 0x56216cc1c780 2024-03-24T10:50:28.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1760ff9640 1 -- 172.21.15.23:0/2557529080 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1774034960 con 0x56216cc1c780 2024-03-24T10:50:28.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1782583740 10 client.18378 fetch_fsmap learned FSMap version 31 2024-03-24T10:50:28.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1782583740 1 -- 172.21.15.23:0/2557529080 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56216cb81bd0 con 0x56216cc1c780 2024-03-24T10:50:28.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1760ff9640 1 -- 172.21.15.23:0/2557529080 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f177405f080 con 0x56216cc1c780 2024-03-24T10:50:28.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1760ff9640 1 -- 172.21.15.23:0/2557529080 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f1774051030 con 0x56216cc1c780 2024-03-24T10:50:28.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1782583740 10 client.18378 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:50:28.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1782583740 10 client.18378 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:50:28.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1782583740 20 client.18378 populate_metadata read hostname 'smithi023' 2024-03-24T10:50:28.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1782583740 1 --2- 172.21.15.23:0/2557529080 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56216cc39790 0x56216cc3bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1782583740 4 client.18378 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:50:28.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.824+0000 7f1782583740 1 -- 172.21.15.23:0/2557529080 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56216c8bf720 con 0x56216cc39790 2024-03-24T10:50:28.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.825+0000 7f177d741640 1 --2- 172.21.15.23:0/2557529080 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56216cc39790 0x56216cc3bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.825+0000 7f1760ff9640 10 client.18378 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:50:28.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.825+0000 7f177d741640 1 --2- 172.21.15.23:0/2557529080 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56216cc39790 0x56216cc3bbd0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9c87aa782b755c70 server_cookie=2834ad3193db1159 in_seq=0 out_seq=0 2024-03-24T10:50:28.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.825+0000 7f1760ff9640 1 -- 172.21.15.23:0/2557529080 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56216c8bf720 con 0x56216cc39790 2024-03-24T10:50:28.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.825+0000 7f1760ff9640 10 client.18378 handle_command_reply: tid=0 2024-03-24T10:50:28.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.825+0000 7f1782583740 1 client.18378 shutdown 2024-03-24T10:50:28.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.826+0000 7f1782583740 1 -- 172.21.15.23:0/2557529080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1c780 msgr2=0x56216cc271c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.826+0000 7f1782583740 1 --2- 172.21.15.23:0/2557529080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1c780 0x56216cc271c0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f1774000c00 tx=0x7f177402cc60 comp rx=0 tx=0).stop 2024-03-24T10:50:28.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.826+0000 7f1782583740 1 -- 172.21.15.23:0/2557529080 shutdown_connections 2024-03-24T10:50:28.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.826+0000 7f1782583740 1 --2- 172.21.15.23:0/2557529080 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56216cc39790 0x56216cc3bbd0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.826+0000 7f1782583740 1 --2- 172.21.15.23:0/2557529080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 0x56216cc27700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.826+0000 7f1782583740 1 --2- 172.21.15.23:0/2557529080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1c780 0x56216cc271c0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.826+0000 7f1760ff9640 0 client.18378 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:50:28.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.826+0000 7f1782583740 1 --2- 172.21.15.23:0/2557529080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1bdd0 0x56216cc26c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.826+0000 7f1782583740 1 -- 172.21.15.23:0/2557529080 >> 172.21.15.23:0/2557529080 conn(0x56216cc16b20 msgr2=0x56216cc18870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.826+0000 7f1782583740 1 -- 172.21.15.23:0/2557529080 shutdown_connections 2024-03-24T10:50:28.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.826+0000 7f1782583740 1 -- 172.21.15.23:0/2557529080 wait complete. 2024-03-24T10:50:28.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.826+0000 7f1782583740 20 client.18378 trim_cache size 0 max 16384 2024-03-24T10:50:28.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.839+0000 7f1782583740 1 Processor -- start 2024-03-24T10:50:28.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.839+0000 7f1782583740 1 -- start start 2024-03-24T10:50:28.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.839+0000 7f1782583740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc1bdd0 0x56216cc293b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f1782583740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 0x56216cc35450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f1782583740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc29d00 0x56216cc18680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f1782583740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56216c8bf720 con 0x56216cc29d00 2024-03-24T10:50:28.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f1782583740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56216cc17920 con 0x56216cc1c780 2024-03-24T10:50:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f1782583740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56216cc1d1d0 con 0x56216cc1bdd0 2024-03-24T10:50:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f177df42640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc29d00 0x56216cc18680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f177df42640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc29d00 0x56216cc18680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41284/0 (socket says 172.21.15.23:41284) 2024-03-24T10:50:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f177df42640 1 -- 172.21.15.23:0/2587521015 learned_addr learned my addr 172.21.15.23:0/2587521015 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f177cf40640 1 --2- 172.21.15.23:0/2587521015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 0x56216cc35450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f177d741640 1 --2- 172.21.15.23:0/2587521015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc1bdd0 0x56216cc293b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f177df42640 1 -- 172.21.15.23:0/2587521015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc1bdd0 msgr2=0x56216cc293b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f177df42640 1 --2- 172.21.15.23:0/2587521015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc1bdd0 0x56216cc293b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f177df42640 1 -- 172.21.15.23:0/2587521015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 msgr2=0x56216cc35450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f177df42640 1 --2- 172.21.15.23:0/2587521015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 0x56216cc35450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f177df42640 1 -- 172.21.15.23:0/2587521015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56216cb1dc80 con 0x56216cc29d00 2024-03-24T10:50:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f177df42640 1 --2- 172.21.15.23:0/2587521015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc29d00 0x56216cc18680 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f1770014010 tx=0x7f1770009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da387a37faea249 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f1762ffd640 1 -- 172.21.15.23:0/2587521015 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770025070 con 0x56216cc29d00 2024-03-24T10:50:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f1762ffd640 1 -- 172.21.15.23:0/2587521015 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17700115c0 con 0x56216cc29d00 2024-03-24T10:50:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.840+0000 7f1762ffd640 1 -- 172.21.15.23:0/2587521015 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770005020 con 0x56216cc29d00 2024-03-24T10:50:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.841+0000 7f1782583740 1 -- 172.21.15.23:0/2587521015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc29d00 msgr2=0x56216cc18680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.841+0000 7f1782583740 1 --2- 172.21.15.23:0/2587521015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc29d00 0x56216cc18680 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f1770014010 tx=0x7f1770009500 comp rx=0 tx=0).stop 2024-03-24T10:50:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.841+0000 7f1782583740 1 -- 172.21.15.23:0/2587521015 shutdown_connections 2024-03-24T10:50:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.841+0000 7f1782583740 1 --2- 172.21.15.23:0/2587521015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc29d00 0x56216cc18680 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.841+0000 7f1782583740 1 --2- 172.21.15.23:0/2587521015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 0x56216cc35450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.841+0000 7f1782583740 1 --2- 172.21.15.23:0/2587521015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc1bdd0 0x56216cc293b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.841+0000 7f1782583740 1 -- 172.21.15.23:0/2587521015 >> 172.21.15.23:0/2587521015 conn(0x56216cc16b20 msgr2=0x56216cc17f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.841+0000 7f1782583740 1 -- 172.21.15.23:0/2587521015 shutdown_connections 2024-03-24T10:50:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.841+0000 7f1782583740 1 -- 172.21.15.23:0/2587521015 wait complete. 2024-03-24T10:50:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.841+0000 7f1782583740 1 -- start start 2024-03-24T10:50:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.841+0000 7f1782583740 1 Processor -- start 2024-03-24T10:50:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f1782583740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1bdd0 0x56216cc2df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f1782583740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 0x56216cc2e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f1782583740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 0x56216cc97ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f1782583740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x56216cc981a0 con 0x56216cc1bdd0 2024-03-24T10:50:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f1782583740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56216cc98310 con 0x56216cc1c780 2024-03-24T10:50:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f1782583740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56216cc984d0 con 0x56216cc29d00 2024-03-24T10:50:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f1782583740 10 client.0 _ll_register_callbacks cb 0x7f176c096990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:50:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f177d741640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1bdd0 0x56216cc2df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f177cf40640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 0x56216cc2e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f177d741640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1bdd0 0x56216cc2df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41292/0 (socket says 172.21.15.23:41292) 2024-03-24T10:50:28.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f177d741640 1 -- 172.21.15.23:0/3174653952 learned_addr learned my addr 172.21.15.23:0/3174653952 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:28.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f177df42640 1 --2- 172.21.15.23:0/3174653952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 0x56216cc97ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f177cf40640 1 -- 172.21.15.23:0/3174653952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 msgr2=0x56216cc97ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f177cf40640 1 --2- 172.21.15.23:0/3174653952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 0x56216cc97ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f177cf40640 1 -- 172.21.15.23:0/3174653952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1bdd0 msgr2=0x56216cc2df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:28.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f177cf40640 1 --2- 172.21.15.23:0/3174653952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1bdd0 0x56216cc2df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f177cf40640 1 -- 172.21.15.23:0/3174653952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1770013e80 con 0x56216cc1c780 2024-03-24T10:50:28.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f177df42640 1 --2- 172.21.15.23:0/3174653952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 0x56216cc97ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:50:28.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f177d741640 1 --2- 172.21.15.23:0/3174653952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1bdd0 0x56216cc2df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:50:28.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f177cf40640 1 --2- 172.21.15.23:0/3174653952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 0x56216cc2e460 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f17740057c0 tx=0x7f1774034670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f0058ba03822b66 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f1760ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T10:50:28.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f1760ff9640 1 -- 172.21.15.23:0/3174653952 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17740035f0 con 0x56216cc1c780 2024-03-24T10:50:28.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f1782583740 1 -- 172.21.15.23:0/3174653952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1774004460 con 0x56216cc1c780 2024-03-24T10:50:28.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.842+0000 7f1760ff9640 1 -- 172.21.15.23:0/3174653952 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f1774000e40 con 0x56216cc1c780 2024-03-24T10:50:28.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.843+0000 7f1760ff9640 1 -- 172.21.15.23:0/3174653952 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1774034a70 con 0x56216cc1c780 2024-03-24T10:50:28.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.843+0000 7f1782583740 10 client.28271 fetch_fsmap learned FSMap version 31 2024-03-24T10:50:28.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.843+0000 7f1782583740 1 -- 172.21.15.23:0/3174653952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56216cb43920 con 0x56216cc1c780 2024-03-24T10:50:28.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.843+0000 7f1760ff9640 1 -- 172.21.15.23:0/3174653952 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1774004460 con 0x56216cc1c780 2024-03-24T10:50:28.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.843+0000 7f1760ff9640 1 -- 172.21.15.23:0/3174653952 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7f1774065030 con 0x56216cc1c780 2024-03-24T10:50:28.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.843+0000 7f1782583740 10 client.28271 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:50:28.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.843+0000 7f1782583740 10 client.28271 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T10:50:28.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.843+0000 7f1782583740 20 client.28271 populate_metadata read hostname 'smithi023' 2024-03-24T10:50:28.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.843+0000 7f1782583740 1 --2- 172.21.15.23:0/3174653952 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56216cca0a20 0x56216cca2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.843+0000 7f1782583740 4 client.28271 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:50:28.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.843+0000 7f1782583740 1 -- 172.21.15.23:0/3174653952 --> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56216c90e890 con 0x56216cca0a20 2024-03-24T10:50:28.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.843+0000 7f177d741640 1 --2- 172.21.15.23:0/3174653952 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56216cca0a20 0x56216cca2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f177d741640 1 --2- 172.21.15.23:0/3174653952 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56216cca0a20 0x56216cca2e40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c5d6ec532b12246a server_cookie=38a526f4e2ad5f39 in_seq=0 out_seq=0 2024-03-24T10:50:28.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1760ff9640 10 client.28271 ms_handle_connect on v2:172.21.15.117:6838/263724286 2024-03-24T10:50:28.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1760ff9640 1 -- 172.21.15.23:0/3174653952 <== mds.1 v2:172.21.15.117:6838/263724286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56216c90e890 con 0x56216cca0a20 2024-03-24T10:50:28.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1760ff9640 10 client.28271 handle_command_reply: tid=0 2024-03-24T10:50:28.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1782583740 1 client.28271 shutdown 2024-03-24T10:50:28.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1782583740 1 -- 172.21.15.23:0/3174653952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 msgr2=0x56216cc2e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1782583740 1 --2- 172.21.15.23:0/3174653952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 0x56216cc2e460 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f17740057c0 tx=0x7f1774034670 comp rx=0 tx=0).stop 2024-03-24T10:50:28.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1782583740 1 -- 172.21.15.23:0/3174653952 shutdown_connections 2024-03-24T10:50:28.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1782583740 1 --2- 172.21.15.23:0/3174653952 >> [v2:172.21.15.117:6838/263724286,v1:172.21.15.117:6839/263724286] conn(0x56216cca0a20 0x56216cca2e40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1782583740 1 --2- 172.21.15.23:0/3174653952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56216cc29d00 0x56216cc97ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1782583740 1 --2- 172.21.15.23:0/3174653952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56216cc1c780 0x56216cc2e460 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1782583740 1 --2- 172.21.15.23:0/3174653952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216cc1bdd0 0x56216cc2df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1760ff9640 0 client.28271 ms_handle_reset on v2:172.21.15.117:6838/263724286 2024-03-24T10:50:28.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1782583740 1 -- 172.21.15.23:0/3174653952 >> 172.21.15.23:0/3174653952 conn(0x56216cc16b20 msgr2=0x56216cc15cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1782583740 1 -- 172.21.15.23:0/3174653952 shutdown_connections 2024-03-24T10:50:28.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1782583740 1 -- 172.21.15.23:0/3174653952 wait complete. 2024-03-24T10:50:28.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f1782583740 20 client.28271 trim_cache size 0 max 16384 2024-03-24T10:50:28.916 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:50:28.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f177f9a5640 1 -- 172.21.15.23:0/329682912 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f17640a2230 msgr2=0x7f17640a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f177f9a5640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f17640a2230 0x7f17640a46f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1770005fd0 tx=0x7f1770009b80 comp rx=0 tx=0).stop 2024-03-24T10:50:28.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.844+0000 7f177f9a5640 1 -- 172.21.15.23:0/329682912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17781016a0 msgr2=0x7f177819da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.845+0000 7f177f9a5640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17781016a0 0x7f177819da60 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f176c00e9c0 tx=0x7f176c00ee80 comp rx=0 tx=0).stop 2024-03-24T10:50:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.845+0000 7f177f9a5640 1 -- 172.21.15.23:0/329682912 shutdown_connections 2024-03-24T10:50:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.845+0000 7f177f9a5640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f17640a2230 0x7f17640a46f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.845+0000 7f177f9a5640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17781076b0 0x7f1778197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.845+0000 7f177f9a5640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1778102060 0x7f177819dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.845+0000 7f177f9a5640 1 --2- 172.21.15.23:0/329682912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17781016a0 0x7f177819da60 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.845+0000 7f177f9a5640 1 -- 172.21.15.23:0/329682912 >> 172.21.15.23:0/329682912 conn(0x7f1778078b10 msgr2=0x7f1778104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.845+0000 7f177f9a5640 1 -- 172.21.15.23:0/329682912 shutdown_connections 2024-03-24T10:50:28.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:28.845+0000 7f177f9a5640 1 -- 172.21.15.23:0/329682912 wait complete. 2024-03-24T10:50:28.919 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T10:50:29.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.026+0000 7ff05f777640 1 -- 172.21.15.23:0/1597667183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0581005c0 msgr2=0x7ff058100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:29.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.026+0000 7ff05f777640 1 --2- 172.21.15.23:0/1597667183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0581005c0 0x7ff058100a40 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7ff04c009930 tx=0x7ff04c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:50:29.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.026+0000 7ff05f777640 1 -- 172.21.15.23:0/1597667183 shutdown_connections 2024-03-24T10:50:29.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.026+0000 7ff05f777640 1 --2- 172.21.15.23:0/1597667183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff058100f80 0x7ff05810d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.026+0000 7ff05f777640 1 --2- 172.21.15.23:0/1597667183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0581005c0 0x7ff058100a40 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.026+0000 7ff05f777640 1 --2- 172.21.15.23:0/1597667183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff058106660 0x7ff058106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.026+0000 7ff05f777640 1 -- 172.21.15.23:0/1597667183 >> 172.21.15.23:0/1597667183 conn(0x7ff058076990 msgr2=0x7ff058076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:29.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.026+0000 7ff05f777640 1 -- 172.21.15.23:0/1597667183 shutdown_connections 2024-03-24T10:50:29.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.027+0000 7ff05f777640 1 -- 172.21.15.23:0/1597667183 wait complete. 2024-03-24T10:50:29.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.027+0000 7ff05f777640 1 Processor -- start 2024-03-24T10:50:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.027+0000 7ff05f777640 1 -- start start 2024-03-24T10:50:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.027+0000 7ff05f777640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0581005c0 0x7ff05819da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.027+0000 7ff05f777640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff058100f80 0x7ff05819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.027+0000 7ff05f777640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff058106660 0x7ff058197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.027+0000 7ff05f777640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff0581982b0 con 0x7ff058106660 2024-03-24T10:50:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.027+0000 7ff05f777640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff058198420 con 0x7ff0581005c0 2024-03-24T10:50:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.027+0000 7ff05f777640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff0581985e0 con 0x7ff058100f80 2024-03-24T10:50:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.027+0000 7ff05d513640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0581005c0 0x7ff05819da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.027+0000 7ff05dd14640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff058106660 0x7ff058197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.027+0000 7ff05d513640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0581005c0 0x7ff05819da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:34686/0 (socket says 172.21.15.23:34686) 2024-03-24T10:50:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.027+0000 7ff05d513640 1 -- 172.21.15.23:0/605394676 learned_addr learned my addr 172.21.15.23:0/605394676 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.027+0000 7ff05cd12640 1 --2- 172.21.15.23:0/605394676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff058100f80 0x7ff05819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.028+0000 7ff05dd14640 1 -- 172.21.15.23:0/605394676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff058100f80 msgr2=0x7ff05819dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.028+0000 7ff05dd14640 1 --2- 172.21.15.23:0/605394676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff058100f80 0x7ff05819dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.028+0000 7ff05dd14640 1 -- 172.21.15.23:0/605394676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0581005c0 msgr2=0x7ff05819da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.028+0000 7ff05dd14640 1 --2- 172.21.15.23:0/605394676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0581005c0 0x7ff05819da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.028+0000 7ff05dd14640 1 -- 172.21.15.23:0/605394676 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04c009560 con 0x7ff058106660 2024-03-24T10:50:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.028+0000 7ff05dd14640 1 --2- 172.21.15.23:0/605394676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff058106660 0x7ff058197b20 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7ff04800e970 tx=0x7ff04800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:29.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.028+0000 7ff0567fc640 1 -- 172.21.15.23:0/605394676 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048009620 con 0x7ff058106660 2024-03-24T10:50:29.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.028+0000 7ff0567fc640 1 -- 172.21.15.23:0/605394676 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff04800ccc0 con 0x7ff058106660 2024-03-24T10:50:29.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.028+0000 7ff05f777640 1 -- 172.21.15.23:0/605394676 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0581988c0 con 0x7ff058106660 2024-03-24T10:50:29.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.028+0000 7ff0567fc640 1 -- 172.21.15.23:0/605394676 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048004a00 con 0x7ff058106660 2024-03-24T10:50:29.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.028+0000 7ff05f777640 1 -- 172.21.15.23:0/605394676 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0581a49e0 con 0x7ff058106660 2024-03-24T10:50:29.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.030+0000 7ff0567fc640 1 -- 172.21.15.23:0/605394676 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7ff048005020 con 0x7ff058106660 2024-03-24T10:50:29.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff06242e740 1 Processor -- start 2024-03-24T10:50:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff06242e740 1 -- start start 2024-03-24T10:50:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff0567fc640 1 --2- 172.21.15.23:0/605394676 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff0400a22b0 0x7ff0400a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff06242e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830a73f0 0x55b2830a77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff06242e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a7d10 0x55b2830b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff06242e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830b08f0 0x55b2830b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff06242e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b283015840 con 0x55b2830a7d10 2024-03-24T10:50:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff06242e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b283012d70 con 0x55b2830a73f0 2024-03-24T10:50:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff06242e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b28300e550 con 0x55b2830b08f0 2024-03-24T10:50:29.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff05cd12640 1 --2- 172.21.15.23:0/605394676 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff0400a22b0 0x7ff0400a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff0567fc640 1 -- 172.21.15.23:0/605394676 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7ff048014070 con 0x7ff058106660 2024-03-24T10:50:29.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff05cd12640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a7d10 0x55b2830b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff05d513640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830a73f0 0x55b2830a77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff05d513640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830a73f0 0x55b2830a77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:34696/0 (socket says 172.21.15.23:34696) 2024-03-24T10:50:29.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff05cd12640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a7d10 0x55b2830b03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41324/0 (socket says 172.21.15.23:41324) 2024-03-24T10:50:29.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff05d513640 1 -- 172.21.15.23:0/4156986849 learned_addr learned my addr 172.21.15.23:0/4156986849 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:29.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff05dd14640 1 --2- 172.21.15.23:0/4156986849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830b08f0 0x55b2830b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff05d513640 1 -- 172.21.15.23:0/4156986849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830b08f0 msgr2=0x55b2830b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:29.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff05d513640 1 --2- 172.21.15.23:0/4156986849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830b08f0 0x55b2830b2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff05d513640 1 -- 172.21.15.23:0/4156986849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a7d10 msgr2=0x55b2830b03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:29.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff05d513640 1 --2- 172.21.15.23:0/4156986849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a7d10 0x55b2830b03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.031+0000 7ff05d513640 1 -- 172.21.15.23:0/4156986849 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b282fa3de0 con 0x55b2830a73f0 2024-03-24T10:50:29.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.032+0000 7ff05cd12640 1 --2- 172.21.15.23:0/605394676 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff0400a22b0 0x7ff0400a4770 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff04c005ab0 tx=0x7ff04c02bc30 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:29.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.032+0000 7ff05d513640 1 --2- 172.21.15.23:0/4156986849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830a73f0 0x55b2830a77d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff050011540 tx=0x7ff050011a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=573ea7a4c7e744c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:29.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.032+0000 7ff03f7fe640 1 -- 172.21.15.23:0/4156986849 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff050003970 con 0x55b2830a73f0 2024-03-24T10:50:29.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.032+0000 7ff03f7fe640 1 -- 172.21.15.23:0/4156986849 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff050004020 con 0x55b2830a73f0 2024-03-24T10:50:29.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.032+0000 7ff03f7fe640 1 -- 172.21.15.23:0/4156986849 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05000f9b0 con 0x55b2830a73f0 2024-03-24T10:50:29.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.032+0000 7ff06242e740 1 -- 172.21.15.23:0/4156986849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830a73f0 msgr2=0x55b2830a77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:29.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.032+0000 7ff06242e740 1 --2- 172.21.15.23:0/4156986849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830a73f0 0x55b2830a77d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff050011540 tx=0x7ff050011a00 comp rx=0 tx=0).stop 2024-03-24T10:50:29.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.032+0000 7ff06242e740 1 -- 172.21.15.23:0/4156986849 shutdown_connections 2024-03-24T10:50:29.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.032+0000 7ff06242e740 1 --2- 172.21.15.23:0/4156986849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830b08f0 0x55b2830b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff06242e740 1 --2- 172.21.15.23:0/4156986849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a7d10 0x55b2830b03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff06242e740 1 --2- 172.21.15.23:0/4156986849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830a73f0 0x55b2830a77d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff06242e740 1 -- 172.21.15.23:0/4156986849 >> 172.21.15.23:0/4156986849 conn(0x55b2830a1b20 msgr2=0x55b2830a1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:29.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff06242e740 1 -- 172.21.15.23:0/4156986849 shutdown_connections 2024-03-24T10:50:29.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff06242e740 1 -- 172.21.15.23:0/4156986849 wait complete. 2024-03-24T10:50:29.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff06242e740 1 -- start start 2024-03-24T10:50:29.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff06242e740 1 Processor -- start 2024-03-24T10:50:29.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff06242e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a73f0 0x55b2830ad890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff06242e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830a7d10 0x55b2830addd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff06242e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830b08f0 0x55b2830ae310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff06242e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b282ce1430 con 0x55b2830b08f0 2024-03-24T10:50:29.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff06242e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b282c9de00 con 0x55b2830a7d10 2024-03-24T10:50:29.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff06242e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b282f93590 con 0x55b2830a73f0 2024-03-24T10:50:29.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff06242e740 10 client.0 _ll_register_callbacks cb 0x55b283002fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:50:29.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.033+0000 7ff05dd14640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830b08f0 0x55b2830ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff05dd14640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830b08f0 0x55b2830ae310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41328/0 (socket says 172.21.15.23:41328) 2024-03-24T10:50:29.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff05dd14640 1 -- 172.21.15.23:0/139760620 learned_addr learned my addr 172.21.15.23:0/139760620 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:29.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff05d513640 1 --2- 172.21.15.23:0/139760620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a73f0 0x55b2830ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff05cd12640 1 --2- 172.21.15.23:0/139760620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830a7d10 0x55b2830addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff05dd14640 1 -- 172.21.15.23:0/139760620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a73f0 msgr2=0x55b2830ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:29.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff05dd14640 1 --2- 172.21.15.23:0/139760620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a73f0 0x55b2830ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff05dd14640 1 -- 172.21.15.23:0/139760620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830a7d10 msgr2=0x55b2830addd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:29.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff05dd14640 1 --2- 172.21.15.23:0/139760620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830a7d10 0x55b2830addd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff05dd14640 1 -- 172.21.15.23:0/139760620 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff050009d30 con 0x55b2830b08f0 2024-03-24T10:50:29.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff05d513640 1 --2- 172.21.15.23:0/139760620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a73f0 0x55b2830ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:50:29.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff05dd14640 1 --2- 172.21.15.23:0/139760620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830b08f0 0x55b2830ae310 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7ff04800e940 tx=0x7ff04800bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d506370041f492a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:29.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff03cff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:50:29.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff03cff9640 1 -- 172.21.15.23:0/139760620 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04800ba30 con 0x55b2830b08f0 2024-03-24T10:50:29.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff03cff9640 1 -- 172.21.15.23:0/139760620 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7ff04800bc00 con 0x55b2830b08f0 2024-03-24T10:50:29.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff06242e740 1 -- 172.21.15.23:0/139760620 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b282d0d7e0 con 0x55b2830b08f0 2024-03-24T10:50:29.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff03cff9640 1 -- 172.21.15.23:0/139760620 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff04809a970 con 0x55b2830b08f0 2024-03-24T10:50:29.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff06242e740 10 client.18396 fetch_fsmap learned FSMap version 31 2024-03-24T10:50:29.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff06242e740 1 -- 172.21.15.23:0/139760620 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b28300cbd0 con 0x55b2830b08f0 2024-03-24T10:50:29.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.034+0000 7ff03cff9640 1 -- 172.21.15.23:0/139760620 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0480b7070 con 0x55b2830b08f0 2024-03-24T10:50:29.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.035+0000 7ff03cff9640 1 -- 172.21.15.23:0/139760620 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7ff0480a2c80 con 0x55b2830b08f0 2024-03-24T10:50:29.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.035+0000 7ff06242e740 10 client.18396 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:50:29.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.035+0000 7ff06242e740 10 client.18396 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:50:29.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.035+0000 7ff06242e740 20 client.18396 populate_metadata read hostname 'smithi023' 2024-03-24T10:50:29.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.035+0000 7ff06242e740 1 --2- 172.21.15.23:0/139760620 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b2830c0970 0x55b2830c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.035+0000 7ff06242e740 4 client.18396 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:50:29.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.035+0000 7ff06242e740 1 -- 172.21.15.23:0/139760620 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b282d4a720 con 0x55b2830c0970 2024-03-24T10:50:29.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.035+0000 7ff05d513640 1 --2- 172.21.15.23:0/139760620 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b2830c0970 0x55b2830c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.035+0000 7ff03cff9640 10 client.18396 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:50:29.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.035+0000 7ff05d513640 1 --2- 172.21.15.23:0/139760620 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b2830c0970 0x55b2830c2d90 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51ac6c3b573b1613 server_cookie=79f683c5e6b22e95 in_seq=0 out_seq=0 2024-03-24T10:50:29.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff03cff9640 1 -- 172.21.15.23:0/139760620 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b282d4a720 con 0x55b2830c0970 2024-03-24T10:50:29.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff03cff9640 10 client.18396 handle_command_reply: tid=0 2024-03-24T10:50:29.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff06242e740 1 client.18396 shutdown 2024-03-24T10:50:29.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff06242e740 1 -- 172.21.15.23:0/139760620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830b08f0 msgr2=0x55b2830ae310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:29.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff06242e740 1 --2- 172.21.15.23:0/139760620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830b08f0 0x55b2830ae310 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7ff04800e940 tx=0x7ff04800bef0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff06242e740 1 -- 172.21.15.23:0/139760620 shutdown_connections 2024-03-24T10:50:29.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff06242e740 1 --2- 172.21.15.23:0/139760620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830b08f0 0x55b2830ae310 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff06242e740 1 --2- 172.21.15.23:0/139760620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830a7d10 0x55b2830addd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff06242e740 1 --2- 172.21.15.23:0/139760620 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b2830c0970 0x55b2830c2d90 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff06242e740 1 --2- 172.21.15.23:0/139760620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a73f0 0x55b2830ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff03cff9640 0 client.18396 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:50:29.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff06242e740 1 -- 172.21.15.23:0/139760620 >> 172.21.15.23:0/139760620 conn(0x55b2830a1b20 msgr2=0x55b2830a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:29.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff06242e740 1 -- 172.21.15.23:0/139760620 shutdown_connections 2024-03-24T10:50:29.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff06242e740 1 -- 172.21.15.23:0/139760620 wait complete. 2024-03-24T10:50:29.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.036+0000 7ff06242e740 20 client.18396 trim_cache size 0 max 16384 2024-03-24T10:50:29.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff06242e740 1 Processor -- start 2024-03-24T10:50:29.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff06242e740 1 -- start start 2024-03-24T10:50:29.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff06242e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a73f0 0x55b2830affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff06242e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a7d10 0x55b2830ae130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff06242e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830b08f0 0x55b2830a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff06242e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b2830a6e30 con 0x55b2830a73f0 2024-03-24T10:50:29.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff06242e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b2830b2ce0 con 0x55b2830b08f0 2024-03-24T10:50:29.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff06242e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b2830a3e50 con 0x55b2830a7d10 2024-03-24T10:50:29.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff05d513640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a73f0 0x55b2830affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff05d513640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a73f0 0x55b2830affc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41342/0 (socket says 172.21.15.23:41342) 2024-03-24T10:50:29.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff05d513640 1 -- 172.21.15.23:0/3756430769 learned_addr learned my addr 172.21.15.23:0/3756430769 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:29.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff05cd12640 1 --2- 172.21.15.23:0/3756430769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a7d10 0x55b2830ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff05dd14640 1 --2- 172.21.15.23:0/3756430769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830b08f0 0x55b2830a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff05d513640 1 -- 172.21.15.23:0/3756430769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a7d10 msgr2=0x55b2830ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff05d513640 1 --2- 172.21.15.23:0/3756430769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a7d10 0x55b2830ae130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff05d513640 1 -- 172.21.15.23:0/3756430769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830b08f0 msgr2=0x55b2830a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff05d513640 1 --2- 172.21.15.23:0/3756430769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830b08f0 0x55b2830a0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff05d513640 1 -- 172.21.15.23:0/3756430769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b282fa8c80 con 0x55b2830a73f0 2024-03-24T10:50:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff05d513640 1 --2- 172.21.15.23:0/3756430769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a73f0 0x55b2830affc0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7ff050011460 tx=0x7ff05000f3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c570c141c5a1547 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff03effd640 1 -- 172.21.15.23:0/3756430769 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff050004880 con 0x55b2830a73f0 2024-03-24T10:50:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff03effd640 1 -- 172.21.15.23:0/3756430769 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff05002b070 con 0x55b2830a73f0 2024-03-24T10:50:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.050+0000 7ff03effd640 1 -- 172.21.15.23:0/3756430769 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05000f640 con 0x55b2830a73f0 2024-03-24T10:50:29.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.051+0000 7ff06242e740 1 -- 172.21.15.23:0/3756430769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a73f0 msgr2=0x55b2830affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:29.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.051+0000 7ff06242e740 1 --2- 172.21.15.23:0/3756430769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a73f0 0x55b2830affc0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7ff050011460 tx=0x7ff05000f3e0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.051+0000 7ff06242e740 1 -- 172.21.15.23:0/3756430769 shutdown_connections 2024-03-24T10:50:29.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.051+0000 7ff06242e740 1 --2- 172.21.15.23:0/3756430769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830b08f0 0x55b2830a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.051+0000 7ff06242e740 1 --2- 172.21.15.23:0/3756430769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a7d10 0x55b2830ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.051+0000 7ff06242e740 1 --2- 172.21.15.23:0/3756430769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a73f0 0x55b2830affc0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.051+0000 7ff06242e740 1 -- 172.21.15.23:0/3756430769 >> 172.21.15.23:0/3756430769 conn(0x55b2830a1b20 msgr2=0x55b2830a0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:29.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.051+0000 7ff06242e740 1 -- 172.21.15.23:0/3756430769 shutdown_connections 2024-03-24T10:50:29.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.051+0000 7ff06242e740 1 -- 172.21.15.23:0/3756430769 wait complete. 2024-03-24T10:50:29.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.051+0000 7ff06242e740 1 -- start start 2024-03-24T10:50:29.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff06242e740 1 Processor -- start 2024-03-24T10:50:29.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff06242e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a73f0 0x55b2830b4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff06242e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a7d10 0x55b2830b4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff06242e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830b08f0 0x55b283125cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff06242e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x55b2830b50a0 con 0x55b2830a73f0 2024-03-24T10:50:29.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff06242e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b2830b5210 con 0x55b2830b08f0 2024-03-24T10:50:29.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff06242e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b2831261f0 con 0x55b2830a7d10 2024-03-24T10:50:29.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff06242e740 10 client.0 _ll_register_callbacks cb 0x7ff04809a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:50:29.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff05d513640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a73f0 0x55b2830b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff05d513640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a73f0 0x55b2830b4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41346/0 (socket says 172.21.15.23:41346) 2024-03-24T10:50:29.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff05d513640 1 -- 172.21.15.23:0/958459686 learned_addr learned my addr 172.21.15.23:0/958459686 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:29.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff05dd14640 1 --2- 172.21.15.23:0/958459686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830b08f0 0x55b283125cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff05cd12640 1 --2- 172.21.15.23:0/958459686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a7d10 0x55b2830b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff05d513640 1 -- 172.21.15.23:0/958459686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a7d10 msgr2=0x55b2830b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:29.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff05d513640 1 --2- 172.21.15.23:0/958459686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a7d10 0x55b2830b4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff05d513640 1 -- 172.21.15.23:0/958459686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830b08f0 msgr2=0x55b283125cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:29.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff05d513640 1 --2- 172.21.15.23:0/958459686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830b08f0 0x55b283125cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff05d513640 1 -- 172.21.15.23:0/958459686 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b282fa3de0 con 0x55b2830a73f0 2024-03-24T10:50:29.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff05cd12640 1 --2- 172.21.15.23:0/958459686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a7d10 0x55b2830b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:50:29.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff05d513640 1 --2- 172.21.15.23:0/958459686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a73f0 0x55b2830b4400 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff0500044d0 tx=0x7ff0500047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33039e82992950d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:29.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.052+0000 7ff03cff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-24T10:50:29.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff03cff9640 1 -- 172.21.15.23:0/958459686 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff050003b80 con 0x55b2830a73f0 2024-03-24T10:50:29.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff03cff9640 1 -- 172.21.15.23:0/958459686 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7ff05003a030 con 0x55b2830a73f0 2024-03-24T10:50:29.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff06242e740 1 -- 172.21.15.23:0/958459686 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff050004020 con 0x55b2830a73f0 2024-03-24T10:50:29.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff03cff9640 1 -- 172.21.15.23:0/958459686 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff050003ce0 con 0x55b2830a73f0 2024-03-24T10:50:29.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff06242e740 10 client.18402 fetch_fsmap learned FSMap version 31 2024-03-24T10:50:29.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff06242e740 1 -- 172.21.15.23:0/958459686 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b282fce920 con 0x55b2830a73f0 2024-03-24T10:50:29.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff03cff9640 1 -- 172.21.15.23:0/958459686 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff050004020 con 0x55b2830a73f0 2024-03-24T10:50:29.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff03cff9640 1 -- 172.21.15.23:0/958459686 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) ==== 8338+0+0 (secure 0 0 0) 0x7ff05000fa40 con 0x55b2830a73f0 2024-03-24T10:50:29.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff06242e740 10 client.18402 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T10:50:29.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff06242e740 10 client.18402 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T10:50:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff06242e740 20 client.18402 populate_metadata read hostname 'smithi023' 2024-03-24T10:50:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff06242e740 1 --2- 172.21.15.23:0/958459686 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b283127610 0x55b283129a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff06242e740 4 client.18402 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T10:50:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff06242e740 1 -- 172.21.15.23:0/958459686 --> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b282d99890 con 0x55b283127610 2024-03-24T10:50:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.053+0000 7ff05cd12640 1 --2- 172.21.15.23:0/958459686 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b283127610 0x55b283129a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff05cd12640 1 --2- 172.21.15.23:0/958459686 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b283127610 0x55b283129a00 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e13b4ba972df1304 server_cookie=c2fbd0e7b8b431e9 in_seq=0 out_seq=0 2024-03-24T10:50:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff03cff9640 10 client.18402 ms_handle_connect on v2:172.21.15.116:6838/2435082224 2024-03-24T10:50:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff03cff9640 1 -- 172.21.15.23:0/958459686 <== mds.0 v2:172.21.15.116:6838/2435082224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b282d99890 con 0x55b283127610 2024-03-24T10:50:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff03cff9640 10 client.18402 handle_command_reply: tid=0 2024-03-24T10:50:29.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff06242e740 1 client.18402 shutdown 2024-03-24T10:50:29.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff06242e740 1 -- 172.21.15.23:0/958459686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a73f0 msgr2=0x55b2830b4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:29.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff06242e740 1 --2- 172.21.15.23:0/958459686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a73f0 0x55b2830b4400 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff0500044d0 tx=0x7ff0500047c0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff06242e740 1 -- 172.21.15.23:0/958459686 shutdown_connections 2024-03-24T10:50:29.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff06242e740 1 --2- 172.21.15.23:0/958459686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2830b08f0 0x55b283125cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff06242e740 1 --2- 172.21.15.23:0/958459686 >> [v2:172.21.15.116:6838/2435082224,v1:172.21.15.116:6839/2435082224] conn(0x55b283127610 0x55b283129a00 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff06242e740 1 --2- 172.21.15.23:0/958459686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2830a7d10 0x55b2830b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff03cff9640 0 client.18402 ms_handle_reset on v2:172.21.15.116:6838/2435082224 2024-03-24T10:50:29.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff06242e740 1 --2- 172.21.15.23:0/958459686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2830a73f0 0x55b2830b4400 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff06242e740 1 -- 172.21.15.23:0/958459686 >> 172.21.15.23:0/958459686 conn(0x55b2830a1b20 msgr2=0x55b2830a0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:29.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff06242e740 1 -- 172.21.15.23:0/958459686 shutdown_connections 2024-03-24T10:50:29.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff06242e740 1 -- 172.21.15.23:0/958459686 wait complete. 2024-03-24T10:50:29.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.054+0000 7ff06242e740 20 client.18402 trim_cache size 0 max 16384 2024-03-24T10:50:29.065 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-24T10:50:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.055+0000 7ff05f777640 1 -- 172.21.15.23:0/605394676 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff0400a22b0 msgr2=0x7ff0400a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.055+0000 7ff05f777640 1 --2- 172.21.15.23:0/605394676 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff0400a22b0 0x7ff0400a4770 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff04c005ab0 tx=0x7ff04c02bc30 comp rx=0 tx=0).stop 2024-03-24T10:50:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.055+0000 7ff05f777640 1 -- 172.21.15.23:0/605394676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff058106660 msgr2=0x7ff058197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.055+0000 7ff05f777640 1 --2- 172.21.15.23:0/605394676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff058106660 0x7ff058197b20 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7ff04800e970 tx=0x7ff04800ee30 comp rx=0 tx=0).stop 2024-03-24T10:50:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.055+0000 7ff05f777640 1 -- 172.21.15.23:0/605394676 shutdown_connections 2024-03-24T10:50:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.055+0000 7ff05f777640 1 --2- 172.21.15.23:0/605394676 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff0400a22b0 0x7ff0400a4770 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.055+0000 7ff05f777640 1 --2- 172.21.15.23:0/605394676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff058106660 0x7ff058197b20 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.055+0000 7ff05f777640 1 --2- 172.21.15.23:0/605394676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff058100f80 0x7ff05819dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.055+0000 7ff05f777640 1 --2- 172.21.15.23:0/605394676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0581005c0 0x7ff05819da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.055+0000 7ff05f777640 1 -- 172.21.15.23:0/605394676 >> 172.21.15.23:0/605394676 conn(0x7ff058076990 msgr2=0x7ff0580ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.055+0000 7ff05f777640 1 -- 172.21.15.23:0/605394676 shutdown_connections 2024-03-24T10:50:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:29.055+0000 7ff05f777640 1 -- 172.21.15.23:0/605394676 wait complete. 2024-03-24T10:50:29.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:28 smithi023 ceph-mon[32466]: pgmap v324: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 91 B/s wr, 10 op/s 2024-03-24T10:50:29.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:28 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3233860325' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:50:29.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:28 smithi117 ceph-mon[36368]: pgmap v324: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 91 B/s wr, 10 op/s 2024-03-24T10:50:29.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:28 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3233860325' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:50:29.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:28 smithi116 ceph-mon[36248]: pgmap v324: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 91 B/s wr, 10 op/s 2024-03-24T10:50:29.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:28 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3233860325' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:50:30.069 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-24T10:50:30.069 INFO:tasks.fwd_scrub:done joining 2024-03-24T10:50:30.069 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-24T10:50:30.080 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:50:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.195+0000 7f0fc53c7640 1 -- 172.21.15.23:0/1344499116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc01066b0 msgr2=0x7f0fc0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.195+0000 7f0fc53c7640 1 --2- 172.21.15.23:0/1344499116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc01066b0 0x7f0fc0106a90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009860 tx=0x7f0fb002bc00 comp rx=0 tx=0).stop 2024-03-24T10:50:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.195+0000 7f0fc53c7640 1 -- 172.21.15.23:0/1344499116 shutdown_connections 2024-03-24T10:50:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.195+0000 7f0fc53c7640 1 --2- 172.21.15.23:0/1344499116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fc0100d40 0x7f0fc010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.195+0000 7f0fc53c7640 1 --2- 172.21.15.23:0/1344499116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fc0106fd0 0x7f0fc0100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.195+0000 7f0fc53c7640 1 --2- 172.21.15.23:0/1344499116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc01066b0 0x7f0fc0106a90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.195+0000 7f0fc53c7640 1 -- 172.21.15.23:0/1344499116 >> 172.21.15.23:0/1344499116 conn(0x7f0fc0077710 msgr2=0x7f0fc0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.195+0000 7f0fc53c7640 1 -- 172.21.15.23:0/1344499116 shutdown_connections 2024-03-24T10:50:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.195+0000 7f0fc53c7640 1 -- 172.21.15.23:0/1344499116 wait complete. 2024-03-24T10:50:30.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.195+0000 7f0fc53c7640 1 Processor -- start 2024-03-24T10:50:30.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.195+0000 7f0fc53c7640 1 -- start start 2024-03-24T10:50:30.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fc53c7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fc0100d40 0x7f0fc0198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fc53c7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fc01066b0 0x7f0fc01990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fc53c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0106fd0 0x7f0fc01a3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fc53c7640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc0199970 con 0x7f0fc0100d40 2024-03-24T10:50:30.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fc53c7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc01a4150 con 0x7f0fc0106fd0 2024-03-24T10:50:30.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fc53c7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc01a4290 con 0x7f0fc01066b0 2024-03-24T10:50:30.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fbeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fc0100d40 0x7f0fc0198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fbeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fc0100d40 0x7f0fc0198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41368/0 (socket says 172.21.15.23:41368) 2024-03-24T10:50:30.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fbeffd640 1 -- 172.21.15.23:0/338535437 learned_addr learned my addr 172.21.15.23:0/338535437 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:30.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fbf7fe640 1 --2- 172.21.15.23:0/338535437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0106fd0 0x7f0fc01a3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fb7fff640 1 --2- 172.21.15.23:0/338535437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fc01066b0 0x7f0fc01990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fbeffd640 1 -- 172.21.15.23:0/338535437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fc01066b0 msgr2=0x7f0fc01990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fbeffd640 1 --2- 172.21.15.23:0/338535437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fc01066b0 0x7f0fc01990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fbeffd640 1 -- 172.21.15.23:0/338535437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0106fd0 msgr2=0x7f0fc01a3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fbeffd640 1 --2- 172.21.15.23:0/338535437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0106fd0 0x7f0fc01a3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fbeffd640 1 -- 172.21.15.23:0/338535437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb0009490 con 0x7f0fc0100d40 2024-03-24T10:50:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fbeffd640 1 --2- 172.21.15.23:0/338535437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fc0100d40 0x7f0fc0198b90 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009780 tx=0x7f0fb00346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fbcff9640 1 -- 172.21.15.23:0/338535437 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb002c7a0 con 0x7f0fc0100d40 2024-03-24T10:50:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fc53c7640 1 -- 172.21.15.23:0/338535437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc01a44e0 con 0x7f0fc0100d40 2024-03-24T10:50:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.196+0000 7f0fbcff9640 1 -- 172.21.15.23:0/338535437 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fb00044d0 con 0x7f0fc0100d40 2024-03-24T10:50:30.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.197+0000 7f0fbcff9640 1 -- 172.21.15.23:0/338535437 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb003d3f0 con 0x7f0fc0100d40 2024-03-24T10:50:30.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.197+0000 7f0fc53c7640 1 -- 172.21.15.23:0/338535437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc01a4990 con 0x7f0fc0100d40 2024-03-24T10:50:30.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.198+0000 7f0fc53c7640 1 -- 172.21.15.23:0/338535437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fc0068500 con 0x7f0fc0100d40 2024-03-24T10:50:30.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.199+0000 7f0fbcff9640 1 -- 172.21.15.23:0/338535437 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f0fb003b040 con 0x7f0fc0100d40 2024-03-24T10:50:30.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.199+0000 7f0fbcff9640 1 --2- 172.21.15.23:0/338535437 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0fac0a2230 0x7f0fac0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.199+0000 7f0fb7fff640 1 --2- 172.21.15.23:0/338535437 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0fac0a2230 0x7f0fac0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.199+0000 7f0fbcff9640 1 -- 172.21.15.23:0/338535437 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f0fb00efc90 con 0x7f0fc0100d40 2024-03-24T10:50:30.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.200+0000 7f0fb7fff640 1 --2- 172.21.15.23:0/338535437 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0fac0a2230 0x7f0fac0a46f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0069420 tx=0x7f0fa8009260 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:30.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.204+0000 7f0fbcff9640 1 -- 172.21.15.23:0/338535437 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0fb00b8ab0 con 0x7f0fc0100d40 2024-03-24T10:50:30.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.387+0000 7f0fc53c7640 1 -- 172.21.15.23:0/338535437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0fc0108b80 con 0x7f0fc0100d40 2024-03-24T10:50:30.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.388+0000 7f0fbcff9640 1 -- 172.21.15.23:0/338535437 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+23169 (secure 0 0 0) 0x7f0fb00bc170 con 0x7f0fc0100d40 2024-03-24T10:50:30.389 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:50:30.389 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":98,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:50:16.243119+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":97,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2350021632":"2024-03-25T10:40:05.291912+0000","172.21.15.23:6800/2350021632":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3614005933":"2024-03-25T10:40:05.291912+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/256452041":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/352422381":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/725846898":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:50:30.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.390+0000 7f0fc53c7640 1 -- 172.21.15.23:0/338535437 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0fac0a2230 msgr2=0x7f0fac0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.390+0000 7f0fc53c7640 1 --2- 172.21.15.23:0/338535437 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0fac0a2230 0x7f0fac0a46f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0069420 tx=0x7f0fa8009260 comp rx=0 tx=0).stop 2024-03-24T10:50:30.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.390+0000 7f0fc53c7640 1 -- 172.21.15.23:0/338535437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fc0100d40 msgr2=0x7f0fc0198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.390+0000 7f0fc53c7640 1 --2- 172.21.15.23:0/338535437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fc0100d40 0x7f0fc0198b90 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009780 tx=0x7f0fb00346f0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.390+0000 7f0fc53c7640 1 -- 172.21.15.23:0/338535437 shutdown_connections 2024-03-24T10:50:30.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.390+0000 7f0fc53c7640 1 --2- 172.21.15.23:0/338535437 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f0fac0a2230 0x7f0fac0a46f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.390+0000 7f0fc53c7640 1 --2- 172.21.15.23:0/338535437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0106fd0 0x7f0fc01a3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.390+0000 7f0fc53c7640 1 --2- 172.21.15.23:0/338535437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fc01066b0 0x7f0fc01990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.390+0000 7f0fc53c7640 1 --2- 172.21.15.23:0/338535437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fc0100d40 0x7f0fc0198b90 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.390+0000 7f0fc53c7640 1 -- 172.21.15.23:0/338535437 >> 172.21.15.23:0/338535437 conn(0x7f0fc0077710 msgr2=0x7f0fc00ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:30.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.390+0000 7f0fc53c7640 1 -- 172.21.15.23:0/338535437 shutdown_connections 2024-03-24T10:50:30.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.390+0000 7f0fc53c7640 1 -- 172.21.15.23:0/338535437 wait complete. 2024-03-24T10:50:30.402 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:39.056452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:38:08.473472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:38:09.414667+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 97, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:50:30.402 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:50:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.507+0000 7f21f0ba1640 1 -- 172.21.15.23:0/2826630856 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21ec10bc70 msgr2=0x7f21ec10e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.507+0000 7f21f0ba1640 1 --2- 172.21.15.23:0/2826630856 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21ec10bc70 0x7f21ec10e110 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009ca0 tx=0x7f21dc02c180 comp rx=0 tx=0).stop 2024-03-24T10:50:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.508+0000 7f21f0ba1640 1 -- 172.21.15.23:0/2826630856 shutdown_connections 2024-03-24T10:50:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.508+0000 7f21f0ba1640 1 --2- 172.21.15.23:0/2826630856 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21ec10bc70 0x7f21ec10e110 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.508+0000 7f21f0ba1640 1 --2- 172.21.15.23:0/2826630856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec103250 0x7f21ec10b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.508+0000 7f21f0ba1640 1 --2- 172.21.15.23:0/2826630856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21ec102930 0x7f21ec102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.508+0000 7f21f0ba1640 1 -- 172.21.15.23:0/2826630856 >> 172.21.15.23:0/2826630856 conn(0x7f21ec0777c0 msgr2=0x7f21ec077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.508+0000 7f21f0ba1640 1 -- 172.21.15.23:0/2826630856 shutdown_connections 2024-03-24T10:50:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.508+0000 7f21f0ba1640 1 -- 172.21.15.23:0/2826630856 wait complete. 2024-03-24T10:50:30.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.508+0000 7f21f0ba1640 1 Processor -- start 2024-03-24T10:50:30.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.508+0000 7f21f0ba1640 1 -- start start 2024-03-24T10:50:30.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.508+0000 7f21f0ba1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec102930 0x7f21ec19d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.508+0000 7f21f0ba1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21ec103250 0x7f21ec19ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.508+0000 7f21f0ba1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21ec10bc70 0x7f21ec197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.508+0000 7f21f0ba1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f21ec1980e0 con 0x7f21ec10bc70 2024-03-24T10:50:30.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21f0ba1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f21ec198250 con 0x7f21ec102930 2024-03-24T10:50:30.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21f0ba1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f21ec198410 con 0x7f21ec103250 2024-03-24T10:50:30.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21ead9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21ec10bc70 0x7f21ec197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21ea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec102930 0x7f21ec19d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21ea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec102930 0x7f21ec19d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.23:34752/0 (socket says 172.21.15.23:34752) 2024-03-24T10:50:30.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21ead9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21ec10bc70 0x7f21ec197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41386/0 (socket says 172.21.15.23:41386) 2024-03-24T10:50:30.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21ea59c640 1 -- 172.21.15.23:0/205789192 learned_addr learned my addr 172.21.15.23:0/205789192 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:30.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21e9d9b640 1 --2- 172.21.15.23:0/205789192 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21ec103250 0x7f21ec19ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21ea59c640 1 -- 172.21.15.23:0/205789192 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21ec103250 msgr2=0x7f21ec19ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21ea59c640 1 --2- 172.21.15.23:0/205789192 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21ec103250 0x7f21ec19ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21ea59c640 1 -- 172.21.15.23:0/205789192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21ec10bc70 msgr2=0x7f21ec197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21ea59c640 1 --2- 172.21.15.23:0/205789192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21ec10bc70 0x7f21ec197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21ea59c640 1 -- 172.21.15.23:0/205789192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d4009490 con 0x7f21ec102930 2024-03-24T10:50:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21ead9d640 1 --2- 172.21.15.23:0/205789192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21ec10bc70 0x7f21ec197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:50:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21ea59c640 1 --2- 172.21.15.23:0/205789192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec102930 0x7f21ec19d890 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f21e400e9f0 tx=0x7f21e400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21e37fe640 1 -- 172.21.15.23:0/205789192 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009700 con 0x7f21ec102930 2024-03-24T10:50:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21f0ba1640 1 -- 172.21.15.23:0/205789192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21dc009900 con 0x7f21ec102930 2024-03-24T10:50:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21e37fe640 1 -- 172.21.15.23:0/205789192 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21e400cd80 con 0x7f21ec102930 2024-03-24T10:50:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21e37fe640 1 -- 172.21.15.23:0/205789192 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4018920 con 0x7f21ec102930 2024-03-24T10:50:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.509+0000 7f21f0ba1640 1 -- 172.21.15.23:0/205789192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21ec1988e0 con 0x7f21ec102930 2024-03-24T10:50:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.510+0000 7f21f0ba1640 1 -- 172.21.15.23:0/205789192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21ec069080 con 0x7f21ec102930 2024-03-24T10:50:30.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.511+0000 7f21e37fe640 1 -- 172.21.15.23:0/205789192 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f21e4018a80 con 0x7f21ec102930 2024-03-24T10:50:30.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.512+0000 7f21e37fe640 1 --2- 172.21.15.23:0/205789192 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f21d80a2230 0x7f21d80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.512+0000 7f21e9d9b640 1 --2- 172.21.15.23:0/205789192 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f21d80a2230 0x7f21d80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.512+0000 7f21e37fe640 1 -- 172.21.15.23:0/205789192 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f21e4014070 con 0x7f21ec102930 2024-03-24T10:50:30.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.512+0000 7f21e9d9b640 1 --2- 172.21.15.23:0/205789192 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f21d80a2230 0x7f21d80a46f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f21ec0fff90 tx=0x7f21d4005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:30.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.517+0000 7f21e37fe640 1 -- 172.21.15.23:0/205789192 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f21e4053280 con 0x7f21ec102930 2024-03-24T10:50:30.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.699+0000 7f21f0ba1640 1 -- 172.21.15.23:0/205789192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f21ec068500 con 0x7f21ec102930 2024-03-24T10:50:30.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.699+0000 7f21e37fe640 1 -- 172.21.15.23:0/205789192 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) ==== 93+0+10 (secure 0 0 0) 0x7f21e4097180 con 0x7f21ec102930 2024-03-24T10:50:30.699 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-24T10:50:30.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.701+0000 7f21f0ba1640 1 -- 172.21.15.23:0/205789192 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f21d80a2230 msgr2=0x7f21d80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.701+0000 7f21f0ba1640 1 --2- 172.21.15.23:0/205789192 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f21d80a2230 0x7f21d80a46f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f21ec0fff90 tx=0x7f21d4005f90 comp rx=0 tx=0).stop 2024-03-24T10:50:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.701+0000 7f21f0ba1640 1 -- 172.21.15.23:0/205789192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec102930 msgr2=0x7f21ec19d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.701+0000 7f21f0ba1640 1 --2- 172.21.15.23:0/205789192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec102930 0x7f21ec19d890 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f21e400e9f0 tx=0x7f21e400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.701+0000 7f21f0ba1640 1 -- 172.21.15.23:0/205789192 shutdown_connections 2024-03-24T10:50:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.701+0000 7f21f0ba1640 1 --2- 172.21.15.23:0/205789192 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f21d80a2230 0x7f21d80a46f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.701+0000 7f21f0ba1640 1 --2- 172.21.15.23:0/205789192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21ec10bc70 0x7f21ec197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.701+0000 7f21f0ba1640 1 --2- 172.21.15.23:0/205789192 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21ec103250 0x7f21ec19ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.701+0000 7f21f0ba1640 1 --2- 172.21.15.23:0/205789192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec102930 0x7f21ec19d890 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.701+0000 7f21f0ba1640 1 -- 172.21.15.23:0/205789192 >> 172.21.15.23:0/205789192 conn(0x7f21ec0777c0 msgr2=0x7f21ec0fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:30.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.701+0000 7f21f0ba1640 1 -- 172.21.15.23:0/205789192 shutdown_connections 2024-03-24T10:50:30.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.701+0000 7f21f0ba1640 1 -- 172.21.15.23:0/205789192 wait complete. 2024-03-24T10:50:30.712 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:50:30.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.818+0000 7f65e7d5e640 1 -- 172.21.15.23:0/2848000739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e0100f60 msgr2=0x7f65e010d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.818+0000 7f65e7d5e640 1 --2- 172.21.15.23:0/2848000739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e0100f60 0x7f65e010d420 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f65d8009900 tx=0x7f65d802bc30 comp rx=0 tx=0).stop 2024-03-24T10:50:30.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.818+0000 7f65e7d5e640 1 -- 172.21.15.23:0/2848000739 shutdown_connections 2024-03-24T10:50:30.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.818+0000 7f65e7d5e640 1 --2- 172.21.15.23:0/2848000739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e0100f60 0x7f65e010d420 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.818+0000 7f65e7d5e640 1 --2- 172.21.15.23:0/2848000739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65e01005a0 0x7f65e0100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.818+0000 7f65e7d5e640 1 --2- 172.21.15.23:0/2848000739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e0106660 0x7f65e0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.818+0000 7f65e7d5e640 1 -- 172.21.15.23:0/2848000739 >> 172.21.15.23:0/2848000739 conn(0x7f65e0077710 msgr2=0x7f65e0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:30.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.819+0000 7f65e7d5e640 1 -- 172.21.15.23:0/2848000739 shutdown_connections 2024-03-24T10:50:30.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.819+0000 7f65e7d5e640 1 -- 172.21.15.23:0/2848000739 wait complete. 2024-03-24T10:50:30.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.819+0000 7f65e7d5e640 1 Processor -- start 2024-03-24T10:50:30.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.819+0000 7f65e7d5e640 1 -- start start 2024-03-24T10:50:30.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.819+0000 7f65e7d5e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65e01005a0 0x7f65e019d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.819+0000 7f65e7d5e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e0100f60 0x7f65e019ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.819+0000 7f65e7d5e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e0106660 0x7f65e0197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.819+0000 7f65e7d5e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f65e0198080 con 0x7f65e0106660 2024-03-24T10:50:30.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.819+0000 7f65e7d5e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f65e01981f0 con 0x7f65e0100f60 2024-03-24T10:50:30.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.819+0000 7f65e7d5e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f65e01983b0 con 0x7f65e01005a0 2024-03-24T10:50:30.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.819+0000 7f65e62fb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e0106660 0x7f65e0197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.819+0000 7f65e62fb640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e0106660 0x7f65e0197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41408/0 (socket says 172.21.15.23:41408) 2024-03-24T10:50:30.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.819+0000 7f65e62fb640 1 -- 172.21.15.23:0/3730118467 learned_addr learned my addr 172.21.15.23:0/3730118467 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:30.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.819+0000 7f65e52f9640 1 --2- 172.21.15.23:0/3730118467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e0100f60 0x7f65e019ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.820+0000 7f65e5afa640 1 --2- 172.21.15.23:0/3730118467 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65e01005a0 0x7f65e019d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.820+0000 7f65e62fb640 1 -- 172.21.15.23:0/3730118467 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65e01005a0 msgr2=0x7f65e019d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.820+0000 7f65e62fb640 1 --2- 172.21.15.23:0/3730118467 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65e01005a0 0x7f65e019d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.820+0000 7f65e62fb640 1 -- 172.21.15.23:0/3730118467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e0100f60 msgr2=0x7f65e019ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.820+0000 7f65e62fb640 1 --2- 172.21.15.23:0/3730118467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e0100f60 0x7f65e019ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.820+0000 7f65e62fb640 1 -- 172.21.15.23:0/3730118467 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d0009490 con 0x7f65e0106660 2024-03-24T10:50:30.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.820+0000 7f65e62fb640 1 --2- 172.21.15.23:0/3730118467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e0106660 0x7f65e0197a30 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f65d8009230 tx=0x7f65d8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:30.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.820+0000 7f65d6ffd640 1 -- 172.21.15.23:0/3730118467 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d80042d0 con 0x7f65e0106660 2024-03-24T10:50:30.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.820+0000 7f65e7d5e640 1 -- 172.21.15.23:0/3730118467 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65d8009560 con 0x7f65e0106660 2024-03-24T10:50:30.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.820+0000 7f65d6ffd640 1 -- 172.21.15.23:0/3730118467 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65d8004430 con 0x7f65e0106660 2024-03-24T10:50:30.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.820+0000 7f65d6ffd640 1 -- 172.21.15.23:0/3730118467 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d8003b50 con 0x7f65e0106660 2024-03-24T10:50:30.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.820+0000 7f65e7d5e640 1 -- 172.21.15.23:0/3730118467 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e01a4740 con 0x7f65e0106660 2024-03-24T10:50:30.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.821+0000 7f65e7d5e640 1 -- 172.21.15.23:0/3730118467 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65e0101cc0 con 0x7f65e0106660 2024-03-24T10:50:30.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.822+0000 7f65d6ffd640 1 -- 172.21.15.23:0/3730118467 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f65d8005460 con 0x7f65e0106660 2024-03-24T10:50:30.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.823+0000 7f65d6ffd640 1 --2- 172.21.15.23:0/3730118467 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f65cc0a2230 0x7f65cc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.823+0000 7f65e5afa640 1 --2- 172.21.15.23:0/3730118467 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f65cc0a2230 0x7f65cc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.823+0000 7f65d6ffd640 1 -- 172.21.15.23:0/3730118467 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f65d80f0820 con 0x7f65e0106660 2024-03-24T10:50:30.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.823+0000 7f65e5afa640 1 --2- 172.21.15.23:0/3730118467 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f65cc0a2230 0x7f65cc0a46f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f65e0198b70 tx=0x7f65dc009500 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:30.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:30.826+0000 7f65d6ffd640 1 -- 172.21.15.23:0/3730118467 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f65d80b9530 con 0x7f65e0106660 2024-03-24T10:50:31.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.009+0000 7f65e7d5e640 1 -- 172.21.15.23:0/3730118467 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f65e0068500 con 0x7f65e0106660 2024-03-24T10:50:31.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.010+0000 7f65d6ffd640 1 -- 172.21.15.23:0/3730118467 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) ==== 104+0+11 (secure 0 0 0) 0x7f65d80bcbf0 con 0x7f65e0106660 2024-03-24T10:50:31.010 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-24T10:50:31.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.012+0000 7f65e7d5e640 1 -- 172.21.15.23:0/3730118467 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f65cc0a2230 msgr2=0x7f65cc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:31.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.012+0000 7f65e7d5e640 1 --2- 172.21.15.23:0/3730118467 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f65cc0a2230 0x7f65cc0a46f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f65e0198b70 tx=0x7f65dc009500 comp rx=0 tx=0).stop 2024-03-24T10:50:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.012+0000 7f65e7d5e640 1 -- 172.21.15.23:0/3730118467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e0106660 msgr2=0x7f65e0197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.012+0000 7f65e7d5e640 1 --2- 172.21.15.23:0/3730118467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e0106660 0x7f65e0197a30 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f65d8009230 tx=0x7f65d8004ab0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.012+0000 7f65e7d5e640 1 -- 172.21.15.23:0/3730118467 shutdown_connections 2024-03-24T10:50:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.012+0000 7f65e7d5e640 1 --2- 172.21.15.23:0/3730118467 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f65cc0a2230 0x7f65cc0a46f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.012+0000 7f65e7d5e640 1 --2- 172.21.15.23:0/3730118467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e0106660 0x7f65e0197a30 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.012+0000 7f65e7d5e640 1 --2- 172.21.15.23:0/3730118467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65e0100f60 0x7f65e019ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.012+0000 7f65e7d5e640 1 --2- 172.21.15.23:0/3730118467 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65e01005a0 0x7f65e019d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.012+0000 7f65e7d5e640 1 -- 172.21.15.23:0/3730118467 >> 172.21.15.23:0/3730118467 conn(0x7f65e0077710 msgr2=0x7f65e00ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:31.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.012+0000 7f65e7d5e640 1 -- 172.21.15.23:0/3730118467 shutdown_connections 2024-03-24T10:50:31.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.012+0000 7f65e7d5e640 1 -- 172.21.15.23:0/3730118467 wait complete. 2024-03-24T10:50:31.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:30 smithi023 ceph-mon[32466]: pgmap v325: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:31.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:30 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/338535437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:50:31.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:30 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/205789192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:50:31.023 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:50:31.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.129+0000 7fc6f4433640 1 -- 172.21.15.23:0/1609107781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6ec107690 msgr2=0x7fc6ec107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:31.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.129+0000 7fc6f4433640 1 --2- 172.21.15.23:0/1609107781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6ec107690 0x7fc6ec107a70 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8009930 tx=0x7fc6e802bc30 comp rx=0 tx=0).stop 2024-03-24T10:50:31.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.129+0000 7fc6f4433640 1 -- 172.21.15.23:0/1609107781 shutdown_connections 2024-03-24T10:50:31.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.129+0000 7fc6f4433640 1 --2- 172.21.15.23:0/1609107781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ec102040 0x7fc6ec10e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.129+0000 7fc6f4433640 1 --2- 172.21.15.23:0/1609107781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6ec101680 0x7fc6ec101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.129+0000 7fc6f4433640 1 --2- 172.21.15.23:0/1609107781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6ec107690 0x7fc6ec107a70 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.129+0000 7fc6f4433640 1 -- 172.21.15.23:0/1609107781 >> 172.21.15.23:0/1609107781 conn(0x7fc6ec0fd1b0 msgr2=0x7fc6ec0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:31.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.129+0000 7fc6f4433640 1 -- 172.21.15.23:0/1609107781 shutdown_connections 2024-03-24T10:50:31.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.129+0000 7fc6f4433640 1 -- 172.21.15.23:0/1609107781 wait complete. 2024-03-24T10:50:31.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f4433640 1 Processor -- start 2024-03-24T10:50:31.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f4433640 1 -- start start 2024-03-24T10:50:31.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f4433640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6ec101680 0x7fc6ec197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:31.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f4433640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ec102040 0x7fc6ec1982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:31.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f4433640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6ec107690 0x7fc6ec19c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:31.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f4433640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec19cc40 con 0x7fc6ec101680 2024-03-24T10:50:31.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f4433640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec19cdb0 con 0x7fc6ec102040 2024-03-24T10:50:31.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f4433640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec19cf70 con 0x7fc6ec107690 2024-03-24T10:50:31.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f21cf640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6ec101680 0x7fc6ec197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:31.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f21cf640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6ec101680 0x7fc6ec197d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41416/0 (socket says 172.21.15.23:41416) 2024-03-24T10:50:31.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f21cf640 1 -- 172.21.15.23:0/2397152945 learned_addr learned my addr 172.21.15.23:0/2397152945 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:31.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f19ce640 1 --2- 172.21.15.23:0/2397152945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ec102040 0x7fc6ec1982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:31.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f29d0640 1 --2- 172.21.15.23:0/2397152945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6ec107690 0x7fc6ec19c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:31.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f21cf640 1 -- 172.21.15.23:0/2397152945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6ec107690 msgr2=0x7fc6ec19c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:31.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f21cf640 1 --2- 172.21.15.23:0/2397152945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6ec107690 0x7fc6ec19c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f21cf640 1 -- 172.21.15.23:0/2397152945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ec102040 msgr2=0x7fc6ec1982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:31.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f21cf640 1 --2- 172.21.15.23:0/2397152945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ec102040 0x7fc6ec1982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f21cf640 1 -- 172.21.15.23:0/2397152945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e8009560 con 0x7fc6ec101680 2024-03-24T10:50:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.130+0000 7fc6f21cf640 1 --2- 172.21.15.23:0/2397152945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6ec101680 0x7fc6ec197d80 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8009230 tx=0x7fc6e802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.131+0000 7fc6e37fe640 1 -- 172.21.15.23:0/2397152945 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e8004350 con 0x7fc6ec101680 2024-03-24T10:50:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.131+0000 7fc6e37fe640 1 -- 172.21.15.23:0/2397152945 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6e80044b0 con 0x7fc6ec101680 2024-03-24T10:50:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.131+0000 7fc6f4433640 1 -- 172.21.15.23:0/2397152945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ec19d1f0 con 0x7fc6ec101680 2024-03-24T10:50:31.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.131+0000 7fc6e37fe640 1 -- 172.21.15.23:0/2397152945 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e8003d00 con 0x7fc6ec101680 2024-03-24T10:50:31.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.131+0000 7fc6f4433640 1 -- 172.21.15.23:0/2397152945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ec1a4ad0 con 0x7fc6ec101680 2024-03-24T10:50:31.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.132+0000 7fc6f4433640 1 -- 172.21.15.23:0/2397152945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6ec068950 con 0x7fc6ec101680 2024-03-24T10:50:31.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.133+0000 7fc6e37fe640 1 -- 172.21.15.23:0/2397152945 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fc6e8004df0 con 0x7fc6ec101680 2024-03-24T10:50:31.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.133+0000 7fc6e37fe640 1 --2- 172.21.15.23:0/2397152945 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc6d80a2230 0x7fc6d80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:31.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.133+0000 7fc6f19ce640 1 --2- 172.21.15.23:0/2397152945 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc6d80a2230 0x7fc6d80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:31.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.133+0000 7fc6e37fe640 1 -- 172.21.15.23:0/2397152945 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fc6e80f06f0 con 0x7fc6ec101680 2024-03-24T10:50:31.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.134+0000 7fc6f19ce640 1 --2- 172.21.15.23:0/2397152945 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc6d80a2230 0x7fc6d80a46f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec0ff890 tx=0x7fc6dc005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:31.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.137+0000 7fc6e37fe640 1 -- 172.21.15.23:0/2397152945 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc6e80b9460 con 0x7fc6ec101680 2024-03-24T10:50:31.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:30 smithi117 ceph-mon[36368]: pgmap v325: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:31.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:30 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/338535437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:50:31.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:30 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/205789192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:50:31.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:30 smithi116 ceph-mon[36248]: pgmap v325: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:31.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:30 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/338535437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:50:31.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:30 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/205789192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:50:31.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.318+0000 7fc6f4433640 1 -- 172.21.15.23:0/2397152945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc6ec109bf0 con 0x7fc6ec101680 2024-03-24T10:50:31.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.319+0000 7fc6e37fe640 1 -- 172.21.15.23:0/2397152945 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) ==== 100+0+11 (secure 0 0 0) 0x7fc6e8003e60 con 0x7fc6ec101680 2024-03-24T10:50:31.319 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-24T10:50:31.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.320+0000 7fc6f4433640 1 -- 172.21.15.23:0/2397152945 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc6d80a2230 msgr2=0x7fc6d80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:31.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.320+0000 7fc6f4433640 1 --2- 172.21.15.23:0/2397152945 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc6d80a2230 0x7fc6d80a46f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec0ff890 tx=0x7fc6dc005f90 comp rx=0 tx=0).stop 2024-03-24T10:50:31.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.320+0000 7fc6f4433640 1 -- 172.21.15.23:0/2397152945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6ec101680 msgr2=0x7fc6ec197d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:31.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.320+0000 7fc6f4433640 1 --2- 172.21.15.23:0/2397152945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6ec101680 0x7fc6ec197d80 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8009230 tx=0x7fc6e802c650 comp rx=0 tx=0).stop 2024-03-24T10:50:31.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.320+0000 7fc6f4433640 1 -- 172.21.15.23:0/2397152945 shutdown_connections 2024-03-24T10:50:31.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.320+0000 7fc6f4433640 1 --2- 172.21.15.23:0/2397152945 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fc6d80a2230 0x7fc6d80a46f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.320+0000 7fc6f4433640 1 --2- 172.21.15.23:0/2397152945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6ec107690 0x7fc6ec19c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.320+0000 7fc6f4433640 1 --2- 172.21.15.23:0/2397152945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6ec102040 0x7fc6ec1982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.320+0000 7fc6f4433640 1 --2- 172.21.15.23:0/2397152945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6ec101680 0x7fc6ec197d80 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.321+0000 7fc6f4433640 1 -- 172.21.15.23:0/2397152945 >> 172.21.15.23:0/2397152945 conn(0x7fc6ec0fd1b0 msgr2=0x7fc6ec0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.321+0000 7fc6f4433640 1 -- 172.21.15.23:0/2397152945 shutdown_connections 2024-03-24T10:50:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.321+0000 7fc6f4433640 1 -- 172.21.15.23:0/2397152945 wait complete. 2024-03-24T10:50:31.331 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-24T10:50:31.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.434+0000 7f196089d640 1 -- 172.21.15.23:0/1505243954 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f195c100f80 msgr2=0x7f195c10d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:31.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.434+0000 7f196089d640 1 --2- 172.21.15.23:0/1505243954 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f195c100f80 0x7f195c10d440 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1954009c10 tx=0x7f195402bef0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.434+0000 7f196089d640 1 -- 172.21.15.23:0/1505243954 shutdown_connections 2024-03-24T10:50:31.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.434+0000 7f196089d640 1 --2- 172.21.15.23:0/1505243954 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f195c100f80 0x7f195c10d440 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.434+0000 7f196089d640 1 --2- 172.21.15.23:0/1505243954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c1005c0 0x7f195c100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.434+0000 7f196089d640 1 --2- 172.21.15.23:0/1505243954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f195c106680 0x7f195c106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.434+0000 7f196089d640 1 -- 172.21.15.23:0/1505243954 >> 172.21.15.23:0/1505243954 conn(0x7f195c0776e0 msgr2=0x7f195c077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:31.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.434+0000 7f196089d640 1 -- 172.21.15.23:0/1505243954 shutdown_connections 2024-03-24T10:50:31.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.434+0000 7f196089d640 1 -- 172.21.15.23:0/1505243954 wait complete. 2024-03-24T10:50:31.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.435+0000 7f196089d640 1 Processor -- start 2024-03-24T10:50:31.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.435+0000 7f196089d640 1 -- start start 2024-03-24T10:50:31.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.435+0000 7f196089d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f195c1005c0 0x7f195c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:31.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.435+0000 7f196089d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c100f80 0x7f195c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:31.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.435+0000 7f196089d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f195c106680 0x7f195c196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:31.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.435+0000 7f196089d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f195c197070 con 0x7f195c106680 2024-03-24T10:50:31.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.435+0000 7f196089d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1971e0 con 0x7f195c100f80 2024-03-24T10:50:31.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.435+0000 7f196089d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1973a0 con 0x7f195c1005c0 2024-03-24T10:50:31.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.435+0000 7f195ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f195c106680 0x7f195c196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:31.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.435+0000 7f195ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f195c106680 0x7f195c196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41434/0 (socket says 172.21.15.23:41434) 2024-03-24T10:50:31.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.435+0000 7f195ad9d640 1 -- 172.21.15.23:0/1191979107 learned_addr learned my addr 172.21.15.23:0/1191979107 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:50:31.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.435+0000 7f1953fff640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c100f80 0x7f195c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:31.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.435+0000 7f195a59c640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f195c1005c0 0x7f195c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:31.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.436+0000 7f195ad9d640 1 -- 172.21.15.23:0/1191979107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f195c1005c0 msgr2=0x7f195c19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:31.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.436+0000 7f195ad9d640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f195c1005c0 0x7f195c19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.436+0000 7f195ad9d640 1 -- 172.21.15.23:0/1191979107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c100f80 msgr2=0x7f195c19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:31.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.436+0000 7f195ad9d640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c100f80 0x7f195c19cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.436+0000 7f195ad9d640 1 -- 172.21.15.23:0/1191979107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194c009560 con 0x7f195c106680 2024-03-24T10:50:31.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.436+0000 7f1953fff640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c100f80 0x7f195c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:50:31.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.436+0000 7f195a59c640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f195c1005c0 0x7f195c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:50:31.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.436+0000 7f195ad9d640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f195c106680 0x7f195c196a20 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f1954036040 tx=0x7f195402cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:31.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.436+0000 7f19537fe640 1 -- 172.21.15.23:0/1191979107 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954003970 con 0x7f195c106680 2024-03-24T10:50:31.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.436+0000 7f196089d640 1 -- 172.21.15.23:0/1191979107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1954009840 con 0x7f195c106680 2024-03-24T10:50:31.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.436+0000 7f19537fe640 1 -- 172.21.15.23:0/1191979107 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1954004040 con 0x7f195c106680 2024-03-24T10:50:31.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.436+0000 7f19537fe640 1 -- 172.21.15.23:0/1191979107 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195403d790 con 0x7f195c106680 2024-03-24T10:50:31.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.436+0000 7f196089d640 1 -- 172.21.15.23:0/1191979107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f195c1a3730 con 0x7f195c106680 2024-03-24T10:50:31.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.437+0000 7f196089d640 1 -- 172.21.15.23:0/1191979107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f195c068950 con 0x7f195c106680 2024-03-24T10:50:31.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.439+0000 7f19537fe640 1 -- 172.21.15.23:0/1191979107 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f1954004b40 con 0x7f195c106680 2024-03-24T10:50:31.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.439+0000 7f19537fe640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f19480a2230 0x7f19480a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:31.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.439+0000 7f195a59c640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f19480a2230 0x7f19480a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:31.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.439+0000 7f19537fe640 1 -- 172.21.15.23:0/1191979107 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f19540f04f0 con 0x7f195c106680 2024-03-24T10:50:31.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.440+0000 7f195a59c640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f19480a2230 0x7f19480a46f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f195c197b60 tx=0x7f194c009260 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:31.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.443+0000 7f19537fe640 1 -- 172.21.15.23:0/1191979107 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f19540b92b0 con 0x7f195c106680 2024-03-24T10:50:31.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.675+0000 7f196089d640 1 -- 172.21.15.23:0/1191979107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f195c1106c0 con 0x7f195c106680 2024-03-24T10:50:31.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.683+0000 7f19537fe640 1 -- 172.21.15.23:0/1191979107 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+736926 (secure 0 0 0) 0x7f1954041090 con 0x7f195c106680 2024-03-24T10:50:31.692 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:50:31.693 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:50:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 25, 2024-03-24T10:50:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: "active_gid": 15288, 2024-03-24T10:50:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: "active_name": "x", 2024-03-24T10:50:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: "active_addrs": { 2024-03-24T10:50:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-24T10:50:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-24T10:50:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6800", 2024-03-24T10:50:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 581236369 2024-03-24T10:50:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-24T10:50:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6801", 2024-03-24T10:50:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 581236369 2024-03-24T10:50:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-24T10:50:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: "active_addr": "172.21.15.23:6801/581236369", 2024-03-24T10:50:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: "active_change": "2024-03-24T10:40:05.292141+0000", 2024-03-24T10:50:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: "active_mgr_features": 4540701547738038271, 2024-03-24T10:50:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: "available": true, 2024-03-24T10:50:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: "standbys": [ 2024-03-24T10:50:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "gid": 15282, 2024-03-24T10:50:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "y", 2024-03-24T10:50:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_features": 4540701547738038271, 2024-03-24T10:50:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-24T10:50:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-24T10:50:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T10:50:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-24T10:50:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-24T10:50:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-24T10:50:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-24T10:50:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-24T10:50:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-24T10:50:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-24T10:50:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-24T10:50:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-24T10:50:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-24T10:50:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-24T10:50:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-24T10:50:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-24T10:50:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-24T10:50:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-24T10:50:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-24T10:50:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.713 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.713 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.713 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-24T10:50:31.713 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.713 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.713 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-24T10:50:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-24T10:50:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-24T10:50:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-24T10:50:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-24T10:50:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T10:50:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-24T10:50:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-24T10:50:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-24T10:50:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-24T10:50:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-24T10:50:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-24T10:50:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T10:50:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-24T10:50:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-24T10:50:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-24T10:50:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T10:50:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T10:50:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-24T10:50:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-24T10:50:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-24T10:50:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-24T10:50:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T10:50:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T10:50:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-24T10:50:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-24T10:50:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-24T10:50:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-24T10:50:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-24T10:50:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T10:50:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-24T10:50:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-24T10:50:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T10:50:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T10:50:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T10:50:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-24T10:50:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-24T10:50:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-24T10:50:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-24T10:50:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-24T10:50:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-24T10:50:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T10:50:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-24T10:50:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-24T10:50:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-24T10:50:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T10:50:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T10:50:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-24T10:50:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-24T10:50:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-24T10:50:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-24T10:50:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T10:50:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T10:50:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:31.738 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:31.738 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:31.738 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.738 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.738 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.738 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.741 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.741 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-24T10:50:31.741 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-24T10:50:31.741 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:31.741 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.741 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T10:50:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-24T10:50:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-24T10:50:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.744 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.744 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-24T10:50:31.744 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.744 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.744 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-24T10:50:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-24T10:50:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: "read", 2024-03-24T10:50:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap", 2024-03-24T10:50:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap-read" 2024-03-24T10:50:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-24T10:50:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-24T10:50:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-24T10:50:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T10:50:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T10:50:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-24T10:50:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-24T10:50:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-24T10:50:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-24T10:50:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T10:50:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T10:50:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-24T10:50:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-24T10:50:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T10:50:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-24T10:50:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-24T10:50:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-24T10:50:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-24T10:50:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T10:50:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-24T10:50:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-24T10:50:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-24T10:50:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.758 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.758 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T10:50:31.758 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.758 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-24T10:50:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-24T10:50:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-24T10:50:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T10:50:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-24T10:50:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-24T10:50:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T10:50:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T10:50:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-24T10:50:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-24T10:50:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.765 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T10:50:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-24T10:50:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T10:50:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-24T10:50:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T10:50:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm_log_destination": { 2024-03-24T10:50:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm_log_destination", 2024-03-24T10:50:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "file", 2024-03-24T10:50:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "file,syslog", 2024-03-24T10:50:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: "syslog" 2024-03-24T10:50:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T10:50:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-24T10:50:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-24T10:50:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T10:50:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-24T10:50:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-24T10:50:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T10:50:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-24T10:50:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-24T10:50:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T10:50:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-24T10:50:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-24T10:50:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T10:50:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-24T10:50:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-24T10:50:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-24T10:50:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T10:50:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-24T10:50:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-24T10:50:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-24T10:50:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T10:50:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-24T10:50:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-24T10:50:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-24T10:50:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T10:50:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-24T10:50:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-24T10:50:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-24T10:50:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T10:50:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-24T10:50:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-24T10:50:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-24T10:50:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T10:50:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-24T10:50:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-24T10:50:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-24T10:50:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T10:50:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-24T10:50:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-24T10:50:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-24T10:50:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T10:50:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-24T10:50:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-24T10:50:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-24T10:50:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T10:50:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-24T10:50:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-24T10:50:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-24T10:50:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T10:50:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-24T10:50:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-24T10:50:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-24T10:50:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T10:50:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-24T10:50:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-24T10:50:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-24T10:50:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-24T10:50:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-24T10:50:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-24T10:50:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-24T10:50:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T10:50:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-24T10:50:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-24T10:50:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-24T10:50:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T10:50:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-24T10:50:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-24T10:50:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-24T10:50:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T10:50:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-24T10:50:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-24T10:50:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-24T10:50:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T10:50:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-24T10:50:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-24T10:50:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T10:50:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-24T10:50:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T10:50:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-24T10:50:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T10:50:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-24T10:50:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-24T10:50:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-24T10:50:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T10:50:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-24T10:50:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-24T10:50:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-24T10:50:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-24T10:50:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-24T10:50:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-24T10:50:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T10:50:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-24T10:50:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-24T10:50:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-24T10:50:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-24T10:50:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-24T10:50:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-24T10:50:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-24T10:50:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-24T10:50:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T10:50:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-24T10:50:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-24T10:50:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T10:50:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-24T10:50:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-24T10:50:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T10:50:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T10:50:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T10:50:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T10:50:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T10:50:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T10:50:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T10:50:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-24T10:50:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T10:50:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-24T10:50:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-24T10:50:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-24T10:50:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-24T10:50:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-24T10:50:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T10:50:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-24T10:50:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-24T10:50:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-24T10:50:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-24T10:50:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-24T10:50:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-24T10:50:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-24T10:50:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-24T10:50:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T10:50:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-24T10:50:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-24T10:50:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-24T10:50:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T10:50:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-24T10:50:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-24T10:50:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T10:50:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T10:50:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-24T10:50:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-24T10:50:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T10:50:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-24T10:50:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-24T10:50:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T10:50:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-24T10:50:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-24T10:50:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T10:50:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-24T10:50:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-24T10:50:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T10:50:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-24T10:50:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-24T10:50:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T10:50:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-24T10:50:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-24T10:50:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-24T10:50:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-24T10:50:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-24T10:50:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-24T10:50:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T10:50:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-24T10:50:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-24T10:50:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T10:50:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-24T10:50:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-24T10:50:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T10:50:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-24T10:50:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-24T10:50:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T10:50:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-24T10:50:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-24T10:50:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T10:50:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-24T10:50:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-24T10:50:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.865 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.865 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.865 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.865 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.865 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T10:50:31.865 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.865 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.865 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-24T10:50:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-24T10:50:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-24T10:50:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-24T10:50:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T10:50:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-24T10:50:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-24T10:50:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-24T10:50:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T10:50:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-24T10:50:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T10:50:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T10:50:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.881 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.881 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.881 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.881 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.881 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.881 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T10:50:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T10:50:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T10:50:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T10:50:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-24T10:50:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T10:50:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T10:50:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T10:50:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T10:50:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T10:50:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T10:50:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T10:50:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T10:50:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T10:50:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T10:50:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T10:50:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T10:50:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T10:50:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T10:50:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T10:50:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T10:50:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T10:50:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T10:50:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T10:50:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T10:50:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T10:50:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T10:50:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T10:50:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-24T10:50:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T10:50:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T10:50:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-24T10:50:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-24T10:50:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T10:50:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T10:50:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-24T10:50:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T10:50:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T10:50:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T10:50:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T10:50:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T10:50:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T10:50:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T10:50:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T10:50:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T10:50:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T10:50:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T10:50:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T10:50:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T10:50:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T10:50:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T10:50:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T10:50:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T10:50:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T10:50:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T10:50:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T10:50:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T10:50:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T10:50:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T10:50:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T10:50:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T10:50:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T10:50:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T10:50:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T10:50:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T10:50:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T10:50:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T10:50:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T10:50:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T10:50:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T10:50:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T10:50:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-24T10:50:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T10:50:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T10:50:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-24T10:50:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T10:50:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T10:50:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T10:50:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T10:50:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-24T10:50:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T10:50:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T10:50:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T10:50:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T10:50:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T10:50:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T10:50:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T10:50:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T10:50:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T10:50:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T10:50:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-24T10:50:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-24T10:50:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-24T10:50:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-24T10:50:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-24T10:50:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-24T10:50:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-24T10:50:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-24T10:50:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-24T10:50:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-24T10:50:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-24T10:50:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-24T10:50:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-24T10:50:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-24T10:50:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-24T10:50:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-24T10:50:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-24T10:50:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T10:50:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-24T10:50:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-24T10:50:31.978 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.978 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.978 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.978 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-24T10:50:31.978 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.978 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.978 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-24T10:50:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-24T10:50:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-24T10:50:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-24T10:50:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-24T10:50:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-24T10:50:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-24T10:50:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-24T10:50:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-24T10:50:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-24T10:50:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-24T10:50:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-24T10:50:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-24T10:50:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-24T10:50:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-24T10:50:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-24T10:50:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-24T10:50:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-24T10:50:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-24T10:50:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-24T10:50:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-24T10:50:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-24T10:50:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-24T10:50:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.994 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:31.994 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:31.994 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:31.994 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:31.994 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:31.994 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-24T10:50:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-24T10:50:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.002 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.002 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.002 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-24T10:50:32.002 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.002 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.002 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.002 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T10:50:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-24T10:50:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-24T10:50:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-24T10:50:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T10:50:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-24T10:50:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-24T10:50:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-24T10:50:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-24T10:50:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-24T10:50:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-24T10:50:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-24T10:50:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T10:50:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-24T10:50:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-24T10:50:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T10:50:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T10:50:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-24T10:50:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-24T10:50:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-24T10:50:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T10:50:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-24T10:50:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-24T10:50:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-24T10:50:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-24T10:50:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-24T10:50:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-24T10:50:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-24T10:50:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-24T10:50:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-24T10:50:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-24T10:50:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-24T10:50:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-24T10:50:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-24T10:50:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T10:50:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-24T10:50:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-24T10:50:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-24T10:50:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T10:50:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-24T10:50:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-24T10:50:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-24T10:50:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-24T10:50:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-24T10:50:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T10:50:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-24T10:50:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-24T10:50:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-24T10:50:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-24T10:50:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-24T10:50:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-24T10:50:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-24T10:50:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-24T10:50:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-24T10:50:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-24T10:50:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T10:50:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-24T10:50:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-24T10:50:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-24T10:50:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-24T10:50:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-24T10:50:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T10:50:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-24T10:50:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-24T10:50:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-24T10:50:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-24T10:50:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-24T10:50:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-24T10:50:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T10:50:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-24T10:50:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-24T10:50:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-24T10:50:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-24T10:50:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-24T10:50:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-24T10:50:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T10:50:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-24T10:50:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-24T10:50:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-24T10:50:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T10:50:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-24T10:50:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.092 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.092 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-24T10:50:32.092 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-24T10:50:32.092 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.092 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.092 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-24T10:50:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-24T10:50:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T10:50:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.098 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.098 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.098 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.098 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.098 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.098 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-24T10:50:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-24T10:50:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-24T10:50:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T10:50:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-24T10:50:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-24T10:50:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-24T10:50:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-24T10:50:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-24T10:50:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-24T10:50:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-24T10:50:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T10:50:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-24T10:50:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-24T10:50:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-24T10:50:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-24T10:50:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-24T10:50:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-24T10:50:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T10:50:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T10:50:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-24T10:50:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-24T10:50:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-24T10:50:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-24T10:50:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-24T10:50:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-24T10:50:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T10:50:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-24T10:50:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-24T10:50:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-24T10:50:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-24T10:50:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-24T10:50:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-24T10:50:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-24T10:50:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-24T10:50:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-24T10:50:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.185 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.185 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:32.185 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:32.185 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.185 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.185 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-24T10:50:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-24T10:50:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-24T10:50:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-24T10:50:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-24T10:50:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T10:50:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-24T10:50:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-24T10:50:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-24T10:50:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T10:50:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-24T10:50:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-24T10:50:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-24T10:50:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-24T10:50:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-24T10:50:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-24T10:50:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-24T10:50:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-24T10:50:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-24T10:50:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-24T10:50:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-24T10:50:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-24T10:50:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-24T10:50:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T10:50:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T10:50:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-24T10:50:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-24T10:50:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T10:50:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T10:50:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-24T10:50:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-24T10:50:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-24T10:50:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-24T10:50:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-24T10:50:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-24T10:50:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-24T10:50:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T10:50:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-24T10:50:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-24T10:50:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-24T10:50:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T10:50:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-24T10:50:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-24T10:50:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-24T10:50:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-24T10:50:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-24T10:50:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-24T10:50:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-24T10:50:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-24T10:50:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-24T10:50:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-24T10:50:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-24T10:50:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-24T10:50:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-24T10:50:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-24T10:50:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.234 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.234 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.234 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.234 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.234 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.234 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.234 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-24T10:50:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-24T10:50:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-24T10:50:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T10:50:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-24T10:50:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-24T10:50:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.248 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.248 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.248 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.248 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.248 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-24T10:50:32.248 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.248 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.249 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.249 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-24T10:50:32.249 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-24T10:50:32.249 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.249 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.249 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.249 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.249 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.250 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.250 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.250 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.250 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.250 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.250 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.250 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.251 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-24T10:50:32.251 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-24T10:50:32.251 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.251 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.251 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.251 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.251 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.251 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.252 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.252 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.252 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.252 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.252 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.252 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.253 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.253 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.253 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.253 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.253 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.253 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.253 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.254 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.254 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.254 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.254 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.254 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.254 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.254 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.254 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.255 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.255 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.255 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.255 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.255 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.255 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.255 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.256 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.256 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.256 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.256 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.256 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.256 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.256 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.257 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.257 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.257 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.257 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.257 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.258 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.258 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.258 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.258 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.258 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.258 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.258 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.259 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.259 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.259 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.259 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.259 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.259 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.259 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.259 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.260 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.260 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.260 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.260 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.260 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.260 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.260 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.261 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.261 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.261 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.261 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.261 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.261 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.261 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.261 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.262 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.262 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-24T10:50:32.262 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-24T10:50:32.262 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.262 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.262 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.263 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.263 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.263 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.263 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.263 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.263 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.263 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.264 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.264 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-24T10:50:32.264 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-24T10:50:32.264 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.264 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.264 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.264 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.265 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.265 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.265 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.265 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.265 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.265 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.265 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.266 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.266 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.266 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.266 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-24T10:50:32.267 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.267 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.267 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.267 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.267 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.267 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.267 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.267 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.268 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.268 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.268 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.268 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.268 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.268 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.268 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.269 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.269 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.269 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.269 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.269 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.269 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.269 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.270 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.270 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.270 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.270 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.270 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.270 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.270 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.270 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.271 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.271 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.271 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.271 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.271 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.271 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.271 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.272 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.272 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.272 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.272 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.272 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.272 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.272 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.272 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.273 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.273 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.273 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.273 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.273 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.273 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.273 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.274 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.274 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.274 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.274 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.274 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.274 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.274 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.275 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.275 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.275 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.275 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.275 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.275 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.275 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.275 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.276 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.276 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.276 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.276 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.276 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.276 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.276 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-24T10:50:32.277 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.277 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.277 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.277 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.277 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.277 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.277 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.278 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.278 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.278 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.278 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.278 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.278 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.278 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.278 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.279 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.279 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.279 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.279 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.279 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.279 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.279 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.279 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.280 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.280 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.280 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.280 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.280 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.280 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.281 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.281 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.281 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.281 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.281 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.281 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.281 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.281 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.282 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.282 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.282 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.282 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.282 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.282 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.282 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.282 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.283 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.283 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.283 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.283 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.283 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.283 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.283 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.284 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.284 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.284 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.284 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.284 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.284 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.285 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.285 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.285 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.285 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.285 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.285 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.285 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.285 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.286 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.286 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.286 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-24T10:50:32.286 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-24T10:50:32.286 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.286 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.286 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.287 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-24T10:50:32.287 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.287 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.287 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.287 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T10:50:32.287 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.287 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.287 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.287 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.288 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.288 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-24T10:50:32.288 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.288 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.288 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.289 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.289 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.289 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.289 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.289 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.289 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.289 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.289 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.290 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.290 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.290 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.290 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.290 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.290 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.290 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.291 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.291 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.291 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.291 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.291 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.291 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.292 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.292 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.292 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.292 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.292 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.292 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.292 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.292 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.292 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.293 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.293 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.293 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.293 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.293 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.293 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.293 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.294 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.294 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.294 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.294 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.294 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.294 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.294 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.294 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.295 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.295 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.295 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.295 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.295 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.295 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.295 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.295 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.296 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.296 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.296 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.296 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.296 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.296 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.296 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.296 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.297 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.297 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.297 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.297 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.297 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.297 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.297 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-24T10:50:32.298 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-24T10:50:32.298 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.298 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.298 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.298 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.298 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.298 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.299 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.299 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.299 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.299 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.299 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.299 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-24T10:50:32.299 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-24T10:50:32.300 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.300 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.300 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.300 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-24T10:50:32.300 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.300 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.300 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.300 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.301 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.301 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.301 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.301 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-24T10:50:32.301 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-24T10:50:32.301 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.301 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.302 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.302 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.302 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.302 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.302 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.302 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.302 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.302 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.303 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.303 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-24T10:50:32.303 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-24T10:50:32.303 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.303 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.303 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.303 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.304 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.304 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.304 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.304 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.304 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.304 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.304 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.305 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-24T10:50:32.305 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-24T10:50:32.305 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:32.305 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.305 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.305 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.305 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.305 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.305 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.306 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.306 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.306 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.306 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.306 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-24T10:50:32.306 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-24T10:50:32.306 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.307 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.307 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.307 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.307 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.307 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.307 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.307 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.307 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.308 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.308 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.308 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.308 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-24T10:50:32.308 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-24T10:50:32.308 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.308 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.309 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.309 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.309 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.309 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.309 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.309 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.309 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.309 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.310 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.310 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.310 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-24T10:50:32.310 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-24T10:50:32.310 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.310 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.310 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.311 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.311 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.311 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.311 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.311 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.311 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.311 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.311 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.312 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-24T10:50:32.312 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-24T10:50:32.312 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.312 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.312 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.312 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.312 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-24T10:50:32.313 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-24T10:50:32.313 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.313 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.313 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.313 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.313 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.313 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.313 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-24T10:50:32.314 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-24T10:50:32.314 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.314 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.314 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.314 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.314 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.314 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.315 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.315 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.315 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.315 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.315 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.315 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-24T10:50:32.315 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-24T10:50:32.316 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.316 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.316 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.316 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.316 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.316 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.316 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.316 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.317 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.317 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.317 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.317 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-24T10:50:32.317 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-24T10:50:32.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.318 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.318 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.318 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.318 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.318 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.318 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.318 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.318 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.319 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.319 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.319 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.319 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.319 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.319 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.319 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-24T10:50:32.320 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.320 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.320 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.320 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-24T10:50:32.320 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-24T10:50:32.320 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.320 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.320 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.320 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.321 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.321 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.321 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.321 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T10:50:32.321 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.321 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.321 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.322 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-24T10:50:32.322 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-24T10:50:32.322 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.322 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.322 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.322 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.322 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-24T10:50:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.324 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.324 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.324 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.324 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.324 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.324 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.324 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.325 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.325 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.325 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.325 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.325 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.325 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.325 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.325 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.326 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.326 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.326 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.326 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.326 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.326 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.326 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.327 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.327 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.327 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.327 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.327 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.327 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.327 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.327 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.330 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.330 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.330 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.330 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.330 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.330 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.331 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.331 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.331 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.331 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.331 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.331 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.331 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.331 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.331 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.332 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.332 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.332 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.332 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.332 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.332 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-24T10:50:32.332 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.332 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.333 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.333 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.333 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.333 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.333 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.333 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.333 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.333 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.335 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.335 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.335 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.335 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.335 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.335 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.335 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.335 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.336 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.336 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.336 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.336 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.336 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.336 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.336 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.336 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.337 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.337 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.337 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.337 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.337 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.337 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.337 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.338 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.338 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.338 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.338 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.338 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.338 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.338 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.338 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.339 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.339 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.339 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.339 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.339 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.339 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.340 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.340 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.340 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.340 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.340 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.340 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.340 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.340 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.341 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.341 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.341 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.341 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.341 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.341 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.342 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.342 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-24T10:50:32.342 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.342 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.342 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.342 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.342 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.342 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.342 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.343 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.343 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.343 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.343 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.343 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.343 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.343 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.344 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.344 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.344 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.344 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.344 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.344 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.344 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.345 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.345 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.345 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.345 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.345 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.345 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.346 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.346 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.346 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.346 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.346 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.346 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.346 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.346 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.347 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.347 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.347 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.347 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.347 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.347 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.347 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.348 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.348 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.348 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.348 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.348 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.348 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.348 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.348 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.349 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.349 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.349 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.349 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.349 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.349 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.349 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.350 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.350 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.350 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.350 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.350 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.350 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.351 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.351 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.351 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.351 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.351 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.351 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.351 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.351 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.352 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.352 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-24T10:50:32.352 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.352 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.352 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.352 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-24T10:50:32.352 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-24T10:50:32.352 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.353 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.353 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.353 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T10:50:32.353 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.353 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.353 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.353 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.353 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.354 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.354 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.354 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.354 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:32.354 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:32.354 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.354 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.354 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.355 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-24T10:50:32.355 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.355 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.355 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.355 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.355 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.355 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.355 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.356 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.356 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.356 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.356 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.356 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.356 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.356 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.356 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.357 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.357 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.357 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.357 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.357 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.357 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.357 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.357 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.358 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.358 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.358 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.358 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.358 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.358 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.359 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.359 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.359 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.359 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.359 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.359 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.359 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.359 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.360 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.360 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.360 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.360 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.360 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.360 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.360 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.361 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.361 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.361 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.361 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.361 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.361 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.361 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.362 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.362 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.362 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.362 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.362 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.362 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.362 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.363 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.363 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.363 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.363 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.363 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.363 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.363 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.363 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.364 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.364 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.364 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.364 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.364 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.364 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.364 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.364 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.365 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.365 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.365 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-24T10:50:32.365 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.365 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.365 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.365 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-24T10:50:32.365 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-24T10:50:32.366 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.366 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.366 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.366 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.366 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.366 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.366 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.366 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T10:50:32.367 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.367 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.367 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.367 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-24T10:50:32.367 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-24T10:50:32.368 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.368 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.368 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.368 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.368 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.368 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.368 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.369 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T10:50:32.369 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.369 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.369 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.369 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-24T10:50:32.369 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-24T10:50:32.369 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.370 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.370 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.370 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.370 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.370 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.370 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.370 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T10:50:32.370 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.371 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.371 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.371 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-24T10:50:32.371 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-24T10:50:32.371 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.371 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.371 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.371 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.372 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.372 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.372 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.372 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T10:50:32.372 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.372 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.372 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.372 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-24T10:50:32.373 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-24T10:50:32.373 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.373 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.373 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.373 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.373 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.373 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.373 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.373 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T10:50:32.373 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.374 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.374 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.374 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-24T10:50:32.374 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-24T10:50:32.374 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.374 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.374 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.375 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.375 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.375 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.375 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.375 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.375 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.375 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.377 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.377 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-24T10:50:32.377 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-24T10:50:32.377 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.377 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.378 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.378 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.378 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.378 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.378 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.378 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.378 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.378 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.378 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.379 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-24T10:50:32.379 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-24T10:50:32.379 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.379 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.379 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.379 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T10:50:32.379 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.380 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.380 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.380 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.380 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.380 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.380 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.381 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-24T10:50:32.381 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-24T10:50:32.381 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.381 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.381 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.381 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.381 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.381 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.382 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.382 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.382 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.382 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.382 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.382 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:32.382 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:32.383 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.383 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.383 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.383 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-24T10:50:32.383 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-24T10:50:32.383 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.383 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.383 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.384 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.384 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.384 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.384 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-24T10:50:32.384 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-24T10:50:32.384 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.385 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.385 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.385 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-24T10:50:32.385 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.385 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.385 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.385 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.385 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.386 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.386 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.386 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.386 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-24T10:50:32.386 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-24T10:50:32.386 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.386 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.386 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.387 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.387 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.387 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.387 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.387 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.387 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.387 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.387 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.388 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-24T10:50:32.388 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-24T10:50:32.388 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.388 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.388 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.388 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.388 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.388 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.388 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.389 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.389 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.389 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.389 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.389 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.389 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.389 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.390 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.390 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.390 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.390 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.390 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.390 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.390 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.390 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.390 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.391 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.391 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.391 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.391 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.391 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.391 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.391 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.391 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.391 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.392 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.392 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.392 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.392 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.392 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.392 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.392 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.392 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.392 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.393 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.393 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.393 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.393 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.393 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.393 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.393 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.394 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.394 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.394 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.394 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.394 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.394 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.394 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.394 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.394 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.395 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.395 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.395 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.395 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.395 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.395 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.395 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.396 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.396 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.396 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.396 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.396 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.396 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.397 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.397 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.397 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.397 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.397 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.397 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.397 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.397 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.398 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-24T10:50:32.398 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-24T10:50:32.398 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.398 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.398 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.399 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.399 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.399 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.399 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.399 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.399 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.399 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.399 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.400 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-24T10:50:32.400 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-24T10:50:32.400 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.400 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.400 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.400 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.401 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.401 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.401 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.401 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.401 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.401 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.401 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.402 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-24T10:50:32.402 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-24T10:50:32.402 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.402 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.402 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.402 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T10:50:32.402 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.402 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.403 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.403 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.403 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.403 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.403 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.403 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.403 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.404 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.404 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-24T10:50:32.404 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.404 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.404 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.404 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.404 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.404 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.405 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.405 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.405 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.405 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.405 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.405 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.405 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.405 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.405 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.406 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.406 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.406 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.406 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.406 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.406 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.406 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.406 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.407 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.407 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.407 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.407 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.407 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.407 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.407 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.408 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.408 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.408 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.408 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.408 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.408 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.408 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.409 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.409 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.409 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.409 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.409 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.409 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.409 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.410 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.410 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.410 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.410 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.410 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.410 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.410 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.411 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.411 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.411 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.411 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.411 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.411 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.411 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.412 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.412 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.412 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.412 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.412 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.412 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.412 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.412 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.413 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.413 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.413 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.413 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.413 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.413 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.413 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.413 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-24T10:50:32.414 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.414 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.414 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.414 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.414 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.414 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.414 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.414 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.415 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.415 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.415 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.415 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.415 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.415 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.415 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.415 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.416 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.416 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.416 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.416 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.416 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.416 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.416 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.417 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.417 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.417 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.417 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.417 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.417 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.417 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.417 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.417 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.418 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.418 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.418 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.418 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.418 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.418 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.419 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.419 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.419 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.419 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.419 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.419 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.419 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.419 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.419 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.419 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.420 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.420 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.420 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.420 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.420 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.420 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.420 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.420 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.421 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.421 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.421 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.421 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.421 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.421 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.421 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.421 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.422 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.422 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.422 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.422 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.422 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.422 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.423 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-24T10:50:32.423 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-24T10:50:32.423 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.423 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.423 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.423 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-24T10:50:32.423 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.424 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.424 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.424 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T10:50:32.424 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.424 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.424 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.424 INFO:teuthology.orchestra.run.smithi023.stdout: "periodic_async_work": { 2024-03-24T10:50:32.425 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "periodic_async_work", 2024-03-24T10:50:32.425 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.425 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.425 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.425 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.425 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.425 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.426 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.426 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Periodically check for async work", 2024-03-24T10:50:32.426 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.426 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.426 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.426 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-24T10:50:32.426 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-24T10:50:32.427 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.427 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.427 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.427 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:32.427 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.427 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.427 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.428 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T10:50:32.428 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.428 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.428 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.428 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_no_wait": { 2024-03-24T10:50:32.428 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T10:50:32.428 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.429 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.429 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.429 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.429 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.429 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.429 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.429 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T10:50:32.429 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.430 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.430 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.430 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.430 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.430 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.430 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-24T10:50:32.431 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.431 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.431 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.431 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-24T10:50:32.431 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-24T10:50:32.431 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:32.431 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.432 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.432 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-24T10:50:32.432 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.432 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.432 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.432 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.432 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.432 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.433 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.433 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-24T10:50:32.433 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-24T10:50:32.433 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.433 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.433 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.433 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.433 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.434 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.434 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.434 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.434 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.434 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.434 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.434 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:32.434 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:32.435 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.435 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.435 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.435 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:32.435 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.435 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.435 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.435 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.435 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.436 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.436 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.436 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.436 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.436 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.436 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.436 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.436 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.437 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.437 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.437 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.437 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.437 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.437 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.437 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.437 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.437 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.438 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.438 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.438 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.438 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.438 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.438 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.438 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.439 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.439 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.439 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.439 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.439 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.439 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.439 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.439 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.439 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.440 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.440 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.440 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.440 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.440 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.440 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.440 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.440 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.441 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.441 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.441 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.441 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.441 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.441 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.441 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.441 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.441 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.442 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.442 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.442 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.442 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.442 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.442 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.443 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.443 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.443 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.443 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.443 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.443 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.443 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.443 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.443 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.444 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.444 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.444 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.444 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.444 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-24T10:50:32.444 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-24T10:50:32.444 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.444 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.445 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.445 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.445 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.445 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.445 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.445 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.445 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.446 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.446 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.446 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-24T10:50:32.446 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-24T10:50:32.446 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.446 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.446 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.447 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-24T10:50:32.447 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.447 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.447 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.447 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.447 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.447 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.447 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.447 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.448 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-24T10:50:32.448 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-24T10:50:32.448 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.448 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.448 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.448 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T10:50:32.448 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.448 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.449 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.449 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.449 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.449 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.449 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.449 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.449 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.450 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.450 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-24T10:50:32.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.450 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.450 INFO:teuthology.orchestra.run.smithi023.stdout: "gid": 25147, 2024-03-24T10:50:32.450 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "z", 2024-03-24T10:50:32.450 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_features": 4540701547738038271, 2024-03-24T10:50:32.450 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-24T10:50:32.451 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.451 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-24T10:50:32.451 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.451 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.451 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.451 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:32.451 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:32.452 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.452 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.452 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.452 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:32.452 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.452 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.453 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.453 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T10:50:32.453 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.453 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.453 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.453 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.453 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.453 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.454 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.454 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.454 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.454 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.454 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.454 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.454 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.455 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.455 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.455 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.455 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.455 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.455 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.455 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.455 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.456 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.456 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.456 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.456 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.456 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.456 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.456 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.456 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.457 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.457 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.457 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.457 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.457 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.457 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.457 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.457 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.458 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.458 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.458 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.458 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.458 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.458 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.458 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.458 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.459 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.459 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.459 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.459 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.459 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.459 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.459 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.459 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.459 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.460 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.460 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.460 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.460 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.460 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.460 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.460 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.460 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.460 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.461 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.461 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.461 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.461 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.461 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.461 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.461 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.461 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.461 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-24T10:50:32.462 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-24T10:50:32.462 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.462 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.462 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.462 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.462 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.462 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.463 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.463 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-24T10:50:32.463 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.463 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.463 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.463 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.463 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-24T10:50:32.463 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-24T10:50:32.464 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.464 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.464 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.464 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-24T10:50:32.464 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.464 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.464 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.464 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-24T10:50:32.465 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.465 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.465 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.465 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.465 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-24T10:50:32.465 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-24T10:50:32.465 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.465 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.466 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.466 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.466 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.466 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.466 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.466 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-24T10:50:32.466 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.466 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.467 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.467 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-24T10:50:32.467 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-24T10:50:32.467 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.467 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.467 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.468 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.468 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.468 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.468 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.468 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-24T10:50:32.468 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.468 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.468 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.469 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-24T10:50:32.469 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-24T10:50:32.469 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.469 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.469 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.469 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-24T10:50:32.470 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.470 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.470 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.470 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-24T10:50:32.470 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.470 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.470 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.470 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.471 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-24T10:50:32.471 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-24T10:50:32.471 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.471 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.471 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.471 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.471 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.471 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.471 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.472 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-24T10:50:32.472 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.472 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.472 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.472 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.472 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-24T10:50:32.472 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-24T10:50:32.473 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.473 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.473 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.473 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.473 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.473 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.473 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.473 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T10:50:32.473 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.473 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.474 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.474 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.474 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-24T10:50:32.474 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-24T10:50:32.474 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.474 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.474 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.474 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.475 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.475 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.475 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.475 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-24T10:50:32.475 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.475 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.475 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.476 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.476 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.476 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.476 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.476 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-24T10:50:32.476 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.476 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.476 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.476 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-24T10:50:32.477 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-24T10:50:32.477 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.477 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.477 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.477 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.477 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.477 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.477 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.477 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T10:50:32.478 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.478 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.478 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.478 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.478 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-24T10:50:32.478 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-24T10:50:32.478 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.479 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.479 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.479 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-24T10:50:32.479 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.479 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.479 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.479 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T10:50:32.479 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T10:50:32.480 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.480 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.480 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.480 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-24T10:50:32.480 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-24T10:50:32.480 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:32.480 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.480 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.481 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:32.481 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-24T10:50:32.481 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-24T10:50:32.481 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.481 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T10:50:32.481 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T10:50:32.481 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.481 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.482 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.482 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-24T10:50:32.482 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-24T10:50:32.482 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:32.482 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.482 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.482 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-24T10:50:32.482 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-24T10:50:32.483 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-24T10:50:32.483 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.483 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T10:50:32.483 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.483 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.483 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.483 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.483 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-24T10:50:32.484 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-24T10:50:32.484 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.484 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.484 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.484 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T10:50:32.484 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.484 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.485 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.485 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T10:50:32.485 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T10:50:32.485 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.485 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.485 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.485 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-24T10:50:32.485 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-24T10:50:32.486 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:32.486 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.486 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.486 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-24T10:50:32.486 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-24T10:50:32.486 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-24T10:50:32.487 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.487 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-24T10:50:32.487 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T10:50:32.487 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.487 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.487 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.487 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-24T10:50:32.487 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-24T10:50:32.488 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.488 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.488 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.488 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-24T10:50:32.488 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.488 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.488 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.488 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T10:50:32.489 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T10:50:32.489 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.489 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.489 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.489 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-24T10:50:32.489 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-24T10:50:32.489 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:32.489 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.489 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.490 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:32.490 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-24T10:50:32.490 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-24T10:50:32.490 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.490 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T10:50:32.490 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T10:50:32.490 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.491 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.491 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.491 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.491 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.491 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.491 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.491 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.491 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.492 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.492 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.492 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.492 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.492 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.492 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.492 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.492 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.492 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.493 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.493 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.493 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.493 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.493 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.493 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.493 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.494 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.494 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.494 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.494 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.494 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.494 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.494 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.495 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.495 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.495 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.495 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.495 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.495 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.495 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.495 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.496 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.496 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.496 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.496 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.496 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.496 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.496 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.496 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.496 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.497 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.497 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.497 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.497 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.497 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.497 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.497 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.498 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.498 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.498 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.498 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.498 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.498 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.498 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.498 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.499 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.499 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.499 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.499 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.499 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.499 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.499 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.499 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.499 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.500 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-24T10:50:32.500 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-24T10:50:32.500 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:32.500 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.500 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.500 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:32.501 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.501 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.501 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.501 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T10:50:32.501 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.501 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.501 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.501 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.502 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-24T10:50:32.502 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-24T10:50:32.502 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.502 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.502 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.502 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-24T10:50:32.502 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.503 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.503 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.503 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-24T10:50:32.503 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-24T10:50:32.503 INFO:teuthology.orchestra.run.smithi023.stdout: "read", 2024-03-24T10:50:32.503 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap", 2024-03-24T10:50:32.503 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap-read" 2024-03-24T10:50:32.503 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.504 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-24T10:50:32.504 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.504 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.504 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.504 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.504 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-24T10:50:32.504 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-24T10:50:32.505 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.505 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.505 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.505 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.505 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.505 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.505 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.505 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T10:50:32.505 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.506 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.506 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.506 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.506 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:32.506 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:32.506 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.506 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.507 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.507 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:32.507 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.507 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.507 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.507 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T10:50:32.507 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.507 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.508 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.508 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.508 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-24T10:50:32.508 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-24T10:50:32.508 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.508 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.508 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.508 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-24T10:50:32.509 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-24T10:50:32.509 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.509 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.509 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T10:50:32.509 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T10:50:32.509 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.509 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.510 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.510 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-24T10:50:32.510 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-24T10:50:32.510 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:32.510 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.510 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.510 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T10:50:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.512 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.512 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.512 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-24T10:50:32.512 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.512 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.512 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.512 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-24T10:50:32.513 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-24T10:50:32.513 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:32.513 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.513 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.513 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-24T10:50:32.513 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.513 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.513 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.514 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T10:50:32.514 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.514 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.514 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.514 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.514 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-24T10:50:32.514 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-24T10:50:32.514 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.515 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.515 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.515 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-24T10:50:32.515 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.515 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.515 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.515 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T10:50:32.515 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.516 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.516 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.516 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.516 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-24T10:50:32.516 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-24T10:50:32.516 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.517 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.517 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.517 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-24T10:50:32.517 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.517 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.517 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.517 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T10:50:32.518 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.518 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.518 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.518 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.518 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-24T10:50:32.518 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-24T10:50:32.518 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.518 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.519 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.519 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.519 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.519 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.519 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.519 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T10:50:32.519 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T10:50:32.519 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.520 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.520 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.520 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-24T10:50:32.520 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-24T10:50:32.520 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.520 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.520 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.520 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:32.521 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.521 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.521 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.521 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T10:50:32.521 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.521 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.521 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.521 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.522 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-24T10:50:32.522 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T10:50:32.522 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:32.522 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.522 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.522 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-24T10:50:32.522 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.523 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.523 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.523 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T10:50:32.523 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.523 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.523 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.523 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.523 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm_log_destination": { 2024-03-24T10:50:32.523 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm_log_destination", 2024-03-24T10:50:32.524 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.524 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.524 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.524 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.524 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.524 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.524 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.524 INFO:teuthology.orchestra.run.smithi023.stdout: "file", 2024-03-24T10:50:32.524 INFO:teuthology.orchestra.run.smithi023.stdout: "file,syslog", 2024-03-24T10:50:32.525 INFO:teuthology.orchestra.run.smithi023.stdout: "syslog" 2024-03-24T10:50:32.525 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.525 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T10:50:32.525 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.525 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.525 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.525 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.526 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-24T10:50:32.526 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-24T10:50:32.526 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.526 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.526 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.526 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.526 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.526 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.527 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.527 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T10:50:32.527 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.527 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.527 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.527 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.527 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-24T10:50:32.527 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-24T10:50:32.528 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.528 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.528 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.528 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.528 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.528 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.528 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.528 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T10:50:32.528 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.529 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.529 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.529 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.529 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-24T10:50:32.529 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-24T10:50:32.529 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.529 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.529 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.530 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.530 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.530 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.530 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.530 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T10:50:32.530 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.530 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.530 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.531 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.531 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-24T10:50:32.531 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-24T10:50:32.531 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.531 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.531 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.531 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T10:50:32.531 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.532 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.532 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.532 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-24T10:50:32.532 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.532 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.532 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.532 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.533 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-24T10:50:32.533 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-24T10:50:32.533 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.533 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.533 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.533 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T10:50:32.533 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.533 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.534 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.534 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-24T10:50:32.534 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.534 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.534 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.534 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.534 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-24T10:50:32.534 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-24T10:50:32.535 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.535 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.535 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.535 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T10:50:32.535 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.535 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.535 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.535 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-24T10:50:32.536 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.536 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.536 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.536 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.536 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-24T10:50:32.536 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-24T10:50:32.536 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.536 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.536 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.537 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T10:50:32.537 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.537 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.537 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.537 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-24T10:50:32.537 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.537 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.538 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.538 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.538 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-24T10:50:32.538 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-24T10:50:32.538 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.538 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.538 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.539 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T10:50:32.539 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.539 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.539 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.539 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-24T10:50:32.539 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.539 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.539 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.540 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.540 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-24T10:50:32.540 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-24T10:50:32.540 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.540 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.540 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.540 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T10:50:32.540 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.540 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.541 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.541 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-24T10:50:32.541 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.541 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.541 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.541 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.541 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-24T10:50:32.541 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-24T10:50:32.541 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.542 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.542 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.542 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T10:50:32.542 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.542 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.542 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.542 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-24T10:50:32.543 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.543 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.543 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.543 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.543 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-24T10:50:32.543 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-24T10:50:32.543 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.543 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.544 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.544 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T10:50:32.544 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.544 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.544 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.544 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-24T10:50:32.544 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.544 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.545 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.545 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.545 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-24T10:50:32.545 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-24T10:50:32.545 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.545 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.545 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.545 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T10:50:32.546 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.546 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.546 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.546 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-24T10:50:32.546 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.546 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.546 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.547 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.547 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-24T10:50:32.547 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-24T10:50:32.547 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.547 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.547 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.547 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T10:50:32.547 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.548 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.548 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.548 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-24T10:50:32.548 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.548 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.548 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.548 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.548 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-24T10:50:32.549 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-24T10:50:32.549 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.549 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.549 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.549 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T10:50:32.549 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.549 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.549 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.549 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-24T10:50:32.550 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.550 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.550 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.550 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.550 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-24T10:50:32.550 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-24T10:50:32.550 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.551 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.551 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.551 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-24T10:50:32.551 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.551 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.551 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.551 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-24T10:50:32.552 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.552 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.552 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.552 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.552 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-24T10:50:32.552 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-24T10:50:32.552 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.552 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.553 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.553 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T10:50:32.553 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.553 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.553 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.553 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-24T10:50:32.553 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.553 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.553 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.554 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.554 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-24T10:50:32.554 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-24T10:50:32.554 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.554 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.554 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.554 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T10:50:32.554 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.555 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.555 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.555 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-24T10:50:32.555 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.555 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.555 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.555 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.555 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-24T10:50:32.556 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-24T10:50:32.556 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.556 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.556 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.556 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T10:50:32.556 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.556 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.556 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.557 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-24T10:50:32.557 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.557 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.557 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.557 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.557 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-24T10:50:32.558 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-24T10:50:32.558 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.558 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.558 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.558 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.558 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.558 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.559 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.559 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T10:50:32.559 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.559 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.559 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.559 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.559 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-24T10:50:32.560 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-24T10:50:32.560 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.560 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.560 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.560 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:32.560 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.561 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.561 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.561 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T10:50:32.561 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.561 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.561 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.561 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.561 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-24T10:50:32.562 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T10:50:32.562 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.562 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.562 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.562 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-24T10:50:32.562 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.562 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.563 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.563 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T10:50:32.563 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.563 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.563 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.563 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.563 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-24T10:50:32.563 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-24T10:50:32.563 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.564 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.564 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.564 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-24T10:50:32.564 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.564 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.564 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.564 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T10:50:32.564 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.565 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.565 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.565 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.565 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-24T10:50:32.565 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-24T10:50:32.565 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.565 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.566 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.566 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-24T10:50:32.566 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.566 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.566 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.566 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-24T10:50:32.566 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.567 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.567 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.567 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.567 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-24T10:50:32.567 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-24T10:50:32.567 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.567 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.568 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.568 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.568 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.568 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.568 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.568 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T10:50:32.568 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.568 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.569 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.569 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.569 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-24T10:50:32.569 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-24T10:50:32.569 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.569 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.569 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.569 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:32.570 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.570 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.570 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.570 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-24T10:50:32.570 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.570 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.570 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.570 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.571 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-24T10:50:32.571 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-24T10:50:32.571 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.571 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.571 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.571 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:32.571 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.571 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.572 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.572 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-24T10:50:32.572 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.572 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.572 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.572 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.572 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-24T10:50:32.573 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-24T10:50:32.573 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.573 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.573 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.573 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.573 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.573 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.573 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.574 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T10:50:32.574 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.574 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.574 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.574 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.574 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-24T10:50:32.574 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-24T10:50:32.574 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.575 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.575 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.575 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.575 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.575 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.575 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.575 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T10:50:32.576 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.576 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.576 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.576 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.576 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.576 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.576 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.576 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.577 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.577 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.577 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.577 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.577 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.577 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.577 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.578 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.578 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.578 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.578 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.578 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.578 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.578 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.578 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.578 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.579 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.579 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-24T10:50:32.579 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-24T10:50:32.579 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.579 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.579 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.579 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.579 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.581 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.581 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.581 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T10:50:32.581 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.582 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.582 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.582 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.582 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.582 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.582 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.582 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.583 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.583 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.583 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.583 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.583 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.583 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T10:50:32.583 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.584 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.584 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.584 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.584 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.584 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.584 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.584 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.584 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.585 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.585 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.585 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.585 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.585 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.585 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.585 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.586 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.586 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.586 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.586 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.586 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.586 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.586 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.587 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.587 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.587 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.587 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.587 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.587 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.587 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.587 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.587 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.588 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.588 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.588 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.588 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.588 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.588 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.588 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.589 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T10:50:32.589 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T10:50:32.589 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.589 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.589 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.589 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.589 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.589 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.590 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.590 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T10:50:32.590 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.590 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.590 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.590 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.590 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T10:50:32.591 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T10:50:32.591 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.591 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.591 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.591 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-24T10:50:32.591 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.591 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.592 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.592 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T10:50:32.592 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.592 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.592 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.592 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.592 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-24T10:50:32.593 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-24T10:50:32.593 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.593 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.593 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.593 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:32.593 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.593 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.593 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.594 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-24T10:50:32.594 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.594 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.594 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.594 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.594 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-24T10:50:32.594 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-24T10:50:32.594 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.595 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.595 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.595 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:32.595 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.595 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.595 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.595 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T10:50:32.596 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.596 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.596 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.596 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.596 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-24T10:50:32.596 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-24T10:50:32.596 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.596 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.596 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.597 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.597 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.597 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.597 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.597 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-24T10:50:32.597 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.597 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.597 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.598 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.598 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-24T10:50:32.598 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-24T10:50:32.598 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.598 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.598 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.598 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-24T10:50:32.598 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.599 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.599 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.599 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-24T10:50:32.599 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-24T10:50:32.599 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.599 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T10:50:32.599 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.599 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.599 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.599 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.600 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-24T10:50:32.600 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-24T10:50:32.600 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.600 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.600 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.600 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-24T10:50:32.600 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.600 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.600 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.601 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T10:50:32.601 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.601 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.601 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.601 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.601 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-24T10:50:32.601 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-24T10:50:32.601 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.602 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.602 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.602 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T10:50:32.602 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.602 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.602 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.602 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T10:50:32.603 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.603 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.603 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.603 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.603 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-24T10:50:32.603 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-24T10:50:32.603 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.603 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.604 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.604 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.604 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.604 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.604 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.604 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T10:50:32.604 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.604 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.605 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.605 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.605 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-24T10:50:32.605 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-24T10:50:32.605 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.605 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.605 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.605 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.605 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.606 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.606 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.606 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T10:50:32.606 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.606 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.606 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.606 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.607 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-24T10:50:32.607 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-24T10:50:32.607 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.607 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.607 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.607 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.607 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.607 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.607 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.607 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T10:50:32.608 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.608 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.608 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.608 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.608 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-24T10:50:32.608 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-24T10:50:32.609 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.609 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.609 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.609 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.609 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.609 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.609 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.610 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T10:50:32.610 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.610 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.610 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.610 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.610 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-24T10:50:32.610 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-24T10:50:32.610 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.610 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.611 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.611 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.611 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.611 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.611 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.611 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T10:50:32.612 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.612 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.612 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.612 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.612 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-24T10:50:32.612 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-24T10:50:32.612 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.612 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.613 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.613 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-24T10:50:32.613 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.613 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.613 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.613 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-24T10:50:32.613 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.614 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.614 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.614 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.614 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-24T10:50:32.614 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-24T10:50:32.614 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.614 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.615 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.615 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.615 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.615 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.615 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.615 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T10:50:32.615 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.616 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.616 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.616 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.616 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-24T10:50:32.616 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-24T10:50:32.616 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.616 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.616 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.617 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.617 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.617 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.617 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.617 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T10:50:32.617 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.617 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.618 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.618 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.618 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-24T10:50:32.618 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-24T10:50:32.618 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.618 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.618 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.618 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.619 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.619 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.619 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.619 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T10:50:32.619 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.619 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.619 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.619 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.619 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-24T10:50:32.620 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-24T10:50:32.620 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.620 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.620 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.620 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.620 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.620 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.621 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.621 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T10:50:32.621 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.621 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.621 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.621 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.621 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-24T10:50:32.621 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-24T10:50:32.622 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.622 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.622 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.622 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.622 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.622 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.622 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.622 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T10:50:32.623 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.623 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.623 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.623 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.623 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-24T10:50:32.623 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-24T10:50:32.623 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.623 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.624 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.624 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.624 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.624 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.624 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.624 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T10:50:32.624 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.624 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.624 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.625 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.625 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.625 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.625 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.625 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-24T10:50:32.625 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.625 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.625 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.625 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.626 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.626 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.626 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.626 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.626 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.626 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.626 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.626 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.627 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.627 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.627 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.627 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.627 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.627 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.627 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.627 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.627 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.628 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.628 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.628 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.628 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.628 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.628 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.628 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.628 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.629 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.629 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.629 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.629 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.629 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.629 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.629 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.629 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.630 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.630 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.630 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.630 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.630 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.630 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.630 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.630 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.630 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.631 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.631 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.631 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.631 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.631 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.631 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.631 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.631 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.632 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.632 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.632 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.632 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.632 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.632 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.632 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.632 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.633 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.633 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.633 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.633 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.633 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.633 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.633 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.633 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.633 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.633 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.634 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.634 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-24T10:50:32.634 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-24T10:50:32.634 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.634 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.634 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.634 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-24T10:50:32.635 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.635 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.635 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.635 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T10:50:32.635 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.635 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.635 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.635 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.635 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-24T10:50:32.636 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-24T10:50:32.636 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.636 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.636 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.636 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-24T10:50:32.636 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.637 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.637 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.637 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T10:50:32.637 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.637 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.637 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.637 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.638 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.638 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.638 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.638 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-24T10:50:32.638 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.638 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.638 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.638 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T10:50:32.638 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T10:50:32.639 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.639 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.639 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.639 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:32.639 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.639 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.639 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.639 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.640 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.640 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.640 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.640 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.640 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T10:50:32.640 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T10:50:32.640 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.640 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.641 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.641 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.641 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.641 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.641 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.641 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.641 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.641 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.642 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.642 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.642 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T10:50:32.642 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T10:50:32.642 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.642 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.642 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.643 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.643 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.643 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.643 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.643 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.643 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.643 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.643 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.643 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.644 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-24T10:50:32.644 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T10:50:32.644 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.644 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.644 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.644 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.644 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.644 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.645 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.645 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.645 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.645 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.645 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.645 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.645 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T10:50:32.645 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T10:50:32.646 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.646 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.646 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.646 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.646 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.646 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.646 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.646 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.647 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.647 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.647 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.647 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.647 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T10:50:32.647 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T10:50:32.647 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.648 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.648 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.648 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.648 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.648 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.648 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.648 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.649 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.649 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.649 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.649 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.649 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T10:50:32.649 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T10:50:32.649 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.649 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.649 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.650 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.650 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.650 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.650 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.650 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.650 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.650 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.651 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.651 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.651 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T10:50:32.651 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T10:50:32.651 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.651 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.651 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.652 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.652 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.652 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.652 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.652 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.652 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.652 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.652 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.653 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.653 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T10:50:32.653 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T10:50:32.653 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.653 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.653 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.653 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.653 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.654 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.654 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.654 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.654 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.654 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.654 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.654 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.654 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T10:50:32.655 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T10:50:32.655 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.655 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.655 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.655 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.655 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.655 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.655 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.656 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.656 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.656 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.656 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.656 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T10:50:32.656 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T10:50:32.656 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.656 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.657 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.657 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.657 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.657 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.657 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.657 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.657 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.658 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.658 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.658 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.658 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T10:50:32.658 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T10:50:32.658 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.658 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.659 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.659 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.659 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.659 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.659 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.659 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.659 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.659 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.660 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.660 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.660 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T10:50:32.660 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T10:50:32.660 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.660 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.660 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.660 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.661 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.661 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.661 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.661 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.661 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.661 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.661 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.661 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.661 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T10:50:32.662 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T10:50:32.662 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.662 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.662 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.662 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.662 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.662 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.663 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.663 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.663 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.663 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.663 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.663 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.663 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T10:50:32.664 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T10:50:32.664 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.664 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.664 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.664 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-24T10:50:32.664 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.664 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.664 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.665 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.665 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.665 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.665 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.665 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T10:50:32.665 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T10:50:32.665 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.666 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.666 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.666 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.666 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.666 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.666 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.666 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.666 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.667 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.667 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.667 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.667 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-24T10:50:32.667 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-24T10:50:32.667 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.667 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.667 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.667 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.668 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.668 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.668 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.668 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.668 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.668 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.668 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.669 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.669 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T10:50:32.669 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T10:50:32.669 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.669 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.669 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.669 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-24T10:50:32.670 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.670 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.670 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.670 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.670 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.670 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.670 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.671 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.671 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T10:50:32.671 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T10:50:32.671 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.671 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.671 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.671 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.671 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T10:50:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T10:50:32.673 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.673 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.673 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.673 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.673 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.673 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.673 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.673 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.674 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.674 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.674 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.674 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.674 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T10:50:32.674 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T10:50:32.674 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.675 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.675 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.675 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.675 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.675 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.675 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.675 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.675 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.676 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.676 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.676 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.676 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T10:50:32.676 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T10:50:32.676 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.676 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.676 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.677 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.677 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.677 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.677 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.677 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.677 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.677 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.677 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.678 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.678 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T10:50:32.678 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T10:50:32.678 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.678 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.678 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.678 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.678 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.679 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.679 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.679 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.679 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.679 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.679 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.679 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.679 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T10:50:32.679 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T10:50:32.680 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.680 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.680 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.680 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.680 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.680 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.681 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.681 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.681 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.681 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.681 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.681 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T10:50:32.682 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T10:50:32.682 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.682 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.682 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.682 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.682 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.682 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.683 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.683 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.683 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.683 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.683 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.683 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T10:50:32.684 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T10:50:32.684 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.684 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.684 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.684 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.684 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.684 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.684 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.685 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.685 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.685 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.685 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.685 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.685 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T10:50:32.685 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T10:50:32.685 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.685 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.686 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.686 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.686 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.686 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.686 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.686 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.686 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.687 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.687 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.687 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.687 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T10:50:32.687 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T10:50:32.687 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.687 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.687 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.688 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.688 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.688 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.688 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.688 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.688 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.688 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.689 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.689 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.689 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T10:50:32.689 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T10:50:32.689 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.689 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.689 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.689 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.690 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.690 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.690 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.690 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.690 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.690 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.690 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.690 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.691 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T10:50:32.691 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T10:50:32.691 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.691 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.691 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.691 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.691 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.692 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.692 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.692 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.692 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.692 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.692 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.692 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.692 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T10:50:32.693 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T10:50:32.693 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.693 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.693 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.693 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.693 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.693 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.693 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.693 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.694 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.694 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.694 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.694 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.694 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T10:50:32.694 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T10:50:32.694 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.695 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.695 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.695 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.695 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.695 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.695 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.695 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.695 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.696 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.696 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.696 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.696 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T10:50:32.696 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T10:50:32.696 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.696 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.696 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.697 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T10:50:32.697 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.697 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.697 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.697 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.697 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.697 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.698 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.698 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.698 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T10:50:32.698 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T10:50:32.698 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.698 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.698 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.698 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:32.698 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.699 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.699 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.699 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.699 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.699 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.699 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.699 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.699 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T10:50:32.699 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T10:50:32.700 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.700 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.700 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.700 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-24T10:50:32.700 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.700 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.700 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.701 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.701 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.701 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.701 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.701 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.701 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T10:50:32.701 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T10:50:32.702 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.702 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.702 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.702 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-24T10:50:32.702 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.702 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.702 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.703 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.703 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.703 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.703 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.703 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.703 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T10:50:32.703 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T10:50:32.703 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.703 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.704 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.704 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.704 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.704 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.704 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.704 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.704 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.705 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.705 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.705 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.705 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T10:50:32.705 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T10:50:32.705 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.705 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.705 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.706 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-24T10:50:32.706 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.706 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.706 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.706 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.706 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.707 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.707 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.707 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T10:50:32.707 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T10:50:32.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.707 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.707 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.708 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.708 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.708 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.708 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.708 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.708 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.708 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.709 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.709 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T10:50:32.709 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T10:50:32.709 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.709 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.709 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.709 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.710 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.710 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.710 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.710 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.710 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.710 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.710 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.711 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T10:50:32.711 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T10:50:32.711 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.711 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.711 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.711 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:32.712 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.712 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.712 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.712 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.712 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.712 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.712 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.713 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T10:50:32.713 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T10:50:32.713 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.713 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.714 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.714 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:32.714 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.714 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.714 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.714 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.714 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.715 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.715 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.715 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T10:50:32.715 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T10:50:32.715 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.715 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.716 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.716 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-24T10:50:32.716 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.716 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.716 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.716 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.716 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.716 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.717 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.717 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.717 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-24T10:50:32.717 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-24T10:50:32.717 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.717 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.717 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.717 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.718 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.718 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.718 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.718 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.718 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.718 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.718 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.719 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-24T10:50:32.719 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-24T10:50:32.719 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.719 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.719 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.719 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.719 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.719 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.720 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.720 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.720 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.720 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.720 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.720 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.720 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-24T10:50:32.720 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-24T10:50:32.721 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.721 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.721 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.721 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.721 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.721 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.721 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.722 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-24T10:50:32.722 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.722 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.722 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.722 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.722 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-24T10:50:32.722 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-24T10:50:32.722 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.723 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.723 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.723 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-24T10:50:32.723 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.723 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.723 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.723 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.724 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.724 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.724 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.724 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.724 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-24T10:50:32.724 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-24T10:50:32.724 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.724 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.725 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.725 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.725 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.725 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.725 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.725 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.725 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.725 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.726 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.726 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.726 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.726 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.726 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.726 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.726 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.727 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.727 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.727 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.727 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.727 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.727 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.727 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.727 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.728 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.728 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.728 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.728 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.728 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.728 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.728 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.728 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.729 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.729 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.729 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.729 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.729 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.729 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.729 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.729 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.730 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.730 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.730 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.730 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.730 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.730 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.730 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.730 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.731 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.731 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.731 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.731 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.731 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.731 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.731 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.731 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.732 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.732 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.732 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.732 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.732 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.732 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.732 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.732 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.733 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.733 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.733 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.733 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.733 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.733 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.733 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.733 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.734 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.734 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.734 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.734 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.734 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.734 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.734 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.735 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.735 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.735 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-24T10:50:32.735 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-24T10:50:32.735 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.735 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.735 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.735 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.736 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.736 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.736 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.736 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-24T10:50:32.736 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.736 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.737 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.737 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.737 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-24T10:50:32.737 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T10:50:32.737 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.737 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.737 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.737 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.738 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.738 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.738 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.738 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.738 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.738 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.739 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.739 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-24T10:50:32.739 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-24T10:50:32.739 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.739 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.739 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.740 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-24T10:50:32.740 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.740 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.740 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.740 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.740 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.740 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.740 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.741 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.741 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-24T10:50:32.741 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-24T10:50:32.741 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.741 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.741 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.742 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-24T10:50:32.742 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.742 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.742 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.742 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.742 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.742 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.742 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.743 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.743 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-24T10:50:32.743 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-24T10:50:32.743 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.743 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.743 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.743 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.743 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.744 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.744 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.744 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.744 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.744 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.744 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.744 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.745 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-24T10:50:32.745 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-24T10:50:32.745 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.745 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.745 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.745 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-24T10:50:32.745 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.746 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.746 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.746 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.746 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.746 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.746 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.747 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.747 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-24T10:50:32.747 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-24T10:50:32.747 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.747 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.747 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.747 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-24T10:50:32.748 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.748 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.748 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.748 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.748 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-24T10:50:32.748 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.748 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.749 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.749 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.749 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.749 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.749 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-24T10:50:32.749 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-24T10:50:32.749 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.749 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.750 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.750 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-24T10:50:32.750 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-24T10:50:32.750 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-24T10:50:32.750 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.750 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.750 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.750 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.750 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.751 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-24T10:50:32.751 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-24T10:50:32.751 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.751 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.751 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.751 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.752 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.752 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.752 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.752 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.752 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.752 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.752 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.752 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.752 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.753 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.753 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.753 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-24T10:50:32.753 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.753 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.753 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.753 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-24T10:50:32.754 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-24T10:50:32.754 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.754 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.754 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.754 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.754 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.754 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-24T10:50:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.756 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.756 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.756 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.756 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.756 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.756 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.756 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.756 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.757 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.757 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.757 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.757 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.757 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.757 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.757 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.757 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.758 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.758 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.758 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.758 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.758 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.758 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.758 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.758 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.759 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.759 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.759 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.759 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.759 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.759 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.760 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.760 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.760 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.760 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.760 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.760 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.760 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.761 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.761 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.761 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.761 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.761 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.761 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.761 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.762 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.762 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.762 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.762 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.762 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.762 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.762 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.763 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.763 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.763 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.763 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.763 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.763 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.763 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.763 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.764 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.764 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.764 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.764 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.764 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.764 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.764 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.765 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.765 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-24T10:50:32.765 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-24T10:50:32.765 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.765 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.765 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.765 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-24T10:50:32.765 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.766 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.766 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.766 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T10:50:32.766 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.766 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.766 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.766 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.766 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-24T10:50:32.767 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-24T10:50:32.767 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.767 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.767 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.767 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-24T10:50:32.767 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.767 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.768 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.768 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T10:50:32.768 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.768 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.768 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.768 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.768 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-24T10:50:32.768 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-24T10:50:32.769 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.769 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.769 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.769 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-24T10:50:32.769 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.769 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.769 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.769 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-24T10:50:32.770 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.770 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.770 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.770 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.770 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-24T10:50:32.770 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-24T10:50:32.770 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.770 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.770 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.771 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-24T10:50:32.771 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.771 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.771 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.771 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T10:50:32.771 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.772 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.772 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.772 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.772 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-24T10:50:32.772 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-24T10:50:32.772 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.772 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.772 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.773 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.773 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.773 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.773 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.773 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T10:50:32.773 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.773 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.774 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.774 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.774 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:32.774 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:32.774 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.774 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.774 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.774 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:32.775 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.775 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.775 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.775 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T10:50:32.775 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.775 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.775 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.776 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.776 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-24T10:50:32.776 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-24T10:50:32.776 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.776 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.776 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.776 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-24T10:50:32.776 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.777 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.777 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.777 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T10:50:32.777 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.777 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.777 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.777 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.778 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.778 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.778 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.778 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-24T10:50:32.778 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.778 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.778 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.779 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.779 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.779 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.779 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.779 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.779 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.779 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.780 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.780 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.780 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.780 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.780 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.780 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.781 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.781 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.781 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.781 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.781 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.781 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.781 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.782 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.782 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.782 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.782 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.782 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.783 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.783 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.783 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.783 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.783 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.783 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.783 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.784 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.784 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.784 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.784 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.784 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.784 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.784 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.784 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.784 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.785 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.785 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.785 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.785 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.785 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.785 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.785 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.785 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.786 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.786 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.786 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.786 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.786 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.786 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.786 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.786 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.787 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.787 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.787 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.787 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.787 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.787 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.787 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.788 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.788 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.788 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.788 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.788 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.788 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.788 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-24T10:50:32.789 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-24T10:50:32.789 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.789 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.789 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.789 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-24T10:50:32.789 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.789 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.790 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.790 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.790 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.790 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.790 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.790 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.790 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-24T10:50:32.791 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-24T10:50:32.791 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.791 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.791 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.791 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-24T10:50:32.791 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.791 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.792 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.792 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.792 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.792 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.792 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.792 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.792 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:32.792 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:32.792 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.792 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.793 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.793 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:32.793 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.793 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.793 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.793 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.793 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.793 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.794 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.794 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.794 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.794 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.794 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.794 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-24T10:50:32.794 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-24T10:50:32.795 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-24T10:50:32.795 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.795 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-24T10:50:32.795 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-24T10:50:32.795 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.795 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.795 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.795 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-24T10:50:32.796 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.796 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.796 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.796 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T10:50:32.796 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.796 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.796 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.796 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.796 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-24T10:50:32.797 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-24T10:50:32.797 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.797 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.797 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.797 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-24T10:50:32.797 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.797 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.797 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.798 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T10:50:32.798 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.798 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.798 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.798 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.798 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-24T10:50:32.798 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-24T10:50:32.798 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.799 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.799 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.799 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.799 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.799 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.799 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.799 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-24T10:50:32.800 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.800 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.800 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.800 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.800 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:32.800 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:32.800 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.800 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.801 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.801 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-24T10:50:32.801 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-24T10:50:32.801 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.801 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.801 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T10:50:32.801 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.801 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.801 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.802 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.802 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.802 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.802 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.802 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.802 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.802 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.802 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.802 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.803 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.803 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.803 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.803 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.803 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.803 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.803 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.803 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.804 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.804 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.804 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.804 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.804 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.804 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.804 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.804 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.804 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.805 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.805 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.805 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.805 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.805 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.805 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.805 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.805 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.806 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.806 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.806 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.806 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.806 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.806 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.806 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.806 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.807 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.807 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.807 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.807 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.807 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.807 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.807 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.807 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.808 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.808 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.808 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.808 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.808 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.808 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.808 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.808 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.808 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.809 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.809 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.809 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.809 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.809 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.809 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.809 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.809 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.809 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.810 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.810 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.810 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.810 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-24T10:50:32.810 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-24T10:50:32.810 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.810 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.810 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.811 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.811 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.811 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.811 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.811 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-24T10:50:32.811 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.811 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.811 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.812 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.812 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-24T10:50:32.812 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-24T10:50:32.812 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.812 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.812 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.812 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-24T10:50:32.812 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.812 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.813 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.813 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-24T10:50:32.813 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.813 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.813 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.813 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.813 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-24T10:50:32.813 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-24T10:50:32.814 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.814 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.814 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.814 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-24T10:50:32.814 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.814 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.814 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.814 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T10:50:32.814 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.815 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.815 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.815 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.815 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-24T10:50:32.815 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-24T10:50:32.815 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.815 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.815 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.816 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-24T10:50:32.816 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-24T10:50:32.816 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-24T10:50:32.816 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.816 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T10:50:32.816 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.816 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.816 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.816 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.817 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-24T10:50:32.817 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-24T10:50:32.817 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.817 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.817 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.817 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.817 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.817 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.817 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.818 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-24T10:50:32.818 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.818 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.818 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.818 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.818 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-24T10:50:32.818 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-24T10:50:32.818 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.819 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.819 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.819 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-24T10:50:32.819 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.819 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.819 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.819 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T10:50:32.819 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.820 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.820 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.820 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.820 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.820 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.820 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.820 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-24T10:50:32.820 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.820 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.821 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.821 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.821 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.821 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.821 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.821 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.821 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.821 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.822 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.822 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.822 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.822 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.822 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.822 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.822 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.822 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.822 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.823 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.823 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.823 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.823 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.823 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.823 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.823 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.823 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.824 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.824 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.824 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.824 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.824 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.824 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.824 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.825 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.825 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.825 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.825 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.825 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.825 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.826 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.826 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.826 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.826 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.827 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.827 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.827 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.827 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.827 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.827 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.827 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.827 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.827 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.828 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.828 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.828 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.828 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.828 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.828 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.828 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.828 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.829 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.829 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.829 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.829 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.829 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.829 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.829 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.829 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.829 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.830 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.830 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.830 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.830 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.830 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.830 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.830 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-24T10:50:32.830 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.830 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.831 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.831 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.831 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.831 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.831 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.831 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.831 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.832 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.832 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.832 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.832 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.832 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.832 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.832 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.832 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.832 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.833 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.833 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.833 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.833 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.833 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.833 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.833 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.833 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.833 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.834 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.834 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.834 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.834 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.834 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.834 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.834 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.834 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.835 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.835 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.835 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.835 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.835 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.835 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.835 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.835 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.835 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.836 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.836 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.836 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.836 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.836 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.836 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.836 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.836 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.836 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.837 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.837 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.837 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.837 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.837 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.837 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.837 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.837 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.837 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.838 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.838 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.838 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.838 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.838 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.838 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.839 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.839 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.839 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.839 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.839 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.839 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.839 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.839 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-24T10:50:32.840 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.840 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.840 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.840 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-24T10:50:32.840 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-24T10:50:32.840 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.840 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.840 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.841 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-24T10:50:32.841 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.841 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.841 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.841 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T10:50:32.841 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.841 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.841 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.841 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.842 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-24T10:50:32.842 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-24T10:50:32.842 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.842 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.842 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.843 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:32.843 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-24T10:50:32.843 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.843 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.843 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T10:50:32.843 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.844 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.844 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.844 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.844 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.844 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.844 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.844 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.844 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.845 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.845 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.845 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.845 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.845 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.845 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.845 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.845 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.846 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.846 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.846 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.846 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.846 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.846 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.846 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.846 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.847 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.847 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.847 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.847 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.847 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.847 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.847 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.847 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.847 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.848 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.848 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.848 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.848 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.848 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.848 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.848 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.848 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.848 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.849 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.849 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.849 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.849 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.849 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.849 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.849 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.849 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.849 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.850 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.850 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.850 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.850 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.850 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.850 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.850 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.850 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.851 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.851 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.851 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.851 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.851 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.851 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.851 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.851 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.851 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.851 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.852 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.852 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.852 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.852 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.852 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.852 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.852 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.852 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-24T10:50:32.853 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.853 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.853 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.853 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-24T10:50:32.853 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-24T10:50:32.853 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.853 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.853 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.853 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-24T10:50:32.854 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.854 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.854 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.854 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-24T10:50:32.854 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T10:50:32.854 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.854 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.854 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.855 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.855 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.855 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.855 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.855 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.855 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.855 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.855 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.855 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.856 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.856 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.856 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.856 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.856 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.856 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.856 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.857 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.857 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.857 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.857 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.857 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.857 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.857 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.857 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.857 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.858 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.858 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.858 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.858 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.858 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.858 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.858 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.858 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.858 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.859 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.859 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.859 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.859 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.859 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.859 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.859 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.859 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.860 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.860 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.860 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.860 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.860 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.860 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.860 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.860 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.860 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.861 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.861 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.861 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.861 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.861 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.861 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.861 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.861 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.861 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.862 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.862 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.862 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.862 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.862 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.862 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.862 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.862 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-24T10:50:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-24T10:50:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.864 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.864 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.864 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-24T10:50:32.864 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T10:50:32.864 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.864 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.864 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.864 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-24T10:50:32.864 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-24T10:50:32.865 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.865 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.865 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.865 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-24T10:50:32.865 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.865 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.865 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.865 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-24T10:50:32.866 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.866 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.866 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.866 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.866 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-24T10:50:32.866 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-24T10:50:32.866 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.866 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.867 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.867 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-24T10:50:32.867 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.867 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.867 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.867 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T10:50:32.867 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.867 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.868 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.868 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.868 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-24T10:50:32.868 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-24T10:50:32.868 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.868 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.868 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.868 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.868 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.869 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.869 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.869 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-24T10:50:32.869 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.869 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.869 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.869 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.869 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-24T10:50:32.870 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-24T10:50:32.870 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.870 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.870 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.870 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-24T10:50:32.870 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.870 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.870 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.870 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T10:50:32.871 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T10:50:32.871 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.871 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.871 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.871 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.871 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.871 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.871 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-24T10:50:32.871 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.872 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.872 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.872 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.872 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.872 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.872 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.872 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.872 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.872 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.873 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.873 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.873 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.873 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.873 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.873 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.873 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.873 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.873 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.874 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.874 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.874 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.874 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.874 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.874 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.875 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.875 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.875 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.875 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.875 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.875 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.875 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.875 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.875 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.876 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.876 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.876 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.876 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.876 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.876 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.876 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.876 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.877 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.877 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.877 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.877 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.877 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.877 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.877 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.877 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.877 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.878 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.878 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.878 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.878 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.878 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.878 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.878 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.879 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.879 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.879 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.879 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.879 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.879 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.879 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.879 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.880 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.880 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.880 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.880 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.880 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.880 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.880 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.880 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-24T10:50:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.882 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.882 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.882 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.882 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.882 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.882 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.882 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.882 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.883 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.883 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.883 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.883 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.883 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.883 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.883 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.883 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.883 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.884 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.884 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.884 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.884 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.884 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.884 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.884 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.884 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.885 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.885 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.885 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.885 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.885 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.885 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.885 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.885 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.885 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.886 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.886 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.886 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.886 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.886 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.886 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.887 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.887 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.887 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.887 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.887 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.887 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.887 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.887 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.888 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.888 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.888 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.888 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.888 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.888 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.888 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.888 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.889 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.889 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.889 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.889 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.889 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.889 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.890 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.890 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.890 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.890 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.890 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.890 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.890 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.890 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.890 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-24T10:50:32.891 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.891 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.891 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.891 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.891 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.891 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.892 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.892 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.892 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.892 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.892 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.892 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.892 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.892 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.892 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.893 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.893 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.893 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.893 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.893 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.893 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.893 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.893 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.894 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.894 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.894 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.894 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.894 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.894 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.894 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.894 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.894 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.895 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.895 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.895 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.895 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.895 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.895 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.895 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.895 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.896 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.896 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.896 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.896 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.896 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.896 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.896 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.896 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.898 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.898 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.898 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.898 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.898 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.898 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.898 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.898 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.899 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.899 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.899 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.899 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.899 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.899 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.899 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.899 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.900 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.900 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.900 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.900 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-24T10:50:32.900 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.900 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.900 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.900 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.902 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.902 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.902 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.902 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.902 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.902 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.902 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.902 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.903 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.903 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.903 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.903 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.903 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.903 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.903 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.903 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.904 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.904 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.904 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.904 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.904 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.904 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.904 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.904 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.904 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.905 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.905 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.905 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.905 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.905 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.905 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.905 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.905 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.906 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.906 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.906 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.906 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.906 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.906 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.906 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.906 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.907 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.907 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.907 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.907 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.907 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.907 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.907 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.907 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.907 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.908 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.908 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.908 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.908 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.908 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.908 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.908 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.908 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.908 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.908 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.909 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.909 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.909 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.909 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.909 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.909 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.909 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.909 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.909 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.910 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.910 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.910 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.910 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.910 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-24T10:50:32.910 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-24T10:50:32.910 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.910 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.911 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.911 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.911 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.911 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.911 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.911 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-24T10:50:32.911 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-24T10:50:32.911 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-24T10:50:32.911 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.912 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-24T10:50:32.912 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.912 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.912 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.912 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.912 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.912 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.912 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.912 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-24T10:50:32.913 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.913 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.913 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.913 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.913 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.913 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.913 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.913 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.913 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.914 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.914 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.914 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.914 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.914 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.914 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.914 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.914 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.914 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.915 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.915 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.915 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.915 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.915 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.915 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.915 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.915 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.915 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.916 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.916 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.916 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.916 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.916 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.916 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.916 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.916 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.916 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.917 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.917 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.917 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.917 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.917 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.917 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.917 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.917 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.917 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.917 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.918 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.918 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.918 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.918 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.918 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.918 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.918 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.918 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.919 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.919 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.919 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.919 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.919 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.919 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.919 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.919 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.919 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.920 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.920 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.920 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.920 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.920 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.920 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.920 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.920 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.920 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.921 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.921 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.921 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.921 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.921 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-24T10:50:32.921 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.921 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.921 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.922 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.922 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.922 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.922 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.922 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.922 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.922 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.922 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.923 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.923 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.923 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.923 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.923 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.923 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.923 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.923 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.923 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.924 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.924 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.924 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.924 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.924 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.924 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.924 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.925 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.925 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.925 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.925 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.925 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.925 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.925 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.925 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.925 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.926 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.926 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.926 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.926 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.926 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.926 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.926 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.926 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.927 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.927 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.927 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.927 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.927 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.927 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.927 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.927 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.927 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.928 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.928 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.928 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.928 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.928 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.928 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.928 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.928 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.928 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.928 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.929 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.929 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.929 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.929 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.929 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.929 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.929 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.929 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.929 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.930 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.930 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.930 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.930 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-24T10:50:32.930 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.930 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.930 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.930 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.931 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.931 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.931 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.931 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.931 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.931 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.931 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.931 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.931 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.932 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.932 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.932 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.932 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.932 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.932 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.932 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.932 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.932 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.932 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.933 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.933 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.933 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.933 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.933 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.933 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.933 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.933 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.933 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.934 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.934 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.934 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.934 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.934 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.934 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.934 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.934 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.934 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.934 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.935 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.935 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.935 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.935 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.935 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.935 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.935 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.935 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.935 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.936 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.936 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.936 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.936 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.936 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.936 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.936 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.936 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.936 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.936 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.937 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.937 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.937 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.937 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.937 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.937 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.937 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.937 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.937 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.938 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.938 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.938 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.938 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:32.938 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:32.938 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.938 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.938 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.938 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:32.938 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.939 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.939 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.939 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.939 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.939 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.939 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.939 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.939 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-24T10:50:32.939 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-24T10:50:32.940 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:32.940 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.940 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.940 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-24T10:50:32.940 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-24T10:50:32.940 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.940 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.940 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-24T10:50:32.940 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T10:50:32.941 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.941 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.941 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.941 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.941 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.941 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.941 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-24T10:50:32.941 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.941 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.941 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.942 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-24T10:50:32.942 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-24T10:50:32.942 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.942 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.942 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.942 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.942 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.942 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.942 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.943 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T10:50:32.943 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.943 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.943 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.943 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.943 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-24T10:50:32.943 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-24T10:50:32.944 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.944 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.944 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.944 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.944 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.944 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.944 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.944 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.944 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.944 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.945 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.945 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.945 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.945 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.945 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.945 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.945 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.946 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.946 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.946 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.946 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.946 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.946 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.946 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.946 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.946 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.947 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.947 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.947 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.947 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.947 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.947 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.947 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.947 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.948 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.948 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.948 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.948 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.948 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.948 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.948 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.948 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.948 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.949 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.949 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.949 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.949 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.949 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.949 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.949 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.949 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.949 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.950 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.950 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.950 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.950 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.950 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.950 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.950 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.950 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.951 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.951 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.951 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.951 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.951 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.951 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.951 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.951 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.952 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.952 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.952 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.952 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.952 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.952 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.952 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.952 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.952 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.952 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.953 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.953 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.953 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.953 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.953 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-24T10:50:32.953 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-24T10:50:32.953 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.953 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.954 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.954 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-24T10:50:32.954 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.954 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.954 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.954 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-24T10:50:32.954 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.954 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.955 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.955 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.955 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:32.955 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:32.955 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:32.955 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.955 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.955 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-24T10:50:32.955 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.956 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.956 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.956 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-24T10:50:32.956 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.956 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.956 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.956 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.956 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.956 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.957 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.957 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-24T10:50:32.957 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.957 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.957 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.957 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-24T10:50:32.957 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-24T10:50:32.957 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.957 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.958 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.958 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.958 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.958 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.958 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.958 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.958 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.958 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.959 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.959 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.959 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-24T10:50:32.959 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-24T10:50:32.959 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.959 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.959 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.960 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:32.960 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.960 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.960 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.960 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T10:50:32.960 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T10:50:32.960 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.960 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.960 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.961 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.961 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.961 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.961 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.961 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.961 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.961 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.961 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.961 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.961 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.962 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.962 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.962 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.962 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.962 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.962 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.962 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.962 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.962 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.963 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.963 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.963 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.963 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.963 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.963 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.963 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.963 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.964 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.964 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.964 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.964 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.964 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.964 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.964 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.964 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.964 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.965 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.965 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.965 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.965 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.965 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.965 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.965 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.965 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.965 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.966 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.966 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.966 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.966 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.966 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.966 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.966 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.966 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.967 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.967 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.967 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.967 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.967 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.967 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.967 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.967 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.967 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.968 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.968 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.968 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.968 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.968 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.968 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.968 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.968 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.968 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-24T10:50:32.969 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-24T10:50:32.969 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.969 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.969 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.969 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.969 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.969 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.969 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.969 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.970 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.970 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.970 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.970 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.970 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T10:50:32.970 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T10:50:32.970 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.971 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.971 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.971 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-24T10:50:32.971 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.971 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.971 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.971 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.971 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.971 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.971 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.972 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.972 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-24T10:50:32.972 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-24T10:50:32.972 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:32.972 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.972 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.972 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-24T10:50:32.972 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.972 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.973 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.973 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.973 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.973 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.973 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.973 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.973 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-24T10:50:32.973 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-24T10:50:32.973 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.973 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.974 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.974 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-24T10:50:32.974 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.974 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.974 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.974 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T10:50:32.974 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.974 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.975 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.975 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.975 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-24T10:50:32.975 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-24T10:50:32.975 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.975 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.975 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.975 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-24T10:50:32.975 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.975 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.976 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.976 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T10:50:32.976 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.976 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.976 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.976 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.976 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-24T10:50:32.977 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-24T10:50:32.977 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.977 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.977 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.977 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-24T10:50:32.977 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.977 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.977 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.977 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.977 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.978 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.978 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.978 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.978 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-24T10:50:32.978 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-24T10:50:32.978 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.978 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.978 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.978 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-24T10:50:32.979 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.979 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.979 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.979 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-24T10:50:32.979 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-24T10:50:32.979 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.979 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.980 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.980 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.980 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.980 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.980 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-24T10:50:32.980 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-24T10:50:32.980 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.980 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.980 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.980 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-24T10:50:32.981 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-24T10:50:32.981 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-24T10:50:32.981 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.981 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.981 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.981 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.981 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.981 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.982 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.982 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.982 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-24T10:50:32.982 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.982 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.982 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.982 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:32.982 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:32.983 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.983 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.983 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.983 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.983 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.983 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.983 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.983 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.984 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.984 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.984 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.984 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.984 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.984 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.984 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.984 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.984 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.984 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.985 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.985 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:32.985 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:32.985 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.985 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.985 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.985 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.985 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.985 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.986 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.986 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.986 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.986 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.986 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.986 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.986 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:32.986 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:32.987 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.987 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.987 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.987 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:32.987 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.989 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.989 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:32.989 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:32.989 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:32.990 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:32.990 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:32.990 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:32.990 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:32.990 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:32.990 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.990 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.991 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.991 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.991 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.991 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:32.991 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:32.991 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.991 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.991 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:32.992 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:32.992 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.992 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.992 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.992 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.992 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.993 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.993 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.993 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.993 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-24T10:50:32.993 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-24T10:50:32.993 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:32.994 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.994 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.994 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:32.994 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.994 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.994 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.994 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.994 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.994 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.995 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.995 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.995 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-24T10:50:32.995 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T10:50:32.995 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.995 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.995 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.995 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.995 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.996 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.996 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.996 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.996 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.996 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.996 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.996 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.997 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-24T10:50:32.997 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-24T10:50:32.997 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:32.997 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.997 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:32.997 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:32.997 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:32.997 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:32.998 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:32.998 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:32.998 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:32.998 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:32.998 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:32.998 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.998 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:32.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:32.999 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:32.999 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-24T10:50:32.999 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:32.999 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:32.999 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:32.999 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-24T10:50:32.999 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-24T10:50:32.999 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:32.999 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:32.999 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.000 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.000 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.000 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.000 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.000 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.000 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.001 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.001 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.001 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.001 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-24T10:50:33.001 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-24T10:50:33.001 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.001 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.001 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.001 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.002 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.002 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.002 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.002 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.002 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.002 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.002 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.002 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.002 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.003 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.003 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.003 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.003 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.003 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.003 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.003 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.003 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.003 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.004 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.004 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.004 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.004 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.004 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.004 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.004 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.004 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.004 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.004 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.005 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.005 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.005 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.005 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.005 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.005 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.005 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.005 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.005 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.005 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.006 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.006 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.006 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.006 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.006 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.006 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.006 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.006 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.007 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.007 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.007 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.007 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.007 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.007 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.007 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.007 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.007 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.007 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.008 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.008 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.008 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.008 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.008 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.008 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.008 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.008 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.008 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.009 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.009 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.009 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.009 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.009 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.009 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.009 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.009 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.009 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.010 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.010 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.010 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.010 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-24T10:50:33.010 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-24T10:50:33.010 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.010 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.010 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.010 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.011 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.011 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.011 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.011 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.011 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.011 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.011 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.011 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.012 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-24T10:50:33.012 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-24T10:50:33.012 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.012 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.012 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.012 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.012 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.012 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.012 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.013 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.013 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.013 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.013 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.013 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.013 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.013 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.013 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.014 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-24T10:50:33.014 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.014 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.014 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.014 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.014 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.014 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.014 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.020 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.020 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.020 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.020 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.020 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.020 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.020 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.020 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-24T10:50:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.024 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.024 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.024 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.024 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.024 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.025 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.025 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.025 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.025 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.025 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.025 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.025 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.026 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.026 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.026 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.026 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.026 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.026 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.028 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.028 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.028 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.028 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.028 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.028 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.028 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.028 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.028 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-24T10:50:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-24T10:50:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-24T10:50:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.033 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.033 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T10:50:33.033 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.033 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.033 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.033 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.033 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.034 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.034 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-24T10:50:33.034 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.034 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.034 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.034 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.034 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.035 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.035 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.035 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.035 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.035 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.035 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.035 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.035 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.036 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.036 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.036 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.036 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.036 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.036 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.036 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.036 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.036 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.037 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.037 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.037 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.037 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.037 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.037 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.037 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.037 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.038 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.038 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.038 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.038 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.038 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.038 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.039 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.039 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.039 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.039 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.039 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.039 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.039 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.040 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.040 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.040 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.040 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.040 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.040 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.040 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.040 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.040 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.041 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.041 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.041 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.041 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.041 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.041 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.041 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.041 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.042 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.042 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.042 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.042 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.042 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.042 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.042 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.042 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.042 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.043 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-24T10:50:33.043 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-24T10:50:33.043 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.043 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.043 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.043 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.043 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.043 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.043 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.044 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.044 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.044 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.044 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.044 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.044 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-24T10:50:33.044 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-24T10:50:33.045 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.045 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.045 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.045 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-24T10:50:33.045 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.045 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.045 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.045 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.045 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.046 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.046 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.046 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-24T10:50:33.046 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-24T10:50:33.046 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.046 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.046 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.047 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.047 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.047 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.047 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.047 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.047 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.047 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.047 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.047 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.048 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-24T10:50:33.048 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-24T10:50:33.048 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.048 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.048 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.048 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.048 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.048 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.048 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.049 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.049 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.049 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.049 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.049 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-24T10:50:33.049 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-24T10:50:33.049 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:33.049 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.050 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.050 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.050 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.050 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.050 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.050 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.050 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.050 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.051 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.051 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-24T10:50:33.051 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-24T10:50:33.051 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.051 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.051 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.051 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.052 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.052 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.052 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.052 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.052 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.052 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.052 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.052 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-24T10:50:33.053 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-24T10:50:33.053 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.053 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.053 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.053 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.053 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.053 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.053 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.053 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.053 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.054 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.054 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.054 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.054 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-24T10:50:33.054 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-24T10:50:33.054 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.054 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.054 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.054 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.055 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.055 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.055 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.055 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.055 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.055 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.055 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.056 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-24T10:50:33.056 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-24T10:50:33.056 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.056 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.056 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.056 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.056 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-24T10:50:33.056 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-24T10:50:33.056 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.057 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.057 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.057 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.057 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.057 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.057 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-24T10:50:33.057 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-24T10:50:33.057 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.057 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.058 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.058 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.058 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.058 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.058 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.058 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.058 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.058 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.059 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.059 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.059 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-24T10:50:33.059 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-24T10:50:33.059 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.059 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.059 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.059 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.059 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.060 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.060 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.060 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.060 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.060 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.060 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.060 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.060 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-24T10:50:33.060 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-24T10:50:33.061 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.061 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.061 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.061 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.061 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.061 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.061 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.061 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.061 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.061 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.062 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.062 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.062 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.062 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.062 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.062 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-24T10:50:33.062 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.062 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.063 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.063 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-24T10:50:33.063 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-24T10:50:33.063 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.063 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.063 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.063 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.063 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.063 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.064 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.064 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T10:50:33.064 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.064 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.064 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.064 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.064 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-24T10:50:33.064 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-24T10:50:33.064 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.064 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.065 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.065 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.065 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.065 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.065 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.065 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-24T10:50:33.065 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.065 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.066 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.066 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.066 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.066 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.066 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.066 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.066 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.066 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.067 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.067 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.067 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.067 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.067 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.067 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.067 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.067 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.067 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.067 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.068 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.068 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.068 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.068 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.068 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.068 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.068 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.068 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.069 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.069 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.069 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.069 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.069 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.069 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.069 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.069 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.070 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.070 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.070 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.070 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.070 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.070 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.070 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.070 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.070 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.071 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.071 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.071 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.071 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.071 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.071 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.071 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.071 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.072 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.072 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.072 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.072 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.072 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.072 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.072 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.072 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.072 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.073 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.073 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.073 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.073 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.073 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.073 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.073 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.073 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.073 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.074 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.074 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.074 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.074 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.074 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-24T10:50:33.074 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.074 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.074 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.074 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.075 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.075 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.075 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.075 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.075 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.075 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.075 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.075 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.075 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.075 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.076 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.076 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.076 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.076 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.076 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.076 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.076 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.076 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.076 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.077 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.077 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.077 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.077 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.077 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.077 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.077 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.077 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.077 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.078 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.078 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.078 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.078 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.078 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.078 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.078 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.078 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.079 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.079 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.079 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.079 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.079 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.079 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.079 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.079 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.080 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.080 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.080 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.080 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.080 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.080 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.080 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.080 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.080 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.081 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.081 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.081 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.081 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.081 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.081 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.081 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.081 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.081 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.082 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.082 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.082 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.082 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.082 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.082 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.082 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-24T10:50:33.082 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.083 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.083 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.083 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.083 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.083 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.083 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.083 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.083 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.084 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.084 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.084 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.084 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.084 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.084 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.084 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.084 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.084 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.084 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.085 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.085 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.085 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.085 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.085 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.085 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.085 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.085 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.085 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.086 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.086 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.086 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.086 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.086 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.086 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.086 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.087 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.087 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.087 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.087 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.087 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.087 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.087 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.087 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.087 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.088 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.088 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.088 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.088 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.088 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.088 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.088 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.088 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.088 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.089 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.089 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.089 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.089 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.089 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.089 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.089 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.089 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.090 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.090 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.090 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.090 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.090 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.090 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.090 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.091 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.091 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.091 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.091 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.091 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.091 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-24T10:50:33.091 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.092 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.092 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.092 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-24T10:50:33.092 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-24T10:50:33.092 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.092 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.092 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.092 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T10:50:33.092 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.092 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.093 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.093 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.093 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.093 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.093 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.093 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:33.093 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:33.094 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.094 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.094 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.094 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-24T10:50:33.094 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.094 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.094 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.094 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.094 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.095 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.095 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.095 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.095 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.095 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.095 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.095 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.095 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.096 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.096 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.096 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.096 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.096 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.096 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.096 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.096 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.096 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.097 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.097 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.097 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.097 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.097 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.097 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.098 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.098 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.098 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.098 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.098 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.098 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.098 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.098 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.099 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.099 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.099 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.099 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.099 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.099 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.099 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.099 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.099 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.099 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.100 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.100 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.100 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.100 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.100 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.100 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.100 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.100 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.100 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.101 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.101 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.101 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.101 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.101 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.101 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.101 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.102 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.102 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.102 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.102 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.102 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.102 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.102 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.102 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.102 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.103 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.103 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.103 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.103 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.103 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.103 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.103 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.103 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-24T10:50:33.103 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.104 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.104 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.104 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-24T10:50:33.104 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-24T10:50:33.104 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.104 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.104 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.104 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.104 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.105 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.105 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.105 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T10:50:33.105 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.105 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.105 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.105 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-24T10:50:33.105 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-24T10:50:33.106 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.106 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.106 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.106 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.106 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.106 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.106 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.106 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T10:50:33.106 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.107 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.107 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.107 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.107 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-24T10:50:33.107 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-24T10:50:33.107 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.107 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.107 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.108 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.108 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.108 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.108 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.108 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T10:50:33.108 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.108 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.108 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.108 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.109 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-24T10:50:33.109 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-24T10:50:33.109 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.109 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.109 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.109 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.109 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.109 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.109 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.109 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T10:50:33.110 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.110 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.110 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.110 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.110 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-24T10:50:33.110 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-24T10:50:33.110 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.110 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.110 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.110 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.111 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.111 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.111 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.111 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T10:50:33.111 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.111 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.111 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.111 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.111 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-24T10:50:33.111 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-24T10:50:33.112 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.112 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.112 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.112 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.112 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.112 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.112 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.112 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.112 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.113 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.113 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.113 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-24T10:50:33.113 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-24T10:50:33.113 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.113 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.113 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.113 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.114 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.114 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.114 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.114 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.114 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.114 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.114 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.115 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-24T10:50:33.115 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-24T10:50:33.115 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.115 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.115 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.115 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T10:50:33.115 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.115 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.116 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.116 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.116 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.116 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.116 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.116 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.116 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-24T10:50:33.116 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-24T10:50:33.116 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.117 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.117 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.117 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.117 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.117 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.117 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.117 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.117 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.117 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.117 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.118 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.118 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:33.118 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:33.118 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.118 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.118 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.118 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-24T10:50:33.118 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-24T10:50:33.118 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.118 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.119 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.119 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.119 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.119 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.119 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.119 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-24T10:50:33.119 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-24T10:50:33.119 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.119 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.120 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.120 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-24T10:50:33.120 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.120 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.120 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.120 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.120 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.120 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.120 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.120 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.121 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-24T10:50:33.121 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-24T10:50:33.121 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.121 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.121 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.121 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.121 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.121 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.121 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.122 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.122 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.122 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.122 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.122 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.122 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-24T10:50:33.122 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-24T10:50:33.123 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.123 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.123 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.123 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.123 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.123 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.123 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.123 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.123 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.124 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.124 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.124 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.124 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.124 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.124 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.124 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.124 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.124 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.125 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.125 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.125 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.125 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.125 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.125 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.125 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.125 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.126 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.126 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.126 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.126 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.126 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.126 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.126 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.126 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.127 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.127 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.127 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.127 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.127 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.127 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.127 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.127 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.128 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.128 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.128 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.128 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.128 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.128 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.128 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.128 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.128 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.128 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.129 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.129 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.129 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.129 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.129 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.129 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.129 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.129 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.129 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.130 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.130 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.130 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.130 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.131 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.131 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.131 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.131 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.132 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.132 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.132 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.132 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.133 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.133 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.133 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.133 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.133 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.133 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.133 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.133 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.133 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.133 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-24T10:50:33.134 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-24T10:50:33.134 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.134 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.134 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.134 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.134 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.134 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.134 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.134 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.135 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.135 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.135 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.135 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-24T10:50:33.135 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-24T10:50:33.135 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.135 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.135 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.135 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.136 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.136 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.136 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.136 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.136 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.136 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.136 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.136 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.137 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-24T10:50:33.137 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-24T10:50:33.137 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.137 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.137 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.137 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T10:50:33.137 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.137 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.137 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.138 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.138 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.138 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.138 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.138 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.138 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.138 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.138 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.138 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-24T10:50:33.138 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.139 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.139 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.139 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.139 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.139 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.139 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.139 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.139 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.139 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.140 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.140 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.140 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.140 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.140 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.140 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.140 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.140 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.141 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.141 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.141 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.141 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.141 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.141 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.141 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.141 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.142 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.142 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.142 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.142 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.142 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.142 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.142 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.142 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.142 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.143 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.143 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.143 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.143 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.143 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.143 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.143 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.143 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.143 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.144 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.144 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.144 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.144 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.144 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.144 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.144 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.144 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.144 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.145 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.145 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.145 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.145 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.145 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.145 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.145 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.146 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.146 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.146 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.146 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.146 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.146 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.146 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.146 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.147 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.147 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.147 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.147 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.147 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.147 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-24T10:50:33.147 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.147 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.148 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.148 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.148 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.148 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.148 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.148 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.148 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.148 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.148 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.149 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.149 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.149 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.149 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.149 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.149 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.149 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.149 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.149 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.149 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.150 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.150 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.150 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.150 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.150 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.150 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.150 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.150 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.150 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.151 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.151 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.151 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.151 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.151 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.151 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.151 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.151 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.151 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.152 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.152 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.152 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.152 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.152 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.152 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.152 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.152 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.153 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.153 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.153 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.153 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.153 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.153 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.153 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.153 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.153 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.153 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.154 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.154 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.154 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.154 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.154 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.154 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.154 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.154 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.155 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.155 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.155 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.155 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.155 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.155 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-24T10:50:33.155 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-24T10:50:33.155 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.156 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.156 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.156 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-24T10:50:33.156 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.156 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.156 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.156 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T10:50:33.156 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.156 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.157 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.157 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.157 INFO:teuthology.orchestra.run.smithi023.stdout: "periodic_async_work": { 2024-03-24T10:50:33.157 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "periodic_async_work", 2024-03-24T10:50:33.157 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.157 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.157 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.157 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.157 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.158 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.158 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.158 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Periodically check for async work", 2024-03-24T10:50:33.158 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.158 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.158 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.158 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.158 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-24T10:50:33.159 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-24T10:50:33.159 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.159 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.159 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.159 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:33.159 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.159 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.159 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.160 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T10:50:33.160 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.160 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.160 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.160 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_no_wait": { 2024-03-24T10:50:33.160 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T10:50:33.160 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.161 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.161 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.161 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.161 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.161 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.161 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.161 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T10:50:33.161 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.161 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.162 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.162 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.162 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.162 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.162 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.162 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-24T10:50:33.162 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.162 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.162 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.163 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-24T10:50:33.163 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-24T10:50:33.163 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:33.163 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.163 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.163 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-24T10:50:33.163 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.163 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.164 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.164 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.164 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.164 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.164 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.164 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.164 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-24T10:50:33.164 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-24T10:50:33.164 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.165 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.165 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.165 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.165 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.165 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.165 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.165 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.165 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.165 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.166 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.166 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:33.166 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:33.166 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.166 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.166 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.166 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:33.166 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.167 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.167 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.167 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.167 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.167 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.167 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.167 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.167 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.168 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.168 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.168 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.168 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.168 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.168 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.168 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.168 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.168 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.169 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.169 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.169 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.169 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.169 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.169 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.169 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.169 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.169 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.170 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.170 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.170 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.170 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.170 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.170 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.170 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.170 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.171 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.171 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.171 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.171 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.171 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.171 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.171 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.171 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.171 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.172 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.172 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.172 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.172 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.172 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.172 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.172 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.172 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.173 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.173 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.173 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.173 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.173 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.173 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.173 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.173 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.173 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.174 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.174 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.174 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.174 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.174 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.174 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.174 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.174 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.175 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.175 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.175 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.175 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.175 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.175 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.175 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-24T10:50:33.176 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-24T10:50:33.176 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.176 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.176 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.176 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.176 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.176 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.176 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.176 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.176 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.177 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.177 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.177 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-24T10:50:33.177 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-24T10:50:33.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.177 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.177 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.177 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-24T10:50:33.177 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.178 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.178 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.178 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.178 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.178 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.178 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.178 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-24T10:50:33.179 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-24T10:50:33.179 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.179 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.179 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.179 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T10:50:33.179 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.179 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.180 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.180 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.180 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.180 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.180 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.180 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.180 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.180 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.181 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-24T10:50:33.181 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.181 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.181 INFO:teuthology.orchestra.run.smithi023.stdout: "modules": [ 2024-03-24T10:50:33.181 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-24T10:50:33.181 INFO:teuthology.orchestra.run.smithi023.stdout: "dashboard", 2024-03-24T10:50:33.181 INFO:teuthology.orchestra.run.smithi023.stdout: "iostat", 2024-03-24T10:50:33.181 INFO:teuthology.orchestra.run.smithi023.stdout: "nfs", 2024-03-24T10:50:33.182 INFO:teuthology.orchestra.run.smithi023.stdout: "restful", 2024-03-24T10:50:33.182 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_schedule" 2024-03-24T10:50:33.182 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.182 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-24T10:50:33.182 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.182 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-24T10:50:33.182 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.182 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.183 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.183 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:33.183 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:33.183 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.183 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.183 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.183 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:33.183 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.184 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.184 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.184 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T10:50:33.184 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.184 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.184 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.184 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.184 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.185 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.185 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.185 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.185 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.185 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.185 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.185 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.185 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.185 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.186 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.186 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.186 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.186 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.186 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.186 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.186 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.186 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.187 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.187 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.187 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.187 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.187 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.187 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.188 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.188 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.188 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.188 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.188 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.188 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.188 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.188 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.189 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.189 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.189 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.189 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.189 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.189 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.189 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.190 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.190 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.190 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.190 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.190 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.190 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.190 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.191 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.191 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.191 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.191 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.191 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.191 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.191 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.192 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.192 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.192 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.192 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.192 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.192 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.192 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.192 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.193 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.193 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.193 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.193 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-24T10:50:33.193 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-24T10:50:33.193 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.193 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.193 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.194 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.194 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.194 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.194 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.194 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-24T10:50:33.194 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-24T10:50:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-24T10:50:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-24T10:50:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-24T10:50:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-24T10:50:33.197 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-24T10:50:33.197 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.197 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.197 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.197 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.197 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.197 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.197 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.197 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-24T10:50:33.198 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.198 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.198 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.198 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.198 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-24T10:50:33.198 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-24T10:50:33.198 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.199 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.199 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.199 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.199 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.199 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.199 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.199 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-24T10:50:33.199 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.200 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.200 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.200 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.200 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-24T10:50:33.200 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-24T10:50:33.200 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.200 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.200 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.200 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-24T10:50:33.201 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.201 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.201 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.201 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-24T10:50:33.201 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.201 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.201 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.202 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-24T10:50:33.202 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-24T10:50:33.202 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.202 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.202 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.202 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.202 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.202 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-24T10:50:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-24T10:50:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-24T10:50:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.204 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.204 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.204 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.204 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.204 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.204 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T10:50:33.204 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.204 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.204 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.205 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.205 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-24T10:50:33.205 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-24T10:50:33.205 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.205 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.205 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.205 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.205 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.205 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.206 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.206 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-24T10:50:33.206 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.206 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.206 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.206 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.206 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.206 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.206 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.207 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-24T10:50:33.207 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.207 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.207 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.207 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-24T10:50:33.207 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-24T10:50:33.207 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.207 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.207 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T10:50:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-24T10:50:33.209 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-24T10:50:33.209 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.209 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.209 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.209 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-24T10:50:33.209 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.209 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.209 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.209 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T10:50:33.209 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T10:50:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-24T10:50:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-24T10:50:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:33.211 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-24T10:50:33.211 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-24T10:50:33.211 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.211 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T10:50:33.211 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T10:50:33.211 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.211 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.211 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-24T10:50:33.212 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-24T10:50:33.212 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:33.212 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.212 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.212 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-24T10:50:33.212 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-24T10:50:33.212 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-24T10:50:33.212 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.212 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T10:50:33.213 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.213 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.213 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.213 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-24T10:50:33.213 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-24T10:50:33.213 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.213 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.213 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T10:50:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T10:50:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T10:50:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.215 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-24T10:50:33.215 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-24T10:50:33.215 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:33.215 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.215 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.215 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-24T10:50:33.215 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-24T10:50:33.215 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-24T10:50:33.215 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.216 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-24T10:50:33.216 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T10:50:33.216 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.216 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.216 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.216 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-24T10:50:33.216 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-24T10:50:33.216 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.217 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.217 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.217 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-24T10:50:33.217 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.217 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.217 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.217 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T10:50:33.217 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T10:50:33.217 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.218 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.218 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-24T10:50:33.218 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-24T10:50:33.218 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:33.218 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.218 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.218 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:33.218 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-24T10:50:33.219 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-24T10:50:33.219 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.219 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T10:50:33.219 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T10:50:33.219 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.219 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.219 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.219 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.220 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.220 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.220 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.220 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.220 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.220 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.220 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.220 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.220 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.221 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.221 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.221 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.221 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.221 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.221 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.221 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.221 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.222 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.222 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.222 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.222 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.222 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.222 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.222 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.222 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.223 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.223 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.223 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.223 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.223 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.223 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.223 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.224 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.224 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.224 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.224 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.224 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.224 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.224 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.224 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.224 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.225 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.225 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.225 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.225 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.225 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.225 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.225 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.225 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.226 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.226 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.226 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.226 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.226 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.226 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.226 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.226 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.227 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.227 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.227 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.227 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.227 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.227 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.227 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.228 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-24T10:50:33.228 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-24T10:50:33.228 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:33.228 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.228 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.228 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:33.228 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.228 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.229 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.229 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T10:50:33.229 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.229 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.229 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.229 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-24T10:50:33.229 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-24T10:50:33.229 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.230 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.230 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.230 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-24T10:50:33.230 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.230 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.230 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.231 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-24T10:50:33.231 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-24T10:50:33.231 INFO:teuthology.orchestra.run.smithi023.stdout: "read", 2024-03-24T10:50:33.231 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap", 2024-03-24T10:50:33.231 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap-read" 2024-03-24T10:50:33.231 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.231 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-24T10:50:33.231 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.231 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.231 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.232 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-24T10:50:33.232 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-24T10:50:33.232 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.232 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.232 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.232 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.232 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.232 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.233 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.233 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T10:50:33.233 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.233 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.233 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.233 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:33.233 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:33.233 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.233 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.234 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.234 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:33.234 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.234 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.234 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.234 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T10:50:33.234 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.234 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.234 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.235 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-24T10:50:33.235 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-24T10:50:33.235 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.235 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.235 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.235 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-24T10:50:33.235 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-24T10:50:33.235 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.235 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.235 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T10:50:33.236 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T10:50:33.236 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.236 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.236 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-24T10:50:33.236 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-24T10:50:33.236 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:33.236 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.236 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.237 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:33.237 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.237 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.237 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.237 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T10:50:33.237 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.237 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.237 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.237 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.237 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.238 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.238 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-24T10:50:33.238 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.238 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.238 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.238 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-24T10:50:33.239 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-24T10:50:33.239 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:33.239 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.239 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.239 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-24T10:50:33.239 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.239 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.239 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.239 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T10:50:33.240 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.240 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.240 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.240 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-24T10:50:33.240 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-24T10:50:33.240 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.240 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.240 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.241 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-24T10:50:33.241 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.241 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.241 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.241 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T10:50:33.241 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.241 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.241 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.242 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-24T10:50:33.242 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-24T10:50:33.242 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.242 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.242 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.242 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-24T10:50:33.242 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.243 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.243 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.243 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T10:50:33.244 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.244 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.244 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.245 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-24T10:50:33.245 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-24T10:50:33.245 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.245 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.246 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.246 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.246 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.246 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.247 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.247 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T10:50:33.247 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T10:50:33.247 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.248 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.248 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-24T10:50:33.248 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-24T10:50:33.249 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.249 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.249 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.249 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:33.250 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.250 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.250 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.250 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T10:50:33.251 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.251 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.251 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.252 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-24T10:50:33.252 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T10:50:33.252 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:33.252 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.253 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.253 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-24T10:50:33.253 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.253 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.254 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.254 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T10:50:33.254 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.254 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.255 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.255 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm_log_destination": { 2024-03-24T10:50:33.255 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm_log_destination", 2024-03-24T10:50:33.256 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.256 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.256 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.256 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.256 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.256 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.256 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.256 INFO:teuthology.orchestra.run.smithi023.stdout: "file", 2024-03-24T10:50:33.256 INFO:teuthology.orchestra.run.smithi023.stdout: "file,syslog", 2024-03-24T10:50:33.256 INFO:teuthology.orchestra.run.smithi023.stdout: "syslog" 2024-03-24T10:50:33.257 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.257 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T10:50:33.257 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.257 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.257 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.257 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-24T10:50:33.257 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-24T10:50:33.257 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.257 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.258 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.258 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.258 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.258 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.258 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.258 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T10:50:33.258 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.259 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.259 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.259 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-24T10:50:33.259 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-24T10:50:33.259 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.259 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.259 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.259 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.260 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.260 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.260 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.260 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T10:50:33.260 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.260 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.260 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.261 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-24T10:50:33.261 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-24T10:50:33.261 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.261 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.261 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.261 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.261 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.261 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.262 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.262 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T10:50:33.262 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.262 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.262 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.263 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-24T10:50:33.263 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-24T10:50:33.263 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.263 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.263 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.263 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T10:50:33.263 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.264 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.264 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.264 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-24T10:50:33.264 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.264 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.264 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.264 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-24T10:50:33.264 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-24T10:50:33.265 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.265 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.265 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.265 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T10:50:33.265 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.265 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.265 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.265 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-24T10:50:33.265 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.266 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.266 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.266 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-24T10:50:33.266 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-24T10:50:33.266 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.266 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.266 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.266 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T10:50:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-24T10:50:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-24T10:50:33.268 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-24T10:50:33.268 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.268 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.268 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.268 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T10:50:33.268 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.268 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.268 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.268 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-24T10:50:33.269 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.269 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.269 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.269 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.269 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-24T10:50:33.269 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-24T10:50:33.269 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.269 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.269 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T10:50:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-24T10:50:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-24T10:50:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-24T10:50:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T10:50:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-24T10:50:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-24T10:50:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-24T10:50:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.273 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T10:50:33.273 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.273 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.273 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.273 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-24T10:50:33.273 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.273 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.274 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.274 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-24T10:50:33.274 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-24T10:50:33.274 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.274 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.274 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.274 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T10:50:33.274 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-24T10:50:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-24T10:50:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-24T10:50:33.276 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.276 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.276 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.277 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T10:50:33.277 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.277 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.277 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.277 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-24T10:50:33.277 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.277 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-24T10:50:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-24T10:50:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T10:50:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-24T10:50:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-24T10:50:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-24T10:50:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T10:50:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.281 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-24T10:50:33.281 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.281 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.281 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.281 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-24T10:50:33.281 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-24T10:50:33.282 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.282 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.282 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.282 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-24T10:50:33.282 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.282 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.282 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.282 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-24T10:50:33.282 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.283 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.283 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.283 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-24T10:50:33.283 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-24T10:50:33.283 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.283 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.283 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T10:50:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-24T10:50:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.285 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-24T10:50:33.285 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-24T10:50:33.285 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.285 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.285 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.285 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T10:50:33.285 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.285 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.286 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.286 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-24T10:50:33.286 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.286 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.286 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.286 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-24T10:50:33.286 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-24T10:50:33.286 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.287 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.287 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.287 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T10:50:33.287 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.287 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.287 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.287 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-24T10:50:33.287 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.287 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-24T10:50:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-24T10:50:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.289 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.289 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T10:50:33.289 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.289 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.289 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.289 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-24T10:50:33.289 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-24T10:50:33.289 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.290 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.290 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.290 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:33.290 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.291 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.291 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.291 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T10:50:33.291 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.291 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.291 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.292 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-24T10:50:33.292 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T10:50:33.292 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.292 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.292 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.292 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-24T10:50:33.292 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.292 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.293 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.293 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T10:50:33.293 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.293 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.293 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.293 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-24T10:50:33.293 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-24T10:50:33.293 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.294 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.294 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.294 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-24T10:50:33.294 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.294 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.294 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.294 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T10:50:33.294 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.295 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.295 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.295 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-24T10:50:33.295 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-24T10:50:33.295 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.296 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.296 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.296 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-24T10:50:33.296 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.296 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.296 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.296 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-24T10:50:33.296 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.296 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.297 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.297 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-24T10:50:33.297 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-24T10:50:33.297 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.297 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.297 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.297 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.298 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.298 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.298 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.298 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T10:50:33.298 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.298 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.298 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.299 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-24T10:50:33.299 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-24T10:50:33.299 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.299 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.299 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.299 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:33.299 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.299 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.300 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.300 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-24T10:50:33.300 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.300 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.300 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.300 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-24T10:50:33.300 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-24T10:50:33.300 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.301 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.301 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.301 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:33.301 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.301 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.301 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.301 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-24T10:50:33.301 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.302 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.302 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.302 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-24T10:50:33.302 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-24T10:50:33.302 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.302 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.303 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.303 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.303 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.303 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.303 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.303 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T10:50:33.303 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.303 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.303 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.304 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-24T10:50:33.304 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-24T10:50:33.304 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.304 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.304 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.304 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.304 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.304 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.305 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.305 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T10:50:33.305 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.305 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.305 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.306 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.306 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.306 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.307 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.307 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.307 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.307 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.307 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.307 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.307 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.308 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.308 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.308 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.308 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.308 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.308 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.308 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.308 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.308 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.309 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.309 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-24T10:50:33.309 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-24T10:50:33.309 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.309 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.309 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.309 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.310 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.310 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.310 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.310 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T10:50:33.310 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.310 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.310 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.310 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.310 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.311 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T10:50:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.312 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.313 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.314 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.315 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.316 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T10:50:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T10:50:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.317 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T10:50:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T10:50:33.318 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T10:50:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-24T10:50:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.319 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T10:50:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-24T10:50:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-24T10:50:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.320 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-24T10:50:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-24T10:50:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-24T10:50:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.322 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T10:50:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-24T10:50:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-24T10:50:33.323 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-24T10:50:33.324 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-24T10:50:33.325 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-24T10:50:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.326 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-24T10:50:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-24T10:50:33.327 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-24T10:50:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T10:50:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-24T10:50:33.328 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-24T10:50:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-24T10:50:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.329 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T10:50:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-24T10:50:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-24T10:50:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.330 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T10:50:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T10:50:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-24T10:50:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-24T10:50:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.332 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T10:50:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-24T10:50:33.333 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-24T10:50:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.334 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T10:50:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-24T10:50:33.335 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-24T10:50:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T10:50:33.336 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-24T10:50:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-24T10:50:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.337 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T10:50:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-24T10:50:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-24T10:50:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.339 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T10:50:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-24T10:50:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-24T10:50:33.340 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-24T10:50:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.341 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-24T10:50:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.342 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.343 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-24T10:50:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-24T10:50:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.344 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T10:50:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-24T10:50:33.345 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-24T10:50:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T10:50:33.346 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-24T10:50:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-24T10:50:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.347 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T10:50:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.348 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-24T10:50:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-24T10:50:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.349 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T10:50:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-24T10:50:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-24T10:50:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.350 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T10:50:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-24T10:50:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-24T10:50:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.352 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T10:50:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.353 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-24T10:50:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.354 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.355 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.357 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.358 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.359 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.361 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.362 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-24T10:50:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-24T10:50:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-24T10:50:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.363 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T10:50:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-24T10:50:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-24T10:50:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.364 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-24T10:50:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T10:50:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.365 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-24T10:50:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.366 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T10:50:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T10:50:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.367 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T10:50:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T10:50:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.368 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T10:50:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T10:50:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.370 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-24T10:50:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T10:50:33.371 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.372 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T10:50:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T10:50:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.373 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.374 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T10:50:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T10:50:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.375 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T10:50:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T10:50:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.376 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.377 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T10:50:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T10:50:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.378 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T10:50:33.379 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T10:50:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.380 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T10:50:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T10:50:33.381 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.382 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T10:50:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T10:50:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.383 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T10:50:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T10:50:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.385 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T10:50:33.386 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T10:50:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.387 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T10:50:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T10:50:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.388 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T10:50:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T10:50:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-24T10:50:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.390 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T10:50:33.391 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T10:50:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.392 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-24T10:50:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-24T10:50:33.393 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.394 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T10:50:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T10:50:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.395 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-24T10:50:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T10:50:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T10:50:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.397 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T10:50:33.398 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T10:50:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.399 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T10:50:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T10:50:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.400 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T10:50:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T10:50:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.402 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T10:50:33.403 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T10:50:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.404 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T10:50:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T10:50:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.405 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.406 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T10:50:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T10:50:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.407 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T10:50:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T10:50:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.408 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.409 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T10:50:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T10:50:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.410 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T10:50:33.411 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T10:50:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.412 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T10:50:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T10:50:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.413 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.414 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.414 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.414 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.414 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.414 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.414 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.414 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.414 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.415 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T10:50:33.415 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T10:50:33.415 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.415 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.415 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.415 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.415 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.415 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.416 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.416 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.416 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.416 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.416 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.416 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T10:50:33.416 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T10:50:33.416 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.417 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.417 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.417 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.417 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.417 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.417 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.417 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.417 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.417 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.418 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.418 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T10:50:33.418 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T10:50:33.418 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.418 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.418 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.418 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.419 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.419 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.419 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.419 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.419 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.419 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.419 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.419 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.419 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T10:50:33.419 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T10:50:33.420 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.420 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.420 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.420 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T10:50:33.420 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.420 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.420 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.420 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.420 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.421 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.421 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.421 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.421 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T10:50:33.421 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T10:50:33.421 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.421 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.421 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.421 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:33.421 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.422 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.422 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.422 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.422 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.422 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.422 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.422 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T10:50:33.423 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T10:50:33.423 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.423 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.423 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.423 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-24T10:50:33.423 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.423 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.423 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.423 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.423 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.424 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.424 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.424 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T10:50:33.424 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T10:50:33.424 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.424 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.424 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.424 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-24T10:50:33.425 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.425 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.425 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.425 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.425 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.425 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.425 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.425 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T10:50:33.426 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T10:50:33.426 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.426 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.426 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.426 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.426 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.426 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.427 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.427 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.427 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.427 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.427 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.428 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T10:50:33.428 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T10:50:33.428 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.428 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.428 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.428 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-24T10:50:33.428 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.428 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.429 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.429 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.429 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.429 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.429 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.429 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T10:50:33.429 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T10:50:33.429 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.430 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.430 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.430 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.430 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.430 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.430 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.430 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.430 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.430 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.431 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.431 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.431 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T10:50:33.431 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T10:50:33.431 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.431 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.431 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.431 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.431 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.432 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.432 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.432 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.432 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.432 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.432 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.432 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T10:50:33.432 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T10:50:33.432 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.433 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.433 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.433 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:33.433 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.433 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.433 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.433 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.433 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.433 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.434 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.434 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T10:50:33.434 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T10:50:33.434 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.434 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.434 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.434 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:33.435 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.435 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.435 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.435 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.435 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.435 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.435 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.435 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.435 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T10:50:33.435 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T10:50:33.436 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.436 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.436 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.436 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-24T10:50:33.436 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.436 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.436 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.436 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.436 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.437 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.437 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.437 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.437 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-24T10:50:33.437 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-24T10:50:33.437 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.437 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.437 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.437 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.438 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.438 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.438 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.438 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.438 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.439 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.439 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.439 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-24T10:50:33.439 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-24T10:50:33.439 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.439 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.439 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.439 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.440 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.440 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.440 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.440 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.440 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.440 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.440 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.440 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-24T10:50:33.441 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-24T10:50:33.441 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.441 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.441 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.441 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.441 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.441 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.441 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.441 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-24T10:50:33.442 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.442 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.442 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.442 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-24T10:50:33.442 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-24T10:50:33.442 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.442 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.443 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.443 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-24T10:50:33.443 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.443 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.443 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.443 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.443 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.443 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.444 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.444 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-24T10:50:33.444 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-24T10:50:33.444 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.444 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.444 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.444 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.445 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.445 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.445 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.445 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.445 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.445 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.445 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.445 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.445 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.446 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.446 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.446 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.446 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.446 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.446 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.446 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.446 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.446 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.447 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.447 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.447 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.447 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.447 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.447 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.447 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.447 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.447 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.448 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.448 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.448 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.448 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.448 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.448 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.448 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.448 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.449 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.449 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.449 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.449 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.449 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.449 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.449 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.449 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.450 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.450 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.450 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.450 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.450 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.450 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.450 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.450 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.450 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.451 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.451 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.451 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.451 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.451 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.451 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.451 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.451 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.451 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.452 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.452 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.452 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.452 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.452 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.452 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.452 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.453 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.453 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.453 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.453 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.453 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.453 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.453 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.453 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-24T10:50:33.453 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-24T10:50:33.454 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.454 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.454 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.454 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.454 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.454 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.454 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.454 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-24T10:50:33.455 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.455 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.455 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.455 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-24T10:50:33.455 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T10:50:33.455 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.455 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.455 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.456 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.456 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.456 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.456 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.456 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.456 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.456 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.456 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.457 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-24T10:50:33.457 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-24T10:50:33.457 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.457 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.457 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.457 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-24T10:50:33.457 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.457 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.457 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.458 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.458 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.458 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.458 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.458 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.458 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-24T10:50:33.458 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-24T10:50:33.458 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.459 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.459 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.459 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-24T10:50:33.459 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.459 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.459 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.459 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.459 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.459 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.460 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.460 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-24T10:50:33.460 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-24T10:50:33.460 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.460 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.460 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.460 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.461 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.461 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.461 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.461 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.461 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.461 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.461 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.461 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.461 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-24T10:50:33.462 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-24T10:50:33.462 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.462 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.462 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.462 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-24T10:50:33.462 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.462 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.463 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.463 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.463 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.463 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.463 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.463 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.463 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-24T10:50:33.463 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-24T10:50:33.464 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.464 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.464 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.464 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-24T10:50:33.464 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.464 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.464 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.464 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.464 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-24T10:50:33.465 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.465 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.465 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.465 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.465 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.465 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.465 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-24T10:50:33.465 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-24T10:50:33.466 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.466 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.466 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.466 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-24T10:50:33.466 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-24T10:50:33.466 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-24T10:50:33.466 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.466 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-24T10:50:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-24T10:50:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.468 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.468 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.468 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.468 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.468 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.468 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.468 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.468 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.468 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.469 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.469 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.469 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-24T10:50:33.469 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.469 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.469 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.469 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-24T10:50:33.469 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-24T10:50:33.470 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.470 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.470 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.470 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.470 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.470 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.470 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.470 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-24T10:50:33.471 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.471 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.471 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.471 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.471 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.471 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.471 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.471 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.471 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.472 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.472 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.472 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.472 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.472 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.472 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.472 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.472 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.472 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.473 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.473 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.473 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.473 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.473 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.473 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.473 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.473 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.473 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.474 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.474 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.474 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.474 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.474 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.474 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.474 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.474 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.475 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.475 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.475 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.475 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.475 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.475 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.475 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.475 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.476 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.476 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.476 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.476 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.476 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.476 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.476 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.476 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.476 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.477 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.477 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.477 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.477 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.477 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.477 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.477 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.477 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.477 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.478 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.478 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.478 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.478 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.478 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.478 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.478 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.478 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.479 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.479 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.479 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.479 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.479 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.479 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-24T10:50:33.479 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-24T10:50:33.479 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.479 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.480 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.480 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-24T10:50:33.480 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.480 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.480 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.480 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T10:50:33.480 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.480 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.481 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.481 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.481 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-24T10:50:33.481 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-24T10:50:33.481 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.481 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.481 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.482 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-24T10:50:33.482 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.482 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.482 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.483 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T10:50:33.483 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.483 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.483 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.484 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.484 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-24T10:50:33.484 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-24T10:50:33.484 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.486 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.486 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.486 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.486 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-24T10:50:33.486 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-24T10:50:33.486 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.487 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.487 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.487 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-24T10:50:33.487 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.487 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.487 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.487 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T10:50:33.488 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.488 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.488 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.488 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.488 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-24T10:50:33.488 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-24T10:50:33.488 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.488 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.489 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.489 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.489 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.489 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.489 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.489 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T10:50:33.489 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.489 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.490 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.490 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.490 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:33.490 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:33.490 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.491 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.491 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.491 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:33.491 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.491 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.492 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.492 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T10:50:33.492 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.492 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.492 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.492 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.492 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-24T10:50:33.493 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-24T10:50:33.493 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.493 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.493 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.493 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-24T10:50:33.493 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.493 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.494 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.494 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T10:50:33.494 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.494 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.494 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.494 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.494 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.494 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.495 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.495 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-24T10:50:33.495 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.495 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.495 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.496 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.496 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.496 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.496 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.497 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.497 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.497 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.498 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.498 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.498 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.498 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.498 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.499 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.499 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.499 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.499 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.500 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.500 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.500 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.501 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.501 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.501 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.501 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.502 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.502 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.502 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.502 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.502 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.502 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.502 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.503 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.503 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.503 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.503 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.503 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.504 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.504 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.504 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.504 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.504 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.504 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.504 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.505 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.505 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.505 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.505 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.505 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.505 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.505 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.505 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.505 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.506 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.506 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.506 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.506 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.506 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.506 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.506 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.506 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.507 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.507 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.507 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.507 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.508 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.508 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.508 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.508 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.508 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.508 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.508 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.508 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-24T10:50:33.508 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-24T10:50:33.509 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.509 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.509 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.509 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-24T10:50:33.509 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.509 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.509 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.510 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.510 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.510 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.510 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.511 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.511 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-24T10:50:33.511 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-24T10:50:33.511 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.512 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.512 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.512 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-24T10:50:33.512 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.512 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.512 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.513 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.513 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.513 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.513 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.513 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.513 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:33.513 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:33.514 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.514 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.514 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.514 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-24T10:50:33.514 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.514 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.514 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.515 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.515 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.515 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.515 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.515 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.516 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.516 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.516 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.516 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-24T10:50:33.517 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-24T10:50:33.517 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-24T10:50:33.517 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.517 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-24T10:50:33.518 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-24T10:50:33.518 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.518 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.518 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.519 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-24T10:50:33.519 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.519 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.519 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.520 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T10:50:33.520 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.520 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.520 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.520 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.521 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-24T10:50:33.521 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-24T10:50:33.521 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.521 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.521 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.521 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-24T10:50:33.521 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.522 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.522 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.522 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T10:50:33.522 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.522 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.523 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.523 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.523 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-24T10:50:33.523 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-24T10:50:33.523 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.523 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.523 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.523 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.523 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.524 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.524 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.524 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-24T10:50:33.524 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.524 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.524 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.524 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.525 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:33.525 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:33.525 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.525 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.525 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.525 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-24T10:50:33.525 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-24T10:50:33.525 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.525 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.526 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T10:50:33.526 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.526 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.526 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.526 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.526 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.526 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.526 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.527 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.527 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.527 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.527 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.527 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.528 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.528 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.528 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.528 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.528 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.528 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.528 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.528 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.529 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.529 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.529 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.529 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.529 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.529 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.529 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.530 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.530 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.530 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.530 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.530 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.530 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.530 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.531 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.531 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.531 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.531 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.531 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.531 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.531 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.531 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.531 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.532 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.532 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.532 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.532 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.532 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.532 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.532 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.532 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.532 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.532 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.533 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.533 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.533 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.533 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.533 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.533 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.533 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.533 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.533 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.534 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.534 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.534 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.534 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.534 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.534 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.534 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.534 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.535 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.535 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.535 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.535 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.535 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-24T10:50:33.535 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-24T10:50:33.535 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.535 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.535 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.536 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.536 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.536 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.536 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.536 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-24T10:50:33.536 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.536 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.536 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.536 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.537 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-24T10:50:33.537 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-24T10:50:33.537 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.537 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.537 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.537 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-24T10:50:33.537 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.537 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.537 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.538 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-24T10:50:33.538 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.538 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.538 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.538 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.538 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-24T10:50:33.538 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-24T10:50:33.538 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.539 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.539 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.539 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-24T10:50:33.539 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.539 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.539 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.539 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T10:50:33.539 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.539 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.540 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.540 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.540 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-24T10:50:33.540 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-24T10:50:33.540 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.540 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.540 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.540 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-24T10:50:33.540 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-24T10:50:33.541 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-24T10:50:33.541 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.541 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T10:50:33.541 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.541 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.541 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.541 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.541 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-24T10:50:33.541 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-24T10:50:33.542 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.542 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.542 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.542 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.542 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.542 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.542 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.542 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-24T10:50:33.542 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.543 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.543 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.543 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.543 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-24T10:50:33.543 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-24T10:50:33.543 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.543 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.543 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.543 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-24T10:50:33.544 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.544 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.544 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.544 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T10:50:33.544 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.544 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.544 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.544 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-24T10:50:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.545 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.546 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.546 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.546 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.546 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.546 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.546 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.546 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.546 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.546 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.547 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.547 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.547 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.547 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.547 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.547 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.547 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.547 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.547 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.548 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.548 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.548 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.548 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.548 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.548 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.548 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.548 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.548 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.549 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.549 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.549 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.549 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.549 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.549 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.549 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.549 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.550 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.550 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.550 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.550 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.550 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.550 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.550 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.550 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.551 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.551 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.551 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.551 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.551 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.551 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.551 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.551 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.551 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.552 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.552 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.552 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.552 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.552 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.552 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.552 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.552 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.552 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.553 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.553 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.553 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.553 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.553 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.553 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.553 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.553 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.554 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.554 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.554 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-24T10:50:33.554 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.554 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.554 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.554 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.554 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.555 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.555 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.555 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.555 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.555 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.555 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.555 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.555 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.555 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.556 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.556 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.556 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.556 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.556 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.556 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.556 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.556 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.557 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.557 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.557 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.557 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.557 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.557 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.557 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.557 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.557 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.558 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.558 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.558 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.558 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.558 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.558 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.558 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.558 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.559 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.559 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.559 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.559 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.559 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.559 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.559 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.559 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.559 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.560 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.560 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.560 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.560 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.560 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.560 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.560 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.560 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.560 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.561 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.561 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.561 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.561 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.561 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.561 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.561 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.561 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.561 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.562 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.562 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.562 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.562 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.562 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.562 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.562 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.562 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.562 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.563 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.563 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-24T10:50:33.563 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.563 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.563 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.563 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-24T10:50:33.563 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-24T10:50:33.564 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.564 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.564 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.564 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-24T10:50:33.564 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.564 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.564 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.564 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T10:50:33.565 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.565 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.565 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.565 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.565 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-24T10:50:33.565 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-24T10:50:33.565 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.565 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.565 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.566 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:33.566 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-24T10:50:33.566 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.566 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.566 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T10:50:33.566 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.566 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.566 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.567 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.567 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.567 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.567 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.567 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.567 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.567 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.567 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.567 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.568 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.568 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.568 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.568 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.568 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.568 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.568 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.568 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.568 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.569 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.569 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.569 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.569 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.569 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.569 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.569 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.569 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.569 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.570 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.570 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.570 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.570 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.570 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.570 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.570 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.570 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.571 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.571 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.571 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.571 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.571 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.571 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.571 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.571 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.571 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.571 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.572 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.572 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.572 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.572 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.572 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.572 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.572 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.572 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.572 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.573 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.573 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.573 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.573 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.573 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.573 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.573 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.573 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.573 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.573 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.574 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.574 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.574 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.574 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.574 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.574 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.574 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.574 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.574 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.575 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.575 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-24T10:50:33.575 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.575 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.575 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.575 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-24T10:50:33.575 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-24T10:50:33.576 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.576 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.576 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.576 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-24T10:50:33.576 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.576 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.576 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.576 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-24T10:50:33.576 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T10:50:33.577 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.577 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.577 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.577 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.577 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.577 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.577 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.577 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.577 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.577 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.578 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.578 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.578 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.578 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.578 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.578 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.578 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.578 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.578 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.579 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.579 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.579 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.579 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.579 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.579 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.579 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.579 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.579 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.579 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.580 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.580 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.580 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.580 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.580 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.580 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.580 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.580 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.580 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.581 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.581 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.581 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.581 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.581 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.581 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.581 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.581 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.582 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.582 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.582 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.582 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.582 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.582 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.582 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.583 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.583 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.583 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.583 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.583 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.583 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.583 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.583 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.583 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.584 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.584 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.584 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.584 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.584 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.584 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.584 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.584 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.584 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.585 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.585 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.585 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-24T10:50:33.585 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-24T10:50:33.585 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.585 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.585 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.585 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.586 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.586 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.586 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.586 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-24T10:50:33.586 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T10:50:33.586 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.586 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.586 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.587 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-24T10:50:33.587 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-24T10:50:33.587 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.587 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.587 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.587 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-24T10:50:33.587 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.587 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.587 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.588 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-24T10:50:33.588 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.588 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.588 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.588 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.588 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-24T10:50:33.588 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-24T10:50:33.588 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.589 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.589 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.589 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-24T10:50:33.589 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.589 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.589 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.589 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T10:50:33.589 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.589 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.590 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.590 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.590 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-24T10:50:33.590 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-24T10:50:33.590 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.590 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.590 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.590 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.590 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.591 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.591 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.591 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-24T10:50:33.591 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.591 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.591 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.591 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.591 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-24T10:50:33.592 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-24T10:50:33.592 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.592 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.592 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.592 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-24T10:50:33.592 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.592 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.592 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.592 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T10:50:33.593 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T10:50:33.593 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.593 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.593 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.593 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.593 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.593 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.593 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-24T10:50:33.594 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.594 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.594 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.594 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.594 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.594 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.594 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.594 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.594 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.595 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.595 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.595 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.595 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.595 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.595 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.595 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.595 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.596 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.596 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.596 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.596 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.596 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.596 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.596 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.596 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.596 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.597 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.597 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.597 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.597 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.597 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.597 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.597 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.597 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.597 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.598 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.598 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.598 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.598 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.598 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.598 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.598 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.598 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.599 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.599 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.599 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.599 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.599 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.599 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.599 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.599 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.599 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.600 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.600 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.600 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.600 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.600 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.600 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.600 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.600 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.600 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.601 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.601 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.601 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.601 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.601 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.601 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.601 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.601 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.601 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.601 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.602 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.602 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.602 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.602 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.602 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.602 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-24T10:50:33.602 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.602 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.603 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.603 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.603 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.603 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.603 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.603 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.604 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.604 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.604 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.604 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.604 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.604 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.604 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.604 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.605 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.605 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.605 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.605 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.605 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.605 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.605 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.605 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.605 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.607 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.607 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.607 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.607 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.607 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.607 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.607 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.607 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.608 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.608 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.608 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.608 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.608 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.608 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.608 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.608 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.608 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.610 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.610 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.610 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.610 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.610 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.610 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.610 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.610 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.610 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-24T10:50:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.612 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.612 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.612 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.612 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.612 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.612 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.612 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.612 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.612 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.613 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.613 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.613 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.613 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.613 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.613 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.613 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.613 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.613 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.614 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.614 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.614 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.614 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.614 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.614 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.614 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.614 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.614 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.615 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.615 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.615 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.615 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.615 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.615 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.615 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.615 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.616 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.616 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.616 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.616 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.616 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.616 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.616 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.616 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.616 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.617 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.617 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.617 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.617 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.617 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.617 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.617 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.617 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.617 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.618 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.618 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.618 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.618 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.618 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.618 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.618 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.618 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.618 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.619 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.619 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.619 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.619 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.619 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.619 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.619 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.619 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-24T10:50:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-24T10:50:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-24T10:50:33.621 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.621 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.621 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.621 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.621 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.621 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.621 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.621 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T10:50:33.622 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.622 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.622 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.622 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.622 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.622 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.622 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.622 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.622 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.623 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.623 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.623 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.623 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.623 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.623 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.623 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.623 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.624 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.624 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.624 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.624 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.624 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.624 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.624 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.624 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.624 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.625 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.625 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.625 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.625 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.625 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.625 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.625 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.625 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.627 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.627 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.627 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.627 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.627 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.627 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.627 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.627 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.627 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.629 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.629 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.629 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.629 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.629 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.629 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.629 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.629 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.630 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.630 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.630 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.630 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-24T10:50:33.630 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-24T10:50:33.630 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.630 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.630 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.630 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-24T10:50:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-24T10:50:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-24T10:50:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-24T10:50:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.632 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.632 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.632 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.632 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.632 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.632 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-24T10:50:33.632 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.632 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.632 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.633 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.633 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.633 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.633 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.633 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.633 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.633 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.633 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.633 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.633 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.634 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.634 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.634 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.634 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.634 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.634 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.634 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.634 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.634 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.635 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.635 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.635 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.635 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.635 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.635 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.635 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.635 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.635 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.636 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.636 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.636 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.636 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.636 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.636 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.636 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.636 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.636 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.637 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.637 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.637 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.637 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.637 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.637 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.637 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.637 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.638 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.638 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.638 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.638 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.638 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.638 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.638 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.639 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.639 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.639 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.639 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.639 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.639 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.639 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.639 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.640 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.640 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.640 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.640 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.640 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.640 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.640 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.640 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.640 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.641 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.641 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.641 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.641 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-24T10:50:33.641 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.641 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.641 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.641 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.642 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.642 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.642 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.642 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.642 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.642 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.642 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.642 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.642 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.643 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.643 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.643 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.643 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.643 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.643 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.643 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.643 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.643 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.643 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.644 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.644 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.644 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.644 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.644 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.644 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.644 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.644 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.644 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.645 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.645 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.645 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.645 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.645 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.645 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.645 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.645 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.645 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.646 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.646 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.646 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.646 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.646 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.646 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.646 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.646 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.646 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.646 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.647 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.647 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.647 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.647 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.647 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.647 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.647 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.647 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.647 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.647 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.648 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.648 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.648 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.648 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.648 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.648 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.648 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.648 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.648 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.649 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.649 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.649 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.649 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.649 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.649 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.649 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-24T10:50:33.649 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.649 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.650 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.650 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.650 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.650 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.650 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.650 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.650 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.650 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.650 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.652 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.652 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.652 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.652 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.652 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.652 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.652 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.652 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.652 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.654 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.654 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.654 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.654 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.654 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.654 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.654 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.654 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.654 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.655 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.655 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.655 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.655 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.655 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.655 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.655 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.655 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.655 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.656 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.656 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.656 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.656 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.656 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.656 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.656 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.657 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.657 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.657 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.657 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.657 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.657 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.657 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:33.657 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:33.658 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.658 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.658 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.658 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:33.658 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.658 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.658 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.658 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.658 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.659 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.659 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.659 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.659 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-24T10:50:33.659 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-24T10:50:33.659 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:33.659 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.659 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.660 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-24T10:50:33.660 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-24T10:50:33.660 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.660 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.660 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-24T10:50:33.660 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T10:50:33.660 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.660 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.660 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.661 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.661 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.661 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.661 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-24T10:50:33.661 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.661 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.661 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.661 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-24T10:50:33.661 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-24T10:50:33.661 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.662 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.662 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.662 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.662 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.662 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.662 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.662 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T10:50:33.662 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.662 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.663 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.663 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.663 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-24T10:50:33.663 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-24T10:50:33.663 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.663 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.663 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.663 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.663 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.664 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.664 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.664 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.664 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.664 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.664 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.664 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.664 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.665 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.665 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.665 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.665 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.665 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.665 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.665 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.665 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.666 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.666 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.666 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.666 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.666 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.666 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.666 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.666 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.666 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.667 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.667 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.667 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.667 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.667 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.667 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.667 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.667 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.667 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.669 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.669 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.669 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.669 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.669 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.669 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.669 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.669 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.670 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.670 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.670 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.670 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.670 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.670 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.670 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.670 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.670 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.670 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.671 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.671 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.671 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.671 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.671 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.671 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.671 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.671 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.671 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-24T10:50:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-24T10:50:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.673 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.673 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-24T10:50:33.673 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.673 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.673 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.673 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-24T10:50:33.673 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.673 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.673 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.674 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.674 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-24T10:50:33.674 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-24T10:50:33.674 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.674 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.674 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.674 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-24T10:50:33.674 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.674 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.675 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.675 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-24T10:50:33.675 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.675 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.675 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.675 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.675 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.675 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.675 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.676 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-24T10:50:33.676 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.676 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.676 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.676 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-24T10:50:33.676 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-24T10:50:33.676 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.676 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.676 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.677 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.677 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.677 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.677 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.677 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.677 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.677 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.677 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.677 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.678 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-24T10:50:33.678 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-24T10:50:33.678 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.678 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.678 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.678 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.678 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.678 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.678 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.678 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T10:50:33.679 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T10:50:33.679 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.679 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.679 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.679 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.679 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.679 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.679 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.679 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.680 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.680 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.680 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.680 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.680 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.680 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.680 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.680 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.680 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.681 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.681 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.681 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.681 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.681 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.681 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.681 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.681 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.682 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.682 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.682 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.682 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.682 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.682 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.682 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.682 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.683 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.683 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.683 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.683 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.683 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.683 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.683 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.683 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.684 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.684 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.684 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.684 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.684 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.684 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.684 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.684 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.684 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.684 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.685 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.685 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.685 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.685 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.685 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.685 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.685 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.685 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.686 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.686 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.686 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.686 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.686 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.686 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.686 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.686 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.686 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.687 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.687 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.687 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.687 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-24T10:50:33.687 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-24T10:50:33.687 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.687 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.687 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.687 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.687 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.688 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.688 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.688 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.688 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.688 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.688 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.688 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.688 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T10:50:33.688 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T10:50:33.689 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.689 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.689 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.689 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-24T10:50:33.689 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.689 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.689 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.689 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.689 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.690 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.690 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.690 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.690 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-24T10:50:33.690 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-24T10:50:33.690 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:33.690 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.690 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.690 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-24T10:50:33.690 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.691 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.691 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.691 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.691 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.691 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.691 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.691 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.691 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-24T10:50:33.691 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-24T10:50:33.692 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.692 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.692 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.692 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-24T10:50:33.692 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.692 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.692 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.692 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T10:50:33.692 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.693 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.693 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.693 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.693 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-24T10:50:33.693 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-24T10:50:33.693 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.693 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.693 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.693 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-24T10:50:33.693 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.694 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.694 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.694 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T10:50:33.694 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.694 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.694 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.694 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.694 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-24T10:50:33.695 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-24T10:50:33.695 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.695 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.695 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.695 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-24T10:50:33.695 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.695 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.695 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.696 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.696 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.696 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.696 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.696 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.696 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-24T10:50:33.696 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-24T10:50:33.696 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.696 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.697 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.697 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-24T10:50:33.697 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.697 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.697 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.697 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-24T10:50:33.697 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-24T10:50:33.697 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.697 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.698 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.698 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.698 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.698 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.698 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-24T10:50:33.698 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-24T10:50:33.698 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.698 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.698 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.699 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-24T10:50:33.699 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-24T10:50:33.699 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-24T10:50:33.699 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.699 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.699 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.699 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.699 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.699 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.699 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.700 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.700 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.700 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-24T10:50:33.700 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.700 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.700 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.700 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.700 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.700 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.701 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.701 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.701 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.701 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.701 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.701 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.701 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.701 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.701 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.702 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.702 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.702 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.702 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.702 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.702 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.702 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.702 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.702 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.703 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.703 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.703 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.703 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.703 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.703 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.703 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.703 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.704 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.704 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.704 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.704 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.704 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.704 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.704 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.705 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.705 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.705 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.705 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.705 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.705 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.705 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.705 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.706 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.706 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.706 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.706 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.706 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.706 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.706 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.706 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.707 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.707 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.707 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.707 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.707 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.707 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.707 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.707 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.708 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.708 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.708 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.708 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.708 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.708 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.708 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.708 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.708 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-24T10:50:33.709 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-24T10:50:33.709 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.709 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.709 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.709 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-24T10:50:33.709 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.709 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.709 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.709 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.710 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.710 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.710 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.710 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-24T10:50:33.710 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T10:50:33.710 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.710 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.710 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.711 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.711 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.711 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.711 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.711 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.711 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.711 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.711 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.712 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-24T10:50:33.712 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-24T10:50:33.712 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.712 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.712 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.712 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.712 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.712 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.712 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.713 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.713 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.713 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.713 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.713 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.713 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.713 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.713 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.713 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-24T10:50:33.714 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.714 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.714 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.714 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-24T10:50:33.714 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-24T10:50:33.714 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.714 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.714 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.715 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.715 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.715 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.715 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.715 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.715 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.715 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.715 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.716 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-24T10:50:33.716 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-24T10:50:33.716 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.716 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.716 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.716 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.716 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.716 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.716 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.717 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.717 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.717 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.717 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.717 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.717 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.717 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.717 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.717 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.718 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.718 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.718 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.718 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.718 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.718 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.718 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.718 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.719 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.719 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.719 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.719 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.719 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.719 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.719 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.719 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.720 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.720 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.720 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.720 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.720 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.720 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.720 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.720 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.720 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.721 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.721 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.721 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.721 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.721 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.721 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.721 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.721 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.721 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.721 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.722 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.722 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.722 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.722 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.722 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.722 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.722 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.722 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.723 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.723 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.723 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.723 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.723 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.723 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.723 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.723 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.724 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.724 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.724 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.724 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.724 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.724 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.724 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.724 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.724 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.725 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.725 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.725 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.725 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.725 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-24T10:50:33.725 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-24T10:50:33.725 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.725 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.725 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.726 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.726 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.726 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.726 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.726 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.726 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.726 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.726 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.726 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.727 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-24T10:50:33.727 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-24T10:50:33.727 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.727 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.727 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.727 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.727 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.727 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.727 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.728 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.728 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.728 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.728 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.728 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.728 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.728 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.728 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.728 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-24T10:50:33.729 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.729 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.729 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.729 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.729 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.729 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.729 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.729 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.729 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.730 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.730 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.730 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.730 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.730 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.730 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.730 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.730 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.730 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.731 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.731 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.731 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.731 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.731 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.731 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.731 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.731 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.731 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.732 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.732 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.732 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.732 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.732 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.732 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.732 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.732 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.732 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.732 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.733 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.733 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.733 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.733 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.733 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.733 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.733 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.733 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.733 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.734 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.734 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.734 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.734 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.734 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.734 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.734 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.734 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.734 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.734 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.735 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.735 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.735 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.735 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.735 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.735 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.735 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.735 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.736 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.736 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.736 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.736 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.736 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.736 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.736 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.736 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.736 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.737 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.737 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.737 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.737 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-24T10:50:33.737 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.737 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.737 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.737 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.737 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.738 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.738 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.738 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.738 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.738 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.738 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.738 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.738 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.738 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.739 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.739 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.739 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.739 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.739 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.739 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.739 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.739 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.739 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.740 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.740 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.740 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.740 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.740 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.740 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.740 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.740 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.740 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.741 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.741 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.741 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.741 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.741 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.741 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.741 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.741 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.741 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.741 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.742 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.742 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.742 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.742 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.742 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.742 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.742 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.742 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.743 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.743 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.743 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.743 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.743 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.743 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.743 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.743 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.743 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.743 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.744 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.744 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.744 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.744 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.744 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.744 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.744 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.744 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.745 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.745 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.745 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.745 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.745 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.745 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-24T10:50:33.745 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-24T10:50:33.745 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.745 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.746 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.746 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-24T10:50:33.746 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.746 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.746 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.746 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T10:50:33.746 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.746 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.746 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.747 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.747 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.747 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.747 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.747 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-24T10:50:33.747 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.747 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.747 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.747 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.748 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.748 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.748 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.748 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.748 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.748 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.748 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.748 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.748 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.748 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.749 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.749 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.749 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.749 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.749 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.749 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.749 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.749 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.749 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.750 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.750 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.750 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.750 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.750 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.750 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.750 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.750 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.751 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.751 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.751 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.751 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.751 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.751 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.751 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.751 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.752 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.752 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.752 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.752 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.752 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.752 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.752 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.752 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.753 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.753 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.753 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.753 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.753 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.753 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.753 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.753 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.753 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.754 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.754 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.754 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.754 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.754 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.754 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.754 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.754 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.754 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.755 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.755 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.755 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.755 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.755 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.755 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.755 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.755 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-24T10:50:33.756 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-24T10:50:33.756 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.756 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.756 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.756 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.756 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.756 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.756 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.756 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.757 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.757 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.757 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.757 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.757 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-24T10:50:33.757 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-24T10:50:33.757 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.757 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.757 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.758 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-24T10:50:33.758 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.758 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.758 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.758 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.758 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.758 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.758 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.758 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.759 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-24T10:50:33.759 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-24T10:50:33.759 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.759 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.759 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.759 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.759 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.759 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.759 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.760 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.760 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.760 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.760 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.760 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.760 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-24T10:50:33.760 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-24T10:50:33.760 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.760 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.761 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.761 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.761 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.761 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.761 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.761 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.761 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.761 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.762 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.762 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.762 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-24T10:50:33.762 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-24T10:50:33.762 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-24T10:50:33.762 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.762 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.762 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.762 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.763 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.763 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.763 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.763 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.763 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.763 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.763 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.763 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-24T10:50:33.763 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-24T10:50:33.764 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.764 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.764 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.764 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.764 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.764 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.764 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.764 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.764 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.765 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.765 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.765 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.765 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-24T10:50:33.765 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-24T10:50:33.765 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.765 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.766 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.766 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.766 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.766 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.766 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.766 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.766 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.766 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.767 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.767 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.767 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-24T10:50:33.767 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-24T10:50:33.767 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.767 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.767 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.767 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.767 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.768 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.768 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.768 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.768 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.768 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.768 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.768 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.768 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-24T10:50:33.768 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-24T10:50:33.769 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.769 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.769 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.769 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.769 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-24T10:50:33.769 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-24T10:50:33.769 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.769 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.770 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.770 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.770 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.770 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.770 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-24T10:50:33.770 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-24T10:50:33.770 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.770 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.770 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.771 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.771 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.771 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.771 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.771 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.771 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.771 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.771 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.771 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.772 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-24T10:50:33.772 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-24T10:50:33.772 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.772 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.772 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.772 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.772 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.772 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.772 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.773 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.773 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.773 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.773 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.773 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.773 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-24T10:50:33.773 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-24T10:50:33.773 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.774 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.774 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.774 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.774 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.774 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.774 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.774 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.774 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.774 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.775 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.775 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.775 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.775 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.775 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.775 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-24T10:50:33.775 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.775 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.776 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.776 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-24T10:50:33.776 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-24T10:50:33.776 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.776 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.776 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.776 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.776 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.776 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.777 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.777 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T10:50:33.777 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.777 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.777 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.777 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.777 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-24T10:50:33.777 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-24T10:50:33.778 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.778 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.778 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.778 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.778 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.778 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.778 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.778 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-24T10:50:33.778 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.779 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.779 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.779 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.779 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.779 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.779 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.779 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.779 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.780 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.780 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.780 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.780 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.780 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.780 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.780 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.780 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.780 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.781 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.781 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.781 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.781 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.781 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.781 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.781 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.782 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.782 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.782 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.782 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.782 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.782 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.782 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.782 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.783 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.783 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.783 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.783 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.783 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.783 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.783 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.783 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.784 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.784 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.784 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.784 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.784 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.784 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.784 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.784 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.784 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.785 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.785 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.785 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.785 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.785 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.785 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.785 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.785 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.786 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.786 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.786 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.786 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.786 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.786 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.786 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.786 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.786 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.787 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.787 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.787 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.787 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.787 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.787 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.787 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.787 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.787 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.788 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.788 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-24T10:50:33.788 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.788 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.788 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.788 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.788 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.788 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.788 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.789 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.789 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.789 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.789 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.789 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.789 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.789 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.789 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.790 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.790 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.790 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.790 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.790 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.790 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.790 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.790 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.790 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.791 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.791 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.791 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.791 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.791 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.791 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.791 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.791 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.791 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.792 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.792 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.792 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.792 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.792 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.792 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.792 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.792 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.792 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.793 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.793 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.793 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.793 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.793 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.793 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.793 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.793 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.793 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.794 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.794 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.794 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.794 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.794 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.794 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.794 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.794 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.794 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.795 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.795 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.795 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.795 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.795 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.795 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.795 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.795 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.796 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.796 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.796 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.796 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.796 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.796 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.796 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.796 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.797 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-24T10:50:33.797 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.797 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.797 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.797 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.797 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.797 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.797 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.797 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.798 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.798 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.798 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.798 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.798 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.798 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.798 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.798 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.799 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.799 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.799 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.799 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.799 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.799 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.799 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.799 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.799 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.800 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.800 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.800 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.800 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.800 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.800 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.800 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.800 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.800 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.801 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.801 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.801 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.801 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.801 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.801 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.801 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.801 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.802 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.802 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.802 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.802 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.802 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.802 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.802 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.802 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.802 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.803 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.803 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.803 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.803 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.803 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.803 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.803 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.805 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.805 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.805 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.805 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.805 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.806 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.806 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.806 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.806 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.806 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.806 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.806 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.806 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.807 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.807 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.807 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.807 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.807 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.807 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-24T10:50:33.807 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.807 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.807 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.808 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-24T10:50:33.808 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-24T10:50:33.808 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.808 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.808 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.808 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T10:50:33.808 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.808 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.808 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.809 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.809 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.809 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.809 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.809 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.809 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:33.809 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:33.809 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.809 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.810 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.810 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-24T10:50:33.810 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.810 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.810 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.810 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.810 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.810 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.810 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.811 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.811 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.811 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.811 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.811 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.811 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.811 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.811 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.811 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.811 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.812 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.812 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.812 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.812 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.812 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.812 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.812 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.813 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.813 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.813 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.813 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.813 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.813 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.813 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.813 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.813 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.814 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.814 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.814 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.814 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.814 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.814 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.814 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.814 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.814 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.815 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.815 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.815 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.815 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.815 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.815 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.815 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.815 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.815 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.815 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.816 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.816 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.816 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.816 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.816 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.816 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.816 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.817 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.817 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.817 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.817 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.817 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.817 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.817 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.817 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.817 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.818 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.818 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.818 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.818 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.818 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.818 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.818 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.818 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.818 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.819 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.819 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.819 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-24T10:50:33.819 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.819 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.819 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.819 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-24T10:50:33.819 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-24T10:50:33.819 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.820 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.820 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.820 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.820 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.820 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.820 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.820 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T10:50:33.820 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.821 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.821 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.821 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-24T10:50:33.821 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-24T10:50:33.821 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.821 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.821 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.821 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.822 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.822 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.822 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.822 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T10:50:33.822 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.822 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.822 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.822 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.822 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-24T10:50:33.823 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-24T10:50:33.823 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.823 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.823 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.823 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.823 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.823 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.823 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.823 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T10:50:33.824 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.824 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.824 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.824 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.824 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-24T10:50:33.824 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-24T10:50:33.824 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.824 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.824 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.825 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.825 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.825 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.825 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.825 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T10:50:33.825 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.825 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.825 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.826 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.826 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-24T10:50:33.826 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-24T10:50:33.826 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.826 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.826 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.826 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.826 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.827 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.827 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.827 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T10:50:33.827 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.827 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.827 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.827 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.827 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-24T10:50:33.827 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-24T10:50:33.828 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.828 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.828 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.828 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.828 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.828 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.828 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.828 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.828 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.829 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.829 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.829 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.829 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-24T10:50:33.829 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-24T10:50:33.829 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.829 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.829 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.829 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.830 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.830 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.830 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.830 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.830 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.830 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.830 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.830 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.830 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-24T10:50:33.831 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-24T10:50:33.831 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.831 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.831 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.831 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T10:50:33.831 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.831 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.831 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.831 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.832 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.832 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.832 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.832 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.832 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-24T10:50:33.832 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-24T10:50:33.832 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.832 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.832 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.832 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.833 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.833 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.833 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.833 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.833 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.833 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.833 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.833 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.833 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:33.834 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:33.834 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.834 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.834 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.834 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-24T10:50:33.834 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-24T10:50:33.834 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.834 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.834 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-24T10:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-24T10:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.835 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-24T10:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.836 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-24T10:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-24T10:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.837 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.838 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-24T10:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-24T10:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.839 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.840 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.840 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.840 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.840 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.840 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.840 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.840 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.840 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.841 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.841 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.841 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.841 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.841 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.841 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.841 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.841 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.841 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.842 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.842 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.842 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.842 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.842 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.842 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.842 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.842 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.842 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.843 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.843 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.843 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.843 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.843 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.843 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.843 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.843 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.843 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.844 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.844 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.844 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.844 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.844 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.844 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.844 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.844 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.844 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.845 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.845 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.845 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.845 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.845 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.845 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.845 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.845 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.845 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.846 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.846 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.846 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.846 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.846 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.846 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.846 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.846 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.846 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.847 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.847 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.847 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.847 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.847 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.847 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.847 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.847 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.847 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.848 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.848 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.848 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.848 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.848 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.848 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-24T10:50:33.848 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-24T10:50:33.848 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.848 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.849 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.849 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.849 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.849 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.849 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.849 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.849 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.849 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.849 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.849 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.850 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-24T10:50:33.850 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-24T10:50:33.850 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.850 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.850 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.850 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.850 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.850 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.850 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.851 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.851 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.851 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.851 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.851 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.851 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-24T10:50:33.851 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-24T10:50:33.851 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.851 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.852 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.852 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T10:50:33.852 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.852 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.852 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.852 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.852 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.852 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.852 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.853 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.853 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.853 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.853 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.853 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-24T10:50:33.853 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.853 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.853 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.853 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.854 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.854 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.854 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.854 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.854 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.854 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.854 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.854 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.855 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.855 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.855 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.855 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.855 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.855 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.855 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.855 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.855 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.856 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.856 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.856 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.856 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.856 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.856 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.856 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.856 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.856 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.857 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.857 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.857 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.857 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.857 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.857 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.857 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.857 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.857 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.857 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.858 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.858 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.858 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.858 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.858 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.858 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.858 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.858 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.859 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.859 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.859 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.859 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.859 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.859 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.859 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.859 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.859 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.859 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.860 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.860 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.860 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.860 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.860 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.860 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.860 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.860 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.860 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.861 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.861 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.861 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.861 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.861 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.861 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.861 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.861 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.862 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.862 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-24T10:50:33.862 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.862 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.862 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.862 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.862 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.862 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.862 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.863 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.863 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.863 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.863 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.863 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.863 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.863 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.863 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.864 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.864 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.864 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.864 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.864 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.864 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.864 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.864 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.864 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.865 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.865 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.865 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.865 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.865 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.865 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.865 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.865 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.865 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.866 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.866 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.866 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.866 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.866 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.866 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.866 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.866 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.867 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.867 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.867 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.867 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.867 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.867 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.867 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.867 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.867 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.868 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.868 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.868 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.868 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.868 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.868 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.868 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.868 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.868 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.869 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.869 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.869 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.869 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.869 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.869 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.869 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.869 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.869 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.869 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.870 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.870 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.870 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.870 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.870 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-24T10:50:33.870 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-24T10:50:33.870 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.870 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.870 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.871 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-24T10:50:33.871 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.871 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.871 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.871 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T10:50:33.871 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.871 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.871 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.871 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.872 INFO:teuthology.orchestra.run.smithi023.stdout: "periodic_async_work": { 2024-03-24T10:50:33.872 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "periodic_async_work", 2024-03-24T10:50:33.872 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.872 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.872 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.872 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.872 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.872 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.872 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.872 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Periodically check for async work", 2024-03-24T10:50:33.873 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.873 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.873 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.873 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.873 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-24T10:50:33.873 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-24T10:50:33.873 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.873 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.873 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.874 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-24T10:50:33.874 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.874 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.874 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.874 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T10:50:33.874 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.874 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.874 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.875 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.875 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_no_wait": { 2024-03-24T10:50:33.875 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T10:50:33.875 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.875 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.875 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.875 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-24T10:50:33.875 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.875 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.876 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.876 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T10:50:33.876 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.876 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.876 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.876 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.876 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.876 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.876 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.877 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-24T10:50:33.877 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-24T10:50:33.877 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-24T10:50:33.877 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-24T10:50:33.877 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-24T10:50:33.877 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-24T10:50:33.877 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-24T10:50:33.877 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.878 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.878 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-24T10:50:33.878 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.878 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.878 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.878 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.878 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.878 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.878 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.879 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.879 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-24T10:50:33.879 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-24T10:50:33.879 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.879 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.879 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.879 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.879 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.880 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.880 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.880 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.880 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.880 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.880 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.880 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.880 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-24T10:50:33.880 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-24T10:50:33.880 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-24T10:50:33.881 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.881 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.881 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-24T10:50:33.881 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.881 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.881 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.881 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.881 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.881 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.881 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.882 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.882 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-24T10:50:33.882 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-24T10:50:33.882 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.882 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.882 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.882 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.882 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.883 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.883 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.883 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.883 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.883 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.883 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.883 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.883 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.884 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.884 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.884 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.884 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.884 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.884 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.884 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-24T10:50:33.884 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-24T10:50:33.884 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.885 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.885 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.885 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.885 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.885 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.885 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.885 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.885 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.885 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.886 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.886 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.886 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-24T10:50:33.886 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-24T10:50:33.886 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.886 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.886 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.886 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-24T10:50:33.886 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.887 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.887 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-24T10:50:33.887 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-24T10:50:33.887 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-24T10:50:33.887 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-24T10:50:33.887 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-24T10:50:33.887 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-24T10:50:33.887 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-24T10:50:33.887 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.887 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.888 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.888 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.888 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.888 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.888 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-24T10:50:33.888 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-24T10:50:33.888 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-24T10:50:33.888 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.888 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-24T10:50:33.889 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-24T10:50:33.889 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.889 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.889 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.889 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.889 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.889 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.889 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.889 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.890 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-24T10:50:33.890 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-24T10:50:33.890 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.890 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.890 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.890 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-24T10:50:33.890 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.890 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.891 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.891 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.891 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.891 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.891 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.891 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.891 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-24T10:50:33.891 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-24T10:50:33.891 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-24T10:50:33.891 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.892 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.892 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-24T10:50:33.892 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.892 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.892 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.892 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.892 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.892 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.893 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.893 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.893 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-24T10:50:33.893 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-24T10:50:33.893 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-24T10:50:33.893 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-24T10:50:33.893 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-24T10:50:33.893 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T10:50:33.893 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-24T10:50:33.894 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-24T10:50:33.894 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-24T10:50:33.894 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-24T10:50:33.894 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-24T10:50:33.894 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-24T10:50:33.894 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-24T10:50:33.894 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.894 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.895 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.895 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.895 INFO:teuthology.orchestra.run.smithi023.stdout: "services": { 2024-03-24T10:50:33.895 INFO:teuthology.orchestra.run.smithi023.stdout: "dashboard": "https://172.21.15.23:8443/" 2024-03-24T10:50:33.895 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.895 INFO:teuthology.orchestra.run.smithi023.stdout: "always_on_modules": { 2024-03-24T10:50:33.895 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus": [ 2024-03-24T10:50:33.895 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-24T10:50:33.895 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-24T10:50:33.896 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-24T10:50:33.896 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-24T10:50:33.896 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-24T10:50:33.896 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-24T10:50:33.896 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-24T10:50:33.896 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-24T10:50:33.896 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-24T10:50:33.896 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-24T10:50:33.897 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.897 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific": [ 2024-03-24T10:50:33.897 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-24T10:50:33.897 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-24T10:50:33.897 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-24T10:50:33.897 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-24T10:50:33.897 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-24T10:50:33.897 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-24T10:50:33.898 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-24T10:50:33.898 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-24T10:50:33.898 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-24T10:50:33.898 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-24T10:50:33.898 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.898 INFO:teuthology.orchestra.run.smithi023.stdout: "quincy": [ 2024-03-24T10:50:33.898 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-24T10:50:33.898 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-24T10:50:33.899 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-24T10:50:33.899 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-24T10:50:33.899 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-24T10:50:33.899 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-24T10:50:33.899 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-24T10:50:33.899 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-24T10:50:33.899 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-24T10:50:33.899 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-24T10:50:33.899 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.900 INFO:teuthology.orchestra.run.smithi023.stdout: "reef": [ 2024-03-24T10:50:33.900 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-24T10:50:33.900 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-24T10:50:33.900 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-24T10:50:33.900 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-24T10:50:33.900 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-24T10:50:33.900 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-24T10:50:33.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-24T10:50:33.900 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-24T10:50:33.901 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-24T10:50:33.901 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-24T10:50:33.901 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-24T10:50:33.901 INFO:teuthology.orchestra.run.smithi023.stdout: "squid": [ 2024-03-24T10:50:33.901 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-24T10:50:33.901 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-24T10:50:33.901 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-24T10:50:33.901 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-24T10:50:33.901 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-24T10:50:33.902 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-24T10:50:33.902 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-24T10:50:33.902 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-24T10:50:33.902 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-24T10:50:33.902 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-24T10:50:33.902 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-24T10:50:33.902 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.902 INFO:teuthology.orchestra.run.smithi023.stdout: "last_failure_osd_epoch": 88, 2024-03-24T10:50:33.902 INFO:teuthology.orchestra.run.smithi023.stdout: "active_clients": [ 2024-03-24T10:50:33.903 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.903 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-24T10:50:33.903 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-24T10:50:33.903 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.903 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-24T10:50:33.903 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-24T10:50:33.903 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 963721920 2024-03-24T10:50:33.903 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.904 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-24T10:50:33.904 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.904 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.904 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "libcephsqlite", 2024-03-24T10:50:33.904 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-24T10:50:33.904 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.904 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-24T10:50:33.904 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-24T10:50:33.905 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2580361571 2024-03-24T10:50:33.905 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.905 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-24T10:50:33.905 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.905 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.905 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-24T10:50:33.905 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-24T10:50:33.905 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.905 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-24T10:50:33.906 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-24T10:50:33.906 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3394129176 2024-03-24T10:50:33.906 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.906 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-24T10:50:33.906 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.906 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.907 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-24T10:50:33.907 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-24T10:50:33.907 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.907 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-24T10:50:33.907 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-24T10:50:33.907 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3851630068 2024-03-24T10:50:33.907 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.907 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-24T10:50:33.907 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.908 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.908 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-24T10:50:33.908 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-24T10:50:33.908 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.908 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-24T10:50:33.908 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-24T10:50:33.908 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2199230519 2024-03-24T10:50:33.908 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.908 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-24T10:50:33.909 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:33.909 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.909 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-24T10:50:33.909 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-24T10:50:33.909 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-24T10:50:33.909 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-24T10:50:33.909 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-24T10:50:33.909 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1825899174 2024-03-24T10:50:33.910 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.910 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-24T10:50:33.910 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:33.910 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-24T10:50:33.910 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:50:33.913 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:31 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3730118467' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:50:33.913 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:31 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2397152945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:50:33.913 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:31 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1191979107' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T10:50:33.913 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:32 smithi116 ceph-mon[36248]: pgmap v326: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:33.913 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:31 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3730118467' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:50:33.914 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:31 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2397152945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:50:33.914 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:31 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1191979107' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T10:50:33.914 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:32 smithi117 ceph-mon[36368]: pgmap v326: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:33.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.688+0000 7f196089d640 1 -- 172.21.15.23:0/1191979107 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f19480a2230 msgr2=0x7f19480a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:33.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.688+0000 7f196089d640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f19480a2230 0x7f19480a46f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f195c197b60 tx=0x7f194c009260 comp rx=0 tx=0).stop 2024-03-24T10:50:33.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.688+0000 7f196089d640 1 -- 172.21.15.23:0/1191979107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f195c106680 msgr2=0x7f195c196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:33.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.688+0000 7f196089d640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f195c106680 0x7f195c196a20 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f1954036040 tx=0x7f195402cb70 comp rx=0 tx=0).stop 2024-03-24T10:50:33.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.688+0000 7f196089d640 1 -- 172.21.15.23:0/1191979107 shutdown_connections 2024-03-24T10:50:33.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.688+0000 7f196089d640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f19480a2230 0x7f19480a46f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:33.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.688+0000 7f196089d640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f195c106680 0x7f195c196a20 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:33.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.688+0000 7f196089d640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c100f80 0x7f195c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:33.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.688+0000 7f196089d640 1 --2- 172.21.15.23:0/1191979107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f195c1005c0 0x7f195c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:33.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.688+0000 7f196089d640 1 -- 172.21.15.23:0/1191979107 >> 172.21.15.23:0/1191979107 conn(0x7f195c0776e0 msgr2=0x7f195c0ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:33.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.688+0000 7f196089d640 1 -- 172.21.15.23:0/1191979107 shutdown_connections 2024-03-24T10:50:33.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:50:31.688+0000 7f196089d640 1 -- 172.21.15.23:0/1191979107 wait complete. 2024-03-24T10:50:33.924 DEBUG:tasks.check_counter:Getting stats from a 2024-03-24T10:50:33.924 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-24T10:50:33.925 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:31 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3730118467' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:50:33.925 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:31 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2397152945' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:50:33.925 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:31 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1191979107' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T10:50:33.925 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:32 smithi023 ceph-mon[32466]: pgmap v326: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:34.181 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:50:34.876 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:34 smithi023 ceph-mon[32466]: pgmap v327: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:35.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:34 smithi117 ceph-mon[36368]: pgmap v327: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:35.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:34 smithi116 ceph-mon[36248]: pgmap v327: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:35.915 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:50:35.915 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T10:50:35.915 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 213, 2024-03-24T10:50:35.915 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 203, 2024-03-24T10:50:35.916 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1430016, 2024-03-24T10:50:35.916 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 124704, 2024-03-24T10:50:35.916 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-24T10:50:35.916 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-24T10:50:35.916 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.038785100, 2024-03-24T10:50:35.916 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.011051092, 2024-03-24T10:50:35.916 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.031994593, 2024-03-24T10:50:35.917 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003570380, 2024-03-24T10:50:35.917 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:35.917 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 202, 2024-03-24T10:50:35.917 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.008087270, 2024-03-24T10:50:35.917 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000040035 2024-03-24T10:50:35.917 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:35.917 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:35.917 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:35.917 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:35.918 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:35.918 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:35.918 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1430016, 2024-03-24T10:50:35.918 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 124704 2024-03-24T10:50:35.918 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:35.918 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T10:50:35.918 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 6, 2024-03-24T10:50:35.918 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 157, 2024-03-24T10:50:35.919 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 3344, 2024-03-24T10:50:35.919 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 183104, 2024-03-24T10:50:35.919 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-24T10:50:35.919 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-24T10:50:35.919 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.012428744, 2024-03-24T10:50:35.919 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.007821435, 2024-03-24T10:50:35.919 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.002573525, 2024-03-24T10:50:35.919 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T10:50:35.919 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:35.920 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 155, 2024-03-24T10:50:35.920 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.013332036, 2024-03-24T10:50:35.920 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000086013 2024-03-24T10:50:35.920 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:35.920 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:35.920 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:35.920 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:35.920 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:35.921 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:35.921 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-24T10:50:35.921 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 183104 2024-03-24T10:50:35.921 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:35.921 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T10:50:35.921 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 0, 2024-03-24T10:50:35.921 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 2, 2024-03-24T10:50:35.921 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 0, 2024-03-24T10:50:35.922 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 0, 2024-03-24T10:50:35.922 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-24T10:50:35.922 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-24T10:50:35.922 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.000362095, 2024-03-24T10:50:35.922 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T10:50:35.922 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000304845, 2024-03-24T10:50:35.922 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T10:50:35.922 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:35.923 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:35.923 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:35.923 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:35.923 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:35.923 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:35.923 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:35.923 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:35.923 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:35.923 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:35.924 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T10:50:35.924 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T10:50:35.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:35.924 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-24T10:50:35.924 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-24T10:50:35.924 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-24T10:50:35.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:35.924 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-24T10:50:35.924 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-24T10:50:35.925 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-24T10:50:35.925 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T10:50:35.925 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-24T10:50:35.925 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T10:50:35.925 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-24T10:50:35.925 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T10:50:35.925 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T10:50:35.926 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T10:50:35.926 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T10:50:35.926 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T10:50:35.926 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-24T10:50:35.927 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T10:50:35.927 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T10:50:35.927 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-24T10:50:35.927 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-24T10:50:35.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-24T10:50:35.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-24T10:50:35.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T10:50:35.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T10:50:35.929 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T10:50:35.929 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T10:50:35.929 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T10:50:35.930 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-24T10:50:35.930 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-24T10:50:35.930 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-24T10:50:35.930 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T10:50:35.931 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T10:50:35.931 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-24T10:50:35.931 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-24T10:50:35.931 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-24T10:50:35.932 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-24T10:50:35.932 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T10:50:35.932 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-24T10:50:35.932 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T10:50:35.933 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-24T10:50:35.933 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 148977, 2024-03-24T10:50:35.933 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 30, 2024-03-24T10:50:35.933 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-24T10:50:35.934 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-24T10:50:35.934 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-24T10:50:35.934 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-24T10:50:35.934 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-24T10:50:35.935 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-24T10:50:35.935 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-24T10:50:35.935 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-24T10:50:35.935 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-24T10:50:35.936 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-24T10:50:35.936 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T10:50:35.936 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-24T10:50:35.936 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-24T10:50:35.937 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-24T10:50:35.937 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-24T10:50:35.937 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-24T10:50:35.937 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-24T10:50:35.938 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-24T10:50:35.938 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-24T10:50:35.938 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-24T10:50:35.938 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:35.938 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T10:50:35.938 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-24T10:50:35.938 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-24T10:50:35.938 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-24T10:50:35.938 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 215, 2024-03-24T10:50:35.939 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1399996, 2024-03-24T10:50:35.939 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:35.939 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 215, 2024-03-24T10:50:35.939 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-24T10:50:35.939 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-24T10:50:35.939 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 215, 2024-03-24T10:50:35.939 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1399996, 2024-03-24T10:50:35.939 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-24T10:50:35.939 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:35.940 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:35.940 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:35.940 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:35.940 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:35.940 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:50:36.662 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-24T10:50:36.662 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-24T10:50:36.663 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-24T10:50:36.663 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-24T10:50:36.663 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-24T10:50:36.663 DEBUG:tasks.check_counter:Getting stats from d 2024-03-24T10:50:36.663 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-24T10:50:36.913 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:50:36.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:36 smithi023 ceph-mon[32466]: pgmap v328: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:37.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:36 smithi117 ceph-mon[36368]: pgmap v328: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:37.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:36 smithi116 ceph-mon[36248]: pgmap v328: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:38.421 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:50:38.421 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T10:50:38.422 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 4, 2024-03-24T10:50:38.422 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 4, 2024-03-24T10:50:38.422 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 3024, 2024-03-24T10:50:38.422 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 448, 2024-03-24T10:50:38.422 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-24T10:50:38.422 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-24T10:50:38.422 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.001616481, 2024-03-24T10:50:38.422 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000041272, 2024-03-24T10:50:38.423 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000615557, 2024-03-24T10:50:38.423 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T10:50:38.423 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:38.423 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 3, 2024-03-24T10:50:38.423 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000666053, 2024-03-24T10:50:38.423 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000222017 2024-03-24T10:50:38.423 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:38.423 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:38.423 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:38.424 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:38.424 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:38.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:38.424 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-24T10:50:38.424 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 448 2024-03-24T10:50:38.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:38.424 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T10:50:38.424 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 2, 2024-03-24T10:50:38.424 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 159, 2024-03-24T10:50:38.425 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 320, 2024-03-24T10:50:38.425 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 183360, 2024-03-24T10:50:38.425 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-24T10:50:38.425 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-24T10:50:38.425 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.012604517, 2024-03-24T10:50:38.425 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.008514337, 2024-03-24T10:50:38.425 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001781411, 2024-03-24T10:50:38.425 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T10:50:38.425 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:38.426 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 156, 2024-03-24T10:50:38.426 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.009669438, 2024-03-24T10:50:38.426 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000061983 2024-03-24T10:50:38.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:38.426 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:38.426 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:38.426 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:38.426 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:38.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:38.427 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T10:50:38.427 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 183360 2024-03-24T10:50:38.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:38.427 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T10:50:38.427 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 218, 2024-03-24T10:50:38.427 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 208, 2024-03-24T10:50:38.427 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1431376, 2024-03-24T10:50:38.427 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 127904, 2024-03-24T10:50:38.427 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-24T10:50:38.428 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-24T10:50:38.428 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.038037247, 2024-03-24T10:50:38.428 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.010732032, 2024-03-24T10:50:38.428 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.032626572, 2024-03-24T10:50:38.428 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003648628, 2024-03-24T10:50:38.428 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:38.428 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 207, 2024-03-24T10:50:38.428 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.007769189, 2024-03-24T10:50:38.428 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000037532 2024-03-24T10:50:38.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:38.429 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:38.429 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:38.429 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:38.429 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:38.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:38.429 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1431376, 2024-03-24T10:50:38.429 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 127904 2024-03-24T10:50:38.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:38.429 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-24T10:50:38.430 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-24T10:50:38.430 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-24T10:50:38.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:38.430 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-24T10:50:38.430 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-24T10:50:38.430 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-24T10:50:38.430 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T10:50:38.430 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-24T10:50:38.430 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T10:50:38.431 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-24T10:50:38.431 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T10:50:38.431 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T10:50:38.431 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T10:50:38.431 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T10:50:38.431 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T10:50:38.431 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-24T10:50:38.431 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T10:50:38.431 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T10:50:38.432 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-24T10:50:38.432 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-24T10:50:38.432 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-24T10:50:38.432 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-24T10:50:38.432 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T10:50:38.432 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T10:50:38.432 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T10:50:38.432 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T10:50:38.432 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T10:50:38.432 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-24T10:50:38.433 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-24T10:50:38.433 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-24T10:50:38.433 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T10:50:38.433 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T10:50:38.433 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-24T10:50:38.433 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-24T10:50:38.433 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-24T10:50:38.433 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-24T10:50:38.433 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T10:50:38.434 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-24T10:50:38.434 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T10:50:38.434 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-24T10:50:38.434 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 149017, 2024-03-24T10:50:38.434 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 31, 2024-03-24T10:50:38.434 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-24T10:50:38.434 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-24T10:50:38.434 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-24T10:50:38.434 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-24T10:50:38.435 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-24T10:50:38.435 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-24T10:50:38.435 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-24T10:50:38.435 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-24T10:50:38.435 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-24T10:50:38.435 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-24T10:50:38.435 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T10:50:38.435 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-24T10:50:38.435 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-24T10:50:38.435 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-24T10:50:38.436 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-24T10:50:38.436 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-24T10:50:38.436 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-24T10:50:38.436 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-24T10:50:38.436 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-24T10:50:38.436 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-24T10:50:38.436 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:38.436 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T10:50:38.436 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-24T10:50:38.437 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-24T10:50:38.437 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-24T10:50:38.437 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 220, 2024-03-24T10:50:38.437 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1400646, 2024-03-24T10:50:38.437 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:38.437 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 220, 2024-03-24T10:50:38.437 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-24T10:50:38.437 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-24T10:50:38.437 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 220, 2024-03-24T10:50:38.437 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1400646, 2024-03-24T10:50:38.438 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-24T10:50:38.438 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:38.438 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:38.438 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:38.438 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:38.438 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:38.438 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:50:39.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:38 smithi023 ceph-mon[32466]: pgmap v329: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:39.059 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-24T10:50:39.059 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-24T10:50:39.059 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-24T10:50:39.059 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-24T10:50:39.059 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-24T10:50:39.060 DEBUG:tasks.check_counter:Getting stats from g 2024-03-24T10:50:39.060 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-24T10:50:39.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:38 smithi117 ceph-mon[36368]: pgmap v329: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:39.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:38 smithi116 ceph-mon[36248]: pgmap v329: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:39.308 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:50:40.918 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:40 smithi023 ceph-mon[32466]: pgmap v330: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:40.929 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:50:40.929 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T10:50:40.929 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 0, 2024-03-24T10:50:40.929 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 2, 2024-03-24T10:50:40.929 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 0, 2024-03-24T10:50:40.930 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 0, 2024-03-24T10:50:40.930 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-24T10:50:40.930 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-24T10:50:40.930 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.001255436, 2024-03-24T10:50:40.930 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T10:50:40.930 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000300792, 2024-03-24T10:50:40.930 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T10:50:40.930 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:40.931 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:40.931 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:40.931 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:40.931 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:40.931 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:40.931 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:40.931 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:40.931 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:40.932 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:40.932 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T10:50:40.932 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T10:50:40.932 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:40.932 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T10:50:40.932 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 2, 2024-03-24T10:50:40.932 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 156, 2024-03-24T10:50:40.932 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 320, 2024-03-24T10:50:40.932 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 182832, 2024-03-24T10:50:40.933 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-24T10:50:40.933 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-24T10:50:40.933 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.012290523, 2024-03-24T10:50:40.933 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.008006723, 2024-03-24T10:50:40.933 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.002303324, 2024-03-24T10:50:40.933 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T10:50:40.933 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:40.933 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 153, 2024-03-24T10:50:40.934 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.009416925, 2024-03-24T10:50:40.934 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000061548 2024-03-24T10:50:40.934 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:40.934 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:40.934 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:40.934 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:40.934 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:40.934 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:40.934 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T10:50:40.935 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 182832 2024-03-24T10:50:40.935 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:40.935 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T10:50:40.935 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 219, 2024-03-24T10:50:40.935 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 207, 2024-03-24T10:50:40.935 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1433584, 2024-03-24T10:50:40.935 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 126432, 2024-03-24T10:50:40.935 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-24T10:50:40.935 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-24T10:50:40.936 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.036224279, 2024-03-24T10:50:40.936 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.011422357, 2024-03-24T10:50:40.936 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.029669473, 2024-03-24T10:50:40.936 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003602770, 2024-03-24T10:50:40.936 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:40.936 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 207, 2024-03-24T10:50:40.936 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.008123778, 2024-03-24T10:50:40.936 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000039245 2024-03-24T10:50:40.937 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:40.937 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:40.937 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:40.937 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:40.937 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:40.937 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:40.937 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1433584, 2024-03-24T10:50:40.937 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 126432 2024-03-24T10:50:40.937 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:40.938 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-24T10:50:40.938 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-24T10:50:40.938 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-24T10:50:40.938 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:40.938 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-24T10:50:40.938 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-24T10:50:40.938 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-24T10:50:40.938 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T10:50:40.939 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-24T10:50:40.939 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T10:50:40.939 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-24T10:50:40.939 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T10:50:40.939 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T10:50:40.939 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T10:50:40.939 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T10:50:40.939 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T10:50:40.940 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-24T10:50:40.940 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T10:50:40.940 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T10:50:40.940 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-24T10:50:40.940 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-24T10:50:40.940 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-24T10:50:40.940 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-24T10:50:40.940 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T10:50:40.941 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T10:50:40.941 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T10:50:40.941 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T10:50:40.941 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T10:50:40.941 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-24T10:50:40.941 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-24T10:50:40.941 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-24T10:50:40.942 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T10:50:40.942 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T10:50:40.942 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-24T10:50:40.942 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-24T10:50:40.942 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-24T10:50:40.942 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-24T10:50:40.942 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T10:50:40.942 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-24T10:50:40.943 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T10:50:40.943 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-24T10:50:40.943 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 149057, 2024-03-24T10:50:40.943 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 32, 2024-03-24T10:50:40.943 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-24T10:50:40.943 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-24T10:50:40.943 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-24T10:50:40.943 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-24T10:50:40.944 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-24T10:50:40.944 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-24T10:50:40.944 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-24T10:50:40.944 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-24T10:50:40.944 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-24T10:50:40.944 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-24T10:50:40.944 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T10:50:40.944 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-24T10:50:40.945 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-24T10:50:40.945 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-24T10:50:40.945 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-24T10:50:40.945 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-24T10:50:40.945 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-24T10:50:40.945 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-24T10:50:40.945 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-24T10:50:40.945 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-24T10:50:40.945 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:40.946 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T10:50:40.946 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-24T10:50:40.946 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-24T10:50:40.946 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-24T10:50:40.946 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 217, 2024-03-24T10:50:40.946 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1400256, 2024-03-24T10:50:40.946 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:40.946 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 217, 2024-03-24T10:50:40.947 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-24T10:50:40.947 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-24T10:50:40.947 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 217, 2024-03-24T10:50:40.947 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1400256, 2024-03-24T10:50:40.947 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-24T10:50:40.947 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:40.947 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:40.947 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:40.948 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:40.948 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:40.948 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:50:41.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:40 smithi117 ceph-mon[36368]: pgmap v330: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:41.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:40 smithi116 ceph-mon[36248]: pgmap v330: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:41.791 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-24T10:50:41.791 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-24T10:50:41.791 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-24T10:50:41.791 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-24T10:50:41.791 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-24T10:50:41.791 DEBUG:tasks.check_counter:Getting stats from j 2024-03-24T10:50:41.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-24T10:50:42.039 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T10:50:43.145 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:42 smithi023 ceph-mon[32466]: pgmap v331: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:43.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:42 smithi117 ceph-mon[36368]: pgmap v331: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:43.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:42 smithi116 ceph-mon[36248]: pgmap v331: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:43.675 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-24T10:50:43.675 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T10:50:43.676 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 4, 2024-03-24T10:50:43.676 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 4, 2024-03-24T10:50:43.676 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 3024, 2024-03-24T10:50:43.676 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 448, 2024-03-24T10:50:43.676 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-24T10:50:43.676 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-24T10:50:43.676 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.001651305, 2024-03-24T10:50:43.676 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000044476, 2024-03-24T10:50:43.677 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000687310, 2024-03-24T10:50:43.677 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T10:50:43.677 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:43.677 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 3, 2024-03-24T10:50:43.677 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000876063, 2024-03-24T10:50:43.677 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000292021 2024-03-24T10:50:43.677 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:43.677 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:43.678 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:43.678 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:43.678 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:43.678 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:43.678 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-24T10:50:43.678 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 448 2024-03-24T10:50:43.678 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:43.678 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T10:50:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 2, 2024-03-24T10:50:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 161, 2024-03-24T10:50:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 320, 2024-03-24T10:50:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 183712, 2024-03-24T10:50:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-24T10:50:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-24T10:50:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.013190196, 2024-03-24T10:50:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.008868434, 2024-03-24T10:50:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001996299, 2024-03-24T10:50:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T10:50:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 158, 2024-03-24T10:50:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.008323157, 2024-03-24T10:50:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000052678 2024-03-24T10:50:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:43.681 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:43.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:43.681 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T10:50:43.681 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 183712 2024-03-24T10:50:43.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:43.681 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T10:50:43.681 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 220, 2024-03-24T10:50:43.681 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 210, 2024-03-24T10:50:43.681 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1431920, 2024-03-24T10:50:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 129184, 2024-03-24T10:50:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-24T10:50:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-24T10:50:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.037394506, 2024-03-24T10:50:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.011903975, 2024-03-24T10:50:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.030606495, 2024-03-24T10:50:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003881674, 2024-03-24T10:50:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 209, 2024-03-24T10:50:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.008331277, 2024-03-24T10:50:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000039862 2024-03-24T10:50:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1431920, 2024-03-24T10:50:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 129184 2024-03-24T10:50:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-24T10:50:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-24T10:50:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-24T10:50:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-24T10:50:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-24T10:50:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-24T10:50:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T10:50:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-24T10:50:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T10:50:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-24T10:50:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T10:50:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T10:50:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T10:50:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T10:50:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T10:50:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-24T10:50:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T10:50:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T10:50:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-24T10:50:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-24T10:50:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-24T10:50:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-24T10:50:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T10:50:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T10:50:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T10:50:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T10:50:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T10:50:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-24T10:50:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-24T10:50:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-24T10:50:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T10:50:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T10:50:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-24T10:50:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-24T10:50:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-24T10:50:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-24T10:50:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T10:50:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-24T10:50:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T10:50:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-24T10:50:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 148977, 2024-03-24T10:50:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 30, 2024-03-24T10:50:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-24T10:50:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-24T10:50:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-24T10:50:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-24T10:50:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-24T10:50:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-24T10:50:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-24T10:50:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-24T10:50:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-24T10:50:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-24T10:50:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T10:50:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-24T10:50:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-24T10:50:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-24T10:50:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-24T10:50:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-24T10:50:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-24T10:50:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-24T10:50:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-24T10:50:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-24T10:50:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-24T10:50:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T10:50:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-24T10:50:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-24T10:50:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-24T10:50:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 222, 2024-03-24T10:50:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1400906, 2024-03-24T10:50:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 222, 2024-03-24T10:50:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-24T10:50:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-24T10:50:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 222, 2024-03-24T10:50:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1400906, 2024-03-24T10:50:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-24T10:50:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-24T10:50:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-24T10:50:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-24T10:50:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:43.694 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-24T10:50:43.694 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-24T10:50:44.315 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-24T10:50:44.315 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-24T10:50:44.315 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-24T10:50:44.315 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-24T10:50:44.315 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-24T10:50:44.315 DEBUG:tasks.check_counter:Getting stats from b 2024-03-24T10:50:44.315 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-24T10:50:44.558 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T10:50:44.999 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:44 smithi116 ceph-mon[36248]: pgmap v332: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:45.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:44 smithi023 ceph-mon[32466]: pgmap v332: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:45.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:44 smithi117 ceph-mon[36368]: pgmap v332: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:46.461 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T10:50:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T10:50:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 1952, 2024-03-24T10:50:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 1126, 2024-03-24T10:50:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 2123089, 2024-03-24T10:50:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 3005325, 2024-03-24T10:50:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 229, 2024-03-24T10:50:46.463 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 6, 2024-03-24T10:50:46.463 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.201513910, 2024-03-24T10:50:46.463 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.037075067, 2024-03-24T10:50:46.463 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.121073456, 2024-03-24T10:50:46.463 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.042537331, 2024-03-24T10:50:46.463 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:46.463 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1121, 2024-03-24T10:50:46.463 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.278410684, 2024-03-24T10:50:46.464 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000248359 2024-03-24T10:50:46.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.464 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:46.464 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1438, 2024-03-24T10:50:46.464 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000183540, 2024-03-24T10:50:46.464 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000127 2024-03-24T10:50:46.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.464 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1968960, 2024-03-24T10:50:46.464 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 277904 2024-03-24T10:50:46.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.465 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T10:50:46.465 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 174221, 2024-03-24T10:50:46.465 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 110285, 2024-03-24T10:50:46.465 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 67346893, 2024-03-24T10:50:46.465 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 24281595, 2024-03-24T10:50:46.466 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 24, 2024-03-24T10:50:46.466 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 6, 2024-03-24T10:50:46.466 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 8.082672053, 2024-03-24T10:50:46.466 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 3.634691921, 2024-03-24T10:50:46.466 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 5.963862725, 2024-03-24T10:50:46.466 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.045334351, 2024-03-24T10:50:46.466 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:46.466 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 110287, 2024-03-24T10:50:46.467 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 1.786976878, 2024-03-24T10:50:46.467 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000016202 2024-03-24T10:50:46.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.467 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:46.467 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 276713, 2024-03-24T10:50:46.467 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.034571241, 2024-03-24T10:50:46.467 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000124 2024-03-24T10:50:46.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.467 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 10171072, 2024-03-24T10:50:46.468 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 11617776 2024-03-24T10:50:46.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.468 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T10:50:46.468 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 2063, 2024-03-24T10:50:46.468 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 1945, 2024-03-24T10:50:46.468 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 823434, 2024-03-24T10:50:46.468 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 5190596, 2024-03-24T10:50:46.468 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 12, 2024-03-24T10:50:46.468 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 6, 2024-03-24T10:50:46.469 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.151363851, 2024-03-24T10:50:46.469 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.057622335, 2024-03-24T10:50:46.469 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.043883815, 2024-03-24T10:50:46.469 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.043854685, 2024-03-24T10:50:46.469 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:46.469 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1943, 2024-03-24T10:50:46.469 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.602880153, 2024-03-24T10:50:46.469 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000310283 2024-03-24T10:50:46.470 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.470 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:46.470 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1335, 2024-03-24T10:50:46.470 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000218803, 2024-03-24T10:50:46.470 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000163 2024-03-24T10:50:46.470 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.470 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 42560, 2024-03-24T10:50:46.470 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 1936 2024-03-24T10:50:46.470 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.471 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-24T10:50:46.471 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 1, 2024-03-24T10:50:46.471 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-24T10:50:46.471 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.471 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-MDSRank": { 2024-03-24T10:50:46.471 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-24T10:50:46.471 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-24T10:50:46.471 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1867, 2024-03-24T10:50:46.471 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 1.020225801, 2024-03-24T10:50:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000546451 2024-03-24T10:50:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-PurgeQueue": { 2024-03-24T10:50:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-24T10:50:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-24T10:50:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 322, 2024-03-24T10:50:46.472 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.054057959, 2024-03-24T10:50:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000167881 2024-03-24T10:50:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-24T10:50:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: "request": 23258, 2024-03-24T10:50:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: "reply": 1610, 2024-03-24T10:50:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: "reply_latency": { 2024-03-24T10:50:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1610, 2024-03-24T10:50:46.473 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 4.322496869, 2024-03-24T10:50:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.002684780 2024-03-24T10:50:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_reply": 0, 2024-03-24T10:50:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: "forward": 21648, 2024-03-24T10:50:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_complete": 5, 2024-03-24T10:50:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_keys": 2, 2024-03-24T10:50:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_commit": 177, 2024-03-24T10:50:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_split": 0, 2024-03-24T10:50:46.474 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_merge": 0, 2024-03-24T10:50:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes": 47, 2024-03-24T10:50:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_top": 17, 2024-03-24T10:50:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_bottom": 9, 2024-03-24T10:50:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pin_tail": 21, 2024-03-24T10:50:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pinned": 22, 2024-03-24T10:50:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_expired": 172, 2024-03-24T10:50:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_with_caps": 6, 2024-03-24T10:50:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": 6, 2024-03-24T10:50:46.475 INFO:teuthology.orchestra.run.smithi116.stdout: "subtrees": 4, 2024-03-24T10:50:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse": 23852, 2024-03-24T10:50:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_hit": 1461, 2024-03-24T10:50:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_forward": 21648, 2024-03-24T10:50:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_discover": 0, 2024-03-24T10:50:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_dir_fetch": 0, 2024-03-24T10:50:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_remote_ino": 0, 2024-03-24T10:50:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_lock": 17, 2024-03-24T10:50:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: "load_cent": 0, 2024-03-24T10:50:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: "q": 0, 2024-03-24T10:50:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: "exported": 9, 2024-03-24T10:50:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: "exported_inodes": 158, 2024-03-24T10:50:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: "imported": 0, 2024-03-24T10:50:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: "imported_inodes": 0, 2024-03-24T10:50:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_dir_fetch": 0, 2024-03-24T10:50:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_backtrace_fetch": 239, 2024-03-24T10:50:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_peer_discover": 0, 2024-03-24T10:50:46.477 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rfiles": 68, 2024-03-24T10:50:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rbytes": 21196213, 2024-03-24T10:50:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rsnaps": 6, 2024-03-24T10:50:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_fetch": 239, 2024-03-24T10:50:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_set_tag": 0, 2024-03-24T10:50:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T10:50:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_inotable_repaired": 0, 2024-03-24T10:50:46.478 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_inodes": 159, 2024-03-24T10:50:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_base_inodes": 20, 2024-03-24T10:50:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dirfrag_rstats": 139, 2024-03-24T10:50:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_file_inodes": 80, 2024-03-24T10:50:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_inode_file_caps": 1, 2024-03-24T10:50:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_revoke": 6395, 2024-03-24T10:50:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_grant": 7367, 2024-03-24T10:50:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T10:50:46.479 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T10:50:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T10:50:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps": 6643, 2024-03-24T10:50:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps_dirty": 167, 2024-03-24T10:50:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_cap_release": 74, 2024-03-24T10:50:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: "process_request_cap_release": 963 2024-03-24T10:50:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_cache": { 2024-03-24T10:50:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update": 2, 2024-03-24T10:50:46.480 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update_receipt": 0, 2024-03-24T10:50:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_try_discover": 0, 2024-03-24T10:50:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_send_discover": 0, 2024-03-24T10:50:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_handle_discover": 18, 2024-03-24T10:50:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays": 0, 2024-03-24T10:50:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_delayed": 0, 2024-03-24T10:50:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_enqueuing": 0, 2024-03-24T10:50:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_created": 220, 2024-03-24T10:50:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_enqueued": 220, 2024-03-24T10:50:46.481 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_reintegrated": 0, 2024-03-24T10:50:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_migrated": 0, 2024-03-24T10:50:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_processing": 0, 2024-03-24T10:50:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_enqueued": 0, 2024-03-24T10:50:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_prioritized": 0, 2024-03-24T10:50:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_started": 0, 2024-03-24T10:50:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_completed": 0, 2024-03-24T10:50:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_path": 0, 2024-03-24T10:50:46.482 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_inode": 0, 2024-03-24T10:50:46.483 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_enqueue_scrub": 20, 2024-03-24T10:50:46.483 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_exportdir": 10, 2024-03-24T10:50:46.483 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_flush": 0, 2024-03-24T10:50:46.483 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragmentdir": 0, 2024-03-24T10:50:46.483 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragstats": 0, 2024-03-24T10:50:46.484 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_inodestats": 0 2024-03-24T10:50:46.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.484 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_log": { 2024-03-24T10:50:46.484 INFO:teuthology.orchestra.run.smithi116.stdout: "evlrg": 0, 2024-03-24T10:50:46.484 INFO:teuthology.orchestra.run.smithi116.stdout: "evadd": 1256, 2024-03-24T10:50:46.484 INFO:teuthology.orchestra.run.smithi116.stdout: "evex": 1219, 2024-03-24T10:50:46.484 INFO:teuthology.orchestra.run.smithi116.stdout: "evtrm": 1219, 2024-03-24T10:50:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: "ev": 37, 2024-03-24T10:50:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: "evexg": 0, 2024-03-24T10:50:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: "evexd": 4, 2024-03-24T10:50:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: "segadd": 37, 2024-03-24T10:50:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: "segex": 36, 2024-03-24T10:50:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: "segtrm": 36, 2024-03-24T10:50:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: "seg": 1, 2024-03-24T10:50:46.485 INFO:teuthology.orchestra.run.smithi116.stdout: "segmjr": 1, 2024-03-24T10:50:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: "segexg": 0, 2024-03-24T10:50:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: "segexd": 1, 2024-03-24T10:50:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: "expos": 8068325, 2024-03-24T10:50:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: "wrpos": 8350225, 2024-03-24T10:50:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: "rdpos": 0, 2024-03-24T10:50:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: "jlat": { 2024-03-24T10:50:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 607, 2024-03-24T10:50:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 1.043106119, 2024-03-24T10:50:46.486 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001718461 2024-03-24T10:50:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: "replayed": 0 2024-03-24T10:50:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_mem": { 2024-03-24T10:50:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 51, 2024-03-24T10:50:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: "ino+": 384, 2024-03-24T10:50:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: "ino-": 333, 2024-03-24T10:50:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: "dir": 47, 2024-03-24T10:50:46.487 INFO:teuthology.orchestra.run.smithi116.stdout: "dir+": 59, 2024-03-24T10:50:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: "dir-": 12, 2024-03-24T10:50:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: "dn": 47, 2024-03-24T10:50:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: "dn+": 664, 2024-03-24T10:50:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: "dn-": 617, 2024-03-24T10:50:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: "cap": 6, 2024-03-24T10:50:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: "cap+": 335, 2024-03-24T10:50:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: "cap-": 329, 2024-03-24T10:50:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: "rss": 60556, 2024-03-24T10:50:46.488 INFO:teuthology.orchestra.run.smithi116.stdout: "heap": 231804 2024-03-24T10:50:46.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.489 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_server": { 2024-03-24T10:50:46.489 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_client_request": 23645, 2024-03-24T10:50:46.489 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_server_request": 1, 2024-03-24T10:50:46.489 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_request": 23258, 2024-03-24T10:50:46.489 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_session": 78, 2024-03-24T10:50:46.489 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_peer_request": 3, 2024-03-24T10:50:46.489 INFO:teuthology.orchestra.run.smithi116.stdout: "req_create_latency": { 2024-03-24T10:50:46.489 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 308, 2024-03-24T10:50:46.490 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.383239321, 2024-03-24T10:50:46.490 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001244283 2024-03-24T10:50:46.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.490 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getattr_latency": { 2024-03-24T10:50:46.490 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 115, 2024-03-24T10:50:46.490 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.106195003, 2024-03-24T10:50:46.490 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000923434 2024-03-24T10:50:46.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.491 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getfilelock_latency": { 2024-03-24T10:50:46.491 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.491 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.491 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.491 INFO:teuthology.orchestra.run.smithi116.stdout: "req_link_latency": { 2024-03-24T10:50:46.491 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.491 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.491 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.492 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookup_latency": { 2024-03-24T10:50:46.492 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 452, 2024-03-24T10:50:46.492 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 2.312358444, 2024-03-24T10:50:46.492 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.005115837 2024-03-24T10:50:46.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.492 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookuphash_latency": { 2024-03-24T10:50:46.492 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.492 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.493 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.493 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupino_latency": { 2024-03-24T10:50:46.493 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1, 2024-03-24T10:50:46.493 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000131144, 2024-03-24T10:50:46.493 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000131144 2024-03-24T10:50:46.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.493 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupname_latency": { 2024-03-24T10:50:46.494 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.494 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.494 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.494 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupparent_latency": { 2024-03-24T10:50:46.494 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.494 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.494 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.495 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupsnap_latency": { 2024-03-24T10:50:46.495 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.495 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.495 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.495 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lssnap_latency": { 2024-03-24T10:50:46.496 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 10, 2024-03-24T10:50:46.496 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.001855364, 2024-03-24T10:50:46.496 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000185536 2024-03-24T10:50:46.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.496 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mkdir_latency": { 2024-03-24T10:50:46.496 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 20, 2024-03-24T10:50:46.497 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.021588101, 2024-03-24T10:50:46.497 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001079405 2024-03-24T10:50:46.497 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.497 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mknod_latency": { 2024-03-24T10:50:46.497 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.497 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.497 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.497 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.498 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mksnap_latency": { 2024-03-24T10:50:46.498 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 10, 2024-03-24T10:50:46.498 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.099781396, 2024-03-24T10:50:46.498 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.009978139 2024-03-24T10:50:46.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.498 INFO:teuthology.orchestra.run.smithi116.stdout: "req_open_latency": { 2024-03-24T10:50:46.498 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1, 2024-03-24T10:50:46.498 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000273364, 2024-03-24T10:50:46.498 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000273364 2024-03-24T10:50:46.499 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.499 INFO:teuthology.orchestra.run.smithi116.stdout: "req_readdir_latency": { 2024-03-24T10:50:46.499 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 398, 2024-03-24T10:50:46.499 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.936908497, 2024-03-24T10:50:46.499 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.002354041 2024-03-24T10:50:46.499 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.499 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rename_latency": { 2024-03-24T10:50:46.499 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 54, 2024-03-24T10:50:46.500 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.084746789, 2024-03-24T10:50:46.500 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001569384 2024-03-24T10:50:46.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.500 INFO:teuthology.orchestra.run.smithi116.stdout: "req_renamesnap_latency": { 2024-03-24T10:50:46.500 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.500 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.500 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.500 INFO:teuthology.orchestra.run.smithi116.stdout: "req_snapdiff_latency": { 2024-03-24T10:50:46.501 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.501 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.501 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.501 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.501 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmdir_latency": { 2024-03-24T10:50:46.501 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.501 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.501 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.502 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.502 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmsnap_latency": { 2024-03-24T10:50:46.502 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 4, 2024-03-24T10:50:46.502 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.024754009, 2024-03-24T10:50:46.502 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.006188502 2024-03-24T10:50:46.502 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.502 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmxattr_latency": { 2024-03-24T10:50:46.502 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.502 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.503 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.503 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.503 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setattr_latency": { 2024-03-24T10:50:46.503 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1, 2024-03-24T10:50:46.503 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.001643729, 2024-03-24T10:50:46.503 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001643729 2024-03-24T10:50:46.503 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.503 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setdirlayout_latency": { 2024-03-24T10:50:46.503 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.504 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.504 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.504 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setfilelock_latency": { 2024-03-24T10:50:46.504 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 8, 2024-03-24T10:50:46.504 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.002766927, 2024-03-24T10:50:46.504 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000345865 2024-03-24T10:50:46.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.504 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setlayout_latency": { 2024-03-24T10:50:46.505 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.505 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.505 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.505 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setxattr_latency": { 2024-03-24T10:50:46.505 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 8, 2024-03-24T10:50:46.505 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.017327396, 2024-03-24T10:50:46.505 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.002165924 2024-03-24T10:50:46.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.506 INFO:teuthology.orchestra.run.smithi116.stdout: "req_symlink_latency": { 2024-03-24T10:50:46.506 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.506 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.506 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.506 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.506 INFO:teuthology.orchestra.run.smithi116.stdout: "req_unlink_latency": { 2024-03-24T10:50:46.506 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 220, 2024-03-24T10:50:46.506 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.328927385, 2024-03-24T10:50:46.507 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001495124 2024-03-24T10:50:46.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.507 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_revoke_eviction": 0, 2024-03-24T10:50:46.507 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_acquisition_throttle": 0, 2024-03-24T10:50:46.507 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getvxattr_latency": { 2024-03-24T10:50:46.507 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.507 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.507 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.507 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:46.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.508 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_sessions": { 2024-03-24T10:50:46.508 INFO:teuthology.orchestra.run.smithi116.stdout: "session_count": 2, 2024-03-24T10:50:46.508 INFO:teuthology.orchestra.run.smithi116.stdout: "session_add": 6, 2024-03-24T10:50:46.508 INFO:teuthology.orchestra.run.smithi116.stdout: "session_remove": 4, 2024-03-24T10:50:46.508 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_open": 2, 2024-03-24T10:50:46.508 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_stale": 0, 2024-03-24T10:50:46.508 INFO:teuthology.orchestra.run.smithi116.stdout: "total_load": 32, 2024-03-24T10:50:46.508 INFO:teuthology.orchestra.run.smithi116.stdout: "average_load": 16, 2024-03-24T10:50:46.509 INFO:teuthology.orchestra.run.smithi116.stdout: "avg_session_uptime": 613, 2024-03-24T10:50:46.509 INFO:teuthology.orchestra.run.smithi116.stdout: "mdthresh_evicted": 0 2024-03-24T10:50:46.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.509 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-24T10:50:46.509 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-24T10:50:46.509 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-24T10:50:46.509 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T10:50:46.509 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-24T10:50:46.509 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T10:50:46.510 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-24T10:50:46.510 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T10:50:46.510 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T10:50:46.510 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T10:50:46.510 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T10:50:46.510 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T10:50:46.510 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-24T10:50:46.510 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T10:50:46.511 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T10:50:46.511 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-24T10:50:46.511 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-24T10:50:46.511 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-24T10:50:46.511 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-24T10:50:46.511 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T10:50:46.511 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T10:50:46.511 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T10:50:46.511 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T10:50:46.512 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T10:50:46.512 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-24T10:50:46.512 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-24T10:50:46.512 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-24T10:50:46.512 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T10:50:46.512 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T10:50:46.512 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-24T10:50:46.512 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-24T10:50:46.512 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-24T10:50:46.513 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-24T10:50:46.513 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T10:50:46.513 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-24T10:50:46.513 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T10:50:46.513 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-24T10:50:46.513 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 315396, 2024-03-24T10:50:46.513 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 152, 2024-03-24T10:50:46.513 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 352, 2024-03-24T10:50:46.514 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 4, 2024-03-24T10:50:46.514 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-24T10:50:46.514 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-24T10:50:46.514 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-24T10:50:46.514 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-24T10:50:46.514 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-24T10:50:46.514 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-24T10:50:46.514 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 8224, 2024-03-24T10:50:46.514 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 171, 2024-03-24T10:50:46.515 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T10:50:46.515 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-24T10:50:46.515 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-24T10:50:46.515 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-24T10:50:46.515 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 263635, 2024-03-24T10:50:46.515 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 2039, 2024-03-24T10:50:46.515 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-24T10:50:46.515 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-24T10:50:46.515 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-24T10:50:46.516 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-24T10:50:46.516 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.516 INFO:teuthology.orchestra.run.smithi116.stdout: "objecter": { 2024-03-24T10:50:46.516 INFO:teuthology.orchestra.run.smithi116.stdout: "op_active": 0, 2024-03-24T10:50:46.516 INFO:teuthology.orchestra.run.smithi116.stdout: "op_laggy": 0, 2024-03-24T10:50:46.516 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send": 2117, 2024-03-24T10:50:46.516 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send_bytes": 4808973, 2024-03-24T10:50:46.516 INFO:teuthology.orchestra.run.smithi116.stdout: "op_resend": 27, 2024-03-24T10:50:46.516 INFO:teuthology.orchestra.run.smithi116.stdout: "op_reply": 2117, 2024-03-24T10:50:46.517 INFO:teuthology.orchestra.run.smithi116.stdout: "op_latency": { 2024-03-24T10:50:46.517 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2117, 2024-03-24T10:50:46.517 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 19.130687468, 2024-03-24T10:50:46.517 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.009036696 2024-03-24T10:50:46.517 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.517 INFO:teuthology.orchestra.run.smithi116.stdout: "op_inflight": 0, 2024-03-24T10:50:46.517 INFO:teuthology.orchestra.run.smithi116.stdout: "oplen_avg": { 2024-03-24T10:50:46.517 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2117, 2024-03-24T10:50:46.518 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 2679 2024-03-24T10:50:46.518 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.518 INFO:teuthology.orchestra.run.smithi116.stdout: "op": 2117, 2024-03-24T10:50:46.518 INFO:teuthology.orchestra.run.smithi116.stdout: "op_r": 293, 2024-03-24T10:50:46.518 INFO:teuthology.orchestra.run.smithi116.stdout: "op_w": 1824, 2024-03-24T10:50:46.518 INFO:teuthology.orchestra.run.smithi116.stdout: "op_rmw": 0, 2024-03-24T10:50:46.518 INFO:teuthology.orchestra.run.smithi116.stdout: "op_pg": 0, 2024-03-24T10:50:46.518 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_stat": 158, 2024-03-24T10:50:46.518 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_create": 101, 2024-03-24T10:50:46.519 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_read": 27, 2024-03-24T10:50:46.519 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_write": 614, 2024-03-24T10:50:46.519 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writefull": 137, 2024-03-24T10:50:46.519 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writesame": 0, 2024-03-24T10:50:46.519 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_append": 0, 2024-03-24T10:50:46.519 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_zero": 0, 2024-03-24T10:50:46.519 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_truncate": 0, 2024-03-24T10:50:46.519 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_delete": 732, 2024-03-24T10:50:46.519 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_mapext": 0, 2024-03-24T10:50:46.520 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_sparse_read": 0, 2024-03-24T10:50:46.520 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_clonerange": 0, 2024-03-24T10:50:46.520 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_getxattr": 266, 2024-03-24T10:50:46.520 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_setxattr": 202, 2024-03-24T10:50:46.520 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_cmpxattr": 0, 2024-03-24T10:50:46.520 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_rmxattr": 0, 2024-03-24T10:50:46.520 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_resetxattrs": 0, 2024-03-24T10:50:46.520 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_call": 0, 2024-03-24T10:50:46.521 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_watch": 0, 2024-03-24T10:50:46.521 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_notify": 0, 2024-03-24T10:50:46.521 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T10:50:46.521 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls": 0, 2024-03-24T10:50:46.521 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls_filter": 0, 2024-03-24T10:50:46.521 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_other": 0, 2024-03-24T10:50:46.521 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_active": 0, 2024-03-24T10:50:46.521 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_send": 0, 2024-03-24T10:50:46.521 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_resend": 0, 2024-03-24T10:50:46.522 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_ping": 0, 2024-03-24T10:50:46.522 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_active": 0, 2024-03-24T10:50:46.522 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_send": 0, 2024-03-24T10:50:46.522 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_resend": 0, 2024-03-24T10:50:46.522 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_active": 0, 2024-03-24T10:50:46.522 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_send": 0, 2024-03-24T10:50:46.522 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_resend": 0, 2024-03-24T10:50:46.522 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_active": 0, 2024-03-24T10:50:46.523 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_send": 0, 2024-03-24T10:50:46.523 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_resend": 0, 2024-03-24T10:50:46.523 INFO:teuthology.orchestra.run.smithi116.stdout: "command_active": 0, 2024-03-24T10:50:46.523 INFO:teuthology.orchestra.run.smithi116.stdout: "command_send": 0, 2024-03-24T10:50:46.523 INFO:teuthology.orchestra.run.smithi116.stdout: "command_resend": 0, 2024-03-24T10:50:46.523 INFO:teuthology.orchestra.run.smithi116.stdout: "map_epoch": 98, 2024-03-24T10:50:46.523 INFO:teuthology.orchestra.run.smithi116.stdout: "map_full": 0, 2024-03-24T10:50:46.523 INFO:teuthology.orchestra.run.smithi116.stdout: "map_inc": 14, 2024-03-24T10:50:46.523 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_sessions": 12, 2024-03-24T10:50:46.524 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_open": 12, 2024-03-24T10:50:46.524 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_close": 0, 2024-03-24T10:50:46.524 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_laggy": 0, 2024-03-24T10:50:46.524 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_wr": 416, 2024-03-24T10:50:46.524 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_rd": 14, 2024-03-24T10:50:46.524 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_del": 12 2024-03-24T10:50:46.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.524 INFO:teuthology.orchestra.run.smithi116.stdout: "oft": { 2024-03-24T10:50:46.524 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_objs": 1, 2024-03-24T10:50:46.525 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_kv_pairs": 0, 2024-03-24T10:50:46.525 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_updates": 0, 2024-03-24T10:50:46.525 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_removes": 0 2024-03-24T10:50:46.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.525 INFO:teuthology.orchestra.run.smithi116.stdout: "osd-slow-ops": { 2024-03-24T10:50:46.525 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_ops_count": 0 2024-03-24T10:50:46.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.525 INFO:teuthology.orchestra.run.smithi116.stdout: "purge_queue": { 2024-03-24T10:50:46.525 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops": 0, 2024-03-24T10:50:46.526 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops_high_water": 10, 2024-03-24T10:50:46.526 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing": 0, 2024-03-24T10:50:46.526 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_high_water": 10, 2024-03-24T10:50:46.526 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed_ops": 220, 2024-03-24T10:50:46.526 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed": 220, 2024-03-24T10:50:46.526 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_item_in_journal": 0 2024-03-24T10:50:46.526 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.526 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T10:50:46.526 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:46.527 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-24T10:50:46.527 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:46.527 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 178232, 2024-03-24T10:50:46.527 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 52765828, 2024-03-24T10:50:46.527 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:46.527 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 178232, 2024-03-24T10:50:46.527 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T10:50:46.527 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T10:50:46.527 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 178232, 2024-03-24T10:50:46.528 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 52765828, 2024-03-24T10:50:46.528 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:46.528 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.528 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.528 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.528 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:46.528 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.528 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_bytes": { 2024-03-24T10:50:46.528 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:46.529 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-24T10:50:46.529 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:46.529 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T10:50:46.529 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T10:50:46.529 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:46.529 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:46.529 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 2117, 2024-03-24T10:50:46.529 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 4831105, 2024-03-24T10:50:46.530 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 2117, 2024-03-24T10:50:46.530 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 4831105, 2024-03-24T10:50:46.530 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:46.530 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.530 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.530 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.530 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:46.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.530 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_ops": { 2024-03-24T10:50:46.531 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:46.531 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 1024, 2024-03-24T10:50:46.531 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:46.531 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T10:50:46.531 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T10:50:46.531 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:46.531 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:46.531 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 2117, 2024-03-24T10:50:46.531 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 2117, 2024-03-24T10:50:46.532 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 2117, 2024-03-24T10:50:46.532 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 2117, 2024-03-24T10:50:46.532 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:46.532 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.532 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.532 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:46.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.532 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle": { 2024-03-24T10:50:46.533 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:46.533 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-24T10:50:46.533 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:46.533 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 220, 2024-03-24T10:50:46.533 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 22220, 2024-03-24T10:50:46.533 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:46.533 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 220, 2024-03-24T10:50:46.533 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T10:50:46.533 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T10:50:46.534 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 7, 2024-03-24T10:50:46.534 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 22220, 2024-03-24T10:50:46.534 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:46.534 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.534 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.534 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:46.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:46.535 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle-0x562907e72500": { 2024-03-24T10:50:46.535 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:46.535 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-24T10:50:46.535 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:46.535 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 1256, 2024-03-24T10:50:46.535 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 4155921, 2024-03-24T10:50:46.535 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:46.535 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 1256, 2024-03-24T10:50:46.535 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T10:50:46.536 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T10:50:46.536 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 607, 2024-03-24T10:50:46.536 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 4155921, 2024-03-24T10:50:46.536 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:46.536 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:46.536 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:46.536 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:46.536 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:46.537 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:46.537 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T10:50:47.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:46 smithi116 ceph-mon[36248]: pgmap v333: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:47.167 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:46 smithi023 ceph-mon[32466]: pgmap v333: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:47.200 INFO:tasks.check_counter:Daemon mds.b mds.exported=9 2024-03-24T10:50:47.200 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-24T10:50:47.200 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-24T10:50:47.200 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-03-24T10:50:47.200 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-03-24T10:50:47.200 DEBUG:tasks.check_counter:Getting stats from e 2024-03-24T10:50:47.201 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-24T10:50:47.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:46 smithi117 ceph-mon[36368]: pgmap v333: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:47.480 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T10:50:49.097 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:48 smithi116 ceph-mon[36248]: pgmap v334: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:49.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:48 smithi023 ceph-mon[32466]: pgmap v334: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:49.227 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T10:50:49.228 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T10:50:49.228 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 728, 2024-03-24T10:50:49.228 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 709, 2024-03-24T10:50:49.228 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 148415698, 2024-03-24T10:50:49.228 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 281901, 2024-03-24T10:50:49.229 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 7, 2024-03-24T10:50:49.229 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 3, 2024-03-24T10:50:49.229 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.137932632, 2024-03-24T10:50:49.229 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.016322472, 2024-03-24T10:50:49.229 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.115480866, 2024-03-24T10:50:49.229 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.013239259, 2024-03-24T10:50:49.229 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:49.230 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 708, 2024-03-24T10:50:49.230 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.015942079, 2024-03-24T10:50:49.230 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000022517 2024-03-24T10:50:49.230 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.230 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:49.230 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.230 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.231 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.231 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.231 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1490880, 2024-03-24T10:50:49.231 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 132304 2024-03-24T10:50:49.231 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.231 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T10:50:49.231 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 1010, 2024-03-24T10:50:49.231 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 1169, 2024-03-24T10:50:49.232 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 198827961, 2024-03-24T10:50:49.232 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 758165, 2024-03-24T10:50:49.232 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 9, 2024-03-24T10:50:49.232 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 7, 2024-03-24T10:50:49.232 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.179538856, 2024-03-24T10:50:49.232 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.020319081, 2024-03-24T10:50:49.232 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.131967972, 2024-03-24T10:50:49.233 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.020058223, 2024-03-24T10:50:49.233 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:49.233 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1166, 2024-03-24T10:50:49.233 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.031047929, 2024-03-24T10:50:49.233 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000026627 2024-03-24T10:50:49.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.233 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:49.234 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.234 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.234 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.234 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T10:50:49.234 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 444672 2024-03-24T10:50:49.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.235 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T10:50:49.235 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 1626, 2024-03-24T10:50:49.235 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 1626, 2024-03-24T10:50:49.235 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 143085907, 2024-03-24T10:50:49.235 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 504948, 2024-03-24T10:50:49.235 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 6, 2024-03-24T10:50:49.235 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 5, 2024-03-24T10:50:49.235 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.170760022, 2024-03-24T10:50:49.235 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.031049793, 2024-03-24T10:50:49.236 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.096191287, 2024-03-24T10:50:49.236 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.034956272, 2024-03-24T10:50:49.236 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:49.236 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1625, 2024-03-24T10:50:49.236 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.035091374, 2024-03-24T10:50:49.236 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000021594 2024-03-24T10:50:49.236 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.236 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:49.237 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.237 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.237 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.237 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-24T10:50:49.237 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 448 2024-03-24T10:50:49.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.237 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-24T10:50:49.237 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 1, 2024-03-24T10:50:49.238 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-24T10:50:49.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.238 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-MDSRank": { 2024-03-24T10:50:49.238 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-24T10:50:49.238 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-24T10:50:49.238 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 4931, 2024-03-24T10:50:49.238 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.111388019, 2024-03-24T10:50:49.238 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000022589 2024-03-24T10:50:49.239 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:49.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.239 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-PurgeQueue": { 2024-03-24T10:50:49.239 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-24T10:50:49.239 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-24T10:50:49.239 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.239 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.239 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.239 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:49.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.240 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-24T10:50:49.240 INFO:teuthology.orchestra.run.smithi116.stdout: "request": 0, 2024-03-24T10:50:49.240 INFO:teuthology.orchestra.run.smithi116.stdout: "reply": 0, 2024-03-24T10:50:49.240 INFO:teuthology.orchestra.run.smithi116.stdout: "reply_latency": { 2024-03-24T10:50:49.240 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.240 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.240 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.241 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_reply": 0, 2024-03-24T10:50:49.241 INFO:teuthology.orchestra.run.smithi116.stdout: "forward": 0, 2024-03-24T10:50:49.241 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_complete": 0, 2024-03-24T10:50:49.241 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_keys": 0, 2024-03-24T10:50:49.241 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_commit": 0, 2024-03-24T10:50:49.241 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_split": 0, 2024-03-24T10:50:49.241 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_merge": 0, 2024-03-24T10:50:49.242 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes": 27691, 2024-03-24T10:50:49.242 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_top": 5307, 2024-03-24T10:50:49.242 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_bottom": 22384, 2024-03-24T10:50:49.242 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pin_tail": 0, 2024-03-24T10:50:49.242 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pinned": 20110, 2024-03-24T10:50:49.242 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_expired": 0, 2024-03-24T10:50:49.242 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_with_caps": 0, 2024-03-24T10:50:49.242 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": 0, 2024-03-24T10:50:49.242 INFO:teuthology.orchestra.run.smithi116.stdout: "subtrees": 3, 2024-03-24T10:50:49.243 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse": 0, 2024-03-24T10:50:49.243 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_hit": 0, 2024-03-24T10:50:49.243 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_forward": 0, 2024-03-24T10:50:49.243 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_discover": 0, 2024-03-24T10:50:49.243 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_dir_fetch": 0, 2024-03-24T10:50:49.243 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_remote_ino": 0, 2024-03-24T10:50:49.243 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_lock": 0, 2024-03-24T10:50:49.244 INFO:teuthology.orchestra.run.smithi116.stdout: "load_cent": 0, 2024-03-24T10:50:49.244 INFO:teuthology.orchestra.run.smithi116.stdout: "q": 0, 2024-03-24T10:50:49.244 INFO:teuthology.orchestra.run.smithi116.stdout: "exported": 0, 2024-03-24T10:50:49.244 INFO:teuthology.orchestra.run.smithi116.stdout: "exported_inodes": 0, 2024-03-24T10:50:49.244 INFO:teuthology.orchestra.run.smithi116.stdout: "imported": 0, 2024-03-24T10:50:49.244 INFO:teuthology.orchestra.run.smithi116.stdout: "imported_inodes": 0, 2024-03-24T10:50:49.245 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_dir_fetch": 0, 2024-03-24T10:50:49.245 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_backtrace_fetch": 0, 2024-03-24T10:50:49.245 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_peer_discover": 0, 2024-03-24T10:50:49.245 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rfiles": 0, 2024-03-24T10:50:49.245 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rbytes": 0, 2024-03-24T10:50:49.245 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rsnaps": 0, 2024-03-24T10:50:49.245 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T10:50:49.245 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_set_tag": 0, 2024-03-24T10:50:49.246 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T10:50:49.246 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_inotable_repaired": 0, 2024-03-24T10:50:49.246 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_inodes": 0, 2024-03-24T10:50:49.246 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T10:50:49.246 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T10:50:49.246 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_file_inodes": 0, 2024-03-24T10:50:49.246 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_inode_file_caps": 0, 2024-03-24T10:50:49.246 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T10:50:49.247 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_grant": 0, 2024-03-24T10:50:49.247 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T10:50:49.247 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T10:50:49.247 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T10:50:49.247 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps": 0, 2024-03-24T10:50:49.247 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps_dirty": 0, 2024-03-24T10:50:49.247 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_cap_release": 0, 2024-03-24T10:50:49.247 INFO:teuthology.orchestra.run.smithi116.stdout: "process_request_cap_release": 0 2024-03-24T10:50:49.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.248 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_cache": { 2024-03-24T10:50:49.248 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update": 0, 2024-03-24T10:50:49.248 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update_receipt": 0, 2024-03-24T10:50:49.248 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_try_discover": 0, 2024-03-24T10:50:49.248 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_send_discover": 0, 2024-03-24T10:50:49.248 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_handle_discover": 0, 2024-03-24T10:50:49.249 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays": 0, 2024-03-24T10:50:49.249 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_delayed": 0, 2024-03-24T10:50:49.249 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_enqueuing": 0, 2024-03-24T10:50:49.249 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_created": 0, 2024-03-24T10:50:49.249 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_enqueued": 0, 2024-03-24T10:50:49.249 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_reintegrated": 0, 2024-03-24T10:50:49.249 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_migrated": 0, 2024-03-24T10:50:49.249 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_processing": 0, 2024-03-24T10:50:49.250 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_enqueued": 0, 2024-03-24T10:50:49.250 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_prioritized": 0, 2024-03-24T10:50:49.250 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_started": 0, 2024-03-24T10:50:49.250 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_completed": 0, 2024-03-24T10:50:49.250 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_path": 0, 2024-03-24T10:50:49.251 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_inode": 0, 2024-03-24T10:50:49.251 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T10:50:49.251 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_exportdir": 0, 2024-03-24T10:50:49.251 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_flush": 0, 2024-03-24T10:50:49.251 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragmentdir": 0, 2024-03-24T10:50:49.251 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragstats": 0, 2024-03-24T10:50:49.251 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_inodestats": 0 2024-03-24T10:50:49.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.251 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_log": { 2024-03-24T10:50:49.252 INFO:teuthology.orchestra.run.smithi116.stdout: "evlrg": 0, 2024-03-24T10:50:49.252 INFO:teuthology.orchestra.run.smithi116.stdout: "evadd": 0, 2024-03-24T10:50:49.252 INFO:teuthology.orchestra.run.smithi116.stdout: "evex": 0, 2024-03-24T10:50:49.252 INFO:teuthology.orchestra.run.smithi116.stdout: "evtrm": 0, 2024-03-24T10:50:49.252 INFO:teuthology.orchestra.run.smithi116.stdout: "ev": 123545, 2024-03-24T10:50:49.252 INFO:teuthology.orchestra.run.smithi116.stdout: "evexg": 0, 2024-03-24T10:50:49.253 INFO:teuthology.orchestra.run.smithi116.stdout: "evexd": 0, 2024-03-24T10:50:49.253 INFO:teuthology.orchestra.run.smithi116.stdout: "segadd": 0, 2024-03-24T10:50:49.253 INFO:teuthology.orchestra.run.smithi116.stdout: "segex": 0, 2024-03-24T10:50:49.253 INFO:teuthology.orchestra.run.smithi116.stdout: "segtrm": 0, 2024-03-24T10:50:49.253 INFO:teuthology.orchestra.run.smithi116.stdout: "seg": 130, 2024-03-24T10:50:49.253 INFO:teuthology.orchestra.run.smithi116.stdout: "segmjr": 12, 2024-03-24T10:50:49.253 INFO:teuthology.orchestra.run.smithi116.stdout: "segexg": 0, 2024-03-24T10:50:49.253 INFO:teuthology.orchestra.run.smithi116.stdout: "segexd": 0, 2024-03-24T10:50:49.254 INFO:teuthology.orchestra.run.smithi116.stdout: "expos": 110886252, 2024-03-24T10:50:49.254 INFO:teuthology.orchestra.run.smithi116.stdout: "wrpos": 492163027, 2024-03-24T10:50:49.254 INFO:teuthology.orchestra.run.smithi116.stdout: "rdpos": 492162453, 2024-03-24T10:50:49.254 INFO:teuthology.orchestra.run.smithi116.stdout: "jlat": { 2024-03-24T10:50:49.254 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.254 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.254 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.255 INFO:teuthology.orchestra.run.smithi116.stdout: "replayed": 123545 2024-03-24T10:50:49.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.255 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_mem": { 2024-03-24T10:50:49.255 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 464, 2024-03-24T10:50:49.255 INFO:teuthology.orchestra.run.smithi116.stdout: "ino+": 27612, 2024-03-24T10:50:49.255 INFO:teuthology.orchestra.run.smithi116.stdout: "ino-": 27148, 2024-03-24T10:50:49.256 INFO:teuthology.orchestra.run.smithi116.stdout: "dir": 63, 2024-03-24T10:50:49.256 INFO:teuthology.orchestra.run.smithi116.stdout: "dir+": 145, 2024-03-24T10:50:49.256 INFO:teuthology.orchestra.run.smithi116.stdout: "dir-": 82, 2024-03-24T10:50:49.256 INFO:teuthology.orchestra.run.smithi116.stdout: "dn": 27691, 2024-03-24T10:50:49.256 INFO:teuthology.orchestra.run.smithi116.stdout: "dn+": 27691, 2024-03-24T10:50:49.256 INFO:teuthology.orchestra.run.smithi116.stdout: "dn-": 0, 2024-03-24T10:50:49.256 INFO:teuthology.orchestra.run.smithi116.stdout: "cap": 0, 2024-03-24T10:50:49.256 INFO:teuthology.orchestra.run.smithi116.stdout: "cap+": 0, 2024-03-24T10:50:49.257 INFO:teuthology.orchestra.run.smithi116.stdout: "cap-": 0, 2024-03-24T10:50:49.257 INFO:teuthology.orchestra.run.smithi116.stdout: "rss": 79512, 2024-03-24T10:50:49.257 INFO:teuthology.orchestra.run.smithi116.stdout: "heap": 215420 2024-03-24T10:50:49.257 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.257 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_server": { 2024-03-24T10:50:49.257 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_client_request": 0, 2024-03-24T10:50:49.257 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_server_request": 0, 2024-03-24T10:50:49.257 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_request": 0, 2024-03-24T10:50:49.257 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_session": 0, 2024-03-24T10:50:49.258 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_peer_request": 0, 2024-03-24T10:50:49.258 INFO:teuthology.orchestra.run.smithi116.stdout: "req_create_latency": { 2024-03-24T10:50:49.258 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.258 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.258 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.258 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getattr_latency": { 2024-03-24T10:50:49.258 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.258 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.259 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.259 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getfilelock_latency": { 2024-03-24T10:50:49.259 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.259 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.259 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.259 INFO:teuthology.orchestra.run.smithi116.stdout: "req_link_latency": { 2024-03-24T10:50:49.260 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.260 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.260 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.260 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookup_latency": { 2024-03-24T10:50:49.260 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.260 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.260 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.261 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookuphash_latency": { 2024-03-24T10:50:49.261 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.261 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.261 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.261 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupino_latency": { 2024-03-24T10:50:49.261 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.261 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.262 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.262 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.262 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupname_latency": { 2024-03-24T10:50:49.262 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.262 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.262 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.262 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.262 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupparent_latency": { 2024-03-24T10:50:49.262 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupsnap_latency": { 2024-03-24T10:50:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lssnap_latency": { 2024-03-24T10:50:49.264 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.264 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.264 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.265 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:48 smithi117 ceph-mon[36368]: pgmap v334: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:49.265 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mkdir_latency": { 2024-03-24T10:50:49.265 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.265 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.265 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.265 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.265 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mknod_latency": { 2024-03-24T10:50:49.266 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.266 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.266 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.266 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mksnap_latency": { 2024-03-24T10:50:49.266 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.266 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.266 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: "req_open_latency": { 2024-03-24T10:50:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: "req_readdir_latency": { 2024-03-24T10:50:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rename_latency": { 2024-03-24T10:50:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.269 INFO:teuthology.orchestra.run.smithi116.stdout: "req_renamesnap_latency": { 2024-03-24T10:50:49.269 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.269 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.269 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.269 INFO:teuthology.orchestra.run.smithi116.stdout: "req_snapdiff_latency": { 2024-03-24T10:50:49.270 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.270 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.270 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.270 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmdir_latency": { 2024-03-24T10:50:49.271 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.271 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.271 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.271 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmsnap_latency": { 2024-03-24T10:50:49.271 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.271 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.271 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmxattr_latency": { 2024-03-24T10:50:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setattr_latency": { 2024-03-24T10:50:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.273 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.273 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.273 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setdirlayout_latency": { 2024-03-24T10:50:49.273 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.273 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.273 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.274 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setfilelock_latency": { 2024-03-24T10:50:49.274 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.274 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.274 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.274 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setlayout_latency": { 2024-03-24T10:50:49.274 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.275 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.275 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.275 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setxattr_latency": { 2024-03-24T10:50:49.275 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.275 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.275 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.276 INFO:teuthology.orchestra.run.smithi116.stdout: "req_symlink_latency": { 2024-03-24T10:50:49.276 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.276 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.276 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.276 INFO:teuthology.orchestra.run.smithi116.stdout: "req_unlink_latency": { 2024-03-24T10:50:49.276 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.276 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.277 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.277 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_revoke_eviction": 0, 2024-03-24T10:50:49.277 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_acquisition_throttle": 0, 2024-03-24T10:50:49.277 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getvxattr_latency": { 2024-03-24T10:50:49.277 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.277 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.277 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.278 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:49.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.278 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_sessions": { 2024-03-24T10:50:49.278 INFO:teuthology.orchestra.run.smithi116.stdout: "session_count": 2, 2024-03-24T10:50:49.278 INFO:teuthology.orchestra.run.smithi116.stdout: "session_add": 4, 2024-03-24T10:50:49.278 INFO:teuthology.orchestra.run.smithi116.stdout: "session_remove": 2, 2024-03-24T10:50:49.279 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_open": 3, 2024-03-24T10:50:49.279 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_stale": 0, 2024-03-24T10:50:49.279 INFO:teuthology.orchestra.run.smithi116.stdout: "total_load": 0, 2024-03-24T10:50:49.279 INFO:teuthology.orchestra.run.smithi116.stdout: "average_load": 0, 2024-03-24T10:50:49.279 INFO:teuthology.orchestra.run.smithi116.stdout: "avg_session_uptime": 2769, 2024-03-24T10:50:49.279 INFO:teuthology.orchestra.run.smithi116.stdout: "mdthresh_evicted": 0 2024-03-24T10:50:49.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.279 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-24T10:50:49.280 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-24T10:50:49.280 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-24T10:50:49.280 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T10:50:49.280 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-24T10:50:49.280 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T10:50:49.280 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-24T10:50:49.280 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T10:50:49.280 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T10:50:49.280 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T10:50:49.281 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T10:50:49.281 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T10:50:49.281 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-24T10:50:49.281 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T10:50:49.281 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T10:50:49.281 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-24T10:50:49.281 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-24T10:50:49.281 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-24T10:50:49.282 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-24T10:50:49.282 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T10:50:49.282 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T10:50:49.282 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T10:50:49.282 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T10:50:49.282 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T10:50:49.282 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-24T10:50:49.282 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-24T10:50:49.283 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-24T10:50:49.283 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T10:50:49.283 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T10:50:49.283 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-24T10:50:49.283 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-24T10:50:49.283 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-24T10:50:49.283 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-24T10:50:49.283 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T10:50:49.283 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-24T10:50:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T10:50:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-24T10:50:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 259966, 2024-03-24T10:50:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 633, 2024-03-24T10:50:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-24T10:50:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-24T10:50:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-24T10:50:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-24T10:50:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-24T10:50:49.285 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-24T10:50:49.285 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-24T10:50:49.285 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-24T10:50:49.285 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 8224, 2024-03-24T10:50:49.285 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 171, 2024-03-24T10:50:49.285 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T10:50:49.285 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-24T10:50:49.285 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-24T10:50:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-24T10:50:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 20796216, 2024-03-24T10:50:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 123473, 2024-03-24T10:50:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-24T10:50:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-24T10:50:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-24T10:50:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-24T10:50:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: "objecter": { 2024-03-24T10:50:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "op_active": 0, 2024-03-24T10:50:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "op_laggy": 0, 2024-03-24T10:50:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send": 3111, 2024-03-24T10:50:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send_bytes": 21, 2024-03-24T10:50:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "op_resend": 3, 2024-03-24T10:50:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "op_reply": 3111, 2024-03-24T10:50:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "op_latency": { 2024-03-24T10:50:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3111, 2024-03-24T10:50:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 2.702957123, 2024-03-24T10:50:49.288 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000868838 2024-03-24T10:50:49.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.288 INFO:teuthology.orchestra.run.smithi116.stdout: "op_inflight": 0, 2024-03-24T10:50:49.288 INFO:teuthology.orchestra.run.smithi116.stdout: "oplen_avg": { 2024-03-24T10:50:49.288 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3111, 2024-03-24T10:50:49.288 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 3112 2024-03-24T10:50:49.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.288 INFO:teuthology.orchestra.run.smithi116.stdout: "op": 3111, 2024-03-24T10:50:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "op_r": 3111, 2024-03-24T10:50:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "op_w": 0, 2024-03-24T10:50:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "op_rmw": 0, 2024-03-24T10:50:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "op_pg": 0, 2024-03-24T10:50:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_stat": 1871, 2024-03-24T10:50:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_create": 0, 2024-03-24T10:50:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_read": 1238, 2024-03-24T10:50:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_write": 0, 2024-03-24T10:50:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writefull": 0, 2024-03-24T10:50:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writesame": 0, 2024-03-24T10:50:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_append": 0, 2024-03-24T10:50:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_zero": 0, 2024-03-24T10:50:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_truncate": 0, 2024-03-24T10:50:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_delete": 0, 2024-03-24T10:50:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_mapext": 0, 2024-03-24T10:50:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_sparse_read": 0, 2024-03-24T10:50:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_clonerange": 0, 2024-03-24T10:50:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_getxattr": 1, 2024-03-24T10:50:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_setxattr": 0, 2024-03-24T10:50:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_cmpxattr": 0, 2024-03-24T10:50:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_rmxattr": 0, 2024-03-24T10:50:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_resetxattrs": 0, 2024-03-24T10:50:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_call": 0, 2024-03-24T10:50:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_watch": 0, 2024-03-24T10:50:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_notify": 0, 2024-03-24T10:50:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T10:50:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls": 0, 2024-03-24T10:50:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls_filter": 0, 2024-03-24T10:50:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_other": 0, 2024-03-24T10:50:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_active": 0, 2024-03-24T10:50:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_send": 0, 2024-03-24T10:50:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_resend": 0, 2024-03-24T10:50:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_ping": 0, 2024-03-24T10:50:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_active": 0, 2024-03-24T10:50:49.293 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_send": 0, 2024-03-24T10:50:49.293 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_resend": 0, 2024-03-24T10:50:49.293 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_active": 0, 2024-03-24T10:50:49.293 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_send": 0, 2024-03-24T10:50:49.293 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_resend": 0, 2024-03-24T10:50:49.293 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_active": 0, 2024-03-24T10:50:49.293 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_send": 0, 2024-03-24T10:50:49.293 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_resend": 0, 2024-03-24T10:50:49.294 INFO:teuthology.orchestra.run.smithi116.stdout: "command_active": 0, 2024-03-24T10:50:49.294 INFO:teuthology.orchestra.run.smithi116.stdout: "command_send": 0, 2024-03-24T10:50:49.294 INFO:teuthology.orchestra.run.smithi116.stdout: "command_resend": 0, 2024-03-24T10:50:49.294 INFO:teuthology.orchestra.run.smithi116.stdout: "map_epoch": 98, 2024-03-24T10:50:49.294 INFO:teuthology.orchestra.run.smithi116.stdout: "map_full": 0, 2024-03-24T10:50:49.294 INFO:teuthology.orchestra.run.smithi116.stdout: "map_inc": 13, 2024-03-24T10:50:49.294 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_sessions": 11, 2024-03-24T10:50:49.294 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_open": 11, 2024-03-24T10:50:49.294 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_close": 0, 2024-03-24T10:50:49.295 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_laggy": 0, 2024-03-24T10:50:49.295 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_wr": 0, 2024-03-24T10:50:49.295 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_rd": 2, 2024-03-24T10:50:49.295 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_del": 0 2024-03-24T10:50:49.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.295 INFO:teuthology.orchestra.run.smithi116.stdout: "oft": { 2024-03-24T10:50:49.295 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_objs": 0, 2024-03-24T10:50:49.295 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_kv_pairs": 0, 2024-03-24T10:50:49.295 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_updates": 0, 2024-03-24T10:50:49.296 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_removes": 0 2024-03-24T10:50:49.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.296 INFO:teuthology.orchestra.run.smithi116.stdout: "osd-slow-ops": { 2024-03-24T10:50:49.296 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_ops_count": 0 2024-03-24T10:50:49.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.296 INFO:teuthology.orchestra.run.smithi116.stdout: "purge_queue": { 2024-03-24T10:50:49.296 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops": 0, 2024-03-24T10:50:49.296 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T10:50:49.296 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing": 0, 2024-03-24T10:50:49.296 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_high_water": 0, 2024-03-24T10:50:49.297 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed_ops": 0, 2024-03-24T10:50:49.297 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed": 0, 2024-03-24T10:50:49.297 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_item_in_journal": 0 2024-03-24T10:50:49.297 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.297 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T10:50:49.297 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:49.297 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-24T10:50:49.297 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:49.298 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 3360, 2024-03-24T10:50:49.298 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 490011525, 2024-03-24T10:50:49.298 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:49.298 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 3360, 2024-03-24T10:50:49.298 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T10:50:49.298 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T10:50:49.298 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 3360, 2024-03-24T10:50:49.298 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 490011525, 2024-03-24T10:50:49.299 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:49.299 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.299 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.299 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.299 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:49.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.299 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_bytes": { 2024-03-24T10:50:49.299 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:49.299 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-24T10:50:49.300 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:49.300 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T10:50:49.300 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T10:50:49.300 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:49.300 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:49.300 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 3111, 2024-03-24T10:50:49.300 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 487978254, 2024-03-24T10:50:49.300 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 3111, 2024-03-24T10:50:49.301 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 487978254, 2024-03-24T10:50:49.301 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:49.301 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.301 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.301 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.301 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:49.301 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.301 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_ops": { 2024-03-24T10:50:49.302 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:49.302 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 1024, 2024-03-24T10:50:49.302 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:49.302 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T10:50:49.302 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T10:50:49.302 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:49.302 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:49.302 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 3111, 2024-03-24T10:50:49.302 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 3111, 2024-03-24T10:50:49.303 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 3111, 2024-03-24T10:50:49.303 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 3111, 2024-03-24T10:50:49.303 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:49.303 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.303 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.303 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.303 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:49.303 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.303 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle": { 2024-03-24T10:50:49.304 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:49.304 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-24T10:50:49.304 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:49.304 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T10:50:49.304 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T10:50:49.304 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:49.304 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:49.304 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T10:50:49.304 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T10:50:49.304 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-24T10:50:49.305 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-24T10:50:49.305 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:49.305 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.305 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.305 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.305 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:49.305 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:49.305 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle-0x56362ff3a6e0": { 2024-03-24T10:50:49.305 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:49.306 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-24T10:50:49.306 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:49.306 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T10:50:49.306 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T10:50:49.306 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:49.306 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:49.306 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T10:50:49.306 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T10:50:49.307 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-24T10:50:49.307 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-24T10:50:49.307 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:49.307 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:49.307 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:49.307 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:49.307 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:49.307 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:49.307 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T10:50:50.062 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-24T10:50:50.062 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-24T10:50:50.062 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-03-24T10:50:50.062 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-24T10:50:50.062 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T10:50:50.063 DEBUG:tasks.check_counter:Getting stats from h 2024-03-24T10:50:50.063 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-24T10:50:50.313 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T10:50:51.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:50 smithi116 ceph-mon[36248]: pgmap v335: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:51.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:50 smithi023 ceph-mon[32466]: pgmap v335: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:51.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:50 smithi117 ceph-mon[36368]: pgmap v335: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:51.911 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T10:50:51.911 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T10:50:51.911 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 68417, 2024-03-24T10:50:51.911 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 112693, 2024-03-24T10:50:51.912 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 14786673, 2024-03-24T10:50:51.912 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 245631272, 2024-03-24T10:50:51.912 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 16, 2024-03-24T10:50:51.912 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 4, 2024-03-24T10:50:51.912 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 5.288326354, 2024-03-24T10:50:51.912 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 2.405555915, 2024-03-24T10:50:51.912 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 2.237175759, 2024-03-24T10:50:51.912 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.616390713, 2024-03-24T10:50:51.912 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:51.913 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 112692, 2024-03-24T10:50:51.913 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 2.286856848, 2024-03-24T10:50:51.913 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000020292 2024-03-24T10:50:51.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.913 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:51.913 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 132504, 2024-03-24T10:50:51.913 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.023021511, 2024-03-24T10:50:51.913 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000173 2024-03-24T10:50:51.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.913 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1510768, 2024-03-24T10:50:51.914 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 156272 2024-03-24T10:50:51.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.914 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T10:50:51.914 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 269088, 2024-03-24T10:50:51.914 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 382849, 2024-03-24T10:50:51.914 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 90843139, 2024-03-24T10:50:51.914 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 504387166, 2024-03-24T10:50:51.914 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 46, 2024-03-24T10:50:51.915 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 6, 2024-03-24T10:50:51.915 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 16.357242320, 2024-03-24T10:50:51.915 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 7.554681839, 2024-03-24T10:50:51.915 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 11.623602485, 2024-03-24T10:50:51.915 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.828021894, 2024-03-24T10:50:51.915 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:51.915 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 382847, 2024-03-24T10:50:51.915 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 7.506681905, 2024-03-24T10:50:51.915 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000019607 2024-03-24T10:50:51.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.916 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:51.916 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 565394, 2024-03-24T10:50:51.916 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.069871915, 2024-03-24T10:50:51.916 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000123 2024-03-24T10:50:51.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.916 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 84218928, 2024-03-24T10:50:51.916 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 323920768 2024-03-24T10:50:51.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.916 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T10:50:51.916 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 95361, 2024-03-24T10:50:51.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 166713, 2024-03-24T10:50:51.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 20860834, 2024-03-24T10:50:51.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 188761107, 2024-03-24T10:50:51.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 12, 2024-03-24T10:50:51.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 6, 2024-03-24T10:50:51.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 8.641738598, 2024-03-24T10:50:51.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 4.284618984, 2024-03-24T10:50:51.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 3.965612084, 2024-03-24T10:50:51.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.699613734, 2024-03-24T10:50:51.917 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:51.918 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 166711, 2024-03-24T10:50:51.918 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 3.251969116, 2024-03-24T10:50:51.918 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000019506 2024-03-24T10:50:51.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.918 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:51.918 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 214601, 2024-03-24T10:50:51.918 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.035377854, 2024-03-24T10:50:51.918 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000164 2024-03-24T10:50:51.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.919 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1568, 2024-03-24T10:50:51.919 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 5152 2024-03-24T10:50:51.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.919 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-24T10:50:51.919 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 1, 2024-03-24T10:50:51.919 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-24T10:50:51.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.919 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-MDSRank": { 2024-03-24T10:50:51.919 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-24T10:50:51.919 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-24T10:50:51.920 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 85275, 2024-03-24T10:50:51.920 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 91.663912406, 2024-03-24T10:50:51.920 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001074921 2024-03-24T10:50:51.920 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:51.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.920 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-PurgeQueue": { 2024-03-24T10:50:51.920 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-24T10:50:51.920 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-24T10:50:51.921 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 39309, 2024-03-24T10:50:51.921 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 7.522061168, 2024-03-24T10:50:51.921 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000191357 2024-03-24T10:50:51.921 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:51.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.921 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-24T10:50:51.921 INFO:teuthology.orchestra.run.smithi116.stdout: "request": 185428, 2024-03-24T10:50:51.921 INFO:teuthology.orchestra.run.smithi116.stdout: "reply": 185420, 2024-03-24T10:50:51.921 INFO:teuthology.orchestra.run.smithi116.stdout: "reply_latency": { 2024-03-24T10:50:51.922 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 185420, 2024-03-24T10:50:51.922 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 284.793416583, 2024-03-24T10:50:51.922 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001535936 2024-03-24T10:50:51.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.922 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_reply": 0, 2024-03-24T10:50:51.922 INFO:teuthology.orchestra.run.smithi116.stdout: "forward": 8, 2024-03-24T10:50:51.922 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_complete": 0, 2024-03-24T10:50:51.922 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_keys": 0, 2024-03-24T10:50:51.923 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_commit": 163, 2024-03-24T10:50:51.923 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_split": 12, 2024-03-24T10:50:51.923 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_merge": 19, 2024-03-24T10:50:51.923 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes": 1985, 2024-03-24T10:50:51.923 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_top": 877, 2024-03-24T10:50:51.923 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_bottom": 1108, 2024-03-24T10:50:51.923 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pin_tail": 0, 2024-03-24T10:50:51.923 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pinned": 733, 2024-03-24T10:50:51.923 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_expired": 0, 2024-03-24T10:50:51.924 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_with_caps": 46, 2024-03-24T10:50:51.924 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": 46, 2024-03-24T10:50:51.924 INFO:teuthology.orchestra.run.smithi116.stdout: "subtrees": 5, 2024-03-24T10:50:51.924 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse": 291281, 2024-03-24T10:50:51.924 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_hit": 172405, 2024-03-24T10:50:51.924 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_forward": 3, 2024-03-24T10:50:51.924 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_discover": 11, 2024-03-24T10:50:51.924 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_dir_fetch": 0, 2024-03-24T10:50:51.924 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_remote_ino": 0, 2024-03-24T10:50:51.925 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_lock": 3063, 2024-03-24T10:50:51.925 INFO:teuthology.orchestra.run.smithi116.stdout: "load_cent": 0, 2024-03-24T10:50:51.925 INFO:teuthology.orchestra.run.smithi116.stdout: "q": 0, 2024-03-24T10:50:51.925 INFO:teuthology.orchestra.run.smithi116.stdout: "exported": 0, 2024-03-24T10:50:51.925 INFO:teuthology.orchestra.run.smithi116.stdout: "exported_inodes": 0, 2024-03-24T10:50:51.925 INFO:teuthology.orchestra.run.smithi116.stdout: "imported": 9, 2024-03-24T10:50:51.925 INFO:teuthology.orchestra.run.smithi116.stdout: "imported_inodes": 158, 2024-03-24T10:50:51.925 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_dir_fetch": 0, 2024-03-24T10:50:51.925 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_backtrace_fetch": 1317, 2024-03-24T10:50:51.926 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_peer_discover": 1, 2024-03-24T10:50:51.926 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rfiles": 68, 2024-03-24T10:50:51.926 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rbytes": 21196213, 2024-03-24T10:50:51.926 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rsnaps": 6, 2024-03-24T10:50:51.926 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_fetch": 1317, 2024-03-24T10:50:51.926 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_set_tag": 0, 2024-03-24T10:50:51.926 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T10:50:51.926 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_inotable_repaired": 0, 2024-03-24T10:50:51.926 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_inodes": 208, 2024-03-24T10:50:51.927 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T10:50:51.927 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dirfrag_rstats": 208, 2024-03-24T10:50:51.927 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_file_inodes": 1109, 2024-03-24T10:50:51.927 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_inode_file_caps": 2, 2024-03-24T10:50:51.927 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_revoke": 3367, 2024-03-24T10:50:51.927 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_grant": 83952, 2024-03-24T10:50:51.927 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T10:50:51.928 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T10:50:51.928 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T10:50:51.928 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps": 40421, 2024-03-24T10:50:51.928 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps_dirty": 21329, 2024-03-24T10:50:51.928 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_cap_release": 10912, 2024-03-24T10:50:51.928 INFO:teuthology.orchestra.run.smithi116.stdout: "process_request_cap_release": 128966 2024-03-24T10:50:51.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.928 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_cache": { 2024-03-24T10:50:51.929 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update": 3, 2024-03-24T10:50:51.929 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update_receipt": 3, 2024-03-24T10:50:51.929 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_try_discover": 2, 2024-03-24T10:50:51.929 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_send_discover": 13, 2024-03-24T10:50:51.929 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_handle_discover": 6, 2024-03-24T10:50:51.929 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays": 0, 2024-03-24T10:50:51.929 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_delayed": 0, 2024-03-24T10:50:51.929 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_enqueuing": 0, 2024-03-24T10:50:51.929 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_created": 27148, 2024-03-24T10:50:51.929 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_enqueued": 27148, 2024-03-24T10:50:51.930 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_reintegrated": 0, 2024-03-24T10:50:51.930 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_migrated": 0, 2024-03-24T10:50:51.930 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_processing": 0, 2024-03-24T10:50:51.930 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_enqueued": 0, 2024-03-24T10:50:51.930 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_prioritized": 0, 2024-03-24T10:50:51.930 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_started": 0, 2024-03-24T10:50:51.930 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_completed": 0, 2024-03-24T10:50:51.930 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_path": 0, 2024-03-24T10:50:51.930 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_inode": 0, 2024-03-24T10:50:51.931 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T10:50:51.931 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_exportdir": 0, 2024-03-24T10:50:51.931 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_flush": 0, 2024-03-24T10:50:51.931 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragmentdir": 31, 2024-03-24T10:50:51.931 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragstats": 0, 2024-03-24T10:50:51.931 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_inodestats": 0 2024-03-24T10:50:51.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.931 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_log": { 2024-03-24T10:50:51.932 INFO:teuthology.orchestra.run.smithi116.stdout: "evlrg": 0, 2024-03-24T10:50:51.932 INFO:teuthology.orchestra.run.smithi116.stdout: "evadd": 123545, 2024-03-24T10:50:51.932 INFO:teuthology.orchestra.run.smithi116.stdout: "evex": 38592, 2024-03-24T10:50:51.932 INFO:teuthology.orchestra.run.smithi116.stdout: "evtrm": 36871, 2024-03-24T10:50:51.932 INFO:teuthology.orchestra.run.smithi116.stdout: "ev": 86674, 2024-03-24T10:50:51.932 INFO:teuthology.orchestra.run.smithi116.stdout: "evexg": 0, 2024-03-24T10:50:51.932 INFO:teuthology.orchestra.run.smithi116.stdout: "evexd": 1721, 2024-03-24T10:50:51.932 INFO:teuthology.orchestra.run.smithi116.stdout: "segadd": 184, 2024-03-24T10:50:51.932 INFO:teuthology.orchestra.run.smithi116.stdout: "segex": 56, 2024-03-24T10:50:51.932 INFO:teuthology.orchestra.run.smithi116.stdout: "segtrm": 54, 2024-03-24T10:50:51.933 INFO:teuthology.orchestra.run.smithi116.stdout: "seg": 130, 2024-03-24T10:50:51.933 INFO:teuthology.orchestra.run.smithi116.stdout: "segmjr": 8, 2024-03-24T10:50:51.933 INFO:teuthology.orchestra.run.smithi116.stdout: "segexg": 0, 2024-03-24T10:50:51.933 INFO:teuthology.orchestra.run.smithi116.stdout: "segexd": 2, 2024-03-24T10:50:51.933 INFO:teuthology.orchestra.run.smithi116.stdout: "expos": 110886252, 2024-03-24T10:50:51.933 INFO:teuthology.orchestra.run.smithi116.stdout: "wrpos": 492163027, 2024-03-24T10:50:51.933 INFO:teuthology.orchestra.run.smithi116.stdout: "rdpos": 0, 2024-03-24T10:50:51.933 INFO:teuthology.orchestra.run.smithi116.stdout: "jlat": { 2024-03-24T10:50:51.933 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 43003, 2024-03-24T10:50:51.934 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 79.833822727, 2024-03-24T10:50:51.934 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001856471 2024-03-24T10:50:51.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.934 INFO:teuthology.orchestra.run.smithi116.stdout: "replayed": 0 2024-03-24T10:50:51.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.934 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_mem": { 2024-03-24T10:50:51.934 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 464, 2024-03-24T10:50:51.934 INFO:teuthology.orchestra.run.smithi116.stdout: "ino+": 27612, 2024-03-24T10:50:51.934 INFO:teuthology.orchestra.run.smithi116.stdout: "ino-": 27148, 2024-03-24T10:50:51.935 INFO:teuthology.orchestra.run.smithi116.stdout: "dir": 63, 2024-03-24T10:50:51.935 INFO:teuthology.orchestra.run.smithi116.stdout: "dir+": 146, 2024-03-24T10:50:51.935 INFO:teuthology.orchestra.run.smithi116.stdout: "dir-": 83, 2024-03-24T10:50:51.935 INFO:teuthology.orchestra.run.smithi116.stdout: "dn": 1985, 2024-03-24T10:50:51.935 INFO:teuthology.orchestra.run.smithi116.stdout: "dn+": 54433, 2024-03-24T10:50:51.935 INFO:teuthology.orchestra.run.smithi116.stdout: "dn-": 52448, 2024-03-24T10:50:51.935 INFO:teuthology.orchestra.run.smithi116.stdout: "cap": 46, 2024-03-24T10:50:51.935 INFO:teuthology.orchestra.run.smithi116.stdout: "cap+": 28850, 2024-03-24T10:50:51.935 INFO:teuthology.orchestra.run.smithi116.stdout: "cap-": 28804, 2024-03-24T10:50:51.935 INFO:teuthology.orchestra.run.smithi116.stdout: "rss": 81440, 2024-03-24T10:50:51.936 INFO:teuthology.orchestra.run.smithi116.stdout: "heap": 223612 2024-03-24T10:50:51.936 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.936 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_server": { 2024-03-24T10:50:51.936 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_client_request": 272831, 2024-03-24T10:50:51.936 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_server_request": 1, 2024-03-24T10:50:51.936 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_request": 185428, 2024-03-24T10:50:51.936 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_session": 3853, 2024-03-24T10:50:51.936 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_peer_request": 2, 2024-03-24T10:50:51.936 INFO:teuthology.orchestra.run.smithi116.stdout: "req_create_latency": { 2024-03-24T10:50:51.937 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 27128, 2024-03-24T10:50:51.937 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 45.407219165, 2024-03-24T10:50:51.937 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001673813 2024-03-24T10:50:51.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.937 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getattr_latency": { 2024-03-24T10:50:51.937 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 7124, 2024-03-24T10:50:51.937 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 16.908415939, 2024-03-24T10:50:51.937 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.002373444 2024-03-24T10:50:51.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.937 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getfilelock_latency": { 2024-03-24T10:50:51.938 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.938 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.938 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.938 INFO:teuthology.orchestra.run.smithi116.stdout: "req_link_latency": { 2024-03-24T10:50:51.938 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.938 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.938 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.939 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookup_latency": { 2024-03-24T10:50:51.939 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 57995, 2024-03-24T10:50:51.939 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 30.016360708, 2024-03-24T10:50:51.939 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000517568 2024-03-24T10:50:51.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.939 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookuphash_latency": { 2024-03-24T10:50:51.939 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.939 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.939 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.940 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupino_latency": { 2024-03-24T10:50:51.940 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1, 2024-03-24T10:50:51.940 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000121983, 2024-03-24T10:50:51.940 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000121983 2024-03-24T10:50:51.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.940 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupname_latency": { 2024-03-24T10:50:51.940 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.940 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.940 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.941 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupparent_latency": { 2024-03-24T10:50:51.941 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.941 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.941 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.941 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupsnap_latency": { 2024-03-24T10:50:51.941 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.941 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.941 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.942 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lssnap_latency": { 2024-03-24T10:50:51.942 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.942 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.942 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.942 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mkdir_latency": { 2024-03-24T10:50:51.942 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.942 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.943 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.943 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mknod_latency": { 2024-03-24T10:50:51.943 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.943 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.943 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.943 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mksnap_latency": { 2024-03-24T10:50:51.943 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.943 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.944 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.944 INFO:teuthology.orchestra.run.smithi116.stdout: "req_open_latency": { 2024-03-24T10:50:51.944 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.944 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.944 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.944 INFO:teuthology.orchestra.run.smithi116.stdout: "req_readdir_latency": { 2024-03-24T10:50:51.944 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 59472, 2024-03-24T10:50:51.945 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 130.270156584, 2024-03-24T10:50:51.945 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.002190445 2024-03-24T10:50:51.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.945 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rename_latency": { 2024-03-24T10:50:51.945 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 5680, 2024-03-24T10:50:51.945 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 13.009406739, 2024-03-24T10:50:51.945 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.002290388 2024-03-24T10:50:51.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.945 INFO:teuthology.orchestra.run.smithi116.stdout: "req_renamesnap_latency": { 2024-03-24T10:50:51.946 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.946 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.946 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.946 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.946 INFO:teuthology.orchestra.run.smithi116.stdout: "req_snapdiff_latency": { 2024-03-24T10:50:51.946 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.946 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.946 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.946 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.947 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmdir_latency": { 2024-03-24T10:50:51.947 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.947 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.947 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.947 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.947 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmsnap_latency": { 2024-03-24T10:50:51.947 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.947 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.947 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.947 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.948 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmxattr_latency": { 2024-03-24T10:50:51.948 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.948 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.948 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.948 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setattr_latency": { 2024-03-24T10:50:51.948 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.948 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.948 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.949 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setdirlayout_latency": { 2024-03-24T10:50:51.949 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.949 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.949 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.949 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setfilelock_latency": { 2024-03-24T10:50:51.949 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 872, 2024-03-24T10:50:51.949 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.620769325, 2024-03-24T10:50:51.950 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000711891 2024-03-24T10:50:51.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.950 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setlayout_latency": { 2024-03-24T10:50:51.950 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.950 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.950 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.950 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setxattr_latency": { 2024-03-24T10:50:51.950 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.950 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.951 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.951 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.951 INFO:teuthology.orchestra.run.smithi116.stdout: "req_symlink_latency": { 2024-03-24T10:50:51.951 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.951 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.951 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.951 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.951 INFO:teuthology.orchestra.run.smithi116.stdout: "req_unlink_latency": { 2024-03-24T10:50:51.952 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 27148, 2024-03-24T10:50:51.952 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 48.560966140, 2024-03-24T10:50:51.952 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001788749 2024-03-24T10:50:51.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.952 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_revoke_eviction": 0, 2024-03-24T10:50:51.952 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_acquisition_throttle": 0, 2024-03-24T10:50:51.952 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getvxattr_latency": { 2024-03-24T10:50:51.952 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.952 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.953 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.953 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:51.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.953 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_sessions": { 2024-03-24T10:50:51.953 INFO:teuthology.orchestra.run.smithi116.stdout: "session_count": 2, 2024-03-24T10:50:51.953 INFO:teuthology.orchestra.run.smithi116.stdout: "session_add": 4, 2024-03-24T10:50:51.953 INFO:teuthology.orchestra.run.smithi116.stdout: "session_remove": 2, 2024-03-24T10:50:51.953 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_open": 2, 2024-03-24T10:50:51.953 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_stale": 0, 2024-03-24T10:50:51.954 INFO:teuthology.orchestra.run.smithi116.stdout: "total_load": 43178, 2024-03-24T10:50:51.954 INFO:teuthology.orchestra.run.smithi116.stdout: "average_load": 21589, 2024-03-24T10:50:51.954 INFO:teuthology.orchestra.run.smithi116.stdout: "avg_session_uptime": 611, 2024-03-24T10:50:51.954 INFO:teuthology.orchestra.run.smithi116.stdout: "mdthresh_evicted": 0 2024-03-24T10:50:51.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.954 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-24T10:50:51.954 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-24T10:50:51.954 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-24T10:50:51.955 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T10:50:51.955 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-24T10:50:51.955 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T10:50:51.955 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-24T10:50:51.955 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T10:50:51.955 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T10:50:51.955 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T10:50:51.955 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T10:50:51.955 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T10:50:51.956 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-24T10:50:51.956 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T10:50:51.956 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T10:50:51.956 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-24T10:50:51.956 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-24T10:50:51.956 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-24T10:50:51.956 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-24T10:50:51.956 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T10:50:51.956 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T10:50:51.956 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T10:50:51.957 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T10:50:51.957 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T10:50:51.957 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-24T10:50:51.957 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-24T10:50:51.957 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-24T10:50:51.957 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T10:50:51.957 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T10:50:51.957 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-24T10:50:51.957 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-24T10:50:51.958 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-24T10:50:51.958 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-24T10:50:51.958 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T10:50:51.958 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-24T10:50:51.958 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T10:50:51.958 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-24T10:50:51.958 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 315980, 2024-03-24T10:50:51.958 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 335, 2024-03-24T10:50:51.958 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 264, 2024-03-24T10:50:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 3, 2024-03-24T10:50:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-24T10:50:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-24T10:50:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-24T10:50:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-24T10:50:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-24T10:50:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-24T10:50:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 8224, 2024-03-24T10:50:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 171, 2024-03-24T10:50:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T10:50:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-24T10:50:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-24T10:50:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-24T10:50:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 2767617, 2024-03-24T10:50:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 28259, 2024-03-24T10:50:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-24T10:50:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-24T10:50:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-24T10:50:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-24T10:50:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "objecter": { 2024-03-24T10:50:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "op_active": 0, 2024-03-24T10:50:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "op_laggy": 0, 2024-03-24T10:50:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send": 73898, 2024-03-24T10:50:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send_bytes": 494946728, 2024-03-24T10:50:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "op_resend": 33, 2024-03-24T10:50:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "op_reply": 73898, 2024-03-24T10:50:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "op_latency": { 2024-03-24T10:50:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 73898, 2024-03-24T10:50:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 150.943613544, 2024-03-24T10:50:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.002042594 2024-03-24T10:50:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "op_inflight": 0, 2024-03-24T10:50:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "oplen_avg": { 2024-03-24T10:50:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 73898, 2024-03-24T10:50:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 74864 2024-03-24T10:50:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: "op": 73898, 2024-03-24T10:50:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: "op_r": 1983, 2024-03-24T10:50:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: "op_w": 71915, 2024-03-24T10:50:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: "op_rmw": 0, 2024-03-24T10:50:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: "op_pg": 0, 2024-03-24T10:50:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_stat": 24, 2024-03-24T10:50:51.964 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_create": 34, 2024-03-24T10:50:51.964 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_read": 666, 2024-03-24T10:50:51.964 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_write": 43669, 2024-03-24T10:50:51.964 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writefull": 274, 2024-03-24T10:50:51.964 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writesame": 0, 2024-03-24T10:50:51.964 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_append": 0, 2024-03-24T10:50:51.964 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_zero": 0, 2024-03-24T10:50:51.964 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_truncate": 12, 2024-03-24T10:50:51.964 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_delete": 27371, 2024-03-24T10:50:51.965 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_mapext": 0, 2024-03-24T10:50:51.965 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_sparse_read": 0, 2024-03-24T10:50:51.965 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_clonerange": 0, 2024-03-24T10:50:51.965 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_getxattr": 1317, 2024-03-24T10:50:51.965 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_setxattr": 68, 2024-03-24T10:50:51.965 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_cmpxattr": 0, 2024-03-24T10:50:51.965 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_rmxattr": 0, 2024-03-24T10:50:51.965 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_resetxattrs": 0, 2024-03-24T10:50:51.965 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_call": 0, 2024-03-24T10:50:51.966 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_watch": 0, 2024-03-24T10:50:51.966 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_notify": 0, 2024-03-24T10:50:51.966 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T10:50:51.966 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls": 0, 2024-03-24T10:50:51.966 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls_filter": 0, 2024-03-24T10:50:51.966 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_other": 0, 2024-03-24T10:50:51.966 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_active": 0, 2024-03-24T10:50:51.966 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_send": 0, 2024-03-24T10:50:51.967 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_resend": 0, 2024-03-24T10:50:51.967 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_ping": 0, 2024-03-24T10:50:51.967 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_active": 0, 2024-03-24T10:50:51.967 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_send": 0, 2024-03-24T10:50:51.967 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_resend": 0, 2024-03-24T10:50:51.967 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_active": 0, 2024-03-24T10:50:51.967 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_send": 0, 2024-03-24T10:50:51.967 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_resend": 0, 2024-03-24T10:50:51.967 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_active": 0, 2024-03-24T10:50:51.967 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_send": 0, 2024-03-24T10:50:51.968 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_resend": 0, 2024-03-24T10:50:51.968 INFO:teuthology.orchestra.run.smithi116.stdout: "command_active": 0, 2024-03-24T10:50:51.968 INFO:teuthology.orchestra.run.smithi116.stdout: "command_send": 0, 2024-03-24T10:50:51.968 INFO:teuthology.orchestra.run.smithi116.stdout: "command_resend": 0, 2024-03-24T10:50:51.968 INFO:teuthology.orchestra.run.smithi116.stdout: "map_epoch": 98, 2024-03-24T10:50:51.968 INFO:teuthology.orchestra.run.smithi116.stdout: "map_full": 0, 2024-03-24T10:50:51.968 INFO:teuthology.orchestra.run.smithi116.stdout: "map_inc": 13, 2024-03-24T10:50:51.968 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_sessions": 12, 2024-03-24T10:50:51.968 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_open": 12, 2024-03-24T10:50:51.968 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_close": 0, 2024-03-24T10:50:51.969 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_laggy": 0, 2024-03-24T10:50:51.969 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_wr": 1084, 2024-03-24T10:50:51.969 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_rd": 0, 2024-03-24T10:50:51.969 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_del": 345 2024-03-24T10:50:51.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.969 INFO:teuthology.orchestra.run.smithi116.stdout: "oft": { 2024-03-24T10:50:51.969 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_objs": 1, 2024-03-24T10:50:51.969 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_kv_pairs": 0, 2024-03-24T10:50:51.969 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_updates": 0, 2024-03-24T10:50:51.970 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_removes": 0 2024-03-24T10:50:51.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.970 INFO:teuthology.orchestra.run.smithi116.stdout: "osd-slow-ops": { 2024-03-24T10:50:51.970 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_ops_count": 0 2024-03-24T10:50:51.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.970 INFO:teuthology.orchestra.run.smithi116.stdout: "purge_queue": { 2024-03-24T10:50:51.970 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops": 0, 2024-03-24T10:50:51.970 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops_high_water": 10, 2024-03-24T10:50:51.970 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing": 0, 2024-03-24T10:50:51.970 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_high_water": 10, 2024-03-24T10:50:51.971 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed_ops": 27148, 2024-03-24T10:50:51.971 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed": 27148, 2024-03-24T10:50:51.971 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_item_in_journal": 0 2024-03-24T10:50:51.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.971 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T10:50:51.971 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:51.971 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-24T10:50:51.971 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:51.972 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 432862, 2024-03-24T10:50:51.972 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 76311754, 2024-03-24T10:50:51.972 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:51.972 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 432862, 2024-03-24T10:50:51.972 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T10:50:51.972 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T10:50:51.972 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 432862, 2024-03-24T10:50:51.972 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 76311754, 2024-03-24T10:50:51.972 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:51.973 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.973 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.973 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.973 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:51.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.974 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_bytes": { 2024-03-24T10:50:51.974 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:51.974 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-24T10:50:51.974 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:51.974 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T10:50:51.975 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T10:50:51.975 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:51.975 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:51.975 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 73898, 2024-03-24T10:50:51.975 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 497691060, 2024-03-24T10:50:51.975 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 73898, 2024-03-24T10:50:51.975 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 497691060, 2024-03-24T10:50:51.975 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:51.976 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.976 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.976 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.976 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:51.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.976 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_ops": { 2024-03-24T10:50:51.976 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:51.976 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 1024, 2024-03-24T10:50:51.977 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:51.977 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T10:50:51.977 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T10:50:51.977 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:51.977 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:51.977 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 73898, 2024-03-24T10:50:51.977 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 73898, 2024-03-24T10:50:51.977 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 73898, 2024-03-24T10:50:51.977 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 73898, 2024-03-24T10:50:51.977 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:51.978 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.978 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.978 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.978 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:51.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.978 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle": { 2024-03-24T10:50:51.978 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:51.978 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-24T10:50:51.978 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:51.978 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 27148, 2024-03-24T10:50:51.979 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 2744332, 2024-03-24T10:50:51.979 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:51.979 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 27148, 2024-03-24T10:50:51.979 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T10:50:51.979 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T10:50:51.979 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 666, 2024-03-24T10:50:51.979 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 2744332, 2024-03-24T10:50:51.979 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:51.979 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.979 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.980 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.980 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:51.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:51.980 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle-0x559ad38ba500": { 2024-03-24T10:50:51.980 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:51.980 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-24T10:50:51.980 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:51.980 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 123545, 2024-03-24T10:50:51.980 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 487968723, 2024-03-24T10:50:51.981 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:51.981 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 123545, 2024-03-24T10:50:51.981 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T10:50:51.981 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T10:50:51.981 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 43003, 2024-03-24T10:50:51.981 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 487968723, 2024-03-24T10:50:51.981 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:51.981 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:51.981 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:51.982 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:51.982 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:51.982 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:51.982 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T10:50:52.683 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-24T10:50:52.683 INFO:tasks.check_counter:Daemon mds.h mds.imported=9 2024-03-24T10:50:52.683 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-24T10:50:52.684 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-24T10:50:52.684 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T10:50:52.684 DEBUG:tasks.check_counter:Getting stats from k 2024-03-24T10:50:52.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-24T10:50:52.953 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T10:50:53.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:52 smithi023 ceph-mon[32466]: pgmap v336: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:53.211 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:52 smithi116 ceph-mon[36248]: pgmap v336: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:53.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:52 smithi117 ceph-mon[36368]: pgmap v336: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:54.453 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T10:50:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T10:50:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 0, 2024-03-24T10:50:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2, 2024-03-24T10:50:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 0, 2024-03-24T10:50:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 0, 2024-03-24T10:50:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-24T10:50:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 0, 2024-03-24T10:50:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.001077620, 2024-03-24T10:50:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T10:50:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.000249591, 2024-03-24T10:50:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T10:50:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T10:50:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T10:50:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T10:50:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 229, 2024-03-24T10:50:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 217, 2024-03-24T10:50:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 1436368, 2024-03-24T10:50:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 132352, 2024-03-24T10:50:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 2, 2024-03-24T10:50:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 1, 2024-03-24T10:50:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.034050174, 2024-03-24T10:50:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.008028756, 2024-03-24T10:50:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.031781105, 2024-03-24T10:50:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.003942464, 2024-03-24T10:50:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 217, 2024-03-24T10:50:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.006686626, 2024-03-24T10:50:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000030813 2024-03-24T10:50:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1436368, 2024-03-24T10:50:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 132352 2024-03-24T10:50:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T10:50:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 2, 2024-03-24T10:50:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 163, 2024-03-24T10:50:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 320, 2024-03-24T10:50:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 184064, 2024-03-24T10:50:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 5, 2024-03-24T10:50:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 3, 2024-03-24T10:50:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.008112125, 2024-03-24T10:50:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.004697521, 2024-03-24T10:50:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.001915747, 2024-03-24T10:50:54.460 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T10:50:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 160, 2024-03-24T10:50:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.007999784, 2024-03-24T10:50:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000049998 2024-03-24T10:50:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:54.461 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T10:50:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 184064 2024-03-24T10:50:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-24T10:50:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 0, 2024-03-24T10:50:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-24T10:50:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:54.462 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-24T10:50:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-24T10:50:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-24T10:50:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T10:50:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-24T10:50:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T10:50:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-24T10:50:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T10:50:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T10:50:54.463 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T10:50:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T10:50:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T10:50:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-24T10:50:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T10:50:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T10:50:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-24T10:50:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-24T10:50:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-24T10:50:54.464 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-24T10:50:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T10:50:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T10:50:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T10:50:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T10:50:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T10:50:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-24T10:50:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-24T10:50:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-24T10:50:54.465 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T10:50:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T10:50:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-24T10:50:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-24T10:50:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-24T10:50:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-24T10:50:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T10:50:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-24T10:50:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T10:50:54.466 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-24T10:50:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 149057, 2024-03-24T10:50:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 32, 2024-03-24T10:50:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-24T10:50:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-24T10:50:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-24T10:50:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-24T10:50:54.467 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-24T10:50:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-24T10:50:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-24T10:50:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-24T10:50:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 0, 2024-03-24T10:50:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 0, 2024-03-24T10:50:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T10:50:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-24T10:50:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-24T10:50:54.468 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-24T10:50:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 928, 2024-03-24T10:50:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 2, 2024-03-24T10:50:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-24T10:50:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-24T10:50:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-24T10:50:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-24T10:50:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T10:50:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T10:50:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T10:50:54.469 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-24T10:50:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T10:50:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 227, 2024-03-24T10:50:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 1401620, 2024-03-24T10:50:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 227, 2024-03-24T10:50:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T10:50:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T10:50:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 227, 2024-03-24T10:50:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 1401620, 2024-03-24T10:50:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T10:50:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T10:50:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T10:50:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T10:50:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:54.471 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T10:50:54.471 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T10:50:55.066 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-24T10:50:55.066 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-24T10:50:55.066 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-24T10:50:55.066 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-24T10:50:55.066 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-24T10:50:55.066 DEBUG:tasks.check_counter:Getting stats from c 2024-03-24T10:50:55.067 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-24T10:50:55.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:54 smithi117 ceph-mon[36368]: pgmap v337: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:55.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:54 smithi116 ceph-mon[36248]: pgmap v337: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:55.310 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:50:55.417 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:54 smithi023 ceph-mon[32466]: pgmap v337: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:57.036 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:56 smithi117 ceph-mon[36368]: pgmap v338: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:57.037 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-24T10:50:57.037 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T10:50:57.037 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 546, 2024-03-24T10:50:57.038 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 545, 2024-03-24T10:50:57.038 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 775306, 2024-03-24T10:50:57.038 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 185863, 2024-03-24T10:50:57.038 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 5, 2024-03-24T10:50:57.038 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 1, 2024-03-24T10:50:57.038 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.035136233, 2024-03-24T10:50:57.038 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.010189326, 2024-03-24T10:50:57.038 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.012066592, 2024-03-24T10:50:57.039 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.010230489, 2024-03-24T10:50:57.039 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:57.039 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 544, 2024-03-24T10:50:57.039 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.012829096, 2024-03-24T10:50:57.039 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000023582 2024-03-24T10:50:57.039 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.039 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:57.039 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.039 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.040 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.040 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.040 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 175472, 2024-03-24T10:50:57.040 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 40000 2024-03-24T10:50:57.040 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.040 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T10:50:57.040 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 9, 2024-03-24T10:50:57.040 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 53, 2024-03-24T10:50:57.041 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 5559, 2024-03-24T10:50:57.041 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 168014, 2024-03-24T10:50:57.041 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 5, 2024-03-24T10:50:57.041 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 4, 2024-03-24T10:50:57.041 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.004157585, 2024-03-24T10:50:57.041 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.001400449, 2024-03-24T10:50:57.041 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.002301195, 2024-03-24T10:50:57.041 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000095981, 2024-03-24T10:50:57.041 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:57.042 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 52, 2024-03-24T10:50:57.042 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.004928905, 2024-03-24T10:50:57.042 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000094786 2024-03-24T10:50:57.042 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.042 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:57.042 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.042 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.042 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.043 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.043 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-03-24T10:50:57.043 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 166704 2024-03-24T10:50:57.043 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.043 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T10:50:57.043 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 223, 2024-03-24T10:50:57.043 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 225, 2024-03-24T10:50:57.043 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 55462, 2024-03-24T10:50:57.044 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 69360, 2024-03-24T10:50:57.044 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 4, 2024-03-24T10:50:57.044 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 3, 2024-03-24T10:50:57.044 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.015729345, 2024-03-24T10:50:57.044 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.004900346, 2024-03-24T10:50:57.044 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.003997825, 2024-03-24T10:50:57.044 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.005520785, 2024-03-24T10:50:57.044 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:57.045 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 223, 2024-03-24T10:50:57.045 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.006362399, 2024-03-24T10:50:57.045 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000028530 2024-03-24T10:50:57.045 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.045 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:57.045 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.045 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.045 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.046 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.046 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T10:50:57.046 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T10:50:57.046 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.046 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-24T10:50:57.046 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-03-24T10:50:57.046 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-24T10:50:57.047 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.047 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-03-24T10:50:57.047 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-24T10:50:57.047 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-24T10:50:57.047 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1146, 2024-03-24T10:50:57.047 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.021243624, 2024-03-24T10:50:57.047 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000018537 2024-03-24T10:50:57.047 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:57.047 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.048 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-03-24T10:50:57.048 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-24T10:50:57.048 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-24T10:50:57.048 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.048 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.048 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.048 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:57.048 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.049 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-03-24T10:50:57.049 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 0, 2024-03-24T10:50:57.049 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 0, 2024-03-24T10:50:57.049 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-03-24T10:50:57.049 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.049 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.049 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.049 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.049 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-03-24T10:50:57.050 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 0, 2024-03-24T10:50:57.050 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-03-24T10:50:57.050 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-03-24T10:50:57.050 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 0, 2024-03-24T10:50:57.050 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 0, 2024-03-24T10:50:57.050 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-03-24T10:50:57.050 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 5, 2024-03-24T10:50:57.050 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 1, 2024-03-24T10:50:57.051 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 4, 2024-03-24T10:50:57.051 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 0, 2024-03-24T10:50:57.051 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 5, 2024-03-24T10:50:57.051 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 0, 2024-03-24T10:50:57.051 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 0, 2024-03-24T10:50:57.051 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 0, 2024-03-24T10:50:57.051 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 3, 2024-03-24T10:50:57.051 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 0, 2024-03-24T10:50:57.052 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 0, 2024-03-24T10:50:57.052 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 0, 2024-03-24T10:50:57.052 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 0, 2024-03-24T10:50:57.052 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-03-24T10:50:57.052 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-03-24T10:50:57.052 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 0, 2024-03-24T10:50:57.053 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-03-24T10:50:57.053 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-03-24T10:50:57.053 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 0, 2024-03-24T10:50:57.053 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 0, 2024-03-24T10:50:57.053 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 0, 2024-03-24T10:50:57.053 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 0, 2024-03-24T10:50:57.053 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-03-24T10:50:57.053 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 0, 2024-03-24T10:50:57.054 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-03-24T10:50:57.054 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 68, 2024-03-24T10:50:57.054 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 21196213, 2024-03-24T10:50:57.054 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 6, 2024-03-24T10:50:57.054 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T10:50:57.054 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-03-24T10:50:57.054 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T10:50:57.054 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-03-24T10:50:57.054 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 0, 2024-03-24T10:50:57.055 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T10:50:57.055 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T10:50:57.055 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 0, 2024-03-24T10:50:57.055 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 0, 2024-03-24T10:50:57.055 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T10:50:57.055 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 0, 2024-03-24T10:50:57.056 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T10:50:57.056 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T10:50:57.056 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T10:50:57.056 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 0, 2024-03-24T10:50:57.056 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 0, 2024-03-24T10:50:57.056 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 0, 2024-03-24T10:50:57.056 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 0 2024-03-24T10:50:57.056 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.057 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-03-24T10:50:57.057 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 0, 2024-03-24T10:50:57.057 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 0, 2024-03-24T10:50:57.057 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 0, 2024-03-24T10:50:57.057 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 0, 2024-03-24T10:50:57.057 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 0, 2024-03-24T10:50:57.057 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 0, 2024-03-24T10:50:57.057 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-03-24T10:50:57.058 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-03-24T10:50:57.058 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 0, 2024-03-24T10:50:57.058 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 0, 2024-03-24T10:50:57.058 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 0, 2024-03-24T10:50:57.058 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-03-24T10:50:57.058 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-03-24T10:50:57.058 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-03-24T10:50:57.058 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-03-24T10:50:57.059 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-03-24T10:50:57.059 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-03-24T10:50:57.059 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_path": 0, 2024-03-24T10:50:57.059 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_inode": 0, 2024-03-24T10:50:57.059 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T10:50:57.059 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 0, 2024-03-24T10:50:57.059 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-03-24T10:50:57.059 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 0, 2024-03-24T10:50:57.059 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-03-24T10:50:57.060 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-03-24T10:50:57.060 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.060 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-03-24T10:50:57.060 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-03-24T10:50:57.060 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 0, 2024-03-24T10:50:57.060 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 0, 2024-03-24T10:50:57.060 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 0, 2024-03-24T10:50:57.060 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 45, 2024-03-24T10:50:57.061 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-03-24T10:50:57.061 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 0, 2024-03-24T10:50:57.061 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 0, 2024-03-24T10:50:57.061 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 0, 2024-03-24T10:50:57.061 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 0, 2024-03-24T10:50:57.061 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 2, 2024-03-24T10:50:57.061 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 4, 2024-03-24T10:50:57.061 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-03-24T10:50:57.061 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 0, 2024-03-24T10:50:57.062 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 8068325, 2024-03-24T10:50:57.062 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 8350225, 2024-03-24T10:50:57.062 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 8350113, 2024-03-24T10:50:57.062 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-03-24T10:50:57.062 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.062 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.062 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.063 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.063 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 45 2024-03-24T10:50:57.063 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.063 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-03-24T10:50:57.063 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 8, 2024-03-24T10:50:57.063 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 8, 2024-03-24T10:50:57.063 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 0, 2024-03-24T10:50:57.063 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 7, 2024-03-24T10:50:57.063 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 7, 2024-03-24T10:50:57.064 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 0, 2024-03-24T10:50:57.064 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 5, 2024-03-24T10:50:57.064 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 5, 2024-03-24T10:50:57.064 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 0, 2024-03-24T10:50:57.064 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 0, 2024-03-24T10:50:57.064 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 0, 2024-03-24T10:50:57.064 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 0, 2024-03-24T10:50:57.064 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 47820, 2024-03-24T10:50:57.065 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 215420 2024-03-24T10:50:57.065 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.065 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-03-24T10:50:57.065 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 0, 2024-03-24T10:50:57.065 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 0, 2024-03-24T10:50:57.065 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 0, 2024-03-24T10:50:57.065 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 0, 2024-03-24T10:50:57.065 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 0, 2024-03-24T10:50:57.066 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-03-24T10:50:57.066 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.066 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.066 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.066 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.066 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-03-24T10:50:57.066 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.066 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.067 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.067 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.067 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-03-24T10:50:57.067 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.067 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.067 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.067 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.067 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-03-24T10:50:57.067 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.068 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.068 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.068 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.068 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-03-24T10:50:57.068 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.068 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.068 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.068 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.069 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-03-24T10:50:57.069 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.069 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.069 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.069 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.069 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-03-24T10:50:57.069 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.069 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.070 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.070 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.070 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-03-24T10:50:57.070 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.070 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.070 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.070 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.070 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-03-24T10:50:57.071 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.071 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.071 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.071 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.071 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-03-24T10:50:57.071 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.072 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.072 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.072 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.072 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-03-24T10:50:57.072 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.072 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.072 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.072 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.073 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-03-24T10:50:57.073 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.073 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.073 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.074 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.074 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-03-24T10:50:57.074 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.074 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.074 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.074 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.074 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-03-24T10:50:57.074 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.075 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.075 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.075 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.075 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-03-24T10:50:57.075 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.075 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.075 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.076 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.076 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-03-24T10:50:57.076 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.076 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.076 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.076 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.076 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-03-24T10:50:57.076 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.076 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.077 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.077 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.077 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-03-24T10:50:57.077 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.077 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.077 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.077 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.077 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-03-24T10:50:57.078 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.078 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.078 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.078 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.078 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-03-24T10:50:57.078 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.078 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.078 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.078 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.079 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-03-24T10:50:57.079 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.079 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.079 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.079 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.079 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-03-24T10:50:57.079 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.079 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.080 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.080 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.080 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-03-24T10:50:57.080 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.080 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.080 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.080 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.080 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-03-24T10:50:57.081 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.081 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.081 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.081 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.081 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-03-24T10:50:57.081 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.081 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.081 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.081 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.082 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-03-24T10:50:57.082 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.082 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.082 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.082 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.082 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-03-24T10:50:57.082 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.082 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.083 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.083 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.083 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-03-24T10:50:57.083 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.083 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.083 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.083 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.083 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-03-24T10:50:57.084 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.084 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.084 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.084 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.084 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-03-24T10:50:57.084 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-03-24T10:50:57.084 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-03-24T10:50:57.084 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.085 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.085 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.085 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:57.085 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.085 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-03-24T10:50:57.085 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 2, 2024-03-24T10:50:57.085 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 2, 2024-03-24T10:50:57.085 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 0, 2024-03-24T10:50:57.085 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 0, 2024-03-24T10:50:57.086 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-03-24T10:50:57.086 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 0, 2024-03-24T10:50:57.086 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 0, 2024-03-24T10:50:57.086 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 2070, 2024-03-24T10:50:57.086 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-03-24T10:50:57.086 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.086 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-24T10:50:57.086 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-24T10:50:57.087 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-24T10:50:57.087 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T10:50:57.087 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-24T10:50:57.087 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T10:50:57.087 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-24T10:50:57.087 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T10:50:57.087 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T10:50:57.087 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T10:50:57.088 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T10:50:57.088 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T10:50:57.088 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-24T10:50:57.088 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T10:50:57.088 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T10:50:57.088 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-24T10:50:57.088 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-24T10:50:57.088 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-24T10:50:57.089 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-24T10:50:57.089 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T10:50:57.089 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T10:50:57.089 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T10:50:57.089 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T10:50:57.089 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T10:50:57.089 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-24T10:50:57.089 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-24T10:50:57.089 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-24T10:50:57.090 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T10:50:57.090 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T10:50:57.090 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-24T10:50:57.090 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-24T10:50:57.090 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-24T10:50:57.090 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-24T10:50:57.090 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T10:50:57.090 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-24T10:50:57.091 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T10:50:57.091 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-24T10:50:57.091 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 190603, 2024-03-24T10:50:57.091 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 58, 2024-03-24T10:50:57.091 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-03-24T10:50:57.091 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-03-24T10:50:57.091 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-24T10:50:57.091 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-24T10:50:57.092 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-24T10:50:57.092 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-24T10:50:57.092 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-24T10:50:57.092 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-24T10:50:57.092 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 8224, 2024-03-24T10:50:57.092 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 171, 2024-03-24T10:50:57.092 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T10:50:57.092 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-24T10:50:57.092 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-24T10:50:57.093 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-24T10:50:57.093 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 66694, 2024-03-24T10:50:57.093 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 333, 2024-03-24T10:50:57.093 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-24T10:50:57.093 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-24T10:50:57.093 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-24T10:50:57.093 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-24T10:50:57.094 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.094 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-03-24T10:50:57.094 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-03-24T10:50:57.094 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-03-24T10:50:57.094 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 696, 2024-03-24T10:50:57.094 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 26, 2024-03-24T10:50:57.094 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 4, 2024-03-24T10:50:57.094 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 696, 2024-03-24T10:50:57.094 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-03-24T10:50:57.095 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 696, 2024-03-24T10:50:57.095 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.260995049, 2024-03-24T10:50:57.095 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000374992 2024-03-24T10:50:57.095 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.095 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-03-24T10:50:57.095 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-03-24T10:50:57.095 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 696, 2024-03-24T10:50:57.095 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 697 2024-03-24T10:50:57.096 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.096 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 696, 2024-03-24T10:50:57.096 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 696, 2024-03-24T10:50:57.096 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 0, 2024-03-24T10:50:57.096 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-03-24T10:50:57.096 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-03-24T10:50:57.096 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 440, 2024-03-24T10:50:57.096 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 0, 2024-03-24T10:50:57.096 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 253, 2024-03-24T10:50:57.097 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 0, 2024-03-24T10:50:57.097 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 0, 2024-03-24T10:50:57.097 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-03-24T10:50:57.097 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-03-24T10:50:57.097 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-03-24T10:50:57.097 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 0, 2024-03-24T10:50:57.097 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 0, 2024-03-24T10:50:57.097 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-03-24T10:50:57.098 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-03-24T10:50:57.098 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-03-24T10:50:57.098 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 2, 2024-03-24T10:50:57.098 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 0, 2024-03-24T10:50:57.098 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-03-24T10:50:57.098 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-03-24T10:50:57.098 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-03-24T10:50:57.098 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-03-24T10:50:57.099 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-03-24T10:50:57.099 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-03-24T10:50:57.099 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T10:50:57.099 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-03-24T10:50:57.099 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-03-24T10:50:57.099 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 0, 2024-03-24T10:50:57.099 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-03-24T10:50:57.099 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-03-24T10:50:57.099 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-03-24T10:50:57.100 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-03-24T10:50:57.100 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-03-24T10:50:57.100 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-03-24T10:50:57.100 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-03-24T10:50:57.100 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-03-24T10:50:57.100 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-03-24T10:50:57.100 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-03-24T10:50:57.100 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-03-24T10:50:57.100 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-03-24T10:50:57.101 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-03-24T10:50:57.101 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-03-24T10:50:57.101 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-03-24T10:50:57.101 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-03-24T10:50:57.101 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 98, 2024-03-24T10:50:57.101 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-03-24T10:50:57.101 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 6, 2024-03-24T10:50:57.101 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 5, 2024-03-24T10:50:57.102 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 5, 2024-03-24T10:50:57.102 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-03-24T10:50:57.102 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-03-24T10:50:57.102 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 0, 2024-03-24T10:50:57.102 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 2, 2024-03-24T10:50:57.102 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 0 2024-03-24T10:50:57.102 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.102 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-03-24T10:50:57.103 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 0, 2024-03-24T10:50:57.103 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-03-24T10:50:57.103 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-03-24T10:50:57.103 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-03-24T10:50:57.103 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.103 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-03-24T10:50:57.103 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-03-24T10:50:57.103 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.104 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-03-24T10:50:57.104 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-03-24T10:50:57.104 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T10:50:57.104 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-03-24T10:50:57.104 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 0, 2024-03-24T10:50:57.104 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 0, 2024-03-24T10:50:57.104 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 0, 2024-03-24T10:50:57.104 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-03-24T10:50:57.104 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.105 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T10:50:57.105 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:50:57.105 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T10:50:57.105 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:50:57.105 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 774, 2024-03-24T10:50:57.105 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 759734, 2024-03-24T10:50:57.105 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:57.105 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 774, 2024-03-24T10:50:57.106 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T10:50:57.106 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T10:50:57.106 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 774, 2024-03-24T10:50:57.106 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 759734, 2024-03-24T10:50:57.106 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:50:57.106 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.106 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.106 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.107 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:57.107 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.107 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-03-24T10:50:57.107 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:50:57.107 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T10:50:57.107 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:50:57.107 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T10:50:57.107 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T10:50:57.108 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:57.108 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:57.108 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 696, 2024-03-24T10:50:57.108 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 457228, 2024-03-24T10:50:57.108 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 696, 2024-03-24T10:50:57.108 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 457228, 2024-03-24T10:50:57.108 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:50:57.108 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.109 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.109 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.109 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:57.109 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.109 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-03-24T10:50:57.109 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:50:57.109 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-03-24T10:50:57.109 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:50:57.110 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T10:50:57.110 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T10:50:57.110 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:57.110 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:57.110 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 696, 2024-03-24T10:50:57.110 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 696, 2024-03-24T10:50:57.110 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 696, 2024-03-24T10:50:57.110 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 696, 2024-03-24T10:50:57.110 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:50:57.111 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.111 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.111 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.111 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:57.111 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.111 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-03-24T10:50:57.111 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:50:57.111 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-24T10:50:57.112 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:50:57.112 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T10:50:57.112 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T10:50:57.112 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:57.112 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:57.112 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T10:50:57.112 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T10:50:57.112 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-03-24T10:50:57.113 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-03-24T10:50:57.113 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:50:57.113 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.113 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.113 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.113 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:57.113 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:57.113 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x55e81cec06e0": { 2024-03-24T10:50:57.113 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:50:57.114 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-24T10:50:57.114 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:50:57.114 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T10:50:57.114 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T10:50:57.114 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:57.114 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:57.114 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T10:50:57.114 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T10:50:57.115 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-03-24T10:50:57.115 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-03-24T10:50:57.115 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:50:57.115 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:57.115 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:57.115 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:57.115 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:57.115 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:57.116 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-24T10:50:57.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:56 smithi116 ceph-mon[36248]: pgmap v338: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:57.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:56 smithi023 ceph-mon[32466]: pgmap v338: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:50:57.781 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-24T10:50:57.782 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-24T10:50:57.782 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-03-24T10:50:57.782 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-24T10:50:57.782 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T10:50:57.782 DEBUG:tasks.check_counter:Getting stats from f 2024-03-24T10:50:57.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-24T10:50:58.081 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:50:59.153 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:50:58 smithi117 ceph-mon[36368]: pgmap v339: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:59.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:50:58 smithi116 ceph-mon[36248]: pgmap v339: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:59.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:50:58 smithi023 ceph-mon[32466]: pgmap v339: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T10:50:59.777 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-24T10:50:59.778 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T10:50:59.778 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 2193, 2024-03-24T10:50:59.778 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 2195, 2024-03-24T10:50:59.778 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 623968, 2024-03-24T10:50:59.778 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 682030, 2024-03-24T10:50:59.778 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 6, 2024-03-24T10:50:59.779 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 2, 2024-03-24T10:50:59.779 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.119752590, 2024-03-24T10:50:59.779 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.035994170, 2024-03-24T10:50:59.779 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.030462450, 2024-03-24T10:50:59.779 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.044522551, 2024-03-24T10:50:59.779 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:59.779 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2193, 2024-03-24T10:50:59.780 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.045707980, 2024-03-24T10:50:59.780 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000020842 2024-03-24T10:50:59.780 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.780 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:59.780 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.780 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.780 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.781 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.781 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T10:50:59.781 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T10:50:59.781 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.781 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T10:50:59.781 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 8, 2024-03-24T10:50:59.781 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 166, 2024-03-24T10:50:59.781 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 3900, 2024-03-24T10:50:59.782 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 447596, 2024-03-24T10:50:59.782 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 6, 2024-03-24T10:50:59.782 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 4, 2024-03-24T10:50:59.782 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.008103207, 2024-03-24T10:50:59.782 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.004425071, 2024-03-24T10:50:59.782 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.002220320, 2024-03-24T10:50:59.782 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000066892, 2024-03-24T10:50:59.783 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:59.783 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 164, 2024-03-24T10:50:59.783 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.009048177, 2024-03-24T10:50:59.783 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000055171 2024-03-24T10:50:59.783 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.783 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:59.784 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.784 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.784 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.784 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.784 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-24T10:50:59.784 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 446912 2024-03-24T10:50:59.784 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.785 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T10:50:59.785 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 251, 2024-03-24T10:50:59.785 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 232, 2024-03-24T10:50:59.785 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 1492028, 2024-03-24T10:50:59.785 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 134055, 2024-03-24T10:50:59.785 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 3, 2024-03-24T10:50:59.785 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 2, 2024-03-24T10:50:59.786 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.033856435, 2024-03-24T10:50:59.786 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.008085228, 2024-03-24T10:50:59.786 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.032762521, 2024-03-24T10:50:59.786 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.003639893, 2024-03-24T10:50:59.786 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:50:59.786 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 231, 2024-03-24T10:50:59.786 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.007531063, 2024-03-24T10:50:59.787 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000032602 2024-03-24T10:50:59.787 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.787 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:50:59.787 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.787 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.787 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.787 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.787 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1491760, 2024-03-24T10:50:59.788 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 133744 2024-03-24T10:50:59.788 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.788 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-24T10:50:59.788 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-03-24T10:50:59.788 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-24T10:50:59.788 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.788 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-03-24T10:50:59.788 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-24T10:50:59.789 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-24T10:50:59.789 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 3728, 2024-03-24T10:50:59.789 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.068608687, 2024-03-24T10:50:59.789 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000018403 2024-03-24T10:50:59.789 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:59.789 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.789 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-03-24T10:50:59.789 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-24T10:50:59.790 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-24T10:50:59.790 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.790 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.790 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.790 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:59.790 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.790 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-03-24T10:50:59.790 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 0, 2024-03-24T10:50:59.791 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 0, 2024-03-24T10:50:59.791 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-03-24T10:50:59.791 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.791 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.791 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.791 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.791 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-03-24T10:50:59.792 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 0, 2024-03-24T10:50:59.792 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-03-24T10:50:59.792 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-03-24T10:50:59.792 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 0, 2024-03-24T10:50:59.792 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 0, 2024-03-24T10:50:59.792 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-03-24T10:50:59.792 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 0, 2024-03-24T10:50:59.792 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 0, 2024-03-24T10:50:59.793 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 0, 2024-03-24T10:50:59.793 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 0, 2024-03-24T10:50:59.793 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 0, 2024-03-24T10:50:59.793 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 0, 2024-03-24T10:50:59.793 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 0, 2024-03-24T10:50:59.793 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 0, 2024-03-24T10:50:59.793 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 1, 2024-03-24T10:50:59.794 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 0, 2024-03-24T10:50:59.794 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 0, 2024-03-24T10:50:59.794 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 0, 2024-03-24T10:50:59.794 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 0, 2024-03-24T10:50:59.794 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-03-24T10:50:59.794 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-03-24T10:50:59.794 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 0, 2024-03-24T10:50:59.794 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-03-24T10:50:59.795 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-03-24T10:50:59.795 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 0, 2024-03-24T10:50:59.795 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 0, 2024-03-24T10:50:59.795 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 0, 2024-03-24T10:50:59.795 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 0, 2024-03-24T10:50:59.795 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-03-24T10:50:59.795 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 0, 2024-03-24T10:50:59.796 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-03-24T10:50:59.796 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 0, 2024-03-24T10:50:59.796 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 0, 2024-03-24T10:50:59.796 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 0, 2024-03-24T10:50:59.796 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T10:50:59.796 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-03-24T10:50:59.796 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T10:50:59.796 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-03-24T10:50:59.797 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 0, 2024-03-24T10:50:59.797 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T10:50:59.797 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T10:50:59.797 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 0, 2024-03-24T10:50:59.797 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 0, 2024-03-24T10:50:59.797 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T10:50:59.797 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 0, 2024-03-24T10:50:59.798 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T10:50:59.798 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T10:50:59.798 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T10:50:59.798 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 0, 2024-03-24T10:50:59.798 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 0, 2024-03-24T10:50:59.798 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 0, 2024-03-24T10:50:59.798 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 0 2024-03-24T10:50:59.798 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.799 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-03-24T10:50:59.799 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 0, 2024-03-24T10:50:59.799 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 0, 2024-03-24T10:50:59.799 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 0, 2024-03-24T10:50:59.799 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 0, 2024-03-24T10:50:59.799 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 0, 2024-03-24T10:50:59.799 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 0, 2024-03-24T10:50:59.800 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-03-24T10:50:59.800 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-03-24T10:50:59.800 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 0, 2024-03-24T10:50:59.800 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 0, 2024-03-24T10:50:59.800 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 0, 2024-03-24T10:50:59.800 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-03-24T10:50:59.800 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-03-24T10:50:59.800 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-03-24T10:50:59.801 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-03-24T10:50:59.801 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-03-24T10:50:59.801 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-03-24T10:50:59.801 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_path": 0, 2024-03-24T10:50:59.801 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_inode": 0, 2024-03-24T10:50:59.801 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T10:50:59.801 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 0, 2024-03-24T10:50:59.801 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-03-24T10:50:59.802 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 0, 2024-03-24T10:50:59.802 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-03-24T10:50:59.802 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-03-24T10:50:59.802 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.802 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-03-24T10:50:59.802 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-03-24T10:50:59.802 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 0, 2024-03-24T10:50:59.803 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 0, 2024-03-24T10:50:59.803 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 0, 2024-03-24T10:50:59.803 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 10, 2024-03-24T10:50:59.803 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-03-24T10:50:59.803 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 0, 2024-03-24T10:50:59.803 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 0, 2024-03-24T10:50:59.803 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 0, 2024-03-24T10:50:59.804 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 0, 2024-03-24T10:50:59.804 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 2, 2024-03-24T10:50:59.804 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 2, 2024-03-24T10:50:59.804 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-03-24T10:50:59.804 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 0, 2024-03-24T10:50:59.804 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 4194304, 2024-03-24T10:50:59.804 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 4197187, 2024-03-24T10:50:59.804 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 4197046, 2024-03-24T10:50:59.805 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-03-24T10:50:59.805 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.805 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.805 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.805 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.805 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 10 2024-03-24T10:50:59.805 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.805 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-03-24T10:50:59.806 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 3, 2024-03-24T10:50:59.806 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 3, 2024-03-24T10:50:59.806 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 0, 2024-03-24T10:50:59.806 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 1, 2024-03-24T10:50:59.806 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 1, 2024-03-24T10:50:59.806 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 0, 2024-03-24T10:50:59.806 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 0, 2024-03-24T10:50:59.806 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 0, 2024-03-24T10:50:59.807 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 0, 2024-03-24T10:50:59.807 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 0, 2024-03-24T10:50:59.807 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 0, 2024-03-24T10:50:59.807 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 0, 2024-03-24T10:50:59.807 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 48936, 2024-03-24T10:50:59.807 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 215420 2024-03-24T10:50:59.807 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.808 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-03-24T10:50:59.808 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 0, 2024-03-24T10:50:59.808 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 0, 2024-03-24T10:50:59.808 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 0, 2024-03-24T10:50:59.808 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 0, 2024-03-24T10:50:59.808 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 0, 2024-03-24T10:50:59.808 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-03-24T10:50:59.808 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.809 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.809 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.809 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.809 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-03-24T10:50:59.809 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.809 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.809 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.809 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.810 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-03-24T10:50:59.810 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.810 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.810 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.810 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.810 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-03-24T10:50:59.811 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.811 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.811 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.811 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.811 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-03-24T10:50:59.811 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.811 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.811 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.812 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.812 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-03-24T10:50:59.812 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.812 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.812 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.812 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.813 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-03-24T10:50:59.813 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.813 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.813 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.813 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.813 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-03-24T10:50:59.813 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.813 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.814 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.814 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.814 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-03-24T10:50:59.814 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.814 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.814 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.814 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.815 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-03-24T10:50:59.815 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.815 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.815 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.815 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.815 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-03-24T10:50:59.815 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.816 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.816 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.816 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.816 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-03-24T10:50:59.816 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.817 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.817 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.817 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.817 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-03-24T10:50:59.817 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.817 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.817 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.818 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.818 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-03-24T10:50:59.818 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.818 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.818 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.818 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.819 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-03-24T10:50:59.819 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.819 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.819 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.819 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.819 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-03-24T10:50:59.819 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.819 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.820 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.820 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.820 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-03-24T10:50:59.820 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.820 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.820 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.820 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.820 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-03-24T10:50:59.821 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.821 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.821 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.821 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.821 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-03-24T10:50:59.821 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.821 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.822 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.822 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.822 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-03-24T10:50:59.822 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.822 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.822 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.822 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.822 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-03-24T10:50:59.823 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.823 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.823 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.823 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.823 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-03-24T10:50:59.823 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.823 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.823 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.824 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.824 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-03-24T10:50:59.824 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.824 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.824 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.824 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.824 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-03-24T10:50:59.825 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.825 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.825 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.825 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.825 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-03-24T10:50:59.825 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.825 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.825 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.826 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.826 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-03-24T10:50:59.826 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.826 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.826 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.826 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.826 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-03-24T10:50:59.827 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.827 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.827 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.827 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.827 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-03-24T10:50:59.827 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.827 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.827 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.828 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.828 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-03-24T10:50:59.828 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.828 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.828 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.828 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.828 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-03-24T10:50:59.828 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-03-24T10:50:59.829 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-03-24T10:50:59.829 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.829 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.829 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.829 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:59.829 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.829 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-03-24T10:50:59.830 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 2, 2024-03-24T10:50:59.830 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 5, 2024-03-24T10:50:59.830 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 3, 2024-03-24T10:50:59.830 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 5, 2024-03-24T10:50:59.830 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-03-24T10:50:59.830 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 0, 2024-03-24T10:50:59.830 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 0, 2024-03-24T10:50:59.830 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 4203, 2024-03-24T10:50:59.831 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-03-24T10:50:59.831 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.831 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-24T10:50:59.831 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-24T10:50:59.831 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-24T10:50:59.831 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T10:50:59.831 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-24T10:50:59.831 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T10:50:59.832 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-24T10:50:59.832 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T10:50:59.832 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T10:50:59.832 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T10:50:59.832 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T10:50:59.832 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T10:50:59.832 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-24T10:50:59.833 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T10:50:59.833 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T10:50:59.833 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-24T10:50:59.833 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-24T10:50:59.833 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-24T10:50:59.833 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-24T10:50:59.833 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T10:50:59.833 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T10:50:59.834 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T10:50:59.834 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T10:50:59.834 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T10:50:59.834 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-24T10:50:59.834 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-24T10:50:59.834 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-24T10:50:59.834 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T10:50:59.835 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T10:50:59.835 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-24T10:50:59.835 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-24T10:50:59.835 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-24T10:50:59.835 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-24T10:50:59.835 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T10:50:59.835 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-24T10:50:59.835 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T10:50:59.836 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-24T10:50:59.836 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 181330, 2024-03-24T10:50:59.836 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 48, 2024-03-24T10:50:59.836 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-03-24T10:50:59.836 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-03-24T10:50:59.836 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-24T10:50:59.836 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-24T10:50:59.837 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-24T10:50:59.837 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-24T10:50:59.837 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-24T10:50:59.837 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-24T10:50:59.837 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 8224, 2024-03-24T10:50:59.837 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 171, 2024-03-24T10:50:59.837 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T10:50:59.837 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-24T10:50:59.838 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-24T10:50:59.838 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-24T10:50:59.838 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 8464, 2024-03-24T10:50:59.838 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 13, 2024-03-24T10:50:59.838 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-24T10:50:59.838 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-24T10:50:59.838 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-24T10:50:59.838 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-24T10:50:59.839 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.839 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-03-24T10:50:59.839 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-03-24T10:50:59.839 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-03-24T10:50:59.839 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 2196, 2024-03-24T10:50:59.839 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 21, 2024-03-24T10:50:59.839 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 3, 2024-03-24T10:50:59.840 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 2196, 2024-03-24T10:50:59.840 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-03-24T10:50:59.840 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2196, 2024-03-24T10:50:59.840 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.890982854, 2024-03-24T10:50:59.840 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000405729 2024-03-24T10:50:59.840 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.840 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-03-24T10:50:59.840 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-03-24T10:50:59.841 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2196, 2024-03-24T10:50:59.841 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 2197 2024-03-24T10:50:59.841 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.841 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 2196, 2024-03-24T10:50:59.841 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 2196, 2024-03-24T10:50:59.841 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 0, 2024-03-24T10:50:59.841 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-03-24T10:50:59.842 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-03-24T10:50:59.842 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 1421, 2024-03-24T10:50:59.842 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 0, 2024-03-24T10:50:59.842 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 773, 2024-03-24T10:50:59.842 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 0, 2024-03-24T10:50:59.842 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 0, 2024-03-24T10:50:59.842 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-03-24T10:50:59.842 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-03-24T10:50:59.843 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-03-24T10:50:59.843 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 0, 2024-03-24T10:50:59.843 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 0, 2024-03-24T10:50:59.843 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-03-24T10:50:59.843 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-03-24T10:50:59.843 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-03-24T10:50:59.843 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 1, 2024-03-24T10:50:59.843 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 0, 2024-03-24T10:50:59.844 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-03-24T10:50:59.844 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-03-24T10:50:59.844 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-03-24T10:50:59.844 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-03-24T10:50:59.844 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-03-24T10:50:59.844 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-03-24T10:50:59.844 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T10:50:59.845 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-03-24T10:50:59.845 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-03-24T10:50:59.845 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 0, 2024-03-24T10:50:59.845 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-03-24T10:50:59.845 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-03-24T10:50:59.845 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-03-24T10:50:59.845 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-03-24T10:50:59.846 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-03-24T10:50:59.846 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-03-24T10:50:59.846 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-03-24T10:50:59.846 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-03-24T10:50:59.846 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-03-24T10:50:59.846 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-03-24T10:50:59.846 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-03-24T10:50:59.846 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-03-24T10:50:59.847 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-03-24T10:50:59.847 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-03-24T10:50:59.847 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-03-24T10:50:59.847 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-03-24T10:50:59.847 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 98, 2024-03-24T10:50:59.847 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-03-24T10:50:59.847 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 13, 2024-03-24T10:50:59.847 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 4, 2024-03-24T10:50:59.848 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 4, 2024-03-24T10:50:59.848 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-03-24T10:50:59.848 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-03-24T10:50:59.848 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 0, 2024-03-24T10:50:59.848 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 2, 2024-03-24T10:50:59.848 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 0 2024-03-24T10:50:59.848 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.848 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-03-24T10:50:59.849 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 0, 2024-03-24T10:50:59.849 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-03-24T10:50:59.849 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-03-24T10:50:59.849 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-03-24T10:50:59.849 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.849 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-03-24T10:50:59.849 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-03-24T10:50:59.849 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.850 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-03-24T10:50:59.850 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-03-24T10:50:59.850 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T10:50:59.850 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-03-24T10:50:59.850 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 0, 2024-03-24T10:50:59.850 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 0, 2024-03-24T10:50:59.850 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 0, 2024-03-24T10:50:59.851 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-03-24T10:50:59.851 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.851 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T10:50:59.851 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:50:59.851 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T10:50:59.851 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:50:59.851 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 2448, 2024-03-24T10:50:59.851 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 1883779, 2024-03-24T10:50:59.852 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:59.852 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 2448, 2024-03-24T10:50:59.852 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T10:50:59.852 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T10:50:59.852 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2448, 2024-03-24T10:50:59.852 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 1883779, 2024-03-24T10:50:59.852 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:50:59.852 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.853 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.853 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.853 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:59.853 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.853 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-03-24T10:50:59.853 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:50:59.853 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T10:50:59.853 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:50:59.854 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T10:50:59.854 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T10:50:59.854 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:59.854 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:59.854 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 2196, 2024-03-24T10:50:59.854 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 2888, 2024-03-24T10:50:59.854 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2196, 2024-03-24T10:50:59.854 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 2888, 2024-03-24T10:50:59.855 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:50:59.855 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.855 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.855 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.855 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:59.855 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.855 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-03-24T10:50:59.856 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:50:59.856 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-03-24T10:50:59.856 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:50:59.856 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T10:50:59.856 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T10:50:59.856 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:59.856 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:59.856 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 2196, 2024-03-24T10:50:59.857 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 2196, 2024-03-24T10:50:59.857 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2196, 2024-03-24T10:50:59.857 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 2196, 2024-03-24T10:50:59.857 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:50:59.857 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.857 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.857 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.857 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:59.858 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.858 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-03-24T10:50:59.858 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:50:59.858 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-24T10:50:59.858 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:50:59.858 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T10:50:59.858 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T10:50:59.859 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:59.859 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:59.859 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T10:50:59.859 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T10:50:59.859 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-03-24T10:50:59.859 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-03-24T10:50:59.859 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:50:59.860 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.860 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.860 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.860 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:59.860 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:50:59.860 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x55cad18b46e0": { 2024-03-24T10:50:59.860 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:50:59.861 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-24T10:50:59.861 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:50:59.861 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T10:50:59.861 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T10:50:59.861 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:50:59.861 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T10:50:59.862 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T10:50:59.862 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T10:50:59.862 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-03-24T10:50:59.862 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-03-24T10:50:59.862 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:50:59.862 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:50:59.862 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:50:59.863 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:50:59.863 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:59.863 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:50:59.863 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-24T10:51:00.546 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-24T10:51:00.547 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-24T10:51:00.547 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-03-24T10:51:00.547 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-24T10:51:00.547 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T10:51:00.547 DEBUG:tasks.check_counter:Getting stats from i 2024-03-24T10:51:00.547 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-24T10:51:00.794 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:51:01.181 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:00 smithi117 ceph-mon[36368]: pgmap v340: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:51:01.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:00 smithi116 ceph-mon[36248]: pgmap v340: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:51:01.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:00 smithi023 ceph-mon[32466]: pgmap v340: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T10:51:02.538 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-24T10:51:02.538 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T10:51:02.538 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 670, 2024-03-24T10:51:02.538 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 721, 2024-03-24T10:51:02.538 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 208691, 2024-03-24T10:51:02.538 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 450980, 2024-03-24T10:51:02.539 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 14, 2024-03-24T10:51:02.539 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 3, 2024-03-24T10:51:02.539 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.047205543, 2024-03-24T10:51:02.539 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.016465104, 2024-03-24T10:51:02.539 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.026586784, 2024-03-24T10:51:02.539 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.002985476, 2024-03-24T10:51:02.539 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:51:02.539 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 722, 2024-03-24T10:51:02.540 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.034685882, 2024-03-24T10:51:02.540 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000048041 2024-03-24T10:51:02.540 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.540 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:51:02.540 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1295, 2024-03-24T10:51:02.540 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000201714, 2024-03-24T10:51:02.540 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000155 2024-03-24T10:51:02.540 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.540 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 6608, 2024-03-24T10:51:02.541 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 9184 2024-03-24T10:51:02.541 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.541 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T10:51:02.541 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 95629, 2024-03-24T10:51:02.541 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 51532, 2024-03-24T10:51:02.541 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 42590470, 2024-03-24T10:51:02.541 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 8893516, 2024-03-24T10:51:02.541 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 18, 2024-03-24T10:51:02.541 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 7, 2024-03-24T10:51:02.542 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 3.458242004, 2024-03-24T10:51:02.542 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 1.466358534, 2024-03-24T10:51:02.542 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 2.761983387, 2024-03-24T10:51:02.542 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000243824, 2024-03-24T10:51:02.542 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:51:02.542 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 51527, 2024-03-24T10:51:02.542 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.570627583, 2024-03-24T10:51:02.542 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000011074 2024-03-24T10:51:02.543 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.543 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:51:02.543 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 146915, 2024-03-24T10:51:02.543 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.013592684, 2024-03-24T10:51:02.543 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000092 2024-03-24T10:51:02.543 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.543 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 2261552, 2024-03-24T10:51:02.543 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 1490384 2024-03-24T10:51:02.543 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.543 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T10:51:02.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 334, 2024-03-24T10:51:02.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 312, 2024-03-24T10:51:02.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 1514964, 2024-03-24T10:51:02.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 544329, 2024-03-24T10:51:02.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 41, 2024-03-24T10:51:02.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 6, 2024-03-24T10:51:02.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.044420724, 2024-03-24T10:51:02.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.009618352, 2024-03-24T10:51:02.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.041564190, 2024-03-24T10:51:02.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.003788322, 2024-03-24T10:51:02.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:51:02.545 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 311, 2024-03-24T10:51:02.545 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.017806364, 2024-03-24T10:51:02.545 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000057255 2024-03-24T10:51:02.545 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:51:02.546 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 42, 2024-03-24T10:51:02.546 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000003833, 2024-03-24T10:51:02.546 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000091 2024-03-24T10:51:02.546 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.546 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1506656, 2024-03-24T10:51:02.546 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 139520 2024-03-24T10:51:02.546 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.546 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-24T10:51:02.547 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-03-24T10:51:02.547 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-24T10:51:02.547 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.547 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-03-24T10:51:02.547 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-24T10:51:02.547 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-24T10:51:02.547 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 62, 2024-03-24T10:51:02.547 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.002224509, 2024-03-24T10:51:02.547 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000035879 2024-03-24T10:51:02.548 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:51:02.548 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.548 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-03-24T10:51:02.548 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-24T10:51:02.548 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-24T10:51:02.548 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2, 2024-03-24T10:51:02.548 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000007227, 2024-03-24T10:51:02.548 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000003613 2024-03-24T10:51:02.548 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:51:02.549 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.549 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-03-24T10:51:02.549 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 7136, 2024-03-24T10:51:02.549 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 6, 2024-03-24T10:51:02.549 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-03-24T10:51:02.549 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 6, 2024-03-24T10:51:02.549 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.002864999, 2024-03-24T10:51:02.549 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000477499 2024-03-24T10:51:02.549 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.550 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-03-24T10:51:02.550 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 7130, 2024-03-24T10:51:02.550 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-03-24T10:51:02.550 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-03-24T10:51:02.550 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 11, 2024-03-24T10:51:02.550 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 0, 2024-03-24T10:51:02.550 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-03-24T10:51:02.550 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 23, 2024-03-24T10:51:02.550 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 2, 2024-03-24T10:51:02.551 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 21, 2024-03-24T10:51:02.551 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 0, 2024-03-24T10:51:02.551 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 19, 2024-03-24T10:51:02.551 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 1, 2024-03-24T10:51:02.551 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 1, 2024-03-24T10:51:02.551 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 1, 2024-03-24T10:51:02.551 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 3, 2024-03-24T10:51:02.551 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 7151, 2024-03-24T10:51:02.552 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 13, 2024-03-24T10:51:02.552 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 7129, 2024-03-24T10:51:02.552 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 6, 2024-03-24T10:51:02.552 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-03-24T10:51:02.552 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-03-24T10:51:02.552 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 0, 2024-03-24T10:51:02.552 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-03-24T10:51:02.552 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-03-24T10:51:02.552 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 0, 2024-03-24T10:51:02.553 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 0, 2024-03-24T10:51:02.553 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 0, 2024-03-24T10:51:02.553 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 0, 2024-03-24T10:51:02.553 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-03-24T10:51:02.553 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 0, 2024-03-24T10:51:02.553 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-03-24T10:51:02.553 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 68, 2024-03-24T10:51:02.553 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 21196213, 2024-03-24T10:51:02.553 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 6, 2024-03-24T10:51:02.554 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T10:51:02.554 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-03-24T10:51:02.554 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T10:51:02.554 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-03-24T10:51:02.554 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 0, 2024-03-24T10:51:02.554 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T10:51:02.554 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T10:51:02.554 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 0, 2024-03-24T10:51:02.554 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 0, 2024-03-24T10:51:02.555 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 1, 2024-03-24T10:51:02.555 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 2, 2024-03-24T10:51:02.555 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T10:51:02.555 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T10:51:02.555 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T10:51:02.555 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 1, 2024-03-24T10:51:02.555 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 0, 2024-03-24T10:51:02.555 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 0, 2024-03-24T10:51:02.556 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 1 2024-03-24T10:51:02.556 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.556 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-03-24T10:51:02.556 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 0, 2024-03-24T10:51:02.556 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 12, 2024-03-24T10:51:02.556 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 8, 2024-03-24T10:51:02.556 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 11, 2024-03-24T10:51:02.556 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 0, 2024-03-24T10:51:02.556 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 0, 2024-03-24T10:51:02.557 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-03-24T10:51:02.557 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-03-24T10:51:02.557 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 0, 2024-03-24T10:51:02.557 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 0, 2024-03-24T10:51:02.557 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 0, 2024-03-24T10:51:02.557 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-03-24T10:51:02.557 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-03-24T10:51:02.557 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-03-24T10:51:02.557 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-03-24T10:51:02.558 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-03-24T10:51:02.558 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-03-24T10:51:02.558 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_path": 0, 2024-03-24T10:51:02.558 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_inode": 0, 2024-03-24T10:51:02.558 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T10:51:02.558 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 0, 2024-03-24T10:51:02.558 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-03-24T10:51:02.558 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 0, 2024-03-24T10:51:02.559 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-03-24T10:51:02.559 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-03-24T10:51:02.559 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.559 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-03-24T10:51:02.559 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-03-24T10:51:02.559 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 10, 2024-03-24T10:51:02.559 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 0, 2024-03-24T10:51:02.559 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 0, 2024-03-24T10:51:02.560 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 10, 2024-03-24T10:51:02.560 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-03-24T10:51:02.560 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 0, 2024-03-24T10:51:02.560 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 2, 2024-03-24T10:51:02.560 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 0, 2024-03-24T10:51:02.560 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 0, 2024-03-24T10:51:02.560 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 2, 2024-03-24T10:51:02.560 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 2, 2024-03-24T10:51:02.560 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-03-24T10:51:02.560 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 0, 2024-03-24T10:51:02.561 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 4194304, 2024-03-24T10:51:02.561 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 4197187, 2024-03-24T10:51:02.561 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 0, 2024-03-24T10:51:02.561 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-03-24T10:51:02.561 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 9, 2024-03-24T10:51:02.561 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.012228826, 2024-03-24T10:51:02.561 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001358758 2024-03-24T10:51:02.561 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.561 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 0 2024-03-24T10:51:02.562 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.562 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-03-24T10:51:02.562 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 26, 2024-03-24T10:51:02.562 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 26, 2024-03-24T10:51:02.562 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 0, 2024-03-24T10:51:02.562 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 32, 2024-03-24T10:51:02.562 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 34, 2024-03-24T10:51:02.562 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 2, 2024-03-24T10:51:02.563 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 23, 2024-03-24T10:51:02.563 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 24, 2024-03-24T10:51:02.563 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 1, 2024-03-24T10:51:02.563 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 1, 2024-03-24T10:51:02.563 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 5, 2024-03-24T10:51:02.563 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 4, 2024-03-24T10:51:02.563 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 56320, 2024-03-24T10:51:02.563 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 223612 2024-03-24T10:51:02.563 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.564 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-03-24T10:51:02.564 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 7141, 2024-03-24T10:51:02.564 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 0, 2024-03-24T10:51:02.564 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 7136, 2024-03-24T10:51:02.564 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 30, 2024-03-24T10:51:02.564 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 1, 2024-03-24T10:51:02.564 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-03-24T10:51:02.564 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.565 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.565 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.565 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.565 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-03-24T10:51:02.565 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 3, 2024-03-24T10:51:02.565 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.002142812, 2024-03-24T10:51:02.565 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000714270 2024-03-24T10:51:02.565 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.565 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-03-24T10:51:02.566 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.566 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.566 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.566 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.566 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-03-24T10:51:02.566 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.566 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.566 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.567 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.567 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-03-24T10:51:02.567 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 3, 2024-03-24T10:51:02.567 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000722187, 2024-03-24T10:51:02.567 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000240729 2024-03-24T10:51:02.567 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.567 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-03-24T10:51:02.567 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.567 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.568 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.568 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.568 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-03-24T10:51:02.568 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.568 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.568 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.568 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.568 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-03-24T10:51:02.568 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.569 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.569 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.569 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.569 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-03-24T10:51:02.569 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.569 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.569 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.569 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.569 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-03-24T10:51:02.570 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.570 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.570 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.570 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.570 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-03-24T10:51:02.570 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.570 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.570 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.571 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.571 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-03-24T10:51:02.571 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.571 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.572 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.572 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-03-24T10:51:02.572 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.572 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.572 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.572 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-03-24T10:51:02.572 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.573 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.573 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.573 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.573 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-03-24T10:51:02.573 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.573 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.573 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.573 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.573 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-03-24T10:51:02.574 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.574 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.574 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.574 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.574 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-03-24T10:51:02.574 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.574 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.575 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.575 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.575 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-03-24T10:51:02.575 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.575 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.575 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.575 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.575 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-03-24T10:51:02.576 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.576 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.576 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.576 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.576 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-03-24T10:51:02.576 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.576 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.576 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.576 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.577 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-03-24T10:51:02.577 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.577 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.577 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.577 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.577 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-03-24T10:51:02.577 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.577 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.578 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.578 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.578 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-03-24T10:51:02.578 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.578 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.578 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.578 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.578 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-03-24T10:51:02.579 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.579 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.579 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.579 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.579 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-03-24T10:51:02.579 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.579 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.579 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.579 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.580 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-03-24T10:51:02.580 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.580 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.580 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.580 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.580 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-03-24T10:51:02.580 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.580 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.580 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.581 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.581 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-03-24T10:51:02.581 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.581 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.581 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.581 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.581 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-03-24T10:51:02.581 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.581 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.582 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.582 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.582 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-03-24T10:51:02.582 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-03-24T10:51:02.582 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-03-24T10:51:02.582 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.582 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.582 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.582 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:51:02.583 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.583 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-03-24T10:51:02.583 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 2, 2024-03-24T10:51:02.583 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 5, 2024-03-24T10:51:02.583 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 3, 2024-03-24T10:51:02.583 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 2, 2024-03-24T10:51:02.583 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-03-24T10:51:02.583 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 5, 2024-03-24T10:51:02.584 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 1, 2024-03-24T10:51:02.584 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 630, 2024-03-24T10:51:02.584 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-03-24T10:51:02.584 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.584 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-24T10:51:02.584 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-24T10:51:02.584 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-24T10:51:02.584 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T10:51:02.584 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-24T10:51:02.585 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T10:51:02.585 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-24T10:51:02.585 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T10:51:02.585 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T10:51:02.585 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T10:51:02.585 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T10:51:02.585 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T10:51:02.585 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-24T10:51:02.586 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T10:51:02.586 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T10:51:02.586 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-24T10:51:02.586 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-24T10:51:02.586 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-24T10:51:02.586 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-24T10:51:02.586 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T10:51:02.586 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T10:51:02.587 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T10:51:02.587 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T10:51:02.587 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T10:51:02.587 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-24T10:51:02.587 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-24T10:51:02.587 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-24T10:51:02.587 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T10:51:02.587 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T10:51:02.587 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-24T10:51:02.588 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-24T10:51:02.588 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-24T10:51:02.588 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-24T10:51:02.588 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T10:51:02.588 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-24T10:51:02.588 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T10:51:02.588 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-24T10:51:02.588 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 275213, 2024-03-24T10:51:02.588 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 127, 2024-03-24T10:51:02.589 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-03-24T10:51:02.589 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-03-24T10:51:02.589 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-24T10:51:02.589 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-24T10:51:02.589 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-24T10:51:02.589 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-24T10:51:02.589 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-24T10:51:02.589 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-24T10:51:02.589 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 8224, 2024-03-24T10:51:02.590 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 171, 2024-03-24T10:51:02.590 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T10:51:02.590 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-24T10:51:02.590 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-24T10:51:02.590 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-24T10:51:02.590 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 121273, 2024-03-24T10:51:02.590 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 658, 2024-03-24T10:51:02.590 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-24T10:51:02.591 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-24T10:51:02.591 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-24T10:51:02.591 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-24T10:51:02.591 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.591 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-03-24T10:51:02.591 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-03-24T10:51:02.591 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-03-24T10:51:02.591 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 50, 2024-03-24T10:51:02.591 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 13864, 2024-03-24T10:51:02.591 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 33, 2024-03-24T10:51:02.592 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 50, 2024-03-24T10:51:02.592 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-03-24T10:51:02.592 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 50, 2024-03-24T10:51:02.592 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.232995429, 2024-03-24T10:51:02.592 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.004659908 2024-03-24T10:51:02.592 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.592 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-03-24T10:51:02.592 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-03-24T10:51:02.593 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 50, 2024-03-24T10:51:02.593 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 76 2024-03-24T10:51:02.593 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.593 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 50, 2024-03-24T10:51:02.593 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 0, 2024-03-24T10:51:02.593 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 50, 2024-03-24T10:51:02.593 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-03-24T10:51:02.593 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-03-24T10:51:02.593 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 0, 2024-03-24T10:51:02.594 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 10, 2024-03-24T10:51:02.594 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 0, 2024-03-24T10:51:02.594 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 9, 2024-03-24T10:51:02.594 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 8, 2024-03-24T10:51:02.594 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-03-24T10:51:02.594 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-03-24T10:51:02.594 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-03-24T10:51:02.594 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 0, 2024-03-24T10:51:02.594 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 6, 2024-03-24T10:51:02.595 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-03-24T10:51:02.595 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-03-24T10:51:02.595 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-03-24T10:51:02.595 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 0, 2024-03-24T10:51:02.595 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 20, 2024-03-24T10:51:02.595 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-03-24T10:51:02.595 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-03-24T10:51:02.595 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-03-24T10:51:02.596 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-03-24T10:51:02.596 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-03-24T10:51:02.596 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-03-24T10:51:02.596 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T10:51:02.596 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-03-24T10:51:02.596 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-03-24T10:51:02.596 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 0, 2024-03-24T10:51:02.596 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-03-24T10:51:02.596 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-03-24T10:51:02.597 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-03-24T10:51:02.597 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-03-24T10:51:02.597 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-03-24T10:51:02.597 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-03-24T10:51:02.597 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-03-24T10:51:02.597 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-03-24T10:51:02.597 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-03-24T10:51:02.597 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-03-24T10:51:02.597 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-03-24T10:51:02.598 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-03-24T10:51:02.598 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-03-24T10:51:02.598 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-03-24T10:51:02.598 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-03-24T10:51:02.598 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-03-24T10:51:02.598 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 98, 2024-03-24T10:51:02.598 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-03-24T10:51:02.598 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 13, 2024-03-24T10:51:02.599 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 10, 2024-03-24T10:51:02.599 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 10, 2024-03-24T10:51:02.599 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-03-24T10:51:02.599 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-03-24T10:51:02.599 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 22, 2024-03-24T10:51:02.599 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 0, 2024-03-24T10:51:02.599 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 1 2024-03-24T10:51:02.599 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.599 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-03-24T10:51:02.600 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 1, 2024-03-24T10:51:02.600 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-03-24T10:51:02.600 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-03-24T10:51:02.600 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-03-24T10:51:02.600 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.600 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-03-24T10:51:02.600 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-03-24T10:51:02.600 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.600 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-03-24T10:51:02.601 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-03-24T10:51:02.601 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T10:51:02.601 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-03-24T10:51:02.601 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 0, 2024-03-24T10:51:02.601 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 0, 2024-03-24T10:51:02.601 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 0, 2024-03-24T10:51:02.601 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-03-24T10:51:02.601 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.602 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T10:51:02.602 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:51:02.602 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T10:51:02.602 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:51:02.602 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 96629, 2024-03-24T10:51:02.602 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 35250335, 2024-03-24T10:51:02.602 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:51:02.602 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 96629, 2024-03-24T10:51:02.602 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T10:51:02.603 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T10:51:02.603 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 96629, 2024-03-24T10:51:02.603 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 35250335, 2024-03-24T10:51:02.603 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:51:02.603 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.603 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.603 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.603 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:51:02.604 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.604 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-03-24T10:51:02.604 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:51:02.604 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T10:51:02.604 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:51:02.604 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T10:51:02.604 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T10:51:02.604 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:51:02.605 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T10:51:02.605 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 50, 2024-03-24T10:51:02.605 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 13864, 2024-03-24T10:51:02.605 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 50, 2024-03-24T10:51:02.605 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 13864, 2024-03-24T10:51:02.605 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:51:02.605 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.605 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.606 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.606 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:51:02.606 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.606 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-03-24T10:51:02.606 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:51:02.606 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-03-24T10:51:02.606 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:51:02.606 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T10:51:02.606 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T10:51:02.607 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:51:02.607 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T10:51:02.607 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 50, 2024-03-24T10:51:02.607 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 50, 2024-03-24T10:51:02.607 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 50, 2024-03-24T10:51:02.607 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 50, 2024-03-24T10:51:02.607 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:51:02.607 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.608 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.608 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.608 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:51:02.608 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.608 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-03-24T10:51:02.608 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:51:02.608 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-24T10:51:02.608 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:51:02.608 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T10:51:02.609 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T10:51:02.609 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:51:02.609 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T10:51:02.609 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T10:51:02.609 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T10:51:02.609 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-03-24T10:51:02.609 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-03-24T10:51:02.609 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:51:02.609 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.610 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.610 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.610 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:51:02.610 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:02.610 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x564918902500": { 2024-03-24T10:51:02.610 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:51:02.610 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-24T10:51:02.610 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:51:02.611 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 10, 2024-03-24T10:51:02.611 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 2883, 2024-03-24T10:51:02.611 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:51:02.611 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 10, 2024-03-24T10:51:02.611 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T10:51:02.611 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T10:51:02.611 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 9, 2024-03-24T10:51:02.611 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 2883, 2024-03-24T10:51:02.611 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:51:02.612 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:02.612 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:02.612 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:02.612 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:51:02.612 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:51:02.612 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-24T10:51:03.233 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:02 smithi117 ceph-mon[36368]: pgmap v341: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:51:03.259 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-24T10:51:03.259 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-24T10:51:03.260 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-24T10:51:03.260 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-24T10:51:03.260 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T10:51:03.260 DEBUG:tasks.check_counter:Getting stats from l 2024-03-24T10:51:03.260 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-24T10:51:03.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:02 smithi116 ceph-mon[36248]: pgmap v341: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:51:03.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:02 smithi023 ceph-mon[32466]: pgmap v341: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:51:03.506 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T10:51:05.084 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:04 smithi117 ceph-mon[36368]: pgmap v342: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-24T10:51:05.085 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-24T10:51:05.085 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T10:51:05.085 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 0, 2024-03-24T10:51:05.085 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 2, 2024-03-24T10:51:05.086 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 0, 2024-03-24T10:51:05.086 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 0, 2024-03-24T10:51:05.086 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 4, 2024-03-24T10:51:05.086 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 0, 2024-03-24T10:51:05.086 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.001148945, 2024-03-24T10:51:05.086 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T10:51:05.086 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.000368141, 2024-03-24T10:51:05.086 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T10:51:05.086 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:51:05.087 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:05.087 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:05.087 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:05.087 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:05.087 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:51:05.087 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:05.087 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:05.087 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:05.087 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:05.088 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T10:51:05.088 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T10:51:05.088 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:05.088 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T10:51:05.088 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 232, 2024-03-24T10:51:05.088 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 220, 2024-03-24T10:51:05.088 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 1437184, 2024-03-24T10:51:05.088 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 134272, 2024-03-24T10:51:05.088 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 2, 2024-03-24T10:51:05.089 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 1, 2024-03-24T10:51:05.089 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.035394473, 2024-03-24T10:51:05.089 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.008457547, 2024-03-24T10:51:05.089 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.032968410, 2024-03-24T10:51:05.089 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.003689769, 2024-03-24T10:51:05.089 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:51:05.089 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 220, 2024-03-24T10:51:05.089 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.007593654, 2024-03-24T10:51:05.089 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000034516 2024-03-24T10:51:05.090 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:05.090 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:51:05.090 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:05.090 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:05.090 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:05.090 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:05.090 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1437184, 2024-03-24T10:51:05.090 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 134272 2024-03-24T10:51:05.090 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:05.091 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T10:51:05.091 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 2, 2024-03-24T10:51:05.091 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 166, 2024-03-24T10:51:05.091 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 320, 2024-03-24T10:51:05.091 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 184592, 2024-03-24T10:51:05.091 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 5, 2024-03-24T10:51:05.091 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 3, 2024-03-24T10:51:05.091 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.007901403, 2024-03-24T10:51:05.091 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.004554734, 2024-03-24T10:51:05.092 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.001750228, 2024-03-24T10:51:05.092 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T10:51:05.092 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T10:51:05.092 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 163, 2024-03-24T10:51:05.092 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.014690917, 2024-03-24T10:51:05.092 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000090128 2024-03-24T10:51:05.092 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:05.092 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T10:51:05.092 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:05.093 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:05.093 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:05.093 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:05.093 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T10:51:05.093 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 184592 2024-03-24T10:51:05.093 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:05.093 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-24T10:51:05.093 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 0, 2024-03-24T10:51:05.093 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-24T10:51:05.094 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:05.094 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-24T10:51:05.094 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-24T10:51:05.094 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-24T10:51:05.094 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T10:51:05.094 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-24T10:51:05.094 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T10:51:05.094 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-24T10:51:05.095 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T10:51:05.095 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T10:51:05.095 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T10:51:05.095 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T10:51:05.095 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T10:51:05.095 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-24T10:51:05.095 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T10:51:05.095 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T10:51:05.095 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-24T10:51:05.096 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-24T10:51:05.096 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-24T10:51:05.096 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-24T10:51:05.096 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T10:51:05.096 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T10:51:05.096 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T10:51:05.096 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T10:51:05.096 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T10:51:05.096 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-24T10:51:05.097 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-24T10:51:05.097 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-24T10:51:05.097 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T10:51:05.097 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T10:51:05.097 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-24T10:51:05.097 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-24T10:51:05.097 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-24T10:51:05.097 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-24T10:51:05.097 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T10:51:05.098 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-24T10:51:05.098 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T10:51:05.098 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-24T10:51:05.098 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 148967, 2024-03-24T10:51:05.098 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 29, 2024-03-24T10:51:05.098 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-03-24T10:51:05.098 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-03-24T10:51:05.098 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-24T10:51:05.098 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-24T10:51:05.098 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-24T10:51:05.099 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-24T10:51:05.099 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-24T10:51:05.099 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-24T10:51:05.099 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 0, 2024-03-24T10:51:05.099 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 0, 2024-03-24T10:51:05.099 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T10:51:05.099 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-24T10:51:05.099 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-24T10:51:05.099 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-24T10:51:05.100 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 928, 2024-03-24T10:51:05.100 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 2, 2024-03-24T10:51:05.100 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-24T10:51:05.100 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-24T10:51:05.100 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-24T10:51:05.100 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-24T10:51:05.100 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T10:51:05.100 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T10:51:05.100 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T10:51:05.101 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T10:51:05.101 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T10:51:05.101 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 230, 2024-03-24T10:51:05.101 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 1402010, 2024-03-24T10:51:05.101 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T10:51:05.101 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 230, 2024-03-24T10:51:05.101 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T10:51:05.101 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T10:51:05.101 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 230, 2024-03-24T10:51:05.101 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 1402010, 2024-03-24T10:51:05.102 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T10:51:05.102 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T10:51:05.102 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T10:51:05.102 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T10:51:05.102 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:51:05.102 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T10:51:05.102 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-24T10:51:05.220 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:04 smithi023 ceph-mon[32466]: pgmap v342: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-24T10:51:05.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:04 smithi116 ceph-mon[36248]: pgmap v342: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-24T10:51:05.688 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-24T10:51:05.689 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-24T10:51:05.689 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-24T10:51:05.689 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-24T10:51:05.689 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-24T10:51:05.689 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-24T10:51:05.699 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-24T10:51:05.699 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:51:05.699 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:51:05.719 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-24T10:51:05.719 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:51:05.777 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:51:05.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.926+0000 7f63fce8e640 1 -- 172.21.15.23:0/1193004423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63f81066b0 msgr2=0x7f63f8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:05.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.926+0000 7f63fce8e640 1 --2- 172.21.15.23:0/1193004423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63f81066b0 0x7f63f8106a90 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f63e8009900 tx=0x7f63e802bcd0 comp rx=0 tx=0).stop 2024-03-24T10:51:05.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.926+0000 7f63fce8e640 1 -- 172.21.15.23:0/1193004423 shutdown_connections 2024-03-24T10:51:05.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.926+0000 7f63fce8e640 1 --2- 172.21.15.23:0/1193004423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63f8100d40 0x7f63f810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:05.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.926+0000 7f63fce8e640 1 --2- 172.21.15.23:0/1193004423 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63f8106fd0 0x7f63f8100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:05.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.926+0000 7f63fce8e640 1 --2- 172.21.15.23:0/1193004423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63f81066b0 0x7f63f8106a90 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:05.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.926+0000 7f63fce8e640 1 -- 172.21.15.23:0/1193004423 >> 172.21.15.23:0/1193004423 conn(0x7f63f8077710 msgr2=0x7f63f8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:05.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.926+0000 7f63fce8e640 1 -- 172.21.15.23:0/1193004423 shutdown_connections 2024-03-24T10:51:05.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.926+0000 7f63fce8e640 1 -- 172.21.15.23:0/1193004423 wait complete. 2024-03-24T10:51:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.926+0000 7f63fce8e640 1 Processor -- start 2024-03-24T10:51:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.926+0000 7f63fce8e640 1 -- start start 2024-03-24T10:51:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63fce8e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63f8100d40 0x7f63f8199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63fce8e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63f81066b0 0x7f63f819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63fce8e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63f8106fd0 0x7f63f819e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63fce8e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f63f819ee50 con 0x7f63f8100d40 2024-03-24T10:51:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63fce8e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f63f819efc0 con 0x7f63f81066b0 2024-03-24T10:51:05.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63fce8e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f63f819f180 con 0x7f63f8106fd0 2024-03-24T10:51:05.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63f659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63f8100d40 0x7f63f8199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:05.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63f659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63f8100d40 0x7f63f8199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44722/0 (socket says 172.21.15.23:44722) 2024-03-24T10:51:05.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63f659c640 1 -- 172.21.15.23:0/4101273009 learned_addr learned my addr 172.21.15.23:0/4101273009 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:51:05.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63f5d9b640 1 --2- 172.21.15.23:0/4101273009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63f81066b0 0x7f63f819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:05.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63f6d9d640 1 --2- 172.21.15.23:0/4101273009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63f8106fd0 0x7f63f819e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:05.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63f659c640 1 -- 172.21.15.23:0/4101273009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63f8106fd0 msgr2=0x7f63f819e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:05.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63f659c640 1 --2- 172.21.15.23:0/4101273009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63f8106fd0 0x7f63f819e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:05.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63f659c640 1 -- 172.21.15.23:0/4101273009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63f81066b0 msgr2=0x7f63f819a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:05.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63f659c640 1 --2- 172.21.15.23:0/4101273009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63f81066b0 0x7f63f819a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:05.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63f659c640 1 -- 172.21.15.23:0/4101273009 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e8009560 con 0x7f63f8100d40 2024-03-24T10:51:05.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63f659c640 1 --2- 172.21.15.23:0/4101273009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63f8100d40 0x7f63f8199f90 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f63e8036040 tx=0x7f63e802c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:05.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63e77fe640 1 -- 172.21.15.23:0/4101273009 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e8004350 con 0x7f63f8100d40 2024-03-24T10:51:05.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.927+0000 7f63fce8e640 1 -- 172.21.15.23:0/4101273009 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63f819f400 con 0x7f63f8100d40 2024-03-24T10:51:05.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.928+0000 7f63e77fe640 1 -- 172.21.15.23:0/4101273009 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63e80044b0 con 0x7f63f8100d40 2024-03-24T10:51:05.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.928+0000 7f63e77fe640 1 -- 172.21.15.23:0/4101273009 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e8003da0 con 0x7f63f8100d40 2024-03-24T10:51:05.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.928+0000 7f63fce8e640 1 -- 172.21.15.23:0/4101273009 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63f81a6ce0 con 0x7f63f8100d40 2024-03-24T10:51:05.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.928+0000 7f63fce8e640 1 -- 172.21.15.23:0/4101273009 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63f8101d50 con 0x7f63f8100d40 2024-03-24T10:51:05.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.930+0000 7f63e77fe640 1 -- 172.21.15.23:0/4101273009 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f63e8004df0 con 0x7f63f8100d40 2024-03-24T10:51:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.930+0000 7f63e77fe640 1 --2- 172.21.15.23:0/4101273009 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f63e00a2230 0x7f63e00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.930+0000 7f63e77fe640 1 -- 172.21.15.23:0/4101273009 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f63e80f0c60 con 0x7f63f8100d40 2024-03-24T10:51:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.930+0000 7f63f5d9b640 1 --2- 172.21.15.23:0/4101273009 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f63e00a2230 0x7f63e00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.931+0000 7f63f5d9b640 1 --2- 172.21.15.23:0/4101273009 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f63e00a2230 0x7f63e00a46f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f63f80ff6e0 tx=0x7f63ec005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:05.934+0000 7f63e77fe640 1 -- 172.21.15.23:0/4101273009 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f63e80b9a60 con 0x7f63f8100d40 2024-03-24T10:51:06.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.118+0000 7f63fce8e640 1 -- 172.21.15.23:0/4101273009 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f63f8068500 con 0x7f63f8100d40 2024-03-24T10:51:06.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.120+0000 7f63e77fe640 1 -- 172.21.15.23:0/4101273009 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+23169 (secure 0 0 0) 0x7f63e80bd120 con 0x7f63f8100d40 2024-03-24T10:51:06.121 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:51:06.121 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":98,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:50:16.243119+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":97,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2350021632":"2024-03-25T10:40:05.291912+0000","172.21.15.23:6800/2350021632":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3614005933":"2024-03-25T10:40:05.291912+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/256452041":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/352422381":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/725846898":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:51:06.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.122+0000 7f63fce8e640 1 -- 172.21.15.23:0/4101273009 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f63e00a2230 msgr2=0x7f63e00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.122+0000 7f63fce8e640 1 --2- 172.21.15.23:0/4101273009 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f63e00a2230 0x7f63e00a46f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f63f80ff6e0 tx=0x7f63ec005f90 comp rx=0 tx=0).stop 2024-03-24T10:51:06.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.122+0000 7f63fce8e640 1 -- 172.21.15.23:0/4101273009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63f8100d40 msgr2=0x7f63f8199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.122+0000 7f63fce8e640 1 --2- 172.21.15.23:0/4101273009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63f8100d40 0x7f63f8199f90 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f63e8036040 tx=0x7f63e802c720 comp rx=0 tx=0).stop 2024-03-24T10:51:06.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.123+0000 7f63fce8e640 1 -- 172.21.15.23:0/4101273009 shutdown_connections 2024-03-24T10:51:06.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.123+0000 7f63fce8e640 1 --2- 172.21.15.23:0/4101273009 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f63e00a2230 0x7f63e00a46f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.123+0000 7f63fce8e640 1 --2- 172.21.15.23:0/4101273009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63f8106fd0 0x7f63f819e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.123+0000 7f63fce8e640 1 --2- 172.21.15.23:0/4101273009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63f81066b0 0x7f63f819a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.123+0000 7f63fce8e640 1 --2- 172.21.15.23:0/4101273009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63f8100d40 0x7f63f8199f90 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.123+0000 7f63fce8e640 1 -- 172.21.15.23:0/4101273009 >> 172.21.15.23:0/4101273009 conn(0x7f63f8077710 msgr2=0x7f63f80688a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:06.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.123+0000 7f63fce8e640 1 -- 172.21.15.23:0/4101273009 shutdown_connections 2024-03-24T10:51:06.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.123+0000 7f63fce8e640 1 -- 172.21.15.23:0/4101273009 wait complete. 2024-03-24T10:51:06.134 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:33:39.056452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:38:08.473472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:38:09.414667+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 97, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:51:06.135 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:51:06.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.249+0000 7fee55655640 1 -- 172.21.15.23:0/2595512832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee5006a490 msgr2=0x7fee5010e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.249+0000 7fee55655640 1 --2- 172.21.15.23:0/2595512832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee5006a490 0x7fee5010e550 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fee48009ca0 tx=0x7fee4802c0f0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.249+0000 7fee55655640 1 -- 172.21.15.23:0/2595512832 shutdown_connections 2024-03-24T10:51:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.249+0000 7fee55655640 1 --2- 172.21.15.23:0/2595512832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee5006a490 0x7fee5010e550 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.249+0000 7fee55655640 1 --2- 172.21.15.23:0/2595512832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee50069af0 0x7fee50069f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.249+0000 7fee55655640 1 --2- 172.21.15.23:0/2595512832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee50107700 0x7fee50107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.249+0000 7fee55655640 1 -- 172.21.15.23:0/2595512832 >> 172.21.15.23:0/2595512832 conn(0x7fee500773a0 msgr2=0x7fee500777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.249+0000 7fee55655640 1 -- 172.21.15.23:0/2595512832 shutdown_connections 2024-03-24T10:51:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.249+0000 7fee55655640 1 -- 172.21.15.23:0/2595512832 wait complete. 2024-03-24T10:51:06.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.249+0000 7fee55655640 1 Processor -- start 2024-03-24T10:51:06.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee55655640 1 -- start start 2024-03-24T10:51:06.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee55655640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee50069af0 0x7fee50100930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:06.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee55655640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee5006a490 0x7fee50100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee55655640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee50107700 0x7fee501013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee55655640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fee501a60b0 con 0x7fee50107700 2024-03-24T10:51:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee55655640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fee501a6220 con 0x7fee5006a490 2024-03-24T10:51:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee55655640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fee501a63e0 con 0x7fee50069af0 2024-03-24T10:51:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee4f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee50107700 0x7fee501013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee4f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee50107700 0x7fee501013b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44732/0 (socket says 172.21.15.23:44732) 2024-03-24T10:51:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee4f7fe640 1 -- 172.21.15.23:0/1422239477 learned_addr learned my addr 172.21.15.23:0/1422239477 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:51:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee4effd640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee50069af0 0x7fee50100930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee4e7fc640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee5006a490 0x7fee50100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:06.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee4f7fe640 1 -- 172.21.15.23:0/1422239477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee50069af0 msgr2=0x7fee50100930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee4f7fe640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee50069af0 0x7fee50100930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee4f7fe640 1 -- 172.21.15.23:0/1422239477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee5006a490 msgr2=0x7fee50100e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee4f7fe640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee5006a490 0x7fee50100e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee4f7fe640 1 -- 172.21.15.23:0/1422239477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee44009490 con 0x7fee50107700 2024-03-24T10:51:06.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee4effd640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee50069af0 0x7fee50100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:51:06.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee4e7fc640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee5006a490 0x7fee50100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:51:06.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.250+0000 7fee4f7fe640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee50107700 0x7fee501013b0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fee48005e30 tx=0x7fee48035aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:06.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.251+0000 7fee37fff640 1 -- 172.21.15.23:0/1422239477 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee48035ea0 con 0x7fee50107700 2024-03-24T10:51:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.251+0000 7fee55655640 1 -- 172.21.15.23:0/1422239477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee48009900 con 0x7fee50107700 2024-03-24T10:51:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.251+0000 7fee37fff640 1 -- 172.21.15.23:0/1422239477 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee48003e50 con 0x7fee50107700 2024-03-24T10:51:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.251+0000 7fee37fff640 1 -- 172.21.15.23:0/1422239477 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee48004830 con 0x7fee50107700 2024-03-24T10:51:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.251+0000 7fee55655640 1 -- 172.21.15.23:0/1422239477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee501a69d0 con 0x7fee50107700 2024-03-24T10:51:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.252+0000 7fee55655640 1 -- 172.21.15.23:0/1422239477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee50067c70 con 0x7fee50107700 2024-03-24T10:51:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.253+0000 7fee37fff640 1 -- 172.21.15.23:0/1422239477 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fee4803b040 con 0x7fee50107700 2024-03-24T10:51:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.253+0000 7fee37fff640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fee380a2230 0x7fee380a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.254+0000 7fee4effd640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fee380a2230 0x7fee380a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.254+0000 7fee37fff640 1 -- 172.21.15.23:0/1422239477 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fee480f0cb0 con 0x7fee50107700 2024-03-24T10:51:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.254+0000 7fee4effd640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fee380a2230 0x7fee380a46f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fee44011550 tx=0x7fee44005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:06.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.257+0000 7fee37fff640 1 -- 172.21.15.23:0/1422239477 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fee480b9a20 con 0x7fee50107700 2024-03-24T10:51:06.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.442+0000 7fee55655640 1 -- 172.21.15.23:0/1422239477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fee50109ba0 con 0x7fee50107700 2024-03-24T10:51:06.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.443+0000 7fee37fff640 1 -- 172.21.15.23:0/1422239477 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) ==== 93+0+10 (secure 0 0 0) 0x7fee480bd0e0 con 0x7fee50107700 2024-03-24T10:51:06.443 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-24T10:51:06.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.445+0000 7fee55655640 1 -- 172.21.15.23:0/1422239477 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fee380a2230 msgr2=0x7fee380a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.445+0000 7fee55655640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fee380a2230 0x7fee380a46f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fee44011550 tx=0x7fee44005f90 comp rx=0 tx=0).stop 2024-03-24T10:51:06.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.445+0000 7fee55655640 1 -- 172.21.15.23:0/1422239477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee50107700 msgr2=0x7fee501013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.445+0000 7fee55655640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee50107700 0x7fee501013b0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fee48005e30 tx=0x7fee48035aa0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.445+0000 7fee55655640 1 -- 172.21.15.23:0/1422239477 shutdown_connections 2024-03-24T10:51:06.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.445+0000 7fee55655640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fee380a2230 0x7fee380a46f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.445+0000 7fee55655640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee50107700 0x7fee501013b0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.445+0000 7fee55655640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee5006a490 0x7fee50100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.445+0000 7fee55655640 1 --2- 172.21.15.23:0/1422239477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee50069af0 0x7fee50100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.445+0000 7fee55655640 1 -- 172.21.15.23:0/1422239477 >> 172.21.15.23:0/1422239477 conn(0x7fee500773a0 msgr2=0x7fee500fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:06.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.445+0000 7fee55655640 1 -- 172.21.15.23:0/1422239477 shutdown_connections 2024-03-24T10:51:06.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.445+0000 7fee55655640 1 -- 172.21.15.23:0/1422239477 wait complete. 2024-03-24T10:51:06.456 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:51:06.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.562+0000 7fcfe59a2640 1 -- 172.21.15.23:0/339012543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe01015a0 msgr2=0x7fcfe0101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.562+0000 7fcfe59a2640 1 --2- 172.21.15.23:0/339012543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe01015a0 0x7fcfe0101a20 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc009900 tx=0x7fcfcc02bc30 comp rx=0 tx=0).stop 2024-03-24T10:51:06.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.562+0000 7fcfe59a2640 1 -- 172.21.15.23:0/339012543 shutdown_connections 2024-03-24T10:51:06.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.562+0000 7fcfe59a2640 1 --2- 172.21.15.23:0/339012543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0101f60 0x7fcfe010e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.562+0000 7fcfe59a2640 1 --2- 172.21.15.23:0/339012543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe01015a0 0x7fcfe0101a20 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.562+0000 7fcfe59a2640 1 --2- 172.21.15.23:0/339012543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcfe0107660 0x7fcfe0107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.562+0000 7fcfe59a2640 1 -- 172.21.15.23:0/339012543 >> 172.21.15.23:0/339012543 conn(0x7fcfe00fd0b0 msgr2=0x7fcfe00ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:06.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.562+0000 7fcfe59a2640 1 -- 172.21.15.23:0/339012543 shutdown_connections 2024-03-24T10:51:06.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.562+0000 7fcfe59a2640 1 -- 172.21.15.23:0/339012543 wait complete. 2024-03-24T10:51:06.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.563+0000 7fcfe59a2640 1 Processor -- start 2024-03-24T10:51:06.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.563+0000 7fcfe59a2640 1 -- start start 2024-03-24T10:51:06.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.563+0000 7fcfe59a2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcfe01015a0 0x7fcfe0073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:06.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.563+0000 7fcfe59a2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe0101f60 0x7fcfe0073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:06.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.563+0000 7fcfe59a2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0107660 0x7fcfe0074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:06.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.563+0000 7fcfe59a2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe01a5f30 con 0x7fcfe0101f60 2024-03-24T10:51:06.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.563+0000 7fcfe59a2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe01a60a0 con 0x7fcfe0107660 2024-03-24T10:51:06.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.563+0000 7fcfe59a2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe01a6260 con 0x7fcfe01015a0 2024-03-24T10:51:06.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.563+0000 7fcfdffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe0101f60 0x7fcfe0073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:06.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.563+0000 7fcfdffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe0101f60 0x7fcfe0073f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44750/0 (socket says 172.21.15.23:44750) 2024-03-24T10:51:06.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.563+0000 7fcfdffff640 1 -- 172.21.15.23:0/580812534 learned_addr learned my addr 172.21.15.23:0/580812534 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:51:06.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.563+0000 7fcfe49a0640 1 --2- 172.21.15.23:0/580812534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcfe01015a0 0x7fcfe0073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:06.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.563+0000 7fcfe51a1640 1 --2- 172.21.15.23:0/580812534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0107660 0x7fcfe0074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:06.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.564+0000 7fcfdffff640 1 -- 172.21.15.23:0/580812534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcfe01015a0 msgr2=0x7fcfe0073a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.564+0000 7fcfdffff640 1 --2- 172.21.15.23:0/580812534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcfe01015a0 0x7fcfe0073a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.564+0000 7fcfdffff640 1 -- 172.21.15.23:0/580812534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0107660 msgr2=0x7fcfe0074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.564+0000 7fcfdffff640 1 --2- 172.21.15.23:0/580812534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0107660 0x7fcfe0074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.564+0000 7fcfdffff640 1 -- 172.21.15.23:0/580812534 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd4009490 con 0x7fcfe0101f60 2024-03-24T10:51:06.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.564+0000 7fcfdffff640 1 --2- 172.21.15.23:0/580812534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe0101f60 0x7fcfe0073f50 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc005940 tx=0x7fcfcc02c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:06.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.564+0000 7fcfddffb640 1 -- 172.21.15.23:0/580812534 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc0042d0 con 0x7fcfe0101f60 2024-03-24T10:51:06.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.564+0000 7fcfe59a2640 1 -- 172.21.15.23:0/580812534 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfcc009560 con 0x7fcfe0101f60 2024-03-24T10:51:06.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.564+0000 7fcfddffb640 1 -- 172.21.15.23:0/580812534 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfcc004430 con 0x7fcfe0101f60 2024-03-24T10:51:06.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.564+0000 7fcfddffb640 1 -- 172.21.15.23:0/580812534 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc003c50 con 0x7fcfe0101f60 2024-03-24T10:51:06.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.564+0000 7fcfe59a2640 1 -- 172.21.15.23:0/580812534 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe01a6860 con 0x7fcfe0101f60 2024-03-24T10:51:06.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.565+0000 7fcfe59a2640 1 -- 172.21.15.23:0/580812534 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfe0068420 con 0x7fcfe0101f60 2024-03-24T10:51:06.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.566+0000 7fcfddffb640 1 -- 172.21.15.23:0/580812534 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fcfcc0054f0 con 0x7fcfe0101f60 2024-03-24T10:51:06.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.567+0000 7fcfddffb640 1 --2- 172.21.15.23:0/580812534 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcfd00a2230 0x7fcfd00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:06.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.567+0000 7fcfe49a0640 1 --2- 172.21.15.23:0/580812534 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcfd00a2230 0x7fcfd00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:06.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.567+0000 7fcfddffb640 1 -- 172.21.15.23:0/580812534 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fcfcc0f05e0 con 0x7fcfe0101f60 2024-03-24T10:51:06.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.567+0000 7fcfe49a0640 1 --2- 172.21.15.23:0/580812534 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcfd00a2230 0x7fcfd00a46f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009ce0 tx=0x7fcfd4005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:06.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.575+0000 7fcfddffb640 1 -- 172.21.15.23:0/580812534 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcfcc0b9350 con 0x7fcfe0101f60 2024-03-24T10:51:06.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.757+0000 7fcfe59a2640 1 -- 172.21.15.23:0/580812534 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcfe01099b0 con 0x7fcfe0101f60 2024-03-24T10:51:06.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.757+0000 7fcfddffb640 1 -- 172.21.15.23:0/580812534 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) ==== 104+0+11 (secure 0 0 0) 0x7fcfcc0bca10 con 0x7fcfe0101f60 2024-03-24T10:51:06.758 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-24T10:51:06.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.759+0000 7fcfe59a2640 1 -- 172.21.15.23:0/580812534 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcfd00a2230 msgr2=0x7fcfd00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.759+0000 7fcfe59a2640 1 --2- 172.21.15.23:0/580812534 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcfd00a2230 0x7fcfd00a46f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009ce0 tx=0x7fcfd4005f90 comp rx=0 tx=0).stop 2024-03-24T10:51:06.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.759+0000 7fcfe59a2640 1 -- 172.21.15.23:0/580812534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe0101f60 msgr2=0x7fcfe0073f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.759+0000 7fcfe59a2640 1 --2- 172.21.15.23:0/580812534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe0101f60 0x7fcfe0073f50 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc005940 tx=0x7fcfcc02c670 comp rx=0 tx=0).stop 2024-03-24T10:51:06.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.759+0000 7fcfe59a2640 1 -- 172.21.15.23:0/580812534 shutdown_connections 2024-03-24T10:51:06.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.759+0000 7fcfe59a2640 1 --2- 172.21.15.23:0/580812534 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fcfd00a2230 0x7fcfd00a46f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.759+0000 7fcfe59a2640 1 --2- 172.21.15.23:0/580812534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0107660 0x7fcfe0074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.759+0000 7fcfe59a2640 1 --2- 172.21.15.23:0/580812534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe0101f60 0x7fcfe0073f50 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.759+0000 7fcfe59a2640 1 --2- 172.21.15.23:0/580812534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcfe01015a0 0x7fcfe0073a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.759+0000 7fcfe59a2640 1 -- 172.21.15.23:0/580812534 >> 172.21.15.23:0/580812534 conn(0x7fcfe00fd0b0 msgr2=0x7fcfe00fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:06.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.759+0000 7fcfe59a2640 1 -- 172.21.15.23:0/580812534 shutdown_connections 2024-03-24T10:51:06.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.759+0000 7fcfe59a2640 1 -- 172.21.15.23:0/580812534 wait complete. 2024-03-24T10:51:06.770 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:51:06.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.876+0000 7fa6d8f93640 1 -- 172.21.15.23:0/3367739694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6d4106670 msgr2=0x7fa6d4106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.876+0000 7fa6d8f93640 1 --2- 172.21.15.23:0/3367739694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6d4106670 0x7fa6d4106a50 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4009900 tx=0x7fa6c402bc30 comp rx=0 tx=0).stop 2024-03-24T10:51:06.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.876+0000 7fa6d8f93640 1 -- 172.21.15.23:0/3367739694 shutdown_connections 2024-03-24T10:51:06.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.876+0000 7fa6d8f93640 1 --2- 172.21.15.23:0/3367739694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6d4100f70 0x7fa6d410d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.876+0000 7fa6d8f93640 1 --2- 172.21.15.23:0/3367739694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6d41005b0 0x7fa6d4100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.876+0000 7fa6d8f93640 1 --2- 172.21.15.23:0/3367739694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6d4106670 0x7fa6d4106a50 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.876+0000 7fa6d8f93640 1 -- 172.21.15.23:0/3367739694 >> 172.21.15.23:0/3367739694 conn(0x7fa6d40777c0 msgr2=0x7fa6d4077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:06.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.876+0000 7fa6d8f93640 1 -- 172.21.15.23:0/3367739694 shutdown_connections 2024-03-24T10:51:06.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.876+0000 7fa6d8f93640 1 -- 172.21.15.23:0/3367739694 wait complete. 2024-03-24T10:51:06.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.877+0000 7fa6d8f93640 1 Processor -- start 2024-03-24T10:51:06.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.877+0000 7fa6d8f93640 1 -- start start 2024-03-24T10:51:06.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.877+0000 7fa6d8f93640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6d41005b0 0x7fa6d4198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:06.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.877+0000 7fa6d8f93640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6d4100f70 0x7fa6d4199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:06.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.877+0000 7fa6d8f93640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6d4106670 0x7fa6d41a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:06.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.877+0000 7fa6d8f93640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d4199910 con 0x7fa6d4106670 2024-03-24T10:51:06.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.877+0000 7fa6d8f93640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d41a4110 con 0x7fa6d41005b0 2024-03-24T10:51:06.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.877+0000 7fa6d8f93640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d41a4250 con 0x7fa6d4100f70 2024-03-24T10:51:06.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.877+0000 7fa6d2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6d4106670 0x7fa6d41a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:06.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.877+0000 7fa6d2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6d4106670 0x7fa6d41a3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44766/0 (socket says 172.21.15.23:44766) 2024-03-24T10:51:06.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.877+0000 7fa6d259c640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6d41005b0 0x7fa6d4198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:06.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.877+0000 7fa6d2d9d640 1 -- 172.21.15.23:0/996834968 learned_addr learned my addr 172.21.15.23:0/996834968 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:51:06.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.877+0000 7fa6cbfff640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6d4100f70 0x7fa6d4199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:06.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.878+0000 7fa6d2d9d640 1 -- 172.21.15.23:0/996834968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6d4100f70 msgr2=0x7fa6d4199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.878+0000 7fa6d2d9d640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6d4100f70 0x7fa6d4199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.878+0000 7fa6d2d9d640 1 -- 172.21.15.23:0/996834968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6d41005b0 msgr2=0x7fa6d4198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.878+0000 7fa6d2d9d640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6d41005b0 0x7fa6d4198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.878+0000 7fa6d2d9d640 1 -- 172.21.15.23:0/996834968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c4009560 con 0x7fa6d4106670 2024-03-24T10:51:06.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.878+0000 7fa6d259c640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6d41005b0 0x7fa6d4198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:51:06.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.878+0000 7fa6cbfff640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6d4100f70 0x7fa6d4199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:51:06.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.878+0000 7fa6d2d9d640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6d4106670 0x7fa6d41a3bd0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc00eb40 tx=0x7fa6cc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:06.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.878+0000 7fa6cb7fe640 1 -- 172.21.15.23:0/996834968 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc00ceb0 con 0x7fa6d4106670 2024-03-24T10:51:06.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.878+0000 7fa6d8f93640 1 -- 172.21.15.23:0/996834968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d41a44b0 con 0x7fa6d4106670 2024-03-24T10:51:06.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.878+0000 7fa6cb7fe640 1 -- 172.21.15.23:0/996834968 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6cc005020 con 0x7fa6d4106670 2024-03-24T10:51:06.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.878+0000 7fa6cb7fe640 1 -- 172.21.15.23:0/996834968 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc010690 con 0x7fa6d4106670 2024-03-24T10:51:06.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.878+0000 7fa6d8f93640 1 -- 172.21.15.23:0/996834968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d41a49c0 con 0x7fa6d4106670 2024-03-24T10:51:06.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.879+0000 7fa6d8f93640 1 -- 172.21.15.23:0/996834968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6d4068500 con 0x7fa6d4106670 2024-03-24T10:51:06.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.881+0000 7fa6cb7fe640 1 -- 172.21.15.23:0/996834968 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fa6cc005190 con 0x7fa6d4106670 2024-03-24T10:51:06.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.881+0000 7fa6cb7fe640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa6c00a2230 0x7fa6c00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:06.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.881+0000 7fa6d259c640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa6c00a2230 0x7fa6c00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:06.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.882+0000 7fa6cb7fe640 1 -- 172.21.15.23:0/996834968 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fa6cc014070 con 0x7fa6d4106670 2024-03-24T10:51:06.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.882+0000 7fa6d259c640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa6c00a2230 0x7fa6c00a46f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4006010 tx=0x7fa6c4005e40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:06.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:06.884+0000 7fa6cb7fe640 1 -- 172.21.15.23:0/996834968 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa6cc09afe0 con 0x7fa6d4106670 2024-03-24T10:51:07.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.072+0000 7fa6d8f93640 1 -- 172.21.15.23:0/996834968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa6d4108a60 con 0x7fa6d4106670 2024-03-24T10:51:07.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.073+0000 7fa6cb7fe640 1 -- 172.21.15.23:0/996834968 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) ==== 100+0+11 (secure 0 0 0) 0x7fa6cc09ab70 con 0x7fa6d4106670 2024-03-24T10:51:07.074 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-24T10:51:07.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.075+0000 7fa6d8f93640 1 -- 172.21.15.23:0/996834968 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa6c00a2230 msgr2=0x7fa6c00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:07.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.075+0000 7fa6d8f93640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa6c00a2230 0x7fa6c00a46f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4006010 tx=0x7fa6c4005e40 comp rx=0 tx=0).stop 2024-03-24T10:51:07.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.075+0000 7fa6d8f93640 1 -- 172.21.15.23:0/996834968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6d4106670 msgr2=0x7fa6d41a3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:07.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.075+0000 7fa6d8f93640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6d4106670 0x7fa6d41a3bd0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc00eb40 tx=0x7fa6cc00c2d0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.075+0000 7fa6d8f93640 1 -- 172.21.15.23:0/996834968 shutdown_connections 2024-03-24T10:51:07.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.075+0000 7fa6d8f93640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fa6c00a2230 0x7fa6c00a46f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.075+0000 7fa6d8f93640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6d4106670 0x7fa6d41a3bd0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.075+0000 7fa6d8f93640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6d4100f70 0x7fa6d4199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.075+0000 7fa6d8f93640 1 --2- 172.21.15.23:0/996834968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6d41005b0 0x7fa6d4198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.075+0000 7fa6d8f93640 1 -- 172.21.15.23:0/996834968 >> 172.21.15.23:0/996834968 conn(0x7fa6d40777c0 msgr2=0x7fa6d40ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:07.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.075+0000 7fa6d8f93640 1 -- 172.21.15.23:0/996834968 shutdown_connections 2024-03-24T10:51:07.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.075+0000 7fa6d8f93640 1 -- 172.21.15.23:0/996834968 wait complete. 2024-03-24T10:51:07.085 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:06 smithi023 ceph-mon[32466]: pgmap v343: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-24T10:51:07.085 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:06 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4101273009' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:51:07.085 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:06 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1422239477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:51:07.085 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:06 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/580812534' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:51:07.086 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T10:51:07.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.190+0000 7ff078db6640 1 -- 172.21.15.23:0/1742427599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff074100990 msgr2=0x7ff07410e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:07.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.190+0000 7ff078db6640 1 --2- 172.21.15.23:0/1742427599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff074100990 0x7ff07410e590 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7ff06c009cd0 tx=0x7ff06c02c180 comp rx=0 tx=0).stop 2024-03-24T10:51:07.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.190+0000 7ff078db6640 1 -- 172.21.15.23:0/1742427599 shutdown_connections 2024-03-24T10:51:07.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.190+0000 7ff078db6640 1 --2- 172.21.15.23:0/1742427599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff074100990 0x7ff07410e590 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.190+0000 7ff078db6640 1 --2- 172.21.15.23:0/1742427599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0740fffd0 0x7ff074100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.190+0000 7ff078db6640 1 --2- 172.21.15.23:0/1742427599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff074107910 0x7ff074107cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.190+0000 7ff078db6640 1 -- 172.21.15.23:0/1742427599 >> 172.21.15.23:0/1742427599 conn(0x7ff074075950 msgr2=0x7ff074075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:07.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.190+0000 7ff078db6640 1 -- 172.21.15.23:0/1742427599 shutdown_connections 2024-03-24T10:51:07.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.190+0000 7ff078db6640 1 -- 172.21.15.23:0/1742427599 wait complete. 2024-03-24T10:51:07.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff078db6640 1 Processor -- start 2024-03-24T10:51:07.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff078db6640 1 -- start start 2024-03-24T10:51:07.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff078db6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740fffd0 0x7ff0741011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:07.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff078db6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff074100990 0x7ff0741016f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:07.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff078db6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff074107910 0x7ff074101c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:07.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff078db6640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7ff0741a61e0 con 0x7ff074107910 2024-03-24T10:51:07.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff078db6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff0741a6350 con 0x7ff0740fffd0 2024-03-24T10:51:07.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff078db6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff0741a6510 con 0x7ff074100990 2024-03-24T10:51:07.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff072d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff074107910 0x7ff074101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:07.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff07259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740fffd0 0x7ff0741011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:07.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff072d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff074107910 0x7ff074101c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44790/0 (socket says 172.21.15.23:44790) 2024-03-24T10:51:07.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff072d9d640 1 -- 172.21.15.23:0/3410865110 learned_addr learned my addr 172.21.15.23:0/3410865110 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:51:07.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff071d9b640 1 --2- 172.21.15.23:0/3410865110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff074100990 0x7ff0741016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:07.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff07259c640 1 -- 172.21.15.23:0/3410865110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff074100990 msgr2=0x7ff0741016f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:07.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.191+0000 7ff07259c640 1 --2- 172.21.15.23:0/3410865110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff074100990 0x7ff0741016f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.192+0000 7ff07259c640 1 -- 172.21.15.23:0/3410865110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff074107910 msgr2=0x7ff074101c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:07.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.192+0000 7ff07259c640 1 --2- 172.21.15.23:0/3410865110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff074107910 0x7ff074101c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.192+0000 7ff07259c640 1 -- 172.21.15.23:0/3410865110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff06c009900 con 0x7ff0740fffd0 2024-03-24T10:51:07.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.192+0000 7ff072d9d640 1 --2- 172.21.15.23:0/3410865110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff074107910 0x7ff074101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:51:07.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.192+0000 7ff07259c640 1 --2- 172.21.15.23:0/3410865110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740fffd0 0x7ff0741011b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff06400e9f0 tx=0x7ff06400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:07.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.192+0000 7ff0637fe640 1 -- 172.21.15.23:0/3410865110 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06400ccc0 con 0x7ff0740fffd0 2024-03-24T10:51:07.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.192+0000 7ff078db6640 1 -- 172.21.15.23:0/3410865110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0741a67f0 con 0x7ff0740fffd0 2024-03-24T10:51:07.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.192+0000 7ff0637fe640 1 -- 172.21.15.23:0/3410865110 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff06400ce20 con 0x7ff0740fffd0 2024-03-24T10:51:07.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.192+0000 7ff0637fe640 1 -- 172.21.15.23:0/3410865110 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0640188c0 con 0x7ff0740fffd0 2024-03-24T10:51:07.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.192+0000 7ff078db6640 1 -- 172.21.15.23:0/3410865110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0741a6d50 con 0x7ff0740fffd0 2024-03-24T10:51:07.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.193+0000 7ff078db6640 1 -- 172.21.15.23:0/3410865110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff074069820 con 0x7ff0740fffd0 2024-03-24T10:51:07.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.194+0000 7ff0637fe640 1 -- 172.21.15.23:0/3410865110 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7ff064018a20 con 0x7ff0740fffd0 2024-03-24T10:51:07.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.194+0000 7ff0637fe640 1 --2- 172.21.15.23:0/3410865110 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff05c0a2260 0x7ff05c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:07.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.195+0000 7ff071d9b640 1 --2- 172.21.15.23:0/3410865110 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff05c0a2260 0x7ff05c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:07.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.195+0000 7ff0637fe640 1 -- 172.21.15.23:0/3410865110 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7ff064014070 con 0x7ff0740fffd0 2024-03-24T10:51:07.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.195+0000 7ff071d9b640 1 --2- 172.21.15.23:0/3410865110 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff05c0a2260 0x7ff05c0a4720 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff074102bf0 tx=0x7ff068005f90 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:07.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.203+0000 7ff0637fe640 1 -- 172.21.15.23:0/3410865110 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff064096fa0 con 0x7ff0740fffd0 2024-03-24T10:51:07.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:06 smithi117 ceph-mon[36368]: pgmap v343: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-24T10:51:07.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:06 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4101273009' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:51:07.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:06 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1422239477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:51:07.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:06 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/580812534' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:51:07.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:06 smithi116 ceph-mon[36248]: pgmap v343: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-24T10:51:07.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:06 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4101273009' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:51:07.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:06 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1422239477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T10:51:07.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:06 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/580812534' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T10:51:07.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.400+0000 7ff078db6640 1 -- 172.21.15.23:0/3410865110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff074068950 con 0x7ff0740fffd0 2024-03-24T10:51:07.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.401+0000 7ff0637fe640 1 -- 172.21.15.23:0/3410865110 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v98) ==== 81+0+943 (secure 0 0 0) 0x7ff06409a660 con 0x7ff0740fffd0 2024-03-24T10:51:07.401 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/2350021632 2024-03-25T10:40:05.291912+0000 2024-03-24T10:51:07.401 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/2350021632 2024-03-25T10:40:05.291912+0000 2024-03-24T10:51:07.401 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3980246761 2024-03-25T10:31:14.712690+0000 2024-03-24T10:51:07.402 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1376242368 2024-03-25T10:31:14.712690+0000 2024-03-24T10:51:07.402 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3663301421 2024-03-25T10:31:14.712690+0000 2024-03-24T10:51:07.402 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3614005933 2024-03-25T10:40:05.291912+0000 2024-03-24T10:51:07.402 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/1774258590 2024-03-25T10:30:48.096659+0000 2024-03-24T10:51:07.402 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1794809128 2024-03-25T10:30:48.096659+0000 2024-03-24T10:51:07.402 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/2159398914 2024-03-25T10:31:14.712690+0000 2024-03-24T10:51:07.402 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/1774258590 2024-03-25T10:30:48.096659+0000 2024-03-24T10:51:07.402 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/256452041 2024-03-25T10:40:05.291912+0000 2024-03-24T10:51:07.402 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3434856686 2024-03-25T10:30:48.096659+0000 2024-03-24T10:51:07.403 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/352422381 2024-03-25T10:40:05.291912+0000 2024-03-24T10:51:07.403 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/725846898 2024-03-25T10:40:05.291912+0000 2024-03-24T10:51:07.403 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3666340600 2024-03-25T10:30:48.096659+0000 2024-03-24T10:51:07.403 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/2159398914 2024-03-25T10:31:14.712690+0000 2024-03-24T10:51:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:listed 16 entries 2024-03-24T10:51:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.403+0000 7ff078db6640 1 -- 172.21.15.23:0/3410865110 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff05c0a2260 msgr2=0x7ff05c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.403+0000 7ff078db6640 1 --2- 172.21.15.23:0/3410865110 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff05c0a2260 0x7ff05c0a4720 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff074102bf0 tx=0x7ff068005f90 comp rx=0 tx=0).stop 2024-03-24T10:51:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.403+0000 7ff078db6640 1 -- 172.21.15.23:0/3410865110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740fffd0 msgr2=0x7ff0741011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.403+0000 7ff078db6640 1 --2- 172.21.15.23:0/3410865110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740fffd0 0x7ff0741011b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff06400e9f0 tx=0x7ff06400eeb0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.403+0000 7ff078db6640 1 -- 172.21.15.23:0/3410865110 shutdown_connections 2024-03-24T10:51:07.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.403+0000 7ff078db6640 1 --2- 172.21.15.23:0/3410865110 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7ff05c0a2260 0x7ff05c0a4720 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.403+0000 7ff078db6640 1 --2- 172.21.15.23:0/3410865110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff074107910 0x7ff074101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.403+0000 7ff078db6640 1 --2- 172.21.15.23:0/3410865110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff074100990 0x7ff0741016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.403+0000 7ff078db6640 1 --2- 172.21.15.23:0/3410865110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0740fffd0 0x7ff0741011b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.403+0000 7ff078db6640 1 -- 172.21.15.23:0/3410865110 >> 172.21.15.23:0/3410865110 conn(0x7ff074075950 msgr2=0x7ff0740febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:07.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.403+0000 7ff078db6640 1 -- 172.21.15.23:0/3410865110 shutdown_connections 2024-03-24T10:51:07.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.403+0000 7ff078db6640 1 -- 172.21.15.23:0/3410865110 wait complete. 2024-03-24T10:51:07.414 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T10:51:07.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:51:07.414 DEBUG:teuthology.orchestra.run.smithi023:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T10:51:07.463 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T10:51:07.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:51:07.463 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T10:51:07.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:51:07.507 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-24T10:51:07.538 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T10:51:07.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:51:07.538 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T10:51:07.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:51:07.605 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2024-03-24T10:51:07.708 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T10:51:07.719 INFO:tasks.cephadm:Teardown begin 2024-03-24T10:51:07.719 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T10:51:07.719 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:07.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.823+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/2741039595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b4106670 msgr2=0x7fd9b4106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.823+0000 7fd9b9e2e640 1 --2- 172.21.15.23:0/2741039595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b4106670 0x7fd9b4106a50 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009930 tx=0x7fd9a402bc30 comp rx=0 tx=0).stop 2024-03-24T10:51:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.823+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/2741039595 shutdown_connections 2024-03-24T10:51:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.823+0000 7fd9b9e2e640 1 --2- 172.21.15.23:0/2741039595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b4100f90 0x7fd9b410d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.823+0000 7fd9b9e2e640 1 --2- 172.21.15.23:0/2741039595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b41005d0 0x7fd9b4100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.823+0000 7fd9b9e2e640 1 --2- 172.21.15.23:0/2741039595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b4106670 0x7fd9b4106a50 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.823+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/2741039595 >> 172.21.15.23:0/2741039595 conn(0x7fd9b40776b0 msgr2=0x7fd9b4077ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.823+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/2741039595 shutdown_connections 2024-03-24T10:51:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.823+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/2741039595 wait complete. 2024-03-24T10:51:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b9e2e640 1 Processor -- start 2024-03-24T10:51:07.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b9e2e640 1 -- start start 2024-03-24T10:51:07.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b9e2e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b41005d0 0x7fd9b4197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:07.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b9e2e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b4100f90 0x7fd9b41982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:07.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b9e2e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b4106670 0x7fd9b419c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b9e2e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b419cc30 con 0x7fd9b4106670 2024-03-24T10:51:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b9e2e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b419cda0 con 0x7fd9b4100f90 2024-03-24T10:51:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b9e2e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b419cf60 con 0x7fd9b41005d0 2024-03-24T10:51:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b4106670 0x7fd9b419c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b4106670 0x7fd9b419c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54852/0 (socket says 172.21.15.23:54852) 2024-03-24T10:51:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b3fff640 1 -- 172.21.15.23:0/4284742185 learned_addr learned my addr 172.21.15.23:0/4284742185 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:51:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b2ffd640 1 --2- 172.21.15.23:0/4284742185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b4100f90 0x7fd9b41982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:07.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b37fe640 1 --2- 172.21.15.23:0/4284742185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b41005d0 0x7fd9b4197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:07.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b3fff640 1 -- 172.21.15.23:0/4284742185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b41005d0 msgr2=0x7fd9b4197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:07.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b3fff640 1 --2- 172.21.15.23:0/4284742185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b41005d0 0x7fd9b4197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b3fff640 1 -- 172.21.15.23:0/4284742185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b4100f90 msgr2=0x7fd9b41982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:07.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b3fff640 1 --2- 172.21.15.23:0/4284742185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b4100f90 0x7fd9b41982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:07.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.824+0000 7fd9b3fff640 1 -- 172.21.15.23:0/4284742185 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a4009560 con 0x7fd9b4106670 2024-03-24T10:51:07.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.825+0000 7fd9b3fff640 1 --2- 172.21.15.23:0/4284742185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b4106670 0x7fd9b419c630 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00eb10 tx=0x7fd9ac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:07.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.825+0000 7fd9b0ff9640 1 -- 172.21.15.23:0/4284742185 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac004b20 con 0x7fd9b4106670 2024-03-24T10:51:07.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.825+0000 7fd9b0ff9640 1 -- 172.21.15.23:0/4284742185 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9ac004c80 con 0x7fd9b4106670 2024-03-24T10:51:07.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.825+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/4284742185 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b419d240 con 0x7fd9b4106670 2024-03-24T10:51:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.825+0000 7fd9b0ff9640 1 -- 172.21.15.23:0/4284742185 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac010650 con 0x7fd9b4106670 2024-03-24T10:51:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.825+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/4284742185 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b41a4ac0 con 0x7fd9b4106670 2024-03-24T10:51:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.826+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/4284742185 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9b4067c70 con 0x7fd9b4106670 2024-03-24T10:51:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.827+0000 7fd9b0ff9640 1 -- 172.21.15.23:0/4284742185 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fd9ac005020 con 0x7fd9b4106670 2024-03-24T10:51:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.827+0000 7fd9b0ff9640 1 --2- 172.21.15.23:0/4284742185 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd99c0a2230 0x7fd99c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.827+0000 7fd9b37fe640 1 --2- 172.21.15.23:0/4284742185 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd99c0a2230 0x7fd99c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:07.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.827+0000 7fd9b0ff9640 1 -- 172.21.15.23:0/4284742185 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fd9ac014070 con 0x7fd9b4106670 2024-03-24T10:51:07.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.828+0000 7fd9b37fe640 1 --2- 172.21.15.23:0/4284742185 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd99c0a2230 0x7fd99c0a46f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4000c00 tx=0x7fd9a4005fd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:07.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:07.832+0000 7fd9b0ff9640 1 -- 172.21.15.23:0/4284742185 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd9ac097220 con 0x7fd9b4106670 2024-03-24T10:51:08.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.054+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/4284742185 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9b4108b70 con 0x7fd9b4106670 2024-03-24T10:51:08.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.055+0000 7fd9b0ff9640 1 -- 172.21.15.23:0/4284742185 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11869 (secure 0 0 0) 0x7fd9ac09a8e0 con 0x7fd9b4106670 2024-03-24T10:51:08.055 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:51:08.055 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:47:17.474711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27526":{"gid":27526,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/2017557046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":2017557046},{"type":"v1","addr":"172.21.15.117:6837","nonce":2017557046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:51:08.055 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 31 2024-03-24T10:51:08.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.056+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/4284742185 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd99c0a2230 msgr2=0x7fd99c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.056+0000 7fd9b9e2e640 1 --2- 172.21.15.23:0/4284742185 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd99c0a2230 0x7fd99c0a46f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4000c00 tx=0x7fd9a4005fd0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.056+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/4284742185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b4106670 msgr2=0x7fd9b419c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.056+0000 7fd9b9e2e640 1 --2- 172.21.15.23:0/4284742185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b4106670 0x7fd9b419c630 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00eb10 tx=0x7fd9ac00c2d0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.056+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/4284742185 shutdown_connections 2024-03-24T10:51:08.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.056+0000 7fd9b9e2e640 1 --2- 172.21.15.23:0/4284742185 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd99c0a2230 0x7fd99c0a46f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.057+0000 7fd9b9e2e640 1 --2- 172.21.15.23:0/4284742185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9b4106670 0x7fd9b419c630 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.057+0000 7fd9b9e2e640 1 --2- 172.21.15.23:0/4284742185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b4100f90 0x7fd9b41982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.057+0000 7fd9b9e2e640 1 --2- 172.21.15.23:0/4284742185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b41005d0 0x7fd9b4197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.057+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/4284742185 >> 172.21.15.23:0/4284742185 conn(0x7fd9b40776b0 msgr2=0x7fd9b40ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:08.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.057+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/4284742185 shutdown_connections 2024-03-24T10:51:08.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.057+0000 7fd9b9e2e640 1 -- 172.21.15.23:0/4284742185 wait complete. 2024-03-24T10:51:08.067 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:08.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.170+0000 7fd459d43640 1 -- 172.21.15.23:0/3937115422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4541005d0 msgr2=0x7fd454100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.170+0000 7fd459d43640 1 --2- 172.21.15.23:0/3937115422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4541005d0 0x7fd454100a50 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fd448009900 tx=0x7fd44802bc30 comp rx=0 tx=0).stop 2024-03-24T10:51:08.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.170+0000 7fd459d43640 1 -- 172.21.15.23:0/3937115422 shutdown_connections 2024-03-24T10:51:08.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.170+0000 7fd459d43640 1 --2- 172.21.15.23:0/3937115422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd454100f90 0x7fd45410d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.170+0000 7fd459d43640 1 --2- 172.21.15.23:0/3937115422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4541005d0 0x7fd454100a50 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.170+0000 7fd459d43640 1 --2- 172.21.15.23:0/3937115422 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd454106690 0x7fd454106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.170+0000 7fd459d43640 1 -- 172.21.15.23:0/3937115422 >> 172.21.15.23:0/3937115422 conn(0x7fd454076990 msgr2=0x7fd454076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:08.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.170+0000 7fd459d43640 1 -- 172.21.15.23:0/3937115422 shutdown_connections 2024-03-24T10:51:08.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.170+0000 7fd459d43640 1 -- 172.21.15.23:0/3937115422 wait complete. 2024-03-24T10:51:08.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.170+0000 7fd459d43640 1 Processor -- start 2024-03-24T10:51:08.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd459d43640 1 -- start start 2024-03-24T10:51:08.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd459d43640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4541005d0 0x7fd45419d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd459d43640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd454100f90 0x7fd45419de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd459d43640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd454106690 0x7fd454197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd459d43640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd454198130 con 0x7fd4541005d0 2024-03-24T10:51:08.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd459d43640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd4541982a0 con 0x7fd454100f90 2024-03-24T10:51:08.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd459d43640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd454198460 con 0x7fd454106690 2024-03-24T10:51:08.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd4537fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4541005d0 0x7fd45419d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd4537fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4541005d0 0x7fd45419d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54868/0 (socket says 172.21.15.23:54868) 2024-03-24T10:51:08.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd4537fe640 1 -- 172.21.15.23:0/4159510316 learned_addr learned my addr 172.21.15.23:0/4159510316 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:51:08.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd452ffd640 1 --2- 172.21.15.23:0/4159510316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd454100f90 0x7fd45419de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd453fff640 1 --2- 172.21.15.23:0/4159510316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd454106690 0x7fd454197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd4537fe640 1 -- 172.21.15.23:0/4159510316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd454106690 msgr2=0x7fd454197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd4537fe640 1 --2- 172.21.15.23:0/4159510316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd454106690 0x7fd454197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd4537fe640 1 -- 172.21.15.23:0/4159510316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd454100f90 msgr2=0x7fd45419de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd4537fe640 1 --2- 172.21.15.23:0/4159510316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd454100f90 0x7fd45419de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.171+0000 7fd4537fe640 1 -- 172.21.15.23:0/4159510316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd448009560 con 0x7fd4541005d0 2024-03-24T10:51:08.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.172+0000 7fd4537fe640 1 --2- 172.21.15.23:0/4159510316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4541005d0 0x7fd45419d930 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fd44400eb90 tx=0x7fd44400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:08.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.172+0000 7fd450ff9640 1 -- 172.21.15.23:0/4159510316 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4440099b0 con 0x7fd4541005d0 2024-03-24T10:51:08.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.172+0000 7fd459d43640 1 -- 172.21.15.23:0/4159510316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd454198740 con 0x7fd4541005d0 2024-03-24T10:51:08.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.172+0000 7fd450ff9640 1 -- 172.21.15.23:0/4159510316 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd44400ee90 con 0x7fd4541005d0 2024-03-24T10:51:08.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.172+0000 7fd450ff9640 1 -- 172.21.15.23:0/4159510316 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4440107a0 con 0x7fd4541005d0 2024-03-24T10:51:08.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.172+0000 7fd459d43640 1 -- 172.21.15.23:0/4159510316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4541a49d0 con 0x7fd4541005d0 2024-03-24T10:51:08.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.173+0000 7fd459d43640 1 -- 172.21.15.23:0/4159510316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd454101cf0 con 0x7fd4541005d0 2024-03-24T10:51:08.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.174+0000 7fd450ff9640 1 -- 172.21.15.23:0/4159510316 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fd444005000 con 0x7fd4541005d0 2024-03-24T10:51:08.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.175+0000 7fd450ff9640 1 --2- 172.21.15.23:0/4159510316 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd43c0a2230 0x7fd43c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.175+0000 7fd452ffd640 1 --2- 172.21.15.23:0/4159510316 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd43c0a2230 0x7fd43c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.175+0000 7fd450ff9640 1 -- 172.21.15.23:0/4159510316 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7fd444014070 con 0x7fd4541005d0 2024-03-24T10:51:08.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.175+0000 7fd452ffd640 1 --2- 172.21.15.23:0/4159510316 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd43c0a2230 0x7fd43c0a46f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd448006010 tx=0x7fd448005e40 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:08.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.178+0000 7fd450ff9640 1 -- 172.21.15.23:0/4159510316 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd444097850 con 0x7fd4541005d0 2024-03-24T10:51:08.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:07 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/996834968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:51:08.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:07 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/3410865110' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T10:51:08.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:07 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/996834968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:51:08.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:07 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/3410865110' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T10:51:08.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.400+0000 7fd459d43640 1 -- 172.21.15.23:0/4159510316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd454198fd0 con 0x7fd4541005d0 2024-03-24T10:51:08.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.400+0000 7fd450ff9640 1 -- 172.21.15.23:0/4159510316 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11869 (secure 0 0 0) 0x7fd44409af10 con 0x7fd4541005d0 2024-03-24T10:51:08.401 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:51:08.401 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3844921538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":3844921538},{"type":"v1","addr":"172.21.15.23:6835","nonce":3844921538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/405041362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":405041362},{"type":"v1","addr":"172.21.15.23:6837","nonce":405041362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/4279444411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":4279444411},{"type":"v1","addr":"172.21.15.23:6839","nonce":4279444411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/644733222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":644733222},{"type":"v1","addr":"172.21.15.23:6841","nonce":644733222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/134162564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":134162564},{"type":"v1","addr":"172.21.15.117:6833","nonce":134162564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/566816400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":566816400},{"type":"v1","addr":"172.21.15.116:6833","nonce":566816400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T10:38:10.206899+0000","modified":"2024-03-24T10:47:17.474711+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24505,"mds_2":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/1535852767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1535852767},{"type":"v1","addr":"172.21.15.116:6835","nonce":1535852767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/1664408853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1664408853},{"type":"v1","addr":"172.21.15.117:6835","nonce":1664408853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6837/3576031724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3576031724},{"type":"v1","addr":"172.21.15.116:6837","nonce":3576031724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.117:6839/263724286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":263724286},{"type":"v1","addr":"172.21.15.117:6839","nonce":263724286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.116:6839/2435082224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2435082224},{"type":"v1","addr":"172.21.15.116:6839","nonce":2435082224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27526":{"gid":27526,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/2017557046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":2017557046},{"type":"v1","addr":"172.21.15.117:6837","nonce":2017557046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24503,24505,24539]},"id":1}]} 2024-03-24T10:51:08.401 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 31 2024-03-24T10:51:08.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.402+0000 7fd459d43640 1 -- 172.21.15.23:0/4159510316 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd43c0a2230 msgr2=0x7fd43c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.402+0000 7fd459d43640 1 --2- 172.21.15.23:0/4159510316 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd43c0a2230 0x7fd43c0a46f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd448006010 tx=0x7fd448005e40 comp rx=0 tx=0).stop 2024-03-24T10:51:08.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.403+0000 7fd459d43640 1 -- 172.21.15.23:0/4159510316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4541005d0 msgr2=0x7fd45419d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.403+0000 7fd459d43640 1 --2- 172.21.15.23:0/4159510316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4541005d0 0x7fd45419d930 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fd44400eb90 tx=0x7fd44400c6a0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.403+0000 7fd459d43640 1 -- 172.21.15.23:0/4159510316 shutdown_connections 2024-03-24T10:51:08.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.403+0000 7fd459d43640 1 --2- 172.21.15.23:0/4159510316 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd43c0a2230 0x7fd43c0a46f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.403+0000 7fd459d43640 1 --2- 172.21.15.23:0/4159510316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd454106690 0x7fd454197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.403+0000 7fd459d43640 1 --2- 172.21.15.23:0/4159510316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd454100f90 0x7fd45419de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.403+0000 7fd459d43640 1 --2- 172.21.15.23:0/4159510316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4541005d0 0x7fd45419d930 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.403+0000 7fd459d43640 1 -- 172.21.15.23:0/4159510316 >> 172.21.15.23:0/4159510316 conn(0x7fd454076990 msgr2=0x7fd4540ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:08.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.403+0000 7fd459d43640 1 -- 172.21.15.23:0/4159510316 shutdown_connections 2024-03-24T10:51:08.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.403+0000 7fd459d43640 1 -- 172.21.15.23:0/4159510316 wait complete. 2024-03-24T10:51:08.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:07 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/996834968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T10:51:08.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:07 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/3410865110' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T10:51:08.414 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:51:08.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.517+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/936609314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d74100f80 msgr2=0x7f9d7410d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.517+0000 7f9d7c4aa640 1 --2- 172.21.15.23:0/936609314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d74100f80 0x7f9d7410d490 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c009be0 tx=0x7f9d6c02bef0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.517+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/936609314 shutdown_connections 2024-03-24T10:51:08.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.517+0000 7f9d7c4aa640 1 --2- 172.21.15.23:0/936609314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d74100f80 0x7f9d7410d490 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.517+0000 7f9d7c4aa640 1 --2- 172.21.15.23:0/936609314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d741005c0 0x7f9d74100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.517+0000 7f9d7c4aa640 1 --2- 172.21.15.23:0/936609314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d74106660 0x7f9d74106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.517+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/936609314 >> 172.21.15.23:0/936609314 conn(0x7f9d74076990 msgr2=0x7f9d74076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:08.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.517+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/936609314 shutdown_connections 2024-03-24T10:51:08.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.517+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/936609314 wait complete. 2024-03-24T10:51:08.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.517+0000 7f9d7c4aa640 1 Processor -- start 2024-03-24T10:51:08.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.517+0000 7f9d7c4aa640 1 -- start start 2024-03-24T10:51:08.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d7c4aa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d741005c0 0x7f9d7419d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d7c4aa640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d74100f80 0x7f9d7419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d7c4aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d74106660 0x7f9d74197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d7c4aa640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f9d741980b0 con 0x7f9d74100f80 2024-03-24T10:51:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d7c4aa640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9d74198220 con 0x7f9d74106660 2024-03-24T10:51:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d7c4aa640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9d741983e0 con 0x7f9d741005c0 2024-03-24T10:51:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d79a45640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d74100f80 0x7f9d7419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d7a246640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d741005c0 0x7f9d7419d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d79a45640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d74100f80 0x7f9d7419ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54892/0 (socket says 172.21.15.23:54892) 2024-03-24T10:51:08.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d7aa47640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d74106660 0x7f9d74197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d79a45640 1 -- 172.21.15.23:0/4170858016 learned_addr learned my addr 172.21.15.23:0/4170858016 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:51:08.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d79a45640 1 -- 172.21.15.23:0/4170858016 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d741005c0 msgr2=0x7f9d7419d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d79a45640 1 --2- 172.21.15.23:0/4170858016 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d741005c0 0x7f9d7419d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d79a45640 1 -- 172.21.15.23:0/4170858016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d74106660 msgr2=0x7f9d74197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d79a45640 1 --2- 172.21.15.23:0/4170858016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d74106660 0x7f9d74197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d79a45640 1 -- 172.21.15.23:0/4170858016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d68009560 con 0x7f9d74100f80 2024-03-24T10:51:08.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d79a45640 1 --2- 172.21.15.23:0/4170858016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d74100f80 0x7f9d7419ddf0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f9d7000e9f0 tx=0x7f9d7000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:08.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d677fe640 1 -- 172.21.15.23:0/4170858016 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70009720 con 0x7f9d74100f80 2024-03-24T10:51:08.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d677fe640 1 -- 172.21.15.23:0/4170858016 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d7000cd80 con 0x7f9d74100f80 2024-03-24T10:51:08.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/4170858016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d6c009840 con 0x7f9d74100f80 2024-03-24T10:51:08.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.518+0000 7f9d677fe640 1 -- 172.21.15.23:0/4170858016 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d70010690 con 0x7f9d74100f80 2024-03-24T10:51:08.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.519+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/4170858016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d741a4770 con 0x7f9d74100f80 2024-03-24T10:51:08.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.519+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/4170858016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d74101d00 con 0x7f9d74100f80 2024-03-24T10:51:08.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.521+0000 7f9d677fe640 1 -- 172.21.15.23:0/4170858016 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f9d700046c0 con 0x7f9d74100f80 2024-03-24T10:51:08.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.521+0000 7f9d677fe640 1 --2- 172.21.15.23:0/4170858016 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f9d600a2230 0x7f9d600a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.521+0000 7f9d7a246640 1 --2- 172.21.15.23:0/4170858016 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f9d600a2230 0x7f9d600a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.521+0000 7f9d677fe640 1 -- 172.21.15.23:0/4170858016 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f9d70014070 con 0x7f9d74100f80 2024-03-24T10:51:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.522+0000 7f9d7a246640 1 --2- 172.21.15.23:0/4170858016 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f9d600a2230 0x7f9d600a46f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9d74198ba0 tx=0x7f9d68009260 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.525+0000 7f9d677fe640 1 -- 172.21.15.23:0/4170858016 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9d70097280 con 0x7f9d74100f80 2024-03-24T10:51:08.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.708+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/4170858016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9d74068950 con 0x7f9d74100f80 2024-03-24T10:51:08.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.710+0000 7f9d677fe640 1 -- 172.21.15.23:0/4170858016 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+23169 (secure 0 0 0) 0x7f9d7009a940 con 0x7f9d74100f80 2024-03-24T10:51:08.710 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-24T10:51:08.711 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":98,"fsid":"4f8fb8f6-e9c9-11ee-95cd-87774f69a715","created":"2024-03-24T10:30:20.456703+0000","modified":"2024-03-24T10:50:16.243119+0000","last_up_change":"2024-03-24T10:37:20.714379+0000","last_in_change":"2024-03-24T10:37:00.563157+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:33:39.056452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:38:08.473472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:38:09.414667+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":97,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e6c1781a-5af4-4191-b299-02371336d02a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6803","nonce":65775038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6805","nonce":65775038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6809","nonce":65775038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":65775038},{"type":"v1","addr":"172.21.15.23:6807","nonce":65775038}]},"public_addr":"172.21.15.23:6803/65775038","cluster_addr":"172.21.15.23:6805/65775038","heartbeat_back_addr":"172.21.15.23:6809/65775038","heartbeat_front_addr":"172.21.15.23:6807/65775038","state":["exists","up"]},{"osd":1,"uuid":"8fd17c0c-e834-46bf-bb82-b247e4d5bb01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6801","nonce":1598773126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6803","nonce":1598773126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6807","nonce":1598773126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1598773126},{"type":"v1","addr":"172.21.15.116:6805","nonce":1598773126}]},"public_addr":"172.21.15.116:6801/1598773126","cluster_addr":"172.21.15.116:6803/1598773126","heartbeat_back_addr":"172.21.15.116:6807/1598773126","heartbeat_front_addr":"172.21.15.116:6805/1598773126","state":["exists","up"]},{"osd":2,"uuid":"8ec590e7-eda9-4938-94c9-b1f61f8dfc6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6801","nonce":3979033351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6803","nonce":3979033351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6807","nonce":3979033351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3979033351},{"type":"v1","addr":"172.21.15.117:6805","nonce":3979033351}]},"public_addr":"172.21.15.117:6801/3979033351","cluster_addr":"172.21.15.117:6803/3979033351","heartbeat_back_addr":"172.21.15.117:6807/3979033351","heartbeat_front_addr":"172.21.15.117:6805/3979033351","state":["exists","up"]},{"osd":3,"uuid":"61dcad05-5af6-4faf-860c-54f5e3566b84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6811","nonce":1780984549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6813","nonce":1780984549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6817","nonce":1780984549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1780984549},{"type":"v1","addr":"172.21.15.23:6815","nonce":1780984549}]},"public_addr":"172.21.15.23:6811/1780984549","cluster_addr":"172.21.15.23:6813/1780984549","heartbeat_back_addr":"172.21.15.23:6817/1780984549","heartbeat_front_addr":"172.21.15.23:6815/1780984549","state":["exists","up"]},{"osd":4,"uuid":"7d56de8f-6140-477e-b851-42928411277d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6809","nonce":3074814953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6811","nonce":3074814953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6815","nonce":3074814953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3074814953},{"type":"v1","addr":"172.21.15.116:6813","nonce":3074814953}]},"public_addr":"172.21.15.116:6809/3074814953","cluster_addr":"172.21.15.116:6811/3074814953","heartbeat_back_addr":"172.21.15.116:6815/3074814953","heartbeat_front_addr":"172.21.15.116:6813/3074814953","state":["exists","up"]},{"osd":5,"uuid":"9f52e187-a40f-4360-8e5e-b665e241657e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6809","nonce":1549866608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6811","nonce":1549866608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6815","nonce":1549866608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1549866608},{"type":"v1","addr":"172.21.15.117:6813","nonce":1549866608}]},"public_addr":"172.21.15.117:6809/1549866608","cluster_addr":"172.21.15.117:6811/1549866608","heartbeat_back_addr":"172.21.15.117:6815/1549866608","heartbeat_front_addr":"172.21.15.117:6813/1549866608","state":["exists","up"]},{"osd":6,"uuid":"32e3081d-af77-4879-ab4c-e77a0303b741","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6819","nonce":1181430315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6821","nonce":1181430315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6825","nonce":1181430315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":1181430315},{"type":"v1","addr":"172.21.15.23:6823","nonce":1181430315}]},"public_addr":"172.21.15.23:6819/1181430315","cluster_addr":"172.21.15.23:6821/1181430315","heartbeat_back_addr":"172.21.15.23:6825/1181430315","heartbeat_front_addr":"172.21.15.23:6823/1181430315","state":["exists","up"]},{"osd":7,"uuid":"dddddbc7-f15a-4844-8d5a-869421e3dead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6817","nonce":2721290459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6819","nonce":2721290459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6823","nonce":2721290459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2721290459},{"type":"v1","addr":"172.21.15.116:6821","nonce":2721290459}]},"public_addr":"172.21.15.116:6817/2721290459","cluster_addr":"172.21.15.116:6819/2721290459","heartbeat_back_addr":"172.21.15.116:6823/2721290459","heartbeat_front_addr":"172.21.15.116:6821/2721290459","state":["exists","up"]},{"osd":8,"uuid":"e84fd4bc-7173-410c-84f2-5c9dcf91b790","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6817","nonce":3633840563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6819","nonce":3633840563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6823","nonce":3633840563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3633840563},{"type":"v1","addr":"172.21.15.117:6821","nonce":3633840563}]},"public_addr":"172.21.15.117:6817/3633840563","cluster_addr":"172.21.15.117:6819/3633840563","heartbeat_back_addr":"172.21.15.117:6823/3633840563","heartbeat_front_addr":"172.21.15.117:6821/3633840563","state":["exists","up"]},{"osd":9,"uuid":"7a1d9119-fd0f-45b4-a0f7-051b118e1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6827","nonce":3239156530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6829","nonce":3239156530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6833","nonce":3239156530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3239156530},{"type":"v1","addr":"172.21.15.23:6831","nonce":3239156530}]},"public_addr":"172.21.15.23:6827/3239156530","cluster_addr":"172.21.15.23:6829/3239156530","heartbeat_back_addr":"172.21.15.23:6833/3239156530","heartbeat_front_addr":"172.21.15.23:6831/3239156530","state":["exists","up"]},{"osd":10,"uuid":"cef53e0f-9f79-44c7-b8ea-6849ade8bb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6825","nonce":2413381037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6827","nonce":2413381037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6831","nonce":2413381037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2413381037},{"type":"v1","addr":"172.21.15.116:6829","nonce":2413381037}]},"public_addr":"172.21.15.116:6825/2413381037","cluster_addr":"172.21.15.116:6827/2413381037","heartbeat_back_addr":"172.21.15.116:6831/2413381037","heartbeat_front_addr":"172.21.15.116:6829/2413381037","state":["exists","up"]},{"osd":11,"uuid":"d763fbde-fd35-4847-b8f1-8f22553ee1c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6825","nonce":3654755153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6827","nonce":3654755153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6831","nonce":3654755153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3654755153},{"type":"v1","addr":"172.21.15.117:6829","nonce":3654755153}]},"public_addr":"172.21.15.117:6825/3654755153","cluster_addr":"172.21.15.117:6827/3654755153","heartbeat_back_addr":"172.21.15.117:6831/3654755153","heartbeat_front_addr":"172.21.15.117:6829/3654755153","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:32:50.059467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:13.436219+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:33:37.266649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:01.921430+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:25.268915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:34:49.497037+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:14.937389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:35:39.047713+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:03.356442+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:29.386144+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:36:53.609003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:37:18.628260+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2350021632":"2024-03-25T10:40:05.291912+0000","172.21.15.23:6800/2350021632":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3980246761":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/1376242368":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3663301421":"2024-03-25T10:31:14.712690+0000","172.21.15.23:0/3614005933":"2024-03-25T10:40:05.291912+0000","172.21.15.23:6800/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/1794809128":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6801/2159398914":"2024-03-25T10:31:14.712690+0000","172.21.15.23:6801/1774258590":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/256452041":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3434856686":"2024-03-25T10:30:48.096659+0000","172.21.15.23:0/352422381":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/725846898":"2024-03-25T10:40:05.291912+0000","172.21.15.23:0/3666340600":"2024-03-25T10:30:48.096659+0000","172.21.15.23:6800/2159398914":"2024-03-25T10:31:14.712690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:51:08.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.712+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/4170858016 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f9d600a2230 msgr2=0x7f9d600a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.712+0000 7f9d7c4aa640 1 --2- 172.21.15.23:0/4170858016 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f9d600a2230 0x7f9d600a46f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9d74198ba0 tx=0x7f9d68009260 comp rx=0 tx=0).stop 2024-03-24T10:51:08.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.712+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/4170858016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d74100f80 msgr2=0x7f9d7419ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.712+0000 7f9d7c4aa640 1 --2- 172.21.15.23:0/4170858016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d74100f80 0x7f9d7419ddf0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f9d7000e9f0 tx=0x7f9d7000edf0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.712+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/4170858016 shutdown_connections 2024-03-24T10:51:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.712+0000 7f9d7c4aa640 1 --2- 172.21.15.23:0/4170858016 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f9d600a2230 0x7f9d600a46f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.712+0000 7f9d7c4aa640 1 --2- 172.21.15.23:0/4170858016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d74106660 0x7f9d74197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.712+0000 7f9d7c4aa640 1 --2- 172.21.15.23:0/4170858016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d74100f80 0x7f9d7419ddf0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.712+0000 7f9d7c4aa640 1 --2- 172.21.15.23:0/4170858016 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d741005c0 0x7f9d7419d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.712+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/4170858016 >> 172.21.15.23:0/4170858016 conn(0x7f9d74076990 msgr2=0x7f9d740fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.712+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/4170858016 shutdown_connections 2024-03-24T10:51:08.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.712+0000 7f9d7c4aa640 1 -- 172.21.15.23:0/4170858016 wait complete. 2024-03-24T10:51:08.724 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T10:51:08.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.827+0000 7f422c18a640 1 -- 172.21.15.23:0/1846394646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4224100f80 msgr2=0x7f422410d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.827+0000 7f422c18a640 1 --2- 172.21.15.23:0/1846394646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4224100f80 0x7f422410d490 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f421c009c10 tx=0x7f421c02bef0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.827+0000 7f422c18a640 1 -- 172.21.15.23:0/1846394646 shutdown_connections 2024-03-24T10:51:08.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.827+0000 7f422c18a640 1 --2- 172.21.15.23:0/1846394646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4224100f80 0x7f422410d490 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.827+0000 7f422c18a640 1 --2- 172.21.15.23:0/1846394646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42241005c0 0x7f4224100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.827+0000 7f422c18a640 1 --2- 172.21.15.23:0/1846394646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4224106660 0x7f4224106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.827+0000 7f422c18a640 1 -- 172.21.15.23:0/1846394646 >> 172.21.15.23:0/1846394646 conn(0x7f4224076990 msgr2=0x7f4224076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:08.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.827+0000 7f422c18a640 1 -- 172.21.15.23:0/1846394646 shutdown_connections 2024-03-24T10:51:08.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.827+0000 7f422c18a640 1 -- 172.21.15.23:0/1846394646 wait complete. 2024-03-24T10:51:08.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422c18a640 1 Processor -- start 2024-03-24T10:51:08.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422c18a640 1 -- start start 2024-03-24T10:51:08.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422c18a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42241005c0 0x7f422410de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422c18a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4224100f80 0x7f422410e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422c18a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4224106660 0x7f422410e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422c18a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f42241a6050 con 0x7f4224106660 2024-03-24T10:51:08.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422c18a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f42241a61c0 con 0x7f42241005c0 2024-03-24T10:51:08.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422c18a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f42241a6380 con 0x7f4224100f80 2024-03-24T10:51:08.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422a727640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4224106660 0x7f422410e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422a727640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4224106660 0x7f422410e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54918/0 (socket says 172.21.15.23:54918) 2024-03-24T10:51:08.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422a727640 1 -- 172.21.15.23:0/2994381987 learned_addr learned my addr 172.21.15.23:0/2994381987 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:51:08.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f4229725640 1 --2- 172.21.15.23:0/2994381987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4224100f80 0x7f422410e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f4229f26640 1 --2- 172.21.15.23:0/2994381987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42241005c0 0x7f422410de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422a727640 1 -- 172.21.15.23:0/2994381987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4224100f80 msgr2=0x7f422410e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422a727640 1 --2- 172.21.15.23:0/2994381987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4224100f80 0x7f422410e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422a727640 1 -- 172.21.15.23:0/2994381987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42241005c0 msgr2=0x7f422410de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422a727640 1 --2- 172.21.15.23:0/2994381987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42241005c0 0x7f422410de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422a727640 1 -- 172.21.15.23:0/2994381987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4218009560 con 0x7f4224106660 2024-03-24T10:51:08.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.828+0000 7f422a727640 1 --2- 172.21.15.23:0/2994381987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4224106660 0x7f422410e8e0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f421c036040 tx=0x7f421c02cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:08.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.829+0000 7f4216ffd640 1 -- 172.21.15.23:0/2994381987 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c003970 con 0x7f4224106660 2024-03-24T10:51:08.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.829+0000 7f422c18a640 1 -- 172.21.15.23:0/2994381987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f421c009840 con 0x7f4224106660 2024-03-24T10:51:08.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.829+0000 7f4216ffd640 1 -- 172.21.15.23:0/2994381987 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f421c004040 con 0x7f4224106660 2024-03-24T10:51:08.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.829+0000 7f4216ffd640 1 -- 172.21.15.23:0/2994381987 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c03d790 con 0x7f4224106660 2024-03-24T10:51:08.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.829+0000 7f422c18a640 1 -- 172.21.15.23:0/2994381987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42241a6920 con 0x7f4224106660 2024-03-24T10:51:08.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.830+0000 7f422c18a640 1 -- 172.21.15.23:0/2994381987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4224101d00 con 0x7f4224106660 2024-03-24T10:51:08.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.831+0000 7f4216ffd640 1 -- 172.21.15.23:0/2994381987 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f421c003ad0 con 0x7f4224106660 2024-03-24T10:51:08.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.831+0000 7f4216ffd640 1 --2- 172.21.15.23:0/2994381987 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f42100a2230 0x7f42100a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.831+0000 7f4229f26640 1 --2- 172.21.15.23:0/2994381987 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f42100a2230 0x7f42100a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.831+0000 7f4216ffd640 1 -- 172.21.15.23:0/2994381987 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8571+0+0 (secure 0 0 0) 0x7f421c0f0560 con 0x7f4224106660 2024-03-24T10:51:08.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.832+0000 7f4229f26640 1 --2- 172.21.15.23:0/2994381987 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f42100a2230 0x7f42100a46f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4218009530 tx=0x7f4218009260 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:08.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:08.835+0000 7f4216ffd640 1 -- 172.21.15.23:0/2994381987 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f421c0b9380 con 0x7f4224106660 2024-03-24T10:51:09.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.059+0000 7f422c18a640 1 -- 172.21.15.23:0/2994381987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4224108b50 con 0x7f4224106660 2024-03-24T10:51:09.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.061+0000 7f4216ffd640 1 -- 172.21.15.23:0/2994381987 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) ==== 161+0+0 (secure 0 0 0) 0x7f421c0bca40 con 0x7f4224106660 2024-03-24T10:51:09.061 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:08 smithi023 ceph-mon[32466]: pgmap v344: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:51:09.061 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:08 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4284742185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:51:09.061 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:08 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4159510316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:51:09.062 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:08 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4170858016' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:51:09.062 INFO:teuthology.orchestra.run.smithi023.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T10:51:09.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.064+0000 7f422c18a640 1 -- 172.21.15.23:0/2994381987 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f42100a2230 msgr2=0x7f42100a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:09.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.064+0000 7f422c18a640 1 --2- 172.21.15.23:0/2994381987 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f42100a2230 0x7f42100a46f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4218009530 tx=0x7f4218009260 comp rx=0 tx=0).stop 2024-03-24T10:51:09.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.064+0000 7f422c18a640 1 -- 172.21.15.23:0/2994381987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4224106660 msgr2=0x7f422410e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:09.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.064+0000 7f422c18a640 1 --2- 172.21.15.23:0/2994381987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4224106660 0x7f422410e8e0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f421c036040 tx=0x7f421c02cb70 comp rx=0 tx=0).stop 2024-03-24T10:51:09.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.064+0000 7f422c18a640 1 -- 172.21.15.23:0/2994381987 shutdown_connections 2024-03-24T10:51:09.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.064+0000 7f422c18a640 1 --2- 172.21.15.23:0/2994381987 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f42100a2230 0x7f42100a46f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:09.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.064+0000 7f422c18a640 1 --2- 172.21.15.23:0/2994381987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4224106660 0x7f422410e8e0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:09.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.064+0000 7f422c18a640 1 --2- 172.21.15.23:0/2994381987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4224100f80 0x7f422410e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:09.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.064+0000 7f422c18a640 1 --2- 172.21.15.23:0/2994381987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42241005c0 0x7f422410de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:09.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.064+0000 7f422c18a640 1 -- 172.21.15.23:0/2994381987 >> 172.21.15.23:0/2994381987 conn(0x7f4224076990 msgr2=0x7f42240fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:09.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.064+0000 7f422c18a640 1 -- 172.21.15.23:0/2994381987 shutdown_connections 2024-03-24T10:51:09.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.064+0000 7f422c18a640 1 -- 172.21.15.23:0/2994381987 wait complete. 2024-03-24T10:51:09.075 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T10:51:09.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.180+0000 7fd4b9074640 1 -- 172.21.15.23:0/3775800689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4b4107760 msgr2=0x7fd4b4107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:09.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.180+0000 7fd4b9074640 1 --2- 172.21.15.23:0/3775800689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4b4107760 0x7fd4b4107b40 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac009930 tx=0x7fd4ac02bc30 comp rx=0 tx=0).stop 2024-03-24T10:51:09.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.180+0000 7fd4b9074640 1 -- 172.21.15.23:0/3775800689 shutdown_connections 2024-03-24T10:51:09.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.180+0000 7fd4b9074640 1 --2- 172.21.15.23:0/3775800689 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b40692d0 0x7fd4b410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:09.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.180+0000 7fd4b9074640 1 --2- 172.21.15.23:0/3775800689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b4068910 0x7fd4b4068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:09.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.180+0000 7fd4b9074640 1 --2- 172.21.15.23:0/3775800689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4b4107760 0x7fd4b4107b40 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:09.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.180+0000 7fd4b9074640 1 -- 172.21.15.23:0/3775800689 >> 172.21.15.23:0/3775800689 conn(0x7fd4b4077710 msgr2=0x7fd4b4077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:09.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.180+0000 7fd4b9074640 1 -- 172.21.15.23:0/3775800689 shutdown_connections 2024-03-24T10:51:09.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.180+0000 7fd4b9074640 1 -- 172.21.15.23:0/3775800689 wait complete. 2024-03-24T10:51:09.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.181+0000 7fd4b9074640 1 Processor -- start 2024-03-24T10:51:09.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.181+0000 7fd4b9074640 1 -- start start 2024-03-24T10:51:09.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.181+0000 7fd4b9074640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b4068910 0x7fd4b419d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:09.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.181+0000 7fd4b9074640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4b40692d0 0x7fd4b419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:09.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b9074640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4107760 0x7fd4b4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:09.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b9074640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b41980e0 con 0x7fd4b40692d0 2024-03-24T10:51:09.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b9074640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b4198250 con 0x7fd4b4068910 2024-03-24T10:51:09.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b9074640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b4198410 con 0x7fd4b4107760 2024-03-24T10:51:09.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4b40692d0 0x7fd4b419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:09.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4b40692d0 0x7fd4b419de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54930/0 (socket says 172.21.15.23:54930) 2024-03-24T10:51:09.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b259c640 1 -- 172.21.15.23:0/1909665918 learned_addr learned my addr 172.21.15.23:0/1909665918 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:51:09.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b359e640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4107760 0x7fd4b4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:09.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b2d9d640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b4068910 0x7fd4b419d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:09.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b259c640 1 -- 172.21.15.23:0/1909665918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4107760 msgr2=0x7fd4b4197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:09.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b259c640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4107760 0x7fd4b4197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:09.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b259c640 1 -- 172.21.15.23:0/1909665918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b4068910 msgr2=0x7fd4b419d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:09.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b259c640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b4068910 0x7fd4b419d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:09.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b259c640 1 -- 172.21.15.23:0/1909665918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4ac009560 con 0x7fd4b40692d0 2024-03-24T10:51:09.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b359e640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4107760 0x7fd4b4197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:51:09.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b259c640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4b40692d0 0x7fd4b419de20 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fd4a800c920 tx=0x7fd4a800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:09.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd49bfff640 1 -- 172.21.15.23:0/1909665918 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8007c70 con 0x7fd4b40692d0 2024-03-24T10:51:09.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd49bfff640 1 -- 172.21.15.23:0/1909665918 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4a8007dd0 con 0x7fd4b40692d0 2024-03-24T10:51:09.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b9074640 1 -- 172.21.15.23:0/1909665918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4b41986f0 con 0x7fd4b40692d0 2024-03-24T10:51:09.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd49bfff640 1 -- 172.21.15.23:0/1909665918 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8018860 con 0x7fd4b40692d0 2024-03-24T10:51:09.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.182+0000 7fd4b9074640 1 -- 172.21.15.23:0/1909665918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4b41a4980 con 0x7fd4b40692d0 2024-03-24T10:51:09.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.184+0000 7fd4b9074640 1 -- 172.21.15.23:0/1909665918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4b4068500 con 0x7fd4b40692d0 2024-03-24T10:51:09.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.185+0000 7fd49bfff640 1 -- 172.21.15.23:0/1909665918 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7fd4a800fac0 con 0x7fd4b40692d0 2024-03-24T10:51:09.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.185+0000 7fd49bfff640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd49c0a2230 0x7fd49c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:09.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.185+0000 7fd4b2d9d640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd49c0a2230 0x7fd49c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:09.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.186+0000 7fd49bfff640 1 -- 172.21.15.23:0/1909665918 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8785+0+0 (secure 0 0 0) 0x7fd4a80ce320 con 0x7fd4b40692d0 2024-03-24T10:51:09.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.186+0000 7fd4b2d9d640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd49c0a2230 0x7fd49c0a46f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac009900 tx=0x7fd4ac005fd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:09.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.190+0000 7fd49bfff640 1 -- 172.21.15.23:0/1909665918 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd4a8096ea0 con 0x7fd4b40692d0 2024-03-24T10:51:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:08 smithi117 ceph-mon[36368]: pgmap v344: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:51:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:08 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4284742185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:51:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:08 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4159510316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:51:09.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:08 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4170858016' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:51:09.253 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71561]: ignoring --setuser ceph since I am not root 2024-03-24T10:51:09.254 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71561]: ignoring --setgroup ceph since I am not root 2024-03-24T10:51:09.254 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-mds[71565]: -- 172.21.15.117:0/1868972804 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5590335f61a0 con 0x5590335cd000 2024-03-24T10:51:09.254 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71561]: starting mds.c at 2024-03-24T10:51:09.254 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-f[66812]: ignoring --setuser ceph since I am not root 2024-03-24T10:51:09.254 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-f[66812]: ignoring --setgroup ceph since I am not root 2024-03-24T10:51:09.255 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-mds[66816]: -- 172.21.15.117:0/3776488405 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56488f6f01a0 con 0x56488f6c6c00 2024-03-24T10:51:09.255 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-f[66812]: starting mds.f at 2024-03-24T10:51:09.255 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-i[67442]: ignoring --setuser ceph since I am not root 2024-03-24T10:51:09.255 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-i[67442]: ignoring --setgroup ceph since I am not root 2024-03-24T10:51:09.255 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-mds[67446]: -- 172.21.15.117:0/2943559346 <== mon.1 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b2539381a0 con 0x55b25390ec00 2024-03-24T10:51:09.255 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-i[67442]: starting mds.i at 2024-03-24T10:51:09.255 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-i[67442]: 2024-03-24T10:51:09.105+0000 7f26f7585700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6835/0: (98) Address already in use 2024-03-24T10:51:09.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:08 smithi116 ceph-mon[36248]: pgmap v344: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-24T10:51:09.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:08 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4284742185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:51:09.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:08 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4159510316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T10:51:09.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:08 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4170858016' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T10:51:09.268 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-h[66318]: ignoring --setuser ceph since I am not root 2024-03-24T10:51:09.268 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-h[66318]: ignoring --setgroup ceph since I am not root 2024-03-24T10:51:09.268 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-h[66318]: starting mds.h at 2024-03-24T10:51:09.269 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-b[66641]: ignoring --setuser ceph since I am not root 2024-03-24T10:51:09.269 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-b[66641]: ignoring --setgroup ceph since I am not root 2024-03-24T10:51:09.269 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-mds[66645]: -- 172.21.15.116:0/412143963 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e1628a41a0 con 0x55e16287ac00 2024-03-24T10:51:09.269 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-b[66641]: starting mds.b at 2024-03-24T10:51:09.269 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-e[66007]: ignoring --setuser ceph since I am not root 2024-03-24T10:51:09.269 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-e[66007]: ignoring --setgroup ceph since I am not root 2024-03-24T10:51:09.269 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-mds[66011]: -- 172.21.15.116:0/1562575430 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55756d9041a0 con 0x55756d8dac00 2024-03-24T10:51:09.270 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-e[66007]: starting mds.e at 2024-03-24T10:51:09.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:09.415+0000 7fd4b9074640 1 -- 172.21.15.23:0/1909665918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd4b4111990 con 0x7fd4b40692d0 2024-03-24T10:51:09.417 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:09 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a[32462]: 2024-03-24T10:51:09.060+0000 7f0006ad2700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T10:51:10.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.069+0000 7fd49bfff640 1 -- 172.21.15.23:0/1909665918 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) ==== 104+0+0 (secure 0 0 0) 0x7fd4a809a560 con 0x7fd4b40692d0 2024-03-24T10:51:10.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.072+0000 7fd4b9074640 1 -- 172.21.15.23:0/1909665918 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd49c0a2230 msgr2=0x7fd49c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:10.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.072+0000 7fd4b9074640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd49c0a2230 0x7fd49c0a46f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac009900 tx=0x7fd4ac005fd0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.072+0000 7fd4b9074640 1 -- 172.21.15.23:0/1909665918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4b40692d0 msgr2=0x7fd4b419de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:10.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.072+0000 7fd4b9074640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4b40692d0 0x7fd4b419de20 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fd4a800c920 tx=0x7fd4a800cde0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.072+0000 7fd4b9074640 1 -- 172.21.15.23:0/1909665918 shutdown_connections 2024-03-24T10:51:10.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.072+0000 7fd4b9074640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7fd49c0a2230 0x7fd49c0a46f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.072+0000 7fd4b9074640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4107760 0x7fd4b4197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.072+0000 7fd4b9074640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4b40692d0 0x7fd4b419de20 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.072+0000 7fd4b9074640 1 --2- 172.21.15.23:0/1909665918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4b4068910 0x7fd4b419d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.072+0000 7fd4b9074640 1 -- 172.21.15.23:0/1909665918 >> 172.21.15.23:0/1909665918 conn(0x7fd4b4077710 msgr2=0x7fd4b40fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:10.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.072+0000 7fd4b9074640 1 -- 172.21.15.23:0/1909665918 shutdown_connections 2024-03-24T10:51:10.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.072+0000 7fd4b9074640 1 -- 172.21.15.23:0/1909665918 wait complete. 2024-03-24T10:51:10.087 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T10:51:10.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.199+0000 7f072178a640 1 -- 172.21.15.23:0/8126789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c106670 msgr2=0x7f071c106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:10.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.199+0000 7f072178a640 1 --2- 172.21.15.23:0/8126789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c106670 0x7f071c106a50 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f0714009900 tx=0x7f071402bc50 comp rx=0 tx=0).stop 2024-03-24T10:51:10.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.199+0000 7f072178a640 1 -- 172.21.15.23:0/8126789 shutdown_connections 2024-03-24T10:51:10.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.199+0000 7f072178a640 1 --2- 172.21.15.23:0/8126789 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f071c100f70 0x7f071c10d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.199+0000 7f072178a640 1 --2- 172.21.15.23:0/8126789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f071c1005b0 0x7f071c100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.199+0000 7f072178a640 1 --2- 172.21.15.23:0/8126789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c106670 0x7f071c106a50 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.199+0000 7f072178a640 1 -- 172.21.15.23:0/8126789 >> 172.21.15.23:0/8126789 conn(0x7f071c0777c0 msgr2=0x7f071c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.199+0000 7f072178a640 1 -- 172.21.15.23:0/8126789 shutdown_connections 2024-03-24T10:51:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.199+0000 7f072178a640 1 -- 172.21.15.23:0/8126789 wait complete. 2024-03-24T10:51:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.200+0000 7f072178a640 1 Processor -- start 2024-03-24T10:51:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.200+0000 7f072178a640 1 -- start start 2024-03-24T10:51:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.200+0000 7f072178a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f071c1005b0 0x7f071c19d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.200+0000 7f072178a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c100f70 0x7f071c19dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:10.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.200+0000 7f072178a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f071c106670 0x7f071c1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.200+0000 7f072178a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f071c197ff0 con 0x7f071c100f70 2024-03-24T10:51:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.200+0000 7f072178a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f071c198160 con 0x7f071c1005b0 2024-03-24T10:51:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.200+0000 7f072178a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f071c198320 con 0x7f071c106670 2024-03-24T10:51:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f071a7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c100f70 0x7f071c19dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f071a7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c100f70 0x7f071c19dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54942/0 (socket says 172.21.15.23:54942) 2024-03-24T10:51:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f071a7fc640 1 -- 172.21.15.23:0/521257519 learned_addr learned my addr 172.21.15.23:0/521257519 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:51:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f071affd640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f071c1005b0 0x7f071c19d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f071b7fe640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f071c106670 0x7f071c1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f071a7fc640 1 -- 172.21.15.23:0/521257519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f071c106670 msgr2=0x7f071c1979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f071a7fc640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f071c106670 0x7f071c1979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f071a7fc640 1 -- 172.21.15.23:0/521257519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f071c1005b0 msgr2=0x7f071c19d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f071a7fc640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f071c1005b0 0x7f071c19d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f071a7fc640 1 -- 172.21.15.23:0/521257519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0714009560 con 0x7f071c100f70 2024-03-24T10:51:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f071affd640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f071c1005b0 0x7f071c19d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T10:51:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f071a7fc640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c100f70 0x7f071c19dd30 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f071000c920 tx=0x7f071000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f0703fff640 1 -- 172.21.15.23:0/521257519 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0710007c70 con 0x7f071c100f70 2024-03-24T10:51:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f072178a640 1 -- 172.21.15.23:0/521257519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071c198600 con 0x7f071c100f70 2024-03-24T10:51:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f0703fff640 1 -- 172.21.15.23:0/521257519 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0710007dd0 con 0x7f071c100f70 2024-03-24T10:51:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f0703fff640 1 -- 172.21.15.23:0/521257519 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0710018860 con 0x7f071c100f70 2024-03-24T10:51:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.201+0000 7f072178a640 1 -- 172.21.15.23:0/521257519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f071c1a4890 con 0x7f071c100f70 2024-03-24T10:51:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.202+0000 7f072178a640 1 -- 172.21.15.23:0/521257519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f071c068500 con 0x7f071c100f70 2024-03-24T10:51:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.203+0000 7f0703fff640 1 -- 172.21.15.23:0/521257519 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f071000fac0 con 0x7f071c100f70 2024-03-24T10:51:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.204+0000 7f0703fff640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f07040a2230 0x7f07040a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.204+0000 7f071affd640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f07040a2230 0x7f07040a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.204+0000 7f0703fff640 1 -- 172.21.15.23:0/521257519 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8785+0+0 (secure 0 0 0) 0x7f07100ce3a0 con 0x7f071c100f70 2024-03-24T10:51:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.205+0000 7f071affd640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f07040a2230 0x7f07040a46f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0714006010 tx=0x7f071402c5f0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:10.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.209+0000 7f0703fff640 1 -- 172.21.15.23:0/521257519 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0710096f20 con 0x7f071c100f70 2024-03-24T10:51:10.252 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2994381987' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T10:51:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-mon[36368]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T10:51:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-mon[36368]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T10:51:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/2994381987' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T10:51:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-mon[36368]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T10:51:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-mon[36368]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-24T10:51:10.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:09 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1909665918' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T10:51:10.267 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2994381987' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T10:51:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-mon[36248]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T10:51:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-mon[36248]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T10:51:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/2994381987' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T10:51:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-mon[36248]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T10:51:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-mon[36248]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-24T10:51:10.268 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:09 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1909665918' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T10:51:10.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:10.390+0000 7f072178a640 1 -- 172.21.15.23:0/521257519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f071c198ed0 con 0x7f071c100f70 2024-03-24T10:51:10.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:09 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2994381987' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T10:51:10.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:09 smithi023 ceph-mon[32466]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T10:51:10.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:09 smithi023 ceph-mon[32466]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T10:51:10.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:09 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/2994381987' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T10:51:10.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:09 smithi023 ceph-mon[32466]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T10:51:10.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:09 smithi023 ceph-mon[32466]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-24T10:51:10.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:09 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1909665918' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T10:51:10.969 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: pgmap v346: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-24T10:51:10.969 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T10:51:10.969 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T10:51:10.969 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: Cluster is now healthy 2024-03-24T10:51:10.969 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/1909665918' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T10:51:10.970 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.117:6836/1075673190,v1:172.21.15.117:6839/1075673190] up:boot 2024-03-24T10:51:10.970 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.117:6835/2542881495,v1:172.21.15.117:6838/2542881495] up:boot 2024-03-24T10:51:10.970 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.117:6834/7324734,v1:172.21.15.117:6837/7324734] up:boot 2024-03-24T10:51:10.970 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.116:6834/3185979398,v1:172.21.15.116:6835/3185979398] up:boot 2024-03-24T10:51:10.970 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.116:6836/3096976825,v1:172.21.15.116:6837/3096976825] up:boot 2024-03-24T10:51:10.970 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: mds.? [v2:172.21.15.116:6838/1616314084,v1:172.21.15.116:6839/1616314084] up:boot 2024-03-24T10:51:10.970 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: fsmap 12 up:standby 2024-03-24T10:51:10.970 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:51:10.971 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T10:51:10.971 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T10:51:10.971 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T10:51:10.971 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T10:51:10.971 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T10:51:10.971 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:10 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/521257519' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T10:51:11.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.071+0000 7f0703fff640 1 -- 172.21.15.23:0/521257519 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v100) ==== 181+0+0 (secure 0 0 0) 0x7f071009a5e0 con 0x7f071c100f70 2024-03-24T10:51:11.071 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_metadata' removed 2024-03-24T10:51:11.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.074+0000 7f072178a640 1 -- 172.21.15.23:0/521257519 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f07040a2230 msgr2=0x7f07040a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:11.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.074+0000 7f072178a640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f07040a2230 0x7f07040a46f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0714006010 tx=0x7f071402c5f0 comp rx=0 tx=0).stop 2024-03-24T10:51:11.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.074+0000 7f072178a640 1 -- 172.21.15.23:0/521257519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c100f70 msgr2=0x7f071c19dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:11.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.074+0000 7f072178a640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c100f70 0x7f071c19dd30 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f071000c920 tx=0x7f071000cde0 comp rx=0 tx=0).stop 2024-03-24T10:51:11.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.074+0000 7f072178a640 1 -- 172.21.15.23:0/521257519 shutdown_connections 2024-03-24T10:51:11.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.074+0000 7f072178a640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f07040a2230 0x7f07040a46f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:11.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.074+0000 7f072178a640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f071c106670 0x7f071c1979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:11.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.074+0000 7f072178a640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c100f70 0x7f071c19dd30 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:11.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.074+0000 7f072178a640 1 --2- 172.21.15.23:0/521257519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f071c1005b0 0x7f071c19d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:11.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.074+0000 7f072178a640 1 -- 172.21.15.23:0/521257519 >> 172.21.15.23:0/521257519 conn(0x7f071c0777c0 msgr2=0x7f071c0ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:11.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.075+0000 7f072178a640 1 -- 172.21.15.23:0/521257519 shutdown_connections 2024-03-24T10:51:11.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.075+0000 7f072178a640 1 -- 172.21.15.23:0/521257519 wait complete. 2024-03-24T10:51:11.091 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T10:51:11.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.207+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/3339585184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d28106660 msgr2=0x7f8d28106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:11.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.207+0000 7f8d2d3f4640 1 --2- 172.21.15.23:0/3339585184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d28106660 0x7f8d28106a40 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f8d20009900 tx=0x7f8d2002bc30 comp rx=0 tx=0).stop 2024-03-24T10:51:11.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.207+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/3339585184 shutdown_connections 2024-03-24T10:51:11.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.207+0000 7f8d2d3f4640 1 --2- 172.21.15.23:0/3339585184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d28100f60 0x7f8d2810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:11.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.207+0000 7f8d2d3f4640 1 --2- 172.21.15.23:0/3339585184 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d281005a0 0x7f8d28100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:11.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.207+0000 7f8d2d3f4640 1 --2- 172.21.15.23:0/3339585184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d28106660 0x7f8d28106a40 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:11.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.207+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/3339585184 >> 172.21.15.23:0/3339585184 conn(0x7f8d28077710 msgr2=0x7f8d28077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:11.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.208+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/3339585184 shutdown_connections 2024-03-24T10:51:11.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.208+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/3339585184 wait complete. 2024-03-24T10:51:11.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.208+0000 7f8d2d3f4640 1 Processor -- start 2024-03-24T10:51:11.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.208+0000 7f8d2d3f4640 1 -- start start 2024-03-24T10:51:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.208+0000 7f8d2d3f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d281005a0 0x7f8d2819d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.208+0000 7f8d2d3f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d28100f60 0x7f8d2819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.208+0000 7f8d2d3f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d28106660 0x7f8d28197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.208+0000 7f8d2d3f4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 -- 0x7f8d28198150 con 0x7f8d28100f60 2024-03-24T10:51:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d2d3f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8d281982c0 con 0x7f8d28106660 2024-03-24T10:51:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d2d3f4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8d28198480 con 0x7f8d281005a0 2024-03-24T10:51:11.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d267fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d28100f60 0x7f8d2819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:11.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d267fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d28100f60 0x7f8d2819de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54952/0 (socket says 172.21.15.23:54952) 2024-03-24T10:51:11.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d267fc640 1 -- 172.21.15.23:0/4260584199 learned_addr learned my addr 172.21.15.23:0/4260584199 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-24T10:51:11.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d277fe640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d28106660 0x7f8d28197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:11.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d26ffd640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d281005a0 0x7f8d2819d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:11.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d267fc640 1 -- 172.21.15.23:0/4260584199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d281005a0 msgr2=0x7f8d2819d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:11.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d267fc640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d281005a0 0x7f8d2819d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:11.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d267fc640 1 -- 172.21.15.23:0/4260584199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d28106660 msgr2=0x7f8d28197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:11.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d267fc640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d28106660 0x7f8d28197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:11.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d267fc640 1 -- 172.21.15.23:0/4260584199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d20009560 con 0x7f8d28100f60 2024-03-24T10:51:11.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d26ffd640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d281005a0 0x7f8d2819d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T10:51:11.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d267fc640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d28100f60 0x7f8d2819de40 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000c970 tx=0x7f8d1000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:11.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d1ffff640 1 -- 172.21.15.23:0/4260584199 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d10007bb0 con 0x7f8d28100f60 2024-03-24T10:51:11.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d1ffff640 1 -- 172.21.15.23:0/4260584199 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d10007d10 con 0x7f8d28100f60 2024-03-24T10:51:11.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/4260584199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d28198760 con 0x7f8d28100f60 2024-03-24T10:51:11.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d1ffff640 1 -- 172.21.15.23:0/4260584199 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1000f450 con 0x7f8d28100f60 2024-03-24T10:51:11.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.209+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/4260584199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d281a49a0 con 0x7f8d28100f60 2024-03-24T10:51:11.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.210+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/4260584199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d28068500 con 0x7f8d28100f60 2024-03-24T10:51:11.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.211+0000 7f8d1ffff640 1 -- 172.21.15.23:0/4260584199 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) ==== 136887+0+0 (secure 0 0 0) 0x7f8d10004f70 con 0x7f8d28100f60 2024-03-24T10:51:11.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.212+0000 7f8d1ffff640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8d140a2230 0x7f8d140a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:11.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.212+0000 7f8d26ffd640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8d140a2230 0x7f8d140a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:11.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.212+0000 7f8d1ffff640 1 -- 172.21.15.23:0/4260584199 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8231+0+0 (secure 0 0 0) 0x7f8d100cdcd0 con 0x7f8d28100f60 2024-03-24T10:51:11.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.212+0000 7f8d26ffd640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8d140a2230 0x7f8d140a46f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8d20036040 tx=0x7f8d20005fd0 comp rx=0 tx=0).ready entity=mgr.15288 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:11.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.216+0000 7f8d1ffff640 1 -- 172.21.15.23:0/4260584199 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8d10096a80 con 0x7f8d28100f60 2024-03-24T10:51:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: pgmap v346: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-24T10:51:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T10:51:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T10:51:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: Cluster is now healthy 2024-03-24T10:51:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/1909665918' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T10:51:11.253 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.117:6836/1075673190,v1:172.21.15.117:6839/1075673190] up:boot 2024-03-24T10:51:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.117:6835/2542881495,v1:172.21.15.117:6838/2542881495] up:boot 2024-03-24T10:51:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.117:6834/7324734,v1:172.21.15.117:6837/7324734] up:boot 2024-03-24T10:51:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.116:6834/3185979398,v1:172.21.15.116:6835/3185979398] up:boot 2024-03-24T10:51:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.116:6836/3096976825,v1:172.21.15.116:6837/3096976825] up:boot 2024-03-24T10:51:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: mds.? [v2:172.21.15.116:6838/1616314084,v1:172.21.15.116:6839/1616314084] up:boot 2024-03-24T10:51:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: fsmap 12 up:standby 2024-03-24T10:51:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:51:11.254 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T10:51:11.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T10:51:11.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T10:51:11.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T10:51:11.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T10:51:11.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:10 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/521257519' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T10:51:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:11.401+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/4260584199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8d28198fe0 con 0x7f8d28100f60 2024-03-24T10:51:11.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: pgmap v346: 97 pgs: 97 active+clean; 416 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-24T10:51:11.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T10:51:11.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T10:51:11.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: Cluster is now healthy 2024-03-24T10:51:11.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/1909665918' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T10:51:11.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.117:6836/1075673190,v1:172.21.15.117:6839/1075673190] up:boot 2024-03-24T10:51:11.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.117:6835/2542881495,v1:172.21.15.117:6838/2542881495] up:boot 2024-03-24T10:51:11.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.117:6834/7324734,v1:172.21.15.117:6837/7324734] up:boot 2024-03-24T10:51:11.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.116:6834/3185979398,v1:172.21.15.116:6835/3185979398] up:boot 2024-03-24T10:51:11.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.116:6836/3096976825,v1:172.21.15.116:6837/3096976825] up:boot 2024-03-24T10:51:11.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: mds.? [v2:172.21.15.116:6838/1616314084,v1:172.21.15.116:6839/1616314084] up:boot 2024-03-24T10:51:11.419 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: fsmap 12 up:standby 2024-03-24T10:51:11.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T10:51:11.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T10:51:11.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T10:51:11.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T10:51:11.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T10:51:11.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: from='mgr.15288 172.21.15.23:0/3016485405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T10:51:11.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:10 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/521257519' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T10:51:12.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:12.077+0000 7f8d1ffff640 1 -- 172.21.15.23:0/4260584199 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v101) ==== 169+0+0 (secure 0 0 0) 0x7f8d1009a140 con 0x7f8d28100f60 2024-03-24T10:51:12.077 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data' removed 2024-03-24T10:51:12.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:12.079+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/4260584199 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8d140a2230 msgr2=0x7f8d140a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:12.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:12.079+0000 7f8d2d3f4640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8d140a2230 0x7f8d140a46f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8d20036040 tx=0x7f8d20005fd0 comp rx=0 tx=0).stop 2024-03-24T10:51:12.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:12.079+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/4260584199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d28100f60 msgr2=0x7f8d2819de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:12.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:12.079+0000 7f8d2d3f4640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d28100f60 0x7f8d2819de40 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000c970 tx=0x7f8d1000ce30 comp rx=0 tx=0).stop 2024-03-24T10:51:12.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:12.079+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/4260584199 shutdown_connections 2024-03-24T10:51:12.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:12.079+0000 7f8d2d3f4640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.23:6800/581236369,v1:172.21.15.23:6801/581236369] conn(0x7f8d140a2230 0x7f8d140a46f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:12.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:12.079+0000 7f8d2d3f4640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d28106660 0x7f8d28197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:12.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:12.079+0000 7f8d2d3f4640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d28100f60 0x7f8d2819de40 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:12.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:12.079+0000 7f8d2d3f4640 1 --2- 172.21.15.23:0/4260584199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d281005a0 0x7f8d2819d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:12.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:12.079+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/4260584199 >> 172.21.15.23:0/4260584199 conn(0x7f8d28077710 msgr2=0x7f8d280ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:12.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:12.079+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/4260584199 shutdown_connections 2024-03-24T10:51:12.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-24T10:51:12.080+0000 7f8d2d3f4640 1 -- 172.21.15.23:0/4260584199 wait complete. 2024-03-24T10:51:12.097 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T10:51:12.126 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T10:51:12.159 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T10:51:12.208 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T10:51:12.208 DEBUG:teuthology.orchestra.run.smithi023:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T10:51:12.224 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T10:51:12.238 DEBUG:teuthology.orchestra.run.smithi117:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T10:51:12.257 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T10:51:12.258 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-24T10:51:12.258 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mon.a 2024-03-24T10:51:12.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:12 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/521257519' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T10:51:12.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:12 smithi023 ceph-mon[32466]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T10:51:12.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:12 smithi023 ceph-mon[32466]: from='client.? 172.21.15.23:0/4260584199' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T10:51:12.418 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:12 smithi023 systemd[1]: Stopping Ceph mon.a for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:51:12.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:12 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/521257519' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T10:51:12.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:12 smithi117 ceph-mon[36368]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T10:51:12.503 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:12 smithi117 ceph-mon[36368]: from='client.? 172.21.15.23:0/4260584199' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T10:51:12.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:12 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/521257519' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T10:51:12.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:12 smithi116 ceph-mon[36248]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T10:51:12.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:12 smithi116 ceph-mon[36248]: from='client.? 172.21.15.23:0/4260584199' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T10:51:12.834 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:12 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a[32462]: 2024-03-24T10:51:12.549+0000 7f000c2dd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:51:12.834 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:12 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a[32462]: 2024-03-24T10:51:12.549+0000 7f000c2dd700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T10:51:12.835 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:12 smithi023 podman[98492]: 2024-03-24 10:51:12.591377669 +0000 UTC m=+0.141289248 container died fc7c0651aa5d9f239e23074707cb733c8310e78c6cea809c0babdd5f5d7ac629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:51:13.101 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:12 smithi023 podman[98492]: 2024-03-24 10:51:12.925815951 +0000 UTC m=+0.475727530 container cleanup fc7c0651aa5d9f239e23074707cb733c8310e78c6cea809c0babdd5f5d7ac629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-24T10:51:13.101 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:12 smithi023 bash[98492]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a 2024-03-24T10:51:13.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 24 10:51:13 smithi023 podman[98506]: 2024-03-24 10:51:13.10187645 +0000 UTC m=+0.511636420 container remove fc7c0651aa5d9f239e23074707cb733c8310e78c6cea809c0babdd5f5d7ac629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T10:51:13.354 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mon.a.service' 2024-03-24T10:51:13.405 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:51:13.405 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-24T10:51:13.405 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-24T10:51:13.405 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mon.b 2024-03-24T10:51:13.695 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:13 smithi116 systemd[1]: Stopping Ceph mon.b for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:51:14.017 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:13 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-b[36244]: 2024-03-24T10:51:13.694+0000 7fe56fee8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:51:14.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:13 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-b[36244]: 2024-03-24T10:51:13.694+0000 7fe56fee8700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-24T10:51:14.018 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:13 smithi116 podman[72321]: 2024-03-24 10:51:13.766351225 +0000 UTC m=+0.165210024 container died c9b7a171dc13f415698635351432a1892e06b7e154809f17150bcdcea2e56ee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T10:51:14.328 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:14 smithi116 podman[72321]: 2024-03-24 10:51:14.111077155 +0000 UTC m=+0.509935960 container cleanup c9b7a171dc13f415698635351432a1892e06b7e154809f17150bcdcea2e56ee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T10:51:14.328 INFO:journalctl@ceph.mon.b.smithi116.stdout:Mar 24 10:51:14 smithi116 bash[72321]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-b 2024-03-24T10:51:14.532 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mon.b.service' 2024-03-24T10:51:14.589 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:51:14.589 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-24T10:51:14.590 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-24T10:51:14.590 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mon.c 2024-03-24T10:51:15.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:14 smithi117 systemd[1]: Stopping Ceph mon.c for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:51:15.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:14 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-c[36364]: 2024-03-24T10:51:14.865+0000 7f1bf58df700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:51:15.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:14 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-c[36364]: 2024-03-24T10:51:14.865+0000 7f1bf58df700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T10:51:15.383 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:15 smithi117 podman[73205]: 2024-03-24 10:51:15.090838662 +0000 UTC m=+0.303789216 container died 08756fdf33ea003f76ab7a05a3dc11a54c7643ca45220adbb4369f6a326c9f78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-24T10:51:15.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:15 smithi117 podman[73205]: 2024-03-24 10:51:15.482602434 +0000 UTC m=+0.695552970 container cleanup 08756fdf33ea003f76ab7a05a3dc11a54c7643ca45220adbb4369f6a326c9f78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T10:51:15.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Mar 24 10:51:15 smithi117 bash[73205]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mon-c 2024-03-24T10:51:15.963 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mon.c.service' 2024-03-24T10:51:16.009 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:51:16.009 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-24T10:51:16.010 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-24T10:51:16.010 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mgr.x 2024-03-24T10:51:16.330 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:51:16 smithi023 systemd[1]: Stopping Ceph mgr.x for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:51:16.647 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:51:16 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x[32703]: 2024-03-24T10:51:16.329+0000 7f5ce91f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:51:16.647 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:51:16 smithi023 podman[98702]: 2024-03-24 10:51:16.360567047 +0000 UTC m=+0.156862495 container died 52b2cf3da6ce9c07188ce6370346ae54541ec08c62de1476e5c509fbc4bea37e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T10:51:16.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:51:16 smithi023 podman[98702]: 2024-03-24 10:51:16.747100767 +0000 UTC m=+0.543396214 container cleanup 52b2cf3da6ce9c07188ce6370346ae54541ec08c62de1476e5c509fbc4bea37e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-24T10:51:16.918 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 24 10:51:16 smithi023 bash[98702]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-x 2024-03-24T10:51:17.106 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-24T10:51:17.143 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:51:17.143 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-24T10:51:17.143 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-24T10:51:17.143 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mgr.y 2024-03-24T10:51:17.460 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:51:17 smithi116 systemd[1]: Stopping Ceph mgr.y for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:51:17.767 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:51:17 smithi116 podman[72444]: 2024-03-24 10:51:17.460964209 +0000 UTC m=+0.128229886 container died b69d43a495ff0e5fe9ca439b9c853074e65639d771038b300202b26358717eba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T10:51:18.159 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:51:17 smithi116 podman[72444]: 2024-03-24 10:51:17.806361726 +0000 UTC m=+0.473627388 container cleanup b69d43a495ff0e5fe9ca439b9c853074e65639d771038b300202b26358717eba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:51:18.159 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:51:17 smithi116 bash[72444]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y 2024-03-24T10:51:18.159 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:51:17 smithi116 podman[72457]: 2024-03-24 10:51:17.957486601 +0000 UTC m=+0.492727398 container remove b69d43a495ff0e5fe9ca439b9c853074e65639d771038b300202b26358717eba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-24T10:51:18.160 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Mar 24 10:51:17 smithi116 systemd[1]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-24T10:51:18.174 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-24T10:51:18.211 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:51:18.211 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-24T10:51:18.211 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-24T10:51:18.211 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mgr.z 2024-03-24T10:51:18.502 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:51:18 smithi117 systemd[1]: Stopping Ceph mgr.z for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:51:18.810 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:51:18 smithi117 podman[73328]: 2024-03-24 10:51:18.542002386 +0000 UTC m=+0.143750859 container died 574d0e6e20f704c9a979dc96f821e7c547a7603a41f8b711ff9f3ae2960b3b65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T10:51:19.062 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:51:18 smithi117 podman[73328]: 2024-03-24 10:51:18.811547202 +0000 UTC m=+0.413295669 container cleanup 574d0e6e20f704c9a979dc96f821e7c547a7603a41f8b711ff9f3ae2960b3b65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T10:51:19.062 INFO:journalctl@ceph.mgr.z.smithi117.stdout:Mar 24 10:51:18 smithi117 bash[73328]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mgr-z 2024-03-24T10:51:19.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mgr.z.service' 2024-03-24T10:51:19.259 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:51:19.259 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-24T10:51:19.259 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T10:51:19.259 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.0 2024-03-24T10:51:19.551 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:51:19 smithi023 systemd[1]: Stopping Ceph osd.0 for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:51:19.917 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:51:19 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-0[45050]: 2024-03-24T10:51:19.551+0000 7f9646a76700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:51:19.918 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:51:19 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-0[45050]: 2024-03-24T10:51:19.551+0000 7f9646a76700 -1 osd.0 101 *** Got signal Terminated *** 2024-03-24T10:51:19.918 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:51:19 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-0[45050]: 2024-03-24T10:51:19.551+0000 7f9646a76700 -1 osd.0 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:51:25.168 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:51:24 smithi023 podman[98823]: 2024-03-24 10:51:24.910549833 +0000 UTC m=+5.457414518 container died a71253101cd55cfa9074c5b79a7abad24a03c03c2953587d353d7056a58f8860 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-24T10:51:25.668 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:51:25 smithi023 podman[98823]: 2024-03-24 10:51:25.468632017 +0000 UTC m=+6.015496695 container cleanup a71253101cd55cfa9074c5b79a7abad24a03c03c2953587d353d7056a58f8860 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T10:51:25.668 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:51:25 smithi023 bash[98823]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-0 2024-03-24T10:51:26.085 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:51:25 smithi023 podman[98840]: 2024-03-24 10:51:25.736562492 +0000 UTC m=+0.827834874 container remove a71253101cd55cfa9074c5b79a7abad24a03c03c2953587d353d7056a58f8860 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-24T10:51:26.418 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:51:26 smithi023 podman[98928]: 2024-03-24 10:51:25.988693532 +0000 UTC m=+0.018796904 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:51:26.418 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:51:26 smithi023 podman[98928]: 2024-03-24 10:51:26.112653469 +0000 UTC m=+0.142756831 container create 20039e53f2ff78958fbe2bb41db2ab5dcd8d53b5f8ed2a73948a792765431bc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-0-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T10:51:26.774 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:51:26 smithi023 podman[98928]: 2024-03-24 10:51:26.474333389 +0000 UTC m=+0.504436754 container init 20039e53f2ff78958fbe2bb41db2ab5dcd8d53b5f8ed2a73948a792765431bc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319) 2024-03-24T10:51:26.774 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:51:26 smithi023 podman[98928]: 2024-03-24 10:51:26.477905664 +0000 UTC m=+0.508009035 container start 20039e53f2ff78958fbe2bb41db2ab5dcd8d53b5f8ed2a73948a792765431bc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-24T10:51:26.775 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:51:26 smithi023 podman[98928]: 2024-03-24 10:51:26.56533893 +0000 UTC m=+0.595442330 container attach 20039e53f2ff78958fbe2bb41db2ab5dcd8d53b5f8ed2a73948a792765431bc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T10:51:27.067 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 24 10:51:26 smithi023 podman[98928]: 2024-03-24 10:51:26.775270142 +0000 UTC m=+0.805373511 container died 20039e53f2ff78958fbe2bb41db2ab5dcd8d53b5f8ed2a73948a792765431bc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux ) 2024-03-24T10:51:27.336 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T10:51:27.379 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:51:27.379 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T10:51:27.380 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T10:51:27.380 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.1 2024-03-24T10:51:27.676 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:51:27 smithi116 systemd[1]: Stopping Ceph osd.1 for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:51:28.017 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:51:27 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-1[40858]: 2024-03-24T10:51:27.676+0000 7f8dd9dbd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:51:28.018 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:51:27 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-1[40858]: 2024-03-24T10:51:27.676+0000 7f8dd9dbd700 -1 osd.1 101 *** Got signal Terminated *** 2024-03-24T10:51:28.018 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:51:27 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-1[40858]: 2024-03-24T10:51:27.676+0000 7f8dd9dbd700 -1 osd.1 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:51:33.517 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:51:33 smithi116 podman[72579]: 2024-03-24 10:51:33.241287198 +0000 UTC m=+5.673146156 container died 9424386b0c33b7c01fee65ddbc4b11beeb4099f341a1c7c0cb7a99fb4a7b39fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-1, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-24T10:51:33.919 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:51:33 smithi116 podman[72579]: 2024-03-24 10:51:33.610354335 +0000 UTC m=+6.042213300 container cleanup 9424386b0c33b7c01fee65ddbc4b11beeb4099f341a1c7c0cb7a99fb4a7b39fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-1, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-24T10:51:33.919 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:51:33 smithi116 bash[72579]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-1 2024-03-24T10:51:34.267 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:51:33 smithi116 podman[72595]: 2024-03-24 10:51:33.91997654 +0000 UTC m=+0.674903085 container remove 9424386b0c33b7c01fee65ddbc4b11beeb4099f341a1c7c0cb7a99fb4a7b39fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T10:51:34.564 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:51:34 smithi116 podman[72687]: 2024-03-24 10:51:34.173981797 +0000 UTC m=+0.017885650 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:51:34.564 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:51:34 smithi116 podman[72687]: 2024-03-24 10:51:34.27983709 +0000 UTC m=+0.123740963 container create 7b8a974394cf1f1d90e9c950b855edd52dd5a4a0684180e282784de6e27a7e54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:51:34.565 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:51:34 smithi116 podman[72687]: 2024-03-24 10:51:34.490096754 +0000 UTC m=+0.334000610 container init 7b8a974394cf1f1d90e9c950b855edd52dd5a4a0684180e282784de6e27a7e54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T10:51:34.565 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:51:34 smithi116 podman[72687]: 2024-03-24 10:51:34.493111736 +0000 UTC m=+0.337015591 container start 7b8a974394cf1f1d90e9c950b855edd52dd5a4a0684180e282784de6e27a7e54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-24T10:51:34.565 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:51:34 smithi116 podman[72687]: 2024-03-24 10:51:34.564949723 +0000 UTC m=+0.408853591 container attach 7b8a974394cf1f1d90e9c950b855edd52dd5a4a0684180e282784de6e27a7e54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True) 2024-03-24T10:51:35.018 INFO:journalctl@ceph.osd.1.smithi116.stdout:Mar 24 10:51:34 smithi116 podman[72687]: 2024-03-24 10:51:34.779490576 +0000 UTC m=+0.623394433 container died 7b8a974394cf1f1d90e9c950b855edd52dd5a4a0684180e282784de6e27a7e54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:51:35.417 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T10:51:35.459 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:51:35.459 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T10:51:35.460 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T10:51:35.460 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.2 2024-03-24T10:51:35.753 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:51:35 smithi117 systemd[1]: Stopping Ceph osd.2 for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:51:36.253 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:51:35 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-2[41648]: 2024-03-24T10:51:35.763+0000 7fd358baa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:51:36.253 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:51:35 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-2[41648]: 2024-03-24T10:51:35.763+0000 7fd358baa700 -1 osd.2 101 *** Got signal Terminated *** 2024-03-24T10:51:36.253 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:51:35 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-2[41648]: 2024-03-24T10:51:35.763+0000 7fd358baa700 -1 osd.2 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:51:41.578 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:51:41 smithi117 podman[73452]: 2024-03-24 10:51:41.261041913 +0000 UTC m=+5.608175952 container died 4e9e5e415fa392d05bf10ef63d83d6b5ab6b17923cdd1a6d5e06133c605d9b7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T10:51:42.002 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:51:41 smithi117 podman[73452]: 2024-03-24 10:51:41.579199645 +0000 UTC m=+5.926333672 container cleanup 4e9e5e415fa392d05bf10ef63d83d6b5ab6b17923cdd1a6d5e06133c605d9b7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T10:51:42.003 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:51:41 smithi117 bash[73452]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-2 2024-03-24T10:51:42.003 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:51:41 smithi117 podman[73482]: 2024-03-24 10:51:41.789001159 +0000 UTC m=+0.523461459 container remove 4e9e5e415fa392d05bf10ef63d83d6b5ab6b17923cdd1a6d5e06133c605d9b7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T10:51:42.392 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:51:42 smithi117 podman[73571]: 2024-03-24 10:51:42.106332966 +0000 UTC m=+0.103664259 container create e0a011b94d646ca1e28998cd736d14f91e059730c9a6035c4b07eeb905812e0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-2-deactivate, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T10:51:42.392 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:51:42 smithi117 podman[73571]: 2024-03-24 10:51:42.020466861 +0000 UTC m=+0.017798157 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:51:42.392 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:51:42 smithi117 podman[73571]: 2024-03-24 10:51:42.317001289 +0000 UTC m=+0.314332592 container init e0a011b94d646ca1e28998cd736d14f91e059730c9a6035c4b07eeb905812e0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-24T10:51:42.392 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:51:42 smithi117 podman[73571]: 2024-03-24 10:51:42.320135414 +0000 UTC m=+0.317466709 container start e0a011b94d646ca1e28998cd736d14f91e059730c9a6035c4b07eeb905812e0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T10:51:42.392 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:51:42 smithi117 podman[73571]: 2024-03-24 10:51:42.391829541 +0000 UTC m=+0.389160842 container attach e0a011b94d646ca1e28998cd736d14f91e059730c9a6035c4b07eeb905812e0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-24T10:51:42.753 INFO:journalctl@ceph.osd.2.smithi117.stdout:Mar 24 10:51:42 smithi117 podman[73571]: 2024-03-24 10:51:42.609090891 +0000 UTC m=+0.606422202 container died e0a011b94d646ca1e28998cd736d14f91e059730c9a6035c4b07eeb905812e0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:51:43.259 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T10:51:43.302 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:51:43.303 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T10:51:43.303 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-24T10:51:43.303 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.3 2024-03-24T10:51:43.595 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:43 smithi023 systemd[1]: Stopping Ceph osd.3 for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:51:43.870 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:43 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3[51619]: 2024-03-24T10:51:43.595+0000 7f4fdb302700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:51:43.870 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:43 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3[51619]: 2024-03-24T10:51:43.595+0000 7f4fdb302700 -1 osd.3 101 *** Got signal Terminated *** 2024-03-24T10:51:43.871 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:43 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3[51619]: 2024-03-24T10:51:43.595+0000 7f4fdb302700 -1 osd.3 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:51:45.503 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:45 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:45.054+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:45.668 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:45 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:45.350+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:46.502 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:46 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:46.040+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:46.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:46 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:46.229+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:51:46.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:46 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:46.020+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:51:46.668 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:46 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:46.351+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:47.502 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:47 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:47.059+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:47.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:47 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:47.241+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:51:47.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:47 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:47.068+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:51:47.668 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:47 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:47.308+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:48.503 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:48 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:48.030+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:48.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:48 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:48.240+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:51:48.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:48 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:48.051+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:51:48.533 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:48 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:48.282+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:48.918 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:48 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3[51619]: 2024-03-24T10:51:48.533+0000 7f4fd80f7700 -1 osd.3 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.280617+0000 front 2024-03-24T10:51:22.280662+0000 (oldest deadline 2024-03-24T10:51:48.180830+0000) 2024-03-24T10:51:49.274 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:49 smithi023 podman[99035]: 2024-03-24 10:51:49.130981065 +0000 UTC m=+5.635436528 container died caadd884494e44b3de7d3feee1811d7b2ad1bdc6e96bf668a3b279dbd7d3651d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-24T10:51:49.275 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:48 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:48.923+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:51:49.502 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:49 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:49.067+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:49.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:49 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:49.226+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:51:49.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:49 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:49.092+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:51:49.646 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:49 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:49.275+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:49.646 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:49 smithi023 podman[99035]: 2024-03-24 10:51:49.479271012 +0000 UTC m=+5.983726462 container cleanup caadd884494e44b3de7d3feee1811d7b2ad1bdc6e96bf668a3b279dbd7d3651d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T10:51:49.647 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:49 smithi023 bash[99035]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3 2024-03-24T10:51:49.918 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:49 smithi023 podman[99065]: 2024-03-24 10:51:49.646997246 +0000 UTC m=+0.510248038 container remove caadd884494e44b3de7d3feee1811d7b2ad1bdc6e96bf668a3b279dbd7d3651d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T10:51:49.918 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:49 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:49.890+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:51:50.312 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:50 smithi023 podman[99156]: 2024-03-24 10:51:50.047979434 +0000 UTC m=+0.108035484 container create 06cd4ee6593faacaa964d358b3d1ebca7cf5371489ef199029779ef3a8606b87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T10:51:50.312 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:50 smithi023 podman[99156]: 2024-03-24 10:51:49.958372952 +0000 UTC m=+0.018429030 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:51:50.312 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:50 smithi023 podman[99156]: 2024-03-24 10:51:50.283814497 +0000 UTC m=+0.343870565 container init 06cd4ee6593faacaa964d358b3d1ebca7cf5371489ef199029779ef3a8606b87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-24T10:51:50.312 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:50 smithi023 podman[99156]: 2024-03-24 10:51:50.287068457 +0000 UTC m=+0.347124511 container start 06cd4ee6593faacaa964d358b3d1ebca7cf5371489ef199029779ef3a8606b87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319) 2024-03-24T10:51:50.502 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:50 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:50.060+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:50.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:50 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:50.218+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:51:50.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:50 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:50.067+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:51:50.579 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:50 smithi023 podman[99156]: 2024-03-24 10:51:50.367540047 +0000 UTC m=+0.427596130 container attach 06cd4ee6593faacaa964d358b3d1ebca7cf5371489ef199029779ef3a8606b87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T10:51:50.579 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:50 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:50.311+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:50.861 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 24 10:51:50 smithi023 podman[99156]: 2024-03-24 10:51:50.580428342 +0000 UTC m=+0.640484407 container died 06cd4ee6593faacaa964d358b3d1ebca7cf5371489ef199029779ef3a8606b87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T10:51:51.111 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:50 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:50.933+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:51:51.129 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.3.service' 2024-03-24T10:51:51.171 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:51:51.171 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-24T10:51:51.171 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-24T10:51:51.171 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.4 2024-03-24T10:51:51.418 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:51 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:51.358+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:51.461 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:51:51 smithi116 systemd[1]: Stopping Ceph osd.4 for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:51:51.462 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:51 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:51.022+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:51:51.502 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:51 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:51.048+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:51.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:51 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:51.225+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:51:51.767 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:51:51 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-4[47712]: 2024-03-24T10:51:51.461+0000 7f179a9c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:51:51.768 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:51:51 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-4[47712]: 2024-03-24T10:51:51.461+0000 7f179a9c0700 -1 osd.4 101 *** Got signal Terminated *** 2024-03-24T10:51:51.768 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:51:51 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-4[47712]: 2024-03-24T10:51:51.461+0000 7f179a9c0700 -1 osd.4 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:51:52.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:51 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:51.892+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:51:52.502 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:52 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:52.071+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:52.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:52 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:52.198+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:51:52.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:52 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:52.044+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:51:52.667 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:52.392+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:53.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:53 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:53.014+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:51:53.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:52 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:52.935+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:51:53.502 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:53 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:53.114+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:53.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:53 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:53.174+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:51:53.904 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:53 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:53.432+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:54.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:53 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:53.905+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:51:54.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:53 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:53.905+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:51:54.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:53 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:53.997+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:51:54.503 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:54 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:54.153+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:54.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:54 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:54.223+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:51:54.503 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:51:54 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:51:54.060+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:51:54.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:51:54 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:51:54.344+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:51:54.915 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:54 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:54.458+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:55.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:54 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:54.916+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:51:55.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:54 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:54.916+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:51:55.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:54 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:54.976+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:51:55.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:54 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:54.976+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:51:55.503 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:55 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:55.155+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:55.503 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:51:55 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:51:55.095+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:51:55.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:55 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:55.244+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:51:55.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:51:55 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:51:55.303+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:51:55.917 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:55 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:55.489+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:56.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:55 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:55.946+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:51:56.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:55 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:55.946+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:51:56.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:55 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:55.957+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:51:56.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:55 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:55.957+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:51:56.502 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:51:56 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:51:56.049+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:51:56.503 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:56 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:56.176+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:56.503 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:56 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:56.176+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:32.014747+0000 front 2024-03-24T10:51:32.014600+0000 (oldest deadline 2024-03-24T10:51:55.514634+0000) 2024-03-24T10:51:56.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:56 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:56.246+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:51:56.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:56 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:56.246+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:51:56.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:51:56 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:51:56.270+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:51:56.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:56 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:56.511+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:56.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:56 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:56.511+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:51:57.121 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:51:56 smithi116 podman[72792]: 2024-03-24 10:51:56.845015351 +0000 UTC m=+5.487704600 container died bbb20a66ceae3fd2f60c804a6ad414f4c15bd0903aae2a58929f2d61873405c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-4, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-24T10:51:57.121 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:56 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:56.947+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:51:57.122 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:56 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:56.947+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:51:57.274 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:51:57 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:51:57.022+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:51:57.274 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:57 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:57.133+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:57.274 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:57 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:57.133+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:32.014747+0000 front 2024-03-24T10:51:32.014600+0000 (oldest deadline 2024-03-24T10:51:55.514634+0000) 2024-03-24T10:51:57.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:56 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:56.936+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:51:57.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:56 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:56.936+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:51:57.455 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:51:57 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:51:57.315+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:51:57.455 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:51:57 smithi116 podman[72792]: 2024-03-24 10:51:57.221445624 +0000 UTC m=+5.864134877 container cleanup bbb20a66ceae3fd2f60c804a6ad414f4c15bd0903aae2a58929f2d61873405c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-4, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T10:51:57.455 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:51:57 smithi116 bash[72792]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-4 2024-03-24T10:51:57.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:57 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:57.274+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:51:57.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:57 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:57.274+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:51:57.764 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:51:57 smithi116 podman[72810]: 2024-03-24 10:51:57.455931813 +0000 UTC m=+0.612295431 container remove bbb20a66ceae3fd2f60c804a6ad414f4c15bd0903aae2a58929f2d61873405c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-4, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-24T10:51:57.914 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:57 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:57.541+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:57.915 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:57 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:57.541+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:51:58.017 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:57 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:57.925+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:51:58.018 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:57 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:57.925+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:51:58.018 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:51:57 smithi116 podman[72905]: 2024-03-24 10:51:57.765121025 +0000 UTC m=+0.104608710 container create 6ce7172b69e454d5cfd2e958833033ac82b1dbd24ee96f176be89d32c65dbdca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-24T10:51:58.018 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:51:57 smithi116 podman[72905]: 2024-03-24 10:51:57.678256379 +0000 UTC m=+0.017744083 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:51:58.018 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:51:57 smithi116 podman[72905]: 2024-03-24 10:51:57.959006435 +0000 UTC m=+0.298494139 container init 6ce7172b69e454d5cfd2e958833033ac82b1dbd24ee96f176be89d32c65dbdca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-4-deactivate, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-24T10:51:58.018 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:51:57 smithi116 podman[72905]: 2024-03-24 10:51:57.962007102 +0000 UTC m=+0.301494793 container start 6ce7172b69e454d5cfd2e958833033ac82b1dbd24ee96f176be89d32c65dbdca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:51:58.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:57 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:57.914+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:51:58.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:57 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:57.914+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:51:58.311 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:51:58 smithi116 podman[72905]: 2024-03-24 10:51:58.037819671 +0000 UTC m=+0.377307369 container attach 6ce7172b69e454d5cfd2e958833033ac82b1dbd24ee96f176be89d32c65dbdca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-24T10:51:58.311 INFO:journalctl@ceph.osd.4.smithi116.stdout:Mar 24 10:51:58 smithi116 podman[72905]: 2024-03-24 10:51:58.243744523 +0000 UTC m=+0.583232221 container died 6ce7172b69e454d5cfd2e958833033ac82b1dbd24ee96f176be89d32c65dbdca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:51:58.324 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:51:58 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:51:58.043+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:51:58.324 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:58 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:58.135+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:58.324 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:58 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:58.135+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:32.014747+0000 front 2024-03-24T10:51:32.014600+0000 (oldest deadline 2024-03-24T10:51:55.514634+0000) 2024-03-24T10:51:58.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:58 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:58.324+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:51:58.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:58 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:58.324+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:51:58.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:51:58 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:51:58.311+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:51:58.802 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.4.service' 2024-03-24T10:51:58.843 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:51:58.843 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-24T10:51:58.843 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T10:51:58.843 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.5 2024-03-24T10:51:58.905 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:58 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:58.508+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:58.905 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:58 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:58.508+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:51:59.137 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:58 smithi117 systemd[1]: Stopping Ceph osd.5 for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:51:59.138 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:51:59 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:51:59.052+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:51:59.167 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:58 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:58.904+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:51:59.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:58 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:58.904+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:51:59.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:58 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:58.906+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:51:59.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:58 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:58.906+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:51:59.502 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:59 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:59.371+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:51:59.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:51:59 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:51:59.371+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:51:59.503 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:59 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:59.137+0000 7f0503696700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:51:59.503 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:59 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:59.137+0000 7f0503696700 -1 osd.5 101 *** Got signal Terminated *** 2024-03-24T10:51:59.503 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:59 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:59.137+0000 7f0503696700 -1 osd.5 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:51:59.503 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:59 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:59.167+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:51:59.504 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:51:59 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:51:59.167+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:32.014747+0000 front 2024-03-24T10:51:32.014600+0000 (oldest deadline 2024-03-24T10:51:55.514634+0000) 2024-03-24T10:51:59.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:51:59 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:51:59.318+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:51:59.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:59 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:59.491+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:51:59.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:51:59 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:51:59.491+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:52:00.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:59 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:59.922+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:00.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:51:59 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:51:59.922+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:00.371 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:00 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:00.049+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:00.371 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:00 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:52:00.126+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:52:00.371 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:00 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:52:00.126+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:32.014747+0000 front 2024-03-24T10:51:32.014600+0000 (oldest deadline 2024-03-24T10:51:55.514634+0000) 2024-03-24T10:52:00.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:59 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:59.925+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:00.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:51:59 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:51:59.925+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:00.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:00 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:00.370+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:00.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:00 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:00.370+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:00.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:00 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:00.369+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:00.896 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:00 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:00.481+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:52:00.897 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:00 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:00.481+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:52:01.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:00 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:00.896+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:01.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:00 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:00.896+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:01.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:00 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:00.873+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:01.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:00 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:00.873+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:01.373 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:01 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:01.037+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:01.373 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:01 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:52:01.104+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:52:01.373 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:01 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:52:01.104+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:32.014747+0000 front 2024-03-24T10:51:32.014600+0000 (oldest deadline 2024-03-24T10:51:55.514634+0000) 2024-03-24T10:52:01.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:01 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:01.373+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:01.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:01 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:01.373+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:01.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:01 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:01.394+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:01.901 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:01 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:01.443+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:52:01.902 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:01 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:01.443+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:52:02.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:01 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:01.901+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:02.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:01 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:01.901+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:02.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:01 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:01.900+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:02.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:01 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:01.900+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:02.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:01 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:01.900+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:02.387 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:02 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:02.074+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:02.388 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:02 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:52:02.135+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:52:02.388 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:02 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:52:02.135+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:32.014747+0000 front 2024-03-24T10:51:32.014600+0000 (oldest deadline 2024-03-24T10:51:55.514634+0000) 2024-03-24T10:52:02.388 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:02 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:52:02.135+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:40.215564+0000 front 2024-03-24T10:51:40.215647+0000 (oldest deadline 2024-03-24T10:52:01.915732+0000) 2024-03-24T10:52:02.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:02 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:02.387+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:02.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:02 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:02.387+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:02.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:02 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:02.431+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:02.907 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:02 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:02.473+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:52:02.907 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:02 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:02.473+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:52:03.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:02 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:02.907+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:03.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:02 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:02.907+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:03.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:02 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:02.907+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:03.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:02 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:02.853+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:03.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:02 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:02.853+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:03.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:02 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:02.853+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:03.406 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:03 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:03.122+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:03.407 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:03 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:52:03.154+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:19.713192+0000 front 2024-03-24T10:51:19.713351+0000 (oldest deadline 2024-03-24T10:51:45.013262+0000) 2024-03-24T10:52:03.407 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:03 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:52:03.154+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:32.014747+0000 front 2024-03-24T10:51:32.014600+0000 (oldest deadline 2024-03-24T10:51:55.514634+0000) 2024-03-24T10:52:03.407 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:03 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5[48652]: 2024-03-24T10:52:03.154+0000 7f050048b700 -1 osd.5 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:40.215564+0000 front 2024-03-24T10:51:40.215647+0000 (oldest deadline 2024-03-24T10:52:01.915732+0000) 2024-03-24T10:52:03.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:03 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:03.406+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:03.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:03 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:03.406+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:03.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:03 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:03.406+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:03.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:03 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:03.444+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:03.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:03 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:03.449+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:52:03.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:03 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:03.449+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:52:04.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:03 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:03.871+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:04.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:03 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:03.871+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:04.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:03 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:03.871+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:04.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:03 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:03.956+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:04.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:03 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:03.956+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:04.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:03 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:03.956+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:04.430 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:04.150+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:04.430 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:04.150+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:04.753 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:04 smithi117 podman[73678]: 2024-03-24 10:52:04.521411855 +0000 UTC m=+5.494121533 container died 0cf35610b52a5ede056ef10dd281bede26454350f940a4ac3859220137f0a82e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=) 2024-03-24T10:52:04.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:04.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:04.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:04.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:04.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:04 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:04.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:04.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:04 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:04.431+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:04.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:04 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:04.431+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:04.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:04 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:04.460+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:52:04.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:04 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:04.460+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:52:04.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:04 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:04.460+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:40.076559+0000 front 2024-03-24T10:51:40.076524+0000 (oldest deadline 2024-03-24T10:52:04.076968+0000) 2024-03-24T10:52:05.146 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:04 smithi117 podman[73678]: 2024-03-24 10:52:04.912251502 +0000 UTC m=+5.884961180 container cleanup 0cf35610b52a5ede056ef10dd281bede26454350f940a4ac3859220137f0a82e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-24T10:52:05.146 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:04 smithi117 bash[73678]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5 2024-03-24T10:52:05.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:04 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:04.888+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:05.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:04 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:04.888+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:05.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:04 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:04.888+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:05.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:04 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:04.922+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:05.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:04 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:04.922+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:05.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:04 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:04.922+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:05.465 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:05 smithi117 podman[73695]: 2024-03-24 10:52:05.146848846 +0000 UTC m=+0.626862155 container remove 0cf35610b52a5ede056ef10dd281bede26454350f940a4ac3859220137f0a82e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T10:52:05.465 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:05.188+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:05.465 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:05.188+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:05.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:05.465+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:05.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:05.465+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:05.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:05 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:05.465+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:05.753 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:05 smithi117 podman[73785]: 2024-03-24 10:52:05.372386911 +0000 UTC m=+0.018106277 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:52:05.753 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:05 smithi117 podman[73785]: 2024-03-24 10:52:05.489546855 +0000 UTC m=+0.135266254 container create 45a6339a629bf8df96072892d9acd372a93de2bd5cb3604509d3fcd1efe65102 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T10:52:05.753 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:05 smithi117 podman[73785]: 2024-03-24 10:52:05.708924958 +0000 UTC m=+0.354644323 container init 45a6339a629bf8df96072892d9acd372a93de2bd5cb3604509d3fcd1efe65102 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319) 2024-03-24T10:52:05.753 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:05 smithi117 podman[73785]: 2024-03-24 10:52:05.712204109 +0000 UTC m=+0.357923458 container start 45a6339a629bf8df96072892d9acd372a93de2bd5cb3604509d3fcd1efe65102 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-24T10:52:05.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:05 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:05.412+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:05.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:05 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:05.412+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:05.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:05.458+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:52:05.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:05.458+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:52:05.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:05.458+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:40.076559+0000 front 2024-03-24T10:51:40.076524+0000 (oldest deadline 2024-03-24T10:52:04.076968+0000) 2024-03-24T10:52:06.181 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:05 smithi117 podman[73785]: 2024-03-24 10:52:05.775017324 +0000 UTC m=+0.420736697 container attach 45a6339a629bf8df96072892d9acd372a93de2bd5cb3604509d3fcd1efe65102 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:52:06.181 INFO:journalctl@ceph.osd.5.smithi117.stdout:Mar 24 10:52:05 smithi117 podman[73785]: 2024-03-24 10:52:05.996678562 +0000 UTC m=+0.642397926 container died 45a6339a629bf8df96072892d9acd372a93de2bd5cb3604509d3fcd1efe65102 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T10:52:06.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:05 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:05.885+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:06.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:05 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:05.885+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:06.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:05 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:05.885+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:06.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:05.922+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:06.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:05.922+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:06.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:05 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:05.922+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:06.469 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:06 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:06.181+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:06.469 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:06 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:06.181+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:06.469 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:06 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:06.426+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:06.470 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:06 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:06.426+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:06.470 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:06 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:06.426+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:06.491 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T10:52:06.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:52:06.532 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T10:52:06.532 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-24T10:52:06.532 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.6 2024-03-24T10:52:06.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:06 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:06.416+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:06.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:06 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:06.416+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:06.822 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:06 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:06.418+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:52:06.823 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:06 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:06.418+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:52:06.823 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:06 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:06.418+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:40.076559+0000 front 2024-03-24T10:51:40.076524+0000 (oldest deadline 2024-03-24T10:52:04.076968+0000) 2024-03-24T10:52:06.823 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:06 smithi023 systemd[1]: Stopping Ceph osd.6 for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:52:06.823 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:06 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:06.822+0000 7f00df70f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:52:07.168 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:06 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:06.822+0000 7f00df70f700 -1 osd.6 101 *** Got signal Terminated *** 2024-03-24T10:52:07.168 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:06 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:06.822+0000 7f00df70f700 -1 osd.6 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:52:07.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:06 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:06.968+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:07.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:06 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:06.968+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:07.168 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:06 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:06.968+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:07.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:06 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:06.859+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:07.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:06 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:06.859+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:07.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:06 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:06.859+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:07.502 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:07 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:07.176+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:07.503 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:07 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:07.176+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:07.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:07 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:07.420+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:07.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:07 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:07.420+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:07.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:07 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:07.420+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:07.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:07 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:07.419+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:07.768 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:07 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:07.419+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:07.917 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:07 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:07.453+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:52:07.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:07 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:07.453+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:52:07.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:07 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:07.453+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:40.076559+0000 front 2024-03-24T10:51:40.076524+0000 (oldest deadline 2024-03-24T10:52:04.076968+0000) 2024-03-24T10:52:08.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:07 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:07.870+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:08.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:07 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:07.870+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:08.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:07 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:07.870+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:08.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:07 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:07.947+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:08.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:07 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:07.947+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:08.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:07 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:07.947+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:08.435 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:08 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:08.134+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:08.435 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:08 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:08.134+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:08.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:08 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:08.434+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:08.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:08 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:08.434+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:08.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:08 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:08.434+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:08.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:08 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:08.377+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:08.768 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:08 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:08.377+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:08.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:08 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:08.429+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:52:08.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:08 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:08.429+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:52:08.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:08 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:08.429+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:40.076559+0000 front 2024-03-24T10:51:40.076524+0000 (oldest deadline 2024-03-24T10:52:04.076968+0000) 2024-03-24T10:52:09.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:08 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:08.880+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:09.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:08 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:08.880+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:09.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:08 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:08.880+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:09.394 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:08 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:08.993+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:09.395 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:08 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:08.993+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:09.395 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:08 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:08.993+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:09.441 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:09.171+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:09.441 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:09.171+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:09.668 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:09 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:09.393+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:52:09.668 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:09 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:09.393+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:52:09.668 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:09 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:09.393+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:40.076559+0000 front 2024-03-24T10:51:40.076524+0000 (oldest deadline 2024-03-24T10:52:04.076968+0000) 2024-03-24T10:52:09.668 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:09 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:09.393+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.577927+0000 front 2024-03-24T10:51:47.577971+0000 (oldest deadline 2024-03-24T10:52:09.278010+0000) 2024-03-24T10:52:09.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:09.441+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:09.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:09.441+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:09.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:09 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:09.441+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:09.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:09 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:09.390+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:09.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:09 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:09.390+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:10.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:09 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:09.851+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:10.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:09 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:09.851+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:10.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:09 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:09.851+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:10.403 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:10 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:10.042+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:10.403 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:10 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:10.042+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:10.404 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:10 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:10.042+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:10.404 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:10 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:10.042+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:10.488 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:10 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:10.184+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:10.488 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:10 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:10.184+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:10.668 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:10 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:10.403+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:52:10.668 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:10 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:10.403+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:52:10.668 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:10 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:10.403+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:40.076559+0000 front 2024-03-24T10:51:40.076524+0000 (oldest deadline 2024-03-24T10:52:04.076968+0000) 2024-03-24T10:52:10.668 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:10 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:10.403+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.577927+0000 front 2024-03-24T10:51:47.577971+0000 (oldest deadline 2024-03-24T10:52:09.278010+0000) 2024-03-24T10:52:10.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:10 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:10.488+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:10.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:10 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:10.488+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:10.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:10 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:10.488+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:10.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:10 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:10.488+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:10.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:10 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:10.425+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:10.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:10 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:10.425+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:11.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:10 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:10.844+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:11.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:10 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:10.844+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:11.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:10 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:10.844+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:11.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:11 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:11.060+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:11.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:11 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:11.060+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:11.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:11 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:11.060+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:11.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:11 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:11.060+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:11.501 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:11 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:11.159+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:11.502 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:11 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:11.159+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:11.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:11 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:11.501+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:11.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:11 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:11.501+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:11.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:11 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:11.501+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:11.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:11 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:11.501+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:11.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:11 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:11.474+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:11.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:11 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:11.474+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:11.768 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:11 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:11.474+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.053949+0000 front 2024-03-24T10:51:48.054091+0000 (oldest deadline 2024-03-24T10:52:10.954015+0000) 2024-03-24T10:52:11.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:11 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:11.420+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:22.772475+0000 front 2024-03-24T10:51:22.772504+0000 (oldest deadline 2024-03-24T10:51:45.072669+0000) 2024-03-24T10:52:11.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:11 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:11.420+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:30.274588+0000 front 2024-03-24T10:51:30.274564+0000 (oldest deadline 2024-03-24T10:51:55.574577+0000) 2024-03-24T10:52:11.918 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:11 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:11.420+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:40.076559+0000 front 2024-03-24T10:51:40.076524+0000 (oldest deadline 2024-03-24T10:52:04.076968+0000) 2024-03-24T10:52:11.919 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:11 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6[58493]: 2024-03-24T10:52:11.420+0000 7f00d84fc700 -1 osd.6 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.577927+0000 front 2024-03-24T10:51:47.577971+0000 (oldest deadline 2024-03-24T10:52:09.278010+0000) 2024-03-24T10:52:12.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:11 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:11.889+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:12.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:11 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:11.889+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:12.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:11 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:11.889+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:12.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:11 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:11.889+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:12.418 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:12 smithi023 podman[99263]: 2024-03-24 10:52:12.22541487 +0000 UTC m=+5.497575278 container died 029b4e00ff5ed62a98e5d427fd0a875c7e57db830ef55a992e010a4e9bdd8a6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T10:52:12.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:12 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:12.073+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:12.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:12 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:12.073+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:12.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:12 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:12.073+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:12.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:12 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:12.073+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:12.472 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:12 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:12.129+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:12.472 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:12 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:12.129+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:12.472 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:12 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:12.129+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:12.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:12 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:12.472+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:12.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:12 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:12.472+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:12.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:12 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:12.472+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:12.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:12 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:12.472+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:12.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:12 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:12.443+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:12.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:12 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:12.443+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:12.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:12 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:12.444+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.053949+0000 front 2024-03-24T10:51:48.054091+0000 (oldest deadline 2024-03-24T10:52:10.954015+0000) 2024-03-24T10:52:12.848 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:12 smithi023 podman[99263]: 2024-03-24 10:52:12.581553403 +0000 UTC m=+5.853713818 container cleanup 029b4e00ff5ed62a98e5d427fd0a875c7e57db830ef55a992e010a4e9bdd8a6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-24T10:52:12.848 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:12 smithi023 bash[99263]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6 2024-03-24T10:52:13.105 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:12 smithi023 podman[99281]: 2024-03-24 10:52:12.849216816 +0000 UTC m=+0.622404027 container remove 029b4e00ff5ed62a98e5d427fd0a875c7e57db830ef55a992e010a4e9bdd8a6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2) 2024-03-24T10:52:13.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:12 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:12.874+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:13.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:12 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:12.874+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:13.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:12 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:12.874+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:13.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:12 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:12.874+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:13.418 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:13 smithi023 podman[99370]: 2024-03-24 10:52:13.107402406 +0000 UTC m=+0.017657190 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:52:13.418 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:13 smithi023 podman[99370]: 2024-03-24 10:52:13.208351629 +0000 UTC m=+0.118606406 container create a3b09d01a7c9dc9de5cbf3cd7d75dc606c8559d3fb12725999663e3b382d8012 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T10:52:13.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:13 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:13.105+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:13.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:13 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:13.105+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:13.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:13 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:13.105+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:13.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:13 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:13.105+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:13.495 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:13 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:13.088+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:13.495 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:13 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:13.088+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:13.495 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:13 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:13.088+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:13.723 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:13 smithi023 podman[99370]: 2024-03-24 10:52:13.419265643 +0000 UTC m=+0.329520452 container init a3b09d01a7c9dc9de5cbf3cd7d75dc606c8559d3fb12725999663e3b382d8012 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T10:52:13.723 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:13 smithi023 podman[99370]: 2024-03-24 10:52:13.422344408 +0000 UTC m=+0.332599187 container start a3b09d01a7c9dc9de5cbf3cd7d75dc606c8559d3fb12725999663e3b382d8012 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-24T10:52:13.723 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:13 smithi023 podman[99370]: 2024-03-24 10:52:13.494162411 +0000 UTC m=+0.404417190 container attach a3b09d01a7c9dc9de5cbf3cd7d75dc606c8559d3fb12725999663e3b382d8012 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T10:52:13.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:13 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:13.495+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:13.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:13 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:13.495+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:13.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:13 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:13.495+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:13.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:13 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:13.495+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:13.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:13 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:13.416+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:13.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:13 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:13.416+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:13.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:13 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:13.416+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.053949+0000 front 2024-03-24T10:51:48.054091+0000 (oldest deadline 2024-03-24T10:52:10.954015+0000) 2024-03-24T10:52:14.004 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 24 10:52:13 smithi023 podman[99370]: 2024-03-24 10:52:13.723791425 +0000 UTC m=+0.634046200 container died a3b09d01a7c9dc9de5cbf3cd7d75dc606c8559d3fb12725999663e3b382d8012 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:52:14.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:13 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:13.922+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:14.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:13 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:13.922+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:14.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:13 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:13.922+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:14.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:13 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:13.922+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:14.273 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:14 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:14.083+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:14.273 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:14 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:14.083+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:14.273 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:14 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:14.083+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:14.274 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:14 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:14.083+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:14.299 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.6.service' 2024-03-24T10:52:14.341 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:52:14.341 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-24T10:52:14.341 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-24T10:52:14.342 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.7 2024-03-24T10:52:14.471 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:14 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:14.057+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:14.471 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:14 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:14.057+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:14.471 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:14 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:14.057+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:14.631 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:14 smithi116 systemd[1]: Stopping Ceph osd.7 for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:52:14.631 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:14 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:14.424+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:14.631 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:14 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:14.424+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:14.632 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:14 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:14.424+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.053949+0000 front 2024-03-24T10:51:48.054091+0000 (oldest deadline 2024-03-24T10:52:10.954015+0000) 2024-03-24T10:52:14.632 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:14 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:14.630+0000 7fd049616700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:52:14.632 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:14 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:14.630+0000 7fd049616700 -1 osd.7 101 *** Got signal Terminated *** 2024-03-24T10:52:14.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:14 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:14.471+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:14.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:14 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:14.471+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:14.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:14 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:14.471+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:14.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:14 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:14.471+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:14.963 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:14 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:14.630+0000 7fd049616700 -1 osd.7 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:52:15.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:14 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:14.963+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:15.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:14 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:14.963+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:15.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:14 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:14.963+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:15.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:14 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:14.963+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:15.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:15 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:15.047+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:15.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:15 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:15.047+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:15.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:15 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:15.047+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:15.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:15 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:15.047+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:15.432 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:15 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:15.053+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:15.432 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:15 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:15.053+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:15.432 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:15 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:15.053+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:15.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:15 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:15.432+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:15.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:15 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:15.432+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:15.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:15 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:15.432+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:15.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:15 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:15.432+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:15.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:15 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:15.380+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:15.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:15 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:15.380+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:15.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:15 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:15.380+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.053949+0000 front 2024-03-24T10:51:48.054091+0000 (oldest deadline 2024-03-24T10:52:10.954015+0000) 2024-03-24T10:52:16.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:15 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:15.935+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:16.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:15 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:15.935+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:16.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:15 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:15.935+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:16.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:15 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:15.935+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:16.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:16 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:16.054+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:16.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:16 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:16.054+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:16.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:16 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:16.054+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:16.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:16 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:16.054+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:16.445 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:16 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:16.053+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:16.445 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:16 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:16.053+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:16.445 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:16 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:16.053+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:16.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:16 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:16.445+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:16.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:16 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:16.445+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:16.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:16 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:16.445+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:16.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:16 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:16.445+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:16.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:16 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:16.338+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:16.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:16 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:16.338+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:16.768 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:16 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:16.338+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.053949+0000 front 2024-03-24T10:51:48.054091+0000 (oldest deadline 2024-03-24T10:52:10.954015+0000) 2024-03-24T10:52:17.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:16 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:16.950+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:17.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:16 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:16.950+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:17.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:16 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:16.950+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:17.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:16 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:16.950+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:17.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:17 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:17.061+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:17.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:17 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:17.061+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:17.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:17 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:17.061+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:17.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:17 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:17.061+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:17.468 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:17 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:17.067+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:17.468 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:17 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:17.067+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:17.468 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:17 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:17.067+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:17.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:17 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:17.468+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:17.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:17 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:17.468+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:17.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:17 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:17.468+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:17.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:17 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:17.468+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:17.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:17 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:17.301+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:17.768 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:17 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:17.301+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:17.768 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:17 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:17.301+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.053949+0000 front 2024-03-24T10:51:48.054091+0000 (oldest deadline 2024-03-24T10:52:10.954015+0000) 2024-03-24T10:52:18.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:17 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:17.947+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:18.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:17 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:17.947+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:18.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:17 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:17.947+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:18.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:17 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:17.947+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:18.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:18 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:18.091+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:18.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:18 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:18.091+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:18.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:18 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:18.091+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:18.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:18 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:18.091+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:18.471 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:18 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:18.024+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:18.472 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:18 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:18.024+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:18.472 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:18 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:18.024+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:18.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:18 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:18.472+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:18.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:18 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:18.472+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:18.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:18 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:18.472+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:18.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:18 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:18.472+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:18.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:18 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:18.314+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:18.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:18 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:18.314+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:18.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:18 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:18.314+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.053949+0000 front 2024-03-24T10:51:48.054091+0000 (oldest deadline 2024-03-24T10:52:10.954015+0000) 2024-03-24T10:52:19.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:18 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:18.991+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:19.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:18 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:18.991+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:19.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:18 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:18.991+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:19.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:18 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:18.991+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:19.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:19 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:19.136+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:19.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:19 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:19.136+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:19.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:19 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:19.136+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:19.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:19 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:19.136+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:19.479 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:19 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:19.057+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:19.479 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:19 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:19.057+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:19.479 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:19 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:19.057+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:19.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:19 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:19.478+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:19.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:19 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:19.478+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:19.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:19 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:19.478+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:19.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:19 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:19.478+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:19.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:19 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:19.353+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.849993+0000 front 2024-03-24T10:51:31.849934+0000 (oldest deadline 2024-03-24T10:51:53.450824+0000) 2024-03-24T10:52:19.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:19 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:19.353+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.751954+0000 front 2024-03-24T10:51:38.752117+0000 (oldest deadline 2024-03-24T10:52:03.452104+0000) 2024-03-24T10:52:19.767 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:19 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7[54760]: 2024-03-24T10:52:19.353+0000 7fd04640b700 -1 osd.7 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.053949+0000 front 2024-03-24T10:51:48.054091+0000 (oldest deadline 2024-03-24T10:52:10.954015+0000) 2024-03-24T10:52:20.381 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:20 smithi116 podman[73011]: 2024-03-24 10:52:20.136579955 +0000 UTC m=+5.600929636 container died dccc619f7d34155bb9eae323dde019bbcc2c090939271198f91046e90e68572c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T10:52:20.381 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:20 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:20.022+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:20.382 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:20 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:20.023+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:20.382 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:20 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:20.023+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:20.382 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:20 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:20.023+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:20.382 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:20 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:20.023+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:20.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:20 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:20.095+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:20.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:20 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:20.095+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:20.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:20 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:20.095+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:20.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:20 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:20.095+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:20.476 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:20 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:20.103+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:20.476 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:20 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:20.103+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:20.476 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:20 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:20.103+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:20.477 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:20 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:20.103+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:53.575685+0000 front 2024-03-24T10:51:53.575781+0000 (oldest deadline 2024-03-24T10:52:19.475621+0000) 2024-03-24T10:52:20.640 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:20 smithi116 podman[73011]: 2024-03-24 10:52:20.464354233 +0000 UTC m=+5.928703904 container cleanup dccc619f7d34155bb9eae323dde019bbcc2c090939271198f91046e90e68572c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-24T10:52:20.640 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:20 smithi116 bash[73011]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7 2024-03-24T10:52:20.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:20 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:20.477+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:20.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:20 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:20.477+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:20.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:20 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:20.477+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:20.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:20 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:20.477+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:20.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:20 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:20.477+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:20.979 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:20 smithi116 podman[73030]: 2024-03-24 10:52:20.64031689 +0000 UTC m=+0.501715339 container remove dccc619f7d34155bb9eae323dde019bbcc2c090939271198f91046e90e68572c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T10:52:21.243 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:21 smithi116 podman[73121]: 2024-03-24 10:52:20.936573235 +0000 UTC m=+0.017763283 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:52:21.244 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:21 smithi116 podman[73121]: 2024-03-24 10:52:21.049920054 +0000 UTC m=+0.131110080 container create 568310d72f5d783b29c3a699edc9063de8d432202f3fa801159b5885ef78f765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T10:52:21.244 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:20 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:20.979+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:21.244 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:20 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:20.979+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:21.244 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:20 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:20.979+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:21.244 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:20 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:20.979+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:21.244 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:20 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:20.979+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:21.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:21 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:21.067+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:21.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:21 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:21.067+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:21.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:21 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:21.067+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:21.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:21 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:21.067+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:21.458 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:21 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:21.152+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:21.458 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:21 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:21.152+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:21.458 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:21 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:21.152+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:21.458 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:21 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:21.152+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:53.575685+0000 front 2024-03-24T10:51:53.575781+0000 (oldest deadline 2024-03-24T10:52:19.475621+0000) 2024-03-24T10:52:21.517 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:21 smithi116 podman[73121]: 2024-03-24 10:52:21.244071559 +0000 UTC m=+0.325261595 container init 568310d72f5d783b29c3a699edc9063de8d432202f3fa801159b5885ef78f765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:52:21.517 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:21 smithi116 podman[73121]: 2024-03-24 10:52:21.247908966 +0000 UTC m=+0.329098998 container start 568310d72f5d783b29c3a699edc9063de8d432202f3fa801159b5885ef78f765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-24T10:52:21.518 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:21 smithi116 podman[73121]: 2024-03-24 10:52:21.335438786 +0000 UTC m=+0.416628820 container attach 568310d72f5d783b29c3a699edc9063de8d432202f3fa801159b5885ef78f765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7-deactivate, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T10:52:21.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:21 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:21.458+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:21.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:21 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:21.458+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:21.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:21 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:21.458+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:21.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:21 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:21.458+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:21.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:21 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:21.458+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:21.812 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:21 smithi116 podman[73121]: 2024-03-24 10:52:21.528393491 +0000 UTC m=+0.609583527 container died 568310d72f5d783b29c3a699edc9063de8d432202f3fa801159b5885ef78f765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-24T10:52:22.063 INFO:journalctl@ceph.osd.7.smithi116.stdout:Mar 24 10:52:22 smithi116 podman[73147]: 2024-03-24 10:52:22.046990925 +0000 UTC m=+0.513263660 container remove 568310d72f5d783b29c3a699edc9063de8d432202f3fa801159b5885ef78f765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T10:52:22.064 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:21 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:21.943+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:22.064 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:21 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:21.943+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:22.064 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:21 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:21.943+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:22.064 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:21 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:21.943+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:22.064 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:21 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:21.943+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:22.069 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.7.service' 2024-03-24T10:52:22.109 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:52:22.110 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-24T10:52:22.110 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-24T10:52:22.110 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.8 2024-03-24T10:52:22.410 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:22 smithi117 systemd[1]: Stopping Ceph osd.8 for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:52:22.410 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:22 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:22.157+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:22.410 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:22 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:22.157+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:22.410 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:22 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:22.157+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:22.410 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:22 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:22.157+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:53.575685+0000 front 2024-03-24T10:51:53.575781+0000 (oldest deadline 2024-03-24T10:52:19.475621+0000) 2024-03-24T10:52:22.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:22 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:22.108+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:22.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:22 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:22.108+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:22.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:22 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:22.108+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:22.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:22 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:22.108+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:22.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:22 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:22.108+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:56.125541+0000 front 2024-03-24T10:51:56.125724+0000 (oldest deadline 2024-03-24T10:52:22.025476+0000) 2024-03-24T10:52:22.752 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:22 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:22.420+0000 7f1a24d68700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:52:22.753 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:22 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:22.420+0000 7f1a24d68700 -1 osd.8 101 *** Got signal Terminated *** 2024-03-24T10:52:22.753 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:22 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:22.420+0000 7f1a24d68700 -1 osd.8 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:52:22.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:22 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:22.410+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:22.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:22 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:22.410+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:22.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:22 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:22.410+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:22.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:22 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:22.410+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:22.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:22 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:22.410+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:23.236 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:22 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:22.943+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:23.237 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:22 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:22.943+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:23.237 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:22 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:22.943+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:23.237 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:22 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:22.944+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:23.237 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:22 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:22.944+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:23.413 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:23 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:23.125+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:23.414 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:23 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:23.125+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:23.414 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:23 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:23.125+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:23.414 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:23 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:23.125+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:53.575685+0000 front 2024-03-24T10:51:53.575781+0000 (oldest deadline 2024-03-24T10:52:19.475621+0000) 2024-03-24T10:52:23.417 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:23 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:23.130+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:23.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:23 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:23.130+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:23.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:23 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:23.130+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:23.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:23 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:23.130+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:23.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:23 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:23.130+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:56.125541+0000 front 2024-03-24T10:51:56.125724+0000 (oldest deadline 2024-03-24T10:52:22.025476+0000) 2024-03-24T10:52:23.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:23 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:23.413+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:23.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:23 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:23.413+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:23.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:23 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:23.413+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:23.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:23 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:23.413+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:23.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:23 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:23.413+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:24.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:23 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:23.904+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:24.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:23 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:23.904+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:24.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:23 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:23.904+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:24.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:23 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:23.904+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:24.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:23 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:23.904+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:24.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:24 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:24.092+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:24.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:24 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:24.092+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:24.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:24 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:24.092+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:24.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:24 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:24.092+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:24.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:24 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:24.092+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:56.125541+0000 front 2024-03-24T10:51:56.125724+0000 (oldest deadline 2024-03-24T10:52:22.025476+0000) 2024-03-24T10:52:24.454 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:24 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:24.136+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:24.454 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:24 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:24.136+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:24.455 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:24 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:24.136+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:24.455 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:24 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:24.136+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:53.575685+0000 front 2024-03-24T10:51:53.575781+0000 (oldest deadline 2024-03-24T10:52:19.475621+0000) 2024-03-24T10:52:24.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:24 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:24.455+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:24.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:24 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:24.455+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:24.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:24 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:24.455+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:24.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:24 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:24.455+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:24.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:24 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:24.455+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:25.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:24 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:24.942+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:25.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:24 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:24.942+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:25.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:24 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:24.942+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:25.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:24 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:24.942+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:25.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:24 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:24.942+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:25.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:25 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:25.050+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:25.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:25 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:25.050+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:25.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:25 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:25.050+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:25.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:25 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:25.050+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:25.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:25 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:25.050+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:56.125541+0000 front 2024-03-24T10:51:56.125724+0000 (oldest deadline 2024-03-24T10:52:22.025476+0000) 2024-03-24T10:52:25.446 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:25.161+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:25.446 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:25.161+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:25.446 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:25.161+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:25.446 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:25.161+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:53.575685+0000 front 2024-03-24T10:51:53.575781+0000 (oldest deadline 2024-03-24T10:52:19.475621+0000) 2024-03-24T10:52:25.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:25.446+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:25.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:25.446+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:25.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:25.446+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:25.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:25.446+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:25.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:25 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:25.446+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:26.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:25 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:25.992+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:26.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:25 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:25.992+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:26.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:25 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:25.992+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:26.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:25 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:25.992+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:26.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:25 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:25.992+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:26.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:25 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:25.992+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:26.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:26 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:26.008+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:26.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:26 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:26.008+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:26.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:26 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:26.008+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:26.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:26 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:26.008+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:26.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:26 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:26.008+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:56.125541+0000 front 2024-03-24T10:51:56.125724+0000 (oldest deadline 2024-03-24T10:52:22.025476+0000) 2024-03-24T10:52:26.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:26 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:26.008+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:52:02.026050+0000 front 2024-03-24T10:52:02.025976+0000 (oldest deadline 2024-03-24T10:52:25.525902+0000) 2024-03-24T10:52:26.474 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:26 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:26.136+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:26.474 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:26 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:26.136+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:26.474 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:26 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:26.136+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:26.474 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:26 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:26.136+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:53.575685+0000 front 2024-03-24T10:51:53.575781+0000 (oldest deadline 2024-03-24T10:52:19.475621+0000) 2024-03-24T10:52:26.474 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:26 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:26.136+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:52:01.776624+0000 front 2024-03-24T10:52:01.776606+0000 (oldest deadline 2024-03-24T10:52:25.876714+0000) 2024-03-24T10:52:26.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:26 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:26.474+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:26.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:26 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:26.474+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:26.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:26 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:26.474+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:26.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:26 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:26.474+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:26.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:26 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:26.474+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:26.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:26 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:26.474+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:27.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:27 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:27.011+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:27.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:27 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:27.011+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:27.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:27 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:27.011+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:27.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:27 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:27.011+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:27.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:27 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:27.011+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:27.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:27 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:27.011+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:27.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:27 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:27.008+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:27.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:27 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:27.008+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:27.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:27 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:27.008+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:27.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:27 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:27.008+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:27.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:27 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:27.008+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:56.125541+0000 front 2024-03-24T10:51:56.125724+0000 (oldest deadline 2024-03-24T10:52:22.025476+0000) 2024-03-24T10:52:27.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:27 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:27.008+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:52:02.026050+0000 front 2024-03-24T10:52:02.025976+0000 (oldest deadline 2024-03-24T10:52:25.525902+0000) 2024-03-24T10:52:27.481 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:27.142+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.070812+0000 front 2024-03-24T10:51:28.070828+0000 (oldest deadline 2024-03-24T10:51:53.370819+0000) 2024-03-24T10:52:27.481 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:27.142+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.572442+0000 front 2024-03-24T10:51:38.572614+0000 (oldest deadline 2024-03-24T10:52:03.272570+0000) 2024-03-24T10:52:27.481 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:27.142+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:48.374682+0000 front 2024-03-24T10:51:48.374526+0000 (oldest deadline 2024-03-24T10:52:11.274517+0000) 2024-03-24T10:52:27.482 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:27.142+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:53.575685+0000 front 2024-03-24T10:51:53.575781+0000 (oldest deadline 2024-03-24T10:52:19.475621+0000) 2024-03-24T10:52:27.482 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8[55405]: 2024-03-24T10:52:27.142+0000 7f1a1db55700 -1 osd.8 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:52:01.776624+0000 front 2024-03-24T10:52:01.776606+0000 (oldest deadline 2024-03-24T10:52:25.876714+0000) 2024-03-24T10:52:27.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:27.481+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:27.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:27.481+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:27.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:27.481+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:27.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:27.481+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:27.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:27.481+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:27.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:27 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:27.481+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:28.179 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:27 smithi117 podman[73891]: 2024-03-24 10:52:27.925238417 +0000 UTC m=+5.627717877 container died 8b8176871a14d4fdbba8da04240cc771356eaa3766fac5ab4790993cafbe951a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:52:28.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:28 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:28.004+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:28.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:28 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:28.004+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:28.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:28 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:28.004+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:28.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:28 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:28.004+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:28.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:28 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:28.004+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:28.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:28 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:28.004+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:28.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:28 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:28.051+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:28.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:28 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:28.051+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:28.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:28 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:28.051+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:28.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:28 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:28.051+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:28.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:28 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:28.051+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:56.125541+0000 front 2024-03-24T10:51:56.125724+0000 (oldest deadline 2024-03-24T10:52:22.025476+0000) 2024-03-24T10:52:28.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:28 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:28.051+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:52:02.026050+0000 front 2024-03-24T10:52:02.025976+0000 (oldest deadline 2024-03-24T10:52:25.525902+0000) 2024-03-24T10:52:28.444 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:28 smithi117 podman[73891]: 2024-03-24 10:52:28.303097346 +0000 UTC m=+6.005576791 container cleanup 8b8176871a14d4fdbba8da04240cc771356eaa3766fac5ab4790993cafbe951a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T10:52:28.444 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:28 smithi117 bash[73891]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8 2024-03-24T10:52:28.752 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:28 smithi117 podman[73908]: 2024-03-24 10:52:28.545832071 +0000 UTC m=+0.617954539 container remove 8b8176871a14d4fdbba8da04240cc771356eaa3766fac5ab4790993cafbe951a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:52:28.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:28 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:28.444+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:28.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:28 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:28.444+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:28.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:28 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:28.444+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:28.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:28 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:28.444+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:28.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:28 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:28.444+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:28.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:28 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:28.444+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:29.131 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:28 smithi117 podman[74000]: 2024-03-24 10:52:28.846572511 +0000 UTC m=+0.085956435 container create e59c5a8f9125896034d5f2622cbba8cc2d0f66ecff0bcd253d0b167427154dc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:52:29.132 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:28 smithi117 podman[74000]: 2024-03-24 10:52:28.778248644 +0000 UTC m=+0.017632578 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:52:29.132 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:29 smithi117 podman[74000]: 2024-03-24 10:52:29.04952085 +0000 UTC m=+0.288904779 container init e59c5a8f9125896034d5f2622cbba8cc2d0f66ecff0bcd253d0b167427154dc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:52:29.132 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:29 smithi117 podman[74000]: 2024-03-24 10:52:29.052671707 +0000 UTC m=+0.292055632 container start e59c5a8f9125896034d5f2622cbba8cc2d0f66ecff0bcd253d0b167427154dc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:52:29.132 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:29 smithi117 podman[74000]: 2024-03-24 10:52:29.132229662 +0000 UTC m=+0.371613598 container attach e59c5a8f9125896034d5f2622cbba8cc2d0f66ecff0bcd253d0b167427154dc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-24T10:52:29.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:29 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:29.004+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:29.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:29 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:29.004+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:29.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:29 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:29.004+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:29.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:29 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:29.004+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:29.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:29 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:29.004+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:29.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:29 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:29.004+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:29.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:29 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:29.054+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:29.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:29 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:29.054+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:29.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:29 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:29.054+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:29.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:29 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:29.054+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:29.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:29 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:29.054+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:56.125541+0000 front 2024-03-24T10:51:56.125724+0000 (oldest deadline 2024-03-24T10:52:22.025476+0000) 2024-03-24T10:52:29.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:29 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:29.054+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:52:02.026050+0000 front 2024-03-24T10:52:02.025976+0000 (oldest deadline 2024-03-24T10:52:25.525902+0000) 2024-03-24T10:52:29.430 INFO:journalctl@ceph.osd.8.smithi117.stdout:Mar 24 10:52:29 smithi117 podman[74000]: 2024-03-24 10:52:29.342390428 +0000 UTC m=+0.581774357 container died e59c5a8f9125896034d5f2622cbba8cc2d0f66ecff0bcd253d0b167427154dc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-24T10:52:29.684 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:29 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:29.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:29.685 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:29 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:29.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:29.685 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:29 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:29.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:29.685 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:29 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:29.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:29.685 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:29 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:29.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:29.685 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:29 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:29.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:29.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.8.service' 2024-03-24T10:52:29.999 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:52:30.000 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-24T10:52:30.000 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-24T10:52:30.000 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.9 2024-03-24T10:52:30.291 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:30.045+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:30.292 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:30.045+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:30.292 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:30.045+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:30.292 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:30.045+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:30.292 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:30.045+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:56.125541+0000 front 2024-03-24T10:51:56.125724+0000 (oldest deadline 2024-03-24T10:52:22.025476+0000) 2024-03-24T10:52:30.292 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:30.045+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:52:02.026050+0000 front 2024-03-24T10:52:02.025976+0000 (oldest deadline 2024-03-24T10:52:25.525902+0000) 2024-03-24T10:52:30.292 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:30 smithi023 systemd[1]: Stopping Ceph osd.9 for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:52:30.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:30.034+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:30.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:30.034+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:30.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:30.034+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:30.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:30.034+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:30.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:30.034+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:30.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:30.034+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:30.668 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:30.291+0000 7febcba9a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:52:30.668 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:30.291+0000 7febcba9a700 -1 osd.9 101 *** Got signal Terminated *** 2024-03-24T10:52:30.668 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:30 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:30.291+0000 7febcba9a700 -1 osd.9 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:52:30.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:30 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:30.439+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:30.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:30 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:30.439+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:30.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:30 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:30.439+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:30.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:30 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:30.439+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:30.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:30 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:30.439+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:30.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:30 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:30.439+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:31.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:30.993+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:31.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:30.993+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:31.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:30.993+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:31.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:30.993+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:31.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:30.993+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:31.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:30 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:30.993+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:31.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:31 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:31.091+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:31.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:31 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:31.091+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:31.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:31 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:31.091+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:31.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:31 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:31.091+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:31.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:31 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:31.091+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:56.125541+0000 front 2024-03-24T10:51:56.125724+0000 (oldest deadline 2024-03-24T10:52:22.025476+0000) 2024-03-24T10:52:31.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:31 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:31.091+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:52:02.026050+0000 front 2024-03-24T10:52:02.025976+0000 (oldest deadline 2024-03-24T10:52:25.525902+0000) 2024-03-24T10:52:31.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:31 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:31.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:31.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:31 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:31.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:31.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:31 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:31.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:31.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:31 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:31.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:31.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:31 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:31.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:31.503 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:31 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:31.429+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:32.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:31 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:31.967+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:32.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:31 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:31.967+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:32.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:31 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:31.967+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:32.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:31 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:31.967+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:32.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:31 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:31.967+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:32.269 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:31 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:31.967+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:32.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:32 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:32.083+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:32.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:32 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:32.083+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:32.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:32 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:32.083+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:32.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:32 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:32.083+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:32.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:32 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:32.083+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:56.125541+0000 front 2024-03-24T10:51:56.125724+0000 (oldest deadline 2024-03-24T10:52:22.025476+0000) 2024-03-24T10:52:32.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:32 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:32.083+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:52:02.026050+0000 front 2024-03-24T10:52:02.025976+0000 (oldest deadline 2024-03-24T10:52:25.525902+0000) 2024-03-24T10:52:32.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:32 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:32.461+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:32.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:32 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:32.461+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:32.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:32 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:32.461+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:32.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:32 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:32.461+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:32.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:32 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:32.461+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:32.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:32 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:32.461+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:33.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:32 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:32.929+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:33.269 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:32 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:32.929+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:33.269 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:32 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:32.929+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:33.269 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:32 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:32.929+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:33.269 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:32 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:32.929+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:33.270 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:32 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:32.929+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:33.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:33 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:33.129+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:33.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:33 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:33.129+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:33.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:33 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:33.129+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:33.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:33 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:33.129+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:33.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:33 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:33.129+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:56.125541+0000 front 2024-03-24T10:51:56.125724+0000 (oldest deadline 2024-03-24T10:52:22.025476+0000) 2024-03-24T10:52:33.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:33 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:33.129+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:52:02.026050+0000 front 2024-03-24T10:52:02.025976+0000 (oldest deadline 2024-03-24T10:52:25.525902+0000) 2024-03-24T10:52:33.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:33 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:33.495+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:33.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:33 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:33.495+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:33.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:33 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:33.495+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:33.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:33 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:33.495+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:33.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:33 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:33.495+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:33.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:33 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:33.495+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:34.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:33 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:33.957+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:34.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:33 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:33.957+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:34.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:33 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:33.957+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:34.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:33 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:33.957+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:34.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:33 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:33.957+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:34.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:33 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:33.957+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:34.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:33 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:33.957+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-24T10:52:09.528258+0000 front 2024-03-24T10:52:09.528245+0000 (oldest deadline 2024-03-24T10:52:33.628193+0000) 2024-03-24T10:52:34.668 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:34 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:34.175+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:34.668 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:34 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:34.175+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:34.668 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:34 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:34.175+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:34.668 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:34 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:34.175+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:34.668 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:34 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:34.175+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:56.125541+0000 front 2024-03-24T10:51:56.125724+0000 (oldest deadline 2024-03-24T10:52:22.025476+0000) 2024-03-24T10:52:34.668 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:34 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:34.175+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:52:02.026050+0000 front 2024-03-24T10:52:02.025976+0000 (oldest deadline 2024-03-24T10:52:25.525902+0000) 2024-03-24T10:52:34.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:34 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:34.463+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:34.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:34 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:34.463+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:34.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:34 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:34.463+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:34.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:34 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:34.463+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:34.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:34 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:34.463+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:34.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:34 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:34.463+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:35.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:34 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:34.970+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:35.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:34 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:34.970+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:35.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:34 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:34.970+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:35.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:34 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:34.970+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:35.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:34 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:34.970+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:35.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:34 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:34.970+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:35.269 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:34 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:34.970+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-24T10:52:09.528258+0000 front 2024-03-24T10:52:09.528245+0000 (oldest deadline 2024-03-24T10:52:33.628193+0000) 2024-03-24T10:52:35.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:35 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:35.142+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:23.920712+0000 front 2024-03-24T10:51:23.920758+0000 (oldest deadline 2024-03-24T10:51:48.620787+0000) 2024-03-24T10:52:35.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:35 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:35.142+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:28.621285+0000 front 2024-03-24T10:51:28.621309+0000 (oldest deadline 2024-03-24T10:51:53.321501+0000) 2024-03-24T10:52:35.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:35 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:35.142+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:38.022861+0000 front 2024-03-24T10:51:38.022798+0000 (oldest deadline 2024-03-24T10:52:02.722990+0000) 2024-03-24T10:52:35.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:35 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:35.142+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.424057+0000 front 2024-03-24T10:51:47.424172+0000 (oldest deadline 2024-03-24T10:52:09.724130+0000) 2024-03-24T10:52:35.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:35 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:35.142+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:56.125541+0000 front 2024-03-24T10:51:56.125724+0000 (oldest deadline 2024-03-24T10:52:22.025476+0000) 2024-03-24T10:52:35.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:35 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:35.142+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:52:02.026050+0000 front 2024-03-24T10:52:02.025976+0000 (oldest deadline 2024-03-24T10:52:25.525902+0000) 2024-03-24T10:52:35.419 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:35 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9[65151]: 2024-03-24T10:52:35.142+0000 7febc888f700 -1 osd.9 101 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-24T10:52:10.727269+0000 front 2024-03-24T10:52:10.727334+0000 (oldest deadline 2024-03-24T10:52:34.827433+0000) 2024-03-24T10:52:35.959 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:35 smithi023 podman[99488]: 2024-03-24 10:52:35.702844652 +0000 UTC m=+5.507365097 container died 515edaca3bf38cd72195f896808ecde56acb3dd8ded9d562e92f71aa52124d2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:52:36.002 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:35 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:35.507+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:36.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:35 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:35.507+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:36.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:35 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:35.507+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:36.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:35 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:35.508+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:36.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:35 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:35.508+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:36.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:35 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:35.508+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:36.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:36 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:36.016+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:36.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:36 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:36.016+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:36.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:36 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:36.016+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:36.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:36 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:36.016+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:36.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:36 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:36.016+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:36.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:36 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:36.016+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:36.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:36 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:36.016+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-24T10:52:09.528258+0000 front 2024-03-24T10:52:09.528245+0000 (oldest deadline 2024-03-24T10:52:33.628193+0000) 2024-03-24T10:52:36.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:36 smithi023 podman[99488]: 2024-03-24 10:52:36.058196295 +0000 UTC m=+5.862716741 container cleanup 515edaca3bf38cd72195f896808ecde56acb3dd8ded9d562e92f71aa52124d2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T10:52:36.418 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:36 smithi023 bash[99488]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9 2024-03-24T10:52:36.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:36 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:36.500+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:36.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:36 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:36.500+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:36.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:36 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:36.500+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:36.755 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:36 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:36.500+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:36.755 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:36 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:36.500+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:36.755 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:36 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:36.500+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:36.809 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:36 smithi023 podman[99506]: 2024-03-24 10:52:36.450856481 +0000 UTC m=+0.743063843 container remove 515edaca3bf38cd72195f896808ecde56acb3dd8ded9d562e92f71aa52124d2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319) 2024-03-24T10:52:37.086 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:36 smithi023 podman[99596]: 2024-03-24 10:52:36.810052742 +0000 UTC m=+0.094801097 container create a44d4efb4037f03d55b9282342038242e47f321642887ed00817dd9c6e7790b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T10:52:37.087 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:36 smithi023 podman[99596]: 2024-03-24 10:52:36.732510115 +0000 UTC m=+0.017258445 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:52:37.087 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:37 smithi023 podman[99596]: 2024-03-24 10:52:37.012515079 +0000 UTC m=+0.297263405 container init a44d4efb4037f03d55b9282342038242e47f321642887ed00817dd9c6e7790b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T10:52:37.087 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:37 smithi023 podman[99596]: 2024-03-24 10:52:37.015694216 +0000 UTC m=+0.300442530 container start a44d4efb4037f03d55b9282342038242e47f321642887ed00817dd9c6e7790b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-24T10:52:37.087 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:37 smithi023 podman[99596]: 2024-03-24 10:52:37.087416393 +0000 UTC m=+0.372164715 container attach a44d4efb4037f03d55b9282342038242e47f321642887ed00817dd9c6e7790b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9-deactivate, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T10:52:37.339 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 24 10:52:37 smithi023 podman[99596]: 2024-03-24 10:52:37.314392138 +0000 UTC m=+0.599140454 container died a44d4efb4037f03d55b9282342038242e47f321642887ed00817dd9c6e7790b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-9-deactivate, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-24T10:52:37.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:37 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:37.036+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:37.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:37 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:37.036+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:37.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:37 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:37.036+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:37.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:37 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:37.036+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:37.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:37 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:37.036+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:37.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:37 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:37.036+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:37.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:37 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:37.036+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-24T10:52:09.528258+0000 front 2024-03-24T10:52:09.528245+0000 (oldest deadline 2024-03-24T10:52:33.628193+0000) 2024-03-24T10:52:37.913 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.9.service' 2024-03-24T10:52:37.953 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:52:37.954 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-24T10:52:37.954 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-24T10:52:37.954 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.10 2024-03-24T10:52:38.002 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:37 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:37.520+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:38.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:37 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:37.520+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:38.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:37 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:37.520+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:38.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:37 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:37.520+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:38.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:37 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:37.520+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:38.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:37 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:37.520+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:38.249 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:38 smithi116 systemd[1]: Stopping Ceph osd.10 for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:52:38.249 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:38 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:38.043+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:38.249 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:38 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:38.043+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:38.250 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:38 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:38.043+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:38.250 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:38 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:38.043+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:38.250 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:38 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:38.043+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:38.250 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:38 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:38.043+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:38.250 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:38 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:38.043+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-24T10:52:09.528258+0000 front 2024-03-24T10:52:09.528245+0000 (oldest deadline 2024-03-24T10:52:33.628193+0000) 2024-03-24T10:52:38.250 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:38 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:38.248+0000 7f832b957700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:52:38.250 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:38 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:38.248+0000 7f832b957700 -1 osd.10 101 *** Got signal Terminated *** 2024-03-24T10:52:38.250 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:38 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:38.248+0000 7f832b957700 -1 osd.10 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:52:39.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:38 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:38.503+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:39.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:38 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:38.503+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:39.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:38 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:38.503+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:39.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:38 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:38.503+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:39.003 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:38 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:38.503+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:39.004 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:38 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:38.503+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:39.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:39 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:39.029+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:39.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:39 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:39.029+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:39.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:39 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:39.029+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:39.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:39 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:39.029+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:39.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:39 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:39.029+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:39.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:39 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:39.029+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:39.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:39 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:39.029+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-24T10:52:09.528258+0000 front 2024-03-24T10:52:09.528245+0000 (oldest deadline 2024-03-24T10:52:33.628193+0000) 2024-03-24T10:52:39.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:39 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:39.456+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:39.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:39 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:39.456+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:39.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:39 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:39.456+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:39.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:39 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:39.456+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:39.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:39 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:39.456+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:39.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:39 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:39.456+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:40.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:40 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:40.046+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:40.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:40 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:40.046+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:40.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:40 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:40.046+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:40.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:40 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:40.046+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:40.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:40 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:40.046+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:40.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:40 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:40.046+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:40.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:40 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:40.046+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-24T10:52:09.528258+0000 front 2024-03-24T10:52:09.528245+0000 (oldest deadline 2024-03-24T10:52:33.628193+0000) 2024-03-24T10:52:40.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:40 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:40.474+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:40.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:40 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:40.474+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:40.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:40 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:40.474+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:40.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:40 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:40.474+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:40.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:40 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:40.474+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:40.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:40 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:40.474+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:41.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:41 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:41.023+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:41.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:41 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:41.023+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:41.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:41 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:41.023+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:41.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:41 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:41.023+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:41.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:41 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:41.023+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:41.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:41 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:41.023+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:41.518 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:41 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:41.023+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-24T10:52:09.528258+0000 front 2024-03-24T10:52:09.528245+0000 (oldest deadline 2024-03-24T10:52:33.628193+0000) 2024-03-24T10:52:41.752 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:41 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:41.488+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:41.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:41 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:41.488+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:41.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:41 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:41.488+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:41.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:41 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:41.488+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:41.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:41 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:41.488+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:41.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:41 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:41.488+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:42.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:41 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:41.975+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:42.267 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:41 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:41.975+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:42.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:41 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:41.975+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:42.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:41 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:41.975+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:42.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:41 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:41.975+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:42.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:41 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:41.975+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:42.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:41 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:41.975+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-24T10:52:09.528258+0000 front 2024-03-24T10:52:09.528245+0000 (oldest deadline 2024-03-24T10:52:33.628193+0000) 2024-03-24T10:52:42.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:42 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:42.497+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:42.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:42 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:42.497+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:42.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:42 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:42.497+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:42.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:42 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:42.497+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:42.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:42 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:42.497+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:42.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:42 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:42.497+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:43.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:42 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:42.927+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.320619+0000 front 2024-03-24T10:51:21.320559+0000 (oldest deadline 2024-03-24T10:51:46.020397+0000) 2024-03-24T10:52:43.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:42 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:42.927+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:31.722350+0000 front 2024-03-24T10:51:31.722375+0000 (oldest deadline 2024-03-24T10:51:54.622468+0000) 2024-03-24T10:52:43.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:42 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:42.927+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:35.723417+0000 front 2024-03-24T10:51:35.723571+0000 (oldest deadline 2024-03-24T10:52:01.623519+0000) 2024-03-24T10:52:43.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:42 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:42.927+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:46.224902+0000 front 2024-03-24T10:51:46.224808+0000 (oldest deadline 2024-03-24T10:52:11.524876+0000) 2024-03-24T10:52:43.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:42 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:42.927+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:55.025673+0000 front 2024-03-24T10:51:55.025741+0000 (oldest deadline 2024-03-24T10:52:19.725763+0000) 2024-03-24T10:52:43.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:42 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:42.927+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.726323+0000 front 2024-03-24T10:51:59.726235+0000 (oldest deadline 2024-03-24T10:52:25.626263+0000) 2024-03-24T10:52:43.269 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:42 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10[61367]: 2024-03-24T10:52:42.927+0000 7f8324744700 -1 osd.10 101 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-24T10:52:09.528258+0000 front 2024-03-24T10:52:09.528245+0000 (oldest deadline 2024-03-24T10:52:33.628193+0000) 2024-03-24T10:52:43.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:43 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:43.454+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:43.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:43 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:43.454+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:43.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:43 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:43.454+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:43.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:43 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:43.454+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:43.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:43 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:43.454+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:43.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:43 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:43.454+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:44.098 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:43 smithi116 podman[73241]: 2024-03-24 10:52:43.777369716 +0000 UTC m=+5.629961995 container died 3def853a5cac62d8d7a8a744441c251b4f4720494e9c2d760e265e5f9794c6af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T10:52:44.098 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:44 smithi116 podman[73241]: 2024-03-24 10:52:44.098635675 +0000 UTC m=+5.951227966 container cleanup 3def853a5cac62d8d7a8a744441c251b4f4720494e9c2d760e265e5f9794c6af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:52:44.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:44 smithi116 bash[73241]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10 2024-03-24T10:52:44.517 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:44 smithi116 podman[73258]: 2024-03-24 10:52:44.291293571 +0000 UTC m=+0.508175635 container remove 3def853a5cac62d8d7a8a744441c251b4f4720494e9c2d760e265e5f9794c6af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T10:52:44.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:44 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:44.484+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:44.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:44 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:44.484+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:44.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:44 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:44.484+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:44.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:44 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:44.484+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:44.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:44 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:44.484+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:44.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:44 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:44.484+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:44.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:44 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:44.484+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-24T10:52:19.272184+0000 front 2024-03-24T10:52:19.272197+0000 (oldest deadline 2024-03-24T10:52:43.972150+0000) 2024-03-24T10:52:44.977 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:44 smithi116 podman[73352]: 2024-03-24 10:52:44.708874555 +0000 UTC m=+0.103480187 container create e798cf6261af524b88cfc318e8f306003d00836e7f378eacab71b6c1a8dc078d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T10:52:44.978 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:44 smithi116 podman[73352]: 2024-03-24 10:52:44.622221604 +0000 UTC m=+0.016827207 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:52:44.978 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:44 smithi116 podman[73352]: 2024-03-24 10:52:44.91098717 +0000 UTC m=+0.305592761 container init e798cf6261af524b88cfc318e8f306003d00836e7f378eacab71b6c1a8dc078d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T10:52:44.978 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:44 smithi116 podman[73352]: 2024-03-24 10:52:44.914050733 +0000 UTC m=+0.308656323 container start e798cf6261af524b88cfc318e8f306003d00836e7f378eacab71b6c1a8dc078d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:52:44.978 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:44 smithi116 podman[73352]: 2024-03-24 10:52:44.977552139 +0000 UTC m=+0.372157748 container attach e798cf6261af524b88cfc318e8f306003d00836e7f378eacab71b6c1a8dc078d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T10:52:45.268 INFO:journalctl@ceph.osd.10.smithi116.stdout:Mar 24 10:52:45 smithi116 podman[73352]: 2024-03-24 10:52:45.208691332 +0000 UTC m=+0.603296942 container died e798cf6261af524b88cfc318e8f306003d00836e7f378eacab71b6c1a8dc078d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True) 2024-03-24T10:52:45.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:45 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:45.489+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:45.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:45 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:45.489+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:45.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:45 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:45.489+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:45.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:45 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:45.489+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:45.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:45 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:45.489+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:45.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:45 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:45.489+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:45.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:45 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:45.489+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-24T10:52:19.272184+0000 front 2024-03-24T10:52:19.272197+0000 (oldest deadline 2024-03-24T10:52:43.972150+0000) 2024-03-24T10:52:45.760 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.10.service' 2024-03-24T10:52:45.799 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:52:45.799 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-24T10:52:45.800 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-24T10:52:45.800 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.11 2024-03-24T10:52:46.087 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:45 smithi117 systemd[1]: Stopping Ceph osd.11 for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:52:46.485 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:46 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:46.086+0000 7f4a61543700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:52:46.485 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:46 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:46.086+0000 7f4a61543700 -1 osd.11 101 *** Got signal Terminated *** 2024-03-24T10:52:46.486 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:46 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:46.086+0000 7f4a61543700 -1 osd.11 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:52:46.486 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:46 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:46.485+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:46.486 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:46 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:46.485+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:46.486 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:46 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:46.485+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:46.486 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:46 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:46.485+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:46.486 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:46 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:46.485+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:46.486 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:46 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:46.485+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:46.487 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:46 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:46.485+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-24T10:52:19.272184+0000 front 2024-03-24T10:52:19.272197+0000 (oldest deadline 2024-03-24T10:52:43.972150+0000) 2024-03-24T10:52:47.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:47 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:47.469+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:47.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:47 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:47.469+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:47.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:47 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:47.469+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:47.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:47 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:47.469+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:47.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:47 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:47.469+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:47.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:47 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:47.469+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:47.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:47 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:47.469+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-24T10:52:19.272184+0000 front 2024-03-24T10:52:19.272197+0000 (oldest deadline 2024-03-24T10:52:43.972150+0000) 2024-03-24T10:52:48.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:48 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:48.431+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:48.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:48 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:48.431+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:48.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:48 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:48.431+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:48.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:48 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:48.431+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:48.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:48 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:48.431+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:48.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:48 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:48.431+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:48.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:48 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:48.431+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-24T10:52:19.272184+0000 front 2024-03-24T10:52:19.272197+0000 (oldest deadline 2024-03-24T10:52:43.972150+0000) 2024-03-24T10:52:49.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:49 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:49.470+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:49.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:49 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:49.470+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:49.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:49 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:49.470+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:49.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:49 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:49.470+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:49.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:49 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:49.470+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:49.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:49 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:49.470+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:49.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:49 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:49.470+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-24T10:52:19.272184+0000 front 2024-03-24T10:52:19.272197+0000 (oldest deadline 2024-03-24T10:52:43.972150+0000) 2024-03-24T10:52:49.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:49 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:49.470+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6820 osd.8 since back 2024-03-24T10:52:23.972740+0000 front 2024-03-24T10:52:23.972768+0000 (oldest deadline 2024-03-24T10:52:48.672712+0000) 2024-03-24T10:52:50.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:50 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:50.461+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-24T10:51:21.364744+0000 front 2024-03-24T10:51:21.364640+0000 (oldest deadline 2024-03-24T10:51:45.464670+0000) 2024-03-24T10:52:50.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:50 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:50.461+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6804 osd.1 since back 2024-03-24T10:51:29.565816+0000 front 2024-03-24T10:51:29.565960+0000 (oldest deadline 2024-03-24T10:51:55.465653+0000) 2024-03-24T10:52:50.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:50 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:50.461+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6804 osd.2 since back 2024-03-24T10:51:36.566644+0000 front 2024-03-24T10:51:36.566667+0000 (oldest deadline 2024-03-24T10:52:02.466793+0000) 2024-03-24T10:52:50.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:50 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:50.461+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-24T10:51:47.767815+0000 front 2024-03-24T10:51:47.767864+0000 (oldest deadline 2024-03-24T10:52:09.467740+0000) 2024-03-24T10:52:50.753 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:50 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:50.461+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6812 osd.4 since back 2024-03-24T10:51:54.570152+0000 front 2024-03-24T10:51:54.570291+0000 (oldest deadline 2024-03-24T10:52:19.870041+0000) 2024-03-24T10:52:50.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:50 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:50.461+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6812 osd.5 since back 2024-03-24T10:51:59.870363+0000 front 2024-03-24T10:51:59.870337+0000 (oldest deadline 2024-03-24T10:52:25.770412+0000) 2024-03-24T10:52:50.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:50 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:50.461+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.116:6820 osd.7 since back 2024-03-24T10:52:19.272184+0000 front 2024-03-24T10:52:19.272197+0000 (oldest deadline 2024-03-24T10:52:43.972150+0000) 2024-03-24T10:52:50.754 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:50 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11[62079]: 2024-03-24T10:52:50.461+0000 7f4a5a330700 -1 osd.11 101 heartbeat_check: no reply from 172.21.15.117:6820 osd.8 since back 2024-03-24T10:52:23.972740+0000 front 2024-03-24T10:52:23.972768+0000 (oldest deadline 2024-03-24T10:52:48.672712+0000) 2024-03-24T10:52:51.804 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:51 smithi117 podman[74116]: 2024-03-24 10:52:51.539615284 +0000 UTC m=+5.541870375 container died 38e3f907c64cee12690b4c382a69271bcd1e3f15a164b3f22386f8b949300564 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T10:52:52.147 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:51 smithi117 podman[74116]: 2024-03-24 10:52:51.887419611 +0000 UTC m=+5.889674709 container cleanup 38e3f907c64cee12690b4c382a69271bcd1e3f15a164b3f22386f8b949300564 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-24T10:52:52.147 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:51 smithi117 bash[74116]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11 2024-03-24T10:52:52.422 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:52 smithi117 podman[74133]: 2024-03-24 10:52:52.147716381 +0000 UTC m=+0.606515360 container remove 38e3f907c64cee12690b4c382a69271bcd1e3f15a164b3f22386f8b949300564 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0) 2024-03-24T10:52:52.708 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:52 smithi117 podman[74223]: 2024-03-24 10:52:52.423433287 +0000 UTC m=+0.086528500 container create b92af4bb588d38d28183a4f8ea0c15f875ea53fa51bc1c3e0f5eb5f38d2d2f76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T10:52:52.709 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:52 smithi117 podman[74223]: 2024-03-24 10:52:52.354330216 +0000 UTC m=+0.017425418 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T10:52:52.709 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:52 smithi117 podman[74223]: 2024-03-24 10:52:52.650767233 +0000 UTC m=+0.313862436 container init b92af4bb588d38d28183a4f8ea0c15f875ea53fa51bc1c3e0f5eb5f38d2d2f76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-24T10:52:52.709 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:52 smithi117 podman[74223]: 2024-03-24 10:52:52.653970732 +0000 UTC m=+0.317065928 container start b92af4bb588d38d28183a4f8ea0c15f875ea53fa51bc1c3e0f5eb5f38d2d2f76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T10:52:52.709 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:52 smithi117 podman[74223]: 2024-03-24 10:52:52.708795679 +0000 UTC m=+0.371890887 container attach b92af4bb588d38d28183a4f8ea0c15f875ea53fa51bc1c3e0f5eb5f38d2d2f76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11-deactivate, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-24T10:52:53.002 INFO:journalctl@ceph.osd.11.smithi117.stdout:Mar 24 10:52:52 smithi117 podman[74223]: 2024-03-24 10:52:52.94576535 +0000 UTC m=+0.608860552 container died b92af4bb588d38d28183a4f8ea0c15f875ea53fa51bc1c3e0f5eb5f38d2d2f76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:52:53.522 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@osd.11.service' 2024-03-24T10:52:53.562 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:52:53.562 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-24T10:52:53.562 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-24T10:52:53.563 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.a 2024-03-24T10:52:53.842 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:52:53 smithi023 systemd[1]: Stopping Ceph mds.a for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:52:54.168 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:52:53 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-a[71711]: 2024-03-24T10:52:53.842+0000 7fb985b1a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:52:54.168 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:52:53 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-a[71711]: 2024-03-24T10:52:53.842+0000 7fb985b1a700 -1 mds.a *** got signal Terminated *** 2024-03-24T10:52:58.168 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:52:57 smithi023 podman[99717]: 2024-03-24 10:52:57.851620961 +0000 UTC m=+4.095955822 container died 21230139e0cf1ca90d02679a06ac64724b8d21ce98b577d048dffc902b23d358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319) 2024-03-24T10:52:58.559 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:52:58 smithi023 podman[99717]: 2024-03-24 10:52:58.309160497 +0000 UTC m=+4.553495374 container cleanup 21230139e0cf1ca90d02679a06ac64724b8d21ce98b577d048dffc902b23d358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T10:52:58.559 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 24 10:52:58 smithi023 bash[99717]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-a 2024-03-24T10:52:58.692 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.a.service' 2024-03-24T10:52:58.728 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:52:58.729 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-24T10:52:58.729 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-24T10:52:58.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.d 2024-03-24T10:52:58.918 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:52:58 smithi023 systemd[1]: Stopping Ceph mds.d for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:52:59.418 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:52:59 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-d[70773]: 2024-03-24T10:52:59.045+0000 7f3ba9d3e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:52:59.418 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:52:59 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-d[70773]: 2024-03-24T10:52:59.045+0000 7f3ba9d3e700 -1 mds.d *** got signal Terminated *** 2024-03-24T10:53:03.329 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:53:03 smithi023 podman[99838]: 2024-03-24 10:53:03.053609777 +0000 UTC m=+4.105666030 container died 8b05d03599b867c52813141946a0912575e20904ba4443d2c9f8285ba42d1be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:53:03.668 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:53:03 smithi023 podman[99838]: 2024-03-24 10:53:03.429251137 +0000 UTC m=+4.481307402 container cleanup 8b05d03599b867c52813141946a0912575e20904ba4443d2c9f8285ba42d1be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-d, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T10:53:03.668 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 24 10:53:03 smithi023 bash[99838]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-d 2024-03-24T10:53:03.808 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.d.service' 2024-03-24T10:53:03.843 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:53:03.843 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-24T10:53:03.843 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-24T10:53:03.843 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.g 2024-03-24T10:53:04.149 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:53:03 smithi023 systemd[1]: Stopping Ceph mds.g for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:53:04.417 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:53:04 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-g[71333]: 2024-03-24T10:53:04.149+0000 7fdc55035700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:53:04.418 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:53:04 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-g[71333]: 2024-03-24T10:53:04.149+0000 7fdc55035700 -1 mds.g *** got signal Terminated *** 2024-03-24T10:53:07.624 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:53:07 smithi023 podman[99958]: 2024-03-24 10:53:07.360270422 +0000 UTC m=+3.300488305 container died 399d431e658793bea2efb19c8f2a47880a3fde663f484d1c54d831edfc952d70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-g, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:53:07.918 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:53:07 smithi023 podman[99958]: 2024-03-24 10:53:07.716188942 +0000 UTC m=+3.656406825 container cleanup 399d431e658793bea2efb19c8f2a47880a3fde663f484d1c54d831edfc952d70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T10:53:07.918 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 24 10:53:07 smithi023 bash[99958]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-g 2024-03-24T10:53:08.305 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.g.service' 2024-03-24T10:53:08.346 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:53:08.346 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-24T10:53:08.346 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-24T10:53:08.346 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.j 2024-03-24T10:53:08.668 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:53:08 smithi023 systemd[1]: Stopping Ceph mds.j for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:53:09.168 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:53:08 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-j[70387]: 2024-03-24T10:53:08.677+0000 7fa90fcbd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:53:09.168 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:53:08 smithi023 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-j[70387]: 2024-03-24T10:53:08.677+0000 7fa90fcbd700 -1 mds.j *** got signal Terminated *** 2024-03-24T10:53:12.793 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:53:12 smithi023 podman[100082]: 2024-03-24 10:53:12.44931496 +0000 UTC m=+3.870680021 container died e83389afb056a2bc5dd5b5a169c8cd035a2cc4393eb555d0f627951fef1c611e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-j, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:53:13.168 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:53:12 smithi023 podman[100082]: 2024-03-24 10:53:12.869493134 +0000 UTC m=+4.290858194 container cleanup e83389afb056a2bc5dd5b5a169c8cd035a2cc4393eb555d0f627951fef1c611e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T10:53:13.168 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:53:12 smithi023 bash[100082]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-j 2024-03-24T10:53:13.437 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 24 10:53:13 smithi023 podman[100096]: 2024-03-24 10:53:13.187196008 +0000 UTC m=+0.733114263 container remove e83389afb056a2bc5dd5b5a169c8cd035a2cc4393eb555d0f627951fef1c611e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-j, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T10:53:13.439 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.j.service' 2024-03-24T10:53:13.473 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:53:13.473 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-24T10:53:13.473 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-24T10:53:13.473 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.b 2024-03-24T10:53:13.767 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:53:13 smithi116 systemd[1]: Stopping Ceph mds.b for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:53:14.031 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:53:13 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-b[66641]: 2024-03-24T10:53:13.864+0000 7f8c6cf37700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:53:14.031 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:53:13 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-b[66641]: 2024-03-24T10:53:13.864+0000 7f8c6cf37700 -1 mds.b *** got signal Terminated *** 2024-03-24T10:53:17.456 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:53:17 smithi116 podman[73457]: 2024-03-24 10:53:17.133444777 +0000 UTC m=+3.469010506 container died db10c050566fe99171a8d9c38999003b0d436e755c92ac2ceb33548b99f97f0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:53:17.731 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:53:17 smithi116 podman[73457]: 2024-03-24 10:53:17.547160674 +0000 UTC m=+3.882726391 container cleanup db10c050566fe99171a8d9c38999003b0d436e755c92ac2ceb33548b99f97f0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-b, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:53:17.731 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:53:17 smithi116 bash[73457]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-b 2024-03-24T10:53:17.982 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:53:17 smithi116 podman[73484]: 2024-03-24 10:53:17.731519077 +0000 UTC m=+0.591706766 container remove db10c050566fe99171a8d9c38999003b0d436e755c92ac2ceb33548b99f97f0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T10:53:17.983 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:53:17 smithi116 systemd[1]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.b.service: Deactivated successfully. 2024-03-24T10:53:17.983 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:53:17 smithi116 systemd[1]: Stopped Ceph mds.b for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715. 2024-03-24T10:53:17.983 INFO:journalctl@ceph.mds.b.smithi116.stdout:Mar 24 10:53:17 smithi116 systemd[1]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.b.service: Consumed 4min 57.269s CPU time. 2024-03-24T10:53:17.984 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.b.service' 2024-03-24T10:53:18.021 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:53:18.021 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-24T10:53:18.022 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-24T10:53:18.022 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.e 2024-03-24T10:53:18.267 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:53:18 smithi116 systemd[1]: Stopping Ceph mds.e for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:53:18.767 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:53:18 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-e[66007]: 2024-03-24T10:53:18.358+0000 7ff436e5c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:53:18.767 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:53:18 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-e[66007]: 2024-03-24T10:53:18.358+0000 7ff436e5c700 -1 mds.e *** got signal Terminated *** 2024-03-24T10:53:21.418 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:53:21 smithi116 podman[73590]: 2024-03-24 10:53:21.12635065 +0000 UTC m=+2.888502499 container died dd163f521ebde1d39b93fa4749fb1db1d48bf701b1dc32a596aa9eda90b04ec2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:53:21.751 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:53:21 smithi116 podman[73590]: 2024-03-24 10:53:21.509521516 +0000 UTC m=+3.271673353 container cleanup dd163f521ebde1d39b93fa4749fb1db1d48bf701b1dc32a596aa9eda90b04ec2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-e, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:53:21.751 INFO:journalctl@ceph.mds.e.smithi116.stdout:Mar 24 10:53:21 smithi116 bash[73590]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-e 2024-03-24T10:53:21.881 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.e.service' 2024-03-24T10:53:21.916 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:53:21.917 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-24T10:53:21.917 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-24T10:53:21.917 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.h 2024-03-24T10:53:22.017 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:53:21 smithi116 systemd[1]: Stopping Ceph mds.h for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:53:22.517 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:53:22 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-h[66318]: 2024-03-24T10:53:22.229+0000 7f056f613700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:53:22.517 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:53:22 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-h[66318]: 2024-03-24T10:53:22.229+0000 7f056f613700 -1 mds.h *** got signal Terminated *** 2024-03-24T10:53:25.446 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:53:25 smithi116 podman[73710]: 2024-03-24 10:53:25.139926513 +0000 UTC m=+3.008524059 container died f886966040763359b713dd539b517e75eed6d5da42c730a8dffaca2c2b35a4bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T10:53:25.729 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:53:25 smithi116 podman[73710]: 2024-03-24 10:53:25.53781515 +0000 UTC m=+3.406412693 container cleanup f886966040763359b713dd539b517e75eed6d5da42c730a8dffaca2c2b35a4bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T10:53:25.730 INFO:journalctl@ceph.mds.h.smithi116.stdout:Mar 24 10:53:25 smithi116 bash[73710]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-h 2024-03-24T10:53:25.858 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.h.service' 2024-03-24T10:53:25.893 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:53:25.893 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-24T10:53:25.893 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-24T10:53:25.893 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.k 2024-03-24T10:53:26.017 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:53:25 smithi116 systemd[1]: Stopping Ceph mds.k for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:53:26.517 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:53:26 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-k[65694]: 2024-03-24T10:53:26.207+0000 7f57fa048700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:53:26.518 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:53:26 smithi116 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-k[65694]: 2024-03-24T10:53:26.207+0000 7f57fa048700 -1 mds.k *** got signal Terminated *** 2024-03-24T10:53:30.091 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:53:29 smithi116 podman[73831]: 2024-03-24 10:53:29.816333476 +0000 UTC m=+3.707646246 container died 44f26d052fc5d85f41c0a73f9b89e63b182eca2c0452bcc6368439b3b9e54510 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-k, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T10:53:30.424 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:53:30 smithi116 podman[73831]: 2024-03-24 10:53:30.182937105 +0000 UTC m=+4.074249880 container cleanup 44f26d052fc5d85f41c0a73f9b89e63b182eca2c0452bcc6368439b3b9e54510 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-k, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T10:53:30.425 INFO:journalctl@ceph.mds.k.smithi116.stdout:Mar 24 10:53:30 smithi116 bash[73831]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-k 2024-03-24T10:53:30.560 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.k.service' 2024-03-24T10:53:30.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:53:30.593 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-24T10:53:30.593 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-24T10:53:30.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.c 2024-03-24T10:53:30.891 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:53:30 smithi117 systemd[1]: Stopping Ceph mds.c for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:53:31.252 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:53:30 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71561]: 2024-03-24T10:53:30.890+0000 7f1b1fcf8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:53:31.253 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:53:30 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c[71561]: 2024-03-24T10:53:30.890+0000 7f1b1fcf8700 -1 mds.c *** got signal Terminated *** 2024-03-24T10:53:33.373 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:53:33 smithi117 podman[74329]: 2024-03-24 10:53:33.121442035 +0000 UTC m=+2.319198163 container died 48528e1915dac0eed881f9ac4c2ead61f4c323d23c458a92affd0de6d41ad4ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-24T10:53:33.752 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:53:33 smithi117 podman[74329]: 2024-03-24 10:53:33.440248368 +0000 UTC m=+2.638004468 container cleanup 48528e1915dac0eed881f9ac4c2ead61f4c323d23c458a92affd0de6d41ad4ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:53:33.753 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:53:33 smithi117 bash[74329]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c 2024-03-24T10:53:33.753 INFO:journalctl@ceph.mds.c.smithi117.stdout:Mar 24 10:53:33 smithi117 podman[74354]: 2024-03-24 10:53:33.607974821 +0000 UTC m=+0.481659449 container remove 48528e1915dac0eed881f9ac4c2ead61f4c323d23c458a92affd0de6d41ad4ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-c, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T10:53:33.826 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.c.service' 2024-03-24T10:53:33.864 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:53:33.925 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-24T10:53:33.925 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-24T10:53:33.925 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.f 2024-03-24T10:53:34.193 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:53:33 smithi117 systemd[1]: Stopping Ceph mds.f for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:53:34.503 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:53:34 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-f[66812]: 2024-03-24T10:53:34.193+0000 7fc6979b7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:53:34.503 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:53:34 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-f[66812]: 2024-03-24T10:53:34.193+0000 7fc6979b7700 -1 mds.f *** got signal Terminated *** 2024-03-24T10:53:37.403 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:53:37 smithi117 podman[74459]: 2024-03-24 10:53:37.124156312 +0000 UTC m=+3.020514621 container died aac3988c0274261a20017ab628ba77e3de3844bd1bb684f5431f15e16a99dff8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-f, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD) 2024-03-24T10:53:37.753 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:53:37 smithi117 podman[74459]: 2024-03-24 10:53:37.486153666 +0000 UTC m=+3.382511957 container cleanup aac3988c0274261a20017ab628ba77e3de3844bd1bb684f5431f15e16a99dff8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-f, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T10:53:37.753 INFO:journalctl@ceph.mds.f.smithi117.stdout:Mar 24 10:53:37 smithi117 bash[74459]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-f 2024-03-24T10:53:37.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.f.service' 2024-03-24T10:53:37.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:53:37.934 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-24T10:53:37.934 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-24T10:53:37.934 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.i 2024-03-24T10:53:38.503 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:53:38 smithi117 systemd[1]: Stopping Ceph mds.i for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:53:38.503 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:53:38 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-i[67442]: 2024-03-24T10:53:38.247+0000 7f26f5581700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:53:38.503 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:53:38 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-i[67442]: 2024-03-24T10:53:38.247+0000 7f26f5581700 -1 mds.i *** got signal Terminated *** 2024-03-24T10:53:41.396 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:53:41 smithi117 podman[74581]: 2024-03-24 10:53:41.127113747 +0000 UTC m=+2.976558736 container died 467f37d8aace28a632957e863c6e78cee9d21e3ffeda81f33e3907eca4aa4743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-i, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T10:53:41.735 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:53:41 smithi117 podman[74581]: 2024-03-24 10:53:41.3974995 +0000 UTC m=+3.246944493 container cleanup 467f37d8aace28a632957e863c6e78cee9d21e3ffeda81f33e3907eca4aa4743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-i, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:53:41.735 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:53:41 smithi117 bash[74581]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-i 2024-03-24T10:53:41.735 INFO:journalctl@ceph.mds.i.smithi117.stdout:Mar 24 10:53:41 smithi117 podman[74595]: 2024-03-24 10:53:41.632146776 +0000 UTC m=+0.507475272 container remove 467f37d8aace28a632957e863c6e78cee9d21e3ffeda81f33e3907eca4aa4743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T10:53:41.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.i.service' 2024-03-24T10:53:41.787 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:53:41.788 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-24T10:53:41.788 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-24T10:53:41.788 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.l 2024-03-24T10:53:42.002 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:53:41 smithi117 systemd[1]: Stopping Ceph mds.l for 4f8fb8f6-e9c9-11ee-95cd-87774f69a715... 2024-03-24T10:53:42.502 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:53:42 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-l[66496]: 2024-03-24T10:53:42.117+0000 7f4716183700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:53:42.503 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:53:42 smithi117 ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-l[66496]: 2024-03-24T10:53:42.117+0000 7f4716183700 -1 mds.l *** got signal Terminated *** 2024-03-24T10:53:43.443 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:53:43 smithi117 podman[74699]: 2024-03-24 10:53:43.149612439 +0000 UTC m=+1.134899925 container died f2de3d30da4abe3e1d0d157068914c39a1cb860389b661bc9cfacef1c733c8c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-l, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T10:53:43.752 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:53:43 smithi117 podman[74699]: 2024-03-24 10:53:43.526923628 +0000 UTC m=+1.512211108 container cleanup f2de3d30da4abe3e1d0d157068914c39a1cb860389b661bc9cfacef1c733c8c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-l, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:53:43.753 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:53:43 smithi117 bash[74699]: ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-l 2024-03-24T10:53:43.753 INFO:journalctl@ceph.mds.l.smithi117.stdout:Mar 24 10:53:43 smithi117 podman[74713]: 2024-03-24 10:53:43.686202429 +0000 UTC m=+0.536825108 container remove f2de3d30da4abe3e1d0d157068914c39a1cb860389b661bc9cfacef1c733c8c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715-mds-l, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T10:53:43.891 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4f8fb8f6-e9c9-11ee-95cd-87774f69a715@mds.l.service' 2024-03-24T10:53:43.925 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:53:43.925 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-24T10:53:43.925 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T10:53:44.122 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:53:46.484 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T10:53:46.674 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:53:49.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T10:53:49.540 INFO:teuthology.orchestra.run.smithi117.stdout:Deleting cluster with fsid: 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:53:51.966 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T10:53:52.001 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T10:53:52.035 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T10:53:52.063 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T10:53:52.065 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048/remote/smithi023/crash 2024-03-24T10:53:52.066 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/crash -- . 2024-03-24T10:53:52.103 INFO:teuthology.orchestra.run.smithi023.stderr:tar: /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T10:53:52.103 INFO:teuthology.orchestra.run.smithi023.stderr:tar: Error is not recoverable: exiting now 2024-03-24T10:53:52.105 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048/remote/smithi116/crash 2024-03-24T10:53:52.106 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/crash -- . 2024-03-24T10:53:52.148 INFO:teuthology.orchestra.run.smithi116.stderr:tar: /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T10:53:52.148 INFO:teuthology.orchestra.run.smithi116.stderr:tar: Error is not recoverable: exiting now 2024-03-24T10:53:52.150 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048/remote/smithi117/crash 2024-03-24T10:53:52.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/crash -- . 2024-03-24T10:53:52.183 INFO:teuthology.orchestra.run.smithi117.stderr:tar: /var/lib/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T10:53:52.183 INFO:teuthology.orchestra.run.smithi117.stderr:tar: Error is not recoverable: exiting now 2024-03-24T10:53:52.184 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T10:53:52.184 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-24T10:53:52.219 INFO:tasks.cephadm:Compressing logs... 2024-03-24T10:53:52.219 DEBUG:teuthology.orchestra.run.smithi023:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T10:53:52.262 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T10:53:52.264 DEBUG:teuthology.orchestra.run.smithi117:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T10:53:52.287 INFO:teuthology.orchestra.run.smithi023.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T10:53:52.288 INFO:teuthology.orchestra.run.smithi023.stderr:: No such file or directory 2024-03-24T10:53:52.288 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-24T10:53:52.290 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-24T10:53:52.290 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T10:53:52.290 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.g.log 2024-03-24T10:53:52.290 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.j.log 2024-03-24T10:53:52.290 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-24T10:53:52.304 INFO:teuthology.orchestra.run.smithi116.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T10:53:52.304 INFO:teuthology.orchestra.run.smithi116.stderr:: No such file or directory 2024-03-24T10:53:52.304 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.10.log 2024-03-24T10:53:52.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T10:53:52.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.b.log 2024-03-24T10:53:52.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-24T10:53:52.305 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.b.log: -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T10:53:52.305 INFO:teuthology.orchestra.run.smithi116.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T10:53:52.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.k.log 2024-03-24T10:53:52.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.h.log 2024-03-24T10:53:52.305 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-24T10:53:52.306 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.e.log 2024-03-24T10:53:52.306 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.7.log: 90.2% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T10:53:52.306 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-24T10:53:52.306 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.e.log: 94.5% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.k.log.gz 2024-03-24T10:53:52.306 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T10:53:52.312 INFO:teuthology.orchestra.run.smithi117.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T10:53:52.312 INFO:teuthology.orchestra.run.smithi117.stderr:: No such file or directory 2024-03-24T10:53:52.312 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.l.log 2024-03-24T10:53:52.312 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T10:53:52.312 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.i.log 2024-03-24T10:53:52.312 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.l.log: /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.11.log 2024-03-24T10:53:52.313 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mgr.z.log 2024-03-24T10:53:52.313 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.8.log 2024-03-24T10:53:52.313 INFO:teuthology.orchestra.run.smithi117.stderr: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T10:53:52.313 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-24T10:53:52.313 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.f.log 2024-03-24T10:53:52.313 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T10:53:52.314 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.f.log: 90.2% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T10:53:52.314 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.log 2024-03-24T10:53:52.314 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.cephadm.log: 83.8% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T10:53:52.314 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T10:53:52.314 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.log: 94.6% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.l.log.gz 2024-03-24T10:53:52.314 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.c.log 2024-03-24T10:53:52.315 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.2.log: 86.8% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T10:53:52.315 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T10:53:52.315 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.c.log: 91.9% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mgr.z.log.gz 2024-03-24T10:53:52.315 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-24T10:53:52.317 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-volume.log: 91.0% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T10:53:52.319 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.j.log: /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.3.log: 90.0% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T10:53:52.319 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-24T10:53:52.319 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T10:53:52.319 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T10:53:52.319 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.0.log: 94.6% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.g.log.gz 2024-03-24T10:53:52.320 INFO:teuthology.orchestra.run.smithi023.stderr: 94.6% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.j.log.gz 2024-03-24T10:53:52.320 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.log 2024-03-24T10:53:52.320 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T10:53:52.320 INFO:teuthology.orchestra.run.smithi023.stderr: 93.4% 84.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T10:53:52.320 INFO:teuthology.orchestra.run.smithi023.stderr: -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.cephadm.log.gz/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.log: 2024-03-24T10:53:52.320 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.9.log 2024-03-24T10:53:52.320 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.d.log 2024-03-24T10:53:52.321 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.9.log: 86.8% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T10:53:52.321 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.a.log 2024-03-24T10:53:52.321 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.d.log.gz 2024-03-24T10:53:52.321 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.a.log.gz 2024-03-24T10:53:52.321 INFO:teuthology.orchestra.run.smithi023.stderr: 90.9% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T10:53:52.324 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mgr.y.log: 91.8% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-24T10:53:52.324 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.log 2024-03-24T10:53:52.324 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.cephadm.log: 83.9% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T10:53:52.324 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T10:53:52.324 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.log: 86.8% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T10:53:52.324 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-24T10:53:52.328 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T10:53:52.359 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mon.c.log: 94.5% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.f.log.gz 2024-03-24T10:53:52.509 INFO:teuthology.orchestra.run.smithi117.stderr: 93.7% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.c.log.gz 2024-03-24T10:53:52.663 INFO:teuthology.orchestra.run.smithi023.stderr: 90.1% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-24T10:53:52.762 INFO:teuthology.orchestra.run.smithi116.stderr: 92.0% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-24T10:53:53.158 INFO:teuthology.orchestra.run.smithi023.stderr: 90.5% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-24T10:53:53.221 INFO:teuthology.orchestra.run.smithi117.stderr: 92.8% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-24T10:53:55.107 INFO:teuthology.orchestra.run.smithi117.stderr: 94.1% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.11.log.gz 2024-03-24T10:53:55.196 INFO:teuthology.orchestra.run.smithi116.stderr: 94.3% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-24T10:53:55.326 INFO:teuthology.orchestra.run.smithi117.stderr: 96.4% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.i.log.gz 2024-03-24T10:53:55.466 INFO:teuthology.orchestra.run.smithi023.stderr: 94.5% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.9.log.gz 2024-03-24T10:53:55.576 INFO:teuthology.orchestra.run.smithi023.stderr: 93.8% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T10:53:55.866 INFO:teuthology.orchestra.run.smithi023.stderr: 94.3% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-24T10:53:56.205 INFO:teuthology.orchestra.run.smithi117.stderr: 94.4% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.8.log.gz 2024-03-24T10:53:56.278 INFO:teuthology.orchestra.run.smithi116.stderr: 93.9% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T10:53:56.318 INFO:teuthology.orchestra.run.smithi117.stderr: 94.4% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-24T10:53:56.416 INFO:teuthology.orchestra.run.smithi117.stderr: 94.3% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T10:53:56.417 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-24T10:53:56.418 INFO:teuthology.orchestra.run.smithi117.stderr:real 0m4.140s 2024-03-24T10:53:56.418 INFO:teuthology.orchestra.run.smithi117.stderr:user 0m18.450s 2024-03-24T10:53:56.418 INFO:teuthology.orchestra.run.smithi117.stderr:sys 0m0.660s 2024-03-24T10:53:56.673 INFO:teuthology.orchestra.run.smithi023.stderr: 94.3% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-24T10:53:56.675 INFO:teuthology.orchestra.run.smithi023.stderr: 2024-03-24T10:53:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:real 0m4.400s 2024-03-24T10:53:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:user 0m15.160s 2024-03-24T10:53:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:sys 0m0.546s 2024-03-24T10:53:56.917 INFO:teuthology.orchestra.run.smithi116.stderr: 94.2% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.10.log.gz 2024-03-24T10:53:57.380 INFO:teuthology.orchestra.run.smithi116.stderr: 94.2% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-24T10:53:59.190 INFO:teuthology.orchestra.run.smithi116.stderr: 95.8% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.b.log.gz 2024-03-24T10:53:59.788 INFO:teuthology.orchestra.run.smithi116.stderr: 95.5% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.e.log.gz 2024-03-24T10:56:12.059 INFO:teuthology.orchestra.run.smithi116.stderr: 93.7% -- replaced with /var/log/ceph/4f8fb8f6-e9c9-11ee-95cd-87774f69a715/ceph-mds.h.log.gz 2024-03-24T10:56:12.060 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-24T10:56:12.061 INFO:teuthology.orchestra.run.smithi116.stderr:real 2m19.783s 2024-03-24T10:56:12.061 INFO:teuthology.orchestra.run.smithi116.stderr:user 2m45.615s 2024-03-24T10:56:12.061 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m5.279s 2024-03-24T10:56:12.061 INFO:tasks.cephadm:Archiving logs... 2024-03-24T10:56:12.062 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048/remote/smithi023/log 2024-03-24T10:56:12.063 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T10:56:14.220 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048/remote/smithi116/log 2024-03-24T10:56:14.220 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T10:56:37.475 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048/remote/smithi117/log 2024-03-24T10:56:37.477 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T10:56:39.731 INFO:tasks.cephadm:Removing cluster... 2024-03-24T10:56:39.731 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 --force 2024-03-24T10:56:39.930 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:56:40.196 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 --force 2024-03-24T10:56:40.388 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:56:40.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 --force 2024-03-24T10:56:41.023 INFO:teuthology.orchestra.run.smithi117.stdout:Deleting cluster with fsid: 4f8fb8f6-e9c9-11ee-95cd-87774f69a715 2024-03-24T10:56:41.376 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T10:56:41.376 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T10:56:41.395 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T10:56:41.410 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T10:56:41.426 INFO:tasks.cephadm:Teardown complete 2024-03-24T10:56:41.426 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T10:56:41.437 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T10:56:41.437 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T10:56:41.440 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T10:56:41.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T10:56:41.554 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T10:56:41.565 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T10:56:41.565 DEBUG:teuthology.orchestra.run.smithi023:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:56:41.568 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:56:41.570 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:56:41.582 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:56:41.583 INFO:teuthology.orchestra.run.smithi023.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:56:41.584 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:56:41.586 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:56:41.586 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-24T10:56:41.587 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +567us[ +579us] +/- 91ms 2024-03-24T10:56:41.587 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -257us[ -245us] +/- 54ms 2024-03-24T10:56:41.587 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 -53us[ -41us] +/- 61ms 2024-03-24T10:56:41.587 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:56:41.588 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:56:41.589 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-03-24T10:56:41.589 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +600us[ +621us] +/- 91ms 2024-03-24T10:56:41.589 INFO:teuthology.orchestra.run.smithi117.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -244us[ -224us] +/- 54ms 2024-03-24T10:56:41.589 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 -49us[ -49us] +/- 61ms 2024-03-24T10:56:41.589 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:56:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:56:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2024-03-24T10:56:41.589 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +580us[ +595us] +/- 91ms 2024-03-24T10:56:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -254us[ -239us] +/- 54ms 2024-03-24T10:56:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -65us[ -50us] +/- 61ms 2024-03-24T10:56:41.590 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:56:41.590 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T10:56:41.601 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T10:56:41.602 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T10:56:41.613 DEBUG:teuthology.orchestra.run.smithi023:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T10:56:41.719 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T10:56:41.790 DEBUG:teuthology.orchestra.run.smithi117:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T10:56:41.862 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T10:56:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:56:41.895 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T10:56:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:56:41.926 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T10:56:41.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:56:41.956 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T10:56:41.968 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T10:56:42.033 INFO:teuthology.task.internal:Duration was 2058.864415 seconds 2024-03-24T10:56:42.033 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T10:56:42.045 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T10:56:42.046 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T10:56:42.048 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T10:56:42.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T10:56:42.091 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:56:42.092 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:56:42.093 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:56:42.549 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T10:56:42.549 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi023.front.sepia.ceph.com 2024-03-24T10:56:42.550 DEBUG:teuthology.orchestra.run.smithi023:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T10:56:42.572 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T10:56:42.572 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T10:56:42.613 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-03-24T10:56:42.614 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T10:56:42.641 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T10:56:42.641 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T10:56:42.643 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T10:56:42.656 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T10:56:42.709 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T10:56:42.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T10:56:42.715 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T10:56:42.724 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T10:56:43.096 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T10:56:43.113 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T10:56:43.113 DEBUG:teuthology.orchestra.run.smithi023:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T10:56:43.140 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T10:56:43.167 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T10:56:43.194 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T10:56:43.206 DEBUG:teuthology.orchestra.run.smithi023:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T10:56:43.209 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T10:56:43.211 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T10:56:43.232 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = core 2024-03-24T10:56:43.234 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-03-24T10:56:43.260 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-03-24T10:56:43.282 DEBUG:teuthology.orchestra.run.smithi023:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T10:56:43.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:56:43.330 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T10:56:43.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:56:43.344 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T10:56:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:56:43.358 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T10:56:43.372 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T10:56:43.373 DEBUG:teuthology.misc:Transferring archived files from smithi023:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048/remote/smithi023 2024-03-24T10:56:43.373 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T10:56:43.416 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048/remote/smithi116 2024-03-24T10:56:43.416 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T10:56:43.454 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048/remote/smithi117 2024-03-24T10:56:43.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T10:56:43.493 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T10:56:43.493 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T10:56:43.495 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T10:56:43.497 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T10:56:43.541 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T10:56:43.556 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T10:56:43.556 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T10:56:43.567 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T10:56:43.567 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T10:56:43.570 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T10:56:43.572 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T10:56:43.581 INFO:teuthology.orchestra.run.smithi023.stdout: 265306 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 10:56 /home/ubuntu/cephtest 2024-03-24T10:56:43.584 INFO:teuthology.orchestra.run.smithi116.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 10:56 /home/ubuntu/cephtest 2024-03-24T10:56:43.596 INFO:teuthology.orchestra.run.smithi117.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 10:56 /home/ubuntu/cephtest 2024-03-24T10:56:43.597 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T10:56:43.609 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T10:56:43.634 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T10:56:43.661 DEBUG:teuthology.nuke:shortname: smithi023 2024-03-24T10:56:43.661 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T10:56:43.685 DEBUG:teuthology.nuke:shortname: smithi116 2024-03-24T10:56:43.685 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T10:56:43.690 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:12:26.468685', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFz//K9/uxvwtd/dn2kIuJgguQItPWZvjS/bTfHZLeUy8O0lrTfHSbPY96xhGanB6CHhZvJhDHfwl2LzBT84ZwM='} 2024-03-24T10:56:43.710 DEBUG:teuthology.nuke:shortname: smithi117 2024-03-24T10:56:43.710 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T10:56:43.715 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:12:26.471945', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMVMb3emcxPjzlzZK0ROwX1kMluUy7TScFma5MqLnZq4clQHyRD9Bs1LHx7vkYOSezWsaadnMz5l3egWLF/kA8w='} 2024-03-24T10:56:43.743 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619048', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:12:26.470526', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN9q4Z7n2upgxsNmEi6Qmkm6Ii9ePAVs2Lqax+P+SdVfRXfKwUHf4Zj99lMAhPCbVZPu5aiDeWLM7VkW4PiOIco='} 2024-03-24T10:56:43.750 INFO:teuthology.orchestra.console.smithi023:Power off 2024-03-24T10:56:43.750 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T10:56:43.772 DEBUG:teuthology.orchestra.console.smithi023:power off output: Chassis Power Control: Down/Off 2024-03-24T10:56:43.773 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:56:43.786 INFO:teuthology.orchestra.console.smithi116:Power off 2024-03-24T10:56:43.786 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T10:56:43.800 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-24T10:56:43.809 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2024-03-24T10:56:43.809 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:56:43.828 INFO:teuthology.orchestra.console.smithi117:Power off 2024-03-24T10:56:43.828 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T10:56:43.837 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-24T10:56:43.850 DEBUG:teuthology.orchestra.console.smithi117:power off output: Chassis Power Control: Down/Off 2024-03-24T10:56:43.851 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:56:43.874 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-24T10:56:47.801 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:56:47.839 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:56:47.875 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:56:47.913 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-24T10:56:47.949 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-24T10:56:47.984 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-24T10:56:51.918 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:56:51.950 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:56:51.985 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:56:52.029 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is off 2024-03-24T10:56:52.060 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-24T10:56:52.095 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-24T10:56:52.131 INFO:teuthology.orchestra.console.smithi023:Power off completed 2024-03-24T10:56:56.062 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:56:56.096 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:56:56.173 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2024-03-24T10:56:56.207 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is off 2024-03-24T10:56:56.275 INFO:teuthology.orchestra.console.smithi116:Power off completed 2024-03-24T10:56:56.309 INFO:teuthology.orchestra.console.smithi117:Power off completed 2024-03-24T10:56:56.416 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} duration: 2058.864414691925 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi023 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=887abaa611ed40c8b7bd91097dff5e36 status: fail success: false 2024-03-24T10:56:56.417 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T10:56:56.510 INFO:teuthology.run:FAIL